Balanced within subject design | XM Community
Skip to main content

I’m trying to design a balanced within-subjects design using embedded data, but I’m struggling to make my design work. 

 

Here’s a description of my study:

  1. IV1: Label type scenarios (Three labels: green, red, blue ) 
  2. IV2: Target type scenarios (Nine targets: A, B, C, D, E, F, G, H, I)
  3. Each target is paired with each label: so i have Target red A, blue A, green A.

Here’s a description of what I want my design to look like: 

  1. each participant should see all 9 targets
  2. each participant should see each target once only (e.g., the target is a square. if they see the red square, they shouldn’t see the green or blue square)
  3. each participant should see three “green label”, three “red label”, and three “blue label” scenarios. for example: a participants might see the scenarios in this order: green A, blue B, red C, green D,  blue E, red F, blue G, green H, red I. In this example, there are  3 greens, 3 blues, 3 reds, but each letter appears once.

my current embedded data looks something like this:

red_cond=0

If red_cond < 3, RED_eligibility = 1 

dsame for green and blue]

 

RANDOMIZER FOR ALL 9 TARGETS: 

GROUP TARGET A (but same method used for all groups):

RANDOMIZER:

If red Is Equal to 1:

SHOW TargetA_RED block 

Red_cond  = ${e://Field/Red_cond + 1}

If GREEN Is Equal to 1:

SHOW TargetA_GREEN block 

GREEN_cond  = ${e://Field/GREEN_cond + 1}

>SAME FOR BLUE]

 

The current logic constrains the targets correctly: I see each target only once. However, it doesn’t constrain the colors correctly. so i’m seeing something like (the order of the colors is random; it’s just that im not seeing three of each color):

Red A

Red B

Red C

Red D

Green E

Blue F

Green G

Blue H

Green I

 

Can you please explain how I can modify my survey flow to fix this?

The issue here lies in the fact that while you're correctly ensuring each target (A–I) is shown only once, you're not explicitly controlling the number of times each label type (red, green, blue) is used across the 9 targets. This leads to the imbalance you're observing—where you might get, say, 4 reds and only 2 blues.

To fix this, you'll need to shift your logic from per-target group randomization to a more global label-quota constrained randomization.

Here's how to do it:

Step 1: Predefine the 9 target–label combinations

You need to construct a list of 9 pairs where each of the 9 targets appears exactly once, and each label is used exactly 3 times. For example:

Shuffle this full list before assigning blocks to participants, ensuring you always get 3 per color and 1 per target.

Since Qualtrics doesn't have a native way to enforce this logic dynamically across multiple randomizers, you have to preconstruct this balance either:

Externally (e.g., using R, Python, or Excel) to generate a fixed list of 9 combinations, and then upload those as embedded data into Qualtrics, or
Manually create 12–18 such sets and randomly assign participants to one of those sets at the start of the survey using a randomizer.

Step 2: Use a custom embedded data variable (e.g., ConditionSet)

In Survey Flow, add a branch that assigns one of the balanced sets like:
ConditionSet = Set1 → Green A, Red B, Blue C, ...
ConditionSet = Set2 → ...

Each Set corresponds to a unique combination of 9 (target, label) pairs with balanced label distribution and unique targets.

 

Step 3: Use branching logic to show blocks

Now, use embedded data logic to show the correct blocks. For instance:
IF ConditionSet = Set1 AND Target = A → Show TargetA_GREEN
IF ConditionSet = Set1 AND Target = B → Show TargetB_RED
...
Yes, it's manual—but it's precise and guaranteed to enforce balance.

Leave a Reply


Terms & Conditions