Hi everyone,

Running into very strange behavior with the Qualtrics randomizer that presents elements evenly, and I’m wondering if anyone could shed light on what might be going awry in the backend.

Basic context: We have coded up an attention-based work task in a Qualtrics survey, where participants evaluate a series of images & we measure performance. Respondents should be randomized into 2 different work conditions.

Our problem: We are in theory randomizing respondents into 1 of 2 conditions (control vs treatment). This is happening in the correct ratio but there is very strong evidence this is happening non-randomly by performance (high-performing respondents are more likely to be in control).

Current working hypothesis: Our current guess is that the non-random sorting is happening based on timing. Because of how much overhead our survey has (see details below), “lags” differ across respondents based on their tech set-up/internet/etc. We think faster (higher performing) respondents are being non-randomly sorted into one condition.

Survey flow: We are using “Randomly Present Elements Evenly” to set an embedded variable (treat_cond = 0 or 1) that then determines the survey flow for respondents. Everyone does the same task for 2 blocks before randomization so performance should be balanced there (but we see that in these blocks the control group is doing better)

More survey details: The big concern/caveat is that our survey has quite a bit of “overhead” and creates some lag -- since we have some custom Javascript throughout to auto-advance pages, and we use loop & merge to show respondents hundreds of images to respond to as part of our academic study.

Any insight into how the randomizer works & interacts with timing would be helpful and deeply appreciated! Thank you!