I am having trouble understanding the "evenly present elements" option.
I am designing a survey with 3 conditions and I'd like to randomize the participants into those conditions. Currently I have set up a randomizer with 3 embedded data elements which assigns one of three values (1,2 or 3) to the variable "CONDITION". I set the number of elements presented to a participant to 1.
So far so good. But should I check the "evenly present elements" option? If what condition one is assigned to is randomized, i.e. there is a 33.3% chance in my case, then in the end there will be roughly the same amount of participants in each condition, without having to execute any feature that "evenly presents" the options, right? If I throw a coin 100 times, it's gonna be about 50 times heads and 50 times tails.
So what is this feature good for?
Isn't "randomly present elements" unnecessary for high enough numbers of participants?
Best answer by TomG
https://www.qualtrics.com/community/discussion/comment/31170#Comment_31170Yes, you should use it. It will ensure that respondents are randomly assigned to your three conditions evenly instead of just randomly. It will reduce the variability in how many respondents are assigned to each condition. It reduces the need to over sample and makes statistical analysis better since you are less likely to run into issues of small sample sizes if you cut the data by other criteria.
It does not guarantee that you will have equal numbers of completes because people could abandon the survey after being assigned but before completing, but it improves the odds.
On average, yes. But it might come up heads 98 times. This feature helps guard against the unlikely, which becomes less important as the number of responses goes up.
Is using this feature still appropriate for randomizing participants in a controlled experiment though? I don't quite understand it, and I'm unsure whether I should use it or not.
+27https://www.qualtrics.com/community/discussion/comment/31170#Comment_31170Yes, you should use it. It will ensure that respondents are randomly assigned to your three conditions evenly instead of just randomly. It will reduce the variability in how many respondents are assigned to each condition. It reduces the need to over sample and makes statistical analysis better since you are less likely to run into issues of small sample sizes if you cut the data by other criteria.
It does not guarantee that you will have equal numbers of completes because people could abandon the survey after being assigned but before completing, but it improves the odds.
https://www.qualtrics.com/community/discussion/comment/31171#Comment_31171Great, thank you for your insights. I will use it then.
https://www.qualtrics.com/community/discussion/comment/31171#Comment_31171Thank you!
I'm still not fully convinced about what it means for the scientific rigor of an experiment.
Is there also a more profound discussion of this somewhere? E.g. how does the evening out work in qualtrics? What assumptions do I have to make to argue this is not causing any biases?
I doubt you'll find a more detailed explanation of these anywhere. Qualtrics isn't aimed at scientific studies, its more for business purposes.
From my small testing of the randomizer (3-4 experiments, 10-12 draws, 2-4 elements in the randomizer), here's the difference that I noticed.
- Without evenly present: Qualtrics does a random draw and presents one element to the respondent.
- With evenly present:
- Qualtrics checks the number of times each element has been presented, shows the one which has been presented least
- If more than one element has been presented the same number of times (lowest), it does a random draw to present one
- It does this till all elements have been presented an equal number of times
But as for the main question, does it have an impact for a large enough sample, I don't think it should make a difference.
+27https://www.qualtrics.com/community/discussion/comment/34518#Comment_34518It works by keeping count of how many times it picks each item in the randomization. I can't speak specifically to the Qualtrics algorithm, but typically it is done by first shuffling the set of items, then sorting them by count. Once that's done, as many items as are needed are picked from the top of the list.
Leave a Reply
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.