I am trying to design a survey where people are assigned questions from one of three pools of questions. For simplicity let's call them P1, P2, and P3 for pools one, two, and three respectively. In the control condition, every time participants are asked a question, that question will be drawn randomly from each pool with equal probability (i.e., 33%). In the experimental condition they will draw from the pools with different probability. I think I more or less have a handle on that part of things.
The way I have approached the problem is that every time I want to ask participants a question I use the survey flow randomizer to call one of my three blocks (P1, P2, P3) then within each block I use the advanced randomization options for questions to select one question from a random set of all questions in the block any time the block is called. The issue I'm having with my current approach is that participants can see the same question more than once. What I would like to have happen instead is for the survey flow randomizer to randomly select one of my pools each time participants are asked a question (this part currently works properly) and then have the question block display one of its questions randomly excluding questions participants have already seen.
How Can I set up randomization without replacement?
Best answer by TomG
https://www.qualtrics.com/community/discussion/comment/29173#Comment_29173You could use JS to do randomization without replacement.
At the beginning of the survey (maybe your intro question). Add JS to randomize all the P questions and save them as comma separated values in embedded data fields (e.g. P1list, P2list, P3list). For example, let's say your P1 questions are p1.1, p1.2, p1.3 and p1.4. Put the values in an array, shuffle the array, then save it to an embedded data field as a comma separated list.
Put all your P questions in separate blocks, then use a randomizer, groups, and branches to randomly show one of the P questions from one of the P groups. You would get the question to show by adding a block the beginning of a P group that gets the first question from the list, sets an embedded variable, updates the embedded data field that contains the randomized list, and clicks the next button. Survey flow would look something like:
Randomizer (1 of 3)
P1 Group
Block: JS to set p1display as next random P1 question
Branch: If p1display = p1.1
Block p1.1
Branch: If p1display = p1.2
Block p1.2
...etc...
P2 Group
...etc...
P3 Group
...etc....
You could duplicate the flow above as many times as needed.
+27You'll need to keep track of which questions in each of the P blocks have been shown. Instead of Advanced Randomization use display logic on each question and set an embedded variable so only one question is shown each time the block is displayed. The easiest way to do this will be to use some code to predetermine a random order for each P block, and grab the next item in the random order before executing the block.
Thanks for the advice Tom. I'm afraid I do not entirely follow though can you elaborate a bit more on how that would actually work?
Here is my understanding of what you are saying as well as the parts I do not know how to implement. I think you are saying I can use the survey flow randomizer to call one of my P blocks at random the way I currently have it set up. The randomizer then selects a block, for example P1. Once P1 has been called it will check the first question in the block to see if it has been displayed before and if it has then it will move on to the next question instead and keep performing this check until it gets to a question that has not been displayed before. When the question is displayed it will then create some embedded data to tell Qualtrics that the participant has now seen this question.
Assuming that I have accurately characterized your suggestion, and please correct me if I have not, then here is what I still don't quite understand how to do:
How do I make it so Qualtrics only displays one question when it calls a block? Previously this was accomplished via use of the randomizer. You've suggested the use of embedded data but I'm not quite sure what that looks like.
How do I create embedded data when a question is displayed so that Qualtrics understands that question has now be seen for the next time the block is called?
Also I want to clarify something. Am I correct that your method will randomize the question ordering one time, when I create the survey but then every participant will see questions in the same pre-determined random order? That may not be a problem, I just want to make sure I've understood correctly.
Thanks again Tom.
+27https://www.qualtrics.com/community/discussion/comment/29141#Comment_29141No, sorry, that is not what I meant, but it doesn't matter because your general approach isn't going to work at all. I did a quick test, and if you repeat a block Qualtrics will only record the answers from the last iteration of the block. Any question not displayed in the last iteration of the block will be blank.
+8Hi,
I have not been able to find a way to do random numbers without replacement. I have figured out some work-arounds though. Take a look at my video tutorials on randomization: https://researchguides.library.syr.edu/dataservices/Qualtrics .
Paul
https://www.qualtrics.com/community/discussion/comment/29175#Comment_29175Thanks for your reply Paul. I watched your videos. Just to make sure I haven't missed it, is your workaround creating a block for each combination of questions like you did with your three paragraphs example? Unfortunately I don't think that'd be viable in my case since I'll have no fewer than 100 questions which would be 100! blocks.
Regardless, I did learn some neat stuff about creating test data so thanks for sharing.
+8Hmm. I need to better understand your design.
I'm surmising that you have three experimental groups - one control and two treatment. Do I have that correct? If so, do these groups correspond to your three pools of questions? For example, the control group sees questions from Pool 1, Group 1 see questions for Pool 2 and Group 3 sees questions from Pool 3?
There are 100 questions total, or in each block?
Does each question get displayed to one and only one respondent?
How many respondents will you have?
+27https://www.qualtrics.com/community/discussion/comment/29173#Comment_29173You could use JS to do randomization without replacement.
At the beginning of the survey (maybe your intro question). Add JS to randomize all the P questions and save them as comma separated values in embedded data fields (e.g. P1list, P2list, P3list). For example, let's say your P1 questions are p1.1, p1.2, p1.3 and p1.4. Put the values in an array, shuffle the array, then save it to an embedded data field as a comma separated list.
Put all your P questions in separate blocks, then use a randomizer, groups, and branches to randomly show one of the P questions from one of the P groups. You would get the question to show by adding a block the beginning of a P group that gets the first question from the list, sets an embedded variable, updates the embedded data field that contains the randomized list, and clicks the next button. Survey flow would look something like:
Randomizer (1 of 3)
P1 Group
Block: JS to set p1display as next random P1 question
Branch: If p1display = p1.1
Block p1.1
Branch: If p1display = p1.2
Block p1.2
...etc...
P2 Group
...etc...
P3 Group
...etc....
You could duplicate the flow above as many times as needed.
https://www.qualtrics.com/community/discussion/comment/29206#Comment_29206At a conceptual level I think I follow what you're saying here and it seems like it would work. I have done some minimal coding in the past, mostly just making small changes to existing code, but have never touched Javascript. Can you recommend a good resource that illustrates how to add something like this in Qualtrics?
Also as a clarifying question, if, for example, each of my pools had 50 questions then I would be creating 150 unique blocks since each question is its own block? That's not a problem if so, I just want to be sure I've understood.
Thanks so much for your help.
https://www.qualtrics.com/community/discussion/comment/29187#Comment_29187We have just two groups a control group and a treatment group. In either group a participant will be asked one of the questions chosen at random from one of the three pools. This process repeats for a total of ten questions. They are then given a few seconds break and do another ten questions. We will need to do some pilot testing and power analysis to determine exactly how many blocks of ten questions each participant will see, but it will likely need to be at least 100 total questions seen by each participant (the questions are very short).
This process is actually exactly the same for the treatment and the control conditions. The only difference is that in the control condition question pools are always sampled with a constant 33% probability throughout and in the treatment condition one of the pools sees a decline in the probability of being sampled over time with a corresponding increase in the opposite pool. It is not important to us that any specific question is evenly represented only that a given type of question (each pool is a different type of question) is represented roughly commensurate with the probability distribution given by the participant's condition.
Because of the probabilistic nature of sampling questions each block will probably need about 40 to 60 questions.
We are using professional participants who are not easy to gain access to but we are hoping to have at least 50 participants so 25 per cell.
+8wow, that's a tuff one! :-)
I don't think you'll be able to do this without some JS scripting as TomG said. Unfortunately, I haven't done any in about 15 years.
If Qualtrics had a looping element in Survey Flow you might be able to do it.
Sorry I couldn't be more help. Good luck - let us know if you figure it out!
https://www.qualtrics.com/community/discussion/comment/29237#Comment_29237No worries mate. Thanks for taking the time to weigh in.
Leave a Reply
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.