Testing Javascript randomization using Automatic Test Responses doesn't work —is there a better way? | XM Community
Skip to main content

TL;DR: 'tools> generate test responses' fails with javascript hidden questions? Multiple-choice display order randomization doesn't work either?
Hi all, I recently posted a question about advanced randomization in a 3x3x12 factorial study design. I have solved this issue myself, but now I cannot really test if the randomization is working as intended. My solution relies on passing the displayed choices of a multiple-choice question into a JS script to randomize the stimuli. If a human completes the survey, the displayed choices look like this once I export the data: {6, 8, 10, 5, 9, 4, 11, 7, 1}. However, when I use "Tools > Generate Automatic Responses" the results look like this: {0, 1, 2, 3, 4, 5, 6, 7, 8} or {0, 1, 3, 5, 6, 7, 8, 9, 10} or {2, 3, 4, 5, 6, 8, 9, 10, 11}, etc.. Note that the order is no longer randomized.
One way I could check if randomization is working would be checking the actual dummy responses to my survey generated by the automatic response generation system. However, my solution works by hiding questions using JS and unfortunately, the automatic response system responds to all questions regardless of whether they would be displayed or not to participants. So doing this doesn't help me figure out what questions would 've actually been displayed to participants.
Is there any way I can test my randomization is working properly other than myself completing my survey 300 times?

If anyone cares, here is how I implemented my randomization (perhaps someone who knows more than I do can tell if the randomization will work just by looking at this).
I created three blocks based on one of my experimental variables. Participants are assigned to one of these blocks randomly and evenly, so this is fine. Inside of each block, I do this:

  1. I created a multiple-choice question. The question has 12 choices —numbers 0 through 11. In advanced randomization, I set that only 9 choices are presented to participants, evenly and in randomized order. This question is invisible and I'm passing the displayed choices into an embedded field, and later into an array (called randomNumbers).

  2. I have a block with 36 questions (3x12 variables). One variable is room_type (let's calle them A, B or C). The other one is flooring material (let's call them FL0 through FL11). I need to present each participants 9 of these questions, such that they see 3 rooms A, 3 rooms B, and 3 rooms C; and such that each one of the rooms has a different material out of the 12 possible ones.

  3. To do this, I assigned each stimuli (i.e. question) a number from 0 to 11 based on the variable FL. Then, I created a custom script where I check if this number is contained within specific portions of the randomNumbers array. The portion of the array depends on the room type, such that room_A will search array indices 0-2; room_B will search in array indices 3-5, and room_C will search in indices 6-8.

  4. This is pseudo code of how the whole block works (just one flooring material and three room types in the example):

  5. FL0_Room_A: I check if the number "0" is contained within the indices 0-2 of the array.

  6. FL0_Room_B: I check if the number "0" is contained within the indices 3-5 of the array. 

  7. FL0_Room_C: I check if the number "0" is contained within the indices 6-8 of the array.

  8. For each question (i.e. stimulus), only one statement is evaluated. If the statement is TRUE, then the question is shown. If it is FALSE, the question is hidden. This approach guarantees that for each block, I see 9 questions, in this fashion:
    FL0_room_A; FL1_room_A; FL2_room_A; FL3_roomB; FL4_roomB; FL4_roomB; FL6_roomC; FL7_roomC; FL8_roomC.
    The permutations are actually different, of course, but this shows we get 9 different materials and 3 rooms of each type displayed to a given participant. 

This is the actual JS code for one question. For the same material but a different room type, the for loop would iterate between 3-5 or 6-8. For a different material, the if statement would be
if(rndNumdi] == N)
where N would be another number from 0 to 11, depending on the material:
code.jpgI know that because of the advanced randomization, each possible choice in the multiple-choice question will be displayed equal number of times. However, will it also be displayed in all possible positions (e.g. first, second, third, etc.) the same number of times?
Thank you for your help


I figured out a way to test my randomization. I created a survey with my multiple choice question only, and then manually completed this one-question survey many times.
I also got in touch with Qualtrics support so they fix this feature, as I think it makes sense that the automatic response generation: a) does not respond hidden questions; and b) displays multiple choices in aleatory order when applicable, even if it is irrelevant from the bot's perspective.

As for the randomization itself, it doesn't work as intended. For 70 participants in the same condition, some questions would be viewed as little as 15 times will others would be seen as much as 30 times. However, when I add responses by material (e.g. roomA_material1 + roomB_material1 + roomC_material1) then all materials have equal exposure. This makes sense as the randomizer displays each number evenly, which is determines material selection. However, the aleatory order does not display each number uniformly across all possible 9 choices, meaning number 4,for example, may show up more as 3rd choice than 9th choice. Since choice order determines which room type is selected for display, this lack of uniformity explains why, for a given material, some room types are displayed more than others.
I wonder if there's a workaround for this in Qualtrics that is not too hard to implement. Any thoughts anyone? Here's an image that hopefully illustrates what my problem is.
randomization result.jpg


Hi everyone,
I am having the same problem, namely, I have a survey which uses javascript for display logic, randomization, and populating some embedded variables; when using the Generate Test Responses feature, none of this works. I unfortunately need to generate an amount of test data that is impossible to do manually and need the javascript to run in the tests; I was wondering if anyone has a solution to this?


https://community.qualtrics.com/XMcommunity/discussion/comment/40322#Comment_40322I am also having the same issue.


When you use 'Generate Text Data' in Qualtrics, the test data generation is done on the server. Since JavaScript runs on the client, JS is ignored when test data is generated.
Other than manually completing the survey, the only way to generate test responses that run JS would be to write JS to automate taking the survey - randomly answer all the questions and click Next on each page.


Thanks for the reply!
In the end I had to use Selenium to automate responses. It's a bit tedious...


Hi,
I am having issue running a javascript which is supposed to select randomly 5 grocery stores from a survey question where respondent is asked where they had shopped in last year (in a list of 40 stores).
The script works ok but for some reason it automatically runs twice and that caused the issue.

  • In the 1st run it select 5 groceries and perfectly fine

  • In the repeated run it sometimes selects different set of stores and that causes select more than 5.

Here is the script.
var i;
var val1;

var choiceInputsQ1="${q://QID29/SelectedChoicesRecode}";
var arrQ1 = new Array();
 arrQ1 = choiceInputsQ1.split(",");

 arrQ1=ArrayShuffle(arrQ1);
var n=0;
for (i = 0; i < arrQ1.length; i++) // All selected in Q1
{
 val1=parseInt(arrQ1ri]);
 this.setChoiceValueByRecodeValue(val1,true); 
 n=n+1;
  if (n==5)
  {break;}
 }

Please help me if there is any solution to force to run script only once.
Jalal


Hi diliprr,

Would you mind sharing your Selenium code? This is an awesome idea, and I'd like to adapt it to my surveys, but it would probably save me (and others reading this thread) a lot of time not to have to start from scratch. Thanks a lot in advance! :)

Best wishes,
Constantin


Any updated solution to this? I’ve tried Selenium code to generate surveys to no avail and would love to know how others handled this problem.


Leave a Reply