TL;DR: 'tools> generate test responses' fails with javascript hidden questions? Multiple-choice display order randomization doesn't work either?
Hi all, I recently posted a question about advanced randomization in a 3x3x12 factorial study design. I have solved this issue myself, but now I cannot really test if the randomization is working as intended. My solution relies on passing the displayed choices of a multiple-choice question into a JS script to randomize the stimuli. If a human completes the survey, the displayed choices look like this once I export the data: {6, 8, 10, 5, 9, 4, 11, 7, 1}. However, when I use "Tools > Generate Automatic Responses" the results look like this: {0, 1, 2, 3, 4, 5, 6, 7, 8} or {0, 1, 3, 5, 6, 7, 8, 9, 10} or {2, 3, 4, 5, 6, 8, 9, 10, 11}, etc.. Note that the order is no longer randomized.
One way I could check if randomization is working would be checking the actual dummy responses to my survey generated by the automatic response generation system. However, my solution works by hiding questions using JS and unfortunately, the automatic response system responds to all questions regardless of whether they would be displayed or not to participants. So doing this doesn't help me figure out what questions would 've actually been displayed to participants.
Is there any way I can test my randomization is working properly other than myself completing my survey 300 times?

If anyone cares, here is how I implemented my randomization (perhaps someone who knows more than I do can tell if the randomization will work just by looking at this).
I created three blocks based on one of my experimental variables. Participants are assigned to one of these blocks randomly and evenly, so this is fine. Inside of each block, I do this:

1. I created a multiple-choice question. The question has 12 choices —numbers 0 through 11. In advanced randomization, I set that only 9 choices are presented to participants, evenly and in randomized order. This question is invisible and I'm passing the displayed choices into an embedded field, and later into an array (called randomNumbers).


2. I have a block with 36 questions (3x12 variables). One variable is room_type (let's calle them A, B or C). The other one is flooring material (let's call them FL0 through FL11). I need to present each participants 9 of these questions, such that they see 3 rooms A, 3 rooms B, and 3 rooms C; and such that each one of the rooms has a different material out of the 12 possible ones.


3. To do this, I assigned each stimuli (i.e. question) a number from 0 to 11 based on the variable FL. Then, I created a custom script where I check if this number is contained within specific portions of the randomNumbers array. The portion of the array depends on the room type, such that room_A will search array indices 0-2; room_B will search in array indices 3-5, and room_C will search in indices 6-8.


4. This is pseudo code of how the whole block works (just one flooring material and three room types in the example):



5. FL0_Room_A: I check if the number "0" is contained within the indices 0-2 of the array.




6. FL0_Room_B: I check if the number "0" is contained within the indices 3-5 of the array. 




7. FL0_Room_C: I check if the number "0" is contained within the indices 6-8 of the array.



8. For each question (i.e. stimulus), only one statement is evaluated. If the statement is TRUE, then the question is shown. If it is FALSE, the question is hidden. This approach guarantees that for each block, I see 9 questions, in this fashion:

   FL0_room_A; FL1_room_A; FL2_room_A; FL3_roomB; FL4_roomB; FL4_roomB; FL6_roomC; FL7_roomC; FL8_roomC.

   The permutations are actually different, of course, but this shows we get 9 different materials and 3 rooms of each type displayed to a given participant. 

if(rndNumdi] == N)