A conversation I had with another community member at the X4 Summit reminded me that others might benefit from the process we use to generate test responses. This is a long post, but I've tried to keep it as focused and concise as possible. My goal is to help you make effective use of the Generate Test Responses feature in Qualtrics.



Why Generate Test Data?

Generating a robust set of test data and carefully examining it ensures that your survey delivers the data you expect and helps avoid costly mistakes. By covering every possible answer option and logic branch, it can be used to uncover logic errors that may have been missed in manual testing or flaws in the survey design.



Issues with Generated Test Responses

The primary problem with Generate Test Responses that many run into is that it can be nearly impossible get past screening questions randomly. You may need to generate thousands of responses to get one complete. Some people, myself included, have added variables and survey flow logic to force test responses past screening questions. The problems with that approach are that it can be complex to set up, partially invalidates the test, adds the risk of introducing new errors, and makes the resulting data confusing.



Q_PopulateResponse to the Rescue

Q_PopulateResponse is a parameter you can add to your survey url to pre-answer questions. When used with Generate Test Responses, the questions specified in Q_PopulateResponse are answered as specified instead of randomly. Using Q_PopulateResponse with Generate Test Responses allows you to create a robust set of test data with relatively few test responses and no testing specific variables or survey flow logic.



A Q_PopulateResponse parameter value specifies the questions and their answers as a json string. Here are some examples (The json string starts with { and ends with }. Also note the answer values are choice ids, NOT recodes):

1. Multiple Choice, Single Select: {"QID1":"1","QID2":"3"} //QID1 answer 1, QID2 answer 3

2. Multiple Choice, Multiple Select: {"QID1":"1,3"} //QID1 answers 1 and 3

3. Matrix, Single Select: {"QID1":{"1":"2","2":"3"}} //QID1 choice (row) 1 answer (col) 2 and choice 2 answer 3

4. Multiple Choice, Carryover: {"QID2":"x1"} //QID2 carryover choice 1

5. Text Entry: {"QID1":"text"} //QID1 text value



Creating the Q_PopulateResponse Json String

Here is what I've found to be the easiest way to generate the json string:

1. Manually take the survey to completion

2. Download a CSV of the data and open in Excel

3. Transpose the data to a new sheet

4. Build Q_PopulateResponse strings for each screening question and other questions where you need to force an answer using concatenation formulas. To include a quote in your string you need two quotes. So, a formula to generate the string "QID1":"1" where QID1 is in cell A1 and 1 is in cell B1 would be `=""""&A1&""":"""&B1&""""`.

5. Concatenate the strings for all the questions into one comma separated Q_PopulateResponse string. In newer versions of Excel (e.g. Office 365) use the TEXTJOIN function to concatenate strings `="Q_PopulateResponse={"&TEXTJOIN(",",TRUE,D1:D20)&"}"`. If using an older version of Excel, adding a vb concatenate range function to Excel makes this easy: `="Q_PopulateResponse={"&concat(D1:D20)&"}"`. Here is a link to the concat function code: https://gist.github.com/marketinview/141bb2307fac90a6c94c



Generating Test Responses

Using the following process will ensure that responses are generated for all possible scenarios of completes and screen-outs.

1. Check Show Verbose and generate one test response with the complete Q_PopulateResponse string to make sure your string results in a complete. If needed, modify and repeat until you have the string correct.

2. Generate more responses with the complete Q_PopulateResponse string. The number of responses needed will depend on the complexity of your survey. Generally, you don’t need that many - usually from 25 for a simple survey to 100 for a complex survey.

3. Remove the last question from the Q_PopulateResponse string and generate more test responses. The number of responses needed depends on the number of choices/answers in the question you removed. A rule of thumb is 3x the number of choices/answers.

4. Repeat the process in (3) until the Q_PopulateResponse string is empty, then remove Q_PopulateResponse completely and generate test responses one last time.



If you've read this far, I hope I've provided enough information to get you started. Feel free to post follow-up questions.