A/B testing a survey | XM Community
Solved

A/B testing a survey

  • 7 February 2020
  • 3 replies
  • 75 views
JoycaV
Rank
Userlevel 7
Badge +26
  • JoycaV
  • QPN Level 4 ●●●●
  • 216 replies
I stumbled into the following Qualtrics blog post today:
https://www.qualtrics.com/eng/building-a-platform-for-distribution-a-b-testing/
and I was wondering if this is available in Qualtrics today? The blog dates from 2018.
The blog is about A/B testing a survey (so NOT about sending a survey for A/B testing a commercial or product features like on https://www.qualtrics.com/support/survey-platform/common-use-cases-rc/a-b-testing/, but A/B testing the actual survey, e.g. what subject line gets higher response rates, if you use a question directly in your email or not, if the sequence of your questions has an impact e.g. start or end with demographics, ...).
In the blog there's also a part about seeing the perfomance of the initial email distribution vs a first, second reminder, but I can't seem to find this distinction in my system.
Or, was the feature that the blog describes never made publicly available?
icon

Best answer by MattBroffman_Orlando 11 February 2020, 17:41

View original

3 replies

MattBroffman_Orlando
Rank
Userlevel 4
Badge +4
The idea here...as I read it...is to manually put people into your A or B group using embedded data at the distribution level (which you can now do) and then putting them in different lists. Once you have set that up, you can then do different things with each group/list. Different emails or reminder delays etc. Once the survey collection is complete, you can report on the embedded data to see which group performed better.

On the reminder reporting...I don't see how to do that either. You could maybe add embedded data in the URL on the reminder that would be unique to each message and report on that unless someone knows how to see which message sent.

Also, if you haven't already check out the respondent funnel (note it doesn't track reminders) but may be useful to you.

Also, thanks for sharing that post.. It was a good read!
JoycaV
Rank
Userlevel 7
Badge +26
Hi @MattBroffman_Orlando ,
Yeah I read about that "using embedded data at the distribution level" in the blog, but I must be searching on the wrong keywords but I can't find on the support pages how that works?
MattBroffman_Orlando
Rank
Userlevel 4
Badge +4

You can set them in the contact list: https://www.qualtrics.com/support/survey-platform/survey-module/survey-flow/standard-elements/embedded-data/#SettingValuesFromAContactList

Or with query strings in the URL of the survey: https://www.qualtrics.com/support/survey-platform/survey-module/survey-flow/standard-elements/passing-information-through-query-strings/

If you are doing a/b testing on distributions, I would set your groups into different directory lists and then use Panel ID to track them. You can then do different distributions for each list.

Leave a Reply


Terms & Conditions