A/B Testing Email Distribution | XM Community
Skip to main content

We have a fairly high Open Rate of 47% but the Survey Start Rate is 8%. I believe there is opportunity to potentially increase the CTR and Survey Start rate by optimizing the email.

I have not found any features that allow me to A/B Test subject lines and/or email body (layout, design, copy, CTA). These features are fairly standard in an email distribution platform.

Can I find features like this in Qualtrics to easily test variations and have Qualtrics distribute the email traffic and measure the outcomes?

 

Or is there another approach to accomplish this such as creating a Contact Field that allows me to distribute different emails (A/B) based on a field value? This is really not an optimal approach as our contact lists are automated from source systems internally.

 

@nkerzman I’m afraid you have to do these kind of test manually by making different distributions and analyze on it’s result. The only pre-build invitation A/B testing I can find is with website feedback.
You can suggest the A/B testing for email here: Evolve: Product Ideas | XM Community (qualtrics.com)

@nkerzman I’ve faced into similar before; 

  1. First I recommend you have bot detection and security scan monitor on if you don’t have already.  There is ultimately a few different software complications based on recipients email client and software etc and this is the first step to helping ensure any clicks are by a human as opposed to spam scanners etc. Without this; I wouldn’t trust any open/start rate reporting.
  2. You & @dxconnamnguyen are right there is no default options for A/B tests; you will need to set these up as different distributions.  We also have automated distributions as well and this prevents us doing as much a/b testing as we would otherwise.
  3. Personally I would ignore open rate; these days open rates through Qualtrics or any other email/marketing platform are typically quite overstated - not limited to but significant impacted by iOS15 changes in 2021.  One year later: iOS 15’s impact on email marketing (omnisend.com).  We focus on start rates ourselves; but only after point 1 as a large % surveys started were from email scanners clicking links before we turned this on.

 

@ScottG and @dxconnamnguyen  thank you both for your thoughtful responses and clarifications on my observations around testing within Qualtrics. I think we would either need to approach testing our manual programs first, or modify the automation file to support a “flag” for bifurcating the distribution for testing. Not optimal, but doable. Long term it may involve moving distribution activity external of Qualtrics and into a email management solution already in our martech stack.

@ScottG great reminder on the iOS changes and the potential open rate impact. 

Leave a Reply


Terms & Conditions