Determining if your CX survey drill-downs are statistially relevant | XM Community
Skip to main content

I have definitely been looking at online calculators for ideal sample size and confidence intervals like this one (these mostly seem geared towards marketing surveys and picking ideal sample sizes, no?), but I am still unsure how to get the answer I need. I will be presenting customer NPS results to executives, and at times breaking out by region, segment, etc. I am looking for a fairly straightforward way to gut check when a drill-down goes too far, and is therefore only directionally relevant.
For example, if one segment that represents 30% of global NPS respondents is then broken down further--where the smaller segment there represents 20% of that segment's respondents at about 50 responses, how would you go about determining how conservative to be in presenting these ratings? Is there a different online calculator I'm looking for, or even just a rule of thumb people use? The survey currently has about an 8% response rate.
Thanks so much!

Hi, AnneG -- this is a really good question and one that I've dealt with often, especially since my surveys often have a limited population that we're able to survey. The answer depends on your audience's understanding of statistics (in terms of how you can explain it when you present). I find that the rule of thumb that a sample of 30 or more generally is statistically significant, and anything less is directionally relevant (which can often be insightful as well).
I hope that's helpful!

Thanks, AdamK12. I've heard that rule of thumb as well, and will certainly be happy to use it if my audience accepts it 🙂 Do you know if the 30 responses needs to be limited to a certain time period? I have more than 50 responses for one metric but it's over a 12 month period. Maybe that seems pushing it a bit?

AnneG It depends what the level of analysis is. If you're analyzing over the year, it's fine, but your sample size will be smaller for smaller periods depending on seasonality, etc. So in that situation, what I would do is show the annual analysis, and show data/analysis per quarter or month but with the caveat that those findings aren't statistically significant.

Leave a Reply


Terms & Conditions