In a recent survey, we had a series of 11 statements that we’re asking our members. We decided to do an A/B test of the matrix display using the standard grid vs. a carousel. In the grid test, we split the statements between two questions, and in both configurations the statements were randomized. The scale was a 5pt agree/disagree, and there was a progress bar displayed.
When looking at the results, the top 2 box scores were nearly identical between the two tests. However, the top box score was significantly lower on the carousel for most of the statements (5-12% with similar volume). I’m wondering why there is such a difference between the two display setups. I initially thought that the grid format could be more prone to straight-lining the answers.
Has anyone had an experience like this? Are you aware of any research/best practices as to when to use a grid vs. carousel display? Meanwhile, we are going to be looking at shifting to individual multiple-choice questions, but if needed in the future I want to make sure we're setting up the Matrix correctly.
Thank you for your consideration and your time.
Does displaying a matrix question as a carousel reduce scores?
+4
Interesting finding!
You may have already controlled for this (I see you have some nice controls set up in the survey design) - but some other things to think about as you assess your findings 1) Did you randomize your test groups? 2) Did you run an actual statistical test to determine there was a significant difference in scores across each statement? The standard deviation between may be yielding insignificant results if you have not tested yet. 3) End of the day, one test is one test. You probably want several more before making calls about the method long-term.
That said, I do try to use Matrix questions only when asking respondents to give relative rankings across a set of statements. I only use these when I want respondents to really think about "Statement A" and how that relates to "Statement B" - the matrix table is really great for comparisons like this. Otherwise, a carousel or individual questions will help anchor the respondent in that metric/statement only.
Leave a Reply
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.