Skeptical of Survey Data? Community Poll Open Now ✔️ | XM Community
Sticky Poll

Skeptical of Survey Data? Community Poll Open Now ✔️

  • 18 April 2024
  • 9 replies
  • 162 views
Skeptical of Survey Data? Community Poll Open Now ✔️
Michael_Cooksey
Rank
Userlevel 7
Badge +32

Are you or your stakeholders skeptical of survey data?

 

Many are becoming increasingly skeptical of survey data, in terms of relevance, inherent bias, and retrospective views. How can a broader cross-section of distribution methods can improve this sampling? 

Do you have any examples of how broadening your sampling methods across multiple channels improved data quality and reduced bias? Has anyone done this analysis across your own internal or external customer base? 

Tell us in the comments how one can improve the integrity of survey results 🤔

How can a broader cross-section of distribution methods improve survey results?

9 replies

Michael_Cooksey
Rank
Userlevel 7
Badge +32

Follow up question: How are we thinking about using Gen AI to overcome known survey bias / plugging sample gaps?

Curious if experts like @AdamK12, @bstrahin, @JessicaGregory_CHNw, @ArunDubey or @Deepak have thoughts to share on this poll or follow up question? 🧑‍💻

PSmith
Userlevel 1
Badge +1

There is a lot of known survey bias, and I would love to see AI being used to overcome this. One question that I have is how we can train AI to look for and recognize bias, which is an inherently human condition. That being said, addressing those survey gaps is going to become increasingly important over the years! 

https://www.forbes.com/sites/glenngow/2022/07/17/how-to-use-ai-to-eliminate-bias/?sh=4199523e1f1f

JeffDavis
Rank
Userlevel 3
Badge +4

To eliminate bias, you must first make sure that the data you're using to train the algorithm is itself free of bias or that the algorithm can recognize bias in that data and bring the bias to a human's attention (source: SAP). 

I don’t personally have any examples of this from my work, but I am quite keen to see how Qualtrics uses AI in the future around this topic.

 

egreenberg
Rank
Userlevel 3
Badge +6

I personally think avoiding bias can be achieved by making sure your distribution channels are themselves not unintentionally inviting bias. This may be what is being hinted at in the original question. But making sure that you are segmenting your audience personas properly and leveraging channels to reach those segments is key. 

This is something AI might be able to suggest or prompt survey designers on currently! Maybe there is a product idea in there for Qualtrics (building these AI prompts into the product). Just my two cents!

GabrielleH
Rank
Userlevel 3
Badge +3

Using a broad range of survey distribution methods is the best way to go. I don’t think there is any way to totally remove survey bias, but making sure you’re funneling your surveys through a wide variety of channels to reach the most diverse audience possible is best. 

S
Rank
Userlevel 6
Badge +23

Working in financial services, we are seeing a narrowing of our survey distribution channels as opposed to increasing them, given both organisational choices and respondents own preferences and concerns for security. 

 

For reducing our bias though our focus is very much on our survey triggers and ensuring we capture at different stages of journeys i.e. not capturing successful journeys only. This often includes looking at our internal data points and considering balancing our survey data with other passive listening (non-survey measures).  

 

I also find organisational culture can drive bias across any survey program and so fostering a culture that is curious and is open to both good and bad results is key to reducing bias through surveying. 

 

Less around distribution, I believe AI detection in the quality of feedback received and where feedback/results may not be representative based on a range of parameters given would be highly useful.  

 

Michael_Cooksey
Rank
Userlevel 7
Badge +32

Working in financial services, we are seeing a narrowing of our survey distribution channels as opposed to increasing them, given both organisational choices and respondents own preferences and concerns for security. 

 

For reducing our bias though our focus is very much on our survey triggers and ensuring we capture at different stages of journeys i.e. not capturing successful journeys only. This often includes looking at our internal data points and considering balancing our survey data with other passive listening (non-survey measures).  

 

I also find organisational culture can drive bias across any survey program and so fostering a culture that is curious and is open to both good and bad results is key to reducing bias through surveying. 

 

Less around distribution, I believe AI detection in the quality of feedback received and where feedback/results may not be representative based on a range of parameters given would be highly useful.  

 

@ScottG thank you so much for the detailed reply! I personally would not have thought that narrowing distribution channels would help in this regard, but I fully see your logic and reasoning! 

Regarding AI detection, what would be something that should exist in product? Eg., inside the Qualtrics survey platform? Or are you currently using another tool to accomplish this?

S
Rank
Userlevel 6
Badge +23

Working in financial services, we are seeing a narrowing of our survey distribution channels as opposed to increasing them, given both organisational choices and respondents own preferences and concerns for security. 

 

For reducing our bias though our focus is very much on our survey triggers and ensuring we capture at different stages of journeys i.e. not capturing successful journeys only. This often includes looking at our internal data points and considering balancing our survey data with other passive listening (non-survey measures).  

 

I also find organisational culture can drive bias across any survey program and so fostering a culture that is curious and is open to both good and bad results is key to reducing bias through surveying. 

 

Less around distribution, I believe AI detection in the quality of feedback received and where feedback/results may not be representative based on a range of parameters given would be highly useful.  

 

@ScottG thank you so much for the detailed reply! I personally would not have thought that narrowing distribution channels would help in this regard, but I fully see your logic and reasoning! 

Regarding AI detection, what would be something that should exist in product? Eg., inside the Qualtrics survey platform? Or are you currently using another tool to accomplish this?

Thanks @Michael_Cooksey we don’t have a tool that is able to proactively review and monitor the quality of our feedback, which it would be something that be great to see Qualtrics introduce with the investment in AI capabilities. 


At present this review and governance is manual outside of platform, with the benefit of having it integrated would keep it consistent and less subjective to analysis of any individual.  As tools such as Qualtrics get easier to use & decentralise insight generation, I believe there is a need for stronger programmatic guidance/governance to help maintain quality of insight.  Just like Qualtrics helps assess quality of questionnaire, it could also help assess the quality of output.

Michael_Cooksey
Rank
Userlevel 7
Badge +32

Working in financial services, we are seeing a narrowing of our survey distribution channels as opposed to increasing them, given both organisational choices and respondents own preferences and concerns for security. 

 

For reducing our bias though our focus is very much on our survey triggers and ensuring we capture at different stages of journeys i.e. not capturing successful journeys only. This often includes looking at our internal data points and considering balancing our survey data with other passive listening (non-survey measures).  

 

I also find organisational culture can drive bias across any survey program and so fostering a culture that is curious and is open to both good and bad results is key to reducing bias through surveying. 

 

Less around distribution, I believe AI detection in the quality of feedback received and where feedback/results may not be representative based on a range of parameters given would be highly useful.  

 

@ScottG thank you so much for the detailed reply! I personally would not have thought that narrowing distribution channels would help in this regard, but I fully see your logic and reasoning! 

Regarding AI detection, what would be something that should exist in product? Eg., inside the Qualtrics survey platform? Or are you currently using another tool to accomplish this?

Thanks @Michael_Cooksey we don’t have a tool that is able to proactively review and monitor the quality of our feedback, which it would be something that be great to see Qualtrics introduce with the investment in AI capabilities. 


At present this review and governance is manual outside of platform, with the benefit of having it integrated would keep it consistent and less subjective to analysis of any individual.  As tools such as Qualtrics get easier to use & decentralise insight generation, I believe there is a need for stronger programmatic guidance/governance to help maintain quality of insight.  Just like Qualtrics helps assess quality of questionnaire, it could also help assess the quality of output.

Thanks for the feedback! Much appreciated. I will make sure it’s passed along internally.

Leave a Reply


Terms & Conditions