Predicting participants' responses via machine learning algorithms (while my survey is running) | XM Community
Skip to main content
Hi there,



In our survey, we want to predict the choices of our participants based on their earlier responses, and then display either one of two blocks based on this prediction.



In the past we have done this via linear regression: In the survey flow, we could set embedded data to, for example:



p = 0.5 + 0.3 * [score of participant in question 1] - 0.7 * [score of participant in scale 2] ….. + 0.4 * *score of participant in question 8]



And then we use branch logic: e.g. if p > 0, the participant sees block A, if p < 0, the participant sees block B.



See screenshot below:



!



Now we want to use more complicated machine learning algorithms (probably random forest) rather than a logistic regression. The problem is that this can’t really be implemented via survey flow (you would need something like a 1000 if-then statements which is not really practical).



Here are a couple of ideas we had:



1)



Somehow integrate Python and Qualtrics via the Qualtrics API. In the first part of our survey, participants answer several questions and scales, then we transfer these values to Python, run the machine learning algorithm in Python, it spits out a number, and we transfer this number back to Qualtrics. In Qualtrics, we can use again simply conditional branching (e.g. if this value is below 0, show block A, if above 0, show block B ).



However, I couldn’t find a method to do this in real-time. Most of the existing posts transfer data between Python and Qualtrics either before the survey or after the survey, but don’t seem to work *during* the survey.



Another huge problem is that one needs administrator rights for anything involving using API stuff and it might be very costly to upgrade one’s account to get this API feature.



2)



Somehow use web service tasks in Qualtrics? Probably won’t work but I don’t understand web service tasks.





3)



Using Javascript as it seems automatically integrated. But can you install certain machine learning libraries during the survey? (Here a post on Javascript for Qualtrics I found relevant.



4)



Displaying to the participants the values of their scores, telling them to copy them, redirecting participants to an online Python code editor, paste the values of their scores there, click on run, and type in the resulting number in the next Qualtrics block. Of course, that would be a suboptimal solution because many participants would probably make mistakes.







What do you think about the above ideas? Could you help me in making them work?



If you have any other ideas, please let me know, I'd be very grateful.



Many thanks in advance!



Best,

David
I don't see how you could do this in real time, unless, as you said, there is a way to embed the code for the prediction in the javascript integration, which probably cannot be done, because you would need at least to download a library once and to be able to use it across all your respondents in that survey, which I doubt is implemented.



I don't understand exactly how you modeled the problem with logistic regression, what the targets where, etc., but from what I understood you ran all of that offline, retrieved the weights, and hardcoded them in the survey flow.

In the case of trees in general, whether random forests or gbms, you could run the model and calculate the importance of each feature (in this case it would give you the importance of the score of each answer) with a library like 'shap' (https://github.com/slundberg/shap), and use those numbers as your weights. It's messy, but you could end up with something 'similar' to what you had with the logistic regression.
Thanks for your comment!



Actually, it seems like we found a way to make it work. We use web service tasks and send the data to some external server where we run the random forest. Then we send the result back to Qualtrics.
Hi David,



We are thinking of doing something very similar with our survey. Is it possible to share more details on your solution?



Which web service tasks do you use?



Thanks,

Joanne

Leave a Reply