Storing large amount of data JSON vs embedded variables

Question

Dear all,

Thank you in advance for any reply/advice.

I have a question about storing (embedded) data in qualtrics. I use qualtrics to run a study and collect data for a modified public goods game.
200 participants will play approximately 50 trials and I have about 50 variables. I use some of those values in order to construct the next trials. Qualtrics, stores all these data in one excel shell in JSON format for each participant. As a result, I have data consisting of too many characters (approx. 16000) stored in only one shell. Also, qualtrics support expressed concerns that as my data file becomes large enough, I might encounter extreme difficulty exporting the data, and due to the slow speed, the export data file might become corrupted.

I thought of using embedded variables, but based on my task, that would require 2500 embedded variables per participant, which seems a lot. Also, as my file size gets bigger, I’m afraid of a drop in performance and speed when accessing my data.
Qualtrics support suggested using Web Services elements and fire off my data to an external server as it is created. Once the data is secured on that server, clear the variables for the next trial in my Javascript (keeping what's needed for the next trial of course). This way I can steer clear of data storage issue on the platform, whilst keeping your data safe.

I was wondering whether any of you has faced a similar problem and has any suggestions (e.g., how to deal with the problem/what is the best solution; storing a large amount of data on a JSON format without problems; using embedded variables for a similar purpose, etc.)?

Best,

Konstantinos

TJUH_Kai · Answer

Have you look into normalization to reduce the size of your dataset?What is Normalization? 1NF, 2NF, 3NF & BCNF with Examples

Sign up

Login to the Community

Scanning file for viruses.

This file cannot be downloaded