Have you ever written a survey and felt your stomach sink when you ran it through an automated review? Maybe you knew it was a bit long, but you were hoping you could squeeze in enough questions to cover everything that you needed to know? Or maybe the survey had too many objectives, but you really had no choice?
Even so, you probably didn't get a review that looked like the one above! Yes, that was a real survey we ran. We coded the questions in Qualtrics using their standard interfaces to get the projection, and just looking at all the 'Serious' violations makes us cringe. After all, we included:
11 open ended questions (max recommendation of 3)
74 matrix table rows (max recommendation of 20)
6 separate rank questions with a total of 100 ranked items
Lots of other questions
And it was all on mobile.
Qualtrics' ExpertReview warned us the survey would take nearly 20 minutes and that we would suffer severe dropoff, especially on mobile devices. Had we fielded it with standard interfaces and interactions, it would indeed have been a disaster. We would have expected high abandonment rates, cheating, severe fatigue in the later parts of the survey, and minimal (if any) useful open-ended responses.
Instead, we constructed the survey using our proprietary ZenSight interactions and our MindFlo seamlessly integrated streaming media technology. The result surprised even us! Take a look below:
Respondents completed the survey at a median time of 14 minutes. That means the typical respondent answered questions over 40% faster than predicted. This is even more impressive since the survey was exclusively fielded was on mobile, which suffers from worse download speeds and typically takes much longer for grid questions and open ends due to scrolling and thumb typing.
Instead of the severe projected dropoff, we achieved a completion rate of 86%, which is strong by any standard for a survey of this length. Moreover, the vast majority of respondents actually enjoyed the survey. When we asked them to grade the survey, they gave us an 'A' average, and 47% rated the survey at 'A+'. Not one single person complained about the length in our open ends, and most expressed strong enthusiasm for the survey. If you'd like to hear the proof, give us a call and we'll share the video reels of the respondents telling us how much they loved it. (In respect for respondent privacy, we won't release such content to the public domain.)
But what about survey quality? The results speak for themselves - if you'd like to see them, ask us for a copy of our results for our Path to Purchase demo for auto insurance. However, you can get a high level sense of the response quality by considering the data in the chart below. The left side shows an analysis of the final open-ended question in which we asked respondents if there was anything they disliked in the survey. Most of them said nothing and instead offered very positive reviews! Meanwhile, if you consider the distribution of time spent on our multi-part open-ended storytelling questions, we see that the average person talked for 43 seconds, yielding a treasure trove of feedback that we could not possibly have sourced from standard text-entry boxes.
In our excitement to bring advances in cognitive science, natural language processing, and intuitive mobile interactions into market research, we sometimes lose sight of some simple but powerful benefits of our technologies. While we usually focus on how we bring pre-conscious insights to the surface, it's important to recognize some of the very practical and measurable benefits that Intuify's technology platforms provide.
We hope you found this inspirational. The next time your survey gets overloaded with questions that you just can't drop, don't give up! Consider some creative alternatives. Technology has advanced by leaps and bounds since the pen-and-paper days, even though most online tools still use pen-and-paper interactions. Surveys are powerful instruments to connect with people directly in ways that Big Data still can't emulate, and it's time that our industry show the world what we can really do.