My friend works for a company that just ran a survey on their entire client base to guide 2021 strategy. They spent money and precious credibility getting 800 responses from busy firms that heavily use their software. They needed detailed responses, so they added an open-ended question about how they could help their clients in the coming year.
Of the 800 people that responded to the survey, only 4 typed in an answer. Of these 4 answers, one was literally: “Improve the product”. The other 3 weren’t much better.
That was not a fun report to take to the CEO. (Fortunately, my friend wasn’t the sponsor.)
I tell this story because I hope I can help others avoid this kind of embarrassment – friends don’t let friends write bad surveys! So I put together a few simple suggestions that we can all use to ask better questions and get meaningful answers.
1) Make the question specific
Our brains are hardwired to avoid stress, and softball questions are guaranteed to get softball answers. If I ask someone on an elevator “How’s your day?”, I’ll usually get an equally thoughtless response. Yet that’s how many questions are written.
Make your question meaningful and specific. Sometimes I ask myself: WWMTS? What Would My Teenagers Say? When I ask them about their day, they always take the easy way out, so I force the issue: “What was the best thing that happened to you today?” Even simple changes like this can help a lot.
2) Use framing effects to help you
Stop obsessing over bias! Instead, focus on usefulness. Many freshly minted college graduates are so afraid of influencing respondent answers that they water down questions to the point of emptiness.
Cognitive biases like Framing Effects are common in surveys, so use that do your advantage! “Leading questions” are great if you offer the right lead-in. Consider changing the perspective of the question to prompt deeper thinking. For example, you might ask: “If you were managing this product and could only focus on 3 strategic platform improvements next year, which would you choose?”
In this question, we did three things:
Changed the frame of reference
Used the Scarcity Effect by limiting the number of answers
Focused on a specific need (platform improvements)
3) Use Modern Tools
OK, this is a bit self-serving, but seriously: Why are we asking open-ended questions the same way we asked them in 1994 (remember Greenfield Online)? That was before fast internet, before Facebook, before the iPhone, before tablets… before some of you were even born!
It wasn’t a good format then, and it isn’t a good format now. New technologies like using voice narratives (so people don’t have to thumb-type answers) increase content by up to 800%. Dedicated interfaces and interactions – like Intuify’s own storytelling mode or conversational AI chatbots (like The Evolved Group) – get even more content.
I sometimes wonder if survey writers just don’t want too much data that they have to manually code. That should no longer be a concern. New technologies (including transcription AI, NLP, and media processing platforms like Intuify) have cut the time and effort of processing these answers by more than 95%.
In a recent B2B project, we captured an average of 45 seconds of audio content per question. With over 1,300 answers, we gathered over 16 hours of content that we auto-transcribed and coded (topic, sentiment, emotion, etc.) in minutes. Imagine taking a 3-minute highlight reel to the CEO with authentic customer voices, backed by quantitative support?
Sounds crazy? It should be a standard deliverable with current technology.
4) Prompt and Pivot
Unless you’re running a study for the NIH or CDC, stop thinking about your survey as a clinical experiment and start thinking about it as a user experience. A survey has a story and a flow, just like the analysis.
Asking a respondent to tell you what’s most important without any build-up is like asking a random person to create a movie script on the spot. Our brains need context to trigger memories.
Cognitive scientists have studied this for years, and qualitative researchers have mastered it. Quants need to catch up and use proven methods to actively stimulate memory recall. At Intuify, for example, we identified six strategies from academic and industry best practice and simplified to an easy system we call E3C3. E3C3 stands for: Emotions, Environment, Extremes, Context, Cues, and Categories. (If you’d like a one page summary of these strategies, email us and we’ll send it over.)
Does it make a difference? Yes! In a recent auto insurance survey we used these techniques as part of a recall-based Path To Purchase study with past 2 year shoppers, and we elicited incredibly detailed responses on how and why people made their decisions.
5) Cut Through the Clutter
A lot of people multitask while taking surveys. This hurts both the survey and the respondent’s creativity. If you want their attention, then earn it. Make the survey interesting.
For example, have you considered asking the question with a 20-second video spot? It’s easy, and it gives your respondents a person and a face to attach to your organization, which is a lot more engaging than some 12-point-font words on a screen.
Is that too much work? Consider that if you ask 800 customers to answer an 8-minute survey, you’re eating up 107 hours of their combined time (that’s over 13 business days!). If you really want them to care about your business, show them that you care about their voices.
“To be able to ask a question clearly is two-thirds of the way to getting it answered. ― John Ruskin
How do I get started?
So there you go… some easy ways to ensure you get great answers that support fantastic insights. Do you have any other tricks and tips? If so, share them below!