It was awesome! There was tremendous energy and enthusiasm and a lot of great ideas and people. Quirks actually SOLD OUT the conference – the rooms were jam packed, and there were a lot of great presentations. I can’t cover everything, but here are a few highlights.
One of the standout presentations was by Pepsi, P&G, and others through the CASE consortium. Basically, there’s a lot of cheating… A LOT. We’ve been calling this out for a while, but it’s finally getting the attention it needs.
Some key points:
In a short, easy 10-minute survey, cheating was 40%
It was much worse for hard-to-reach cohorts, because cheaters know how to game the screener
Something like 20% of all surveys were taken by 1-2% of respondents and those respondents took 40+ surveys a day
Here’s a few things they didn’t mention…
Cheating on longer survey is much worse… 80% at the 20-minute mark according to a SurveyMonkey review
Cheating on low incidence surveys is also much worse because cheaters game the screener and honest respondents fall out. If you have a 3-5% incidence survey, up to 50% could be cheaters before the survey even starts.
A lot of times people ignore the cheating issue because they have a sense that it might flatten the results but doesn’t really change the answer. The presenters did a good job showing just how bad of an impact it has. Wow! 5 stars for courage! The slide below shows a product that tested great among people who said they used the product, but terribly among people who actually used it.
That is a massive difference… and this is pervasive. Everyone has known about this for ages, but no one wants to talk about it. They had several good suggestions, but it’s not enough. There are really two issues…
People lie to get incentives – but the baseline for that is 15% to 20%
The survey experience itself sucks, and so people speed through it… that’s the other 20% to 60%
There was a lot of talk about fixing problem #1, but not much about bad survey experiences. That’s a problem because that 20% to 60% includes a lot of real people who have short attention spans – but guess what, that’s the real world. If you only market to people who will finish bad surveys, you’re missing a key part of your market.
Full disclosure: Intuify fixes BOTH of these problems, so I obviously have a viewpoint here. ;)