A year ago, we published a series of best practice for questionnaire development. This year, we would like to highlight some common mistakes that research practitioners should avoid, to ensure their data collection effort is not wasted. We are calling them The Seven Deadly Sins Of Questionnaire Design.
The Seven Deadly Sins emerged from an ecclesiastic era, and since then have evolved as broader ethic markers for those who prefer disinfected consciences. Gwyneth Paltrow lost her head over them in the movie Seven, and the secular world has incorporated them as business credos (an example being The Seven Deadly Sins of Management, from the Harvard Business Review).
As long as people are dropping the ball in their professions, the Seven Deadly Sins work as a values template. They certainly work in market research, specifically when it comes to online survey questionnaire design.
Below are the Seven Deadly Sins of Questionnaire Design that every researcher should avoid.
LUST
You are passionate about your project. You let that lust pollute your wording, even allow bias to possess the questions like a Linda Blair dream. Sometimes you don’t even know you’re doing it! As one research expert put it: “Bias is the mortal enemy of all surveys, and as a survey creator, it’s important to guard against it to make sure you get reliable results.”
To avoid this Hades, keep your language neutral and dry; employ a sensible number of opt-outs and open-ended questions; and make sure you use a second and even third set of eyes while crafting questions.
GLUTTONY
You crave that data or have a reprehensible voracity for it. You write an extremely long questionnaire, which ultimately results in crappy data collection. You ignore the fact that respondents don’t care much for long, boring surveys, or your greediness for the data causes tunnel vision in your data collection methodology. It’s all part of your questionnaire design.
As some of our own research has shown, respondent fatigue sets in after 20 minutes of a survey. This may result in respondents exerting less effort and spending less time thinking about their answers as they journey deeper into the survey. A survey over 35 minutes is an indicator that your craving for data is approaching its peak with little consideration for the survey participant.
Some had predicted that online surveys completed from mobile devices would approach 50%. In an era where mobile devices are displacing computers, long survey questionnaires are just a sin.
SLOTH
Put it simply, edit, edit, edit! Mistakes won’t make you look slothful to respondents, just demonic. Furthermore, on the side effects of sloth, a good research practitioner explained:
But the problem is rarely “bad respondents” – instead the problem is lazy researchers. When people discover that the survey they just agreed to take is boring, tedious, repetitive, or too long, they either quit altogether or they stop providing good answers. As I’ve stated many times in the past, when it comes to data quality, the burden should always be on the researcher first.
WRATH
You want to defend your research project or the sanctity of your data. Full of wrath, you will strike down upon thee with great vengeance and furious anger those unsuitable respondents who access your holy survey. Okay, maybe not Ezekiel 25:17, but you’re going to place a lot of screeners and trap questions to weed out the unfaithful.
Recent studies indicate the methodology of trap questions for surveys may not be as effective as originally thought. In reality, trap questions might have unexpected results—such as shifting the thinking of respondents to critical thinking from “optimal thinking” (that is the state of mind they reason as they normally would in daily life, which is typically necessary for reliable data).
There are other analytical ways to evaluate respondent data that don’t include placing land mines in your questionnaire.
GREED
Budgets are rarely fun, unless you’re working on the next Marvel film or you’re a Congressman(not from the state of Illinois). At the same time, a sense of greed within you assumes that the internet ought to make research economical.
Therefore, you skimp on incentives. Bad move.
Some reports claim that 175,000 online surveys are conducted a day. This volume has influenced a drop in participation rates to historic lows, which some estimates to be at 2%. On the other hand, studies have shown that the proper incentive will have a positive effect on survey response rate. We touched base on this topic on a previous blog.
This form of avarice can be avoided by rewarding your respondents properly for their time, and never assume they care about your brand as much as you do.
ENVY
Okay, you’ve done all this work and soon respondents will joyfully complete surveys, while vying for an iPad, that trip to Hawaii or Starbucks gift card. You say to yourself: “this is a lot for a 40min IDI or telephone interview and it’s all from my blood, sweat and tears”.
This attitude of envy will harm your research project. You’re not just envious, you lack empathy—the key ingredient for a successful online survey questionnaire. As stated in a previous blog, empathy is significant.
“Companies need to have more empathy for the research participant. The person(s) who writes the survey instrument should ask themselves if they could sit through that survey for 25-30 minutes. Companies should make surveys fun and engaging, regardless of the topic. They should test their surveys over and over again to identify the fatigue points in the survey. This is usually the area where data integrity is compromised.”
This quote addresses all the other deadly sins, mind you, because they all overlap. It certainly overlaps with:
PRIDE
Pride is also known as vainglory. It basically means you think you know better than everyone, including study participants. Pride has become a positive quality in western culture, but don’t let it fool you.
Pride also tends to stifle the ability to be open-minded. With all the tech innovations changing market research this year, such as eye tracking technology, social listening etc., don’t let your pride assume your ways are absolute and unchangeable.
The opposite of the Seven Deadly Sins are the Seven Heavenly Virtues. We’ll discuss them pertaining to online surveys in the future. Right now, though, avoiding the above list will likely create a paradise for your next online survey undertaking. Hell might not freeze over, but neither will your data.
