Open-ended Comments on Surveys
Part 1 of 2 — This is the first part of a two-part blog post on open-ended comments on surveys.
Surveys are often associated with analytics. Almost all surveys are heavily weighted towards collecting, what market researchers call, closed-ended data – i.e. where users are typically asked to choose between a set of choices (either discrete choice or a rating scale.) Open-Ended text data or open-ended comments on surveys (aka freeform text), however, is different in the sense that there is no concrete question – users are asked to give their opinions on a subject via free-form text.
Open-ended comments on surveys are a curse and a boon for conducting customer perception and satisfaction studies. They are a curse because they cannot be consolidated and analyzed easily. Although there are textual analysis programs that claim to be a by-product of artificial intelligence – I have yet to meet a serious market researcher who uses these automated text analysis tools for real decision making. They are a boon because it allows your respondents to “open up” and give you comments, ideas, and suggestions that you were not even asking them.
Here’s an example:
As many of you use QuestionPro regularly know when you log out we ask you to give us comments. This is essentially a survey, with a single open-ended question in it. We realize that it can be annoying at times (especially if you are logging in and out of QuestionPro several times a day) but the log-out survey is entirely optional and you can simply ignore it if you have nothing to say and close the window.
One day, about 4 years ago a senior hot-shot VP of Research at Catalina Marketing told us about TURF analysis and suggested that we have an integrated TURF Tool – That is how our interest into integrated TURF analysis began. It’s an entirely different story that they made us go through a 40 page RFP and then decided to “build something in-house” – but the point here is that we never asked our users about TURF analysis or for that matter what kinds of analytical tools they would like integrated into the system. The question on the open-ended comment question was pretty simple – “Enter in comments and suggestion for QuestionPro.” – And Dave Suedcamp from Catalina Marketing thought about educating us on TURF analysis!
Now, we realize that many companies cannot afford to go through each and every comment and suggestion, especially if you have thousands of responses – but you can put together a human filter and escalation procedure that allows bubbling of interesting comments up the chain of command. We here at QuestionPro, periodically look at all the comments once a month and think about them and see if we need to do something about them. The other thing that has worked really well for us is that we share the comments with everyone in the company. Some of them are simply hilarious and witty and many of them are general complaints about pricing etc. and yet a few of them are actually constructive suggestions that we’ve implemented. We’re even thinking about putting some of the witty and humorous comments on the blog periodically (we’ll hide the identity of course!)
One last thing I also want to mention is the fact that it is generally considered good survey design/practice to have a “catch-all” open-ended comment question towards the end of a survey. Even if you do not really want users giving open-ended comments for your research, a simple open-ended comment question can give you valuable insight into areas of research you’ve not even thought about. Users may even give you feedback on the survey itself – they may point our grammatical inconsistencies or even offer suggestions on how you can structure your survey better – Trust us on this one – we are speaking from experience!
So, if you are putting together questionnaires and you do not have open-ended comments on surveys, think about the valuable data that you are missing on. You already have a respondent’s attention – why not give him the opportunity to express himself in his own words in addition to conforming to your structured survey?
Here are more tips on Using Open Versus Closed-Ended Questions in Surveys.
