Yesterday’s discussion of Likert scale and Likert-style scales included the assertion that presenting the scales with the positive items to the left will result in a research bias towards the positive side of the scale, as opposed to presenting the same scale with the negative items to the left. Let’s dive into some issues with the way scales are presented to respondents, and the research-approved recommendation for how to present a scale in your online survey.
Left-side bias
When using Likert-type scales for measuring respondent attitudes and opinions, the way in which you present the scale matters. A number of studies have been conducted to measure the difference between responses gathered when the scale was presented with positive options to the left versus negative options to the left. One by Jason Chan from University of Texas at Austin (pdf download) showed that, while respondents were more likely in general to choose the left-most options in a scale, when the most positive options were listed first on the left, the response bias was higher than when listing the same options with the negative options on the left. A similar study was repeated and had the same results (pdf download): whatever is listed to the left is chosen more, but it’s more pronounced when the positive options are listed to the left.
Here’s a caveat, though – the bias is more pronounced when you’re dealing with 100 or more responses. The bias is not measured as strongly for fewer than 100 responses.
Vertical versus horizontal representation
This left-side bias presents a new question: what if we listed the same items in a vertical order, one per line, instead of the left-to-right horizontal order?
Researchers investigated this question, too. In 2009, an article was published in the Journal of Official Statistics (pdf download) describing the results of a variety of tests for rating scale layouts for online surveys. Here’s the net result: horizontal scales seemed to produce less bias than vertical scales. Interestingly, when scales were laid out vertically, it seemed respondents were less likely to read through the entire list of options than if the scale was laid out horizontally in one line.
They also tested showing respondents the five-point scale in three columns with the options reading across horizontally, the five-point scale in three columns with the options reading vertically, and showing the scale options with numbers. Their summary recommendation based on their research is to use a horizontal layout with words to minimize bias in responses. Combine that with the left-side bias, and it’s better to arrange the items with the negative options first which will produce less bias than if the same items are arranged with the positive options first.
Re-arranging response layout in QuestionPro
When it comes to putting this into practice for your own surveys created with QuestionPro, the steps to changing the layout from vertical to horizontal are simple. The steps below assume you are working with a single-select question (the most likely for using a Likert-style scale or any other rating scale).
- Hover over the question you want to modify. Click More on the right-side fly-out menu.
- Select Settings.
- Under Answer Display select Next to each other from the second drop-down (default shown is One Per Line (vertical)).
- Click Save Settings.
