A Question of Scale: Thoughts on Questionnaire Design

March 12, 2012 Written by Richard Jenkins

I have a love-hate relationship with the 10-point scale. Although it has considerable advantages over its much smaller cousins, the 7-point and 5-point scales, it also comes with some significant drawbacks.

It seems to promise so much to the survey researcher. These advantages include:

  1. the ability to one number to summarize the result. Yes, there are occasions when the mean is not a very good measure of central tendency (e.g. when the distribution is bi-modal or when outliers pull the mean too far away from the center) but overall, it is often a very good indicator of the balance of opinion. The mean is often better than choosing just one category (very strongly) or a net of more than one category (very or somewhat) because it actually uses all of the information.
  2. the ability to better use correlation and regression analyses. All too often market researchers rely on cross tabs to tell the story and some of this reliance is a function of the use of scales that are not as good for this purpose.

The problem, however, is that a 10-point scale creates other headaches (ordered below from least to most important).

  1. A 10-point scale is messy, particularly on-line. All of those check boxes/ radio buttons make for intimidating surveys.
  2. There is not really a natural middle. 5 is half way to 10 but on a 10 point scale both 5 ad 6 are equally in the middle. This has to mess with respondents who want to be neutral.
  3. There are no natural categories. Since we often want to group the scale items into smaller buckets the lack of obvious buckets makes it awkward. One can go with 5 (2 item) buckets but there are no natural labels for these combinations.
  4. Means don’t carry meaning very well. There is nothing like a presentation or client discussion about the results when the mean says 7.2/10. There are blank stares. As humans, we have ways of understanding average age or average income (because the quantity makes sense) but averages on 10 point scales of agreement or importance don’t mean much. Most of us think in terms of key breaking points — e.g. 50%+1 is a majority but the mean does not translate as well. The result is uncertain clients who can only rely on relative differences — this mean is higher than this other one.

Like many questionnaire design choices, there are both good and bad that go with the use of a 10-point scale. For me, the 10-point scale comes out most often when I want to be able to use other statistics with more confidence. Using them with scales of less than 10 points can be problematic.

When do you use a 10-point scale?

Survey methodology
, , ,
Similar posts
  • Implications of survey mode on public opinion content | a look at the number of questions — Survey mode is an interesting variable in the current market research industry landscape. Historically we moved from in-person to mail to telephone and online as the principle methods of polling. The latest entry into the survey mode toolkit is the IVR or interactive voice response survey. Several market research firms in Canada release polls into [...]
  • More election polls and fewer policy related polls in Q2 — Q2 saw a rise from Q1in the number of polls released into the public domain as 173 unique surveys produced 257 different releases compared with 195 in Q1. Overall, however, the number of polls is consistent with Q2 of 2016. More than half of these were national polls with provincial and local polls representing the [...]
  • Unbiased Questions: The Use of Endorsements — Bias has considerable negative consequences and the easiest critique of a public opinion survey is to suggest that the questions are biased. But bias is inherent in asking questions and relying on truly unbiased questions risks stripping the questions of meaning, leaving them stale and irrelevant. While probably everyone could recognize extreme bias, bias can be [...]
  • Questions I Hate on Surveys: A Continuing Series — I spend a lot of time on the Internet, as you probably do as well. It is there in the background when I am working, it is in my pocket when I am walking, it is playing music in the kitchen as I prepare dinner, and it is sending me alarms and notices. When I [...]
  • We all want relevant advertising.. so what? — Do you want companies to send you or expose you to irrelevant ads? I think not. But maybe this intuition of mine is wrong. What is needed is a scientific study. Thankfully, SAS and Leger Marketing asked some questions on this topic recently. They found that: 60% of online Canadians agree that “I would like to receive offers and promotions [...]