The number of questions asked is a good indicator of the depth of insight that is in a public opinion release. On average, each public opinion release has 10 questions that have been asked. The average does not reflect, however, the considerable variation in the number nor the fact that in 19% of releases there is no way to confidently tell how many questions were asked.

The 2177 unique releases identified since April 2016 represent a whopping 17,737 questions (though not necessarily unique questions)

Each release is on average
made up of 10 questions

A Lack of Information about the Number of Questions

Sometimes it is hard to tell how many questions were asked. In many cases this is because the new release only includes general high-level observations about public opinion rather than the actual detail. Polling companies put the question wordings on their sites so it is a matter of counting them.

The number of questions is most likely to be known for economic policy releases (2% did not identify the number of questions) and least likely to be known for general perceptions of life and world (9.5%). This is consistent with private sector companies that are using polling as content strategy deciding not to share the questions asked. 

Topics Most Likely to Have the Most Questions

On average, releases that focus on generally perceptions of life and world tend to have the highest number of questions (15.28). Public policy releases (10.1) also tend to have a higher number of questions. Electoral and political competition releases (8.5) and economic releases (4.4) have the fewest questions per releases.

Many releases only have 1 of 2 questions. Nine per cent of public policy releases only have 1 or 2. This is very thin. Electoral and political competition also tend to have fewer (5%). 

Number of Questions Per Release by Type of Release

Online Research Allows for More Questions

No surprise; IVR polls have the fewest questions on them. On average only 4.7 questions are part of IVR poll releases.  This is one third of the average number on online polls (13.3). Telephone, which is the most costly, tend to have on average 6.7 questions. IVR has clear limitations in terms of its ability to do more that provide a superficial sense of public opinion.

Polls sponsored by the media have, on average, the fewest questions (average 7.4) followed by those funded by research firms. So the potentially most visible survey results are the most likely to be based on fewer questions.

Final Observations

Most public opinion releases are based on a large number of questions. On average each release has 10 questions. There are, however, many surveys that only provide superficial insights into public opinion. Twenty per cent of releases are based on only 2 questions. In particular, IVR surveys tend to have only a few questions.

Public polls are all about the Business of Reading our Minds

Richard Jenkins

Surveys with few questions are potentially concerning in that they provide such a superficial view of what Canadians think. IVR surveys fall into this category as do many media sponsored polls.