The number of questions asked is a good indicator of the depth of insight that is in a public opinion release. Is it light and fluffy or deep and extensive? On average, each public opinion release has 10 questions that have been asked. The average does not reflect, however, the considerable variation in the number nor the fact that in 15% of releases there is no way to confidently tell how many questions were asked.
The 3676 unique releases identified since April 2016 represent a 29,922 questions (though not necessarily unique questions).
A Lack of Information about the Number of Questions
It was sometimes hard to tell how many questions were asked. This happens when there are no figures or tables reported. See the Public Polls Project for info on how the database was constructed.
In many cases this is because the news release only includes general high-level observations about public opinion rather than the actual detail. Most polling companies put the question wordings on their sites so it is a matter of counting them.
We could not determine how many questions were asked in 15% of the releases. The number of questions is most likely to be known for electoral and party competition policy releases (4% did not identify the number of questions) and least likely to be known for general perceptions of life and world (44%). This is consistent with private sector companies that are using polling as content strategy deciding not to share the questions asked.
Topics Most Likely to Have the Most Questions
On average, releases that focus on generally perceptions of life and world tend to have the highest number of questions (14.6) if the information is available. Public policy releases (10.0) also tend to have a higher number of questions. Electoral and political competition releases (8.5) and economic releases (4.5) have the fewest questions per releases.
Many releases only have 1 (10%) or 2 (10%) questions. Twenty-five per cent of public policy releases only have 1 or 2. Electoral and political competition also tend to have fewer (15% have 1 or 2).
Online Research Allows for More Questions
No surprise; IVR polls have the fewest questions on them. On average only 4.7 questions are part of IVR poll releases. This is one third of the average number on online polls (12.1). Telephone, which is the most costly, tend to have on average 7.2 questions.
IVR has clear limitations in terms of its ability to do more that provide a superficial sense of public opinion.
Sponsors
Polls sponsored by the media have, on average, the fewest questions (average 7.0) followed by those funded by research firms (8.6). So the potentially most visible survey results are the most likely to be based on fewer questions.
Final Observations about Number of Questions
Most public opinion releases are based on a large number of questions. On average each release has 10 questions. More than enough to take a reading on a single topic. Not enough, perhaps, if there are a lot of unique topics.
There are, however, many surveys that only provide superficial insights into public opinion. Twenty per cent of releases are based on only 2 questions. In particular, IVR surveys tend to have only a few questions.
Surveys with few questions are potentially concerning in that they provide such a superficial view of what Canadians think. A headline, “X% of Canadians support <insert idea>” may not tell us much.
Public polls are all about the Business of Reading our Minds
Richard Jenkins