It is well established in the industry that response rates to telephone surveys are declining (between 1997 and 2002, the MRIA Response Rate Committee found that the response rate to omnibus surveys declined from 19% t0 12%). Survey firms are spending more and more effort to reach fewer and fewer people.
Not all surveys are alike. Using publicly available information, I examined over 100 surveys conducted for government departments in the past few years. Each survey was a national general public, telephone survey with no other criteria. The surveys were conducted by a number of firms but the advantage of using government surveys is that all of the firms are supposed to follow similar procedures (including an 8 call minimum for each number) and the topics are all public policy related.
The findings, a summary of which is available here, do suggest that response rates are declining.
Over the past three years the average response rate has changed from 12.3 to 11.1 but the real story is not the trend over time but rather the huge variability in response rates within a year.
At the extremes, in 2009 there were surveys with response rates of 2.5% and of 29%. Even excluding these outliers, there is considerable variation with multiple surveys coming in at 7% and others at 14%. What then accounts for the variation?
Using information, where available on interview length (in minutes) and field length (number of days), it is possible, using regression analysis, to show that each minute of survey length reduces the response rate (by .38) and each extra day of field contributes to a higher response rate (.17). These variables , however, explain a relatively small percentage of the variation.
Variable nature of response rates a quality problem?
It is a simple concept. A better response rate reflects a better survey. It is true because all of the things which are associated with higher response rates reflect good survey practice such as:
- Well designed surveys that are not too long; that are interesting to the respondent; and that are well written (e.g. not confusing)
- Proper field management practices including good interviewer training and supervision, a good call management system to schedule calls, and appropriate sample management procedures.
- A reasonable field period, which increases the likelihood of getting respondent cooperation. Only surveys with particular time sensitivity should have short field periods.
Given the choice between two surveys one with a 7% response rate and one with a 12% response rate, which would one assume is of higher quality? Naturally, it is the 12% one because we must believe that the higher response rate reflects a more focused and potentially conscientious data collection process. There are procedures, such as not having a random selection within the household, that might lead to higher response rates but lower overall quality but information on this was not available.
Even though low response rates may not themselves be a fundamental problem, the variability suggests some studies could be executed better with resulting better value for clients.
What do you think? Are clients getting the response rates (and data quality) they are paying for? Do you associate a higher response rate with higher quality?