Are Lower Response Rates Hazardous to Your Health Survey?

By Michael Davern, Ph.D.
Assistant Professor
Division of Health Policy and Management
University of Minnesota
School of Public Health

The short answer is that lower response are not as hazardous as often assumed, and in many cases they may not be hazardous at all. What is becoming increasingly clear is that response rates, although reported everywhere, are generally not a good indicator of survey quality, according to several recent papers including one by Michael Davern, Timothy J. Beebe (of the Mayo Clinic), Jeanette Ziegenfuss, Donna McAlpine, and Kathleen Call of the University of Minnesota School of Public Health (2006). The paper concludes with a call for researchers to find new and innovative response bias analyses methods instead of relying on only response rates.

Past Assumptions about Response Rates

Surveys are widely used in public health for surveillance, evaluation, and monitoring of important policy issues.The most widely used measure of the quality of such surveys is the response rate (Atrostic et al, 2001; Biemer & Lyberg, 2003). A high response rate was seen as a proxy for the amount of potential bias due to differences between respondents and non-respondents. Recent research, however, has shown that response rates are only weakly associated with response bias (Blumberg et al, 2005; Groves, 2006) and public opinion surveys with much lower response rates can yield similar estimates to those with higher response rates (Keeter et al, 2000; Keeter, Kennedy, Dimock, Best, & Craighill, 2006). Davern and colleagues set out to build upon this later research using general population health surveys.

Analyzing Telephone Survey Rates

Response rates to most telephone surveys are rarely higher than 70 percent (Brehm, 1993; Centers for Disease Control and Prevention, 2006) and they have decreased over the past several years. For example, recent response rates for the Behavioral Risk Factor Surveillance System (BRFSS) survey – a telephone survey conducted by states – hover around the 50 percent range, whereas rates for general population telephone surveys in the late 1980s were typically in the vicinity of 70 percent (Groves et al., 2004).

To secure the highest possible response rates, survey research firms make multiple call attempts (e.g., up to 50 calls) and try to convert people who initially refuse to participate in the survey (Frey, 1983; Groves & Lyberg, 2001; Lavrakas, 1993). The work by Davern and colleagues examined whether refusal conversion and/or making many calls to the same number alter the results of health surveys.Using data from three large general population telephone surveys of Minnesota and Oklahoma residents (conducted by the University of Minnesota’s School of Public Health from 2003 to 2005), they examined differences in demographics and health measures between early respondents and those who took five or more days to complete the survey (a typical field period for a public opinion poll) or those who initially refused to participate (Davern et al., 2006).

Findings

The analysis revealed some differences between early and more reluctant survey responders by demographic characteristics (e.g., people over 65 tended to be early respondents and they also tended to be initial refusers).However there were no differences with respect to health characteristics such as reported health insurance coverage, access to health care, and general health status after controlling for demographic characteristics commonly used to make weighting adjustments to survey data (these variables include age, sex, geography, race and ethnicity). These findings are consistent with those from public opinion surveys (Keeter et al., 2000; Keeter et al., 2006).

In one of the three surveys Davern and colleagues examined, however, they did find a significant difference in the type of reported health insurance coverage and self-reported drug use by refusal status and being a late responder.This survey’s methods differed in important ways from those of the other two surveys examined. Most notably, proxy responses were not allowed as the survey dealt with sensitive mental health and drug abuse issues. Initial refusers were more likely to be current smokers, to have used illicit drugs and to have abused prescription drugs. Those who took 5 or more days to become a respondent were more likely to have private insurance coverage, binge drink, and have an alcohol disorder.

What the Findings Suggest

The results suggest that researchers should carefully examine under what circumstances additional survey resources should be expended toward achieving higher response rates. Making many calls to one number and attempting refusal conversion increases the cost of fielding surveys and generates higher respondent (or nonrespondent) burden.These increased costs reduce the total number of completed surveys one can obtain within a given survey budget (Allison & Yoshida, 1989; Groves, 1989). As a result, a survey’s statistical power will be lower.That is, the effort invested in continued, repeated attempts to reach a given potential respondent could be directed toward calling a new number that has a higher probability of response on the next attempt than one that has been called many times already (Groves, 1989; Triplett, 2002).

Efforts to increase response rates also create more respondent burden. Repeated call attempts create a situation in which people have to refuse twice and they may receive an excessive number of calls that may create a fair amount of respondent annoyance and burden (even for those who may never answer the phone). In the long term, this burden may heighten the downward trend of response rates that these efforts were designed to forestall or eliminate altogether.

Several recent studies suggest that the respondents we work hardest to obtain responses from may be somewhat different sociodemographically, but quite similar in their substantive responses to their more accessible and receptive counterparts (Holle et al., 2006; Keeter et al., 2000; Keeter et al., 2006; Triplett, 2002). Because the extent of bias rests on the differences between the responding sample and those not responding, a survey with a 90 percent response rate could have the same amount of overall response bias as one with a 30 percent response rate. These findings suggest that survey research in public health needs to move away from its current over-reliance on response rates as the main indicator of response quality.Researchers would be well-served by conducting more non-response bias analysis, rather than relying on the response rate as the most important – and often only – measure of quality.

References

• Allison, K. R., & Yoshida, K. K. (1989). Increasing response rates in community health surveys administered by telephone. Canadian Journal of Public Health, 80, 67-70.

• Atrostic, B. K., Bates, N., Burt, G., & Silberstein, A. (2001). Nonresponse in U.S. government household surveys: Consistent measures, recent trends, and new insights. Journal of Official Statistics, 117, 209-226.

• Biemer, P., & Lyberg, L. (2003). Introduction to Survey Quality. New York: Wiley.

• Blumberg, S., Davis, K., Khare, M., & Martinez, M. (2005). The effect of survey follow-up on nonresponse bias: Joint Canada/United States survey of health, 2002-03. Paper Presented at the Annual Meeting of the American Association for Public Opinion Research, Miami FL.

• Brehm, J. (1993). The Phantom Respondents: Opinion Surveys and Political Representation. Ann Arbor, MI: University of Michigan Press.
Centers for Disease Control and Prevention. (2006). 2005 Behavioral Risk Factor Surveillance System Data Quality Report Handbook. Atlanta GA: Centers for Disease Control and Prevention.

• Davern, Michael, Kathleen Thiede Call, Jeanette Ziegenfuss Donna McAlpine and Timothy Beebe. “Are Low Response Rates Hazardous to Your Health?” Paper Presented at the Telephone Survey Methodology II Conference, Miami Florida January 12, 2006.

• Frey, J. H. (1983). Survey Research by Telephone. Beverly Hills, CA: Sage Publications.

• Groves, R. M. (Ed.). (1989). Survey Errors and Survey Costs. New York: Wiley.

• Groves, R. M. (2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70(4), 646-675.

• Groves, R. M., Fowler, F. J., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2004). Survey methodology. New York: Wiley.

• Groves, R. M., & Lyberg, L. E. (2001). An overview of nonresponse issues in telephone surveys. In R. M. Groves, P. P. Biemer, L. E. Lyberg, J. T.

• Massey, W. L. Nicholls & J. Waksberg (Eds.), Telephone survey methodology (pp. 191-212)Wiley.

• Holle, R., Hochadel, M., Reitmeir, P., Meisinger, C., & Wichman, H. E. (2006). Prolonged recruitment efforts in health surveys. Epidemiology, 17(6), 639-643.

• Keeter, S., Kennedy, C., Dimock, M., Best, J., & Craighill, P. (2006). Gauging the impact of growing nonresponse on estimates from a national RDD telephone survey. Public Opinion Quarterly, 70(4), 125-148.

• Keeter, S., Kohut, A., Miller, A., Groves, R., & Presser, S. (2000). Consequences of reducing non-response in a large national telephone survey. Public Opinion Quarterly, 64(2), 125-48.

• Lavrakas, P. J. (1993). Telephone Survey Methods: Sampling, Selection, and Supervision. Thousand Oaks, CA: Sage Publications.