Biased Standard Errors From Complex Survey Analysis: An Example From Applying Ordinary Least Squares to the National Hospital Ambulatory Medical Care Survey

Purpose: A common research interest is to identify whether there is an increasing or decreasing trend for various health-related conditions over time in national complex surveys. We examined whether standard errors from conventional regression approaches appear accurate for trend analysis of complex surveys. Methods: We re-conducted a trend analysis of the national emergency department visit rate from 1997 through 2007 published recently in JAMA. We compared standard errors from classical weighted least squares (CWLS), generalized estimating equation (GEE), information-weighted least squares (IWLS) regression, and nonparametric bootstrapping. Results: The standard errors of the slope estimates from CWLS regression (0.88 per 1000 person-years) and from GEE regression (0.87 per 1000 person-years) were less than half the standard error from IWLS regression (1.98 per 1000 person-years). Nonparametric bootstrapping replicated the IWLS result. The p-value for trend from CWLS was only .002 and the GEE p-value was .00002, both much smaller than the p-value of .09 from IWLS. Conclusions: In ecologic time-trend analyses, standard errors from CWLS and GEE can be much too small. For these settings, IWLS provides more reliable inferential statistics.