Adjusting mortality for reliability provides more stable estimates of hospital quality

New research published in the January issue of the Journal of Vascular Surgery® explores the impact that adjustments in statistical reliability can have on hospital quality rankings for vascular surgery. A total of 14,569 patients who underwent common vascular procedures (open and endovascular abdominal aortic aneurysm repair, carotid endarterectomy, lower extremity bypass and aorto-femoral bypass) were identified through the 2007 National Surgical Quality Improvement Project (NSQIP). For each of the hospitals in the study, a ratio of observed to expected mortality (O-E ratio) using standard NSQIP techniques was calculated.

"We first estimated the extent to which hospital variations in mortality can be attributed to statistical noise (unexplained variations in a sample), patient factors and true difference in quality," said Nicholas H. Osborne, MD, MS, coauthor and research fellow in the University of Michigan Department of Surgery in Ann Arbor.

The study revealed that estimates of hospital mortality are more reliable at high volume hospitals compared to low volume hospitals. The estimated reliability ranged from 0 percent for the lowest volume hospitals to 55 percent for one hospital that performed more than 350 cases annually.

After stratifying hospitals into quartiles of volume, the proportion of the mortality due to statistical noise ranged from 94 percent in hospitals with the lowest volume to 64 percent with the highest number of cases. The proportion of the mortality due to "signal," which could represent the fraction of mortality attributable to quality, was estimated to range from 3 percent in hospitals with the lowest caseload to 18 percent in hospitals with the highest caseload.

To determine how much traditional techniques result in misclassifications of hospitals, estimates were then adjusted for statistical noise using empirical Bayes methods and rankings were compared based on the standard O-E to rankings after the reliability adjustment.

"The 172 hospitals reported an average adjusted mortality rate of 2.4 percent for the five procedures, varying from 0 percent to 17 percent," said Dr. Osborne. He added after adjusting for statistical noise using reliability adjustment, hospital mortality was greatly diminished, varying only from 1.7 percent to 4.1 percent.

"This adjustment for reliability had a dramatic effect on hospital rankings," said Dr. Osborne. "Overall, 43 percent of hospitals were reclassified into either a higher or lower quartile of performance using traditional methods of risk adjustment. After adjusting for statistical noise, 51 percent all hospitals in the best quartile of performance according to traditional O-E ratios were not classified in the best quartile, and 26 percent of hospitals in the worst quartile were no longer classified as such."

"Reliability adjustment should be standard for comparing hospital quality," said Dr. Osborne. "Our findings show that adjusting mortality for reliability reduces statistical noise and provides more stable estimates of hospital quality."