Hospitals Quality

Are hospital quality metrics causal?

Jason Shafrin

That is the question asked by a recent NBER working paper by Chandra et al. (2023).

This question is important for a variety of reasons. First, quality measure data collection is expensive. Saraswathula et al. 2023 found that Johns Hopkins Hospital had to report 162 unique quality metrics, and the cost for collecting these data were over $5.6m dollars ($5.0m plus $0.6m in vendor fees. Moreover, understanding whether a metric is causal is important both for value-based purchasing agreements and also evaluation of for whether hospital mergers result in superior quality or patient satisfaction (e.g., see Beaulieu et al. 2020)

A good quality metric should have at least 2 characteristics:

  • Predictive validity: Patients have better outcomes when assigned to hospitals that are higher rated
  • Forecast unbiased: The gain in patient health outcomes is equal to the value predicted by the quality indicators (i.e., actual quality differences are as large as quality indicators suggest)

Chandra et al. (2023) aim to test whether hospital quality metrics meet these two criteria.

Methodology

An overview of the Chandra et al. methodology is as follows:

To determine whether hospital quality indicators are causal, we exploit the reassignment of patients as a result of a hospital closure to create quasi-experimental variation in the quality of care that patients receive— if quality indicators are causal, patient outcomes ought to improve when a low-quality hospital closes, and worsen when a high-quality hospital closes (predictive validity). Moreover, the magnitude of the change in patient outcomes should be consistent with the predicted change in hospital quality induced by a closure (forecast unbiasedness).

The authors use 20 years of Medicare claims data for patients hospitalized between 1992 and 2015 for heart-attacks, hip-fractures, pneumonia, congestive heart failure or stroke.  To account for the fact that hospitals may close some units first, before a full facility closure is completed, the authors define closure date as the last year for which a hospital had an admission of a given type. The approach for this population uses a 3-step approach:

  1. estimate quality metrics (30-day mortality, 30-day readmissions, 30-day total inpatient (Part A) costs, and length of stay),
  2. Use hospital closures to predict how (risk-adjusted) health outcomes would change for patients in each ZIP code of quality metrics were causal, and
  3. Regress actual change in outcomes on the predicted change in outcomes [from step 2] and test whether the regression yields a coefficient of 1 (which would happen if the indicators were perfectly forecast unbiased). 

The econometric approach largely follows the Chetty et al. (2014) empirical Bayesian methodology, but identification strategy is similar to Doyle et al. (2019) — which uses ambulance referral patterns to validate the predictive validity for a range of hospital performance measures and Finkelstein et al. (2016)—which uses patient migration to validate geographic variation in health outcomes (mortality) and costs.

Results

Using this approach, the authors find that:

…closures reduced mortality by 0.2 percentage points (off a base of 13%) and reduced readmission by 0.1 percentage points (off a base of 18%), while raising costs and length of stay by about 1%. However, the effect of closure varied widely by zip code depending on the relative quality of the closing hospital. For example, mortality among affected patients in zip codes at the 10th percentile fell by about 1.5 percentage points, while at the 90th percentile they rose by about 1 percentage point

The full paper can be found here.

Further reading:

  • Beaulieu ND, Dafny LS, Landon BE, Dalton JB, Kuye I, McWilliams JM. Changes in quality of care after hospital mergers and acquisitions. New England Journal of Medicine. 2020 Jan 2;382(1):51-9.
  • Chetty, Raj, John N. Friedman, and Jonah E. Rockoff. 2014. “Measuring the Impacts of Teachers I: Evaluating Bias in Teacher Value-Added Estimates.” American Economic Review, 104 (9): 2593-2632.
  • Doyle J, Graves J, Gruber J. Evaluating Measures of Hospital Quality:Evidence from Ambulance Referral Patterns. Rev Econ Stat. 2019 Dec;101(5):841-852.
  • Finkelstein, Amy, Matthew Gentzkow, Heidi Williams, Sources of Geographic Variation in Health Care: Evidence From Patient Migration, The Quarterly Journal of Economics, Volume 131, Issue 4,  November 2016, Pages 1681–1726, https://doi.org/10.1093/qje/qjw023
  • Saraswathula, Anirudh, Samantha J. Merck, Ge Bai, Christine M. Weston, Elizabeth Ann Skinner, April Taylor, Allen Kachalia, Renee Demski, Albert W. Wu, and Stephen A. Berry. 2023. “The Volume and Cost of Quality Metric Reporting.” JAMA: The Journal of the American Medical Association 329 (21): 1840–47.