Measuring Hospital Quality

How does one measure hospital quality? Quality occurs along multiple dimensions. Thus, to summarize overall quality, one must create a weighting scheme to compete the distinct quality measures in a single measure. In most cases, quality measures should also account for differences in patient case mix. Hospitals should not be punished with lower quality scores for treating sicker patients. Various risk adjustment methodologies can be applied to account for variation in patient case mix across hospitals.

Today, I review one proposed method for creating composite mortality measures to predict hospital mortality rates after surgery.

Methodology

Step 1: Calculate Risk Adjusted Rates for Each Measure

• Yi = Oi/Ei,
• Oi = ni-1Σjyij
• Ei = ni-1Σjpij

where pij is the predicted probability based on a logit function [i.e., pij=P(yij=1|Xij)] and Xij are patient covariates such as age, gender, race, and health status.  Yi is a Kx1 vector of risk-adjusted mortality rate for hospital i.

Step 2: Calculate Variance-Covariance Matrix

At the first level, the distribution of the mortality estimate conditional on the structural parameter is:
E(Yi | μi) = μi , and Var(Yi | μi) = Vi

where Yi is defined as above and μi is the corresponding underlying structural quality parameter that represents the average mortality rate that a typical patient could expect at this hospital, and Vi is the KxK sampling variance for the estimates in Yi.

Step 3: Conduct Hospital-Level Analysis
Other factors such as hospital-specific factors such as hospital volume, teaching status, and other factors can also affect expected mortality rates. These variables do not vary by patient; only across hospital.  However, one can explicitly incorporate these into a composite measures using the following framework:

• E(μi) = Ziβ
• Var(μi) = Ω

where β is a KxJ matrix of coefficients capturing the effect of hospital characteristic j on patient outcome k and Zi is a Jx1 vector of observable characteristics of hospital i that are thought to be related to patient outcomes. Ω is the variance-covariance matrix in the hospital-level quality parameters summarizing the relationships across different dimensions of hospital quality.

One can estimate β using a weighted least squares regression of Y on Z, where one weights by ni, the number of patients in hospital i.

One can estimate Ω as follows:

• Ω=N-1 Σ {(Yi) – Ziβ)'(Yi) – Ziβ) – Vi}
• Ω=Var(Yi) – mean(Vi)

Here, Vi is the mean sampling-error covariance matrix. The i-j element of Ω is weighted by the product of ni and nj. In certain cases, the variance covariance matrix may not be positive-semi definite (e.g., correlations are greater than 1). If this occurs, one can replace the corelation matrix with the nearest positive-semi-definite correlation matrix.

Step 4: Bayesian Composite Score
The estimates Yi are a noisy measures of hospital quality. Further, on must determine how much each measure should contribute to the overall score. Using an empirical Bayesian approach, the weights depend on both the signal (&Sigma) and noise (Vi) variances. Thus, the final hospital score is:

• mi = YiWi + Ziβ(I-Wi)

where I is a KxK identity matrix and Wi is a KxK weighting matrix estimated by:

• Wi = (Ω+Vi)-1Ω

In other words, the weight is the ratio of the signal variance to the total variance. Thus, estimated composite score places more weight on a hospital’s own outcome (Yi) when the signal is high, but shrinks back toward a conditional mean (Ziβ) when the signal ratio is low.

Attractive Features

1. The composite measure incorporate information in a systematic way from many quality measures into a single outcome.
2. The parameters are consistently estimated as the number of hospitals increases.
3. The estimates maintain many aspects of the existing Bayesian approaches while simplifying the complexity of the estimation.

Drawbacks

There are some drawbacks to this approach. For instance, this method does not incorporate subjective weights. In the example, a hospital has multiple mortality measures. Since the there is no a priori reason to weight one mortality measure more than another, this approach is useful. However, hospitals may be evaluated based on process of care, patient outcome, and patient satisfaction measures simultaneously. One may wish to place more importance on certain measures than others. However, the approach described above may be useful for calculating domain-level composite scores and then one can aggregate the domain level composites using subjective weights.