A common method for measuring the effect of policy interventions is the difference in difference (DiD) approach. In essence, one examines the change in outcomes among observations subject to the policy intervention and compare them agains observations that were not eligible for the policy intervention.
A key assumption for this approach to be valid is that the trends in both groups are the same and any deviation from this trend is due to the policy intervention. This assumption is known as the parallel trends assumption.
A study by Krief et al. (2015) proposes an alternative approach: the synthetic control method. The authors describe this method as follows:
The central idea behind the synthetic control method is that the outcomes from the control units are weighted so as to construct the counterfactual outcome for the treated unit, in the absence of the treatment. More precisely, a synthetic control unit is defined as the time-invariant weighted average of available control units, which prior to the intervention have similar pre-intervention characteristics and outcome trajectory to the treated unit. In contrast to the DiD method, the synthetic control method allows the effects of observed and unobserved predictors of the outcome to change over time, while assuming that pre-intervention covariates have a linear relationship with outcomes post-treatment.
The authors apply this method to the Advancing Quality (AQ) initiative in England:
The AQ scheme was the first hospital-based P4P programme in the UK, introduced in October 2008 for all hospitals in the North West Strategic Health Authority. The AQ scheme aimed to improve hospital performance on a number of clinical processes by rewarding hospitals for achieving quality targets. As the US Hospital Quality Incentive Demonstration scheme, the AQ was initially based on a ‘tournament’ system, in which bonuses were paid to the top performers. The programme required participating hospi- tals to collect and submit data on 28 quality measures of patient care across five clinical areas: pneumo- nia, heart failure, acute myocardial infarction, hip and knee surgery, and coronary artery bypass grafting. In the first year of the programme, bonuses equal to 2–4% of their revenue were paid to hospitals, which reported quality scores in the top two quartiles.
They find the following:
In contrast to the original DiD analysis, the synthetic control method reports that, for the incentivised conditions, the P4P scheme did not significantly reduce mortality and that there is a statistically significant increase in mortality for non-incentivised conditions. This result was robust to alternative specifications of the synthetic control method.
- Kreif, N., Grieve, R., Hangartner, D., Turner, A. J., Nikolova, S., and Sutton, M. (2015) Examination of the Synthetic Control Method for Evaluating Health Policies with Multiple Treated Units. Health Econ., doi: 10.1002/hec.3258.