This weekend, my fire alarms kept going off. I called the local fire department who came out and they checked everything out. They arrived quickly, were extremely friendly and explained everything clearly. The did some tests and said all was well. If I were asked to rate their quality, I would rate them very high. However, this rating would be based largely on their customer service (i.e., how nice they are, how quickly they arrived); I do not have the fire safety expertise to determine if the tests they did were the right ones or if they examined all the correct locations in my house. Perhaps I could measure quality by examining whether my house burned down a few weeks/months from now, but that is a pretty crude measure, and may not take into account any subsequent risk factors that could materialize after they left.
The issue of rating firefighter quality is similar to the issues of rating quality of care for health care providers. Patient-reported quality measures are largely based on bedside manner and amenities; patients don’t have the scientific basis to evaluate how good a job a physician did. Outcome measures would solve the problem, but there are numerous challenges including appropriate risk adjustment and also determining which provider receives the credit (or blame) for the care of a given patient when the patient visits multiple providers.
A similar issue arises when evaluating health plans. A Health Affairs blog post by Soleil Shah and Eric Sun from last week summarize some of the issues:
Research that assesses the impact of the star ratings program on patient outcomes is limited, perhaps because of an implicit assumption that MA plans receiving higher star ratings achieve better outcomes for their enrollees. The little research that does exist shows that star ratings do not predict which MA plans reduce beneficiary mortality on average…
Beneficiaries may find administrative measures important because they reflect the quality of the plans’ customer service. However, since administrative measures comprise such a significant proportion of the star ratings measures, plans may be able to achieve higher star ratings by focusing mostly on administrative improvements that are immediately within the organization’s reach, rather than seeking to improve enrollee health.
The authors also discuss that health plans that reduce readmission rates–a very common quality measure for hospitals–may not show any improvements in post-discharge mortality rates (Dharmarajan et al. 2017); in fact, some studies have even shown that reducing readmissions increases post-discharge mortality (Samarghandi et al. 2019; Gupta et al. 2018).
Further, there is the problem of Goodhart’s Law which states:
When a measure becomes a target, it ceases to be a good measure