What are the best practices for conducting comparative effectiveness research in the real-world? One proposed best practice guildelines are the Good Research for Comparative Effectiveness (GRACE) guidelines. However, most studies do not follow these guidelines. A paper by Dreyer, Bryant and Velentgas (2016) assembled 28 observational comparative effectiveness articles published from 2001 to 2010 that compared treatment effectiveness and/or safety of drugs, medical devices, and medical procedures. They found
The best predictors of quality included the following: use of concurrent comparators, limiting the study to new initiators of the study drug, equivalent measurement of outcomes in study groups, collecting data on most if not all known confounders or effect modifiers, accounting for immortal time bias in the analysis, and use of sensitivity analyses to test how much effect estimates depended on various assumptions.
What are the GRACE guidelines? I list these below.
DATA
- D1: Were treatment and/or important details of treatment exposure adequately recorded for the study purpose in the data source(s)?
-
D2. Were the primary outcomes adequately recorded for the study purpose (e.g., available in sufficient detail through data sources)?
-
D3. Was the primary clinical outcome(s) measured objectively rather than subject to clinical judgment (e.g., opinion about whether the patient’s condition has improved)?
-
D4. Were primary outcomes validated, adjudicated, or otherwise known to be valid in a similar population?
-
D5. Was the primary outcome(s) measured or identified in an equivalent manner between the treatment/intervention group and the comparison group?
-
D6. Were important covariates that may be known confounders or effect modifiers available and recorded? Important covariates depend on the treatment and/or outcome of interest (e.g., body mass index should be available and recorded for studies of diabetes; race should be available and recorded for studies of hypertension and glaucoma).
METHODS
-
M1. Was the study (or analysis) population restricted to new initiators of treatment or those starting a new course of treatment? Efforts to include only new initiators may include restricting the cohort to those who had a washout period (specified period of medication nonuse) before the beginning of study follow-up.
-
M2. If 1 or more comparison groups were used, were they concurrent comparators? If not, did the authors justify the use of historical comparison groups?
-
M3. Were important confounding and effect-modifying variables taken into account in the design and/or analysis? Appropriate methods to take these variables into account may include restriction, stratification, interaction terms, multivariate analysis, propensity score matching, instrumental variables, or other approaches
-
M4. Is the classification of exposed and unexposed person-time free of “immortal time bias,” i.e., “immortal time” in epidemiology refers to a period of cohort follow-up time during which death (or an outcome that determines end of follow-up) cannot occur
-
M5. Were any meaningful analyses conducted to test key assumptions on which primary results are based (e.g., were some analyses reported to evaluate the potential for a biased assessment of exposure or outcome, such as analyses where the impact of varying exposure and/or outcome definitions was tested to examine the impact on results)