Should we trust difference-in-difference estimators?

Difference-in-difference (DD) estimators are appropriate when the interventions are as good as random, conditional on time and group fixed effects.  Although much of the debate in the economic literature regarding DD estimators focuses on the possible endogeneity of the interventions, there is another problem with DD: underestimated standard errors.

A paper by Bertrand, Duflo and Mullainathan (2003) give three reasons why the DD OLS standard errors are often severely underestimated.

  1. Fairly Long Time Series. A survey of DD papers finds an average of 16.5 periods.
  2. Serial Correlation.  The most commonly used dependent variables in DD estimation are typically highly positively serially correlated.
  3. Limited Variation in Treatment Variable.  The treatment variable typically changes little within a state over time, oftentimes only once during the study period.   These three factors reinforce each other so that the standard

How can one address these issues?  The authors propose a number of methods.

  • Block Bootstrap.  This bootstrap which maintains the auto-correlation structure by keeping all the observations that belong to the same group (e.g., state) together.  This method performs well when the number of states is large enough.
  • Aggregate data into pre/post time periods.  Taken literally, however, this solution will work only for laws that are passed at the same time for all the treated states. One can regress the outcomes of interest on state fixed effects, year dummies and relevant covariates.  One can then divide the residuals of the treatment states only into two groups: residuals from years before the laws, and residuals from years after the laws.  The estimate of the laws’ effect and its standard error can then be obtained from an OLS regression in this two-period panel.
  • Unrestricted Covariance Matrix.  One can allow the unrestricted covariance structure to vary over time within states, with or without making the assumption that the error terms in all states follow the same process.  This produces a block diagonal variance-covariance matrix of the error term, with 50 identical blocks of size TxT (where T is the number of time periods). This technique works well when the number of groups is large (e.g. 50 states) but fare more poorly as the number of groups gets small.



Leave a Reply

Your email address will not be published. Required fields are marked *