5.3 Difference-in-difference Analysis
This method contrasts the growth in the variable of interest between a treatment group and a
relevant control group. This approach requires that participants be tracked over time, beginning
with a pre-intervention baseline survey, followed up by subsequent surveys of participants and
non-participants. The estimate of treatment impact is given by the difference in outcomes
for individuals before and after the intervention, and then the difference between that mean
difference for participants and non-participants. The key assumption underlying this method
is that selection bias is invariant over time.
Difference-in-difference estimates may be appropriate where an argument can be made that
outcomes would not have been different over time in regions that received the programme
compared to those that did not, had the programme not been introduced. If this case can
be made, then one can compare differences in the growth of the variable of interest between
programme and non-programme areas. However, this approach requires long-standing time-
series data in order to ensure that the groups are as similar as possible, and to pro ject that
they would have behaved similarly without the presence of the treatment. Moreover, one must
be certain that no other programmes were introduced concurrently, and that a region may has
not been affected by a time persistent shock that may manifest as a treatment effect (Bertrand,
Duflo and Mullainathan, 2003).
A further benefit of the difference-in-difference approach is that it can be used to address
bias in the estimates obtained from a randomized evaluation study if there has been selective
compliance or attrition, and they minimize bias that might arise due to measurement error.
Even so, there can be additional biases to the standard errors from using this method. At
the time of the baseline survey, it may not be apparent which individuals will participate in
the programme and which will not, and hence, the researcher must make their best guess
when drawing a random sample for the baseline survey. This may hold implications for sample
representativeness ex-post, so to minimize this source of possible bias, the researcher should
use any information they have about the details and context of the proposed programme to
help guide their sampling choices, and then over-sample from the likely participant group, in
order to ensure a good comparison group. Secondly, the assumption that selection bias is
unchanging over time may also be problematic, especially if changes in outcome variables due
to the intervention are a function of initial conditions which influenced progamme assignment
to begin with (Ravallion, 2008; Jalan and Ravallion, 1998). In other words, if poor regions
are targeted for intervention because of their poverty status, and if treatment impact depends
on the level of poverty, this will bias impact estimates. Consequently, the researcher needs to
control for initial conditions in deriving their impact estimates (Ravallion, 2008).
Since difference-in-difference estimates require longitudinal data, the researcher will have to
consider the trade-off between relying on a single survey estimate and utilizing PSM to find a
comparable control group, as opposed to incurring the cost of tracking individuals over time in
order to be able to utilize difference-in-difference estimators. Ravallion (2008) argues that such
a decision should be made based on how much is known ex ante about programme placement.
If a single cross-sectional survey is able to provide comprehensive data in this regard, then this
may be a more feasible alternative that collecting longitudinal data.
The difference-in-difference approach has been successfully used to provide estimates of
impact in a number of interventions. For example, Thomas et al (2003) show that iron sup-
plementation amongst iron-deficient individuals, males in particular, yields improved economic
productivity, as well as improved psycho-social and physical health outcomes. Galiani et al
(2005) use difference-in-difference estimates to show that the privatization of water services in
Argentina reduced child mortality.
6 External validity
The non-experimental methods reviewed above may assist in dealing with concerns that arise
over the internal validity of impact estimates based on randomization alone. However, in addi-
11