programme. 2 Institutional or political factors that delay randomized assignment may also
promote selective attrition (Ravallion, 2008; Heckman and Smith, 1995).
In each case, this leads to a difference between the actual allocation and the intended
allocation, and to the extent that this is not controlled for, will result in biased estimates of
impact. Program design should try to anticipate this, and put processes in place to minimize
attrition that might occur. In the Balsakhi program, for example, Banerjee et al (2007) ensured
that when students did not appear at schools, the data collectors went to their homes in order
to ensure that they collected the data and could track the individuals. This resulted in lower
attrition from the study.
However, when attrition or selective compliance is present, researchers typically deal with
these kinds of problems through intention to treat models (Imbens and Angrist, 1994), whereby
the differences in outcomes for treatment and control groups (as per the original assignment)
are scaled up by dividing the difference in outcomes by the difference in the probability of
actually receiving treatment in the two groups. This gives an estimate of the average treatment
effect for those induced to participate by randomization (Ravallion, 1995). Importantly, this
differs from the average treatment effect in the population as a whole, where this kind of
selective compliance does not occur. Rather, intention to treat models account for the fact
that individuals who anticipate benefiting from a programme may be the most likely to take
advantage of it. Arguably, these may be precisely the kinds of individuals that policy makers
are most interested in.
A second important consideration in critically evaluating impact estimates is the presence
of externalities generated by the programme or intervention itself. Externalities may plague
the credibility of impact evaluation estimates if policy makers or aid agencies reallocate their
spending priorities to compensate some communities or individuals for their non-participation
in the intervention. This is difficult to know but vitally important to keep track of, since
to the extent that such re-allocation of spending priorities may occur, this will influence the
magnitude of the impact estimates. In addition, to the extent that an intervention confers
positive externalities on individuals outside of the treatment group, failure to account for these
externalities may lead to an under-estimate of the intervention impact. For example, in the
Miguel and Kremer (2004) study of mass deworming programmes in Kenya, they argue that a
randomized intervention targeted at the individual level in which some children within the same
school were treated while others were not would result in a serous underestimate of treatment
effects, since the control children would enjoy reduced disease transmission by virtue of being
in contact with treated children. 3 Hence, they chose to randomize at the school level.
The presence of externalities generated by an intervention thus points to the need for careful
thought to be given about the level at which randomization should occur as well as the need to
collect detailed information to control for these possible spillovers in arriving at credible impact
estimates. For example, despite randomizing at the school level, Miguel and Kremer (2004)
still find evidence of positive spillovers in the deworming pro ject in that children attending
neighbouring non-treatment schools also enjoy reduced incidence of intestinal worms through
reduced transmission of disease when interacting with children in treatment schools. Since
detailed spatial information about the distance between schools was collected as part of the
evaluation survey, Miguel and Kremer (2004) are able to utilize this data to control for these
spillovers.
Thus, the choice of observational unit should reflect likely spillover effects (Ravallion, 2005).
2 Not only does partial compliance by individuals holds implications for the credibility of the impact estimates,
but it also holds implications for sampling. For example if there were to be approximately an 80% level of
compliance by the treated group, then the entire sample would have to be approximately 50% larger in order to
get commensurable effects relative to a group that had 100% compliance. Thus, in designing a study, one has
to weigh up the costs and benefits between a study that requires high compliance rates but lower sample sizes
relative to a program with lower compliance levels but requiring larger sample sizes in order to have the same
level of power from the results. It may happen that a more comprehensively considered program with higher
(predicted) compliance levels might in fact be less expensive to implement than a project with lower compliance
levels.
3 Similarly, failure to account for negative externalities imposed by an intervention would result in an over-
estimate of the programmes benefits.