have also argued that randomizing eligibility could be coupled with instrumental variables. This
type of quasi-experimental design works quite well when the eligibility rules of the program are
not compromised during implementation. When eligibility is correlated with outcomes however,
the analyst might be forced to look for IVs elsewhere. In such instances, detailed knowledge of
the institutional environment as well as the administration of the program could prove useful
in constructing alternative IVs.
While experimental designs are always desirable when evaluating health impacts, they are
not a panacea to all data problems. Identification strategies that rely solely on randomizing
treatment assignment have to contend with the problem of selective compliance and attrition
from both the treatment and control groups. Guarding against such problems will often involve
combining methods and/or building into studies additional rules concerning participation This
may require conditionality to be imposed on participants, as was the case with PROGRESSA, or
may require significant investments of time and energy by the research team in establishing good
working relationships with survey participants, as well as the ability to maintain contact over
time in the case of longitudinal studies. Moreover, interventions that are simple to administer
and for participants to adhere to have a stronger chance of success than interventions that
require a complex bureaucratic structure in order to be administered, or where the intervention
requires significant education or time commitment on the part of participants.
Where health investments are made at early ages, longitudinal data is ideally required to
assess longer term health impacts on productivity. When the collection of longitudinal data is
not possible, intermediate indicators of long term success should be collected in cross-sectional
surveys. Given the costs involved in data collection exercises, collecting such data might best
be accomplished by partnering with medical randomized controlled studies.
In sum, the evaluation problem is really one of missing data. The credibility of impact esti-
mates will only ever be as good as the data upon which they are based. Randomized evaluations
that do not control adequately for selective compliance and attrition will necessitate the use
of NX methods as well as substantial collection of good quality data, including administra-
tive and process data to provide important insights about the context and inner workings of
the programme, so that additional analytical options are available if important aspects of the
experimental design of a program are prone to unravelling.
A Appendix
A.1 Derivation of the Wald Estimator
Our derivation follows Wooldridge (2002). The the numerator can be written as PiN=1 Pi(yi -
ÿ) = PN=i Piyi - (PN=i Pi)y = Niyi - Niy = N1(y1 — ÿ) where Ni = PN=i Pi is the number
of observations in the sample with Pi = 1 and ÿi is the average of the yi over the observations
with Pi = 1. Next write y as a weighted average: y = NNN-yÿ + NN-yi, where the zero/one
subscripting refers to treatment and control. After some algebra it can be shown that yÿi - yÿ =
( N NN )yi — (Nv )yo = (NN)(yi — yo). So the numerator of the IV estimate is (NN1 )(yi - yo).
The same argument shows that the denominator is ( N'NN'1 )(Ti — T1 ). Taking the ratio completes
the proof.
20