4.4 Pipeline Matching
Delays in implementation of a programme may also facilitate the formation of a comparison
group. In these studies, usually termed pipeline studies, the control group comprises those indi-
viduals who have applied for a programme but not yet received it (for example, see Chase, 2002;
Galasso and Ravallion 2004). For example, in PROGRESSA, one third of eligible participants
didn’t receive a transfer for 18 months during which they formed control group. Thereafter,
they were phased into the programme. Similarly, in a Kenyan deworming programme studied
by Miguel and Kremer (2004), while 75 schools were chosen to participate, in the first year of
the study, only 25 schools were treated, while the other 50 schools formed the control group. In
year two, a further 25 schools were phased into treatment and by the third year, all 75 schools
were receiving the treatment. The advantage of this method is that it deals with selection
bias even on unobservable characteristics to some extent, since the successful applicants not
yet receiving treatment will be very similar in most respects to beneficiaries of the programme.
A key assumption though in pipeline studies is that the timing of treatment be random given
application.
4.5 Comparison with Randomization
The evidence on whether PSM methods and RE methods produce the same results is somewhat
mixed. Agodini and Dynarski (2004) find no consistent evidence that PSM can replicate RE
results of school dropouts programmes in the US. In contrast, work by Heckman et al (1997a,
1998) and Diaz and Handa (2004) suggests that PSM works well as long as the survey instrument
used for measuring outcomes is identical for treatment and control participants (Diaz and
Handa, 2004; Heckman et al, 1997a, 1998). A recent study by Diaz and Handa (2007) shows that
with the collection of a large number of observables, propensity score matching can approximate
RE results.
Hence, the success of PSM hinges critically on the data available, as well as the variables
used for matching. The key challenge for PSM methods is to identify all potentially relevant
covariates and differences between treatment and control groups. If treatment is assigned on
the basis of an unobservable trait, then the estimates obtained will be biased.
The choice of variables should be based on some theoretical reasoning and/or facts about the
intervention and its context, as well as any relevant socio-economic and political considerations.
In this regard, additional qualitative work may be useful (Jalan and Ravallion, 2003b; Godtland
et al, 2004). Ex-post, it is important to test for differences in the covariates between treatment
and comparison groups to ensure that covariate balance can be achieved (Smith and Todd,
2005a). Importantly, then, PSM estimates will be limited to a matched sample and not the
full sample. However, matched sample estimates tend to be less biased and more robust to
misspecification error (Rubin and Thomas, 2000).
5 Other Non-Experimental Methods
Two potential problems remain unexplored with the propensity score approach. The first,
discussed already, concerns the possibility of remaining omitted variable biases. The propensity
score regression uses proxies for the unobserved/omitted variables under the assumption that
the omitted variables are redundant in explaining treatment assignment once their proxies are
accounted for. Matching methods are of little use when such proxies do not exist. Observational
studies - even those based on quasi-experimental designs - with this type of problem are said
to exhibit selection on unobservables. This section deals with three widely used alternatives
to randomization and/or matching when we do not observe the full set of variables influencing
treatment status: instrumental variable estimation, regression discontinuity approaches and
double-differencing.