4 mhbounds
could differ in their odds of receiving the treatment by as much as a factor of 2. In this
sense, eγ is a measure of the degree of departure from a study that is free of hidden bias
(Rosenbaum, 2002).4
2.2 The MH Test Statistic
For binary outcomes, Aakvik (2001) suggests using the Mantel and Haenszel (MH,
1959) test statistic. To do so, some additional notation is needed. We observe the
outcome y for both participants and non-participants. If y is unaffected by different
treatment assignments, treatment d is said to have no effect. If y is different for different
assignments, then the treatment has some positive (or negative) effect. To be significant,
the treatment effect has to cross some test statistic t(d, y). The MH non-parametric test
compares the successful number of individuals in the treatment group against the same
expected number given the treatment effect is zero. Aakvik (2001) notes that the MH
test can be used to test for no treatment effect both within different strata of the sample
and as a weighted average between strata. Under the null-hypothesis of no treatment
effect, the distribution of y is hypergeometric. We notate N1s and N0s as the numbers
of treated and non-treated individuals in stratum s, where Ns = N0s + N1s . Y1s is
the number of successful participants, Y0s is the number of successful non-participants,
and Ys is the number of total successes in stratum s. The test-statistic QMH follows
asymptotically the standard normal distribution and is given by:
|Y1 - ∑S=1 E(Y1S)I- 0.5 = IYi - ∑S=1(NNsYs)|- 0.5
S^s tv ʌ ∖" NisNosYs(Ns-Ys)
V∑s = 1 Var(Yls) S= ∑s=1 N2(Ns-I)
To use such a test-statistic, we first have to make the individuals in the treatment
and control groups as similar as possible, because this test is based on random sampling.
Since this is done by our matching procedure, we can proceed to discuss the possible
influences of eγ > 1. For fixed eγ > 1 and u ∈ {0, 1}, Rosenbaum (2002) shows that the
test-statistic QMH can be bounded by two known distributions. As noted already, if
eγ = 1 the bounds are equal to the ‘base’ scenario of no hidden bias. With increasing eγ,
the bounds move apart reflecting uncertainty about the test-statistics in the presence
of unobserved selection bias. Two scenarios are especially useful. Let Q+MH be the
test-statistic given that we have overestimated the treatment effect and Q-MH the case
where we have underestimated the treatment effect. The two bounds are then given by:
Q+MH =
IYi - ΣS=1 E+I- 0.5
√∑s=ι Var(Es+)
(5)
4. A related approach can be found in Manski (1990, 1995) who proposes ‘worst-case bounds’ which
are somewhat analogous to letting eγ → ∞ in a sensitivity analysis.