for a calculation how impressive is this core of 41% of pupils with consistent
responses over time about staying on? In year 7, Croll reports that over 86% of pupils
who express an intention say they will stay on, and 88% say this in each of the next
three years. If these reports were actually meaningless in relation to their future
intentions, then after four interviews we would expect to find that 59% of pupils (0.86
times 0.88 times 0.88 times 0.88) had ‘consistently’ reported an intention to stay on.
Again, this result is a product of the initial imbalance in the responses, and tells us
nothing about the link between intention and revealed behaviour. If we accept that
some pupils may have had five interviews, that intention to stay on drops to 84% in
year 11, and that the proportion of ‘don’t knows’ reduces over time, then my result
may drop to less than 59% consistent responses (even though just by chance). But
Croll only found 41% of pupils consistently reporting an intention to stay on. This can
hardly be more than chance, and certainly would not lead one to the strong conclusion
that intentions were stable and useful predictors of behaviour.
Summary
In a skewed situation as we face when looking at both reported intentions and
revealed patterns of staying on, it behoves us to consider what the ‘accuracy’ of
intentions would be if they really meant nothing at all. This is just an example of the
standard warranting principle - if the conclusion were false how else could we
explain the evidence found? (see Gorard 2002). The fact that Croll does not use the
warrant principle, as a matter of course, to address any of this is worrying, as is the
fact that none of the referees picked it up before publication. In the same issue of
BJES, Coe (2010) makes an understandable plea for the kind of scepticism underlying
the warranting principle when we consider causal claims based on passive designs. He
cautions against what I have called the post hoc dredging of sullen datasets (Gorard
2006, p.76). But the same thing also applies to the more descriptive associations
presented by Croll here. This is not something that the complex techniques derived
from sampling theory can address at all - it is about care and sceptical judgement in
the use of numeric (and indeed any) data. Otherwise, the consequences, if taken
seriously, could mean research, policy and practice attention being diverted from the
understanding of choice behaviour into a blind alley.