Reconsidering the value of pupil attitudes to studying post-16: a caution for Paul Croll

Just because most pupils say they will stay on and most pupils do stay on, using
intention to ‘predict’ participation will be 66% ([519+34]/843) accurate even though
in my analysis the two things are designed to be completely unrelated. This means
that Croll’s figure for the accuracy of intentions (see above) is only 10% better than
pure chance. Presumably it is possible to imagine around 10% of each cohort already
knowingly on a trajectory towards school avoidance or long-term academic success.
This would mean that Croll is wrong to say that expressed intentions are good
predictors of later participation more generally. Intentions are rather weak predictors,
if indeed they are predictors at all (see below).

Looked at another way, by how much can using pupil year 7 intentions help to make a
more accurate prediction of whether they stayed on post-16 than we could make
without using them? Since 72% of pupils stay on in fact, if we predicted that all year 7
pupils would stay on then we would be right in around 72% of cases. This is even
better than the 66% chance model above, and makes Croll’s accuracy of 76% look
less impressive again. Using intentions would only improve the base prediction by 34
pupils (4% of 843). If this were a logistic regression model, with a binary outcome
variable of participation or not, it would fail because both the outcome and the
predictor are so skewed. It is a shame that Croll did not attempt such a model, or
similar, and that the journal reviewers for BJES did not request it, since it would
quickly reveal the flaw in the claims about intentions being 76% accurate. Intentions
improve the accuracy of predictions by 4% at best.

Croll describes a standard set of potential background determinants associated with
post-16 participation like prior attainment, sex, and family background. These could
have been used in a model to predict participation, and would probably improve
predictions by more than the 4% increase on the baseline due to using intentions (see

<<   <   2   3   4   5   6   7   8   >   >>