intake have, ceteris paribus, low raw-scores at KS2, and that the ‘value-added’ scores
do almost nothing to overcome this clear pattern. Therefore, these value-added scores
are not, as the DfES has claimed, independent of actual levels of raw-score
attainment.
An example of why this matters comes from the revised OFSTED light-touch school
inspections in England. Inspectors from OFSTED are spending less time in schools,
and making fewer lesson observations, on each inspection. The reduced reliance on
primary observation has, in the reports of some school leaders, led to an increased
reliance on prior value-added analyses of the schools (Bald 2006). This has led to
clear anomalies and ‘bizarre judgements’ such as a school being judged largely ‘good’
or ‘outstanding’ on observation, but being reported as merely ‘satisfactory’ because
the best outcome allowable by OFSTED was constrained by a relatively low prior
value-added score (Mansell 2006a, Slater 2006). The increased reliance on
contextualised value-added (at time of writing), which is very sensitive to exclusions
for example, leads to some schools getting lower than expected inspection results
(Mansell 2006b). As Paterson pointed out as early as 1997, pupil-level regression of
the kind now in use by the DfES is a fascinating and productive research tool, which
can be used to inform professional debate. But it should not yet be used directly as a
tool for pupil, teacher or school assessment.
References
Bald, J. (2006) Inspection is now just a numbers game, Times Educational
Supplement, 26/5/06, p.21
DfES (2006) Value-added technical information,
http://www.dfes.gov.uk/performancetables/primary 03/p5.shtml, accessed 15th
March 2006
Gigerenzer, G. (2004) Mindless statistics, American Economic Review, 33, 5, 587-606
Gorard, S. (2000) 'Underachievement' is still an ugly word: reconsidering the relative
effectiveness of schools in England and Wales, Journal of Education Policy, 15,
5, 559-573
Gorard, S. (2001) International comparisons of school effectiveness: a second
component of the 'crisis account'?, Comparative Education , 37, 3, 279-296
Gorard, S. (2002) Fostering scepticism: the importance of warranting claims,
Evaluation and Research in Education, 16, 3, 136-149
Gorard, S. (2005) Academies as the ‘future of schooling’: is this an evidence-based
policy?, Journal of Education Policy, 20, 3, 369-377
Gorard, S. (2006a) Towards a judgement-based statistical analysis, British Journal of
Sociology of Education, 27, 1, 67-80
Gorard, S. (2006b) Using everyday numbers effectively in research, London:
Continuum
Gorard, S. (2006c) Value-added is of little value, Journal of Educational Policy, 21,
2, 233-241
Lunt, P. (2004) The significance of the significance test controversy: comments on
‘size matters’, American Economic Review, 33, 5, 559-564
Luyten, H. (2006) An empirical assessment of the absolute effect of schooling:
regression-discontinuity applied to TIMSS-95, Oxford Review of Education, 32,
3, 397-429