Tuesday, 17 December 2013

Myths of Assessment

by John Yandell, Institute of Education

I want to explore five aspects or strands of the way in which we tend to think about assessment. 

In calling these strands myths, I am suggesting two things. First, that they are both powerful and deeply embedded in our assumptions about assessment: they have become, in other words, common sense.  Second, that they are, in important ways, untrue and unhelpful, obstacles that make it harder for us to arrive at more accurate and adequate understandings of assessment. 

I should make it clear, too, that what I mean by assessment here is not formative assessment or assessment for learning,  but the regimes of high-stakes, summative assessment that figure so prominently in the landscape of schooling in this country.

The effect of high-stakes assessment in distorting and impeding education has a history, more or less as long as the history of state education itself. In 1911, Edmond Holmes, an HMI who had resigned in disgust from the inspectorate, wrote a wonderful little book entitled What is and what might be?  The first half of the book speaks directly to us across the intervening century. Holmes describes a system in which teachers spoon-feed their pupils, a system in which there is precious little room for genuine learning to take place:

Why is the teacher so ready to do everything (or nearly everything) for the children whom he professes to educate?  One obvious answer to this question is that for a third of a century (1862-1895) … "My Lords" required their inspectors to examine every child in every elementary school in England on a syllabus which was binding on all schools alike. In doing this, they put a bit into the mouth of the teacher and drove him, at their pleasure, in this direction and that. And what they did to him they compelled him to do to the child (Holmes 1911: 7).

Holmes identifies the effects of a centralised curriculum, enforced through testing and through inspection.  Within such a system, there is no space for creativity, no space for dialogue, no space to explore and exploit the interests and experiences that the learners bring with them.  Schooling is a transmission process, driven by fear. 

What is also significant about Holmes’s account, though, is that he is writing sixteen years after the ending of the system of payment by results.  What Holmes understood, because he had seen the evidence in the elementary schools he had visited across the country, was that the pernicious effects of such systems of control lived on after the systems themselves had been abandoned.

Within this system, Holmes identified the crucial effect of assessment:

How did the belief that a formal examination is a worthy end for teacher and child to aim at, and an adequate test of success in teaching and in learning, come to establish itself in this country? And not in this country only, but in the whole Western world? In every Western country that is progressive and "up to date," ... the examination system controls education, and in doing so arrests the self-development of the child, and therefore strangles his inward growth.

What is the explanation of this significant fact?  .... The Western belief in the efficacy of examinations is a symptom of a widespread and deep-seated tendency – the tendency to judge according to the appearance of things, to attach supreme importance to visible "results," to measure inward worth by outward standards, to estimate progress in terms of what the "world" reveres as "success" (Holmes 1911: 8-9).

Holmes was describing an education system in which the tail of assessment wagged the dog of learning. How familiar.

Myth 1: learning is linear
Since the imposition, twenty years ago, of the National Curriculum and its attendant levels and level descriptors, it has become increasingly hard to challenge the assumption that learning happens in predictable, identifiable and incremental stages.  Increasingly, too, the levels of the National Curriculum are broken down into sub-levels, in an attempt to describe ever more precisely the progress that learners have made – and also to set ever more precise targets for their future progress. The attainment of literacy or numeracy becomes inextricably associated with achieving a level four before the end of primary education, while press and politicians are quick to make headlines with the assumption that those who have not been awarded a level four are therefore illiterate, innumerate.

The problem with all of this is that it is not true.  At best, the National Curriculum level descriptors are an attempt to describe what progression in a subject might look like. But learning itself is a much messier, more complicated business than the linear scale of levels or GCSE grades would suggest. 

What happens day by day in the classroom depends on factors other than the learners’ existing or target levels: it depends on their interests and experiences beyond school, and whether they can make connections between these interests and experiences and the school curriculum.  It depends on the learners’ motivation.

Even within subjects, such as maths, where learning seems more easily accommodated to the paradigm of linear progression, learners’ understanding of, say, number can be at a markedly different stage of development from their understanding of shape, space and measure. 

Within subjects such as English, where development more obviously involves social and emotional aspects alongside intellectual processes, and where learners’ development as speakers and listeners does not bear any simple or constant relationship to their development as readers and writers, the attempt to place their progress at a single point on a single linear scale is neither meaningful nor educationally justifiable. 

What this amounts to is a bad case of reification, more infectious and far more damaging than swine flu: levels that were, at most, ways of gesturing broadly at progression have been transformed under the pressure of particular forms of accountability into things, as if levels had the same solidity and materiality as, say a child’s shoe size. 

So one encounters with wearying regularity children who announce that they “are” a level six, or a level three (the former with pride verging on smugness, the latter with an air of resignation, a meek acceptance that literacy, or even learning, is not really their thing). 

Even more worryingly, the reification process has affected the way that teachers talk about their pupils – so “she’s a level five” or “he’s a level four” are now not so much shorthand expressions, standing for more complicated pictures of a learner’s development, as bald statements of fact.

Myth 2: learning is context-independent
The idea that learning happens in a vacuum, as it were, and hence that learning can be measured in isolation from the context in which it happened, is closely linked to the myth of linearity.  It is another aspect of the same reductive approach to learning, an approach that seeks to isolate sub-skills and then assess whether the sub-skill has been acquired without any reference to the contexts in which such skills might be used and developed. 

Once again, it is an attempt to sidestep the messy contingencies of real learning, substituting in its place the thin abstractions of the easily transmitted and easily measured.

Always and everywhere, classrooms are populated by real people with particular histories, experiences, cultures.  Learning involves these people interacting with each other and with particular materials – with particular problems, particular texts, particular resources.

Myth 3: learning is individual
Deeply implicated in the history of schooling in this country is the assumption that learning is the property of an individual, that learning happens inside a single learner’s head.  It is a myth that is nurtured by, and helps to sustain, the sense of self that is central to bourgeois culture and values.  Plagiarism, the Manichean other of intellectual property, is the cardinal sin within the religion of schooling precisely because it entails a transgression of this article of faith, learning as the property of the individual. 

With glorious circularity, we know that learning is individual because the assessment regimes constantly demonstrate that this is the case.  Assessment, predicated on the commonsense assumption that learning happens in an individual’s head, proceeds to provide opportunities for the individual to demonstrate this learning, in circumstances – such as the exam hall – where normal human interactions are absolutely forbidden, and then offers conclusive proof of the validity of the procedure by arriving at differential assessments for different individuals: to one a level five, to another a grade A, and so on.

It is in such routines of assessment that the role of education as a sorting mechanism becomes most obvious.  Assessment separates sheep from goats, high-fliers from also-rans, leaders of men from hewers of wood.  It underpins the notion of meritocracy and it sustains the illusion that social justice can be achieved through social mobility.

The myth of the isolated individual, the learner as examination candidate, filters out all that we know about the reality of distributed learning, all we know about learners as irreducibly social beings, situated in history and in culture. 

Myth 4: the assessment of learning is objective and reliable
The dominant discourses of schooling are based on the myth of reliable assessment.  In the creation of this myth, one of the things that happens is that assessment processes assume a kind of autonomy, independent of human agency. 

But assessment is always a social practice.  It always involves the exercise of judgement.  It is always conducted for specific purposes by particular people.  Only in the most trivial cases is assessment merely a matter of measuring.  And reliability comes at a cost: the more reliable a test, the less information it can provide about the breadth of a child’s learning and development.  There is, in other words, an inverse relationship between reliability and validity (Alexander 2010: 320-1).

Myth 5: high stakes assessment is vital for accountability
The argument here is not over whether schools should be accountable – of course they should – but over the role of testing in achieving accountability. The whole machinery of National Curriculum levels and sub-levels does not make it easier for parents and carers to find out about their children’s progress; it is a barrier in the way of communication between teachers and parents, a professionalist jargon impenetrable to most people. 

The myth that national assessment frameworks introduced accountability ignores other forms of accountability such as parents’ evenings and school reports.  To make this statement is not to claim that such systems of accountability are perfect – they aren’t – but the imposition of high-stakes testing has done nothing to address their shortcomings.  As the Cambridge Primary Review reminds us, parents and carers want to know, among other things, whether their children are happy (Alexander 2010: 316). And there is not, as yet, a National Curriculum level for happiness.

There is also a massive problem in conflating the assessment of individual learners with judgements about the effectiveness of teachers, schools or local authorities.  Most obviously, using aggregated test data as an accountability measure tends to leave much that is relevant out of the account: it ignores the particular contexts of schools and their communities, and it ignores all the other aspects of a school’s work that lie beyond the preparation of pupils for high-stakes tests. 

Worse than this, such accountability measures always and inevitably distort schools’ and teachers’ priorities, encouraging an exclusive focus on the curricular areas that are to be tested, on those aspects of a subject that are to be tested (reading and writing rather than speaking and listening, say), on the students who lie at the threshold of success (the 3a/4c pupils, the C/D borderlines, and so on).

To conclude, I want to return to Edmond Holmes’s picture of schooling a hundred years ago.  Two aspects of this picture seem salient to me – and horribly contemporary.  First, that education has become a commodity: children and their learning are reduced to test scores, mere units of analysis.  The process is epitomised by the new verb meaning, to level: this is the process that teachers are engaged in, a process that flattens and renders invisible all that is distinctive, all that is interesting, about real learners. 

Second, that what the system values – and demands – of all its participants, teachers and pupils alike, is compliance – neither originality nor creativity, neither judgement nor responsiveness to individual circumstance, neither questioning nor imagination, but mere mechanical compliance.

Except, of course, this is not a conclusion, simply a diagnosis.  For us, as for Edmond Holmes, the task is not just to see what is, but to begin to envisage, and to work towards, what might be.

Alexander, R., ed. (2010) Children, their World, their Education: final report and recommendations of the Cambridge Primary Review, London & New York: Routledge.

Brooks, R., & Tough, S. (2006) Assessment and Testing: Making space for teaching and learning, London: IPPR.

Holmes, E. (1911) What is and what might be, London: Constable.

A longer version of this paper will be published in a forthcoming issue of the journal, Changing English.

No comments:

Post a Comment