A little information can be a dangerous thing
December 2007
Merlin John
Young Elfed was raising concern at his primary school with the likely results of his reading National Curriculum tests in six months’ time. The management team checked his indicators again and, yes, he was seriously underperforming. So they called in someone from county who was used to working with this value-added data. She visited the school and checked them again, and against other data, and, no doubt about it, Elfed was a good two years behind.
“It must be the upset of his father's latest stay at Her Majesty's pleasure,” one of the senior managers said. “The emotional toll the parents wreak on their little ones…” “Hold on a minute,” said the woman from county. “Let's sit down with Elfed and go through these tests to see what's really happening.” It only took her an hour to get to the root. “Better get Elfed down to the optician,” she advised. “He's very shortsighted and can't see well enough to get on with his work.”
Sure enough, with his new spectacles Elfed's scores climbed back to a happy level.
The name and some of the details might have been changed but it's a true story that researcher Mike Treadaway, who runs the Fischer Family Trust Data Analysis Project, one of the UK's leading independent authorities on pupil data, recounts to show how data can be used to highlight issues. And also to warn of the dangers of trusting data without checking it against a number of other sources.
As the saying goes, “A little information can be a dangerous thing”. In education, however, there is more than just a little information. According to the high master of St Paul's School, Dr Martin Stephen, who recently attacked the league tables as "a cancer on the face of education" (ironically The Sunday Times placed his school fifth in its table), we live in a "data obese" society. He has a point. Besides SATs scores and examination results, schools are awash with data from voluntary schemes like YELLIS and NFER CATS before you even get to the attendance and behaviour data.
The concern extends to the commercial world. Capita, the company that runs SIMS, the leading schools information service, is so concerned that its school customers do not get the most out of the plentiful pupil data that they already hold, that it provides free training programmes. It also offers a free web service, called Improve (Data for School Improvement), which is also freely available to the customers of its competitors (see links top right).
Like the commercial providers, researchers keep clear of the politics of testing and how and why the data is recorded. Their first commitment is to the learners and to schools, and the possibilities of making helpful interventions that can address some of the ‘warnings’ thrown up by the data.
Mike Treadaway says the issue isn’t so much to do with the amount of data collected or what percentage of it is put to good use. “I am tempted to say that it doesn’t matter as long as schools are making the right choices. If you have a vast amount of data and use 2% of it really well, it’s better than having a relatively small amount of data and using all of it badly – if you used all of it you wouldn’t really do much else!”
“There’s a good principle called triangulation. It says that if you’ve got two or three bits of information and they all tell you broadly the same thing, then that’s a pretty solid basis for action. If they tell you different things, what you shouldn’t do is assume that one of them is right and the others are wrong. It’s a flag to say, ‘Delve a bit more’. And that applies at a whole range of levels.”
In this field the delving is crucial. It’s where what happens in the classroom is held up to the ‘mirror’ of the data collection to see why the picture is skewiff – and here the humans hold the keys rather than the computer. “Where there is a lot of data,” explains Mike Treadaway, “the issue is that we should be providing users with the tools to help them know the areas where different bits of data say the same thing (and therefore they should use that and move on and feel solid in their conclusions), and where those things are different enough to recognise that you should delve some more.
“We are probably at the stage where we have a lot of data but we have not yet given folks the tools to ask the right questions. I don’t think you can ever have too much data, but you can present too much data and present it in a way that confuses rather than helps them ask and answer the right questions.”
That’s why the Fischer Family Trust works extensively with local authorities and groups of schools across the UK to get the most from the data they hold, and put it to good use. Self-evaluation tools are available via local authorities from the trust’s website to spread the word that the process is important and not the preserve of number-crunchers.
This practical and supportive use of schools data is getting a welcome boost from the new Institute for Effective Education at the University of York, set up with £11 million funding from the Bowland Charitable Trust. One of its aims is to produce an online ‘best evidence’ encyclopaedia of proven good practice, with comprehensive reviews of existing research. “Shouldn’t take us long: it’s not a very big place,” quips its director Professor Bob Slavin, an American who has moved over from the Centre for Research and Reform in Education (CRRE) at Johns Hopkins University, Maryland, USA, where he retains a position. “We’ll try to do it as soon as we can – in our lifetimes would be good!”
With him at York are two other CRRE colleagues, his wife Professor Nancy Madden and Professor Bette Chambers, a Canadian. All three are also well known for their work with the Success For All Foundation. Their expertise in using data, together with classroom work in schools and extensive experience in successful learning and teaching, has earned them an international reputation. And their passion for education and commitment to helping all learners experience success is palpable.
This is how the institute describes itself: “The Institute for Effective Education has a single-minded focus on developing, evaluating, and disseminating effective programmes and promoting evidence-based policies. It focuses on what works, not on what corresponds to current political trends, not on what can be profitable, not on what contributes to theory or scholarship alone, but what works for children.”
That attitude is reflected in the team. They are committed to extracting meaning from pupil data and applying what is learned to solutions, but only with close attention to what is happening with children in classrooms.
Bob Slavin and Nancy Madden became involved in experiments in ‘cooperative learning’ back in the 1970s. They became convinced of its effectiveness in “well designed experiments in maths, English and science”. However, as they progressed, they could see that identifying success and describing how it could be achieved was not enough to bring about systemic change.
“By the late 1970s a person could sit back and say, ‘We have really found something, anyone could do it and teachers are happy with it. It’s published in the best journals and we are done.’ What immediately became apparent was that it was worth nothing – perhaps we were creating potential but we were not creating change. We could talk to roomful of people and they would go out and what they did bore no relationship to what we were talking about.”
The lesson they learned was that “teaching is heavily driven by curriculum, not by teaching methods, especially in secondary but in primary too… And when you bring in methods not tied to content, where they run into friction the curriculum will always win.”
By the 1980s they were bringing together packages where the curriculum and teaching were “inseparable” and began building successful improvements in the achievements of children in stressed urban US environments – Baltimore in particular, “a fabulous opportunity” ¬– for which they have become renowned. This was the stage when their understanding of ‘cooperative learning’ extended beyond groups of children to groups of teachers, schools and districts.
One aspect of UK classroom practice that particularly interests them is the adoption of
As with Mike Treadaway, there is an insistence on delving. Bette Chambers says, “You have to be in the actual schools with the actual people. It takes more in-school contact that we originally thought. You have to see the data as giving you the hypothesis you have to check, and see the quality of instruction and how to make change. People think the data is the end of the process; it isn’t, it’s the beginning.”
She makes the analogy of a ship, with markings on the side showing her position in the water. “You do assessments and each week we are one fathom deeper. Well, we are a data-driven ship and we don’t want to wait until we are at the bottom before we find the leaks.”
With this sort approach to the oceans of data that schools now hold, it seems some interesting insights and solutions are emerging from the depths.
Merlin John is an editor who created and ran The TES’ Online magazine. He is now freelance and works for publications including The Guardian, and runs his own web service at www.merlinjohnonline.net .