Ofqual’s A-level algorithm: why did it fail to make the grade?

Pkj = (1-rj)Ckj + rj(Ckj + qkj – pkj)

For such a short string of algebraic symbols, there is a lot we can learn from Ofqual’s grading algorithm (though really it is an equation) – and a lot we can learn about what went wrong.

First and most obviously, the size of the algorithm is an issue. With just four unique terms – Ckj, qkj, pkj and rj – it shows the sparseness of the inputs. This is not a “big data” solution, gathering every possible piece of information about a student in an attempt to gain a full view of their capability. In fact, it is the opposite: using the smallest possible amount of information, so it can be rapidly gathered and easily standardised.

So what are those terms? The first are three various distributions of grades, k, at schools, j. Ckj is simple enough: it is the historical grade distribution at the school over the last three years, 2017-19.

Algorithm and blues: Gavin Williamson’s week of A-level chaos

That tells us already that the history of the school is very important to Ofqual. The grades other pupils got in previous years is a huge determinant to the grades this year’s pupils were given in 2020. The regulator argues this is a plausible assumption but for many students it is also an intrinsically unfair one: the grades they are given are decided by the ability of pupils they may have never met.

qkj is where the pupils’ own ability comes in. That is the predicted grade distribution based on the class’s prior attainment at GCSEs. A class with mostly 9s (the top grade) at GCSE will get a lot of predicted A*s; a class with mostly 1s at GCSEs will get a lot of predicted Us.

pkj is the predicted grade distribution of the previous years, based on their GCSEs. You need to know that because, if previous years were predicted to do poorly and did well, then this year might do the same; and, again, vice versa.

The final term, rj, is different: it is not about grades at all, hence the absence of the k. Instead, it is about how many pupils in the class actually have historical data available. If you can perfectly track down every GCSE result, then it is 1; if you cannot track down any, it is 0.

Finally, we can put the terms together. First, the equation is in two halves, one multiplied by that rj term, and one multiplied by the inverse of that term – one minus rj. What that says is: “If we don’t know about this group’s GCSE grades, ignore the right half of this equation, and just base everything on last year’s A-levels; to the extent that we do know about their GCSE grades, use that information as well.”

How ministers defended the A-level results system

“[I have] every confidence that the system we have put in place is a robust system, a system that’s fair”

“The system, for the overwhelming majority of young people, is going to deliver credible, strong results. It’s a robust system, it’s a fair system, it’s making sure that young people get the grades that they’ve worked so hard towards”

“Most young people … will get the grade that the teacher sent in to the exam board that they thought they would get.”

Q) “Can you give a cast-iron guarantee that you will not be forced into the embarrassing U-turn that John Swinney and Nicola Sturgeon were in Scotland?”

A) “Absolutely”

“Let’s be in no doubt about it, the exam results that we’ve got today are robust. They’re good, they’re dependable for employers. It’s very important that for years to come people should be able to look at these grades and think these are robust, these are dependable”

“This is it… No U-turn, no change… [In Scotland] you’ve got a system where there aren’t any controls, you’ve got rampant grade inflation. There’s been no checks and balances in that system; it degrades every single grade as a result and in-baked unfairness” 

The left half, which only gets used if we do not know the GCSE data, is that simple: “Just use the historical A-level results.” And then the right half says: “Use the historical A-level results, but add to them the predictions from this year’s GCSE results, then downgrade them based on how good the last lot of predictions were.” That means a school that regularly gets good A-level results despite having bad GCSEs will get a boost.

Aggregating all those terms together gives us pkj, the predicted grades for the school.

Even in this short equation, we can see the seeds of a fiasco: prior attainment based exclusively on GCSE results; historical grades stretching back just three years; and a refusal to allow the actual success of the pupil to overrule the situation.

In a better system, perhaps the rest of the process could have ironed out these flaws, but in reality they made them worse.

The decision to give small classes the ability to receive their teachers’ recommended grades is not in the algorithm but led to a boost for elite private schools.

The choice to take the results of the algorithm and further tweak the grade boundaries to prevent overall grade inflation is not in the algorithm but further depressed the larger classes in favour of the smaller.

And the choice to focus, not on determining individual grades, but on determining a distribution for a class which students were then matched to on the basis of their rank in the class, is not an error in the algorithm but a fundamental misunderstanding of what the goal was.

Source: Read Full Article