Grade the grader: an eval harness for an LLM IELTS examiner

In the last post I described how EasyIELTS grades IELTS speaking with an LLM: turn the official rubric into a strict-JSON contract, encode the exam’s expectations into the prompt, clamp the output. I claimed this makes a fuzzy human judgement reproducible.

That’s a measurable claim, and I had never actually measured it. So I built the eval harness, pointed it at real production answers, and measured. The grader is less stable than I assumed, my resilience fallback turns out to be a silent grading-policy change, and the top band drifted by almost a full band in three weeks. Here’s the harness, and the numbers.

Step 1: a frozen sample, stratified by the output

In the last 30 days production scored 5,293 speaking turns. Their median band is 5.5 — so a random sample would be almost all mid-band answers, and would tell me nothing about how the grader treats a band-8 monologue or a band-2 ramble. The sampler stratifies by the thing being evaluated:

case
  when band < 5  then '1_lt5'
  when band < 6  then '2_5_to_5_5'
  when band < 7  then '3_6_to_6_5'
  when band < 8  then '4_7_to_7_5'
  else                '5_gte8'
end as band_bucket
-- 8 random turns per bucket → 40 turns, frozen as CSV

Each sampled turn keeps its question, transcript, the four dimension scores production actually served — and a signed 24-hour audio URL, because a transcript-only grader needs a human who can listen as the calibration anchor. The CSV is frozen: every future harness run re-scores the same 40 turns, so runs are comparable across weeks, models and prompt versions.

Step 2: evaluate the production code, not a copy of it

The most common way eval results lie to you is evaluating a copy of the prompt that has quietly diverged from the one deployed. The harness imports the production module:

import {
  SPEAKING_SCORING_SYSTEM_PROMPT,
  buildSpeakingScoringPrompt,
} from '../lib/speakingScoringPrompts.ts'

Same prompt, same temperature (0.3), same JSON mode, same clamping. Then a small matrix of runs, built to answer three questions:

for (let p = 1; p <= 3; p++) runs.push({ spec: primary, pass: p })  // is it stable?
runs.push({ spec: fallback, pass: 1 })                              // does my fallback agree?
// ...and the frozen CSV already holds May's scores                 // has it drifted?

160 calls against deepseek-v4-flash (primary) and the OpenRouter fallback. The whole run costs cents and takes a few minutes — there is no budget excuse for not having one of these.

Finding 1: the grader is a distribution

Same model, same prompt, same transcript, three runs. Only 15% of turns got the same band three times. A third moved by more than half a band between identical runs (mean spread 0.625, worst case 1.5 — one Part 2 answer scored 8, 6.5 and 7).

Per dimension it’s noisier, and the noisiest dimension is exactly the one the model can’t hear: pronunciation (mean spread 0.74, stable on only 10% of turns) — the blind spot from the last post, showing up as variance.

This matters because learners track progress in 0.5-band steps. For a single turn, the noise is the size of the signal. The honest fixes are mechanical: report a session band aggregated over many turns (which production already does — averaging across turns cancels per-turn noise), and gate any future “the new prompt scores better” claim on this harness, because a 0.5-band improvement on one run of one turn is indistinguishable from nothing.

Finding 2: the fallback is a grading-policy change

When the primary API errors, scoring silently falls back to another provider. Resilience feature, right? The fallback model agrees with the primary within half a band on only 72.5% of turns — and the disagreement isn’t symmetric: it grades +0.34 band more lenient on average.

In other words: when DeepSeek has a bad day, EasyIELTS gets easier. Nobody chose that policy; it fell out of an error handler. A fallback provider isn’t an infrastructure detail — it’s a second examiner with different opinions, and it should be calibrated (or at least monitored) like one.

Finding 3: three weeks of drift, invisible in the average

Re-scoring May’s turns with today’s primary model: mean shift only −0.19 — looks fine. Stratified, it’s not fine:

Band bucket	Shift vs May
< 5	+0.44
5–5.5	−0.19
6–6.5	−0.06
7–7.5	−0.25
≥ 8	−0.88

Weak answers now score higher, strong answers score almost a full band lower — regression toward the middle, masked by a near-zero average. (At n=8 per bucket, run-to-run variance explains part of this — but that’s an argument for running the harness weekly, not for ignoring it.) If you only monitor the mean of live scores, this is the kind of change you will never see.

What the harness actually bought me

Before: “the rubric is a contract, so the grading is reproducible” — an architecture claim. After: reproducibility is a number (67.5% of turns within half a band), the fallback’s leniency is a number (+0.34), drift is a per-bucket table, and all three re-compute in minutes for cents. Every future change to the prompt, the model, or the provider now has a regression gate; the frozen sample with audio URLs doubles as the human-audit work queue.

Takeaways

• An unevaluated LLM grader is an opinion. Test-retest variance is the floor — measure it before measuring anything fancier.
• Sample from production and stratify by the output. A random sample of a median-5.5 population tells you nothing about band 8.
• Evaluate the deployed prompt by importing it. A copied prompt in an eval script is already drifting.
• Your fallback provider is part of your grading policy. Treat a second model like a second examiner, not an infrastructure detail.
• Averages hide drift. −0.19 overall was +0.44 at the bottom and −0.88 at the top.

The harness is ~200 lines, no framework: a stratified SQL sampler, a frozen CSV, and a script that re-scores it through the production prompt. EasyIELTS is live, with 1,400+ learners.