Calibration — Three Raters and the Corpus

40 blinded items, labelled independently by Sarah, Claude and Jev, scored against each other and against what the corpus already says. 20 September 2026.

Two results, and both are decisions. skill_code gates at a confidence floor of 0.80, holding 97% accuracy. difficulty cannot gate — and the reason is not Jev.

skill_code — ships, and gates at 0.80

Pairwise agreement

	Sarah	Claude	Jev	Corpus
Sarah	—	90%	82%	78%
Claude	90%	—	88%	88%
Jev	82%	88%	—	90%
Corpus	78%	88%	90%	—

The two humans agree 90% of the time. That is the ceiling. Jev sits at 88% against Claude and 90% against the corpus — at the ceiling, not below it. A dimension cannot be asked to beat the people who define it.

Confidence does exactly what it should

Floor	Kept	Accuracy above it	Sent to review	Wrong, kept
0.0	40	88%	0	5
0.6	34	94%	6	2
0.80	32	97%	8	1

At or above 0.80, Jev is right 31 times in 32. Below it, 4 of 8. That is a clean separation and it is what a confidence floor is for.

Set in the rubric: skill_code now has gates: true and min_confidence: 0.80, with the calibration set recorded beside it. Eight items in forty go to you rather than to Claude.

difficulty — cannot gate, and Jev is the best of the four

Pairwise agreement

	Sarah	Claude	Jev	Corpus
Sarah	—	70%	70%	62%
Claude	70%	—	75%	52%
Jev	70%	75%	—	72%
Corpus	62%	52%	72%	—

Sarah and Claude agree on difficulty only 70% of the time. Two careful readers, the same forty items, and they differ on twelve. 70% is the ceiling, so no threshold above it can mean anything. And Jev agrees with the corpus more often (72%) than either human does (62% and 52%) — calling Jev the weak link here, which I did earlier, was wrong.

Jev's confidence runs backwards

Confidence band	Correct
at or above 0.80	33%
below 0.80	88%

Jev is most certain precisely where it is wrong. Every one of the five high-confidence misses is the same shape:

Confidence	Item	Jev	All three raters
0.88	`w-math-000`	hard	standard
0.87	`w-math-002`	hard	standard
0.87	`l-math-translate-2`	hard	standard
0.85	`l-math-wrench-1`	hard	standard
0.76	`w-math-005`	hard	standard

The level description is the defect, not the model. Level 2 reads "a value appears along the way that a reader could mistake for the answer, so finishing means checking which quantity the stem asked for." That is true of nearly every multi-step math item. Jev applies it faithfully and confidently, and a threshold cannot repair a level that fires on everything. This is the guide's own criterion A, and the guide already records it being over-applied once before, when a backfill put hard on 71% of the Wager.

The repair is to rewrite level 2 so it discriminates within math. That rewrite invalidates this calibration, so the sample has to be re-scored afterwards — which is why it was recorded rather than done here.

Where the corpus is the outlier

Items where all three raters agreed with each other and the corpus did not. These are the ones worth a second look, because three independent reads landed together.

skill_code — 2 items

Item	All three raters	Corpus
`l-reading-extreme-5`	info-and-ideas	craft-and-structure
`w-writing-021`	standard-english	craft-and-structure

difficulty — 7 items, and six run the same direction

Item	All three raters	Corpus
`l-writing-breather-3`	standard	hard
`w-reading-007`	standard	hard
`l-writing-agree-1`	standard	hard
`l-writing-volunteer-2`	standard	hard
`l-writing-volunteer-2-twin`	standard	hard
`w-reading-kr07`	standard	hard
`w-math-020`	hard	standard

Six of seven are the corpus saying hard where all three raters say standard. That is the same over-marking the guide documents, surviving the retune in these items.

What the third rater bought

With two raters, difficulty would have read as "Jev disagrees with the corpus 28% of the time" and the obvious response would have been to blame Jev or raise a threshold. Three raters show the corpus disagreeing with two humans more than Jev does, and the humans disagreeing with each other 30% of the time.

The dimension that ships, ships on evidence — 97% at a floor, against a measured human ceiling of 90%.
The dimension that does not ship fails for a nameable reason — an ambiguous construct plus one level description that fires on everything — rather than for "the model is bad at this".
Nine corpus labels are now under suspicion, which neither a two-rater design nor any amount of model tuning would have surfaced.