Jev Evaluation Loop for the Peregrine Item Bank

Findings · Phases 0 to 2

What is being evaluated

Section split of the 439: reading 178, writing 131, math 130. Two entries (rs-halden, rs-skerry) carry a non-dict options list and are skipped with a named reason rather than silently.

The dimensions, and how each maps onto Jev

Verdict routing

Three bands, following TypeSafe's confidence-gated routing pattern. This is the part that keeps Claude off work it cannot do.

Routing low confidence to a human rather than to Claude is what stops the loop generating work from its own uncertainty.

"evaluation": {
  "rubric_version": "2026-09-20a",
  "model": "jev-1.13.0",
  "run": "20260920T143012",
  "verdict": "pass",
  "dimensions": {
    "skill_code": {
      "type": "choice", "choice": "algebra", "confidence": 0.81,
      "probabilities": {"algebra": 0.88, "advanced-math": 0.12},
      "corpus_value": "algebra", "agrees": true
    },
    "sat_stem_conventions": {"type": "score", "score": 2.4, "confidence": 0.77},
    "engagement": {"type": "score", "score": 1.9, "confidence": 0.52, "advisory": true}
  },
  "composite": {"sat_fidelity": 0.78, "craft": 0.71},
  "flags": ["skill-mismatch"],
  "revision_passes": 0,
  "evaluated": "2026-09-20"
}

Four architecture decisions, each of which had a real alternative

• The rubric is corpus data, not Python. Every config in this system is a file, and the rubric is a content judgement — which skills exist, what "hard" means, what reads as the digital SAT. In evaluation/rubric.json you edit it without touching code, and every stored result names the rubric_version that produced it.
• stdlib urllib, not the TypeSafe SDK. All eleven scripts in tools/ import stdlib only; there is no requirements file, no venv, no lockfile. The SDK's contribution is retry-with-backoff on 429/529, about fifteen lines. Adding the project's first Python dependency to save fifteen lines is the wrong trade. The cost is honest: tools/jev.py owns its own backoff, and if TypeSafe changes the wire format the SDK would have absorbed, that file is where it breaks.
• Results live on the entry, declared in the schema. ResolvedSchema.envelopeKeys covers fourteen keys the app owns outright; anything else undeclared renders nowhere, rides in no packet, and gets reported by StructuralLint. evaluation is declared in schema-extensions.json as type: "object", following the precedent validation_status already set there.
• Evaluation does not gate export in v1. export-machine.py already blocks on validation_status: playable, and the hook to do the same for items is right there. It stays unused until calibration shows the thresholds mean something. Blocking a build on an uncalibrated model score would put Jev in charge of the bank.

Phase 0 · Access, client, and an honest probe complete

• Key stored in the login keychain as openrouter-api-key, entered at a prompt so it reached neither a file nor shell history
• Wrote tools/jev.py: stdlib client, backoff on 429/5xx honouring retry-after, immediate non-retried exit on 400/401/422 carrying the server's own message
• Added evaluate_safe() returning (None, reason) — an error, timeout or malformed body never becomes an invented probability
• Added optional-field readers, because confidence and probabilities are not required by the schema
• --selftest reporting auth, budget, endpoint, served model, round trip and cost; --probe and --math-probe
• Ran the math-competence probe, widened from ten items to a paired design over all 130
• Recorded the result whatever it said — it said the original scoping was wrong

Phase 1 · The rubric as corpus data after Phase 0

• Declare evaluation in schema-extensions.json as a typed object; verify no undeclaredKey lint finding in the app
• Write the state builder: {section, passage, stem, choices[{key,text}], answer_key} and nothing else
• Ensure the blind pass carries no authored labels — agreement is computed in code, because putting the corpus value in the state tells Jev the answer
• Write every Score's levels as concrete situations, never as degrees; TypeSafe's own data shows numbers-only levels collapse confidence to 0.33
• Give every dimension a min_score, min_confidence, weight and gates flag — and ship with every gate off
• Record the engagement caveat in the rubric file itself, with its reason

Phase 2 · The runner, reporting only after Phase 1

• --dry-run is the default: build every request, print counts and estimated cost, send nothing
• Blind pass — one request per item carrying every non-suppressed dimension, since parallel questions cost only their own tokens
• --pass distractors — a second request whose state carries the authored trap codes and mechanisms, asking per distractor whether the mechanism explains that choice
• Move the checks that do not belong in a model into code: forbidden-vocabulary regex, trap-code vocabulary membership, option count, duplicate keys, answer key present among options
• Write a run ledger and a dated HTML report. Write nothing to entry files in this phase.
• Report the distribution per dimension, not just outliers — a dimension where 90% of the bank lands on one level is a broken rubric, not a clean bank

Phase 3 · Calibration — the gate on everything downstream needs your labels

This phase exists because the bank has already been burned once by an uncalibrated judgement: the difficulty backfill put hard on 71% of The Wager and had to be retuned the next day. A model score adopted without checking it against your own reading would repeat that at scale.

• Build a stratified 40-item sample; generate a label template with empty slots
• --calibrate reports per dimension: agreement rate, agreement above each candidate confidence floor, and the confusion pairs
• Report what a threshold would have cost — at each floor, how many items go to Claude, how many to you, and how many of those your labels say were fine
• Flip gates: true only on dimensions clearing a bar you name; record the failures with their numbers rather than deleting them
• Pin jev-1.13.0 rather than jev-latest — an alias moves and would silently retune every gate

Phase 4 · Writing verdicts to the corpus after Phase 3

• --write off by default; writes only the evaluation key and updated
• Skip entries whose evaluation is byte-identical, so an unchanged item causes no write and no watcher reload
• Check git status in the corpus and refuse a dirty tree unless forced — you commit from the app mid-edit, so a surprise diff is easy to lose
• Add an evaluation section to validate-peregrine.py reporting missing evaluations, stale rubric versions, and queue counts as WARN
• Leave export-machine.py untouched, and record why

Phase 5 · The bounded revision loop after Phase 4

No direct Anthropic API client. Revision runs through claude -p in headless mode, the same shape CodeRaven's SimpleClaudeRunner already uses, with limits in front of it.

• --dry-run default: print the prompt and the ids, invoke nothing
• Build the prompt from the failed dimensions only — name, score, the level description it landed on, its confidence, and the rubric's own instruction text. Never a bare number.
• Carry the content bar and forbidden vocabulary into the prompt, so a revision cannot fix a score by breaking the brief
• --max-items per run; --max-passes defaulting to 2, after which an item becomes escalate and is never sent again
• Refuse to revise a final entry or one under a locked guide lifecycle
• Halt on regression: if a run lowers the mean composite for the items it touched, stop and report rather than continuing to the next batch
• Re-evaluate only the revised ids; write before/after into the ledger and a dated HTML report

Phase 6 · App surface deferred, not scheduled

• Decide whether the evaluation queue deserves a card in Press Raven, once there is a run history to look at
• If yes, read evaluation through EntryDocument accessors beside confidenceScore — the app renders corpus state and runs no network calls
• Keep the HTTP client in tools/; do not add one to a @MainActor SwiftUI app for a batch job that belongs in a script

Final review confidence 78 / 100 · was 74

What this solves

The bank's confidence score is an authoring self-assessment — the author's own number about their own work. This adds a second, independent read with a calibrated uncertainty signal, at a cost low enough to re-run on every rubric edit. The routing split is the part that earns its place: a confident negative is work for Claude, an uncertain read is work for you, and the two never get confused.

Pros

• Cheap enough to iterate. Four cents a pass means the rubric can be wrong several times without anyone counting.
• The rubric is data. You retune a level description without touching Python, and every stored result names the version that produced it.
• It uses the corpus's own precedents. validation_status already gates export; schema-extensions.json already declares a gate field; validate-peregrine.py already owns the ERROR/WARN register. Nothing here invents a mechanism the project lacks.
• Confidence routing prevents the obvious failure. Without it, every item Jev could not read becomes a revision request, and the loop manufactures work from its own noise.

Cons and what is genuinely uncertain

• Math correctness is not covered, and cannot be. Overturned by F1. Paired separation of +0.558 across all 130 math items puts answer-key checking back on the table as a candidate dimension, subject to the same calibration as everything else.
• New, from F3 — the audit framing fits only 36% of the bank. For 281 items Jev would be supplying a label rather than checking one, and the plan is not written for that. This is now the largest open question.
• Engagement is the weakest dimension. Including it because it was asked for, and gating on it because it was asked for, are two different decisions. This plan does the first and not the second.
• Nouls carry no confidence. Four dimensions are Nouls, so four have no uncertainty signal and cannot participate in confidence routing.
• Calibration depends on 40 hand labels from you. Phase 3 gates everything downstream and needs a person. If the labels never get made, the honest outcome is a reporting tool with no gates — still useful, and it should be stated rather than quietly upgraded.
• The first Python HTTP client in tools/. Eleven scripts are stdlib-only. This adds outbound network calls to a directory that has never made one.
• Adversarial state. TypeSafe says state is not treated as hostile. Peregrine items contain authored prose about reasoning and error — text that argues about correctness. Whether that moves Jev's answers is unknown until Phase 3's numbers exist.

Where this plan could be inadequate

• If the real goal is catching wrong answer keys, Phase 0's probe will show that early and the plan needs rewriting around a different tool.
• The 173 deferred items have different shapes; each needs its own state builder and possibly its own dimensions. That work is not estimated here.
• revision_passes is per item, not per dimension, so an item failing a different dimension on its second pass still counts against the same cap. The cap is deliberately blunt.
• If Press Raven is mid-edit on an item when --write runs, CorpusStore raises an EntryConflict. The git status check catches an uncommitted tree, not an unsaved draft in the app. Running writes while not editing is a convention, and a convention held by memory is the failure mode this codebase has already documented twice.

Adjustments worth considering

• Run Phase 0's math probe before agreeing to the rest. It determines whether a third of the bank is in scope.
• Consider shipping Phases 0–3 and stopping. A calibrated reporting tool with no gates and no revision loop is most of the value, and it is where the uncertainty resolves.
• Add a code-only answer_key_present_in_options check — it costs nothing and catches a class of error no model should be asked about.
• Unrelated drift noticed while scoping: twelve entries carry status: "draft" where the framework lifecycle is seed / drafted / checked / final. All twelve are prime-suspect or season-documents. Worth a separate one-line fix.