Engagement, Decomposed

A starting point for the meeting, 20 September 2026. Part one is why this one dimension resists evaluation in a way the other ten do not. Part two is a method for breaking it into sub-dimensions and framing them — the tests a candidate has to survive, and the criteria for choosing its question type. What the sub-dimensions turn out to be is the meeting's output, not this document's.

What this document deliberately does not contain. No proposed list of engagement sub-dimensions, and no type assignments. Two worked examples appear in part two, and both are kills — naive candidates run through the tests to show the mechanics. Nothing here pre-decides what survives.

1Why engagement resists evaluation

Seven reasons, ordered by how much each should change the room's approach. The first three are structural and will not be argued away by better prompt wording.

1 · It has no ground truth anywhere in the corpus

Every other dimension in the rubric has an authored value to agree or disagree with. skill_code, section, difficulty and the per-option trap_codes all exist on the entries already, so calibration means comparing Jev's answer to a label that is sitting right there. Engagement has no authored value on any of the 439 items. Calibrating it means manufacturing the labels first, which makes it the only dimension whose calibration cost is paid in your time rather than in a comparison.

2 · The referent is ambiguous, and the brief pushes it away from the item

Engagement of what — the item, the round, the mini game's loop, or the session? The guide's own admission test says a loop has to survive with the SAT removed, which makes engagement a property of the mechanic. The mechanics live in the app and the forty-odd mockups, and the guide is explicit that entries carry content and never behaviour. An item evaluated in isolation has been stripped of the thing the brief says carries the engagement. Settling this referent is the first thing the meeting should do, because every later decision inherits it.

3 · Engagement is relational, and Jev evaluates one state in isolation

Whether a round lands depends on the player's level, what they just missed, streak state, and position in the run. A round that delights as item three is tedious as item thirty if it repeats a con the player has already seen twice. Jev holds no memory across requests and sees one state at a time, so nothing about repetition, pacing or variety is visible to it at the item level. Those properties are real and they are measurable, but not by this instrument in this shape.

4 · A single Score cannot hold a bundled judgement, and TypeSafe says so plainly

From the Score documentation: if a level description says "punctual and smart and experienced," the question is measuring three things, and an input high on one and low on another cannot be placed. Confidence drops and the score means less. Engagement as ordinarily understood bundles at least surprise, stakes, agency, pace, voice and fairness. One Score over that bundle is precisely the named failure mode, which is the technical reason decomposition is the right instinct.

5 · Jev is documented as literal, and "is this fun" is maximum indirection

The jaggedness page lists literal reading and indirection as the first and fourth failure modes: Jev answers the question as written, and multi-hop reasoning costs accuracy. Asking whether a person would enjoy something requires simulating that person, which is several hops. Asking whether the text has a property is one hop. Every candidate dimension will need to be restated as a property of the text before it can be asked at all.

6 · Confidence will look reassuring and mean less than it does elsewhere

Confidence measures how concentrated the probability distribution is, not whether the answer is right. A vaguely framed question can return a tight distribution because the model has a stable prior about generic prose. TypeSafe says it directly: higher confidence alone does not show that a level description is better. On dimensions with a corpus value, a wrong confident answer is caught by the disagreement. On engagement there is nothing to disagree with, so a confident wrong answer looks identical to a confident right one.

7 · Gating on it would invite drift with no way to detect the drift

If engagement gates and Claude revises toward it, items move toward whatever the rubric rewards. With no ground truth, there is no signal that would reveal the drift. This is why the plan currently marks engagement advisory. The meeting can overturn that, and the thing to overturn it with is a calibration design, not a conviction that the dimension matters — which it does.

One more worth naming because it will show up in the room. Peregrine items contain authored prose about temptation and error — mechanism, drill_rx, con lines. TypeSafe says state is not treated as hostile by default, so text arguing about how tempting a wrong answer is may move a model asked how tempting the wrong answers are. Whichever sub-dimensions survive, decide per question which fields belong in its state.

2The method: six tests a candidate must survive

Generate candidates freely, then run each through these in order. The order matters — tests one through three kill fastest and cheapest. A candidate that fails any one of the six is either reframed until it passes or dropped, and a dropped candidate is worth recording with the test that killed it.

Test 1 · Locatable

Can you point at the bytes in the entry that determine the answer?

If the answer depends on the mockup, the streak state, the player's history, or which item came before, it is not evaluable from an entry. It may still be real and worth measuring — by telemetry, by playtest, or by a run-shaped state rather than an item-shaped one. It is not this instrument's job.

Test 2 · Single-axis

Can an item be high on this and low on everything else?

If two items can both score well under the same dimension for opposite reasons, it is two dimensions wearing one name. Splitting is nearly free: questions are evaluated in parallel and an extra one costs only its own tokens. When the room is unsure, split.

Test 3 · Describable as situations

Can you write three to five level or option descriptions that someone else could sort items by, without seeing the numbering?

If a description needs the words "more," "moderately," or "than the previous level," the dimension is not decomposed yet. Jev evaluates each level separately and never sees a level's number or its neighbours, so comparative language is invisible to it. TypeSafe's own data: a scale whose levels were the bare numbers 0, 1, 2 returned confidence 0.33; the same item with three descriptive levels returned confidence 1.0.

Test 4 · Disagreeable

Can you name an item in the bank you would score at the bottom and one you would score at the top?

A dimension on which the whole bank scores the same carries no information regardless of what the model reports. This is checkable on the first run by reading the distribution rather than the outliers, and it is the cheapest way to catch a dimension that sounds meaningful and is not.

Test 5 · Actionable

If an item scores low, does the score name something an author could change in the entry?

This is the test that connects engagement to the revision loop. A dimension that fails without implying an edit produces a queue nobody can work, and under the plan's routing a confident failure goes to Claude — which will then try to revise toward a target it cannot act on. A dimension that passes every other test and fails this one should stay advisory even if the room loves it.

Test 6 · Non-redundant

Does it measure something the rubric already measures?

distractor_pull and difficulty may already absorb much of what "engaging" means for an SAT item under a noir frame. Do not settle this by argument — the first full run gives you the correlation for four cents, so decide it with the numbers on the second pass.

Two worked kills, to show the mechanics

Candidate: "Is this item fun?"

Test 1 · Locatable	FAIL — fun is a property of the experience, and the experience includes the mechanic, which is not in the entry
Test 2 · Single-axis	FAIL — a tense item and a witty item are both fun for unrelated reasons
Test 3 · Describable	FAIL — every level description collapses to a degree word

Dead at test one. Worth running anyway, because most first-draft candidates are this one in better clothes, and seeing it fail three times gives the room a calibrated sense of what "reframe it" means.

Candidate: "Would a player want to keep going after this round?"

Test 1 · Locatable	FAIL — it is a property of the session, and depends entirely on what came before
Test 2 · Single-axis	pass — arguably one axis
Test 5 · Actionable	FAIL — a low score names nothing an author could edit in the entry

This one is more instructive than the first, because it is a genuinely good question that belongs to a different instrument. It is a playtest question or a telemetry question. Record it somewhere other than the rubric rather than discarding it.

3Framing rules for whatever survives

These apply to every surviving sub-dimension regardless of its type, and they come from TypeSafe's own guidance on how Jev fails.

Ask about the text, never about the player. "Would a student enjoy this" requires simulating a person. "Does the stem name a concrete situation rather than an abstract one" is a match against words on the page. The first is the indirection Jev is documented to fail; the second is the kind of question it is built for.
State the exact condition, including the boundary. The tell that an instruction is incomplete: you look at a wrong answer and find yourself explaining what you really meant. That explanation is the missing half of the instruction.
Boundary cases belong in the criteria, not the instruction. The instruction carries the question; the criteria carry what each answer means, including the awkward cases.
Keep instruction and criteria aligned. A Noul whose true maps to something the instruction phrases as a no performs measurably worse. Read the pair aloud and check they ask the same thing in the same direction.
Never number a level or reference its neighbours. Each level is evaluated in isolation against the state.
Give a rare extreme its own level if you would act on it differently. A scale topping out at "very tempting" cannot distinguish a tempting distractor from one that is arguably also correct, and those want different responses.
Filter the state per question. Accuracy falls as the state grows with material the question does not need. A question about the stem's concreteness probably should not see drill_rx.
Write it so the vocabulary rule survives. Rubric text is not UI copy, so the banned words are technically allowed in it. The risk is subtler: phrasing that treats the item as a test question pulls the whole evaluation into the frame the brief rejects.

4Choosing the type, once a dimension exists

Criteria rather than assignments. Run each survivor against these and let the shape of the property pick the type.

Reach for	When	What it costs
Score	The property is genuinely ordered and you can describe each rung as a distinct situation someone could sort by	Two to ten levels, and every level you cannot describe distinctly is a level you should not add
Choice	The answers are discrete kinds rather than degrees, and "in between" is meaningless	Nothing structural. It also hands you a probability per kind, which is how you spot an item sitting between two categories
Noul	The property is a true yes/no and you want the probability rather than a verdict	No confidence is returned. Under the plan's routing that means it cannot gate, and its threshold cannot be borrowed from a Choice's
Combination	A Choice picks the kind while a Noul asks whether the property is present at all; or a rare extreme deserves its own separate question	Almost nothing — parallel evaluation means the second question costs only its own tokens

The combination case has a worked precedent. TypeSafe's skill-suggestion cookbook runs a Choice to rank candidates and Nouls to decide whether any should be chosen at all, on the same state in one request. The Choice is relative and settles which; each Noul is absolute and can be low for all of them. That asymmetry is likely to matter here, because "which of these is the item's strongest hook" and "does this item have a hook at all" are different questions and a room can easily merge them by accident.

One standing caution: do not carry a threshold tuned on a Noul across to a Choice, and do not expect P(x) and 1 − P(not x) to agree. TypeSafe measured 0.72 and 0.47 on the same proposition asked both ways.

5Combining, gating, and the calibration problem

Combining

Weights live in rubric.json and are yours. The working method is the one TypeSafe recommends and the one this corpus already uses for judgement calls: when the combined result disagrees with what the room would decide, change the weights and run again. At four cents a pass, the loop is fast enough to do in the meeting.

Gating

Decide gates per sub-dimension, not for engagement as a whole. A single well-anchored sub-dimension gating is a much smaller claim than the composite gating, and it is a claim the calibration can actually support. Everything else stays advisory and still gets reported.

The calibration problem, and the cheapest way out of it

Absolute labelling is the expensive path: scoring forty items on a four-level scale asks you to hold the whole scale in your head and stay consistent across a sitting. People are poor at that and good at something adjacent.

Proposal for the room: calibrate engagement by forced pairwise ranking rather than by absolute scoring. Take thirty pairs drawn from the bank, pick the better one in each, and derive an ordering from the wins. The rubric's job is then to reproduce your ordering, checked by rank correlation, rather than to hit absolute numbers. Pairwise judgements are faster, more consistent, and produce a defensible target for a dimension that has no authored value to compare against.

One honest caveat on that proposal. TypeSafe blesses using a Score's expectation to test a threshold and warns against using it to reconstruct exact magnitudes between levels. Ordering by expectation sits between those two, so the rank correlation is a calibration diagnostic and the production gate stays a threshold. If the ordering reproduces well, that is evidence the levels are doing real work; it is not itself the gate.

6A run sheet for the meeting

Time	Step	Output
15 min	Settle the referent. Item, round, loop, or session — what are we claiming to measure?	One sentence everything else inherits
20 min	Generate candidates. Divergent, no filtering, no defending	A long list, deliberately including bad ones
30 min	Run the six tests. In order; kill fast	Survivors, plus a record of what killed each casualty
20 min	Draft descriptions out loud for each survivor	The ones nobody can describe distinctly die here
15 min	Assign types and decide what gates	Type per dimension; a short gating list and a long advisory list
10 min	Pick the calibration set	Thirty pairs, or a decision to stay advisory and skip calibration

Red flags to watch for in the room

An argument about whether a specific item is fun. That is the referent problem resurfacing. Go back to step one rather than settling the example.
A level description that needs a comparative. Test three has already failed; reframe rather than wordsmith.
A dimension everyone loves that nobody can act on. Test five. Keep it advisory and say so out loud, so it does not quietly become a gate later.
Reaching for more levels to capture nuance. Levels you cannot describe distinctly make the score worse, not finer.
Agreement arriving quickly. On a dimension with no ground truth, fast consensus is the least reliable signal in the room. Test four exists to break it: name the bottom item and the top item, out loud, before moving on.