Where you lose points: a weakness report that reads skills and cons together
This plan comes from your note on the Trap Atlas. It extends the skill dashboard plan of 2026-10-04 (docs/plans/2026-10-04-skill-dashboard-v1.html), and nothing in it is built yet.
- +10 Run the report on your own exported record and check that it reaches your conclusion: an even spread across cons, with one kind of math question behind it.
- +8 Q1, the category vocabulary, decided.
- +8 Check a 40-item sample of Claude's category tags, to measure tagging accuracy before the full pass.
- +5 Q4, whether the report builds practice sessions, decided.
- +4 Q2, the Trap Atlas's fate, decided.
The problem
The app has two views of a student's misses, and each one answers half the question.
- The Trap Atlas (Underworld) counts misses by con. It cannot tell a con habit from a topic the student hasn't learned. Its counts are also raw. A con that appears in many questions looks expensive just because it is common.
- Where you stand (Account) counts by the eight SAT skills, and "Keeps catching you" lists recurring miss reasons. Eight skills are too coarse for advice. "Advanced Math: needs work" doesn't tell the student what to study.
Your record shows the gap. Your misses spread fairly evenly across the cons, but they gather in one kind of math question. Neither screen says so. The finding sits where the two axes meet.
What the data supports today
| Need | State |
|---|---|
| The con behind each miss | Recorded. Every game that deals Wager-bank questions (Wager, Lineup, Darkroom, Tail, Raid, Kingpin, Rematch, Alibi) stores the con of the wrong choice picked, or the suspect's con in the Lineup. |
| The eight-skill code per item | Exported and joined. SkillLedger resolves every Wager, Lineup, Alibi and Target Practice row to a skill. |
| A finer question category | Missing for the main banks. In the corpus, about 190 of roughly 470 Wager and Lineup items carry a question_type. The other 280 carry none, and the export drops the field for all of them. SkillLedger passes an empty subskill for every Wager-bank row. Drill questions and the phone games do carry a subskill. |
| How often each con had a chance to catch you | Derivable without new storage. The bank records which con each wrong choice belongs to, so every attempt shows which cons were on the table. |
| Confidence on a miss | Recorded: the Wager's lock, lean or coin, and the call timing in the Lineup. |
Results join to the bank by item id. A category added to the bank therefore applies to every past answer at once. No stored data changes, and no migration is needed.
Open Questions (5) — implementation blocked until answered
Q1. Which category vocabulary do items get tagged with?
openBlocks: step 1 (tagging), and every later step.
Recommendation: use the College Board's own skill list. That is about 29 named skills under the eight domains, for example Advanced Math → "Equivalent expressions", "Nonlinear functions" and "Nonlinear equations and systems". Students see these names in Bluebook and on their score reports, and Khan Academy is organised by them. A finding like "Equivalent expressions" then points at study material outside the app.
Alternative: the corpus's 23 question_type values. Some items already carry one, but the list is uneven: Math has only "solve", "setup" and "geometry". The existing tags would also need remapping, since several are format labels such as "vocab-in-context" rather than skills.
Q2. What happens to the Trap Atlas?
openBlocks: step 4.
Recommendation: retire it as a screen and move its three parts:
- the con tallies go into the new report, measured against how often each con had a chance;
- calibration ("are your locks gold?") goes into the report as a note on each finding;
- the gallery of cons caught and named goes into The Usual Suspects, which already shows the files.
The Underworld keeps an in-world line on the case board, along the lines of "The file says you fall for the Swap in any subject". It appears only when the report finds a true habit.
Alternative: keep the Atlas as the in-world face of the con axis, reading the same analysis. That leaves two screens over the same data, which is the redundancy you flagged.
Q3. Who reads the report?
openBlocks: step 3 (wording and layout).
Recommendation: write it for the student, in plain words, on the Account screen. Plain words fit there because it is the one area that talks about the SAT directly. "Export your record" would include the same findings, so a parent or tutor gets them without a second screen.
Alternative: a separate tutor view with the full table and rates. That's more work, and nobody has asked for it yet.
Q4. Should a finding build its own practice?
openBlocks: step 5. Steps 1–4 can ship without it.
Recommendation: yes, as the second release. A finding like "Equivalent expressions" gets a "Work on this" button. It deals a night of that category's questions through the games that use them: Wager, Darkroom, Tail and Kingpin. A habit finding opens the Lineup on My Misses for that con. The first release links each finding to the existing Gameplay plan, filtered to that category.
Cost: each game's session composer has to accept a category filter, and some categories hold fewer than ten questions today. The category counts from step 1 show where the banks are thin.
Q5. How much evidence before the report names a weakness?
openBlocks: step 2 (tuning constants). Every value can change later in one line.
Recommendation:
- at least 5 misses and 10 attempts in a category before naming it;
- at least 8 chances at a con before calling it a habit;
- a window of the last 60 days, with older misses counted at half weight.
A finding the student has since fixed fades out instead of staying in the report forever.
The design
1. One table, read two ways
Each miss falls in one cell of a table. The rows are question categories, grouped under the eight skills, and the columns are the cons. Each cell also counts the times that con had a chance, so its rate can be compared fairly with the rest. The report reads where the misses gather, and the shape gives the diagnosis:
| Shape | Diagnosis | Wording sketch | What it prescribes |
|---|---|---|---|
| One category is high, across many cons | knowledge | "Equivalent expressions are costing you. You missed 9 of 14, and it didn't matter which trick was in play." | Learn the topic: the missed items with explanations, then practice in that category. |
| One con is high, across many categories | habit | "The Swap gets you in algebra, reading and grammar alike. It's how you check, more than what you know." | Con work: its playbook in The Usual Suspects, then the Lineup on My Misses. |
| A single cell is high | trap | "Sign flips in linear equations, 5 times." | The specific items, side by side. |
| Innocents arrested in the Lineup | habit | "You've cuffed 6 suspects whose reasoning was sound." | Over-suspicion, which is a separate habit with its own advice. |
Each finding carries a calibration note when it applies: "4 of these were Locks." A confident miss means the student believes something wrong, and that weighs more than a guess.
Example: your record, as the report would show it (illustrative numbers)
| Category | Swap | Translate | Ear | Units | Other | Rate |
|---|---|---|---|---|---|---|
| Equivalent expressions | 3 | 3 | – | 2 | 2 | 64% |
| Linear equations | 1 | 1 | – | 0 | 1 | 21% |
| Boundaries | – | – | 1 | 0 | 1 | 18% |
| Inferences | 1 | 0 | – | 1 | 0 | 15% |
A dash means that con never had a chance in that category. The columns stay even and one row stands out, so the report names a knowledge finding for Equivalent expressions and no habit.
2. Where it appears
- Account → Where you stand. "How you lose points" replaces "Keeps catching you". It shows up to three findings, each a sentence with its tag and two links: "See the items" and "What to play". A "Full table" link opens the heat table, grouped by skill, with the cons that matter as columns.
- Underworld depends on Q2.
- Export your record carries the findings and the table.
3. When a con and a category always go together
Some cons live in one subject only. The Understudy only works on word choice, so a high Understudy count and a high word-choice count are the same misses. When a con's chances sit in a single category, the report says so plainly and makes one finding, not two.
Steps
| # | Step | Where | Waits on |
|---|---|---|---|
| 1 | Tag every Wager, Lineup, Target Practice and Alibi item with its category. Claude drafts from the stem and passage, Jev checks, and you backstop, as with the other banks. A 40-item sample comes first, to measure Claude's accuracy. The export then carries subskill, and SkillLedger reads it. | Corpus, export, PeregrineCore | Q1 |
| 2 | Build WeaknessReport in PeregrineCore/Skills: the table with each con's chances, the diagnosis rules, the confounding rule and the tuning constants. The tests are synthetic histories: your case of even cons and one category, a true habit, a single trap, too little data, and a confounded con. | PeregrineCore | Q5 |
| 3 | "How you lose points" on Where you stand, the heat table screen, and the findings in the export. | App | Q3 |
| 4 | Change the Trap Atlas as decided, and move the gallery to The Usual Suspects if it is retired. | App | Q2 |
| 5 | "Work on this": a category filter for each composer, and con-focused My Misses. | PeregrineCore, App | Q4 |
| 6 | Acceptance: export your record from the phone, feed it to a test, and check the report finds the kind of math question you see yourself. | Tests | step 2 |
Final review
- It answers your exact case, and that case becomes the acceptance test.
- No new storage: past answers gain categories through the bank.
- One analysis feeds the Account screen, the Underworld line and the export, so the two screens stop disagreeing.
- Step 1 is a tagging pass over about 550 items, and the report is only as good as those tags.
- Categories with few questions will rarely gather enough evidence. A student may wait weeks for a finding in geometry.
- The diagnosis rules are reasoned, not measured. Whether "knowledge" versus "habit" is the right split for a 16-year-old's study time is a judgement, and only use will test it.
- Games without cons (Split Decision, Red Flag, Coordinate Drop, Odd One Out) add to the category axis only. They can confirm a knowledge finding but never a habit.
- Prime Suspect, Red String and Go Undercover still record nothing per item, so they stay out.
- A student who plays one game mostly gets that game's categories. The report can only say "not enough yet" about the rest, and the existing "Not practiced" row covers them.
Adjustment worth considering: start step 1 with Advanced Math and Algebra only. That covers your case, and you could judge the report on real findings within a week. The other categories would follow after the format is proven.
- +10 Your exported record run through the report, reaching your conclusion.
- +8 Q1 decided.
- +8 The 40-item tagging sample checked.
- +5 Q4 decided.
- +4 Q2 decided.