# Creating corpus cells + wiring the bench

audience: AI coding agents first. Imperative, BLUF-ordered.

**Core principle: a cell is RED-on-vuln + GREEN-on-safe + survives discriminators; a class without n≥3 cells is a point estimate, NEVER a recall rate.**

## When to use
Invoke to add a corpus cell, a discriminator, or to extend the bench to a new class.

## Build a cell (ladder)
1. Use `investigate` to reconstruct vuln + safe via RAW git from the fix commit; verify clean end + canonical symbol.
2. Lay it out per `domains/security/corpus/README.md`: `<id>-<slug>/{vuln.ts, safe.ts, canonical.json}`.
3. Add **discriminators** (anti-overfit): rename, cosmetic-reformat, polarity-flip, null-mutant variants. The detector MUST still flag the vuln and still pass the safe.
4. Wire into the bench: `python3 bench.py --inventory` must list the new cell. Recall: feed orchestrator findings via `--findings`.

## Encode the MEASURED band methodology (do NOT re-measure what is settled — see docs/validation/)
- **band-1 single-file logic: COVERED.** 7/11 classes measured CAUGHT (S1,S4,S5,S6,S7,S8,S10). A new single-file logic cell is expected catchable — measure k≥3, don't assume.
- **Shape-A cross-file IDOR: FLAGGED by band-1** as an unconfirmed in-file-ownership gap (recall OK at flag level).
- **Shape-B cross-file (imported insecure default): a band-1 single-file MISS (MEASURED 0/3).** DETECTION is solved once the symbol is RESOLVED into scope — the two-file test caught it 3/3. So a Shape-B cell MUST carry both the single-file `vuln.ts` (the MISS input) and `vuln_unresolved.ts` (the resolved CATCH input), and `canonical.json.requires_resolution:true`. The open band-2 work is DELIVERY (barrel resolver + proactive trigger) — NOT a bespoke omission-detection sub-check (proven unnecessary).

## Statistical rules (spec §6)
- LLM-band recall: k≥3 rolls, report as a RATE. NEVER a single-run claim.
- A recall RATE per class needs n≥3 cells/class — incl. n≥3 Shape-B cells for a real cross-file rate. n=1 is a point estimate; label it so.
- Deterministic detectors: pass/fail.
- no-false-coverage-claim: a class with 0 cells / unmeasured recall is a BLIND SPOT in `docs/taxonomy/security.md`, never "covered".

## DO NOT
- DO NOT ship a cell without a GREEN safe.ts — RED-only overfits.
- DO NOT claim a recall RATE from one cell or one roll.
- DO NOT tune a detector and validate it on the same cell — validate held-out.

## Pointers
Band evidence: `docs/validation/2026-06-17-recall-spike-S1-S5-S7-S8-S10.md`, `docs/validation/2026-06-17-band2-crossfile-spike.md`. Corpus contract: `domains/security/corpus/README.md`. Bench: `bench.py`.

## Learned Rules

### green-residual-classify-tp-fp | fired:1 | 2026-06-17
Labeled a non-canonical finding raised on safe.ts (S2 invite stale-JWT, team TOCTOU) as "precision noise" + claimed it sharpened dedup task #17 → contradiction (a real issue the detector surfaces is a TRUE POSITIVE, not noise; advisor caught it). git-fix safe.ts is canonical-clean, NOT globally-clean: it can still hold distinct real issues, more on subtler cells = the gate working.
Prevent: when safe.ts draws a non-canonical finding, READ the fixed code and classify it TP vs FP before naming it — never file a TP as "precision noise" (that steers dedup toward suppressing real bugs) and never link distinct-issue residuals to #17 (which is paraphrase-dedup, ~N:1 same-issue restatement — a different phenomenon). Only true precision threads (e.g. severity over-rating) belong there.

### test-the-gate-through-its-real-action-not-around-it | fired:2 | 2026-06-18
Validated the prevent pre-commit gate via the commit-SUCCESS path + a direct prevent.py call — neither ran the installed hook, so the hook→`git commit`-abort seam was untested until advisor flagged it (#42 doctrine one level up: green-on-a-direct-call hides a broken integration boundary). fired:2 — the FIX left the real-`git commit` proof as a one-off in a validation doc; the seam stayed un-regression-locked (advisor re-flagged) until ported into `tests/test_prevent_e2e.py`.
Prevent: a gate whose job is to ABORT an action must be tested by performing the REAL action (a guarded real `git commit` in a temp ratchet) and asserting it aborts (exit 1 + HEAD unchanged) — never by calling the checker function directly. AND lock that real-action test IN the suite (a `tests/` file), never as a one-off in a validation doc: a documented rule is not a regression lock, and a direct-call green ≠ wired. To make the entry point hermetic, give it root/confirmed overrides (mirror `load_confirmed(p=None)`) so the through-git test needs no install.

### validation-test-pins-contract-not-AssertionError | fired:1 | 2026-06-18
A rejection test caught `AssertionError`; hardening the impl to `raise ValueError` (so it survives `python3 -O`, which strips asserts) broke the test → a correct hardening looked like a regression because the test coupled to the assert MECHANISM.
Prevent: test input-rejection by asserting the documented exception type (ValueError/contract error), never `AssertionError`; and never use bare `assert` for runtime validation that must hold under `-O`. The test pins the contract, not the mechanism.

### llm-live-selftest-not-a-ship-gate | fired:1 | 2026-06-18
A plan's verification step gated on the LLM-live `--selftest` (real `claude -p`, k-roll, stochastic); its benign "unstable" FAIL then read as a possible regression and cost an investigation detour, while the behavior-preserving refactor was provably clean and the deterministic `--unittest` + pytest were green. Extends §"Statistical rules" (LLM-band = rate, never single-run) into the ship-gate dimension.
Prevent: when a tool exposes BOTH a deterministic mode and an LLM-live mode, the plan's verification command + any CI/ship gate MUST invoke the deterministic mode (pytest + `--unittest`) with its explicit flag — never the bare script (errors) and never `--selftest`. Record an LLM-live run as rate-only evidence; a single stochastic FAIL is NOT a regression and NEVER blocks merge.

### teaching-to-test-is-not-validation | fired:1 | 2026-06-18
Presented S2-role 1/5→4/5 as the headline "specialist lift" without flagging the prompt author had READ that cell's `canonical.json` (the fix description) before writing the detection pass → the pass restated the fix; an in-distribution result reported as generalizable validation. Advisor had to force a held-out control. Sharpens body rule "DO NOT tune+validate on the same cell": contamination is not only tuning-on-the-cell, it is AUTHORING the prompt with the canonical / ground-truth in context.
Prevent: before claiming a catch VALIDATES a detector (not merely "it fired"), verify the prompt was FROZEN before that cell's canonical was ever read — teaching-to-test ≠ validation. A lift/generalization claim needs a HELD-OUT cell (canonical unseen at prompt-freeze); run-first-then-read keeps even the right-reason judgment unprimed. An in-distribution catch is confirmation at most — label it IN-DISTRIBUTION, never as lift. And state precisely WHAT was held out (mechanism vs sub-shape): a specialist only generalizes within sub-shapes its prompt enumerated.

### k3-union-audit-config-is-not-k1-production-rate | fired:1 | 2026-06-18
Reported a k=3 `--no-merge` union recall (the FLAKINESS-AUDIT config — best-of-rolls, paraphrase groups unioned) as if it were the production catch rate. Production = k=1 per target (one roll via the unbuilt Opus DETECT orchestrator). The union inflates recall vs a single production roll → a "99%-looking" audit number masks the true ~86% k=1 floor.
Prevent: when reporting a recall number, STATE the config that produced it (k; --no-merge/union vs semantic_merge; audit vs production) and NEVER quote a k≥3 union number as the production rate. Production recall = k=1 per target; the k≥3 union is diagnostic only. With --no-merge, report the MAX single finding-group roll-count, never a paraphrase-sum across groups.