# Measured-recall RATE gate — `recall_gate.py` (item #5)

date: 2026-06-20 · gate: `recall_gate.py` (repo root, standalone — NOT bench.py) · SoT: `domains/security/recall/records.json` · test: `tests/test_recall_gate.py` (11/11 green) · wired: `check.sh` gate #4

## BLUF
The project's load-bearing LLM-validation doctrine — **"recall measured STATISTICALLY, k≥3, NEVER a single-run
claim"** — was enforced only by prose: `bench.py:72` hardcodes `rate_validated:["S2","S3"]` with no machine backing,
and `ledger.py` deliberately disclaims grading rate (`sample` grades ground-truth DEPTH, "that measurement is bench's,
never derivable here", `ledger.py:218-219`). `recall_gate.py` is the deterministic test that proves the rate claim is
real — the statistical analogue of `ledger.py --check`'s structural 100%-proof. **MEASURED GREEN:** 16 cells
rate-validated at k≥3 bound to the current prompt, 0 hard failures, 3 cells honestly flagged UNMEASURED-at-rate; the
prompt-hash binding reds all 16 STALE on a one-byte prompt drift (demonstrated). Conformance 11/11.

## What it enforces
Over `domains/security/recall/records.json` — each record `{cell_id, detector, catches, k, metric, measured_against, …}`:

**HARD (exit 1):**
- `SINGLE_RUN` — `k<3`. The exact "NEVER a single-run claim" doctrine, as a machine gate.
- `STALE` — `measured_against != sha256(current detector prompt)`. **The load-bearing honesty fix.** Measured recall
  is a function of the prompt; a frozen `catches/k` carrying only a date is an assumption wearing a measurement's
  clothes. Binding to the prompt hash is the no-false-clean DUAL for measured recall — the rate is valid only while
  the prompt it was rolled against still exists. (This also **operationalizes the #14/#15 apply-gates**: both candidates
  edit `baseline.prompt.txt` → every bound record flips STALE → this gate reds until the rolls are re-run. The gate IS
  the "re-measure after any LLM-behavior change" enforcement.)
- `UNKNOWN_CELL` — record names a cell absent from the corpus (no-false-coverage).
- `MALFORMED` — missing field (incl. an absent `metric` — an unlabeled rate is rejected, see below), `catches>k`,
  `k<1`, unresolvable detector/prompt.

**REPORT-ONLY (exit 0; `--strict-floor` to harden):**
- `BELOW_FLOOR` — `rate < floor` (default 0.66 = the 2/3 bar floor) on a record that is NOT waived. The floor is
  **advisory + per-cell waivable** by design: the project ships some cells on no-regression BELOW the bar floor
  (S2-role 3/9, a documented pre-existing flaky cell), so a default hard floor would wrongly red a deliberately-shipped
  cell. A documented-flaky cell carries an explicit `waiver:{below_floor_ok:true, reason:…}` (mirrors the ratchet
  pattern) → `OK_WAIVED`. Note the asymmetry that proves the floor can't be a default hard gate: S2-role at k=9 *passes*
  k≥3 but *would fail* a 0.66 floor.
- `UNMEASURED` — corpus cells with NO record. Honestly unmeasured-at-rate (most cells are n=1 point estimates), surfaced,
  never silently counted as covered.

## Derived output (`--json`) — replaces the prose `bench.py:72`
The **rate-validated set** = every record that is well-formed, known-cell, k≥3 AND hash-current (floor-independent —
"statistically measured" is what validation means; the floor is a separate quality annotation). Current: 16 cells across
classes S1/S2/S3/S4/S5/S6/S7/S8/S10. This is the machine replacement for `bench.py:72`'s hardcoded `["S2","S3"]`.

## Ownership boundary (work split)
`recall_gate.py` derives the set **independently**. Editing `bench.py:72` to consume it and reconciling `ledger.py`'s
`sample` (ground-truth depth) against this measured-rate (orthogonal: rate-capable = *enough cells*; rate-validated =
*actually rolled k≥3 at the current prompt*) are **DOWNSTREAM cursor tasks** (#4/#7 own bench/ledger) — documented here,
not coded. Records live in the new `domains/security/recall/` dir (unambiguously this stream's, no collision with
cursor's corpus-discriminators).

## Seed provenance (faithful, never inflated)
`records.json` transcribes already-rolled measurements from `docs/validation/2026-06-19-citation-prompt-recall.md` — the
**only** validation doc measuring the CURRENT baseline prompt (the 2026-06-18 generalization doc measured the OLD
pre-citation prompt → would bind to a different hash and red STALE, correctly). Transcribing an existing measurement is
not a new single-run claim. The three controlled k=9 cells (S2-role 3/9, S3-report-schedule 9/9, S3-zyncapi 9/9) carry
their k=9 numbers; the rest carry the doc's k=3 per-roll-union values. **S3-calendar is intentionally absent** — the doc
records it only as the imprecise "≥2/3", which is not a faithful `catches/k`, so it stays honestly UNMEASURED rather than
transcribed loosely.

## The metric label is load-bearing (a rate with no metric is soft false-coverage)
Every record carries `metric: "per-roll-union"` and the field is **REQUIRED** — an absent metric is a hard `MALFORMED`,
because the same cell reads two wildly different numbers under two defensible recall metrics:
- **per-roll-union** (what this seed records): the k=1 right-reason catch probability — *P[at least one roll names the
  cell, right-reason]*. This is the citation doc's faithful column.
- **max-single-entry**: the single literal-title max. This is the metric behind **CLAUDE.md's k=1 PRODUCTION ≈86%
  headline** and the 2026-06-18 generalization doc.

For diversified-phrasing cells the two diverge **sharply**: `S3-report-schedule` and `S3-zyncapi` are **9/9
per-roll-union** but **≈1/9 max-single** (the new prompt's added structure diversifies per-roll phrasing, so a naive
literal-title max undercounts). An unlabeled `rate=1.000` sitting next to the 86% headline is a flat contradiction —
and "1.000 on a cell whose max-single is 1/9" is itself the soft false-coverage this gate exists to prevent. Hence the
label rides next to every rate in the report and in `--json`. **Consequence for the downstream `bench.py:72` reconcile:
align on the METRIC, not just import the values.** `bench`'s historical recall notion is the max-single line (that is
where 86% comes from); these exported rates are per-roll-union. Merging the two quantities without aligning the metric
silently conflates incompatible numbers — `--json`'s per-cell `metric` makes that impossible to do by accident. (No
re-roll and no re-litigation of which metric is "correct": per-roll-union is a defensible k=1 catch probability; the
defect this fixes was the *unlabeled* number, not the number.)

## RATE-PENDING (no-false-coverage, the documented gap)
The full-corpus honest recall RATE under fresh k≥3 rolls is the measurement step this session does not perform (directive:
do not roll). What ships is the GATE + schema + conformance + a faithfully-seeded SoT that operates on REAL transcribed
data. Filling the 3 UNMEASURED cells (S3-calendar at a precise k≥3; S12/S13 finance) and re-rolling on any prompt change
are the open follow-ups the gate now makes impossible to forget (STALE reds the moment the prompt moves). This is "real
completion" under the locked done-bar (mechanism + green on available data + documented gap), not a deferral.