# run_bench 17-cell recall — fresh k=1, MEASURED through orchestrator/detect.py

audience: AI coding agents first. Result-first; honesty caveats are load-bearing, not decoration.

## Result (BLUF)

**17/17 right-reason real-catch on one fresh k=1 run_bench pass** (sonnet / effort=medium, the production config).
Measured THROUGH `orchestrator/detect.py` `run_bench` (the directive's required path), one call per cell with a
unique out_dir so each cell's report+titles persist (a single multi-cell out_dir collides on the `vuln.ts`
basename). run_bench is per-cell independent, so the union over single-cell calls == a whole-corpus call's recall.

- **Automated scorer: 6/6 = 1.0** over auto-scorable cells. The hardened `bench.is_flagged` auto-credits only
  no-false-credit matches; it PUNTS the other 11 to hand-judge rather than risk crediting a wrong-reason finding.
- **Hand-judge over THIS run's titles: 11/11 punts CAUGHT right-reason.** Every punt has a top finding naming the
  exact canonical defect (symbol + why + class). Bar = "flagged ≠ caught": must name the specific defect.
- **Combined headline = 6 auto + 11 hand = 17/17.** 0 genuine misses this sample.

This satisfies the acceptance test (advisor reframe): ≥99% of the 17 cells catching in ONE k=1 run_bench pass
(16/17 = 94.1% < 99%, so the bar is effectively 17/17).

## HONESTY — what this is and is NOT

- **POINT ESTIMATE, not a durability cert.** A single k=1 sample cannot certify a per-cell rate (rule of three:
  even 3/3 leaves a wide CI; one sample far more so). 17/17 means *this measured pass* caught all 17, not that
  every future k=1 will.
- **Deferred ≤1% = documented flaky sub-shapes** (per the goal's own "defer the hard-to-detect 1%"). The
  membership-of-a-body-reference S3 sub-shape (assignee cells) was CAUGHT this sample (auto-HIT, contiguous
  `assignee_id`) but prior sweeps measured it ~1/3 — its durability is NOT established by one favorable sample.
- **The scorer under-credits by design (no-false-credit dominates).** 11 punts are the conservative path: a punt
  routes to hand-judge (ground truth at n=17), it never auto-marks a cell clean/caught. A punt is NOT a false-clean.

## Scorer change — `bench.is_flagged` (advisor-signed-off)

Robustified to stop under-reporting correctly-worded catches WITHOUT auto-crediting non-canonical matches:

1. **why-discriminator PUNT** — canonicals whose `why` says hand-judge ("hand-judged" / "not substring" /
   "credit only") return False (route to hand-judge). Generic tokens like bare "privilege escalation" would
   else false-credit. Empirical set: S2-invite, S2-role, S2-team-owner.
2. **code-identifier symbol** (single token, no whitespace) → contiguous substring (exact, safe).
3. **phrase symbol** (multi-word) → ALL distinctive tokens present in ONE title (≥2 distinctive required;
   `_STOP` drops only genuinely-generic words — security-distinctive words stay REQUIRED so a sibling finding
   naming the same symbol for a different reason cannot match).
4. **Dropped the class-id branch** — a bare class tag (s1..s11) in a title is never a right-reason catch
   (verified: it fired on zero cached titles, so removal flips nothing).

Why 11 punt despite being caught: the right-reason finding often paraphrases a distinctive symbol token
(symbol "reset token **single**-use" vs catch "reset token is never invalidated after **first use**" — "single"
absent). The all-distinctive rule abstains rather than guess. Advisor: "systematically misses valid paraphrases —
that's safe; hand-judge fills the punts. Don't chase recall in the matcher."

Locked by pytest (93 passed) + this measured run.

## Per-cell ledger (fresh k=1, this run)

| cell | class | scorer | hand-judge | catching finding (this run) |
|------|-------|--------|------------|------------------------------|
| S3-calendar-tenant-isolation-idor | S3 | AUTO-HIT | — | getSchedulingConnectionById unscoped tenant read |
| S3-report-schedule-tenant-isolation-idor | S3 | AUTO-HIT | — | getReportScheduleById unscoped tenant read |
| S3-session-tenant-isolation-idor | S3 | AUTO-HIT | — | touchSession cross-tenant write |
| S3-task-assignee-membership-idor | S3 | AUTO-HIT | — | assignee_id not validated vs tenant membership |
| S3-task-assignee-membership-zyncapi | S3 | AUTO-HIT | — | assignee_id membership not validated |
| S8-attachment-cross-tenant-idor | S8 | AUTO-HIT | — | hardDeleteAttachmentRow no tenant predicate |
| S1-reset-token-reuse | S1 | punt | CAUGHT | "Reset token is never invalidated after first use" [critical] |
| S1-barrel-2fa-on-refresh | S1 | punt | CAUGHT | "2FA enforcement silently dropped on every token refresh" [critical] |
| S1-xfile-2fa-on-refresh | S1 | punt | CAUGHT | "2FA requirement bypassed on every token refresh" [critical] |
| S10-oauth-code-replay | S10 | punt | CAUGHT | "Authorization code accepted non-atomically — replay via TOCTOU" [critical] |
| S2-invite-privilege-escalation | S2 | punt(why) | CAUGHT | "Unconstrained roleId enables privilege escalation via invitation" [critical] |
| S2-role-privilege-escalation | S2 | punt(why) | CAUGHT | "Permission ceiling not enforced on role create or update" [critical] |
| S2-team-owner-invariant | S2 | punt(why) | CAUGHT | "PATCH /team/:id/role can demote the tenant owner, bypassing the ownership-transfer guard" [high] |
| S4-kb-spaceids-sqli | S4 | punt | CAUGHT | "SQL Injection via sql.raw() in listArticlesBySpaceIds" [critical] |
| S5-email-html-injection | S5 | punt | CAUGHT | "Unescaped HTML injection — all text vars interpolated raw into HTML output" [critical] |
| S6-jira-baseurl-ssrf | S6 | punt | CAUGHT | "SSRF via unvalidated baseUrl with credential forwarding" [critical] |
| S7-webhook-sig-failopen | S7 | punt | CAUGHT | "Signature check fail-open when app_secret is falsy" [critical] |

Right-reason discipline on the 3 hardest (why-discriminator) cells: each catch names the *specific* canonical
backdoor (roleId capability-handle / create-update permission ceiling / owner-demotion-bypassing-transfer), NOT a
generic "coarse RBAC" note that would also fire on safe.ts. Bar held; not loosened under goal pressure.

## Reproduce

Per-cell driver `/tmp/runbench_chunk.py <cid|ALL>` (resumable, foreground-chunkable under the 10-min cap), results
in `/tmp/runbench/<cid>.json`. Each cell = `detect.run_bench([cell], …, k=1, model="sonnet", effort="medium")`.