# Baseline detector generalization — recall measurement (#16)

audience: AI coding agents first. BLUF, [MEASURED] tags, no-false-coverage.

## BLUF

The single finance-titled LLM prompt (`security-generalist.prompt.txt`, "Payments & Financial Module") was SPLIT into
two drop-in detectors: an always-on, domain-agnostic **`baseline`** (full S1–S11 / OWASP breadth) and a **`finance`**
money-depth specialist (S9, payments submodule). `gate.py` now defaults to `baseline` and selects detectors by id
(`--detector <id>` → `<id>/detector.json` → its prompt).

**[MEASURED] baseline catches 17/17 LLM corpus cells right-reason AND passes the safe.ts discriminator 17/17 — at the
k=3 union FLAKINESS-AUDIT config** (10 cells 3/3, 7 cells 2/3), no regression vs the finance prompt, plus a measured
IMPROVEMENT on the one known-hard sub-shape. This is the #16 deliverable (generalize the LLM leg beyond payments)
MEASURED.

**[MEASURED] k=1 PRODUCTION reading — the "catch the bug in 1 go" standard — is ≈86%, NOT 99%. Do not report the k=3
17/17 as the production catch rate.** Production = k=1 per target (the unbuilt Opus orchestrator); k=3 union is the
audit, not production (SoT: `docs/specs/2026-06-18-modular-hierarchy-design.md` §"Strategy placement"). Expected
single-roll recall = (10×1.0 + 7×0.667)/17 = 86%; strict-reliable (3/3-only) = 10/17 = 59%. n=3 rolls/cell → noisy
point estimate, but the direction (well below 99% at one go) is robust. The 7 cells at 2/3 are the real near-term gap
(~14%), addressed by the orchestrator's per-domain targeting + deps-as-context — NOT the deferred elusive ~1%.

## What "real catch" means here
Right-reason judgment against each cell's `canonical.json.why` (the user's standard: a real catch = the true defect
flagged for the right reason). bench.py substring auto-match is used only where `canonical_symbol` is a literal code
symbol; for phrase-canonicals (e.g. S5 "escape email interpolation") it underflags, so catch is hand-judged on the
k=3 union of finding titles + bodies. Blind config (`--config-dir /tmp/sg_cfg`), k=3, `--no-merge`.

**Roll-count is the MAX single-entry roll-count, NOT a paraphrase union.** With `--no-merge`, one roll can emit the
canonical under two paraphrases that land in two groups; summing them ("2/3 + 1/3 = 3/3 union") double-counts and
overstates robustness. Report the highest single finding-group's roll-count. (Corrected 2026-06-18 per advisor.)

**Blind-config creds expire — re-probe before trusting any result.** `--config-dir /tmp/sg_cfg` carries its own
`.credentials.json`; an expired token makes `claude -p` return empty → the gate emits ZERO findings, which reads
identically to a clean pass. ALWAYS probe first: `CLAUDE_CONFIG_DIR=/tmp/sg_cfg claude -p --model sonnet
--dangerously-skip-permissions 'PONG'`. On 401: `cp ~/.claude/.credentials.json /tmp/sg_cfg/.credentials.json`.
Interleave a known-catch vuln canary in every sweep as a live-creds proof (an all-zero safe.ts sweep is a 401 tell).

## [MEASURED] Recall — baseline on the 15-cell LLM corpus (2026-06-18)

| class | cell | canonical | catch (k=3) |
|-------|------|-----------|-------------|
| class | cell | canonical | catch (k=3, max single-entry) | discriminator (safe.ts) |
|-------|------|-----------|-------------|-------------|
| S1 | reset-token-reuse | reset token single-use | ✓ 2/3 | ✓ PASS |
| S1 | xfile-2fa-on-refresh | 2FA dropped on refresh (cross-file builder) | ✓ 3/3 (resolver pulled builder) | ✓ PASS |
| S1 | barrel-2fa-on-refresh | same, through a barrel | ✓ 3/3 (barrel resolved) | ✓ PASS |
| S2 | invite-privilege-escalation | roleId capability handle needs ceiling check | ✓ 3/3 | ✓ PASS |
| S2 | role-privilege-escalation | grant perms beyond actor's own | ✓ 2/3 | ✓ PASS |
| S2 | team-owner-invariant | owner-demotion backdoor in role-change path | ✓ 2/3 (strict canonical) | ✓ PASS |
| S3 | calendar-tenant-isolation-idor | getSchedulingConnectionById no tenant predicate | ✓ 3/3 | ✓ PASS |
| S3 | report-schedule-tenant-isolation-idor | getReportScheduleById no tenant predicate (+self-rationalizing comment) | ✓ 3/3 | ✓ PASS |
| S3 | session-tenant-isolation-idor | touchSession no tenant predicate (write IDOR) | ✓ 2/3 | ✓ PASS |
| S3 | task-assignee-membership-idor | absent getMembership on body FK assignee_id | ✓ 2/3–3/3 — **was 1/3 (finance)** | ✓ PASS |
| S3 | task-assignee-membership-zyncapi | same, FK spread via `input` (harder) | ✓ 2/3 — **was 1/3 (finance)** | ✓ PASS |
| S4 | kb-spaceids-sqli | `listArticlesBySpaceIds` sql.raw ARRAY interpolation (was a named gap) | ✓ 3/3 | ✓ PASS |
| S5 | email-html-injection | unescaped {{}} interpolation → XSS | ✓ 2/3 | ✓ PASS |
| S6 | jira-baseurl-ssrf | `jiraAdapter` tenant `baseUrl` unguarded fetch (was a named gap) | ✓ 3/3 | ✓ PASS |
| S7 | webhook-sig-failopen | sig verify fail-open when app_secret falsy | ✓ 3/3 | ✓ PASS |
| S8 | attachment-cross-tenant-idor | hardDeleteAttachmentRow missing tenant scope | ✓ 3/3 | ✓ PASS |
| S10 | oauth-code-replay | auth-code single-use non-atomic (TOCTOU) | ✓ 3/3 | ✓ PASS |

17/17 caught at the k=3 union audit, every cell ≥2/3: 10 at 3/3, 7 at 2/3. "Robust" is k=3-union ONLY — at k=1
production each 2/3 cell misses ~1 roll in 3 (expected single-roll recall ≈86%, see BLUF). 17/17 discriminator PASS
(see next section).
S4 (SQLi) + S6 (SSRF) added 2026-06-18 as real corpus cells (RAW git: S4 `9b0c485`, S6 `bec8bc8`) — the two
previously-named blind spots, now MEASURED both sides (point-estimate, n=1/class — a catch, NOT yet a rate).

## [MEASURED] Discriminator — baseline on safe.ts (right-reason, not trigger-happy) (2026-06-18)

**Why this exists.** Recall on `vuln.ts` alone is gameable: a detector that flags *every* by-id query as IDOR also
scores 15/15. Each cell ships a `safe.ts` (the fix commit's version) plus discriminator warnings. The fair metric
(per advisor + the project's `green-residual-classify-tp-fp` rule): per cell, **does baseline re-raise THE CANONICAL
defect on the FIXED file?** Canonical-on-safe = the catch was pattern-matching, not understanding. Other findings on
`safe.ts` are NOT automatically false positives — `safe.ts` can hold genuine unrelated/residual issues.

**[MEASURED] 15/15 cells PASS: baseline does NOT re-raise any canonical on its fixed file.** Run blind (`--config-dir
/tmp/sg_cfg`, creds re-probed live), k=3, `--no-merge`. The catches are right-reason. Strongest evidence that the
model *understands* the fix rather than pattern-matching the shape:

- **S2-role**: on the fixed file baseline states verbatim "the anti-escalation invariant **enforced in POST/PATCH**
  is bypassed via the **DELETE** path" — it recognizes the canonical fix is present and instead surfaces a *genuine
  residual* on the unfixed DELETE `reassign_to_role_id` path. Discrimination, not re-flagging.
- **S3-task-assignee-idor** (hardest cell): `vuln` caught 3/3; `safe` recognizes the membership/project/status checks
  now exist (flags only a TOCTOU *on* them), does not re-raise "assignee_id unvalidated".
- **S3-task-assignee-zyncapi**: `safe` does not re-flag the fixed `assignee_id`; it generalizes the same
  membership-of-a-body-reference class to *different* fields (`reporter_id`, `/bulk statusId`) — genuine same-class
  residuals on fields the canonical never enumerated, not a canonical re-flag.
- **S8 / S10 / S7**: `safe` flags a different axis (S8 uploader-ownership vs the cross-tenant canonical), or honestly
  hedges the now-cross-file invariant as "unverifiable" (S10 replay atomicity, S7) — never fabricates the canonical.

**Creds-bug caught mid-measurement (recorded for honesty).** The first safe.ts pass returned all-zero findings —
which looked like a clean discriminator pass but was an expired blind-config token (401 → empty `claude -p` output).
Re-probe + vuln-canary interleave (above) caught it; results here are post-refresh, with live-creds proven by the
S8/S5/S3-task vuln canaries re-catching their canonical in the same sweep.

### Open precision items (NOT discriminator failures, but recorded)
- **Parser noise:** attack-narrative bullets and `Title:**` / `[HIGH]` / `[LOW]` prefixes leak into the FINDING list
  as `[unrated]`/duplicate entries. This is the known precision/dedup problem (band-2 semantic-merge), orthogonal to
  recall and to the discriminator. Does not affect any catch or any PASS verdict.
- **Same-class residuals on `safe.ts`** (S2-role DELETE path, S3-zyncapi `reporter_id`/`/bulk`): plausibly genuine
  TPs (narrow fix commits), but unverified against the fix diff. Classify per `green-residual-classify-tp-fp` before
  citing as either coverage wins or noise.

## [MEASURED] Before/after vs the finance prompt (no regression)
- Head-to-head on S3-calendar + S5-email, finance vs baseline, k=3: both prompts catch the canonical; **baseline ≥
  finance** on roll-count (S3 3/3 vs 2/3) and adds true findings (credential exposure, onConflict tenant re-assert).
- The one known weak spot (taxonomy/security.md): the `assignee_id` membership-of-a-body-reference sub-shape was
  MEASURED 1/3 CONSISTENT with the finance prompt (n=2: zync-api + public-api). With baseline it is **2/3 on both**.
  The generalized Pass 3 ("when the request references a SECOND object — assignee, member, target — is its
  membership/ownership validated, not just the actor's own?") directly closes that sub-shape. Improvement, not
  regression.

## no-false-coverage — what this does NOT yet measure
- Corpus now spans S1/S2/S3/S4/S5/S6/S7/S8/S10 (LLM) + S9 (oracle, deterministic) + S11 (deps/headers,
  deterministic) — **every S1–S11 class has a measured detector.** S4/S6 (added 2026-06-18) are n=1/class
  point-estimates (a measured catch both sides, NOT yet a recall RATE — a RATE needs n≥3 cells/class per
  create-tests). 100% here = 100% on this 17-cell denominator, which is right-reason-validated but small.
- **Still a point estimate, not a population rate.** n≥3 cells/class (esp. for S4/S6) is the next rigor step. Two
  DISTINCT gaps, do not merge them: (a) the k=1 same-model FLAKY tail — the 7×2/3 cells, ~14% — closed by the Opus
  orchestrator's per-domain targeting; (b) the deferred ELUSIVE ~1% (cross-model dice) + band-2 cross-file *precision*.
- k=3 union is the FLAKINESS-AUDIT config, NOT production. Production = k=1 per target via the Opus orchestrator (SoT:
  `docs/specs/2026-06-18-modular-hierarchy-design.md` §"Strategy placement"; `docs/specs/2026-06-18-modular-domain-detectors-design.md`).
  k=1 expected recall on this corpus ≈86%; the gap to 99% is gap (a) above, distinct from the deferred elusive ~1%.
- This measures the `baseline` detector standalone. The per-domain specialists (auth/access-control/injection/xss/
  ssrf/webhook/finance/…) and the Opus DETECT orchestrator that unions baseline + matched specialists are the
  remaining spec work — gap-filled only against a MEASURED miss (there are none on the current corpus).

## Reproduce
```
cd <repo>; for cell in <corpus dirs>; do \
  python3 orchestrator/gate.py domains/security/corpus/$cell/vuln.ts \
    --config-dir /tmp/sg_cfg --k 3 --no-merge --report /tmp/sg_${cell}.md; done
# finance head-to-head "before" used the retired combined prompt (git: `git show 54163ad:domains/security/detectors/security-generalist.prompt.txt`).
# going forward, compare baseline vs the finance specialist: --detector finance
```