# Specialist-lift pilot — does a per-domain detector lift catch-rate over baseline?

audience: AI coding agents first. BLUF-ordered, [MEASURED]-tagged. SoT for the load-bearing premise of the
modular-domain-detectors build (`docs/specs/2026-06-18-modular-domain-detectors-design.md` §"Architecture
decisions" — ACCEPTED "baseline always-on + matched specialists … specialists lift catch-rate"). That premise was
UNTESTED before this pilot; the whole 12-detector fan-out rests on it. This pilot tests it on n=4 cells across 2
domains (access-control S2/S3 + auth S1): one HELD-OUT LIFT cell (S2-invite), two HELD-OUT NEUTRAL cells
(S3-session, S1-reset-token), one in-distribution confound (S2-role). Held-out = prompt frozen before that cell's
canonical was ever read.

## BLUF — the premise HOLDS, validated on a HELD-OUT cell, and it is sub-shape-dependent

[MEASURED] An `access-control` specialist prompt (S2/S3/S8 depth, blind-authored OWASP A01) lifts the subtle
"authorized-but-insufficient" escalation sub-shape and is neutral on the already-recognizable tenant-IDOR sub-shape.
The decisive datum is the **held-out** cell:

- **HELD-OUT — S2-invite-privilege-escalation: baseline 0/5 → specialist 4/5, right-reason.** The prompt was
  committed BEFORE this cell's `canonical.json` was ever read, and the cell's mechanism is HARDER than anything the
  author saw (the granted role hides behind a `roleId` indirection / cross-file `getPermissionsForRole`, not an
  inline `permissions[]`). The specialist reasoned "`roleId` is a capability handle needing a ceiling check" — the
  exact move the canonical says the detector must make. Baseline found it 0/5 (a low rate on one draw, NOT structural
  blindness — baseline's Pass 3 covers escalation; its 15 findings here were all invitation-lifecycle).
  **This is generalization ACROSS MECHANISM within an enumerated sub-shape, held-out.** Pass E enumerates "grant or
  ASSIGN a role you do not hold"; S2-invite is a second, harder-mechanism instance of that same enumerated sub-shape
  (roleId indirection vs the inline `permissions[]` of the cell the author saw). So the lift comes from **sub-shape
  enumeration in the prompt**, not domain-magic. Corollary, load-bearing for the fan-out: each specialist's value is
  bounded by how well its author enumerates that domain's subtle sub-shapes — this is NOT evidence a specialist lifts
  a sub-shape its author never enumerated.
- IN-DISTRIBUTION — S2-role-privilege-escalation: baseline 1/5 → specialist 4/5. CONFOUNDED (teaching-to-test): the
  author read this cell's canonical before writing Pass E, so Pass E encodes its fix. Counts only as in-distribution
  confirmation, NOT as evidence of lift. The held-out cell is what earns the claim.
- NEUTRAL — S3-session-tenant-isolation-idor (recognizable missing-predicate IDOR): baseline 3/5 → specialist 3/5.
  No recall lift; baseline already handles it. The specialist does escalate severity (high→critical) and emits
  fewer, on-domain findings — but its precision/FP rate is UNQUANTIFIED (it also emits unverified depth findings on
  the FIXED file, 2–3/5; see Honest limits — a build-time cost to measure, NOT a settled "cleaner output" win).
  Uncontaminated (its canonical was not used to author the prompt) — so the neutrality is a real finding, not a confound.
- HELD-OUT NEUTRAL — S1-reset-token-reuse (auth domain, recognizable single-use-token sub-shape): baseline 5/5 →
  specialist **4/5 max-group** (the canonical critical splits two phrasings 4/5 + 1/5; ~5/5 under semantic_merge).
  No meaningful regression (4 vs 5 is within k=5 noise) and NO recall headroom — baseline already nails the
  reset-token-replay sub-shape. The `auth` specialist (blind-authored OWASP A07, FROZEN before its canonical was read)
  adds on-domain DEPTH (stateless-reset-JWT survival of `bumpUserVersion`; second-reset non-revocation) but does not
  lift recall. A 2nd NEUTRAL data point in a 2nd domain — confirms the baseline holds the recognizable floor ACROSS
  domains, NOT a 2nd lift point. (See Honest limits: the auth corpus's subtle sub-shapes are cross-file, so a
  prompt-only held-out LIFT cell in auth was not available — this neutral was the only single-file auth held-out.)

All four pass the safe.ts discriminator (the canonical critical fires 0/5 on the fixed file; no discriminator-trap
FP). Config: k=5 `--no-merge`, blind `/tmp/sg_cfg`, v2@sonnet/MEDIUM, single-file (no deps → prompt-only isolation).
Recall judged by MAX single finding-group roll-count (roll-count rule — NOT paraphrase-sum); semantic-merge would
raise the specialist numbers toward ~5/5.

**What this means for the fan-out:** specialists earn their keep on the HARD tail — the subtle "guard-present-but-
insufficient" / capability-handle escalation sub-shapes that ARE the flaky ~14% gap to 99% — and are validated to
generalize ACROSS MECHANISM within an enumerated sub-shape (held-out), NOT to a sub-shape the author never wrote
down. They do NOT lift recall on sub-shapes the baseline already recognizes (recognizable IDOR). So the fan-out
value is real but is bounded by prompt sub-shape coverage — reason per-sub-shape, do not assume flat across all 12.

## Method

- Cells: two contaminated escalation cells (`S2-role`, `S2-invite`) + one uncontaminated IDOR cell (`S3-session`)
  from the access-control corpus, + one uncontaminated single-use-token cell (`S1-reset-token-reuse`) from the auth
  corpus. `S2-invite` and `S1-reset-token` are HELD-OUT — each cell's `canonical.json` was NOT read until AFTER both
  arms were run (access-control prompt frozen in commit 5e77754; auth prompt frozen in commit 43cadc5). The held-out
  property is the freeze; the run-first-then-read order keeps even the right-reason JUDGMENT unprimed.
- Single-file shape chosen deliberately → no deps-as-context → the only variable is the PROMPT (clean isolation).
- Head-to-head: same cell, same conditions, `--detector baseline` k=5 vs `--detector access-control` k=5, plus the
  specialist on safe.ts k=5 (the discriminator / GREEN half — RED-without-GREEN is a logged mistake).
- Catch = the canonical defect flagged with right-reason (canonical.json `why`), hand-judged on the rolls.

## [MEASURED] Results (k=5, max single finding-group roll-count)

### HELD-OUT — S2-invite-privilege-escalation (capability-handle escalation, canonical unseen at prompt-freeze)
- Canonical (`fix_sha 2a70b38`): POST /api/auth/invite passes attacker-controlled `roleId` to `createInvitation`
  with no check the inviter may confer that role → any `users:invite` holder grants OWNER / a cross-tenant role. The
  detector must treat `roleId` as a capability handle (the perms hide behind `getPermissionsForRole`, cross-file),
  not scan an inline `permissions[]`. Discriminator: the public accept endpoint is correctly public (token hash IS
  the credential) — a trigger-happy detector mis-flags it missing-auth.
- **baseline RED: 0/5** — all 15 findings invitation-lifecycle; closest is `[medium] roleId not validated as
  belonging to tenant` 1/5 (tenant scope, NOT the privilege-ceiling escalation).
- **access-control RED: 4/5** — `[critical] Invited roleId unconstrained by actor's own privilege level` (+ a 2nd
  ceiling phrasing 1/5). Right-reason: roleId-as-capability-handle ceiling, the exact canonical move.
- **GREEN (safe.ts): pass** — the "unconstrained roleId / no ceiling" critical fires 0/5; the top residual is a
  DIFFERENT subtler finding (`ceiling enforced against stale JWT claim, not live DB` 3/5) — canonical-clean ≠
  globally-clean, NOT a re-raise. The public accept endpoint is NOT mis-flagged missing-auth (discriminator avoided).
- **LIFT: 0/5 → 4/5, HELD-OUT, harder mechanism than authored-against. The premise's load-bearing evidence.**

### IN-DISTRIBUTION (confounded) — S2-role-privilege-escalation
- Canonical: POST/PATCH /roles let a `users:manage` holder grant permissions BEYOND their own set (fix =
  `permissionsExceedActor`→403).
- **baseline RED: 1/5 → access-control RED: 4/5.** CONFOUNDED — the author read this canonical before writing Pass E
  ("permission ceiling: requested ⊆ actor"), so Pass E restates this fix. Counts as in-distribution confirmation,
  not lift evidence. (Baseline also wobbled: 2/3 prior sweep, 1/5 here — single point-estimates are noisy.)
- **GREEN (safe.ts): pass** — ceiling critical 0/5; residuals are deeper escalation paths the fix didn't address
  (not FPs).

### NEUTRAL (uncontaminated) — S3-session-tenant-isolation-idor (recognizable missing-predicate IDOR)
- Canonical: `touchSession` UPDATE-by-id with no tenant predicate → cross-tenant session-mutation IDOR (write).
  Discriminator: by-token-hash lookups legitimately lack tenant scope (token hash IS the bearer credential).
- **baseline RED: 3/5 → access-control RED: 3/5** (+ `[critical] cross-tenant session mutation` 2/5 → severity
  escalated; 4 findings vs baseline's 10 — fewer, but precision/FP NOT measured, so "cleaner" is unproven).
- **GREEN (safe.ts): pass** — cross-tenant IDOR critical 0/5; by-token-hash lookups NOT mis-flagged.
- **NO recall lift (3/5 → 3/5).** Uncontaminated → the neutrality is a real result: baseline already handles
  recognizable IDOR; the specialist adds severity/precision, not recall, here.

### HELD-OUT NEUTRAL — S1-reset-token-reuse (auth domain, recognizable single-use-token sub-shape)
- Canonical (`fix_sha 59ec337`): password-reset token not invalidated after use → unlimited reuse within TTL
  (replay → account takeover). The `auth` specialist prompt (auth.prompt.txt + detector.json) was FROZEN at commit
  43cadc5 BEFORE this canonical was read.
- **baseline RED: 5/5** — `[critical] Reset token is not invalidated on use — unlimited reuse` every roll (+
  out-of-domain rate-limit/CSRF/timing findings at lower rolls).
- **auth RED: 4/5 max-group** — `[critical] Reset token never invalidated after use — replay enables account takeover`
  4/5 (+ an alt single-use phrasing 1/5; ~5/5 under semantic_merge). NO regression (4 vs 5 within k=5 noise). Plus
  on-domain DEPTH baseline did not surface: stateless-reset-JWT survival of `bumpUserVersion` (1/5); second-reset
  non-revocation (1/5). Fewer out-of-domain findings than baseline = on-domain focus.
- **GREEN (safe.ts): pass** — the reset-token-replay critical fires 0/5. Residuals are DISTINCT subtler issues, not a
  canonical re-raise: JTI single-use consume non-atomic (3/5, a real check-then-act race = TRUE POSITIVE, not FP),
  expiry delegated to cross-file `verifySignedToken` (2/5, unverified cross-file dependency), rate-limiter fail-open
  (5/5). git-fix safe.ts is canonical-clean, NOT globally-clean.
- **NO recall lift (baseline 5/5, auth 4/5 ≈ 5/5).** Recognizable single-use-token sub-shape the baseline already
  nails; the auth specialist matches it + adds depth, no recall headroom. 2nd NEUTRAL data point, 2nd domain.

## Honest limits

- **n=4 cells across 2 domains; the held-out LIFT is still n=1 (S2-invite); the held-out NEUTRAL is now n=2
  (S3-session IDOR + S1-reset-token across 2 domains).** The CONTRAST (held-out lift on a subtle escalation sub-shape,
  neutral on recognizable sub-shapes the baseline already handles) is the robust signal; exact rates are noisy (the
  baseline 2/3→1/5 escalation swing proves a single k=5 estimate has large variance).
- **Auth corpus gap (why there is no 2nd held-out LIFT point).** The auth domain's subtle "guard-present-but-
  insufficient" sub-shapes (e.g. 2FA-secret-zeroed-on-refresh) exist in the corpus only as CROSS-FILE Shape-B cells
  (the insecure default lives in an imported symbol) — on a single-file `vuln.ts` BOTH arms miss (resolution gap, not
  a prompt-lift gap). The only single-file auth held-out cell available (S1-reset-token) is a RECOGNIZABLE sub-shape
  the baseline already nails → it tests the NEUTRAL prong, not the lift prong. A prompt-only held-out LIFT data point
  is not obtainable in auth with the current corpus; it would require either a different domain with a subtle
  single-file cell or a cross-file lift test (deps-as-context, which also exercises untested cross-file specialist
  behavior). Cell-shopping for a single-file cell that happens to show lift would be outcome-shaped selection — NOT
  done.
- The held-out LIFT claim rests on ONE cell. It is genuine held-out (canonical unread at freeze, harder mechanism),
  which is why it carries the premise — but a fuller blind-author + held-out-validate sweep (#22) across more
  escalation cells and other domains is still owed before trusting the fan-out flatly.
- Reported recall = max single finding-group (roll-count rule). Production uses semantic_merge → specialist numbers
  likely higher. Single-file cells only; cross-file specialist behavior (deps-as-context) NOT tested here.
- **Specialist precision / FP rate is UNQUANTIFIED — a build-time cost, not a pilot result.** On the FIXED files the
  specialist still emits depth findings (S2-invite `ceiling enforced against stale JWT claim` 3/5; S2-role
  in-handler-guard delegation 2/5). For the pilot's question (recall lift) these are correctly OUT of scope and the
  GREEN classification holds (the canonical critical fires 0/5). But at fan-out scale these unverified-depth findings
  ARE the precision/dedup cost already on the open-problems list — measure FP rate per specialist before trusting the
  merged union as "clean." (The auth specialist's safe.ts residuals — JTI-consume race 3/5, cross-file expiry 2/5 —
  were classified TP/cross-file-dependency, NOT auto-FP; classification, not a measured FP rate, so it does not close
  this gap.)

## Implication for the build (the earned checkpoint)

The fan-out premise is VALIDATED on n=1 domain held-out for LIFT (access-control / S2-invite: a focused specialist
generalizes ACROSS MECHANISM within an enumerated sub-shape — it caught a harder-mechanism instance of the escalation
sub-shape its prompt enumerated), and the baseline-holds-the-recognizable-floor finding is now confirmed across 2
domains (S3 IDOR + S1 reset-token, both NEUTRAL held-out). The method is proven; the value is sub-shape-dependent,
not flat, and bounded by how well each specialist's author enumerates its domain's subtle sub-shapes.

**Earned conclusion, and the deliberate stop.** A 2nd held-out LIFT data point is NOT obtainable by drawing more
single-file cells without outcome-shaped selection (see Honest limits — auth corpus gap), and a 2nd hand-picked lift
cell would add an instance, not information. The decision the fan-out actually turns on is precision/FP at scale and
whether an orchestrator can route the matched specialist to the right target — neither of which any further recall
cell touches. So specialist authoring STOPS here pending the user's fan-out decision; the productive non-gated next
step is the **Opus DETECT orchestrator** (`orchestrator/detect.py`, its own spec+plan): it routes baseline + matched
specialists per target by the existing `applies_to` signals and semantic-merges, and it is the harness that finally
measures k=1 production recall — on the critical path to 99% REGARDLESS of how many specialists exist. Then:
re-measure k=1 production recall across the union (target 99%) AND quantify per-specialist precision/FP rate (the
depth-finding cost flagged in Honest limits — recall lift is worthless if it floods the union with unverified
findings). 99%-at-k=1 is NOT met (~86% bare baseline) — it remains the open frontier. The fan-out scope itself —
#16/#36/oracle-lexicon priority and committing to all 12 specialists — is the user-gated one-way-door lever: surface,
do not auto-commit. (Spec-vs-memory conflict to surface, not silently resolve: the modular-domain-detectors spec
declares "#16+#22 unified, ungated" while memory records #16 prompt rewrite as advisor-forbidden-autonomous and the
fan-out scope as user-gated.)