# Detection registry — every bug not 1-pass-caught + the detection strategy audience: AI coding agents first. BLUF, imperative, [MEASURED]/[HYPOTHESIS]-tagged. Testing-native vocab only (detector/domain/corpus/cell/canary/band). ## What this is The **detection registry**: the running record of every bug the gate does NOT catch in one orchestrated pass, sorted by what (if anything) would catch it. Built so the whole set can be REASONED ABOUT AS A BATCH later, once the single-pass campaign has populated it. **Scope = "every bug not 1-pass-caught," NOT just the 49.** The 49 P0/P1 audit findings (below) are the Phase-1 *campaign denominator* — a measurement worklist — NOT the registry's scope ceiling. A measured 1-pass miss off that denominator still gets an entry (e.g. FR-1, a held-out frontier miss). Denominator ⊂ scope. **North star:** 99% detection from a broad SINGLE orchestrated run (`mapper.py` enumerate→delegate, one roll per target) — see Strategy. **Operating rule while chasing it:** when the single pass misses a finding, register the miss (entry = shape + measured verdict + bucket + why + evidence pointer), then MOVE ON. Do not chase an entry. Triage the whole registry as one batch when the campaign data is in. This is NOT the coverage ledger — `docs/taxonomy/security.md` owns per-class detector status (one source of truth; point, don't re-inline). This registry is the cut by **1-pass detectability**. ## Strategy — two phases (do them in order) ### Phase 1 — NOW: 99% via one orchestrated pass [ACTIVE] A broad SINGLE orchestrated pass that maps the attack surface and delegates each target to the right detector for MAXIMAL detection. **"1-run" = ONE sweep of the surface, NOT one roll per file.** **Production shape DECIDED (user directive; see `docs/specs/2026-06-18-modular-hierarchy-design.md` §"Strategy placement"):** 1 Opus orchestrator triages the attack-surface map to dedicated per-target subagents, each doing **ONE pass (k=1)**. Running k=3 same-model union over a whole codebase per sweep is "expensive and stupid" (user) for steady state. [MEASURED reality of the shipped system:] `mapper.py` enumerate → prioritize → dispatch `gate.py` per target, and `gate.py` *defaults* to **k=3 SAME-MODEL union** (`gate.py:511` `--k` default 3 + `union_rolls`; mapper injects no `--k`) — so the shipped default is the FLAKINESS-AUDIT mechanism, not the production pass; the production dispatch is k=1. **k=3 same-model union = the one-time FLAKINESS AUDIT** (full re-run ×3 after ~99% to find flaky catches and relocate them to the elusive set), NOT "dice" (the user defined dice as CROSS-MODEL; see Phase 2). The lever here is mapping + delegation quality, not cross-model variance. Get to 99% FIRST. ### Phase 2 — DEFERRED, record-only: cheap, reliable "dice rolling" [PARKED — do NOT run mid-Phase-1] A SEPARATE strategy, recorded now, run later: CROSS-model (and cheaper-model) dice-rolling to lift catch-rate on the bucket-(b) entries below. (Same-model k=3 union is NOT this cross-model dice — it is the one-time FLAKINESS AUDIT; but a same-model union-only/flaky catch — 1–2/3 — DOES feed this elusive set, since it misses at k=1 production.) Open questions to answer THEN, not now: - Run the same targets with codex / cursor / agy / other agent CLIs — where does dice-rolling improve the catch chances vs the Phase-1 single pass? - Where is the CHEAPER dice-roll — which model/roll-count buys the most catch per token? - How cheap can the dice-roll get — is an open-weight model from `/vibeflare-api` as good as sonnet on success-rate? [HYPOTHESIS — unmeasured] - This phase is for bucket-(b) ONLY. Bucket-(a) is dice-immune by construction; bucket-(c) is a frontier-detector problem, not a dice problem. ## Inclusion rule — what counts as "1-pass full detection" [the dividing line] A finding is **Phase-1-caught** (NOT a registry entry) only if it is caught by the production pass: - at **k=1** — one roll per target via the Opus-orchestrator → subagent dispatch (the DECIDED production shape; the shipped `gate.py` default `--k 3` is the flakiness-audit mechanism, not this), AND - on the **fixed default model** (sonnet/MED — CROSS-model variance is the deferred elusive round), AND - **right-reason** (canonical defect named with correct mechanism, not a substring hit). Reliability rule: a finding caught at **3/3 rolls = ROBUST** (k=1 production catches it = Phase-1-caught). A finding caught only by **1–2 of 3 rolls = FLAKY** — it would MISS at k=1, so it is NOT a clean Phase-1 catch; it moves to the **elusive/dice set** (§"Variance watch" is the staging area; this is exactly what the flakiness audit relocates). Anything that needs a **different/bigger model**, a **cross-model run**, or **more effort** is dice-rolling → deferred elusive round → **register it**. Pin the model: recall is measured on ONE fixed model, or "99%" drifts. **DECISION RESOLVED (user directive, `docs/specs/2026-06-18-modular-hierarchy-design.md`):** production = strict **k=1** per subagent; same-model k=3 union is the one-time **flakiness audit**, not steady state. Every union-only catch (the 2/3 cases, §"Variance watch") is therefore an **elusive/dice entry**, and the campaign measures the production number at k=1 (the shipped k=3 default gives an upper bound; the k=3−k=1 delta = the flakiness signal). ## The 99% denominator [pin it — without this the north star is unfalsifiable] 99% **of the i1 audit universe** (`~/Projects/zync.is/docs/plans/audit/security/`, P0/P1 first — the 49 P0/P1 findings; P2/P3 after). Caveat carried, state it on every recall claim: this is 99% of **what a 3-iteration fresh-agent LLM marathon found**, i.e. REPRODUCTION of that marathon — NOT 99% of all bugs that exist. Bugs the marathon never found are off-ledger by construction. ## Primary axis — sort every entry by WHAT CATCHES IT This axis mirrors the strategy: it tells you which phase (if any) owns the entry. - **(a) 1-pass caught by a DETERMINISTIC leg only** (oracle / deps / headers). LLM-irrelevant; SOLVED by delegating to the right detector in Phase 1. **Dice-immune** — no model/roll-count changes it. Not a Phase-2 target. - **(b) NOT 1-pass-caught now, but recoverable by dice-rolling** (multi-roll union and/or another model). **This is the Phase-2 target list** the user asked to build. - **(c) caught by NOTHING tried.** The true frontier gap — needs a new detector (deterministic or cross-file), not more dice. Highest-value to reason about in the batch. ## Secondary axis — why it's LLM-hard (per entry) [HYPOTHESIS, sharpening as the campaign runs] A shape is LLM-band-hard when **its bug-ness is NOT derivable from the code under review**. Sort each (b)/(c) entry into one: - **business POLICY present in no code** — anti-abuse intent in no pre-fix code (→ deterministic/oracle territory). - **imported / black-box code the single-file reviewer cannot see** — present check lives in a resolved helper body (→ cross-file / band-2 territory). - **genuine LLM reasoning-depth gap** — info IS in the code, the single pass missed it (→ the ONLY candidate for Phase-2 model/effort dice). Corollary: **escalating model/effort cannot recover information absent from the input.** Confirm absence-vs- under-retrieval before reaching for a bigger model — a (b) entry whose secondary axis is policy/imported is mis-filed; it belongs in (a)/(c). ## Entries ### KH-1 — C02/C09 self-deal where the rule is a business POLICY (S9) — bucket (a), dice-IMMUNE [MEASURED] - **Shape:** complete-mediation self-deal. Canonical `owner|referee` — a buyer who owns the deal's vendor self-deals an affiliate commission; blocked only by an anti-abuse POLICY that exists in NO pre-fix code. - **Bucket (a): 1-pass caught via the deterministic ORACLE leg only.** The orchestrator catches it in one pass by delegating to the oracle (band-3 complete-mediation). This is the cleanest Phase-1/Phase-2 boundary anchor: caught by routing to the right detector, **untouchable by any amount of dice.** - **No LLM recovers it — at any model, effort, or roll-count [MEASURED].** k=3 blind Opus fed the oracle's log called the real C02 a FALSE POSITIVE 3/3 and crowned a phantom. The information is not in the input → dice-rolling provably CANNOT help. So this is NOT a Phase-2/dice item; it is a permanent deterministic-only catch. [MEASURED: `docs/validation/2026-06-18-oracle-log-llm-interpretation-k3.md`] - **Parked precision-only question (do NOT run mid-Phase-1):** does opus at HIGH/xhigh effort escape the 3/3 inversion? This is precision/interpretation of the LLM-over-log layer ONLY — it cannot beat the oracle's recall, and the gate already catches the bug in Phase 1 via the oracle. `gate.py` has `--model`/`--effort` to run it later. Pre-registered criterion (advisor-set): primary = does GAP-2 (`owner|referee`) flip FP→REAL; why-it-flipped = collusion-policy recognized (STRONG) vs structural-symmetry (WEAK) vs still-FP; GAP-1 demoted? (expected NO); protocol = identical input, change ONLY `--effort high`, score on GAP REAL/FP not "name top bug"; append to the k3 doc. - **Boundary caveat:** the oracle is field-name-lexicon-bound (#13) — off-lexicon names = SILENT MISS, promoted to its own frontier entry **FR-1** below (bucket-c, caught by nothing). See taxonomy §"Oracle (S9 self-deal) generalization boundary." ### FR-1 — off-lexicon self-deal: caught by NOTHING currently (S9 frontier) — bucket (c) [MEASURED] - **Shape:** the SAME complete-mediation self-deal as KH-1, but the beneficiary field uses natural off-payments names (`approverId`/`recipientId`/`award`/`kind`) or an ES6-shorthand sink prop. - **Bucket (c): caught by nothing tried.** The oracle — the ONLY leg that catches the founding self-deal — is field-name-lexicon-bound (#13): off-lexicon names = SILENT, a possible FALSE CLEAN with NO guard. And no LLM recovers self-deal-by-policy (KH-1: the anti-abuse rule is in no code). Neither leg fires. [MEASURED: held-out #13; taxonomy §"Oracle (S9 self-deal) generalization boundary."] - **Secondary why-axis:** business POLICY in no code (the self-deal rule) COMPOUNDED by the oracle's lexicon blind spot — even the deterministic leg that owns KH-1 goes dark. - **Frontier detector needed (not dice):** the #13 fix direction — a NON-SILENT "value-sink-shaped call with off-lexicon fields — UNRELIABLE" notice (cannot overclaim) over an ever-growing field regex. Not built. A bucket-(c) new-detector task, NOT a Phase-2 dice item (escalating model/effort cannot recover absent policy). - **Scope note:** off the 49-finding campaign denominator (held-out synthetic-domain test), but ON the registry's scope ("every bug not 1-pass-caught") — recorded as its own entry, not buried as a KH-1 caveat. ### KH-2 — partial-refund reported as full-refund (settlements earnings) — bucket (b), MODEL-DICE candidate [HYPOTHESIS — TP unverified] - **Shape:** settlement/earnings read misclassifies a PARTIALLY-refunded order as FULLY refunded — revenue/refund misreport on the money path. Source: multideal `apps/web/src/server/admin/resources/settlements/queries.ts` (`getEarningsReport` refund/status aggregation). Mechanism UNVERIFIED — recorded as the reviewer stated it; confirm the true-positive before trusting or scoring it. - **Model-dice delta [MEASURED, n=1 per model — NOT the k≥3 blind protocol]:** single-roll `gpt-5.4/low` did NOT surface it; single-roll `gpt-5.6-sol/low` surfaced it `[high]` 1/1. Same file, same `prompt_v2`, same k=1, same aliases. The info IS in the code under review (refund SQL) → the weaker model under-retrieved, the stronger caught. - **Bucket (b): recoverable by MODEL dice** (newer/bigger model) — the Phase-2 target class. Secondary why-axis: **genuine LLM reasoning-depth gap** (NOT policy-absent, NOT imported/black-box — the refund logic is in-file). This is a clean reasoning-depth dice candidate, unlike KH-1/FR-1 (which are dice-immune by construction). - **Benchmark TODO (user-flagged):** build a corpus cell (vuln/safe pair + `canonical.json`) from this so recall on the reasoning-depth gap is MEASURED across models — does gpt-5.6-sol hold at k=1, does the pinned production model miss it, where is the cheapest model that catches it. Until built + TP-confirmed this is a CANDIDATE, not a scored canonical. Verify the mechanism first. - **Evidence [ephemeral — promote to durable]:** gate reports in the multideal repo `apps/web/tmp/sql-audit/sg2-settlements.md` (gpt-5.6-sol catch) vs `sg-settlements.md` (gpt-5.4, finding absent). ## Precision axis — separate from the recall registry above The registry's inclusion rule is RECALL ("not a 1-pass full detection"). Precision FPs are 1-pass DETECTED but emit a false RED on fixed/safe code — they fail a different test and live here, not in the main line, so they don't muddy the recall headline. ### PX-1 — membership-of-a-body-reference (absent-call sub-shape of S3) — PRECISION FP, recall CLEAN [MEASURED] - **Shape:** a request-body FK to a THIRD entity (`assignee_id`) passed to create/update with no `getMembership` tenant-scope check; the required authorization is a MISSING call, not a present-but-wrong predicate. - **Recall: 1-pass CAUGHT (NOT a registry miss).** band-1 catches the vuln right-reason 6/6 single-roll and discriminates on presence (vuln=absent vs safe=present). [MEASURED: `docs/validation/2026-06-18-16-membership-recognition-pilot.md`] - **The defect is PRECISION:** single-file v2 emits a `[high]` membership/tenant FALSE POSITIVE on the FIXED safe.ts because the present scoping lives in imported (black-box) query bodies it cannot read. xfail in `corpus/S3-task-assignee-membership-idor/canonical.json`. - **Detector that would suppress the FP:** band-2 cross-file resolution (#36/#40) — HYPOTHESIS: resolving the helper bodies into scope MIGHT let the model see the present check. Not assumed; MEASURE at n≥3. USER one-way-door lever (precision track) — do NOT auto-start. ## Variance watch — union-caught but not 3/3-stable [MEASURED in recall spikes; re-confirm under campaign protocol] These are caught by only 2 of 3 same-model rolls in the recall spikes. **Under the RESOLVED k=1 production rule these are FLAKY, not clean Phase-1 catches** — they would MISS on a k=1 production roll, so each is an **elusive/dice entry** (the flakiness audit's exact job is to relocate them here). The shipped k=3 default would union-catch them, which is why they read as "caught" in the spikes; that is the upper bound, not the production number. Caveat: these figures come from the recall spikes, not the blind campaign protocol — re-confirm under blind `--config-dir` + refreshed creds before final scoring. | finding | class | 2/3 source | cell / fix SHA | |---|---|---|---| | reset-token reuse | S1 | recall spike | `S1__password.ts` 59ec337 | | attachments cross-tenant IDOR | S8 | recall spike | `S8__attachments.ts` 30ebf2a | | OAuth non-atomic mark-used | S10 | recall spike | `S10__token.ts` 960bb60 | | team RBAC (THIN/variance-prone) | S2 | recall spike | team 43d8e10 | ## How the registry grows The single-pass campaign (feeder tracker: `docs/validation/2026-06-18-recall-campaign-49.md`) is the data source. When the single orchestrated pass MISSES a finding (per the inclusion rule above — blind, right-reason, fixed model), add an entry: shape, measured verdict, **primary bucket (a/b/c)**, **secondary why-axis**, evidence pointer. When the campaign is broad enough, reason about the whole registry at once: (a) = already solved by delegation; (b) = hand to Phase-2 dice-rolling; (c) = design a new detector. Only bucket-(b) reasoning-depth entries are candidates for model/effort dice. ## Pointers (do not re-inline) - Coverage ledger / per-class detector status: `docs/taxonomy/security.md` - Single-pass recall campaign (the 49 P0/P1 feeder): `docs/validation/2026-06-18-recall-campaign-49.md` - Oracle log inversion (KH-1): `docs/validation/2026-06-18-oracle-log-llm-interpretation-k3.md` - Membership precision FP (PX-1): `docs/validation/2026-06-18-16-membership-recognition-pilot.md` - Oracle lexicon boundary (#13): taxonomy §"Oracle (S9 self-deal) generalization boundary"