# STEP-2 zync-audit pilot — generalization beyond payments (2026-06-17)

audience: AI coding agents first. Measured log; tags [MEASURED] vs [INFERRED]. Do not prettify.

## BLUF
The fable zync.is security audit (11 classes S1–S11, ~6 P0 + ~36 P1 confirmed) is now the STEP-2
ground-truth corpus. Two blind pilots run with the **UNMODIFIED v2 "payments" prompt** (k=3 union)
both CAUGHT their non-payments canonical. The v2 prompt is a strong GENERAL adversarial code reviewer,
not a payments-only one. The step-2 gap is therefore NOT "payments blindness" — it is (1) precision /
semantic dedup, and (2) structurally-different classes that an LLM code-review cannot do by construction.

## Pilots [MEASURED] — blind, isolated single-file cells, /tmp/sg_cfg clean config
| cell | class | canonical | gate (v2@sonnet/MED, k=3) result |
|---|---|---|---|
| S4-001 | SQLi | `kb.ts` `sql.raw(\`ARRAY[…'${id}'…]\`)` injection | **CAUGHT 3/3, ranked #1 critical.** + kbPortalQuery IDOR (1/3) + many real bugs |
| S6-001 | SSRF | PATCH `/:id` updates url without `assertSafeWebhookUrl` (POST has it) | **CAUGHT (union)** — but as 3 separate "1/3" paraphrases |
| S6-004 | SSRF | in-file `assertSafeWebhookUrl` passes non-dotted/octal IP + metadata host | **CAUGHT (partial)** — "non-dotted-decimal/octal IPv4" 1/3 |

Reports: `S4-001_gate_v2.md`, `S6_gate_v2.md`. Pre-fix cells via `git show <fix>~1:<path>`
(S4: 9b0c485, S6: bec8bc8). Ground truth: `/home/user/Projects/zync.is/docs/plans/audit/security/`
(orchestrator-only — gate subagents never see it).

## Reframed gaps
- **GAP 1 — precision / semantic dedup [MEASURED, now #1 priority].** S6-001 surfaced as THREE separate
  `1/3` entries ("Missing SSRF guard on PATCH", "PATCH stores arbitrary URLs", "SSRF bypass via PATCH").
  Word-overlap dedup counts PHRASINGS, not bugs. Two harms: (a) noise (~17–19 findings/file : ~1 canonical);
  (b) it HIDES true recall — one bug found by all 3 rolls in different words reads as flaky `1/3`. The
  roll-count signal is corrupted until dedup is semantic. This is task #17, promoted from "precision nicety"
  to "the measurement instrument is distorted."
- **GAP 2 — structurally-different classes need DETERMINISTIC tool legs [INFERRED from class taxonomy].**
  S11 = dependency CVEs (pdfjs CVE-2024-4367, drizzle GHSA) + missing CSP/HSTS response headers in
  wrangler.toml. An LLM reading a `.ts` route file cannot find a CVE in a lockfile it wasn't given, nor run
  `pnpm audit`, nor probe live response headers. Fable's OWN audit used `pnpm audit` + live `curl` probes for
  exactly S11 — i.e. fable did NOT use an LLM for these. The principled mirror: add deterministic legs
  (pnpm-audit leg, header/config checker, optional live-probe) the way the oracle is a deterministic leg for
  complete-mediation. Do NOT expect the LLM leg to cover deps/headers. NOT yet measured (a run would only
  confirm the obvious miss); stated as a design fact, flagged for validation when the tool legs exist.

## What the pilots did NOT yet establish (honest)
- Recall on classes still untested: S1 auth-session, S2 RBAC (incidentally seen in S6 run), S5 XSS/email-HTML,
  S7 webhook-auth, S10 OAuth, S8 file-authz. Expect mostly catchable (code-logic) — UNMEASURED.
- "ALL bugs in 1 round" bar: NOT met/claimed on 2 cells. Need a recall sweep across classes after dedup fix.
- File coverage: the gate only catches a bug if RUN on the file containing it; the audit spans ~40 files →
  "1 round" = run the gate over the audited file set (file-selection is its own step).

## Recommendation (evidence-based, ordered)
1. **Fix semantic dedup (#17) FIRST** — highest leverage: restores the recall signal + cuts noise. Embedding
   or a small LLM merge pass over the unioned findings.
2. **Add deterministic tool legs for S11-class** — pnpm-audit leg + header/config checker. Mirrors fable.
3. **Light prompt generalization (#16)** — strip residual "payments" framing → "security/correctness",
   broaden the class checklist; re-validate it does NOT regress the 6 payments cells. Likely SMALL given the
   v2 prompt already generalizes.
4. **Recall sweep** across the 11 classes (1–2 cells each, after dedup) → measure true per-class recall →
   then push toward the "all bugs in 1 round" bar.