# security-gate Scaffold Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use /ship (recommended) or /executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Stand up the `~/Projects/security-gate/` project skeleton from its advisor-approved design spec — salvaged orchestrator + oracle + LLM prompt, SoT docs, taxonomy, seeded corpus, bench skeleton, CLAUDE.md, and the 5 operating skills — so the project is a working starting point that can run the gate and grow detectors. **Architecture:** Testing-native layout per spec §7 (`detector`/`domain`/`corpus`/`cell`/`canary`/`band`, never module/layer/tier/seam). v1 = security domain only. Three detector bands: generalist-LLM (`prompt_v2.txt`), orchestrated-LLM (`gate.py` cross-file escalation + k-roll union), deterministic (TS oracle, pnpm-audit deps, header/config). The Rust `engine/` is a DEFERRED honest placeholder — no v1 detector needs it. The corpus (git-pinned RED-on-vuln / GREEN-on-safe cells) feeds `bench.py`, which measures per-class recall and fails on regression. **Tech Stack:** python (orchestrator + bench, LLM-latency-bound — never Rust-ified), TypeScript + `bun` (type-aware oracle), `claude -p` (LLM bands), git (corpus ground truth), markdown skills. **Source of truth:** `docs/specs/2026-06-17-security-gate-design.md`. This plan implements its §7/§8/§9/§10. Spike evidence the build rests on: `docs/validation/2026-06-17-recall-spike-S1-S5-S7-S8-S10.md` + `docs/validation/2026-06-17-band2-crossfile-spike.md`. --- ## Salvage source inventory (verified 2026-06-17) `~/.claude/security-gate/` (the prototype this project SUPERSEDES): - `gate.py` (10897 B) — orchestrator. argparse: `target`, `--alias '@/='` (repeatable), `--k 3`, `--depth 2`, `--config-dir`, `--template` (default `/prompt_v2.txt`), `--oracle` (default `/oracle/oracle2.ts`), `--runtime bun`, `--report`. LLM leg: `claude -p --model sonnet --effort medium` k≥3 unioned. **The `--template`/`--oracle` defaults are computed relative to gate.py's own directory (`_here`) — moving gate.py without moving its siblings or fixing the defaults breaks it.** - `oracle/oracle2.ts` (24690 B) + `oracle/tsconfig.json` — type-aware complete-mediation detector. Entry: `bun oracle2.ts ` (`process.argv[2]`). Self-contained. Prints `=== FLAGS ===` / `oracle SILENT` / `=== UNRESOLVED IMPORTS … UNRELIABLE ===`. - `prompt_v2.txt` (11764 B) — generalist-LLM band prompt. - `GATE_DESIGN.md` (9231 B) — the orchestrator's OWN design (2026-06-16). Project-level superseded by the SoT; still the best description of the orchestrator's 3 non-trivial pieces (scope resolution, cross-file escalation, k-roll union). **Carries STALE claims** ("the legs already work") now corrected by the band-2 spike. - `TOOLBANK_DECISION.md` (20638 B), `VALIDATION.md` (19370 B) — salvage to `docs/` per §10. - `zync-pilot/` — `S4-001_gate_v2.md`, `S6_gate_v2.md`, `STEP2_ZYNC_PILOT.md` (pilot evidence). Spike cells already reconstructed (raw git, verified) — corpus seed: - `/tmp/sg_spike/cells/`: `S1__password.ts` `S5__render.ts` `S7__whatsapp.ts` `S8__attachments.ts` `S10__token.ts` (+ `/tmp/sg_spike/S{1,5,7,8,10}_report.md`) - `/tmp/sg_shapeB/cells/`: `refresh.ts` (Shape-B single-file MISS), `session.ts` (the imported builder), `refresh_plus_session.ts` (two-file CATCH) (+ reports) - Fix SHAs (zync, for safe-cell reconstruction): S1=`59ec337` S5=`a15deb0` S7=`5a2867f` S8=`30ebf2a` S10=`960bb60` Shape-A=`16f510f` Shape-B=`ead618d`. All in `~/Projects/zync.is`. --- ## Wave Plan | Wave | Tasks | Files touched | Safe to parallelize? | |------|-------|---------------|----------------------| | 1 | Task 1 (salvage code), Task 2 (SoT docs + taxonomy) | `orchestrator/`, `domains/security/detectors/oracle/`, `domains/security/detectors/security-generalist.prompt.txt` / `docs/` | ✅ code vs docs, no overlap | | 2 | Task 3 (seed corpus), Task 4 (orchestrator DESIGN salvage) | `domains/security/corpus/` / `orchestrator/DESIGN.md` | ✅ no overlap (Task 4 after Task 1 — both name `orchestrator/` but disjoint files; serialized by wave) | | 3 | Task 5 (bench.py) | `bench.py`, `tests/test_bench.py` | single task | | 4 | Task 6 (CLAUDE.md), Task 7 (harvest), Task 8 (implement), Task 9 (investigate), Task 10 (create-tests), Task 11 (coding-standards) | `CLAUDE.md` / `skills/harvest/` / `skills/implement/` / `skills/investigate/` / `skills/create-tests/` / `skills/coding-standards/` | ✅ all disjoint files | Wave gates: W2 blocked by W1 (corpus + DESIGN reference salvaged paths). W3 blocked by W2 (bench reads corpus). W4 blocked by W3 (CLAUDE.md + skills reference orchestrator/bench/corpus/docs — they must exist so references resolve and the banned-term/reference checks pass). **W4 commit-contention gate (shared `.git/index.lock`):** the six W4 tasks write DISJOINT files (parallel-safe for authoring), but `git add`/`git commit` serialize on `.git/index.lock` — file-disjointness does NOT cover that shared resource. If the executor runs W4 subagents concurrently in ONE working dir, the per-task commit steps WILL race intermittently. Required: either (a) **run W4 tasks serially** (default — they are fast doc authoring; the parallelism saved is marginal), or (b) **worktree-isolate per task** (`isolation: worktree`, per [[feedback_multi_agent_parallel]]) and merge back. Do NOT parallelize W4 in-place without one of these. (If /ship already serializes subagent commits, this is a no-op — but assume it does not.) --- ## Cross-cutting rules (apply in EVERY task) - **Testing-native vocabulary ONLY.** Allowed: detector, domain, detector contract, corpus, cell, fixture, canary, anti-canary, discriminator, bench, band, delete-test. **BANNED** in any authored file: `module`, `package`, `layer`, `subpath`, `tier`, `seam`, `port-adapter`, `north-star`, `semver`, `changeset`. Every doc/skill task ends with a banned-term grep gate (shown in-task). - **Reconstruct cells with RAW git only** — `command git --no-pager show :` (RTK silently truncates routed `git show`; verify each file ends on a clean closing token + contains its canonical symbol). Write cells via shell redirection (Bash), NOT the Write tool, so reconstruction stays byte-faithful. - **No stubs that claim functionality.** A placeholder (e.g. `engine/`) must say it is a placeholder, in its own README, and nothing must import or invoke it as if it worked. - **agent-doc-authoring applied to every authored doc/skill** (CLAUDE.md, taxonomy, all SKILL.md): audience-first line; imperative + scoped + BLUF-ordered rules; before/after examples with anti-patterns tagged `DO NOT`/`// reject:`; recognized principle names only (no private codenames); point to the SoT, never re-inline its content. - **Commit after each task** with explicit paths (this project's git has no restrictive allow-list — docs commit freely, verified by `da9ac2d`). Terse caveman commit message, never co-authored. --- ### Task 1: Salvage the code legs into the testing-native layout **Wave:** 1 **Blocks:** Task 4, Task 5, Task 6, Tasks 7–11 **Blocked by:** — **Files:** - Create: `orchestrator/gate.py` (from salvaged `~/.claude/security-gate/gate.py`, with default paths fixed) - Create: `domains/security/detectors/oracle/oracle2.ts`, `domains/security/detectors/oracle/tsconfig.json` (copied as-is) - Create: `domains/security/detectors/security-generalist.prompt.txt` (from `prompt_v2.txt`, renamed to its role) - Create: `engine/README.md` (honest DEFERRED placeholder) - Create: `.gitignore` - [ ] **Step 1: Create the directory skeleton** Run: ```bash cd ~/Projects/security-gate mkdir -p orchestrator domains/security/detectors/oracle domains/security/detectors/deps domains/security/corpus engine skills tests docs/taxonomy docs/plans ``` - [ ] **Step 2: Copy the oracle + prompt as-is (type-aware leg + generalist prompt)** Run: ```bash cd ~/Projects/security-gate cp ~/.claude/security-gate/oracle/oracle2.ts domains/security/detectors/oracle/oracle2.ts cp ~/.claude/security-gate/oracle/tsconfig.json domains/security/detectors/oracle/tsconfig.json cp ~/.claude/security-gate/prompt_v2.txt domains/security/detectors/security-generalist.prompt.txt ``` Expected: 3 files copied. Verify: `bun domains/security/detectors/oracle/oracle2.ts domains/security/detectors/oracle/oracle2.ts` prints an `=== EXTRACTED …` / `oracle SILENT` block (oracle runs; it analyzing itself is a smoke test, output content irrelevant). - [ ] **Step 3: Copy gate.py and FIX its salvaged default paths** The salvaged defaults assume `prompt_v2.txt` and `oracle/oracle2.ts` sit next to `gate.py`. After the move they live under `domains/security/detectors/`. Copy then repoint the two defaults. Run: ```bash cd ~/Projects/security-gate cp ~/.claude/security-gate/gate.py orchestrator/gate.py ``` Then edit `orchestrator/gate.py`. Locate (around lines 188–192): ```python ap.add_argument("--template", default=os.path.join(_here, "prompt_v2.txt")) ap.add_argument("--oracle", default=os.path.join(_here, "oracle", "oracle2.ts")) ``` Replace with (`_here` is `orchestrator/`; `_root` is the project root one level up): ```python _root = os.path.dirname(_here) ap.add_argument("--template", default=os.path.join(_root, "domains", "security", "detectors", "security-generalist.prompt.txt")) ap.add_argument("--oracle", default=os.path.join(_root, "domains", "security", "detectors", "oracle", "oracle2.ts")) ``` - [ ] **Step 4: Verify the orchestrator still runs end-to-end with the new defaults** Run (uses the clean unseeded config + the Shape-B single-file cell already on disk — a known band-1 SILENT result, so a fast oracle-only smoke): ```bash cp ~/.claude/.credentials.json /tmp/sg_cfg/.credentials.json && chmod 600 /tmp/sg_cfg/.credentials.json cd ~/Projects/security-gate python3 orchestrator/gate.py /tmp/sg_shapeB/cells/refresh.ts --k 1 --config-dir /tmp/sg_cfg --report /tmp/sg_smoke.md ``` Expected: prints a `[scope]` line and `[written] /tmp/sg_smoke.md`; report contains `**Oracle (complete-mediation):** SILENT` and an LLM-leg section. No `FileNotFoundError`/`No such file` for the template or oracle (proves Step 3 repoint worked). (k=1 here is a smoke test, NOT a measurement — measurements are k≥3.) - [ ] **Step 5: Write the engine/ DEFERRED placeholder (honest, non-functional)** Create `engine/README.md`: ```markdown # engine/ — DEFERRED (not built in v1) audience: AI coding agents first. **STATUS: PLACEHOLDER. Nothing here. Do not import or invoke it.** This directory is reserved for the absorbed + modularized Rust syntactic scan engine (declarative regex / ast-grep rules), per design spec §5 + §10. It is **intentionally empty in v1**: no v1 security detector needs syntactic scanning — v1 catches come from the LLM bands + the TS oracle + `pnpm audit` + header/config checks (design spec §8). Build this ONLY after the security domain hits its v1 bar and another domain (or a security class) is shown by the delete-test-vs-LLM (spec §4) to NEED a deterministic syntactic detector. Until then, adding Rust here is YAGNI. See: `docs/specs/2026-06-17-security-gate-design.md` §5 (polyglot-by-bottleneck), §10 (salvage map). ``` - [ ] **Step 6: Write .gitignore** Create `.gitignore`: ``` __pycache__/ *.pyc node_modules/ .DS_Store *.report.tmp ``` - [ ] **Step 7: Banned-term gate + commit** Run: ```bash cd ~/Projects/security-gate grep -rnE '\b(module|package|layer|tier|seam|port-adapter)\b' engine/README.md && echo "BANNED TERM — fix before commit" || echo "clean" git add orchestrator/gate.py domains/security/detectors/oracle/oracle2.ts domains/security/detectors/oracle/tsconfig.json domains/security/detectors/security-generalist.prompt.txt engine/README.md .gitignore git commit -m "salvage: orchestrator + oracle + generalist prompt into testing-native layout; engine deferred placeholder" ``` Expected: grep prints `clean`; commit succeeds. --- ### Task 2: Salvage SoT docs + author the security taxonomy (the coverage contract) **Wave:** 1 **Blocks:** Task 6, Tasks 7–11 **Blocked by:** — **Files:** - Create: `docs/TOOLBANK_DECISION.md`, `docs/VALIDATION.md` (copied salvage) - Create: `docs/validation/2026-06-16-assembled-orchestrator-pilots.md` (zync-pilot evidence, consolidated) - Create: `docs/taxonomy/security.md` (authored — the coverage contract) - [ ] **Step 1: Copy the two salvage decision docs** Run: ```bash cd ~/Projects/security-gate cp ~/.claude/security-gate/TOOLBANK_DECISION.md docs/TOOLBANK_DECISION.md cp ~/.claude/security-gate/VALIDATION.md docs/VALIDATION.md ``` - [ ] **Step 2: Consolidate the zync pilot evidence into docs/validation/** Run: ```bash cd ~/Projects/security-gate { echo "# Assembled-orchestrator pilots — S4-001 + S6 + STEP2 (salvaged 2026-06-16)"; echo; echo "audience: AI coding agents first. Salvaged pilot evidence; superseded for project decisions by the SoT design spec."; echo; echo "---"; echo; cat ~/.claude/security-gate/zync-pilot/STEP2_ZYNC_PILOT.md; echo; echo "---"; echo; cat ~/.claude/security-gate/zync-pilot/S4-001_gate_v2.md; echo; echo "---"; echo; cat ~/.claude/security-gate/zync-pilot/S6_gate_v2.md; } > docs/validation/2026-06-16-assembled-orchestrator-pilots.md ``` Verify: `tail -3 docs/validation/2026-06-16-assembled-orchestrator-pilots.md` shows the S6 doc's tail (concatenation completed). - [ ] **Step 3: Author `docs/taxonomy/security.md` — the coverage contract** This is the no-false-coverage-claim ledger (spec §3 COVERAGE knob, §6). It anchors coverage to OWASP ASVS / CWE Top-25 + the fable S1–S11 worked set, and records per-class MEASURED status. Write exactly: ```markdown # Security taxonomy — the coverage contract audience: AI coding agents first. BLUF-ordered, imperative. Tag [MEASURED]/[INFERRED]. This is the coverage LEDGER: a class with 0 cells or unmeasured recall is a BLIND SPOT, never reported covered (spec §6 no-false-coverage-claim). ## How coverage is anchored Coverage is EXHAUSTIVE and taxonomy-driven (spec §3 COVERAGE knob), NOT donor-gated. Anchors: - **OWASP ASVS** (application security verification standard) — the breadth spine. - **CWE Top-25** — the recognized weakness names. - **S1–S11** — the fable worked set below, the cells we have ground truth for. A new class enters the bench (`bench.py`) the moment it has ≥1 git-pinned cell. Recall becomes a RATE only at n≥3 cells/class. ## S1–S11 worked set — MEASURED status (2026-06-17) | id | class | band that catches it | status | cell(s) / fix SHA | |----|-------|----------------------|--------|-------------------| | S1 | auth / session / token lifecycle | band-1 (single-file logic) + band-2 (cross-file 2FA-on-reissue) | [MEASURED] single-file CAUGHT (reset-token reuse, 2/3); cross-file Shape-B MISS single-file (0/3), CAUGHT once resolved (3/3) | `S1__password.ts` 59ec337; `refresh.ts`/`session.ts` ead618d | | S2 | RBAC / authorization | band-1 (expected) | [INFERRED] unmeasured — expect catchable (logic) | — | | S3 | tenant isolation | band-1 flag + band-2 confirm | [INFERRED] unmeasured; Shape-A IDOR flagged by band-1 (see band-2 spike) | task-statuses 16f510f (Shape-A) | | S4 | SQL injection | band-1 | [MEASURED] CAUGHT (pilot) | S4-001 (assembled-orchestrator pilot) | | S5 | output encoding / XSS | band-1 | [MEASURED] CAUGHT 3/3 | `S5__render.ts` a15deb0 | | S6 | SSRF | band-1 | [MEASURED] CAUGHT (pilot) | S6 (assembled-orchestrator pilot) | | S7 | webhook signature verification | band-1 | [MEASURED] CAUGHT 3/3 (fail-open on falsy secret) | `S7__whatsapp.ts` 5a2867f | | S8 | cross-tenant IDOR (DB query path) | band-1 | [MEASURED] CAUGHT 2/3 | `S8__attachments.ts` 30ebf2a | | S9 | payments / complete-mediation | band-3 oracle (C02/C09) + band-1 | [MEASURED] oracle catches C02 (no LLM config did) | multideal C02/C09 | | S10 | OAuth / auth-code replay | band-1 | [MEASURED] CAUGHT 2/3 (non-atomic mark-used) | `S10__token.ts` 960bb60 | | S11 | dependencies / CVE + security headers | band-3 deterministic (`pnpm audit` + header/config check) | [INFERRED] structural non-LLM class by design — NOT an LLM miss | — | ## Coverage summary [MEASURED 2026-06-17] - **7 of 11 classes MEASURED, all CAUGHT** by the generalist band on single-file canonicals (S1,S4,S5,S6,S7,S8,S10). - **Cross-file:** Shape-A IDOR is FLAGGED by band-1 (recall OK); Shape-B (imported insecure default) is a band-1 single-file MISS whose DETECTION is solved once the symbol is resolved into scope (two-file test 3/3) → the open work is band-2 DELIVERY (barrel resolver + proactive trigger), spec §3. - **BLIND SPOTS (no cells / unmeasured):** S2, S3 (expect catchable); per-class recall RATE everywhere (n=1 point estimates, not rates). Do NOT report these covered. ## Pointers (do not re-inline) - Recall evidence: `docs/validation/2026-06-17-recall-spike-S1-S5-S7-S8-S10.md` - Cross-file evidence: `docs/validation/2026-06-17-band2-crossfile-spike.md` - Coverage/extraction model: `docs/specs/2026-06-17-security-gate-design.md` §3, `docs/TOOLBANK_DECISION.md` ``` - [ ] **Step 4: Banned-term gate + commit** Run: ```bash cd ~/Projects/security-gate grep -rnE '\b(layer|tier|seam|port-adapter|north-star)\b' docs/taxonomy/security.md && echo "BANNED TERM — fix" || echo "clean" git add docs/TOOLBANK_DECISION.md docs/VALIDATION.md docs/validation/2026-06-16-assembled-orchestrator-pilots.md docs/taxonomy/security.md git commit -m "docs: salvage toolbank/validation decisions + zync pilots; author security taxonomy coverage contract" ``` Expected: grep `clean`; commit succeeds. (Note: "tier" must not appear — taxonomy uses "class"/"band".) --- ### Task 3: Seed the corpus — RED-on-vuln / GREEN-on-safe cells + canonical.json **Wave:** 2 **Blocks:** Task 5, Task 10 **Blocked by:** Task 1 **Files:** - Create: `domains/security/corpus/README.md` - Create: `domains/security/corpus//vuln.ts`, `.../safe.ts`, `.../canonical.json` for S1,S5,S7,S8,S10 (single-file) + the S1 cross-file Shape-B cell. **Cell layout convention (single-file class):** ``` domains/security/corpus/-/ vuln.ts # git show ~1: (RED — must be flagged) safe.ts # git show : (GREEN — must NOT be flagged) canonical.json # {id, domain, class, band, shape, file, line, fix_sha, canonical_symbol, why} ``` **Cross-file Shape-B cell:** add `vuln_unresolved.ts` (the two-file concatenation — the resolved input band-2 must construct) and set `canonical.json.shape="B-imported-insecure-default"`, `band:2`, `requires_resolution:true`. - [ ] **Step 1: Write the corpus README (the cell contract)** Create `domains/security/corpus/README.md`: ```markdown # security corpus — git-pinned cells (the bench ground truth) audience: AI coding agents first. Each cell directory holds one defect class with ground truth from a git FIX COMMIT (spec §6): - `vuln.ts` = `git show ~1:` — the pre-fix code. A detector MUST flag the canonical (RED-on-vuln). - `safe.ts` = `git show :` — the post-fix code. A detector MUST NOT flag the canonical (GREEN-on-safe). - `canonical.json` = `{id, domain, class, band, shape, file, line, fix_sha, canonical_symbol, why}`. RULES: - Reconstruct cells with RAW git only: `command git --no-pager show :` (RTK truncates routed git show). - Verify every cell ends on a clean closing token AND contains its `canonical_symbol` before trusting any bench result. - Cross-file (Shape-B) cells add `vuln_unresolved.ts` (call site + the imported symbol concatenated) — the resolved input band-2 must build. band-1 is a MEASURED MISS on the single-file form; the resolved form is a 3/3 CATCH (see `docs/validation/2026-06-17-band2-crossfile-spike.md`). ``` - [ ] **Step 2: Seed the 5 single-file cells (vuln from /tmp, safe from raw git)** The vuln cells already exist (verified) in `/tmp/sg_spike/cells/`. Reconstruct each matching safe cell from zync via RAW git. Run: ```bash cd ~/Projects/security-gate ZR=~/Projects/zync.is declare -A SHA=( [S1]=59ec337 [S5]=a15deb0 [S7]=5a2867f [S8]=30ebf2a [S10]=960bb60 ) declare -A PATHS=( [S1]=apps/zync-api/src/routes/auth/password.ts [S5]=packages/notifications/src/email/render.ts [S7]=apps/zync-api/src/routes/webhooks/whatsapp.ts [S8]=packages/db/src/queries/attachments.ts [S10]=apps/zync-api/src/routes/oauth/token.ts ) declare -A SLUG=( [S1]=S1-reset-token-reuse [S5]=S5-email-html-injection [S7]=S7-webhook-sig-failopen [S8]=S8-attachment-cross-tenant-idor [S10]=S10-oauth-code-replay ) for k in S1 S5 S7 S8 S10; do d=domains/security/corpus/${SLUG[$k]}; mkdir -p "$d" cp /tmp/sg_spike/cells/${k}__*.ts "$d/vuln.ts" command git -C "$ZR" --no-pager show "${SHA[$k]}:${PATHS[$k]}" > "$d/safe.ts" echo "$k vuln=$(wc -l < $d/vuln.ts)L safe=$(wc -l < $d/safe.ts)L" done ``` Expected: each prints non-zero line counts for both vuln and safe. Verify no safe.ts is empty or truncated mid-token (`tail -2 domains/security/corpus/*/safe.ts`). The `PATHS` above are VERIFIED (2026-06-17) against each fix SHA's stat — every one is the exact file that commit touched. If any `git show` ever errors with "exists on disk, but not in ", derive the correct path directly from the commit: `command git -C "$ZR" --no-pager show --stat ` lists the files that fix changed — pick the `.ts` route/query file and fix the `PATHS` entry. (Do NOT consult the recall-spike doc for the path — it carries only partial paths, so it cannot resolve a prefix.) - [ ] **Step 3: Seed the cross-file Shape-B cell (S1 2FA-on-refresh)** Run: ```bash cd ~/Projects/security-gate d=domains/security/corpus/S1-xfile-2fa-on-refresh; mkdir -p "$d" cp /tmp/sg_shapeB/cells/refresh.ts "$d/vuln.ts" cp /tmp/sg_shapeB/cells/session.ts "$d/imported-builder.ts" cp /tmp/sg_shapeB/cells/refresh_plus_session.ts "$d/vuln_unresolved.ts" ZR=~/Projects/zync.is command git -C "$ZR" --no-pager show "ead618d:apps/zync-api/src/routes/auth/refresh.ts" > "$d/safe.ts" echo "vuln=$(wc -l < $d/vuln.ts)L safe=$(wc -l < $d/safe.ts)L resolved=$(wc -l < $d/vuln_unresolved.ts)L" ``` Expected: vuln=92L, safe>92L (the fix adds the 2FA derivation), resolved>170L. Verify `safe.ts` contains `enforce2fa:` being PASSED into `buildSessionPayload` (the fix). - [ ] **Step 4: Write the canonical.json files** Create each canonical. Single-file example — `domains/security/corpus/S1-reset-token-reuse/canonical.json`: ```json { "id": "S1-reset-token-reuse", "domain": "security", "class": "S1", "band": 1, "shape": "single-file", "file": "apps/zync-api/src/routes/auth/password.ts", "line": 0, "fix_sha": "59ec337", "canonical_symbol": "reset token single-use", "why": "reset token not invalidated after use — unlimited reuse within TTL" } ``` Write the analogous canonical.json for `S5-email-html-injection` (class S5, fix a15deb0, why "{{}} interpolation unescaped → HTML/XSS in email"), `S7-webhook-sig-failopen` (class S7, fix 5a2867f, why "inbound webhook sig verify fail-OPEN when app_secret falsy"), `S8-attachment-cross-tenant-idor` (class S8, fix 30ebf2a, why "hardDeleteAttachmentRow missing tenant scope → cross-tenant IDOR"), `S10-oauth-code-replay` (class S10, fix 960bb60, why "auth-code single-use non-atomic — mark-used result ignored → code replay"). For each, set `band:1`, `shape:"single-file"`, `canonical_symbol` to the fixed symbol. Cross-file — `domains/security/corpus/S1-xfile-2fa-on-refresh/canonical.json`: ```json { "id": "S1-xfile-2fa-on-refresh", "domain": "security", "class": "S1", "band": 2, "shape": "B-imported-insecure-default", "file": "apps/zync-api/src/routes/auth/refresh.ts", "line": 79, "fix_sha": "ead618d", "canonical_symbol": "buildSessionPayload enforce2fa omission", "requires_resolution": true, "band1_single_file_recall": "0/3 MISS", "band1_resolved_recall": "3/3 CATCH", "why": "refresh.ts calls buildSessionPayload omitting enforce2fa; builder defaults enforce_2fa: args.enforce2fa ?? false → 2FA dropped on every refresh for a tenant that enforces it" } ``` - [ ] **Step 5: Commit** Run: ```bash cd ~/Projects/security-gate git add domains/security/corpus/ git commit -m "corpus: seed S1/S5/S7/S8/S10 single-file cells + S1 cross-file Shape-B cell (RED vuln + GREEN safe + canonical.json)" ``` --- ### Task 4: Salvage the orchestrator's own design into orchestrator/DESIGN.md (with correction) **Wave:** 2 **Blocks:** Task 6 **Blocked by:** Task 1 **Files:** - Create: `orchestrator/DESIGN.md` (from `GATE_DESIGN.md`, prepended correction header) - [ ] **Step 1: Copy GATE_DESIGN.md with a correction header that fixes its stale claim** `GATE_DESIGN.md` says "the legs already work (validated…)". The band-2 spike MEASURED that the cross-file escalation leg is dead beyond payments. Prepend a correction so the stale claim cannot mislead. Run: ```bash cd ~/Projects/security-gate { echo "# Orchestrator design (salvaged 2026-06-16) — orchestrator-LOCAL only" echo echo "audience: AI coding agents first." echo echo "> **SCOPE:** describes the orchestrator's internals (scope resolution, cross-file escalation, k-roll union)." echo "> Project-level decisions live in the SoT: \`docs/specs/2026-06-17-security-gate-design.md\`." echo ">" echo "> **CORRECTION [MEASURED 2026-06-17, band-2 spike]:** the original claim below that \"the legs already work\"" echo "> is TRUE only for the oracle leg + the generalist LLM leg on single-file inputs. The **cross-file escalation" echo "> leg is DEAD beyond payments** — its trigger gates on a payments-flavored symbol-name regex, co-extensive" echo "> with the oracle's domain. Generalizing the trigger (proactive pull) + a barrel resolver is the #1 open" echo "> problem. See \`docs/validation/2026-06-17-band2-crossfile-spike.md\`. Read this doc for HOW the orchestrator" echo "> works; read the SoT + band-2 spike for WHAT is proven vs open." echo echo "---" echo cat ~/.claude/security-gate/GATE_DESIGN.md } > orchestrator/DESIGN.md ``` Verify: `head -20 orchestrator/DESIGN.md` shows the correction header above the original `# Security-gate — assembled-orchestrator design` title. - [ ] **Step 2: Commit** Run: ```bash cd ~/Projects/security-gate git add orchestrator/DESIGN.md git commit -m "salvage: orchestrator DESIGN.md with band-2 correction header (cross-file leg dead beyond payments)" ``` --- ### Task 5: bench.py — recall over the corpus + regression gate + no-false-coverage-claim **Wave:** 3 **Blocks:** Task 6, Task 10 **Blocked by:** Task 3 **Files:** - Create: `bench.py` - Create: `tests/test_bench.py` **Scope (honest):** v1 bench measures what the seeded corpus supports — it loads every `canonical.json`, reports per-class status, and computes RECALL when given a findings source. It does NOT itself call the LLM (that is the orchestrator's job, k≥3, minutes/roll); bench consumes a findings JSON the orchestrator produced, OR runs in `--inventory` mode (no LLM) to print the coverage ledger. This keeps bench fast + deterministic and avoids a stub that pretends to run the gate. - [ ] **Step 1: Write the failing test** Create `tests/test_bench.py`: ```python import json, subprocess, sys, os ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) def test_inventory_lists_every_corpus_cell(): r = subprocess.run([sys.executable, os.path.join(ROOT, "bench.py"), "--inventory"], capture_output=True, text=True) assert r.returncode == 0, r.stderr out = json.loads(r.stdout) ids = {c["id"] for c in out["cells"]} assert "S1-reset-token-reuse" in ids assert "S1-xfile-2fa-on-refresh" in ids # no-false-coverage-claim: a class with no MEASURED recall is a blind spot, never "covered" assert out["blind_spots"], "must report unmeasured classes as blind spots" def test_recall_marks_missing_canonical_as_regression(): # a findings source that flags nothing must yield recall 0 and a non-zero exit (regression gate) empty = os.path.join(ROOT, "tests", "_empty_findings.json") with open(empty, "w") as f: json.dump({"S1-reset-token-reuse": []}, f) r = subprocess.run([sys.executable, os.path.join(ROOT, "bench.py"), "--findings", empty, "--require", "S1-reset-token-reuse"], capture_output=True, text=True) assert r.returncode != 0, "missing required canonical must fail the bench (regression gate)" os.remove(empty) ``` - [ ] **Step 2: Run the test to verify it fails** Run: `cd ~/Projects/security-gate && python3 -m pytest tests/test_bench.py -v` Expected: FAIL — `bench.py` does not exist yet (`FileNotFoundError` / non-zero from subprocess). - [ ] **Step 3: Implement bench.py** Create `bench.py`: ```python #!/usr/bin/env python3 """ bench.py — recall/precision over the security corpus (design spec §6). Modes: --inventory load every corpus canonical.json, print the coverage ledger as JSON. Reports blind_spots (classes with no MEASURED recall) — no-false-coverage-claim. --findings FILE [--require ID...] FILE = {cell_id: [finding_str, ...]} produced by the orchestrator. For each cell, recall = canonical flagged? (substring match of canonical_symbol OR class id in any finding string). --require makes named cells mandatory: a missing canonical exits non-zero (the regression gate). bench does NOT call the LLM. It consumes orchestrator output. The orchestrator runs the bands (k>=3). """ import argparse, glob, json, os, sys ROOT = os.path.dirname(os.path.abspath(__file__)) CORPUS = os.path.join(ROOT, "domains", "security", "corpus") def load_cells(): cells = [] for cj in sorted(glob.glob(os.path.join(CORPUS, "*", "canonical.json"))): with open(cj) as f: c = json.load(f) c["_dir"] = os.path.dirname(cj) cells.append(c) return cells def is_flagged(canonical, findings): sym = canonical.get("canonical_symbol", "").lower() cls = canonical.get("class", "").lower() for fnd in findings: t = fnd.lower() if (sym and sym in t) or (cls and (cls + " ") in t) or (cls and t.startswith(cls)): return True return False def cmd_inventory(cells): blind = sorted({"S2","S3"} | {"S11"}) # logic-unmeasured + structural-by-design; see taxonomy out = { "cells": [{"id": c["id"], "class": c["class"], "band": c["band"], "shape": c.get("shape")} for c in cells], "n_cells": len(cells), "blind_spots": blind, "note": "per-class recall RATE needs n>=3 cells/class; current cells are point estimates", } print(json.dumps(out, indent=2)) return 0 def cmd_findings(cells, findings_path, required): with open(findings_path) as f: findings_by_cell = json.load(f) by_id = {c["id"]: c for c in cells} rows, missing_required = [], [] for cell_id, findings in findings_by_cell.items(): c = by_id.get(cell_id) if not c: rows.append({"id": cell_id, "status": "UNKNOWN_CELL"}); continue flagged = is_flagged(c, findings) rows.append({"id": cell_id, "recall": int(flagged), "class": c["class"]}) if cell_id in required and not flagged: missing_required.append(cell_id) print(json.dumps({"rows": rows, "missing_required": missing_required}, indent=2)) if missing_required: print(f"REGRESSION: required canonical(s) not flagged: {missing_required}", file=sys.stderr) return 1 return 0 def main(): ap = argparse.ArgumentParser() ap.add_argument("--inventory", action="store_true") ap.add_argument("--findings") ap.add_argument("--require", action="append", default=[]) a = ap.parse_args() cells = load_cells() if a.inventory: sys.exit(cmd_inventory(cells)) if a.findings: sys.exit(cmd_findings(cells, a.findings, set(a.require))) ap.error("need --inventory or --findings") if __name__ == "__main__": main() ``` - [ ] **Step 4: Run the test to verify it passes** Run: `cd ~/Projects/security-gate && python3 -m pytest tests/test_bench.py -v` Expected: both tests PASS. Also run `python3 bench.py --inventory` and confirm it prints the cell list incl. `S1-xfile-2fa-on-refresh` and a non-empty `blind_spots`. - [ ] **Step 5: Commit** Run: ```bash cd ~/Projects/security-gate git add bench.py tests/test_bench.py git commit -m "bench: recall/inventory over corpus + regression gate + no-false-coverage-claim blind-spot ledger" ``` --- ### Task 6: CLAUDE.md — project doctrine **Wave:** 4 **Blocks:** — **Blocked by:** Task 1, Task 2, Task 3, Task 5 (references their paths) **Files:** - Create: `CLAUDE.md` - [ ] **Step 1: Author CLAUDE.md (points to the SoT, never re-inlines)** Create `CLAUDE.md`: ```markdown # security-gate — project doctrine audience: AI coding agents first. Imperative, BLUF-ordered. This file POINTS to the source of truth; it never re-inlines it. ## What this is One standing GATE that catches defects across test domains — **security first** — across all of the user's repos. A defect that ships is the exact failure the gate exists to prevent: **coverage is the product.** Source of truth: `docs/specs/2026-06-17-security-gate-design.md`. Read it before changing anything structural. ## Vocabulary — testing-native ONLY (spec §1) Use: detector, domain, detector contract, corpus, cell, fixture, canary, anti-canary, discriminator, bench, band, delete-test. NEVER use (banned — software-architecture vocabulary): module, package, layer, subpath, tier, seam, port-adapter, north-star, semver, changeset. ## The 3-band detector ladder (spec §3) 1. **generalist-LLM** — one broad pass (`domains/security/detectors/security-generalist.prompt.txt`). 2. **orchestrated-LLM** — cross-file scope + k≥3 union + sub-prompts (`orchestrator/gate.py`). Carries hard-but-not-decidable classes. 3. **deterministic** — narrow decidable invariants (`domains/security/detectors/oracle/` TS, `deps/` pnpm-audit, header/config). ROUTING: a bench-found miss is routed to a band by JUDGMENT. Most hard misses → band 2 (better orchestration), NOT a new band-3 detector. Reserve band 3 for the narrow decidable residue. ## Non-negotiable validation discipline (spec §2, §6) - **LLM bands are measured STATISTICALLY: k≥3 rolls, recall as a rate. NEVER a single-run claim.** Deterministic detectors are pass/fail. - **no-false-clean:** a detector on degraded input (unresolved imports) reports UNRELIABLE, never clean. - **no-false-coverage-claim:** a taxonomy class with 0 cells / unmeasured recall is a flagged BLIND SPOT (`docs/taxonomy/security.md`), never "covered". - **Ground truth = git FIX COMMITS.** vuln cell = `git show ~1:`, safe cell = `git show :`. - **Reconstruct cells with RAW git only** — `command git --no-pager show` (RTK silently truncates routed git show). Verify clean end + canonical symbol present. - **Blind catch-test discipline:** detectors/agents under test never see audit docs / fix commits / ground truth; the orchestrator (judging) may. - **Every detector ships only if its conformance suite is green** — flags every canary, ignores every anti-canary, survives every discriminator (spec §4 admission gate). - **delete-test-vs-LLM:** build a band-3 deterministic detector ONLY if deleting it loses a catch the LLM bands cannot recover (spec §4). Building SQLi/IDOR/SSRF detectors is YAGNI — the LLM catches them. ## Measured state (do not re-derive — see docs) - 7/11 S1–S11 classes MEASURED, all CAUGHT single-file by band-1 (`docs/validation/2026-06-17-recall-spike-*`). - Cross-file Shape-B (imported insecure default) is a band-1 single-file MISS, a 3/3 CATCH once resolved → band-2's open work is DELIVERY (barrel resolver + proactive trigger), `docs/validation/2026-06-17-band2-crossfile-spike.md`. This is the #1 open problem. ## Run it - Gate one file: `python3 orchestrator/gate.py --k 3 --config-dir /tmp/sg_cfg --report .md` (clean unseeded config; refresh creds first). - Coverage ledger: `python3 bench.py --inventory`. Recall over orchestrator findings: `python3 bench.py --findings .json --require `. ## v1 scope (spec §8) SECURITY domain only, to the bar: catch ALL fable (zync S1–S11) bugs in 1 round. The Rust `engine/` is DEFERRED (placeholder) — no v1 detector needs syntactic scanning. Other domains follow only after security hits the bar. ## Operating skills (`skills/`) harvest · implement · investigate · create-tests · coding-standards. Invoke the matching skill before its task. ## When authoring any doc/skill here Apply the `agent-doc-authoring` discipline: audience-first, imperative + scoped + BLUF, before/after examples with anti-patterns tagged DO NOT, recognized principle names only, point to the SoT (never copy it). ``` - [ ] **Step 2: Banned-term gate + reference check + commit** Run: ```bash cd ~/Projects/security-gate grep -nE '\b(module|package|layer|subpath|tier|seam|port-adapter|north-star|semver|changeset)\b' CLAUDE.md | grep -v 'NEVER use' && echo "BANNED TERM LEAK — fix" || echo "clean" for p in docs/specs/2026-06-17-security-gate-design.md docs/taxonomy/security.md orchestrator/gate.py bench.py domains/security/detectors/security-generalist.prompt.txt domains/security/detectors/oracle; do test -e "$p" && echo "ok $p" || echo "MISSING $p"; done git add CLAUDE.md git commit -m "doctrine: CLAUDE.md — testing-native, 3-band ladder, validation discipline, points to SoT" ``` Expected: grep `clean` (the only banned-term line is the `NEVER use` declaration, excluded); all referenced paths `ok`. --- ### Task 7: skills/harvest — census a repo for candidate detector classes **Wave:** 4 **Blocks:** — **Blocked by:** Task 1, Task 2, Task 3, Task 5 **Files:** - Create: `skills/harvest/SKILL.md` - [ ] **Step 1: Author the harvest skill** Create `skills/harvest/SKILL.md`: ```markdown # Harvesting candidate detectors from a repo audience: AI coding agents first. Imperative, BLUF-ordered. **Core principle: census recurring deterministic check-classes; rank by convergence across INDEPENDENT donors, not occurrences within one donor.** ## When to use Invoke before adding detectors for a new repo or domain — to find what classes are worth a detector at all. ## Ladder (stop at the first rung that answers the question) 1. **Census, don't assume.** Read actual call sites / patterns. Never infer a class exists from a name — read it. 2. **Group by class, count by DONOR.** A class supported by 1 donor (one file/feature) is a runner, not core. Convergence = independent donors exhibiting the same check, NOT N features within one donor. (kinds-count ≠ donor-count.) 3. **Route each candidate to a band (spec §3).** Decidable syntactic invariant → band-3 candidate. Hard-but-not-decidable → band-2. Already caught by band-1 (check the corpus/bench) → DROP (YAGNI, delete-test-vs-LLM). 4. **Emit candidates**, each: `{class, donors[], proposed_band, decidable?, why}`. A candidate is NOT a detector until it passes `implement`. ## DO NOT - DO NOT count features within one donor as convergence. // reject: "5 call sites in payments → core" (1 donor). - DO NOT propose a band-3 detector for a class the LLM band already catches 3/3 — check `bench.py --inventory` + the corpus first. - DO NOT recommend a detector you have not measured the need for (delete-test-vs-LLM, spec §4). ## Output → hand to `implement` A ranked candidate list. Coverage targets the taxonomy (`docs/taxonomy/security.md`), not the donor set. ## Pointers Coverage/extraction model: `docs/specs/2026-06-17-security-gate-design.md` §3–§4. Donor-count rule: it is the project's anti-overfit guard. ``` - [ ] **Step 2: agent-doc gate + commit** Run: ```bash cd ~/Projects/security-gate grep -nE '\b(module|package|layer|tier|seam|port-adapter)\b' skills/harvest/SKILL.md && echo "BANNED — fix" || echo "clean" grep -q 'audience: AI coding agents first' skills/harvest/SKILL.md && echo "audience-line ok" || echo "MISSING audience line" git add skills/harvest/SKILL.md git commit -m "skill: harvest — census candidate detector classes, rank by donor-count, route by band" ``` Expected: `clean` + `audience-line ok`. --- ### Task 8: skills/implement — author a detector to the contract + admission gate **Wave:** 4 **Blocks:** — **Blocked by:** Task 1, Task 2, Task 3, Task 5 **Files:** - Create: `skills/implement/SKILL.md` - [ ] **Step 1: Author the implement skill** Create `skills/implement/SKILL.md`: ```markdown # Implementing a detector to the contract audience: AI coding agents first. Imperative, BLUF-ordered. **Core principle: a detector ships ONLY when its conformance suite is green — flags every canary, ignores every anti-canary, survives every discriminator. No green suite, no ship.** ## When to use Invoke to turn a harvested candidate into a shipping detector. ## The detector contract (spec §4 — author ALL fields) - `id` · `domain` · `band` · `severity` · `category` - the check: a **declarative rule** (regex / ast-grep, runs on the Rust engine) OR a **procedural `fn(target) -> Finding[]`** (TS for type-aware, e.g. the oracle). - `appliesTo(target)` — which files it runs on. - `canary` — input it MUST flag (RED-on-vuln). - `antiCanary[]` — inputs it MUST NOT flag (GREEN-on-safe). - `discriminators` — rename / cosmetic / polarity / null-mutant variants it MUST survive (anti-overfit — slopgate lacked these; we require them). - `Finding`: `{id, domain, severity, file, line, text, resolution, band, rollsFound?}`. ## Language by BOTTLENECK (spec §5) — pick by the WORK, never preference - CPU-bound + syntactic → declarative rule on the Rust engine. Authoring = a YAML/JSON rule file, NOT writing Rust. - Semantic / type-aware (needs types, cross-file symbol resolution) → TS. // reject: porting the oracle to ast-grep — loses the type info that is its whole value. - Lockfile / CVE / config → the native tool (`pnpm audit`, a config parser). - LLM band / orchestrator glue → python. // reject: Rust-ifying the orchestrator — it is LLM-latency-bound. ## Admission gate (imperative — run before ship) 1. Write canary + every anti-canary + every discriminator as corpus cells/fixtures (use `create-tests`). 2. Run the detector over them. Green = flags all canaries, ignores all anti-canaries, survives all discriminators. 3. **delete-test-vs-LLM:** if deleting this detector loses NO catch the LLM bands can't recover, DO NOT ship it (YAGNI). [MEASURED] oracle/deps/headers pass; SQLi/IDOR/SSRF fail (LLM catches 3/3). 4. Register by CONVENTION (file in the domain's `detectors/`), NOT a registry. ## DO NOT - DO NOT ship a detector without discriminators — a canary-only suite overfits to one phrasing. - DO NOT add a band-3 detector for an LLM-caught class (delete-test). - DO NOT tune-and-test on the same cell (overfit). Validate on a HELD-OUT cell. ## Pointers Contract + admission gate + delete-test: `docs/specs/2026-06-17-security-gate-design.md` §4–§5. Conformance idiom is Semgrep `.test.yaml` / CodeQL `.expected` / ESLint RuleTester — best practice, not novel. ``` - [ ] **Step 2: agent-doc gate + commit** Run: ```bash cd ~/Projects/security-gate grep -nE '\b(layer|tier|seam|port-adapter)\b' skills/implement/SKILL.md && echo "BANNED — fix" || echo "clean" grep -q 'audience: AI coding agents first' skills/implement/SKILL.md && echo "audience-line ok" || echo "MISSING" git add skills/implement/SKILL.md git commit -m "skill: implement — detector contract, language-by-bottleneck, admission gate, delete-test-vs-LLM" ``` Expected: `clean` + `audience-line ok`. --- ### Task 9: skills/investigate — forensic ground-truth reconstruction from a fix commit **Wave:** 4 **Blocks:** — **Blocked by:** Task 1, Task 2, Task 3, Task 5 **Files:** - Create: `skills/investigate/SKILL.md` - [ ] **Step 1: Author the investigate skill** Create `skills/investigate/SKILL.md`: ```markdown # Investigating a defect — reconstruct ground truth from a fix commit audience: AI coding agents first. Imperative, BLUF-ordered. **Core principle: ground truth comes from the git FIX COMMIT, reconstructed with RAW git, verified before trusted. The orchestrator may read ground truth; detectors/agents under test never may.** ## When to use Invoke to build a corpus cell's ground truth, or to forensically confirm what a fix actually changed. ## Ladder 1. **Find the fix commit.** `git log --oneline -- `; read the diff: `command git --no-pager show `. 2. **Reconstruct cells with RAW git** (RTK truncates routed git show — silent, variable): - vuln: `command git --no-pager show ~1: > vuln.ts` - safe: `command git --no-pager show : > safe.ts` 3. **VERIFY before trusting:** each file ends on a clean closing token AND contains its canonical symbol. A cell missing its tail = a false "miss". 4. **Classify the shape** (spec §3, band-2 spike): - **single-file** — the defect is decidable from the one file. band-1. - **Shape-A cross-file IDOR** — user FK passed to an imported fn with no visible guard. band-1 FLAGS it on the smell. - **Shape-B imported insecure default** — call site is visible AND well-formed; the bug is the default INSIDE the imported symbol. band-1 single-file MISS; a 3/3 CATCH once the symbol is resolved into scope. → band-2. 5. **Record `canonical.json`** `{id, domain, class, band, shape, file, line, fix_sha, canonical_symbol, why}`. For Shape-B add `requires_resolution:true` + the resolved cell. ## DO NOT - DO NOT reconstruct a cell via routed `git show` — // reject: silent truncation drops the tail → false clean. - DO NOT let a detector/agent under test see the fix commit, audit doc, or canonical. Blind catch-test discipline. - DO NOT print reconstructed secrets — secret-scrub every cell. ## Pointers Shapes + measured recall: `docs/validation/2026-06-17-band2-crossfile-spike.md`. Ground-truth-from-git: `docs/specs/2026-06-17-security-gate-design.md` §6. ``` - [ ] **Step 2: agent-doc gate + commit** Run: ```bash cd ~/Projects/security-gate grep -nE '\b(layer|tier|seam|port-adapter)\b' skills/investigate/SKILL.md && echo "BANNED — fix" || echo "clean" grep -q 'audience: AI coding agents first' skills/investigate/SKILL.md && echo "audience-line ok" || echo "MISSING" git add skills/investigate/SKILL.md git commit -m "skill: investigate — raw-git ground-truth reconstruction, shape classification, blind discipline" ``` Expected: `clean` + `audience-line ok`. --- ### Task 10: skills/create-tests — build corpus cells + wire the bench **Wave:** 4 **Blocks:** — **Blocked by:** Task 1, Task 2, Task 3, Task 5 **Files:** - Create: `skills/create-tests/SKILL.md` - [ ] **Step 1: Author the create-tests skill (encodes the measured band methodology)** Create `skills/create-tests/SKILL.md`: ```markdown # Creating corpus cells + wiring the bench audience: AI coding agents first. Imperative, BLUF-ordered. **Core principle: a cell is RED-on-vuln + GREEN-on-safe + survives discriminators; a class without n≥3 cells is a point estimate, NEVER a recall rate.** ## When to use Invoke to add a corpus cell, a discriminator, or to extend the bench to a new class. ## Build a cell (ladder) 1. Use `investigate` to reconstruct vuln + safe via RAW git from the fix commit; verify clean end + canonical symbol. 2. Lay it out per `domains/security/corpus/README.md`: `-/{vuln.ts, safe.ts, canonical.json}`. 3. Add **discriminators** (anti-overfit): rename, cosmetic-reformat, polarity-flip, null-mutant variants. The detector MUST still flag the vuln and still pass the safe. 4. Wire into the bench: `python3 bench.py --inventory` must list the new cell. Recall: feed orchestrator findings via `--findings`. ## Encode the MEASURED band methodology (do NOT re-measure what is settled — see docs/validation/) - **band-1 single-file logic: COVERED.** 7/11 classes measured CAUGHT (S1,S4,S5,S6,S7,S8,S10). A new single-file logic cell is expected catchable — measure k≥3, don't assume. - **Shape-A cross-file IDOR: FLAGGED by band-1** as an unconfirmed in-file-ownership gap (recall OK at flag level). - **Shape-B cross-file (imported insecure default): a band-1 single-file MISS (MEASURED 0/3).** DETECTION is solved once the symbol is RESOLVED into scope — the two-file test caught it 3/3. So a Shape-B cell MUST carry both the single-file `vuln.ts` (the MISS input) and `vuln_unresolved.ts` (the resolved CATCH input), and `canonical.json.requires_resolution:true`. The open band-2 work is DELIVERY (barrel resolver + proactive trigger) — NOT a bespoke omission-detection sub-check (proven unnecessary). ## Statistical rules (spec §6) - LLM-band recall: k≥3 rolls, report as a RATE. NEVER a single-run claim. - A recall RATE per class needs n≥3 cells/class — incl. n≥3 Shape-B cells for a real cross-file rate. n=1 is a point estimate; label it so. - Deterministic detectors: pass/fail. - no-false-coverage-claim: a class with 0 cells / unmeasured recall is a BLIND SPOT in `docs/taxonomy/security.md`, never "covered". ## DO NOT - DO NOT ship a cell without a GREEN safe.ts — RED-only overfits. - DO NOT claim a recall RATE from one cell or one roll. - DO NOT tune a detector and validate it on the same cell — validate held-out. ## Pointers Band evidence: `docs/validation/2026-06-17-recall-spike-S1-S5-S7-S8-S10.md`, `docs/validation/2026-06-17-band2-crossfile-spike.md`. Corpus contract: `domains/security/corpus/README.md`. Bench: `bench.py`. ``` - [ ] **Step 2: agent-doc gate + commit** Run: ```bash cd ~/Projects/security-gate grep -nE '\b(layer|tier|seam|port-adapter)\b' skills/create-tests/SKILL.md && echo "BANNED — fix" || echo "clean" grep -q 'audience: AI coding agents first' skills/create-tests/SKILL.md && echo "audience-line ok" || echo "MISSING" git add skills/create-tests/SKILL.md git commit -m "skill: create-tests — RED/GREEN cells + discriminators + k>=3 statistical rules + Shape-B resolution methodology" ``` Expected: `clean` + `audience-line ok`. --- ### Task 11: skills/coding-standards — the project's own code rules **Wave:** 4 **Blocks:** — **Blocked by:** Task 1, Task 2, Task 3, Task 5 **Files:** - Create: `skills/coding-standards/SKILL.md` - [ ] **Step 1: Author the coding-standards skill** Create `skills/coding-standards/SKILL.md`: ```markdown # security-gate coding standards audience: AI coding agents first. Imperative, BLUF-ordered. **Core principle: testing-native vocabulary, language-by-bottleneck, no stubs, no false-clean, no false-coverage-claim.** ## Vocabulary (spec §1) — enforced Use: detector, domain, detector contract, corpus, cell, fixture, canary, anti-canary, discriminator, bench, band, delete-test. NEVER: module, package, layer, subpath, tier, seam, port-adapter, north-star, semver, changeset. Grep every authored file before commit. ## Language by bottleneck (spec §5) - syntactic + CPU-bound → declarative rule on the Rust engine (author a rule file, not Rust). - type-aware / semantic → TS. // reject: porting the oracle to syntax-only ast-grep. - deps / CVE / config → native tool. - LLM glue / orchestrator → python. // reject: Rust-ifying LLM-latency-bound glue. ## Honesty rules (non-negotiable) - **No stubs that claim functionality.** A placeholder says it is one, in its own README; nothing invokes it as if it worked (e.g. `engine/`). - **No-false-clean:** degraded input (unresolved imports) → report UNRELIABLE, never clean. - **No-false-coverage-claim:** unmeasured class → BLIND SPOT, never covered. - **No ignored signals:** every error/warning/hint from a build/test/commit is fixed or explicitly justified benign before moving on. Green = clean, not passed-with-noise. - **Measure before claiming:** never rank/recommend from inferred proxies; pull live results. LLM claims are k≥3, never single-run. ## Detector code rules - Every detector meets the §4 contract incl. discriminators. Discovered by CONVENTION (file placement), no registry. - delete-test-vs-LLM before any band-3 detector (spec §4). - Reconstruct cells with RAW git (`command git --no-pager show`); verify clean end + canonical symbol. ## Commit rules - Commit completed work; terse caveman messages; NEVER co-author. - Stage explicit paths. ## Pointers Full rationale: `docs/specs/2026-06-17-security-gate-design.md`. Validation discipline: `docs/VALIDATION.md`. ``` - [ ] **Step 2: agent-doc gate + commit** Run: ```bash cd ~/Projects/security-gate grep -nE '\b(layer|tier|seam|port-adapter)\b' skills/coding-standards/SKILL.md | grep -v 'NEVER' && echo "BANNED — fix" || echo "clean" grep -q 'audience: AI coding agents first' skills/coding-standards/SKILL.md && echo "audience-line ok" || echo "MISSING" git add skills/coding-standards/SKILL.md git commit -m "skill: coding-standards — testing-native vocab, language-by-bottleneck, honesty rules, detector code rules" ``` Expected: grep `clean` (only the `NEVER:` declaration line is excluded); `audience-line ok`. --- ## Self-Review **1. Spec coverage** (each §7/§8/§9/§10 item → task): - §7 structure: CLAUDE.md (T6), docs/ (T2,T4), engine/ placeholder (T1), orchestrator/ (T1,T4), domains/security/detectors/oracle + prompt (T1), domains/security/corpus (T3), docs/taxonomy (T2), bench.py (T5), skills/ (T7–T11). ✅ all covered. - §8 build order: step 0 spike DONE (committed da9ac2d); steps 1–4 are the BUILD phase — this plan stands up the apparatus (orchestrator+bench+corpus = T1,T3,T5) that makes those steps runnable; dedup/sweep/route are post-scaffold work, correctly NOT in a scaffold plan. ✅ - §9 skills: harvest T7, implement T8, investigate T9, create-tests T10, coding-standards T11. ✅ - §10 salvage map: gate.py→orchestrator (T1), oracle→domains (T1), prompt (T1), TOOLBANK/VALIDATION/zync-pilot→docs (T2), GATE_DESIGN→orchestrator/DESIGN (T4), slopgate engine DEFERRED placeholder (T1), corpus cells (T3). ✅ - Spike findings encoded: taxonomy (T2), create-tests (T10), CLAUDE.md (T6), corpus canonical.json (T3), orchestrator DESIGN correction (T4). ✅ **2. Placeholder scan:** `engine/` is the only placeholder — honestly labeled non-functional, nothing invokes it (T1 Step 5). bench.py is honestly scoped (consumes findings, does not stub an LLM call). No TBD/TODO. ✅ **3. Type consistency:** `canonical.json` field set `{id, domain, class, band, shape, file, line, fix_sha, canonical_symbol, why}` is identical in T3 (cells), T5 (bench `is_flagged` reads `canonical_symbol`/`class`), T9/T10 (skills). `gate.py` flags: `--k --config-dir --report --template --oracle` consistent T1↔T6. bench modes `--inventory`/`--findings`/`--require` consistent T5↔T6. ✅ **4. Wave plan check:** every task has Wave/Blocks/Blocked-by. Wave table consistent. File-overlap per wave: W1 T1(orchestrator/,domains/,engine/) vs T2(docs/) — disjoint ✅. W2 T3(corpus/) vs T4(orchestrator/DESIGN.md) — disjoint; both blocked-by T1 (W1) ✅. W3 T5 single ✅. W4 T6(CLAUDE.md) vs T7–T11(skills//) — all disjoint files ✅. Dependencies: bench(T5) after corpus(T3) ✅; CLAUDE.md+skills(W4) after orchestrator/docs/corpus/bench(W1–W3) so references resolve ✅. Plan is complete and internally consistent. ```