# security-gate — ARCHITECTURE (canonical SoT)

audience: AI coding agents first. Imperative, BLUF-ordered, caveman-compressed prose. This is the **canonical source of truth for the architecture**: the named pattern, the keystone contract, the day-1 primitive set, and the decision rules that gate new abstraction. It POINTS to deeper docs; it never re-inlines them.

- Vocabulary + bands + validation discipline: `docs/specs/2026-06-17-security-gate-design.md` (read before structural change).
- Goal / roadmap / doctrine: `docs/handoffs/2026-06-19-session-handoff.md`.
- Modular-hierarchy rationale: `docs/specs/2026-06-18-modular-hierarchy-design.md`.
- External prior art + decided borrow list (Semgrep / Vulnhuntr / claude-code-security-review / IRIS): `docs/prior-art.md`.

**Vocabulary scope (reconciles with the spec §1 ban).** This doc cites EXTERNAL architecture patterns (microkernel, pipes-and-filters, detection-as-code, hexagonal/adapter) as **priors only** — training-dense names that switch on the right model behavior. The testing-native ban (`module/package/layer/tier/seam/plugin`) still governs the system's OWN parts: NEVER rename `detector / domain / corpus / cell / canary / band / oracle / roll / recall`. `adapter` is the one licensed design noun. External-pattern name in a "Pattern" cell = prior; our-own-part = testing-native word, always.

---

## 1. What this architecture IS — name it, do not redesign it

**Already ~80% built. Keep it. Do NOT greenfield-redesign.** When asked to "rethink the architecture," extend the registry (§5) — never rewrite the engine.

The pattern, by its training-dense names:

| Concern | Pattern | Where it lives |
|---|---|---|
| Extensibility | **microkernel / plugin** — small stable core, drop-in detectors discovered by manifest | `prevent/registry.py` (discovery), `domains/*/detectors/*/detector.json` |
| Data contract | **SARIF-aligned finding/report** — many producers, ONE shape, one merge | `prevent/contract.py` (§2) |
| Scan pipeline | **pipes-and-filters** — `map → select → dispatch → merge → report` | `orchestrator/{mapper,detect,gate,semantic_merge}.py` |
| Content lifecycle | **detection-as-code** — every detector/cell/rule version-controlled + tested | `domains/`, `tests/` |
| External-dependency boundary | **adapter at the boundary** — ONE boundary only (model backend) | `orchestrator/llm_runner.py` |

Industry analogues to reason from: Semgrep, CodeQL, ESLint, Sigma / detection-engineering.

**Rule: hexagonal/adapter at exactly ONE boundary (the LLM backend). Do NOT hexagonal-everything.** More adapter boundaries = ceremony, not robustness.

## 2. The keystone — the frozen finding/report contract

**Every producer emits ONE finding/report shape. Never fork it.** `prevent/contract.py` is the single source of truth for the shape AND for status derivation. It is a one-way-door contract.

- **no-false-clean is structural, not a convention.** Non-empty `unresolved` ALWAYS degrades status — a partial scan CANNOT serialize as a clean `ok`. Do NOT add a code path that bypasses `emit()`'s status derivation.
- SARIF levels: `error` = blockable/precise, `warning` = surfaced, `note` = info.
- Detectors are language-agnostic because the boundary is **JSON over a subprocess** (`exec` in the manifest). A TS detector and a Python detector are equal citizens. This is why detector language is a per-detector choice, never an engine concern (§7).

## 3. The recurring shape — deterministic rung + LLM rung behind one interface

**Every capability in this system has the same shape: a deterministic rung, an LLM rung, behind one interface, unified by a shared gate.** Design every new interface to this shape so "add the missing rung later" is uniform, never a refactor.

| Interface | deterministic rung (preferred) | LLM rung (backup / majority) | shared gate |
|---|---|---|---|
| detection (bands) | band-3: **wrapped FOSS** (Semgrep/Trivy/Gitleaks/ast-grep) + oracle / deps / headers | band-1/2 LLM rolls | finding contract (§2) |
| **Verdict** (§4) | mechanical match | LLM judge | credit / punt decision |
| **SolutionAdapter** (§4) | codemod | LLM-patch | apply → re-detect clean |
| **PreventionAdapter** (§4) | programmatic detector + ratchet | LLM-rule directive | re-detect after install |

**Preference ladder inside each interface — stop at the first rung that holds:**
1. **deterministic / executable** — prefer ALWAYS when the fix/check is syntactic + local.
2. **LLM rung** — the *majority* case for security (most fixes are semantic, not syntactic). First-class, never bolted-on.
3. **prose / human** — last resort.

**DO NOT build a shared base class / unified framework over these.** Same shape, SEPARATE interfaces, no premature parent. The win is the *pattern*, not a god-object. (over-abstraction is the documented failure mode — `docs/specs/2026-06-18-modular-hierarchy-design.md`.)

**Band-3 deterministic rung = mostly WRAPPED FOSS, not hand-written scanners.** SCA / secrets / syntactic taint SAST are solved — wrap them behind the finding contract (one `kind:exec` `detector.json` + a thin output adapter). A deterministic FOSS catch replaces an LLM dice-roll on the syntactic subset → directly serves "100% in one run". Decided wraps, lane diagram, license preconditions: `docs/prior-art.md` §4 + `.visual/data/toolflow.json`.

**The wrap-decision gate ≠ the build delete-test (§4).** The delete-test gates *building* band-3 (build only if the LLM cannot recover it). Wrapping is ~zero build cost, so the bar drops: **wrap IF (a) passes conformance (flags the canary, quiet on the safe cell) AND (b) removes-a-dice-roll OR adds-recall** — even on classes the LLM sometimes catches. Apply the wrap-gate to wraps, the delete-test to hand-built detectors; do NOT conflate them.
- **deterministic ≠ complete.** A wrap removes the dice ONLY on what it catches; a deterministic miss is still a miss (no-false-coverage-claim §9 still binds — measure recall, never treat "tool ran" as "covered").
- **Free-FOSS is a candidate precondition** — gate runs across private/commercial repos; a tool whose license restricts commercial/private use is NOT a candidate. Kept wraps are all permissive/free (Trivy/osv Apache-2.0, Gitleaks/ast-grep MIT, Semgrep LGPL-2.1 via subprocess). EXCLUDED: CodeQL (OSS-only license) + TruffleHog (AGPL, redundant) — no capability lost (band-2 + oracle cover semantic dataflow).
- **The semantic moat stays ours — do NOT wrap it.** authz / IDOR / self-deal / complete-mediation (S2/S3/S9) is data-model reasoning, not pattern matching; no FOSS does it well. That is why the LLM bands + oracle exist.
- **Second integration point: FOSS output PRIMES the LLM** (run deterministic tools first, feed hits as context) so the LLM spends its budget on the semantic residue instead of re-finding syntactic bugs. Highest-leverage for the security-guard pass.

## 4. The day-1 primitive set — build these, gate everything else

**Decision rule for ANY new primitive: build it only if ≥2 concrete consumers/adapters already exist (in code or in the corpus). 1 consumer → plain data, not a primitive.** This is the deletion / single-adapter test. Apply it before reserving any abstraction.

Build day-1 (each justified by a NAMED imminent task, nothing speculative):

| Primitive | What it is | ≥2-consumer justification | Triggering task |
|---|---|---|---|
| **Verdict** *(new)* | `match(findings, expectation) → credit \| punt`; `expectation` carries polarity (canonical / canary / anti-canary / discriminator / bench-require) | bench scoring · conformance admission · resolution verify-gate · prevention re-detect | registry coverage matrix (now); autoscorer #61 = drop LLM judge behind `punt` |
| **SolutionAdapter** | resolution strategy: `apply(cell) → patch`, shared `verify` (apply → re-detect → canonical no longer credits + behavior preserved) | corpus already has syntactic (S4/S11) AND semantic (S3/S2) cells = 2 fix shapes forced | registry resolution column |
| **PreventionAdapter** | surface-bound prevent strategy: programmatic detector+ratchet OR LLM-rule directive | programmatic (`prevent/`) AND LLM-rule both needed | registry prevention column |
| **Registry index** | the **deliverable**, not a primitive type (§5) | — | FIRST TASK |

**Verdict is the one genuinely new primitive. Its impl axis is real, not a pure function.** Scoring today is mechanical-for-reproducibility WITH an explicit `_PUNT_WHY` path to a human judge (`bench.py`, `is_flagged`). The autoscorer (#61) replaces punt-to-human with punt-to-LLM-judge. **Build the `punt → judge` boundary day-1** so the autoscorer is a drop-in, not a rewrite. `is_flagged` is currently scattered across `bench.py` + `orchestrator/detect.py:219` and has been re-patched repeatedly — unify it into Verdict to stop the drift.

**Admission bar: executable resolution is gated HARDER than detection.** A wrong autofix writes insecure code AND false assurance — strictly worse than no fix. For semantic fixes (which tenant predicate? which ownership scope?) the executable rung emits a *located suggestion*, it does NOT silently rewrite. Auto-apply only the syntactic + local rung.

**NOT day-1 — watch, do not build:** a `Target` / scan-unit primitive (file vs surface vs resolved-import-graph). Consumers not all present; building now = over-abstraction. Build when a 2nd real consumer appears.

## 5. The registry — a derived coverage oracle, NOT a bug database

**The registry's job is to make blind spots structurally impossible to hide — it is the test that proves the "100%" claim is real.** "Coverage is the product." Treat it as a consistency oracle, not storage.

Decision rules:
- **DERIVE, never hand-maintain.** Generate the registry from the content files. A parallel hand-edited registry drifts; a derived one cannot.
- **SoT stays distributed.** `canonical.json` (cells), `detector.json` (capabilities), `confirmed.json` (ratchet) remain the source of truth. **DO NOT introduce a second hand-edited SoT.**
- **Normalize, then project.** Storage is normalized (a detector catches a class = 1:many — never duplicate it onto every bug row). "One row per bug, full lifecycle" is the **generated presentation** (`Defect → Class → detectors → ratchet → resolution`), NOT the storage format.
- **FAIL CI on any gap:** a cell with no green detector, a class with zero cells (flagged BLIND SPOT), a confirmed defect with no ratchet. This is no-false-coverage-claim made structural.

Where new data lives (instance vs class — do not mix axes):
- **provenance** (audit / repo / fix_sha / when / how / who) = **instance-level → field on `canonical.json`.** Plain data, no primitive.
- **resolution** ("how to fix this class") = **class-level metadata.** Plain data + a SolutionAdapter rung tag.
- **dedup identity key** = `class + repo + symbol + fix_sha` so the same defect from two audits (zync + fable) is ONE entry, not two.

## 6. Source is NOT a runtime primitive

**Provenance is a data field, not a load-bearing primitive — because corpus cells are MATERIALIZED.** Each cell dir holds `vuln.ts` + `safe.ts` + `canonical.json`; loaders read files, never `git show` at scan time. Cross-repo (fable audit) bites ONLY at authoring time — the agent reconstructs `vuln/safe` from `command git --no-pager show <repo>` once, into files.

- Repo-root resolution lives in an **authoring helper**, NEVER in the engine.
- When fable lands, **no runtime refactor** — add provenance fields + the dedup key (§5).

## 7. Detector language + the Rust hot-path question

**Detector language is a per-detector choice behind the subprocess/JSON boundary (§2) — NOT an engine decision. Decide nothing globally.**

Rust hot-path (PREVENT on every save): **defer. The latency lever is subprocess cold-start, not language.** Each detector spawns a process (python; node for the TS oracle) per invocation; on the hot path that spawn dominates, and Rust does not fix it while shelling out to node.

**Rule: measure the PREVENT hook cold-start BEFORE porting anything.** If too slow, the architectural answer is a **long-lived dispatcher / daemon**, NOT a rewrite. (doctrine: never recommend without measuring.)

## 8. Schema hygiene — cheap now, expensive after N detectors

Not primitives — normalize these day-1, before beyond-payments multiplies detectors:

- **Unify the detector manifest.** Today bifurcated: `kind:"llm"` + `prompt` vs exec-based (`exec`, no `kind`) with a forked loader. **Add an explicit `kind` discriminant (`llm | exec`) + ONE load path.**
- **Single-source the surface/kind taxonomy.** The surface vocabulary (`http-defn-call`, `webhook-receiver`, …) is duplicated between `mapper.py` `kind_of` and every `detector.json` `applies_to.kinds`. **Freeze ONE taxonomy** so detectors cannot drift against the mapper.

## 9. Doctrine pointers (do not re-inline — see the spec)

LLM bands measured **statistically (k≥3, recall as a rate, NEVER a single-run claim)**; deterministic detectors pass/fail. Ground truth = git FIX COMMITS, reconstructed with RAW git. Blind/isolated testing, right-reason bar, the ratchet, advisor-before-locking-contracts. Full statements: `docs/specs/2026-06-17-security-gate-design.md` §2/§6 + the handoff doctrine section.

## 10. Surface routing — the cost discipline (ADD depth by category; NEVER subtract a class)

**Routing decides WHICH detector fires on WHICH file. It controls cost. It must NEVER manufacture a false-clean.** Routing is the cost lever for "don't look for payment bugs in the emoji picker, or SQLi where there is no SQL." It achieves this by **ADDING depth where a category signal fires + keeping the LLM off whole non-target files — NEVER by programmatically SUBTRACTING a class from a file that is scanned.** Implementation SoT: `docs/specs/2026-06-17-attack-surface-mapping-design.md` (the mapper + the `bench --routing` recall mode). This section is the canonical INVARIANT set; the spec carries the how.

**[DECISION 2026-06-19 — the SUBTRACT was measured redundant; do NOT build it.]** An earlier design proposed a `sink_signal` SUBTRACT (skip a "Narrowable" class — S4/S5/S6/S7 — when its sink-grep is empty). `bench --routing` step 0 proved it has **no coherent target**: the SUBTRACT differs from the existing `applies_to.signal` ADD ONLY for a detector that runs *unconditionally*, and the only such detector is `baseline` — the never-narrowable floor (inv 1). All S4/S5/S6/S7 cells are baseline-floor-only (no specialist); `baseline` is ONE LLM pass over all 11 classes, so there is no "S4 part" to skip. Any real specialist would be **opt-in via `applies_to.signal`, which already skips on sink-absence** — i.e. the ADD. Further, a sink-grep SUBTRACT would be an **imprecise heuristic** (the oracle's documented residual false-positive is the cautionary precedent, inv 8) → noise the LLM must re-scan + a mis-dismissal risk. **The SUBTRACT collapses into the ADD. Narrowing = ADD specialists by category + file-level enumeration. There is no class-exclusion mechanism.**

**The invariants — apply in order; each gates the next:**

1. **The irreducible floor: EVERY scanned file gets the deterministic pass UNCONDITIONALLY** — *target-state, partially built.* The cheap breadth (wrapped FOSS — Semgrep/ast-grep/Gitleaks/Trivy + oracle, §3) is the design floor; **routing NEVER skips it on a file that is scanned.** Current gaps (no-false-coverage-claim binds — state them, do not pretend the floor is complete): FOSS wraps are UNBUILT; `headers` is dead (`scope_globs=[]`, deferred — `bench --routing` xfail); `oracle` is `.ts`-only → a `.py`/other file gets nothing today. **The present-day backstop for never-narrowable classes is baseline-on-target (LLM), which covers only ENUMERATED surfaces — NOT non-surface semantic code (the deferred cross-file gap, dependency note below).**
2. **Routing ADDS depth on top of the floor; routing NEVER SUBTRACTS a class.** The ONLY two cost levers, both ADD/skip-whole-file — never skip-a-class-on-a-scanned-file:
   - **file-level**: surface enumeration (`bench --mapper`) + `.gateignore` keep the expensive LLM off whole non-target files (the README, the emoji-picker). A file the LLM never opens is the cost win — NOT a class silently dropped from a file it does open.
   - **specialist-level**: `applies_to.signal` ADDS a specialist pass where its category signal fires (finance only on money code, etc.). Absence of the signal = the specialist is simply not ADDED; the floor + baseline still cover the file.
3. **LLM depth is ROUTED, never universal.** The expensive LLM bands run only on attack-surface targets (entry points, §spec mapper) + their resolved cross-file neighborhood (reachability). **The cost lever is routing the LLM to surfaces + signal, NOT subtracting a class.**
4. **The decidability partition is RETAINED as the WHY-no-class-is-excluded record, not a skip rule.** No class is ever programmatically excluded; the partition documents which classes *could even in principle* be sink-decidable and why that still does not license a SUBTRACT:
   - **Sink-decidable in principle** (absence is a deterministic fact): **S4** SQLi, **S5** XSS, **S6** SSRF, **S7** webhook-sig, **S11** deps. A specialist for these is signal-GATED (ADD) — it is added on sink-presence, never subtracted on sink-absence; the two are equivalent only because the floor is unconditional.
   - **Semantic / implicit** (money, auth-state, tenancy, ownership cannot be grepped absent): **S1** auth, **S2** RBAC, **S3** tenant, **S8** IDOR, **S9** self-deal, **S10** OAuth-replay. These ride the always-on floor + gain LLM depth on positive signal only.
   - **Uncertain → treat as semantic. Default is SCAN.**
5. **Router = programmatic-first. The LLM is a DETECTOR, NEVER the gatekeeper.** A model may classify/prioritize over the grounded target list (fail-open, measured), but a model NEVER decides a class is absent. An unmeasured LLM router manufactures *silent* blind spots.
6. **Route by category (surface-predicate `kind × sink × data`), NOT per-file.** Category routing generalizes (a new webhook file inherits webhook routing) and is conformance-testable at the kind level; per-file routing is fragile, costly, unmeasurable.
7. **The shipping gate: specialist-routing recall measured on the corpus, BASELINE-EXCLUDED.** For all corpus cells, the mapper MUST route each to its **INTENDED** detector — the class specialist (or that class's deterministic detector), **NOT merely "some always-on detector caught it."** `baseline.always=true` catches the corpus cells, so a baseline-inclusive assertion is trivially green and measures baseline, not routing (the ledger false-green failure mode). Discriminator the gate MUST satisfy: **disable the specialist → its class goes RED** (proven: `tests/test_bench.py::test_routing_excludes_baseline_DISCRIMINATOR`). A routing miss (right bug, specialist never ran) is a NEW failure class — measured by `bench --routing`, DISTINCT from enumeration-recall (§spec) and catch-recall (§9). A class with NO specialist is a routing **BLIND SPOT** (reported, excluded from rate); a known-dead detector is an explicit cell **`routing_xfail`** (reported, non-fatal) — never a silent pass. This gate guards `applies_to` drift; it does not gate a (nonexistent) SUBTRACT.
8. **Band-3 admission bar: deterministic ONLY if PRECISE + decidable — else it is LLM work.** Ship a band-3 deterministic detector ONLY when a finding is **agent-trustable: an agent can act on it directly without re-scanning the code** (deps-grade — a curated-truth source + a decidable invariant; `deps` = advisory-DB CVE, fix = bump version). An **imprecise heuristic** (sink-greps; the `oracle` ships a *documented residual false-positive* — the `referee|referrer` pair flagged in BOTH vuln AND fixed code) hands the LLM a noisy list it must re-reason → net-negative vs the LLM scanning + a mis-dismissal risk (a real hit dismissed alongside the noise). Imprecise → it belongs to the LLM band. This is `delete-test-vs-LLM` (spec §4) with the precision clause made explicit.

**Dependency to surface, never hide:** routing the LLM to surfaces leans on cross-file resolution (the #1 open problem — barrel resolver + proactive trigger, §spec band2). Until it is delivered, recall on non-surface helpers reached only through a target is INCOMPLETE — report it `incomplete`, NEVER clean (no-false-clean §2 binds).