# Finding citation + comment-precision — design

audience: AI coding agents first. BLUF, imperative. Points to SoT, never re-inlines. Vocabulary: testing-native only (CLAUDE.md §Vocabulary).

## BLUF — what to build, in order
1. **SHIP the prompt fix (band-1, MEASURED).** Add one scoped rule to `baseline.prompt.txt`: a defect that lives SOLELY inside commented-out / illustrative / non-executing code is NOT a finding; every finding MUST quote the exact executable substring it flags. MEASURED to eliminate the comment-construct false-positive cluster 0/6 (see §Measurement).
2. **BUILD the citation-spine (plumbing, NOT a primitive).** The prompt fix already MAKES the LLM quote the flagged substring — capture it. `parse_findings` extracts the cited `file:line` + verbatim snippet; `build_emit_dict` wires the REAL `line`/`symbol` into the emit (today both are hard-coded `0`/`""`, gate.py:191 — every emitted finding is non-navigable). Natural completion of #1, near-zero marginal cost.
3. **DEFER the deterministic comment-verifier (band-3 tokenizer).** Do NOT build it now. At n=1 the prompt fix recovers the comment catch, so a hand-built band-3 detector FAILS the delete-test (ARCHITECTURE §4) and the precision admission bar (§4.8). Gate its build on a measured flaky tail (§Deferred).

## Problem — 3 MEASURED gate-precision defects (live-fire SoT: `docs/validation/2026-06-19-live-fire-precision-3repos.md`)
1. **Comment-vs-code blindness [#1 FP cluster].** zync 0/8 precision — every flagged construct sat inside `/* */` blocks; the only executable path returned a mock URL. The LLM read comments as live code. **THIS spec addresses it (via #1+#2 above).**
2. **Cross-file UNRELIABLE-but-confident.** trance TOCTOU/IDOR FALSE at 3/3 — the real guard lived in an unresolved predicate import; the oracle correctly self-reported UNRELIABLE, the LLM leg did not. **NOT this spec.** Addressed by the band-2 resolver (#36, SHIPPED) + the `coverage.unresolved ⇒ status='degraded'` signal already in `build_emit_dict`. The citation-spine's `line` is a future input to a finer per-finding downgrade, not built here.
3. **Roll-count ≠ true-positive probability.** Several 3/3 findings were false. **NOT this spec** — it is the measurement-discipline already in CLAUDE.md (k≥3 = consistency, not correctness; right-reason verification).

## Measurement — the tie-breaker that sized this (n=1 A/B, durable: `/tmp/sg_meas/*.md`)
A = shipped baseline. B = baseline + one rule ("flag only executing code; do NOT report defects solely inside commented-out/illustrative code; quote the exact executable substring"). k=3 semantic-merge union on the 2 zync stub files. Oracle SILENT in all runs → whole delta is the LLM prompt.

| File | A | B | Δ | Comment-construct FPs in B |
|---|---|---|---|---|
| cardcom.ts | 9 | 5 | −4 | **0/6** (currency, process.env, float-math, GET-creds, fetch-timeout, non-null all GONE) |
| stripe.ts | 9 | 4 | −5 | **0/2** (idempotency, amount float-math GONE) |

**Verdict: PROMPT-FIX-SUFFICIENT for the comment class** (0/6 commented constructs flagged), PARTIAL on TOTAL FPs — the residual ~50% is a DIFFERENT class (executable-stub over-flagging: live `console.info` PII log, `terminalNumber` interpolated into the returned URL; some arguably low-severity true-positives). The instruction was never designed to touch that class. See §Non-goals.

## Tier 1 — the prompt rule (before/after)
Edit `domains/security/detectors/baseline/baseline.prompt.txt`, OUTPUT CONTRACT block (lines 69–82).

BEFORE (field 2, loose, never enforced):
```
2. **Location** — `file:line` or `function name` (be specific).
```
AFTER (field 2 tightened to a mandatory verifiable anchor):
```
2. **Location** — `file:line` (mandatory, exact).
3. **Code** — quote the executable line AT or NEAREST the defect, verbatim from the file. For a MISSING check/guard/validation/idempotency, quote the unguarded sink/query/call/return where the check is ABSENT. One line or a short span.
```
(renumber Severity/Trigger/Fix to 4/5/6.)

**Anchor is NOT a filter (no-false-clean — load-bearing).** Field 3 anchors a finding for downstream verification; it MUST NEVER read as "no quotable substring → not a finding." An absence-class defect (missing authorization/IDOR, missing idempotency, missing validation, fail-open compare) has no defective substring — it is a defective ABSENCE anchored at the line where the guard SHOULD be. These are the BULK of the live-fire confirmed catches (platform missing `secretKey` validation, trance missing idempotency, the fail-open compare) and the prompt's own stance (line 6: "an unverifiable cross-file assumption is itself a finding") wants them. The MEASURED B wording ("quote the exact executable substring") already surfaced absence findings ("no runtime validation of `config.secretKey`") — keep that imperative-to-quote; add NO suppression conditional on top of it.

ADD to "Rules for the output" (line 77+):
```
- Flag only code that EXECUTES. A defect that exists SOLELY inside commented-out, illustrative, or otherwise non-executing code is NOT a finding — do not report it. (This is scoped to comment/illustration trivia ONLY; declarative config, schema, and IaC that take effect at deploy/runtime ARE live code and remain in scope.)
```

**Scope discipline (false-negative guard):** the rule targets comment/illustration trivia ONLY. It MUST NOT read as "ignore all non-executing code" — config/schema/IaC are live. The parenthetical above is load-bearing; keep it verbatim.

## Tier 2 — the citation-spine (capture what the prompt now produces)
NOT a new primitive (ARCHITECTURE §4 ≥2-consumer gate does not bite — this enriches existing functions). **Snippet is the PRIMARY anchor, not the LLM's line number.** In bundle mode (`build_bundle`, deps inlined) the LLM reviews a concatenated temp file (header + `===== REVIEW TARGET =====` + target + dep sections), so its cited line counts from the BUNDLE top — off by the header/section offset, wrong as the target's line. Derive the real line by LOCATING the quoted snippet in the ACTUAL target file; the LLM's stated line is best-effort FALLBACK only. Touch points:
- `orchestrator/gate.py::parse_findings` — return `(title, sev, body, snippet, line_hint)`. Extract the verbatim `Code` snippet and the `Location` line from `body`. Tolerant: missing → `snippet="", line_hint=0` (degrade, never crash; finding KEPT and flagged un-anchored, NOT dropped — no-false-clean).
- line resolution — `line = target_file.find_line_of(snippet)` (exact verbatim match in the real target; first match). On no match (snippet from a dep section, paraphrased, or whitespace-drifted) fall back to `line_hint` and mark the finding `anchor=unverified`. NEVER emit a bundle-offset line as a target line.
- `union_rolls` / `merge_groups` — carry `snippet`/`line` onto the group (first non-empty wins; do NOT change dedup keying — title-normalization stays; location-based dedup is a watched follow-up).
- `orchestrator/gate.py::build_emit_dict` — replace `line=0, symbol=""` (line 191): set `line` to the resolved line; put the snippet in a DEDICATED finding field (e.g. `code`), NOT `symbol` (PREVENT consumers expect a symbol NAME there — leave `symbol=""` unless a real symbol is known). The PREVENT emit (`prevent/prevent.py`) becomes navigable.

## Tier 3 — DEFERRED: the deterministic comment-verifier
Do NOT build now. Rationale (ARCHITECTURE §4 delete-test + §4.8 precision bar): the prompt fix recovers the comment catch (measured 0/6), so a band-3 tokenizer that re-checks "is this snippet inside comment trivia" loses NO catch the LLM band cannot recover → YAGNI.
**Build trigger (record, do not act):** if an n≥3 recall/precision measurement of the shipped prompt fix shows a FLAKY comment-FP tail the prompt cannot close, THEN build a thin conservative verifier — for each finding's quoted snippet, use the language's REAL tokenizer (TS compiler API, mirroring the oracle subprocess pattern) to check the snippet appears OUTSIDE comment trivia; DROP only when PROVABLY inside comment/non-code trivia (never otherwise → no-false-clean). Regex-strip is REJECTED (fragile: delimiters in strings/regex/JSX). This is the original "verify a quoted substring, not a drifting line number" design, held in reserve.

## Non-goals (honest scope — do NOT oversell)
- **Executable-stub over-flagging** (the residual ~50%: PII-log, identifier-in-URL on live stub lines) — a severity-calibration / stub-context problem, NOT comment blindness. Out of scope.
- **Cross-file UNRELIABLE downgrade (defect #2)** — band-2 resolver + degraded-status own it. Out of scope.
- **Location-based dedup** — watched; title-normalization stays.
- This spec does NOT make the LLM band "agent-trustable from the list." It removes the #1 FP cluster and makes findings navigable. The oracle leg remains the agent-trustable tier (live-fire SoT §Design-split).

## Validation plan (BINDING before ship — no-false-clean, k≥3 statistical)
1. **Recall-no-regression (the gate the measurement did NOT cover).** The A/B tested PRECISION on 2 FP files only. Before the prompt change ships, re-run the 17-cell right-reason recall at k=3 with the patched prompt; assert NO regression vs the baseline-generalization bar (10 cells 3/3, 7 cells 2/3 — `docs/validation/2026-06-18-baseline-generalization-recall.md`) AND the safe.ts discriminator stays 17/17. A drop = the scope-discipline parenthetical OR the field-3 anchor wording failed → fix wording, re-measure. NEVER ship the prompt on the precision result alone.
   - **Absence-class assertion (guards the blocking-fix risk).** The recall check is only a guard against absence-class suppression IF the 17-cell set CONTAINS absence-class cells (missing auth/IDOR, missing idempotency, fail-open) at finding granularity. FIRST verify the corpus has ≥1 such cell; if it does not, the recall check is BLIND to the exact class the field-3 wording endangered — add a real absence-class cell (RAW git fix commit) to the recall set BEFORE shipping the prompt.
2. **Precision-hold.** Re-confirm the comment-FP elimination at n≥3 (the A/B is n=1) on the 2 zync stub files; record the RATE, not the point estimate.
3. **Spine unit tests** (`tests/`, deterministic, pure parse/emit — no LLM): a finding with `Location: f.ts:42` + a `Code` snippet → `build_emit_dict` emits `line=42` and the snippet (not `0`/`""`); a malformed/missing Location → `line=0`, finding KEPT and flagged un-anchored (no crash, no drop); wire into `check.sh` via pytest.
4. **No band-3 built** — Tier 3 stays a recorded trigger; `ledger.py --check` / `bench.py --routing` unchanged (no new detector, no new class).

## Architecture decisions (ARCHITECTURE §4 / §4.8)
- Citation-spine = plumbing, not a primitive (enriches existing functions; data already in `contract.finding`). ≥2-consumer gate N/A.
- Comment-verifier = a band-3 primitive → delete-test + precision-bar gated → DEFERRED (measured recoverable by band-1). Phase-2 architecture-depth review SKIPPED (size gate: <3 new modules — trivial spec).