# #16 pilot — does the band-1 "absent-call membership" MISS reproduce? NO (6/6 catch)

audience: AI coding agents first. BLUF, [MEASURED]-tagged.

## Result [MEASURED]

**The documented band-1 single-file MISS does NOT reproduce on clean git-reconstructed cells.**
v2 (`security-generalist.prompt.txt`) catches the membership-of-a-body-reference sub-shape **6/6 single-roll, right-reason**, across the two exact files the prior recorded as the n=2 "1/3 miss".

This is a **go/no-go pilot** for the #16 v3 rewrite (advisor gate: *"verify RED before you build GREEN — if v2 catches it 3/3, stop, no measured target"*). It came back GREEN. **NO-GO on the v3 rewrite as scoped.**

## Method [MEASURED]

- Blind: `gate.py <vuln.ts> --k 1 --config-dir /tmp/sg_cfg` (CLAUDE_CONFIG_DIR isolates the reviewing LLM from `~/.claude` user-memory ground truth). Confirmed seedless: blind-config memory dirs empty; only stale creds (refreshed — see Degraded-run note).
- v2@sonnet/MED, single-file (no deps → byte-identical band-1). 3 independent k=1 rolls per cell = single-roll rate (the lever metric; a k=3 *union* would catch even a 1/3-rate shape).
- CATCH = right-reason read of each roll's findings (canonical defect named with correct mechanism), not bare `assignee_id` substring.
- Cells reconstructed RAW-git from the FIX commits the prior named.

## Cells (new, git-pinned)

| cell | fix | file | FK at call site | rolls catching canonical |
|---|---|---|---|---|
| `S3-task-assignee-membership-idor` | `7fb0c02` | zync-public-api `routes/tasks.ts` | **named** (`assignee_id: parsed.data.assignee_id`) | **3/3** ([high], right-reason) |
| `S3-task-assignee-membership-zyncapi` | `efa7486` | zync-api `routes/tasks.ts` | **opaque spread** (`createTask(db, tid, input)`) | **3/3** ([critical], right-reason, multi-paraphrase) |

Both: canonical = body FK refs (`assignee_id`/`project_id`/`status_id` — references to a THIRD entity) passed to create/update with no `getMembership`/`getProjectById`/`getTaskStatusById` tenant-scope check. Fix adds the absent calls.

### Right-reason evidence (verbatim findings)
- 7fb0c02: `IDOR: assignee_id accepted without membership/tenant check`; one roll also flagged the discriminator `project_id ownership check absent in PATCH, present in POST — asymmetric trust`.
- efa7486: `Assignee ID not validated against tenant membership — cross-tenant data leak`; `assignee_id accepted from client body without tenant-membership check`. **Bonus** (all 3 rolls): `reporter_id` client-forgeable (identity spoofing) + bulk-update IDOR — additional real body-trust bugs the prompt surfaced unprompted.

## Hypotheses falsified

1. **Membership-recognition is a band-1 gap (#16/#22 primary motivation).** FALSIFIED — 6/6 right-reason, single-roll, no union needed.
2. **FK-visibility drives the miss** (opaque `input` spread harder than a named field). FALSIFIED — the opaque-spread variant was caught *more* strongly (critical severity, 3 paraphrases merged). The model infers the FKs from the in-file schema.

## Why the prior recorded 1/3

Consistent with memory UPDATE 17's own caveat: the "1/3" empirical leg was **variance-prone, un-persisted per-roll, same-blob batch disagreement** — the attribution already rested on a STRUCTURAL argument, not the empirics. A plausible contributor: the **degraded-run false-clean** hit this session — an expired blind-config credential made `claude -p` return empty (401), which a naive harness would score as "0 findings = miss". Caught here by the no-false-clean reflex (empty ≠ missed); creds refreshed; re-run gave 6/6.

## Implication for #16 [reconcile pending advisor]

- Primary motivation (close membership-recognition gap): **falsified**.
- Prior evidence already showed v2 generalizes across S1/S2/S3/S5/S6/S7/S8/S10 (catches non-payments canonicals).
- => band-1 v2 has **no measured catch-defect**. A framing-only generalization (the secondary motivation) is low/negative-EV: regression risk on the measured-CAUGHT cells, no payments LLM cell exists to measure that regression, and zero catch-rate upside.
- **Recommendation: no-go on the v3 rewrite as scoped.** The band-1 prompt lever appears spent for *catch-rate*. Candidate redirects: precision/dedup (#17b), oracle payment-field lexicon (#13), or band-2 cross-file PRECISION (the n≥3 measurement #36 opened).

## Open / not-done

- GREEN-on-safe MEASURED — see appended section below. Cell-zyncapi discriminates cleanly; cell-idor discriminates on PRESENCE (vuln=absent vs safe=present) but emits a `[high]` precision FP on safe.ts (xfail, known band-1 gap recorded in canonical.json). Both cells VALID (safe.ts genuinely fixed) — keep both.
- Degraded-run hardening: the gate should treat an empty/401 roll as COVERAGE-INCOMPLETE, never a 0-finding clean. Candidate follow-up.

## Blockers run before report — contamination + discrimination [MEASURED]

Two pre-report gates (advisor-set). Both run; the result REFINES the headline (recall no-go holds; a new precision defect surfaces).

### Blocker 1 — sibling-file contamination: RULED OUT [MEASURED]

The blind reviewer ran with cwd=cell dir + full tools (`--dangerously-skip-permissions`); `safe.ts`/`canonical.json` sit in that dir, so a roll *could* read the answer. Checked all 6 roll transcripts (`/tmp/sg_cfg/projects/<cell-hash>/*.jsonl`):
- tool census = `Read`×6 ONLY (one per successful roll), no Bash/Glob/Grep/LS.
- every `file_path` tool input = `vuln.ts`; ZERO reads of `safe.ts` or `canonical.json`.
- ZERO canonical-only strings (`absent-call`, `canonical_symbol`, `Discriminator pressure`) in any transcript.

=> the 6/6 vuln catches are blind, uncontaminated.

### Blocker 2 — GREEN-on-safe discrimination: SPLIT [MEASURED]

v2@sonnet/MED, k=3, on each `safe.ts` (the FIXED file — `getMembership` present).

| cell | safe.ts membership/tenant finding | discriminates? |
|---|---|---|
| `...-zyncapi` (opaque-spread) | canonical drops to `[low]` truthy-nit 1/3; top findings = reporter_id / TOCTOU / audit (other real bugs) | YES — real catch |
| `...-idor` (named-FK) | STILL `[high]` ×2 on the FIXED file: "PATCH does not verify task belongs to tenant" 3/3 (task-`:id` FP noise) + "membership check not verified as tenant member" 2/3 | PARTIAL — presence-discriminated (vuln=absent, safe=present, transcript-confirmed) but emits `[high]` precision FP on safe.ts → precision gap, NOT a recall constant. xfail (known band-1 gap, see canonical.json). |

**Both cell-idor safe.ts `[high]` findings are FALSE POSITIVES — verified against the code:**
- "PATCH does not verify task belongs to tenant" — `updateTask(db, apiKey.tenantId, id, …)` (L180) PASSES tenantId; scoping is in the imported (black-box) query body. Identical shape to GET `getTask(db, tenantId, id)` (L101), which the model did NOT flag → FP from single-file blindness.
- "membership check not verified as tenant member" — code is `getMembership(db, assignee_id, apiKey.tenantId)` (L175), which checks exactly that. Model misreads a correct check → FP.

### Revised conclusion [MEASURED]

1. **Recall no-go HOLDS.** v2 catches the documented miss BLIND (Blocker 1) → no recall-RED target exists for a v3 rewrite. The "membership-recognition is a band-1 gap" premise stays falsified.
2. **The catch is REAL, not a constant — and the new defect is PRECISION (transcript-confirmed).** Cell-idor DISCRIMINATES on presence: VULN findings frame the check as ABSENT (`assignee_id accepted without membership check` / `… without membership/tenant check` / `passes assignee_id directly to createTask/updateTask without any … membership check`, across rolls); SAFE findings reference the PRESENT check (`membership check performed against assignee_id, not verified as a member of the calling tenant`; other rolls read `getMembership(db, assignee_id, tenantId) validates … is a member` CORRECTLY). The model READ the getMembership call on safe.ts, then a subset of rolls misjudged it. => recall is clean (n=2); the failure is a SEPARATE precision FP — on single-file review v2 emits `[high]` membership/tenant FALSE POSITIVES on correctly tenant-scoped code because the scoping lives in imported (black-box) query bodies it cannot see. (The `[high] task-belongs-to-tenant 3/3` finding is pure FP noise about the task `:id` / `updateTask`↔`getTask` pattern — unrelated to the `assignee_id` canonical.)
3. **HYPOTHESIS (not a finding): band-2 cross-file resolution MIGHT suppress this FP.** Resolving `updateTask`/`getMembership` bodies into scope COULD let the model see the tenant scoping → FP gone. But the data argues against certainty: cell-A safe FP'd at `[high]` while cell-B safe only at `[low]` 1/3, and the model flagged `updateTask` but NOT the identical `getTask` — that asymmetry is disposition/noise, not a clean information-gap band-2 is guaranteed to close. Frame as: MEASURE whether band-2 suppresses it (n≥3) — do not assume.
4. **Redirect = PRECISION, but the lever is a USER one-way-door call.** Candidates: measure band-2 cross-file FP suppression (n≥3, on already-built #36/#40); a band-1 *precision* prompt (#16/#22 track); #17b dedup; #13 oracle lexicon. NOT a v3 recall rewrite — the pilot falsified its premise (authoring would manufacture a fake win vs cells v2 already passes). Do not auto-start.
5. **Reusable win — GREEN-on-safe discrimination separates a catch from a constant.** It is what distinguished cell-B (real catch) from cell-A (FP-driven) here. Prior band-1 `n/n catch` claims that were NEVER discrimination-tested may be similarly inflated. Record + surface to user; do NOT autonomously sweep the corpus to re-test them.