# codex gpt-5.5 LLM-leg pilot — S7-webhook-sig-failopen (k=1, blind)

2026-06-27. Question (owner-set): does codex (gpt-5.5) work AT ALL as the LLM leg, k=1, with a result
**identical-or-better** than the recorded sonnet baseline? Seam: codex-exec (production-representative),
NOT OpenRouter-via-ccr. Scope: single blind pilot cell, k=1.

## Rig
- Leg swap lives in the single seam `orchestrator/llm_runner.py` — `SG_LLM_BACKEND=codex` switches
  `claude -p` → `codex exec --skip-git-repo-check -C <cwd> -m gpt-5.5 -c model_reasoning_effort=high <prompt>`
  (prompt as ARG, stdin closed). Call sites (`gate.py:one_roll`, `semantic_merge`) UNCHANGED; default path byte-identical.
- Run: `SG_LLM_BACKEND=codex python3 orchestrator/gate.py domains/security/corpus/S7-webhook-sig-failopen/vuln.ts
  --neutral-path --k 1 --model gpt-5.5 --effort high --no-workspace-alias --emit … --report …`
- Discipline held: BLIND (codex saw only the neutrally-named target copy, cwd=/tmp; no canonical/fix-commit/dir-name),
  NEUTRAL-PATH (`--neutral-path`), same baseline.prompt as the sonnet leg (no harness change beyond the binary).

## Result — CAUGHT (identical to baseline), k=1
codex k=1 findings (single roll):
- [high] **Webhook signature verification fails open when `app_secret` is missing** ← the canonical defect
- [critical] Tenant ID not bound to the WhatsApp phone number
- [high] No replay/idempotency guard before enqueueing inbound events
- [medium] GET verification accepts empty stored verify tokens
- [medium] Webhook body read/parsed without size or schema limits

Auto-score (`prevent/verdict.py` / `bench.py --require`) = **PUNT → recall 0**. This is a SCORER artifact, NOT a miss:
`canonical_symbol = "webhook signature fail-closed"` is written in the FIX direction; codex (correctly) named the
BUG as "fails open", so the phrase-token matcher misses the literal token `closed`. The cell is hand-judge by design.

## Apples-to-apples vs sonnet baseline
The sonnet baseline on THIS cell is scored the SAME way:
- `docs/validation/2026-06-19-runbench-17cell-recall.md:73` → `punt | CAUGHT | "Signature check fail-open when app_secret is falsy"`
- `docs/validation/2026-06-19-citation-prompt-recall.md:49` → `sig verify fail-open ✓ 3/3`

So baseline = punt→hand-judge CAUGHT (3/3 @ k=3). codex = punt→hand-judge CAUGHT @ k=1, same defect, same mechanism,
+ 4 additional real findings. **Verdict: IDENTICAL on the canonical catch, broader on coverage, achieved at k=1.**
Pilot bar MET — codex-exec works end-to-end as the gpt-5.5 leg.

## Effort sweep on S7 (compute hypothesis) — LOW = HIGH on this cell
Re-ran k=1 blind at `--effort low` (does throwing compute solve it, or is it compute-free?).
- LOW caught the canonical too: `[critical] Missing app secret fails open`, anchor RESOLVED to the exact fail-open
  branch `if (hubSignature && cred.app_secret) {` (vuln.ts:102) — same defect, line-precise, + idempotency/replay,
  malformed-parse-as-success, partial-processing, empty-verify-token findings.
- **Verdict: on S7, gpt-5.5 LOW == HIGH == sonnet baseline (all CAUGHT @ k=1).** Compute is NOT the lever here.
- CAVEAT — S7 is a poor discriminator for the compute hypothesis: it is a baseline-EASY cell (sonnet 3/3). LOW=HIGH
  here only shows the cell sits in gpt-5.5's easy-catch zone at any effort. The hypothesis "throwing compute lifts
  recall" is only TESTABLE on the **flaky 2/3 tail** (S1-reset-token-reuse, S2-role/team, S3-idor trio, S5), where the
  baseline is genuinely unreliable. Effort sweep on a flaky cell is the real experiment — NOT YET run.

## NOT yet established (out of pilot scope)
- n=1 point estimate, ONE cell. A RATE needs k≥3 across the corpus (lab discipline: recall is a rate, never single-run).
- The flaky 2/3 tail (S1-reset-token-reuse, S2-role/team, S3-idor trio, S5) — does gpt-5.5 lift or match it?
- high→xhigh escalation sweep (deferred; user asked high first).
- Quota envelope: gpt-5.5-high burns fast (~45k+ tok/roll); a full 17-cell k≥3 sweep will likely hit the 5-hour
  window and need resume. Owner approval required before that run (memory: paid-run-owner-approval).