# coder-impl A/B — gpt-5.6-luna/low (2026-07-11)

**What:** the FIRST coder-seat feature-implementation benchmark, closing the gap between our
existing benchmarks (reviewer-seat security detection + one 3-cell security coding-ab) and the
REAL coder workload. Cells are transcribed from committed tasks in 3 report-backed runplans
(autonomy-p2b, autonomy-p3-daemon, mega-plan-harness) — the seat that actually ran the most real work.

**Provenance (mechanical, no LLM trust):** `scratchpad/provenance_substrate.json` — 28 committed
coder tasks extracted from `runstate/*.log.jsonl` (task.state committed → head sha) + `docs/plans/*.jsonl`
(desc/seat/tier/files). All shas byte-verified to resolve. Journal is truth where a stale `-report.md`
disagrees. 4 pilot cells chosen (2-file, test-carrying, self-contained).

**Cell construction:** codex gpt-5.6-luna/xhigh (avi) transcribed each real task's spec+diff into a
blind, oracle-free prompt (exact exports/signatures pinned; no impl code quoted) →
`scratchpad/codex_ab_cell_proposals.md`. Every provenance field re-verified against the substrate.

**Admission gate (mechanical, `scratchpad/admission_gate.py`):** each cell VALID iff its real test
goes RED at base (test discriminates) and GREEN with the accepted oracle diff (oracle satisfies).
All 4 VALID.

**A/B leg (`scratchpad/ab_runner.py`):** worktree at base → luna/low gets the blind prompt (writes impl
only; no oracle in tree; base carries the spec, as the real coder had) → inject the HEAD oracle test →
grade. Test-hidden; interface pinned in the prompt. k=3, rafa.

## Result — luna/low codes SIMPLE, misses COMPLEX

| cell | real task | seat tier | k=3 rate | failure |
|---|---|---|---|---|
| `w3.t2` | pure canary-first ordering fn (scheduler) | medium | **3/3 PASS** | — |
| `w1.t1` | resolver: 40-row policy table + breaker/backoff/journal | high | **0/3 RED** | `breaker already tripped` throw not implemented (ERR_ASSERTION) |
| `w2.t2` | notifier: ntfy/webhook transport + 3-attempt retry/give-up | medium | **0/3 RED** | consistent assert at notify.test.js:86 (k=1 had 1 lucky PASS → ~1/4) |
| `w1.t2` | decision inbox: sqlite replay, first-answer-wins 409 | high | **0/3 RED** | `answerDecision` 200-status replay not implemented (ERR_ASSERTION) |

**Aggregate: 3/12 = 0.25** · 58.6k avg tok/roll · 156s avg wall. All REDs are genuine assertion
failures on the acceptance contract, NOT harness artifacts (confirmed: consistent failing line, files
written, model rc=0).

## Verdict
**luna/low fills the coder seat only for simple, self-contained tasks (pure functions).** It misses
every multi-contract implementation (large policy tables, transport+retry state machines, sqlite replay
with concurrency invariants) — exactly the tier-medium/high work the real runplans required. This is the
coder analogue of the reviewer-seat finding: luna/low ≈ gpt-5.4 tier, a cheap breadth/simple-task worker,
NOT a drop-in for the complex coder seat. Routing implication: reserve luna/low for the simplest coder
cells; the complex coder seat needs a higher family (mirrors the S2-RBAC → sol/low result on the reviewer side).

**Open:** widen the corpus beyond 4 cells (24 more committed tasks available) for a per-difficulty rate;
run the same 4 cells on terra/sol to place the coder-seat frontier (which family clears the complex cells cheapest).