# gpt-5.3-codex-spark — coding-seat (cx-implement) A/B vs gpt-5.4

2026-06-30. Question (owner): can `gpt-5.3-codex-spark` replace `gpt-5.4` as the run-plan
**coding seat** (`cx-implement` / `implementCodex`)? The lever is NOT capability superiority —
it is **separate usage limits**: spark draws on a different quota pool than gpt-5.4/5.5, so a
spark coding seat is *free parallel capacity* that does not burn shared codex quota. Therefore
the bar is **parity**, not superiority: 5.4-tier coding quality is a PASS for this seat.

## Instrument
No code-gen benchmark existed (the security corpus is a *detection* instrument). Built a small
build+behavioral-test harness that reuses existing ground truth — the corpus `vuln.ts`→`safe.ts`
pairs as **spec'd fix tasks** (spec given; this is implement-to-spec, NOT detection):
- Self-contained pure-logic cells only (S13, S5 — the rest pull Hono/db/env infra → ambiguous fix).
- Each task = git repo: `src/` (vuln code) + `TASK.md` (the spec) + a **gitignored** `oracle.test.ts`
  (bun test) the model never sees; applied only at scoring (post-commit).
- **Scorer validated to discriminate before trusting it**: every oracle FAILS on vuln.ts and
  PASSES on safe.ts / a reference fix. A test green on both measures nothing.
- Each arm: fresh git worktree at the same baseline → `cx-implement.sh --model M --effort E`
  (it commits a draft) → harness runs the oracle on the committed draft. Runner:
  `~/.claude/workflows/scratchpad/run_ab.sh <bench> <model> <effort> <label>`.
- Asymmetric efforts are intentional: **spark/high vs gpt-5.4/low** — separate quota makes
  spark-high affordable, so we compare current-cheap-baseline vs spark-at-its-best.

## Results (k=1 per arm, owner-authorized quota)
| task | class | gpt-5.4/low | spark/high | notes |
|---|---|---|---|---|
| S13-promo-nonpositive | numeric validation | 3/3 green, 84s, 1 commit | 3/3 green, **18s**, 1 commit | both emitted the EXACT canonical fix (`valueAmount <= 0n` → throw), byte-identical to safe.ts |
| S5-email-html-injection | string escaping | 4/4 green, 93s, 1 commit | 3/3 green, **12s**, 1 commit | both wrote idiomatic `escapeHtml` over `[&<>"']`; 5.4 ALSO authored its own test file (marginally more conscientious) |

Both arms on both tasks:
- **acceptance pass** — green (discriminating oracle).
- **protocol compliance** — `cx-implement` reported `ok:true`, 1 clean commit. **spark's known
  schema-compliance risk did NOT bite here.** (That risk showed only in the *detection* JSON path,
  docs/validation/2026-06-30-codex-spark.md; the implement loop is unaffected.)
- **latency** — spark ~7× faster (12–18s vs 84–93s). Not the deciding axis, but a real bonus.

## Discriminating task (S5b — the one that can tell models apart)
The two tasks above did NOT discriminate capability: on S13 both models emitted *byte-identical*
fixes. A task two models solve identically is too easy to be evidence of parity — it can't fail a
weaker model, so passing it proves nothing about capability. Built one that CAN:

- **S5b-invoice-key-aware-escaping**: same email-render code, but the spec mirrors safe.ts's real
  *key-aware* design — escape untrusted text vars (`title`/`body`/…) BUT keep the trusted
  pre-rendered `actionButtons` HTML fragment **raw** (escaping it breaks the rendered buttons).
- **Oracle pre-validated to discriminate THREE references** (not just vuln vs safe):
  `vuln` → fails the escape test; **`escape-everything` → fails the *preserve* test** (the axis the
  easy tasks lacked — a weaker model that naively escapes all vars breaks the feature here);
  `key-aware` → 3/3 green. This is the seam where capability shows.

| arm | result | latency | fix quality |
|---|---|---|---|
| gpt-5.4 / low | 3/3 oracle green (+own test, 4 pass) | 94s | `escapeHtml` map + `key==='actionButtons'` guard |
| **gpt-5.3-codex-spark / high** | **3/3 green** | **18s** | `TRUSTED_HTML_VARS` Set + `escapeHtml` — cleanest/most extensible of the three |
| gpt-5.5 / high (seat ceiling) | 3/3 green | 234s | escape-map constant + regex, same guard |

All three threaded key-aware escaping correctly and idiomatically (full `< > & " '` set → no
attribute-breakout weakness). **On a task where escape-all FAILS, spark cleared the preserve test** —
so parity now rests on a discriminating task, not a trivial one. spark matched both gpt-5.4/low AND
the gpt-5.5/high seat ceiling, at 13× the speed of 5.5/high.

## Scale-out k=3 sweep (owner: "scale to a lot")
Re-ran all 3 benches (S13, S5, S5b — incl. the discriminating one) at **k=3 per model**, spark/high
vs gpt-5.4/low. Tests consistency, the real risk a one-shot run hides.

| model | green | cxi_ok | commits | latency med [range] |
|---|---|---|---|---|
| **gpt-5.3-codex-spark / high** | **9/9** (s13 3/3, s5 3/3, s5b 3/3) | all true | 1 each | **13s** [11–21] |
| gpt-5.4 / low | 9/9 | all true | 1 each | 89s [60–125] |

**Zero flakes** for spark across 9 runs — every roll: 3/3 oracle green (incl. S5b preserve test),
clean 1-commit protocol, ~7× faster than 5.4. Parity is now k=3-stable, not a single lucky roll.

## Feature-scale A/B (real recorded tasks: prompt + known-good tested code)
Closed the feature gap with 3 REAL tasks — each an existing, merged, tested feature where we
have the original spec/prompt AND the known-good code's own test suite as oracle. Protocol per
task: (1) known-good tests GREEN today; (2) baseline (parent SHA) + oracle copied in → RED, and
the failure MODE diagnoses fairness; (3) spark builds from the spec (test-free prompt) in a
fresh worktree at baseline → run the known-good oracle → compare. One task per category.

| category | task (repo) | known-good bar | spark/high | verdict |
|---|---|---|---|---|
| **UI** | 5 React form primitives — Checkbox/RadioGroup/Switch/DataList/Pagination (`platform` ui-primitives) | 15/15 testing-library by-role/by-text | **14/15** | **PARITY** — the 1 red is a coupling artifact: oracle asserts `getByText('2 / 5')` carries `aria-current`; spark put `aria-current` on the page-number element (arguably MORE precise). Spec never pinned the `'2 / 5'` text. nav/bounds/next-prev all green; components idiomatic + token-driven. |
| **auth/money** | `useCheckoutConfirm` provider hook (`platform` commerce-checkout-react) | 4/4 incl. money-truth | **4/4** | **PARITY** — read the existing reducer, wired exact action types; `succeeded → settling` (NOT paid), `error → failed`, `requires_redirect → leave confirming`, guard on orderId+clientSecret. Structurally equal to known-good. |
| **backend** | `config.rs` native TOML resolver, 518 lines, Rust (`slopgate`) | 6/6 (parity vs JS-resolver vector) | **COMPILE FAIL** | **FAIL** — spark's own Rust does not build (`E0308` type mismatch; `.is_empty()`/`.join()` on `Option<T>`; `Pattern: Serialize` unsatisfied), reproduced across 2 attempts. **gpt-5.4/low 6/6 PASS, gpt-5.5/high 6/6 PASS on the identical prompt+oracle.** |

(Harness note: first Rust oracle run was contaminated — HEAD test module had post-Task-15 tests
for project-rule-pack *resolution* that the prompt explicitly DEFERS to a typed error; all models
failed those 7 identically. Re-ran with the `2d3c5b0` oracle that matches the prompt; spark still
fails at COMPILE, independent of the oracle.)

**Finding: spark holds parity on TypeScript/React feature work but FAILS the harder Rust systems
task where both gpt-5.4/low and gpt-5.5/high succeed.** Confound: the Rust task is also larger/
harder (518 lines vs a 36-line hook / 5 small components) — language and complexity are entangled,
n=1 per category, one-shot with no build-feedback loop. But the signal is clean: on a real backend
task spark shipped non-compiling code where the current alternative (5.4) did not. The run-plan
review/build gates WOULD catch spark's broken code (it won't ship) — but it burns a retry, eroding
the free-capacity/speed win exactly on the hard tasks.

**Seat implication:** spark is a viable coding seat for a TS/React-dominant workload; it is NOT a
safe drop-in where Rust/systems work appears — gpt-5.4 IS at parity there, so 5.4 remains the safer
polyglot default. A language-aware seat (spark for TS/React, 5.4+ for systems) is the robust shape.

## Verdict (confidence: k=3-stable parity on a DISCRIMINATING task; gap = multi-file feature scale)
On 3 implement-to-spec tasks — including S5b, whose oracle is proven to fail a naive (escape-all)
fix — **spark/high is at parity with gpt-5.4/low AND with the gpt-5.5/high seat ceiling** for the
coding seat: correct, idiomatic key-aware fixes, clean cx-implement protocol, far faster. Combined
with the separate-quota lever, spark is a viable coding seat — free parallel capacity that here
matched the current 5.5 default, not just 5.4 — with the run-plan **review seats (5.5/opus) as the
downstream safety net** on whatever it produces.

Caveats (honest — still NOT benchmark-grade like the detection campaign):
- n=3, k=1. The discrimination bar is now real (S5b can fail a weaker model), but all tasks are
  single-file **bug-fixes**, not multi-file **feature** implementation (the seat's real job). A
  multi-file authored feature task would extend confidence to feature scale.
- The current coding-seat default is **gpt-5.5**, not 5.4 (`cx-implement.sh` MODEL default; run-plan
  `implementCodex` pins gpt-5.5). On S5b spark MATCHED 5.5/high, which softens (but does not erase)
  the "swap drops below 5.5" concern — one discriminating datapoint, not a campaign.

## Wiring (when owner approves)
One line: pass `--model gpt-5.3-codex-spark` in `run-plan.js::implementCodex`'s `cx-implement.sh`
call (the `--model` seam already exists, default gpt-5.5). NOT done pending owner decision on the
5.5→spark drop.