# gpt-5.6-luna/low — full benchmark suite (2026-07-11)

**Model under test:** `gpt-5.6-luna` reasoning-effort `low`, run as the LLM leg via `SG_LLM_BACKEND=codex`
(bare `codex exec`, `-m gpt-5.6-luna -c model_reasoning_effort=low`). Blind protocol (`--neutral-path`,
`--no-workspace-alias`). Right-reason hand-judged vs canonical (auto-score NOT used — it false-negatives on
fn-name/phrase symbols, per prior campaigns). Baselines are the RECORDED gpt-5.5 / gpt-5.4 / spark docs — no
Anthropic control re-runs.

**Account:** entire run on `rafa` — 60 detection rolls (2.44M tok) + 9 coding builds (0.58M tok) ≈ **3.0M tokens,
~72 min codex wall, and rafa never hit its usage cap.** Telemetry seam added to `orchestrator/llm_runner.py`
(env-gated `SG_TELEMETRY`, default-off, measurement-only — captures per-call wall-ms + codex token count).

## Cost framing
codex-via-subscription is **usage-cap** billed, NOT per-token dollars. `gpt-5.6-luna` is post-cutoff with **no
verified public $/token** — do not fabricate one. **Cost unit = tokens** (the objective measure below);
"how expensive" in dollars is unanswerable until a price is published. Rafa absorbing 3.0M tokens without
capping is itself the practical cost signal: cheap in usage terms at this account tier.

## Per-benchmark verdict (the 4 dimensions asked: good-for-it? · cost/tokens · speed)

| Benchmark | Good for it? | Result vs baseline | Cost (tokens) | Speed |
|---|---|---|---|---|
| **T1 full-corpus detection** (20 LLM+C02 cells, N=3 = 60 rolls) | **YES — strong generalist** | 16/20 catch. Nails IDOR (S3×5, S8), SQLi (S4), SSRF (S6), XSS (S5), webhook fail-open (S7), oauth replay (S10), double-spend (S12), reset-token (S1), C02 (S9). Blind spot: RBAC permission-ceiling. | 2,437,417 tot · **40.6k avg/roll** | **61s avg/roll** |
| **T2 flaky-tail** (7 cells, N=3 = 21 rolls) | **WEAKER — below the OpenAI baselines** | ≈5/7. Solid: S1-reset, S3-session, S3-assignee-idor, S3-assignee-zyncapi, S5. MISS: S2-role (RBAC ceiling), S2-team-owner. Baseline **5.5 = 7/7, 5.4 = 6/7** → luna/low under both. | 858,018 tot · 40.9k avg/roll | 61s avg/roll |
| **T3 C02 hard discriminator** (S9 self-deal, N=3) | **YES — parity** | **3/3 CATCH** "buyer-owned vendor self-deal not blocked". Catches the cell EVERY Anthropic config missed. Matches 5.5 & 5.4 (both 3/3 low). | 98,846 tot · **32.9k avg/roll** | **51s avg/roll** (fastest) |
| **T4 coding-seat A/B** (s13/s5/s5b build+behavioral, N=3 = 9 builds) | **YES — parity, fast** | **9/9 GREEN** vs gitignored oracle, clean 1-commit each. = spark/5.4/5.5 parity bar. Speed ~5.4-tier: **~60s avg vs 5.5's recorded 234s**. | 578,071 tot · 64.2k avg/build | **60s avg build** (40–87s) |

## Detection cell detail
Full right-reason table: `scratchpad/luna_detection_verdicts.md`. Raw per-roll telemetry (tokens+wall+titles):
`scratchpad/sweep_gpt-56-luna_low_instrumented.json`. Coding-seat: `scratchpad/luna_codeseat_results.jsonl`
+ `scratchpad/luna_codeseat_tokens.txt`.

**CATCH 3/3 (14):** S1-reset, S10, S12, S3-calendar/report/session-IDOR, S3-assignee-idor, S3-assignee-zyncapi,
S4-SQLi, S5-XSS, S6-SSRF, S7-webhook, S8-attachment, **S9-C02**.
**CATCH 2/3 flaky (2):** S13-promo-fixed, S2-invite-priv-esc.
**MISS (4):** S1-barrel-2fa + S1-xfile-2fa (STRUCTURAL — cross-file enforce2fa omission run single-file; every
model misses this single-file), **S2-role RBAC permission-ceiling** (GENUINE — same blind spot as gpt-5.4),
S2-team-owner-invariant (5.5 caught 7/7; luna names membership/atomicity, not the owner-gate bypass).

## Bottom line
**gpt-5.6-luna/low sits at roughly the gpt-5.4 capability tier, fast and cheap-in-usage.** It catches the C02
cross-model discriminator (3/3), codes at full parity (9/9), and is a strong breadth detector — but it **shares
gpt-5.4's S2 RBAC privilege-escalation blind spot** (misses S2-role and S2-team-owner that gpt-5.5 catches). So:
- **Good for:** high-volume breadth detection, C02/self-deal, the coding seat (much faster than 5.5).
- **NOT a gpt-5.5 replacement** for the privilege-escalation classes — the right escalation for an S2 miss stays
  a MODEL bump to 5.5, not effort (consistent with the C02/flaky-tail lesson: lever is model, not compute).
- A `low→med/high` effort sweep on the 2 genuine S2 misses is the open follow-up (untested — effort has NOT
  recovered a capability miss in prior campaigns, so expectation is low).

## RUNG-1 EFFORT SWEEP RESULT (2026-07-11, rafa, k=3 per config)
**Effort does NOT recover either S2 RBAC miss. Confirms prior lesson: capability miss needs a MODEL jump, not compute.**
Hand-judged right-reason vs the strict per-cell `why` rubric (credit only the canonical defect, reject the discriminator framing).

| config | S2-role (perms-exceed-actor) | S2-team-owner (owner-invariant backdoor) | tok/roll | s/roll | cost× |
|---|---|---|---|---|---|
| luna/low  | 0/3 MISS | 0/3 MISS | 40.6k | 61s  | 1.00 |
| luna/med  | 0/3 MISS | 0/3 MISS | 88.7k | 103s | 2.18 |
| luna/high | 0/3 MISS | **1/3** (flaky, r1 only names "demote owner via generic role endpoint") | 116.9k | 277s | 2.88 |

- **S2-role: flat 0/3 across low/med/high.** High effort latched onto "system-role immutability unverified" — a DISCRIMINATOR the rubric flags as present+correct in both vuln/safe (a near-FP), never the actual `permissionsExceedActor` escalation.
- **S2-team-owner: 1/6 rolls at high** (below the ≥2/3 bar; 5.5 gets 7/7). One roll named the owner-demotion backdoor; not reliable.
- **Cost verdict: high effort = ~2.9× tokens + ~4.5× wall for ~0 capability gain.** Terrible value. Do NOT raise luna effort for RBAC.
- **NEXT RUNG = model jump (Terra/low on the 2 S2 cells), NOT more luna effort.** Raw: `scratchpad/sweep_gpt-56-luna_{medium,high}_instrumented.json`. Chart facet `s2-rbac-misses` in `scratchpad/seat_bench_chart.png`.

## RUNG-2 RESULT — gpt-5.6-terra/low (2026-07-11, rafa, k=3)
**Terra/low partially recovers S2-team-owner (1/3) but is still BLIND to S2-role (0/3). Neither reliably caught (bar = ≥2/3).**

| config | S2-role | S2-team-owner | tok/roll | s/roll | cost× (vs luna/low) |
|---|---|---|---|---|---|
| terra/low | 0/3 MISS | **1/3** (r2: "grant the special owner role without owner authorization") | 47.4k | 140s | 1.17 |

- **Best value so far on S2-team-owner:** same 1/6 hit-rate as luna/high but at **1.17× cost vs 2.88×**. Terra family sees the owner-invariant backdoor where luna barely does.
- **S2-role is the stubborn cell:** 0/3 across luna low/med/high AND terra/low. Hardest S2 (escalation WITHIN an authorized+validated action). Needs a higher tier.
- **NEXT RUNG = Sol/low** (top family, cheapest effort) on the 2 cells — does the top family see S2-role at all? If Sol/low still misses → sol-medium (preset `reviewer.high` ceiling). Raw: `scratchpad/sweep_gpt-56-terra_low_instrumented.json`.