# Flaky-tail reproducibility + fix-quality — pre-registered rubric

date: 2026-06-24 · written BEFORE any roll (goalpost-lock per advisor) · models: `openrouter/minimax/minimax-m3` (opencode) vs `composer-2.5` (cursor-agent)

## Why this exists
Two questions, pre-registered so grading cannot drift post-hoc:
1. **Reproducibility** — is minimax's cross-model dice value on the flaky tail repeatable? Re-roll FIND x3 on every flaky-tail cell.
2. **Fix quality** — oracle-told-fix: given the localized bug, can minimax PATCH it as well as composer-2.5? If yes, the gate (dice-union detect) + cheap-model-fix pipeline could displace Cursor's expensive fix labor.

**Fix protocol = oracle-told, NOT autonomous find+fix.** Both models get the SAME localized finding + the file, and produce a patch. Rationale: minimax is a flaky *finder* (S2-role 3/9); an autonomous find+fix run would score minimax's fix as failing mostly because it never found — confounding detection with fixing. Oracle-told isolates fix skill and matches the real deployment (gate detects, cheap model fixes). **Caveat:** composer's native cursor mode is autonomous, so identical-task is a fair *fix-skill* head-to-head, not composer "as used today."

## Flaky-tail set (recall SoT `domains/security/recall/records.json`, catches<k)
| cell | recorded | fix_sha (zync.is) | file |
|------|----------|-------------------|------|
| S2-role-privilege-escalation | 3/9 baseline | 1b19ed1 | apps/zync-api/src/routes/roles.ts |
| S2-team-owner-invariant | 2/3 baseline (minimax 1/3 prior) | 43d8e10 | apps/zync-api/src/routes/users/team.ts |
| S6-jira-baseurl-ssrf | 2/3 baseline | bec8bc8 | packages/integrations/src/tasks/jira.ts |
| S3-task-assignee-membership-zyncapi | 3/3 minimax prior (not in records) | efa7486 | apps/zync-api/src/routes/tasks.ts |

k=3 re-rolls are a COARSE reproducibility check, NOT a rate — S2-role's baseline is k=9, not directly rate-comparable to k=3.

## FIND credit (right-reason, per cell) — same as canonical.json `why`
- **S2-role**: name that POST/PATCH `/roles` let a user define a role with permissions EXCEEDING their own → privilege escalation. NOT generic coarse-RBAC.
- **S2-team-owner**: name the owner demote/promote backdoor on PATCH `/team/:id/role` (DELETE guards the owner, PATCH-role does NOT). NOT the perm-subset check (already correct), NOT the transfer path.
- **S6**: name jiraAdapter fetching tenant-controlled `baseUrl` with no SSRF guard (only a falsy check).
- **S3z**: single out `assignee_id` (and project_id/status_id) passed to create/updateTask with NO tenant-membership check → cross-tenant IDOR. The MISSING call, not a present-but-wrong predicate. NOT generic "validate FKs".

## FIX grade (graded by hand vs the canonical file-scoped hunk — multi-concern commits: grade ONLY the cell's file hunk)
Canonical patches (the bar a fix must semantically match; equivalent-but-different counts, wrong-place/condition does not):
- **S2-team-owner** (team.ts): on PATCH `/team/:id/role`, fetch target member; reject if `target.role==='owner'` (403) AND reject granting `role.name==='owner'` via this endpoint (403). Transfer path stays the escape hatch.
- **S2-role** (roles.ts): add a superset guard — reject when requested `permissions` ⊄ `session.permissions` (403), on BOTH POST and PATCH.
- **S6** (jira.ts): call `assertSafeOutboundUrl(baseUrl)` (or equivalent guard blocking loopback / RFC1918 / link-local / metadata / non-https) before the outbound fetch(es) in jiraAdapter.
- **S3z** (tasks.ts): on POST and PATCH, add `getMembership(db, assignee_id, tid)` → 404 if absent (+ getProjectById/getTaskStatusById for project_id/status_id). Core credit = the assignee_id membership check.

Grade levels: **FULL** (closes the specific hole at the right point, safe path intact) · **PARTIAL** (right idea, wrong place/condition, or only one of POST/PATCH) · **MISS** (no check / wrong fix) · **BREAK** (over-blocks / breaks the safe path).

---

## RESULT 1 — FIND reproducibility (minimax k=3, 2026-06-24)

minimax db-cost this batch = **$0.289**. Grades by hand vs the FIND-credit criteria above. All finish=`stop`.

| cell | this batch | prior point-est | verdict |
|------|-----------|-----------------|---------|
| **S6-jira-baseurl-ssrf** | **3/3 CATCH** | 2/3 baseline | **ROBUST** — all 3 name baseUrl SSRF + 169.254 metadata + credential-exfil, propose assertSafeOutboundUrl-equivalent. Reproducibly caught. |
| **S2-team-owner-invariant** | **2/2 CATCH** (roll 1 INVALID) | 1/3 minimax | reproducibly catchable; roll 2&3 name the owner demote/promote backdoor on PATCH role + contrast the DELETE owner-guard, exact right-reason. Prior 1/3 looks like the unlucky tail. |
| **S2-role-privilege-escalation** | **1/3 CATCH** | 3/9 baseline | FLAKY ~⅓ — only roll 1 names "grant a role bearing permissions the caller does not hold"; rolls 2/3 chase cross-tenant reassign / DoS / UUID and miss the absent superset guard. Consistent with the 3/9 baseline. |
| **S3-task-assignee-membership-zyncapi** | **1/3 CATCH** | 3/3 minimax | **prior 3/3 did NOT reproduce.** Rolls 1&2 find the *taskId* IDOR + reporter-spoof (real, but the WRONG bug — taskId is already tid-scoped); only roll 3 singles out `assignee_id` tenant-membership (inside its fix). The earlier 3/3 was an optimistic point estimate — true catch ≈⅓. |

**Per-roll** (finish/chars/cost): s2role 1–3 $0.0281/$0.0179/$0.0216 · s6 $0.0324(1 resume)/$0.0225/$0.0175 · s2 $0.0090(roll1 INVALID, 95 chars)/$0.0369/$0.0202 · s3z $0.0320/$0.0194/$0.0319.

**Harness flake:** s2 roll 1 reply = "No file was read — the read tool call was rejected." minimax's Read was auto-denied once in non-interactive opencode; the resume prompt ("emit from analysis") then had nothing to emit. NOT a model miss — discard as a data point. ~1/12 rolls hit this; a fix harness should pre-seed the file content into the prompt rather than rely on an in-arena Read tool.

**Reproducibility BLUF:** minimax dice value is real but cell-specific — ROBUST on S6 (3/3), reliable on S2-team-owner, genuinely flaky (~⅓) on S2-role and S3z. The headline correction: **S3z's prior 3/3 was optimistic — it reproduces at ~⅓**, so a single k=3 point estimate over-claims; dice value holds only as union-additivity, not as per-cell reliability.

## RESULT 2 — FIX quality (oracle-told)

### minimax-m3 — STRONG (db-cost $0.106 this batch)
Both models told the localized bug, asked for a patch; graded by hand vs the canonical file-scoped hunk.

| cell | grade | note |
|------|-------|------|
| **S6** | **3/3 FULL** | every roll adds an SSRF guard before each outbound fetch: deny-list loopback / RFC1918 / link-local / 169.254 metadata / multicast, `new URL` parse, https-enforce, several also DNS-resolve every A/AAAA + disable redirect-following + reject userinfo. **Exceeds** the canonical (`assertSafeOutboundUrl`). |
| **S3z** | **2/2 FULL** (r3 invalid) | r1/r2 add a tenant-scoped FK check on POST&PATCH — `getUserInTenant(db,tid,assignee_id)` ≈ canonical `getMembership`, 404 on miss, covers project/status too. r3 = 157-char harness fail (resume produced no patch), discard. |

**Per-roll cost / tokens (in/out/reason/total):** s6 $0.0207 27982/1808/8297/41671 · $0.0214 14674/1700/11611/44881 · $0.0137 14691/2193/4710/38490 — s3z $0.0144 16078/2456/3879/54669 · $0.0144 27829/2986/26/71801 · $0.0218(INVALID) 31522/332/6811/101071.

**Headline: minimax FIXES far better than it FINDS.** S3z it found at ~⅓ but, once told, patched 2/2 FULL. S6 it both finds (3/3) and fixes (3/3) cleanly. This is the strongest signal for the *detect-then-cheap-fix* pipeline.

**Caveat (portability):** minimax's S6 fix leans on `node:dns/promises` + `node:net.BlockList`. zync is **Cloudflare Workers** — those Node APIs are not in the Workers runtime; the canonical used a Workers-safe shared util. The guard *logic* is right and thorough, but a minimax patch would need runtime-API review before it runs. A finder/fixer that doesn't know the deploy target over-reaches on imports.

### composer-2.5 (live) — BLOCKED
6 cursor-agent rolls `rc=1`, 0 output. Not the harness: `--force` clears workspace-trust, but the model call returns **`ActionRequiredError: You're out of usage`** — cursor account (royachiron@gmail.com) hit its cap. composer-2.5 unavailable live until the limit resets/raised. Cursor billing is a separate pool from OR.

## RESULT 3 — repo-scale agentic audit: minimax vs composer's KNOWN result

Composer's result is already recorded (no re-run needed): cursor-agent (composer-2.5) blind-audited `multideal@40056d2c6` (`apps/web/src/**`, `apps/web-do/**`, `functions/**`, S1–S13) and produced **7 Claude-verified candidates** (3 S6 SSRF real, 1 Shape-B real, 3 deployment-conditional) — clean, structured, per-class table (`docs/harvest/2026-06-20-multideal-scan-VERIFIED.md`).

**minimax on the IDENTICAL task FAILED.** One pass, opencode/OpenRouter, same scope + audit prompt, worktree @40056d2c6:
- 29 turns / 35 tool calls; read the right security-critical server files first (SSRF, amount-bounds) — capability present — then **fanned out into 4 parallel sub-agent `task` calls** and **never synthesized a final report**.
- `finish=null`, **0 findings emitted**, 1770 chars of scattered intermediate text, no per-class table. The truncation-resume wrapper fired twice ("emit findings now") and still got nothing — there was no converged answer to extract.
- **Cost: $0.72 OR in one pass** (vs ~$0.02–0.05 single-file) — the sub-agent fan-out drained nearly the entire remaining OR budget for zero usable output.

**Verdict (repo-scale): minimax-via-opencode is NOT a composer replacement for unbounded agentic audit/coding.** Failure mode = sprawl + non-convergence + cost blowout, not lack of skill. Composer completed the same haystack cleanly. A bounded scaffold (explicit file enumeration, NO sub-agent fan-out, forced single-pass output) might recover it — untested; out of OR budget ($0.018 left).

---

## OVERALL VERDICT — what minimax-m3 is good for

| flow | minimax-m3 | use it? |
|------|-----------|---------|
| **Fixing** (oracle-told, localized, single-file) | **STRONG** — S6 3/3, S3z 2/2 FULL, exceeds canonical | **YES** — cheap fix labor once a bug is localized. Review imports for deploy-target (Workers) portability. |
| **Finding** (single-file, as a dice-union member) | flaky ~⅓, S6 robust 3/3 | **YES, but only as an additive union roll** — never solo, never as a discriminator (C02/C09 0/3). |
| **Reviewing** (repo-scale agentic audit) | **FAILS** — sprawls, fans out to sub-agents, no convergence, $0.72/pass for 0 output | **NO** (as a drop-in agentic worker). Possibly yes with a bounded non-agentic scaffold — untested. |
| **Coding** (greenfield/feature, agentic) | not directly measured; the repo-audit failure (same agentic mode) is the strong prior | **NO** until proven with a bounded scaffold. |

## RESULT 4 — MiMo V2.5 Free (OpenCode Zen, `opencode/mimo-v2.5-free`, headless)

Re-ran the same tasks (find + fix; repo scan skipped per user). **FREE — cost=0** on every call (sidesteps the exhausted OR budget).

**Behavior — better-shaped than minimax:**
- **Does NOT over-reason** (12–75 reasoning tokens vs minimax's 10k–50k) — completes cleanly, `finish=stop`, no truncation spiral, no sub-agent fan-out. On the single-file tasks it stayed on the file and emitted a structured review directly.
- **BUT free-tier RATE-LIMITED** — back-to-back/burst calls get throttled to instant **0-byte** returns; only spaced calls succeed. A naive k=3 loop fails (all-empty); needs pacing + retry. This blocked the full unattended battery.

**Capability (k=1 probe — too few rolls for a rate):**
| task | grade | note |
|------|-------|------|
| S6 find | **CATCH** | names SSRF + credential-exfil on tenant `baseUrl`, right-reason |
| S2-role find | **MISS** | flagged the cross-tenant `reassign_to_role_id` IDOR but missed the canonical permission-superset guard — the SAME blind spot as minimax rolls 2/3 |

**Verdict (partial):** MiMo is the more *promising harness fit* — free and convergent (none of minimax's over-reasoning/fan-out failure). On the k=1 probe its hit/miss pattern matches minimax (S6 easy, S2-role hard). A real k=3 + fix comparison needs a **paced/retry** runner (the rate-limit, not capability, is the blocker) — not run (token budget). 

## RESULT 5 — OpenRouter FREE models, surveyed the same way (2026-06-24)

User asked to check the *other* OpenRouter free models and compare them the same way via opencode headless. Two findings, both negative.

### 5a — Agentic opencode harness is NOT viable for OR free models
Ran the established opencode-headless agentic path (`opencode run --dir <arena> -m <id>`, model uses Read tool) on the candidate set. The real ~3.9k-token detector prompt **fails on every one**:

| model | direct-API liveness (tiny prompt) | opencode agentic (full detector prompt) |
|-------|-----------------------------------|------------------------------------------|
| `qwen/qwen3-coder:free` | **HTTP 429** upstream rate-limit | rc=124 (SDK retry-stall, 0 bytes) |
| `meta-llama/llama-3.3-70b-instruct:free` | **HTTP 429** | rc=124 |
| `qwen/qwen3-next-80b-a3b-instruct:free` | **HTTP 429** | rc=124 |
| `nousresearch/hermes-3-llama-3.1-405b:free` | **HTTP 429** | rc=124 |
| `nvidia/nemotron-3-ultra-550b-a55b:free` | 300s stall, empty | rc=124 |
| `openai/gpt-oss-120b:free` | **LIVE** (2.6s) | **rc=124** — `session.error` at +3.5s then opencode `run` *hangs* (never exits); works on a TRIVIAL prompt (Glob→Read→finding) but the full detector prompt errors+hangs every time |
| `nvidia/nemotron-3-super-120b-a12b:free` | **LIVE** (1.3s) | rc=124 |

Root causes confirmed from `~/.local/share/opencode/log` + db: (1) most free models are **hard-429** (shared free pool congested); (2) the live ones throw `session.error` on the real multi-turn tool-calling request and **opencode hangs instead of failing fast**. Net: **you cannot run OR free models through the opencode agentic flow** — the exact harness minimax/MiMo use. This is the direct answer to "can we use them in our flows": **no, not the opencode-agentic flow.**

### 5b — Raw capability (single-completion, file inlined — NOT comparable to the agentic minimax/MiMo grades)
To still measure whether the *models* can find the bug, sent the full detector prompt + inlined file as ONE completion (the only path that runs) to the two live models, across the 3 discriminating cells. **Right-reason: 0/6.**

| cell (canonical right-reason) | gpt-oss-120b:free | nemotron-3-super-120b:free |
|---|---|---|
| S6 — SSRF on tenant `baseUrl` before fetch | **MISS** (flagged `external_id` injection / btoa, not baseUrl SSRF) | **MISS** (described baseUrl as benign credentials; truncated at max_tokens before findings) |
| S2-role — permission-superset privilege escalation | **MISS** (caught cross-tenant roleId IDOR — a *different* real bug) | **MISS** (0 relevant) |
| S3z — missing membership check on `assignee_id` | **MISS** (generic ownership/TOCTOU, never singled out assignee_id) | **MISS** (found assignee but declared the `tasks:assign` *permission* check "good" — conflated permission with membership) |

gpt-oss surfaced *adjacent* real bugs (it is a competent generalist) but missed all three canonical right-reasons — including **S6, the easy cell minimax catches 3/3 and MiMo catches k=1**. nemotron is verbose, hits the token cap before converging to findings, and missed all three.

### Verdict (OR free models)
**Not usable for security-gate.** Two independent walls: (a) the opencode *agentic* harness fails for them (429 / session.error+hang) — so they don't even fit the flow; (b) at their capability ceiling (single-shot) the two reachable models miss every canonical cell, including the one the paid/zen models reliably catch. No money was spent (all free tier). The cheap-worker lane that pays off remains **minimax (paid, narrow) + MiMo (free, convergent)** — NOT the OpenRouter free pool.

## RESULT 6 — OR free models: FIXING + CODING (the same battery asked of minimax/MiMo)

Find capability (RESULT 5b) was 0/6. Fixing and coding tell the **opposite** story — same inversion seen in minimax (fixes ≫ finds). All via direct single-completion (the agentic harness is dead for these, RESULT 5a); $0 spent. Live models: `gpt-oss-120b:free`, `nemotron-3-super-120b:free` (qwen/llama still hard-429).

### Oracle-told FIX (k=3 each, graded vs canonical hunk)
| cell | gpt-oss-120b:free | nemotron-3-super-120b:free |
|------|-------------------|----------------------------|
| **S6** (SSRF guard before fetch) | **3/3 FULL** — pure-JS IP-range block (RFC1918/loopback/link-local/CGNAT/metadata), guards BOTH fetch sinks. **Cloudflare-Workers-compatible** — *better CF-compat than minimax's fix*, which reached for `node:dns`/`net.BlockList` (not on Workers). | **3/3 FULL** — equivalent IPv4-regex + IPv6 guard; minor nit: a redundant `import { URL } from 'url'` (URL is a Workers global). |
| **S3z** (tenant-membership on FK refs, both handlers) | **3/3 FULL** — adds a real `belongsToTenant(db,table,id,tid)` DB lookup; guards `project_id`/`status_id`/`assignee_id` on POST **and** PATCH, returns 404 on mismatch. Clean, idiomatic, matches canonical `efa7486` intent. | **PARTIAL** — POST handler complete with real per-FK tenant lookups; **PATCH truncated at the token cap** (`fr=length`). Right approach, incomplete *output* (a verbosity/budget failure, not a capability miss). |

Shared SSRF limit (both models, both = canonical too): IP-literal block only, no DNS-rebind pre-resolve — a Workers runtime constraint, acceptable at the bar.

### Bounded CODING task (objectively executed, not eyeballed)
Spec: implement `isPrivateOrReservedIp(ip): boolean` for Workers (no node:/deps), 7 IPv4 + 4 IPv6 range classes. Generated code **run via `bun` against a 20-case vector set** (private/reserved → true, public/CGNAT-boundary/hostname → false).

| model | result |
|-------|--------|
| **gpt-oss-120b:free** | **PASS 20/20** — single clean fenced block, correct boundary handling (172.16–31, 100.64/10 vs 100.128, fc00::/7, `::`), CF-compatible. A drop-in, runnable deliverable. |
| **nemotron-3-super-120b:free** | **FAIL (no deliverable)** — emitted reasoning prose, never opened a code block, **truncated mid-function** at the token cap. Logic was on-track; instruction-following + verbosity is the defect. |

### Verdict (OR free models, fix/code)
- **`gpt-oss-120b:free` is a real find: a FREE, capable oracle-told FIXER and bounded CODER** — 6/6 FULL fixes + 20/20 executable coding, all Cloudflare-Workers-compatible, follows the "code only" contract. For the **detect-(elsewhere)-then-cheap-fix** lane and small spec'd coding, it is a $0 alternative to paid minimax — *and* cleaner on CF-compat than minimax's S6 fix.
- **`nemotron-3-super-120b:free`: NOT a reliable worker** — correct instincts, but over-verbose, ignores output contracts, and hits token caps → truncated/unusable deliverables.
- **Hard ceiling (binds both):** usable **only via direct single-completion**, NOT the opencode agentic flow (RESULT 5a). So the role is "cheap fix/code oracle you feed a localized task + the file", not an autonomous agent. And neither can FIND the bug (0/6) — they fix what they're TOLD is broken.

## Bottom line (minimax)
minimax is a cheap, capable *narrow* worker — give it a localized, scoped task (fix this bug, find in this one file) and it performs, often beating its price. Give it an *unbounded agentic* job (audit/build across a repo) and it sprawls, fails to converge, and over-spends. The money-saver vs Cursor is real but **only in the detect-(union)-then-cheap-fix lane**, not as an autonomous repo agent.

## Parity / isolation (both models)
Neutral-basename file in an isolated git-root arena (no CLAUDE.md/AGENTS.md/`.cursor/` in any parent — verify cursor-agent does not pull repo context minimax never saw). Same prompt. Blind: neither model sees canonical.json, fix commit, or audit docs. Cost pools kept SEPARATE: minimax = OpenRouter $, composer = cursor account.

## RESULT 7 — Real cursor-cli tasks given to free models (NON-security, user-directed)
**Setup:** Pull tasks cursor-cli actually completed + committed in `~/Projects/multideal` (ground truth = the merged commit, NOT the transcript self-report). Pre-fix repo state = task input; grade free-model output by **byte-diff against cursor-cli's committed file** + symbol checks. Identical verbatim prompt + the same source files cursor-cli would Read, inlined.

### Harness finding (blocks the like-for-like agentic comparison)
cursor-cli's tasks are **agentic** (Read → Edit → run typecheck → `git commit` → `git push`, with a `STATUS/COMMIT` report contract). The opencode agentic harness is **BROKEN this session** — `opencode run` hangs (rc=124, zero events) **before any model call**, for the **MiMo control too** (not free-model-specific): (1) `opencode-timer-plugin` missing-`.tgz` ENOENT fired `session.error` during bootstrap (removed from `~/.config/opencode/opencode.json`); (2) even after removal, bootstrap completes then `run` stalls before issuing the model request. Direct-API liveness is fine (gpt-oss PONG 6s; qwen3-coder still hard-429). **So the free models could NOT be run through cursor-cli's actual agentic loop** — they were graded on the deliverable (the code diff) via direct single-completion, the labeled fallback. "Can't run the cursor-cli agentic loop here" is a real limitation, but this session it's an *environment* break, not a proven model-capability ceiling.

### Task nature (the honest framing)
cursor-cli in multideal is driven by **prescriptive task-cards** — the prompt hands over the exact target code (full file content / `const showBuyButton = ...` recipe / Image→Camera swap). So this measures **instruction-following + exact reproduction**, NOT autonomous problem-solving. Of ~40 small (≤12-turn) cursor-cli sessions surveyed, the code-producing ones are all recipe-style; the rest are agentic infra (run typecheck / run Playwright / commit-only).

| task (cursor-cli committed) | gpt-oss-120b:free | nemotron-3-super-120b:free |
|------|-------------------|----------------------------|
| **DealCardCornerCta** split `showCtaButtons` → `showBuyButton`/`showAddToCart` (commit `24b8c17fb`, 23+/24-) | **EXACT byte-match** to committed file (flag split + separate JSX blocks, all props preserved) | **EXACT byte-match** |
| **checkoutModalStore** extend with exported `DealContext`/`CheckoutModalPreview`/`CheckoutModalPayload` (commit `98fc005da`, 35+/1-) | **EXACT byte-match** to committed file (all symbols, comment banners, GroupData import) | **FAIL** — ignored the explicit spec, **hallucinated `DealContext` fields** (`defaultSkuId`/`csrfToken`/`isGuest`/`stockRemaining`/`maxPerUser`), dropped spec'd `maxQuantityPerOrder`/`requiresShipping`. 9 wrong lines vs golden. |

### Verdict (RESULT 7)
- **gpt-oss-120b:free reproduces cursor-cli's localized prescriptive code tasks EXACTLY (2/2 byte-perfect)** when given the same prompt + files via single-completion. For this class — spec'd, single-file, code-diff deliverable — it is a $0 equal to what cursor-cli did, consistent with RESULT 6's fix/code wins.
- **nemotron stays unreliable (1/2)** — even handed an explicit field-by-field spec it invented its own `DealContext` shape. Confirms RESULT 6: ignores contracts.
- **The gap that remains is the agentic loop, not the edit.** cursor-cli's value here is running the loop (typecheck → commit → push, report back) end-to-end; the free models deliver the *edit* but were not runnable through that loop this session (harness break). To claim free-model agentic parity, fix the opencode `run` hang first, then re-test on these same tasks.

## RESULT 8 — gpt-oss:free THROUGH the Claude Code harness (true agentic, not single-completion)
**Rig (BUILT + PROVEN):** Claude Code → `claude-code-router` translating proxy (Anthropic `/v1/messages` ↔ OpenAI `chat/completions`) → OpenRouter. Isolated `CLAUDE_CONFIG_DIR=~/.claude-openrouter` (clean, no session hooks). Launcher `~/.claude-openrouter/cc-or.sh <model> [args]`. This answers "same harness, swap model" far more fairly than single-completion — the free model now drives Claude Code's REAL Read/Edit/Bash loop.

**gpt-oss-120b:free unblock (was 400 "Reasoning is mandatory…cannot be disabled"):** custom ccr transformer `~/.claude-code-router/plugins/reasoningfix.js` strips the fatal `reasoning.enabled:false` and forces a bounded `effort`. Effort is a wired knob — default `low`, override `CCR_EFFORT=low|medium|high cc-or.sh …` (proven: `medium` → `{"effort":"medium"}` on the wire). `effort:high` on the free "OpenInference" provider degrades into endless whitespace-reasoning → hang; `low` converges.

**Agentic eval — 2 real cursor-cli tasks, n=2 each, graded by diff vs the committed file:**

| task | n | result |
|------|---|--------|
| **DealCardCornerCta** (split a flag — the *change* is described in the prompt) | **2/2 PASS** | full agentic loop (Read→Edit→verify, 4–6 turns, 60–143s), edit semantically identical to cursor-cli `24b8c17fb`. |
| **checkoutModalStore** (replace file with an *exact pasted spec* + a "DealContext must match CheckoutModal.tsx — verify first" caveat) | **2/2 FAIL** | drives the loop fine (8–9 turns, 185–197s) but BOTH runs added the **same 5 unrequested fields** (`defaultSkuId`/`csrfToken`/`isGuest`/`stockRemaining`/`maxPerUser`) pulled from CheckoutModal.tsx — diverging from cursor's committed result. Defensible-but-wrong: it over-prioritized "match the codebase" over the literal "replace with this content". **Same failure mode nemotron showed in single-completion (RESULT 7).** |

### Verdict (RESULT 8) — is gpt-oss:free good for our tasks?
- **YES for localized logic edits where the prompt DESCRIBES the change** (dealcard-style): drives the full Claude Code agentic loop, free, reliable 2/2.
- **NO for verbatim / exact-spec reproduction** (paste-this-file task-cards — which is exactly how cursor-cli is driven here): it editorializes, importing fields/shape it judges relevant → infidelity vs the known-good result. Reproducible 2/2, not a fluke.
- **Operational caveats:** free-pool latency is variable (60–200s, occasional rc=124 hang → retry); must run at `effort:low`; reported `total_cost_usd` is fake (Anthropic rates) — real `:free` cost = $0.
- **Net:** a free, agentic, capable *executor of described changes* — a real $0 lane for the "do this specific edit" class. NOT a drop-in for prescriptive paste-the-spec cards where fidelity to the literal spec is the bar; for those, cursor-cli's literal-reproduction is more trustworthy.

### RESULT 8b — effort sweep on the checkoutModalStore FAIL (does more reasoning fix fidelity?)
Re-ran the failing exact-spec task at each effort (wire-confirmed via ccr log):

| effort | outcome |
|--------|---------|
| low    | FAIL — same 5 unrequested fields (the n=2 baseline) |
| medium | FAIL — same 5 fields (7 turns, 243s) |
| high   | FAIL — same 5 fields **+ rc=124 hang** (720s timeout, whitespace-reasoning degradation returns) |

**Finding:** the over-eager field-add is a stable JUDGMENT / instruction-priority decision, NOT a reasoning-budget shortfall — gpt-oss reads the "DealContext must match CheckoutModal.tsx" caveat and prioritizes it over the literal "replace with this exact content", and MORE reasoning entrenches the same call. Higher effort only adds latency + reintroduces the free-pool hang. **Keep default `effort:low`.** To get fidelity on paste-spec cards you'd fix the PROMPT (drop/clarify the verify-caveat), not the effort.

---

## RESULT 9 — ALL OpenRouter :free models through the real Claude Code harness (2026-06-24, $0)

User asked: overview of ALL free models — good for / not good for / anything actually usable. Rig = the `claude-code-router` translating proxy + isolated `~/.claude-openrouter` config + `reasoningfix.js` (see memory `openrouter-via-claude-code-harness`). Two lanes: **dealcard** = described-edit (prompt narrates the change; primary capability signal), **cms** = exact-spec fidelity (paste-the-file + "verify vs X" caveat; instruction-fidelity-under-tension). Graded vs cursor-cli's committed golden. Failure CLASS recorded (not pass/fail). 17 tool-capable :free models (5 no-tool models can't drive an agentic harness at all). 429s retried across 3 spaced passes.

| model | dealcard | cms | class verdict |
|---|---|---|---|
| **cohere/north-mini-code:free** | **PASS 3/3** | **PASS 3/3 (byte-identical to golden all 3 rolls)** | **USABLE — only model clean on BOTH lanes; zero phantom fields on the fidelity task gpt-oss fails** |
| nvidia/nemotron-nano-9b-v2:free | PASS (n=1) | drives-no-edit | tiny 9b drives described-edits; too small for the bigger fidelity task |
| openai/gpt-oss-120b:free | 2/2 prior, wrong-edit this run | FAIL 0/3 prior | flaky described-edit; adds 5 unrequested fields on fidelity (RESULT 8) |
| nvidia/nemotron-3-super-120b-a12b:free | PASS prior | — | drives described-edit (memory rig proof) |
| nvidia/nemotron-3-nano-30b-a3b:free | wrong-edit (1 of 2 reachable) | — | drives but wrong edit |
| nvidia/nemotron-3-nano-omni-…-reasoning:free | drives-no-edit | — | reads wrong file, no edit |
| poolside/laguna-m.1 / laguna-xs.2:free | output-incompat | — | emits non-text content block harness can't parse ("Content block is not a text block") |
| liquid/lfm-2.5-1.2b-thinking:free | ctx-too-small | — | 32k ctx < ~53k the CC system prompt+tools need → hard 400 |
| nvidia/nemotron-3-ultra-550b-a55b:free | hang-rc124 | — | never converges in budget |
| nvidia/nemotron-nano-12b-v2-vl:free | hang-rc124 | — | never converges |
| qwen/qwen3-coder:free | 429 ×3 | — | chronic free-pool congestion — unreachable all session |
| openai/gpt-oss-20b:free | 429 ×3 | — | chronic 429 (drove 2-6 turns then cut) |
| qwen/qwen3-next-80b-a3b-instruct:free | 429 ×3 | — | chronic 429 |
| meta-llama/llama-3.3-70b-instruct:free | 429 ×3 | — | chronic 429 |
| google/gemma-4-31b-it / gemma-4-26b-a4b-it:free | 429 ×3 | — | chronic 429 |
| (5 no-tool models: dolphin-mistral-venice, lfm-2.5-1.2b-instruct, llama-3.2-3b, hermes-3-405b, nemotron-3.5-content-safety) | n/a | — | no `tools` support → cannot drive harness |

**VERDICT.** The free pool is mostly a wall, but it is NOT empty. **`cohere/north-mini-code:free` is the real find** — the ONLY model green on both lanes (dealcard 3/3, cms 3/3 at the project's k≥3 bar; cms output verified **byte-identical to cursor's golden** all 3 rolls via `diff`, not just heuristic), and crucially it did NOT add the phantom DealContext fields that sink gpt-oss on the fidelity lane (extra=[] every roll, 9–22 turns, 46–113s). It is a genuine $0 drop-in for *described* AND *exact-spec* localized edits — strictly better than gpt-oss-120b here. Second tier: gpt-oss-120b (flaky described-edits, fails fidelity) and nemotron-nano-9b-v2 (small, described-edits only). Everything else is dead on free tier today: the popular coders (qwen3-coder, gpt-oss-20b, qwen3-next, llama-3.3, gemmas) are chronically 429 across 3 spaced passes; the big nemotrons hang; laguna emits harness-incompatible blocks; lfm-1.2b can't fit the system prompt. **Bottom line: one usable free model — north-mini-code — for the cheap localized-edit lane; the rest are blocked by rate limits, size, or output-format incompatibility, not by the rig.**

---

## RESULT 10 — north-mini-code:free on 5 MORE diverse cursor-cli tasks (2026-06-24, $0)

Followup to RESULT 9: stress `cohere/north-mini-code:free` (the one usable free model) across task TYPES beyond the original UI flag-split + type-extend. Harvested 5 real cursor-cli commits (`Co-authored-by: Cursor <cursoragent@cursor.com>`) from `~/Projects/multideal`, each a single-file change, golden = `git show <sha>:<path>`. Lane = described-edit (prompt narrates intent + names any cross-file helper north can't see in an isolated single-file arena). Graded by per-task SEMANTIC checks (key symbols/logic present, no over-reach), THEN every output `diff`ed line-by-line vs golden (per RESULT 9's lesson: heuristic-PASS ≠ fidelity — the diff is the real deliverable). Not byte-exact-required (described-edits legitimately differ in import order/formatting). Run through the real Claude Code harness via the proxy rig, effort=low.

| task | type | cursor sha | result (diff-verified vs golden) | turns/dur |
|---|---|---|---|---|
| ssrf | add SSRF guard (`assertSafeFetchUrl` + reject) | fb198248a | **PASS** — logic identical to golden; diffs = import order + `.js` ext + brace style | 23t / 107s |
| turnstile | fail-CLOSED auth logic (unavailable→503) | c6e5666af | **PASS (code) — near-miss on docs**: 503 branch logically identical to golden, but north left the TOP comment still reading "fails OPEN if siteverify is unreachable", now contradicted by its own code (prompt asked to update it). Code right, explaining comment stale. | 10t / 33s |
| refund | conditional Stripe `reverse_transfer` + diagnostic log/capture | cb8123d2c | **PASS** — equivalent conditional (`? {} : {}` ×2 vs golden's single `&&` spread, same omit-when-not-released semantics) + same log+captureCaught (diff key names `type/code/message` vs `stripeType/...`, diff position in catch, same data) | 12t / 65s |
| ltv | money-precision refactor (`Number`→`minorForApi(BigInt)`, `sql<number>`→`sql<string>` ×4) | dca9608e1 | **PASS** — diff-verified IDENTICAL to golden modulo import order + one Map collapsed onto one line (the earlier "+1 vs golden +3 lines" was pure formatting, NOT a skipped edit; all 4 conversions present). Edit correct, but rc=124 hung after (non-termination). | — |
| dlq | error-handling restructure (ack in `finally`) | 223e4a134 | logic CORRECT, but wrote to a path-corrupted location under the pathological scratch path → real arena untouched (n=2 stable); **PASS / 29s from a clean short path** | 6–46t |

**VERDICT.** north-mini-code produced the **diff-verified-correct edit on all 5 diverse task types** (SSRF guard, fail-closed auth, Stripe conditional+logging, money-precision drizzle refactor, DLQ try/finally restructure) — broadening RESULT 9's UI-only evidence to security, payments, auth, analytics, and queue-handling code. Every output was diffed against cursor's golden, not just symbol-checked: 4/5 are golden-equivalent (cosmetic deltas only — import order, line-wrapping, equivalent ternary-vs-`&&`, key naming); the 5th (dlq) is golden-equivalent from a sane path. It is a genuinely capable $0 executor of *described* localized edits across domains. Caveats observed, none a coding-LOGIC miss:
0. **Stale-doc tail** (turnstile): correct code, but did NOT fully update an explaining comment the prompt asked it to — leaving a comment that now contradicts the code. A reviewer would catch it; worth a "update ALL related comments" nudge in the prompt.
1. **Non-termination tail** (ltv): made the correct edit, then looped past 360s without emitting final JSON (rc=124). The work is done but the harness times out — retry/bounded-turn wrapper needed.
2. **Long-absolute-path corruption** (dlq): under the ugly `/tmp/claude-1000/-home-user-Projects-…` scratch path, north dropped `user` when reconstructing the Write path, wrote the (correct) code to a phantom tree, and reported success — a small-model transcription error on a pathological path, n=2 stable. **From a normal short path the same task PASSes 4/4 in 29s.** Non-issue in real repos (short CWD + relative edits); a caveat only for deeply-nested absolute paths.

**Combined with RESULT 9** (dealcard 3/3 + cms 3/3 byte-exact): north-mini-code:free is a real, broadly-capable $0 cheap-lane coder for localized described edits — modulo a bounded-turn/retry wrapper for the occasional non-termination, and sane (short, relative) working paths. Strongest free-tier finding of this study.

---

## RESULT 11 — north-mini-code DEEP DIVE: the cursor-orchestrator role ceiling (2026-06-24, $0)

**Question (user):** test north across the WIDER range of cursor-orchestrator daily tasks, find the high limit / where it fails, is it on-par with composer-2.5 on all roles we use it for?

**Reframe (advisor):** partition by composer's THREE real cursor-orchestrator roles — **implementer, reviewer/auditor, fixer** — not an abstract difficulty ladder. **Gate asymmetry = the headline:** the orchestrator wraps composer in a two-tier gate; an implementer/fixer miss is caught downstream (tolerable-under-gate), but the **Tier-1 reviewer IS the gate — nothing catches what it misses.** So "on-par" matters far more for the reviewer seat. Composer's "score" = its archived golden commits (composer is out-of-usage, `paid-run-owner-approval`); no live composer run needed. Pre-registered grading per role BEFORE running (fix the RESULT-10 heuristic-vs-diff drift).

**3 real multideal defects (RAW git pre-fix = the latent bug cursor later fixed), spanning a defect-class gradient:**
- `ssrf` (fb198248a, botmaster.ts): outbound channel fetches attacker-configurable `baseUrl` with no SSRF guard. *Surface / pattern class.*
- `turnstile` (c6e5666af, send.ts): captcha fails OPEN when siteverify unavailable (only `outcome==='rejected'` rejected). *Medium / branch-tracing-logic class.*
- `dlq` (223e4a134, index.ts): work-class DLQ loop logs full `message.body`; if `JSON.stringify` throws, `ack()` is skipped → message reloops forever. Fix = try/catch/**finally{ack}**. *Subtle / failure-mode-reasoning class.*

### Scorecard (all $0, north-mini-code:free via Claude Code harness)

| Role | Task | Result |
|------|------|--------|
| Implementer — described-edit | (RESULT 10, 5 tasks) | **5/5 PASS** diff-verified |
| Implementer — **multi-file** src+test | settlement-monitor (d517fbc25, src + 472L test) | **PASS** — edited src (guard semantically equal-or-better than golden; also catches `null`) AND added a coordinated `it('…AdaptiveGroups is undefined')` test asserting its own reason string, matching existing test style |
| Fixer — **goal-only** (symptom given, NOT diff; must FIND then FIX) | ssrf / turnstile, n=1 · dlq **n=3** | ssrf ✓ (wrote own inline SSRF guard: loopback+RFC1918+metadata). turnstile ✓-secure-but-not-exact (`outcome==='rejected' \|\| 'unavailable'` → both return **403**; golden keeps `rejected`→403 and adds a separate `unavailable`→**503** branch, verified `c6e5666af` — north **fails closed but collapses golden's 503/403 split**: hits the security goal, misses the retryable-vs-failed refinement). dlq **1/3** (roll2 guarded the risky full-body work-class loop = correct; rolls 1&3 guarded only the safe small-field settlements loop, leaving the real throw-risk unguarded) — **flaky on locating WHICH operation can throw**, not a clean miss |
| **Reviewer / Tier-1 GATE** — *cold latent-defect detection* (whole static file, NO spec/diff/CLAUDE.md — a LOWER BOUND on the real reviewer role, whose primary job is spec-compliance with all three in hand), AUDIT mode via code-review-prompt.md | ssrf / turnstile / dlq, **k=3** | **ssrf 3/3 · turnstile 1/3 · dlq 0/3**, AND NOISY (3–6 findings/roll, mostly speculative FP: header-injection, path-traversal, credential-memory, waitUntil+await). **Disqualifier rests on turnstile 1/3** — a genuinely review-findable logic bug missed ⅔ of the time. dlq-0/3 NOT counted as a clean reviewer signal: that defect was found by a *2026-06-13 production incident* (per golden's own comment), not by review — a strong reviewer (composer included) plausibly also misses a cold oversized/circular-`JSON.stringify`-throw read. |
| Autonomous find (3-file dir, no symptom/no pointer) | botmaster SSRF among 2 clean siblings | **localized + fixed correctly**, but **42 turns / 218s** for 3 files — converges at small scale; cost trajectory projects non-convergence at repo scale (matches minimax repo-scale wall, RESULT in `paid-run-owner-approval`). Confounded: the planted class was SSRF (north's strong suit). |

### The wall (consistent across BOTH find and fix)
Defect-class gradient: **surface (SSRF) > medium (fail-open) > subtle-robustness (ack-in-finally).** north pattern-matches vulnerability *shapes* reliably; it does not reliably *reason about latent failure modes*. On the subtle class the evidence is flakiness, not a clean wall — as a told-fixer it locates the dlq throw-risk only **1/3** (and as reviewer never, but that's a hard cold read even for a strong reviewer). The cleanest disqualifying datum is the **medium** class: turnstile fail-open caught **1/3** as reviewer, fixed **secure-but-not-exact** (503/403 collapse) as fixer.

### Verdict — NOT a single pass/fail; it's role-shaped
1. **Implementer + fixer: on-par-enough with composer for surface/medium classes, multi-file included** → viable **$0 substitute in the gated coding seat** (the bulk of orchestrator implementer/fixer waves). Mirrors cursor-orchestrator's own IRON LAW (Claude/cheap-engine codes; a stronger model judges). Caveat: fixes hit the security *goal* but can miss refinements (503/403 split) — fine under a reviewer, not as the final word.
2. **Reviewer (Tier-1 gate): NOT on-par — disqualifying on the absolute bar.** Even granting only the cleanest datum: a review-findable logic bug (turnstile fail-open) caught **1/3**, plus high FP noise. Because nothing downstream catches a Tier-1 miss, a sole gate at 1/3 ships the fail-open ~⅔ of the time — that alone disqualifies the seat, no composer-relative number needed (the fix commits show composer *implemented* the fixes, not that composer-as-reviewer *caught* them, so a "composer 3/3 reviewer" baseline is unproven and not claimed). **The gate seat stays Opus/composer + advisor.**
3. **Autonomous repo-scale find: keep bounded** — works small/easy-class, cost blows up with scale.

**Bottom line:** north is a real $0 cheap CODING engine UNDER a strong reviewer — never the reviewer. Slot it into the orchestrator's gated implementer/fixer role only; the two-tier gate + Opus Tier-2 + advisor must stay the strong model. SoT for raw rolls: `scratchpad/north3/` (audit_*, goalfix_*, mf_settle, auto.json).