# Codex grant — one writer, one lock

**Status: SUPERSEDED for the authority target by `docs/plans/2026-08-17-subrouter-authority.md`.**

The replacement preserves the registry-parser, owner-visible lifecycle, redacted transition, and selected-host attestation outcomes from S4/S6/S7/S3. It cancels S1/S2/S5's copy reconciliation or custom refresh trigger: self-hosted Subrouter becomes the only refresh-token owner, and migrated clients receive no provider refresh token. Historical incident analysis below remains authoritative evidence.

Remediation plan for the incident recorded in
`docs/reports/2026-08-17-zync2-codex-token-invalidated.md`. Nothing here is implemented.

## The problem in one paragraph

One OpenAI OAuth grant per account is written by several independent processes and copied
to several machines, with nothing serialising them. The account's own
`CODEX_HOME/auth.json` is refreshed by any local `codex` run and by the systray health
probe. `claudex` copies the same grant to `CLAUDEX_HOME/codex/auth.json` for
`claude-code-proxy`, which refreshes that copy independently; `reconcile_credentials`
merges the two by comparing access-token `exp`, with no lock (`modules/systray/claudex.py`).
`seat-creds.mjs` pushes the file to debian1/2/3, and each seat container copies it again
and discards any refresh by design. The Claude grant already has the hardened
single-writer treatment (`modules/systray/claude_oauth.py`); the codex grant has none. On
2026-08-13 the account's access token was superseded server-side while its refresh token
stayed valid; nothing local noticed, nothing refreshed, and the only owner-facing recovery
path (the tray's Repair button) was itself broken. The card read "Needs repair" for four
days while 270 container runs kept succeeding on the same account.

## Design constraint that shapes everything

**`codex` is the real writer, and it does not participate in any lock we invent.** A
systray-side `flock` can serialise *our* writers (probe, claudex reconcile, converger) but
cannot stop the CLI from refreshing `auth.json` underneath us. So the design goal is not
mutual exclusion with the CLI — it is:

1. never have a second *copy* of the grant that can refresh independently, and
2. never write the file ourselves in a way that can lose the CLI's newer content.

Anything that only adds a lock while leaving the second copy in place fixes the smaller
half of the problem.

A `gpt-5.6-sol`/high review (2026-08-17) sharpened this further by reading the CLI source.
A flock **cannot** guarantee that a directly-invoked `/usr/local/bin/codex` stays out, that
a long-running codex process will not refresh hours after its launcher released the lock,
that a remote box or container observes the lock at all, or that two copied grants are not
redeemed concurrently at the provider. Holding a lock for the full lifetime of every
refresh-capable process is the only external workaround, and it serialises multi-hour
sessions — an availability hazard worse than the defect. **So the target design is one
refresh-token owner per grant**: a single writer performs all rotations; the proxy, the
health probe, and seats receive short-lived access credentials and account metadata but
never the refresh token; a consumer that hits a rejected access token asks the owner to
rotate rather than rotating itself. Codex 0.146's app-server accepts externally-supplied
`chatgptAuthTokens`, which is the plausible integration seam. If the proxy cannot consume
an externally-managed access token, it gets its own separate grant — never a duplicate of
the canonical one.

## Correction: the CLI does not refresh on a 401

The original S5 premise was wrong, and the review proved it from codex 0.146 source:

- The `account/rateLimits/read` path returns a backend 401 as an app-server error. It does
  **not** retry through `refresh_token()`.
- Codex refreshes proactively only when the access JWT is within five minutes of expiry —
  or, when the expiry cannot be parsed, when `last_refresh` is older than eight days.

At four days old with ~six days of `exp` left, a server-invalidated token qualifies for
neither. So the probe's group-kill probably did not interrupt anything, and waiting longer
would have produced the same 401 forever. That matches the observed four-day plateau
better than the interrupted-write theory and removes it from this plan.

**Unresolved against that:** something in the 2026-08-17 13:24 local probe sequence *did*
refresh the grant (`last_refresh` advanced, the file was rewritten, and usage worked
immediately after). Which call did it is not established — `codex login status` ran in the
same sequence. Whatever S5 becomes must be built on an identified trigger, not on this
correlation.

## Slices

Each slice ends in owner-obtainable evidence and lands before the next starts.

Order revised after review — the registry parser split now leads, because a credential
safety path that depends on which copy of a parser happens to be imported cannot be
trusted to hold any of the later invariants.

### S1 — Remove the second copy (P1)

`claudex.py::codex_proxy` stops materialising `CLAUDEX_HOME/codex/auth.json` as an
independently-refreshable copy. Preferred: point `claude-code-proxy` at the account's own
`auth.json`. If its config layout forbids that, keep the copy but hold an exclusive lock
for the proxy's whole lifetime and reconcile under that same lock, treating the proxy copy
as authoritative on exit.

- Evidence: start a claudex session, refresh the grant, exit uncleanly (SIGKILL the
  proxy), and show `CODEX_HOME/auth.json` still holds a working grant — currently this is
  the case that strands an invalidated token.
- Risk if wrong: a proxy that cannot read the credential fails GPT model serving. Gate on
  a real claudex round trip before landing.

### S2 — One lock for our own writers (P1)

A single flock sidecar next to the account's `auth.json`, taken by every overdeck-owned
writer: the claudex reconcile, the seat converger's read, and any future refresher. Copy
the `claude_oauth.py` shape (flock → re-read under lock → atomic replace). Explicitly does
NOT include a POST: systray code never spends a refresh token itself.

- Evidence: a test that runs two reconciles concurrently against a fixture and shows the
  later read observing the earlier write, plus a lock-contention test that does not
  deadlock the tray.

### S3 — Never select a host whose credential is not attested (P1, rewritten)

The original wording ("any host that cannot be proven current makes the dispatch fatal")
was too broad: `convergeCredentials` walks every eligible host *before* the transport picks
one, so one offline buildbox — or the current parser mismatch — would halt the whole fleet.

The correct invariant is narrower: **never select a host unless its required credential
digest was just attested, or is covered by a trustworthy receipt.** An unreachable host
that is not selected is not an error. Unreadable registry stays fatal.

- Evidence: with one box unreachable, a dispatch still runs on a healthy box; with the
  *selected* box holding a stale digest, the dispatch refuses rc 3 and starts no container.
- Risk if wrong: a fatal path can block all offload. Verify a normal dispatch first.

### S4 — Registry parses the same way everywhere (P1, now first)

The repo copy of `modules/workstation/claude/lib/buildbox-registry.mjs` rejects the live
`~/.claude/buildbox-hosts.json` (`unknown key "k3s"`, added 2026-08-13) while the installed
copy accepts it, so whether credentials converge at all depends on which transport is
loaded. Reconcile the two and version the registry schema; an unparseable registry must be
loud, never an empty host list. Highest risk-removed-per-effort of anything here. Owned by
the lane that added the key.

### S5 — Classify the rejection and ask the owner to rotate (P2, rewritten)

Waiting does not heal this — the CLI's rate-limit path never retries through a refresh (see
the correction above). Once S1/S2 give the grant a single writer, the probe should classify
`token_invalidated` distinctly, request exactly one owner-managed refresh through a
supported app-server RPC, and retry the rate-limit read once before painting the card
broken. Systray still never POSTs a refresh itself.

- Depends on S1/S2. Implementing it first would add a second refresher — the defect.
- Risk if wrong: forcing a refresh per poll rotates credentials repeatedly and collides
  with other writers; killing a refresher after the provider succeeds but before the write
  lands orphans the new grant. Bound to one attempt, and keep the broken state reachable —
  a probe that retries forever hides real breakage.

### S6 — Divergence becomes an alarm (P2)

`CredentialConflictError("divergent Codex refresh tokens")` is today an exception in a log.
Make it an owner-visible account state with the now-working Repair button attached.

### S7 — Status transitions are recorded (P3, expanded)

Status alone is not enough to diagnose the next one. Each transition records: time, the
structured error class/code, the codex version, the credential file's mtime, and
**fingerprints only** — access-token fingerprint plus `iat`/`exp`, refresh-token
fingerprint, proxy-copy fingerprint.

- NEVER log a raw error payload or token text; the 401 body and the JWT are both credential
  material. Bound the log — an unbounded append-only file is a disk-exhaustion path on a
  machine that has already hit ENOSPC.

## Recommended order after review

`S4 → S6 → S7 → S3 → S1 → S5`, with S2 folded into S1 rather than shipped as its own
guarantee. The reviewer ranked a lock-only slice last and called it "insufficient and
potentially misleading" — it protects only cooperating overdeck code while reading as
though the grant is now safe. It stays in the plan solely as the mechanism inside S1, never
as the headline fix.

## Explicitly rejected

- **A systray-side periodic refresher, uncoupled.** It would keep the workstation copy
  fresh and make every box copy stale the moment it ran. Only acceptable if each refresh
  triggers a converger push in the same critical section — and even then S1/S2 come first.
- **Any refresh POST from systray code.** The provider rotates refresh tokens; a
  second refresher is the thing this plan exists to remove, not to add.
- **Bind-mounting the credential writable into seat containers.** Copy-and-discard is a
  deliberate containment property; the fix belongs on the host side, not by giving a
  container write access to the owner's grant.

## Open question

The review added a second candidate worth ruling out before the first is accepted: the
same bearer may have stayed acceptable to the *model* endpoint while `/wham/usage` alone
rejected it — an endpoint-specific rejection rather than a global session invalidation.
The discriminator is whether the 270 successful seat runs used the same access-token bytes;
compare the old token's `exp` against the run timestamps. If successful runs happened after
that `exp`, containers refreshed and the fan-out story returns.

Partly settled already: the access token minted 2026-08-17 carries exactly a 10-day life
(`iat` 13:24:55 → `exp` 2026-08-27 13:24:55), so the Aug-13 token would have expired around
2026-08-23 19:57. The last successful seat run was 2026-08-16 18:12 — comfortably inside
that window. **No container ever needed to refresh**, which is consistent with the surviving
refresh token and leaves endpoint-specific rejection or a server-side session event as the
remaining candidates. This rests on the 10-day life being constant across mints, which is
one observation, not a documented guarantee.

Why the access token was superseded on 2026-08-13 while its refresh token stayed valid for
four more days is not attributable from this machine. A straightforward rotation should
have spent that refresh token. Either the provider tolerates the previous refresh token for
a window, or something minted a new session without consuming the token in the account
file. The evidence that would discriminate is gone (old access token overwritten, journal
retention starts after the event, proxy copy deleted). S7 exists so the next occurrence is
datable.