# zync2 codex token — why the card read "Needs repair" for four days

status: investigated 2026-08-17. Account recovered; two defects fixed, three proposals open.

## What was actually wrong

The grant was never revoked. The **access token minted 2026-08-13 12:57Z was rejected by
the usage endpoint only**:

```
GET https://chatgpt.com/backend-api/wham/usage → 401
{"error":{"message":"Your authentication token has been invalidated. Please try signing in
again.","code":"token_invalidated"}}
```

The systray health probe maps that error to `BROKEN`, so the card said "Needs repair".
Meanwhile the same token kept working for real work: `containment.jsonl` records **270
successful containerised `cdx` runs on debian1/2/3 between Aug 13 and Aug 16 18:12**, all
on account `zync2`, rc 0.

One token refresh cleared it. On 2026-08-17 06:24:55Z a `codex app-server` run refreshed
the grant; the usage call then returned normally (66% of the weekly window used) and the
tray flipped to `ok` at 13:24:59 local, unattended.

## Why it never recovered on its own

Nothing on this workstation refreshes that file.

- `codex` refreshes only when the access token is near its own expiry. The Aug-13 token
  had ~10 days of `exp` left, so no local run had a reason to refresh — and the file's
  mtime confirms it: untouched from Aug 13 19:57 to Aug 17 13:24 local.
- Real codex work is offloaded. Each seat container **copies** the credential and discards
  any refresh by design (`modules/harness/seat/seat-entrypoint.sh`: "a refreshed token
  stays inside this container and dies with it"). So 270 runs produced zero convergence
  back to the host.
- The one owner-facing recovery path — the tray's Repair button — was dead. It spawned
  bare `codex login --device-auth`, which resolved to the `~/.claude/bin` PATH shim; the
  shim refuses with exit 78 ("seat authority is not installed") because
  `/usr/local/bin/overdeck-seat-scope-entry` does not exist on this laptop. Fixed on
  `wt/systray-repair-shim` (submitted, ticket `535e0ddc`).

## What invalidated the session — not attributable from this box

The refresh token survived: the Aug-13 value was still accepted on Aug 17, four days
later. So no successful refresh happened anywhere in that window, and the copies on the
buildboxes cannot be the cause — a competing refresh would have consumed it.

`token_invalidated` + a working refresh grant is the signature of the **session** behind
the access token being superseded or revoked server-side — a new sign-in, a
sign-out-everywhere, or an OpenAI-side session policy. The leading local candidate for
that superseding sign-in is the unlocked two-writer design below. Nothing on this machine
records which. The transition date is not recoverable either: the health cache is overwritten
every poll and no status history is kept.

**Owner action if attribution matters:** check the OpenAI account's active sessions/devices
around 2026-08-13.

## Two local writers refresh the same grant, with no lock

Raised by the owner and confirmed in the code. The codex grant for an account has **two
live local copies**, each refreshed by a different process, with nothing serialising them:

- `accounts/<slug>/CODEX_HOME/auth.json` — refreshed by any local `codex` run and by the
  systray health probe (`codex app-server --stdio`, every poll). Proven: the systray's own
  probe is what refreshed this grant on 2026-08-17.
- `accounts/<slug>/CLAUDEX_HOME/codex/auth.json` — `claudex` copies the grant here for
  `claude-code-proxy` to serve GPT models (`claudex.py::codex_proxy`), and the proxy
  refreshes that copy independently.

`reconcile_credentials` merges them by comparing access-token `exp` — newer wins, in
either direction — and there is **no `flock` anywhere in `claudex.py`**. The Claude grant
has a hardened rotator for exactly this reason (`claude_oauth.py:195`: flock sidecar,
re-read under lock, write-ahead journal). The codex grant has no equivalent.

The code already anticipates the failure: `reconcile_credentials` can raise
`CredentialConflictError("claudex: divergent Codex refresh tokens; re-authenticate
selected Systray AI account")` — which is precisely the state the card was reporting.

Two ways this produces the observed symptom:

1. **Lost update.** Both copies refresh close together. The server ends up with the newer
   session; the file that wins the last write holds the older access token, which now 401s.
2. **Refresh never reconciled back.** The proxy refreshes, then `claudex` dies before its
   `finally:` reconcile runs (session OOM-kills are documented on this machine). The native
   copy keeps a superseded access token indefinitely — matching the four-day plateau.

**Caveat, stated plainly:** one datum does not fit a naive rotation model. A competing
refresh should have consumed the native refresh token, yet that token still worked four
days later. That is only consistent if the provider keeps the previous refresh token usable
for a window — plausible, unproven. Definitive attribution is gone: today's refresh
overwrote the old access token, user-journal retention starts 2026-08-13 23:31 (after the
19:57 refresh), and the proxy's own copy was deleted 2026-08-17 01:27.

So: the mechanism is real and unguarded and is the best-supported explanation, but it is
not proven to be what happened on Aug 13.

## Hazards found while investigating

1. **Fan-out of one grant with no convergence back.** The same `auth.json` lives on the
   workstation, on debian1/2/3, and inside every seat container. Refresh-token rotation is
   real — the Aug-17 refresh rotated it — so the moment anything refreshes, every other
   copy holds a spent refresh token. Reuse of a spent refresh token is what revokes a whole
   grant chain.
   - This bit today: after the Aug-17 refresh the three boxes still held the spent token.
     Converged with the project's own converger; all three now hold the current digest
     (verified by remote sha256).
2. **Credential convergence fails silently.** `convergeCredentials` swallows a bad registry
   (`catch { return [] }`) and `remote-seat.mjs` logs "convergence skipped" and dispatches
   anyway. Right now the repo copy of `buildbox-registry.mjs` rejects the live registry
   (`unknown key "k3s"`, added 2026-08-13) while the installed copy at
   `~/.claude/lib/` accepts it — so which transport gets loaded decides whether credentials
   converge at all. A dispatch that proceeds with a stale credential is the exact path to a
   spent-token reuse.
3. **No status history.** Nothing records when an account flipped ok→broken, so the window
   in which to look for a cause is unrecoverable after the fact.

## Proposals

- **One lock for the codex grant (P1).** Every writer — systray probe, local `codex` run,
  `claudex`'s proxy copy — takes the same `flock` sidecar before refreshing or reconciling,
  and re-reads under the lock. The Claude rotator in `claude_oauth.py` is the pattern to
  copy; it exists because this exact class already cost an account.
- **Stop handing the proxy a second copy (P1).** Either point `claude-code-proxy` at the
  account's own `auth.json`, or hold the lock for the proxy's lifetime and reconcile under
  it. A copy that outlives an unclean exit is an invalidated grant nobody notices.
- **Surface `CredentialConflictError` as an owner-visible alarm (P2)** rather than an
  exception in a log — with a Repair button that works (fixed 2026-08-17).
- **Make convergence failure fatal to dispatch (P1).** If `convergeCredentials` cannot
  prove every target host holds the current digest, refuse the dispatch (exit 3, nothing
  dispatched) instead of logging and continuing. Cheap, removes the whole
  spent-token-reuse class.
- **Reconcile the registry validator (P1).** One of the repo/installed `buildbox-registry.mjs`
  copies rejects the live registry. Whichever is authoritative, both must agree, and a
  registry the transport cannot parse should be loud, not an empty host list.
- **Let the probe heal itself (P2).** When the usage call returns 401 `token_invalidated`,
  give `codex` time to complete its own refresh before `_terminate_process` group-kills the
  probe, then re-probe once before painting the card broken. Never POST a refresh from
  systray code — let the CLI own its grant. (Whether the current group-kill actually cuts a
  refresh mid-write is plausible but unproven; do not implement on the assumption alone.)
- **Keep the host copy the freshest (P2, only coupled).** A periodic refresh of the
  workstation's codex grant would keep the canonical copy alive — but it makes every box
  copy stale the moment it runs, so it is only safe if each refresh triggers a converger
  push. Do not ship the refresher without that coupling.
- **Record account status transitions (P3).** One append-only line per ok↔broken flip, so
  the next occurrence has a timestamp to investigate from.

## Evidence

- `containment.jsonl`: 364 zync2 rows since Aug 13, 270 `container`/`podman-rootless` rc 0.
- `auth.json` mtime Aug 13 19:57 → Aug 17 13:24 local; `last_refresh` 2026-08-13T12:57:19Z
  → 2026-08-17T06:24:55Z; refresh-token fingerprint `88c9f6b7…` → `dd765202…`.
- debian1/2/3 held `88c9f6b7…` (identical sha) until converged; now `dd765202…` on all three.
- `health_cache.json` zync2: `broken` at 13:20:28 → `ok` at 13:24:59, no owner action.