# ask-gpt resume, conversation log, live output — design

audience: AI coding agents first. PLAN_SLUG: `ask-gpt-resume-live`.

## Outcome

`ask-gpt` can continue an existing ChatGPT conversation (`--resume <id>`), keeps a
local transcript of every conversation (prompts, reasoning summaries, answers,
artifact paths) in a canonical log plus a per-invocation duplicate, and can stream
the turn live to the terminal (`--live`) — thinking, answer, attachment upload and
artifact download progress — through both the daemon and the direct-browser path.

Owner request (2026-08-10): resume line printed when the server assigns a
conversation; `-r/--resume <id>` sends another message into that conversation;
`Log: ./ask-gpt/<id>.log` printed on completion; `--live` shows output as the web
UI renders it, including formatted thinking, upload progress, and end-of-run
download progress. Logs are REAL duplicate files, never symlinks — sandboxed
agents cannot follow a symlink out of their sandbox.

## Constraints

- Builds on the landed seat pool (slot-aware `Session`, `SeatPool`, engine
  per-conversation key locks in solwebd).
- Default invocation (no new flags, not `--json`) stays byte-identical on stdout.
  New human-facing meta lines go to stderr.
- The web UI exposes reasoning as collapsed summary blocks only; the log records
  what the UI shows — raw chain-of-thought does not exist client-side.
- No metered API; same chat-surface driving as today.

## Architecture

The turn engine (wherever the browser runs) owns ONE authoritative accumulating
turn state and emits TurnEvents as notifications of its changes. Consumers are
projections of that state — none of them reconstructs text on its own.

```
turn engine (chat.py, in CLI or daemon process)
  state: phase, conversation_id, thinking, answer, uploads, artifacts
   └─ TurnEvents ─┬─ CLI renderer (--live, stderr meta)
                  ├─ LogWriter (canonical + cwd duplicate; CLI process only)
                  └─ solwebd SSE (/v1/ask stream) ─→ CLI over HTTP
```

The CLI is the only log writer. On the daemon path the CLI ALWAYS requests the
stream (`--live` only changes rendering, never transport), so partial state for
failure logs exists on every path. The blocking `/v1/ask` shape remains for
other clients, unchanged plus `conversation_id` and `thinking` fields.

### Turn state machine

`attaching → submitted → named (meta) → streaming → response_done → collecting →
done | error`. Every event carries the phase. `done` is the ONLY successful
terminal and fires AFTER artifact collection; `error` is the only failure
terminal and carries the partial state (phase reached, thinking/answer so far,
error class: timeout | capped | chat | marionette | protocol).

## Components

### 1. Turn events — chat.py

```python
TurnEvent = dict  # json-serializable; every event: {"type": ..., "phase": ...}
#  {"type": "meta", "conversation_id": str, "url": str}    once, when /c/<uuid> first seen
#  {"type": "upload", "file": str, "state": "queued"|"uploading"|"attached"}
#  {"type": "thinking", "delta": str}
#  {"type": "answer", "delta": str}
#  {"type": "replace", "thinking": str, "answer": str}     full corrected state, not a delta
#  {"type": "response_done", "thinking": str, "answer": str}  assistant reply frozen
#  {"type": "download", "file": str, "state": "started"|"saved", "path": str|None}
#  {"type": "done", "reply": {text, thinking, images, files, url, conversation_id}}
#  {"type": "error", "class": str, "detail": str, "partial": {phase, thinking, answer,
#                                                             saved_artifacts}}

def send(m, before, timeout=900.0, on_event=None) -> None   # emits through response_done
# ask_on / the harvest-collect phase emits download events and the terminal done/error.
```

- The existing 2.5 s poll additionally harvests the in-progress turn's thinking
  and answer text from the engine's accumulated state; suffix extension → delta
  events, non-prefix DOM rewrite → one `replace` with full state.
- `meta` fires the first poll where the URL matches `chatgpt.com/c/<uuid>`. If
  the response completes and no `meta` ever fired, that is a protocol `error`
  (the CLI cannot name the log or the resume command) — never improvised around.
- `on_event=None` → behavior identical to today.
- Thinking harvest distinguishes three cases: block absent (instant effort /
  image turn → no events, `thinking=""`), block present and read, block present
  but unreadable → `thinking` set to a sentinel `[thinking present but not
  harvested]` and a warning on stderr — a selector regression must never
  masquerade as "no reasoning". Selectors match visible text/roles (MARK style),
  never volatile class names; collapsed blocks are expanded at most once per turn.
- Upload events poll composer chips (state change only). Download events wrap the
  existing collect phase (`started` on fetch/click, `saved` with final path).
- `Reply` gains `thinking: str` and `conversation_id: str`.

### 2. Resume — chat.py + ask_gpt.py

- `-r/--resume <id>` accepts the conversation UUID or a full
  `https://chatgpt.com/c/<uuid>` URL.
- Navigate to the conversation, `wait_ready`; composer present AND ≥1 existing
  turn required, else `ChatError("conversation <id> not found in this account")`.
- **History hydration**: before typing, the CLI reconciles the canonical log with
  the page — any web-visible turn missing from the log (foreign conversation,
  a turn from a crashed run, another tool's turn) is appended from the DOM,
  marked `(hydrated)`, thinking included where expandable. This is both what
  makes the log a real transcript and the crash-recovery mechanism: a turn that
  ran while no CLI survived to log it appears on the next resume.
- `--effort` and attachments work unchanged on resume.

### 3. Concurrency — two layers, one lock order

Lock order everywhere: **conversation before seat**. Both paths acquire the
conversation guard first, then seat/slot; never the reverse — the order is
stated here so a deadlock cannot be built out of two bounded waits.

- **Cross-process (CLI flock)** `~/.overdeck/gptbridge/logs/<id>.lock`:
  - resume: acquired before navigation, held through terminal event + log write.
  - new conversation: acquired the moment `meta` arrives — BEFORE the
    `resume:` line is printed to stderr, so the id is never exposed while
    unlocked — and held through terminal + log write. (A fresh id cannot be
    contended before `meta` by construction; exposing it is what creates the
    race, so exposure and lock are one step.)
  - Bounded wait, then: "another ask-gpt run is using conversation <id>".
- **In-daemon (mutex)**: solwebd holds a per-conversation-id mutex for the whole
  turn it runs. `/v1/ask` with `conversation` is therefore concurrency-safe for
  ANY authenticated caller, not only the flock-taking CLI; a second daemon
  request for the same id waits (bounded) on the mutex. The daemon is the
  correctness floor; the CLI flock additionally covers daemon-down direct runs.
- **Client disconnect**: the daemon finishes the turn it started (the quota is
  already spent; cancelling wastes it) while holding the mutex, so a reconnecting
  or competing client cannot start a second turn mid-generation. The outcome of
  a disconnected turn reaches the log via hydration on the next resume.

### 3b. Dispatch rate limiter (owner request 2026-08-11; DEMOTED to lowest
priority same day — good-to-have, not urgent, implement after everything else;
DONE 2026-08-11, `chat._rate_limit_dispatch`, commit 620494afc)

The account was rate-limited by eager agents dispatching back-to-back. A GLOBAL
minimum gap between prompt dispatches, across every seat and process:

- Before every submit (new turn, resume turn, daemon or direct path alike), wait
  until `last_dispatch + gap` where gap is uniform-random in [60, 75] seconds —
  jittered so N waiting agents don't stampede in lockstep when the window opens.
- Cross-process state: `~/.overdeck/gptbridge/dispatch.stamp` guarded by its own
  flock (`dispatch.lock`): take lock → read stamp → compute wait → sleep →
  re-check (another process may have dispatched while sleeping) → write stamp →
  release → submit. The stamp is written when the dispatch is COMMITTED (just
  before submit clicks), not when the reply lands — the limit is on send rate.
- Randomness: `random.uniform(60, 75)` per wait (allowed at runtime; the
  no-Date.now rule binds workflow scripts, not this module).
- Read-only browser work (sidebar listing, --download-attachments collection,
  navigation) is NOT gated — only prompt dispatches consume model quota.
- Waits surface as a stderr line (`rate-limit: waiting Ns before dispatch`) and,
  with events on, a `{"type": "ratelimit", "wait_s": N}` event.
- Env override `ASKGPT_DISPATCH_GAP=0` for tests only; fakes never sleep.

### 4. Conversation log — new module gptbridge/convlog.py

- Canonical `~/.overdeck/gptbridge/logs/<id>.log` — **authoritative**.
- Duplicate `<invocation-cwd>/ask-gpt/<id>.log` — real file, atomic replace
  (temp + rename), rewritten in full from the canonical after every turn.
- Invariant (exact): the canonical is always the complete record; the CURRENT
  invocation's cwd duplicate matches it at successful exit. Duplicates left in
  other directories by earlier invocations are snapshots — stale by design, not
  tracked, repaired whenever an invocation runs from that cwd again. A crash
  between canonical append and duplicate rewrite leaves a stale duplicate; the
  next run from any cwd repairs its own duplicate from the canonical.
- Partial-turn recovery: convlog records a turn as an `## turn N` block closed by
  an explicit end marker. On open, an unclosed trailing block is preserved and
  annotated `(interrupted)` before anything appends — never blind-appended into.
- Failure flush: terminal `error` writes the partial turn (phase, thinking,
  answer so far, saved artifacts, error class). Pre-ID failures (upload/login/
  navigation died before `meta`) cannot use `<id>.log`: they write
  `logs/failed-<stamp>.log` instead, and the `Log:` stderr line names it — the
  conversation-log promise starts when a conversation id exists.
- Format: human-readable markdown; header (id, url, created), per turn:
  timestamp, effort, prompt, `### thinking`, `### answer`, artifact paths,
  `(hydrated)` / `(interrupted)` annotations. Machine consumers use `--json`.
- Retention: none (small text; YAGNI).

### 4b. Attachment recovery — `--download-attachments <id-or-url>`

Owner request (2026-08-11): recover a dead session's artifacts, pull an old
conversation's files, or fetch via CLI instead of clicking through the browser.

- Read-only mode: no prompt is sent, mutually exclusive with a prompt argument
  and with `--resume` (it IS a form of resume; combining them is an error).
- Navigate + verify the conversation exactly like `--resume` (same not-found
  error), under the same per-conversation lock (collection clicks download
  buttons in the DOM — a concurrent turn on the same conversation would race).
- Collect from EVERY assistant turn, not only the last: generalize the existing
  last-turn harvest to walk all turns — images, downloadable file links, and
  download buttons, same recognition rules as today.
- Destination: `<invocation-cwd>/ask-gpt/` (flat, created if absent). Name
  collisions across turns get a numeric suffix before the extension
  (`report.pdf`, `report-2.pdf`); identical content already present (same size
  + bytes) is skipped, reported as such.
- Emits the standard `download` events (so `--live` shows progress); terminal
  `done` carries the saved paths. Failure of one artifact is reported per file
  and does not abort the rest; exit 1 only when NOTHING requested could be
  fetched or the conversation itself failed to load.
- stdout: today's `saved: <path>` lines (one per file); `--json` lists `files`.
  The conversation log gains a turn-less `### downloaded` record (timestamp +
  paths) — recovery is part of the conversation's history.
- Daemon path: `/v1/ask` accepts `{"conversation": <id>, "download": true}`
  (no prompt required in exactly this combination) — same seat rules, same SSE.

### 4c. Account-bound conversation registry — `--list`, `--search`, `--list-all`

Owner correction (2026-08-11): web sidebar MUST NOT remain listing source.
Registry source of truth: conversations initiated or explicitly imported by
`ask-gpt`. Canonical file:
`~/.overdeck/gptbridge/conversations.jsonl`.

- Record shape MUST contain `account`, `conversation_id`, `title`. MAY also carry
  `url` and timestamps. One JSON object per line. Updates append newer records;
  readers project latest record per `conversation_id`. Same id changing account
  is corruption: fail closed. Serialize reads/appends with
  `conversations.lock`; flush + `fsync` before release.
- New conversation: when `meta` first exposes id, browser-owning process MUST
  append account-bound provisional record immediately; reply failure/client
  disconnect MUST NOT lose an initiated conversation. After UI assigns title,
  append same-account title update. Resume MUST NOT replace existing actual title
  with provisional prompt text. Daemon and direct paths share registry flock.
- Account identity MUST come from authenticated browser session, not profile path,
  static config, prompt, or registry assumption. Resolve via ChatGPT auth-session
  response in page context. Empty/unparseable identity = hard error.
- Resume/download: registry lookup MUST succeed before navigation. CLI preflights
  lookup; daemon independently derives expected account from canonical registry
  and MUST NOT trust caller-supplied account as authority. Compare authenticated
  current account before opening target conversation. Mismatch MUST fail before
  conversation fetch with expected/current accounts named. Unknown id MUST fail
  closed because ownership cannot be verified.
- `--list [N]` and `--search`: read registry only. NEVER open browser, daemon, or
  remote endpoint. Preserve `<conversation-id>\t<title>` output and existing
  regex-then-glob semantics. Order by latest registry record, newest first.
- `--list-all`: read registry only; write real cwd copy
  `./ask-gpt/conversations.jsonl`; print only
  `Log: ./ask-gpt/conversations.jsonl`. NEVER scrape web.
- `--bootstrap-registry`: explicit migration-only command. Refuse when canonical
  registry already has records. Fetch full sidebar once, detect current account,
  append imported rows, then exit. This is sole listing-related remote fetch.
  Deploy procedure MUST run it once for current `chatgpt@alex.org.il` account.
- Old `threads.json`, 15-minute staleness, `--fresh`, and automatic listing
  refresh are superseded. Existing file MAY remain inert; no command reads it.
- Listing/bootstrap remain mutually exclusive with prompt, `--resume`, and
  `--download-attachments`.

### 5. CLI surface — ask_gpt.py

- stderr on `meta` (after the conversation lock is held):
  `resume: ask-gpt --resume <id>`
- stderr at exit (success and failure alike, whenever a log exists):
  `Log: ./ask-gpt/<id>.log` and `Log: ~/.overdeck/gptbridge/logs/<id>.log`.
- `--live` is a HUMAN DISPLAY stream, explicitly not pipe-safe: thinking dimmed/
  prefixed on stdout interleaved with answer text; `replace` redraws by marker +
  corrected tail; upload/download lines on stderr. Mechanical consumers use the
  default mode or `--json` — documented in `--help`. `--live --json` errors.
- `--json` gains `conversation_id`, `resume`, `thinking`, `log`, `log_canonical`.
- Daemon path: CLI always consumes SSE internally (buffering when `--live`
  absent) so failure logs and `--json` fields are identical on both paths. A
  daemon that predates the feature (no SSE, no `conversation_id`) → hard error
  telling the owner to restart solwebd; never a silently degraded log.

### 6. Daemon — solwebd.py

- `/v1/ask` gains `conversation` (uuid) and `stream` (bool).
- Per-conversation mutex as §3; unknown conversation → HTTP 404 with the
  ChatError message. Seat choice free (any seat can navigate); pinning not
  needed for one-shot turns.
- SSE wire schema (frozen): every event is `data: <TurnEvent JSON>\n\n` — the
  type lives INSIDE the JSON, `event:`/`id:`/`retry:` unused; keepalives are
  SSE comments (`: ka\n\n`); newlines inside text arrive JSON-escaped, so one
  event is always exactly one `data:` line. Terminal is one `done` or `error`
  event (same shapes as §1), then the stream closes. `done.reply` equals the
  blocking response body — one payload model, stated once.
- Event delivery is decoupled from the poll loop by a bounded per-request queue
  (browser never blocks on a slow client). Overflow policy: coalesce — drop
  queued deltas and enqueue one `replace` carrying current full state; terminal
  events are never dropped.
- The daemon never writes conversation logs.

## Known limitations (accepted)

- On resume, the first live poll can briefly echo the previous turn's answer
  until the new turn renders; a `replace` event corrects it, and the log
  projects `response_done`, so only `--live` viewers see the flicker.

## Error handling

- Unknown/foreign id → exit 1 with the not-found error (CLI), 404 (daemon).
- Conversation contention → bounded wait then the "another run" error.
- SSE stream break mid-turn: CLI distinguishes transport death (connection
  drops with no terminal event → "solwebd died/socket lost mid-turn" + partial
  log from events so far) from turn failure (`error` event → report its class).
- Missing `meta` by response completion → protocol error, partial log to
  `failed-<stamp>.log`.

## Testing

Fake-Marionette / session_factory pattern; no browser, network, or credentials.

- Events: delta/replace emission, meta-once, thinking absent-vs-unharvestable
  sentinel, upload/download state changes, response_done→collect→done ordering,
  done-after-collection, error carries partial state.
- convlog: create/append, duplicate atomic rewrite, stale-duplicate repair on
  next run from that cwd, unclosed-block annotation, pre-ID failure log,
  failure flush content.
- Resume: UUID/URL parsing, not-found, hydration of missing turns (foreign turn
  + simulated crash gap).
- Concurrency: meta→immediate-competing-resume loses (lock already held);
  daemon mutex serializes two same-id /v1/ask; disconnect mid-turn keeps mutex
  until terminal; lock order (conversation→seat) asserted; CLI kill →
  duplicate stale → repaired next run.
- SSE: wire framing (one data line per event), bounded-queue coalesce under a
  stalled consumer, terminal error event, keepalive comments.
- CLI: exact stderr lines, `--json` fields, `--live --json` rejected, default
  stdout byte-identical (golden), old-daemon hard error.

`python3 -m pytest modules/gptbridge/tests/ -q` and `modules/gptbridge/health.sh`
green.

## Sequencing — release phases (owner-ordered 2026-08-11)

Seat pool landed and live-installed. Three phases, each independently
releasable; PHASE R1 SHIPS FIRST AND ALONE — another session needs `--resume`
soon and must not wait for the rest.

- **R1 — `--resume` minimal, release ASAP**: resume navigation + verification,
  per-conversation flock, `resume:` stderr line on `meta`, canonical log append
  for the resumed turn (no hydration, no duplicate yet if it saves time —
  canonical only is acceptable for R1), `conversation_id` in `--json`, daemon
  `/v1/ask` `conversation` field + per-conversation mutex. NO `--live`, NO SSE,
  NO event seam beyond what `meta` needs. Commit as a standalone reviewable
  slice and REPORT IMMEDIATELY so it can be reviewed/installed/landed while
  work continues.
- **R2 — events + `--live` + full logs**: turn state machine, event types,
  thinking harvest, upload/download progress, convlog duplicates + hydration +
  partial-turn recovery, SSE, golden stdout test.
- **R3 — recovery + listing batch**: `--download-attachments` (§4b) and
  `--list`/`--search`/`--list-all` (§4c) together.

R2 and R3 land later, after R1 is released.

## Architecture Decisions

- Event seam over MutationObserver push: one liveness mechanism feeds all
  consumers. Accepted.
- Single authoritative turn state in the engine; consumers are projections —
  no per-consumer text reconciliation. Accepted (review round 2).
- `response_done` vs `done` split so artifact collection is inside the event
  protocol, not after its end. Accepted (review round 2).
- Daemon-side per-conversation mutex is the correctness floor; CLI flock covers
  daemon-down runs. `/v1/ask` safe for any authenticated caller. Accepted
  (review round 2).
- Disconnected turns run to completion; hydration heals the log gap. Accepted
  (review round 2).
- `/v1/chat` additionally takes the conversation's `ask-conv:<uuid>` mutex once
  known, unifying it with `/v1/ask`'s — one real conversation, one mutex.
- History hydration on first resume: required anyway for crash recovery, and it
  is what makes "full transcript" true for foreign conversations. Accepted.
- Duplicate-file invariant narrowed to canonical-authoritative + current-cwd
  current; stale snapshots repaired on reuse, never tracked. Accepted.
- `--live` is display-only by contract; pipe consumers use default/`--json`.
  Accepted.
- CLI as sole log writer; daemon stateless about logs. Accepted.
- Real duplicate over symlink (sandbox). Owner-decided.
- convlog.py module (deletion test passes); live renderer stays in ask_gpt.py
  (~40 lines of presentation). Accepted.
