# ask-gpt resume, conversation log, live output — design audience: AI coding agents first. PLAN_SLUG: `ask-gpt-resume-live`. ## Outcome `ask-gpt` can continue an existing ChatGPT conversation (`--resume `), keeps a local transcript of every conversation (prompts, reasoning summaries, answers, artifact paths) in a canonical log plus a per-invocation duplicate, and can stream the turn live to the terminal (`--live`) — thinking, answer, attachment upload and artifact download progress — through both the daemon and the direct-browser path. Owner request (2026-08-10): resume line printed when the server assigns a conversation; `-r/--resume ` sends another message into that conversation; `Log: ./ask-gpt/.log` printed on completion; `--live` shows output as the web UI renders it, including formatted thinking, upload progress, and end-of-run download progress. Logs are REAL duplicate files, never symlinks — sandboxed agents cannot follow a symlink out of their sandbox. ## Constraints - Builds on the landed seat pool (slot-aware `Session`, `SeatPool`, engine per-conversation key locks in solwebd). - Default invocation (no new flags, not `--json`) stays byte-identical on stdout. New human-facing meta lines go to stderr. - The web UI exposes reasoning as collapsed summary blocks only; the log records what the UI shows — raw chain-of-thought does not exist client-side. - No metered API; same chat-surface driving as today. ## Architecture The turn engine (wherever the browser runs) owns ONE authoritative accumulating turn state and emits TurnEvents as notifications of its changes. Consumers are projections of that state — none of them reconstructs text on its own. ``` turn engine (chat.py, in CLI or daemon process) state: phase, conversation_id, thinking, answer, uploads, artifacts └─ TurnEvents ─┬─ CLI renderer (--live, stderr meta) ├─ LogWriter (canonical + cwd duplicate; CLI process only) └─ solwebd SSE (/v1/ask stream) ─→ CLI over HTTP ``` The CLI is the only log writer. On the daemon path the CLI ALWAYS requests the stream (`--live` only changes rendering, never transport), so partial state for failure logs exists on every path. The blocking `/v1/ask` shape remains for other clients, unchanged plus `conversation_id` and `thinking` fields. ### Turn state machine `attaching → submitted → named (meta) → streaming → response_done → collecting → done | error`. Every event carries the phase. `done` is the ONLY successful terminal and fires AFTER artifact collection; `error` is the only failure terminal and carries the partial state (phase reached, thinking/answer so far, error class: timeout | capped | chat | marionette | protocol). ## Components ### 1. Turn events — chat.py ```python TurnEvent = dict # json-serializable; every event: {"type": ..., "phase": ...} # {"type": "meta", "conversation_id": str, "url": str} once, when /c/ first seen # {"type": "upload", "file": str, "state": "queued"|"uploading"|"attached"} # {"type": "thinking", "delta": str} # {"type": "answer", "delta": str} # {"type": "replace", "thinking": str, "answer": str} full corrected state, not a delta # {"type": "response_done", "thinking": str, "answer": str} assistant reply frozen # {"type": "download", "file": str, "state": "started"|"saved", "path": str|None} # {"type": "done", "reply": {text, thinking, images, files, url, conversation_id}} # {"type": "error", "class": str, "detail": str, "partial": {phase, thinking, answer, # saved_artifacts}} def send(m, before, timeout=900.0, on_event=None) -> None # emits through response_done # ask_on / the harvest-collect phase emits download events and the terminal done/error. ``` - The existing 2.5 s poll additionally harvests the in-progress turn's thinking and answer text from the engine's accumulated state; suffix extension → delta events, non-prefix DOM rewrite → one `replace` with full state. - `meta` fires the first poll where the URL matches `chatgpt.com/c/`. If the response completes and no `meta` ever fired, that is a protocol `error` (the CLI cannot name the log or the resume command) — never improvised around. - `on_event=None` → behavior identical to today. - Thinking harvest distinguishes three cases: block absent (instant effort / image turn → no events, `thinking=""`), block present and read, block present but unreadable → `thinking` set to a sentinel `[thinking present but not harvested]` and a warning on stderr — a selector regression must never masquerade as "no reasoning". Selectors match visible text/roles (MARK style), never volatile class names; collapsed blocks are expanded at most once per turn. - Upload events poll composer chips (state change only). Download events wrap the existing collect phase (`started` on fetch/click, `saved` with final path). - `Reply` gains `thinking: str` and `conversation_id: str`. ### 2. Resume — chat.py + ask_gpt.py - `-r/--resume ` accepts the conversation UUID or a full `https://chatgpt.com/c/` URL. - Navigate to the conversation, `wait_ready`; composer present AND ≥1 existing turn required, else `ChatError("conversation not found in this account")`. - **History hydration**: before typing, the CLI reconciles the canonical log with the page — any web-visible turn missing from the log (foreign conversation, a turn from a crashed run, another tool's turn) is appended from the DOM, marked `(hydrated)`, thinking included where expandable. This is both what makes the log a real transcript and the crash-recovery mechanism: a turn that ran while no CLI survived to log it appears on the next resume. - `--effort` and attachments work unchanged on resume. ### 3. Concurrency — two layers, one lock order Lock order everywhere: **conversation before seat**. Both paths acquire the conversation guard first, then seat/slot; never the reverse — the order is stated here so a deadlock cannot be built out of two bounded waits. - **Cross-process (CLI flock)** `~/.overdeck/gptbridge/logs/.lock`: - resume: acquired before navigation, held through terminal event + log write. - new conversation: acquired the moment `meta` arrives — BEFORE the `resume:` line is printed to stderr, so the id is never exposed while unlocked — and held through terminal + log write. (A fresh id cannot be contended before `meta` by construction; exposing it is what creates the race, so exposure and lock are one step.) - Bounded wait, then: "another ask-gpt run is using conversation ". - **In-daemon (mutex)**: solwebd holds a per-conversation-id mutex for the whole turn it runs. `/v1/ask` with `conversation` is therefore concurrency-safe for ANY authenticated caller, not only the flock-taking CLI; a second daemon request for the same id waits (bounded) on the mutex. The daemon is the correctness floor; the CLI flock additionally covers daemon-down direct runs. - **Client disconnect**: the daemon finishes the turn it started (the quota is already spent; cancelling wastes it) while holding the mutex, so a reconnecting or competing client cannot start a second turn mid-generation. The outcome of a disconnected turn reaches the log via hydration on the next resume. ### 3b. Dispatch rate limiter (owner request 2026-08-11; DEMOTED to lowest priority same day — good-to-have, not urgent, implement after everything else; DONE 2026-08-11, `chat._rate_limit_dispatch`, commit 620494afc) The account was rate-limited by eager agents dispatching back-to-back. A GLOBAL minimum gap between prompt dispatches, across every seat and process: - Before every submit (new turn, resume turn, daemon or direct path alike), wait until `last_dispatch + gap` where gap is uniform-random in [60, 75] seconds — jittered so N waiting agents don't stampede in lockstep when the window opens. - Cross-process state: `~/.overdeck/gptbridge/dispatch.stamp` guarded by its own flock (`dispatch.lock`): take lock → read stamp → compute wait → sleep → re-check (another process may have dispatched while sleeping) → write stamp → release → submit. The stamp is written when the dispatch is COMMITTED (just before submit clicks), not when the reply lands — the limit is on send rate. - Randomness: `random.uniform(60, 75)` per wait (allowed at runtime; the no-Date.now rule binds workflow scripts, not this module). - Read-only browser work (sidebar listing, --download-attachments collection, navigation) is NOT gated — only prompt dispatches consume model quota. - Waits surface as a stderr line (`rate-limit: waiting Ns before dispatch`) and, with events on, a `{"type": "ratelimit", "wait_s": N}` event. - Env override `ASKGPT_DISPATCH_GAP=0` for tests only; fakes never sleep. ### 4. Conversation log — new module gptbridge/convlog.py - Canonical `~/.overdeck/gptbridge/logs/.log` — **authoritative**. - Duplicate `/ask-gpt/.log` — real file, atomic replace (temp + rename), rewritten in full from the canonical after every turn. - Invariant (exact): the canonical is always the complete record; the CURRENT invocation's cwd duplicate matches it at successful exit. Duplicates left in other directories by earlier invocations are snapshots — stale by design, not tracked, repaired whenever an invocation runs from that cwd again. A crash between canonical append and duplicate rewrite leaves a stale duplicate; the next run from any cwd repairs its own duplicate from the canonical. - Partial-turn recovery: convlog records a turn as an `## turn N` block closed by an explicit end marker. On open, an unclosed trailing block is preserved and annotated `(interrupted)` before anything appends — never blind-appended into. - Failure flush: terminal `error` writes the partial turn (phase, thinking, answer so far, saved artifacts, error class). Pre-ID failures (upload/login/ navigation died before `meta`) cannot use `.log`: they write `logs/failed-.log` instead, and the `Log:` stderr line names it — the conversation-log promise starts when a conversation id exists. - Format: human-readable markdown; header (id, url, created), per turn: timestamp, effort, prompt, `### thinking`, `### answer`, artifact paths, `(hydrated)` / `(interrupted)` annotations. Machine consumers use `--json`. - Retention: none (small text; YAGNI). ### 4b. Attachment recovery — `--download-attachments ` Owner request (2026-08-11): recover a dead session's artifacts, pull an old conversation's files, or fetch via CLI instead of clicking through the browser. - Read-only mode: no prompt is sent, mutually exclusive with a prompt argument and with `--resume` (it IS a form of resume; combining them is an error). - Navigate + verify the conversation exactly like `--resume` (same not-found error), under the same per-conversation lock (collection clicks download buttons in the DOM — a concurrent turn on the same conversation would race). - Collect from EVERY assistant turn, not only the last: generalize the existing last-turn harvest to walk all turns — images, downloadable file links, and download buttons, same recognition rules as today. - Destination: `/ask-gpt/` (flat, created if absent). Name collisions across turns get a numeric suffix before the extension (`report.pdf`, `report-2.pdf`); identical content already present (same size + bytes) is skipped, reported as such. - Emits the standard `download` events (so `--live` shows progress); terminal `done` carries the saved paths. Failure of one artifact is reported per file and does not abort the rest; exit 1 only when NOTHING requested could be fetched or the conversation itself failed to load. - stdout: today's `saved: ` lines (one per file); `--json` lists `files`. The conversation log gains a turn-less `### downloaded` record (timestamp + paths) — recovery is part of the conversation's history. - Daemon path: `/v1/ask` accepts `{"conversation": , "download": true}` (no prompt required in exactly this combination) — same seat rules, same SSE. ### 4c. Account-bound conversation registry — `--list`, `--search`, `--list-all` Owner correction (2026-08-11): web sidebar MUST NOT remain listing source. Registry source of truth: conversations initiated or explicitly imported by `ask-gpt`. Canonical file: `~/.overdeck/gptbridge/conversations.jsonl`. - Record shape MUST contain `account`, `conversation_id`, `title`. MAY also carry `url` and timestamps. One JSON object per line. Updates append newer records; readers project latest record per `conversation_id`. Same id changing account is corruption: fail closed. Serialize reads/appends with `conversations.lock`; flush + `fsync` before release. - New conversation: when `meta` first exposes id, browser-owning process MUST append account-bound provisional record immediately; reply failure/client disconnect MUST NOT lose an initiated conversation. After UI assigns title, append same-account title update. Resume MUST NOT replace existing actual title with provisional prompt text. Daemon and direct paths share registry flock. - Account identity MUST come from authenticated browser session, not profile path, static config, prompt, or registry assumption. Resolve via ChatGPT auth-session response in page context. Empty/unparseable identity = hard error. - Resume/download: registry lookup MUST succeed before navigation. CLI preflights lookup; daemon independently derives expected account from canonical registry and MUST NOT trust caller-supplied account as authority. Compare authenticated current account before opening target conversation. Mismatch MUST fail before conversation fetch with expected/current accounts named. Unknown id MUST fail closed because ownership cannot be verified. - `--list [N]` and `--search`: read registry only. NEVER open browser, daemon, or remote endpoint. Preserve `\t` output and existing regex-then-glob semantics. Order by latest registry record, newest first. - `--list-all`: read registry only; write real cwd copy `./ask-gpt/conversations.jsonl`; print only `Log: ./ask-gpt/conversations.jsonl`. NEVER scrape web. - `--bootstrap-registry`: explicit migration-only command. Refuse when canonical registry already has records. Fetch full sidebar once, detect current account, append imported rows, then exit. This is sole listing-related remote fetch. Deploy procedure MUST run it once for current `chatgpt@alex.org.il` account. - Old `threads.json`, 15-minute staleness, `--fresh`, and automatic listing refresh are superseded. Existing file MAY remain inert; no command reads it. - Listing/bootstrap remain mutually exclusive with prompt, `--resume`, and `--download-attachments`. ### 5. CLI surface — ask_gpt.py - stderr on `meta` (after the conversation lock is held): `resume: ask-gpt --resume <id>` - stderr at exit (success and failure alike, whenever a log exists): `Log: ./ask-gpt/<id>.log` and `Log: ~/.overdeck/gptbridge/logs/<id>.log`. - `--live` is a HUMAN DISPLAY stream, explicitly not pipe-safe: thinking dimmed/ prefixed on stdout interleaved with answer text; `replace` redraws by marker + corrected tail; upload/download lines on stderr. Mechanical consumers use the default mode or `--json` — documented in `--help`. `--live --json` errors. - `--json` gains `conversation_id`, `resume`, `thinking`, `log`, `log_canonical`. - Daemon path: CLI always consumes SSE internally (buffering when `--live` absent) so failure logs and `--json` fields are identical on both paths. A daemon that predates the feature (no SSE, no `conversation_id`) → hard error telling the owner to restart solwebd; never a silently degraded log. ### 6. Daemon — solwebd.py - `/v1/ask` gains `conversation` (uuid) and `stream` (bool). - Per-conversation mutex as §3; unknown conversation → HTTP 404 with the ChatError message. Seat choice free (any seat can navigate); pinning not needed for one-shot turns. - SSE wire schema (frozen): every event is `data: <TurnEvent JSON>\n\n` — the type lives INSIDE the JSON, `event:`/`id:`/`retry:` unused; keepalives are SSE comments (`: ka\n\n`); newlines inside text arrive JSON-escaped, so one event is always exactly one `data:` line. Terminal is one `done` or `error` event (same shapes as §1), then the stream closes. `done.reply` equals the blocking response body — one payload model, stated once. - Event delivery is decoupled from the poll loop by a bounded per-request queue (browser never blocks on a slow client). Overflow policy: coalesce — drop queued deltas and enqueue one `replace` carrying current full state; terminal events are never dropped. - The daemon never writes conversation logs. ## Known limitations (accepted) - On resume, the first live poll can briefly echo the previous turn's answer until the new turn renders; a `replace` event corrects it, and the log projects `response_done`, so only `--live` viewers see the flicker. ## Error handling - Unknown/foreign id → exit 1 with the not-found error (CLI), 404 (daemon). - Conversation contention → bounded wait then the "another run" error. - SSE stream break mid-turn: CLI distinguishes transport death (connection drops with no terminal event → "solwebd died/socket lost mid-turn" + partial log from events so far) from turn failure (`error` event → report its class). - Missing `meta` by response completion → protocol error, partial log to `failed-<stamp>.log`. ## Testing Fake-Marionette / session_factory pattern; no browser, network, or credentials. - Events: delta/replace emission, meta-once, thinking absent-vs-unharvestable sentinel, upload/download state changes, response_done→collect→done ordering, done-after-collection, error carries partial state. - convlog: create/append, duplicate atomic rewrite, stale-duplicate repair on next run from that cwd, unclosed-block annotation, pre-ID failure log, failure flush content. - Resume: UUID/URL parsing, not-found, hydration of missing turns (foreign turn + simulated crash gap). - Concurrency: meta→immediate-competing-resume loses (lock already held); daemon mutex serializes two same-id /v1/ask; disconnect mid-turn keeps mutex until terminal; lock order (conversation→seat) asserted; CLI kill → duplicate stale → repaired next run. - SSE: wire framing (one data line per event), bounded-queue coalesce under a stalled consumer, terminal error event, keepalive comments. - CLI: exact stderr lines, `--json` fields, `--live --json` rejected, default stdout byte-identical (golden), old-daemon hard error. `python3 -m pytest modules/gptbridge/tests/ -q` and `modules/gptbridge/health.sh` green. ## Sequencing — release phases (owner-ordered 2026-08-11) Seat pool landed and live-installed. Three phases, each independently releasable; PHASE R1 SHIPS FIRST AND ALONE — another session needs `--resume` soon and must not wait for the rest. - **R1 — `--resume` minimal, release ASAP**: resume navigation + verification, per-conversation flock, `resume:` stderr line on `meta`, canonical log append for the resumed turn (no hydration, no duplicate yet if it saves time — canonical only is acceptable for R1), `conversation_id` in `--json`, daemon `/v1/ask` `conversation` field + per-conversation mutex. NO `--live`, NO SSE, NO event seam beyond what `meta` needs. Commit as a standalone reviewable slice and REPORT IMMEDIATELY so it can be reviewed/installed/landed while work continues. - **R2 — events + `--live` + full logs**: turn state machine, event types, thinking harvest, upload/download progress, convlog duplicates + hydration + partial-turn recovery, SSE, golden stdout test. - **R3 — recovery + listing batch**: `--download-attachments` (§4b) and `--list`/`--search`/`--list-all` (§4c) together. R2 and R3 land later, after R1 is released. ## Architecture Decisions - Event seam over MutationObserver push: one liveness mechanism feeds all consumers. Accepted. - Single authoritative turn state in the engine; consumers are projections — no per-consumer text reconciliation. Accepted (review round 2). - `response_done` vs `done` split so artifact collection is inside the event protocol, not after its end. Accepted (review round 2). - Daemon-side per-conversation mutex is the correctness floor; CLI flock covers daemon-down runs. `/v1/ask` safe for any authenticated caller. Accepted (review round 2). - Disconnected turns run to completion; hydration heals the log gap. Accepted (review round 2). - `/v1/chat` additionally takes the conversation's `ask-conv:<uuid>` mutex once known, unifying it with `/v1/ask`'s — one real conversation, one mutex. - History hydration on first resume: required anyway for crash recovery, and it is what makes "full transcript" true for foreign conversations. Accepted. - Duplicate-file invariant narrowed to canonical-authoritative + current-cwd current; stale snapshots repaired on reuse, never tracked. Accepted. - `--live` is display-only by contract; pipe consumers use default/`--json`. Accepted. - CLI as sole log writer; daemon stateless about logs. Accepted. - Real duplicate over symlink (sandbox). Owner-decided. - convlog.py module (deletion test passes); live renderer stays in ask_gpt.py (~40 lines of presentation). Accepted.