# ask-gpt attachment download progress design

audience: AI coding agents first.

## Outcome

`ask-gpt --download-attachments <id>` MUST show immediate, continuous, truthful progress without `--live`. Discover full recoverable manifest before first download. Show item count, current file, elapsed time, approximate ETA, per-file terminal state, final summary.

## Scope

Change only attachment recovery path:

- `modules/gptbridge/chat.py`: discovery plan + download lifecycle events.
- `modules/gptbridge/solwebd.py`: preserve plan, failures, terminal metadata through SSE.
- `modules/gptbridge/ask_gpt.py`: progress renderer + summary.
- gptbridge tests.

MUST NOT modify account registry work (`registry.py`, listing, bootstrap, account binding). Registry branch lands independently; merge latest main before implementation/landing as needed.

## Current failure

1. `collect_attachments()` discovers and downloads in one nested loop. CLI cannot print full manifest first.
2. `RunLog.on_event()` calls `render_live()` only when `--live`; ordinary download command stays silent.
3. `_fetch_blob()` returns complete blob after one blocking Marionette call. Byte-level progress unavailable.
4. Daemon result discards `collect_attachments()` failures; direct and daemon UX diverge.
5. One slow blob produces no heartbeat after `started` event.

## Architecture

### 1. Manifest-first collector

Introduce one internal work-item shape. Keep it inside `chat.py`; no new module.

```text
DownloadItem
  index: int              # 1-based across attempted items
  total: int
  turn: int               # 1-based assistant turn
  kind: image | file | button
  label: str              # human-visible preflight name
  source: internal opaque metadata
```

Split responsibilities:

```text
discover_attachments(m) -> (items, unavailable, invalid)
download_item(m, item, out_dir, used, ...) -> (Path, reused_existing: bool)
collect_attachments(...) -> (saved_attempts, failures)
```

- Discovery MUST execute `HARVEST_ALL` once.
- Discovery MUST include every recoverable image/link and latest-turn button download.
- Earlier-turn button-only entries MUST enter `unavailable` with reason; NEVER silently disappear.
- Malformed DOM metadata MUST enter `invalid` with reason; do NOT mislabel schema failure as unavailable.
- Emit full `download_plan` before first network/blob fetch.
- `total` counts attempted items only. Plan lists `unavailable` and `invalid` separately.
- Preserve current collision numbering, duplicate-content reuse, safe filenames, and failure isolation.

Deletion test: removing `DownloadItem`/discovery split forces manifest, index, kind, and metadata logic back into nested fetch loops. Boundary earns existence.

### 2. Download event contract

Extend existing TurnEvent stream; backward-compatible additive fields.

```json
{"type":"download_phase","state":"queued|opening|scanning","conversation_id":"..."}
{"type":"download_plan","attempt":"daemon-1","items":[{"index":1,"total":3,"turn":2,"kind":"file","file":"result.zip"}],"unavailable":[{"file":"Download old.bin","reason":"..."}],"invalid":[{"file":"turn-3-file-2","reason":"invalid metadata"}]}
{"type":"download","state":"started","attempt":"daemon-1","index":1,"total":3,"file":"result.zip","kind":"file","path":null}
{"type":"download","state":"saved","attempt":"daemon-1","index":1,"total":3,"file":"result.zip","kind":"file","path":"...","reused_existing":false}
{"type":"download","state":"failed","attempt":"daemon-1","index":2,"total":3,"file":"plot.png","kind":"image","error":"...","path":null}
```

Rules:

- Every planned item MUST emit exactly one `started` and one terminal `saved|failed` event.
- Known unrecoverable entries MUST appear once in `download_plan.unavailable`; malformed entries in `download_plan.invalid`; neither emits `started`.
- `saved` MUST set `reused_existing`; successful attempts, reused paths, and unique output files are distinct summary counts.
- Terminal `done.reply` shape:

```json
{"files":["/out/a.zip","/out/a.zip"],"failures":[{"file":"plot.png","error":"403"}],"unavailable":[{"file":"old.bin","reason":"older-turn button"}],"invalid":[{"file":"turn-3-file-2","reason":"invalid metadata"}],"planned":3,"reused":1}
```

  `files` is attempt-level successful paths in plan order; reused path MAY repeat. `planned` equals attempted-item count only. `reused` counts successful attempts returning existing identical content. `failures`, `unavailable`, `invalid` are structured lists, never formatted strings. CLI computes unique-file count from `files`. CLI owns one monotonic end-to-end elapsed clock; server MUST NOT report ambiguous `elapsed_s`.
- Exit policy: `0` when ≥1 attempt succeeds, even with partial failures; `1` when zero succeeds and any failed/unavailable/invalid entry exists; fatal errors `1`; interruption `130`; truly empty conversation follows existing tested no-artifact policy.
- Download failures remain item-local. Filesystem failures affecting output integrity (`ENOSPC`, permission, failed move/write) are fatal: stop remaining items and preserve already-written files.
- Daemon and direct paths MUST call one shared `download_attachments_on(m, conversation_id, ...)` seam that creates `TurnState`, emits phases/plan/items, and emits terminal `done|error`. `Engine.ask()` MUST NOT call `collect_attachments()` bare.
- Put stable `attempt` on every attempt-scoped phase, plan, item, `done`, and `error` event.
- SSE delivery MUST split coalescible answer deltas from lossless control events. Assign monotonic sequence numbers; store control events in unbounded deque, answer deltas in bounded/coalescing storage; `get()` returns lowest sequence. Coalesce only contiguous delta ranges between control events—deltas on opposite sides of control event MUST NEVER merge into one later replacement. Queue saturation MUST NEVER drop/reorder `meta`, download phase/plan/item, `response_done`, `done`, or `error`.
- Automatic daemon→direct fallback is allowed only before any `download_plan`; print attempt transition and reset phase. After plan, disconnect is terminal—NEVER start duplicate recovery pass.

### 3. CLI progress renderer

Add one stateful renderer owned by `_run_download`; do not overload generic answer `render_live()`.

```text
DownloadProgress(stderr, *, is_tty, clock, interval)
  on_event(event) -> None
  finish(files, failures, unavailable, invalid) -> None
  interrupt() -> None
```

Behavior:

1. Construct for every download invocation, independent of `--live`.
2. Print phase immediately and keep heartbeat active:
   - `download: starting`
   - daemon before seat acquire: `download: waiting for browser seat`
   - after seat acquire: `download: opening conversation <id>`
   - `download: scanning attachments`
3. On plan, print complete numbered manifest before first download:

```text
Will download 3 attachments; 1 unavailable, 1 invalid:
  1/3  [file]   result.zip
  2/3  [image]  turn-4-image-1
  3/3  [button] Download report.csv
  unavailable [button] Download old.bin — only latest-turn button downloads are recoverable
  invalid turn-3-file-2 — invalid metadata
```

4. TTY: maintain one in-place bar on stderr. Show current item, completed/total, elapsed, ETA. Redraw at most once/second. Use terminal width; truncate filename, NEVER field semantics.

```text
[████████░░░░░░░░] file 2/3 · completed 1/3 · result.zip · elapsed 00:18 · ETA ~00:36
```

5. ETA MUST use CLI renderer's one monotonic clock from command start. Render `ETA --:--` until ≥1 terminal attempt. Thereafter: average terminal-item duration × `(total - completed)`; this includes active item's full average duration and is intentionally approximate, shown with `~`. Failures count as terminal timing samples. NEVER claim byte ETA.
6. Heartbeat MUST cover every silent phase: daemon seat queue, conversation opening, scanning, active item. TTY redraws elapsed at most once/second. Non-TTY prints heartbeat every 30s:
   `download: scanning attachments — elapsed 01:30`
   or `download: file 1/3 · completed 0/3 · result.zip — elapsed 01:30, ETA --:--`.
7. Ticker thread MAY render while seat wait/Marionette blocks. It MUST NOT call browser functions. Protect renderer state/output with one lock; stop/join on done, error, interrupt, and all exits. Thread MUST be daemonized as final process-safety fallback, not relied on for cleanup.
8. `--json`: preserve machine JSON on stdout. Progress remains stderr. Never emit ANSI when stderr is not TTY.
9. `--live`: do not duplicate download lines. Route download events only through `DownloadProgress`; answer renderer ignores them during download-only invocation.

Deletion test: renderer hides TTY/non-TTY/timer complexity from collector and RunLog. Boundary earns existence.

### 4. Completion and interruption UX

Success summary:

```text
Recovered 3/3 attachments (2 files, 1 reused) to ./ask-gpt in 01:14.
```

Partial summary:

```text
Recovered 2/3 attachments (2 files) to ./ask-gpt in 01:14; 1 failed, 1 unavailable.
Failed:
  plot.png — request returned 403
```

Rules:

- Existing `saved: <path>` stdout lines remain for compatibility.
- Summary/progress go stderr.
- Summary MUST distinguish attempted, successful, reused, unique files, unavailable, invalid, failed.
- Output directory MUST be shown relative when caller supplied relative path; otherwise absolute. Never claim current directory if daemon wrote elsewhere.
- Ctrl-C direct path: stop ticker, terminate bar with newline, keep already-written files, print `download: interrupted after X/Y; completed files kept in <dir>`, exit 130, no traceback.
- Ctrl-C daemon path: closing client does NOT cancel worker. Print `download: stopped watching after X/Y; daemon may continue writing to <dir>`, exit 130, no claim of cancellation/partial finality.
- Empty manifest: print `Will download 0 attachments.` then preserve current no-artifact success/error contract as pinned by existing tests; do not invent downloads.

## Data flow

```text
CLI _run_download
  -> DownloadProgress starts: starting + heartbeat
  -> daemon emits queued before seat acquire OR direct emits opening
  -> shared download_attachments_on creates TurnState
  -> browser open + account validation (when registry work lands)
  -> opening + scanning events
  -> discover_attachments
  -> plan event (full manifest + unavailable + invalid)
  -> sequential item lifecycle events
  -> done.reply includes files/failures/unavailable/invalid/counts
  -> renderer stops + summary
  -> existing RunLog download record + saved stdout lines
```

## Error handling

- Conversation open/account mismatch before plan: heartbeat continues, then concise existing error; ticker stops.
- Discovery/session failure: fatal error, no fake manifest/count.
- Malformed single-item metadata: `invalid` preflight failure; continue valid items.
- Known older-turn button limitation: `unavailable`; preserve existing overall exit contract by treating unavailable-only recovery as unsuccessful when nothing recovered.
- Per-item fetch failure: terminal failed event + continue.
- Output filesystem failure (`ENOSPC`, permission, failed write/move): fatal; stop remaining items, preserve completed files.
- Daemon disconnect before `download_plan`: announce direct retry attempt and reset phase. Disconnect after plan: stop renderer and fail; daemon may still continue, NEVER duplicate recovery.
- Renderer exceptions MUST NOT abort downloads. Catch presentation failures, disable renderer, continue core recovery.
- Negative/non-monotonic clock deltas clamp to zero in display/tests.

## Testing

Deterministic clocks; no real sleep/browser/network.

1. Discovery emits full plan before any `_fetch_blob`/button click.
2. Plan indices/totals stable across turns and mixed kinds; current index and completed count remain distinct.
3. Old button downloads appear unavailable; malformed metadata appears invalid.
4. Every attempted item emits started + terminal state, including failures and reused existing content.
5. Daemon uses shared `download_attachments_on`, emits queued/plan/items/done, and terminal reply preserves failures/unavailable/invalid.
6. SSE control queue saturation preserves order and loses no plan/item/done/error events while answer deltas coalesce.
7. Default command (without `--live`) renders starting/queued/opening/scanning/manifest/progress/summary.
8. TTY bar: ANSI/in-place redraw, separate file/completed counts, elapsed, `ETA ~` after first completion, filename truncation.
9. Non-TTY: no ANSI; phase/item lines + bounded heartbeat during queue/open/scan/fetch.
10. `--json`: stdout remains one valid JSON document; progress only stderr.
11. Ctrl-C direct: ticker stopped/newline restored/exit 130/completed-files path. Ctrl-C daemon: says worker may continue, never claims cancellation.
12. Daemon→direct fallback before plan resets attempt; after plan refuses duplicate pass.
13. Skipped-only, invalid-only, partial success, fatal filesystem failure exit policies match pinned behavior.
14. Existing download collision/dedup/failure semantics remain green.
15. Full `python3 -m pytest modules/gptbridge/tests/ -q`; installed-entrypoint live proof on target conversation.

## Architecture decisions

- Keep discovery/work item in `chat.py`: single browser collector implementation; new module would be decorative.
- Use shared `download_attachments_on`: daemon/direct parity requires one TurnState/event owner.
- Create dedicated CLI renderer class: timer/TTY policy is deep and independent of browser internals.
- Use item-count progress, not fake byte progress: current blob boundary exposes no streaming bytes.
- Use one ticker thread, not async rewrite: smallest seam supporting both daemon and direct blocking paths.
- Split SSE delta/control storage: bounded answer coalescing MUST NOT destroy lifecycle control events.
- Do not add daemon cancellation: truthful Ctrl-C messaging wins over an unproven cancellation protocol.
- Emit additive events; preserve existing TurnEvent and SSE transport.