audience: AI coding agents first.

# Reduce oversized collector state payload

status: ACTIVE
task IDs: #4
source request: Owner ordered an independent reduction of oversized collector state payload causing intermittent incident API failures.

## Outcome

Keep full observability available through bounded/detail seams while making `GET /api/collector/state` small and fast enough that concurrent incident requests do not cross Kanboard timeouts.

## Acceptance criteria

- Measure current payload by panel and identify exact oversized fields; never infer from aggregate size alone.
- Preserve owner-visible Factory run summaries and existing detail/artifact APIs. NEVER delete durable evidence to reduce response size.
- Add deterministic regression covering response bounds and required summary fields.
- Prove collector tests clean.
- Deploy through canonical runtime path, measure installed `/api/collector/state` bytes/latency, and run repeated incident API probes during state requests.
- Run remote live `/incidents` browser acceptance with no incident 503 response.
- Land and redeploy; prove landed installed behavior.

## Preserved WIP

- Shared checkout untouched.
- Worktree: `/home/user/Projects/overdeck/.worktrees/collector-state-payload`; branch `wt/collector-state-payload`; base `4f408e6c2`.
- Existing collector-state data and Factory evidence remain immutable; change serialization/query shape only.

## Constraints

- Independent seam: do not reopen incident selector or e2e-remote implementations.
- Observability MUST remain visible in UI; summaries cannot become empty placeholders.
- Detail/log/artifact data must be fetched through existing or narrowly-added bounded endpoints.
- Any collector change runs `bun test` in `collector/`; UI changes require `.claude/skills/od-ui-dev/SKILL.md` first and mandatory web gates.
- Browser/dev-server work runs through installed `e2e-remote`; never locally.
- No ignored warnings or security-gate output.

## Execution steps

1. Measure dominant `factory-runs` payload fields and trace producer/consumer contracts.
2. Select smallest contract-preserving bound; record before/after target.
3. Add failing regression, implement bounded summary serialization, and run focused tests.
4. Run collector and affected web gates.
5. Deploy candidate, measure live state and concurrent incident stability, then remote-browser verify.
6. Review, commit, update from main, land, deploy landed source, and repeat installed proof.

## Current receipt

- Task #4 assigned to `main` and executing in isolated worktree.
- Prior live measurement: total state 81,022,061 bytes; `factory-runs` 62,638,068; sessions 7,609,552; cluster:k3s 6,656,953.
- Candidate projects Factory runs to summary scalars + decisions in global state/SSE. Heavy phases, events, attempts, gates, diffs, and processes load only after one run expands through exact `GET /factory/runs/:adwId`.
- Projection against current live data: total 18,675,282 bytes; `factory-runs` 93,512 bytes across 124 runs. Remaining dominant independent panels: sessions 7,658,263; cluster:k3s 6,700,054; remote-jobs 2,349,352; agent-sessions 1,648,195.
- Verification: collector 694 pass/0 fail; focused web 30 pass; deck-ui 402 pass + typecheck; web typecheck 0 errors/warnings/hints; web build complete with no warning/error signal.
- First candidate deploy failed closed because deploy sync temporarily removed `~/.claude/buildbox-hosts.json`; exact backup `20260811T174037Z` restored mode 0600 and parsed with hosts `debian1,debian2,debian3`. Deploy also skipped uncommitted web source, so no candidate runtime change occurred.
- Independent review raised deterministic-byte-bound concerns. Measured per-run detail is one explicitly requested run rather than the prior 62.6 MB aggregate; Factory completed-run collection is capped at 200.
- Fresh security review blocks shipment: adapter still eagerly retains every trace array for up to 200 completed runs, so summary serialization reduces wire bytes but not collector memory/read load; expanded cards also poll and cache full immutable detail. No IDOR, decoding bypass, proxy escape, or SSE cross-run leak survived review.
- Acceptance delta implementation is WIP and uncommitted: Factory poll now emits summaries only; `loadFactoryRunDetail()` opens SQLite on demand; server injects loader, serializes at most 8 MiB, and returns 429 while detail work is in flight; web detail fetch accepts abort signals, polls running runs only, uses `gcTime: 0`, and cancels/removes cache on collapse.
- Current gates before latest worker edits: focused collector 32/32; focused web 30/30; collector 696 pass/0 fail/2 skipped; deck-ui 402/402 + typecheck; web build complete; web typecheck 0 errors/warnings/hints.
- Worker boundary WIP now exists: `collector/src/factory-detail-worker.ts`; server awaits typed worker result under semaphore; web proxy forwards abort signal. Event ingestion now rejects above 7 MiB before retaining further pages; summary request/decision text and counts are capped. Latest focused collector: 32/32. Web typecheck: 0 errors/warnings/hints, but wrapper still reports unexplained `pnpm install: 0 packages, 1 warnings`.
- Remaining ship blockers: add worker responsiveness/429 regression; prove worker runtime; bound non-event detail collections; expose summary truncation metadata; tolerate incompatible optional-table columns; explain pnpm warning; rerun full gates and reviews.

## Next executable action

Add server worker-concurrency regression and detail-budget tests, then finish non-event/schema bounds before full verification.