# Fix collector memory leak

status: DONE
task IDs: #5
source request: Owner requested root-cause fix for `overdeck-collector` memory leak after measured growth of 156 MiB in five seconds.

audience: AI coding agents first.

## Outcome

Collector memory remains bounded under live polling/streaming workload. Fix runs in installed runtime, lands, deploys, and remains fixed from landed source.

## Acceptance criteria

1. Reproduce growth deterministically. Capture operation/input causing retained memory.
2. Trace allocation ownership to source. NEVER patch with restart, cap, timer, or arbitrary cache clear.
3. Add failing regression proving repeated operation does not retain unbounded objects/buffers/listeners/results.
4. Implement one root-cause fix. Preserve collector behavior and observability.
5. Run collector focused tests and required compile/type checks without warnings.
6. Install candidate before landing; invoke real installed service; prove bounded memory across equivalent workload and owner-visible collector health.
7. Preserve atomic rollback and unrelated deploy-clone/WIP state.
8. Commit, reconcile current `origin/main`, land, deploy landed source, repeat installed memory proof.

## Preserved WIP

- Worktree: `/home/user/Projects/overdeck/.worktrees/collector-memory-leak`.
- Shared main checkout and dirty deploy clones remain untouched.
- Runtime restart was containment only; no source claim derives from it.

## Constraints

- Read `collector/` source and tests before claims.
- Use existing collector test/runtime patterns.
- No UI, LSP, session-lifecycle, disk-cleanup, or unrelated refactor changes.
- Runtime restoration before Git ceremony.

## Execution steps

1. Trace collector startup, poll/stream loops, caches, listeners, subprocess capture, and recent changes.
2. Reproduce with bounded instrumentation and heap/RSS samples.
3. State one root-cause hypothesis; add failing deterministic regression.
4. Implement smallest source fix; run focused verification.
5. Install candidate with rollback; prove installed service memory plateau and health.
6. Review diff; commit; reconcile; land; deploy; re-prove.

## Current receipt

- Live PID grew 156 MiB in five seconds, from ~255 MiB to 411 MiB; restarted process later peaked at 1.7 GiB.
- Exact `overdeck-collector.service` restart reduced memory to ~1.2 MiB and restored availability.
- `/state` is 21.8 MiB. Four panels retain almost all payload: `sessions` 9.0 MiB / 7,639 records, Kubernetes `cluster` 7.9 MiB / 7,500 unplaced sessions, `remote-jobs` 2.8 MiB / 10,457 jobs, `agent-sessions` 1.8 MiB.
- Historical terminal records dominate: 7,282 `FINISHED` sessions and 10,225 succeeded/failed remote jobs. Both 60-second session/Kubernetes polls reread and reclassify the complete ledger; offload repeatedly fetches and maps complete controller job history.
- A 30-second sample ranged 244–334 MiB RSS and ended at 262 MiB, proving large transient allocation cycles rather than monotonic map/listener retention during that window.
- Evidence-backed root cause: operational snapshot adapters ingested complete terminal history. Repeated parse/classify/project/validate/serialize cycles over growing history drove rapid RSS spikes and unbounded future memory cost.
- First installed candidate disproved output-only filtering: `/state` remained 20.9–23.0 MiB and process startup still peaked at 1.7 GiB because both session adapters classified all 7,639 ledger files before trimming output. No memory-fix claim derives from that candidate.
- Corrected source candidate bounds terminal ledger entries before classification, caps terminal session/job output at newest 100, and excludes `FINISHED` sessions from Kubernetes placement.
- Recent runtime samples are invalid as candidate proof. `packaging/deploy-local.sh` installed from `${OVERDECK_DEPLOY_DIR:-$HOME/.local/share/overdeck/deploy}`, not this worktree. Installed source had old timestamps, lacked all three new bounds, and still exposed 8,041 sessions plus 11,135 remote jobs. The changed source has not run in the installed service yet.
- Candidate source identity is now verified: temporary units point at this worktree and package-local dependency links reuse the deployed immutable dependency tree. Initial direct install failed closed because the isolated worktree had no dependencies; units were immediately rolled back, then the candidate was started successfully after package-local links were supplied.
- Candidate `/state` through the authenticated web proxy fell from about 23 MiB to 3.09 MiB: sessions 350 / 441 KiB, remote jobs 347 / 93 KiB, Kubernetes unplaced sessions 0, agent sessions 1.95 MiB / 54 groups.
- Stable candidate PID sampled for two minutes across multiple poll cycles. RSS peaked at 1.76 GiB during startup, then settled between 557–717 MiB and ended near 584 MiB. The previous monotonic 156 MiB-per-five-seconds growth did not recur, but startup allocation and the independent `agent-sessions` panel remain large. Service-level swap reporting from the sample was inconsistent with the configured 1 GiB cap and is not accepted as proof.
- Independent review closed with no findings. Its initial concern was withdrawn after checking the authoritative classifier: `finishedAt` records are excluded from live evidence and always classified `FINISHED` (`agent-session-reader.mjs:555,623-625`).
- Existing regression covers unfinished persisted records and bounds persisted terminal records, matching the classifier's authority contract.
- Focused regressions pass: collector 57/57; controller 4/4. Full suites pass: collector 719/719; controller 236/236. These receipts remain source proof until landed deployment is re-proven.

- Fix committed, reconciled with current main, directly landed as `18c8b664`, and deployed from the normal deploy clone. Deploy completed with existing shim-drift hazards in unrelated worktrees; those were not modified and do not affect collector source identity.
- Landed runtime proof used stable PID `747812` from `/home/user/.local/share/overdeck/deploy/collector` across six 20-second samples. Authenticated `/state` remained 3.07 MiB, sessions remained 349, remote jobs 347–348, and Kubernetes unplaced sessions remained 0.
- Final service memory was 155 MiB current, 181 MiB process RSS, 0 swap, with a 1.80 GiB startup peak. The original sustained 156 MiB growth in five seconds is fixed. High startup allocation remains observable but is transient and no longer drives swap after startup.

## Next executable action

None. Root-cause fix is installed, landed, deployed, and proven in the normal runtime.