# Collector socket front door — design

Slug: `collector-socket-frontdoor`
Date: 2026-08-16
Status: approved for planning

## Problem

Every deploy restarts `overdeck-collector.service`. The old process dies instantly; the new
bun process takes ~35–55s to reach `listening` (large TS graph cold-start under
`CPUQuota=25%` on a loaded box). During that window port 31338 refuses connections and the
requests board looks unreachable, several times a day. A client-side retry (od-requests,
120s deadline, landed 6a88a41c9) masks it for one CLI only; the web app and hooks still see
refusals, and every future client must reimplement the retry.

## Decision

Make the port structurally un-refusable instead of teaching every client to retry:
systemd holds 127.0.0.1:31338 forever via a socket unit; a tiny node relay (socket-activated,
inherits the listening fd) forwards raw TCP to the collector on a new loopback-internal bind
port. While the collector restarts, the relay holds accepted connections and retries the
backend connect until the collector binds. Unreachability becomes bounded latency.

Verified empirically 2026-08-16 on this box:
- node v24 serves HTTP over a systemd-inherited fd (`systemd-socket-activate` test passed).
- bun does NOT (accepts `listen({fd:3})`, logs listening, never serves) — this is why the
  collector itself cannot be socket-activated and a relay is required.
- `systemd-socket-proxyd` is absent on PATH and has no connect-retry, so it cannot bridge
  the startup window; socat is not installed. Hence a repo-owned ~60-line node relay.

Rejected alternatives:
- In-place hot reload / never restart: contradicts immutable-release doctrine; stateful
  init (SQLite stores, schedulers, incident dispatch) makes half-old processes untrustworthy.
- Blue-green overlap (retire old on new-ready): two live collectors double-fire the alerter
  (edge-triggered notifications), incident dispatch, and reconcilers; needs leader/standby
  machinery disproportionate to a loopback single-user service.
- Startup shrink alone (deploy-time bundle): probabilistic, window reopens under load.
  Kept as slice 2 to shorten the queued wait, not as the structural fix.

## Components and seams

### 1. Relay script — `collector/frontdoor/relay.mjs` (plain JS, node, zero deps)

```
node relay.mjs <backendHost:backendPort>
```

- Takes listener from `LISTEN_FDS` (fd 3). Exit non-zero with a clear message if absent —
  the relay is only ever started by its socket unit.
- Per accepted connection: connect to backend; on ECONNREFUSED/ETIMEDOUT retry every 500ms
  until a 120s per-connection deadline, then destroy the client socket.
- Once connected: bidirectional pipe, no buffering beyond node defaults, no protocol
  awareness (SSE and long polls pass through untouched).
- No config parsing — backend address is argv, pinned by the unit file.
- Logs one line per backend-unavailable episode (edge-triggered: first failed connect and
  recovery), never per retry.

### 2. systemd user units — source in `packaging/`, installed like `packaging/overdeck-collector.service`

- `overdeck-collector-front.socket`: `ListenStream=127.0.0.1:31338`, `Backlog=1024`,
  `WantedBy=sockets.target`. IPv4-only is deliberate: every in-repo client builds URLs
  with the literal `127.0.0.1`, never `localhost`.
- `overdeck-collector-front.service`: runs the relay with a node binary resolved at
  migration time (node ≥ v20; the verified v24 preferred), backend `127.0.0.1:31341` as
  argv. `Requires=` its socket only; no dependency on the collector service — the relay
  must survive collector restarts, and the socket must survive relay restarts. Pinned
  crash behavior: `Restart=on-failure`, `RestartSec=1`, `StartLimitIntervalSec=60`,
  `StartLimitBurst=30` — a crash-looping relay must not exhaust the user manager's
  default start limit while the socket queues connections.
- `packaging/overdeck-collector.service` gains no new deps; deploy restart list unchanged
  (front units are NOT in `deploy-local.sh`'s `restart:` line). Relay updates are not
  orphaned: `packaging/install.sh` try-restarts the front SERVICE (never the socket) when
  the installed relay or unit content differs from the running copy.

### 3. Collector bind seam — `collector/src/config.ts`

- New optional config key `bind_port` (zod: positive int, default = `port`).
- `startServer` binds `bind_port`; every other consumer of `config.port` (advertised URLs,
  PermissionQueue self-URL) is unchanged — self-traffic through the front door is correct
  and exercises the relay.
- Install/migration: this box's `~/.config/overdeck/config.toml` gets `bind_port = 31341`
  (port stays 31338). Absent `bind_port` = exact current behavior — tests and any other
  install are unaffected.

### 4. Install and migration

- `packaging/install.sh` keeps its existing semantics: it installs files, it enables and
  starts nothing new.
- Activation is a one-shot idempotent `packaging/frontdoor-migrate.sh`. The running
  collector owns 31338 until it stops, so "socket first" is impossible; the real order:
  precondition gates (effective `port == 31338`, loopback bind host — `tailnetBind`
  disqualifies, node resolvable) → root-scoped atomic `bind_port = 31341` TOML edit,
  verified by re-parsing with the collector's own loader → stop collector → start socket
  (31338 held) → enable relay service → start collector on 31341 → end-to-end verify.
  Any post-stop failure rolls back: front units off, config restored, collector
  restarted on 31338, failed step named. A failed precondition changes nothing — that is
  the guarantee that non-31338 installs are unaffected.

## Data flow

```
client (CLI / web / hooks) → 127.0.0.1:31338 (systemd-held socket)
  → relay.mjs (fd 3) → connect 127.0.0.1:31341 [retry ≤120s] → collector (bun)
```

Collector restart: 31338 keeps accepting; relay retries 31341; requests complete when the
collector binds. Relay restart: socket queues in kernel backlog; systemd re-spawns relay.

## Error handling

- Backend never comes up: per-connection 120s deadline; the client socket is destroyed
  within one retry interval past the deadline (bounded above AND below — a leaked
  never-closed connection is a bug, not a pass).
- Relay lifecycle hygiene: one backend attempt per client, retry timers cancelled on
  client close/success/deadline, failed backend sockets destroyed; half-closed peers get
  EOF forwarded (`end`, not `destroy`) so buffered response bytes flush.
- Relay crash loop: socket unit keeps the port; relay respawns under pinned rate limits
  (see §2); connections queue in the socket `Backlog=1024` while no relay runs.
- In-flight connections at collector death die as they do today; clients reconnect and are
  then held by the relay.

## Testing

- Relay test (in `collector/`, follows /od-testing doctrine): spawn the relay via
  `systemd-socket-activate` (present on the box) against a delayed-start backend; assert a
  request issued BEFORE the backend binds still succeeds, and one whose backend never binds
  fails only at the deadline (deadline overridable via env for a fast test).
- Config test: `bind_port` default equals `port`; explicit value wins (extend existing
  config schema tests in `collector/src`).
- Live proof (owner-obtainable evidence, named first per delivery doctrine):
  `systemctl --user restart overdeck-collector.service` then immediately
  `curl -s -o /dev/null -w '%{http_code} %{time_total}' http://127.0.0.1:31338/health`
  — auth precedes /health, so unauthenticated it returns `401` after a wait; ANY HTTP
  status (vs today's connection refused) is the end-to-end proof; plus one od-requests
  run during a real restart returning the board with no client-side retry log, plus an
  open web SSE stream reconnecting normally after the restart.

## Slice plan

0. **Slice 0 (instant, lands independently, owner-added 2026-08-16):** diff-gated collector
   restart in `packaging/deploy-local.sh`. Today the restart list is unconditional; the
   script already knows the previously deployed sha (`stamp_before` from `DEPLOY_STAMP`).
   Contract: drop `overdeck-collector.service` from the release restart command when
   `git diff --quiet <stamp_before> <deployment_sha> -- collector/ packaging/overdeck-collector.service packaging/install.sh`
   reports no change. Fail-open: missing/invalid stamp, git error, or any ambiguity →
   restart as today (correctness over uptime). This removes the daily downtime for the
   majority of deploys (web/docs-only) immediately, before the front door lands.
1. **Slice 1 (this spec's core):** relay + units + `bind_port`, installed live on this box,
   proof = the curl-during-restart evidence above.
2. **Slice 2 (follow-up, separate):** deploy-time `bun build` bundle for the collector to
   shrink the queued wait (~50s → seconds). Not part of this plan doc's mandatory scope.

## Architecture decisions

- Relay is repo-owned JS instead of socat/proxyd: neither tool is installed; proxyd cannot
  retry; a pinned ~60-line script with a unit test beats a new system package dependency.
- Relay deliberately protocol-blind (TCP pipe): deep enough (callers cannot tell it exists);
  an HTTP-aware proxy would add failure modes for zero benefit.
- Collector keeps a single instance always — no leader election, no standby mode (collapse
  of the blue-green alternative; see Rejected alternatives).
- `bind_port` defaulting to `port` collapses the "front door optional" seam into one config
  key; no second code path.
- sol/low review 2026-08-16 (REJECT → revised): adopted the ordered stop-collector-first
  migration with rollback, root-scoped atomic TOML edit with loader-verified result,
  precondition gating for non-31338/tailnet installs, 401-based live proof, half-close
  piping semantics, per-client cleanup invariants, LISTEN_PID/fd-count validation,
  Backlog + restart-rate pins, install-time node resolution, and relay-refresh via
  install.sh. Rejected none of the review's mandatory fixes.