# CD structural review — 2026-08-16 (Fable interim + sol/medium merged)

Status: interim findings above and beyond `docs/plans/2026-08-16-deploy-idle-work.md`
(whose items — per-service restart gates, kanboard/gateway/reaper/pnpm gates, debounce,
restarted:/kept: evidence — are excluded here). To be merged with the sol/medium full-CD
review when it returns; items graduate into plan docs individually, never as one batch.

## Structural (highest leverage)

1. **One activation seam instead of five hand-rolled variants.** Web has the clean model
   (content-addressed releases, atomic `current`, verify-then-rollback). Collector,
   kanboard, gateway, botmaster each re-invent backup/restart/verify/rollback with
   different semantics (gateway: full rollback; kanboard: canary + empty-plugin fallback;
   botmaster: none). Every variant has produced its own bug class. Contract: a generic
   `activate-service` seam (render unit → cmp → restart-if-changed → identity probe →
   rollback) all installers call. Deletes hundreds of lines of near-duplicate bash;
   robustness properties become uniform instead of per-service folklore. Effort L.

2. **Backend services run FROM the mutable deploy clone.** Web gets immutable
   `releases/<sha>`; the collector/controller WorkingDirectory is the clone itself, which
   re-pins mid-deploy under the running process — the root of the "deploy builds from
   working tree" incident class, and why backend rollback doesn't actually exist (web
   release is restored; backend restarts on whatever the clone holds). Contract: extend
   the release-dir model to collector/controller. Effort L, pairs with item 1.

3. **Verification is liveness-heavy, identity-light.** Kanboard-answering-on-31339 is the
   canonical false-green: "something answered HTTP" is not "the intended revision is
   serving." Contract: every service exposes its git sha (env-injected at start); smoke
   asserts served sha == deployed sha. Cheap; converts restart-happened into
   right-code-serving; also guards against any future restart-gate bug leaving stale
   processes. Effort S–M. Highest error-rate impact per line of the structural items.

4. **The stamp is a single global high-water mark, and tripwires fail the deploy.**
   One flaky late step (live-report-refresh, sandbox parity) fails the whole deploy, the
   stamp doesn't advance, and the next deploy re-does every restart that already
   succeeded. Contracts: (a) per-component stamps or independently stamped phases so one
   weak step doesn't re-trigger the chain; (b) reclassify observability/tripwire steps
   (live-report refresh, sandbox drift, notify) as degraded-not-failed — each spurious
   `fail` today is a full rollback-and-redo cycle. Effort M.

   **Partially closed 2026-08-17:** revision `6c302b19` adds atomically published exact-SHA
   component receipts and uses them as independent dependency/service convergence bases. Installed
   proof removed only the global stamp, retried the exact revision, skipped dependency installation,
   restarted no core service, and preserved all five core PIDs. Reclassifying remaining
   observability/tripwire failures as degraded remains open.

5. **Serialization.** Smoke polls 100×3s serially; one lock spans build+install+smoke.
   Pipelining (build N+1 during smoke of N) is possible but deliberately deferred: get
   the gated/debounced numbers first. Recorded so the deferral is a decision.

## Smaller items, not yet filed anywhere

- `pnpm --filter web deploy --prod` staging block duplicated in bootstrap and normal
  build paths (`deploy-local.sh`) — one function.
- `sandbox-provision` leaves undeletable temp debris (permission-denied on its own `rm`)
  under agent temp dirs — slow cache litter.
- `shim-drift-check: HAZARD` fires on EVERY deploy while uncommitted shim edits sit in
  checkouts — a standing alarm is noise; make it edge-triggered or have it file a fire.
- Deploy failure JSON truncates smoke evidence to a 300-char tail; add the existing
  `deploy-status.smoke.log` path as a named field — today's triage would have been
  minutes faster.
- ~~Conductor no-op for already-merged branches~~ — VERIFIED 2026-08-16:
  `finish-branch.sh:1450` `_stage_a_remote_contains` short-circuits a ticket whose head
  is already an ancestor of origin/main before any gate runs. Today's two stale tickets
  are the live test; expected outcome: cheap recovery, no gate.

## Sol/medium full-CD review — merged verdict (2026-08-16)

Full report: `~/.local/state/overdeck/systray/runtime/logs/cdx/20260816-181947-299665.log`
(931 KB, file:line-cited). Converges with the structural items above on: immutable
backend artifacts (its #1 failure-rate item = item 2 above), identity-typed probes
(= item 3), per-component convergence receipts replacing the global stamp (= item 4),
tripwires fail-open, duplicated pnpm staging, smoke-log truncation, and the
`.releases` chmod walk (it upgraded that from "deprioritized" with a concrete
prune-the-immutable-subtree contract — accepted).

**Sol-unique findings adopted into the backlog (not previously filed anywhere):**
- ~~**Queue-ack ordering race** — deploy requests were deleted before the durable success
  commit (stamp), so a late-step failure silently lost the retry the queue exists to
  guarantee.~~ **CLOSED 2026-08-16:** `c510d7f9` moves acknowledgement after live-report,
  atomic full-SHA stamp publication, release-tree locking, and fail-open checkout sync.
  Production-code failure injection covers every terminal step; installed proof is recorded in
  `docs/plans/2026-08-16-deploy-queue-ack.md`.
- **Lock-holder process-tree containment** — the flock holder must bound and reap its
  entire child tree (the orphaned-pnpm fd-leak class, seen live as the wedged queue).
- ~~**Retry classification + intended-SHA recording** — record the full target sha before
  any failure and classify failures transient-retryable vs permanent.~~ **CLOSED 2026-08-17:**
  `538880f7` records schema-2 full target identity, persists bounded 60s/300s/900s transient
  backoff, stops permanent/exhausted targets, and moves failed covered requests out of the active
  systemd path queue into durable deferred storage. Installed proof recorded zero active requests,
  three deferred requests, no service relaunch, exact live collector state, and owner-facing `/ci`
  wording; see `docs/plans/2026-08-16-deploy-retry-classification.md`.
- **Port-ownership gate before activation** — generalize today's 31339 lesson: a
  declared port registry checked before any unit activation.
- **Docs-only classification before the 15 GiB disk-admission floor** — docs landings
  currently blocked by an admission check they don't need.
- **Land conductor as the only writer to origin/main** — a direct rebuke of today's
  three direct lands; holds once queue depth is fixed (see meta-item below).
- **Systray installer `FileExistsError`** on changed managed desktop/icon files —
  latent deploy-step failure waiting to fire.
- **Wall-time**: artifact-scoped remote builds on the buildboxes, web+gateway built
  concurrently, staged-release reuse after late failure, read-only preflights hoisted
  and parallelized, non-delivery work moved outside the lock.

**Sol's top-5 order (adopted as the graduation order into plan docs):**
1. ~~Queue acknowledgement after durable success + failure-injection tests per terminal step.~~
   **Closed at `c510d7f9`.**
2. ~~Typed readiness probes with monotonic deadlines replacing attempt-count smoke
   (subsumes the /health-must-be-200 quick win).~~ **Closed at `c8c66f66`; installed
   convergence was 79,560 ms under the 300,000 ms deadline with all four services named ready.**
3. ~~Full target identity + retry classification/backoff.~~ **Closed at `538880f7`; installed
   queue/backoff and `/ci` evidence recorded in the task plan.**
4. ~~Component convergence receipts + resumable activation.~~ **Closed at `6c302b19`; installed
   exact-target retry skipped dependency installation, restarted no core service, and preserved all
   five core PIDs.**
5. Immutable revision artifacts for non-web services with real per-component rollback
   (the largest; unlocks shorter locks, parallel builds, component-level recovery).

Central direction, agreed by both reviews: immutable candidates, component-scoped
commits, durable request coverage, typed readiness, explicit failure classes.

## The meta-item: land-queue depth is the real error-rate driver

The direct-land escape hatch was used three times today because the queue sat at 87–88.
When landing takes hours, work routes around the gate and the gate protects nothing.
Whatever drains the conductor faster — batch-gating independent branches, or gate-class
triage so docs/script-only branches don't pay the full suite — likely matters more for
error rates than any single deploy-script fix, because it keeps landings on the safe
path. Needs its own analysis (CI side, out of the deploy plan's scope).