# Personal build/CI/offload system review

## 1. Overall build-process recommendations

### Findings

Current design has strong mechanisms—remote-only intent, immutable snapshot refs, detached systemd jobs, reconnectable logs, adaptive admission—but no single owner for system state. Policy is distributed across JSON flags, per-process caches, flock files, GitHub labels, two runner services, timers, and operator memory. Today’s incident is therefore expected, not exceptional.

Highest-risk defect is artifact publication. `syncPull()` rsyncs the whole remote mirror back into the live checkout (`remote-build.mjs:331-333`), not a declared artifact set. A developer can edit locally while a remote job runs; newer remote mtimes can overwrite unrelated source. Epoch equality proves only that job and remote mirror agree, not that local source still matches submitted snapshot.

Other high-frequency correctness gaps:

- Invalid/missing `build-remote.json` becomes `disabled`; `local-gate` then authorizes local execution (`remote-build.mjs:22-32`, `local-gate:202-210`). Remote-only policy fails open on control-plane corruption.
- Shared mutable mirrors force cancellation/fencing on every newer submission (`remote-build.mjs:156-159, 293-320`). Retries mask structural interference; they do not fix it.
- Environment readiness checks only runner hash and SSH. `remoteDoctor()` omits `tsc`, test runners, repo-specific commands, disk, writable paths, and systemd capability (`remote-build.mjs:106-124, 459-484`). Status 127 is treated as an ordinary build result.
- Admission and remote-job ledgers are unlocked read-modify-write files (`local-gate:151-165`; `remote-build.mjs:336-347, 424-425`). Concurrent writers can lose entries; PID-only identity permits PID-reuse errors.
- CI failover scripts convert API/action failures into successful timer exits and append-only logs (`ci-fallback.sh:72-105`; `platform-fallback.sh:90-104`). Green timer status does not mean policy reconciled.

### Prioritized recommendations

1. **P0—make publication safe:** remote jobs publish only a declared artifact manifest into a job-specific staging directory. Validate submitted snapshot digest and local checkout generation before atomic promotion. Never rsync a remote working tree into a live source tree.
2. **P0—introduce one desired-state reconciler:** one controller owns offload mode, host maintenance, capacity reservations, CI eligibility, runner services, and recovery. Persist desired/observed state plus monotonic revision in SQLite WAL or an equivalent transactional store. All CLIs and Overdeck call this controller; direct flag/label/unit mutation is denied.
3. **P0—fail closed on policy/config corruption:** schema-validate both gate configs; preserve invalid files for diagnosis; emit a typed incident; never rewrite defaults or silently authorize local work.
4. **P1—replace shared working mirrors with per-job workspaces:** retain one bare object cache per repo, then create an immutable worktree/snapshot directory per job with a job-specific overlay and output directory. TTL-GC completed jobs. Epochs become publication leases, not mutual interference control.
5. **P1—replace distributed polling with one admission scheduler:** durable FIFO/priority queue, atomic capacity reservation, PID+starttime/systemd-unit ownership, lease expiry, and queryable state. Remove `lsof` from correctness decisions.
6. **P1—make execution environments declarative:** pin a host image/toolchain manifest; verify command capabilities at admission; quarantine a host after repeated 126/127 capability failures. Manual package installation is not parity management.
7. **P1—emit structured events and metrics:** every state transition carries job/repo/host/snapshot/attempt/stage/reason/rc/duration. Logs remain diagnostic payload, not monitoring state.

## 2. Load-transfer robustness

### Findings

Manual drain/restore must become one crash-resumable state machine. Required states: `available`, `draining`, `maintenance`, `restoring`, `degraded`. A transition owns timers, runner eligibility, local-fallback lease, remote jobs, and CI jobs. `draining → maintenance` is legal only after remote jobs and CI are idle; `restoring → available` requires toolchain, storage, runner, and offload probes. Reconciliation resumes after process or laptop restart.

Per-job remote workspaces are mandatory. Current sequence allocates epochs locally without locking (`remote-build.mjs:146-155`), fences before materialization, and stamps one shared checkout. Two processes can obtain/order epochs independently, arrive remotely out of order, regress `.rb-epoch`, overwrite the shared overlay archive, or kill valid older work. `max_remote_jobs` is also check-then-start and can oversubscribe (`remote-build.mjs:350-366`).

Health caching is not a scheduler. A 30-second cached positive can admit into a newly broken host; a cached negative suppresses recovery. Use controller-owned circuit-breaker states (`closed/open/half-open`), active capability probes, atomic reservations, and reason-specific cooldowns.

Portable runner identity is the wrong primitive. `platform-fallback.sh` duplicates one credential and reaps listeners because `KillMode=process` leaves sessions behind (`platform-fallback.sh:54-77`). Give each host a distinct runner identity; move eligibility through labels/runner groups. Desired state then reconciles labels and service state without session transfer.

Remote-only fail-closed behavior is correct; the one-hour synchronous wait followed by bare exit 97 is not. Return a typed `infra_unavailable` result after a short configured SLO while the durable job remains inspectable/retryable. Never fall back locally without an explicit, expiring maintenance lease. Preserve a documented nonzero CLI code for automation, but do not make the code the only diagnostic.

`BUILD_SLOT_TIMEOUT` requeue is unsafe: an admitted child can legitimately return 75, which `local-gate` misclassifies as queue timeout and reruns (`local-gate:245-279`). Communicate admission timeout through a side-channel/status record, not the child’s exit namespace.

### Hardening order

1. Per-job workspace + artifact-only atomic publication with snapshot CAS.
2. Desired-state reconciler for drain/restore and CI eligibility.
3. Atomic queue/capacity reservations and server-issued monotonic job revisions.
4. Distinct runner identities; remove listener-kill recovery from normal operation.
5. Declarative toolchain/capability admission and repeated-127 circuit breaker.
6. Typed job state/results; bounded retries by failure class.

## 3. Loud failures

### Strategy

Use one incident pipeline: structured producer events → durable incident reducer/deduper → Prometheus metrics → Overdeck Items → independent human notifier. Alert on violated invariants/SLOs, not raw log lines. Every alert has incident key, first/last seen, count, affected jobs, remediation, cooldown, and auto-resolution condition.

| Class | Automatic response | Human signal |
|---|---|---|
| Single SSH/mux interruption, transient push/pull transport failure | Reconnect/reset mux; one bounded retry; record event | None unless retry budget exhausts |
| Dead queue lease proven by controller ownership + PID starttime | Reconcile/reclaim lease | Info item; page only if recurrence threshold trips |
| Repeated 126/127 for same capability, capability-manifest drift | Quarantine host; stop new admissions | Immediate actionable notification |
| Pending queue while remote capacity is idle; oldest-age SLO breached | Reconcile scheduler once | Actionable notification with queue/job traces |
| Local fallback lease expired, desired/actual runner mismatch, drain/restore stuck | Reconciler retries boundedly | Immediate actionable notification |
| Snapshot/epoch/artifact CAS mismatch or possible source overwrite | Stop publication; preserve staging | Immediate high-severity page; never auto-promote |
| CI test failure | No infrastructure retry unless classified transient | Inbox item; rerun available |
| Collector/Prometheus/controller unavailable | Restart through systemd watchdog | Independent notifier after restart budget exhausts |

Remaining silent surfaces:

- Config parse failures and broad empty `catch` blocks.
- Remote status 127, invalid job metadata, epoch-read failures, and pull failures surfaced only to caller/log.
- Pull-back has no stage/progress/byte telemetry.
- `ci-fallback.sh` logs `SKIP` and exits 0 on GitHub/API/unit failures; `platform-fallback.sh` suppresses SSH stderr and logs generic failures.
- Platform runner can remain unavailable forever when SSH works but service start repeatedly fails; connectivity is treated as service health.
- Deduped jobs return success without result provenance (`remote-build.mjs:421-425`; `local-gate:179-182`).
- Overdeck adapters intentionally produce only stale panels when Prometheus/source APIs fail; collector death itself has no observer.

Desktop notification alone is not paging: today’s laptop saturation could impair both Overdeck and its notifier. Run a low-resource watchdog independently—preferably from `debian1` or an external heartbeat—and notify through a channel not hosted on the laptop. If v1 keeps GOLIVE’s “no push” cut, document that loud failure is not achieved.

## 4. Overdeck plan audit vs CI pains

### Tabletop: today’s incident

| Incident facet | Visible in planned v1? | Fixable with current X1 verbs? |
|---|---|---|
| `debian1` idle while laptop overloaded | Partially. A4 shows host health; A5 intends queue depth. No correlated “pending work + idle remote capacity” invariant. | No |
| Wedged buildslot queue/ghost | A5 intends running/queued/p95, but reads `~/.cache/buildslot`; code uses `/run/dev-build-slot` or `/run/user/<uid>/buildslot` (`buildslot.sh:61-67`; plan:205-206). | No. `reap` targets orphan agent PIDs, not queue reconciliation. |
| Repeated remote status 127 | No. A3 observes GitHub CI, not `local-gate`/remote jobs. | No |
| Epoch races/discarded jobs | No remote job, mirror, epoch, retry, or publication telemetry. | No |
| Fallback/runner drift | Only indirect runner status. No desired-vs-actual mode, label, listener/session, or maintenance lease. | No |

Required panels/signals:

- `Panel{id:"build-control"}`: desired/observed state, revision, maintenance lease/expiry, reconciler health.
- `Panel{id:"build-queue"}`: ordered tickets, repo/key, enqueue age, priority, owner PID+starttime/unit, assigned host, state, retries, oldest/p95.
- `Panel{id:"remote-jobs"}`: per-job snapshot, workspace, stage, host, capability image, log tail, rc/reason, pull bytes/duration, artifact publication state.
- `Panel{id:"ci-runners"}`: distinct runner identities, labels/eligibility, service/listener/session state, desired host, queue oldest age.
- Derived Items: `remote-idle-queue-stalled`, `capability-missing`, `fallback-lease-expired`, `runner-drift`, `artifact-publication-blocked`, `controller-down`, and repeated failure-class aggregation.

Required actions are high-level controller operations, not file/lock toggles:

- `box-drain`, `box-restore`—idempotent desired-state transitions with revision precondition.
- `job-cancel`, `job-retry`—job-ID scoped; retry only typed infrastructure failures.
- `admission-reconcile`—controller proves and repairs stale leases. Do not expose `flush-ghost-lock`.
- `ci-reconcile`—restore declared runner eligibility/service state. Do not expose raw label mutation.
- `host-quarantine`, `host-unquarantine`—capability/transport circuit breaker operations.

X1’s default-deny verb list, Zod validation, journaling, and argv spawning are necessary but insufficient (`plan:252-261`). Add capability-scoped handler modules, authorization per handler, `expectedRevision`, idempotency key, fresh-state preconditions, PID+starttime/unit identity, bounded execution, and write-ahead audit. If audit intent cannot be durably written, mutation must fail. Overdeck should call controller APIs; it must not shell out to mutate buildslot files or config.

Additional plan defects:

- A3 assumes runner APIs for platform repos, while `platform-fallback.sh:12-13` states current credentials cannot access runner-list/label APIs. Acceptance cannot pass as specified without a GitHub App/PAT with appropriate administration scope or an alternate authoritative adapter.
- U1 creates new `KpiTile`/phase/lane/table components (`plan:157-165`), but the harness report’s R3.8 contract requires reusable pure observability primitives. Implementing them first in Overdeck duplicates the declared shared library instead of consuming it.
- Plan architecture says existing tools remain untouched/read-only, but decision answers, steering, Settings config writes, and X1 actions mutate them (`plan:7, 11, 49, 192, 244, 257`). Mutation boundaries need explicit ownership, not “read-only except four actions.”

## 5. Verdict

Overdeck v1 does **not** address the incidents it is now expected to operate. It is a broad dashboard with CI-run visibility, but no offload control-plane model. Do not add raw log scraping and lock-flush buttons. Amend waves around an authoritative build controller, then adapt Overdeck to it.

Keep the one-week cap. Defer T1 deck-tui, A7 Bots, MapCanvas polish, and template morning digest; retain plain web status pages. Spend that budget on these contracts:

### Task I1: Build-control desired-state reconciler

**Wave:** 0 · **Blocks:** I2, A8, X1

**Contract:** single writer for `available|draining|maintenance|restoring|degraded`; transactional journal + monotonic revision; owns local-fallback lease, host capacity, CI labels/services, and crash-resumable reconciliation. Read/status API plus typed transition API. Direct mutation paths denied.

**Acceptance:** kill controller during every transition and restart; state converges without two eligible shared identities, forgotten fallback, or premature poweroff. Corrupt config fails closed and creates one incident.

### Task I2: Isolated remote job and artifact protocol

**Wave:** 1 · **Blocks:** A8 · **Blocked by:** I1

**Contract:** shared bare object cache only; unique immutable workspace/overlay/output per job; atomic remote capacity reservation; declared artifact manifest; staged pull and local snapshot CAS before promotion; TTL cleanup. Remove shared mirror epoch fencing from job correctness.

**Acceptance:** two concurrent dirty worktrees from one repo both complete without cancellation or cross-contamination; local edits made during build remain byte-identical; stale job artifacts cannot publish; transport interruption reattaches once.

### Task A8: Build/offload adapter

**Wave:** 2 · **Blocks:** U4, X1 · **Blocked by:** I1, I2, S2

**Contract:** consume controller status/events and Prometheus; emit the four panels and incident Items listed above. No log parsing as authoritative state. Retain failed incidents until explicit auto-resolution.

**Acceptance:** recorded fixture of today’s four-part incident produces queue-stalled, capability-missing, and concurrency/publication Items with exact job links; source outage marks stale and independent controller-down Item fires.

### Task X1 amendment: Typed infrastructure handlers

**Wave:** 4 · **Blocked by:** A8

**Contract:** retain existing verbs; add `box-drain`, `box-restore`, `job-cancel`, `job-retry`, `admission-reconcile`, `ci-reconcile`, and host quarantine verbs. Handlers call versioned controller APIs with idempotency and expected revision. Write-ahead journal is fail-closed; no generic exec handler.

**Acceptance:** stale revision, invalid state transition, replayed idempotency key, unavailable audit store, and stale PID/unit fixtures all deny safely; crash after intent write resumes to one outcome.

### Task N1/V1 amendment: Incident delivery + chaos tabletop

**Contract:** N1 alerts on actionable incident Items with cooldown/quiet-hours rules but bypasses quiet hours for publication-integrity events; independent watchdog observes collector/controller. V1 replays box loss/return, stuck queue, missing executable, concurrent same-repo jobs, runner drift, pull interruption, and collector death.

**Acceptance:** each scenario has an asserted panel/Item/action/auto-resolution path; no scenario depends on reading a log or manually editing config/labels.

### Contradictions with the context bundle

1. **“REMOTE-ONLY … never silent local run” is overstated:** invalid/missing/disabled remote config returns `disabled`, which authorizes local execution.
2. **“Machine-global FIFO” is not strict:** non-head tickets may probe after one second of observed free capacity (`buildslot.sh:426-437`), and priority tickets are LIFO (`buildslot.sh:224-231`).
3. **“Stranded cohorts self-heal” is conditional:** requeue injection is skipped when caller already sets `BUILD_SLOT_TIMEOUT`, and child exit 75 is indistinguishable from admission timeout.
4. **“300s timeout kills a 6+ minute pull” is inaccurate:** code uses rsync’s 300-second I/O-idle timeout, not a 300-second wall-clock deadline (`remote-build.mjs:137-144, 331-333`). Pulls can still appear hung and fail without structured progress.
5. **Literal “flock held with no fd owner” remains unproven:** a live flock requires an open file description. Empty `lsof` plus failed lock acquisition implies visibility/race/path-inode ambiguity; rotating the path may restore service but does not establish root cause. Correctness must not depend on `lsof` visibility.
6. **Overdeck A5’s buildslot path contradicts live code:** plan names `~/.cache/buildslot`; implementation selects `/run/dev-build-slot` or `/run/user/<uid>/buildslot`.
7. **Overdeck A3’s runner visibility contradicts platform fallback assumptions:** the fallback explicitly states current account cannot read runner/label APIs.
8. **Overdeck’s visual reuse direction contradicts harness R3.8:** plan builds a second component set and says harness may adopt later; report requires one reusable pure-props primitive library consumed by other observability systems.