---
name: ci-cd-expert
description: Set up or fix efficient CI/CD on a self-hosted runner sharing a dev workstation — PR-gated trunk, shared build cache, CPU containment, flaky-gate diagnosis, browser/E2E matrix gates (sharding, completeness accounting, fake-green detection). Use when adding CI to a project, when a CI failure just recurred and needs a self-healing remedy rather than another hand fix, when a gate is slow/flaky/starving the machine, when a gate reads green but catches nothing, or when replicating the platform repo's CI stack elsewhere.
---

# CI/CD Expert — self-hosted gate on a shared dev box

Audience: AI coding agents first. Battle-tested in `platform` repo (PR#67→#81, 2026-07). Canonical generic primitives are VENDORED in this skill's `templates/` — **copy from there, don't re-derive and don't hunt for the origin repo** (catalog + replication checklist: `references/primitives.md`). A project's copy is an instance; back-port material improvements to `templates/`.

## Git/CI fatigue contract

MUST follow `/home/user/Projects/0 DOCS/GIT_FATIGUE.md` §12. Valid receipt/log for current tree = proof; NEVER rerun broad typecheck/build/test. Mass diagnostics: capture once, cluster, repair ALL, ONE replacement run. Exhaustive gate runs once on designated executor; workstation hooks MUST finish ≤10s and only hand off. User surface MUST be only `Preview ready` or `Needs product decision`. MUST read §12 before adding or changing any gate, hook, CI step, or ceremony.

## Prime directive (owner constraint)

**MINIMIZE TOTAL ON-BOX COMPUTE. NEVER widen the core budget to go faster.** Levers, in order: (1) don't recompute — share cache; (2) don't collide — serialize; (3) don't recompute per-boot — template caches; (4) only then tune concurrency. Machine-wide aggregate cap = systemd user slice (`build.slice`); per-run knobs live in one wrapper script.

## Self-healing law (any failure that can recur)

**A failure that can happen twice MUST ship with a mechanism that detects and repairs it automatically. Repairing the instance is not repairing the problem.** The expensive part was never the fix — it was the investigation, and a hand-run remedy makes someone pay for that investigation again, in full, every time. Write the mechanism in the same change that fixes the instance, while the failure is still understood.

The failure mode this exists to kill: an outage nobody notices. CI that queues forever instead of erroring, a cache that silently stops being warm, a gate that quietly stops firing. Wall-clock cost is bounded by how long until a human happens to look — nine hours, in the incident that produced this rule.

What makes the mechanism worth having:

- **Key it on the invariant, not on the cause you diagnosed.** "A Ready listener exists" heals every future cause with that symptom; "clear the leaked CNI netns" heals exactly one. The diagnosis is a sample of the failure; the invariant is the contract.
- **Confirm before acting.** Require the same bad state on two consecutive observations. Transient bad states are normal, and a trigger-happy remedy becomes its own outage.
- **Escalate, then stop.** Cheap remedy, then a bigger one, then fail loudly. A remedy that retries the same action forever is indistinguishable from no remedy at all.
- **Failing loudly is PART of the mechanism.** Self-healing narrows the window in which a human is needed; it never removes the case. When automatic repair does not work, leave an artifact that persists on its own (a Failed Job, a non-zero gate) — never just a log line.
- **Silence must be attributable.** Publish a machine-readable verdict and give it exactly ONE command to read it. "It heals silently" must not become "nobody can tell what happened".
- **Record what the repair CANNOT undo.** Restoring the ARC listener does not re-dispatch the jobs GitHub queued while it was down — those need `gh run cancel && gh run rerun`. Residue the remedy cannot clear has to be named in the verdict, or it becomes the next silent failure.
- **Rung 9 below applies — watch it heal once.** Inject the failure, watch the mechanism fire, confirm the state it leaves behind. A watchdog nobody has ever seen fire is fake-green with extra steps.

Reference implementation: `modules/ci/k8s/listener-watchdog.yaml` (detect + repair + verdict) with `modules/ci/arc-watchdog-status.sh` (the one command) and `modules/ci/tests/install-arc.test.sh` (proves the mechanism is actually installed, per rung 8 below).

## Greenfield build order (NEW CI — follow in order, do not defer any rung)

Every rung below exists because skipping it cost a later remediation round. Build them in, don't optimize them in.

1. **Slice first.** Create the systemd user slice with a FINITE `CPUQuota`, `MemoryHigh`, `MemoryMax`, `IOWeight` before the first workflow. Every heavy step goes through one wrapper script from day one. Set `BUILD_SCHED_IDLE=0` in CI at the same moment.
2. **No `actions/cache` on self-hosted.** Gate it to hosted runners on the first commit that adds it.
3. **Decompose before you have a duration problem.** Design every long suite as N shards keyed by shard index on a shared label, with the shard list emitted by the planner. Retrofitting host-pinned jobs into a shard matrix is a schema change that touches planner, runner, aggregate, artifact names and tests simultaneously.
4. **Receipt schema on day one:** `expected`, `executed`, `notStarted`, terminal status. Aggregate validates the UNION. Never let a receipt infer a result it did not observe.
5. **Generate every manifest** (routes, specs, packages) from the filesystem with a `--check` drift mode. Hand-curated lists start drifting the week they are written.
6. **Overlap the critical path** from the start: provisioning in step 1, install/build parallel, fail-closed barrier.
7. **Derive parallelism from the fixture data**, never a literal.
8. **Wire every check script into the gate chain** as it is written. An orphaned `check:*` enforces nothing.
9. **Watch each gate FAIL once** before trusting it. Inject the violation, see it block, revert.
10. **Instrument from run 1:** per-step timings and per-test durations as machine-readable artifacts. Without them every later optimization is guesswork, and you will fabricate a number under pressure.
11. **Self-heal every recurrable failure.** The moment a failure is understood, ship the mechanism that detects and repairs it — not a runbook entry. See *Self-healing law*.

Anti-pattern that generates remediation rounds: shipping a gate that is green, fast and unmeasured, then discovering it never executed what it claimed. Rungs 4, 9 and 10 are the ones that prevent it.

## Topology (decision rules)

- Gate PRs **pre-merge** (`pr-gate.yml` on `pull_request`). Post-merge `main-gate.yml` exists ONLY as delivery trigger — NEVER as the safety net, NEVER with an auto-revert bot.
- Free-tier private repo = no server-side branch protection. Enforce in a **fail-closed merge wrapper** (`safe-merge.mjs`): refuses unless the named gate check is SUCCESS; raw `gh pr merge` blocked by PreToolUse hook. Wrapper must be authoritative on PR STATE, not gh exit code (gh's post-merge local checkout fails in worktree-heavy repos → false negative).
- Pre-push hook = ≤10s handoff only; NEVER typecheck/build/test/e2e. Designated executor owns one authoritative receipt-backed gate.
- Workflow contract = executable: assert workflow topology in a script test (`ci-artifact-gate.test.mjs` pattern) so drift fails the gate.
- Self-hosted runner + `pull_request` = RCE from fork PRs. Gate fork PRs before repo goes public.

## Shared machine cache (biggest win)

ONE turbo/build cache machine-wide: `/var/cache/<project>-turbo`, ACL-shared between dev user and runner user (`setfacl -R -m u:<runner>:rwX -m d:u:<runner>:rwX -m d:u:<dev>:rwX`). Wrapper auto-prefers it when present+writable.

**`actions/cache` on a self-hosted runner is an ANTI-PATTERN. GATE it to hosted runners; do NOT delete it outright.**

The store is already persistent local disk. Wrapping it re-tars and re-untars every job for zero benefit — and a store larger than GitHub's ~10GB per-repo limit can never even hit.

```yaml
# target — keep the step, gate it
- if: runner.environment == 'github-hosted'
  uses: actions/cache@v6
```
Deleting the step breaks any job in the same repo that DOES run on `ubuntu-latest`. Check every workflow's `runs-on` before removing. A reviewer demanding outright deletion is wrong unless the repo has zero hosted jobs.

**Measured (multideal, 2026-08):** composite setup 77.4s avg × 516 jobs = **665.4m = 16% of all CI host time**. Bare `pnpm install --frozen-lockfile` 8.3s, `setup-node` 7.9s → ~60s/job pure cache overhead. Store 8.5G + Playwright browsers 4.2G = 12.7G, over the limit. `Post Run` averaged 1.1s → the cost is on RESTORE, not SAVE.

Before adding any cache step to a self-hosted job, verify the janitor does not prune the path. Prune scripts that target `/tmp` and `$HOME/builds` do NOT touch `~/.local/share/pnpm/store` or `~/.cache/ms-playwright` — those are already safe and need no wrapper. Confirm the store and the build dir share an fsid, so the package manager hardlinks instead of copying.

- Content-hashed cache is portable across executor checkouts and users. Workstation hooks NEVER warm it with heavy checks.
- **Proof numbers:** warm PR gate 1m51s, post-merge main gate 2m02s, vs ~15m cold (vs ~1h original).
- Lockfile or turbo.json change = GLOBAL hash invalidation → full cold run. Expected, not a bug. Don't chase it.
- Trim inputs: `"inputs": ["$TURBO_DEFAULT$", "!**/*.md"]` (verify nothing imports .md first). `typecheck` task: `outputs: ["tsconfig.tsbuildinfo"]` + TS `incremental` with `"tsBuildInfoFile": "${configDir}/tsconfig.tsbuildinfo"` (bare `incremental` breaks tsup DTS, TS5074).

## CPU containment (the wrapper)

One `cpu-limit.sh` wraps every heavy invocation (`templates/cpu-limit.sh`). Chain: `systemd-run --user --scope --slice=build.slice -p CPUQuota/-p MemoryHigh/Max/CPUWeight/IOWeight` → `nice 19` → `ionice -c 3` → `chrt --idle 0` → `taskset`. Plus: `GOMAXPROCS` (esbuild ignores cpusets), `VITEST_MAX_FORKS`+`VITEST_MAX_THREADS` (default pool is forks — capping threads alone is inert), Node heap cap, affinity-aware `nproc` vs `nproc --all` (outer restriction IS the budget — don't subtract reserve twice).

- **SCHED_IDLE (`chrt --idle 0`) is for LOCAL INTERACTIVE runs ONLY. CI MUST set `BUILD_SCHED_IDLE=0`.** Proven failure: busy box (load 55–80) starves an idle-scheduled gate → wall-clock test timeouts fire, DIFFERENT victim package each run. Signature to recognize: hook/test timeouts + trivial tests taking 10x normal + rerun fails elsewhere.
- **Machine-wide flock serialization** in the wrapper: fd held for process lifetime, `BUILD_LOCK_HELD` nested-guard, `BUILD_NO_LOCK` opt-out, unwritable lock path → warn + run unserialized (fail-open on the LOCK, never on correctness). Second gate queues, then replays warm cache — serialization is itself a speed win.

## Utilization: decompose, never build a scheduler

**Measure concurrency as MEAN-WHILE-ACTIVE, never as peak.** Peak tells you the cap was reachable once; the mean tells you what the estate actually does.

Measured (multideal, 2026-08): peak **6/6** slots, mean-while-active **2.10/6**, idle **91%** of the window. Peak alone would have read as "saturated".

Decision rule from that pair:
- **High peak + low mean → the estate is starved of DISPATCHABLE UNITS. Split long jobs.** Every lever that decomposes beats every lever that schedules.
- Low peak + low mean → the trigger never fires (see Unreachable gate).
- High peak + high mean → genuinely capacity-bound; only then buy capacity.

**NEVER build a queue orchestrator to fix low utilization.** The CI provider's matrix dispatcher already is one, and it is deterministic. A hand-rolled scheduler on top re-implements it worse and adds a component that can itself fail green. If jobs are not spreading, the cause is host affinity or job granularity — fix those.

**Fail-closed MUST NOT mean fail-on-a-condition-you-could-repair.** A wrapper that aborts because a lock/cache/state DIRECTORY is absent is a defect: creating the directory preserves the guarantee in full. Reserve fail-closed for conditions that genuinely cannot be made safe (permission denied, read-only fs, unverifiable provenance).

```bash
# reject: aborts CI on a directory it could create
[ -d "$lock_parent" ] || { echo FATAL >&2; exit 78; }
# target
mkdir -p "$lock_parent" 2>/dev/null || true
# ...then the real writability test, still fail-closed under CI
```
Measured cost of getting this wrong: every job on both hosts died at `exit 78` before one test ran, because `~/.cache/ci-box` did not exist.

**A machine-wide serialization lock and sharding are mutually exclusive.** The lock fd is held for the whole process lifetime, so N shards on one host serialize to N × single-shard duration. When you shard, drop the lock and let the slice carry the aggregate ceiling — but verify the slice bounds MEMORY and IO, not only CPU (`MemoryMax`, `MemoryHigh`, `IOWeight`), before trading serialization away. Audit that every workflow using the wrapper agrees on this setting; the one that disagrees is the bug.

## Pipeline shape: overlap the critical path

- **Start slow provisioning in step 1, then install/build in parallel, and join on a fail-closed barrier.** DB branch creation, container pulls and seeding do not depend on `node_modules`. Barrier contract: a state dir with `status` (`ready`/`failed`), `error` and `env` files; the consumer polls with a finite deadline, aborts on `failed`, and aborts on timeout. Never let the barrier fall through on a missing file.
- **Disposable test Postgres: put the data dir on tmpfs and turn durability OFF** — `fsync=off`, `synchronous_commit=off`, `full_page_writes=off`. The cluster is thrown away; crash safety buys nothing. Keep a disk fallback with a free-space floor check.
- **Clone shards with `CREATE DATABASE … TEMPLATE`** instead of re-seeding per shard. Terminate template sessions first, and withhold the template URL from consumers until the clone completes.
- **Use the blob reporter + a single merge** instead of per-shard HTML. Upload heavy evidence `if: failure()` only; receipts and duration JSON stay `if: always()`.
- Artifact retention: evidence on failure only is a large, free saving. Never make the RECEIPT conditional — that is the thing you need on a green run.

## Parallelism ceilings are usually DATA, not CPU

Before raising a worker count, find what the workers contend over. If tests share seeded fixtures, the ceiling is the number of independent seeded actor sets — raising `workers:` past it produces cross-test interference, not speed.

**Derive the worker count from the fixture array so the two cannot drift:**
```ts
export function playwrightWorkerCount(factoryMode: boolean) {
  return factoryMode ? FACTORY_ACTOR_SETS.length : 1;
}
```
Raising parallelism then means seeding another actor set — a data change with an obvious blast radius — not editing a magic number. Give each worker its own actor set via a `scope: 'worker'` fixture keyed on `testInfo.parallelIndex`, and make an out-of-range index THROW.

**Do NOT build a runtime load-sensing worker controller.** The cgroup already arbitrates CPU/memory/IO; a userspace load-reader fights it and adds nondeterminism to a gate. Test a source-string assertion (`expect(config).toContain('workers: 2')`) is NOT a test of this — assert the exported function's behaviour instead.

## Test-suite mechanics (DB-backed suites)

- **Timeouts are factory-owned, sized to contention, one source of truth.** Shared vitest factory preset carries `hookTimeout: 300_000, testTimeout: 180_000` for PG-backed suites. NEVER per-package bumps — with mergeConfig, a per-package 30s silently CAPS the factory value. Contention stretches wall-clock ~10x (measured: <8s-idle tests ran 77s). Timeouts bound genuine hangs, not speed.
- PG suites: `singleFork` per package → ≤1 DB per package, total DBs ≤ task concurrency.
- Embedded PG: **socket-only** (`listen_addresses=` + `-k dataDir`, connect `host: dataDir`). TCP port pre-allocation = TOCTOU race under concurrency. Structural fix, not retry.
- **initdb template cache** (`@tooling/pg-template` `initdbCached()`): per-boot 1675ms→534ms. Design invariants — cache key MUST include initdb `--version` AND binary size+mtime (version string survives package rebuilds) AND user+args AND full `LC_*`+`LANG`+`TZ` (initdb bakes TZ into postgresql.conf — proven stale-template bug); atomic `rename` = lock-free race (EEXIST/ENOTEMPTY/EPERM = use winner); refuse non-empty dataDir (fs.cp silently overlays); ANY cache error → warn + real initdb (fail-open to correctness).

## Measure before optimizing (MUST — first rung, always)

NEVER tune a gate from its total duration. Derive the phase split from the run's OWN machine-readable report before touching anything.

1. Step timings → fixed overhead vs execution (`gh api .../actions/runs/<id>/jobs`).
2. Per-test duration + count, grouped by project, from the reporter's JSON — not from log scraping.
3. Compare max observed test duration against the configured timeout BEFORE tuning timeouts.

Decision rules from those three numbers:
- max observed ≪ timeout → timeout tuning is NOT a lever. Do not touch it.
- mean per-test small × huge count → parallelism lever.
- fixed overhead dominant → cache/reuse lever, NOT parallelism.
- Parallel shards duplicate COMPUTE, not WALL TIME. N independent setups still cost 1× setup on the critical path. Do NOT reject sharding on duplicated-setup grounds.

Measured example (multideal ui-sweep, 2026-08): 31:03 total = 1:05 setup + 3:29 DB branch/seed + 1:18 build + 25:05 execution; 1,284 cells, mean 4.18s, max 22.1s vs 45s timeout. => parallelism + per-cell writes were the levers; timeouts were not.

## Browser/E2E matrix gates

Route-×-role-×-locale-×-viewport suites. Rules here are ADDITIONAL to the sections above.

**Sharding**
- **Shard by SHARD INDEX on a shared runner label. NEVER pin a shard to a named host.** Pinning defeats the dispatcher: a shard routed to a busy host waits while another host sits idle. Emit the shard-id list from the planner as a job output and drive `strategy.matrix` from it via `fromJSON`, so shard count lives in exactly ONE place.
  - Earlier guidance said "shard by physical host". That was a workaround for cpuset contention and port collision. Both have proper fixes (a finite `build.slice` quota; shard-id in the port derivation). Host pinning is the wrong tool — it trades away all dispatch balance to solve a problem the slice already solves.
  - Constrain concurrency with the runner label + slice quota, NOT with host affinity.
- Port derivation MUST include the shard id. Two matrix jobs on one host that derive ports from repo+workflow alone collide.
- **Emit only NON-EMPTY shards.** A fixed shard count (`N=12`) over uneven work produces empty bins; the runner rejects an empty shard and aggregate cannot prove a receipt for a cell that had no work. Never dispatch a shard the plan did not declare.
- Bin-pack **longest-processing-time-first**: sum each group's duration, sort DESCENDING by weight with a deterministic id tie-break, assign to the lowest-weight shard. Sorting groups lexically and assigning to the lightest bin is NOT LPT and packs badly.
- **Indivisible groups.** Where the runner executes whole spec files, the packing unit is the spec (or spec+project), never the individual cell. A spec marked `serial-external` needs every one of its projects in ONE shard. A splitter that ignores grouping silently corrupts execution.
- Correctness keys on SHARD ID. Keep the real hostname as metadata only. Shard id MUST appear in every internal filename and every artifact name — blob/report names built from `host:project` collide across shards on merge.
- Per shard: unique artifact names AND unique run markers.
- Cross-route aggregate checks (nav-consistency fingerprints, dedup, global uniqueness) CANNOT run per shard. Merge raw inputs, fold ONCE globally.

**Union completeness (the part that must not be got wrong)**
Per-shard markers that are each internally consistent prove NOTHING about the run. Validate the UNION against the manifest:
- shard cell sets DISJOINT (no cell twice, none claimed twice),
- every shard id present (missing shard MUST fail, never silently shrink the run),
- executed count == expected count exactly,
- one aggregate fingerprint per cell.

// DO NOT: sum per-shard pass counts and call it green.

**Coverage completeness**
- GENERATE the route manifest from the filesystem + `--check` drift mode. Hand-curated manifests silently omit new pages (measured: 173/177 covered, 4 pages absent, nobody noticed).
- One-role-per-route manifests hide every authed defect on public routes. Fan out roles explicitly; every exclusion carries a machine-readable reason.
- Dynamic-route fixtures MUST resolve from seeded reference data. NEVER by crawling a source page — a blocked source page silently unresolves the fixture.
- Unresolvable fixture MUST fail loudly naming the route. NEVER degrade to skip or pass.
- Lazily-hydrated islands (`client:visible`, `client:idle`) are INVISIBLE unless forced to mount. A cell that only asserts "page loaded" proves nothing about them.
- Expected-error declarations MUST fail when STALE (declared error stops firing), not only when an undeclared error appears.

**Determinism (never buy it with time)**
- Replace every fixed sleep with an awaited signal + deadline. Deadline expiry MUST be a FAILURE naming what never arrived. Measured cost of one fixed 15s hydration wait: 112 cells, 7+ wall-clock minutes.
- Connectivity-dependent features (WebSocket/SSE) expose CONNECTED / DISABLED explicitly. Ambiguous silence MUST fail.
- `repeat-each`/retries are CONFIDENCE checks, NEVER fixes. Never ship one as a determinism fix.
- DO NOT reuse browser contexts across cells to save time — masks role/session leakage, the bug class the suite exists to catch.

**Per-cell cost (apply all; none reduce coverage)**
- Launch flags: `--disable-dev-shm-usage` (load-bearing where `/tmp` is tmpfs — shm pressure with parallel workers crashes renderers and reads as flake), `--disable-gpu`, `--disable-extensions`, `--disable-background-timer-throttling`, `--disable-renderer-backgrounding`, `--disable-backgrounding-occluded-windows`.
- Artifacts: `trace: 'on-first-retry'`, `video: 'off'`, `screenshot: 'only-on-failure'`. Never `trace: 'on'` in CI.
- `page.route`-abort third-party/analytics/font hosts. Use a NAMED BLOCKLIST, never a blanket cross-origin block — over-broad blocking silently changes what is under test.
- Serve a PREBUILT preview. A `webServer.command` of `build && preview` rebuilds inside the Playwright startup timeout, so a slow build reads as a server-start failure. Move the build to its own step; `webServer` only serves.
- Install only the browsers the matrix uses, per job.

**Prove read-only, never assume it**
A suite documented "live-readonly" is an untested assumption. Run the matrix under a SELECT-only DB role; passing IS the proof. Only then may one seeded DB branch serve N shards.
Measured: every authenticated cell performed 9 writes before its SELECT on an owner-level connection, in a suite documented read-only.

## Fake-green failure modes (audit for ALL of these)

A gate reading green proves nothing until each is excluded. Every one below was live in one repo simultaneously.

| Mode | Signature | Fix |
|---|---|---|
| `continue-on-error: true` on the gate job | job conclusion `success` with hundreds of failures in the log | delete it; assert absence in the workflow contract test |
| Gate triggers only on a dead branch / manual dispatch | last run months old; workflow "exists" | trigger on the path that ships code |
| Reporter records only `passed` cells | `executed` field understates; never compared | record TERMINAL cells (passed/failed/timedOut/interrupted) |
| expected vs executed computed, never compared | ratchet accepts marker status `failed` | compare; fail when `executed < expected`, naming missing cells |
| Aggregate check folded AFTER validation | findings land in an artifact, exit code already 0 | fold BEFORE validation; findings participate in pass/fail |
| Changed-files gate maps shared deps to zero routes | one-level dependency mapping | transitive mapping, or full matrix |
| Unit test PINS the unsound behavior | `expect(run('failed').code).toBe(0)` | fix test + state why the change is legitimate |
| Receipt SYNTHESIZES results it never observed | `executed: 0` alongside `failed: 451`; duration wildly over the raw report's | report `executed` / `notStarted` / `incomplete` as DISTINCT facts; never expand "expected" into "failed" |
| Injected auth state applied suite-wide | anonymous + auth-boundary specs silently authenticated; protected-route tests pass without proving protection | `storageState` is OPT-IN per spec; keep a real-login lane covering login, logout, cookie issuance, expiry, revocation, redirect |
| Gate skipped on a self-written receipt | a host-writable file marks a SHA "already passed" | provenance MUST be the CI provider's own authenticated immutable run/check result, bound to repo + exact 40-char SHA + workflow + job + contract version + conclusion + artifact digest |

**Cross-check every receipt against the raw reporter output in the SAME artifact.** Measured: a receipt claimed `expected: 451, executed: 0, counts.failed: 451` while the raw Playwright JSON beside it showed 6 specs, 2 passed, 14 failed, 5.0 minutes. ~16 tests ran; the receipt reported 451 failures. A receipt is a claim, not evidence.

**Run got faster AND quieter = broken gate, NEVER progress.** Check executed-vs-expected on EVERY run. Measured: a blanket capability-disable skipped all 1,286 cells and produced a clean 5-minute run.

**Every gate MUST be seen to FAIL before it is trusted.** Inject the violation, watch it block, revert. A guard nobody saw block is not a verified guard.

## Stall detection (a hung job is the most expensive failure there is)

A red job costs one rerun. A HUNG job burns a runner slot for the whole timeout and is only noticed when a human happens to look — measured on this estate: 12 hangs / 1142 min in 3 days, worst 8.6h. Every long-running CI step and every agent dispatch MUST be supervised.

**NEVER infer liveness from process existence.** A crash-looping, deadlocked or socket-blocked child holds a live PID for its entire timeout while doing nothing. `pgrep`/`pidof`/`kill -0` are NOT liveness. Judge **PROGRESS**.

Canonical implementation: `~/Projects/overdeck/modules/monitor/bin/stall-guard` (+ `README-stall-guard.md`). Do not re-derive it.

**Sample four orthogonal signals across the whole process TREE, not the direct child:**

| Signal | Source | Why it must be there |
|---|---|---|
| `out` | captured log size | the only semantic signal |
| `cpu` | `/proc/<pid>/stat` utime+stime, recursive | a silent long compile is NOT a hang |
| `io` | `/proc/<pid>/io` **rchar+wchar**, recursive | `read_bytes` MISSES socket traffic entirely |
| `beat` | mtime of a heartbeat file | declared legitimate waits (locks, queue slots, barriers) |

**Two kill rules, both required:**
1. every tracked signal flat for the idle window → PASSIVE hang (blocked socket, deadlock, `sleep`).
2. `out` alone flat for a much longer output ceiling, whatever cpu/io do → ACTIVE hang (spin loop, retry loop). Rule 1 ANDs the signals, so ONE perpetually-moving signal vetoes detection forever without rule 2.

**Accumulate POSITIVE deltas only — never compare raw endpoints.** Counters are not monotonic: summed cpu/io FALL when a child exits, `out` falls on log truncation. A decrease then reads as "progress" and silently disables the whole detector. Confirmed live in real traces.

**Kill the enumerated tree AND the process group, then WAIT for death before `SIGKILL`.** `killpg` alone misses grandchildren that called `setsid`; an unconditional sleep before `SIGKILL` lets a recycled PGID absorb the signal.

**Failure of the watchdog MUST be loud.** `except: return` in a sampling loop restores exactly the silent multi-hour blindness the tool exists to remove. Report UNGUARDED to stderr + desktop notification.

**The residual class, stated honestly:** a process emitting NOVEL output forever (`echo waiting for lock...` in a loop) is indistinguishable from progress by any content-agnostic signal. Give it a **non-killing advisory** at ~p99.9 of the measured duration distribution. Wall-clock is the only rule with a real false-positive cost — never let it kill by default.

**Size every window from the measured distribution, never a guess.** 5918 real agent runs: p50 5.7m, p90 19.5m, p99 57m, p99.9 175m, max 531m — a 30m wall cap would false-positive on 4.56% of legitimate runs. One real 11m47s reasoning run went **195s with zero output** while cpu/io never stopped: an output-only watchdog at 120s kills it.

**Both replay arms MUST pass before arming enforcement** — see Verification doctrine. Record every run's sample trace so the corpus grows into a standing regression check.

## Verification doctrine

- Every deflake/cache fix ships with a test that **fails on the old code** (prove RED before trusting GREEN). A test never observed failing proves nothing — neuter the fix, watch it go red, restore.
- **A detector needs BOTH arms.** `--expect quiet` over traces of runs that COMPLETED, and `--expect fired` over known-stuck ones. A detector that can never fire passes the no-false-positive arm alone — the exact defect that shipped here once already.
- Adversarial review before merge; reviewer must VERIFY claimed fixes discriminate, not just read the diff.
- After merge, capture the timing number as proof; report cold-vs-warm honestly (don't attribute global-hash cold runs to a broken cache).

## Gate content (coverage ≠ speed)

Fast gate can still enforce nothing. After perf work, audit WHAT gate checks vs project law. Recurring gap classes (all found live in platform, 2026-07):

1. **Orphaned checks.** Script exists (`check:*` in package.json, or tools/*.mjs) but absent from the gate chain → never runs, law silently unenforced. Audit: every check script reachable from `gate`/`verify`; wire or delete.
2. **Hand-maintained lists drift.** Import-smoke list covered 34/63 packages — every new package ships un-smoked unless someone remembers to edit the list. Fix: GENERATE the check from machine-readable source (walk every package's `exports` map, import every subpath from dist). Catches the proven bug class: subpath declared in `exports`, tsup entry missing → dist 404s while src-importing tests stay GREEN.
3. **Prose-only law.** Packaging conventions (ESM fields, `sideEffects:false`, dep classification, version ceiling) documented but not machine-enforced → regression by drift. Fix: one conformance script over every `packages/*/package.json`, wired into `verify`.
4. **Missing process gate.** PR touches `packages/*/src` with no changeset → merges fine, silently never publishes. Fix: `changeset status --since <base>` step in PR gate (empty changesets stay valid).
5. **Pre-public security.** Self-hosted runner: fork-PR guard `if: github.event.pull_request.head.repo.full_name == github.repository` on EVERY job (see Topology) + pinned gitleaks step.
6. **Unreachable gate.** Workflow is correct AND never fires (dead branch trigger, dispatch-only, schedule dropped). Audit: every gate reachable from the path that ships code. A gate the shipping path does not traverse enforces NOTHING.
7. **Documented-but-unenforced law.** A rule stated in this skill (or any doc) and absent from the workflow. Proven live: `BUILD_SCHED_IDLE=0` documented here since 2026-07, still unset in a 2026-08 gate → idle-scheduled under host load. Fix: assert the rule in the workflow contract test, not in prose.

Every new check: prove RED on synthetic violation, revert, prove GREEN — before calling it wired.

## Three-tier enforcement (place each law at its cheapest catcher)

| Tier | Catches | When |
|---|---|---|
| Static content gate (slopgate) | mechanical code-law violations (forbidden imports, comment law, `.only`, a11y patterns) | write/commit time |
| Conformance + process checks | packaging law, registry sync, changesets, secrets | CI (PR gate) |
| Review agents (seam/security) | semantic law (spec conformance, authz, money invariants) | pre-merge review |

Tiers complement — NEVER duplicate a rule across tiers; put it at the earliest tier that can decide it statically.

**Static-gate wiring pattern (slopgate or equivalent):** (1) self-test MUST pass before trusting the gate; (2) per rule: RED fixture + GREEN fixture, zero false positives on current repo; pre-existing real hits → ratchet baseline (block NEW, don't demand repo-wide cleanup); (3) wire three points: pre-commit staged-files fast tier + agent edit-hooks (write-time) + CI backstop over changed-files vs base.

## Diagnosis table

| Symptom | Cause → fix |
|---|---|
| Hook/test timeouts, different package each run, trivial tests 10x slow | scheduler starvation → CI `BUILD_SCHED_IDLE=0`; timeouts to contention budget |
| `Cached: 0` on a run that should be warm | global hash invalidated (lockfile/turbo.json) → expected, no action |
| `<tool>: not found` in fresh/rebased worktree | stale node_modules → `pnpm install` before gate |
| esbuild `all goroutines are asleep - deadlock!` / silent vitest exit 1 | corrupt node_modules (nondeterministic hoist, esp. after a codex-run install) → `rm -rf node_modules && pnpm install` |
| `postgres exited 1` under gate concurrency | TCP port TOCTOU → socket-only harness |
| gh merge "error" but PR actually merged | gh local-checkout failure → wrapper re-queries PR state |
| Gate job green, log full of failures | `continue-on-error: true` on the job → GitHub rewrites conclusion to `success` → delete it |
| Run suddenly much faster and much quieter | cells skipped, not fixed → check executed-vs-expected before believing it |
| Same failure signature on hundreds of cells | ONE broken precondition, not N bugs → add fail-fast preflight asserting env + each role sentinel |
| E2E flake that retries "fix" | test races an event it does not control → awaited signal + deadline-is-failure; NEVER `repeat-each` |
| Cells fail `no runtime link found` / unresolved fixture | fixture discovered by crawling a blocked page → resolve from seeded reference data |
| CI queued forever, no error anywhere, land gate times out with no check started | the scale-set listener died — nothing polls GitHub → `bash modules/ci/arc-watchdog-status.sh`; never re-diagnose by hand. Runs queued during the outage are never assigned: `gh run cancel <id> && gh run rerun <id>` |
| ONE branch's pre-push slow / runs full gate, others fast | branch (or worktree) predates the CI stack — old lefthook/cpu-limit committed → rebase onto main, `pnpm install` to resync hooks |

## Replicating to a new project (the template question)

Portable as-is (~80%, all in `templates/`): wrapper script, flock, vitest factory, safe-merge, workflow shapes, contract test, pg-template helper. Per-project adaptation (thin): package manager + task runner names, test framework, runner labels, which suites are DB-backed, whether runner shares the dev box (if dedicated: skip ACL cache sharing + SCHED_IDLE entirely; keep everything else). One-time per MACHINE: cache dir + ACLs, `build.slice` unit, runner user. Ordered checklist + per-file adaptation notes: `references/primitives.md`.