---
name: ci-cd-expert
description: Set up or fix efficient CI/CD on a self-hosted runner sharing a dev workstation — PR-gated trunk, shared build cache, CPU containment, flaky-gate diagnosis. Use when adding CI to a project, when a CI failure just recurred and needs a self-healing remedy rather than another hand fix, when a gate is slow/flaky/starving the machine, or when replicating the platform repo's CI stack elsewhere.
---

# CI/CD Expert — self-hosted gate on a shared dev box

Audience: AI coding agents first. Battle-tested in `platform` repo (PR#67→#81, 2026-07). Canonical generic primitives are VENDORED in this skill's `templates/` — **copy from there, don't re-derive and don't hunt for the origin repo** (catalog + replication checklist: `references/primitives.md`). A project's copy is an instance; back-port material improvements to `templates/`.

## Git/CI fatigue contract

MUST follow `/home/user/Projects/0 DOCS/GIT_FATIGUE.md` §12. Valid receipt/log for current tree = proof; NEVER rerun broad typecheck/build/test. Mass diagnostics: capture once, cluster, repair ALL, ONE replacement run. Exhaustive gate runs once on designated executor; workstation hooks MUST finish ≤10s and only hand off. User surface MUST be only `Preview ready` or `Needs product decision`. MUST read §12 before adding or changing any gate, hook, CI step, or ceremony.

## Prime directive (owner constraint)

**MINIMIZE TOTAL ON-BOX COMPUTE. NEVER widen the core budget to go faster.** Levers, in order: (1) don't recompute — share cache; (2) don't collide — serialize; (3) don't recompute per-boot — template caches; (4) only then tune concurrency. Machine-wide aggregate cap = systemd user slice (`build.slice`); per-run knobs live in one wrapper script.

## Self-healing law (any failure that can recur)

**A failure that can happen twice MUST ship with a mechanism that detects and repairs it automatically. Repairing the instance is not repairing the problem.** The expensive part was never the fix — it was the investigation, and a hand-run remedy makes someone pay for that investigation again, in full, every time. Write the mechanism in the same change that fixes the instance, while the failure is still understood.

The failure mode this exists to kill: an outage nobody notices. CI that queues forever instead of erroring, a cache that silently stops being warm, a gate that quietly stops firing. Wall-clock cost is bounded by how long until a human happens to look — nine hours, in the incident that produced this rule.

What makes the mechanism worth having:

- **Key it on the invariant, not on the cause you diagnosed.** "A Ready listener exists" heals every future cause with that symptom; "clear the leaked CNI netns" heals exactly one. The diagnosis is a sample of the failure; the invariant is the contract.
- **Confirm before acting.** Require the same bad state on two consecutive observations. Transient bad states are normal, and a trigger-happy remedy becomes its own outage.
- **Escalate, then stop.** Cheap remedy, then a bigger one, then fail loudly. A remedy that retries the same action forever is indistinguishable from no remedy at all.
- **Failing loudly is PART of the mechanism.** Self-healing narrows the window in which a human is needed; it never removes the case. When automatic repair does not work, leave an artifact that persists on its own (a Failed Job, a non-zero gate) — never just a log line.
- **Silence must be attributable.** Publish a machine-readable verdict and give it exactly ONE command to read it. "It heals silently" must not become "nobody can tell what happened".
- **Record what the repair CANNOT undo.** Restoring the ARC listener does not re-dispatch the jobs GitHub queued while it was down — those need `gh run cancel && gh run rerun`. Residue the remedy cannot clear has to be named in the verdict, or it becomes the next silent failure.
- **Watch it heal once, before trusting it.** Inject the failure, watch the mechanism fire, confirm the state it leaves behind. A watchdog nobody has ever seen fire is fake-green with extra steps.

Reference implementation: `modules/ci/k8s/listener-watchdog.yaml` (detect + repair + verdict) with `modules/ci/arc-watchdog-status.sh` (the one command) and `modules/ci/tests/install-arc.test.sh` (proves the mechanism is actually installed — an uninstalled watchdog enforces nothing).

## Topology (decision rules)

- Gate PRs **pre-merge** (`pr-gate.yml` on `pull_request`). Post-merge `main-gate.yml` exists ONLY as delivery trigger — NEVER as the safety net, NEVER with an auto-revert bot.
- Free-tier private repo = no server-side branch protection. Enforce in a **fail-closed merge wrapper** (`safe-merge.mjs`): refuses unless the named gate check is SUCCESS; raw `gh pr merge` blocked by PreToolUse hook. Wrapper must be authoritative on PR STATE, not gh exit code (gh's post-merge local checkout fails in worktree-heavy repos → false negative).
- Pre-push hook = ≤10s handoff only; NEVER typecheck/build/test/e2e. Designated executor owns one authoritative receipt-backed gate.
- Workflow contract = executable: assert workflow topology in a script test (`ci-artifact-gate.test.mjs` pattern) so drift fails the gate.
- Self-hosted runner + `pull_request` = RCE from fork PRs. Gate fork PRs before repo goes public.

## Shared machine cache (biggest win)

ONE turbo/build cache machine-wide: `/var/cache/<project>-turbo`, ACL-shared between dev user and runner user (`setfacl -R -m u:<runner>:rwX -m d:u:<runner>:rwX -m d:u:<dev>:rwX`). Wrapper auto-prefers it when present+writable. DELETE `actions/cache` tar/untar steps — same box, pure overhead.

- Content-hashed cache is portable across executor checkouts and users. Workstation hooks NEVER warm it with heavy checks.
- **Proof numbers:** warm PR gate 1m51s, post-merge main gate 2m02s, vs ~15m cold (vs ~1h original).
- Lockfile or turbo.json change = GLOBAL hash invalidation → full cold run. Expected, not a bug. Don't chase it.
- Trim inputs: `"inputs": ["$TURBO_DEFAULT$", "!**/*.md"]` (verify nothing imports .md first). `typecheck` task: `outputs: ["tsconfig.tsbuildinfo"]` + TS `incremental` with `"tsBuildInfoFile": "${configDir}/tsconfig.tsbuildinfo"` (bare `incremental` breaks tsup DTS, TS5074).

## CPU containment (the wrapper)

One `cpu-limit.sh` wraps every heavy invocation (`templates/cpu-limit.sh`). Chain: `systemd-run --user --scope --slice=build.slice -p CPUQuota/-p MemoryHigh/Max/CPUWeight/IOWeight` → `nice 19` → `ionice -c 3` → `chrt --idle 0` → `taskset`. Plus: `GOMAXPROCS` (esbuild ignores cpusets), `VITEST_MAX_FORKS`+`VITEST_MAX_THREADS` (default pool is forks — capping threads alone is inert), Node heap cap, affinity-aware `nproc` vs `nproc --all` (outer restriction IS the budget — don't subtract reserve twice).

- **SCHED_IDLE (`chrt --idle 0`) is for LOCAL INTERACTIVE runs ONLY. CI MUST set `BUILD_SCHED_IDLE=0`.** Proven failure: busy box (load 55–80) starves an idle-scheduled gate → wall-clock test timeouts fire, DIFFERENT victim package each run. Signature to recognize: hook/test timeouts + trivial tests taking 10x normal + rerun fails elsewhere.
- **Machine-wide flock serialization** in the wrapper: fd held for process lifetime, `BUILD_LOCK_HELD` nested-guard, `BUILD_NO_LOCK` opt-out, unwritable lock path → warn + run unserialized (fail-open on the LOCK, never on correctness). Second gate queues, then replays warm cache — serialization is itself a speed win.

## Test-suite mechanics (DB-backed suites)

- **Timeouts are factory-owned, sized to contention, one source of truth.** Shared vitest factory preset carries `hookTimeout: 300_000, testTimeout: 180_000` for PG-backed suites. NEVER per-package bumps — with mergeConfig, a per-package 30s silently CAPS the factory value. Contention stretches wall-clock ~10x (measured: <8s-idle tests ran 77s). Timeouts bound genuine hangs, not speed.
- PG suites: `singleFork` per package → ≤1 DB per package, total DBs ≤ task concurrency.
- Embedded PG: **socket-only** (`listen_addresses=` + `-k dataDir`, connect `host: dataDir`). TCP port pre-allocation = TOCTOU race under concurrency. Structural fix, not retry.
- **initdb template cache** (`@tooling/pg-template` `initdbCached()`): per-boot 1675ms→534ms. Design invariants — cache key MUST include initdb `--version` AND binary size+mtime (version string survives package rebuilds) AND user+args AND full `LC_*`+`LANG`+`TZ` (initdb bakes TZ into postgresql.conf — proven stale-template bug); atomic `rename` = lock-free race (EEXIST/ENOTEMPTY/EPERM = use winner); refuse non-empty dataDir (fs.cp silently overlays); ANY cache error → warn + real initdb (fail-open to correctness).

## Verification doctrine

- Every deflake/cache fix ships with a test that **fails on the old code** (prove RED before trusting GREEN).
- Adversarial review before merge; reviewer must VERIFY claimed fixes discriminate, not just read the diff.
- After merge, capture the timing number as proof; report cold-vs-warm honestly (don't attribute global-hash cold runs to a broken cache).

## Gate content (coverage ≠ speed)

Fast gate can still enforce nothing. After perf work, audit WHAT gate checks vs project law. Recurring gap classes (all found live in platform, 2026-07):

1. **Orphaned checks.** Script exists (`check:*` in package.json, or tools/*.mjs) but absent from the gate chain → never runs, law silently unenforced. Audit: every check script reachable from `gate`/`verify`; wire or delete.
2. **Hand-maintained lists drift.** Import-smoke list covered 34/63 packages — every new package ships un-smoked unless someone remembers to edit the list. Fix: GENERATE the check from machine-readable source (walk every package's `exports` map, import every subpath from dist). Catches the proven bug class: subpath declared in `exports`, tsup entry missing → dist 404s while src-importing tests stay GREEN.
3. **Prose-only law.** Packaging conventions (ESM fields, `sideEffects:false`, dep classification, version ceiling) documented but not machine-enforced → regression by drift. Fix: one conformance script over every `packages/*/package.json`, wired into `verify`.
4. **Missing process gate.** PR touches `packages/*/src` with no changeset → merges fine, silently never publishes. Fix: `changeset status --since <base>` step in PR gate (empty changesets stay valid).
5. **Pre-public security.** Self-hosted runner: fork-PR guard `if: github.event.pull_request.head.repo.full_name == github.repository` on EVERY job (see Topology) + pinned gitleaks step.

Every new check: prove RED on synthetic violation, revert, prove GREEN — before calling it wired.

## Three-tier enforcement (place each law at its cheapest catcher)

| Tier | Catches | When |
|---|---|---|
| Static content gate (slopgate) | mechanical code-law violations (forbidden imports, comment law, `.only`, a11y patterns) | write/commit time |
| Conformance + process checks | packaging law, registry sync, changesets, secrets | CI (PR gate) |
| Review agents (seam/security) | semantic law (spec conformance, authz, money invariants) | pre-merge review |

Tiers complement — NEVER duplicate a rule across tiers; put it at the earliest tier that can decide it statically.

**Static-gate wiring pattern (slopgate or equivalent):** (1) self-test MUST pass before trusting the gate; (2) per rule: RED fixture + GREEN fixture, zero false positives on current repo; pre-existing real hits → ratchet baseline (block NEW, don't demand repo-wide cleanup); (3) wire three points: pre-commit staged-files fast tier + agent edit-hooks (write-time) + CI backstop over changed-files vs base.

## Diagnosis table

| Symptom | Cause → fix |
|---|---|
| Hook/test timeouts, different package each run, trivial tests 10x slow | scheduler starvation → CI `BUILD_SCHED_IDLE=0`; timeouts to contention budget |
| `Cached: 0` on a run that should be warm | global hash invalidated (lockfile/turbo.json) → expected, no action |
| `<tool>: not found` in fresh/rebased worktree | stale node_modules → `pnpm install` before gate |
| esbuild `all goroutines are asleep - deadlock!` / silent vitest exit 1 | corrupt node_modules (nondeterministic hoist, esp. after a codex-run install) → `rm -rf node_modules && pnpm install` |
| `postgres exited 1` under gate concurrency | TCP port TOCTOU → socket-only harness |
| gh merge "error" but PR actually merged | gh local-checkout failure → wrapper re-queries PR state |
| CI queued forever, no error anywhere, land gate times out with no check started | the scale-set listener died — nothing polls GitHub → `bash modules/ci/arc-watchdog-status.sh`; never re-diagnose by hand. Runs queued during the outage are never assigned: `gh run cancel <id> && gh run rerun <id>` |
| ONE branch's pre-push slow / runs full gate, others fast | branch (or worktree) predates the CI stack — old lefthook/cpu-limit committed → rebase onto main, `pnpm install` to resync hooks |

## Replicating to a new project (the template question)

Portable as-is (~80%, all in `templates/`): wrapper script, flock, vitest factory, safe-merge, workflow shapes, contract test, pg-template helper. Per-project adaptation (thin): package manager + task runner names, test framework, runner labels, which suites are DB-backed, whether runner shares the dev box (if dedicated: skip ACL cache sharing + SCHED_IDLE entirely; keep everything else). One-time per MACHINE: cache dir + ACLs, `build.slice` unit, runner user. Ordered checklist + per-file adaptation notes: `references/primitives.md`.