---
name: ci-cd-expert
description: Set up or fix efficient CI/CD on a self-hosted runner sharing a dev workstation — PR-gated trunk, shared build cache, CPU containment, flaky-gate diagnosis, browser/E2E matrix gates (sharding, completeness accounting, fake-green detection). Use when adding CI to a project, when a gate is slow/flaky/starving the machine, when a gate reads green but catches nothing, or when replicating the platform repo's CI stack elsewhere.
---

# CI/CD Expert — self-hosted gate on a shared dev box

Audience: AI coding agents first. Battle-tested in `platform` repo (PR#67→#81, 2026-07). Canonical generic primitives are VENDORED in this skill's `templates/` — **copy from there, don't re-derive and don't hunt for the origin repo** (catalog + replication checklist: `references/primitives.md`). A project's copy is an instance; back-port material improvements to `templates/`.

## Git/CI fatigue contract

MUST follow `/home/user/Projects/0 DOCS/GIT_FATIGUE.md` §12. Valid receipt/log for current tree = proof; NEVER rerun broad typecheck/build/test. Mass diagnostics: capture once, cluster, repair ALL, ONE replacement run. Exhaustive gate runs once on designated executor; workstation hooks MUST finish ≤10s and only hand off. User surface MUST be only `Preview ready` or `Needs product decision`. MUST read §12 before adding or changing any gate, hook, CI step, or ceremony.

## Prime directive (owner constraint)

**MINIMIZE TOTAL ON-BOX COMPUTE. NEVER widen the core budget to go faster.** Levers, in order: (1) don't recompute — share cache; (2) don't collide — serialize; (3) don't recompute per-boot — template caches; (4) only then tune concurrency. Machine-wide aggregate cap = systemd user slice (`build.slice`); per-run knobs live in one wrapper script.

## Greenfield build order (NEW CI — follow in order, do not defer any rung)

Every rung below exists because skipping it cost a later remediation round. Build them in, don't optimize them in.

1. **Slice first.** Create the systemd user slice with a FINITE `CPUQuota`, `MemoryHigh`, `MemoryMax`, `IOWeight` before the first workflow. Every heavy step goes through one wrapper script from day one. Set `BUILD_SCHED_IDLE=0` in CI at the same moment.
2. **No `actions/cache` on self-hosted.** Gate it to hosted runners on the first commit that adds it.
3. **Decompose before you have a duration problem.** Design every long suite as N shards keyed by shard index on a shared label, with the shard list emitted by the planner. Retrofitting host-pinned jobs into a shard matrix is a schema change that touches planner, runner, aggregate, artifact names and tests simultaneously.
4. **Receipt schema on day one:** `expected`, `executed`, `notStarted`, terminal status. Aggregate validates the UNION. Never let a receipt infer a result it did not observe.
5. **Generate every manifest** (routes, specs, packages) from the filesystem with a `--check` drift mode. Hand-curated lists start drifting the week they are written.
6. **Overlap the critical path** from the start: provisioning in step 1, install/build parallel, fail-closed barrier.
7. **Derive parallelism from the fixture data**, never a literal.
8. **Wire every check script into the gate chain** as it is written. An orphaned `check:*` enforces nothing.
9. **Watch each gate FAIL once** before trusting it. Inject the violation, see it block, revert.
10. **Instrument from run 1:** per-step timings and per-test durations as machine-readable artifacts. Without them every later optimization is guesswork, and you will fabricate a number under pressure.

Anti-pattern that generates remediation rounds: shipping a gate that is green, fast and unmeasured, then discovering it never executed what it claimed. Rungs 4, 9 and 10 are the ones that prevent it.

## Topology (decision rules)

- Gate PRs **pre-merge** (`pr-gate.yml` on `pull_request`). Post-merge `main-gate.yml` exists ONLY as delivery trigger — NEVER as the safety net, NEVER with an auto-revert bot.
- Free-tier private repo = no server-side branch protection. Enforce in a **fail-closed merge wrapper** (`safe-merge.mjs`): refuses unless the named gate check is SUCCESS; raw `gh pr merge` blocked by PreToolUse hook. Wrapper must be authoritative on PR STATE, not gh exit code (gh's post-merge local checkout fails in worktree-heavy repos → false negative).
- Pre-push hook = ≤10s handoff only; NEVER typecheck/build/test/e2e. Designated executor owns one authoritative receipt-backed gate.
- Workflow contract = executable: assert workflow topology in a script test (`ci-artifact-gate.test.mjs` pattern) so drift fails the gate.
- Self-hosted runner + `pull_request` = RCE from fork PRs. Gate fork PRs before repo goes public.

## Shared machine cache (biggest win)

ONE turbo/build cache machine-wide: `/var/cache/<project>-turbo`, ACL-shared between dev user and runner user (`setfacl -R -m u:<runner>:rwX -m d:u:<runner>:rwX -m d:u:<dev>:rwX`). Wrapper auto-prefers it when present+writable.

**`actions/cache` on a self-hosted runner is an ANTI-PATTERN. GATE it to hosted runners; do NOT delete it outright.**

The store is already persistent local disk. Wrapping it re-tars and re-untars every job for zero benefit — and a store larger than GitHub's ~10GB per-repo limit can never even hit.

```yaml
# target — keep the step, gate it
- if: runner.environment == 'github-hosted'
  uses: actions/cache@v6
```
Deleting the step breaks any job in the same repo that DOES run on `ubuntu-latest`. Check every workflow's `runs-on` before removing. A reviewer demanding outright deletion is wrong unless the repo has zero hosted jobs.

**Measured (multideal, 2026-08):** composite setup 77.4s avg × 516 jobs = **665.4m = 16% of all CI host time**. Bare `pnpm install --frozen-lockfile` 8.3s, `setup-node` 7.9s → ~60s/job pure cache overhead. Store 8.5G + Playwright browsers 4.2G = 12.7G, over the limit. `Post Run` averaged 1.1s → the cost is on RESTORE, not SAVE.

Before adding any cache step to a self-hosted job, verify the janitor does not prune the path. Prune scripts that target `/tmp` and `$HOME/builds` do NOT touch `~/.local/share/pnpm/store` or `~/.cache/ms-playwright` — those are already safe and need no wrapper. Confirm the store and the build dir share an fsid, so the package manager hardlinks instead of copying.

- Content-hashed cache is portable across executor checkouts and users. Workstation hooks NEVER warm it with heavy checks.
- **Proof numbers:** warm PR gate 1m51s, post-merge main gate 2m02s, vs ~15m cold (vs ~1h original).
- Lockfile or turbo.json change = GLOBAL hash invalidation → full cold run. Expected, not a bug. Don't chase it.
- Trim inputs: `"inputs": ["$TURBO_DEFAULT$", "!**/*.md"]` (verify nothing imports .md first). `typecheck` task: `outputs: ["tsconfig.tsbuildinfo"]` + TS `incremental` with `"tsBuildInfoFile": "${configDir}/tsconfig.tsbuildinfo"` (bare `incremental` breaks tsup DTS, TS5074).

## CPU containment (the wrapper)

One `cpu-limit.sh` wraps every heavy invocation (`templates/cpu-limit.sh`). Chain: `systemd-run --user --scope --slice=build.slice -p CPUQuota/-p MemoryHigh/Max/CPUWeight/IOWeight` → `nice 19` → `ionice -c 3` → `chrt --idle 0` → `taskset`. Plus: `GOMAXPROCS` (esbuild ignores cpusets), `VITEST_MAX_FORKS`+`VITEST_MAX_THREADS` (default pool is forks — capping threads alone is inert), Node heap cap, affinity-aware `nproc` vs `nproc --all` (outer restriction IS the budget — don't subtract reserve twice).

- **SCHED_IDLE (`chrt --idle 0`) is for LOCAL INTERACTIVE runs ONLY. CI MUST set `BUILD_SCHED_IDLE=0`.** Proven failure: busy box (load 55–80) starves an idle-scheduled gate → wall-clock test timeouts fire, DIFFERENT victim package each run. Signature to recognize: hook/test timeouts + trivial tests taking 10x normal + rerun fails elsewhere.
- **Machine-wide flock serialization** in the wrapper: fd held for process lifetime, `BUILD_LOCK_HELD` nested-guard, `BUILD_NO_LOCK` opt-out, unwritable lock path → warn + run unserialized (fail-open on the LOCK, never on correctness). Second gate queues, then replays warm cache — serialization is itself a speed win.

## Utilization: decompose, never build a scheduler

**Measure concurrency as MEAN-WHILE-ACTIVE, never as peak.** Peak tells you the cap was reachable once; the mean tells you what the estate actually does.

Measured (multideal, 2026-08): peak **6/6** slots, mean-while-active **2.10/6**, idle **91%** of the window. Peak alone would have read as "saturated".

Decision rule from that pair:
- **High peak + low mean → the estate is starved of DISPATCHABLE UNITS. Split long jobs.** Every lever that decomposes beats every lever that schedules.
- Low peak + low mean → the trigger never fires (see Unreachable gate).
- High peak + high mean → genuinely capacity-bound; only then buy capacity.

**NEVER build a queue orchestrator to fix low utilization.** The CI provider's matrix dispatcher already is one, and it is deterministic. A hand-rolled scheduler on top re-implements it worse and adds a component that can itself fail green. If jobs are not spreading, the cause is host affinity or job granularity — fix those.

**Fail-closed MUST NOT mean fail-on-a-condition-you-could-repair.** A wrapper that aborts because a lock/cache/state DIRECTORY is absent is a defect: creating the directory preserves the guarantee in full. Reserve fail-closed for conditions that genuinely cannot be made safe (permission denied, read-only fs, unverifiable provenance).

```bash
# reject: aborts CI on a directory it could create
[ -d "$lock_parent" ] || { echo FATAL >&2; exit 78; }
# target
mkdir -p "$lock_parent" 2>/dev/null || true
# ...then the real writability test, still fail-closed under CI
```
Measured cost of getting this wrong: every job on both hosts died at `exit 78` before one test ran, because `~/.cache/ci-box` did not exist.

**A machine-wide serialization lock and sharding are mutually exclusive.** The lock fd is held for the whole process lifetime, so N shards on one host serialize to N × single-shard duration. When you shard, drop the lock and let the slice carry the aggregate ceiling — but verify the slice bounds MEMORY and IO, not only CPU (`MemoryMax`, `MemoryHigh`, `IOWeight`), before trading serialization away. Audit that every workflow using the wrapper agrees on this setting; the one that disagrees is the bug.

## Pipeline shape: overlap the critical path

- **Start slow provisioning in step 1, then install/build in parallel, and join on a fail-closed barrier.** DB branch creation, container pulls and seeding do not depend on `node_modules`. Barrier contract: a state dir with `status` (`ready`/`failed`), `error` and `env` files; the consumer polls with a finite deadline, aborts on `failed`, and aborts on timeout. Never let the barrier fall through on a missing file.
- **Disposable test Postgres: put the data dir on tmpfs and turn durability OFF** — `fsync=off`, `synchronous_commit=off`, `full_page_writes=off`. The cluster is thrown away; crash safety buys nothing. Keep a disk fallback with a free-space floor check.
- **Clone shards with `CREATE DATABASE … TEMPLATE`** instead of re-seeding per shard. Terminate template sessions first, and withhold the template URL from consumers until the clone completes.
- **Use the blob reporter + a single merge** instead of per-shard HTML. Upload heavy evidence `if: failure()` only; receipts and duration JSON stay `if: always()`.
- Artifact retention: evidence on failure only is a large, free saving. Never make the RECEIPT conditional — that is the thing you need on a green run.

## Parallelism ceilings are usually DATA, not CPU

Before raising a worker count, find what the workers contend over. If tests share seeded fixtures, the ceiling is the number of independent seeded actor sets — raising `workers:` past it produces cross-test interference, not speed.

**Derive the worker count from the fixture array so the two cannot drift:**
```ts
export function playwrightWorkerCount(factoryMode: boolean) {
  return factoryMode ? FACTORY_ACTOR_SETS.length : 1;
}
```
Raising parallelism then means seeding another actor set — a data change with an obvious blast radius — not editing a magic number. Give each worker its own actor set via a `scope: 'worker'` fixture keyed on `testInfo.parallelIndex`, and make an out-of-range index THROW.

**Do NOT build a runtime load-sensing worker controller.** The cgroup already arbitrates CPU/memory/IO; a userspace load-reader fights it and adds nondeterminism to a gate. Test a source-string assertion (`expect(config).toContain('workers: 2')`) is NOT a test of this — assert the exported function's behaviour instead.

## Test-suite mechanics (DB-backed suites)

- **Timeouts are factory-owned, sized to contention, one source of truth.** Shared vitest factory preset carries `hookTimeout: 300_000, testTimeout: 180_000` for PG-backed suites. NEVER per-package bumps — with mergeConfig, a per-package 30s silently CAPS the factory value. Contention stretches wall-clock ~10x (measured: <8s-idle tests ran 77s). Timeouts bound genuine hangs, not speed.
- PG suites: `singleFork` per package → ≤1 DB per package, total DBs ≤ task concurrency.
- Embedded PG: **socket-only** (`listen_addresses=` + `-k dataDir`, connect `host: dataDir`). TCP port pre-allocation = TOCTOU race under concurrency. Structural fix, not retry.
- **initdb template cache** (`@tooling/pg-template` `initdbCached()`): per-boot 1675ms→534ms. Design invariants — cache key MUST include initdb `--version` AND binary size+mtime (version string survives package rebuilds) AND user+args AND full `LC_*`+`LANG`+`TZ` (initdb bakes TZ into postgresql.conf — proven stale-template bug); atomic `rename` = lock-free race (EEXIST/ENOTEMPTY/EPERM = use winner); refuse non-empty dataDir (fs.cp silently overlays); ANY cache error → warn + real initdb (fail-open to correctness).

## Measure before optimizing (MUST — first rung, always)

NEVER tune a gate from its total duration. Derive the phase split from the run's OWN machine-readable report before touching anything.

1. Step timings → fixed overhead vs execution (`gh api .../actions/runs/<id>/jobs`).
2. Per-test duration + count, grouped by project, from the reporter's JSON — not from log scraping.
3. Compare max observed test duration against the configured timeout BEFORE tuning timeouts.

Decision rules from those three numbers:
- max observed ≪ timeout → timeout tuning is NOT a lever. Do not touch it.
- mean per-test small × huge count → parallelism lever.
- fixed overhead dominant → cache/reuse lever, NOT parallelism.
- Parallel shards duplicate COMPUTE, not WALL TIME. N independent setups still cost 1× setup on the critical path. Do NOT reject sharding on duplicated-setup grounds.

Measured example (multideal ui-sweep, 2026-08): 31:03 total = 1:05 setup + 3:29 DB branch/seed + 1:18 build + 25:05 execution; 1,284 cells, mean 4.18s, max 22.1s vs 45s timeout. => parallelism + per-cell writes were the levers; timeouts were not.

## Browser/E2E matrix gates

Route-×-role-×-locale-×-viewport suites. Rules here are ADDITIONAL to the sections above.

**Sharding**
- **Shard by SHARD INDEX on a shared runner label. NEVER pin a shard to a named host.** Pinning defeats the dispatcher: a shard routed to a busy host waits while another host sits idle. Emit the shard-id list from the planner as a job output and drive `strategy.matrix` from it via `fromJSON`, so shard count lives in exactly ONE place.
  - Earlier guidance said "shard by physical host". That was a workaround for cpuset contention and port collision. Both have proper fixes (a finite `build.slice` quota; shard-id in the port derivation). Host pinning is the wrong tool — it trades away all dispatch balance to solve a problem the slice already solves.
  - Constrain concurrency with the runner label + slice quota, NOT with host affinity.
- Port derivation MUST include the shard id. Two matrix jobs on one host that derive ports from repo+workflow alone collide.
- **Emit only NON-EMPTY shards.** A fixed shard count (`N=12`) over uneven work produces empty bins; the runner rejects an empty shard and aggregate cannot prove a receipt for a cell that had no work. Never dispatch a shard the plan did not declare.
- Bin-pack **longest-processing-time-first**: sum each group's duration, sort DESCENDING by weight with a deterministic id tie-break, assign to the lowest-weight shard. Sorting groups lexically and assigning to the lightest bin is NOT LPT and packs badly.
- **Indivisible groups.** Where the runner executes whole spec files, the packing unit is the spec (or spec+project), never the individual cell. A spec marked `serial-external` needs every one of its projects in ONE shard. A splitter that ignores grouping silently corrupts execution.
- Correctness keys on SHARD ID. Keep the real hostname as metadata only. Shard id MUST appear in every internal filename and every artifact name — blob/report names built from `host:project` collide across shards on merge.
- Per shard: unique artifact names AND unique run markers.
- Cross-route aggregate checks (nav-consistency fingerprints, dedup, global uniqueness) CANNOT run per shard. Merge raw inputs, fold ONCE globally.

**Union completeness (the part that must not be got wrong)**
Per-shard markers that are each internally consistent prove NOTHING about the run. Validate the UNION against the manifest:
- shard cell sets DISJOINT (no cell twice, none claimed twice),
- every shard id present (missing shard MUST fail, never silently shrink the run),
- executed count == expected count exactly,
- one aggregate fingerprint per cell.

// DO NOT: sum per-shard pass counts and call it green.

**Coverage completeness**
- GENERATE the route manifest from the filesystem + `--check` drift mode. Hand-curated manifests silently omit new pages (measured: 173/177 covered, 4 pages absent, nobody noticed).
- One-role-per-route manifests hide every authed defect on public routes. Fan out roles explicitly; every exclusion carries a machine-readable reason.
- Dynamic-route fixtures MUST resolve from seeded reference data. NEVER by crawling a source page — a blocked source page silently unresolves the fixture.
- Unresolvable fixture MUST fail loudly naming the route. NEVER degrade to skip or pass.
- Lazily-hydrated islands (`client:visible`, `client:idle`) are INVISIBLE unless forced to mount. A cell that only asserts "page loaded" proves nothing about them.
- Expected-error declarations MUST fail when STALE (declared error stops firing), not only when an undeclared error appears.

**Determinism (never buy it with time)**
- Replace every fixed sleep with an awaited signal + deadline. Deadline expiry MUST be a FAILURE naming what never arrived. Measured cost of one fixed 15s hydration wait: 112 cells, 7+ wall-clock minutes.
- Connectivity-dependent features (WebSocket/SSE) expose CONNECTED / DISABLED explicitly. Ambiguous silence MUST fail.
- `repeat-each`/retries are CONFIDENCE checks, NEVER fixes. Never ship one as a determinism fix.
- DO NOT reuse browser contexts across cells to save time — masks role/session leakage, the bug class the suite exists to catch.

**Per-cell cost (apply all; none reduce coverage)**
- Launch flags: `--disable-dev-shm-usage` (load-bearing where `/tmp` is tmpfs — shm pressure with parallel workers crashes renderers and reads as flake), `--disable-gpu`, `--disable-extensions`, `--disable-background-timer-throttling`, `--disable-renderer-backgrounding`, `--disable-backgrounding-occluded-windows`.
- Artifacts: `trace: 'on-first-retry'`, `video: 'off'`, `screenshot: 'only-on-failure'`. Never `trace: 'on'` in CI.
- `page.route`-abort third-party/analytics/font hosts. Use a NAMED BLOCKLIST, never a blanket cross-origin block — over-broad blocking silently changes what is under test.
- Serve a PREBUILT preview. A `webServer.command` of `build && preview` rebuilds inside the Playwright startup timeout, so a slow build reads as a server-start failure. Move the build to its own step; `webServer` only serves.
- Install only the browsers the matrix uses, per job.

**Prove read-only, never assume it**
A suite documented "live-readonly" is an untested assumption. Run the matrix under a SELECT-only DB role; passing IS the proof. Only then may one seeded DB branch serve N shards.
Measured: every authenticated cell performed 9 writes before its SELECT on an owner-level connection, in a suite documented read-only.

## Fake-green failure modes (audit for ALL of these)

A gate reading green proves nothing until each is excluded. Every one below was live in one repo simultaneously.

| Mode | Signature | Fix |
|---|---|---|
| `continue-on-error: true` on the gate job | job conclusion `success` with hundreds of failures in the log | delete it; assert absence in the workflow contract test |
| Gate triggers only on a dead branch / manual dispatch | last run months old; workflow "exists" | trigger on the path that ships code |
| Reporter records only `passed` cells | `executed` field understates; never compared | record TERMINAL cells (passed/failed/timedOut/interrupted) |
| expected vs executed computed, never compared | ratchet accepts marker status `failed` | compare; fail when `executed < expected`, naming missing cells |
| Aggregate check folded AFTER validation | findings land in an artifact, exit code already 0 | fold BEFORE validation; findings participate in pass/fail |
| Changed-files gate maps shared deps to zero routes | one-level dependency mapping | transitive mapping, or full matrix |
| Unit test PINS the unsound behavior | `expect(run('failed').code).toBe(0)` | fix test + state why the change is legitimate |
| Receipt SYNTHESIZES results it never observed | `executed: 0` alongside `failed: 451`; duration wildly over the raw report's | report `executed` / `notStarted` / `incomplete` as DISTINCT facts; never expand "expected" into "failed" |
| Injected auth state applied suite-wide | anonymous + auth-boundary specs silently authenticated; protected-route tests pass without proving protection | `storageState` is OPT-IN per spec; keep a real-login lane covering login, logout, cookie issuance, expiry, revocation, redirect |
| Gate skipped on a self-written receipt | a host-writable file marks a SHA "already passed" | provenance MUST be the CI provider's own authenticated immutable run/check result, bound to repo + exact 40-char SHA + workflow + job + contract version + conclusion + artifact digest |

**Cross-check every receipt against the raw reporter output in the SAME artifact.** Measured: a receipt claimed `expected: 451, executed: 0, counts.failed: 451` while the raw Playwright JSON beside it showed 6 specs, 2 passed, 14 failed, 5.0 minutes. ~16 tests ran; the receipt reported 451 failures. A receipt is a claim, not evidence.

**Run got faster AND quieter = broken gate, NEVER progress.** Check executed-vs-expected on EVERY run. Measured: a blanket capability-disable skipped all 1,286 cells and produced a clean 5-minute run.

**Every gate MUST be seen to FAIL before it is trusted.** Inject the violation, watch it block, revert. A guard nobody saw block is not a verified guard.

## Stall detection (a hung job is the most expensive failure there is)

A red job costs one rerun. A HUNG job burns a runner slot for the whole timeout and is only noticed when a human happens to look — measured on this estate: 12 hangs / 1142 min in 3 days, worst 8.6h. Every long-running CI step and every agent dispatch MUST be supervised.

**NEVER infer liveness from process existence.** A crash-looping, deadlocked or socket-blocked child holds a live PID for its entire timeout while doing nothing. `pgrep`/`pidof`/`kill -0` are NOT liveness. Judge **PROGRESS**.

Canonical implementation: `~/Projects/overdeck/modules/monitor/bin/stall-guard` (+ `README-stall-guard.md`). Do not re-derive it.

**Sample four orthogonal signals across the whole process TREE, not the direct child:**

| Signal | Source | Why it must be there |
|---|---|---|
| `out` | captured log size | the only semantic signal |
| `cpu` | `/proc/<pid>/stat` utime+stime, recursive | a silent long compile is NOT a hang |
| `io` | `/proc/<pid>/io` **rchar+wchar**, recursive | `read_bytes` MISSES socket traffic entirely |
| `beat` | mtime of a heartbeat file | declared legitimate waits (locks, queue slots, barriers) |

**Two kill rules, both required:**
1. every tracked signal flat for the idle window → PASSIVE hang (blocked socket, deadlock, `sleep`).
2. `out` alone flat for a much longer output ceiling, whatever cpu/io do → ACTIVE hang (spin loop, retry loop). Rule 1 ANDs the signals, so ONE perpetually-moving signal vetoes detection forever without rule 2.

**Accumulate POSITIVE deltas only — never compare raw endpoints.** Counters are not monotonic: summed cpu/io FALL when a child exits, `out` falls on log truncation. A decrease then reads as "progress" and silently disables the whole detector. Confirmed live in real traces.

**Kill the enumerated tree AND the process group, then WAIT for death before `SIGKILL`.** `killpg` alone misses grandchildren that called `setsid`; an unconditional sleep before `SIGKILL` lets a recycled PGID absorb the signal.

**Failure of the watchdog MUST be loud.** `except: return` in a sampling loop restores exactly the silent multi-hour blindness the tool exists to remove. Report UNGUARDED to stderr + desktop notification.

**The residual class, stated honestly:** a process emitting NOVEL output forever (`echo waiting for lock...` in a loop) is indistinguishable from progress by any content-agnostic signal. Give it a **non-killing advisory** at ~p99.9 of the measured duration distribution. Wall-clock is the only rule with a real false-positive cost — never let it kill by default.

**Size every window from the measured distribution, never a guess.** 5918 real agent runs: p50 5.7m, p90 19.5m, p99 57m, p99.9 175m, max 531m — a 30m wall cap would false-positive on 4.56% of legitimate runs. One real 11m47s reasoning run went **195s with zero output** while cpu/io never stopped: an output-only watchdog at 120s kills it.

**Both replay arms MUST pass before arming enforcement** — see Verification doctrine. Record every run's sample trace so the corpus grows into a standing regression check.

## Verification doctrine

- Every deflake/cache fix ships with a test that **fails on the old code** (prove RED before trusting GREEN). A test never observed failing proves nothing — neuter the fix, watch it go red, restore.
- **A detector needs BOTH arms.** `--expect quiet` over traces of runs that COMPLETED, and `--expect fired` over known-stuck ones. A detector that can never fire passes the no-false-positive arm alone — the exact defect that shipped here once already.
- Adversarial review before merge; reviewer must VERIFY claimed fixes discriminate, not just read the diff.
- After merge, capture the timing number as proof; report cold-vs-warm honestly (don't attribute global-hash cold runs to a broken cache).

## Gate content (coverage ≠ speed)

Fast gate can still enforce nothing. After perf work, audit WHAT gate checks vs project law. Recurring gap classes (all found live in platform, 2026-07):

1. **Orphaned checks.** Script exists (`check:*` in package.json, or tools/*.mjs) but absent from the gate chain → never runs, law silently unenforced. Audit: every check script reachable from `gate`/`verify`; wire or delete.
2. **Hand-maintained lists drift.** Import-smoke list covered 34/63 packages — every new package ships un-smoked unless someone remembers to edit the list. Fix: GENERATE the check from machine-readable source (walk every package's `exports` map, import every subpath from dist). Catches the proven bug class: subpath declared in `exports`, tsup entry missing → dist 404s while src-importing tests stay GREEN.
3. **Prose-only law.** Packaging conventions (ESM fields, `sideEffects:false`, dep classification, version ceiling) documented but not machine-enforced → regression by drift. Fix: one conformance script over every `packages/*/package.json`, wired into `verify`.
4. **Missing process gate.** PR touches `packages/*/src` with no changeset → merges fine, silently never publishes. Fix: `changeset status --since <base>` step in PR gate (empty changesets stay valid).
5. **Pre-public security.** Self-hosted runner: fork-PR guard `if: github.event.pull_request.head.repo.full_name == github.repository` on EVERY job (see Topology) + pinned gitleaks step.
6. **Unreachable gate.** Workflow is correct AND never fires (dead branch trigger, dispatch-only, schedule dropped). Audit: every gate reachable from the path that ships code. A gate the shipping path does not traverse enforces NOTHING.
7. **Documented-but-unenforced law.** A rule stated in this skill (or any doc) and absent from the workflow. Proven live: `BUILD_SCHED_IDLE=0` documented here since 2026-07, still unset in a 2026-08 gate → idle-scheduled under host load. Fix: assert the rule in the workflow contract test, not in prose.

Every new check: prove RED on synthetic violation, revert, prove GREEN — before calling it wired.

## Three-tier enforcement (place each law at its cheapest catcher)

| Tier | Catches | When |
|---|---|---|
| Static content gate (slopgate) | mechanical code-law violations (forbidden imports, comment law, `.only`, a11y patterns) | write/commit time |
| Conformance + process checks | packaging law, registry sync, changesets, secrets | CI (PR gate) |
| Review agents (seam/security) | semantic law (spec conformance, authz, money invariants) | pre-merge review |

Tiers complement — NEVER duplicate a rule across tiers; put it at the earliest tier that can decide it statically.

**Static-gate wiring pattern (slopgate or equivalent):** (1) self-test MUST pass before trusting the gate; (2) per rule: RED fixture + GREEN fixture, zero false positives on current repo; pre-existing real hits → ratchet baseline (block NEW, don't demand repo-wide cleanup); (3) wire three points: pre-commit staged-files fast tier + agent edit-hooks (write-time) + CI backstop over changed-files vs base.

## Diagnosis table

| Symptom | Cause → fix |
|---|---|
| Hook/test timeouts, different package each run, trivial tests 10x slow | scheduler starvation → CI `BUILD_SCHED_IDLE=0`; timeouts to contention budget |
| `Cached: 0` on a run that should be warm | global hash invalidated (lockfile/turbo.json) → expected, no action |
| `<tool>: not found` in fresh/rebased worktree | stale node_modules → `pnpm install` before gate |
| esbuild `all goroutines are asleep - deadlock!` / silent vitest exit 1 | corrupt node_modules (nondeterministic hoist, esp. after a codex-run install) → `rm -rf node_modules && pnpm install` |
| `postgres exited 1` under gate concurrency | TCP port TOCTOU → socket-only harness |
| gh merge "error" but PR actually merged | gh local-checkout failure → wrapper re-queries PR state |
| Gate job green, log full of failures | `continue-on-error: true` on the job → GitHub rewrites conclusion to `success` → delete it |
| Run suddenly much faster and much quieter | cells skipped, not fixed → check executed-vs-expected before believing it |
| Same failure signature on hundreds of cells | ONE broken precondition, not N bugs → add fail-fast preflight asserting env + each role sentinel |
| E2E flake that retries "fix" | test races an event it does not control → awaited signal + deadline-is-failure; NEVER `repeat-each` |
| Cells fail `no runtime link found` / unresolved fixture | fixture discovered by crawling a blocked page → resolve from seeded reference data |
| ONE branch's pre-push slow / runs full gate, others fast | branch (or worktree) predates the CI stack — old lefthook/cpu-limit committed → rebase onto main, `pnpm install` to resync hooks |

## Replicating to a new project (the template question)

Portable as-is (~80%, all in `templates/`): wrapper script, flock, vitest factory, safe-merge, workflow shapes, contract test, pg-template helper. Per-project adaptation (thin): package manager + task runner names, test framework, runner labels, which suites are DB-backed, whether runner shares the dev box (if dedicated: skip ACL cache sharing + SCHED_IDLE entirely; keep everything else). One-time per MACHINE: cache dir + ACLs, `build.slice` unit, runner user. Ordered checklist + per-file adaptation notes: `references/primitives.md`.