# bin/runplan Worktree Isolation — Design ## Problem `src/runner.js` (the engine behind `bin/runplan`) runs every task directly against `repoRoot`: `worktree = repoRoot` unconditionally in `main()`, `runWrapper()` passes `--workspace repoRoot` to the coder/reviewer/fixer wrapper, `commitTask()` runs `git add -A && git commit` straight in `repoRoot`, and `gate0`/`risk` checks run against `repoRoot` too. There is no per-task branch and no per-task worktree — unlike the Workflow-controller engine (`~/.claude/workflows/run-plan.js`), which isolates each task on its own worktree (`/tmp/wt--`) and branch (`plan/--`), and only integrates onto `plan/` after that task's gates pass. Two concrete failures follow from this: 1. **Concurrency race, not just a safety gap.** `runPool()` already dispatches up to `defaultConcurrency()` (os.cpus()-2, capped 8) tasks from the same wave concurrently. With `worktree = repoRoot` for every task, two same-wave tasks write to the same working tree and commit onto the same branch at the same time — the exact case `dag-parallel` plans (the recommended default per `[[plan]]` whenever a wave holds ≥2 disjoint tasks) produce. 2. **The documented Land step is a no-op.** `run-plan` skill's "Harness engine path" already documents step 4: `bash /.claude/scripts/ship.sh land` after `bin/runplan` finishes, landing per the frozen method (`pr`/`merge-to-main`/`deploy-verify`). But today `commitTask()` commits directly onto `base_branch` (normally `main`) as each task finishes — there is nothing left for `ship.sh land` to land; the documented step is vestigial. `lib/gates.sh`'s `gate0` CLI already accepts `` and `` as distinct positional args (`gates.sh gate0 `) — `runner.js` just always passes `repoRoot` for both. This is strong evidence the split was anticipated and never wired up, not an intentional single-tree design. ## Is a second engine with this machinery justified? Two engines now converge on near-identical isolation (worktree+branch per task, integration branch, serialized merge). Worth asking why `bin/runplan` doesn't just delegate to the Workflow controller instead of reimplementing its topology. It can't: `~/.claude/workflows/run-plan.js` lives outside this repo and is driven by the `Workflow` tool inside a Claude session — there is no way for a plain Node CLI process (`bin/runplan`, invoked from cron, CI, or a headless shell with no Claude session at all) to call into it. `bin/runplan`'s reason to exist is exactly that: a blocking, dependency-free CLI loop that runs the wave/gate0/journal/commit cycle without a Claude session in the loop. That's a real, distinct requirement, not an accident of two teams building the same thing twice — so the duplication here is deliberate, not DRY-violating. A simpler alternative was considered and rejected: force `concurrency=1` in `runner.js`, run all tasks serially against one `plan/` integration worktree, no per-task worktree/branch, no merge step, no conflict handling, no resume ladder. This fixes both problems in the "Two concrete failures" section above with far less code. It was rejected because `defaultConcurrency()` (os.cpus()-2, capped 8) and `runPool()`'s concurrent dispatch are existing, exercised behavior in `runner.js` today — collapsing to serial would be a silent regression for any plan whose wave-parallelism is load-bearing (a `dag-parallel` plan is explicitly the recommended default from `[[plan]]` whenever a wave has ≥2 disjoint tasks), not a neutral simplification. Full per-task isolation preserves that existing capability instead of removing it. ## Goal Give `src/runner.js` the same per-task worktree + branch isolation the Workflow controller has: each task implements/gates/reviews on its own worktree+branch; a serialized merge lands each green task onto a `plan/` integration branch/worktree; `base_branch` stays untouched until `ship.sh land` runs. This closes the concurrency race as a direct consequence (each task now writes to its own filesystem tree) and makes the skill's documented Land step real. ## Non-goals - No change to `lib/journal.sh`'s `VALID_STATES` or its JSONL record shape — this is a git-topology change inside `runner.js`, not a journal schema change. - No LLM-mediated merge (the Workflow controller dispatches codex for its integration merge because it is orchestrating subagents anyway). `runner.js` is a plain deterministic CLI loop with no subagent dispatch for orchestration-level git operations — the integration merge here is a deterministic `git merge`, run directly. - No change to `resolve-seat.sh` or the preset/adapter resolution ladder. - No change to `ship.sh` itself — it already lands whatever is on `plan/`; today that branch just happens to be empty/unused by `bin/runplan`. ## Naming — reuse the Workflow controller's convention exactly Same slug-derived paths as `~/.claude/workflows/run-plan.js`, so the two engines are visibly compatible and a human inspecting the repo sees one convention, not two: - Integration worktree: `/.wt--int`, checked out on branch `plan/`. - Per-task worktree: `/tmp/wt--`, checked out on branch `plan/--`. Both are **deterministic from `(slug, taskId)`** — never persisted in the journal. Resume recomputes the same paths; this is what makes resume work without adding journal fields. ## Git topology per task 1. **Worktree ensure — runs on EVERY non-terminal entry into a task, not only fresh lease.** A resumed task can enter the per-task loop with journal state already `implemented`/`gated`/`reviewed` (its `leased` record already written by a prior run), skipping straight to that state's branch — so ensuring the worktree exists cannot be gated behind the `leased` step alone; it must run unconditionally before ANY state branch, immediately after the "already done" check short-circuits out. Ensure the integration worktree exists first: if `intWt` is already present on disk, reuse it untouched. Else, check whether `plan/` already exists as a branch: if so, `git worktree add plan/` (attach, no reset — a prior run may have already integrated commits onto this branch even though its worktree dir was cleaned up); if not, `git worktree add -b plan/ `. **Never `git worktree add -B`** — force-resetting the branch on every fresh `worktree add` would silently drop any commits already merged onto `plan/` by a prior run whose worktree directory no longer exists (see Failure modes below). Read `intHead = gitHead(intWt)`. Ensure the task worktree exists using the identical exists-on-disk / branch-exists / branch-absent primitive: if `taskWt` exists on disk, reuse it (the resume case — branch may be ahead of `intHead`, see Resume below); else if `plan/--` exists as a branch (surviving a `/tmp` wipe, e.g. reboot, while the worktree dir did not), `git worktree add plan/--` (attach, no reset); else `git worktree add -b plan/-- `. `taskBase` becomes `intHead` computed at this point (was `gitHead(repoRoot)`) — its computation moves to AFTER the worktree-ensure step, since it now depends on `intHead`. **`git worktree add` (both the integration worktree, once, and every per-task worktree) runs under the SAME serialization lock as the merge step (step 4) — `git worktree add`/`remove` contend on `.git/worktrees` and fail if run concurrently, exactly like the Workflow controller's own `// A1 — lease serially` comment documents. Only implement/gate/review (step 2/3) run unserialized, inside `runPool`'s concurrency limit; every `git worktree` mutation, on both the integration and task trees, goes through one mutex.** The ensure step is skipped only when the task is already fully `done` — required so a completed, already-merged-and-removed task never re-creates its `taskWt` directory on a later resume (would otherwise leave a lingering, never-cleaned-up worktree). 2. **Implement / gate / review** (`implemented`/`gated`/`reviewed` states): every `dispatchWithFallback` call and every `gates.sh gate0`/`risk` call targets the task worktree, not `repoRoot`. `runWrapper()`'s `--workspace` argument becomes the task worktree path. `gates.sh gate0 ` — worktree and repoRoot now genuinely differ, exercising the split `gates.sh` already supports. 3. **Commit** (`commitTask`): `git add -A && git commit` runs inside the task worktree, not `repoRoot`. **`commitTask` must be idempotent on a clean tree**: if `git status --porcelain` is empty AND the task worktree's head is already ahead of `taskBase` (this task's work was already committed by a prior run that crashed before the merge landed — see Resume below), skip `git add`/`git commit` and return the existing head unchanged, rather than erroring on "nothing to commit." The `committed` journal record is appended **here, immediately after the commit (or the no-op-commit skip) — strictly before step 4's merge runs.** This ordering is load-bearing: resume's crash-detection (below) distinguishes "committed but merge never landed" from "not yet committed" purely by whether the task branch is an ancestor of `plan/`, which only works if a `committed` record with no corresponding merge is a reachable, expected intermediate state. 4. **Merge into integration** (separate step, run immediately after step 3 inside the same task iteration — not a new journal state): serialized under the same worktree-mutation lock as step 1 — "checkout `intWt`, `git merge --no-ff plan/--`". Fast-forward or clean merge (expected: same-wave tasks are file-disjoint by planning convention) → success, remove the task worktree (`git worktree remove `; keep the task branch for audit/resume, mirroring the Workflow controller). Conflict → fail closed: abort the merge, leave the task worktree and branch intact, raise the same kind of error `runGateLoop` already raises on a non-green gate (task ends up BLOCKED, not silently resolved). The journal's `committed` record's `head` field is the task branch head (the pre-existing convention — after a `--no-ff` merge the task head is always an ancestor of `plan/`, so `reconcileJournal()`'s reachability check passes either way; recording the integration head would add a field with no reconcile benefit). ## Changes to existing seams - `main()` / `createRunner()` context: add the integration worktree path and `plan/` branch name, computed once at startup (deterministic from `repoRoot`/`slug`, not passed in). - `reconcileJournal()`: reconcile against `plan/` (the integration branch), not `context.branch` (`base_branch`). `base_branch` is now purely the branch the integration branch was cut FROM, not the branch task commits land on. - `runTask()`: `taskBase` sourcing changes from `gitHead(context.repoRoot)` to `gitHead(intWt)`; all `gitHead`/`spawnPassthrough` calls that currently pass `context.repoRoot` as the working directory for implement/gate/review/commit steps pass the task worktree path instead. - `runGateLoop()` / fixer path: same worktree retargeting; `taskBase..head` risk diff range now diffs within the task worktree/branch. - End of `run()`: after all waves complete, no merge to `base_branch` happens — that remains `ship.sh land`'s job, unchanged. ## Resume behavior On `--resume`, `reconcileJournal()` runs once at startup, before any task is dispatched, checked against `plan/` (the integration branch — see Changes to existing seams). It downgrades any task whose journal `committed` record has a head unreachable from `plan/` back to that task's last prior non-`committed` state — the existing `lib/journal.sh` reconcile/downgrade mechanism, retargeted. This is the SOLE crash-mid-merge recovery mechanism; there is no separate "is this task committed-but-unmerged" check in the per-task loop. Its effect: a resumed task's journal state is never `committed`-with-unmerged-head by the time its turn comes — it re-enters at the downgraded state, the state machine naturally drives it back to `committed` (re-running `commitTask`, now a no-op thanks to the idempotency above), and the merge step (step 4) re-attempts. Worktree reattachment (step 1) already handles the case where `taskWt`'s directory is gone but its branch survives. Branch absent entirely → fresh lease from current integration tip, as in the non-resume path. **Re-attempted merge that conflicts again is NOT auto-recovered.** If the re-attempt in the paragraph above hits the same (or a new, since integration may have moved) conflict, resume does not retry a second time or re-implement automatically — it fails closed to BLOCKED again, identically to the first attempt (step 4). Recovery is a human call, same as any other BLOCKED task: either resolve the conflict by hand on the task branch and re-run `--resume` (merge re-attempts against the now-conflict-free branch), or discard the draft via the existing escape hatch (`git branch -D plan/--` + `git worktree remove /tmp/wt-- --force`) so the next lease re-implements from the current integration tip. This is a deliberate scope line: automatic re-implementation-on-conflict is out of scope for this design (YAGNI — same-wave tasks are file-disjoint by planning convention, so a real conflict here indicates a planning error worth surfacing to a human, not silently working around). ## Concurrency `runPool`'s existing concurrency limit and wave-sequential structure are unchanged — this design makes same-wave concurrent dispatch **safe** (each task has its own filesystem tree) rather than removing concurrency. Two steps are serialized critical sections (worktree-create at lease, and merge-into-integration at commit — see step 1/4 above); implement/gate/review for multiple same-wave tasks still run fully in parallel, since neither touches `.git/worktrees`. ## Failure modes to handle explicitly - **Merge conflict at integration**: fail closed (see step 4 above) — never auto-resolve, never force-push over the integration branch. - **Integration worktree/branch already exists from a prior stalled run**: reuse it (matches `git worktree add` idempotency the Workflow controller already relies on) — do not delete and recreate, that would drop in-flight merged work from other tasks. - **Task worktree lingers after a crash mid-merge**: resume path (above) detects and re-drives it; there is no separate cleanup sweep in this design (YAGNI — the Workflow controller's `rp-isolate.sh gc` is for its clone-isolation feature, a different mechanism, out of scope here). - **Task or integration worktree DIRECTORY is gone but its branch survives** (e.g. `/tmp` cleared on reboot for a task worktree, or manual `git worktree prune` for the integration worktree): the branch is the durable state, the worktree directory is disposable. `git worktree add ` (no `-b`, no `-B`) reattaches without touching branch history. Using `-B` here would be a silent data-loss bug — see step 1 above. ## Dependency provisioning Audience: AI coding agents first. A per-task worktree at `/tmp/wt---` is a git worktree — it carries no `node_modules` (gitignored, never materialized into a worktree) and no build cache. `gate0` in the default `strict` mode runs the repo's own `npm run build` / `npm run test` there and fails closed on the first non-zero exit. With no deps installed, that red is environmental, not a task defect: the gate throws `gate-not-green-after-N` after `GATE_MAX_ATTEMPTS`, the task is BLOCKED, and the failure is indistinguishable from a real break. This section closes the gap the topology above leaves open. ### Rule 1. Provision deps into EVERY per-task worktree AND the integration worktree, once, in `runner.js` — immediately after the worktree-ensure step (step 1 of Git topology), BEFORE the coder dispatch. Not just before the gate: the coder needs deps to run and verify its own work; provisioning here deterministically replaces the old engine's reliance on the agent installing deps itself on every dispatch. 2. NEVER symlink the parent repo's `node_modules` into a worktree. It leaks worktree writes (postinstall scripts, test caches, an agent-run `npm install`) into the user's real store, is blind to monorepo nested `node_modules` and yarn-PnP (no `node_modules` at all), and fails cold when a headless clone has no parent store to point at. 3. NEVER run a full `npm ci` per task. Correct but heavy — a clean wipe-and-reinstall per worktree, dozens of times per run. Install once per unique lockfile; reuse everywhere. 4. Provisioning failure fails closed with a DISTINCT error (`dep-provision-failed`), never conflated with `gate-not-green`. Operators MUST be able to separate environmental from real. ### Mechanism — lockfile-keyed immutable cache + copy-on-write materialization Interface: `provisionDeps(taskWt, repoRoot) -> void | throws`. Dispatches by detected toolchain; logs-and-no-ops for toolchains it does not recognize (Makefile-only / non-Node repos still run — gate proceeds unchanged). 1. **Detect toolchain authoritatively.** Honor `packageManager` in `package.json` (corepack) first, else lockfile precedence: `pnpm-lock.yaml`→pnpm, `package-lock.json`/`npm-shrinkwrap.json`→npm, `yarn.lock`→yarn (berry if `.yarnrc.yml`/`yarnPath`, classic otherwise), `bun.lockb`→bun. No lockfile → fail closed with an actionable message (a repo gating on tests without a committed lock is itself a defect), unless `--allow-no-lockfile` opts into a non-deterministic `install` fallback. 2. **Cache key = `sha256(pmId + nodeVersion + arch + lockfileBytes)`**, hashing the TASK worktree's OWN lockfile at its HEAD — never the parent's. Consequence, load-bearing: a task that changes a dependency has a different lockfile → different key → its own freshly-installed entry. Correctness under dependency change falls out of the key definition; no staleness, no false green. Do NOT fold `package.json` into the key (it churns on version/script edits with no dep change) — the frozen install below is the manifest/lock consistency guard instead. 3. **Populate once per key, atomically, serialized.** Cache root `/.runplan-cache/deps/` (same filesystem as worktrees → reflink/hardlink valid; NEVER `/tmp` — tmpfs/cross-device breaks links and gets wiped; gitignore `.runplan-cache/`). Install into `.tmp.` with the deterministic, offline-preferring frozen form (`npm ci --prefer-offline --no-audit --no-fund`, `pnpm install --frozen-lockfile --prefer-offline`, `yarn install --immutable`, `bun install --frozen-lockfile`), then `flock` `.lock` and `rename` `.tmp.`→`/` — the rename is the atomic commit point, so a reader sees a fully-populated entry or none. Same-key tasks serialize on the flock; different-key tasks install fully in parallel. `chmod -R a-w /` after populate (immutable). A frozen install erroring on a `package.json`/lock mismatch is the CORRECT signal (lock out of sync is a real defect) — surface it, do not paper over it. 4. **Materialize into the worktree by the cheapest isolation-preserving method, probed once at startup, first rung that holds:** (1) reflink/CoW `cp --reflink=auto -a` — O(1) metadata on btrfs/XFS/APFS, writes copy-on-write so the cache is never touched; (2) hardlink clone `cp -al` — O(files), no data copy, safe because the gate only reads `node_modules`; (3) plain `cp -a` — always correct, O(data), final fallback; (4) overlayfs (cache read-only lower + per-worktree upper) where mount privileges exist — zero-copy, fully isolated. 5. **Delegate to native stores where superior.** pnpm: skip the custom cache — its global content-addressed store IS the cache and worktree `node_modules` are symlink farms; run `pnpm install --frozen-lockfile --prefer-offline` in the worktree directly. yarn berry PnP: no `node_modules` exists; `yarn install --immutable` populates `.yarn/cache` from the global mirror. The custom cache is ONLY for npm / yarn-classic, which have no native store. The toolchain adapter MUST NOT assume `node_modules` exists. 6. **GC.** LRU by mtime, keep last N entries AND cap total bytes (bound both — big `node_modules` × several repos still reaches GBs on count alone); sweep orphan `*.tmp.*` older than a threshold at run start (crash-interrupted installs). Never evict an entry referenced by a live worktree. ### Failure modes to handle explicitly (provisioning) - **Cold clone, no parent `node_modules`** (cron/CI): irrelevant — install is from the lockfile into the cache, never from a parent store. - **Task changes a dependency**: different key → own fresh install → no false green. One real install for that unique lock, unavoidable and correct. - **`package.json`/lock disagreement**: frozen install errors → provisioning fails closed with the correct "out of sync" signal. - **Monorepo / workspaces**: key on the ROOT lock (captures the whole workspace graph); cache and materialize the full tree including nested `packages/*/node_modules`. - **yarn PnP**: no `node_modules`; adapter uses `--immutable`, never assumes the dir. - **Native modules / ABI**: key includes node version + arch; same host → arch matches; binaries preserved by reflink/copy. - **Concurrent same-key installs**: flock + atomic rename — one installs, the rest reuse. - **Crash mid-install**: orphan `*.tmp.*` never renamed → never seen as a valid cache; GC sweeps it. `/` exists only after a successful rename. - **Offline / flaky registry**: `--prefer-offline`/`--immutable` use the global mirror; only genuinely-new deps need network, else fail closed. - **CoW-unsupported filesystem** (ext4): reflink probe fails → hardlink; hardlink safe because gate is read-only over `node_modules`. - **Security**: the cache runs the repo's own install/postinstall scripts once and all same-key tasks reuse the result — identical trust model to `gate0` running the repo's own test scripts (already documented as not a security boundary against an adversarial repo). No new surface. ## Testing - Extend `test/runner-integration.sh`'s fixture (`plan.jsonl` with 2+ same-wave file-disjoint tasks) to assert: `base_branch` (`main` in the fixture) is unchanged after the run; `plan/` contains both tasks' commits; no `/tmp/wt-*` or `.wt-*-int` directories remain for a fully-committed run (cleaned up per the merge step); a forced merge conflict (two same-wave tasks editing the same file, deliberately violating the disjointness convention) leaves the task BLOCKED and the integration branch unaffected by the conflicting task's half-merged state. - New assertion for the concurrency claim: run the fixture with `concurrency: 2` and two same-wave tasks, assert both task worktrees existed simultaneously (or at minimum that both tasks' isolated commits appear on separate branches before the merge step, proving no shared-tree interleaving). - Dependency provisioning: fixture repo with a lockfile and a `build`/`test` script whose success depends on an installed dep. Assert (a) a `strict`-mode gate goes green in a fresh task worktree with NO manual install (provisioning ran); (b) the cache entry is created exactly once across two same-key tasks (install count == 1); (c) a task that mutates the lockfile gets a distinct cache entry; (d) a deliberately broken install (bad lockfile) yields `dep-provision-failed`, NOT `gate-not-green`; (e) the user's real `/node_modules` is byte-unchanged after the run (no leak). ## Architecture Decisions - **Deterministic path naming over persisted worktree paths in the journal** — accepted. Single-adapter test: there is exactly one plausible worktree path per `(slug, taskId)` pair; persisting it would be redundant state that could drift from the deterministic value. Recompute, never store. - **Plain `git merge` over an LLM-mediated merge (unlike the Workflow controller)** — accepted. `runner.js` has no subagent dispatch loop; introducing one solely for a deterministic git operation would be a new, heavier machinery boundary for no behavioral gain, and fails the deletion test (an LLM merge step deleted here loses nothing `git merge --no-ff` doesn't already provide, since gate0 already guarantees the task branch is green before merge). - **No new journal states** — accepted. "Committed" already means "done, in git" from the outside; changing what `head` points to (integration branch head instead of task branch head) keeps that external contract intact while fixing what `reconcile` checks against. - **Lockfile-keyed immutable dep cache + CoW materialize over symlink-parent or `npm ci`-per-task** — accepted. Symlink-parent fails the deletion test the wrong way (it removes isolation, leaking worktree writes into the user's real store) and is blind to monorepo/PnP; `npm ci`-per-task is correct but pays a full install N times. The cache pays one install per unique lockfile and materializes per worktree in O(1) on a CoW filesystem — same correctness as `npm ci`, amortized. Rejected going further to a Nix/derivation-level env cache: strictly more robust but requires the target repo to adopt Nix, out of scope for "works on any Node repo as-is." - **Provision before the coder, not only before the gate** — accepted. The coder must run the project to verify its own work; provisioning at the gate only would force every coder dispatch to install deps itself (the old engine's non-deterministic behavior this design replaces). One deterministic provision per worktree serves coder, gate, and fixer. - **Delegate to the package manager's native store where it has one (pnpm, yarn-berry)** — accepted. Reimplementing a content-addressed cache on top of pnpm's store would be duplicate machinery for no gain; the custom cache exists only for npm / yarn-classic, which lack one.