# Controller-Owned Dark Delivery

audience: AI coding agents first.

status: ACTIVE
task IDs: #5, #25
source request: Owner approved full recommended execution order on 2026-08-11: preserve current WIP; implement minimal fail-closed feature control; convert retained WIP into current-main dark slices; move lifecycle ownership into existing controller/harness; enforce short-lived lanes. Owner approved recommended activation authority and final architecture. Owner explicitly reasserted on 2026-08-11: continue without stopping until all five stages are completed end-to-end. Offline package remains reference evidence only and MUST NOT be installed. Owner directed guarded landing of committed clean slice `80302f7e2b5ed5424e4aab223330a755ebf74d4a`, canonical local deployment, installed-SHA/controller/evidence proof with feature remaining OFF, accurate receipts, emergency containment of forgeable evidence, then completion of all remaining execution steps.

## Outcome

Deliver trunk-based, controller-owned, dark-launch workflow inside existing Overdeck architecture. Preserve offline package as reference evidence; NEVER install it or treat it as authoritative source. Reuse verified concepts; reimplement against current seams.

## Package disposition

Classify `temp-user/overdeck-git-automation` as `RECOVER + CONSOLIDATE`, not worktree #118.

- Preserve original ZIP, outer documents/scripts, SHA-256 inventory, manifest verification, and review receipt as plan evidence.
- NEVER run `apply-overdeck-git-automation.sh`, package installer, migration tool, or standalone service.
- Reuse contracts selectively: short-lived leased workspaces, automatic checkpoints, disposable candidates, exact-SHA publication, dark feature state, reachability-gated retirement.
- Reject implementation: red authoritative test; working-tree deployment; non-idempotent publication; broad sandbox exposure; no verified recovery bundle; parallel daemon; filename-based entrypoint rewrite; local heavy gates; no content-addressed receipt reuse.
- Delete package only after implemented controller behavior has parity receipts and preserved reference archive remains recoverable. Current action: keep.

## Acceptance criteria

1. Preserve every current committed, dirty, staged, untracked, ignored-important, detached, and stashed WIP carrier under verified durable recovery refs/snapshots plus manifest. Never expose secrets.
2. Add one tracked feature-definition contract for applicable behavior changes. Default/failure state MUST be off. Require stable ID, change class, owner, fallback, readiness checks, observability, review date, removal condition, introduced task/SHA.
3. Store mutable feature state outside Git in existing authoritative controller/harness state. Support `OFF → INTERNAL → CANARY → ON → RETIRED` and rollback to `OFF`.
4. Agent code MUST NOT mutate feature runtime state directly. Controller MAY auto-activate after exact deployed-SHA acceptance and smoke receipts when source request authorizes activation. Require owner only for unresolved product rollout scope.
5. Off-path regression and on-path feature tests MUST bind to exact candidate/deployed SHA. Provider/state failure MUST preserve old behavior or stable disabled response.
6. Candidate assembly MUST use disposable current-main workspace; NEVER rebase stale source lane in place.
7. Gate receipt identity MUST include candidate tree, gate graph, declared inputs, lockfile, environment/toolchain, and fixture/schema identity. Identical proof MUST be reused.
8. Heavy gates MUST execute through existing remote admission/buildbox/k3s containment. NEVER fall back to workstation.
9. Deployment MUST use exact verified artifact/tree from pristine deploy source. Never build or deploy from enrolled/dirty task worktree.
10. Publication/deployment/activation transitions MUST persist intent before side effect and reconcile observed external state after crash/lost response.
11. Cleanup MUST create and verify task-local recovery bundle/snapshot and restore drill before removing source/candidate worktrees or refs.
12. Integrate into existing harness/controller/ledger. NEVER add independent lifecycle daemon or competing SQLite authority.
13. New green feature slices land to current main within one task/session while behavior remains dark. Red/incoherent slices remain checkpointed recovery tasks.
14. Temporary flags MUST open cleanup task and be removed after stable-on rollback window. Do not flag refactors/docs/tests or ordinary atomic fixes.
15. Owner-visible Delivery surface MUST show editing, saved, verifying, landed, deployed dark, activation, blocked reason, and retirement without Git operation choices.
16. Factory reliability suite MUST pass for any phase/retry/gate/permission/commit behavior change.
17. Installed runtime MUST be proven through real entrypoint before landing local-infra changes; product changes follow branch→gate→land→deploy.

## Preserved WIP

- Canonical audit: `docs/plans/2026-08-11-wip-worktree-triage.md` and `.wip-triage-*` artifacts in `wt/wip-triage-audit`.
- Offline reference package: `/home/user/Projects/overdeck/temp-user/overdeck-git-automation/`; 49 ZIP entries, manifest hashes/sizes verified; required focused validation red.
- Existing controller/harness plans remain source of truth: `docs/specs/2026-07-30-harness-reliability-lessons-and-plan.md`, `docs/plans/2026-08-10-factory-reliability.md`, and `/home/user/Projects/0 DOCS/GIT_FATIGUE.md`.
- Current worktree inventory: 272 worktrees, 99 branch-only refs, 8 stashes, 117 recovery lanes. Counts are audit evidence, not deletion authorization.

## Constraints

- Preserve first. Verify second. Mutate last.
- Never install supplied package.
- Never create second daemon/controller/state store.
- Never mass-rebase or mass-merge stale lanes.
- Never delete uncertain work.
- Never run heavy gates locally.
- Never make owner choose Git mechanics.
- Never make routine activation a manual owner chore when request already authorizes shipping.
- Keep existing live work and unrelated shared-checkout WIP untouched.

## Execution steps

1. Preserve package and full live WIP inventory with verified recovery manifest.
2. Map existing harness/controller/receipt/deploy/feature seams; pin minimal contracts and migration boundaries.
3. Implement tracked feature definitions, schema, fail-closed evaluators, state transitions, exact-SHA readiness, and controller activation policy.
4. Implement crash-idempotent candidate→gate→publish→artifact deploy→dark/activate→retire transitions in existing ledger.
5. Route gates through existing remote execution and add content-addressed receipt reuse.
6. Add verified recovery bundles and restore drill before cleanup.
7. Integrate wrappers/factory incrementally; preserve old path until parity tests pass. Never rewrite entrypoints by filename.
8. Convert retained WIP plan-by-plan into small current-main dark slices; land low-conflict lanes first.
9. Add owner-visible Delivery projection using existing UI primitives.
10. Prove chaos cases, installed runtime, automatic closure, flag rollback/removal, and bounded worktree debt.
11. Retire superseded package/legacy code only after parity and recovery receipts.

## Current receipt

- 2026-08-11: Owner approved architecture and full execution order.
- 2026-08-11: Offline package classified `RECOVER + CONSOLIDATE`; keep as reference, do not install, do not call worktree #118.
- 2026-08-11: Factory `9c8de67e` succeeded at commit `7b925a9d0`; Factory recorded `pnpm run test`, `pnpm run typecheck`, and `pnpm --filter web build` passing. Independent review rejected shipment: runtime definition load throws instead of failing closed; exact-SHA evaluation accepts arbitrary equal strings; `INTERNAL`/`CANARY` expose a universal enabled boolean without audience eligibility. Commit implements execution step 3 only partially; acceptance 1, 4, and 6–17 remain open. Do not install, land, deploy, or activate this commit.

## Current receipt

- 2026-08-11 (implementation range `3852d5538..HEAD`): repaired the three independent-review blockers with fail-closed runtime definition resolution, exact lowercase 40-hex identity validation, and bound internal/canary eligibility decisions. Added immutable component deployment observations and acceptance/smoke receipt storage, a pure exact-identity readiness evaluator, and controller-only foreground `OFF -> INTERNAL` reconciliation through the existing intent journal. The tracked `dark-only` feature remained OFF; nothing was installed, deployed, or activated.
- Verification run: `pnpm --filter overdeck-controller test` exited 0 (211 passing), `pnpm --filter overdeck-controller typecheck` exited 0, and `git diff --check` exited 0. A later broad `pnpm run test` exited 1 only because the installed `build.slice` had stale finite `MemoryHigh`/`MemoryMax` values while this checkout declares no size ceiling; no deployed runtime configuration was mutated by this slice.
- Execution step 3 and acceptance criteria 4/5 advanced but remain partial. Acceptance criterion 1, criteria 6–17, and all unimplemented portions remain open. Remaining boundaries include trusted CI receipt producer/admission, exact deploy producer and component release manifest, build-once/deploy-exact artifact consumption, credential isolation, foreground Factory/controller integration, publication intent/reconciliation, recovery cleanup, UI, retirement, and installed-entrypoint proof.

- 2026-08-11: Factory repair `52451072` completed and focused controller tests/typecheck/diff-check passed, but independent review rejected shipment. Blockers: cancelled journal keys are not replayable; readiness can satisfy checks with a wrong-candidate acceptance receipt; readiness/activation lacks direct crash/retry/refusal/CAS tests; public store transitions bypass receipt/readiness authority. Expiry and cleanup are absent. Nothing installed, landed, deployed, or activated.
- 2026-08-11 (current implementation): fixed all recorded dark-delivery blockers. Cancelled refusals and pending lost-response retries replay deterministically; generic startup recovery preserves delivery intents; exact candidate/deployment identity binds every required receipt; controller-owned lifecycle covers `OFF → INTERNAL → CANARY → ON → RETIRED`, rollback, crash resume, CAS rollback, and evidence drift. Delivery mutation capability is module-private; authenticated `/delivery/evidence` admission validates and atomically persists exact deployment/receipt evidence. Definition review expiry fails closed during steady-state evaluation; tracked definitions require cleanup task, rollback window, and removal condition. Existing admission loop runs idempotent expiry reconciliation without overlap, retires expired ON/INTERNAL/CANARY, removes state/evidence after rollback window, and isolates lifecycle failures from host admission. Nothing was installed, landed, deployed, or activated.
- Verification: `pnpm --filter overdeck-controller test` exited 0 (225 passing, 0 failing, 0 warnings), `pnpm --filter overdeck-controller typecheck` exited 0 with no warnings, and `git diff --check` exited 0. Independent review’s final periodic-loop blocker was fixed with same-tick host-admission regression coverage.
- 2026-08-11 guarded landing: initial queue ticket `ticket.4a420340b0fd4e2badc2fd9104fc124c` stopped on a current-main candidate conflict. Integrated both unique `docs/plans/INDEX.md` rows per conflict rubric and preserved current-main work. Retry ticket `ticket.3a5f49815aa049beb5c7ace9cfdbce62` ran candidate typecheck/test attempts but initially produced no verdict because repository ref `refs/heads/wt/factory-gpt-engine` pointed to missing object `73d14eb08f9ac904fe1f7c299c1201f65bafab0c`. The owning peer repaired that preserved branch to existing commit `05fff831f3d5971f1be1cee635978d5fac8afe83` from base `8fbbd303d`; `show-ref`, commit object validation, and `fetch --dry-run origin` then passed.
- 2026-08-11 emergency security correction: dark-delivery commit `80302f7e2b5ed5424e4aab223330a755ebf74d4a` is contained in deployed `origin/main`/deploy SHA `b5c85ea48a16fe14298d1f4fbc467e01422addf0`. Installed controller is active. Deployed `POST /delivery/evidence` authenticates only with the controller-wide bearer token and then accepts caller-supplied deployment and receipt authority. Generic authenticated callers can therefore forge rollout evidence. This violates acceptance criteria 4, 5, 9, and 10. Feature remains OFF. Task #25 owns emergency containment in isolated lane `wt/delivery-evidence-incident`; preserve rollback and never activate.
- Acceptance delta: evidence admission MUST fail closed until a producer trust contract exists. Generic controller authorization MUST NOT authorize evidence writes. Final producer path MUST require dedicated evidence-writer authority and cryptographically verified attestations bound to allowlisted issuer, audience, repository/workflow, feature, target, commit/tree, artifact digest, result, and timestamp. Request fields MUST NOT become evidence authority directly. Add forged-token/body, wrong issuer/audience/identity/digest/result/time, replay, and valid-attestation tests before enabling admission.
- 2026-08-11 emergency restoration: confirmed deployed source `b5c85ea48a16fe14298d1f4fbc467e01422addf0` constructed `DeliveryEvidenceAdmissionService` behind controller-wide bearer authorization, parsed caller JSON, and persisted caller-supplied deployment/receipt authority. Restoration keeps `POST /delivery/evidence` present but returns HTTP 403 `{ "error": "delivery-evidence-admission-unavailable" }` immediately after generic authentication; it does not parse or persist request data. Feature remains OFF. Installed controller code is commit `8b124bc31cb9300e4ec139bcc86d79b8849e3b20`; pre-commit scoped code/test diff SHA-256 was `e4375bb79f677d074f3834f8065a43bff75ed1e8e037796d24fe193836c355bb`.
- Emergency verification: focused `pnpm --filter overdeck-controller exec bun test --timeout 20000 src/server.test.ts --test-name-pattern 'valid generic bearer'` passed 1 test/5 assertions; full `pnpm --filter overdeck-controller test` passed 226 tests across 22 files with 0 failures/warnings; `pnpm --filter overdeck-controller typecheck` and `git diff --check` exited 0 without warnings. Independent trust-boundary self-audit confirmed generic authentication cannot reach any evidence parser/writer, the admission service is not constructed by the server, and the regression proves revision/deployment/receipt storage remains unchanged.
- Live restoration: canonical `packaging/install-controller.sh` installed the candidate unit. A stale `task176-candidate.conf` drop-in overrode its working directory, so final-priority rollbackable drop-in `/home/user/.config/systemd/user/overdeck-controller.service.d/zzz-delivery-evidence-emergency.conf` pinned only the exact controller service to this isolated candidate. Initial restart exposed missing worktree dependencies; `bun install --frozen-lockfile` installed 6 packages without warnings, then exact-service restart succeeded. Runtime proof reported active/running PID `353016`, working directory `/home/user/Projects/overdeck/.worktrees/delivery-evidence-incident/controller`, and an internally authenticated request returned only HTTP 403 plus `delivery-evidence-admission-unavailable`; token contents were neither read into shell nor printed. Rollback: remove emergency drop-in, restore `/home/user/.cache/overdeck-emergency-rollback/overdeck-controller.service.pre-delivery-evidence-20260811` (SHA-256 `dbdc097879ecc411ff4bd75f6f90a19a5597af6631f6944721b35bec849bae71`) to the unit path, daemon-reload, and restart only `overdeck-controller.service`.

- 2026-08-11 orchestration expansion: owner requested all remaining work use main-authored `/gpt-orchestrator` contracts with at most eight concurrent GPT-5.6 Luna/max babysitters. Each implementation babysitter drives one ask-gpt Pro conversation; a separate Luna/max babysitter drives a fresh ask-gpt Pro adversarial review conversation; confirmed findings return through same implementation conversation for correction. Main thread alone pins decomposition, file ownership, dependencies, packages, prompts, acceptance, and review rubrics. Luna workers MUST remain mechanical executors. One global prompt-send queue MUST serialize all new/resumed ask-gpt prompts 55–75 seconds apart; eight workers do not authorize eight independent senders. Generation/review MAY run concurrently; overlapping application/landing remains dependency-ordered against refreshed current main.

- Review-input invariant: main coordinator snapshots each task's exact initial baseline commit/tree before implementation dispatch. Fresh review package MUST contain that immutable baseline source, accepted implementation artifact, exact `baseline..candidate` binary-safe diff plus diff SHA-256, changed-file manifest, implementation conversation/attachment identities, mechanical archive-validation receipt, authoritative task contract, and applicable instructions. Reviewer MUST compare only that pair, reject any candidate whose parent/base/path manifest differs, and never review a moving worktree or narration-only summary. Luna driver only transports and validates identities/hashes; ask-gpt Pro performs judgment. Coordinator MUST verify every Luna receipt against local Git/artifact evidence before correction, application, or landing.

- Landing-role invariant: reviewer approval does not authorize direct push. For each approved artifact, one GPT-5.6 Terra/medium integration orchestrator owns refreshed-main application, semantic conflict resolution, local deterministic verification, guarded `.claude/scripts/ship.sh` landing, canonical deploy/install, installed-entrypoint proof, and exact receipt. Luna MAY mechanically prepare hashes, invoke already-pinned commands, and monitor outputs, but MUST NOT resolve conflicts, reinterpret warnings, choose between competing implementations, weaken gates, or decide whether a changed baseline preserves acceptance. Integration tasks serialize when paths/contracts overlap; disjoint approved tasks MAY run concurrently in separate fresh worktrees, but guarded landing revalidates against latest main. Any conflict, changed contract, warning, failed gate, identity drift, or reviewer scope mismatch returns to correction/re-review; never patch opportunistically during landing.

- 2026-08-11 trusted evidence candidate: added `controller/src/evidence.ts` Ed25519/WebCrypto envelope verification, validated `deliveryEvidence` trust configuration, dedicated `x-delivery-evidence-authority` authorization, and an atomic SQLite replay claim with evidence persistence. Signed payloads bind issuer, audience, repository, workflow, feature, definition, target, candidate/deployed commit and tree, artifact digest, result, validity window, deployment, and receipts. Strict canonical base64url decoding enforces 32-byte public keys and 64-byte signatures. Observation times must fall inside the signed window and not in the future. Generic bearer-only, absent/wrong/equal dedicated authority, malformed identity/signature/key/time, tampering, and replay fail closed. No private-key producer was added; a trusted deployment boundary must own signing. Feature remains OFF.
- Verification from `/home/user/Projects/overdeck/.worktrees/delivery-evidence-attestation`: focused evidence/server tests passed 25 tests with 109 assertions; full `pnpm --filter overdeck-controller test` exited 0; `pnpm --filter overdeck-controller typecheck` exited 0; `git diff --check` exited 0. No warnings were emitted. Candidate is uncommitted and not installed, landed, deployed, or enabled.

- 2026-08-12 lost-session reconstruction: ground-truth transcript `/home/user/.claude/projects/-home-user-Projects-overdeck/6dd0b227-c961-451d-82d8-2c1d13cdce2c.jsonl.live.jsonl` contains progress after previous receipt. Shell-wrapper identity repair landed/deployed through `346ce595`. Installed bootstrap then exposed nested `node buildbox-registry.mjs` interception. Follow-up `d381525ca` added exact first-script-operand identity for allowlisted `node`, configured canonical deployed registry path, and added positive plus `node -e`/later-argument rejection tests. Focused remote-build suites and diff check passed; guarded landing succeeded to `origin/main`.
- Final repair deployment remains incomplete. Canonical deploy refused because deploy clone carries another session's uncommitted `modules/workstation/claude/bin/_human-session` edit. NEVER overwrite it. Exact binary recovery patch: `/home/user/.cache/overdeck-emergency-rollback/deploy-clone-human-session-20260812.patch`; SHA-256 `0f1fc48a43d6e62b04a252fcbcc5f76bb1e2cd798958aa0eeba6c1b538b72a2a`; preserved diff size 22 insertions/47 deletions. Ownership queries were sent to likely sessions; no reply was recorded before compaction.
- Installed bootstrap/audit and `buildbox-parity.timer` proof remain pending because `d381525ca` is landed but not deployed. Automated commit review also emitted unresolved `Agent/Subprocess Permission Bypass in modules/workstation/claude/lib/remote-build.mjs`; no detailed verdict was captured. Independently verify final executable/interpreter identity logic before calling Task #39 clean. Existing unused-symbol diagnostics in `remote-build.test.mjs` are not introduced by this repair but MUST be explicitly resolved or justified.

- Installed bootstrap retry after deploy clone became clean failed deterministically: `local-gate --remote-only` receives trusted `local_only:config:<canonical-registry>` from `tryRemoteBuild` but refuses at the early remote-only branch before `resolveGateDecision` can authorize exact local execution. This is the remaining admission defect; repair local-only handling without permitting any unavailable/transient remote failure to fall back locally.

- 2026-08-14 owner-requested re-review of the offline package; disposition reconfirmed as `RECOVER + CONSOLIDATE`, do not merge, do not install. Supersession verified from the repository rather than from narration: `git merge-base --is-ancestor 80302f7e2b5ed5424e4aab223330a755ebf74d4a HEAD` succeeded, `controller/src/delivery` and `delivery/feature-definition.schema.json` exist in tree. Package baseline `9990b8cb9b5ebf40acba7c22a2b9763ca4134480` is 484 commits behind HEAD `7b3f5d188`. All 31 package paths are pure additions (`tools/git-automation/**` plus `modules/workstation/claude/bin/overdeck-managed-agent`), so it would apply without textual conflict — the disqualifier is architectural, not textual: `tools/git-automation/install.sh` enables its own `overdeck-git-controller.service`, its own state authority under `$XDG_STATE_HOME/overdeck/git-controller`, its own `tools/git-automation/bin/git` broker and `overdeck-git-deny`, alongside the live `modules/workstation/claude/bin/git` guard, `od-worktree`, and the land queue. That is a third publication protocol, forbidden by acceptance criterion 12 and the "never create second daemon/controller/state store" constraint.
- Package carries zero passing evidence for its 31 files, not merely a failing test: `tools/git-automation/test.sh` runs `bash -n` over `tools/git-automation/bin/*`, which includes the Node ESM file `bin/github-app-token-provider.mjs`; under `set -euo pipefail` the run aborts there, so `node --test` and all three adversarial test files never executed. This matches the supplied `OVERDECK_GIT_AUTOMATION_DELIVERY_VERIFICATION.md` overall verdict FAIL (`focused` exit 2, `clean_apply_and_test` exit 3).
- Duplicate on disk: `temp-user/git-automation` and `temp-user/overdeck-git-automation` hold byte-identical ZIP (`1846a9d9993b4939a7fec31e2f73c5ecb8a923d09d0eb22ee1d52580f7d613d4`), apply script, and agent prompt. Canonical copy remains `temp-user/overdeck-git-automation` (holds `OVERDECK_GLOBAL_GIT_AUTOMATION_PLAN.md`). The only content unique to `temp-user/git-automation` is `OVERDECK_GIT_AUTOMATION_DELIVERY_VERIFICATION.md`. Consolidating that one file into the canonical directory was refused by `main-checkout-guard` because both directories are untracked inside the shared main checkout; consolidation therefore requires owner action and MUST precede any removal of the second copy. Nothing was moved or deleted.
- Incidental, pre-existing and unrelated to this plan: `git log --all -- 'tools/*'` reports `fatal: unable to read tree 44742e6abe0970bb913ceb3b69516292776c7b1b`, a dangling tree object in repository history. Not introduced by this review; not blocking.

## Next executable action

1. Add a failing `--remote-only` exact-local-only regression proving trusted configured command identity runs locally while unavailable/transient remote dispatch still refuses.
2. Restore only that deploy-clone path to deployed HEAD ONLY after proof that live edit is no longer active and recovery is sufficient. NEVER discard uncertain WIP.
3. Run canonical queued deployment for landed `d381525ca`.
4. Run installed `buildbox bootstrap debian1 debian2 debian3`, installed audits, and `systemctl --user is-active buildbox-parity.timer`.
5. Independently verify reported subprocess-permission bypass and address every diagnostic.
6. Mark Task #39 complete only after installed proof is clean; then resume Task #37 and remaining delivery stages.
7. NEVER execute or install `temp-user/k3s-phase*`, `temp-user/overdeck-git-automation`, or its byte-identical copy `temp-user/git-automation`; preserve as evidence only.