# AWP I1 Dogfood Session Handoff — 2026-08-22 ## Stop point Implementation was stopped at the owner's request after restoring the live dogfood stack and verifying the current revision. Do not assume any later work occurred. Resume by inspecting this file, `docs/progress/2026-08-22-i1-drift-audit.md`, git state, and live debian3 state first. ## Exact repository state at handoff - repo/worktree: `/home/user/Projects/.worktrees/awp-i1-dogfood-deploy` - branch: `ops/i1-dogfood-deploy` - worktree: clean and synchronized with `origin/ops/i1-dogfood-deploy` at handoff - latest branch commit at handoff: `644da01` — `Record system image convergence diagnosis` - recent relevant commits: - `a5332e1` — `Persist verification evidence and protect merge` - `67f9eb5` — image publisher commit pinning the evidence-emitting agent runner digest - `9cb0020` — `Converge system images through immutable registry pulls` - `24a2a86` — `Fix system image publisher package auth` - `9d40d20` — image publisher commit pinning Subrouter/model-gateway digests - `644da01` — durable diagnosis/update after the system-image drift incident ## Exact live dogfood state at handoff Host: `debian3`; owner URL remains `http://100.101.104.41:4173`. Verified immediately before writing this handoff: - `awp-dogfood-control-plane.service`: active - `awp-dogfood-web.service`: active - `awp-dogfood-postgres.service`: active - `awp-subrouter`: 1/1 Ready/Available - `awp-model-gateway`: 1/1 Ready/Available - `awp-fabro`: 1/1 Ready/Available - control-plane health: `status=ok` - migration health: `applied=9`, `expected=9`, `pending=0`, `drift=[]` - account inventory: `accounts=[]` - execution DB rows: - `factory_runs=0` - `agent_runs=0` - `changesets=0` - `verification_evidence=0` - `review_findings=0` - no owner login was initiated by the assistant - no FactoryRun/AgentRun was initiated by the assistant Live immutable system images: - agent runner desired state: `ghcr.io/platform-modules/awp-agent-runner@sha256:af9aee7f4dcab70b743d6d2e619d3f16a7cee330cc09af5beccf512f1c179d5d` - Subrouter: `ghcr.io/platform-modules/awp-subrouter@sha256:ad546a6882e5430743e8122a8922115f94017349730f56efbb0062c493e5cd18` - model gateway: `ghcr.io/platform-modules/awp-model-gateway@sha256:1e572c0593063fe9830dbcf6036b764754643795c48c53cbece8821988447373` - Fabro: `ghcr.io/fabro-sh/fabro@sha256:94e65f3da935a3b269818a0238c8f304fe5cf2aad5db856859d5ba3050eef6e7` ## What was completed in this session ### 1. Slice 1 image convergence / configuration-drift repair The original drift class was: AgentRun/system pods could be scheduled or recreated on a node that did not have a node-local image, producing `ErrImageNeverPull` and requiring manual `k3s ctr images import`. That violated the Slice 1 requirement that image/version management be transparent and self-healing. Completed repair: - AgentRun image is registry-backed and digest-locked in GHCR. - Clean-worker proof already succeeded on debian2: ordinary Kubernetes/GHCR pull, pod Ready, exact imageID match, runner IPC capability proof. - removed obsolete per-node agent-runner readiness labels/annotations; scheduling no longer depends on manual image-preload metadata. - added registry pull authority to the required namespaces rather than handing credentials to workloads. - extended the same invariant to required `awp-system` pods after a real kubelet eviction exposed that Subrouter/Fabro/model-gateway still used `localhost/...` + `imagePullPolicy: Never`. - added `.github/workflows/system-images.yml` to build/publish immutable Subrouter + model-gateway images and commit digest locks. - pinned Fabro to its verified immutable upstream digest. - dogfood deployment no longer performs normal-path local build/import for Subrouter/model-gateway/Fabro. - Subrouter/model-gateway use the scoped GHCR pull secret in `awp-system`; Fabro pulls its public immutable digest. - live eviction incident was recovered through the durable registry-backed deployment, not by restoring node-local imports. Important incident context: - debian3 temporarily hit legitimate ephemeral-storage pressure. Kubelet evicted Subrouter/Fabro/model-gateway. - replacements initially failed because the previously deployed manifests used `Never` + localhost images. - debian3 subsequently recovered (`DiskPressure=False`, Ready, ~23 GiB free when checked). - after the registry convergence deployment all three workloads returned 1/1 Ready. ### 2. Authoritative VerificationEvidence and protected merge gate Implemented and deployed: - first-class `VerificationEvidence` and `ReviewFinding` entities, candidate-digest-bound; - persistence tables/repositories plus Drizzle migration `0008_purple_scalphunter.sql`; - canonical command paths for recording verification evidence, recording findings, and resolving findings; - event/audit/outbox recording for these mutations; - Attempt completion requires structured timestamped evidence; - ACP runner emits real required evidence for candidate-tree creation and `git diff --cached --check`; - live U4/U5 hierarchy/projection consumes persisted evidence/findings instead of hard-coded empties; - review approval no longer blindly marks a ChangeSet mergeable; - `requestMerge()` independently reloads and rechecks evidence/findings and fails closed; - latest observation wins per `(source, check name)`, so a rerun may supersede a stale failed observation; - unresolved blocking finding revokes merge readiness; resolution recomputes readiness. Canonical PGlite proof added: `complete -> failed evidence -> approve review -> merge refused -> newer passing evidence -> ready -> blocking finding -> merge refused -> resolve finding -> ready -> trusted merge` That focused proof passed. Full remote repository gate for the evidence slice passed on debian1: - 39 test files passed / 2 skipped - 135 tests passed / 5 skipped - typecheck - lint - build - architecture tests - dependency-cruiser zero violations - formatting The current schema is live: health shows 11/11 migrations with zero drift. ### 3. Previously completed I1 foundations that remain valid - approved live U1 shell/routes are wired; - one FactoryRun per PlanRevision orchestration is implemented; - account Add/select UI is Slice 1-required and backed by K3s Subrouter; - account inventory starts empty, user must add/login separately; - account/model provenance propagates into execution; - native Fabro HTTP integration is live and DBOS durability is composed; - native ACP runner exists in K3s workspaces; - short-lived Attempt/account-scoped model-gateway capability prevents workspace possession of reusable Subrouter proxy/OAuth credentials; - real Git candidate tree/diff evidence is produced by runner; - image management is now registry/digest based for AgentRun plus required system workloads. ## Overall I1 status I1 is **not complete**. Do not restore the old `30/30` claim. The authoritative drift audit remains `docs/progress/2026-08-22-i1-drift-audit.md`, and GOLIVE acceptance must only be advanced from verified evidence. Major repaired areas: 1. canonical live UI — materially repaired 2. correct FactoryRun orchestration — repaired 3. provider composition Subrouter/Fabro/ACP/Kubernetes/DBOS — materially repaired/live 4. real workspace/runner execution — materially repaired 5. VerificationEvidence merge gating — now materially repaired/deployed 6. image/version convergence — materially repaired for AgentRun and required awp-system pods 7. repository-scoped trusted GitHub publication/merge — repaired, tested, pushed, and deployed fail-closed without owner credential 8. independent reviewer AgentRun — repaired, tested, immutable runner published, and deployed Major remaining gaps are below. ## Remaining work / next tasks ### P0 — Repository-scoped trusted publication/merge — repaired Completed at code candidate `138a7d5` (`Add repository-scoped trusted GitHub publication`) and deployed to debian3: - ChangeSet authority derives from `Project.repositoryUrl` and canonical GitHub `owner/repository` identity; - concrete server-side GitHub Git Data/merge HTTP transport added with credential-reference resolution; - trusted publication occurs before Review creation and persists immutable publication/target identity; - trusted merge rechecks Project repository identity, expected target/base, candidate manifest/digest, evidence, and findings; - publication/merge replay is idempotent and ambiguous-success reconciliation is covered; - local-checkout merge authority (`AWP_TRUSTED_REPOSITORY_PATH` / `LocalGitTrustedMergeAdapter`) was removed; - source checkout remains only as `AWP_SOURCE_REPOSITORY_PATH` for clean ACP source seeding; - migration `0009_tricky_mandrill.sql` persists repository/publication/merge/result revision state; - full debian2 `pnpm check` passed: 142 active tests, 5 skipped, architecture zero violations; - live debian3 is healthy at 10/10 migrations with zero execution rows and empty account inventory. Dogfood GitHub publication authority is intentionally **not** provisioned: `/home/user/.config/awp-dogfood/github-publication-token` is absent and the generated control-plane environment contains no `AWP_GITHUB_PUBLICATION_TOKEN_FILE`. This is a deliberate fail-closed boundary until the owner authorizes/provisions repository-scoped publication credentials. No existing workstation credential was copied or read. ### P0 — Independent review — repaired Completed at `e4746b2` plus publisher digest-lock commit `0c06b3a` and deployed to debian3: - every dogfood Review is assigned to a distinct reviewer AgentRun/Principal, never the coder/owner principal; - AgentRun role and Review -> reviewer AgentRun binding are durable via migration `0010_perfect_victor_mancha.sql`; - trusted publication idempotently creates/reuses exactly one reviewer AgentRun/Attempt/Workspace; - native ACP reviewer materializes the exact ChangeSet base+patch, verifies candidate tree digest before and after review, uses a read-only reviewer prompt/permission posture, and submits via a reviewer-only callback; - reviewer callback is Attempt-token authenticated and candidate/Review/AgentRun bound; structured findings/disposition are durable; - agent-assigned Review cannot be approved through the owner `/approve` path; ready-to-merge and trusted merge require the approved independent reviewer binding; - go-live scripts 19/22/23/27 were reconciled to wait for the reviewer AgentRun rather than self-approving; - GitHub Actions run `32586960856` published and smoke-tested immutable runner digest `sha256:af9aee7f...`; - final head `0c06b3a` passed full debian2 `pnpm check`: 146 tests passed, 5 intentionally skipped, architecture zero violations; - debian3 is healthy at 11/11 migrations with the new runner digest configured and all execution/review/evidence rows still zero. ### P0/P1 — Repository-required GitHub checks — repaired and live-verified Source implementation is complete and final exact-tree `pnpm check` is green on debian2. Canonical evidence: `docs/progress/2026-08-23-i1-github-required-check-gates.md`. Implemented: - durable `Project.requiredChecks` policy with migration `0011_hesitant_mojo.sql`; dogfood policy is `CI`; - GitHub branch-protection required-check discovery plus check-run/commit-status readback; - observations are tied to the trusted publication commit SHA while VerificationEvidence remains bound to the immutable ChangeSet tree digest; - Project policy and provider-required checks are unioned fail-closed; - reserved provider evidence namespace prevents coder/reviewer self-certification; - required checks refresh after publication, after independent Review, on explicit canonical refresh, and immediately before trusted Merge; - `.github/workflows/ci.yml` runs on trusted `awp/**` publication branches; - U4/U5 now distinguish independent review, pending checks, changes requested, and ready-to-merge; pending checks expose refresh but not Merge/correction; - final debian2 source gate: 41 test files passed / 2 skipped; 154 tests passed / 5 skipped; architecture 6/6; dependency-cruiser zero violations; Prettier clean. Live deployment is now complete at source commit `9064252`. debian3 is healthy at 12/12 migrations with zero drift; Project required-check policy is `CI`; Subrouter/model-gateway/Fabro are 1/1 Ready at their immutable digests; FactoryRun/AgentRun/ChangeSet/Review/evidence/finding counts remain zero; account inventory remains empty; no GitHub publication token/env binding exists. Canonical gate G1-G9 is complete. The next autonomous implementation priority is workspace lifecycle / durable WIP correctness: terminal cleanup, content-derived checkpoint identity, retry/recovery preservation of persistent WIP, and pod replacement/restart resume behavior. ### P1 — Workspace lifecycle / durable WIP correctness — repaired and live-verified Canonical evidence: `docs/progress/2026-08-23-i1-workspace-lifecycle-gates.md`. Implemented and remotely verified: - real Git-tree content checkpoint identity from `/workspace/repo`; - durable Workspace checkpoint/collection/cleanup state via migration `0012_dear_surge.sql`; - PVC checkpoint attestation and exact-digest destructive cleanup handshake; - failed Attempt retry persists+attests checkpoint before replacing compute on the same Workspace/PVC; - coder recovery does not reset/clean/clone/reseed an existing repository; - publication failure keeps WIP; trusted merge is the only composed terminal cleanup path; - merged ChangeSet cleans producer + independent reviewer PVCs only after exact checkpoints are collected; replay converges idempotently; - canonical U3 now reports persisted WIP checkpoint/cleanup truth rather than hard-coded unknown; - final exact debian2 `pnpm check`: 159 tests passed / 5 skipped, architecture 6/6, dependency-cruiser zero violations (111 modules / 194 dependencies), Prettier clean. Live closure: - source commit `6e5e4da` passed full debian2 `pnpm check`; - agent-runner workflow `32618189832` succeeded and published/smoke-tested digest `sha256:afed7f0708643f8bc5fa365ef1f7569675eab93fdf607dce312eec26177366e5`; - automated lock commit `10b5b98` also passed the exact full debian2 gate and was deployed zero-run to debian3; - live health is 13/13 migrations with zero drift; execution/review/evidence/Workspace counts remain zero; accounts remain empty; no GitHub publication authority was provisioned; `AWP_WORKSPACE_IMAGE` is the new immutable digest; required system workloads are Ready. Canonical workspace lifecycle G1-G10 is complete. ### P1 — Execution/retry final reconciliation — repaired The current ACP/Kubernetes retry path is reconciled and covered by focused regression evidence: - failed Attempt is immutable terminal history; retry is a new Attempt on the same AgentRun + Workspace/PVC with explicit `selection.kind=retry` and `previousAttemptId`; - exact Git-tree checkpoint is persisted and PVC-attested before disposable compute replacement; - `replaceCompute()` receives the new Attempt identity while preserving Workspace/PVC; - composed AgentProvider `startAttempt()` receives the new retry Attempt ID plus the same AgentRun/Task/Workspace; - retry keeps provider/account/model provenance; - duplicate failure callback reuses the existing retry Attempt and does not replace compute or start another agent-provider session; - native ACP recovery regression proves an existing coder repository is reattached without destructive reset/clean/clone/reseed. `tests/application/execution-retry.test.ts` now explicitly proves provider call order `attest -> replace -> agent-start`, exact retry identity, and one provider start across callback replay. Focused proof passes on debian2. ### P1 — GitHub/CI provider live composition — repaired Repository-scoped GitHub trusted publication/merge and authoritative required-check ingestion are both composed in the dogfood control plane behind the same control-plane-only credential boundary. With no owner credential provisioned, both remain intentionally fail-closed. Required-check source/live evidence is recorded in `docs/progress/2026-08-23-i1-github-required-check-gates.md`. ### P1 — AC-29 / historical acceptance and GOLIVE rebuild Audit found AC-29 definitely false at the prior historical claim. After the actual vertical slice is real: - rebuild `GOLIVE.md` acceptance from current verified behavior; - prove AC-29 using actual AWP history, not an absent/fake commit; - rerun all acceptance checks against current branch/live stack; - do not inflate unattended/real-provider claims. ### Final quality/deployment sequence Before declaring I1 complete: - exact full `pnpm check` on debian1/2/3-authorized remote buildbox at final HEAD; - live dogfood migration health clean; - all required awp-system pods Ready and registry-backed; - account inventory/read path healthy; - owner account Add/select flow proven by owner when they choose to initiate it; - real owner FactoryRun only when user explicitly decides to use Factory; - trusted repository-scoped publication proven end-to-end; - required checks/review/merge verified against the real candidate; - workspace lifecycle cleanup/recovery proof; - only then update GOLIVE acceptance and consider I1 complete. ## Important owner constraints to preserve - **Do not initiate Factory.** Owner explicitly said they want it set up and will use Factory themselves. Do not create a FactoryRun/AgentRun unless they explicitly change this. - **Do not initiate account login.** Account login is user-initiated through the UI unless the owner explicitly asks the assistant to start it. - K3s Subrouter account state must stay separate from workstation/Systray/Overdeck auth. Never import/copy workstation OAuth into AWP K3s. - Workspace receives only short-lived Attempt-scoped model gateway capability; never OAuth or reusable Subrouter proxy token. - AWP is authority; provider state is observation/reference, not authority. - PostgreSQL is authoritative persistence; DBOS supplies durable workflow mechanics. - Tests/E2E/builds run only on debian1/debian2/debian3, not on the main workstation display. - Do not restart LibreWolf without owner approval. - No subagents/assistant lanes for this AWP implementation unless owner changes that instruction; current thread was sole implementor. ## Recommended resume procedure 1. Read this handoff and `docs/progress/2026-08-22-i1-drift-audit.md`. 2. Inspect `git status`, branch/head, and fetch origin before changing anything. 3. Inspect live debian3: systemd services, awp-system deployments, `/health`, account inventory, execution row counts, migration drift. 4. Confirm no owner FactoryRun/account login occurred since this handoff. If owner has used UI since then, adapt to actual state rather than resetting it. 5. Start with independent-review semantics, then GitHub CI/check-run ingestion and required-check policy. Repository-scoped trusted GitHub publication/merge is already repaired at `138a7d5`; do not regress it or provision owner publication credentials without explicit authorization. 6. Continue updating this handoff or the drift-audit durable file after each substantial verified checkpoint. ## Definition of completion The I1 vertical slice is complete only when a real owner-configured Project/Goal/Plan can execute through one FactoryRun, dispatch legal Tasks to real K3s ACP AgentRuns under explicit account/model provenance, preserve WIP/recover durably, produce a real ChangeSet with candidate-bound required verification evidence, obtain independent review, publish/merge through a trusted repository-scoped boundary with expected-base/candidate checks, show U1-U6 accurately, clean up workspace state safely, and prove the acceptance criteria from real persisted/live evidence. ## 2026-08-23 workspace-lifecycle continuation checkpoint Required-check gating is fully deployed. The automated image publisher advanced the branch with `c511cc8`; it was merged normally (no rebase/force) and the resulting head `a4cde44` was deployed. Live immutable pins are now Subrouter `sha256:28b809b6...`, model gateway `sha256:36337978...`, Fabro unchanged `sha256:94e65f3...`. Health remains 12/12 migrations with zero drift, all execution/review/evidence rows zero, accounts empty, required policy `CI`, and GitHub publication authority absent. Current implementation slice: `docs/progress/2026-08-23-i1-workspace-lifecycle-gates.md`. G1-G10 are now complete and live-verified: content-derived Git-tree checkpoints, durable checkpoint/collection/cleanup state, attest-before-replace retry, non-destructive coder recovery, exact-digest terminal producer+reviewer PVC cleanup, canonical U3 WIP truth, immutable runner publication, and zero-run debian3 deployment are all proven. Live health is 13/13 migrations with zero owner execution/account state.