# I0-C Durable Execution / Workspace / CI Progress - Lane: Development Agent 4 — Lane C, Durable Execution / Workspace / CI Infrastructure - Branch: `impl/i0-c-execution` - Worktree: `/home/user/Projects/awp-i0-c-execution` - Required start point: current `origin/main` after C0, initially `75d80546d78380e7ee317a9df46964d951832ac1` - Recorded C0 implementation baseline: `114b57a4ad37f9178b9c6309497c39663d4580e9` - C1 landed during Lane C: `08276e7` (`feat(core): establish C1 domain application persistence contracts`) - Lane C rebased cleanly after C1 onto current main history (`89d5300` at convergence time); no shared provider contract change was required. - State: independent I0 Lane C implementation complete; post-C1 repository gates green; available live proofs executed; unavailable live proofs remain explicitly red/pending. ## Delivered implementation ### DBOS durable workflow provider - Added `@awp/provider-workflow-dbos` behind the existing `DurableWorkflowProvider` port. - Pinned the actual provider dependency at `@dbos-inc/dbos-sdk@4.26.10` in the Lane C-owned provider manifest. - Bound the structural adapter to the real SDK API and compiled it against the pinned SDK; DBOS SDK types do not leak into contracts/domain/application. - Deterministic AWP `OperationId` -> DBOS workflow identity (`awp:`). - Explicit DBOS workflow identity is the durable idempotency identity. - Ambiguous start outcomes reconcile by deterministic workflow ID before an upstream retry can duplicate the operation. - Cancellation is explicit and reconciled from persisted provider state. - DBOS statuses are normalized into AWP provider observations; raw DBOS workflow state is not product/domain authority. - Added `runIdempotentSideEffect()` mechanics: persist `started`, reconcile after an ambiguous crash/restart, and persist `completed` rather than blindly repeating an external mutation. - Added a real PostgreSQL/DBOS live-proof worker and bounded buildbox harness. ### Kubernetes WorkspaceProvider - Added `@awp/provider-workspace-kubernetes` behind the existing `WorkspaceProvider` port. - Deterministically creates Pod, PVC when persistent, ServiceAccount, NetworkPolicy and optional Service resources. - Added server-side-apply Kubernetes HTTP transport for only the WorkspaceProvider resource families. - In-cluster transport explicitly reads the mounted ServiceAccount bearer token **and Kubernetes CA certificate**, using the CA for API-server TLS rather than assuming the host trust store contains the cluster CA. - Pod security baseline: non-root UID/GID, RuntimeDefault seccomp, no privilege escalation, non-privileged, all Linux capabilities dropped, read-only root filesystem, explicit CPU/memory requests/limits. - Workspace ServiceAccount and Pod service-account-token automount are disabled. - Namespace/default and per-workspace networking are deny-by-default with only declared DNS/CIDR/service access. - RuntimeClass is an immutable execution-profile property; intended untrusted baseline is `gvisor` where the cluster actually provides it. - Durable WIP is separate from disposable compute: pod replacement reuses the same PVC; logical `destroy()` removes compute/network identity but retains persistent WIP. - Destructive checkpoint cleanup requires `collected=true` plus the current checkpoint digest. - Workspace profiles reject Git-publication, merge and production-deploy credential projections before resource creation. ### Execution-profile resolution Resolution follows the approved project-native order: 1. `.devcontainer/devcontainer.json` / `.devcontainer.json`; 2. Docker Compose; 3. Dockerfile; 4. mise/Devbox; 5. language/tool manifests; 6. AWP fallback profile only for missing orchestration metadata. Dev Container JSONC is supported. A Dev Container `build` declaration is surfaced as `requiresImageBuild=true`; the provider does not pretend a fallback image is the project-native build output. Resolution records provenance. ### GitHub Actions CI provider - Added `@awp/provider-ci-github` behind the existing `CIProvider` port. - `workflow_dispatch` correlation uses deterministic `awp::` run-name identity plus exact candidate digest. - Retry searches for an existing correlated run before dispatch. - Timeout/ambiguous dispatch reconciles before failure; GitHub visibility gaps use a temporary correlation reference that later resolves to the concrete run. - Added cancel/readback and actual branch-protection required-check discovery. - HTTP transport exposes Actions/check mechanics only; it has no ref publication, PR merge or Git mutation method. - GitHub credentials are resolved by `CredentialReferenceId` per request and disposed after the request. ### K3s / gVisor / ARC substrate - Added bounded `awp-system` / `awp-workspaces` namespace and namespace-scoped workspace-controller RBAC. - Workspace-controller RBAC has no Secret, pod-exec, Node, Namespace, CRD, ClusterRole, impersonation or cluster-admin authority. - Added resource quota/defaults and namespace default-deny NetworkPolicy. - Added `gvisor` RuntimeClass manifest (`handler: runsc`) without treating the manifest itself as runtime evidence. - Added ARC trust-tier namespaces/scale-set values: - `awp-untrusted` — default/general, gVisor, non-root/unprivileged, service-account token disabled; - `awp-trusted-internal` — separate internal trust pool, still non-root/gVisor/token-disabled; - `awp-privileged-build` — explicit exceptional Docker-in-Docker profile, never default/fork/untrusted routing. - Versioned ARC configuration contains only a GitHub authentication Secret **reference** (`awp-arc-github-app`), never credential material. ### Preflight / live-proof harnesses - `workspace-wip-smoke.sh --run [--yes]` — bounded namespace/PVC pod-replacement proof with synchronous cleanup; optional explicit RuntimeClass. - `gvisor-toolchain-smoke.sh --run [--yes]` — representative Node/TypeScript/React-base and PHP/WordPress toolchains under `runtimeClassName: gvisor`; fails rather than silently weakening to native. - `arc-ephemeral-proof.sh` — AWP-compatible workflow-dispatch/runner observation harness. - `dbos-live-proof.sh --run [--yes]` — temporary user-local PostgreSQL + real DBOS crash/restart/retry/cancel/idempotency/persistence proof on an authorized build/test host. ## Automated tests Lane C tests cover: - DBOS deterministic identity, duplicate suppression, ambiguous-start reconciliation, persisted cancellation and crash-after-side-effect reconciliation. - Kubernetes Pod/PVC/ServiceAccount/NetworkPolicy/RuntimeClass mechanics, server-side-apply request shape, WIP checkpoint identity across pod replacement, safe cleanup handshake and Dev Container resolution. - GitHub CI deterministic correlation, ambiguous dispatch reconciliation, cancellation/readback, required checks and candidate validation. - ARC trust-profile separation, default unprivileged/gVisor/ephemeral baseline and secret-reference-only versioned configuration. - execution security: forbidden publication/merge/deployment credential projections, restricted RBAC, default-deny networking, no reusable publication credential markers, and no CI ref-publication/merge API path. ## Post-C1 repository validation Validation ran on authorized buildbox `debian1`, not on the workstation. Toolchain: - Node `v22.23.2` via mise. - pnpm `9.7.0`. - DBOS SDK `4.26.10` installed from the Lane C provider manifest in the temporary buildbox copy. Latest aggregate result after C1 rebase and real SDK/Kubernetes-CA binding: - `pnpm check` -> PASS. - `pnpm typecheck` -> PASS. - `pnpm lint` -> PASS. - `pnpm test` -> PASS: **16 files / 48 tests**. - `pnpm build` -> PASS, including the real DBOS SDK binding. - `pnpm test:architecture` -> PASS: 6 architecture tests; dependency-cruiser **0 violations across 55 modules / 71 dependencies**. - `pnpm format:check` -> PASS. - Final aggregate rerun after the live-proof harness lint correction -> PASS with the same 16 files / 48 tests and 55 modules / 71 dependencies. The temporary buildbox copy uses `pnpm install --no-frozen-lockfile` because root `pnpm-lock.yaml` is outside Lane C ownership; see the convergence request below. ## Live infrastructure discovery Intended K3s cluster reached through `overdeck-vm` and the buildboxes: - `debian3` is K3s control-plane. - `debian1`, `debian2`, `debian3` are Kubernetes `Ready` on K3s `v1.36.3+k3s1`. - default StorageClass: `local-path`, `WaitForFirstConsumer`. - ARC CRDs installed: autoscaling runner sets/listeners plus ephemeral runner/runner-set resources. - ARC controller and existing `arc-k3s` listener are running. - Existing scale set `arc-ci/arc-k3s`: GitHub config URL `https://github.com/alexcodeplace/overdeck`, `minRunners=0`, `maxRunners=1`. - Current RuntimeClasses include `crun`, Wasm-related profiles and NVIDIA profiles; **no `gvisor` RuntimeClass is installed**. Installing gVisor would require a real K3s/containerd runtime change and likely node service restart. Lane C did not disrupt the shared active cluster merely to make that row green. ## Live proof evidence ### DBOS + PostgreSQL — GREEN Executed `tools/preflight/dbos-live-proof.sh --run --yes` on `debian1` using: - Node `v22.23.2`; - PostgreSQL `17` in a temporary user-local data directory; - real `@dbos-inc/dbos-sdk@4.26.10`; - workflow `awp-live-1787279640-3385892`. Observed/proved: - workflow reached a durable `recv` wait; - process terminated abruptly with exit code `137` **without `DBOS.shutdown()`**; - next DBOS process recovered and resumed the same workflow from PostgreSQL; - retrying step failed intentionally twice and succeeded on attempt 3; - duplicate workflow start with the same workflow ID returned the persisted result; - checkpointed side-effect marker executed exactly once; - duplicate message delivery used the same DBOS idempotency key; - cancellation persisted as `CANCELLED`; - terminal workflow status persisted as `SUCCESS`; - terminal result persisted across a third DBOS launch: `{message:"go", retryAttempts:3}`; - temporary PostgreSQL/state was removed by the harness cleanup. ### Native K3s WIP/pod replacement — GREEN for this slice Executed the bounded WIP smoke on the intended K3s control-plane with native runtime because gVisor is unavailable. Observed: - temporary namespace created; - RWO `local-path` PVC created; - non-root writer pod wrote `checkpoint-awp`; - writer pod deleted; - replacement non-root reader pod mounted the same PVC; - reader emitted `WIP_PERSISTED`; - temporary namespace/resources were subsequently removed. This proves PVC/pod-replacement durability. It does **not** claim gVisor, recoverable node interruption, or an AWP control-plane restart as green. ### ARC ephemeral runner lifecycle/security — GREEN for lifecycle/isolation substrate A bounded real proof used the existing `arc-k3s` scale set and a temporary `wt/awp-arc-proof-20260821021905` ref in `alexcodeplace/overdeck`. The ref was deleted after the proof. GitHub run: `32439433857`. ARC runner pod observed: `arc-k3s-z25rk-runner-55z6p`. Live pod inspection showed: - `runAsNonRoot: true`, UID/GID `1001`; - `seccompProfile: RuntimeDefault`; - `allowPrivilegeEscalation: false`; - all Linux capabilities dropped; - `automountServiceAccountToken: false`; - no `GH_TOKEN`, `GITHUB_TOKEN`, publication, or merge credential marker in the runner pod environment; - runner pod disappeared after the GitHub job reached terminal state. The GitHub job itself concluded **failure** because the existing Overdeck collector test suite is currently red (the run executed 1006 passing tests plus current collector failures). A prior current-main run on the same main SHA was already failing, so this is not evidence of ARC runner failure. Therefore: - ARC ephemeral lifecycle/security substrate: **GREEN**; - “successful real repository workflow” readiness row: **PENDING/RED** until the repository's own collector workflow is green or an AWP-compatible successful workflow is available. ### gVisor — PENDING/RED No `gvisor` RuntimeClass/runsc runtime exists on the intended cluster. Static manifests and automated tests are green, but live Node and PHP/WordPress gVisor compatibility is **not** claimed. ### Remaining disruptive/unavailable live slices — PENDING/RED - recoverable node interruption WIP proof; - AWP control-plane restart while a live Workspace is active; - target AWP gVisor runner/workspace proof; - successful repository workflow on the live ARC scale set while the existing collector workflow remains red. These are intentionally not substituted with workstation-local, simulated, or native-runtime evidence. ## Live-proof status matrix | Gate | Status | Evidence / reason | |---|---|---| | DBOS crash/restart/resume | **GREEN live** | Real DBOS 4.26.10 + temporary PostgreSQL 17 on `debian1`; abrupt exit 137, recovery and terminal persistence proved. | | DBOS retry/cancel/duplicate/idempotent-side-effect | **GREEN live + automated** | Attempts 1/2 failed, attempt 3 succeeded; cancellation `CANCELLED`; duplicate start returned same result; side effect count 1. | | Native K3s PVC/pod replacement WIP | **GREEN live** | `WIP_PERSISTED` after writer pod deletion/replacement on intended cluster. | | AWP/control restart + recoverable node interruption | **PENDING** | Not safely exercised on the shared active cluster. | | gVisor RuntimeClass/toolchain compatibility | **PENDING / environment blocker** | No gVisor/runsc runtime registered on intended K3s cluster. | | Dev Container resolution | **GREEN automated** | Precedence, JSONC and build-required behavior tested. | | ARC installed / CRDs / listener | **GREEN live discovery** | Controller, CRDs, listener and scale set observed. | | ARC ephemeral runner pod lifecycle | **GREEN live** | Real runner pod observed and removed after terminal job. | | ARC successful repository workflow | **PENDING** | Existing Overdeck collector suite currently fails independently of runner lifecycle. | | No reusable Git publication credentials in normal workspace/runner | **GREEN static + live ARC inspection** | Workspace rejects publication/merge purposes; live runner pod had no reusable publication credential marker and no SA token mount. | | Workspace RBAC/network/non-root isolation | **GREEN automated/static** | Exact resources/manifests tested; no broader live admission claim. | ## Shared root lockfile convergence — COMPLETE Final convergence regenerated and validated the root `pnpm-lock.yaml` with the Lane C provider workspace importers: - `packages/providers/workflow-dbos`, including `@dbos-inc/dbos-sdk@4.26.10`; - `packages/providers/workspace-kubernetes`; - `packages/providers/ci-github`. The final buildbox convergence run used authorized `debian1`, Node `v22.23.1`, and pnpm `9.7.0`. From a truly clean install state with no pre-existing `node_modules`: - `CI=true pnpm install --frozen-lockfile` -> **PASS** (exit 0); all 13 workspace projects were recognized and pnpm reported `Lockfile is up to date, resolution step is skipped`. - `pnpm check` -> **PASS** (exit 0): typecheck, lint, test, build, architecture and format gates all green; Vitest **16 files / 48 tests**; dependency-cruiser **0 violations across 55 modules / 71 dependencies**. - focused Lane C integration/security run -> **PASS** (exit 0): **5 files / 23 tests** covering DBOS, Kubernetes workspace, GitHub CI, ARC manifests and execution security. ## Final convergence live-gate status The convergence work does **not** change the previously recorded live readiness status. These three gates remain explicitly red/pending: - **gVisor compatibility:** PENDING/RED because the intended K3s cluster still has no `gvisor` RuntimeClass/runsc runtime. - **recoverable node interruption / AWP control-plane restart:** PENDING/RED; not safely exercised on the shared active cluster. - **successful repository workflow:** PENDING/RED because the existing repository collector workflow remains independently red; ARC lifecycle/security substrate proof remains green but is not substituted for a successful repository workflow. ## Completion / handoff Lane C implementation and final shared-root lockfile convergence are complete at the owned boundary. The clean frozen install, full repository gate, and focused Lane C integration/security suite all pass on authorized build/test infrastructure. The three live gates above remain intentionally red and unchanged; no claim is made that they are green.