# Target-bound deploy retries — request

**Owner-visible evidence first:** the installed `/ci` deploy warning distinguishes failures that
will retry from failures that need a person, names the exact landed commit, and shows when the next
bounded retry is due. Installed status JSON records that same full target SHA and classification
before an injected early failure.

**Goal:** make one retry decision from one exact deployment target. Never count a previous deploy's
terminal record against current main, retry a deterministic blocker, or enqueue on every watcher
tick.

## Scope

- `packaging/deploy-local.sh`: put a validated full target SHA in every queue request; select and
  publish the consumer's exact target before admission failures; deploy that pinned commit; publish
  schema-2 progress with `target_sha` and `failure_class`.
- `controller/src/deploy-watcher.ts`: match terminal outcomes by full `targetSha`, classify legacy
  outcomes conservatively, stop permanent failures immediately, and schedule transient retries with
  bounded backoff.
- `controller/src/store.ts` and status schema: persist/expose classification and `nextRetryAt` across
  controller restarts.
- Collector deploy-status adapter and the existing `/ci` delivery model: carry the fields through and
  render truthful owner language. No new component or primitive.
- Focused shell failure-injection plus controller/collector/web behavior tests.

## Contracts

1. Queue request content is one 40-character lowercase commit SHA. Legacy empty requests remain
   readable during rollout, but every new request is target-bound.
2. The consumer resolves one immutable target, publishes it in progress before disk,
   lock, checkout, build, or activation failure, and checks out that exact commit rather than a
   moving `origin/main` ref. New requests provide the target directly; targetless `--now` and legacy
   requests fetch before resolving it so they cannot deploy a stale remote-tracking ref.
3. Failure class is explicit: `transient`, `permanent`, or `none`. Unknown/legacy failures default to
   permanent so ambiguity cannot create a retry storm. Lock contention remains a non-failure.
4. Transient retries use persisted UTC eligibility deadlines from an injected watcher clock and
   bounded delays of 60s, 300s, then 900s. Persisted wall-clock deadlines survive controller
   restarts; monotonic clocks remain the contract for in-process readiness timeouts. A due
   retry advances its next eligible time before enqueueing, so repeated reconcile ticks cannot flood
   the queue. Three distinct failed runs exhaust the target.
5. Permanent failures never auto-retry the same target. A new target resets state. A later successful
   manual deploy still supersedes the stopped record.
6. A failed consumer run moves each exact-target or ancestor request atomically from the active path
   queue to a durable sibling deferred queue, while unrelated requests remain active. This prevents
   `DirectoryNotEmpty` from relaunching the same failed deploy continuously while preserving request
   evidence. The watcher alone reactivates transient work at its persisted deadline; permanent work
   stays deferred. A later successful deploy acknowledges exact-target and ancestor requests from both
   active and deferred queues.
7. Existing rollback and per-service restart gates are unchanged.

## Acceptance

- Early disk/fetch failure fixture contains the exact full intended target and an explicit class.
- Moving `origin/main` after target selection cannot change the checked-out target.
- Fake-clock tests cover not-before scheduling, one enqueue per backoff window, increasing bounded
  delays, permanent stop, exhaustion, restart persistence, new-target reset, legacy-safe fallback,
  and success superseding a stopped record.
- Collector schemas accept both legacy schema 1 and target-bound schema 2 during rollout.
- `/ci` labels say `retrying in …` only for scheduled transient work and `needs a person` immediately
  for permanent work.
- Focused suites, static checks, controller/collector/web typechecks, then installed full-flow proof.

## Local verification — 2026-08-16

- `packaging/test-deploy-local.sh`: `PASS=48 FAIL=0`, including targetless `--now`, early
  transient fetch failure, unavailable-target permanent failure, pinned checkout, durable failure
  deferral outside the active systemd path queue, exact/ancestor active and deferred acknowledgement,
  and preservation of unrelated requests.
- Controller watcher: 20 focused tests passed; controller and collector broad Bun suites passed.
- Changed `/ci` model/component tests: 39 passed. Controller, collector, and web typechecks passed;
  web reported zero errors and two pre-existing `FormEvent` deprecation hints. Web production build passed.
- The broad web suite has an unrelated gap in untouched tests: `DecisionsApp.test.tsx` expects the
  contiguous text node `2 waiting` although the rendered counter splits the number and label, and
  `RequestsBoard.test.tsx` times out restoring a card after a mocked send failure. Both repeated in
  an isolated diagnostic run, so they were not retried again under the two-failure rule. The changed
  `/ci` tests are green.
- Installed `/ci` and schema-2 runtime evidence remain required after the exact landed revision is
  deployed.

## Installed verification — 2026-08-17

- `origin/main` and the deploy clone both resolved to
  `538880f7ff830c8e5b6eac64471320c05d53373f`; the installed deploy entrypoint emitted schema 2 with
  that full `target_sha` before its terminal failure.
- A real permanent failure left zero active requests and three durable deferred requests. With
  `overdeck-deploy.path` enabled and active, `overdeck-deploy.service` remained stopped at its prior
  `ExecMainStartTimestamp`, with `NRestarts=0`; deferred work did not retrigger `DirectoryNotEmpty`.
- Authenticated installed collector `/state` exposed the same target with `attempts: 1`,
  `failureClass: permanent`, and `nextRetryAt: null`. The installed `/ci` delivery formatter rendered:
  `Auto-deploy could not install 538880f (...) — needs a person to look`.
- The installed controller source passed the focused fake-clock retry proof: one transient failure
  stayed inactive before 60 seconds, enqueued exactly once when due, and persisted the 300-second
  next window before enqueue (`1 pass`, `8 expect() calls`).
- Installed typed readiness passed for collector, controller, botmaster, and web.
- The complete deploy could not publish a new success stamp because the independent Debian 3 sandbox
  image parity gate is red. The deploy remained fail-closed; this slice's source, queue units, and
  entrypoint were installed and proved, but direct browser automation of `/ci` was not run because the
  installed service is loopback-only and the required browser runner executes remotely.