# Land queue loses a live ticket

- status: DONE
- outcome: the canonical land queue never drops a ticket whose owner is alive; every live owner receives a verdict, or fails closed with an accurate reason.
- source request: real incident — ticket.3b8ce7f97c0747c58ac662687382a6bc waited 810s, conducted, then hit `land-queue-failed: conducted the queue without a verdict for this ticket` (finish-branch.sh:1185). Post-mortem: the ticket's `.job` and `.lock` were gone and `queue` was empty.
- scope: `modules/workstation/claude/workflows/lib/finish-branch.sh` (`_landq_live`, `_landq_release`, `_landq_serve`, `_landq_conduct`, `_landq_verdict_write`, `_stage_a_land`) and `test-finish-branch.sh`.
- constraints: never weaken a gate; a missing verdict never reads as success; FIFO, single-conductor and ownerless-ticket pruning stay intact; genuinely unexpected state stays fail-closed. Worktree `.worktrees/ask-gpt-download-progress` only — no push, no deploy, no landing controller.

## Findings

- `flock` conflict detection is intact: a fresh OFD on an existing lock file is blocked from a foreign process, a plain subshell, a process-substitution child, and even the same process. The "process substitution inherits the ticket-lock fd, so the child's lock is redundant" hypothesis is disproven by probe.
- `_landq_live` opens the ticket lock with `exec {ofd}>"$dir/$t.lock"`, which is `O_CREAT`. Once the path is unlinked, every later lister recreates it as a *fresh inode*, acquires it trivially, and prunes — proven: after `rm -f`, a foreign process acquires the recreated path while the original owner still holds the old inode. The prune is therefore self-perpetuating, so the real question is what unlinks the path first.
- No component outside `finish-branch.sh` unlinks queue state: the collector source is read-only and both pre-push hooks touch only `conductor.lock`.
- The incident returned at line 1185, not 1182, so `_landq_conduct` returned 0 — the owner's ticket was absent from `_landq_live`'s output entirely (pruned, or lost from `queue`).

## Root cause (proven)

`_landq_conduct` piped the lister into its loop: `done < <(_landq_live "$dir")`. A process
substitution's exit status is unreachable, so a lister that FAILED was indistinguishable from a
lister that found an EMPTY queue — and "no ticket to serve" is the success exit
(`(( served )) || return 0`). The conductor therefore returned 0 having served nothing.

`_landq_live` has real failure paths: `exec {qfd}>"$dir/queue.lock"`, `flock`, and
`exec {ofd}>"$dir/$t.lock"` (one open per ticket per pass) all `return 3` on `open(2)` failure —
reachable under fd exhaustion, EACCES or ENOSPC on a loaded box.

Proof, driving the real functions:
- stubbed failing lister → `_landq_conduct` returns **0** on HEAD.
- real fd starvation (`ulimit -n 12`) with one live queued ticket → `_landq_live` returns 3 and
  `_landq_conduct` returns **0** on HEAD.

The owner then reached the `[[ ! -f "$qdir/$ticket.verdict" ]]` branch and emitted exactly
`land-queue-failed: conducted the queue without a verdict for this ticket` — the incident, and a
wrong diagnosis of what was really a lister failure.

Disproven along the way (each by experiment, not reasoning): pruning a live owner's ticket is
impossible — a fresh OFD on the ticket lock is blocked from a foreign process, a subshell, a
process-substitution child that inherits the owner's fd, and the owner's own process; the lock
survives command substitutions, process substitutions, external commands and varfd churn;
`_landq_live` never drops a live ticket from `queue`, including mixed live/dead queues. The
post-mortem residue (`.job`/`.lock` gone, `queue` empty) is not incident evidence: line 1185
returns 20 *before* `_landq_release`, so the owner exited still holding its lock, and the next
lister pruned the now-genuinely-ownerless ticket legitimately.

Separate observation, not fixed (out of scope): `_landq_conduct` serves from a snapshot, so a
ticket released by its owner mid-pass is still served, writing an orphan
`ticket-rejected: job description missing` verdict. Several such orphans exist on disk. They land
on tickets whose owners already succeeded, so no owner is harmed.

## Fix

1. `_landq_conduct` captures the listing (`listing=$(_landq_live "$dir") || return 4`) so the
   lister's status propagates. New exit 4 = "could not list the queue", distinct from 3 =
   "a live ticket got no verdict".
2. `_stage_a_land` reports rc 4 as `conductor could not list the land queue` — an accurate
   diagnosis instead of the misleading one.
3. The enqueue appends to `$ticket.job` and `queue` are checked; an unchecked append leaves a
   ticket nobody serves while its owner waits forever.
4. The fail-closed `return 20` paths now `_landq_release` the ticket. Abandoning it without
   releasing left an entry no owner would ever collect, inflating `queue_depth` for every later
   lander — which is why the incident ticket queued behind depth 2 and waited 810s.

Gates unchanged: FIFO, single conductor, ownerless pruning and every verdict check are intact;
no missing verdict can read as success.

Known residuals of the fix, both benign and deliberate:
- On the rc≠0 branch, `_landq_release` also deletes `$ticket.verdict`. If this owner's ticket was
  served successfully earlier in the same pass and a later sibling then failed the conduct, the
  owner erases its own good verdict. It already returned 20 there on HEAD, and `_landq_serve`'s
  git-ancestry recovery still short-circuits the retry, so no correctness is lost — only a
  forensic artifact.
- Partial-append guard: if the `.job` append succeeds and the `queue` append fails, `.job`/`.lock`
  linger with the ticket never queued, so no lister prunes them. Rare, cosmetic.

## Acceptance criteria

1. Mechanism proven by a reproduction that drives the real functions, not by reasoning.
2. A permanent regression in `test-finish-branch.sh` that is deterministic — red on the unfixed lib, green after — asserting a live owner's ticket is never pruned and always receives a verdict.
3. `bash test-finish-branch.sh`, `node modules/workstation/claude/tests/local-gate.test.mjs`, `LC_ALL=C python3 -m pytest modules/harness/factory/tests/ -q` all green, warnings fixed or justified.
4. Local commit only.

## Execution steps

1. Instrumented concurrent stress against throwaway repos to observe the first unlink. (in progress)
2. Convert the observed interleaving into a forced-interleaving deterministic regression.
3. Fix; re-run the regression against the pre-fix lib to confirm it was red.
4. Run the three suites; commit locally.

## Receipt

- status: DONE. Landed in `origin/main` and locally deployed at `052b1f706d44a00c2f0d42bcb2dda68ad2322443`; installed `finish-branch.sh` contains `listing=$(_landq_live "$dir") || return 4`.
- Root cause proven by experiment; fix landed in the worktree with three permanent regressions
  (LQ-L1, LQ-L2, LQ-L3) verified RED on HEAD (`_landq_conduct` rc=0) and GREEN after (rc=4).
- `bash modules/workstation/claude/workflows/lib/test-finish-branch.sh` → PASS=109 FAIL=0.
- `node modules/workstation/claude/tests/local-gate.test.mjs` → passed.
- `LC_ALL=C python3 -m pytest modules/harness/factory/tests/ -q` → 151 passed in 154s.
- No warnings emitted by any suite.

Next executable action: none. Installed runtime verified. If the orphan-verdict/stale-snapshot observation is to be fixed, open a separate plan.