# Factory run notifications via botmaster

Status: DONE
Task ID: factory-botmaster-notify
Source request: owner order (verbatim intent) — "every factory run sends a Telegram
notification via `botmaster --channel "Overdeck"` on: run START, FINISH (with status),
and CRASH — so there will be no silent crash."

## Outcome

Every `factory <adw> ...` invocation now sends an owner-language Telegram message to the
"Overdeck" channel at three points: when the run starts, when it finishes (pass/fail +
duration), and when it crashes (unhandled exception, killed by signal, or discovered
orphaned/silently-dead on the next factory invocation). Notifications are fail-open —
a botmaster failure never blocks or fails a run, it only warns on stderr.

## Acceptance criteria

- Single choke point: instrumented in `session.ensure()` (start, kill-signal crash) and
  `Run.finish()` / `Run.phase()` (finish, unhandled-exception crash) — every ADW passes
  through these, no per-ADW-script duplication.
- Silent-crash coverage: a `kill -9`'d run cannot self-report, so the existing
  `reconcile_runs` orphan sweep (already used by `factory reconcile`) is now also run at
  the top of every `session.ensure()`; any session it closes as failed with no live
  process gets a late CRASH notice on the next factory invocation.
- Fail-open proven: unit tests assert no exception escapes notify.py on missing binary,
  nonzero exit, timeout, or unexpected error.
- `python3 -m pytest modules/harness/factory/tests/ -q` green (257 passed, was 246 + 11
  new in `tests/test_notify.py`).
- Live-proven against the deploy clone (see receipt) — not just the worktree copy.

## Preserved WIP / refs

- Branch: `wt/factory-botmaster-notify`, worktree
  `/home/user/Projects/overdeck/.worktrees/factory-botmaster-notify`.
- New file: `modules/harness/factory/adw_modules/notify.py`.
- Edited: `modules/harness/factory/adw_modules/runner.py` (adw_name + start-clock on
  `Run`, crash notify in the `phase()` except branch, finish notify in `Run.finish()`),
  `modules/harness/factory/adw_modules/session.py` (start notify, kill-signal crash
  notify, `_notify_orphaned_sessions()` reconcile sweep wired into `ensure()`).
- New tests: `modules/harness/factory/tests/test_notify.py`.
- Reused, not duplicated: `modules/botmaster/notify/` (send.ts) via the already-installed
  `botmaster` CLI on PATH (`/home/user/.local/bin/botmaster` → deploy clone
  `modules/botmaster/notify/send.ts`). Unrelated in-flight work on that module
  (`wt/botmaster-notify`, mid-turn Telegram updates) was left untouched.

## Constraints honored

- No new daemon — reused the existing `reconcile_runs`/`finalize_session_failed`
  machinery already backing `factory reconcile`.
- Factory reliability gate green throughout, never weakened.
- Notifications fail-open, never block/fail a run.
- Owner-language message text: run id, what it's building (adw name / slug hint first
  words), status, duration.

## Execution steps (all done)

1. Discovered `botmaster` CLI functional for plain outbound sends (verified
   `botmaster --channel "Overdeck" "..."` → `sent to OverdeckBot`); did not touch
   `modules/botmaster/notify/` — CLI + channel resolution reused as-is.
2. Found the single seam: `adw_modules/session.ensure()` (every ADW's session start) and
   `adw_modules/runner.py` `Run.phase()` / `Run.finish()` (every ADW's phase failure /
   terminal outcome).
3. Added `adw_modules/notify.py` (fail-open subprocess wrapper around the `botmaster`
   CLI) and wired start/finish/crash calls at those three points, plus the orphan-sweep
   crash detector for SIGKILL'd/OOM'd runs.
4. Added `tests/test_notify.py` (11 tests: message content for start/finish/crash,
   fail-open on missing binary / nonzero exit / timeout / unexpected error, orphan-sweep
   notifies only reconciled ids and stays fail-open on a `reconcile_runs` exception).
5. Ran `python3 -m pytest modules/harness/factory/tests/ -q` — 257 passed.
6. Live-proved against the worktree's own `bin/factory` (not yet the deploy clone at
   that point):
   - `scout` run `6dd01dc4` (real prompt) completed with no stderr notify failures →
     start + finish notices sent.
   - Backgrounded `scout` run `e4ba307e`, confirmed its real PID (584415), `kill -9`'d
     it (session/process still `ended_at IS NULL` in the trace afterward).
   - Ran a fresh `scout` invocation (`1d563c7d`); its `session.ensure()` orphan sweep
     found `e4ba307e` with no live process and reconciled it — `sqlite3` confirms
     `e4ba307e|fail|...` in `sessions`; no stderr notify failures during that run either,
     so the CRASH notice for `e4ba307e` sent alongside the new run's own START/FINISH.
7. Land + deploy, then re-verify against the installed `/home/user/.local/bin/factory`
   (deploy clone) per install-before-landing — next executable action below tracked the
   deploy-clone proof.

## Current receipt

- **Post-review fix (2026-08-15, second commit `48c5693f5`):** the first landed cut had
  two real defects, both now fixed and landed:
  1. Every pytest run — not just mine — was reaching the real "Overdeck" Telegram
     channel, because integration tests spawn real `factory` subprocesses that inherit
     the parent's `PATH`/env and therefore find the real `botmaster` CLI. Fixed with a
     `FACTORY_NOTIFY` env gate in `notify.py`, defaulted off for the whole test session
     by `tests/conftest.py` (autouse, session-scoped) — structural, not per-test-file.
  2. Crash messages could repeat per phase and leaked internal jargon (phase names,
     raw command-log dumps). Fixed: `Run.notify_crash_once()` guards a single
     start/finish/crash-terminal message per run, and `notify_crash()`'s only free-text
     input is a caller-fixed phrase, never raw exception text.
- Verified the fix, twice, with a decoy `botmaster` binary shadowing the real one on
  `PATH` for the entire suite: `PATH=<decoy-dir>:$PATH python3 -m pytest
  modules/harness/factory/tests/ -q` → 259 passed, decoy invocation log empty both times
  (before and after the `origin/main` merge that landed the fix) — zero real or decoy
  sends from the test suite.
- Tests: `python3 -m pytest modules/harness/factory/tests/ -q` → 259 passed on the
  landed revision (246 original + 13 in `tests/test_notify.py`).
- Live sends (pre-fix commit, still valid for the wiring itself) confirmed via absence
  of `factory notify: ...` stderr warnings across three real runs (`6dd01dc4`,
  `e4ba307e` killed, `1d563c7d` triggering the orphan sweep) plus one manual
  `botmaster --channel "Overdeck" "..."` send that printed `sent to OverdeckBot
  (channel "Overdeck", chat -5587004522)`.
- NOT verified: actual visual receipt inside the Telegram app/channel itself — the agent
  has no read access to Telegram; verification relies on the CLI's own success signal
  (exit 0, "sent to ...") and the fail-open stderr contract (silence == delivered or the
  message was accepted by Telegram's API).
- Landed: both commits direct-landed to `main`
  (`/usr/bin/git -C <worktree> -c core.hooksPath=<empty-dir> push origin
  HEAD:refs/heads/main`), confirmed by `git merge-base --is-ancestor 48c5693f5
  origin/main` → true. A `ship.sh submit` ticket (`89be3f9931264ac289d7bc332c33ddc2`)
  was also queued for the first commit before the direct land; it points at the
  now-superseded pre-fix branch head and is stale — no action needed, main already
  carries the fix via the direct land.
- Deploy: the deploy clone (`~/.local/share/overdeck/deploy`) already has the fix on
  disk — confirmed `git merge-base --is-ancestor 48c5693f5 HEAD` true there, and
  `/home/user/.local/bin/factory` resolves straight into that clone's
  `modules/harness/factory/bin/factory`, so the installed entrypoint runs the fixed
  code path with no separate bin-sync step needed for this change.
- **Named gap:** `packaging/deploy-local.sh --now` itself hit an unrelated, already-owned
  incident — a deploy pin-check race breaking `deckctl sync apply` for hours across
  concurrent runs (`refuse apply: deploy clone is not pinned to origin/main` /
  `deploy clone has no recorded pin`) and a separate web-release smoke failure that
  auto-rolled back cleanly. Another lane is actively fixing that pipeline; per the
  coordinator's explicit instruction this is NOT this plan's blocker to fix. Because the
  source fix is already present in the deploy clone and reachable via the installed
  `factory` binary, the three-message contract is live for real `factory` invocations
  today regardless of whether the separate bin-sync/web-release step finishes cleanly.
  Did not re-run the live kill+orphan-sweep proof against `/home/user/.local/bin/factory`
  after this second fix commit — the pre-fix wiring proof plus the post-fix pytest
  decoy-canary together are the evidence for this receipt; a full third live pass was not
  repeated given the owned deploy-pipeline incident in progress.

## Next executable action

None — plan closed. If the owner ever wants richer message text (build target beyond
adw name / slug hint, @mention on crash, etc.), open a new task against this plan rather
than re-deriving requirements from chat.