---
name: od-recovery
description: Machine-down incidents — system crash, lockout, graphics/compositor death, kernel panic, black screen, unresponsive desktop, won't boot. Triggers on hang, frozen UI, GPU/Xorg/Cinnamon crash, watchdog reboot, SSH lockout. Invoke BEFORE rebooting.
---

# Machine down — incident playbook

audience: AI coding agents first. BLUF: capture evidence → check VT/console → minimal recovery → reboot only with journal in hand.

## Covers

**Absorbs:** `system-crash`, `machine-lockout`, `graphics-crash`.

**Keywords:** crash, lockout, locked out, hang, frozen, black screen, reboot, graphics, gpu, xorg, cinnamon, compositor, kernel panic, watchdog, unresponsive, dead, won't boot.

**Registry coverage:** 16 of 217 (2026-08-08). Often follows resource overload on shared kernel — OOM/IO storm takes down graphics env.

## Doctrine

Capture evidence BEFORE reboot. Check `/sys/class/tty/tty0/active` first.

## First steps — evidence before power cycle

1. **Still responsive?** SSH from another host if possible. If SSH up → gather remote; if not → physical/serial path only.
2. **VT state:** `cat /sys/class/tty/tty0/active` — which VT active. Graphics crash often VT7 dead, VT1-6 may work (`Ctrl+Alt+F2`).
3. **Journal (before reboot):** `journalctl -b -p err..alert --no-pager -n 200`; `journalctl -b -u user@*.service --no-pager -n 100`. Save output to artifact file.
4. **OOM/GPU signals:** `journalctl -b -k | grep -iE 'oom|gpu|drm|xorg|cinnamon'`; `dmesg -T | tail -100` if permitted.
5. **kdump/pstore:** if configured, note whether crash dump captured — do not overwrite.
6. **Recovery ladder:** switch VT → restart compositor/session ONLY if journal points there → full reboot LAST. Reboot without journal = evidence destroyed.
7. **Lockout root cause check:** past lockouts traced to sshd misconfig on buildboxes — NEVER edit sshd config (never-touch).

## Never-touch — applies here

From `modules/workstation/claude/incidents/never-touch.md`:

- NEVER edit sshd config on buildboxes.
- NEVER read `~/stupid.user` — root via `deck-sudo` only; absent password file = no root.
- NEVER kill owner gnome-terminal sessions.
- NEVER destructive git in shared main checkout.

## Placement map — read these paths

`modules/workstation/claude/incidents/placement-map.md`:

- `~/.local/state/overdeck` — journals, `actions.jsonl`, harness crash evidence.
- `~/.overdeck` — systray/runtime state.
- Buildbox registry `~/.claude/buildbox-hosts.json` — if crash was on remote box, reboot box not laptop.

## Resolve — exact CLI (never guess syntax)

```
od-incidents list [--type machine-down] [--state S]
od-incidents show <id>
od-incidents search <query>
od-incidents resolve <id> --artifact <ref> [--summary <line>]
```

`resolve` REFUSES without `--artifact` (exit 2). Artifact = saved journal excerpt + recovery command output, or fix sha preventing recurrence.

## Self-update — MANDATORY

This doc is the fleet's memory. You changed the architecture, wiring, commands, or
doctrine this skill describes — or a live incident just proved a rule here wrong or
missing → EDIT THIS FILE in the same landing as the change. Adopted AND rejected
decisions both go in, with the measured why. An owner reminder to record a lesson is
a failure of this rule, not the trigger for it.