autonomic-bot
cb20bea7cd
The 2026-08-03 cc-ci outage was recovered with ad-hoc tooling living in /tmp (leftover from a PREVIOUS incident, half-evaporated). Promoted to the repo: - scripts/recovery/hetzner.py — Hetzner API helper (status/actions/reboot/reset/ power/rescue-on|off/console), knows cc-ci=134485294 + orchestrator=134487234 by name; token from HCLOUD_TOKEN or /srv/cc-ci/.hcloud-token (0600, never in git). - scripts/recovery/hetzner-console.sh — shell-only VGA console: fresh console session -> websocat bridge -> vncdotool (venv auto-bootstrapped in ~/.cache). screenshot / key / type subcommands; encodes the reset-invalidates-session and single-connection-bridge gotchas. - scripts/recovery/README.md — the condensed 10-minute unreachable-server drill, incl. the GRUB submenu 1>N ids + clear-grubenv-after-switch rule. - hetzner-server-recovery skill: console/API sections now point at the repo tools instead of describing /tmp rebuilds. Smoke-tested: hetzner.py cc-ci status OK.
38 lines
2.5 KiB
Markdown
38 lines
2.5 KiB
Markdown
# Recovery tooling — permanent home
|
|
|
|
Promoted from `/tmp` ad-hocery after the 2026-08-03 cc-ci 26.05 outage (see
|
|
`.cc-ci-logs/server-update-2026-08-03.md` and the `hetzner-server-recovery` skill, which is
|
|
the *procedure*; these are the *tools* it uses).
|
|
|
|
| Tool | What it does |
|
|
|---|---|
|
|
| `hetzner.py <server> <cmd>` | Hetzner Cloud API: status, actions, reboot/reset/poweroff/poweron, rescue-on/off, console credentials. Knows `cc-ci` (134485294) and `orchestrator` (134487234) by name. |
|
|
| `hetzner-console.sh <server> screenshot\|key\|type` | Shell-only access to the VGA console: fresh console session → websocat bridge → vncdotool (venv auto-bootstrapped at `~/.cache/hetzner-console-venv`). |
|
|
|
|
**Token:** `HCLOUD_TOKEN` env, or `/srv/cc-ci/.hcloud-token` (chmod 600). Not in git, not in
|
|
`.testenv`. Prefer per-incident tokens and revoke after — and never paste tokens into chat
|
|
transcripts (the 2026-08-03 incident token was pasted mid-incident and had to be flagged for
|
|
rotation).
|
|
|
|
## The 10-minute unreachable-server drill (condensed from 2026-08-03)
|
|
|
|
1. `hetzner.py cc-ci status` — "running" + no SSH/ping means booted-but-broken, not crashed.
|
|
2. `hetzner-console.sh cc-ci screenshot /tmp/console.png` — look at the actual screen: which
|
|
generation booted? login prompt or emergency shell?
|
|
3. Plain `reboot` first. If the default boot is the broken generation, DON'T fight GRUB
|
|
timing over VNC — go to rescue:
|
|
`rescue-on` → `poweroff` → wait `status=off` → `poweron` → `ssh root@<public-ip>` (key
|
|
113082420 = `~/.ssh/cc-ci-root-ed25519`; fresh `UserKnownHostsFile`).
|
|
4. In rescue: `mount /dev/sda1 /mnt` and fix the boot default:
|
|
- Generations live in a GRUB **submenu**: entry ids are `1>N` (top level: 0 = default
|
|
entry, 1 = the submenu). A bare index like `2` silently falls back to the broken default.
|
|
- Persistent: `grub-editenv /mnt/boot/grub/grubenv set 'default=1>N'` (survives reboots).
|
|
- **Clear it after the next successful `nixos-rebuild switch`** — the regenerated grub.cfg
|
|
shifts indices and a stale override points at the wrong generation.
|
|
- Journal of the failed boot: `journalctl -D /mnt/var/log/journal --list-boots` / `-b <id>`.
|
|
5. `rescue-off` → `poweroff` → `poweron` → verify → write the incident up in
|
|
`cc-ci-plan/JOURNAL.md` and (if server) `.cc-ci-logs/`.
|
|
|
|
**Prevention:** `nixos-rebuild test` before `switch`, always (see AGENTS.md / the update
|
|
skills) — `test` leaves the bootloader alone, so a power-cycle recovers by itself.
|