Files

T

History

autonomic-bot cb20bea7cd recovery: give the incident tooling a permanent home (scripts/recovery/)

The 2026-08-03 cc-ci outage was recovered with ad-hoc tooling living in /tmp
(leftover from a PREVIOUS incident, half-evaporated). Promoted to the repo:
- scripts/recovery/hetzner.py — Hetzner API helper (status/actions/reboot/reset/
  power/rescue-on|off/console), knows cc-ci=134485294 + orchestrator=134487234 by
  name; token from HCLOUD_TOKEN or /srv/cc-ci/.hcloud-token (0600, never in git).
- scripts/recovery/hetzner-console.sh — shell-only VGA console: fresh console
  session -> websocat bridge -> vncdotool (venv auto-bootstrapped in ~/.cache).
  screenshot / key / type subcommands; encodes the reset-invalidates-session and
  single-connection-bridge gotchas.
- scripts/recovery/README.md — the condensed 10-minute unreachable-server drill,
  incl. the GRUB submenu 1>N ids + clear-grubenv-after-switch rule.
- hetzner-server-recovery skill: console/API sections now point at the repo tools
  instead of describing /tmp rebuilds.
Smoke-tested: hetzner.py cc-ci status OK.

2026-08-04 01:57:34 +00:00

hetzner-console.sh

recovery: give the incident tooling a permanent home (scripts/recovery/)

2026-08-04 01:57:34 +00:00

hetzner.py

recovery: give the incident tooling a permanent home (scripts/recovery/)

2026-08-04 01:57:34 +00:00

README.md

recovery: give the incident tooling a permanent home (scripts/recovery/)

2026-08-04 01:57:34 +00:00

README.md

Recovery tooling — permanent home

Promoted from /tmp ad-hocery after the 2026-08-03 cc-ci 26.05 outage (see .cc-ci-logs/server-update-2026-08-03.md and the hetzner-server-recovery skill, which is the procedure; these are the tools it uses).

Tool	What it does
`hetzner.py <server> <cmd>`	Hetzner Cloud API: status, actions, reboot/reset/poweroff/poweron, rescue-on/off, console credentials. Knows `cc-ci` (134485294) and `orchestrator` (134487234) by name.
`hetzner-console.sh <server> screenshot\|key\|type`	Shell-only access to the VGA console: fresh console session → websocat bridge → vncdotool (venv auto-bootstrapped at `~/.cache/hetzner-console-venv`).

Token: HCLOUD_TOKEN env, or /srv/cc-ci/.hcloud-token (chmod 600). Not in git, not in .testenv. Prefer per-incident tokens and revoke after — and never paste tokens into chat transcripts (the 2026-08-03 incident token was pasted mid-incident and had to be flagged for rotation).

The 10-minute unreachable-server drill (condensed from 2026-08-03)

hetzner.py cc-ci status — "running" + no SSH/ping means booted-but-broken, not crashed.
hetzner-console.sh cc-ci screenshot /tmp/console.png — look at the actual screen: which generation booted? login prompt or emergency shell?
Plain reboot first. If the default boot is the broken generation, DON'T fight GRUB timing over VNC — go to rescue: rescue-on → poweroff → wait status=off → poweron → ssh root@<public-ip> (key 113082420 = ~/.ssh/cc-ci-root-ed25519; fresh UserKnownHostsFile).
In rescue: mount /dev/sda1 /mnt and fix the boot default:
- Generations live in a GRUB submenu: entry ids are 1>N (top level: 0 = default entry, 1 = the submenu). A bare index like 2 silently falls back to the broken default.
- Persistent: grub-editenv /mnt/boot/grub/grubenv set 'default=1>N' (survives reboots).
- Clear it after the next successful nixos-rebuild switch — the regenerated grub.cfg shifts indices and a stale override points at the wrong generation.
- Journal of the failed boot: journalctl -D /mnt/var/log/journal --list-boots / -b <id>.
rescue-off → poweroff → poweron → verify → write the incident up in cc-ci-plan/JOURNAL.md and (if server) .cc-ci-logs/.

Prevention: nixos-rebuild test before switch, always (see AGENTS.md / the update skills) — test leaves the bootloader alone, so a power-cycle recovers by itself.