--- name: hetzner-server-recovery description: Recover a Hetzner-hosted server when SSH or Tailscale access is broken. Use when a Hetzner cloud server is unreachable, drops into emergency mode, or needs rescue-mode repair through the Hetzner API and web console. Invoke as /hetzner-server-recovery. --- # hetzner-server-recovery Use this skill when a **Hetzner Cloud server** is no longer reachable over normal SSH/Tailscale and needs recovery through the Hetzner API, rescue mode, or the web console. This procedure was proven on the cc-ci server (`server id 134485294`) when it booted into emergency mode because `/dev/disk/by-label/ESP` was missing. ## Preconditions - You have a valid `HCLOUD_TOKEN` in `.testenv`. - You know the Hetzner server ID. - You have the SSH private key that is registered in Hetzner rescue mode. - You can install temporary tooling locally if needed. ## 1. First triage Check all three access paths before changing anything: ```bash ssh -o BatchMode=yes -o ConnectTimeout=10 hostname ssh -o BatchMode=yes -o ConnectTimeout=10 root@ hostname tailscale ping -c 3 ``` Interpretation: - `tailscale ssh` timeout + public `Connection refused` often means the box booted but `sshd` or boot dependencies failed. - `Host key verification failed` on the public IP after rescue/power-cycle is normal; use a temporary `UserKnownHostsFile`. ## 2. Try the least-destructive recovery first Request a plain reboot: ```bash set -a && . /srv/cc-ci/.testenv && set +a curl -s -X POST \ -H "Authorization: Bearer ${HCLOUD_TOKEN}" \ -H "Content-Type: application/json" \ "https://api.hetzner.cloud/v1/servers//actions/reboot" ``` Then poll: ```bash ssh -o BatchMode=yes -o ConnectTimeout=10 hostname ssh -o BatchMode=yes -o ConnectTimeout=10 root@ hostname tailscale ping -c 3 ``` If the host still does not come back, continue. ## 3. Use the repo recovery tools (permanent home — do NOT rebuild these in /tmp) The API and console tooling live in **`scripts/recovery/`** (see its README for the condensed 10-minute drill, proven 2026-08-03): ```bash # API: status / actions / reboot / reset / poweroff / poweron / rescue-on / rescue-off / console python3 /srv/cc-ci-orch/scripts/recovery/hetzner.py cc-ci status python3 /srv/cc-ci-orch/scripts/recovery/hetzner.py cc-ci actions 10 # Shell-only console access (fresh console session + websocat bridge + vncdotool, # venv auto-bootstrapped at ~/.cache/hetzner-console-venv): bash /srv/cc-ci-orch/scripts/recovery/hetzner-console.sh cc-ci screenshot /tmp/console.png bash /srv/cc-ci-orch/scripts/recovery/hetzner-console.sh cc-ci key Down Down Return ``` Known server names: `cc-ci` (134485294), `orchestrator` (134487234). Token: `HCLOUD_TOKEN` env or `/srv/cc-ci/.hcloud-token` (0600, not in git; prefer per-incident revocable tokens, and never paste tokens into a chat transcript). Notes that used to cost time: - Each console command requests a **fresh** console session — old sessions die on hard reset, and the websocat bridge is single-connection anyway. - A GRUB one-shot/default for a NixOS generation needs the **submenu id `1>N`** (top level: 0 = default entry, 1 = the "All configurations" submenu). A bare index silently falls back to the default entry. Clear any grubenv override after the next `switch` regenerates grub.cfg — indices shift. Capture a screenshot from the console: ```bash /tmp/opencode/hetzner-console-venv/bin/python - <<'PY' from vncdotool import api client = api.connect('127.0.0.1::5905', password='') client.captureScreen('/tmp/opencode/console.png') PY ``` Read the screenshot with your local tooling to inspect the boot state. ## 5. If rescue mode is needed Enable rescue mode using the registered SSH key ID: ```bash curl -s -X POST \ -H "Authorization: Bearer ${HCLOUD_TOKEN}" \ -H "Content-Type: application/json" \ -d '{"type":"linux64","ssh_keys":[]}' \ "https://api.hetzner.cloud/v1/servers//actions/enable_rescue" ``` Then do a **full power cycle** if a normal reboot does not actually switch into rescue: ```bash curl -s -X POST -H "Authorization: Bearer ${HCLOUD_TOKEN}" \ -H "Content-Type: application/json" \ "https://api.hetzner.cloud/v1/servers//actions/poweroff" curl -s -X POST -H "Authorization: Bearer ${HCLOUD_TOKEN}" \ -H "Content-Type: application/json" \ "https://api.hetzner.cloud/v1/servers//actions/poweron" ``` Then connect with a temporary known-hosts file: ```bash ssh -o BatchMode=yes -o ConnectTimeout=10 \ -o StrictHostKeyChecking=no \ -o UserKnownHostsFile=/tmp/opencode/known_hosts_rescue \ -i root@ ``` ## 6. Mount the installed system In rescue mode: ```bash lsblk -f mkdir -p /mnt/recover mount /dev/sda1 /mnt/recover ls -l /mnt/recover/nix/var/nix/profiles readlink -f /mnt/recover/nix/var/nix/profiles/system ``` Useful follow-ups: ```bash journalctl --directory=/mnt/recover/var/log/journal -p err..alert -n 120 --no-pager grep -R "by-label/ESP\|/boot\|/boot/efi" -n /mnt/recover/etc /mnt/recover/etc/nixos 2>/dev/null ``` ## 7. Proven repair: restore the EFI label On the cc-ci host, the failure was: - boot dropped into emergency mode - console showed root locked / emergency prompt - journal showed: - `Timed out waiting for device /dev/disk/by-label/ESP` - `Dependency failed for /boot` - `Dependency failed for Local File Systems` Disk inspection showed the EFI partition existed as `/dev/sda15` with UUID `D978-69EE`, but it had no filesystem label. Repair: ```bash blkid /dev/sda15 fatlabel /dev/sda15 ESP blkid /dev/sda15 sync ``` Expected after repair: ```text LABEL="ESP" ``` ## 8. Return to normal boot Disable rescue mode and reboot: ```bash curl -s -X POST -H "Authorization: Bearer ${HCLOUD_TOKEN}" \ -H "Content-Type: application/json" \ "https://api.hetzner.cloud/v1/servers//actions/disable_rescue" curl -s -X POST -H "Authorization: Bearer ${HCLOUD_TOKEN}" \ -H "Content-Type: application/json" \ "https://api.hetzner.cloud/v1/servers//actions/reboot" ``` Then verify both paths again: ```bash ssh -o BatchMode=yes -o ConnectTimeout=10 hostname ssh -o BatchMode=yes -o ConnectTimeout=10 \ -o StrictHostKeyChecking=no \ -o UserKnownHostsFile=/tmp/opencode/known_hosts_postfix \ root@ hostname tailscale ping -c 3 ``` ## 9. Post-recovery follow-up - Re-run the exact operation that caused the outage, but more cautiously. - Compare the live boot blocker against the committed Nix config. - If the server runs helper sessions or agents, notify them that SSH is back and they can resume work. - Record the failure and the repair in `JOURNAL.md`. ## cc-ci-specific notes - cc-ci Hetzner server ID: `134485294` - Public IP observed during this recovery: `91.98.47.73` - Tailscale alias: `cc-ci` - Registered rescue SSH key that matched local `/home/loops/.ssh/cc-ci-root-ed25519`: `cc-ci-orchestrator-deploy` (`SSH key id 113082420`) ## Hard guardrails - Prefer reboot, then rescue, before destructive rebuild. - Never force-rebuild a server disk unless recovery/rescue is exhausted. - Avoid editing data on mounted filesystems unless you understand the exact boot blocker. - For cloud-hosted NixOS, filesystem labels/UUID expectations can be just as critical as service config.