cc-ci-orchestrator/.claude/skills/hetzner-server-recovery/SKILL.md

name, description

name	description
hetzner-server-recovery	Recover a Hetzner-hosted server when SSH or Tailscale access is broken. Use when a Hetzner cloud server is unreachable, drops into emergency mode, or needs rescue-mode repair through the Hetzner API and web console. Invoke as /hetzner-server-recovery.

hetzner-server-recovery

Use this skill when a Hetzner Cloud server is no longer reachable over normal SSH/Tailscale and needs recovery through the Hetzner API, rescue mode, or the web console.

This procedure was proven on the cc-ci server (server id 134485294) when it booted into emergency mode because /dev/disk/by-label/ESP was missing.

Preconditions

• You have a valid HCLOUD_TOKEN in .testenv.
• You know the Hetzner server ID.
• You have the SSH private key that is registered in Hetzner rescue mode.
• You can install temporary tooling locally if needed.

1. First triage

Check all three access paths before changing anything:

ssh -o BatchMode=yes -o ConnectTimeout=10 <host-alias> hostname
ssh -o BatchMode=yes -o ConnectTimeout=10 root@<public-ip> hostname
tailscale ping -c 3 <host-alias>

Interpretation:

• tailscale ssh timeout + public Connection refused often means the box booted but sshd or boot dependencies failed.
• Host key verification failed on the public IP after rescue/power-cycle is normal; use a temporary UserKnownHostsFile.

2. Try the least-destructive recovery first

Request a plain reboot:

set -a && . /srv/cc-ci/.testenv && set +a
curl -s -X POST \
  -H "Authorization: Bearer ${HCLOUD_TOKEN}" \
  -H "Content-Type: application/json" \
  "https://api.hetzner.cloud/v1/servers/<SERVER_ID>/actions/reboot"

Then poll:

ssh -o BatchMode=yes -o ConnectTimeout=10 <host-alias> hostname
ssh -o BatchMode=yes -o ConnectTimeout=10 root@<public-ip> hostname
tailscale ping -c 3 <host-alias>

If the host still does not come back, continue.

3. Use the repo recovery tools (permanent home — do NOT rebuild these in /tmp)

The API and console tooling live in scripts/recovery/ (see its README for the condensed 10-minute drill, proven 2026-08-03):

# API: status / actions / reboot / reset / poweroff / poweron / rescue-on / rescue-off / console
python3 /srv/cc-ci-orch/scripts/recovery/hetzner.py cc-ci status
python3 /srv/cc-ci-orch/scripts/recovery/hetzner.py cc-ci actions 10

# Shell-only console access (fresh console session + websocat bridge + vncdotool,
# venv auto-bootstrapped at ~/.cache/hetzner-console-venv):
bash /srv/cc-ci-orch/scripts/recovery/hetzner-console.sh cc-ci screenshot /tmp/console.png
bash /srv/cc-ci-orch/scripts/recovery/hetzner-console.sh cc-ci key Down Down Return

Known server names: cc-ci (134485294), orchestrator (134487234). Token: HCLOUD_TOKEN env or /srv/cc-ci/.hcloud-token (0600, not in git; prefer per-incident revocable tokens, and never paste tokens into a chat transcript).

Notes that used to cost time:

• Each console command requests a fresh console session — old sessions die on hard reset, and the websocat bridge is single-connection anyway.
• A GRUB one-shot/default for a NixOS generation needs the submenu id 1>N (top level: 0 = default entry, 1 = the "All configurations" submenu). A bare index silently falls back to the default entry. Clear any grubenv override after the next switch regenerates grub.cfg — indices shift.

Capture a screenshot from the console:

/tmp/opencode/hetzner-console-venv/bin/python - <<'PY'
from vncdotool import api
client = api.connect('127.0.0.1::5905', password='<PASSWORD>')
client.captureScreen('/tmp/opencode/console.png')
PY

Read the screenshot with your local tooling to inspect the boot state.

5. If rescue mode is needed

Enable rescue mode using the registered SSH key ID:

curl -s -X POST \
  -H "Authorization: Bearer ${HCLOUD_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"type":"linux64","ssh_keys":[<SSH_KEY_ID>]}' \
  "https://api.hetzner.cloud/v1/servers/<SERVER_ID>/actions/enable_rescue"

Then do a full power cycle if a normal reboot does not actually switch into rescue:

curl -s -X POST -H "Authorization: Bearer ${HCLOUD_TOKEN}" \
  -H "Content-Type: application/json" \
  "https://api.hetzner.cloud/v1/servers/<SERVER_ID>/actions/poweroff"

curl -s -X POST -H "Authorization: Bearer ${HCLOUD_TOKEN}" \
  -H "Content-Type: application/json" \
  "https://api.hetzner.cloud/v1/servers/<SERVER_ID>/actions/poweron"

Then connect with a temporary known-hosts file:

ssh -o BatchMode=yes -o ConnectTimeout=10 \
  -o StrictHostKeyChecking=no \
  -o UserKnownHostsFile=/tmp/opencode/known_hosts_rescue \
  -i <private-key> root@<public-ip>

6. Mount the installed system

In rescue mode:

lsblk -f
mkdir -p /mnt/recover
mount /dev/sda1 /mnt/recover
ls -l /mnt/recover/nix/var/nix/profiles
readlink -f /mnt/recover/nix/var/nix/profiles/system

Useful follow-ups:

journalctl --directory=/mnt/recover/var/log/journal -p err..alert -n 120 --no-pager
grep -R "by-label/ESP\|/boot\|/boot/efi" -n /mnt/recover/etc /mnt/recover/etc/nixos 2>/dev/null

7. Proven repair: restore the EFI label

On the cc-ci host, the failure was:

• boot dropped into emergency mode
• console showed root locked / emergency prompt
• journal showed:
  • Timed out waiting for device /dev/disk/by-label/ESP
  • Dependency failed for /boot
  • Dependency failed for Local File Systems

Disk inspection showed the EFI partition existed as /dev/sda15 with UUID D978-69EE, but it had no filesystem label.

Repair:

blkid /dev/sda15
fatlabel /dev/sda15 ESP
blkid /dev/sda15
sync

Expected after repair:

LABEL="ESP"

8. Return to normal boot

Disable rescue mode and reboot:

curl -s -X POST -H "Authorization: Bearer ${HCLOUD_TOKEN}" \
  -H "Content-Type: application/json" \
  "https://api.hetzner.cloud/v1/servers/<SERVER_ID>/actions/disable_rescue"

curl -s -X POST -H "Authorization: Bearer ${HCLOUD_TOKEN}" \
  -H "Content-Type: application/json" \
  "https://api.hetzner.cloud/v1/servers/<SERVER_ID>/actions/reboot"

Then verify both paths again:

ssh -o BatchMode=yes -o ConnectTimeout=10 <host-alias> hostname
ssh -o BatchMode=yes -o ConnectTimeout=10 \
  -o StrictHostKeyChecking=no \
  -o UserKnownHostsFile=/tmp/opencode/known_hosts_postfix \
  root@<public-ip> hostname
tailscale ping -c 3 <host-alias>

9. Post-recovery follow-up

• Re-run the exact operation that caused the outage, but more cautiously.
• Compare the live boot blocker against the committed Nix config.
• If the server runs helper sessions or agents, notify them that SSH is back and they can resume work.
• Record the failure and the repair in JOURNAL.md.

cc-ci-specific notes

• cc-ci Hetzner server ID: 134485294
• Public IP observed during this recovery: 91.98.47.73
• Tailscale alias: cc-ci
• Registered rescue SSH key that matched local /home/loops/.ssh/cc-ci-root-ed25519: cc-ci-orchestrator-deploy (SSH key id 113082420)

Hard guardrails

• Prefer reboot, then rescue, before destructive rebuild.
• Never force-rebuild a server disk unless recovery/rescue is exhausted.
• Avoid editing data on mounted filesystems unless you understand the exact boot blocker.
• For cloud-hosted NixOS, filesystem labels/UUID expectations can be just as critical as service config.