cc-ci-orchestrator/.claude/skills/hetzner-server-recovery/SKILL.md

---
name: hetzner-server-recovery
description: Recover a Hetzner-hosted server when SSH or Tailscale access is broken. Use when a Hetzner cloud server is unreachable, drops into emergency mode, or needs rescue-mode repair through the Hetzner API and web console. Invoke as /hetzner-server-recovery.
---

# hetzner-server-recovery

Use this skill when a **Hetzner Cloud server** is no longer reachable over normal SSH/Tailscale and
needs recovery through the Hetzner API, rescue mode, or the web console.

This procedure was proven on the cc-ci server (`server id 134485294`) when it booted into emergency
mode because `/dev/disk/by-label/ESP` was missing.

## Preconditions

- You have a valid `HCLOUD_TOKEN` in `.testenv`.
- You know the Hetzner server ID.
- You have the SSH private key that is registered in Hetzner rescue mode.
- You can install temporary tooling locally if needed.

## 1. First triage

Check all three access paths before changing anything:

```bash
ssh -o BatchMode=yes -o ConnectTimeout=10 <host-alias> hostname
ssh -o BatchMode=yes -o ConnectTimeout=10 root@<public-ip> hostname
tailscale ping -c 3 <host-alias>
```

Interpretation:

- `tailscale ssh` timeout + public `Connection refused` often means the box booted but `sshd` or boot
  dependencies failed.
- `Host key verification failed` on the public IP after rescue/power-cycle is normal; use a temporary
  `UserKnownHostsFile`.

## 2. Try the least-destructive recovery first

Request a plain reboot:

```bash
set -a && . /srv/cc-ci/.testenv && set +a
curl -s -X POST \
  -H "Authorization: Bearer ${HCLOUD_TOKEN}" \
  -H "Content-Type: application/json" \
  "https://api.hetzner.cloud/v1/servers/<SERVER_ID>/actions/reboot"
```

Then poll:

```bash
ssh -o BatchMode=yes -o ConnectTimeout=10 <host-alias> hostname
ssh -o BatchMode=yes -o ConnectTimeout=10 root@<public-ip> hostname
tailscale ping -c 3 <host-alias>
```

If the host still does not come back, continue.

## 3. Use the repo recovery tools (permanent home — do NOT rebuild these in /tmp)

The API and console tooling live in **`scripts/recovery/`** (see its README for the condensed
10-minute drill, proven 2026-08-03):

```bash
# API: status / actions / reboot / reset / poweroff / poweron / rescue-on / rescue-off / console
python3 /srv/cc-ci-orch/scripts/recovery/hetzner.py cc-ci status
python3 /srv/cc-ci-orch/scripts/recovery/hetzner.py cc-ci actions 10

# Shell-only console access (fresh console session + websocat bridge + vncdotool,
# venv auto-bootstrapped at ~/.cache/hetzner-console-venv):
bash /srv/cc-ci-orch/scripts/recovery/hetzner-console.sh cc-ci screenshot /tmp/console.png
bash /srv/cc-ci-orch/scripts/recovery/hetzner-console.sh cc-ci key Down Down Return
```

Known server names: `cc-ci` (134485294), `orchestrator` (134487234). Token: `HCLOUD_TOKEN`
env or `/srv/cc-ci/.hcloud-token` (0600, not in git; prefer per-incident revocable tokens,
and never paste tokens into a chat transcript).

Notes that used to cost time:
- Each console command requests a **fresh** console session — old sessions die on hard reset,
  and the websocat bridge is single-connection anyway.
- A GRUB one-shot/default for a NixOS generation needs the **submenu id `1>N`** (top level:
  0 = default entry, 1 = the "All configurations" submenu). A bare index silently falls back
  to the default entry. Clear any grubenv override after the next `switch` regenerates
  grub.cfg — indices shift.

Capture a screenshot from the console:

```bash
/tmp/opencode/hetzner-console-venv/bin/python - <<'PY'
from vncdotool import api
client = api.connect('127.0.0.1::5905', password='<PASSWORD>')
client.captureScreen('/tmp/opencode/console.png')
PY
```

Read the screenshot with your local tooling to inspect the boot state.

## 5. If rescue mode is needed

Enable rescue mode using the registered SSH key ID:

```bash
curl -s -X POST \
  -H "Authorization: Bearer ${HCLOUD_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"type":"linux64","ssh_keys":[<SSH_KEY_ID>]}' \
  "https://api.hetzner.cloud/v1/servers/<SERVER_ID>/actions/enable_rescue"
```

Then do a **full power cycle** if a normal reboot does not actually switch into rescue:

```bash
curl -s -X POST -H "Authorization: Bearer ${HCLOUD_TOKEN}" \
  -H "Content-Type: application/json" \
  "https://api.hetzner.cloud/v1/servers/<SERVER_ID>/actions/poweroff"

curl -s -X POST -H "Authorization: Bearer ${HCLOUD_TOKEN}" \
  -H "Content-Type: application/json" \
  "https://api.hetzner.cloud/v1/servers/<SERVER_ID>/actions/poweron"
```

Then connect with a temporary known-hosts file:

```bash
ssh -o BatchMode=yes -o ConnectTimeout=10 \
  -o StrictHostKeyChecking=no \
  -o UserKnownHostsFile=/tmp/opencode/known_hosts_rescue \
  -i <private-key> root@<public-ip>
```

## 6. Mount the installed system

In rescue mode:

```bash
lsblk -f
mkdir -p /mnt/recover
mount /dev/sda1 /mnt/recover
ls -l /mnt/recover/nix/var/nix/profiles
readlink -f /mnt/recover/nix/var/nix/profiles/system
```

Useful follow-ups:

```bash
journalctl --directory=/mnt/recover/var/log/journal -p err..alert -n 120 --no-pager
grep -R "by-label/ESP\|/boot\|/boot/efi" -n /mnt/recover/etc /mnt/recover/etc/nixos 2>/dev/null
```

## 7. Proven repair: restore the EFI label

On the cc-ci host, the failure was:

- boot dropped into emergency mode
- console showed root locked / emergency prompt
- journal showed:
  - `Timed out waiting for device /dev/disk/by-label/ESP`
  - `Dependency failed for /boot`
  - `Dependency failed for Local File Systems`

Disk inspection showed the EFI partition existed as `/dev/sda15` with UUID `D978-69EE`, but it had no
filesystem label.

Repair:

```bash
blkid /dev/sda15
fatlabel /dev/sda15 ESP
blkid /dev/sda15
sync
```

Expected after repair:

```text
LABEL="ESP"
```

## 8. Return to normal boot

Disable rescue mode and reboot:

```bash
curl -s -X POST -H "Authorization: Bearer ${HCLOUD_TOKEN}" \
  -H "Content-Type: application/json" \
  "https://api.hetzner.cloud/v1/servers/<SERVER_ID>/actions/disable_rescue"

curl -s -X POST -H "Authorization: Bearer ${HCLOUD_TOKEN}" \
  -H "Content-Type: application/json" \
  "https://api.hetzner.cloud/v1/servers/<SERVER_ID>/actions/reboot"
```

Then verify both paths again:

```bash
ssh -o BatchMode=yes -o ConnectTimeout=10 <host-alias> hostname
ssh -o BatchMode=yes -o ConnectTimeout=10 \
  -o StrictHostKeyChecking=no \
  -o UserKnownHostsFile=/tmp/opencode/known_hosts_postfix \
  root@<public-ip> hostname
tailscale ping -c 3 <host-alias>
```

## 9. Post-recovery follow-up

- Re-run the exact operation that caused the outage, but more cautiously.
- Compare the live boot blocker against the committed Nix config.
- If the server runs helper sessions or agents, notify them that SSH is back and they can resume work.
- Record the failure and the repair in `JOURNAL.md`.

## cc-ci-specific notes

- cc-ci Hetzner server ID: `134485294`
- Public IP observed during this recovery: `91.98.47.73`
- Tailscale alias: `cc-ci`
- Registered rescue SSH key that matched local `/home/loops/.ssh/cc-ci-root-ed25519`:
  `cc-ci-orchestrator-deploy` (`SSH key id 113082420`)

## Hard guardrails

- Prefer reboot, then rescue, before destructive rebuild.
- Never force-rebuild a server disk unless recovery/rescue is exhausted.
- Avoid editing data on mounted filesystems unless you understand the exact boot blocker.
- For cloud-hosted NixOS, filesystem labels/UUID expectations can be just as critical as service config.