recovery: give the incident tooling a permanent home (scripts/recovery/)

The 2026-08-03 cc-ci outage was recovered with ad-hoc tooling living in /tmp
(leftover from a PREVIOUS incident, half-evaporated). Promoted to the repo:
- scripts/recovery/hetzner.py — Hetzner API helper (status/actions/reboot/reset/
  power/rescue-on|off/console), knows cc-ci=134485294 + orchestrator=134487234 by
  name; token from HCLOUD_TOKEN or /srv/cc-ci/.hcloud-token (0600, never in git).
- scripts/recovery/hetzner-console.sh — shell-only VGA console: fresh console
  session -> websocat bridge -> vncdotool (venv auto-bootstrapped in ~/.cache).
  screenshot / key / type subcommands; encodes the reset-invalidates-session and
  single-connection-bridge gotchas.
- scripts/recovery/README.md — the condensed 10-minute unreachable-server drill,
  incl. the GRUB submenu 1>N ids + clear-grubenv-after-switch rule.
- hetzner-server-recovery skill: console/API sections now point at the repo tools
  instead of describing /tmp rebuilds.
Smoke-tested: hetzner.py cc-ci status OK.

.claude/skills/hetzner-server-recovery/SKILL.md

@@ -57,60 +57,33 @@ tailscale ping -c 3 <host-alias>
 
 If the host still does not come back, continue.
 
-## 3. Request the Hetzner console
+## 3. Use the repo recovery tools (permanent home — do NOT rebuild these in /tmp)
 
-Request a remote console session:
+The API and console tooling live in **`scripts/recovery/`** (see its README for the condensed
+10-minute drill, proven 2026-08-03):
 
 ```bash
-curl -s -X POST \
-  -H "Authorization: Bearer ${HCLOUD_TOKEN}" \
-  -H "Content-Type: application/json" \
-  "https://api.hetzner.cloud/v1/servers/<SERVER_ID>/actions/request_console"
+# API: status / actions / reboot / reset / poweroff / poweron / rescue-on / rescue-off / console
+python3 /srv/cc-ci-orch/scripts/recovery/hetzner.py cc-ci status
+python3 /srv/cc-ci-orch/scripts/recovery/hetzner.py cc-ci actions 10
+
+# Shell-only console access (fresh console session + websocat bridge + vncdotool,
+# venv auto-bootstrapped at ~/.cache/hetzner-console-venv):
+bash /srv/cc-ci-orch/scripts/recovery/hetzner-console.sh cc-ci screenshot /tmp/console.png
+bash /srv/cc-ci-orch/scripts/recovery/hetzner-console.sh cc-ci key Down Down Return
 ```
 
-The API returns:
+Known server names: `cc-ci` (134485294), `orchestrator` (134487234). Token: `HCLOUD_TOKEN`
+env or `/srv/cc-ci/.hcloud-token` (0600, not in git; prefer per-incident revocable tokens,
+and never paste tokens into a chat transcript).
 
-- `wss_url`
-- `password`
-
-If you have a browser, use the Hetzner console directly.
-
-If you only have shell access, you can still drive it locally because the console is **raw VNC over
-websocket**.
-
-## 4. Shell-only console access (websocket VNC bridge)
-
-Install temporary tools:
-
-```bash
-nix shell nixpkgs#websocat -c websocat --version
-python3 -m venv /tmp/opencode/hetzner-console-venv
-/tmp/opencode/hetzner-console-venv/bin/pip install --disable-pip-version-check pillow websocket-client vncdotool
-```
-
-Bridge the websocket console to a local VNC TCP port:
-
-```bash
-nohup nix shell nixpkgs#websocat -c \
-  websocat -b -E tcp-l:127.0.0.1:5905 '<WSS_URL>' \
-  >/tmp/opencode/hetzner-websockify.log 2>&1 &
-```
-
-Validate the RFB banner:
-
-```bash
-python3 - <<'PY'
-import socket
-s=socket.socket(); s.settimeout(5); s.connect(('127.0.0.1',5905))
-print(repr(s.recv(32)))
-PY
-```
-
-Expected:
-
-```text
-b'RFB 003.008\n'
-```
+Notes that used to cost time:
+- Each console command requests a **fresh** console session — old sessions die on hard reset,
+  and the websocat bridge is single-connection anyway.
+- A GRUB one-shot/default for a NixOS generation needs the **submenu id `1>N`** (top level:
+  0 = default entry, 1 = the "All configurations" submenu). A bare index silently falls back
+  to the default entry. Clear any grubenv override after the next `switch` regenerates
+  grub.cfg — indices shift.
 
 Capture a screenshot from the console: