autonomic-bot
cb20bea7cd
The 2026-08-03 cc-ci outage was recovered with ad-hoc tooling living in /tmp (leftover from a PREVIOUS incident, half-evaporated). Promoted to the repo: - scripts/recovery/hetzner.py — Hetzner API helper (status/actions/reboot/reset/ power/rescue-on|off/console), knows cc-ci=134485294 + orchestrator=134487234 by name; token from HCLOUD_TOKEN or /srv/cc-ci/.hcloud-token (0600, never in git). - scripts/recovery/hetzner-console.sh — shell-only VGA console: fresh console session -> websocat bridge -> vncdotool (venv auto-bootstrapped in ~/.cache). screenshot / key / type subcommands; encodes the reset-invalidates-session and single-connection-bridge gotchas. - scripts/recovery/README.md — the condensed 10-minute unreachable-server drill, incl. the GRUB submenu 1>N ids + clear-grubenv-after-switch rule. - hetzner-server-recovery skill: console/API sections now point at the repo tools instead of describing /tmp rebuilds. Smoke-tested: hetzner.py cc-ci status OK.
77 lines
3.2 KiB
Bash
Executable File
77 lines
3.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# hetzner-console.sh — drive a Hetzner web-console (raw VNC over websocket) from a shell.
|
|
# The screenshot/keyboard half of hetzner-server-recovery; proven in the 2026-08-03
|
|
# cc-ci 26.05 outage (console screenshots identified the wrong-generation boot).
|
|
#
|
|
# Usage:
|
|
# hetzner-console.sh <server> screenshot <out.png>
|
|
# hetzner-console.sh <server> key <key> [key ...] # e.g. key Down Down Return
|
|
# hetzner-console.sh <server> type "<text>"
|
|
#
|
|
# <server> = name/id understood by hetzner.py. Each invocation requests a FRESH console
|
|
# session (they are cheap, and hard resets invalidate old ones), bridges it to a local
|
|
# TCP port with websocat, and runs vncdo against it. The bridge is single-connection —
|
|
# that is why every command re-requests + re-bridges.
|
|
#
|
|
# Gotchas encoded here so nobody rediscovers them at 2am:
|
|
# - The wss_url from the API contains literal '&' — when it arrives via JSON it may be
|
|
# &-escaped; parse the JSON properly (as below), never paste from raw output.
|
|
# - A hard `reset` drops the console websocket mid-session: re-request and reconnect.
|
|
# - GRUB menus: generations live in a SUBMENU — one-shot boot ids are "1>N", and any
|
|
# persistent grubenv `default` must be cleared after the next switch regenerates
|
|
# grub.cfg (indices shift). See the hetzner-server-recovery skill.
|
|
set -o errexit -o nounset -o pipefail
|
|
|
|
HERE="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")"
|
|
SERVER="${1:?usage: hetzner-console.sh <server> screenshot|key|type ...}"
|
|
CMD="${2:?need a command: screenshot|key|type}"
|
|
shift 2
|
|
|
|
PORT="${CONSOLE_PORT:-5905}"
|
|
VENV="${HOME}/.cache/hetzner-console-venv"
|
|
|
|
# 1. vncdotool venv (bootstrap once; durable across incidents, unlike /tmp)
|
|
if [ ! -x "${VENV}/bin/vncdo" ]; then
|
|
echo "bootstrapping vncdotool venv at ${VENV}..." >&2
|
|
python3 -m venv "${VENV}"
|
|
"${VENV}/bin/pip" -q install vncdotool
|
|
fi
|
|
|
|
# 2. fresh console session
|
|
CREDS="$(python3 "${HERE}/hetzner.py" "${SERVER}" console)"
|
|
WSS="$(printf '%s' "${CREDS}" | python3 -c 'import json,sys; print(json.load(sys.stdin)["wss_url"])')"
|
|
PW="$(printf '%s' "${CREDS}" | python3 -c 'import json,sys; print(json.load(sys.stdin)["password"])')"
|
|
|
|
# 3. bridge (single-connection; killed on exit)
|
|
pkill -f "websocat.*${PORT}" 2>/dev/null || true
|
|
sleep 0.5
|
|
nix shell nixpkgs#websocat -c websocat --binary "tcp-listen:127.0.0.1:${PORT}" "${WSS}" \
|
|
> /tmp/hetzner-console-websocat.log 2>&1 &
|
|
BRIDGE=$!
|
|
trap 'kill ${BRIDGE} 2>/dev/null || true; pkill -f "websocat.*${PORT}" 2>/dev/null || true' EXIT
|
|
sleep 2
|
|
|
|
# 4. run the vncdo command
|
|
case "${CMD}" in
|
|
screenshot)
|
|
OUT="${1:?screenshot needs an output path}"
|
|
timeout 40 "${VENV}/bin/vncdo" -s "127.0.0.1::${PORT}" -p "${PW}" capture "${OUT}"
|
|
echo "captured ${OUT}"
|
|
;;
|
|
key)
|
|
[ $# -ge 1 ] || { echo "key needs at least one key name" >&2; exit 1; }
|
|
ARGS=()
|
|
for k in "$@"; do ARGS+=(key "$k" pause 0.3); done
|
|
timeout 60 "${VENV}/bin/vncdo" -s "127.0.0.1::${PORT}" -p "${PW}" "${ARGS[@]}"
|
|
echo "sent: $*"
|
|
;;
|
|
type)
|
|
TEXT="${1:?type needs text}"
|
|
timeout 60 "${VENV}/bin/vncdo" -s "127.0.0.1::${PORT}" -p "${PW}" type "${TEXT}"
|
|
echo "typed"
|
|
;;
|
|
*)
|
|
echo "unknown command ${CMD} (screenshot|key|type)" >&2; exit 1
|
|
;;
|
|
esac
|