autonomic-bot
cb20bea7cd
The 2026-08-03 cc-ci outage was recovered with ad-hoc tooling living in /tmp (leftover from a PREVIOUS incident, half-evaporated). Promoted to the repo: - scripts/recovery/hetzner.py — Hetzner API helper (status/actions/reboot/reset/ power/rescue-on|off/console), knows cc-ci=134485294 + orchestrator=134487234 by name; token from HCLOUD_TOKEN or /srv/cc-ci/.hcloud-token (0600, never in git). - scripts/recovery/hetzner-console.sh — shell-only VGA console: fresh console session -> websocat bridge -> vncdotool (venv auto-bootstrapped in ~/.cache). screenshot / key / type subcommands; encodes the reset-invalidates-session and single-connection-bridge gotchas. - scripts/recovery/README.md — the condensed 10-minute unreachable-server drill, incl. the GRUB submenu 1>N ids + clear-grubenv-after-switch rule. - hetzner-server-recovery skill: console/API sections now point at the repo tools instead of describing /tmp rebuilds. Smoke-tested: hetzner.py cc-ci status OK.
cc-ci-orchestrator
Orchestrator workspace for building the cc-ci Co-op Cloud recipe CI server. The plan, launch
tooling, and loop prompts live in cc-ci-plan/; see AGENTS.md for the
roles and operating model. Secrets (.testenv) are gitignored — never commit them.
Run the orchestrator in tmux (survives disconnects + closing your laptop)
Keep this supervising session alive on the host with tmux, and use --remote-control so you can
watch/steer it from claude.ai/code (or the mobile app).
# 0. Exit any running orchestrator session first — a conversation can't be resumed while it's live:
# /exit (inside Claude) or Ctrl-D
# 1. Start a detachable tmux session on this host
tmux new -s orchestrator
# 2. Inside tmux, resume the orchestrator conversation WITH remote control:
claude --resume autonomous-orchestrator \
--remote-control "autonomous-orchestrator" \
--dangerously-skip-permissions
# - If name-resume opens a picker instead of resuming directly, choose "autonomous-orchestrator".
# - Or resume by the stable session id (more deterministic in a fresh pane):
# claude --resume 34a80a99-b37e-4809-b8da-ccc9fafe785e \
# --remote-control "autonomous-orchestrator" --dangerously-skip-permissions
# 3. Detach — the process keeps running: press Ctrl-b, then d
Reconnect later
- On this host:
tmux attach -t orchestrator - From anywhere: claude.ai/code → the
autonomous-orchestratorsession
Why it survives: tmux keeps the claude process alive across SSH disconnects and your laptop
closing; remote-control runs outbound from this host to Anthropic, so it stays connected
regardless of the viewer. After a host reboot, re-run steps 1–2.
Two different "names":
--resume <name|id>selects the conversation to restore (shown in the/resumepicker); the--remote-control "<name>"value is only the web display label and resumes nothing. Resuming reuses the same session id each time (stays34a8…) — don't pass--fork-sessionunless you intend to branch a new conversation.Already inside a live session and just want the web surface? Run
/remote-control— no exit/resume.
Kick off / supervise the loops
cd /srv/cc-ci/cc-ci-plan
./launch.sh start # Builder + Adversary loops (interactive --remote-control in tmux) + watchdog
./launch.sh status # session + DONE state
./launch.sh logs builder|adversary|watchdog
./launch.sh stop
Full supervision guide, credential map, and the Incus VM fallback are in
cc-ci-plan/kickoff.md and cc-ci-plan/plan.md §1.5.