The 2026-08-03 cc-ci outage was recovered with ad-hoc tooling living in /tmp
(leftover from a PREVIOUS incident, half-evaporated). Promoted to the repo:
- scripts/recovery/hetzner.py — Hetzner API helper (status/actions/reboot/reset/
power/rescue-on|off/console), knows cc-ci=134485294 + orchestrator=134487234 by
name; token from HCLOUD_TOKEN or /srv/cc-ci/.hcloud-token (0600, never in git).
- scripts/recovery/hetzner-console.sh — shell-only VGA console: fresh console
session -> websocat bridge -> vncdotool (venv auto-bootstrapped in ~/.cache).
screenshot / key / type subcommands; encodes the reset-invalidates-session and
single-connection-bridge gotchas.
- scripts/recovery/README.md — the condensed 10-minute unreachable-server drill,
incl. the GRUB submenu 1>N ids + clear-grubenv-after-switch rule.
- hetzner-server-recovery skill: console/API sections now point at the repo tools
instead of describing /tmp rebuilds.
Smoke-tested: hetzner.py cc-ci status OK.
Operator decision: no policy difference between cc-ci and recipe-maintainer. On
inspection ARM already agrees (recipe-upgrade-cron-all: 'PRs are reviewed and merged
manually by a human afterwards... never merges anything'; 'no human review in the
middle' = skip the mid-run plan confirmation only). Wrappers previously framed this
as a cc-ci override over ARM auto-merge flows — wrong reading; now stated as ONE
unified rule. /help conventions updated to match.
The repo already vendored ARM as the references/recipe-maintainer submodule (old
repo name recipe-maintainers/recipe-maintainer, pinned 460eba0). Rather than carry
two copies, drop the just-added vendor/ duplicate and:
- retarget references/recipe-maintainer to
ssh://git.autonomic.zone/recipe-maintainers/autonomic-recipe-maintainer.git
(same lineage — 460eba0 is an ancestor) and bump to latest acd5cfb, which also
refreshes the parity-test SOURCE reference the tests cite.
- gen-cctest-skills.py + all 30 cctest-* wrappers + /help now point at
references/recipe-maintainer.
- JOURNAL.md: pending session entries (server-update policy addendum, tests-update,
orchestrator-update, upgrade-run notes).
One operator interface for both toolkits (operator decision 2026-08-04):
- vendor/autonomic-recipe-maintainer: ARM pinned as a submodule at acd5cfb (latest).
- scripts/gen-cctest-skills.py: generates a cctest-<name> wrapper pair
(.opencode canonical + .claude thin) for every ARM skill — frontmatter carries ARM's
own description tagged [recipe-maintainer/cctest]; body points at the canonical
submodule SKILL.md, sets cwd/sandbox context, and states the policy overrides
(auto-merge-style ARM flows need per-run operator opt-in; never touch cc-ci infra
from an ARM skill; submodule is read-only here). Re-run after every submodule bump.
- 30 cctest-* wrappers generated.
- /help: cctest family section + situation-table rows + the cc-ci-vs-cctest rule of thumb.
cctest = the recipe-maintainer test server; these skills run against it + the ARM
sandbox, never against the cc-ci CI server/swarm.
autonomic-bot