The 2026-08-03 cc-ci outage was recovered with ad-hoc tooling living in /tmp
(leftover from a PREVIOUS incident, half-evaporated). Promoted to the repo:
- scripts/recovery/hetzner.py — Hetzner API helper (status/actions/reboot/reset/
power/rescue-on|off/console), knows cc-ci=134485294 + orchestrator=134487234 by
name; token from HCLOUD_TOKEN or /srv/cc-ci/.hcloud-token (0600, never in git).
- scripts/recovery/hetzner-console.sh — shell-only VGA console: fresh console
session -> websocat bridge -> vncdotool (venv auto-bootstrapped in ~/.cache).
screenshot / key / type subcommands; encodes the reset-invalidates-session and
single-connection-bridge gotchas.
- scripts/recovery/README.md — the condensed 10-minute unreachable-server drill,
incl. the GRUB submenu 1>N ids + clear-grubenv-after-switch rule.
- hetzner-server-recovery skill: console/API sections now point at the repo tools
instead of describing /tmp rebuilds.
Smoke-tested: hetzner.py cc-ci status OK.
The repo already vendored ARM as the references/recipe-maintainer submodule (old
repo name recipe-maintainers/recipe-maintainer, pinned 460eba0). Rather than carry
two copies, drop the just-added vendor/ duplicate and:
- retarget references/recipe-maintainer to
ssh://git.autonomic.zone/recipe-maintainers/autonomic-recipe-maintainer.git
(same lineage — 460eba0 is an ancestor) and bump to latest acd5cfb, which also
refreshes the parity-test SOURCE reference the tests cite.
- gen-cctest-skills.py + all 30 cctest-* wrappers + /help now point at
references/recipe-maintainer.
- JOURNAL.md: pending session entries (server-update policy addendum, tests-update,
orchestrator-update, upgrade-run notes).
One operator interface for both toolkits (operator decision 2026-08-04):
- vendor/autonomic-recipe-maintainer: ARM pinned as a submodule at acd5cfb (latest).
- scripts/gen-cctest-skills.py: generates a cctest-<name> wrapper pair
(.opencode canonical + .claude thin) for every ARM skill — frontmatter carries ARM's
own description tagged [recipe-maintainer/cctest]; body points at the canonical
submodule SKILL.md, sets cwd/sandbox context, and states the policy overrides
(auto-merge-style ARM flows need per-run operator opt-in; never touch cc-ci infra
from an ARM skill; submodule is read-only here). Re-run after every submodule bump.
- 30 cctest-* wrappers generated.
- /help: cctest family section + situation-table rows + the cc-ci-vs-cctest rule of thumb.
cctest = the recipe-maintainer test server; these skills run against it + the ARM
sandbox, never against the cc-ci CI server/swarm.
Seven check areas: weekly-run recency+outcome, report publishing, stale recipes/tests,
open recipe PRs (CVE-carrying PRs open >14d flagged high-priority, ready-to-merge PRs
listed as normal), server+orchestrator flake-update recency vs channel tip, host/service
health incl. the bridge !testme path (silent-401 stale-secret check from the 2026-08-03
finding), and maintained-set consistency. Ends with ALL HEALTHY or a prioritized findings
list, each mapped to the skill to invoke. Strictly read-only.
Codifies the full path walked for the 2026-08-03 wordpress enrollment (cc-ci PR #14):
survey -> mirror create+sync from coopcloud -> author test suite (health floor +
non-vacuous recipe-specific tests incl. sec4.3 create-an-object round-trip, recipe-local
setup helper, PARITY.md) -> bridge POLL_REPOS -> used-recipes.md weekly row +
upstream/<recipe>.md registry -> full-suite-green verification with the new tests ->
bridge deploy via test-before-switch -> merge-on-green + report (PR-visibility policy).
Includes the traps hit live: creds injection over stdin for cc-ci-side helpers, fresh-
deploy wizard state in HEALTH_OK, repo-dev-shell ruff, swarm serialization, stale bridge
secret 401s silently dropping !testme.
- cc-ci-server-update / cc-ci-tests-update / cc-ci-update: operator policy change —
open PRs for visibility/historical record and merge directly once verified
(invocation = authorization); reports must list merged-PR links + change summaries.
(tests-update still never merges the paired recipe upgrade PRs.)
- cc-ci-server-update: mandatory nixos-rebuild test step (5d) before switch, with
detached-activation + transient-unit notes from the 2026-08-03 26.05 deploy.
- AGENTS.md: test-before-switch policy for orchestrator host rebuilds.
- NEW skill cc-ci-orchestrator-update (+ thin .claude wrapper): the /cc-ci-server-update
analogue for this host (flake /srv/cc-ci-orch, .#cc-ci-orchestrator-hetzner, Hetzner
server 134487234), with self-update caveats.
- JOURNAL.md: 2026-08-03 server-update incident/recovery handoff entry.
Reverse the skill location: the full definitions now live in
.opencode/skills/<name>/SKILL.md (opencode's native project-skill
location, auto-scanned), and .claude/skills/<name>/SKILL.md are thin
pointer wrappers that keep the frontmatter for discoverability and
redirect to the .opencode/ canonical file for the procedure.
Also drops the 'operator-authorized mid-run' gating from
/cc-ci-server-update: the operator's choice to invoke the skill IS the
authorization for the live nixos-rebuild switch — no second check to
wait for. The quiescent-moment preference stays (don't disrupt an
in-flight !testme), but it's not a gate.
Three new skills for keeping cc-ci itself current (manual triggers for
what should ideally be automatic):
- /cc-ci-server-update — bump the cc-ci SERVER host's nixpkgs/sops-nix
flake inputs (hard rev pin, moves deliberately), open a cc-ci PR, then
deploy to the live host via nixos-rebuild switch (operator-authorized,
no-CI window) + health gate (no failed units, infra up, endpoints 200).
Manual trigger for periodic nixpkgs bumps.
- /cc-ci-tests-update — sweep all maintained recipes for stale cc-ci
tests (carry-over from /upgrade-all's commented stale tests + a fresh
/ci-test-review sweep), author the minimal test update per stale
recipe, open a cc-ci test PR, verify via the standard PR + !testme ->
CI flow. Closes the loop the weekly /upgrade-all deliberately defers
(it comments-only; this fixes the tests). Never weakens a test.
- /cc-ci-update — umbrella: run server-update then tests-update, in
that order, gated on the server health check (don't sweep tests on a
sick/rolled-back host). Server-first because the harness toolchain
rides on the host nixpkgs.
All three: create + verify, NEVER merge. Reuse the existing ci-test-review
helpers (open-cc-ci-pr.sh, run-all-recipes.sh, verify-pr.sh) and the
recipe-upgrade --with-tests discipline.
Root-cause fix for the 2026-07-03 run stalling: the cc-ci host disk filled to
100% (ENOSPC) mid-run (Wave 6, lasuite-drive), the agent stopped to reclaim
space, and nothing resumed it — the log-idle/429 watchdog only covers opencode-go
usage-limit stalls, not an environmental wedge.
- launch-upgrader.py: step-0 prereclaim_cc_ci() prunes STALE cc-ci docker images
(unused AND older than a week, so this week's likely-reused images stay) before
each weekly run. Best-effort; env-tunable (UPGRADER_PRERECLAIM*).
- launch-supervisor.py (new): hourly glm-5.2 orchestrator wake-up. Cheap
deterministic gate — no-ops (zero tokens) when the run is complete or
progressing; only when a run stalled/died before completing does it launch a
short-lived glm-5.2 agent to diagnose + drive it to a clean DONE. Progress is
judged by live run-proc + log mtime (session_busy() is claude-tuned and misreads
a headless opencode run as idle).
- configuration.nix: cc-ci-upgrade-supervisor service + hourly timer (:07).
- upgrade-all SKILL §0: note the stale-image reclaim for manual runs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
Make explicit that ALL formatting/HTML is owned by recipe-report.py render() and
the model's only artifact is the spec JSON — never hand-write/edit HTML. Matters
now that glm-5.2 drives the report. Also fix stale 'default opus' refs (report
now defaults to opencode-go/glm-5.2, overridable via REPORT_BACKEND/REPORT_MODEL).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Trust abra fully for any image whose tag it can read — a normal semver/calver
tag with no newer version is genuinely up-to-date. Only cross-check upstream for
images abra physically can't parse (tag+digest pins, digest-only pins), which is
the actual immich blind spot. Avoids redundant upstream checks on every recipe.
abra recipe upgrade is the first approach, but it silently contributes no
candidate for tag+digest pins (FATA: tag and digest not supported), digest-only
pins, and non-semver tags. immich kept getting skipped this way. Before
concluding SKIPPED — up-to-date, do a direct upstream tag check for every image
abra could not cleanly evaluate; only skip when BOTH agree nothing is newer.
§2 no longer bumps the coop-cloud version label in the recipe PR (no
--dry-run compute, no tag). It records the recommended 'abra recipe release
<recipe> -<x|y|z>' (no --dry-run) in the PR body (§3) as the operator's
final publish step — run after the upstream PR merges, it bumps the label +
tags + pushes in one go. Bumps recipe-maintainer submodule to 9daddac (same
change across its /recipe-upstream, -upgrade-apply, -upgrade-plan, -new-tag).
cc-ci recipe-upgrade skill now computes the version via 'abra recipe release --dry-run'
(not a hand-edit) and requires the PR body to link upstream release notes per service.
Bumps the recipe-maintainer submodule pointer to the matching change.
Operator: uptime-kuma is maintained elsewhere — drop it from the weekly upgrade
but keep it in the used-recipes inventory. New cc-ci-plan/used-recipes.md is the
canonical list of every recipe cc-ci deploys/tests, with a weekly|external tier;
upgrade-all §1 now excludes 'external' rows from the candidate list (explicit
--args still override). uptime-kuma = external; all others weekly.
The durable /16 proxy fix landed in phase pvfix (2026-06-13).
Update the guard description from "safety net until that lands"
to "belt-and-suspenders even after the /16 fix" — guard logic
unchanged, description now accurate.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Root-caused (empirically, dockerd logs) the discourse/ghost deploy wedges:
the shared proxy overlay (/24=254 VIPs) exhausts as concurrent stack rm leaks
endpoints over many days -> tasks stuck in Swarm 'New'. Add a per-run safety
net to Step 0 (network prune + docker restart when VIP-allocation failures are
logged). Plans + memory for the durable fix (enlarge proxy to /16 in swarm.nix,
maintenance window) and for debugging/fixing the ghost PR afterward.
Per operator: just work through recipes alphabetically keeping CAP (=
DRONE_RUNNER_CAPACITY=2) subagents running at once, starting the next the moment
one finishes (rolling pool via run_in_background). Removes the wave-barrier and
the heavy/light classification entirely — simpler and no slot ever idles.
Per operator: always fill all CAP slots. Heavy/light alternation only spreads
heavies across waves while a light is available; once only heavies remain, run
two-per-wave (capacity is the tuned ceiling) instead of one-per-wave.
Host memory is the binding limit, so never schedule two HEAVY recipes in the
same capacity wave — pair each heavy (discourse/immich/matrix-synapse/
lasuite-drive/mattermost-lts/ghost) with a light one to bound peak memory while
keeping both slots busy. Heaviest-first could co-schedule two heavies and OOM/
wedge the box (the disc-50cc8a 'New'-state wedge). For CAP>2 cap heavies at
~CAP/2; if only heavies remain, run one-per-wave.
Now that the 2026-06-10 concurrency restructure makes concurrent recipe runs
safe (per-run trees, app-domain locks, isolation), default /upgrade-all to run
up to DRONE_RUNNER_CAPACITY (the drone runner's slots, currently 2) recipe
subagents at a time instead of strictly sequential — using all available
concurrency without oversubscribing. Query the live capacity from
'systemctl show drone-runner-exec' (fallback 2); process recipes in waves of
CAP (emit CAP Agent calls per message, await, next wave). Flags: --capacity N,
--sequential (CAP=1, old default — use when the build loops share the box),
--parallel (unbounded). Applies to the NEXT run; the in-flight run is unaffected.
Per operator: drop the hourly cc-ci-reap-dev-deploys systemd timer; instead run the
dev-* reaper at the START (Step 0, alongside the orphan sweep) and END (new step 4b)
of each /upgrade-all run, with THRESHOLD=0 (the run is quiescent then, so clear all
dev-* unconditionally). The reaper keeps its safe default (4h) for ad-hoc use.
Step-2b mandatory teardown is unchanged (primary mechanism); this is the backstop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- /recipe-upgrade step 2b: teardown is now MANDATORY on every exit path (finally),
with a verify-no-leak check; tear down even on failure before reporting.
- reap-dev-deploys.sh: safe, age-gated backstop that removes only idle dev-* stacks
(never CI per-run stacks, warm-*, infra; an active dev loop stays fresh).
- orchestrator: hourly cc-ci-reap-dev-deploys systemd timer runs it against cc-ci,
bounding any leaked dev deploy from a crashed/abandoned loop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Absolute, mode-gated rule reinforced in /recipe-upgrade (Guardrails + the new
step-2b direct-deploy loop where the upgrader has cc-ci host access) and noted as
the interim safeguard in IDEAS.md until the deploy loop moves to isolated infra.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The upgrader now deploys the WIP recipe directly on cc-ci (abra app deploy --chaos
under a dev-<recipe> domain on the local swarm) and inspects live logs
(docker service logs) to SEE what the upgrade does, before/alongside the !testme
CI gate. ADDITIONAL to — not a replacement for — the 3-attempt !testme verification;
it front-loads diagnosis so fewer CI attempts are spent on basics. Always torn down
(orphan-sweep is the backstop). /upgrade-all dispatch references the new step 2b.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
abra hard-FATAs on image refs with both a tag and a digest (immich:
postgres:14-vectorchord...@sha256:..., valkey:9@sha256:...), aborting the whole
recipe survey so immich was silently dropped. Per operator: don't normalize the
recipe; catch the failure and check the upstream registry directly.
- /upgrade-all box item 4: a tag+digest parse FATA is NOT not-fetchable. Use abra
for the images it parses; for the rest, list upstream tags (Docker Hub / ghcr /
buildx imagetools) and judge availability (match the variant the app supports,
not blindly the max). Upgradeable if abra OR the direct check finds a newer tag.
- /recipe-upgrade implement: hand-bump tag+digest pins (abra can't), and re-resolve
+ re-append the digest for the new tag so the pin is preserved (never drop it).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename the table's Status column -> TESTS (the CI/test verdict, unchanged
content). Add a new STATUS column showing the PR's LIVE state, fetched
client-side: 'open' vs a ✓ for any not-open state (merged or closed). The cell
is a JS hook (data-repo/data-pr) derived from existing recipe+pr fields; an
inline, dependency-free, CSP-safe script GETs the same-origin /pr/<recipe>/<n>
proxy (cc-ci nix/modules/reports.nix) on load and every 30s, and degrades to a
muted '?' if the proxy/repo is unreachable. Blank cell when a row has no PR.
Doc + SKILL updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Documents the end-to-end workflow used to land the intentional-skips/4-rung-ladder
feature: explore harness → branch a local cc-ci clone → implement + unit-verify
cold on cc-ci → live full-stage check → open PR (never push main) → independent
adversary verdict → squash-merge on PASS → deploy via /root/builder-clone rebuild.
Includes the adversary-verify-pr6.md plan as a reusable template.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds sweep-orphans.sh (safe-by-allowlist: removes orphan test stacks, standalone
debug containers >30m old, leaked dangling volumes, and reparented docker-run
wrappers; spares infra + warm-* canonicals and their retained volumes) and wires
it as Step 0 of /upgrade-all so a prior run's leaked stack/container/process can't
contend for the shared Swarm or skew the survey. Idempotent; no-op when clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New page order: short lead -> the full wire table (sorted by priority-to-address,
CVE recipes first, new CVEs count column) -> Addendum (bullets of real special
issues, omitted if clean) -> Security Bulletin -> per-recipe "What changed".
- recipe-report.py: _table() gains a CVEs column + recipe-name linking; new
_changes() helper; render() reordered; docstring SPEC SHAPE updated
(cve/addendum/changes added, needs_attention/routine removed).
- recipe-report/SKILL.md + example-spec.json: new procedure, spec shape, and
gold-standard template (2026-06-05, new format).
- launch-report.py: kickoff text reflects the new priority-ordered structure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A stale cc-ci-report session (from a prior week's run, gone idle) caused this week's
launch-report.py 'start' (use-or-create) to leave it and never run a fresh report.
Fix: upgrade-all step 6 now calls 'fresh', and start only leaves a session that's
actively busy producing a report — an idle/leftover session is killed + restarted.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The reconcile that's supposed to make the mirror main == upstream main was fetching origin/main —
but origin is the cc-ci MIRROR, so it synced the mirror to itself (a no-op) and never pulled real
upstream. Fix: fetch coopcloud explicitly (git.coopcloud.tech/coop-cloud/<recipe>, default branch
main OR master) via an 'upstream' remote and force-sync the mirror main + tags from it. Every recipe
has a coopcloud correspondent; none are forked. Also reorder the skill so the reconcile runs BEFORE
the upgrade check, so the check sees the real current recipe. Verified by divergence test (diverged a
mirror, reconcile snapped it back to coopcloud HEAD).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Close the two gaps vs recipe-maintainer's recipe-upgrade-plan:
- Per-recipe release-notes registry at cc-ci-plan/upstream/<recipe>.md (discover the source repo +
releases/changelog URL for each image once, persist+commit, reuse) — fetch release notes FROM those
URLs instead of rediscovering ad-hoc each run. Format doc + cryptpad seed included.
- Explicitly read the recipe's README for shipped upgrade/migration notes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
render() auto-links whole-word recipe mentions in the editorial lead to
git.autonomic.zone/recipe-maintainers/<recipe> (single regex pass, longest-name-first,
no href corruption). Skill: lead is ~3 short paragraphs (~150-180 words) incl. an
'anything strange worth looking into' paragraph. example-spec.json lead updated to the
concise target.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Save the operator-approved 2026-06-02 spec as example-spec.json (gold standard
for voice/structure/specificity). Skill now tells the agent to match it, with
one deliberate change: keep the editorial lead TIGHT (~2 short paragraphs,
~120 words). The live 2026-06-02 page stays as the reference.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Masthead + opus 'lead' editorial (overall fleet state + what to focus on), a Security Bulletin of
critical-CVE upgrades up top (mined from per-recipe upgrade_notes_md), then needs-attention/routine,
and the comprehensive table as 'the full wire' at the end. survey now includes each recipe's
upgrade_notes_md (breaking-change/CVE analysis) so opus can lead with security.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- recipe-report.py: survey (run + per-recipe PRs + CI verdicts) / render (spec->HTML) / publish
(copy to cc-ci:/var/lib/cc-ci-reports + regen index).
- skill .claude/skills/recipe-report: review the weekly run, classify needs-attention vs routine,
publish one public HTML page per week + index at report.ci.commoninternet.net. Read-only.
- launch-report.py: one-shot cc-ci-report agent, REPORT_MODEL default opus (separate from the
sonnet upgrader), REPORT_BACKEND default claude.
- upgrade-all SKILL: closing step launches the report agent.
Serving (nix/modules/reports.nix) already deployed + live.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The actual 'abra auth error' that skipped 8 recipes was go-git failing to
fetch tags from the PRIVATE git.autonomic.zone mirror ('authentication
required: Unauthorized'), NOT the TTY issue. abra/go-git reads
remote.origin.url literally and IGNORES git url.insteadOf + credential
helpers (confirmed: insteadOf left immich Unauthorized; literal embedded URL
fixed it). Skill now bakes $GITEA_USERNAME:$GITEA_PASSWORD into origin for
git.autonomic.zone recipes before the version check, and stashes the
untracked cc-ci overlay so it isn't mis-counted as dirty-worktree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
abra over plain 'ssh cc-ci abra ...' has no TTY -> FATA 'inappropriate ioctl
for device' (the abra error). The working harness (runner/harness/abra.py)
wraps abra in util-linux 'script' for a pseudo-TTY + passes -n. Apply the
same in the recipe-upgrade and upgrade-all skills: every abra call becomes
ssh cc-ci 'script -qec "abra <args> -n" /dev/null'. Confirmed: abra server
ls FATAs plain, works pty-wrapped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Instead of force-pushing HEAD onto the existing PR branch (history rewrite),
add a commit ON TOP of the branch tip (fast-forward) when it already exists,
so the PR's history is preserved and it re-tests. Fresh branches still push
normally. The only remaining force-push is the mirror-main->upstream sync
(intentional mirroring), never a PR branch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When an open upgrade PR already exists for a recipe (branch upgrade-*), push
the new work onto ITS branch and update+re-test that PR — one evolving
upgrade PR per recipe instead of spawning a second parallel PR. Only open a
fresh upgrade-<version> PR when none exists. Unrelated open PRs (e.g. backup
fixes) are still never touched; merged-upstream PRs still close.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per operator: opening a new upgrade PR should stack ON TOP of any other
still-open PRs, not close them. Only PRs already merged into upstream
main are closed (merging them is a no-op). This prevents the phase-7
incident where an unrelated open ghost PR was auto-closed as 'superseded'.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve the recipe branch/ref to its head commit sha via the Gitea API
before invoking the cold full-suite run, so the upgrade tier deploys the
exact PR head. From the phase-5 upgrade-flow verification.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When multiple commit statuses exist (e.g. an Adversary probe + the real run),
the first status in the array may not be the cc-ci run. Filter by context
'cc-ci/testme' to get the correct Drone build URL.
autonomic-bot
autonomic-bot