Persistent agent memories now live in memory/ in this repo; the Claude
auto-memory path is symlinked here so future memories land in the repo
and get committed like any other change.
After a live incident: plausible build 220 (ClickHouse exit-1 crash-loop) held the
single serial runner for its full 1200s DEPLOY_TIMEOUT, starving immich PR-2's
queued builds for ~12min until manually torn down. Logs the two fixes (fail-fast
on crash-loop; head-of-line blocking on the serial runner) + the interim
mitigations (step-2b dev loop for debugging; SIGINT to free a wedged run).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per operator: drop the hourly cc-ci-reap-dev-deploys systemd timer; instead run the
dev-* reaper at the START (Step 0, alongside the orphan sweep) and END (new step 4b)
of each /upgrade-all run, with THRESHOLD=0 (the run is quiescent then, so clear all
dev-* unconditionally). The reaper keeps its safe default (4h) for ad-hoc use.
Step-2b mandatory teardown is unchanged (primary mechanism); this is the backstop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- /recipe-upgrade step 2b: teardown is now MANDATORY on every exit path (finally),
with a verify-no-leak check; tear down even on failure before reporting.
- reap-dev-deploys.sh: safe, age-gated backstop that removes only idle dev-* stacks
(never CI per-run stacks, warm-*, infra; an active dev loop stays fresh).
- orchestrator: hourly cc-ci-reap-dev-deploys systemd timer runs it against cc-ci,
bounding any leaked dev deploy from a crashed/abandoned loop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Absolute, mode-gated rule reinforced in /recipe-upgrade (Guardrails + the new
step-2b direct-deploy loop where the upgrader has cc-ci host access) and noted as
the interim safeguard in IDEAS.md until the deploy loop moves to isolated infra.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The step-2b direct deploy-and-inspect runs on the cc-ci server's own swarm today, so
the upgrader holds write access to the host that owns the tests + CI verdict — a
trust hole (could hack the tests). Parked idea: a dedicated throwaway test server
with scoped creds, so the upgrader can deploy+inspect but not modify the gate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The upgrader now deploys the WIP recipe directly on cc-ci (abra app deploy --chaos
under a dev-<recipe> domain on the local swarm) and inspects live logs
(docker service logs) to SEE what the upgrade does, before/alongside the !testme
CI gate. ADDITIONAL to — not a replacement for — the 3-attempt !testme verification;
it front-loads diagnosis so fewer CI attempts are spent on basics. Always torn down
(orphan-sweep is the backstop). /upgrade-all dispatch references the new step 2b.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
abra hard-FATAs on image refs with both a tag and a digest (immich:
postgres:14-vectorchord...@sha256:..., valkey:9@sha256:...), aborting the whole
recipe survey so immich was silently dropped. Per operator: don't normalize the
recipe; catch the failure and check the upstream registry directly.
- /upgrade-all box item 4: a tag+digest parse FATA is NOT not-fetchable. Use abra
for the images it parses; for the rest, list upstream tags (Docker Hub / ghcr /
buildx imagetools) and judge availability (match the variant the app supports,
not blindly the max). Upgradeable if abra OR the direct check finds a newer tag.
- /recipe-upgrade implement: hand-bump tag+digest pins (abra can't), and re-resolve
+ re-append the digest for the new tag so the pin is preserved (never drop it).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename the table's Status column -> TESTS (the CI/test verdict, unchanged
content). Add a new STATUS column showing the PR's LIVE state, fetched
client-side: 'open' vs a ✓ for any not-open state (merged or closed). The cell
is a JS hook (data-repo/data-pr) derived from existing recipe+pr fields; an
inline, dependency-free, CSP-safe script GETs the same-origin /pr/<recipe>/<n>
proxy (cc-ci nix/modules/reports.nix) on load and every 30s, and degrades to a
muted '?' if the proxy/repo is unreachable. Blank cell when a row has no PR.
Doc + SKILL updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Documents the end-to-end workflow used to land the intentional-skips/4-rung-ladder
feature: explore harness → branch a local cc-ci clone → implement + unit-verify
cold on cc-ci → live full-stage check → open PR (never push main) → independent
adversary verdict → squash-merge on PASS → deploy via /root/builder-clone rebuild.
Includes the adversary-verify-pr6.md plan as a reusable template.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds sweep-orphans.sh (safe-by-allowlist: removes orphan test stacks, standalone
debug containers >30m old, leaked dangling volumes, and reparented docker-run
wrappers; spares infra + warm-* canonicals and their retained volumes) and wires
it as Step 0 of /upgrade-all so a prior run's leaked stack/container/process can't
contend for the shared Swarm or skew the survey. Idempotent; no-op when clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New page order: short lead -> the full wire table (sorted by priority-to-address,
CVE recipes first, new CVEs count column) -> Addendum (bullets of real special
issues, omitted if clean) -> Security Bulletin -> per-recipe "What changed".
- recipe-report.py: _table() gains a CVEs column + recipe-name linking; new
_changes() helper; render() reordered; docstring SPEC SHAPE updated
(cve/addendum/changes added, needs_attention/routine removed).
- recipe-report/SKILL.md + example-spec.json: new procedure, spec shape, and
gold-standard template (2026-06-05, new format).
- launch-report.py: kickoff text reflects the new priority-ordered structure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A stale cc-ci-report session (from a prior week's run, gone idle) caused this week's
launch-report.py 'start' (use-or-create) to leave it and never run a fresh report.
Fix: upgrade-all step 6 now calls 'fresh', and start only leaves a session that's
actively busy producing a report — an idle/leftover session is killed + restarted.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The reconcile that's supposed to make the mirror main == upstream main was fetching origin/main —
but origin is the cc-ci MIRROR, so it synced the mirror to itself (a no-op) and never pulled real
upstream. Fix: fetch coopcloud explicitly (git.coopcloud.tech/coop-cloud/<recipe>, default branch
main OR master) via an 'upstream' remote and force-sync the mirror main + tags from it. Every recipe
has a coopcloud correspondent; none are forked. Also reorder the skill so the reconcile runs BEFORE
the upgrade check, so the check sees the real current recipe. Verified by divergence test (diverged a
mirror, reconcile snapped it back to coopcloud HEAD).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Close the two gaps vs recipe-maintainer's recipe-upgrade-plan:
- Per-recipe release-notes registry at cc-ci-plan/upstream/<recipe>.md (discover the source repo +
releases/changelog URL for each image once, persist+commit, reuse) — fetch release notes FROM those
URLs instead of rediscovering ad-hoc each run. Format doc + cryptpad seed included.
- Explicitly read the recipe's README for shipped upgrade/migration notes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_stories() now auto-links whole-word recipe mentions in story titles + bodies to their mirror
repos (same single-pass linkify as the lead); explicit PR/build links are untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
render() auto-links whole-word recipe mentions in the editorial lead to
git.autonomic.zone/recipe-maintainers/<recipe> (single regex pass, longest-name-first,
no href corruption). Skill: lead is ~3 short paragraphs (~150-180 words) incl. an
'anything strange worth looking into' paragraph. example-spec.json lead updated to the
concise target.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Save the operator-approved 2026-06-02 spec as example-spec.json (gold standard
for voice/structure/specificity). Skill now tells the agent to match it, with
one deliberate change: keep the editorial lead TIGHT (~2 short paragraphs,
~120 words). The live 2026-06-02 page stays as the reference.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Masthead + opus 'lead' editorial (overall fleet state + what to focus on), a Security Bulletin of
critical-CVE upgrades up top (mined from per-recipe upgrade_notes_md), then needs-attention/routine,
and the comprehensive table as 'the full wire' at the end. survey now includes each recipe's
upgrade_notes_md (breaking-change/CVE analysis) so opus can lead with security.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- recipe-report.py: survey (run + per-recipe PRs + CI verdicts) / render (spec->HTML) / publish
(copy to cc-ci:/var/lib/cc-ci-reports + regen index).
- skill .claude/skills/recipe-report: review the weekly run, classify needs-attention vs routine,
publish one public HTML page per week + index at report.ci.commoninternet.net. Read-only.
- launch-report.py: one-shot cc-ci-report agent, REPORT_MODEL default opus (separate from the
sonnet upgrader), REPORT_BACKEND default claude.
- upgrade-all SKILL: closing step launches the report agent.
Serving (nix/modules/reports.nix) already deployed + live.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The actual 'abra auth error' that skipped 8 recipes was go-git failing to
fetch tags from the PRIVATE git.autonomic.zone mirror ('authentication
required: Unauthorized'), NOT the TTY issue. abra/go-git reads
remote.origin.url literally and IGNORES git url.insteadOf + credential
helpers (confirmed: insteadOf left immich Unauthorized; literal embedded URL
fixed it). Skill now bakes $GITEA_USERNAME:$GITEA_PASSWORD into origin for
git.autonomic.zone recipes before the version check, and stashes the
untracked cc-ci overlay so it isn't mis-counted as dirty-worktree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The raw 'tmux pipe-pane' logs are TUI-escape soup (the 191MB builder log).
agent-log.py renders Claude's own JSONL transcript into a clean one-event-
per-line <agent>.clean.log — read-only on a file the agent writes anyway, so
zero agent slowdown and zero extra tokens. Resolves each agent's transcript
(disambiguating the shared project dir by kickoff signature; tracks restarts).
'follow-all' runs as the cc-ci-cleanlogs session, wired into launch.py start
so it comes up with the loops. render/tail subcommands for ad-hoc use.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
abra over plain 'ssh cc-ci abra ...' has no TTY -> FATA 'inappropriate ioctl
for device' (the abra error). The working harness (runner/harness/abra.py)
wraps abra in util-linux 'script' for a pseudo-TTY + passes -n. Apply the
same in the recipe-upgrade and upgrade-all skills: every abra call becomes
ssh cc-ci 'script -qec "abra <args> -n" /dev/null'. Confirmed: abra server
ls FATAs plain, works pty-wrapped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bash runner (cheap polling, no claude budget) that gates on the assistant's
PR-consolidation done-marker, waits past the usage-limit reset (~03:30 UTC)
and for the loops to idle, runs the weekly /upgrade-all (DEFAULT, never
merges), then writes overnight-report-<date>.md and pings the orchestrator
to notify. One-off; the Sunday 02:00 timer is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Instead of force-pushing HEAD onto the existing PR branch (history rewrite),
add a commit ON TOP of the branch tip (fast-forward) when it already exists,
so the PR's history is preserved and it re-tests. Fresh branches still push
normally. The only remaining force-push is the mirror-main->upstream sync
(intentional mirroring), never a PR branch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When an open upgrade PR already exists for a recipe (branch upgrade-*), push
the new work onto ITS branch and update+re-test that PR — one evolving
upgrade PR per recipe instead of spawning a second parallel PR. Only open a
fresh upgrade-<version> PR when none exists. Unrelated open PRs (e.g. backup
fixes) are still never touched; merged-upstream PRs still close.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One root doc maps every agent (Builder, Adversary, Orchestrator, Assistant,
Upgrader) -> its prompt + plan, with the watchdog and git coordination
protocol as the subtlety beneath. Fold the orchestrator supervision routine
into it (remove orchestrator-supervision.md). The hourly wake prompt and
AGENTS.md now just point at orchestration.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The hourly wake prompt was hardcoding phase 5 / STATUS-5.md and going stale
as the build advanced. Make it a one-line pointer to a maintained doc
(orchestrator-supervision.md) that looks the CURRENT phase up live via
launch.py status — so the wake prompt never needs editing as phases change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One deliberately-broken custom-html-tiny fixture per lifecycle tier so the
suite proves the server reports RED at EVERY tier (not just one) — each
asserts RED at the intended tier with prior tiers PASS, so it's 'catches a
failure at this tier', not 'fails somewhere'. Fast (simplest recipe); the
fast subset of the suite vs the slow good canaries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The gate existed because a wrong-target nixos-rebuild #cc-ci once dropped
the cc-ci server into emergency mode. That footgun is fixed (be4f451 maps
#cc-ci -> the Hetzner host config), and deploying cc-ci is the loops'
normal operation, so Phase 4 now runs autonomously with verify + rollback
as the safety net.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
E2E pytest canaries proving the server confirms a healthy app healthy
(semantic per-tier assertions, not just exit codes) AND catches a broken
one (false-green guard). Good canaries: custom-html-tiny + lasuite-docs;
known-bad fixture must report RED. Queued as the loops' next phase after
mirror-enroll.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mirror the .loop-backend pattern: env wins, else the persisted file, else
the default build sequence. Without this, a custom single-phase run was
invisible to bare 'launch.py status' and would NOT survive a reboot (the
service has no PHASES_SPEC env). Now the current phase set is durable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The watchdog is spawned into the existing tmux server and didn't reliably
inherit a custom PHASES_SPEC — it would fall back to the default 11-phase
spec and mis-detect completion. Forward PHASES_SPEC/PHASE_IDX_FILE/
LOOP_BACKEND/LOOP_MODEL explicitly in the watchdog command so custom
single-phase runs (like the mirror-enroll plan) work end-to-end. Also make
the mirror-enroll plan's live-host-deploy step an explicit claim-and-wait
operator gate for the loops.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- SSH config: replace REPLACE_WITH_CC_CI_HETZNER_TAILNET_IP placeholder with
the real tailnet IP 100.95.31.88 (so a fresh re-provision is correct).
- nix/configuration.nix + nix/README.md: mark HISTORICAL/dead (old Incus VM,
superseded by the Hetzner host) to prevent a wrong-host deploy.
- nginx oc.commoninternet.net vhost: note it's PARKED alongside opencode-web
(kept for one-step re-enable, not deleted).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per operator: opening a new upgrade PR should stack ON TOP of any other
still-open PRs, not close them. Only PRs already merged into upstream
main are closed (merging them is a no-op). This prevents the phase-7
incident where an unrelated open ghost PR was auto-closed as 'superseded'.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Keep the unit definition in the flake for easy re-enable; just stop it
auto-starting. Restore wantedBy = [ "multi-user.target" ] to bring it back.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the boot-fragile busybox-crond-in-tmux (phase 5 §4) with a
systemd service+timer. Service is timer-triggered only (not wantedBy
multi-user.target) so it never runs on boot/activation; mirrors the
cc-ci-loops env fix (CLAUDE_BIN + /home/loops/.local/bin on PATH).
Timer fires Sundays 02:00 UTC, Persistent=true so a missed run (box
down) fires once on next boot. Runs launch-upgrader.py start ->
cc-ci-upgrader agent -> /upgrade-all DEFAULT (opens recipe PRs, never
merges). Activate via nixos-rebuild + retire the old Monday crond after
the phase-5 T0-fire verification completes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve the recipe branch/ref to its head commit sha via the Gitea API
before invoking the cold full-suite run, so the upgrade tier deploys the
exact PR head. From the phase-5 upgrade-flow verification.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The service path lacked /home/loops/.local/bin, so launch.py preflight's
which(claude) failed on every boot and the loops never auto-started
(they were restarted by hand). Set CLAUDE_BIN to the standalone CLI's
absolute path and prepend the dir to PATH so the tmux server every agent
session inherits resolves bare claude.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The standalone ai-progress-monitor.sh waker pinged a hardcoded
orchestrator session every 15m. Move that into the watchdog loop:
ORCH_WAKE_INTERVAL (default 3600s) types the supervision prompt into
the live orchestrator session, retrying each tick until it lands so a
busy or briefly-absent orchestrator is never interrupted and no hour is
skipped. Delete the now-redundant waker script; the prompt file is now
driven by the watchdog. Reboot-safe by inheritance (the watchdog is
started by cc-ci-loops.service).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Same fix as the loops: opencode run --attach exits after one turn;
plain opencode TUI stays alive in tmux. Send startup prompt via
ping_session (Enter) after 8s init wait. Bootstrap points to
JOURNAL.md rather than sending the full prompt inline.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1. API key: opencode doesn't support env: substitution in apiKey — write
actual key value to ~/.config/opencode/opencode.jsonc at setup time
(file is not committed to git; key sourced from .testenv).
2. Permission system: add permission:"allow" to opencode config (equivalent
to --dangerously-skip-permissions) to avoid interactive prompts.
3. Submit key: opencode TUI uses Enter (return) to submit; Ctrl+S not
needed. ping_session already uses Enter — keep as is.
4. Startup timing: bump opencode TUI init wait from 4s to 8s so the TUI
is fully connected to the server before bootstrap is sent.
5. Backend persistence: LOOP_BACKEND/LOOP_MODEL written to .loop-backend /
.loop-model so the watchdog uses them when restarting dead sessions.
All tested: both builder and adversary sessions alive, deepseek-v4-pro
processing kickoffs via tinfoil inference.tinfoil.sh, no API/permission
errors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three fixes discovered during first live run:
- inference host is inference.tinfoil.sh not api.tinfoil.sh (control plane
only serves /v1/models, not /v1/chat/completions)
- opencode run exits after one turn; switch to opencode attach for the
persistent TUI, then ping_session sends the kickoff prompt
- NO_COLOR=1 suppresses the first-run interactive theme picker
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Bash scripts are now one-liner wrappers: exec python3 <script>.py "$@"
All logic lives in the Python scripts (pure stdlib, no deps).
launch.py — loops + watchdog:
Full port of launch.sh: phase sequencing, start/stop/status/logs/watchdog,
handoff signalling, stall detection, heal_session, heal_orchestrator.
Cleaner structure: config block → helpers → phase/kickoff/agent/healing/
handoff/watchdog/main. LOOP_BACKEND + LOOP_MODEL switches throughout.
launch-orchestrator.py — orchestrator session:
claude path: --resume <id> preserved (conversation survives reboots).
opencode path: run --attach --title (no --resume; STARTUP_PROMPT orients
the new session; reads JOURNAL.md for context).
STARTUP_PROMPT updated to reference JOURNAL.md on startup.
launch-upgrader.py — one-shot upgrade job:
LOOP_BACKEND / LOOP_MODEL take precedence over UPGRADER_BACKEND / UPGRADER_MODEL.
Both claude and opencode paths supported.
cc-ci-plan/JOURNAL.md — new orchestrator handoff file:
Persistent across conversation resets. Documents the handoff format and
carries the current session's summary: migration complete, phase 5 in
progress (V3/V7 PASS), phase 4 deferred, open items for next session.
AGENTS.md: step 1 on startup = read JOURNAL.md; step 5 = append on handoff.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
configuration.nix:
- systemd.services.opencode-web: one shared opencode server on 127.0.0.1:4096,
EnvironmentFile=/srv/cc-ci/.testenv (TINFOIL_API_KEY), ExecStartPre clears
stale /tmp/opencode so restarts never fail on the EEXIST race.
- services.nginx: reverse-proxy oc.commoninternet.net → localhost:4096,
bound to tailscale IP 100.84.190.30 (tailnet-only, plain HTTP).
DNS: A record oc.commoninternet.net → 100.84.190.30 (operator step).
launch.sh + launch-upgrader.sh:
- Drop per-session ports / OPENCODE_HOST; add OPENCODE_SERVER=http://127.0.0.1:4096.
- opencode backend: agents use `opencode run --attach $OPENCODE_SERVER --title $session`
so each shows up as a named session in the web UI.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When multiple commit statuses exist (e.g. an Adversary probe + the real run),
the first status in the array may not be the cc-ci run. Filter by context
'cc-ci/testme' to get the correct Drone build URL.
Adds UPGRADER_MODEL env var (default: sonnet) passed as --model to the
claude invocation. The cron runs the upgrader on Sonnet so it doesn't
consume Opus weekly credits. Override with UPGRADER_MODEL=opus if needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Now the workspace is staged on the Hetzner cpx22 (server 134487234, public
91.98.47.73, tailnet cc-ci-orchestrator-1 @ 100.84.190.30):
- configuration.nix: enable cc-ci-loops.service (wantedBy multi-user.target) so the
loops + watchdog auto-resume on boot; wire reboot-log.sh as ExecStartPre so reboots
auto-log to REBOOTS.md (boot_id-gated).
- plan-orchestrator-hetzner-migration.md: full migration record.
- REBOOTS.md / AGENTS.md: point the orchestrator host at Hetzner; first auto-logged
reboot line.
- launch-orchestrator.sh: default session id -> the Hetzner orchestrator session.
- flake.lock: pin inputs.
Verified: nixos-rebuild switch applied; systemctl is-enabled cc-ci-loops.service =
enabled; ExecStartPre logged this boot to REBOOTS.md; loops healthy on phase 2.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
nixos-rebuild removed the infect-provisioned authorized_keys — declare it
explicitly so rebuilds don't lock out root access.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
