Bash runner (cheap polling, no claude budget) that gates on the assistant's
PR-consolidation done-marker, waits past the usage-limit reset (~03:30 UTC)
and for the loops to idle, runs the weekly /upgrade-all (DEFAULT, never
merges), then writes overnight-report-<date>.md and pings the orchestrator
to notify. One-off; the Sunday 02:00 timer is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One root doc maps every agent (Builder, Adversary, Orchestrator, Assistant,
Upgrader) -> its prompt + plan, with the watchdog and git coordination
protocol as the subtlety beneath. Fold the orchestrator supervision routine
into it (remove orchestrator-supervision.md). The hourly wake prompt and
AGENTS.md now just point at orchestration.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The hourly wake prompt was hardcoding phase 5 / STATUS-5.md and going stale
as the build advanced. Make it a one-line pointer to a maintained doc
(orchestrator-supervision.md) that looks the CURRENT phase up live via
launch.py status — so the wake prompt never needs editing as phases change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One deliberately-broken custom-html-tiny fixture per lifecycle tier so the
suite proves the server reports RED at EVERY tier (not just one) — each
asserts RED at the intended tier with prior tiers PASS, so it's 'catches a
failure at this tier', not 'fails somewhere'. Fast (simplest recipe); the
fast subset of the suite vs the slow good canaries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The gate existed because a wrong-target nixos-rebuild #cc-ci once dropped
the cc-ci server into emergency mode. That footgun is fixed (be4f451 maps
#cc-ci -> the Hetzner host config), and deploying cc-ci is the loops'
normal operation, so Phase 4 now runs autonomously with verify + rollback
as the safety net.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
E2E pytest canaries proving the server confirms a healthy app healthy
(semantic per-tier assertions, not just exit codes) AND catches a broken
one (false-green guard). Good canaries: custom-html-tiny + lasuite-docs;
known-bad fixture must report RED. Queued as the loops' next phase after
mirror-enroll.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mirror the .loop-backend pattern: env wins, else the persisted file, else
the default build sequence. Without this, a custom single-phase run was
invisible to bare 'launch.py status' and would NOT survive a reboot (the
service has no PHASES_SPEC env). Now the current phase set is durable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The watchdog is spawned into the existing tmux server and didn't reliably
inherit a custom PHASES_SPEC — it would fall back to the default 11-phase
spec and mis-detect completion. Forward PHASES_SPEC/PHASE_IDX_FILE/
LOOP_BACKEND/LOOP_MODEL explicitly in the watchdog command so custom
single-phase runs (like the mirror-enroll plan) work end-to-end. Also make
the mirror-enroll plan's live-host-deploy step an explicit claim-and-wait
operator gate for the loops.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The standalone ai-progress-monitor.sh waker pinged a hardcoded
orchestrator session every 15m. Move that into the watchdog loop:
ORCH_WAKE_INTERVAL (default 3600s) types the supervision prompt into
the live orchestrator session, retrying each tick until it lands so a
busy or briefly-absent orchestrator is never interrupted and no hour is
skipped. Delete the now-redundant waker script; the prompt file is now
driven by the watchdog. Reboot-safe by inheritance (the watchdog is
started by cc-ci-loops.service).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Same fix as the loops: opencode run --attach exits after one turn;
plain opencode TUI stays alive in tmux. Send startup prompt via
ping_session (Enter) after 8s init wait. Bootstrap points to
JOURNAL.md rather than sending the full prompt inline.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1. API key: opencode doesn't support env: substitution in apiKey — write
actual key value to ~/.config/opencode/opencode.jsonc at setup time
(file is not committed to git; key sourced from .testenv).
2. Permission system: add permission:"allow" to opencode config (equivalent
to --dangerously-skip-permissions) to avoid interactive prompts.
3. Submit key: opencode TUI uses Enter (return) to submit; Ctrl+S not
needed. ping_session already uses Enter — keep as is.
4. Startup timing: bump opencode TUI init wait from 4s to 8s so the TUI
is fully connected to the server before bootstrap is sent.
5. Backend persistence: LOOP_BACKEND/LOOP_MODEL written to .loop-backend /
.loop-model so the watchdog uses them when restarting dead sessions.
All tested: both builder and adversary sessions alive, deepseek-v4-pro
processing kickoffs via tinfoil inference.tinfoil.sh, no API/permission
errors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three fixes discovered during first live run:
- inference host is inference.tinfoil.sh not api.tinfoil.sh (control plane
only serves /v1/models, not /v1/chat/completions)
- opencode run exits after one turn; switch to opencode attach for the
persistent TUI, then ping_session sends the kickoff prompt
- NO_COLOR=1 suppresses the first-run interactive theme picker
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Bash scripts are now one-liner wrappers: exec python3 <script>.py "$@"
All logic lives in the Python scripts (pure stdlib, no deps).
launch.py — loops + watchdog:
Full port of launch.sh: phase sequencing, start/stop/status/logs/watchdog,
handoff signalling, stall detection, heal_session, heal_orchestrator.
Cleaner structure: config block → helpers → phase/kickoff/agent/healing/
handoff/watchdog/main. LOOP_BACKEND + LOOP_MODEL switches throughout.
launch-orchestrator.py — orchestrator session:
claude path: --resume <id> preserved (conversation survives reboots).
opencode path: run --attach --title (no --resume; STARTUP_PROMPT orients
the new session; reads JOURNAL.md for context).
STARTUP_PROMPT updated to reference JOURNAL.md on startup.
launch-upgrader.py — one-shot upgrade job:
LOOP_BACKEND / LOOP_MODEL take precedence over UPGRADER_BACKEND / UPGRADER_MODEL.
Both claude and opencode paths supported.
cc-ci-plan/JOURNAL.md — new orchestrator handoff file:
Persistent across conversation resets. Documents the handoff format and
carries the current session's summary: migration complete, phase 5 in
progress (V3/V7 PASS), phase 4 deferred, open items for next session.
AGENTS.md: step 1 on startup = read JOURNAL.md; step 5 = append on handoff.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
configuration.nix:
- systemd.services.opencode-web: one shared opencode server on 127.0.0.1:4096,
EnvironmentFile=/srv/cc-ci/.testenv (TINFOIL_API_KEY), ExecStartPre clears
stale /tmp/opencode so restarts never fail on the EEXIST race.
- services.nginx: reverse-proxy oc.commoninternet.net → localhost:4096,
bound to tailscale IP 100.84.190.30 (tailnet-only, plain HTTP).
DNS: A record oc.commoninternet.net → 100.84.190.30 (operator step).
launch.sh + launch-upgrader.sh:
- Drop per-session ports / OPENCODE_HOST; add OPENCODE_SERVER=http://127.0.0.1:4096.
- opencode backend: agents use `opencode run --attach $OPENCODE_SERVER --title $session`
so each shows up as a named session in the web UI.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds UPGRADER_MODEL env var (default: sonnet) passed as --model to the
claude invocation. The cron runs the upgrader on Sonnet so it doesn't
consume Opus weekly credits. Override with UPGRADER_MODEL=opus if needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Now the workspace is staged on the Hetzner cpx22 (server 134487234, public
91.98.47.73, tailnet cc-ci-orchestrator-1 @ 100.84.190.30):
- configuration.nix: enable cc-ci-loops.service (wantedBy multi-user.target) so the
loops + watchdog auto-resume on boot; wire reboot-log.sh as ExecStartPre so reboots
auto-log to REBOOTS.md (boot_id-gated).
- plan-orchestrator-hetzner-migration.md: full migration record.
- REBOOTS.md / AGENTS.md: point the orchestrator host at Hetzner; first auto-logged
reboot line.
- launch-orchestrator.sh: default session id -> the Hetzner orchestrator session.
- flake.lock: pin inputs.
Verified: nixos-rebuild switch applied; systemctl is-enabled cc-ci-loops.service =
enabled; ExecStartPre logged this boot to REBOOTS.md; loops healthy on phase 2.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The orchestrator Pi is retired (2026-05-31). All agents now run on the
cc-ci-orchestrator VM (NixOS, loops user, /srv/cc-ci). The VM is a
direct tailnet peer to cc-ci — no SOCKS proxy, no userspace tailscaled,
no ProxyCommand. Updated across all affected files:
AGENTS.md
- Remove Pi from reboot description; migration complete (not "parked")
- cc-ci access: direct ssh, not via proxy
kickoff.md
- Prerequisites: direct tailnet peer, not proxy
- Host deps: NixOS (not apt)
- Fallback/Incus: b1 reachable directly, no --proxy curl flag
plan.md §1 + §1.5
- §1 bootstrap: direct SSH, check tailscale status (not restart proxy)
- §1.5 intro: "VM" not "sandbox host"; no proxy
- Credentials table: remove TS_AUTH_KEY row; update cc-ci SSH row
- Replace "Tailscale connection (proxy)" subsection with direct-peer description
plan-orchestrator-migration.md
- Mark COMPLETE (2026-05-31); historical record only
plan-phase1c-full-reproducibility.md
- Incus access: direct, not via SOCKS proxy
prompts/builder.md + prompts/adversary.md
- cc-ci access language only: direct ssh, no proxy restart instructions
- adversary: *.ci.commoninternet.net via plain curl, no proxy flag
REBOOTS.md
- Retitle for VM; note Pi retired; Pi entries marked historical
systemd/cc-ci-loops.service
- User/Group/HOME/PATH: notplants → loops
- Remove cc-ci-tailscaled.service dependency (no proxy on VM)
- Add note about nix/configuration.nix as the authoritative VM declaration
test-e2e-testme-acceptance.md
- tailscale status: no --socket flag
- ssh to throwaway: no ProxyCommand
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Gitea repos renamed:
cc-ci-autonomous-orchestrator → cc-ci-orchestrator
cc-ci-orchestrator → archived-cc-ci-orchestrator
Updated in this workspace:
- README.md, AGENTS.md: repo title
- cc-ci-plan/plan-orchestrator-migration.md: cc-ci-autonomous-orchestrator refs
- cc-ci-plan/plan-repo-consolidation.md: marked complete + Pi remote-update notice
- cc-ci-plan/launch-orchestrator.sh, launch.sh: session naming comment cleanup
NOTE: Pi clone still has the old origin URL. On the Pi, run:
git remote set-url origin https://git.autonomic.zone/recipe-maintainers/cc-ci-orchestrator.git
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
A general-purpose Claude session sharing the orchestrator's workspace + access,
under remote-control (cc-ci-assistant), NOT on a loop. Sits idle until the
orchestrator/operator hands it a plan/task, does it, reports, waits. Modelled on
launch-orchestrator.sh: persistent pinned session-id (resume across relaunch),
root-aware --dangerously-skip-permissions handling, start/fresh/status/attach/stop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator: use a single uniform filename `compose.ccci.yml` per recipe (one file
holding all cc-ci-side deploy tweaks) rather than per-purpose suffixes like
compose.ccci-health.yml. Updated §9 + plan-ccci-compose-overlay-policy.md; added
a DoD item to rename tests/{ghost,discourse}/compose.ccci-health.yml ->
compose.ccci.yml and update their install_steps.sh cp target + recipe_meta
COMPOSE_FILE.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The prior commit only captured the file deletion (git add aborted on the
already-removed pathspec). This adds the actual content: the reworked §9
guardrail (justified ccci overlays OK; abra can't env start_period; always test
upgrade-to-latest, from-version custom tests skippable) and the new policy doc.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator correction (builder was right): abra does NOT support an env value for
healthcheck start_period, so the earlier "parameterize via APP_START_PERIOD env
PR" approach is impossible — a ccci compose overlay is the right tool there.
- plan.md §9: replace the "don't fork compose / use env PR" guardrail with
"avoid where possible + justify each + prefer upstream PR, BUT a uniform
optional compose.ccci-*.yml overlay is an acceptable fallback" (esp. for
abra-unparameterizable values like start_period). Add the upgrade-tier rule:
ALWAYS test the upgrade to latest; a from-version's custom tests may be
skipped if it can't fully run, but never drop upgrade-to-latest.
- replace plan-prefer-env-over-compose-overlay.md with
plan-ccci-compose-overlay-policy.md: ghost/discourse start_period overlays
STAY (justified); discourse image re-pin STAYS (keeps the upgrade-to-latest
testable; 0.7.0 custom tests may be skipped); mumble old-base host-ports copy
DROPPED (skip 0.2.0 voice tests, still upgrade to latest + test there). Each
surviving overlay must be minimal + header-justified + Adversary-confirmed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase A done before the Pi's reboot #3 (commit was interrupted): the loops VM
cc-ci-orchestrator is on the tailnet (100.116.55.106) and ssh-able; TS-key
finding recorded (VM-creator .test.env key revoked; cc-ci .testenv key valid +
persisted). REBOOTS.md carries the auto-logged 2026-05-30 17:03 reboot
(cc-ci-loops.service auto-recovered the loops at phase 2; swapfile persisted).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator go-ahead (Pi is OOM-thrashing/slow). Created the dedicated loops VM
cc-ci-orchestrator (2GB RAM / 2 vCPU / 30GB, incus-base-vm NixOS) on b1 via the
Incus API, mirroring the known-good cc-nix-test spec; started it — cloud-init is
running nixos-rebuild boot + reboot + tailnet join. Status flipped DRAFT->IN
PROGRESS with the remaining Phase-A items noted (add cc-ci-root key via incus
exec, confirm tailnet+ssh, write the reproducible TF project).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tmux `send-keys -l <long msg>` often leaves the text UNSENT in the input box (the
immediate Enter is swallowed while the TUI ingests the paste). Both now type the
message then retry Enter/C-m until the leading text is no longer in the input box
(= submitted) or a bounded loop gives up.
- msg-loop.sh: standalone reliable messenger for orchestrator use.
- launch.sh ping_session: same retry-submit (loads on next watchdog restart).
Live-tested: delivered first try.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator (2026-05-30): a cc-ci-authored compose overlay risks silent drift from
the recipe users actually run — avoid it wherever possible.
- plan.md §9 guardrail: when a recipe needs a cc-ci-env-tuned value (e.g. a longer
healthcheck start_period for the slow single node), the preferred fix is an
UPSTREAM recipe PR exposing it as an env var (e.g. APP_START_PERIOD) with the
current value as the default in env.sample — CI sets the env, no new compose.
For making the upgrade tier work from an older base version, prefer DECLARING
that version not-testable under this CI env over crafting a custom compose.
Overlay = last resort, Adversary-confirmed non-drifting + paired with the env PR.
- plan-prefer-env-over-compose-overlay.md: migrates the existing debt —
ghost/discourse compose.ccci-health.yml start_period -> APP_START_PERIOD recipe
PRs (default=current) then drop the overlays; discourse image re-pin + mumble
old-base host-ports copy -> declare those old versions untestable instead of
forking compose. No test weakened; untestable-version is an honest outcome.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
For a long deploy/convergence, arm a Monitor that polls the node every ~30s and
wakes on convergence OR failure, with a longer fallback heartbeat (ScheduleWakeup)
as a backstop. Proceeds the instant it converges (no over-waiting), surfaces
failures promptly, and the heartbeat bounds the wait. Size the timeout sanely
(longer if justified, never absurd like the ~40-min ghost case). Credit: builder.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator (2026-05-30): the real deploy-speed bottleneck was hardware (cc-ci VM
was 2 vCPU on a 4-core host + disk-I/O-bound; RAM fine), now fixed directly
(bumped to 4 vCPU, made cc-nix-test the only running VM on b1). The 2b software
micro-optimizations are judged unlikely to help, so:
- IDEAS.md: parked the whole empirical-perf program (instrumentation, baseline,
attribution) + the optimization menu (image cache/prepull, readiness tuning,
warm-SSO start/stop, runner caching, concurrency sizing, resources, secret
overhead) under "Phase-2b empirical performance work", revisit only if
measurement later proves a specific software bottleneck.
- plan-phase2b: reduced to ONE goal — confirm (and fix if needed) that the
per-recipe test sequence already uses the minimum deploys (1 base shared by
install+functional+backup/restore, +1 for the upgrade tier, +1 per dep),
enforced by the existing DG4.1 deploy-count check, WITHOUT weakening any test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
