Replace the boot-fragile busybox-crond-in-tmux (phase 5 §4) with a
systemd service+timer. Service is timer-triggered only (not wantedBy
multi-user.target) so it never runs on boot/activation; mirrors the
cc-ci-loops env fix (CLAUDE_BIN + /home/loops/.local/bin on PATH).
Timer fires Sundays 02:00 UTC, Persistent=true so a missed run (box
down) fires once on next boot. Runs launch-upgrader.py start ->
cc-ci-upgrader agent -> /upgrade-all DEFAULT (opens recipe PRs, never
merges). Activate via nixos-rebuild + retire the old Monday crond after
the phase-5 T0-fire verification completes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve the recipe branch/ref to its head commit sha via the Gitea API
before invoking the cold full-suite run, so the upgrade tier deploys the
exact PR head. From the phase-5 upgrade-flow verification.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The service path lacked /home/loops/.local/bin, so launch.py preflight's
which(claude) failed on every boot and the loops never auto-started
(they were restarted by hand). Set CLAUDE_BIN to the standalone CLI's
absolute path and prepend the dir to PATH so the tmux server every agent
session inherits resolves bare claude.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The standalone ai-progress-monitor.sh waker pinged a hardcoded
orchestrator session every 15m. Move that into the watchdog loop:
ORCH_WAKE_INTERVAL (default 3600s) types the supervision prompt into
the live orchestrator session, retrying each tick until it lands so a
busy or briefly-absent orchestrator is never interrupted and no hour is
skipped. Delete the now-redundant waker script; the prompt file is now
driven by the watchdog. Reboot-safe by inheritance (the watchdog is
started by cc-ci-loops.service).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Same fix as the loops: opencode run --attach exits after one turn;
plain opencode TUI stays alive in tmux. Send startup prompt via
ping_session (Enter) after 8s init wait. Bootstrap points to
JOURNAL.md rather than sending the full prompt inline.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1. API key: opencode doesn't support env: substitution in apiKey — write
actual key value to ~/.config/opencode/opencode.jsonc at setup time
(file is not committed to git; key sourced from .testenv).
2. Permission system: add permission:"allow" to opencode config (equivalent
to --dangerously-skip-permissions) to avoid interactive prompts.
3. Submit key: opencode TUI uses Enter (return) to submit; Ctrl+S not
needed. ping_session already uses Enter — keep as is.
4. Startup timing: bump opencode TUI init wait from 4s to 8s so the TUI
is fully connected to the server before bootstrap is sent.
5. Backend persistence: LOOP_BACKEND/LOOP_MODEL written to .loop-backend /
.loop-model so the watchdog uses them when restarting dead sessions.
All tested: both builder and adversary sessions alive, deepseek-v4-pro
processing kickoffs via tinfoil inference.tinfoil.sh, no API/permission
errors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three fixes discovered during first live run:
- inference host is inference.tinfoil.sh not api.tinfoil.sh (control plane
only serves /v1/models, not /v1/chat/completions)
- opencode run exits after one turn; switch to opencode attach for the
persistent TUI, then ping_session sends the kickoff prompt
- NO_COLOR=1 suppresses the first-run interactive theme picker
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Bash scripts are now one-liner wrappers: exec python3 <script>.py "$@"
All logic lives in the Python scripts (pure stdlib, no deps).
launch.py — loops + watchdog:
Full port of launch.sh: phase sequencing, start/stop/status/logs/watchdog,
handoff signalling, stall detection, heal_session, heal_orchestrator.
Cleaner structure: config block → helpers → phase/kickoff/agent/healing/
handoff/watchdog/main. LOOP_BACKEND + LOOP_MODEL switches throughout.
launch-orchestrator.py — orchestrator session:
claude path: --resume <id> preserved (conversation survives reboots).
opencode path: run --attach --title (no --resume; STARTUP_PROMPT orients
the new session; reads JOURNAL.md for context).
STARTUP_PROMPT updated to reference JOURNAL.md on startup.
launch-upgrader.py — one-shot upgrade job:
LOOP_BACKEND / LOOP_MODEL take precedence over UPGRADER_BACKEND / UPGRADER_MODEL.
Both claude and opencode paths supported.
cc-ci-plan/JOURNAL.md — new orchestrator handoff file:
Persistent across conversation resets. Documents the handoff format and
carries the current session's summary: migration complete, phase 5 in
progress (V3/V7 PASS), phase 4 deferred, open items for next session.
AGENTS.md: step 1 on startup = read JOURNAL.md; step 5 = append on handoff.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
configuration.nix:
- systemd.services.opencode-web: one shared opencode server on 127.0.0.1:4096,
EnvironmentFile=/srv/cc-ci/.testenv (TINFOIL_API_KEY), ExecStartPre clears
stale /tmp/opencode so restarts never fail on the EEXIST race.
- services.nginx: reverse-proxy oc.commoninternet.net → localhost:4096,
bound to tailscale IP 100.84.190.30 (tailnet-only, plain HTTP).
DNS: A record oc.commoninternet.net → 100.84.190.30 (operator step).
launch.sh + launch-upgrader.sh:
- Drop per-session ports / OPENCODE_HOST; add OPENCODE_SERVER=http://127.0.0.1:4096.
- opencode backend: agents use `opencode run --attach $OPENCODE_SERVER --title $session`
so each shows up as a named session in the web UI.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When multiple commit statuses exist (e.g. an Adversary probe + the real run),
the first status in the array may not be the cc-ci run. Filter by context
'cc-ci/testme' to get the correct Drone build URL.
Adds UPGRADER_MODEL env var (default: sonnet) passed as --model to the
claude invocation. The cron runs the upgrader on Sonnet so it doesn't
consume Opus weekly credits. Override with UPGRADER_MODEL=opus if needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Now the workspace is staged on the Hetzner cpx22 (server 134487234, public
91.98.47.73, tailnet cc-ci-orchestrator-1 @ 100.84.190.30):
- configuration.nix: enable cc-ci-loops.service (wantedBy multi-user.target) so the
loops + watchdog auto-resume on boot; wire reboot-log.sh as ExecStartPre so reboots
auto-log to REBOOTS.md (boot_id-gated).
- plan-orchestrator-hetzner-migration.md: full migration record.
- REBOOTS.md / AGENTS.md: point the orchestrator host at Hetzner; first auto-logged
reboot line.
- launch-orchestrator.sh: default session id -> the Hetzner orchestrator session.
- flake.lock: pin inputs.
Verified: nixos-rebuild switch applied; systemctl is-enabled cc-ci-loops.service =
enabled; ExecStartPre logged this boot to REBOOTS.md; loops healthy on phase 2.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
nixos-rebuild removed the infect-provisioned authorized_keys — declare it
explicitly so rebuilds don't lock out root access.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds terraform/ to provision a Hetzner cpx11 (2 vCPU / 2 GB dedicated AMD / 40 GB NVMe)
for the loops runtime, and a flake + NixOS host config to converge it — replacing the slow
b1 Incus VM. Mirrors the cc-ci server terraform (same nixos-infect pin, same pattern).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The orchestrator Pi is retired (2026-05-31). All agents now run on the
cc-ci-orchestrator VM (NixOS, loops user, /srv/cc-ci). The VM is a
direct tailnet peer to cc-ci — no SOCKS proxy, no userspace tailscaled,
no ProxyCommand. Updated across all affected files:
AGENTS.md
- Remove Pi from reboot description; migration complete (not "parked")
- cc-ci access: direct ssh, not via proxy
kickoff.md
- Prerequisites: direct tailnet peer, not proxy
- Host deps: NixOS (not apt)
- Fallback/Incus: b1 reachable directly, no --proxy curl flag
plan.md §1 + §1.5
- §1 bootstrap: direct SSH, check tailscale status (not restart proxy)
- §1.5 intro: "VM" not "sandbox host"; no proxy
- Credentials table: remove TS_AUTH_KEY row; update cc-ci SSH row
- Replace "Tailscale connection (proxy)" subsection with direct-peer description
plan-orchestrator-migration.md
- Mark COMPLETE (2026-05-31); historical record only
plan-phase1c-full-reproducibility.md
- Incus access: direct, not via SOCKS proxy
prompts/builder.md + prompts/adversary.md
- cc-ci access language only: direct ssh, no proxy restart instructions
- adversary: *.ci.commoninternet.net via plain curl, no proxy flag
REBOOTS.md
- Retitle for VM; note Pi retired; Pi entries marked historical
systemd/cc-ci-loops.service
- User/Group/HOME/PATH: notplants → loops
- Remove cc-ci-tailscaled.service dependency (no proxy on VM)
- Add note about nix/configuration.nix as the authoritative VM declaration
test-e2e-testme-acceptance.md
- tailscale status: no --socket flag
- ssh to throwaway: no ProxyCommand
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Gitea repos renamed:
cc-ci-autonomous-orchestrator → cc-ci-orchestrator
cc-ci-orchestrator → archived-cc-ci-orchestrator
Updated in this workspace:
- README.md, AGENTS.md: repo title
- cc-ci-plan/plan-orchestrator-migration.md: cc-ci-autonomous-orchestrator refs
- cc-ci-plan/plan-repo-consolidation.md: marked complete + Pi remote-update notice
- cc-ci-plan/launch-orchestrator.sh, launch.sh: session naming comment cleanup
NOTE: Pi clone still has the old origin URL. On the Pi, run:
git remote set-url origin https://git.autonomic.zone/recipe-maintainers/cc-ci-orchestrator.git
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Merging recipe-maintainers/cc-ci-orchestrator (the VM NixOS config repo)
into this repo as nix/ — the next step toward consolidating the two
orchestrator repos into a single cc-ci-orchestrator.
The source repo will be renamed to archived-cc-ci-orchestrator on Gitea.
This repo will be renamed cc-ci-orchestrator.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The symlink pointed to /srv/recipe-maintainer which does not exist on
the VM, leaving the parity corpus empty and blocking Phase-2 porting
of not-yet-migrated recipes.
Submodule URL: https://git.autonomic.zone/notplants/recipe-maintainer
(notplants/recipe-maintainer on git.autonomic.zone)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
A general-purpose Claude session sharing the orchestrator's workspace + access,
under remote-control (cc-ci-assistant), NOT on a loop. Sits idle until the
orchestrator/operator hands it a plan/task, does it, reports, waits. Modelled on
launch-orchestrator.sh: persistent pinned session-id (resume across relaunch),
root-aware --dangerously-skip-permissions handling, start/fresh/status/attach/stop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator: use a single uniform filename `compose.ccci.yml` per recipe (one file
holding all cc-ci-side deploy tweaks) rather than per-purpose suffixes like
compose.ccci-health.yml. Updated §9 + plan-ccci-compose-overlay-policy.md; added
a DoD item to rename tests/{ghost,discourse}/compose.ccci-health.yml ->
compose.ccci.yml and update their install_steps.sh cp target + recipe_meta
COMPOSE_FILE.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The prior commit only captured the file deletion (git add aborted on the
already-removed pathspec). This adds the actual content: the reworked §9
guardrail (justified ccci overlays OK; abra can't env start_period; always test
upgrade-to-latest, from-version custom tests skippable) and the new policy doc.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator correction (builder was right): abra does NOT support an env value for
healthcheck start_period, so the earlier "parameterize via APP_START_PERIOD env
PR" approach is impossible — a ccci compose overlay is the right tool there.
- plan.md §9: replace the "don't fork compose / use env PR" guardrail with
"avoid where possible + justify each + prefer upstream PR, BUT a uniform
optional compose.ccci-*.yml overlay is an acceptable fallback" (esp. for
abra-unparameterizable values like start_period). Add the upgrade-tier rule:
ALWAYS test the upgrade to latest; a from-version's custom tests may be
skipped if it can't fully run, but never drop upgrade-to-latest.
- replace plan-prefer-env-over-compose-overlay.md with
plan-ccci-compose-overlay-policy.md: ghost/discourse start_period overlays
STAY (justified); discourse image re-pin STAYS (keeps the upgrade-to-latest
testable; 0.7.0 custom tests may be skipped); mumble old-base host-ports copy
DROPPED (skip 0.2.0 voice tests, still upgrade to latest + test there). Each
surviving overlay must be minimal + header-justified + Adversary-confirmed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase A done before the Pi's reboot #3 (commit was interrupted): the loops VM
cc-ci-orchestrator is on the tailnet (100.116.55.106) and ssh-able; TS-key
finding recorded (VM-creator .test.env key revoked; cc-ci .testenv key valid +
persisted). REBOOTS.md carries the auto-logged 2026-05-30 17:03 reboot
(cc-ci-loops.service auto-recovered the loops at phase 2; swapfile persisted).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
