Gitea repos renamed:
cc-ci-autonomous-orchestrator → cc-ci-orchestrator
cc-ci-orchestrator → archived-cc-ci-orchestrator
Updated in this workspace:
- README.md, AGENTS.md: repo title
- cc-ci-plan/plan-orchestrator-migration.md: cc-ci-autonomous-orchestrator refs
- cc-ci-plan/plan-repo-consolidation.md: marked complete + Pi remote-update notice
- cc-ci-plan/launch-orchestrator.sh, launch.sh: session naming comment cleanup
NOTE: Pi clone still has the old origin URL. On the Pi, run:
git remote set-url origin https://git.autonomic.zone/recipe-maintainers/cc-ci-orchestrator.git
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Merging recipe-maintainers/cc-ci-orchestrator (the VM NixOS config repo)
into this repo as nix/ — the next step toward consolidating the two
orchestrator repos into a single cc-ci-orchestrator.
The source repo will be renamed to archived-cc-ci-orchestrator on Gitea.
This repo will be renamed cc-ci-orchestrator.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The symlink pointed to /srv/recipe-maintainer which does not exist on
the VM, leaving the parity corpus empty and blocking Phase-2 porting
of not-yet-migrated recipes.
Submodule URL: https://git.autonomic.zone/notplants/recipe-maintainer
(notplants/recipe-maintainer on git.autonomic.zone)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
A general-purpose Claude session sharing the orchestrator's workspace + access,
under remote-control (cc-ci-assistant), NOT on a loop. Sits idle until the
orchestrator/operator hands it a plan/task, does it, reports, waits. Modelled on
launch-orchestrator.sh: persistent pinned session-id (resume across relaunch),
root-aware --dangerously-skip-permissions handling, start/fresh/status/attach/stop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator: use a single uniform filename `compose.ccci.yml` per recipe (one file
holding all cc-ci-side deploy tweaks) rather than per-purpose suffixes like
compose.ccci-health.yml. Updated §9 + plan-ccci-compose-overlay-policy.md; added
a DoD item to rename tests/{ghost,discourse}/compose.ccci-health.yml ->
compose.ccci.yml and update their install_steps.sh cp target + recipe_meta
COMPOSE_FILE.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The prior commit only captured the file deletion (git add aborted on the
already-removed pathspec). This adds the actual content: the reworked §9
guardrail (justified ccci overlays OK; abra can't env start_period; always test
upgrade-to-latest, from-version custom tests skippable) and the new policy doc.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator correction (builder was right): abra does NOT support an env value for
healthcheck start_period, so the earlier "parameterize via APP_START_PERIOD env
PR" approach is impossible — a ccci compose overlay is the right tool there.
- plan.md §9: replace the "don't fork compose / use env PR" guardrail with
"avoid where possible + justify each + prefer upstream PR, BUT a uniform
optional compose.ccci-*.yml overlay is an acceptable fallback" (esp. for
abra-unparameterizable values like start_period). Add the upgrade-tier rule:
ALWAYS test the upgrade to latest; a from-version's custom tests may be
skipped if it can't fully run, but never drop upgrade-to-latest.
- replace plan-prefer-env-over-compose-overlay.md with
plan-ccci-compose-overlay-policy.md: ghost/discourse start_period overlays
STAY (justified); discourse image re-pin STAYS (keeps the upgrade-to-latest
testable; 0.7.0 custom tests may be skipped); mumble old-base host-ports copy
DROPPED (skip 0.2.0 voice tests, still upgrade to latest + test there). Each
surviving overlay must be minimal + header-justified + Adversary-confirmed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase A done before the Pi's reboot #3 (commit was interrupted): the loops VM
cc-ci-orchestrator is on the tailnet (100.116.55.106) and ssh-able; TS-key
finding recorded (VM-creator .test.env key revoked; cc-ci .testenv key valid +
persisted). REBOOTS.md carries the auto-logged 2026-05-30 17:03 reboot
(cc-ci-loops.service auto-recovered the loops at phase 2; swapfile persisted).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator go-ahead (Pi is OOM-thrashing/slow). Created the dedicated loops VM
cc-ci-orchestrator (2GB RAM / 2 vCPU / 30GB, incus-base-vm NixOS) on b1 via the
Incus API, mirroring the known-good cc-nix-test spec; started it — cloud-init is
running nixos-rebuild boot + reboot + tailnet join. Status flipped DRAFT->IN
PROGRESS with the remaining Phase-A items noted (add cc-ci-root key via incus
exec, confirm tailnet+ssh, write the reproducible TF project).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tmux `send-keys -l <long msg>` often leaves the text UNSENT in the input box (the
immediate Enter is swallowed while the TUI ingests the paste). Both now type the
message then retry Enter/C-m until the leading text is no longer in the input box
(= submitted) or a bounded loop gives up.
- msg-loop.sh: standalone reliable messenger for orchestrator use.
- launch.sh ping_session: same retry-submit (loads on next watchdog restart).
Live-tested: delivered first try.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator (2026-05-30): a cc-ci-authored compose overlay risks silent drift from
the recipe users actually run — avoid it wherever possible.
- plan.md §9 guardrail: when a recipe needs a cc-ci-env-tuned value (e.g. a longer
healthcheck start_period for the slow single node), the preferred fix is an
UPSTREAM recipe PR exposing it as an env var (e.g. APP_START_PERIOD) with the
current value as the default in env.sample — CI sets the env, no new compose.
For making the upgrade tier work from an older base version, prefer DECLARING
that version not-testable under this CI env over crafting a custom compose.
Overlay = last resort, Adversary-confirmed non-drifting + paired with the env PR.
- plan-prefer-env-over-compose-overlay.md: migrates the existing debt —
ghost/discourse compose.ccci-health.yml start_period -> APP_START_PERIOD recipe
PRs (default=current) then drop the overlays; discourse image re-pin + mumble
old-base host-ports copy -> declare those old versions untestable instead of
forking compose. No test weakened; untestable-version is an honest outcome.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
For a long deploy/convergence, arm a Monitor that polls the node every ~30s and
wakes on convergence OR failure, with a longer fallback heartbeat (ScheduleWakeup)
as a backstop. Proceeds the instant it converges (no over-waiting), surfaces
failures promptly, and the heartbeat bounds the wait. Size the timeout sanely
(longer if justified, never absurd like the ~40-min ghost case). Credit: builder.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator (2026-05-30): the real deploy-speed bottleneck was hardware (cc-ci VM
was 2 vCPU on a 4-core host + disk-I/O-bound; RAM fine), now fixed directly
(bumped to 4 vCPU, made cc-nix-test the only running VM on b1). The 2b software
micro-optimizations are judged unlikely to help, so:
- IDEAS.md: parked the whole empirical-perf program (instrumentation, baseline,
attribution) + the optimization menu (image cache/prepull, readiness tuning,
warm-SSO start/stop, runner caching, concurrency sizing, resources, secret
overhead) under "Phase-2b empirical performance work", revisit only if
measurement later proves a specific software bottleneck.
- plan-phase2b: reduced to ONE goal — confirm (and fix if needed) that the
per-recipe test sequence already uses the minimum deploys (1 base shared by
install+functional+backup/restore, +1 for the upgrade tier, +1 per dep),
enforced by the existing DG4.1 deploy-count check, WITHOUT weakening any test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator: when the final phase completes, install the weekly cron anchored to
actual completion — first run ~1h after the build finishes, weekly from then on
(supersedes the fixed "Sat 03:00 UTC" placeholder).
- plan-phase5 §4: orchestrator computes T0=now+1h, installs a weekly job at T0's
DOW+HH:MM running launch-upgrader.sh start; cron env needs claude on PATH +
tmux + claude.ai login (mirror cc-ci-loops.service). VERIFY the first kickoff:
cheap --dry-run pre-check, then confirm the real T0 fire launched the
cc-ci-upgrader agent (status RUNNING, ran /upgrade-all, summary produced);
record schedule + verified kickoff in DECISIONS.md.
- upgrade-all skill Cron section + cron memory updated to the completion-anchored
schedule + first-kickoff verification.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The weekly upgrade run now executes inside a dedicated, remote-control agent
(cc-ci-upgrader) — viewable/steerable at claude.ai/code like the Builder — rather
than buried in headless cron output.
- launch-upgrader.sh: spins up the cc-ci-upgrader tmux session under
--remote-control with a kickoff that runs /upgrade-all (DEFAULT mode) to
completion. On finish the agent STOPS and stays idle (does NOT self-terminate)
so the run + summary stay reviewable in the web UI. `start` = use-or-create:
leaves an in-flight (busy) run alone, else clears a finished/idle/wedged
session and runs fresh; `fresh` always restarts. UPGRADER_ARGS passes flags
(e.g. --dry-run); never --with-tests.
- launch.sh: orchestrator_alive() now also skips the cc-ci-upgrader
remote-control name, so the upgrader job isn't mistaken for the orchestrator.
- upgrade-all skill: documents it runs as the cc-ci-upgrader agent; the weekly
cron invokes `launch-upgrader.sh start` (not /upgrade-all inline).
- Phase 5: V8a verifies the agent lifecycle (launch → run to completion → stay
idle/viewable → next start clears it); V9 stops the verification session.
- cron memory: weekly task = launch-upgrader.sh start at 0 3 * * 6 UTC.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Appended as the LAST phase in the launcher sequence (… 3 4 5). It can only run
once cc-ci is fully built — the !testme-on-recipe-PR flow depends on Phase 3
(results UX) surfacing the run result back on the PR for testme-on-pr.sh to read.
DoD (Adversary cold-verifies): !testme on a recipe PR is the real gate + results
land in the PR (V1); testme-on-pr.sh reads GREEN/RED/PENDING + BUILD url, POST=0
polls without re-triggering (V2); /recipe-upgrade default end-to-end green on a
sandbox recipe, nothing merged (V3); the ≤3 !testme regression loop (V4); stale
test DEFAULT = comment-only, no test edit (V5); --with-tests opens+verifies a
cc-ci test PR, paired (V6); mirror reconcile closes merged/superseded PRs and
main==upstream (V7); /upgrade-all default dry-run + small live run never edits
tests (V8); all verification PRs closed + deploys torn down (V9). Use a sandbox
recipe; never merge; never weaken tests. Watchdog reloaded (seq …3 4 5).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per operator:
- Verify via `!testme` posted ON the recipe PR (the real CI path) so results are
viewable in the PR; iterate up to 3 !testme runs (fix a real regression + re-test).
New helper testme-on-pr.sh posts !testme and polls the PR head commit status
for the verdict (POST=0 to keep polling without re-triggering).
- Test updates are now OPT-IN via `--with-tests`. DEFAULT: recipe PR only using
existing tests; if a test fails and is genuinely stale, leave an explanatory
COMMENT on the PR (upgrade looks correct; re-run --with-tests to update tests)
and do NOT touch any test. --with-tests keeps the verified cc-ci test-update PR
path (verified via the branch-checkout harness run, since !testme uses prod tests).
- upgrade-all (weekly cron) calls the DEFAULT — never auto-edits tests unattended;
surfaces "tests look stale" PRs in the summary for the operator to opt in per-recipe.
- New RESULT: SUCCESS-PENDING-TESTS for the recipe-green-but-test-stale default case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of the adversary "overrun": stall_check rebooted the instant
now >= WAITING-UNTIL (zero grace), but the loop's own ScheduleWakeup fires AT
that stated time — and the runtime scheduled it ~40s later than the marker
(date-vs-scheduler skew). So the watchdog pre-empted a HEALTHY self-wake by
~37s; the loop wasn't wedged, it was killed just before it woke. That was the
single false reboot at 18:55Z.
Fix: split the two cases cleanly.
- Marker present: reboot only when now > WAITING-UNTIL + STALL_GRACE (180s) —
covers wake+start latency + marker/scheduler skew, so the watchdog only fires
if the self-wake GENUINELY failed.
- No marker: unchanged — reboot when idle >= STALL_IDLE (300s).
Verified post-fix: adversary self-woke on time and re-paced (WAITING-UNTIL
19:19:30Z); no new stall reboots.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The builder wedged at the context limit (garbled output) — alive but matching
none of heal_session's signals (dead/FATAL/limit), so the watchdog left it
stuck. Fix: loops now declare every wait, and the watchdog reboots a wait that
never resumes.
- plan.md §7 + both prompts: cap every wait at 10 min (chunk longer waits);
before going idle, the loop's FINAL line must be `WAITING-UNTIL: <ISO8601 UTC>`
(the resume time, matching its ScheduleWakeup); run /compact proactively at
~80% context to avoid wedging near the limit.
- launch.sh: new stall_check (runs every 30s signal tick) — reboots a loop idle
>= STALL_IDLE (300s) when it has NO current WAITING-UNTIL marker as its last
message OR is past the time the marker named; a healthy paced wait (marker
present, before its time) is left alone. Complements heal_session's
dead/FATAL/limit cases. Reboot is safe — loops re-orient from git + STATUS.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per operator: an open mirror PR must mean "genuinely still open against true
current upstream main". On every run the recipe-upgrade flow now:
- force-syncs the recipe-maintainers/<recipe> mirror `main` to be IDENTICAL to
upstream main (origin/main of the abra checkout = coopcloud);
- closes any open mirror PR whose changes are already in upstream main (merged
upstream, no-op merge detected via `git merge-tree` vs main's tree) — even
when the recipe is up to date (new `--reconcile-only` mode, run in step 1);
- when opening a new upgrade PR, closes any other still-open PR for that recipe
(superseded) and opens the new one IN ITS PLACE; same-version re-runs just
update the existing same-branch PR.
open-recipe-pr.sh gains the --reconcile-only mode + the close logic (with an
auto-close comment naming the reason). upgrade-all reconciles every candidate's
mirror during the survey so merged PRs are closed fleet-wide. Still never merges.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator: don't run the weekly upgrade-all while the build loops are still
constructing cc-ci (shared-host contention). Activate the Sat 03:00 UTC
(0 3 * * 6) cron only once the build is complete; on-demand until then.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per-recipe and fleet-wide upgrade skills modelled on recipe-maintainer's
recipe-upgrade-full / recipe-upgrade-cron-all, but gated by the cc-ci CI server
and inheriting ci-test-review's create+verify+never-merge discipline.
- recipe-upgrade/: plan (release notes, breaking changes) -> implement (abra
recipe upgrade + version bump + config, lint) -> open the recipe PR -> VERIFY
green on cc-ci (full suite cold against the PR head via verify-pr.sh). If the
upgrade is correct but a cc-ci TEST went stale, also update the test, verify
it, and open a second PR to recipe-maintainers/cc-ci. Never merges; never
weakens a test; prefers a recipe-only PR. Emits a parseable RESULT line.
+ open-recipe-pr.sh: adapted recipe-create-pr; runs on cc-ci (has the recipe
checkout + bot token), creds passed from the orchestrator .testenv;
force-syncs the mirror main so the PR diff is exactly the upgrade.
- upgrade-all/: weekly fan-out — enumerate enrolled recipes, survey upgrades,
run /recipe-upgrade per upgradeable recipe via subagent (sequential default,
--parallel / --dry-run), collect into one PR-list summary. Coordination +
single-writer + shared-Swarm-teardown guardrails; built for a weekly cron.
- ci-test-review/verify-pr.sh: pass SRC (recipe-maintainers/<recipe>) alongside
REF so the harness clones the mirror PR head correctly (its real contract).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The earlier `git add` included an already-`git rm`'d pathspec, so it errored and
staged nothing — launch.sh (3r removal) and .gitignore (track .claude/skills/)
were left uncommitted while the skill files went in via a separate -f add.
Runtime was already correct (watchdog reads the working-tree launch.sh); this
just syncs git HEAD to the working tree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per operator: the skill should not just propose, it should CREATE the fix PR
(recipe repo or cc-ci repo) and VERIFY it green on its own CI server — but not
merge. It drives cc-ci like the loops do.
- SKILL.md: diagnose+classify (recipe vs CI-server) -> author the fix + open a
PR (recipe-create-pr for recipe PRs; Gitea API for cc-ci PRs, dedicated branch
in a separate clone, single-writer safe) -> VERIFY on cc-ci (full suite cold
against the PR head = the !testme dogfood path) -> report a verified,
ready-to-merge PR. Never merges; never weakens a test; flake != bug. General
bar = one cold green; repeated-green (REPEAT=3) only for a known-flaky recipe.
Adds coordination/single-writer guardrails (shared Swarm is stateful; tear
down deploys; never push main or touch the loops' clones).
- verify-pr.sh: deterministic recipe-PR gate — RECIPE + REF -> cold full suite
on cc-ci, green iff every repeat exits 0. CI-server-PR verification stays
bespoke (branch checkout + rebuild + regression sample) per SKILL.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The on-demand AI review layer is now an orchestration-repo skill built directly
by the orchestrator, NOT a loops phase in the cc-ci product repo:
- .claude/skills/ci-test-review/{SKILL.md,run-all-recipes.sh}: runs the real
cc-ci harness across all enrolled recipes (deterministic, AI-free execution),
then AI diagnoses each failure and classifies it as needing a recipe PR / a
CI-server PR / a stale-test update — or reports "ALL PASSED, recipes + tests
up to date". Proposes PRs; never decides pass/fail; never auto-merges.
- .gitignore: track .claude/skills/ (shareable) while still ignoring local
claude session state (locks, history) under .claude/.
- launch.sh: remove Phase 3r from PHASES_SPEC; loops sequence back to
1c 1b 1d 1e 2w 2pc 2 2b 3 4. Deleted plan-phase3r (superseded by the skill).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Deterministic CI stays the primary, AI-free path. Adds a separate on-demand skill (ships in the
cc-ci repo .claude/skills/ci-test-review/) that runs the full suite across all recipes and, per
failure, AI-diagnoses + classifies: recipe PR (+ proposed change) vs CI-server PR vs stale-test;
or 'all passed, recipes+tests up to date' (incl. a latest-version freshness check). Proposes, never
auto-merges (operator-merge rule). Slotted 3 -> 3r -> 4. AI only diagnoses; execution stays
deterministic.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve a recipe's images (docker compose config --images) and docker pull them (skip-if-present for
pinned tags) at the start of the recipe sequence + before the upgrade-new-version deploy, then the
normal abra deploy. Separates pull from converge (clear pull failures vs murky convergence timeouts),
speeds convergence (fits abra-native window). No layer re-download on warm cache; nightly all-recipes
run warms everything. Complements (not replaces) the recipe healthcheck for slow-init convergence.
Near-term Phase-2 harness unit; real abra deploy unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator (2026-05-29): dedicated sub-plan for the upstream recipe PR. Fixes collabora WOPI
healthcheck/start_period (keystone — fixes F2-12 at the source so cc-ci can return to abra-native
convergence + drop the -c/READY_PROBE backstop), backend WOPI retry, gunicorn-perms race, lazy OIDC.
PR is 'working' only when cc-ci runs the full suite incl. upgrade tier green + Adversary cold-verify,
then operator merges. Broken out from plan-lasuite-drive-oidc-robustness.md Part B (now points here).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cheap guard against the deploy/git divergence: a fix built locally but uncommitted/un-pushed is a
guaranteed Adversary cold-build mismatch. Added to the builder prompt claim discipline + plan.md
§6.1. (Lighter than binding the deploy to a git rev — iteration speed + the Adversary's
cold-from-git verify is the real safety net.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator (2026-05-29): on one host Docker's local image store already IS the cache; the churn was
over-pruning, not a missing cache. So 2pc = conservative prune policy + confirm local-store retention
+ daemon auth (PC1-3). Registry pull-through cache deferred to IDEAS with a concrete revisit
condition (multi-node, or measured cold-deploy bottleneck on recreate-surviving storage).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deferred lasuite-drive [~] (Q3.2). Two parts: (A) cc-ci wires OIDC at INSTALL against the live-warm
keycloak (WC1) so there's no flaky mid-run 12-service --chaos reconverge — using REAL abra commands
only (no docker service update bypass; operator decision); (B) a lasuite-drive recipe PR fixing the
root cause (collabora WOPI healthcheck-gating + gunicorn-perms race + lazy/retrying OIDC discovery).
Operator rule: a recipe change is "working" only once cc-ci runs the full suite on the PR and it's
repeatedly green (Adversary cold-verified) — then the operator merges. A+B reinforce (lazy OIDC makes
install-time wiring safe for the generic-first invariant). Ground the fix in captured failure logs first.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the brittle markdown prose-match ("Gate: … CLAIMED, awaiting Adversary") with detection of
the loops' conventional commit prefixes on origin/main: a new `claim(...)` commit pings the
Adversary; a new `review(...)` commit pings the Builder. Edge-triggered on the origin/main SHA
(append-only — no force-push), no file parsing, can't mis-route. The loops already use these
prefixes consistently; codified as a load-bearing contract in plan.md §6.1 + both prompts so it
stays reliable. INBOX detection unchanged (pushed-state, file-routed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The handoff pings fired on the writer's LOCAL working-tree write (before push), so the receiver
pulled a stale origin/main, saw "no formal gate", and a clarifying inbox round-trip ensued
(several minutes + wasted turns per handoff). And the gate-id parser read "WC1" as "C1" and could
fire on prose mentions.
Fix (1): handoff_check now `git fetch`es and reads origin/main (what the receiver will pull), via
_wd_fetch_origin + _wd_show_pushed, for STATUS / REVIEW / both INBOXes — a ping only fires once the
claim/verdict is actually pushed, so the receiver's pull always sees it. Eliminates the stale-pull
"premature" dance.
Fix (2): gate-claim detection matches ONLY a formal line (Gate: <id> … CLAIMED, awaiting Adversary)
and edge-triggers on a genuinely-new such line compared whole — no firing on historical
"CLAIMED detail" lines or prose; gate-id is a best-effort label only.
Loops' clones have a credential helper (reads .testenv) so the watchdog's fetch works
non-interactively. Verified.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator (2026-05-29): a passing health check does NOT prove a required manual migration ran, so
auto-update needs a PRE-deploy gate in addition to the post-deploy health rollback. Reconciler
auto-applies only non-major (patch/minor) upgrades with no manual-migration release notes; a MAJOR
recipe-version bump (or release notes flagging a manual migration) is held on the current version
with a PushNotification carrying the release notes (operator upgrades manually). Leans on abra's
own major-bump caution + recipe releaseNotes/. Updated WC1.2/WC6/principles/decisions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator decision (2026-05-28): traefik + keycloak stay UNPINNED (fetch latest + chaos deploy);
a nightly `nixos-rebuild switch` rolls them to latest, then the full-cold sweep runs. The nix
closure stays byte-identical (recipe fetched at runtime, not in the store) so D8 holds.
Health-gated rollback is built INTO the reconciler (not nix-generation rollback, since the swarm
app isn't in the generation): record last-good -> deploy latest -> health-check -> commit or
roll back + PushNotification. Stateful apps (keycloak): snapshot the data volume before upgrade
(undeploy->snapshot->deploy-latest) and restore it on rollback, reusing the WC3 snapshot helper;
traefik = version rollback only. Added WC1.1 + updated WC1/WC6/milestones/guardrails/decisions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator-directed: pause Phase 2, build the warm-data + --quick system, then resume Phase 2.
- live-warm keycloak (SSO dep, realm-per-run), data-warm canonicals (undeploy keeps volume),
cold = authoritative default. --quick reattaches the canonical, upgrades to PR head, asserts,
and rolls back to the last-known-good snapshot on failure (never loses working data).
- known-good = raw volume copy taken while undeployed (consistent), one per app, advanced ONLY
by green cold runs; a nightly full-cold sweep refreshes canonicals + is a daily regression run.
- launch.sh: insert 2w at the current index (Phase 2 -> resumes after 2w DONE); seq is now
1c 1b 1d 1e 2w 2 2b 3 4.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- heal_session: detect the unrecoverable "thinking/redacted_thinking blocks cannot
be modified" 400 (recurs every turn, session stays alive so the dead-check misses
it) and kill+restart the loop fresh (re-orients from repo). Consolidates the
dead/fatal/limit handling for builder+adversary.
- heal_orchestrator: keep the orchestrator alive too, conflict-safe. Restarts via
launch-orchestrator.sh ONLY when no orchestrator is alive anywhere — liveness
detects both a managed cc-ci-orchestrator tmux session AND a hand-launched
terminal session (any non-loop claude), so it never double-resumes the
conversation (the likely cause of the thinking-block crashes). Kill+restart if
the managed session is wedged on the FATAL error. Toggle: WATCH_ORCHESTRATOR=0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
handoff_check's now="$(grep CLAIMED.*awaiting ... )" returned non-zero when a phase's STATUS
has no claimed-awaiting lines yet (normal early in a phase); under set -euo pipefail that
assignment exited the whole watchdog. Append `|| true` to the now= and cur= command
substitutions. Verified: watchdog survives the handoff tick on a freshly-created STATUS-1c.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Strengthen C4/W5: the genuine reproducibility proof is a clean-room repeat — the Adversary
destroys any existing throwaway VM, creates a brand-new blank VM, and runs the entire install
from scratch per docs/install.md so nothing from the Builder's setup attempt can mask a gap.
Cold, with logged evidence (VM id, exact install commands, convergence + TLS-from-git-cert).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Make the launcher drive an ordered phase sequence (default 1c then 1b). Each phase has its own
plan + phase-namespaced loop-state files (STATUS-<id>.md/BACKLOG/REVIEW/JOURNAL); the watchdog
auto-transitions when the current phase's STATUS-<id>.md shows ## DONE, and STOPS after the last
phase (writes SEQUENCE-COMPLETE, exits) as a manual gate before Phase 2. start_agent injects a
phase preamble (source-of-truth = phase plan; phase-namespaced state) ahead of the base role
prompt. DONE detection reads the builder's local clone (reliable, no push-lag). Handoff signalling
+ resilience preserved and made phase-scoped (reset baseline on transition).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
