- heal_session: detect the unrecoverable "thinking/redacted_thinking blocks cannot
be modified" 400 (recurs every turn, session stays alive so the dead-check misses
it) and kill+restart the loop fresh (re-orients from repo). Consolidates the
dead/fatal/limit handling for builder+adversary.
- heal_orchestrator: keep the orchestrator alive too, conflict-safe. Restarts via
launch-orchestrator.sh ONLY when no orchestrator is alive anywhere — liveness
detects both a managed cc-ci-orchestrator tmux session AND a hand-launched
terminal session (any non-loop claude), so it never double-resumes the
conversation (the likely cause of the thinking-block crashes). Kill+restart if
the managed session is wedged on the FATAL error. Toggle: WATCH_ORCHESTRATOR=0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
handoff_check's now="$(grep CLAIMED.*awaiting ... )" returned non-zero when a phase's STATUS
has no claimed-awaiting lines yet (normal early in a phase); under set -euo pipefail that
assignment exited the whole watchdog. Append `|| true` to the now= and cur= command
substitutions. Verified: watchdog survives the handoff tick on a freshly-created STATUS-1c.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Strengthen C4/W5: the genuine reproducibility proof is a clean-room repeat — the Adversary
destroys any existing throwaway VM, creates a brand-new blank VM, and runs the entire install
from scratch per docs/install.md so nothing from the Builder's setup attempt can mask a gap.
Cold, with logged evidence (VM id, exact install commands, convergence + TLS-from-git-cert).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Make the launcher drive an ordered phase sequence (default 1c then 1b). Each phase has its own
plan + phase-namespaced loop-state files (STATUS-<id>.md/BACKLOG/REVIEW/JOURNAL); the watchdog
auto-transitions when the current phase's STATUS-<id>.md shows ## DONE, and STOPS after the last
phase (writes SEQUENCE-COMPLETE, exits) as a manual gate before Phase 2. start_agent injects a
phase preamble (source-of-truth = phase plan; phase-namespaced state) ahead of the base role
prompt. DONE detection reads the builder's local clone (reliable, no push-lag). Handoff signalling
+ resilience preserved and made phase-scoped (reset baseline on transition).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1c (full git reproducibility: cc-ci-secrets split, cert-in-sops, genuine D8 live rebuild)
now runs before 1b. This way 1b's review/lint and its final cold re-verification of all
D1-D10 cover the final refactored state (incl. the secrets split) and the genuine post-1c
D8 — rather than reviewing pre-refactor code and re-verifying a flawed D8. Updated status
lines in 1b/1c and the README ordering. Sequence: 1 -> 1c -> 1b -> 2 -> 2b -> 3.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per operator: the split boundary is secrecy, not modularity. Only the sops-encrypted secrets
(incl. the wildcard cert) move to a separate private repo `cc-ci-secrets` (extra access-control
layer), consumed by the base via a flake input. Instance non-secret vars (domain, gateway,
recipients) stay in the well-parameterized base cc-ci repo — another admin repoints by editing
params, no second config repo. Guardrail reworded: instance vars in base are fine; only plaintext
SECRETS must never leak into base/store. Updated model/C1/C2/W2/§6/§7 + README.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per operator: don't downsize cc-nix-test to 2GB. Instead raise the terraform-ci running-RAM
guideline to ~12GB (it's doc-only — the project has no enforced limits.memory; b1 is 16GB),
resize cc-nix-test 6->4GB, and create the throwaway VM at 4GB (4+4+lichen 4 = 12 <= 16).
Updated W1/W3/C6/§4 and the incus memory note.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
D8's throwaway-VM live rebuild was wrongly marked "infeasible by design" — the master
recovery age key defeats the sops-host-key reason, DNS/cert is a precondition not a
rebuild blocker, and Incus was available. Phase 1c (loop-driven): (A) make the VM fully
reproducible from git including ALL secrets — move the wildcard cert + every secret into
sops-in-git, split generic base repo from a private instance repo composed via a flake
input (the only out-of-band secret is the bootstrap age key); (B) actually perform +
cold-verify a blank-VM nixos-rebuild and rewrite D8 honestly. Resize cc-nix-test to 2GB
first to free b1 headroom for a sized throwaway VM; destroy it after; restore/promote
sizing. Gandi token stays out of repo/agent (only the cert artifact is committed). Linked README.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Adversary got a spurious "gate CLAIMED" ping: STATUS.md keeps historical
"Gate: Mn — CLAIMED, awaiting Adversary" lines after they PASS, and on watchdog restart the
first observation pinged on those already-passed lines. Now track the SET of gate ids on
CLAIMED-awaiting lines and ping only when an id NEWLY appears vs the prior observation, after a
silent baseline. A gate passing (line kept) or evidence edits don't re-ping; restart re-baselines
without pinging. Verified: watchdog restart no longer pings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug: start_watchdog used $0, which breaks when launch.sh is called by a relative path
(the watchdog tmux session cd's into PLAN_DIR, so a relative $0 no longer resolves —
"No such file or directory", watchdog dies instantly). Resolve BASH_SOURCE to an absolute
SELF once and use it for the watchdog self-invocation. Verified: watchdog now starts and
its handoff_check immediately pinged the Adversary about a standing CLAIMED gate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
launch.sh watchdog now runs a fast (~30s) handoff_check alongside the heavy (300s) restart/DONE
check: when the Builder writes a CLAIMED gate it pings the Adversary to verify now; when the
Adversary updates REVIEW.md it pings the Builder to proceed (edge-triggered, reads local clones).
So a pending handoff resolves in <~30s instead of a whole idle interval. Pacing revised: the
Adversary may idle freely when nothing's pending (no pointless re-verify/busy-poll) and is woken
by the watchdog; Builder waits on the ping + a fallback ~2-4m self-poll. kickoff documents the
new "handoff signalling" role.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause of "both waiting": parked-at-gate was lumped into the long idle sleep, so a
pending handoff sat while both loops slept on desynced timers. Fix: three cases — (1) in
flight → ~4m; (2) BLOCKED ON THE OTHER LOOP (Builder at CLAIMED gate / Adversary awaiting a
fix) → ~4m poll for the counterpart, never long-idle; (3) genuinely nothing pending → ~10-15m.
Adversary: a CLAIMED gate is immediate top-priority; otherwise run background probes, rarely
idle while Builder is active. Builder: keep an unblocked item in hand to rarely be fully gated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
§7 pacing + Builder/Adversary prompts: idle/parked sleep lowered to ~10–15 min so the next
unit of work (or a gate claim) is picked up without long gaps. Unchanged: ~4m polling while
a build/deploy is in flight; keep polling something clearly in-flight rather than treating
it as idle; don't spin on a minutes-long build. Adversary aligned to §7 for consistency.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
RL3 strengthened: after lint/review findings are responded to and fixed, the Adversary
independently re-verifies EVERY Phase-1 Definition-of-Done item (D1–D10) from a cold start
to the same bar as Phase 1's own DONE (fresh PASS + evidence in REVIEW.md), proving the
cleanup regressed nothing. 1b cannot be DONE until all D1–D10 are re-confirmed green
post-cleanup. Method/W2 updated to make the ordering explicit (tooling -> fixes -> re-verify).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Before scaling to many recipes: (1) deterministic style/hygiene via linters/formatters
(alejandra/statix/deadnix, ruff, shellcheck/shfmt) wired as a .drone.yml stage so commits
stay clean; (2) a white-box review checklist with teeth (real tests not health-only/skipped,
DRY harness, Nix-declared idempotent bring-up, no footguns/secrets-in-code, architecture
matches plan) — blocking fixed, advisory triaged. Bounded pass; never weaken a test for a
nit. Phase 2 now follows 1b. Linked in README.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 2b (after Phase 2, before Phase 3): instrument per-phase timings, baseline a
representative recipe set (cold vs warm), attribute where time goes (Pareto), then try
improvements as controlled before/after experiments and keep measured winners — image
pull cache/pre-pull, readiness-wait tuning, dedup deploy cycles, warm/shared infra
(isolation-proven), runner caching, concurrency sizing, vCPU. Speed never weakens tests
or isolation (Adversary re-measures + re-verifies). Phase 3 now follows 2b. Linked in README.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 3 (after Phase-2 DONE, manual transition): compute a per-run quality LEVEL, post an
image-forward Gitea PR comment in the YunoHost shape (marker + status/level badge + a
rendered summary card containing a real app screenshot, linking to the run), and polish the
overview dashboard to a ci-apps.yunohost.org look/feel with per-recipe level badges +
screenshots. Reuses the Phase-1 dashboard/bridge/Playwright; presentation never changes the
verdict; no secrets in any artifact; cosmetics never block the pipeline. Linked from README.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add §7.1 Adversary mandate: default assumption is everything meaningful is testable
(OIDC/SSO, federation, media, WOPI, WebRTC connectivity, backup data survival) — the job
is a good test, not declaring impossibility. Adversary reads test bodies, rejects
skip/xfail/mock/health-only/empty-assertion tests and bogus parity renames, re-runs cold.
"Untestable" is a rare exception needing a true environment blocker + maximal subset +
Adversary sign-off; "needs browser/SSO/another app" is not valid. Tighten P7 and §8 to match.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 2 fills the CI machine with good tests for every maintained Co-op Cloud app,
using references/recipe-maintainer as the corpus: port a comparable cc-ci test for
EACH existing recipe-maintainer test (parity, tracked in PARITY.md) + >=2 new
recipe-specific functional tests per recipe, plus real backup data-integrity and SSO
dependency handling. Reuses the Phase-1 harness/stages/trigger/resource-caps; adds
test content + small shared-harness ports from helpers.py. Linked from the package README.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Don't overload the single node: cap concurrent test builds at a configurable MAX_TESTS
(= DRONE_RUNNER_CAPACITY); Drone natively queues excess builds and times out hung ones,
freeing slots — no custom queue. Each run deploys one app then undeploys; the run-start
janitor is the backstop for timed-out/killed builds. At most MAX_TESTS apps live at once.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First item: later, for environments where the CI server has repo-admin, consider an
opt-in (off-by-default) feature to auto-register + idempotently reconcile the issue_comment
webhook — preserving the read-only/polling default. Parked, out of current scope.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Finalize trigger model per operator: polling is the primary trigger (outbound, read-only,
no admin); the server never self-registers webhooks (that needs admin) — webhook is an
optional push optimization an admin registers manually, documented in enroll-recipe.md.
Commenter auth via org-membership endpoint (read-level), not the admin-only permission
endpoint. Bot's required privilege is read + comment + org-membership, never repo-admin.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The repo's explicit collaborator list is empty — bot and maintainers (trav/notplants)
all access via org ownership, so the collaborators check 404s for everyone. Authorize via
GET /collaborators/{user}/permission requiring owner/admin/write (matches the builder's fix).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Record the trigger design: webhook (default/primary, confirmed working) and polling
(kept but disabled behind a flag) are mutually exclusive — only one runs at a time, so no
cross-path dedupe. Poll is the fallback when webhook delivery fails. Also note the
commenter-auth check must count recipe-maintainers org members/admins, not just repo
collaborators (the bot is org admin and was being rejected).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Strengthen the idempotency guardrail: every infra piece (swarm, traefik recipe deploy,
drone, bridge, dashboard) is a systemd oneshot that re-runs each activation/boot and
converges to desired state (like swarm-init) — no manual post-steps, no run-once
sentinels. Goal: from-scratch install = clone + nixos-rebuild switch + preconditions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Supersedes the original modules/traefik.nix hand-rolled proxy. cc-ci now deploys the
coop-cloud/traefik recipe via abra in wildcard/file-provider mode, serving the operator's
pre-issued wildcard cert as the recipe's ssl_cert/ssl_key swarm secrets — canonical
web/web-secure + proxy/swarm conventions every recipe expects, no ACME, DNS token never
on cc-ci. Updated §1, §1.5, §3, §4.0, §4.2, §5 (M1), §8.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Records the exact sequence to keep the orchestrator alive in tmux and resume it
with remote control (survives disconnects/laptop close), reconnect commands, and
pointers to launch/supervision docs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Clarify the two distinct names (--resume <conversation> vs --remote-control display
label), the in-session /remote-control shortcut, and the persist-vs-reconnect model.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Recipe repos under test live on the private mirror git.autonomic.zone/recipe-maintainers,
mirrored from upstream git.coopcloud.tech. autonomic-bot is admin on that org (can create
repos + add webhooks). A recipe missing from the mirror is not a blocker — fetch from
upstream and open a PR via the recipe-create-pr procedure. Updated D10 (§2) and enrollment (§4.1).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Documents the three roles (orchestrator vs Builder/Adversary loops), how to keep
this orchestrator session alive under --remote-control for check-ins/steering via
claude.ai/code, launch/supervision pointers, access/cred locations, and the VM
fallback. Secrets remain gitignored.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Planning + launch + setup material for the cc-ci Co-op Cloud recipe CI server:
plan.md (single source of truth), kickoff/launch supervision, and the
Builder/Adversary loop prompts. Secrets (.testenv) and runtime dirs are gitignored.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
