M2 finding (Adversary-flagged): promote_canonical did a bare `abra app deploy` that lacked the
cold install's wiring, so recipes that passed the cold test still failed to promote:
- ghost: `abra app new` FATA 'locally unstaged changes' — the CCCI_SKIP_FETCH per-run tree was
left dirty by the tier suite. Fix: force re-checkout the tag + `git clean -fd` before deploy.
- bluesky-pds: missing pds_plc_rotation_key (install_steps inserts it, #generate=false).
- custom-html-tiny: 404 (install_steps seeds index.html). Fix: run install_steps_hook in promote.
- OIDC recipes would miss their realm. Fix: provision DEPS in promote like the cold install.
promote_canonical now: clean tree → provision deps → deploy_app with install_steps_hook + overlay +
ready-probes, then snapshot. Also: sweep result label now derives from whether the canonical was
actually written (promote is non-fatal; rc==0 did not imply promoted) — fixes the misleading
'PASS (promoted)'.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the head_version-vs-latest_version divergence: should_promote gates on head_version
(code under test) but promote_canonical recorded latest_version(recipe_tags). In a manual
RECIPE=<r> run whose main checkout sits on a tag OLDER than the newest published tag, the gate
would pass on the older tag yet promote the newer (never-tested) one. promote_canonical now
takes the tested `version` (head_version, guaranteed a release tag by the tagged-gate) and
records exactly that. Sweep path unaffected (head==tag by construction).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
M1.4: run the sweep from the deployed checkout (CCCI_REPO=/etc/cc-ci, cd there, exec
$CCCI_REPO/runner/nightly_sweep.py) instead of a nix-store runner copy. The store copy
had no tests/, so enrolled_recipes() resolved TESTS_DIR to a missing dir and returned []
— the root cause of the hollow no-op sweep. /etc/cc-ci has runner/ AND tests/ and is the
same checkout run_recipe_ci already runs from.
M1.5: timer OnCalendar daily -> weekly (Sun 03:00 UTC), Persistent kept.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
WARM_CANONICAL=True added to every recipe in cc-ci-plan/used-recipes.md (20 weekly +
uptime-kuma external). enrolled_recipes() now returns all 21. Test fixtures
(custom-html-*-bad, concurrency, regression) intentionally left unenrolled.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- should_promote_canonical gains a `tagged` requirement (canon §2.A): a green cold
latest run promotes only when the tested head version is a published release tag;
an untagged main commit never becomes a canonical.
- warm_reconcile.is_released_version(recipe, version): release-tag membership (exact or
by version_key). Caller computes `tagged` so the gate stays pure.
- unit tests: untagged -> no promote; is_released_version cases.
- drive-by (pre-existing reds, unrelated to canon, now green): test_warm_reconcile
traefik assertion was stale vs the phase-pxgate spec (probes /api/version, no
health_domain); meta.py UPGRADE_BASE_VERSION KEYS help synced to the prevb doc text.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
resolve_upgrade_base now reads the head's published version (abra.head_compose_version,
the coop-cloud.<stack>.version label) and, when the last-green warm-canonical version
equals it, steps back to the newest published version strictly older than head instead
of deploying a same-version no-op. warm_reconcile gains version_key + newest_older_version
(single coop-cloud ordering source; sort_versions refactored onto version_key, no behavior
change). Skip only when no older published predecessor exists. Step-back returns kind=version
so it inherits F1d-2 pinned-tag checkout. Extends tests/unit/test_upgrade_base.py (13 pass).
21/21 recipes GREEN post-prevb. 0 prevb regressions. A-regall-2 closed
(plausible backup_restore=fail was recipe bug in 3.0.1+v2.0.0, NOT prevb;
run 758 / PR#3 / 3.1.0+v2.0.0 confirms L5 pass with fixed backup mechanism).
All batches 1-6 complete. M1+M2 both claimed 2026-06-17T04:45Z.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Builder diagnosis (a3d115d) accepted:
- backupbot.backup.path in 3.0.1+v2.0.0 places dump in writable layer (not restic volume)
- PR#4 (trivial regall trigger at 3.0.1+v2.0.0) exposes the bug; PR#3 (3.1.0+v2.0.0) fixes it
- Baseline run 658 used PR#3 (d77adba4698b) — same passing ref as run 758
Cold-verified: run 758 (PR#3, d77adba4698b) → level=5, backup_restore=pass ✓
Plausible regall result = L5 GREEN. Sweep now 21/21 complete.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
backupbot.backup.path: "/postgres.dump.gz" places dump in container writable
layer (not a volume), so restic never captures it. Restore post-hook fails
with "No such file or directory". PR#3 (3.1.0+v2.0.0) fixes this with
backupbot.backup.volumes.db-data.path. Baseline run 658 tested PR#3 (working
mechanism), not 3.0.1+v2.0.0 (broken). Re-opened PR#3 + !testme triggered
(comment 14651) to demonstrate backup_restore=pass. BUILDER-INBOX consumed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Runs 750 and 754 both fail: ci_marker absent after restore.
No-op upgrade (3.0.1+v2.0.0→3.0.1+v2.0.0) via UPGRADE_BASE_VERSION path is prevb-specific.
Baseline run 658 had genuine git-ref upgrade and passed L5.
Builder-INBOX written. M1 blocked pending plausible fix.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The custom tier runs on the PR head — now genuinely the official discourse/discourse image (prevb
stopped the overlay reverting it to bitnamilegacy). mint_admin hardcoded /opt/bitnami/discourse (404 on
official) → create-topic roundtrip failed. Detect /var/www/discourse, re-export DISCOURSE_DB_PASSWORD
from /run/secrets (entrypoint exports it only for boot), run bin/rails; keep bitnami fallback.
docker stack deploy doesn't prune services the head compose dropped (discourse PR#4 drops sidekiq),
leaving them orphaned on the base image. perform_upgrade now reconciles the live stack to the head
compose service set (lifecycle.prune_orphan_services). Makes the deployed stack faithfully reflect
the head — no test weakened. No-op when service sets match / compose unresolvable.
Build #695 (RECIPE=gitea PR=1 REF=357926f26e69): level=5/5, test_lfs_roundtrip PASS (18s).
Build #692 (RECIPE=drone REF=main): level=5/5, dep path confirmed.
All 6 M2 DoD conditions met per Adversary REVIEW-gtea.md @2026-06-15T22:10Z.
Phase gtea complete. Gitea enrolled as a fully-tested recipe with LFS PR verified.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Build #695 (gitea PR=1 REF=357926f26e69): level=5, all stages PASS, test_lfs_roundtrip
PASS (18s) — LFS roundtrip verified in real CI on lfs-plain-gitea PR #1.
Build #692 (drone dep path PR=0 REF=main): level=5, drone recipe unaffected.
Build #684 (gitea main PR=0): level=5 (verified in prior round).
cc-ci self-test lint green. Unit tests 53/53. no_secret_leak in all runs.
Also records build #691 FAIL finding: STACK_NAME not in .env (fixed in ad53b5a).
Gate M2: ADVERSARY PASS.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
abra does NOT write STACK_NAME to the app's .env file — it derives it at runtime
by replacing dots with underscores (e.g. gite-e1cb78.ci.commoninternet.net →
gite-e1cb78_ci_commoninternet_net). Build #691 failed with 'STACK_NAME not found'
because the env file read was looking for a key that doesn't exist.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Blocker 4 fix: abra `secret generate --all` uses .env.sample for length hints; the
lfs-plain-gitea PR has SECRET_LFS_JWT_SECRET_VERSION=v1 COMMENTED OUT, so abra produces
a wrong-length secret. gitea requires exactly 43 chars (32 bytes base64 URL-safe); wrong
length → gitea fatals trying to save the JWT secret to the read-only Docker Config
app.ini → health check fails → swarm rolls back.
Fix: new UPGRADE_SECRET_PREP hook (meta.py) called before `abra secret generate --all`
in the upgrade path. abra's `--all` is idempotent (skips existing secrets), so the
correctly pre-inserted secret survives. gitea's recipe_meta.py implements the hook using
`docker secret create` directly to guarantee correct format regardless of .env.sample.
Also consumes machine-docs/BUILDER-INBOX.md (Adversary Blocker 4 digest).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Build #684 (RECIPE=gitea REF=main PR=0): PASS level=5 — all tiers pass, LFS correctly
SKIP on main, HC1 SHA match (e6a1cc79=e6a1cc79). M2 main-branch DoD MET.
Build #685 (RECIPE=gitea PR=1 REF=357926f26e69): FAIL level=1 — new critical blocker:
upgrade chaos redeploy to PR head with compose.lfs.yml fails with rollback_completed.
Root cause: lfs_jwt_secret generated by abra --all with wrong length/format because
.env.sample in PR #1 has `SECRET_LFS_JWT_SECRET_VERSION=v1 # length=43` COMMENTED OUT.
Gitea starts but fails health check on bad JWT secret → Docker swarm rolls back.
Also filed: cc-ci self-test lint failures (9 ruff format violations in gtea files),
drone dep path not re-verified via live CI since a121d2c.
M2 still NOT claimable — Builder must fix lfs_jwt_secret generation and re-trigger #685.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Blocker 1 (LFS roundtrip fails on PR #1):
- Add UPGRADE_EXTRA_ENV to gitea recipe_meta.py — after PR-head checkout
(compose.lfs.yml now in ABRA_DIR), add compose.lfs.yml to COMPOSE_FILE
and set SECRET_LFS_JWT_SECRET_VERSION=v1 so the upgrade chaos redeploy
actually runs with LFS enabled. Without this, the base install checks out
the 3.5.x tag (compose.lfs.yml removed), EXTRA_ENV sees no LFS, and the
upgrade chaos redeploy inherits the no-LFS .env — so the LFS test runs
(compose.lfs.yml is restored by recipe_checkout_ref) but LFS is off.
- Add abra.secret_generate(domain) in generic.perform_upgrade when
upgrade_env is non-empty — generates lfs_jwt_secret before chaos redeploy.
Blocker 2 (REF=main upgrade fails HC1):
- Always use recipe_head_commit (git rev-parse HEAD) for head_ref instead
of using ref directly. When ref="main" (a branch name), the HC1 commit
check "head_ref.startswith(chaos_commit)" always fails since "main" ≠ SHA.
recipe_head_commit returns the actual SHA after the fetch/checkout.
Side-fix (stale creds — build #675):
- ops.py pre_install: delete the per-domain creds file before calling
_ensure_admin. A fresh install wipes gitea's DB; any creds file from a
prior run on the same domain is stale and causes 401s in all API calls.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two critical issues prevent M2: (1) lfs_jwt_secret not generated via disk .env → LFS disabled in
container; (2) upgrade tier fails when REF=main. Details + fix hints in BUILDER-INBOX.md.
Run 674 (main): upgrade FAIL ("not intended PR-head"); run 676 (PR#1 LFS): test_lfs_roundtrip
fails at git-push batch endpoint (LFS not enabled in deployed container). Builder must fix before M2.
level=5/5 verified; 53/53 unit tests PASS (Adversary cold run from adv-clone);
code review: all test hooks have teeth; dep path correct; LFS skip correct.
One non-blocking finding: stale screenshot (pre-existing harness bug, manual run_id reuse).
Prerequisite for M2: enables the bridge to pick up !testme comments
on gitea recipe PRs (PR #1 lfs-plain-gitea) and post results back.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
LIVENESS PROTOCOL: declared per 10-min rule. Adversary pre-checks done
at 950ab8b, ready to verify. Claim posted at bac3662 (~20:13Z).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Required by test_lfs_roundtrip.py for the M2 LFS capstone run on the
lfs-plain-gitea PR branch. Also revert the same change from the Incus
host (cc-ci/configuration.nix) where it was mistakenly added.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Gitea 1.22+ (including 1.24.2 on cc-ci) requires explicit scopes
when creating API tokens. Add read:user + read:organization to satisfy
the token creation endpoint and the read-back assertions that follow.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Empty-repo HTTPS push with git clone exits 0 but silently fails (remote
branch creation on an empty clone is unreliable). Fix:
- Create repo with auto_init=True + default_branch=main (initial commit present)
- Clone into a non-existing subdir (git clone must target non-existing path)
- Push via explicit cred_url (bypasses remote config; no tracking needed)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- test_git_push.py + test_lfs_roundtrip.py: use cred_url (https://user:pass@host/...)
instead of GIT_CONFIG_COUNT insteadOf rewriting, which silently failed to
propagate credentials to the push step (repo remained empty after push exit 0).
Also add GIT_SSL_NO_VERIFY=true and GIT_TERMINAL_PROMPT=0.
- test_lfs_roundtrip.py: fix restart health-poll path /api/v1/version → /version
(_api() already prepends /api/v1; double prefix produced 404 and a 120s timeout).
- nix/hosts/cc-ci/configuration.nix: add git-lfs to systemPackages (required for
the LFS capstone test on the lfs-plain-gitea PR branch).
Adversary pre-M1 findings: Issue 1 (git-lfs absent) + Issue 2 (double path) both fixed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
autonomic-botandClaude Opus 4.8