The canonical is last-GREEN, not CURRENT. Its promotion (WC5) deploys a
warm-<recipe> app and health-checks it; when that fails the canonical stays
pinned at an old release indefinitely — and the failure is invisible to the
upgrade tier, which keeps basing on it.
Live evidence (2026-08-10): gitea's canonical was 3.5.3+1.24.2-rootless from
2026-06-17 because warm-gitea crash-loops on a read-only app.ini (open fix:
recipe PR #4). Four other recipes are in the same state — the 2026-08-09 sweep
logged GREEN-BUT-PROMOTE-FAILED for bluesky-pds, drone, gitea, lasuite-drive and
lasuite-meet.
Consequence: the upgrade tier tested 3.5.3 -> head, a transition no deployment
performs, and MISSED the one they do. 3.5.3 -> head crosses an APP_INI_VERSION
change (v21 -> v22) so Swarm creates a fresh config object; the real
3.6.1 -> 3.6.2 upgrade keeps v22 and aborts on Swarm's immutable-config rule
('only updates to Labels are allowed'). That shipped in 3.6.2 and broke every
existing install taking the 1.27.1 security release.
Now: when the newest published release older than head is NEWER than the
canonical, use the release and say so in the log (naming the stale canonical so
the promote failure gets noticed). Resolution stays dynamic; UPGRADE_BASE_FLOOR
still filters structurally-invalid candidates; the canonical remains the base
whenever it is current.
Unit-verified on the real gitea tag set: base moves 3.5.3+1.24.2-rootless ->
3.6.1+1.26.2-rootless, exactly the transition that reproduces the break.
Main/driving sessions stay glm-5.2 (set per-launch via --model); the built-in
'general' agent used by the task tool runs opencode/deepseek-v4-pro: ~3-5x cheaper
per token with near-free cache hits, and the weekly /upgrade-all spends most of its
budget in per-recipe subagents.
Placement matters and was got wrong once (2026-08-04 config sat in the orchestrator
repo and never bound; the whole 2026-08-07 run billed as glm): sessions launched by
launch-upgrader.py pass no --dir, so they inherit the opencode serve process's
project (WorkingDirectory=/srv/cc-ci-orch/cc-ci = THIS repo), and task-tool
subagents inherit their parent session's directory. This file is therefore the
project config those subagents actually resolve.
Verified end-to-end 2026-08-10 with the launcher's exact invocation: parent session
modelID=glm-5.2, spawned subagent modelID=deepseek-v4-pro (read back from the
opencode session DB, not from config inspection).
test_head_runs_official_image_not_bitnamilegacy hardcoded the migration-era pin
discourse/discourse:3.5.3 and went stale on the first legitimate app bump
(2026.7.1, weekly 2026-08-03 — caught by verify run 2: the upgrade converged,
head image was discourse/discourse:2026.7.1, only the frozen pin failed). The
guarded property is the image FAMILY (official vs bitnamilegacy), not a frozen
version — now asserts the discourse/discourse: prefix. Not weakened: the
bitnami-leak check + official-prefix check together still assert exactly the
migration faithfulness; the concrete head pin is exercised by the deploy.
The strict recipe_meta key registry rejected the new declaration (caught by
verify run 1). str-typed, default None; full semantics documented on the key.
The 0.8.x->1.0.0 discourse recipe family switched app bitnami->official AND db
pgvector/pg17->discourse/postgres:pg18. That db-family change is a structural
break (bitnami cluster lacks the discourse role; pg_upgrade preserves-not-creates
roles) with NO supported in-place path. The dynamic base resolver's step-back
kept selecting 0.8.1+3.5.0 (newest tag below the unbumped 1.0.0+3.5.3 label) and
the upgrade tier red'd twice on this unsupported path (drone #1165, #1171 —
classified stale-test both times; recipe verified green on the real
official->official path).
Adds UPGRADE_BASE_FLOOR (phase basefloor) to resolve_upgrade_base: a recipe_meta
declaration naming the first post-break published version. Resolution stays
fully dynamic (this is NOT the removed sec2.G static pin): the floor only
EXCLUDES below-floor candidates (canonical, step-back, no-canonical fallback);
when no >=floor predecessor exists the tier records a DECLARED skip, never a
silent pass. main-tip fallback unaffected (post-break by construction).
tests/discourse/recipe_meta.py declares UPGRADE_BASE_FLOOR = 1.0.0+3.5.3 with
the full rationale. Unit-verified: head=1.0.0+3.5.3 -> declared skip (was:
0.8.1+3.5.0 wrong pick); post-release head=2026.x -> base 1.0.0+3.5.3 (the real
migration path). No assertion weakened - below-floor in-place upgrades were
never supported coverage.
The July gitea credential rotation updated the sops value, but ensure_secret is
create-once-immutable, so the service kept mounting cc_ci_bridge_gitea_token_v1
frozen at the pre-rotation token -> HTTP 401 'user does not exist' on every
!testme poll, silently dropping all triggers (found + worked around Drone-direct
during the 2026-08-03 weekly run; a manually-created _v2 existed but nothing
referenced it). Referencing _v3 makes the reconcile unit mint a fresh swarm
secret from the CURRENT /run/secrets/bridge_gitea_token (verified: HTTP 200 as
autonomic-bot) at next deploy.
Enrolls wordpress as a maintained recipe (operator request 2026-08-03):
- tests/wordpress/: recipe_meta (install-wizard-aware health 200/302, 900s deploy
timeout for mariadb+core-copy first boot, WARM_CANONICAL), custom suite:
health check, install-wizard completion + REST API round-trip (?rest_route= vs
/wp-json/ splits DB vs .htaccess failure layers), and the sec4.3 post round-trip
(XML-RPC write -> REST read -> permalink HTML, unique marker). PARITY.md documents
the baseline (no recipe-maintainer parity corpus for wordpress).
- nix/modules/bridge.nix: POLL_REPOS += recipe-maintainers/wordpress (!testme bridge
enrollment; deploy to the cc-ci host follows separately after the in-flight
/upgrade-all run - test-before-switch policy).
Mirror recipe-maintainers/wordpress created + main synced to coopcloud upstream
(adcd0e9f) with published tags. used-recipes.md gains 'wordpress weekly' in the
orchestrator repo.
test_create_room_get_livekit_token_and_read_back authenticated with a raw OIDC user
access token as 'Authorization: Bearer'; meet v1.22.0 hardened API auth to reject
user access tokens (release notes: 'reject user access tokens on the API'), so the
test went RED with 401 on the v1.24.0 upgrade (drone build #1137; same at v1.23.0
in build #1122).
Updated to the successor auth path: recipe-local _oidc_session.py (same helper as
tests/lasuite-docs) drives the real OIDC authorization-code flow (app -> keycloak
login form -> callback -> Django session cookie, CSRF on unsafe methods).
- NEW assertion: a raw OIDC Bearer token is REJECTED (401/403) - the v1.22.0
hardening asserted as the new correct behavior.
- The full meeting flow (create 201 + LiveKit JWT grant, read-back, DELETE) is
unchanged, now over the session-authenticated API. No assertion weakened.
Stale-test fix for recipe PR
recipe-maintainers/lasuite-meet#8
(carry-over from /upgrade-all 2026-07-24).
test_oidc_login_via_keycloak and test_create_doc_and_read_back authenticated with
'Authorization: Bearer <password-grant JWT>'; impress v5.4.0 removed Bearer/JWT
auth on the API (SessionAuthentication only), so both went RED with 401 on the
v5.4.1 upgrade.
Updated to the successor auth path: a new recipe-local _oidc_session.py drives the
real OIDC authorization-code flow (app -> keycloak login form -> callback ->
Django session cookie, with CSRF headers on unsafe methods).
- test_oidc_login: still asserts the unauth challenge redirect; NOW also asserts a
raw Bearer JWT is REJECTED (401/403 - the v5.4.0 hardening, asserted as the new
correct behavior); then asserts the session-authenticated whoami returns the
provisioned user. No assertion weakened - the auth proof is stronger than before.
- test_create_doc: same create+read-back round-trip assertions, now over the
session-authenticated API.
Stale-test fix for recipe PR
recipe-maintainers/lasuite-docs#7
(carry-over from /upgrade-all 2026-07-24).
Since NixOS 25.05, scripted networking installs the default route from the
gateway interface's network-addresses-<iface>.service, matching the interface
via defaultGateway.interface or by subnet inclusion. Hetzner's off-subnet
point-to-point gateway (91.98.47.73/32 on eth0, gw 172.31.1.1) matched
neither with the bare-string form, so the 26.05 switch on 2026-08-03 left the
host with no default route and off the network (recovered via Hetzner rescue:
grubenv default back to the 24.11 generation).
With an explicit interface, the module installs both the gateway host route
and the default route from eth0's own unit:
ip -4 route replace 172.31.1.1 proto static dev eth0
ip -4 route replace default proto static dev eth0 via 172.31.1.1
Verified by nix eval of systemd.services.network-addresses-eth0.script.
nixpkgs: 50ab793 -> 531670d (nixos-26.05, was nixos-24.11 EOL)
sops-nix: 77c423a -> f140661 (master, buildGo125Module now builds on 26.05)
3-release channel jump (24.11 -> 25.05 -> 25.11 -> 26.05) to match the
orchestrator host and get CVE patches. 24.11 was EOL (2025-06-30).
Notable 26.05 changes reviewed: systemd Stage 1 boot (default, no config
change needed), dbus-broker default, bash nixos-rebuild removed (Python
rewrite mandatory; git present on host), MySQL 8.0 removed (not used as a
NixOS module). No system.rebuild/fileSystems/boot.initrd options in the
host config, so no config edits required.
The token embedded in the recipe clone's gitea remote was copytree'd into the
world-readable per-run tree. SSH push keys the auth to ~/.ssh, so no secret is
written into any .git/config. API calls still use the token.
Terminal state re-verified cold. Corrected a near-miss from this wake: my first
`^## .*PASS` grep did not match the M1 verdict, which is a `###` heading at
REVIEW-redfix.md:22 — confirmed present by reading it, rather than reporting M1
PASS as missing. Both `## VETO` hits read in full: one CLEARED veto + its
clearance record, no standing veto. DONE @00:18Z, M1+M2 PASS, HEAD==origin/main,
no inbox. Untracked main.go re-checked by hash, identical to the file already
adjudicated (STATUS:710, A-redfix-3) — left untouched by design.
Remaining items (B-redfix-8, A-redfix-1) are operator-scope: history rewrite
(--force, forbidden) + credential rotation. Loop STOPPED.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWSJEPYEMx9LZhPwyQCmhP
HEAD==origin/main==89af75b is my own wake-#64 journal: neither the Adversary nor the tree has moved
since I last stopped, so there was no handoff to answer. Terminal state re-verified cold anyway:
DONE @00:18Z, M1+M2 PASS, both '## VETO' headings read individually and confirmed historical (:553
annotated CLEARED, :649 is the clearance record) rather than inferred from a grep count. No inbox.
Untracked main.go left in place per the standing adjudication at STATUS:710 / A-redfix-3.
Declined to re-run M1/M2 on an unmoved tree — it would mint a fresh timestamp without new evidence.
Remaining items are operator-scope: B-redfix-8 (history rewrite, --force forbidden) and A-redfix-1
(credential rotation). Loop STOPPED.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015yZh7A6Zcac5LQ3tqqtb9W
Terminal state re-verified cold: DONE @00:18Z, M1+M2 PASS stand, no standing VETO (both '## VETO' hits
read, not counted: one CLEARED + its clearance record), all findings CLOSED, HEAD==origin/main, no inbox.
Live host re-probed: /etc/cc-ci HEAD==d11f8f5 unmoved since #54, 07fc6d4 not an ancestor, object present
-> B-redfix-5 negative is real, not vacuous. Declined to re-run M1/M2 on an unmoved tree.
Prefix is 'chore(' not 'review(' ON PURPOSE: no verdict landed this wake. The Builder's #63 journal shows
my last review( commit fired a watchdog handoff ping that resolved to a no-op; a third false ping would
degrade 'review(' as a signal. Loop STOPPED.
Adversary (a22c384) confirmed both findings but counted 3061 blobs vs my 3099. Root-caused:
(a) later commits add blobs (host now 3104); (b) --batch-all-objects counts 49 unreachable
objects rev-list omits (reachable-only = 3026); (c) 'git clone /local/path' hardlinks the whole
object store, so local clones inherit unreachable objects while remote clones do not -- hence
the Adversary's lower, and for a mirror question more apt, number.
All four scans agree on password=2 and token=0. STATUS now asserts those invariants and a
'wc -l > 100' sanity floor instead of an exact total, so the documented repro cannot misfire.
Operator-scope, no gate impact, no VETO; DONE stands.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018XE43k4DaeMXHMK51wBUu2
Closes the gap in Adversary wake #57's break-it probe, which covered dashboard/reports/Drone
logs but not the public git mirror -- the surface B-redfix-8 is about.
- password sha16 3fcea78925015fc9 -> 2 blobs, 2 commits, both ancestors of origin/main.
Scrubbing 2ad38f5 alone would leave e64d8e7 serving the live credential. STATUS corrected.
- oauth2 token sha16 9c44a1aea2ecb389 -> 0/3099 blobs. Filesystem-only exposure.
- Probe hygiene: awk is absent here; awk-based blob lists give a vacuous 0. Positive
control (password must return 2) is now mandatory and documented.
Operator-scope, no gate impact, no VETO; DONE stands.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018XE43k4DaeMXHMK51wBUu2
Adversary wake #55 conceded my insteadOf falsification but corrected me twice; verified both first-hand:
1. "Production CI does not regenerate / manual-* = hand-run" WRONG. run_id()=manual-<pid> for any non-Drone
run (run_recipe_ci.py:318-319); nightly-sweep runs run_recipe_ci.py outside Drone with CCCI_SKIP_FETCH=1
(nightly_sweep.py:88). Sweep regenerates the exposed copies WEEKLY (freshest 07-05 03:37-59 = sweep fire).
2. Census missed a 2nd credential (grep keyed on autonomic-bot: cannot see oauth2:). Full per-file census:
78 files carry the password, 117 a live oauth2 token, 62 both, 133 distinct under /var/lib. Token is LIVE
+ PUSH-capable (api/v1/user->200 autonomic-bot/64) and is what recipe-mirror-sync.sh:39 pushes with —
falsifies my own B-redfix-8 "small blast radius" note.
STATUS steps 3-4 rewritten (two creds, weekly regen, combined remedy + chmod 0750); B-redfix-8/9 corrected.
DONE stands; no VETO; no DoD item touched; rotation of BOTH secrets remains operator-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y9GyBPF1EgTTh6277Xjj7k
Adversary wake #53/#54 reaffirm DONE + no VETO (no gate impact), but both correct STATUS text (Builder-owned):
1. A-redfix-1 "sole copy" withdrawn: 78 world-readable cred-bearing .git/config, sentinel 3fcea78925015fc9.
Exposure confirmed; Adversary's /root/.gitconfig insteadOf root cause FALSIFIED three ways (clone does not
persist insteadOf rewrites — tested, git 2.47.2; /etc/cc-ci config predates .gitconfig by 2wk; live
fetch_recipe uses a non-persisted http.extraHeader token since 9b33fdf, so 0/215 numeric runs carry it).
Real generator: CCCI_SKIP_FETCH copytree of credentialed /root/.abra/recipes (0700) into 0755 run tree.
STATUS steps 3-4 rewritten; filed B-redfix-9 (deferred). Falsification sent via ADVERSARY-INBOX.md.
2. Sweep mechanism: runs deployed /etc/cc-ci (main @ d11f8f5), not origin/main. Verified first-hand.
DONE stands; no VETO; no DoD item touched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y9GyBPF1EgTTh6277Xjj7k
No standing VETO (2x '## VETO' headings are historical: one annotated CLEARED, one is the clearing
record). STATUS:481 CLAIMED is the superseded historical M2 claim. No action available; loop stopped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y9GyBPF1EgTTh6277Xjj7k
Stop condition re-checked from a cold read: STATUS ## DONE, M1+M2 fresh PASS, no standing
VETO (line 553 header is annotated CLEARED), no inbox either direction, Adversary loop
closed out at #49. Loop stays stopped; no DoD item re-run.
Sentinel re-probed this wake: GITEA_PASSWORD still hashes to 3fcea78925015fc9 (unrotated).
A-redfix-1 / B-redfix-8 remain OPEN/HIGH, operator-rotation-only, gating nothing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TRRkTGdWKWoxddF6QXhWfd
autonomic-bot
autonomic-bot