The live /root/.ssh/authorized_keys on cc-ci had drifted from this
declarative list: several keys were added manually over time and would
have been wiped by the next nixos-rebuild switch.
- sync the declared list to the live file (10 keys, verified blob-for-blob)
- add the new notplants-orchestrator and nptest keys (added live first)
- drop claude-sandbox keys at operator request (incl. one unnamed live
key identified as a sandbox key, fingerprint SHA256:Wlhj5g4IjCId1wvR
HnxEiwVVu2N+aYa4gRsY1XfOkts) and the never-live claude@claude-vm key
- live file updated in the same change, so state is converged now
- acme-dns.nix: ONE dual-zone SAN cert (ci+*.ci.autonomic.zone AND ci+*.ci.commoninternet.net)
via the same acmedns account — storage re-keyed by cc-ci-acme-storage-seed.service; handoff
reads the new cert dir. Single secret pair => zero changes to the traefik reconciler.
- dashboard/bridge/reports: dual Host rules during the bake window (bridge gets explicit
parentheses so && does not shadow the dashboard on the new host).
- drone abra app renamed to drone.ci.autonomic.zone (fresh DB supported: DRONE_USER_CREATE
re-injects the sops bridge token); runner RPC + bootstrap-drone-oauth.sh follow.
- harness: app_domain() issues *.ci.autonomic.zone run domains; RUN_APP_RE / stack-name
regexes / docker-prune accept BOTH zones during the bake. Warm stacks deliberately stay
on the legacy zone (data-warm volumes; post-bake migration).
- URLs in bridge/dashboard defaults + recipe-report.py follow the new names.
Route deepseek-v4-flash through the opencode zen subscription endpoint
(opencode/deepseek-v4-flash) instead of tinfoil pay-per-use. The zen
subscription is already paid for; flash consumes the balance far more
slowly than glm-5.2 did (flash cost ~$0.0000002/tok vs glm's higher rate),
so a full weekly run is much less likely to exhaust the balance — and
the operator has reset it.
Completes the fleet-wide move: upgrader parent (LOOP_MODEL) + report
(REPORT_MODEL) in upgrader.env, and this PR sets the subagents. Supervisor
stays on opencode-go/glm-5.2.
Verified: zen endpoint serves deepseek-v4-flash (HTTP 200, 1.2s).
Switch the general subagent model from opencode/deepseek-v4-pro (zen
endpoint, subject to workspace balance limits) to tinfoil/deepseek-v4-flash
(pay-per-use API, no rolling balance limit, cheaper than pro).
This completes the fleet-wide move to deepseek-flash on tinfoil: the upgrader
parent (LOOP_MODEL) and report (REPORT_MODEL) are set in upgrader.env, and
this changes the subagents. The hourly supervisor stays on glm-5.2.
Verified: tinfoil endpoint accepts deepseek-v4-flash (HTTP 200, 2.3s response).
Trial run next Friday (2026-08-21) to evaluate flash capability on the
upgrade workload; fall back to pro if it struggles on complex recipes.
Verified GREEN (level 5/5) on cc-ci against plausible PR head 867ebfaf, twice: once with the SQL fixture, once with the app-native rewrite. Regression sample green.
Main/driving sessions stay glm-5.2 (set per-launch via --model); the built-in
'general' agent used by the task tool runs opencode/deepseek-v4-pro: ~3-5x cheaper
per token with near-free cache hits, and the weekly /upgrade-all spends most of its
budget in per-recipe subagents.
Placement matters and was got wrong once (2026-08-04 config sat in the orchestrator
repo and never bound; the whole 2026-08-07 run billed as glm): sessions launched by
launch-upgrader.py pass no --dir, so they inherit the opencode serve process's
project (WorkingDirectory=/srv/cc-ci-orch/cc-ci = THIS repo), and task-tool
subagents inherit their parent session's directory. This file is therefore the
project config those subagents actually resolve.
Verified end-to-end 2026-08-10 with the launcher's exact invocation: parent session
modelID=glm-5.2, spawned subagent modelID=deepseek-v4-pro (read back from the
opencode session DB, not from config inspection).
test_head_runs_official_image_not_bitnamilegacy hardcoded the migration-era pin
discourse/discourse:3.5.3 and went stale on the first legitimate app bump
(2026.7.1, weekly 2026-08-03 — caught by verify run 2: the upgrade converged,
head image was discourse/discourse:2026.7.1, only the frozen pin failed). The
guarded property is the image FAMILY (official vs bitnamilegacy), not a frozen
version — now asserts the discourse/discourse: prefix. Not weakened: the
bitnami-leak check + official-prefix check together still assert exactly the
migration faithfulness; the concrete head pin is exercised by the deploy.
The strict recipe_meta key registry rejected the new declaration (caught by
verify run 1). str-typed, default None; full semantics documented on the key.
The 0.8.x->1.0.0 discourse recipe family switched app bitnami->official AND db
pgvector/pg17->discourse/postgres:pg18. That db-family change is a structural
break (bitnami cluster lacks the discourse role; pg_upgrade preserves-not-creates
roles) with NO supported in-place path. The dynamic base resolver's step-back
kept selecting 0.8.1+3.5.0 (newest tag below the unbumped 1.0.0+3.5.3 label) and
the upgrade tier red'd twice on this unsupported path (drone #1165, #1171 —
classified stale-test both times; recipe verified green on the real
official->official path).
Adds UPGRADE_BASE_FLOOR (phase basefloor) to resolve_upgrade_base: a recipe_meta
declaration naming the first post-break published version. Resolution stays
fully dynamic (this is NOT the removed sec2.G static pin): the floor only
EXCLUDES below-floor candidates (canonical, step-back, no-canonical fallback);
when no >=floor predecessor exists the tier records a DECLARED skip, never a
silent pass. main-tip fallback unaffected (post-break by construction).
tests/discourse/recipe_meta.py declares UPGRADE_BASE_FLOOR = 1.0.0+3.5.3 with
the full rationale. Unit-verified: head=1.0.0+3.5.3 -> declared skip (was:
0.8.1+3.5.0 wrong pick); post-release head=2026.x -> base 1.0.0+3.5.3 (the real
migration path). No assertion weakened - below-floor in-place upgrades were
never supported coverage.
The July gitea credential rotation updated the sops value, but ensure_secret is
create-once-immutable, so the service kept mounting cc_ci_bridge_gitea_token_v1
frozen at the pre-rotation token -> HTTP 401 'user does not exist' on every
!testme poll, silently dropping all triggers (found + worked around Drone-direct
during the 2026-08-03 weekly run; a manually-created _v2 existed but nothing
referenced it). Referencing _v3 makes the reconcile unit mint a fresh swarm
secret from the CURRENT /run/secrets/bridge_gitea_token (verified: HTTP 200 as
autonomic-bot) at next deploy.
Enrolls wordpress as a maintained recipe (operator request 2026-08-03):
- tests/wordpress/: recipe_meta (install-wizard-aware health 200/302, 900s deploy
timeout for mariadb+core-copy first boot, WARM_CANONICAL), custom suite:
health check, install-wizard completion + REST API round-trip (?rest_route= vs
/wp-json/ splits DB vs .htaccess failure layers), and the sec4.3 post round-trip
(XML-RPC write -> REST read -> permalink HTML, unique marker). PARITY.md documents
the baseline (no recipe-maintainer parity corpus for wordpress).
- nix/modules/bridge.nix: POLL_REPOS += recipe-maintainers/wordpress (!testme bridge
enrollment; deploy to the cc-ci host follows separately after the in-flight
/upgrade-all run - test-before-switch policy).
Mirror recipe-maintainers/wordpress created + main synced to coopcloud upstream
(adcd0e9f) with published tags. used-recipes.md gains 'wordpress weekly' in the
orchestrator repo.
test_create_room_get_livekit_token_and_read_back authenticated with a raw OIDC user
access token as 'Authorization: Bearer'; meet v1.22.0 hardened API auth to reject
user access tokens (release notes: 'reject user access tokens on the API'), so the
test went RED with 401 on the v1.24.0 upgrade (drone build #1137; same at v1.23.0
in build #1122).
Updated to the successor auth path: recipe-local _oidc_session.py (same helper as
tests/lasuite-docs) drives the real OIDC authorization-code flow (app -> keycloak
login form -> callback -> Django session cookie, CSRF on unsafe methods).
- NEW assertion: a raw OIDC Bearer token is REJECTED (401/403) - the v1.22.0
hardening asserted as the new correct behavior.
- The full meeting flow (create 201 + LiveKit JWT grant, read-back, DELETE) is
unchanged, now over the session-authenticated API. No assertion weakened.
Stale-test fix for recipe PR
recipe-maintainers/lasuite-meet#8
(carry-over from /upgrade-all 2026-07-24).
test_oidc_login_via_keycloak and test_create_doc_and_read_back authenticated with
'Authorization: Bearer <password-grant JWT>'; impress v5.4.0 removed Bearer/JWT
auth on the API (SessionAuthentication only), so both went RED with 401 on the
v5.4.1 upgrade.
Updated to the successor auth path: a new recipe-local _oidc_session.py drives the
real OIDC authorization-code flow (app -> keycloak login form -> callback ->
Django session cookie, with CSRF headers on unsafe methods).
- test_oidc_login: still asserts the unauth challenge redirect; NOW also asserts a
raw Bearer JWT is REJECTED (401/403 - the v5.4.0 hardening, asserted as the new
correct behavior); then asserts the session-authenticated whoami returns the
provisioned user. No assertion weakened - the auth proof is stronger than before.
- test_create_doc: same create+read-back round-trip assertions, now over the
session-authenticated API.
Stale-test fix for recipe PR
recipe-maintainers/lasuite-docs#7
(carry-over from /upgrade-all 2026-07-24).
Since NixOS 25.05, scripted networking installs the default route from the
gateway interface's network-addresses-<iface>.service, matching the interface
via defaultGateway.interface or by subnet inclusion. Hetzner's off-subnet
point-to-point gateway (91.98.47.73/32 on eth0, gw 172.31.1.1) matched
neither with the bare-string form, so the 26.05 switch on 2026-08-03 left the
host with no default route and off the network (recovered via Hetzner rescue:
grubenv default back to the 24.11 generation).
With an explicit interface, the module installs both the gateway host route
and the default route from eth0's own unit:
ip -4 route replace 172.31.1.1 proto static dev eth0
ip -4 route replace default proto static dev eth0 via 172.31.1.1
Verified by nix eval of systemd.services.network-addresses-eth0.script.
nixpkgs: 50ab793 -> 531670d (nixos-26.05, was nixos-24.11 EOL)
sops-nix: 77c423a -> f140661 (master, buildGo125Module now builds on 26.05)
3-release channel jump (24.11 -> 25.05 -> 25.11 -> 26.05) to match the
orchestrator host and get CVE patches. 24.11 was EOL (2025-06-30).
Notable 26.05 changes reviewed: systemd Stage 1 boot (default, no config
change needed), dbus-broker default, bash nixos-rebuild removed (Python
rewrite mandatory; git present on host), MySQL 8.0 removed (not used as a
NixOS module). No system.rebuild/fileSystems/boot.initrd options in the
host config, so no config edits required.
Terminal state re-verified cold. Corrected a near-miss from this wake: my first
`^## .*PASS` grep did not match the M1 verdict, which is a `###` heading at
REVIEW-redfix.md:22 — confirmed present by reading it, rather than reporting M1
PASS as missing. Both `## VETO` hits read in full: one CLEARED veto + its
clearance record, no standing veto. DONE @00:18Z, M1+M2 PASS, HEAD==origin/main,
no inbox. Untracked main.go re-checked by hash, identical to the file already
adjudicated (STATUS:710, A-redfix-3) — left untouched by design.
Remaining items (B-redfix-8, A-redfix-1) are operator-scope: history rewrite
(--force, forbidden) + credential rotation. Loop STOPPED.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWSJEPYEMx9LZhPwyQCmhP
HEAD==origin/main==89af75b is my own wake-#64 journal: neither the Adversary nor the tree has moved
since I last stopped, so there was no handoff to answer. Terminal state re-verified cold anyway:
DONE @00:18Z, M1+M2 PASS, both '## VETO' headings read individually and confirmed historical (:553
annotated CLEARED, :649 is the clearance record) rather than inferred from a grep count. No inbox.
Untracked main.go left in place per the standing adjudication at STATUS:710 / A-redfix-3.
Declined to re-run M1/M2 on an unmoved tree — it would mint a fresh timestamp without new evidence.
Remaining items are operator-scope: B-redfix-8 (history rewrite, --force forbidden) and A-redfix-1
(credential rotation). Loop STOPPED.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015yZh7A6Zcac5LQ3tqqtb9W
Adversary (a22c384) confirmed both findings but counted 3061 blobs vs my 3099. Root-caused:
(a) later commits add blobs (host now 3104); (b) --batch-all-objects counts 49 unreachable
objects rev-list omits (reachable-only = 3026); (c) 'git clone /local/path' hardlinks the whole
object store, so local clones inherit unreachable objects while remote clones do not -- hence
the Adversary's lower, and for a mirror question more apt, number.
All four scans agree on password=2 and token=0. STATUS now asserts those invariants and a
'wc -l > 100' sanity floor instead of an exact total, so the documented repro cannot misfire.
Operator-scope, no gate impact, no VETO; DONE stands.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018XE43k4DaeMXHMK51wBUu2
Closes the gap in Adversary wake #57's break-it probe, which covered dashboard/reports/Drone
logs but not the public git mirror -- the surface B-redfix-8 is about.
- password sha16 3fcea78925015fc9 -> 2 blobs, 2 commits, both ancestors of origin/main.
Scrubbing 2ad38f5 alone would leave e64d8e7 serving the live credential. STATUS corrected.
- oauth2 token sha16 9c44a1aea2ecb389 -> 0/3099 blobs. Filesystem-only exposure.
- Probe hygiene: awk is absent here; awk-based blob lists give a vacuous 0. Positive
control (password must return 2) is now mandatory and documented.
Operator-scope, no gate impact, no VETO; DONE stands.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018XE43k4DaeMXHMK51wBUu2
Adversary wake #55 conceded my insteadOf falsification but corrected me twice; verified both first-hand:
1. "Production CI does not regenerate / manual-* = hand-run" WRONG. run_id()=manual-<pid> for any non-Drone
run (run_recipe_ci.py:318-319); nightly-sweep runs run_recipe_ci.py outside Drone with CCCI_SKIP_FETCH=1
(nightly_sweep.py:88). Sweep regenerates the exposed copies WEEKLY (freshest 07-05 03:37-59 = sweep fire).
2. Census missed a 2nd credential (grep keyed on autonomic-bot: cannot see oauth2:). Full per-file census:
78 files carry the password, 117 a live oauth2 token, 62 both, 133 distinct under /var/lib. Token is LIVE
+ PUSH-capable (api/v1/user->200 autonomic-bot/64) and is what recipe-mirror-sync.sh:39 pushes with —
falsifies my own B-redfix-8 "small blast radius" note.
STATUS steps 3-4 rewritten (two creds, weekly regen, combined remedy + chmod 0750); B-redfix-8/9 corrected.
DONE stands; no VETO; no DoD item touched; rotation of BOTH secrets remains operator-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y9GyBPF1EgTTh6277Xjj7k
Adversary wake #53/#54 reaffirm DONE + no VETO (no gate impact), but both correct STATUS text (Builder-owned):
1. A-redfix-1 "sole copy" withdrawn: 78 world-readable cred-bearing .git/config, sentinel 3fcea78925015fc9.
Exposure confirmed; Adversary's /root/.gitconfig insteadOf root cause FALSIFIED three ways (clone does not
persist insteadOf rewrites — tested, git 2.47.2; /etc/cc-ci config predates .gitconfig by 2wk; live
fetch_recipe uses a non-persisted http.extraHeader token since 9b33fdf, so 0/215 numeric runs carry it).
Real generator: CCCI_SKIP_FETCH copytree of credentialed /root/.abra/recipes (0700) into 0755 run tree.
STATUS steps 3-4 rewritten; filed B-redfix-9 (deferred). Falsification sent via ADVERSARY-INBOX.md.
2. Sweep mechanism: runs deployed /etc/cc-ci (main @ d11f8f5), not origin/main. Verified first-hand.
DONE stands; no VETO; no DoD item touched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y9GyBPF1EgTTh6277Xjj7k
No standing VETO (2x '## VETO' headings are historical: one annotated CLEARED, one is the clearing
record). STATUS:481 CLAIMED is the superseded historical M2 claim. No action available; loop stopped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y9GyBPF1EgTTh6277Xjj7k
Stop condition re-checked from a cold read: STATUS ## DONE, M1+M2 fresh PASS, no standing
VETO (line 553 header is annotated CLEARED), no inbox either direction, Adversary loop
closed out at #49. Loop stays stopped; no DoD item re-run.
Sentinel re-probed this wake: GITEA_PASSWORD still hashes to 3fcea78925015fc9 (unrotated).
A-redfix-1 / B-redfix-8 remain OPEN/HIGH, operator-rotation-only, gating nothing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TRRkTGdWKWoxddF6QXhWfd
Adversary confirmed all three claims and failed to refute "strip userinfo is safe" three ways:
no push through /etc/cc-ci origin; secrets submodule anonymously fetchable so submodule update
survives; nothing auto-pulls the checkout (/root/.git-credentials inert, no helper wired).
Folded those three into STATUS step 3 so the operator sees why the command is safe. The submodule
angle was an inference on my part, not a probe -- the Adversary actually tested it. Recorded in
JOURNAL.
Incidental (not a finding): cc-ci-secrets.git is public but SOPS-encrypted ciphertext.
STATUS/JOURNAL text only. No code. ## DONE stands; M1+M2 PASS stand; no gate reopens.
Adversary wake #45 established A-redfix-1 and B-redfix-8 are the same credential. Re-derived
every claim first-hand before writing it (the A-redfix-2 lesson), incl. value identity:
.git/config userinfo and .testenv GITEA_PASSWORD both hash to 3fcea78925015fc9; control
sha256("")=e3b0c44298fc1c14 rules out the empty-input probe artifact.
Rotation does NOT scrub /etc/cc-ci/.git/config (644 root:root, manual clone per
configuration.nix:7, not nix-generated). Remedy is now a 3-step operator procedure that
strips the userinfo rather than re-embedding the rotated password (re-embedding would
recreate the exposure). Verified the strip is safe: anonymous ls-remote succeeds rc=0.
Blast radius recorded: GITEA_PASSWORD is consumed only by bootstrap-drone-oauth.sh;
recipe-mirror-sync.sh uses an OAuth token. No reason to delay rotation.
STATUS text only. No code. ## DONE stands; M1+M2 PASS stand; no gate reopens.
autonomic-bot
