The first cut of this fix added a team INSERT next to the existing sites INSERT.
That fixed the symptom and kept the cause: a fixture that knows the apps table
layout breaks whenever the app changes it, which is exactly what happened here.
_register_site now calls Plausible.Sites.create/2 through the app release console,
so the app provisions whatever its data model currently requires - including the
team it introduced in v3. Verified against BOTH versions on cc-ci: the identical
expression works on v2.0.0, which has no teams table at all, and on v3.2.1. No
version gate is needed because the fixture no longer depends on the schema.
The HTTP provisioning API (POST /api/v1/sites) would have been first choice, but
it is gated behind a paid plan and answers :upgrade_required on CE. That is
recorded in the code so the next person does not re-derive it.
tests/STYLE.md writes the rule down, along with the others this failure exercised:
gate on version rather than supporting both schemas (old-version tests can just be
deleted - the older version is only exercised through the upgrade tier); correct
the fixture or the wait but never the assertion; assert stored state rather than a
202 ack; size waits from the recipes declared readiness; and read the apps own
telemetry before deciding a test is stale.
Full cold suite against the recipe PR head: level 5 of 5, GREEN.
plausible v3 (community-edition) only ingests events for a site that belongs to
a TEAM. The custom tier registered a site row and nothing else, which was enough
for v2 — under v3 the POST still acks 202 and the row still exists in postgres,
but every event is discarded. ClickHouse records the reason itself in
ingest_counters as dropped_not_found, and events_v2 stays empty, so it presents
as a silent ingestion stall.
Verified on cc-ci against v3.2.1: identical site row with no team ->
dropped_not_found and 0 rows; with a team linked -> buffered and the rows land.
_register_site now provisions a team and links the site, guarded on the schema
actually having teams so it stays a no-op on v2 (the upgrade tier deploys the
older base first).
Separately, the custom health check waited 60s for /api/health. That tier runs
after backup/restore, which disrupts postgres under the app and restarts it, and
v3 boots through sleep 10 + createdb + migrate + cache warmers before health
flips to 200. Widened to 300s, still far inside the recipe HTTP_TIMEOUT of 1200.
The assertion is unchanged: a hard 200 from the real readiness endpoint.
Neither change weakens a test - the event tests still require the row to arrive
in ClickHouse and match what was sent.
Main/driving sessions stay glm-5.2 (set per-launch via --model); the built-in
'general' agent used by the task tool runs opencode/deepseek-v4-pro: ~3-5x cheaper
per token with near-free cache hits, and the weekly /upgrade-all spends most of its
budget in per-recipe subagents.
Placement matters and was got wrong once (2026-08-04 config sat in the orchestrator
repo and never bound; the whole 2026-08-07 run billed as glm): sessions launched by
launch-upgrader.py pass no --dir, so they inherit the opencode serve process's
project (WorkingDirectory=/srv/cc-ci-orch/cc-ci = THIS repo), and task-tool
subagents inherit their parent session's directory. This file is therefore the
project config those subagents actually resolve.
Verified end-to-end 2026-08-10 with the launcher's exact invocation: parent session
modelID=glm-5.2, spawned subagent modelID=deepseek-v4-pro (read back from the
opencode session DB, not from config inspection).
test_head_runs_official_image_not_bitnamilegacy hardcoded the migration-era pin
discourse/discourse:3.5.3 and went stale on the first legitimate app bump
(2026.7.1, weekly 2026-08-03 — caught by verify run 2: the upgrade converged,
head image was discourse/discourse:2026.7.1, only the frozen pin failed). The
guarded property is the image FAMILY (official vs bitnamilegacy), not a frozen
version — now asserts the discourse/discourse: prefix. Not weakened: the
bitnami-leak check + official-prefix check together still assert exactly the
migration faithfulness; the concrete head pin is exercised by the deploy.
The strict recipe_meta key registry rejected the new declaration (caught by
verify run 1). str-typed, default None; full semantics documented on the key.
The 0.8.x->1.0.0 discourse recipe family switched app bitnami->official AND db
pgvector/pg17->discourse/postgres:pg18. That db-family change is a structural
break (bitnami cluster lacks the discourse role; pg_upgrade preserves-not-creates
roles) with NO supported in-place path. The dynamic base resolver's step-back
kept selecting 0.8.1+3.5.0 (newest tag below the unbumped 1.0.0+3.5.3 label) and
the upgrade tier red'd twice on this unsupported path (drone #1165, #1171 —
classified stale-test both times; recipe verified green on the real
official->official path).
Adds UPGRADE_BASE_FLOOR (phase basefloor) to resolve_upgrade_base: a recipe_meta
declaration naming the first post-break published version. Resolution stays
fully dynamic (this is NOT the removed sec2.G static pin): the floor only
EXCLUDES below-floor candidates (canonical, step-back, no-canonical fallback);
when no >=floor predecessor exists the tier records a DECLARED skip, never a
silent pass. main-tip fallback unaffected (post-break by construction).
tests/discourse/recipe_meta.py declares UPGRADE_BASE_FLOOR = 1.0.0+3.5.3 with
the full rationale. Unit-verified: head=1.0.0+3.5.3 -> declared skip (was:
0.8.1+3.5.0 wrong pick); post-release head=2026.x -> base 1.0.0+3.5.3 (the real
migration path). No assertion weakened - below-floor in-place upgrades were
never supported coverage.
The July gitea credential rotation updated the sops value, but ensure_secret is
create-once-immutable, so the service kept mounting cc_ci_bridge_gitea_token_v1
frozen at the pre-rotation token -> HTTP 401 'user does not exist' on every
!testme poll, silently dropping all triggers (found + worked around Drone-direct
during the 2026-08-03 weekly run; a manually-created _v2 existed but nothing
referenced it). Referencing _v3 makes the reconcile unit mint a fresh swarm
secret from the CURRENT /run/secrets/bridge_gitea_token (verified: HTTP 200 as
autonomic-bot) at next deploy.
Enrolls wordpress as a maintained recipe (operator request 2026-08-03):
- tests/wordpress/: recipe_meta (install-wizard-aware health 200/302, 900s deploy
timeout for mariadb+core-copy first boot, WARM_CANONICAL), custom suite:
health check, install-wizard completion + REST API round-trip (?rest_route= vs
/wp-json/ splits DB vs .htaccess failure layers), and the sec4.3 post round-trip
(XML-RPC write -> REST read -> permalink HTML, unique marker). PARITY.md documents
the baseline (no recipe-maintainer parity corpus for wordpress).
- nix/modules/bridge.nix: POLL_REPOS += recipe-maintainers/wordpress (!testme bridge
enrollment; deploy to the cc-ci host follows separately after the in-flight
/upgrade-all run - test-before-switch policy).
Mirror recipe-maintainers/wordpress created + main synced to coopcloud upstream
(adcd0e9f) with published tags. used-recipes.md gains 'wordpress weekly' in the
orchestrator repo.
test_create_room_get_livekit_token_and_read_back authenticated with a raw OIDC user
access token as 'Authorization: Bearer'; meet v1.22.0 hardened API auth to reject
user access tokens (release notes: 'reject user access tokens on the API'), so the
test went RED with 401 on the v1.24.0 upgrade (drone build #1137; same at v1.23.0
in build #1122).
Updated to the successor auth path: recipe-local _oidc_session.py (same helper as
tests/lasuite-docs) drives the real OIDC authorization-code flow (app -> keycloak
login form -> callback -> Django session cookie, CSRF on unsafe methods).
- NEW assertion: a raw OIDC Bearer token is REJECTED (401/403) - the v1.22.0
hardening asserted as the new correct behavior.
- The full meeting flow (create 201 + LiveKit JWT grant, read-back, DELETE) is
unchanged, now over the session-authenticated API. No assertion weakened.
Stale-test fix for recipe PR
recipe-maintainers/lasuite-meet#8
(carry-over from /upgrade-all 2026-07-24).
test_oidc_login_via_keycloak and test_create_doc_and_read_back authenticated with
'Authorization: Bearer <password-grant JWT>'; impress v5.4.0 removed Bearer/JWT
auth on the API (SessionAuthentication only), so both went RED with 401 on the
v5.4.1 upgrade.
Updated to the successor auth path: a new recipe-local _oidc_session.py drives the
real OIDC authorization-code flow (app -> keycloak login form -> callback ->
Django session cookie, with CSRF headers on unsafe methods).
- test_oidc_login: still asserts the unauth challenge redirect; NOW also asserts a
raw Bearer JWT is REJECTED (401/403 - the v5.4.0 hardening, asserted as the new
correct behavior); then asserts the session-authenticated whoami returns the
provisioned user. No assertion weakened - the auth proof is stronger than before.
- test_create_doc: same create+read-back round-trip assertions, now over the
session-authenticated API.
Stale-test fix for recipe PR
recipe-maintainers/lasuite-docs#7
(carry-over from /upgrade-all 2026-07-24).
Since NixOS 25.05, scripted networking installs the default route from the
gateway interface's network-addresses-<iface>.service, matching the interface
via defaultGateway.interface or by subnet inclusion. Hetzner's off-subnet
point-to-point gateway (91.98.47.73/32 on eth0, gw 172.31.1.1) matched
neither with the bare-string form, so the 26.05 switch on 2026-08-03 left the
host with no default route and off the network (recovered via Hetzner rescue:
grubenv default back to the 24.11 generation).
With an explicit interface, the module installs both the gateway host route
and the default route from eth0's own unit:
ip -4 route replace 172.31.1.1 proto static dev eth0
ip -4 route replace default proto static dev eth0 via 172.31.1.1
Verified by nix eval of systemd.services.network-addresses-eth0.script.
nixpkgs: 50ab793 -> 531670d (nixos-26.05, was nixos-24.11 EOL)
sops-nix: 77c423a -> f140661 (master, buildGo125Module now builds on 26.05)
3-release channel jump (24.11 -> 25.05 -> 25.11 -> 26.05) to match the
orchestrator host and get CVE patches. 24.11 was EOL (2025-06-30).
Notable 26.05 changes reviewed: systemd Stage 1 boot (default, no config
change needed), dbus-broker default, bash nixos-rebuild removed (Python
rewrite mandatory; git present on host), MySQL 8.0 removed (not used as a
NixOS module). No system.rebuild/fileSystems/boot.initrd options in the
host config, so no config edits required.
The token embedded in the recipe clone's gitea remote was copytree'd into the
world-readable per-run tree. SSH push keys the auth to ~/.ssh, so no secret is
written into any .git/config. API calls still use the token.
Terminal state re-verified cold. Corrected a near-miss from this wake: my first
`^## .*PASS` grep did not match the M1 verdict, which is a `###` heading at
REVIEW-redfix.md:22 — confirmed present by reading it, rather than reporting M1
PASS as missing. Both `## VETO` hits read in full: one CLEARED veto + its
clearance record, no standing veto. DONE @00:18Z, M1+M2 PASS, HEAD==origin/main,
no inbox. Untracked main.go re-checked by hash, identical to the file already
adjudicated (STATUS:710, A-redfix-3) — left untouched by design.
Remaining items (B-redfix-8, A-redfix-1) are operator-scope: history rewrite
(--force, forbidden) + credential rotation. Loop STOPPED.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWSJEPYEMx9LZhPwyQCmhP
HEAD==origin/main==89af75b is my own wake-#64 journal: neither the Adversary nor the tree has moved
since I last stopped, so there was no handoff to answer. Terminal state re-verified cold anyway:
DONE @00:18Z, M1+M2 PASS, both '## VETO' headings read individually and confirmed historical (:553
annotated CLEARED, :649 is the clearance record) rather than inferred from a grep count. No inbox.
Untracked main.go left in place per the standing adjudication at STATUS:710 / A-redfix-3.
Declined to re-run M1/M2 on an unmoved tree — it would mint a fresh timestamp without new evidence.
Remaining items are operator-scope: B-redfix-8 (history rewrite, --force forbidden) and A-redfix-1
(credential rotation). Loop STOPPED.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015yZh7A6Zcac5LQ3tqqtb9W
Terminal state re-verified cold: DONE @00:18Z, M1+M2 PASS stand, no standing VETO (both '## VETO' hits
read, not counted: one CLEARED + its clearance record), all findings CLOSED, HEAD==origin/main, no inbox.
Live host re-probed: /etc/cc-ci HEAD==d11f8f5 unmoved since #54, 07fc6d4 not an ancestor, object present
-> B-redfix-5 negative is real, not vacuous. Declined to re-run M1/M2 on an unmoved tree.
Prefix is 'chore(' not 'review(' ON PURPOSE: no verdict landed this wake. The Builder's #63 journal shows
my last review( commit fired a watchdog handoff ping that resolved to a no-op; a third false ping would
degrade 'review(' as a signal. Loop STOPPED.
Adversary (a22c384) confirmed both findings but counted 3061 blobs vs my 3099. Root-caused:
(a) later commits add blobs (host now 3104); (b) --batch-all-objects counts 49 unreachable
objects rev-list omits (reachable-only = 3026); (c) 'git clone /local/path' hardlinks the whole
object store, so local clones inherit unreachable objects while remote clones do not -- hence
the Adversary's lower, and for a mirror question more apt, number.
All four scans agree on password=2 and token=0. STATUS now asserts those invariants and a
'wc -l > 100' sanity floor instead of an exact total, so the documented repro cannot misfire.
Operator-scope, no gate impact, no VETO; DONE stands.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018XE43k4DaeMXHMK51wBUu2
Closes the gap in Adversary wake #57's break-it probe, which covered dashboard/reports/Drone
logs but not the public git mirror -- the surface B-redfix-8 is about.
- password sha16 3fcea78925015fc9 -> 2 blobs, 2 commits, both ancestors of origin/main.
Scrubbing 2ad38f5 alone would leave e64d8e7 serving the live credential. STATUS corrected.
- oauth2 token sha16 9c44a1aea2ecb389 -> 0/3099 blobs. Filesystem-only exposure.
- Probe hygiene: awk is absent here; awk-based blob lists give a vacuous 0. Positive
control (password must return 2) is now mandatory and documented.
Operator-scope, no gate impact, no VETO; DONE stands.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018XE43k4DaeMXHMK51wBUu2
Adversary wake #55 conceded my insteadOf falsification but corrected me twice; verified both first-hand:
1. "Production CI does not regenerate / manual-* = hand-run" WRONG. run_id()=manual-<pid> for any non-Drone
run (run_recipe_ci.py:318-319); nightly-sweep runs run_recipe_ci.py outside Drone with CCCI_SKIP_FETCH=1
(nightly_sweep.py:88). Sweep regenerates the exposed copies WEEKLY (freshest 07-05 03:37-59 = sweep fire).
2. Census missed a 2nd credential (grep keyed on autonomic-bot: cannot see oauth2:). Full per-file census:
78 files carry the password, 117 a live oauth2 token, 62 both, 133 distinct under /var/lib. Token is LIVE
+ PUSH-capable (api/v1/user->200 autonomic-bot/64) and is what recipe-mirror-sync.sh:39 pushes with —
falsifies my own B-redfix-8 "small blast radius" note.
STATUS steps 3-4 rewritten (two creds, weekly regen, combined remedy + chmod 0750); B-redfix-8/9 corrected.
DONE stands; no VETO; no DoD item touched; rotation of BOTH secrets remains operator-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y9GyBPF1EgTTh6277Xjj7k
Adversary wake #53/#54 reaffirm DONE + no VETO (no gate impact), but both correct STATUS text (Builder-owned):
1. A-redfix-1 "sole copy" withdrawn: 78 world-readable cred-bearing .git/config, sentinel 3fcea78925015fc9.
Exposure confirmed; Adversary's /root/.gitconfig insteadOf root cause FALSIFIED three ways (clone does not
persist insteadOf rewrites — tested, git 2.47.2; /etc/cc-ci config predates .gitconfig by 2wk; live
fetch_recipe uses a non-persisted http.extraHeader token since 9b33fdf, so 0/215 numeric runs carry it).
Real generator: CCCI_SKIP_FETCH copytree of credentialed /root/.abra/recipes (0700) into 0755 run tree.
STATUS steps 3-4 rewritten; filed B-redfix-9 (deferred). Falsification sent via ADVERSARY-INBOX.md.
2. Sweep mechanism: runs deployed /etc/cc-ci (main @ d11f8f5), not origin/main. Verified first-hand.
DONE stands; no VETO; no DoD item touched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y9GyBPF1EgTTh6277Xjj7k
No standing VETO (2x '## VETO' headings are historical: one annotated CLEARED, one is the clearing
record). STATUS:481 CLAIMED is the superseded historical M2 claim. No action available; loop stopped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y9GyBPF1EgTTh6277Xjj7k
Stop condition re-checked from a cold read: STATUS ## DONE, M1+M2 fresh PASS, no standing
VETO (line 553 header is annotated CLEARED), no inbox either direction, Adversary loop
closed out at #49. Loop stays stopped; no DoD item re-run.
Sentinel re-probed this wake: GITEA_PASSWORD still hashes to 3fcea78925015fc9 (unrotated).
A-redfix-1 / B-redfix-8 remain OPEN/HIGH, operator-rotation-only, gating nothing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TRRkTGdWKWoxddF6QXhWfd
cc-ci
autonomic-bot
autonomic-bot