Compare commits

...
Author SHA1 Message Date
autonomic-bot 10ecb741e7 fix: let acme own wildcard certificate files 2026-08-31 18:59:14 +00:00
autonomic-bot 04e50c7c17 fix: roll out rotated traefik certificate secrets 2026-08-31 18:57:22 +00:00
autonomic-bot 611e16f62d fix: roll out rotated traefik certificate secrets 2026-08-31 18:57:06 +00:00
autonomic-bot f42dbc3f82 feat: promote acme-dns renewal to production 2026-08-31 18:54:09 +00:00
autonomic-bot 1415cc53c6 feat: promote acme-dns renewal to production 2026-08-31 18:53:52 +00:00
autonomic-bot 12dee8bf75 fix: serve acme-dns nameserver address 2026-08-31 17:15:33 +00:00
autonomic-bot 8de2b125e9 fix: serve acme-dns nameserver address 2026-08-31 17:15:21 +00:00
autonomic-bot 1c70b9e61a harden: disable acme-dns registration 2026-08-31 17:14:15 +00:00
autonomic-bot b7bf41057a harden: disable acme-dns registration 2026-08-31 17:14:03 +00:00
autonomic-bot 1c2d5e9f7f fix: use acme-dns sqlite backend 2026-08-31 17:11:52 +00:00
autonomic-bot f6e977c69e fix: use acme-dns sqlite backend 2026-08-31 17:11:34 +00:00
autonomic-bot 0db8194dd5 feat: prepare restricted acme-dns renewal 2026-08-31 17:09:17 +00:00
autonomic-bot 148d4c9381 feat: prepare restricted acme-dns renewal 2026-08-31 17:08:15 +00:00
autonomic-bot cb315f8ab4 Merge pull request 'plan: add restricted acme-dns renewal design' (#23) from plan/acme-dns-renewal into main 2026-08-31 17:00:39 +00:00
autonomic-bot c24bd0c62c plan: add restricted acme-dns renewal design 2026-08-31 17:00:16 +00:00
autonomic-bot 5380997543 Merge pull request 'config: track public cc-ci runtime environment' (#22) from chore/public-runtime-config into main 2026-08-31 16:49:51 +00:00
autonomic-bot 4176b48a7b config: track public cc-ci runtime environment 2026-08-31 16:49:14 +00:00
autonomic-bot c0b473328d Merge pull request 'config: subagents -> opencode/deepseek-v4-flash (zen, not tinfoil)' (#21) from config/deepseek-flash-zen into main
continuous-integration/drone/push Build is failing
continuous-integration/drone Build is passing
2026-08-16 02:40:17 +00:00
autonomic-bot 1e0accbda7 config: subagents -> opencode/deepseek-v4-flash (zen endpoint, not tinfoil)
continuous-integration/drone/push Build is failing
Route deepseek-v4-flash through the opencode zen subscription endpoint
(opencode/deepseek-v4-flash) instead of tinfoil pay-per-use. The zen
subscription is already paid for; flash consumes the balance far more
slowly than glm-5.2 did (flash cost ~$0.0000002/tok vs glm's higher rate),
so a full weekly run is much less likely to exhaust the balance — and
the operator has reset it.

Completes the fleet-wide move: upgrader parent (LOOP_MODEL) + report
(REPORT_MODEL) in upgrader.env, and this PR sets the subagents. Supervisor
stays on opencode-go/glm-5.2.

Verified: zen endpoint serves deepseek-v4-flash (HTTP 200, 1.2s).
2026-08-16 02:40:06 +00:00
autonomic-bot 5083c51430 Merge pull request 'config: subagents -> tinfoil/deepseek-v4-flash (cheaper, pay-per-use)' (#20) from config/deepseek-flash-subagents into main
continuous-integration/drone/push Build is failing
2026-08-16 02:34:32 +00:00
autonomic-bot 65063efdaa config: subagents -> tinfoil/deepseek-v4-flash (cheaper, pay-per-use)
continuous-integration/drone/push Build is failing
Switch the general subagent model from opencode/deepseek-v4-pro (zen
endpoint, subject to workspace balance limits) to tinfoil/deepseek-v4-flash
(pay-per-use API, no rolling balance limit, cheaper than pro).

This completes the fleet-wide move to deepseek-flash on tinfoil: the upgrader
parent (LOOP_MODEL) and report (REPORT_MODEL) are set in upgrader.env, and
this changes the subagents. The hourly supervisor stays on glm-5.2.

Verified: tinfoil endpoint accepts deepseek-v4-flash (HTTP 200, 2.3s response).
Trial run next Friday (2026-08-21) to evaluate flash capability on the
upgrade workload; fall back to pro if it struggles on complex recipes.
2026-08-16 02:34:21 +00:00
autonomic-bot b1c9ec1464 upstream(gitea): release-notes sources
continuous-integration/drone/push Build is failing
continuous-integration/drone Build is passing
2026-08-14 02:35:20 +00:00
autonomic-bot a3e63660f3 Merge pull request 'AGENTS.md: ship work as PRs, self-merge, operator reviews retrospectively' (#19) from policy/pr-then-merge into main
continuous-integration/drone/push Build is failing
continuous-integration/drone Build is passing
2026-08-11 19:09:26 +00:00
cc-ci de658cf40a AGENTS.md: ship work as PRs, self-merge, operator reviews retrospectively
continuous-integration/drone/push Build is failing
Operator policy (2026-08-11), matching cc-ci-orchestrator. Branch, PR, merge once
verified, operator reviews after. The PR is not a gate - it is how the work stays
legible - so the description carries what changed, why, and the evidence.

Recipe repos are explicitly excluded: created and verified, never agent-merged.
2026-08-11 19:09:12 +00:00
autonomic-bot 92ac9a4a4a Merge pull request 'style(plausible): ruff format the rewritten event-tracking fixture' (#18) from fix/plausible-format into main
continuous-integration/drone/push Build is failing
continuous-integration/drone Build is passing
2026-08-11 14:59:05 +00:00
cc-ci 4bc92c44eb style(plausible): ruff format the rewritten event-tracking fixture
continuous-integration/drone/push Build is failing
Self-inflicted: the rewrite in eecc4aa left the file unformatted, which the push
lint gate flags. Formatting only - no behaviour change.

(Note for the operator: the gate is red on main for unrelated reasons - 90 other
files also fail ruff format, and tests/unit/test_f211_sso_skip.py fails ruff
check C420. Neither is touched here; both predate this branch.)
2026-08-11 14:58:51 +00:00
autonomic-bot 8aa21356af Merge pull request 'fix(plausible): register a team so v3 ingests events; widen post-restore health wait' (#17) from fix/plausible-v3-custom-tests into main
continuous-integration/drone/push Build is failing
continuous-integration/drone Build is failing
Verified GREEN (level 5/5) on cc-ci against plausible PR head 867ebfaf, twice: once with the SQL fixture, once with the app-native rewrite. Regression sample green.
2026-08-11 14:52:06 +00:00
cc-ci eecc4aaa51 test(plausible): provision the site through the app, not SQL; add tests/STYLE.md
continuous-integration/drone/push Build is failing
The first cut of this fix added a team INSERT next to the existing sites INSERT.
That fixed the symptom and kept the cause: a fixture that knows the apps table
layout breaks whenever the app changes it, which is exactly what happened here.

_register_site now calls Plausible.Sites.create/2 through the app release console,
so the app provisions whatever its data model currently requires - including the
team it introduced in v3. Verified against BOTH versions on cc-ci: the identical
expression works on v2.0.0, which has no teams table at all, and on v3.2.1. No
version gate is needed because the fixture no longer depends on the schema.

The HTTP provisioning API (POST /api/v1/sites) would have been first choice, but
it is gated behind a paid plan and answers :upgrade_required on CE. That is
recorded in the code so the next person does not re-derive it.

tests/STYLE.md writes the rule down, along with the others this failure exercised:
gate on version rather than supporting both schemas (old-version tests can just be
deleted - the older version is only exercised through the upgrade tier); correct
the fixture or the wait but never the assertion; assert stored state rather than a
202 ack; size waits from the recipes declared readiness; and read the apps own
telemetry before deciding a test is stale.

Full cold suite against the recipe PR head: level 5 of 5, GREEN.
2026-08-11 14:50:41 +00:00
cc-ci eb1d6d9161 fix(plausible): register a team so v3 ingests events; widen post-restore health wait
continuous-integration/drone/push Build is failing
plausible v3 (community-edition) only ingests events for a site that belongs to
a TEAM. The custom tier registered a site row and nothing else, which was enough
for v2 — under v3 the POST still acks 202 and the row still exists in postgres,
but every event is discarded. ClickHouse records the reason itself in
ingest_counters as dropped_not_found, and events_v2 stays empty, so it presents
as a silent ingestion stall.

Verified on cc-ci against v3.2.1: identical site row with no team ->
dropped_not_found and 0 rows; with a team linked -> buffered and the rows land.

_register_site now provisions a team and links the site, guarded on the schema
actually having teams so it stays a no-op on v2 (the upgrade tier deploys the
older base first).

Separately, the custom health check waited 60s for /api/health. That tier runs
after backup/restore, which disrupts postgres under the app and restarts it, and
v3 boots through sleep 10 + createdb + migrate + cache warmers before health
flips to 200. Widened to 300s, still far inside the recipe HTTP_TIMEOUT of 1200.
The assertion is unchanged: a hard 200 from the real readiness endpoint.

Neither change weakens a test - the event tests still require the row to arrive
in ClickHouse and match what was sent.
2026-08-11 05:18:18 +00:00
autonomic-bot 0a229ac016 opencode: run task-tool subagents on deepseek-v4-pro
continuous-integration/drone/push Build is failing
continuous-integration/drone Build is passing
Main/driving sessions stay glm-5.2 (set per-launch via --model); the built-in
'general' agent used by the task tool runs opencode/deepseek-v4-pro: ~3-5x cheaper
per token with near-free cache hits, and the weekly /upgrade-all spends most of its
budget in per-recipe subagents.

Placement matters and was got wrong once (2026-08-04 config sat in the orchestrator
repo and never bound; the whole 2026-08-07 run billed as glm): sessions launched by
launch-upgrader.py pass no --dir, so they inherit the opencode serve process's
project (WorkingDirectory=/srv/cc-ci-orch/cc-ci = THIS repo), and task-tool
subagents inherit their parent session's directory. This file is therefore the
project config those subagents actually resolve.

Verified end-to-end 2026-08-10 with the launcher's exact invocation: parent session
modelID=glm-5.2, spawned subagent modelID=deepseek-v4-pro (read back from the
opencode session DB, not from config inspection).
2026-08-10 15:57:04 +00:00
autonomic-bot de1eb1ca75 Merge pull request 'test(discourse): UPGRADE_BASE_FLOOR — exclude structurally-invalid upgrade bases' (#15) from test/discourse-upgrade-base-floor-20260804 into main
continuous-integration/drone/push Build is failing
continuous-integration/drone Build is failing
2026-08-04 17:57:46 +00:00
autonomic-bot 877aea3814 test(discourse): version-agnostic official-image assertion
test_head_runs_official_image_not_bitnamilegacy hardcoded the migration-era pin
discourse/discourse:3.5.3 and went stale on the first legitimate app bump
(2026.7.1, weekly 2026-08-03 — caught by verify run 2: the upgrade converged,
head image was discourse/discourse:2026.7.1, only the frozen pin failed). The
guarded property is the image FAMILY (official vs bitnamilegacy), not a frozen
version — now asserts the discourse/discourse: prefix. Not weakened: the
bitnami-leak check + official-prefix check together still assert exactly the
migration faithfulness; the concrete head pin is exercised by the deploy.
2026-08-04 17:46:33 +00:00
autonomic-bot ae40545491 meta: register UPGRADE_BASE_FLOOR key (phase basefloor)
continuous-integration/drone/push Build is failing
The strict recipe_meta key registry rejected the new declaration (caught by
verify run 1). str-typed, default None; full semantics documented on the key.
2026-08-04 17:37:02 +00:00
autonomic-bot 5086b2f8bb test(discourse): UPGRADE_BASE_FLOOR — exclude structurally-invalid upgrade bases
continuous-integration/drone/push Build is failing
The 0.8.x->1.0.0 discourse recipe family switched app bitnami->official AND db
pgvector/pg17->discourse/postgres:pg18. That db-family change is a structural
break (bitnami cluster lacks the discourse role; pg_upgrade preserves-not-creates
roles) with NO supported in-place path. The dynamic base resolver's step-back
kept selecting 0.8.1+3.5.0 (newest tag below the unbumped 1.0.0+3.5.3 label) and
the upgrade tier red'd twice on this unsupported path (drone #1165, #1171 —
classified stale-test both times; recipe verified green on the real
official->official path).

Adds UPGRADE_BASE_FLOOR (phase basefloor) to resolve_upgrade_base: a recipe_meta
declaration naming the first post-break published version. Resolution stays
fully dynamic (this is NOT the removed sec2.G static pin): the floor only
EXCLUDES below-floor candidates (canonical, step-back, no-canonical fallback);
when no >=floor predecessor exists the tier records a DECLARED skip, never a
silent pass. main-tip fallback unaffected (post-break by construction).

tests/discourse/recipe_meta.py declares UPGRADE_BASE_FLOOR = 1.0.0+3.5.3 with
the full rationale. Unit-verified: head=1.0.0+3.5.3 -> declared skip (was:
0.8.1+3.5.0 wrong pick); post-release head=2026.x -> base 1.0.0+3.5.3 (the real
migration path). No assertion weakened - below-floor in-place upgrades were
never supported coverage.
2026-08-04 17:15:37 +00:00
autonomic-bot 5327a24faa Merge pull request 'enroll(wordpress): test suite + bridge POLL_REPOS entry' (#14) from test/wordpress-enroll-20260803 into main
continuous-integration/drone/push Build is failing
continuous-integration/drone Build is passing
2026-08-04 17:01:35 +00:00
autonomic-bot f5c97117d6 bridge: bump gitea-token swarm secret to v3 (fix silent !testme drop)
continuous-integration/drone/push Build is failing
The July gitea credential rotation updated the sops value, but ensure_secret is
create-once-immutable, so the service kept mounting cc_ci_bridge_gitea_token_v1
frozen at the pre-rotation token -> HTTP 401 'user does not exist' on every
!testme poll, silently dropping all triggers (found + worked around Drone-direct
during the 2026-08-03 weekly run; a manually-created _v2 existed but nothing
referenced it). Referencing _v3 makes the reconcile unit mint a fresh swarm
secret from the CURRENT /run/secrets/bridge_gitea_token (verified: HTTP 200 as
autonomic-bot) at next deploy.
2026-08-04 16:55:59 +00:00
autonomic-bot d9a446cd36 enroll(wordpress): test suite + bridge POLL_REPOS entry
continuous-integration/drone/push Build is failing
Enrolls wordpress as a maintained recipe (operator request 2026-08-03):
- tests/wordpress/: recipe_meta (install-wizard-aware health 200/302, 900s deploy
  timeout for mariadb+core-copy first boot, WARM_CANONICAL), custom suite:
  health check, install-wizard completion + REST API round-trip (?rest_route= vs
  /wp-json/ splits DB vs .htaccess failure layers), and the sec4.3 post round-trip
  (XML-RPC write -> REST read -> permalink HTML, unique marker). PARITY.md documents
  the baseline (no recipe-maintainer parity corpus for wordpress).
- nix/modules/bridge.nix: POLL_REPOS += recipe-maintainers/wordpress (!testme bridge
  enrollment; deploy to the cc-ci host follows separately after the in-flight
  /upgrade-all run - test-before-switch policy).

Mirror recipe-maintainers/wordpress created + main synced to coopcloud upstream
(adcd0e9f) with published tags. used-recipes.md gains 'wordpress weekly' in the
orchestrator repo.
2026-08-03 21:05:33 +00:00
autonomic-bot 04ae8f55c8 Merge pull request 'test(lasuite-meet): update stale meeting-flow test for meet v1.22.0+ API auth hardening' (#13) from test/lasuite-meet-stale-test-20260803 into main
continuous-integration/drone/push Build is failing
continuous-integration/drone Build is passing
2026-08-03 20:50:37 +00:00
autonomic-bot 304b1610b5 Merge pull request 'test(lasuite-docs): update stale OIDC tests for impress v5.4.0 Bearer-auth removal' (#12) from test/lasuite-docs-stale-test-20260803 into main
continuous-integration/drone/push Build is failing
2026-08-03 20:47:50 +00:00
autonomic-bot 972f5ec4ad test(lasuite-meet): update stale meeting-flow test for meet v1.22.0+ API auth hardening
continuous-integration/drone/push Build is failing
test_create_room_get_livekit_token_and_read_back authenticated with a raw OIDC user
access token as 'Authorization: Bearer'; meet v1.22.0 hardened API auth to reject
user access tokens (release notes: 'reject user access tokens on the API'), so the
test went RED with 401 on the v1.24.0 upgrade (drone build #1137; same at v1.23.0
in build #1122).

Updated to the successor auth path: recipe-local _oidc_session.py (same helper as
tests/lasuite-docs) drives the real OIDC authorization-code flow (app -> keycloak
login form -> callback -> Django session cookie, CSRF on unsafe methods).
- NEW assertion: a raw OIDC Bearer token is REJECTED (401/403) - the v1.22.0
  hardening asserted as the new correct behavior.
- The full meeting flow (create 201 + LiveKit JWT grant, read-back, DELETE) is
  unchanged, now over the session-authenticated API. No assertion weakened.

Stale-test fix for recipe PR
recipe-maintainers/lasuite-meet#8
(carry-over from /upgrade-all 2026-07-24).
2026-08-03 20:45:37 +00:00
28 changed files with 1338 additions and 87 deletions
+7
View File
@@ -0,0 +1,7 @@
# Non-sensitive runtime configuration shared by the cc-ci orchestrator and agents.
#
# Keep credentials, tokens, and keys in /srv/cc-ci/.testenv. The orchestrator
# loads this file first via cc-ci-plan/load-env.sh.
GITEA_USERNAME=autonomic-bot
TINFOIL_MODEL=deepseek-v4-pro
TINFOIL_BASE_URL=https://inference.tinfoil.sh/v1
+16
View File
@@ -36,3 +36,19 @@ Two kinds of tests live here — run them on **different** cadences:
A red test is information. Never skip, delete, or relax a test to make a run green — fix the root
cause or record it in `machine-docs/DEFERRED.md`. (This is a standing build guardrail.)
## Ship work as PRs, merge them yourself, operator reviews retrospectively
Work on this repo goes: **branch → PR → merge it yourself once verified → operator reviews
retrospectively.** Do not commit straight to `main`, and do not wait for review before merging — the
invocation is the authorization, and blocking would stall the CI this repo runs.
The PR is therefore not a gate; it is how the work stays legible after the fact. Write the
description to be read later: what changed, why, and the evidence it works (harness output, a
verified run, a before/after number). A PR that says "fix test" has failed at its only job.
The same policy covers `recipe-maintainers/cc-ci-orchestrator`. It does **NOT** cover recipe repos —
any `coop-cloud/<recipe>` or its mirror is created and verified but **never agent-merged**, because
those change what deploys on other people's infrastructure.
Before editing a test, read `tests/STYLE.md`.
+19
View File
@@ -0,0 +1,19 @@
# gitea upstream sources
## gitea/gitea
- image: gitea/gitea
- source: https://github.com/go-gitea/gitea
- releases: https://github.com/go-gitea/gitea/releases
- security: https://blog.gitea.com/
## postgres
- image: postgres
- source: https://github.com/postgres/postgres
- releases: https://www.postgresql.org/docs/release/
- security: https://www.postgresql.org/support/security/
## mariadb
- image: mariadb
- source: https://github.com/MariaDB/server
- releases: https://mariadb.com/kb/en/release-notes/
- security: https://mariadb.com/kb/en/security/
+283
View File
@@ -0,0 +1,283 @@
# Plan: restricted ACME DNS renewal for cc-ci
## Outcome
Replace the manually issued, sops-stored wildcard certificate with unattended
DNS-01 renewal for these exact names:
```text
ci.commoninternet.net
*.ci.commoninternet.net
```
The cc-ci host will run an authoritative `acme-dns` instance only for
`acme.commoninternet.net`. It will never receive a Gandi credential or any
credential that can edit the parent `commoninternet.net` zone.
The only enduring delegation from the parent zone is:
```text
_acme-challenge.ci.commoninternet.net. CNAME <account-id>.acme.commoninternet.net.
```
That CNAME authorizes the generated acme-dns account to answer ACME TXT
challenges for the ci wildcard, not to edit any parent-zone DNS record.
## Project facts and constraints
- The target is the production `cc-ci-hetzner` NixOS 26.05 host, not the
orchestrator. Its public IPv4 is `91.98.47.73`; it has no public IPv6.
- The wildcard currently points at the public gateway, which TLS-passthroughs
to cc-ci's Traefik. DNS authority for `acme.commoninternet.net` must point
directly to `91.98.47.73`; the gateway is not involved in DNS.
- Nothing listens on TCP or UDP 53 today. The Nix firewall permits 22, 80, and
443 only; any Hetzner Cloud firewall must also be checked before deployment.
- TLS terminates in the Docker Swarm Traefik service. It currently reads
`ssl_cert` and `ssl_key` **Swarm secrets** populated from
`/var/lib/ci-certs/live/{fullchain.pem,privkey.pem}` by
`runner/warm_reconcile.py`. A normal host-service reload cannot install a
renewed certificate.
- The existing certificate is expired: its served validity ended
`2026-08-24 18:18:52 UTC`. Keep the current files as rollback material until
the new production certificate and Traefik rotation have both been verified.
- `pkgs.acme-dns` and `pkgs.lego` are available in the pinned nixpkgs. NixOS
`security.acme` uses Lego and supports a DNS provider plus an environment
file and post-renew hook. Confirm the pinned provider spelling with
`lego --help` during implementation; Lego's current documented provider code
is `acmedns`.
## Security invariants
1. Do not request, add, store, or use `GANDI_API_KEY`, a Gandi PAT, or any
parent-zone update credential on cc-ci or the orchestrator.
2. Bind the acme-dns HTTP API to `127.0.0.1` only. Its API may use plain HTTP
because it is loopback-only; do not create a circular API TLS dependency.
3. Allow public DNS only on TCP/UDP 53 and only for the authoritative zone.
4. The generated acme-dns account data is a secret. Keep it as a root/acme-only
persistent state file under `/var/lib/acme/`; never put it in Nix text, the
Nix store, git, `.env.public`, or a log.
5. After the account exists, set `disable_registration = true`. The existing
account must still be able to call `/update`.
6. Limit the account's update source with `ACME_DNS_ALLOWLIST=127.0.0.1/32`.
This is defence in depth in addition to the loopback API binding.
## Intended DNS design
Use an **out-of-bailiwick** nameserver name to avoid in-bailiwick glue
ambiguity:
```text
ns-acme.commoninternet.net. A 91.98.47.73
acme.commoninternet.net. NS ns-acme.commoninternet.net.
```
`acme-dns` itself serves the delegated zone and returns its matching NS record:
```text
acme.commoninternet.net. NS ns-acme.commoninternet.net.
```
This host is authoritative for `acme.commoninternet.net` and its generated
children only. It is not authoritative for `ci.commoninternet.net` or for
`commoninternet.net`.
## Implementation phases
### 1. Preflight and safety checks
Before changing Nix configuration, record:
```bash
ssh cc-ci 'ss -lntup "( sport = :53 )"'
ssh cc-ci 'systemctl list-units --type=service --all "*acme*" "*dns*"'
ssh cc-ci 'nft list ruleset'
ssh cc-ci 'docker service ls'
```
Confirm that no service owns port 53, that the Traefik Swarm services are
healthy, and that the Hetzner Cloud firewall will permit both 53/tcp and
53/udp. Do not replace an existing DNS service.
Obtain the operator's ACME contact email before enabling `security.acme`.
### 2. Add a dedicated acme-dns Nix module
Create `nix/modules/acme-dns.nix` and import it from
`nix/hosts/cc-ci-hetzner/configuration.nix`. The module should:
- create a dedicated unprivileged `acme-dns` user and group;
- run `${pkgs.acme-dns}/bin/acme-dns -c <public generated config>` with a
persistent working/state directory `/var/lib/acme-dns`;
- grant only `CAP_NET_BIND_SERVICE` to bind DNS port 53;
- use SQLite at `/var/lib/acme-dns/acme-dns.db` with mode `0600`;
- bind DNS to `91.98.47.73:53` with `protocol = "both4"`;
- set `domain = "acme.commoninternet.net"`,
`nsname = "ns-acme.commoninternet.net"`, and a public hostmaster-style
`nsadmin` value;
- include the public NS record above in `general.records`;
- bind `[api]` to `127.0.0.1:8080`, set `tls = "none"`, use a restrictive
CORS list, and initially leave `disable_registration = false`;
- use a hardened systemd unit: `NoNewPrivileges`, `PrivateTmp`,
`ProtectSystem = "strict"`, `ProtectHome`, `PrivateDevices`, and only the
state directory as writable; and
- open `networking.firewall.allowedTCPPorts = [ 53 ]` and
`allowedUDPPorts = [ 53 ]` in the **cc-ci Hetzner host** configuration.
The configuration file is public data and may be generated by Nix. It must not
contain account credentials.
Deploy this phase with the normal cc-ci deployment discipline: first
`nixos-rebuild test --flake /etc/cc-ci#cc-ci-hetzner`, verify SSH, Traefik, and
the host remain healthy, then run the identical `switch` target. Verify local
DNS on both transports:
```bash
dig @91.98.47.73 acme.commoninternet.net NS
dig +tcp @91.98.47.73 acme.commoninternet.net NS
```
### 3. Operator gate: delegate the narrow DNS zone
After the service is healthy, ask the operator to add exactly these records at
Gandi (using its DNS UI, never a token on this host):
```dns
ns-acme.commoninternet.net. A 91.98.47.73
acme.commoninternet.net. NS ns-acme.commoninternet.net.
```
If Gandi models delegation as a nameserver/glue form rather than ordinary zone
records, use its equivalent UI flow. Do not proceed until public recursive DNS
shows the delegation and direct queries work from an external network:
```bash
dig NS acme.commoninternet.net @1.1.1.1
dig TXT test.acme.commoninternet.net @91.98.47.73
dig +tcp TXT test.acme.commoninternet.net @91.98.47.73
```
### 4. Configure NixOS ACME in staging mode and obtain the account target
Extend the new module with one `security.acme.certs` entry for the base name
`ci.commoninternet.net`:
```nix
{
domain = "ci.commoninternet.net";
extraDomainNames = [ "*.ci.commoninternet.net" ];
dnsProvider = "acmedns"; # verify against the pinned Lego binary
environmentFile = "/etc/acme-dns/lego.env";
dnsResolver = "1.1.1.1:53";
}
```
`/etc/acme-dns/lego.env` contains only non-secret wiring:
```text
ACME_DNS_API_BASE=http://127.0.0.1:8080
ACME_DNS_STORAGE_PATH=/var/lib/acme/ci.commoninternet.net/acme-dns-accounts.json
ACME_DNS_ALLOWLIST=127.0.0.1/32
```
Lego registers and persists its per-domain acme-dns account in the storage
path. The path is writable only by the ACME service user and is not Nix-managed
content. Do not hand-create its JSON: let the pinned Lego provider establish
the account format.
Set the ACME CA to Let's Encrypt staging for this phase. Start the certificate
unit manually after the NS delegation is confirmed. The first staging run is
expected to create the account and may fail validation because the CNAME is not
yet present. Read the storage file only with a root-only helper that prints the
generated **fulldomain** and never its username or password.
### 5. Operator gate: permanent challenge CNAME
Ask the operator to create the exact target reported in phase 4:
```dns
_acme-challenge.ci.commoninternet.net. CNAME <generated-id>.acme.commoninternet.net.
```
This is a permanent record. It must not be created, changed, or removed by an
agent. Confirm the complete chain through a public recursive resolver before
continuing:
```bash
dig CNAME _acme-challenge.ci.commoninternet.net @1.1.1.1
dig TXT <generated-id>.acme.commoninternet.net @91.98.47.73
dig +tcp TXT <generated-id>.acme.commoninternet.net @91.98.47.73
```
### 6. Staging issuance, then production issuance
Run the NixOS ACME certificate unit against staging and verify all of the
following:
1. it updates only the generated acme-dns TXT target;
2. public recursive DNS sees the CNAME and the TXT value;
3. staging issues a certificate containing both requested names; and
4. no Gandi variable, credential file, or API request appears in the unit.
Only then select the production Let's Encrypt directory and issue the real
certificate. Keep the old sops certificate live during both attempts.
### 7. Make Traefik consume renewals safely
Do **not** use only `reloadServices`: Traefik receives Docker Swarm secrets and
cannot see an updated host file. Add a root-only renewal handoff service,
serialized with all other Traefik reconciliation, and call it from the ACME
certificate's `postRun` hook.
The handoff must:
1. atomically copy the new `fullchain.pem` and key from the NixOS ACME output
into `/var/lib/ci-certs/live`, with the existing `0444`/`0400` modes;
2. generate a new, content-derived **non-secret** Swarm secret version;
3. insert new `ssl_cert` and `ssl_key` Swarm secrets, update the Traefik recipe
environment to reference those versions, and reconcile/redeploy Traefik;
4. health-check `https://traefik.ci.commoninternet.net/api/version` with SNI;
5. retain the prior secret version until the new task is healthy, then remove
it; and
6. record a failure clearly without deleting the last-known-good certificate.
Implement this as a tested extension of `runner/warm_reconcile.py` (or a
small, explicitly locked companion) rather than an ad-hoc shell command. The
renewal path and the normal `deploy-proxy` path must share a lock so they cannot
race over Swarm secret versions.
After production issuance and a successful Traefik rotation, remove the
`wildcard_cert` and `wildcard_key` sops declarations from
`nix/modules/secrets.nix`; otherwise later Nix activations would overwrite the
renewed host files. Remove the obsolete encrypted values from the private
`cc-ci-secrets` repository only after rollback is no longer needed.
### 8. Lock registration and prove unattended renewal
In a follow-up Nix change, set `api.disable_registration = true`, test that the
existing account can still update its TXT record, and confirm `/register` is
rejected. Then verify:
```bash
systemctl list-timers 'acme-*'
systemctl start acme-ci.commoninternet.net.service
journalctl -u acme-ci.commoninternet.net.service -b
```
Perform a controlled staging renewal after registration is disabled, observe
the renewed Traefik secret version, and confirm the certificate served through
the gateway has the expected names and a new validity window.
## Final acceptance checklist
- [ ] cc-ci and the orchestrator contain no Gandi API credential.
- [ ] Gandi delegates only `acme.commoninternet.net` to cc-ci.
- [ ] Only `_acme-challenge.ci.commoninternet.net` CNAMEs into that zone.
- [ ] The acme-dns API is loopback-only; only 53/tcp and 53/udp are public.
- [ ] External UDP and TCP authoritative DNS checks pass.
- [ ] Registration is disabled after the one account is created.
- [ ] The ACME account can update only its generated TXT record.
- [ ] The certificate covers both `ci.commoninternet.net` and its wildcard.
- [ ] A renewal rotates Traefik's Swarm secrets and preserves a working prior
version until the replacement passes health checks.
- [ ] No credential or private key has been committed, logged, or written into
the Nix store.
+15
View File
@@ -0,0 +1,15 @@
# Public runtime environment
`.env.public` contains non-sensitive configuration that the cc-ci orchestrator
and its agent sessions need at runtime. It is intentionally tracked so it can
be inspected and reproduced with the rest of the CI configuration.
Load it together with the local secret file by sourcing
`/srv/cc-ci/cc-ci-plan/load-env.sh`. The helper reads `.env.public` first and
then `/srv/cc-ci/.testenv`; credentials, tokens, and keys belong only in the
latter file.
Do not put a value in `.env.public` merely because it is convenient. If it
would grant access or require rotation, it is a secret and belongs in
`.testenv`. Public service endpoints, model names, and account identifiers may
be tracked here.
@@ -14,6 +14,7 @@
./networking.nix
../../modules/packages.nix
../../modules/secrets.nix
../../modules/acme-dns.nix
../../modules/swarm.nix
../../modules/docker-prune.nix
../../modules/abra.nix
+155
View File
@@ -0,0 +1,155 @@
# Restricted DNS-01 certificate issuance for ci.commoninternet.net.
#
# This host is authoritative only for acme.commoninternet.net. Gandi continues
# to own commoninternet.net; it delegates this narrow zone and one permanent
# _acme-challenge CNAME manually. No Gandi credential is present here.
{ pkgs, ... }:
let
acmeDnsConfig = pkgs.writeText "cc-ci-acme-dns.conf" ''
[general]
listen = "91.98.47.73:53"
protocol = "both4"
domain = "acme.commoninternet.net"
nsname = "ns-acme.commoninternet.net"
nsadmin = "hostmaster.commoninternet.net"
records = [
"acme.commoninternet.net. NS ns-acme.commoninternet.net.",
"ns-acme.commoninternet.net. A 91.98.47.73",
]
debug = false
[database]
# acme-dns 2.x registers the embedded driver under `sqlite` (not the
# legacy `sqlite3` identifier).
engine = "sqlite"
connection = "/var/lib/acme-dns/acme-dns.db"
[api]
ip = "127.0.0.1"
port = "8080"
tls = "none"
# The one Lego account was bootstrapped before this configuration was
# hardened. Updates authenticated by that account remain available.
disable_registration = true
corsorigins = []
[logconfig]
loglevel = "info"
logtype = "stdout"
logformat = "json"
'';
# These are wiring values only. The acme-dns account JSON is generated by
# Lego below /var/lib/acme and never enters Nix, git, or /etc.
legoEnvironment = pkgs.writeText "cc-ci-acme-dns-lego.env" ''
ACME_DNS_API_BASE=http://127.0.0.1:8080
ACME_DNS_STORAGE_PATH=/var/lib/acme/ci.commoninternet.net/acme-dns-accounts.json
ACME_DNS_ALLOWLIST=127.0.0.1/32
'';
in
{
users.groups.acme-dns = { };
users.users.acme-dns = {
isSystemUser = true;
group = "acme-dns";
home = "/var/lib/acme-dns";
};
environment.etc."acme-dns/lego.env".source = legoEnvironment;
# The staging order has completed successfully. This marker permits the
# production ACME post-run hook to hand a renewed certificate to Traefik.
systemd.tmpfiles.rules = [
"f /var/lib/ci-certs/acme-production-enabled 0600 root root -"
];
networking.firewall = {
allowedTCPPorts = [ 53 ];
allowedUDPPorts = [ 53 ];
};
systemd.services.acme-dns = {
description = "Restricted authoritative DNS for cc-ci ACME DNS-01";
wantedBy = [ "multi-user.target" ];
after = [ "network-online.target" ];
wants = [ "network-online.target" ];
serviceConfig = {
User = "acme-dns";
Group = "acme-dns";
StateDirectory = "acme-dns";
StateDirectoryMode = "0700";
WorkingDirectory = "/var/lib/acme-dns";
ExecStart = "${pkgs.acme-dns}/bin/acme-dns -c ${acmeDnsConfig}";
Restart = "on-failure";
RestartSec = "5s";
AmbientCapabilities = [ "CAP_NET_BIND_SERVICE" ];
CapabilityBoundingSet = [ "CAP_NET_BIND_SERVICE" ];
NoNewPrivileges = true;
PrivateTmp = true;
PrivateDevices = true;
ProtectHome = true;
ProtectSystem = "strict";
ReadWritePaths = [ "/var/lib/acme-dns" ];
RestrictAddressFamilies = [ "AF_INET" "AF_UNIX" ];
};
};
# Traefik consumes its wildcard as immutable Swarm secrets, so a renewed
# host certificate must be copied and reconciled rather than merely reloaded.
# This service is started only by the production-mode ACME postRun hook.
systemd.services.cc-ci-acme-traefik-handoff = {
description = "Install renewed cc-ci wildcard into Traefik Swarm secrets";
after = [ "docker.service" "deploy-proxy.service" ];
requires = [ "docker.service" ];
path = [ pkgs.coreutils pkgs.docker pkgs.systemd pkgs.gnugrep ];
serviceConfig = {
Type = "oneshot";
UMask = "0077";
};
script = ''
src=/var/lib/acme/ci.commoninternet.net
dst=/var/lib/ci-certs/live
test -s "$src/fullchain.pem"
test -s "$src/key.pem"
install -d -m 0700 "$dst"
install -m 0444 "$src/fullchain.pem" "$dst/fullchain.pem.new"
install -m 0400 "$src/key.pem" "$dst/privkey.pem.new"
mv -f "$dst/fullchain.pem.new" "$dst/fullchain.pem"
mv -f "$dst/privkey.pem.new" "$dst/privkey.pem"
# deploy-proxy performs the health-gated Swarm rollout. Its reconciler
# derives a fresh version from the public certificate chain and inserts
# the matching ssl_cert/ssl_key secrets before deploying Traefik.
systemctl restart deploy-proxy.service
# A successful rollout no longer references old wildcard versions. Best
# effort removal retains any secret Docker still reports as in use.
keep="v$(sha256sum "$dst/fullchain.pem" | cut -c1-16)"
docker secret ls --format '{{.Name}}' | \
grep -E '^traefik_ci_commoninternet_net_ssl_(cert|key)_v' | \
grep -v -E "_(ssl_cert|ssl_key)_$keep\$" | \
while IFS= read -r stale; do docker secret rm "$stale" || true; done
'';
};
security.acme = {
acceptTerms = true;
certs."ci.commoninternet.net" = {
domain = "ci.commoninternet.net";
extraDomainNames = [ "*.ci.commoninternet.net" ];
# Staging issuance succeeded using the permanent, narrowly delegated
# CNAME. Production uses the same restricted acme-dns account.
dnsProvider = "acmedns";
environmentFile = "/etc/acme-dns/lego.env";
dnsResolver = "1.1.1.1:53";
server = "https://acme-v02.api.letsencrypt.org/directory";
postRun = ''
# The production marker is deployed only after staging proves the
# permanent CNAME and restricted acme-dns account work end to end.
if [ -e /var/lib/ci-certs/acme-production-enabled ]; then
${pkgs.systemd}/bin/systemctl --no-block start cc-ci-acme-traefik-handoff.service
fi
'';
};
};
}
+3 -3
View File
@@ -40,7 +40,7 @@ let
# admin-registered push optimization deduped against the poller (§4.1). Enrollment = add
# the repo to POLL_REPOS (csv) + ensure tests/<recipe>/ exists.
- POLL_INTERVAL=30
- POLL_REPOS=recipe-maintainers/cc-ci,recipe-maintainers/custom-html,recipe-maintainers/custom-html-tiny,recipe-maintainers/keycloak,recipe-maintainers/cryptpad,recipe-maintainers/matrix-synapse,recipe-maintainers/lasuite-docs,recipe-maintainers/lasuite-meet,recipe-maintainers/n8n,recipe-maintainers/hedgedoc,recipe-maintainers/uptime-kuma,recipe-maintainers/bluesky-pds,recipe-maintainers/discourse,recipe-maintainers/ghost,recipe-maintainers/immich,recipe-maintainers/lasuite-drive,recipe-maintainers/mailu,recipe-maintainers/mattermost-lts,recipe-maintainers/mumble,recipe-maintainers/plausible,recipe-maintainers/drone,recipe-maintainers/gitea
- POLL_REPOS=recipe-maintainers/cc-ci,recipe-maintainers/custom-html,recipe-maintainers/custom-html-tiny,recipe-maintainers/keycloak,recipe-maintainers/cryptpad,recipe-maintainers/matrix-synapse,recipe-maintainers/lasuite-docs,recipe-maintainers/lasuite-meet,recipe-maintainers/n8n,recipe-maintainers/hedgedoc,recipe-maintainers/uptime-kuma,recipe-maintainers/bluesky-pds,recipe-maintainers/discourse,recipe-maintainers/ghost,recipe-maintainers/immich,recipe-maintainers/lasuite-drive,recipe-maintainers/mailu,recipe-maintainers/mattermost-lts,recipe-maintainers/mumble,recipe-maintainers/plausible,recipe-maintainers/drone,recipe-maintainers/gitea,recipe-maintainers/wordpress
- HMAC_FILE=/run/secrets/webhook_hmac
- DRONE_TOKEN_FILE=/run/secrets/drone_token
- GITEA_TOKEN_FILE=/run/secrets/gitea_token
@@ -72,7 +72,7 @@ let
name: cc_ci_bridge_drone_token_v1
gitea_token:
external: true
name: cc_ci_bridge_gitea_token_v1
name: cc_ci_bridge_gitea_token_v3
'';
reconcile = pkgs.writeShellApplication {
@@ -95,7 +95,7 @@ let
}
ensure_secret /run/secrets/bridge_webhook_hmac cc_ci_bridge_webhook_hmac_v1
ensure_secret /run/secrets/bridge_drone_token cc_ci_bridge_drone_token_v1
ensure_secret /run/secrets/bridge_gitea_token cc_ci_bridge_gitea_token_v1
ensure_secret /run/secrets/bridge_gitea_token cc_ci_bridge_gitea_token_v3
docker stack deploy --detach=true -c ${stack} ccci-bridge
'';
+4 -11
View File
@@ -37,17 +37,10 @@
bridge_drone_token = { };
bridge_gitea_token = { };
# Phase-1c C2: the wildcard TLS cert+key are now sops secrets (in cc-ci-secrets), decrypted at
# activation to /var/lib/ci-certs/live/{fullchain.pem,privkey.pem} — the exact path the traefik
# reconcile (modules/proxy.nix) already reads. Replaces the prior operator-drops-a-cert-file step.
wildcard_cert = {
path = "/var/lib/ci-certs/live/fullchain.pem";
mode = "0444"; # leaf+intermediate chain — not secret
};
wildcard_key = {
path = "/var/lib/ci-certs/live/privkey.pem";
mode = "0400"; # private key — root only
};
# The wildcard certificate and private key are issued and renewed locally
# by security.acme. Do not restore the retired SOPS pair here: activation
# would overwrite a freshly renewed ACME certificate before Traefik can
# consume it.
# Phase-2 rate-limit fix (Class A1 registry creds, operator-2026-05-28). Authenticated Docker
# Hub pulls (200/6h per-account) replace the exhausted 100/6h shared-IP anonymous limit that
+8
View File
@@ -0,0 +1,8 @@
{
"$schema": "https://opencode.ai/config.json",
"agent": {
"general": {
"model": "opencode/deepseek-v4-flash"
}
}
}
+11
View File
@@ -132,6 +132,17 @@ KEYS: tuple[Key, ...] = (
"Callable `(ctx)` invoked after UPGRADE_EXTRA_ENV env_set but before `abra secret generate --all` in the upgrade path. Use to pre-insert secrets that `generate --all` would produce with wrong format (e.g. when the .env.sample spec is commented out).",
hook_params=("ctx",),
),
Key(
"UPGRADE_BASE_FLOOR",
"str",
None,
"Declared STRUCTURAL breaking boundary for the upgrade tier (phase basefloor): the first "
"post-break published version tag. Bases strictly below it are excluded from resolution "
"(canonical / step-back / no-canonical fallback) because an in-place upgrade across the "
"boundary is not supported upstream (e.g. a db-family change). When no ≥-floor predecessor "
"exists the tier records a DECLARED skip. NOT a static base pin (§2.G stays removed) — "
"resolution remains dynamic above the floor.",
),
# (CHAOS_BASE_DEPLOY, OIDC_AT_INSTALL and SKIP_GENERIC were deleted in restructure P2:
# compose.ccci.yml is first-class + auto-chaos; install-time deps wiring is the only mode;
# the generic floor is suppressible only via the dev-only CCCI_SKIP_GENERIC* env form.)
+48 -11
View File
@@ -151,8 +151,30 @@ def resolve_upgrade_base(
flush=True,
)
return BasePlan("skip", None, None, f"declared EXPECTED_NA[upgrade]: {declared}")
# UPGRADE_BASE_FLOOR (phase basefloor): a recipe_meta declaration marking a STRUCTURAL breaking
# boundary — published versions strictly below the floor are not valid in-place upgrade sources
# (e.g. discourse 0.8.x→1.0.0 changed the db family bitnami/pgvector → discourse/postgres; the
# data layout+roles are incompatible, upstream supports no in-place path across it). This is NOT
# the removed UPGRADE_BASE_VERSION pin (§2.G): resolution stays fully dynamic — the floor only
# EXCLUDES structurally-impossible bases, and when no candidate ≥ floor exists the tier records
# a DECLARED skip (never a silent pass). Never weakens: below-floor upgrades were never a
# supported path, so no real coverage is lost.
floor = getattr(meta, "UPGRADE_BASE_FLOOR", None)
def _below_floor(version: str) -> bool:
return bool(floor) and warm_reconcile.version_key(version) < warm_reconcile.version_key(
floor
)
skip_canonicals = settings_mod.get().skip_canonicals_for_upgrade
rec = canonical.read_registry(recipe)
if rec and rec.get("version") and not skip_canonicals and _below_floor(rec["version"]):
print(
f"== upgrade tier: last-green canonical {rec['version']} is below the declared "
f"UPGRADE_BASE_FLOOR {floor} (structural break) — excluded as a base",
flush=True,
)
rec = None
if rec and rec.get("version") and not skip_canonicals:
canon = rec["version"]
same = head_version is not None and warm_reconcile.version_key(
@@ -168,10 +190,20 @@ def resolve_upgrade_base(
f"last-green (warm canonical, status={rec.get('status')})",
)
# canonical == head version → deploying it would be a same-version no-op. Step back to the
# newest published version strictly older than the head (phase samever).
older = warm_reconcile.newest_older_version(
warm_reconcile.recipe_tags(recipe), head_version
)
# newest published version strictly older than the head (phase samever). Candidates below a
# declared UPGRADE_BASE_FLOOR are excluded (phase basefloor — structurally invalid bases).
_tags = warm_reconcile.recipe_tags(recipe)
if floor:
_tags = [t for t in _tags if not _below_floor(t)]
older = warm_reconcile.newest_older_version(_tags, head_version)
if older is None and floor:
return BasePlan(
"skip",
None,
None,
f"declared UPGRADE_BASE_FLOOR {floor}: no published predecessor ≥ floor below "
f"head {head_version} (all older tags cross a structural break)",
)
if older:
return BasePlan(
"version",
@@ -189,10 +221,12 @@ def resolve_upgrade_base(
# No canonical in play — none recorded, OR SKIP_CANONICALS_FOR_UPGRADE=true (canonical lookup
# bypassed entirely, behaving as if none exists). Improved fallback (phase settings §2.C): prefer
# a REAL published predecessor (newest release tag < head) over the raw main-tip.
return _no_canonical_base(recipe, head_ref, head_version)
return _no_canonical_base(recipe, head_ref, head_version, floor=floor)
def _no_canonical_base(recipe: str, head_ref: str | None, head_version: str | None) -> BasePlan:
def _no_canonical_base(
recipe: str, head_ref: str | None, head_version: str | None, floor: str | None = None
) -> BasePlan:
"""Upgrade base when no canonical is used (none recorded, its promote failed, or
SKIP_CANONICALS_FOR_UPGRADE is true). Release-tag-first fallback (phase settings §2.C):
1. most recent release TAG with version strictly older than the PR head — a clean published
@@ -202,11 +236,14 @@ def _no_canonical_base(recipe: str, head_ref: str | None, head_version: str | No
3. skip — no predecessor (no older tag and head == main-tip, or no main at all).
This replaces the old jump-straight-to-main-tip path, so an un-promoted recipe upgrades from a real
release base instead of a possibly-untagged WIP commit."""
older = (
warm_reconcile.newest_older_version(warm_reconcile.recipe_tags(recipe), head_version)
if head_version
else None
)
_tags = warm_reconcile.recipe_tags(recipe)
if floor:
# phase basefloor: exclude structurally-invalid bases below the declared floor; the
# main-tip fallback below remains available (it is post-break by definition of the
# declaration — the floor names the first post-break published version).
_fk = warm_reconcile.version_key(floor)
_tags = [t for t in _tags if warm_reconcile.version_key(t) >= _fk]
older = warm_reconcile.newest_older_version(_tags, head_version) if head_version else None
if older:
return BasePlan(
"version",
+65 -8
View File
@@ -24,6 +24,7 @@ Run as root on cc-ci (direct docker/volume access). CLI: `warm_reconcile.py <app
from __future__ import annotations
import hashlib
import json
import os
import re
@@ -37,12 +38,51 @@ from harness import abra, lifecycle, warmsnap # noqa: E402
# --------------------------------------------------------------------------- specs
def _traefik_setup(recipe: str, domain: str, version: str) -> None:
CERT_DIR = "/var/lib/ci-certs/live"
def wildcard_secret_version(cert_dir: str = CERT_DIR) -> str:
"""Stable Swarm-secret version for the public certificate chain.
The certificate chain is public material, so its digest is safe to use as a
version label. The key is deliberately never read or hashed for logging.
"""
chain = os.path.join(cert_dir, "fullchain.pem")
if not os.path.isfile(chain):
raise RuntimeError(f"FATAL: wildcard certificate missing at {chain}")
with open(chain, "rb") as certificate:
digest = hashlib.sha256(certificate.read()).hexdigest()
return "v" + digest[:16]
def _traefik_requires_certificate_rollout(domain: str, secret_version: str) -> bool:
"""Whether Traefik's active service still references an older cert version."""
stack = lifecycle._stack_name(domain) # noqa: SLF001
service = f"{stack}_app"
result = _run(
[
"docker",
"service",
"inspect",
service,
"--format",
"{{range .Spec.TaskTemplate.ContainerSpec.Secrets}}{{.SecretName}} {{end}}",
],
timeout=30,
)
expected = {
f"{stack}_ssl_cert_{secret_version}",
f"{stack}_ssl_key_{secret_version}",
}
return not expected.issubset(set(result.stdout.split()))
def _traefik_setup(recipe: str, domain: str, version: str) -> bool:
"""Per-app config for the traefik reverse-proxy reconcile — preserves EXACTLY what the prior
proxy.nix bash reconcile did (wildcard/file-provider mode serving the pre-issued cert as
ssl_cert/ssl_key swarm secrets; NO ACME). Uses the proven abra.env_set (newline-safe, unlike the
bash set_env that bit keycloak)."""
cert_dir = "/var/lib/ci-certs/live"
cert_dir = CERT_DIR
if not (
os.path.isfile(f"{cert_dir}/fullchain.pem") and os.path.isfile(f"{cert_dir}/privkey.pem")
):
@@ -56,14 +96,15 @@ def _traefik_setup(recipe: str, domain: str, version: str) -> None:
abra.env_set(domain, "DOMAIN", domain)
abra.env_set(domain, "LETS_ENCRYPT_ENV", "")
abra.env_set(domain, "WILDCARDS_ENABLED", "1")
abra.env_set(domain, "SECRET_WILDCARD_CERT_VERSION", "v1")
abra.env_set(domain, "SECRET_WILDCARD_KEY_VERSION", "v1")
secret_version = wildcard_secret_version(cert_dir)
abra.env_set(domain, "SECRET_WILDCARD_CERT_VERSION", secret_version)
abra.env_set(domain, "SECRET_WILDCARD_KEY_VERSION", secret_version)
abra.env_set(domain, "COMPOSE_FILE", '"compose.yml:compose.wildcard.yml"')
stack = lifecycle._stack_name(domain) # noqa: SLF001
have = set(lifecycle._docker_names("secret", stack)) # noqa: SLF001
def _has(name):
return any(s.endswith(f"_{name}_v1") for s in have)
return any(s.endswith(f"_{name}_{secret_version}") for s in have)
if not _has("ssl_cert"):
_run(
@@ -74,7 +115,7 @@ def _traefik_setup(recipe: str, domain: str, version: str) -> None:
"insert",
domain,
"ssl_cert",
"v1",
secret_version,
f"{cert_dir}/fullchain.pem",
"-f",
"-n",
@@ -91,7 +132,7 @@ def _traefik_setup(recipe: str, domain: str, version: str) -> None:
"insert",
domain,
"ssl_key",
"v1",
secret_version,
f"{cert_dir}/privkey.pem",
"-f",
"-n",
@@ -99,6 +140,7 @@ def _traefik_setup(recipe: str, domain: str, version: str) -> None:
timeout=120,
check=True,
)
return _traefik_requires_certificate_rollout(domain, secret_version)
SPECS: dict[str, dict] = {
@@ -457,8 +499,9 @@ def reconcile(app: str) -> str:
# Per-app config/secrets: a spec may provide its own `setup` (traefik's cert/file-provider wiring);
# otherwise the default keycloak-shaped path (app new + DOMAIN/LETS_ENCRYPT + generate secrets).
setup = spec.get("setup")
setup_needs_rollout = False
if setup:
setup(recipe, domain, latest)
setup_needs_rollout = bool(setup(recipe, domain, latest))
else:
ensure_app_config(recipe, domain, latest)
ensure_secrets(domain)
@@ -476,6 +519,20 @@ def reconcile(app: str) -> str:
write_last_good(recipe, target)
return f"deployed-fresh:{target}"
# A certificate rotation changes Traefik's immutable Swarm secrets but
# must not be held hostage by an unrelated recipe-major upgrade policy.
# Redeploy the current recipe version so its compose spec references the
# just-created cert/key secret pair, then apply the usual health gate.
if setup_needs_rollout:
if not current:
raise RuntimeError(f"{app} has services but no current version")
print(f"[{app}] certificate changed → redeploy {current}", flush=True)
deploy_version(recipe, domain, current, dt)
if not wait_healthy(spec):
raise RuntimeError(f"{app} certificate rollout {current} did not become healthy")
write_last_good(recipe, current)
return f"certificate-rolled-out:{current}"
# Deployed & already on latest → converge to a no-op (commit last-good if healthy).
if current == latest:
if wait_healthy(spec, timeout=60):
+116
View File
@@ -0,0 +1,116 @@
# cc-ci test style guide
Rules for writing and changing tests under `tests/`. Read this before any test edit — in particular
before a `/recipe-upgrade <recipe> --with-tests` or `/ci-test-review` fix, where the temptation is to
make a red run green rather than to make the test right.
The tests are the **independent gate** on recipe upgrades. Their value is entirely in being hard to
fool, so every rule below exists to keep them (a) honest and (b) alive across upgrades.
---
## 1. Set up state through the application, not its database
**Order of preference for any fixture that must create state:**
1. **The app's public HTTP API.**
2. **The app's official CLI or release console** (`docker exec … <app-cli>`).
3. **Writing rows into its database — last resort only**, and only with a comment saying which of the
above were tried and why they did not work.
Direct SQL couples the test to the app's *internal schema*, which upgrades are free to change. The
app's own interface is the thing it promises to keep working.
> **Why this rule exists.** `tests/plausible/custom/test_event_tracking.py` used to register its test
> site with `INSERT INTO sites (...)`. That was sufficient for plausible v2. In v3 a site must belong
> to a **team**, and the app silently discards events for a teamless site — `POST /api/event` still
> returns **202** and the row is still in postgres, so the only visible symptom was that nothing ever
> reached ClickHouse. It read as a mysterious ingestion stall and held the recipe RED for six weeks.
>
> The fix was not to also INSERT a team row. It was to stop writing rows: the fixture now calls
> `Plausible.Sites.create/2` through the app's release console, and the app provisions whatever its
> data model currently needs. The same expression works unchanged on v2 (which has no `teams` table
> at all) **and** v3 — not because the test handles both, but because it stopped depending on the
> schema.
When the ideal interface is unavailable, say so in the code. plausible's HTTP provisioning API
(`POST /api/v1/sites`) is gated behind a paid plan and answers `:upgrade_required` on CE, so the test
drops to option 2 and records that in a comment.
## 2. Gate on version rather than writing dual-path fixtures
If a behaviour genuinely only exists from version X, **gate the test on the version** instead of
branching inside it:
```python
pytest.mark.skipif(app_version < (3,), reason="teams were introduced in v3")
```
Do **not** write a fixture that carefully supports both schemas. Version-portable code is harder to
read, harder to trust, and quietly rots once nobody runs the old path.
Corollary: **old tests can simply be deleted** once the fleet has moved past that version. The older
version is only ever exercised through the *upgrade* tier (deploy base → upgrade → assert), so tests
that only make sense for a superseded version are dead weight, not coverage.
Prefer §1 first: an app-level fixture often makes the version difference disappear, and then no gate
is needed at all.
## 3. Never weaken an assertion to turn a run green
There is a hard line between these two, and only the second is allowed as a way out of a red run:
* **Weakening** — relaxing *what* is asserted: dropping a field check, accepting a wider status set,
asserting a 202 ack instead of the stored result, deleting the read-back.
* **Correcting the fixture or the wait** — fixing *how* the test sets up or how long it allows, with
the assertion untouched.
If a test can only pass by asserting less, it has found a real regression. Report it; do not edit it.
## 4. Assert real state, not acknowledgements
An HTTP 202 means "accepted", not "done". Read the effect back out of the system that owns it — the
row in the analytics store, the file on disk, the record in the API — and assert on the values you
sent. plausible's ingestion returns 202 for events it goes on to discard entirely; a test that
stopped at the ack would have been permanently, silently green.
## 5. Derive waits from the recipe's declared readiness, not a guess
A per-recipe `recipe_meta.py` already declares `DEPLOY_TIMEOUT` / `HTTP_TIMEOUT` because someone
measured that app's boot profile. A custom test that hard-codes a shorter window contradicts it and
will flake or fail on a slower version.
Remember the **tier order**: `custom` runs after `backup`/`restore`, which disrupts the datastore and
restarts the app. A window sized for a warm app is not sized for that. plausible's health check
allowed 60s; v3 boots through `sleep 10``createdb``migrate` → cache warmers first.
## 6. Diagnose from the app's own telemetry before touching a test
Before concluding a test is stale, find the app's account of what happened. It is usually definitive
and it stops you fixing the wrong thing. plausible records dropped events in ClickHouse's
`ingest_counters`: `dropped_not_found` with 0 rows before the fix, `buffered` with rows after — that
single counter identified the root cause after the HTTP status had suggested everything was fine.
Prove the diagnosis both ways where you can: same input, broken state → symptom; corrected state →
no symptom.
## 7. Fixtures must be idempotent
A fixture may run against a warm canonical, a restored volume, or a re-run. Creating state must be
safe to repeat — look the object up first and reuse it, rather than assuming a clean database.
## 8. Keep test identities obviously synthetic
Use `ccci-`-prefixed names and `.example` / `.invalid` domains for anything a test creates, so state
it leaves behind is instantly attributable and can never be confused with real data.
---
## Changing a test: the checklist
1. Reproduce the failure and get the **app's own** explanation (§6).
2. Classify: recipe bug, or stale test? Only a stale test justifies a test edit.
3. Fix the **fixture, wait, or setup** — never the assertion (§3).
4. Prefer the app's interface over its database (§1); gate on version rather than branching (§2).
5. Verify green against the recipe PR head with the changed test, plus a regression sample.
6. Say in the commit and PR **what evidence** proves the diagnosis, not just what changed.
+15 -5
View File
@@ -23,11 +23,21 @@ HTTP_TIMEOUT = 1200
#
# UPGRADE-tier BASE (phase prevb — DYNAMIC, no hardcoded UPGRADE_BASE_VERSION): the base the head
# upgrades from is resolved at run time — last-green (warm canonical) → fallback target-branch (`main`)
# tip → else skip (run_recipe_ci.resolve_upgrade_base). discourse has no warm canonical, so the base is
# the `main` tip = bitnamilegacy/discourse:3.5.0, which deploys clean (bitnamilegacy exists) with NO
# `previous/` repair needed. The PR head (recipe-maintainers/discourse#4) switches app to the official
# `discourse/discourse:3.5.3` and drops the sidekiq service, so the upgrade tier now exercises the REAL
# bitnamilegacy→official image migration the PR claims to support.
# tip → else skip (run_recipe_ci.resolve_upgrade_base).
#
# UPGRADE_BASE_FLOOR (phase basefloor, 2026-08-04): the 0.8.x→1.0.0 recipe family switched the app
# bitnamilegacy/discourse → official discourse/discourse AND the db pgvector/pgvector:pg17 →
# discourse/postgres:pg18. That db-family change is a structural break: the bitnami cluster has no
# `discourse` role and pg_upgrade preserves-not-creates roles, so an in-place 0.8.x→1.x deploy can
# NEVER converge (app FATALs `role "discourse" does not exist`, swarm rolls back) — upstream ships
# no in-place path across it. Without the floor, the resolver's step-back/fallback selected
# 0.8.1+3.5.0 (newest tag below the head label) and the upgrade tier red'd on this unsupported
# path twice (drone #1165 2026-07-31 diagnosis, #1171/weekly 2026-08-03 — both classified
# stale-test, recipe verified green on the real official→official path). Declaring the floor keeps
# resolution dynamic and only excludes the structurally-impossible bases; when no ≥-floor
# predecessor exists the tier records a DECLARED skip (never a silent pass). No assertion weakened:
# below-floor in-place upgrades were never supported coverage.
UPGRADE_BASE_FLOOR = "1.0.0+3.5.3"
#
# compose.ccci.yml is now the ENVIRONMENTAL overlay (all deploys): only app.deploy.update_config.order:
# stop-first (node memory reality on the upgrade crossover — see its header). The version-specific
+7 -3
View File
@@ -6,7 +6,11 @@ migration was never tested. With the version-specific config removed from the al
and the dynamic base (last-green/main = bitnamilegacy:3.5.0) deployed only as the *base*, the upgrade
chaos redeploy must land the PR head UNMODIFIED. This overlay asserts exactly that, post-upgrade:
1. the running `app` service image IS the official discourse/discourse:3.5.3 — NOT bitnamilegacy;
1. the running `app` service image IS from the official `discourse/discourse` repository —
NOT bitnamilegacy. (Version-agnostic since 2026-08-04: the original assertion hardcoded the
migration-era pin `:3.5.3` and went stale on the first legitimate app bump (2026.7.1, weekly
2026-08-03). The property this test guards is the IMAGE FAMILY — official vs bitnami — not a
frozen version; the exact head pin is already exercised by the deploy itself.)
2. the `sidekiq` service the PR deletes is GONE from the deployed stack.
If either fails, the head did not really run (the overlay leaked onto it) → RED. Assertion-only,
@@ -26,8 +30,8 @@ def test_head_runs_official_image_not_bitnamilegacy(live_app):
f"app image is {image!r} — the bitnamilegacy base leaked onto the PR head "
"(the version-specific overlay was applied to the head, the prevb bug)"
)
assert image.startswith("discourse/discourse:3.5.3"), (
f"app image is {image!r}, expected the PR head's official discourse/discourse:3.5.3 "
assert image.startswith("discourse/discourse:"), (
f"app image is {image!r}, expected the PR head's official discourse/discourse image "
"— the head's image migration was not exercised"
)
+160
View File
@@ -0,0 +1,160 @@
"""Recipe-local OIDC *session* login helper (authorization-code flow + session cookie).
meet v1.22.0 hardened API auth — raw OIDC user access tokens are rejected as Bearer
credentials; the app accepts only its own session cookie, established through the standard OIDC
authorization-code browser flow (app login URL → keycloak login form → callback → Django
session). This helper drives that flow with urllib + a CookieJar so the custom tests can
exercise the API the way a real client does.
Kept recipe-local (cf. tests/ghost/custom/_ghost.py precedent; same helper as
tests/lasuite-docs/custom/_oidc_session.py) rather than in runner/harness — promote it there
if a third recipe needs it.
Usage:
sess = OidcSession(f"https://{live_app}")
me = sess.login(kc["user"], kc["password"]) # asserts whoami 200; returns the user dict
status, body = sess.post("/api/v1.0/rooms/", {"name": "x"}) # CSRF handled
"""
from __future__ import annotations
import contextlib
import html
import http.cookiejar
import json
import re
import ssl
import urllib.error
import urllib.parse
import urllib.request
# Per-run *.ci.commoninternet.net domains serve the operator's wildcard cert via the Traefik file
# provider; chain verification is done once in the install tier (generic.served_cert).
_CTX = ssl.create_default_context()
_CTX.check_hostname = False
_CTX.verify_mode = ssl.CERT_NONE
_LOGIN_PATHS = ("/api/v1.0/authenticate/", "/oidc/authenticate/", "/api/v1.0/users/me/")
_WHOAMI = "/api/v1.0/users/me/"
class OidcSession:
"""A cookie-carrying HTTP session logged in via the app's OIDC authorization-code flow."""
def __init__(self, base: str):
self.base = base.rstrip("/")
self.jar = http.cookiejar.CookieJar()
self.opener = urllib.request.build_opener(
urllib.request.HTTPCookieProcessor(self.jar),
urllib.request.HTTPSHandler(context=_CTX),
)
# -- low-level ---------------------------------------------------------------------------
def _open(
self,
url: str,
data: bytes | None = None,
headers: dict[str, str] | None = None,
method: str | None = None,
timeout: int = 30,
) -> tuple[int, str, bytes]:
"""Open a URL (following redirects, carrying cookies). Returns (status, final_url, body)."""
req = urllib.request.Request(url, data=data, method=method)
for k, v in (headers or {}).items():
req.add_header(k, v)
try:
with self.opener.open(req, timeout=timeout) as resp:
return resp.getcode(), resp.geturl(), resp.read()
except urllib.error.HTTPError as e:
body = b""
with contextlib.suppress(Exception):
body = e.read()
return e.code, e.filename or url, body
def _csrf_token(self) -> str | None:
for c in self.jar:
if "csrftoken" in c.name.lower():
return c.value
return None
# -- login -------------------------------------------------------------------------------
def login(
self,
username: str,
password: str,
login_paths: tuple[str, ...] = _LOGIN_PATHS,
whoami: str = _WHOAMI,
) -> dict:
"""OIDC authorization-code login: app → keycloak form → callback → session cookie.
Asserts the resulting session GETs `whoami` with HTTP 200 and returns the parsed user.
"""
page, page_url, last = None, None, (0, "", b"")
for path in login_paths:
status, final_url, body = self._open(self.base + path)
last = (status, final_url, body)
text = body.decode(errors="replace")
if "kc-form-login" in text or (
"/protocol/openid-connect/" in final_url and "<form" in text
):
page, page_url = text, final_url
break
assert page is not None, (
f"could not reach the keycloak login form via {login_paths}: last URL "
f"{last[1]!r} HTTP {last[0]} body[:200]={last[2][:200]!r}"
)
m = re.search(r'<form[^>]*id="kc-form-login"[^>]*action="([^"]+)"', page) or re.search(
r'<form[^>]*action="([^"]+)"[^>]*method=["\']?post', page, re.I
)
assert m, f"no login form action on keycloak page {page_url!r}: {page[:300]!r}"
action = html.unescape(m.group(1))
form = urllib.parse.urlencode(
{"username": username, "password": password, "credentialId": ""}
).encode()
status, landed, body = self._open(
action, data=form, headers={"Content-Type": "application/x-www-form-urlencoded"}
)
status, _, who = self._open(self.base + whoami)
assert status == 200, (
f"OIDC session login failed: GET {whoami} -> HTTP {status} after submitting the "
f"keycloak form (landed at {landed!r}; excerpt: {body[:200]!r})"
)
parsed = json.loads(who)
assert isinstance(parsed, dict), f"unexpected whoami payload: {who[:200]!r}"
return parsed
# -- API calls with the session ----------------------------------------------------------
def request(self, method: str, path: str, data: dict | None = None) -> tuple[int, object]:
"""Issue an API call with the session cookie (+ CSRF header on unsafe methods)."""
url = path if path.startswith("http") else self.base + path
headers: dict[str, str] = {}
body: bytes | None = None
if data is not None:
body = json.dumps(data).encode()
headers["Content-Type"] = "application/json"
if method.upper() not in ("GET", "HEAD", "OPTIONS"):
tok = self._csrf_token()
if tok:
headers["X-CSRFToken"] = tok
headers["Referer"] = self.base + "/"
headers["Origin"] = self.base
status, _, raw = self._open(url, data=body, headers=headers, method=method.upper())
try:
return status, json.loads(raw)
except (json.JSONDecodeError, ValueError):
return status, None
def get(self, path: str) -> tuple[int, object]:
return self.request("GET", path)
def post(self, path: str, data: dict | None = None) -> tuple[int, object]:
return self.request("POST", path, data)
def delete(self, path: str) -> tuple[int, object]:
return self.request("DELETE", path)
+41 -28
View File
@@ -5,8 +5,14 @@ SOURCE: references/recipe-maintainer/recipe-info/lasuite-meet/tests/meeting_flow
Meet's characteristic behavior is real-time meetings: a user creates a room and receives a LiveKit
(SFU) join token for WebSocket signaling. This is the §4.3 create-an-object + read-it-back, plus the
distinctive WebRTC-signaling feature (LiveKit token issuance) — not a health/200 stand-in. Flow:
1. OIDC password grant (the per-run keycloak user) → a Meet API bearer token.
distinctive WebRTC-signaling feature (LiveKit token issuance) — not a health/200 stand-in.
Updated for meet v1.22.0+ API auth hardening: the API now REJECTS raw OIDC user access tokens
sent as Bearer credentials; authenticated calls run on the app's session cookie from the real
OIDC authorization-code login (see _oidc_session.py). Flow:
1. OIDC password grant (the per-run keycloak user) → assert the API rejects it as Bearer
(the v1.22.0 hardening — a 2xx here would mean the hardening regressed); then log in via
the OIDC authorization-code flow → session cookie.
2. POST /api/v1.0/rooms/ {name, access_level:public} → 201 with id/slug AND a LiveKit room+token.
3. GET /api/v1.0/rooms/{id}/ (read-it-back) → 200, again with a LiveKit token for the same room.
4. Assert the LiveKit token is a real JWT carrying a video grant for that room (token issuance —
@@ -27,9 +33,11 @@ import sys
import pytest
sys.path.insert(0, os.path.dirname(__file__))
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "..", "runner"))
from _oidc_session import OidcSession # noqa: E402 (recipe-local helper, same dir)
from harness import http as harness_http # noqa: E402
from harness import sso
from harness import sso # noqa: E402
def _b64url(seg: str) -> bytes:
@@ -57,17 +65,28 @@ def _creds(deps: dict) -> dict:
@pytest.mark.requires_deps
def test_create_room_get_livekit_token_and_read_back(live_app, deps):
assert "keycloak" in deps, f"keycloak creds missing; got {list(deps.keys())}"
kc = deps["keycloak"]
base = f"https://{live_app}"
# meet v1.22.0+ hardening: a raw OIDC user access token must be REJECTED as Bearer.
token = sso.oidc_password_grant(_creds(deps))
assert isinstance(token, str) and token.count(".") == 2, "OIDC access token is not a JWT"
auth = {"Authorization": f"Bearer {token}"}
# --- create a room (the object) ---
status, body = harness_http.http_post(
f"{base}/api/v1.0/rooms/",
data={"name": "ccci-meeting", "access_level": "public"},
headers=auth,
headers={"Authorization": f"Bearer {token}"},
)
assert status in (401, 403), (
f"POST /api/v1.0/rooms/ with a raw OIDC Bearer token returned HTTP {status} — meet >= "
f"v1.22.0 must reject user access tokens on the API (body {body!r})"
)
# The successor auth path: session cookie via the real OIDC authorization-code login.
sess = OidcSession(base)
sess.login(kc["user"], kc["password"])
# --- create a room (the object) ---
status, body = sess.post("/api/v1.0/rooms/", {"name": "ccci-meeting", "access_level": "public"})
assert status == 201, f"room create returned HTTP {status} (expected 201); body={body!r}"
assert isinstance(body, dict), f"room create body not JSON: {body!r}"
room_id = body.get("id")
@@ -75,36 +94,32 @@ def test_create_room_get_livekit_token_and_read_back(live_app, deps):
lk_room = livekit.get("room")
lk_token = livekit.get("token")
assert room_id, f"room created but no id: {body!r}"
assert (
lk_token and isinstance(lk_token, str) and lk_token.count(".") == 2
), f"room created but no LiveKit JWT token: {livekit!r}"
assert lk_token and isinstance(lk_token, str) and lk_token.count(".") == 2, (
f"room created but no LiveKit JWT token: {livekit!r}"
)
try:
# --- read it back (a fresh authenticated GET of the created room) ---
status, got = harness_http.http_request(
"GET", f"{base}/api/v1.0/rooms/{room_id}/", headers=auth
)
status, got = sess.get(f"/api/v1.0/rooms/{room_id}/")
assert status == 200, f"room read-back returned HTTP {status} (expected 200); body={got!r}"
assert (
isinstance(got, dict) and got.get("id") == room_id
), f"read-back room id mismatch: {got!r}"
assert isinstance(got, dict) and got.get("id") == room_id, (
f"read-back room id mismatch: {got!r}"
)
got_lk = got.get("livekit") or {}
assert got_lk.get("token"), f"read-back room missing LiveKit token: {got!r}"
assert (
got_lk.get("room") == lk_room
), f"read-back LiveKit room {got_lk.get('room')!r} != create-time {lk_room!r}"
assert got_lk.get("room") == lk_room, (
f"read-back LiveKit room {got_lk.get('room')!r} != create-time {lk_room!r}"
)
# --- the LiveKit token is a real signaling grant for this room (WebRTC subset) ---
payload = json.loads(_b64url(lk_token.split(".")[1]))
video = payload.get("video") or {}
assert (
video.get("room") == lk_room or payload.get("room") == lk_room
), f"LiveKit JWT does not grant the created room {lk_room!r}: {payload!r}"
assert video.get("room") == lk_room or payload.get("room") == lk_room, (
f"LiveKit JWT does not grant the created room {lk_room!r}: {payload!r}"
)
finally:
# --- delete the room (cleanup + a real DELETE mutation) ---
del_status, _ = harness_http.http_request(
"DELETE", f"{base}/api/v1.0/rooms/{room_id}/", headers=auth
)
del_status, _ = sess.delete(f"/api/v1.0/rooms/{room_id}/")
assert del_status in (
204,
200,
@@ -120,9 +135,7 @@ def test_create_room_get_livekit_token_and_read_back(live_app, deps):
gone = False
for _ in range(5):
status, _ = harness_http.http_request(
"GET", f"{base}/api/v1.0/rooms/{room_id}/", headers=auth
)
status, _ = sess.get(f"/api/v1.0/rooms/{room_id}/")
if status == 404:
gone = True
break
+68 -17
View File
@@ -14,7 +14,11 @@ Both assert real app state (the event reached the analytics store), not just the
plausible only ingests events for *known* sites — the in-memory `sites_cache` gates ingestion and
drops events for unregistered domains (empirically confirmed: an event for an unregistered domain
never appears in events_v2). So each test first registers a site row in the metadata postgres, then
never appears in events_v2). Sites are therefore provisioned through plausible's OWN creation path
rather than by writing rows — under v3 a site must belong to a TEAM, and a teamless site is dropped as
`dropped_not_found` while the POST still acks 202, which reads as a silent ingestion stall. Letting the
app create the site sidesteps that entirely, and works unchanged on v2. So each test first provisions
the site, then
POSTs repeatedly while polling ClickHouse: the sites_cache must refresh to admit the new site and the
event write-buffer must flush to ClickHouse, so the first landing is not instantaneous. Re-POSTing the
same event is safe — we assert the row count is >= 1.
@@ -40,6 +44,10 @@ _UA = (
"(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
)
# Identity the harness provisions sites under. Ephemeral per-run deploy, never a real account.
_HARNESS_EMAIL = "cc-ci@ci.invalid"
_HARNESS_PW = "ccci-harness-passphrase-2026"
def _ch(domain: str, sql: str) -> str:
"""Run a ClickHouse query against the `plausible_events_db` service; return stdout (stripped)."""
@@ -50,18 +58,61 @@ def _ch(domain: str, sql: str) -> str:
).strip()
# plausible's own provisioning path. Creating a site through the app (rather than INSERTing rows)
# means the app applies whatever its current data model requires — which is what makes this work
# unchanged across the v2→v3 jump, where sites gained a mandatory owning TEAM. Verified on cc-ci
# against BOTH v2.0.0 (no `teams` table at all) and v3.2.1: identical expression, site usable, events
# ingested. See tests/STYLE.md.
#
# The HTTP provisioning API (`POST /api/v1/sites`) would be the first choice, but it is gated behind
# a paid plan — on CE it answers `:upgrade_required` — so the app's release console is the closest
# public interface available here.
_PROVISION_SITE_EXS = """
pw = "__PW__"
email = "__EMAIL__"
user =
case Plausible.Auth.find_user_by(email: email) do
nil ->
{:ok, u} =
Plausible.Auth.User.new(%{name: "cc-ci", email: email, password: pw, password_confirmation: pw})
|> Plausible.Repo.insert()
u
u -> u
end
site = "__SITE__"
result =
case Plausible.Sites.get_by_domain(site) do
nil -> Plausible.Sites.create(user, %{"domain" => site, "timezone" => "UTC"})
s -> {:ok, s}
end
case result do
{:ok, _} -> IO.puts("CCCI_SITE_OK " <> site)
other -> IO.puts("CCCI_SITE_ERR " <> inspect(other))
end
"""
def _register_site(domain: str, site: str) -> None:
"""Insert a site row into the metadata postgres (`db` service) so plausible will ingest events for
it. Idempotent (ON CONFLICT DO NOTHING)."""
sql = (
"INSERT INTO sites (domain, timezone, inserted_at, updated_at, native_stats_start_at) "
f"VALUES ('{site}','UTC', now(), now(), now()) ON CONFLICT (domain) DO NOTHING; "
f"SELECT domain FROM sites WHERE domain = '{site}';"
"""Provision `site` via plausible's own site-creation path, so it is a site the app will ingest for.
Idempotent: an existing domain is reused rather than re-created.
Do NOT reach into postgres to do this. A `sites` INSERT was enough under v2, but v3 requires the
site to belong to a TEAM and silently discards events for a teamless site — `POST /api/event`
still acks 202 and the row still exists, so the only symptom is that nothing reaches ClickHouse
(ClickHouse's own `ingest_counters` records it as `dropped_not_found`). That is what put this
recipe RED on build 1224. Going through the app removes the whole class of problem: it provisions
the team itself.
"""
exs = (
_PROVISION_SITE_EXS.replace("__PW__", _HARNESS_PW)
.replace("__EMAIL__", _HARNESS_EMAIL)
.replace("__SITE__", site)
)
out = lifecycle.exec_in_app(domain, ["/app/bin/plausible", "rpc", exs], service="app")
assert f"CCCI_SITE_OK {site}" in out, (
f"could not provision site {site!r} via the app: {out.strip()[-400:]}"
)
out = lifecycle.exec_in_app(
domain, ["psql", "-q", "-U", "plausible", "-d", "plausible", "-tAc", sql], service="db"
).strip()
assert out == site, f"site {site!r} not registered in postgres (got {out!r})"
def _post_event(base_domain: str, site: str, name: str, pathname: str) -> int:
@@ -93,9 +144,9 @@ def _ingest_and_count(
last_status = None
while True:
last_status = _post_event(base_domain, site, name, pathname)
assert (
last_status == 202
), f"POST /api/event for {name!r} → HTTP {last_status} (expected 202)"
assert last_status == 202, (
f"POST /api/event for {name!r} → HTTP {last_status} (expected 202)"
)
time.sleep(interval)
raw = _ch(base_domain, count_sql)
count = int(raw) if raw.isdigit() else 0
@@ -143,6 +194,6 @@ def test_custom_event_roundtrip(live_app):
live_app,
f"SELECT name FROM events_v2 WHERE pathname = '{pathname}' LIMIT 1",
)
assert (
stored_name == event_name
), f"custom event stored as {stored_name!r}, expected {event_name!r}"
assert stored_name == event_name, (
f"custom event stored as {stored_name!r}, expected {event_name!r}"
)
+7 -1
View File
@@ -17,6 +17,12 @@ def test_plausible_root_serves(live_app):
62-char SECRET_KEY_BASE, see recipe_meta.EXTRA_ENV); the dedicated
/api/health endpoint is.
"""
# The custom tier runs AFTER the backup/restore tier, which disrupts postgres under the app and
# restarts it. v3 (community-edition) then boots through `sleep 10` + `db createdb` + `db migrate`
# + cache warmers before /api/health flips to 200, which does not fit in 60s — that is what put
# this recipe RED on build 1224 while install/upgrade/backup/restore all passed. The assertion is
# unchanged (still a hard 200 from the real readiness endpoint); only the wait matches the boot
# profile the recipe already declares via recipe_meta.HTTP_TIMEOUT (1200).
url = f"https://{live_app}/api/health"
status, _ = harness_http.retry_http_get(url, expect_status=(200,), max_wait=60, interval=3)
status, _ = harness_http.retry_http_get(url, expect_status=(200,), max_wait=300, interval=5)
assert status == 200, f"GET {url} HTTP {status}"
+21
View File
@@ -0,0 +1,21 @@
"""Unit coverage for the public wildcard-secret version label."""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parents[2] / "runner"))
import warm_reconcile as wr # noqa: E402
def test_wildcard_secret_version_is_stable_and_does_not_need_key(tmp_path):
(tmp_path / "fullchain.pem").write_text("public certificate chain\n")
assert wr.wildcard_secret_version(str(tmp_path)) == wr.wildcard_secret_version(str(tmp_path))
assert wr.wildcard_secret_version(str(tmp_path)).startswith("v")
def test_wildcard_secret_version_changes_with_certificate_chain(tmp_path):
chain = tmp_path / "fullchain.pem"
chain.write_text("first public certificate chain\n")
first = wr.wildcard_secret_version(str(tmp_path))
chain.write_text("replacement public certificate chain\n")
assert wr.wildcard_secret_version(str(tmp_path)) != first
+15
View File
@@ -9,6 +9,7 @@ from __future__ import annotations
import os
import sys
from types import SimpleNamespace
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", ".."))
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner"))
@@ -115,6 +116,20 @@ def test_traefik_spec_is_stateless_with_setup():
assert "setup" not in wr.SPECS["keycloak"]
def test_traefik_certificate_rollout_detects_active_secret_version(monkeypatch):
stack = "traefik_ci_commoninternet_net"
monkeypatch.setattr(wr.lifecycle, "_stack_name", lambda _domain: stack)
monkeypatch.setattr(
wr,
"_run",
lambda *_args, **_kwargs: SimpleNamespace(
stdout=f"{stack}_ssl_cert_vnew {stack}_ssl_key_vnew"
),
)
assert not wr._traefik_requires_certificate_rollout("traefik.ci.commoninternet.net", "vnew")
assert wr._traefik_requires_certificate_rollout("traefik.ci.commoninternet.net", "vold")
def test_manual_migration_markers():
assert wr.notes_flag_manual_migration("This release requires a MANUAL MIGRATION of the DB.")
assert wr.notes_flag_manual_migration("Breaking change: action required before upgrade.")
+29
View File
@@ -0,0 +1,29 @@
# Parity — wordpress
The recipe-maintainer corpus has **no** `recipe-info/wordpress/tests/` directory — wordpress was
not in the recipe-maintainer parity suite. This PARITY.md documents the Phase-2-style
recipe-specific tests + health_check as the parity-aligned baseline (enrolled 2026-08-03 on
operator request).
## Recipe-specific tests (≥2 beyond parity)
wordpress is a classic PHP CMS on mariadb. A fresh CI deploy serves the install wizard (the CI
env sets no `POST_DEPLOY_CMDS core_install`); the custom tier completes the wizard itself with
run-scoped credentials (`custom/_wp.py`) and then exercises the installed site's real APIs.
| cc-ci file | what's verified | rationale |
|---|---|---|
| `custom/test_health_check.py` | GET `/` → 200 or 302 (install-wizard redirect). | traefik → app wiring floor. |
| `custom/test_install_and_api.py` | Completes the install wizard (writes site options + admin user to mariadb), then asserts `/?rest_route=/` AND `/wp-json/` both return the configured site name. | Non-vacuous: the name only comes back if the install round-tripped through the DB. The two REST routes split failure layers: `?rest_route=` isolates "REST + DB", `/wp-json/` additionally proves the recipe's `.htaccess` rewrites are live. A DB-wiring failure is caught earlier by the installer's own "database connection" error, asserted in `_wp.ensure_installed`. |
| `custom/test_post_roundtrip.py` | §4.3 create-an-object + read-it-back: publish a post with a unique marker via **XML-RPC** `wp.newPost` (admin user/pass), read it back via the **public REST API** (`/wp/v2/posts/<id>`, title must contain the marker), then fetch the public permalink `/?p=<id>` and assert the marker in the served HTML. | The marker round-trips app → mariadb → app across three distinct subsystems (XML-RPC write, REST read, themed HTML render). A post that didn't persist, a broken DB, or a wedged PHP fails at the layer that broke. |
## Backup data-integrity (P4)
The recipe stores content in the `wordpress_content` volume + mariadb; backup-capable detection is
automatic (compose.yml `backupbot.backup` labels via the standard recipe mechanism). Lifecycle
overlays not yet authored — catch-up if backup data-integrity proves needed for this recipe.
## Playwright (P6)
Not authored. The install + post round-trip is API-driven; a Playwright pass over wp-admin would
add browser coverage of the dashboard. Follow-up if wanted.
+85
View File
@@ -0,0 +1,85 @@
"""Shared wordpress test helper — install-wizard client + admin credentials.
A fresh CI deploy of the recipe does NOT run `core_install` (no POST_DEPLOY_CMDS in the CI
env), so the app serves the WP install wizard until something completes it. The custom tests
complete it here (run-scoped class-B credentials; the whole app — DB volume + secrets — is
destroyed at teardown) and then exercise the real APIs (wp-json + XML-RPC) as an installed
site. `ensure_installed` is idempotent so any custom test can call it first regardless of
alphabetical ordering.
"""
from __future__ import annotations
import ssl
import urllib.error
import urllib.parse
import urllib.request
# Per-run *.ci.commoninternet.net domains use the operator wildcard cert via the Traefik file
# provider; the real-cert chain check is done once in the generic install assertion.
_CTX = ssl.create_default_context()
_CTX.check_hostname = False
_CTX.verify_mode = ssl.CERT_NONE
ADMIN_USER = "ccci-admin"
ADMIN_PW = "Ccci-Wp-Test-Pw-2026!x" # strong so the wizard needs no pw_weak confirmation
ADMIN_EMAIL = "ccci-admin@ccci.example.com"
BLOG_TITLE = "CCCI Test Site"
def _open(url: str, data: bytes | None = None, timeout: int = 60) -> tuple[int, str]:
req = urllib.request.Request(url, data=data)
if data is not None:
req.add_header("Content-Type", "application/x-www-form-urlencoded")
try:
with urllib.request.urlopen(req, timeout=timeout, context=_CTX) as resp:
return resp.getcode(), resp.read().decode(errors="replace")
except urllib.error.HTTPError as e:
try:
return e.code, e.read().decode(errors="replace")
except Exception: # noqa: BLE001
return e.code, ""
def fetch_text(url: str, timeout: int = 60) -> tuple[int, str]:
"""GET a URL and return (status, body-text) — for HTML-content assertions."""
return _open(url, timeout=timeout)
def ensure_installed(domain: str) -> bool:
"""Complete the WP install wizard if it hasn't been completed yet.
Returns True if THIS call ran the install, False if the site was already installed.
Fails the calling test (assert) if the wizard is reachable but the install POST fails.
"""
base = f"https://{domain}"
status, body = _open(f"{base}/wp-admin/install.php")
assert status == 200, f"GET /wp-admin/install.php HTTP {status} (body[:200]={body[:200]!r})"
if "already installed" in body.lower():
return False
assert "wordpress" in body.lower(), (
f"/wp-admin/install.php does not look like the WP installer: {body[:300]!r}"
)
assert "database connection" not in body.lower(), (
"WP installer reports a database connection problem — app→mariadb wiring is broken"
)
form = urllib.parse.urlencode(
{
"weblog_title": BLOG_TITLE,
"user_name": ADMIN_USER,
"admin_password": ADMIN_PW,
"admin_password2": ADMIN_PW,
"admin_email": ADMIN_EMAIL,
"blog_public": "1",
"Submit": "Install WordPress",
"language": "",
}
).encode()
status, body = _open(f"{base}/wp-admin/install.php?step=2", data=form, timeout=180)
assert status == 200, f"install POST HTTP {status} (body[:300]={body[:300]!r})"
lowered = body.lower()
assert "success" in lowered or "wordpress has been installed" in lowered, (
f"install POST did not report success: {body[:400]!r}"
)
return True
@@ -0,0 +1,21 @@
"""wordpress — Phase-2 health_check (no recipe-maintainer parity corpus for wordpress).
Asserts the served root responds: 200 (installed site) or 302 (redirect to the install
wizard on a fresh deploy). Either proves traefik → app wiring; the deeper DB/API proofs
live in test_install_and_api / test_post_roundtrip.
"""
from __future__ import annotations
import os
import sys
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "..", "runner"))
from harness import http as harness_http # noqa: E402
def test_wordpress_root_serves(live_app):
"""GET / → 200 or 302 (install-wizard redirect on a fresh deploy)."""
url = f"https://{live_app}/"
status, _ = harness_http.retry_http_get(url, expect_status=(200, 302), max_wait=90, interval=5)
assert status in (200, 302), f"GET {url} HTTP {status} (expected 200 or 302)"
@@ -0,0 +1,41 @@
"""wordpress — complete the install wizard, then read the site back via the REST API.
Non-vacuous: the install POST writes the site options + admin user to mariadb through the
app's DB wiring; `/wp-json/` only returns the site name after WP can read those options back
from the DB, and the pretty-permalink REST route additionally proves the recipe's .htaccess
rewrites are live. A wedged DB fails the install; a missing htaccess breaks /wp-json/ (the
`?rest_route=` fallback is asserted separately so the failure names the broken layer).
"""
from __future__ import annotations
import os
import sys
sys.path.insert(0, os.path.dirname(__file__))
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "..", "runner"))
from _wp import BLOG_TITLE, ensure_installed # noqa: E402
from harness import http as harness_http # noqa: E402
def test_install_and_rest_api_roundtrip(live_app):
ensure_installed(live_app)
# ?rest_route= works regardless of rewrites — isolates "REST API up + DB readable"
status, body = harness_http.http_get(f"https://{live_app}/?rest_route=/", timeout=60)
assert status == 200, f"GET /?rest_route=/ HTTP {status}"
assert isinstance(body, dict), f"REST index is not JSON: {body!r}"
assert body.get("name") == BLOG_TITLE, (
f"site name mismatch: got {body.get('name')!r}, expected {BLOG_TITLE!r}"
"install options did not round-trip through the DB"
)
# /wp-json/ additionally requires the recipe's .htaccess rewrite rules
status, body = harness_http.http_get(f"https://{live_app}/wp-json/", timeout=60)
assert status == 200, (
f"GET /wp-json/ HTTP {status} — REST works via ?rest_route= but the pretty route "
"fails: the recipe's .htaccess rewrites are not active"
)
assert isinstance(body, dict) and body.get("name") == BLOG_TITLE, (
f"unexpected /wp-json/ payload: {body!r}"
)
@@ -0,0 +1,65 @@
"""wordpress — §4.3 create-an-object + read-it-back: publish a post, read it back twice.
Flow:
1. `ensure_installed` (idempotent — run-scoped admin credentials from _wp.py).
2. Create a post with a unique marker title via **XML-RPC** `wp.newPost` (stdlib
xmlrpc.client; XML-RPC ships enabled in WP and authenticates with the admin
user/password directly — no cookie/nonce dance).
3. Read it back via the **public REST API** (`?rest_route=/wp/v2/posts/<id>`) — a different
subsystem than the one that wrote it — asserting id + rendered title match.
4. Fetch the public permalink `/?p=<id>` and assert the marker is in the served HTML.
Non-vacuous: the marker round-trips app → mariadb → app across three distinct read paths;
a post that didn't persist, a broken DB, or a wedged PHP-FPM fails at the layer that broke.
"""
from __future__ import annotations
import os
import ssl
import sys
import uuid
import xmlrpc.client
sys.path.insert(0, os.path.dirname(__file__))
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "..", "runner"))
from _wp import ADMIN_PW, ADMIN_USER, ensure_installed, fetch_text # noqa: E402
from harness import http as harness_http # noqa: E402
_CTX = ssl.create_default_context()
_CTX.check_hostname = False
_CTX.verify_mode = ssl.CERT_NONE
def test_create_post_and_read_back(live_app):
ensure_installed(live_app)
marker = f"ccci-post-{uuid.uuid4().hex[:8]}"
proxy = xmlrpc.client.ServerProxy(f"https://{live_app}/xmlrpc.php", context=_CTX)
post_id = proxy.wp.newPost(
0,
ADMIN_USER,
ADMIN_PW,
{
"post_title": marker,
"post_content": f"cc-ci round-trip body for {marker}",
"post_status": "publish",
},
)
assert post_id and str(post_id).isdigit(), f"wp.newPost returned no post id: {post_id!r}"
# Read back via the public REST API (different subsystem than XML-RPC)
status, body = harness_http.http_get(
f"https://{live_app}/?rest_route=/wp/v2/posts/{post_id}", timeout=60
)
assert status == 200, f"GET rest_route /wp/v2/posts/{post_id} HTTP {status}: {body!r}"
assert isinstance(body, dict) and str(body.get("id")) == str(post_id), (
f"read-back id mismatch: {body!r}"
)
rendered = (body.get("title") or {}).get("rendered", "")
assert marker in rendered, f"read-back title {rendered!r} missing marker {marker!r}"
# And the public permalink serves the marker in HTML
status, raw = fetch_text(f"https://{live_app}/?p={post_id}")
assert status == 200, f"GET /?p={post_id} HTTP {status}"
assert marker in raw, f"permalink page does not contain the marker {marker!r}"
+12
View File
@@ -0,0 +1,12 @@
# Per-recipe harness config for wordpress (classic PHP app + mariadb). A fresh deploy
# (no POST_DEPLOY_CMDS core_install in the CI env) serves the WP install wizard: GET /
# redirects 302 to /wp-admin/install.php until the custom tier completes the install.
HEALTH_PATH = "/"
HEALTH_OK = (200, 302)
# First boot copies the WP core into the content volume and waits for mariadb init;
# the recipe's own healthcheck has start_period 1m — give the deploy headroom.
DEPLOY_TIMEOUT = 900
HTTP_TIMEOUT = 300
# canon §2.B: enroll as a DATA-WARM canonical (all recipes enrolled — operator 2026-06-17).
WARM_CANONICAL = True