diff --git a/.opencode/skills/cctest-recipe-upstream/SKILL.md b/.opencode/skills/cctest-recipe-upstream/SKILL.md index 992364b..e48a4c1 100644 --- a/.opencode/skills/cctest-recipe-upstream/SKILL.md +++ b/.opencode/skills/cctest-recipe-upstream/SKILL.md @@ -24,3 +24,68 @@ run `/cctest-intro` / `/cctest-setup-sandbox` first. timers) from an ARM skill — cc-ci work goes through the cc-ci skills. - The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration. + +**Sandboxed vs non-sandboxed mode.** This skill needs only git + (optionally) the Gitea API — +not the test server — so it can run either way. Probe, then follow that branch: + +- **Sandboxed** (ARM env configured: `test-ssh/.testenv` with `GITEA_USERNAME`/`GITEA_PASSWORD`/ + `GITEA_URL`, sandbox/test instances): the canonical `recipe-upstream` script in + `references/recipe-maintainer/.claude/commands/recipe-upstream.md` runs as written. Note its + WORKSPACE probing expects `/workspace` or `~/Documents/recipe-maintainer`; on a bare host pass + the submodule dir explicitly instead. +- **Non-sandboxed** (no ARM env on the host — e.g. the orchestrator, where the recipe-maintainer + checkout is only a pinned submodule): no sandbox/test instances are needed and NONE of the + setup skills are. Recipe-maintainer mirrors on `git.autonomic.zone` are publicly readable, so: + 1. Check out the recipe if missing: `abra recipe fetch ` (lands in `~/.abra/recipes/`) + — or a plain anonymous `git clone https://git.autonomic.zone/recipe-maintainers/.git` + if abra is unavailable. + 2. Fetch the PR head branch from the mirror **anonymously** — no credentials in the remote URL: + `git remote add gitea https://git.autonomic.zone/recipe-maintainers/.git` + (remote update rather than re-add if it exists), then + `git fetch gitea +refs/pull//head:refs/heads/`. + 3. Fetch PR metadata (head/base refs, merged flag, release bump line) from + `https://git.autonomic.zone/api/v1/repos/recipe-maintainers//pulls/` — + unauthenticated; use bot creds only if the repo turns out to be private (orchestrator hosts + can read them from `/srv/cc-ci-orch/.testenv` — never written anywhere else). + 4. Everything else in the canonical script (origin/dev remote setup, release recommendation, + emitted next-steps) is identical. + +Every time the branch was prepared **here**, remember it exists only on this host — the +operator's machine must fetch it first. Always emit this **step 0** before the push step +(anonymous public fetch, no credentials needed): + +``` + # 0. On a machine WITHOUT the branch pre-fetched, get it from the autonomic mirror + # (fetch by URL — works regardless of what the local remotes are named): + cd + git fetch https://git.autonomic.zone/recipe-maintainers/.git +refs/pull//head:refs/heads/ + git checkout +``` + +If the operator's checkout does NOT yet have the mirror remote, emit once before the fetch: + +``` + git remote add gitea https://git.autonomic.zone/recipe-maintainers/.git +``` + +In both modes the final output is a set of commands for the operator to run on a machine **with +push access to `git.coopcloud.tech`** — always print them, even when everything local is +already prepared. + +**Verify the merge-base BEFORE rebasing — bases can be traps.** Upstream repos can +carry a stray, divergent `main` ALONGSIDE the real `master` base branch. Rebase targets must be +chosen by evidence: `git ls-remote origin | grep -E 'refs/heads/(main|master)$'`, then +`git merge-base origin/` — the correct base is the one where the upgrade +branch's merge-base is its tip's parent (i.e. `` is already directly on that line — +then NO cherry-pick is needed, the raw branch compares clean). Cherry-picking onto the wrong +base produces the very messy 3-dot compare this guard exists to prevent (real case: gitea — +first "rebase" landed on a stray `main` and had to be redone onto `master`). + +**Branch-name mismatch mirror vs upstream.** The mirror and upstream can use different base-branch +names, and a mirror can even carry a stray, divergent `main` alongside the real base (gitea: mirror +PR against `main`, upstream `master`; keycloak PR #9: base `main`, both mirror and upstream on +`master`). The canonical script now detects the upstream default branch by evidence (`git +ls-remote --symref origin HEAD`, then the git.coopcloud.tech API, then the PR base with a warning) +and uses it for the compare URL AND the post-merge release steps — never hardcode `main`, and when +it prints the base-mismatch warning, confirm the upgrade branch actually sits on that line +(`git merge-base` against both candidates) before emitting the push step. diff --git a/.opencode/skills/recipe-enroll/SKILL.md b/.opencode/skills/recipe-enroll/SKILL.md index de21841..ae57150 100644 --- a/.opencode/skills/recipe-enroll/SKILL.md +++ b/.opencode/skills/recipe-enroll/SKILL.md @@ -43,7 +43,10 @@ On cc-ci, after `abra recipe fetch `, read `~/.abra/recipes//`: ### 2. Create + sync the mirror Create `recipe-maintainers/` (Gitea API: `POST /orgs/recipe-maintainers/repos`, -`{"name":…,"private":true,"default_branch":"main","auto_init":false}`), then force-sync from +`{"name":…,"private":false,"default_branch":"main","auto_init":false}`) — a recipe mirror MUST be +created **public**: the cc-ci recipe mirrors are public (fleet invariant; the report's live PR-STATUS +proxy `report./pr//` is tokenless and only sees public repos — private-from-birth mirrors +like gitea/wordpress left its status cells dark until flipped). Then force-sync from coopcloud with the existing helper **run on cc-ci with creds injected via stdin** (the host has no `.testenv`): ``` diff --git a/cc-ci-plan/JOURNAL.md b/cc-ci-plan/JOURNAL.md index 190a6b4..0911be5 100644 --- a/cc-ci-plan/JOURNAL.md +++ b/cc-ci-plan/JOURNAL.md @@ -31,8 +31,44 @@ handoff). --- -## Session 2026-05-31 ~18:30 UTC — Claude Sonnet 4.6 +## Session 2026-09-14 ~16:45 UTC — opencode glm-5.3-flash (orchestrator) — round 2: Anubis UA +**Left off:** The real root cause turned out to be TWO independent layers; the mirror-privacy +fix (earlier session entry today) was necessary but not sufficient. Operator's browser console +showed CORS failures redirecting to `anubis.swarm.autonomic.zone/.within.website/?redir=…`. +Reproduced exactly: the `/pr/` proxy forwards the END browser's User-Agent to Gitea; Gitea sits +behind **Anubis**, which 307-challenges browser-like UAs to `anubis.swarm.autonomic.zone` +(no CORS headers) → every fetch throws in the browser → all cells "?" (curl passed clean, which +is why server-side checks and my earlier headless test never saw it — intermittent/rate-dependent +for my playwright run). Fix: cc-ci **PR #38** adds +`proxy_set_header User-Agent "ccci-reports-proxy/1.0";` to the reports.nix `/pr/` location. +Hot-verified on the host by mount-swapping a fixed conf into the running task (one mis-step: +`--mount-rm`+`--mount-add` same-target order wiped the mount; re-added), scoped live, all 16 +cells rendering with a real Chromium. Merged PR #38, `nix flake update cc-ci`, +`nixos-rebuild test` → healthy (reports 200, no failed units) → `switch` (flake.lock commit +9e7770f). Final verify: browser-UA curl 200 both gitea/9 + full headless-Chromium sweep 16/16 +OPEN, zero non-200 /pr fetches. Also flipped memory: `memory/gitea-anubis-ua-challenge.md` + +MEMORY.md index. +**Open:** nothing blocking; next weekly /recipe-report and STATUS live-checks carry the fix. + +## Session 2026-09-14 ~15:00 UTC — opencode glm-5.3-flash (orchestrator) + +**Left off:** Report STATUS column fix. Operator reported the week-2026-09-11 report's live +PR-STATUS column all "?" — root cause: the tokenless same-origin proxy +`report./pr//` (cc-ci `nix/modules/reports.nix`) 404s on **private** mirrors; two +late-enrolled mirrors, `recipe-maintainers/gitea` (2026-06-11) and `wordpress` (2026-08-03), +had been created `"private":true` from birth — by the stale instruction in +`/recipe-enroll`'s mirror step (the other 21 mirrors were flipped public on 2026-06-09, and +the old 'org is private' blocker is long resolved). Fixed: secret-scanned both repos, flipped +`private=false` (PATCH with bot creds), patched `.opencode/skills/recipe-enroll/SKILL.md` to +create mirrors `private:false`, updated memory/recipe-mirrors-public-org-blocker.md + +MEMORY.md index. **Verified in a real headless Chromium (nixpkgs chromium + playwright)**: +all 16 STATUS rows render `open`, every `/pr/` fetch 200 JSON. Commit 6e93922 pushed. The +STATUS column refreshes live every 30s; cells go ✓ when a PR merges. No reports.nix change +was needed (proxy itself was healthy). +**Open:** nothing on this; the report index regenerates next weekly run. + +## Session 2026-05-31 ~18:30 UTC — Claude Sonnet 4.6 **Left off:** Got opencode/deepseek-v4-pro working as the loop backend. Both builder and adversary are actively running on `tinfoil/deepseek-v4-pro` (via `inference.tinfoil.sh`). Phase 5 [11/11] in progress. The operator is debugging the opencode web UI visibility and @@ -1202,6 +1238,56 @@ the host: `opencode-go/deepseek-v4-flash` and `opencode-go/glm-5.3-flash` answer `upgrader.env` (`LOOP_TIER=go` maps to the `opencode-go` auth entry; `LOOP_MODEL` overrides the tier default). Next fire Fri 2026-09-11 02:00 UTC. - The steering orchestrator agent stays on `opencode-go/glm-5.2` (not asked to change). + +## Session 2026-09-21 — domain cutover to ci.autonomic.zone (orchestrator) + +**What happened.** Morning: hourly supervisor resumed the stalled 09-18 weekly run on the GO tier +(ZEN endpoint dead server-side — `UnknownError`; run completed 13 green PRs, 0 failed). Published +the missing week-2026-09-18 report (PR-finding #4; launcher defaults flipped to `go` in +cc-ci-orchestrator PR #23). Then executed the full domain cutover per +`cc-ci-plan/plan-domain-migration-ci-autonomic-zone.md` (PR #25). + +**Plan deviation (simplification).** No dual-cert SNI: ONE Let's Encrypt cert carries SANs for +BOTH zones (`ci` + `*.ci` of autonomic.zone AND commoninternet.net) — the unchanged single-pair +`ssl_cert/ssl_key` traefik reconciler keeps working; Phase 4 reissues without the legacy SANs. +The new zone's DNS-01 challenge reuses the SAME acme-dns account: storage re-keyed by +`cc-ci-acme-storage-seed.service` (jq clone of the legacy entry under `ci.autonomic.zone`), +CNAME already delegated. Proven by a hand lego **staging** run before any production change. + +**Merged:** cc-ci #39 (front doors dual Host rules, bridge/dashboard env URLs, drone abra rename, +runner RPC, naming.py → `*.ci.autonomic.zone`, dual-zone name regexes in lifecycle/warm/prune, +recipe-report URLs) · cc-ci #40 (seed-unit nesting fix) · cc-ci #41 (have_secret stack-scope — +caught live: the old stack's `*_rpc_secret_v1` satisfied the check post-rename) · cc-ci #42 (nix +interpolation escape) · orchestrator #27 (oc.ci host + `opencodeUiExtraHosts`, host self-pins, +flake bump). + +**Deployed** via `nixos-rebuild test` → verify → `switch` (generation `nn1vwiv7v1k…`, running == +boot). Cert SANs confirmed 4-name; all 5 front doors answer on BOTH zones +(200/200/303/401 + traefik 200), TLS verify=0 from outside; zero failed units; disk dropped +88%→45% after prune. + +**Drone migration.** New abra app `drone.ci.autonomic.zone`, FRESH DB (module's +`DRONE_USER_CREATE` re-injected the sops bridge token). The Gitea OAuth app redirect now has both +URIs; the client secret was rotated (each Gitea PATCH regenerates it) and synced through +sops → `sops-install-secrets` → swarm secret v1 → drone. Bootstrapped OAuth +(`drone login ok (admin=true)`), re-enabled cc-ci + discourse repos, build timeout 60m. +Webhooks: ghost + discourse repointed (secrets preserved); cc-ci repo's bridge webhook → +`ci.autonomic.zone/hook`, stale drone hook deleted, Drone's auto-created new-zone hook active. +Old stack removed + orphaned secrets reaped. + +**E2E proof.** `!testme` on keycloak PR #9 → bridge → drone build #1 (new DB numbering) → runner → +harness → `results.json` + PR card `✅ passed` linking `ci.autonomic.zone/runs/1/summary.png`. + +**Deferred / open.** +- Warm stacks + backupbot stay on the legacy zone (data-warm volumes / restic password tied to + abra app names) — post-bake migration; harness regexes accept both zones meanwhile. +- Docs sweep (~60 references: AGENTS.md, README, skills incl. cc-ci-status front-door list, + launcher printed URLs). +- Phase 4 (after ≥7 clean days): drop legacy SANs (reissue), remove legacy Host arms + host + self-pins + Gandi records (`ci`, `*.ci`, `_acme-challenge.ci`), TTLs back to 3600. +- Host auto-update was `failed` 2026-09-15 (health check) — `/cc-ci-orchestrator-update` still + pending; next auto-attempt Tue 09-22. + ## Session 2026-09-28 20:00 UTC — operator-broken cc-ci recovered by plain hard reset - Operator reported ci.autonomic.zone down after their own change, supplied a Hetzner API token diff --git a/cc-ci-plan/launch-report.py b/cc-ci-plan/launch-report.py index 8fbf297..dcaadb0 100755 --- a/cc-ci-plan/launch-report.py +++ b/cc-ci-plan/launch-report.py @@ -2,8 +2,9 @@ """cc-ci recipe-report launcher — one-shot agent that runs /recipe-report after the weekly upgrade. The report agent's model is configured SEPARATELY from the upgrader (REPORT_BACKEND/REPORT_MODEL), -but defaults to the same OpenCode subscription on glm-5.2 as the upgrader. The model default tracks -the backend+tier (opencode+zen→opencode/glm-5.2, opencode+go→opencode-go/glm-5.2, claude→opus). +but defaults to the same OpenCode subscription as the upgrader (now OpenCode GO — the only tier +with a key on this host; ZEN models are dead here, see launch-upgrader.py). The model default tracks +the backend+tier (opencode+go→opencode-go/glm-5.3-flash, opencode+zen→opencode/glm-5.2, claude→opus). Usage: launch-report.py start [DATE] use-or-create the session; runs /recipe-report [DATE] @@ -12,7 +13,8 @@ Usage: launch-report.py status show session state Env: REPORT_BACKEND (default opencode), REPORT_MODEL (default tracks tier), - LOOP_TIER / REPORT_TIER (default 'zen'; 'go' or 'zen'; only for opencode), + LOOP_TIER / REPORT_TIER (default 'go' — the only tier with a key on this host; ZEN + models are dead here; 'go' or 'zen'; only for opencode), OPENCODE_SHARE (1=attach web server + public --share link), REPORT_SESSION, REPORT_DIR. """ import os, subprocess, sys, time @@ -23,9 +25,9 @@ SESSION = os.environ.get("REPORT_SESSION", "cc-ci-report") WORKDIR = os.environ.get("REPORT_DIR", "/srv/cc-ci") LOG_DIR = os.environ.get("LOG_DIR", "/srv/cc-ci/.cc-ci-logs") BACKEND = os.environ.get("REPORT_BACKEND", "opencode") -TIER = os.environ.get("LOOP_TIER", os.environ.get("REPORT_TIER", "zen")) +TIER = os.environ.get("LOOP_TIER", os.environ.get("REPORT_TIER", "go")) _TIER_MODEL = {"go": "opencode-go/glm-5.2", "zen": "opencode/glm-5.2"} -_DEFAULT_MODEL = _TIER_MODEL.get(TIER, _TIER_MODEL["zen"]) if BACKEND == "opencode" else "opus" +_DEFAULT_MODEL = _TIER_MODEL.get(TIER, _TIER_MODEL["go"]) if BACKEND == "opencode" else "opus" MODEL = os.environ.get("REPORT_MODEL", _DEFAULT_MODEL) CLAUDE_BIN = os.environ.get("CLAUDE_BIN", "claude") CLAUDE_FLAGS = os.environ.get("CLAUDE_FLAGS", "--dangerously-skip-permissions") diff --git a/cc-ci-plan/launch-upgrader.py b/cc-ci-plan/launch-upgrader.py index a0c7a9f..0f48fe0 100644 --- a/cc-ci-plan/launch-upgrader.py +++ b/cc-ci-plan/launch-upgrader.py @@ -15,14 +15,16 @@ Usage: Env: LOOP_BACKEND opencode (default) | claude — also accepts UPGRADER_BACKEND - LOOP_TIER opencode subscription tier: "zen" (OpenCode ZEN, default) or "go" - (OpenCode Go). Selects the default model + the usage-limit probe - endpoint/key. Go hit a monthly limit; ZEN is the working alternative. - Only affects the opencode backend (ignored for claude). + LOOP_TIER opencode subscription tier: "go" (OpenCode Go, DEFAULT — the only tier + with a key on this host) or "zen" (OpenCode ZEN, opt-in on a host that + has a ZEN key). Selects the default model + the usage-limit probe + endpoint/key. ZEN models are DEAD here (no key, gone from the + provider catalogue — see the TIER block below). + Only affects the opencode backend (ignored for claude). LOOP_MODEL model flag (overrides UPGRADER_MODEL); default tracks backend+tier — - opencode+zen→opencode/glm-5.2, opencode+go→opencode-go/glm-5.2, claude→sonnet - UPGRADER_MODEL provider/model for opencode, e.g. opencode/glm-5.2 (OpenCode ZEN), - opencode-go/glm-5.2 (OpenCode Go), or tinfoil/deepseek-v4-pro; sonnet for claude + opencode+go→opencode-go/glm-5.3-flash, opencode+zen→opencode/glm-5.2, claude→sonnet + UPGRADER_MODEL provider/model for opencode, e.g. opencode-go/glm-5.3-flash (OpenCode Go, default), + opencode/glm-5.2 (OpenCode ZEN), or tinfoil/deepseek-v4-pro; sonnet for claude UPGRADER_ARGS extra args passed to /upgrade-all (e.g. "n8n ghost", "--dry-run") claude backend: @@ -42,11 +44,13 @@ WORKDIR = os.environ.get("UPGRADER_DIR", "/srv/cc-ci") LOG_DIR = os.environ.get("LOG_DIR", "/srv/cc-ci/.cc-ci-logs") # LOOP_BACKEND / LOOP_MODEL take precedence (unified control from the operator). -# LOOP_TIER selects the OpenCode subscription: "zen" (default) or "go". Go hit a monthly -# usage limit; ZEN is the working alternative. Set LOOP_TIER=go when the Go limit resets. +# LOOP_TIER selects the OpenCode subscription: "go" (DEFAULT — the only tier with a key on +# this host) or "zen" (opt-in on a host that has a ZEN key). ZEN models are DEAD here — the +# endpoint errors server-side, and env loss in the tmux→agent→launch-report chain is exactly +# how it silently ate the 2026-09-18 weekly report. Never default back to zen. # Only affects the opencode backend; claude ignores the tier. BACKEND = os.environ.get("LOOP_BACKEND", os.environ.get("UPGRADER_BACKEND", "opencode")) -TIER = os.environ.get("LOOP_TIER", os.environ.get("UPGRADER_TIER", "zen")) +TIER = os.environ.get("LOOP_TIER", os.environ.get("UPGRADER_TIER", "go")) _TIER_CFG = { "go": {"model": "opencode-go/glm-5.2", "endpoint": "https://opencode.ai/zen/go/v1/chat/completions", "auth": "opencode-go", "label": "OpenCode Go"}, @@ -56,7 +60,7 @@ _TIER_CFG = { if BACKEND == "opencode" and TIER not in _TIER_CFG: print(f"[upgrader] ERROR: unknown LOOP_TIER '{TIER}' — use 'go' or 'zen'", flush=True) sys.exit(1) -_tier = _TIER_CFG.get(TIER, _TIER_CFG["zen"]) +_tier = _TIER_CFG.get(TIER, _TIER_CFG["go"]) _DEFAULT_MODEL = _tier["model"] if BACKEND == "opencode" else "sonnet" MODEL = os.environ.get("LOOP_MODEL", os.environ.get("UPGRADER_MODEL", _DEFAULT_MODEL)) diff --git a/cc-ci-plan/plan-domain-migration-ci-autonomic-zone.md b/cc-ci-plan/plan-domain-migration-ci-autonomic-zone.md new file mode 100644 index 0000000..86e63bd --- /dev/null +++ b/cc-ci-plan/plan-domain-migration-ci-autonomic-zone.md @@ -0,0 +1,172 @@ +# Plan: migrate cc-ci domains from `*.commoninternet.net` to `ci.autonomic.zone` + +**Date:** 2026-09-21 · **Status:** DRAFT — blocked only on operator DNS access (Gandi zone `autonomic.zone`) +**Principle:** same host, same IP (195.201.88.249), same services — only names change. Dual-run new +alongside old; old names retire last; rollback at any point is "change nothing further" because the +old DNS zone keeps answering until Phase 4. + +--- + +## 1. Inventory — what answers on what today + +| Name | Served by | Router/config source | +|---|---|---| +| `ci.commoninternet.net` | results dashboard + bridge `/hook` | `cc-ci/nix/modules/dashboard.nix`, `bridge.nix` (traefik Host rules) | +| `report.ci.commoninternet.net` | weekly report static site | `cc-ci/nix/modules/reports.nix` | +| `drone.ci.commoninternet.net` | Drone server | `cc-ci/nix/modules/drone.nix` (abra app; env `DOMAIN`, `DRONE_SERVER_HOST`); runner RPC: `drone-runner.nix` | +| `oc.ci.commoninternet.net` | opencode UI (traefik → host nginx basic auth) | `cc-ci-orch/nix/modules/orchestrator-host.nix` (`opencodeUiHost`), set in `nix/hosts/cc-ci/configuration.nix` | +| `traefik.ci.commoninternet.net` | traefik stack identity + deploy-proxy health gate | `cc-ci/nix/modules/proxy.nix` | +| `warm-*.ci.commoninternet.net` | warm canonical recipe stacks | `warm-keycloak.nix`, warm-bluesky-pds (caddy labels) | +| `-<6hex>.ci.commoninternet.net` | per-run recipe test deploys | `cc-ci/runner/harness/naming.py` `app_domain()` — **resolves via public DNS** (containers use 1.1.1.1/8.8.8.8) | +| `backups.ci.commoninternet.net` | backup-bot-two — stack name only, no web route | `backupbot.nix` | +| `acme.commoninternet.net` (NS `ns-acme`) | local acme-dns (systemd, port 53) | `cc-ci/nix/modules/acme-dns.nix` | + +**Cert chain today:** nixos `security.acme` (lego) issues `ci.commoninternet.net` + `*.ci.commoninternet.net` +via DNS-01 through the local acme-dns; Gandi delegates `_acme-challenge.ci.commoninternet.net` CNAME → +`9d3f0428-eb97-4b14-bb07-da2f0f6c2073.acme.commoninternet.net`. `postRun` fires +`cc-ci-acme-traefik-handoff.service` → installs cert into `/var/lib/ci-certs/live` → restarts +`deploy-proxy.service`, whose reconciler re-derives the `traefik_ci_commoninternet_net_ssl_{cert,key}_v` +swarm secrets (traefik runs co-op-cloud wildcard/no-ACME mode off those secrets). +**DNS:** both zones on Gandi. IPv4-only (no AAAA records anywhere; none needed). + +**External references:** Gitea OAuth app for Drone (redirect URI → drone.ci…); per-repo webhooks → +`/hook` — **only `recipe-maintainers/ghost` and `recipe-maintainers/discourse` have any** (everything +else is bridge-driven via the Drone API); ~60 doc references across cc-ci-orch (AGENTS.md, README, +plan.md, skills). + +## 2. Target naming + +| Old | New | +|---|---| +| ci.commoninternet.net | ci.autonomic.zone | +| report.ci.commoninternet.net | report.ci.autonomic.zone | +| drone.ci.commoninternet.net | drone.ci.autonomic.zone | +| oc.ci.commoninternet.net | oc.ci.autonomic.zone | +| traefik.ci.commoninternet.net | traefik.ci.autonomic.zone | +| *.ci.commoninternet.net (warm-\*, per-run apps, backups identity) | *.ci.autonomic.zone | +| acme.commoninternet.net | **stays** through Phases 1–2 (infra, not a front door); optional Phase 3 | + +Swarm **stack names** (`traefik_ci_commoninternet_net`, `ccci-*`, warm stacks) are identifiers, not +domains — keep them where possible to avoid churn in keep-lists/prune filters, EXCEPT abra-managed +apps whose name **is** the domain (drone, backup-bot, per-run apps). + +## 3. DNS records to set (zone `autonomic.zone`, Gandi — operator) + +| # | Name | Type | Value | TTL | Purpose | +|---|---|---|---|---|---| +| 1 | `ci.autonomic.zone` | A | `195.201.88.249` | 300 | apex front door (wildcard does NOT cover the apex) | +| 2 | `*.ci.autonomic.zone` | A | `195.201.88.249` | 300 | report/drone/oc/traefik/warm-\*/`-<6hex>` run apps | +| 3 | `_acme-challenge.ci.autonomic.zone` | CNAME | `9d3f0428-eb97-4b14-bb07-da2f0f6c2073.acme.commoninternet.net.` | 300 | DNS-01 for the new wildcard — **reuses the existing restricted acme-dns account** (`disable_registration=true`; the lego *account* authenticates, not the FQDN, so one acme-dns subdomain serves both zones) | + +Also: lower TTL to 300 on the old `ci`/`*.ci` records a few days ahead (fast rollback later). +Phase 3 (optional) adds: `acme.ci.autonomic.zone` + `ns-acme.ci.autonomic.zone` A records. + +**DNS verification (before touching the host):** +```bash +dig +short ci.autonomic.zone A @ns-187-a.gandi.net # 195.201.88.249 +dig +short report.ci.autonomic.zone @1.1.1.1 # 195.201.88.249 (via wildcard) +dig +short drone.ci.autonomic.zone @8.8.8.8 # 195.201.88.249 +dig +short _acme-challenge.ci.autonomic.zone CNAME # 9d3f0428-….acme.commoninternet.net. +``` + +## 4. Phase 1a — certificate (issue BEFORE any routing change) + +1. `cc-ci/nix/modules/acme-dns.nix`: add a second `security.acme.certs."ci.autonomic.zone"` entry + (domain + `*.ci.autonomic.zone` SAN, `dnsProvider = "acmedns"`, `dnsResolver = "1.1.1.1:53"`). + Split the lego env per cert — today `ACME_DNS_STORAGE_PATH` is baked to + `/var/lib/acme/ci.commoninternet.net/…`; generate one env file per cert so each keeps its own + acme-dns account file. +2. Extend `cc-ci-acme-traefik-handoff` to stage the new cert alongside the old + (`/var/lib/ci-certs/live` holds both, e.g. `fullchain.pem` = new + `fullchain-old.pem`). +3. Extend the **deploy-proxy reconciler** (`proxy.nix`) for **dual-cert SNI**: new wildcard pair as + `ssl_cert/ssl_key` (default store) + old pair as a second secret pair wired through a small + file-provider `[[tls.certificates]]` so `*.ci.commoninternet.net` keeps serving during the bake + window. This is the one fiddly bit — validate against the **Let's Encrypt staging** endpoint + first (the module already has the staging-marker pattern). +4. `nixos-rebuild test` → confirm both certs issued and served (`openssl s_client -servername` for a + name in each zone, check SANs) → `switch`. + +## 5. Phase 1b — routing + service config (one stack at a time, health-gated) + +During the bake window every router rule carries **both** hostnames +(`Host(`ci.autonomic.zone`) || Host(`ci.commoninternet.net`)`); old names drop out only in Phase 4. + +- `dashboard.nix` + `bridge.nix` (share the `ci` host: dashboard `/` + bridge `/hook`), `reports.nix`, + and the traefik health-gate name in `proxy.nix`. +- `bridge.py` / `dashboard.py` code defaults for `DRONE_URL` / `DASH_URL`. +- `recipe-report.py` — footer, publish URL text and its verification curl → `report.ci.autonomic.zone` + (regenerate the index after, so the home page stops linking the old name). +- Drone **abra app rename** (name = domain, so this is a new stack): + 1. Export enabled-repo list from the old Drone API. + 2. Deploy new abra app `drone.ci.autonomic.zone` (same module; secrets `rpc_secret`, + `client_secret` re-insert from `/run/secrets` as today). **Fresh DB** — Drone data volume is + stack-name-prefixed; repos re-sync on Gitea login, build history is lost (accepted; export the + old repo list and re-enable via API). + 3. Point `drone-runner.nix` `DRONE_RPC_HOST` at the new name; restart runner. + 4. Re-run `scripts/bootstrap-drone-oauth.sh` with `DRONE=https://drone.ci.autonomic.zone` + (updates the Gitea OAuth redirect URI; script already uses `--resolve` to hit loopback). + 5. Update the two webhooks (ghost, discourse) to `https://drone.ci.autonomic.zone/hook?secret=` + — read the secret off the existing hook config before deleting it. + 6. `abra app undeploy` + remove the old drone app once green. +- `configuration.nix`: `opencodeUiHost = "oc.ci.autonomic.zone"`; extend the `networking.hosts` + self-pin list with the new names (keep old pins during bake). +- Gitea `GITEA_SERVER` stays `git.autonomic.zone` — unchanged. + +## 6. Phase 2 — harness / warm / per-run naming (the `_ci_commoninternet_net` blast radius) + +- `cc-ci/runner/harness/naming.py` `app_domain()` → `.ci.autonomic.zone` (single source of truth). +- Regexes/patterns that match the derived names: `docker-prune.nix` run-app filter, + `tests/concurrency/concutil.py` (`DOMAIN` const + `RUN_APP_RE`), any separate `RUN_APP_RE` in + `lifecycle.py` / `warm.py` / `warm_reconcile.py` / `canonical.py` / `generic.py` / `http.py`, + `nightly-sweep` keep-list patterns, `tests/gitea/recipe_meta.py` comment. +- Redeploy warm stacks under new names (abra `warm-.ci.autonomic.zone`), reconcile, drain old. +- `backupbot.nix` `DOMAIN` (stack-name-only change). +- Verify: full deterministic test suite + one live `!testme` run end-to-end. + +## 7. Phase 3 (optional) — move acme-dns off commoninternet.net + +Only if `commoninternet.net` is being dropped entirely: new `domain`/`nsname` in acme-dns config +(`acme.ci.autonomic.zone` / `ns-acme.ci.autonomic.zone` + Gandi records), briefly re-enable +registration or hand-migrate the account DB, re-point `_acme-challenge` CNAMEs, re-issue both certs. + +## 8. Phase 4 — retirement (after ≥7 clean days) + +Remove old names from the dual router rules → remove `security.acme.certs."ci.commoninternet.net"` ++ its env file + old cert secret pair → drop old `networking.hosts` pins → delete old Gandi records +(`ci`, `*.ci`, `_acme-challenge.ci`) → restore TTLs to 3600. Keep `acme.commoninternet.net` unless +Phase 3 was done. + +## 9. Verification checklist + +- **DNS:** the dig table in §3 against a Gandi NS and a public resolver. +- **Cert:** `openssl s_client -connect 195.201.88.249:443 -servername ` for one name per zone — + SANs correct for both, `ssl_verify_result=0`. +- **Front doors:** `ci` → 200 · `report.ci` → 200 · `drone.ci` → 200/303 · `oc.ci` → **401** + (basic auth intact) · `traefik.ci/api/version` → 200. +- **SSO:** Drone login through Gitea on the new redirect URI. +- **Webhook:** empty commit to `recipe-maintainers/ghost` main → build fires on the new host. +- **Bridge/!testme:** `!testme` on a scratch PR → build runs, `results.json` written, PR comment + contains **new** domain links. +- **Report:** publish one report page and verify it at `https://report.ci.autonomic.zone`. +- **Host:** `systemctl --failed` empty; `deploy-proxy` reconcile logs clean. + +## 10. Risks / gotchas + +- **Dual-cert SNI in the traefik reconciler** is the highest-risk change — stage-first, verify SNI + selection for both zones before switching. +- **Drone DB** does not follow the abra rename — accept fresh build history; export/re-apply the + enabled-repo list; re-insert secrets from `/run/secrets` (module already does). +- **Per-run recipe domains resolve via public DNS** — the `*.ci.autonomic.zone` wildcard A record is + load-bearing for every test deploy; never merge Phase 2 before it propagates. +- Old PR comments/badges link old names — they keep working until Phase 4 removes the old records. +- ~60 doc references (AGENTS.md, README, plan.md, skills, status skill) — one cosmetic doc PR at the + end so the docs don't lie. +- No domain-bearing secret values were found (webhook secrets are per-repo and copied as-is). + +## 11. Sequencing + +1. **Operator:** TTLs down + 3 Gandi records (§3) — 5 minutes. +2. **Agent:** cert PR (§4) → `test` → verify dual SNI → `switch`. +3. **Agent:** routing/env PRs (§5) stack-by-stack, health-gated; OAuth + webhooks + drone rename. +4. **Agent:** end-to-end verification (§9) + report republish under the new name. +5. **Bake ≥7 days** → Phase 2 PR → full suite → bake → Phase 4 retirement. diff --git a/cc-ci-plan/recipe-report.py b/cc-ci-plan/recipe-report.py index 34f9ea4..f114c04 100755 --- a/cc-ci-plan/recipe-report.py +++ b/cc-ci-plan/recipe-report.py @@ -303,7 +303,7 @@ def render(spec_path, out_path): for p in lead.split("\n\n") if p.strip()) body = (_mast() + f'' + f'report.ci.autonomic.zone{gen}' f'
{lead}
') # 1) the full wire — every recipe, in the agent's recommended priority order (CVEs first); CVEs column. wire = ("The full wire — every recipe, in priority order" if kind == "week" @@ -322,7 +322,7 @@ def render(spec_path, out_path): if s.get("changes"): body += f'

What changed

{_changes(s.get("changes"), repo_url)}' body += (f'') + f'dashboard · archive') open(out_path, "w").write(_page(f"{title} · " + s["date"], body)) print("wrote", out_path) @@ -347,10 +347,10 @@ def publish(html_path, date, kind="week"): for d, k in sorted(set(entries), reverse=True)) idx = _page("The Recipe Report — Archive", _mast() + '' + 'report.ci.autonomic.zone' f'
    {lis or "
  • No reports yet.
  • "}
') subprocess.run(["ssh", "cc-ci", f"cat > {HOST_REPORTS}/index.html"], input=idx.encode(), check=True) - print(f"published https://report.ci.commoninternet.net/{page} (+ index)") + print(f"published https://report.ci.autonomic.zone/{page} (+ index)") def main(): diff --git a/cc-ci-plan/upstream/bluesky-pds.md b/cc-ci-plan/upstream/bluesky-pds.md index 917f486..62511ff 100644 --- a/cc-ci-plan/upstream/bluesky-pds.md +++ b/cc-ci-plan/upstream/bluesky-pds.md @@ -2,7 +2,7 @@ | service | image | source repo | releases / changelog | |---------|-------|-------------|----------------------| -| app | ghcr.io/bluesky-social/pds | https://github.com/bluesky-social/pds | git tags `vX.Y.Z` (no GitHub Releases objects — `releases/latest` 404s) | +| app | ghcr.io/bluesky-social/pds | https://github.com/bluesky-social/pds | git tags `vX.Y.Z` 1:1 with image tags (no GitHub Releases objects — `releases/latest` 404s); app-level changes live in the atproto monorepo changelog: https://github.com/bluesky-social/atproto/blob/main/packages/pds/CHANGELOG.md (image `0.4.5NNN` == `@atproto/pds` `0.5.NN`) | | caddy | docker.io/caddy:2 | https://github.com/caddyserver/caddy | https://github.com/caddyserver/caddy/releases | ## Standing notes (researched 2026-06-11, bsky phase) diff --git a/cc-ci-plan/upstream/custom-html.md b/cc-ci-plan/upstream/custom-html.md index 2030d69..125a6a9 100644 --- a/cc-ci-plan/upstream/custom-html.md +++ b/cc-ci-plan/upstream/custom-html.md @@ -9,6 +9,7 @@ - nginx even-numbered minor versions (1.28.x, 1.30.x) are mainline; odd (1.27.x, 1.29.x) are stable. The recipe tracks mainline. - compose.git-pull.yml is an optional overlay for git-pull functionality; alpine/git version lives there. - compose.sftp.yml and compose.sso.yml are other optional overlays; linuxserver/openssh-server uses `latest` tag and is not version-pinned. +- nginx 1.31.6 (15 Sep 2026) is a security patch release: CVE-2026-90439 (heap buffer overflow in a worker under certain configurations when using HTTP/3 with OpenSSL 3.5.0 and earlier). Change: QUIC transport parameters extension received in an SSL connection is now always ignored. Bugfixes: binary upgrade with control API socket + perl module; predicate-location evaluation error ignored; nested location lookup with regex/predicate locations; segfault reading config with `geo` … `ranges` + corrupted binary base file. None affect the custom-html static-file-serving use case (no HTTP/3/QUIC, no predicate locations, no perl, no `geo`). - Breaking change in nginx 1.31.0: HTTP/2 and HTTP/3 requests with Connection/Proxy-Connection/Keep-Alive/Transfer-Encoding/Upgrade/TE headers are now rejected. - Breaking change in nginx 1.29.7: keepalive in upstream block is enabled by default; proxy_http_version changed to 1.1; Connection proxy header no longer sent by default. - nginx 1.31.3 (15 Jul 2026) is a security patch release: CVE-2026-42533 (heap buffer overflow in `map` directive with regex matching / non-cacheable variable in string expression), CVE-2026-60005 (uninitialized memory access with unnamed regex captures + `slice` directive / background cache update → memory disclosure or worker crash), CVE-2026-56434 (use-after-free in `ngx_http_ssi_filter_module` processing crafted proxied backend response). Two non-security `Change`s: HTTP/2 response header/trailer size now limited by `proxy_buffer_size`/`grpc_buffer_size` in `ngx_http_proxy_v2_module`+`ngx_http_grpc_module`; external-entity loading disabled by default in `ngx_http_xslt_filter_module` (new `xml_external_entities` directive re-enables it). Plus bugfixes (HTTP/2 out-of-order pseudo-headers, flow control, Upgrade header in HTTP/2/3 responses). NONE of these changes affect the custom-html static-file-serving use case (no `map`, no `slice`/cache, no SSI, no xslt, no proxy_v2/grpc). No breaking config changes — no operator action required beyond the image pull. diff --git a/cc-ci-plan/upstream/gitea.md b/cc-ci-plan/upstream/gitea.md index a522edc..33e214b 100644 --- a/cc-ci-plan/upstream/gitea.md +++ b/cc-ci-plan/upstream/gitea.md @@ -10,8 +10,27 @@ This is where CVE-2026-60004 + CVE-2026-59774 (both CVSS 9.8, fixed in 1.27.1) w and why the 2026-08-03/07 reports under-counted gitea's CVEs. advisory-scan.py fetches every URL in this file, so keep vendor security pages listed here. | db | postgres | https://github.com/postgres/postgres | https://www.postgresql.org/docs/release/ | +| db (mariadb overlay) | mariadb | https://github.com/MariaDB/server | https://mariadb.com/docs/release-notes/community-server/10.11 (series index — the CVE tables live on the PER-RELEASE pages, e.g. https://mariadb.com/docs/release-notes/community-server/10.11/10.11.19) | ## Standing notes +- **mariadb overlay** (compose.mariadb.yml): pinned `mariadb:10.11.x` (10.11 LTS, maintained until + Feb 2028). ⚠ BOTH db overlays (postgres + mariadb) define the same service name `db`, and + `abra recipe upgrade -m` machine output keeps only ONE `db` entry — the postgres one (files are + processed alphabetically; the mariadb result is clobbered). ALWAYS check the mariadb pin against + Docker Hub directly; the weekly survey's single "db:" line hides it. Per-release security notes + (CVE tables — GitHub release bodies only link out): + https://mariadb.com/docs/release-notes/community-server/10.11/10.11.3.md + https://mariadb.com/docs/release-notes/community-server/10.11/10.11.6.md + https://mariadb.com/docs/release-notes/community-server/10.11/10.11.8.md + https://mariadb.com/docs/release-notes/community-server/10.11/10.11.11.md + https://mariadb.com/docs/release-notes/community-server/10.11/10.11.12.md + https://mariadb.com/docs/release-notes/community-server/10.11/10.11.15.md + https://mariadb.com/docs/release-notes/community-server/10.11/10.11.16.md + https://mariadb.com/docs/release-notes/community-server/10.11/10.11.17.md + https://mariadb.com/docs/release-notes/community-server/10.11/10.11.18.md + https://mariadb.com/docs/release-notes/community-server/10.11/10.11.19.md + (the 10 CVE-bearing releases of the 10.11.2→10.11.19 window, 2026-09-11; append newer pages for + later windows). - **postgres major version**: recipe uses plain postgres (not pgautoupgrade); pg_backup.sh uses pg_dump/psql (logical backup, not pg_upgrade). Major version bump (e.g. 15→16) requires operator to manually restore from logical backup — no auto-migration tooling. Bump only within same major (e.g. 15.13→15.18) unless recipe adds pg_upgrade support. - **compose.postgres.yml**: postgres is an optional overlay (not in the main compose.yml). The cc-ci tests determine whether the postgres overlay is used. `abra recipe upgrade` reads it too and will propose a db bump — hold db unless doing a migration-aware major bump. - **1.25.0**: Removed deprecated auth sources — legacy auth setups may need reconfiguration. diff --git a/cc-ci-plan/upstream/hedgedoc.md b/cc-ci-plan/upstream/hedgedoc.md index ef952f9..dd6a766 100644 --- a/cc-ci-plan/upstream/hedgedoc.md +++ b/cc-ci-plan/upstream/hedgedoc.md @@ -7,5 +7,7 @@ ## Standing notes - hedgedoc 1.11.0 (2026): 4 security CVEs fixed (HTML injection, YAML DoS, CSRF via Gist export, rate-limit bypass). No breaking changes, no migrations, no schema changes. Optional new env var `CMD_RATE_LIMIT_USING_CLOUDFLARE` only needed if running behind Cloudflare — not required for standard deployments. -- pgautoupgrade: handles Postgres major-version upgrades automatically on container start. Bump ONE major at a time (16→17, then 17→18 on next cycle). The image tag is `-alpine`. -- cc-ci tests use the sqlite backend (default compose.yml), not the postgresql compose override — so pgautoupgrade bumps do not affect CI test coverage. +- pgautoupgrade: handles Postgres major-version upgrades automatically on container start; multi-major in ONE deploy works (16.15→18.6 verified live 2026-09-18 — pg_upgrade ran directly on the v16 files to 18.6, data survived). The image tag is `-alpine`. +- pgautoupgrade 18+ (RESOLVED on PR #3, 2026-09-18): PG 18 changes the data-directory layout (docker-library/postgres#1259; PGDATA → `/var/lib/postgresql//docker`, `/var/lib/postgresql/data` becomes a symlink → mount error or EMPTY instance if the mount stays at .../data). The fix carried by PR #3: move the volume mount to `postgres:/var/lib/postgresql` (NO PGDATA env pin needed) — the entrypoint's `MOVING_TO_NEW_STRUCTURE` detects `/var/lib/postgresql/PG_VERSION`, upgrades, and moves data into `/docker` automatically. Verified live on cc-ci 2026-09-18: `Upgrade to PostgreSQL 18.6 complete.`, pad data byte-identical, app green; 16→17 separately verified 2026-09-11 (`Upgrade to PostgreSQL 17.11 complete.`). Upstream coopcloud renovate PR #22 bumps the tag WITHOUT this mount fix — do not merge that alone. +- hedgedoc 1.12.0 (2026-08): requires Node 20.17+ (app image bundles its own node — not recipe-relevant); webp uploads; some highlight.js languages removed/changed. No breaking changes or migrations for the recipe. +- cc-ci tests use the sqlite backend (default compose.yml), not the postgresql compose override — so pgautoupgrade bumps do not affect CI test coverage (verify pg bumps via a direct --chaos deploy with the postgresql override instead). diff --git a/cc-ci-plan/upstream/immich.md b/cc-ci-plan/upstream/immich.md index 29e16e7..3a35964 100644 --- a/cc-ci-plan/upstream/immich.md +++ b/cc-ci-plan/upstream/immich.md @@ -50,6 +50,35 @@ Immich is now **up-to-date** (v3.1.0 is latest stable per GitHub releases; no v3.1.x patch / v3.2.x). PR #4 now diverges from upstream main only on `database` (pgvectors0.3.0 vs 0.2.0) + `redis` (valkey:9@8e8d64b4 vs @3acc0687) pins — operator decides merge-vs-close. +- **immich-server v3.2.0 (2026-09-10) pins** (from its `docker/docker-compose.yml` at the v3.2.0 tag): + `valkey:9@sha256:70739f85ad2ee01a726a965584a0f94895f01b0c60b3cc8b0aeef11eaa6888cf` + (= valkey **9.1.1** per image env; a NEWER build than v3.1.0's `8e8d64b4…` = 9.1.0 — but OLDER + than the live `9` tag, which by 2026-09-11 moved to `c123e371…` = 9.1.2; pin immich's tested + `70739f85…`, NOT the live tag, per the 2026-07-17 precedent) and + `postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357…` (SAME combo as v3.1.0 — DB + unchanged). No breaking server-side changes in v3.2.0; `example.env` byte-identical to v3.1.0; + TypeORM migrations auto-run. Operator-info only: the new opt-in cluster-groups feature requires + a per-group facial-recognition reset (user-triggered button, not a deploy migration). **ghcr + pagination gotcha (hit 2026-09-11):** `ghcr.io/v2/immich-app/postgres/tags/list` pages are ~100 + tags, NOT purely lexicographic, and the `Link:` header returns a RELATIVE url (`/v2/...`) — + prefix the host or pagination silently stops at page 1; the pinned pg tag `14-vectorchord0.4.3- + pgvectors0.2.0` is on **page 2** of 174 (a first-page probe wrongly suggests it's gone). Upstream + coopcloud has open renovate PRs #18 (v3.2.0 monorepo bump = server+ML only), #17 (valkey digest + → live c123e37 — not immich's tested pin), #16 (postgres → pg16 — unsupported by immich v3.2.0, + reject). Done in the 2026-09-11 upgrade (v3.1.0→v3.2.0, extending PR #4 from upstream main + 794560f; PR #4's pgvectors0.3.0/valkey-8e8d64b4 divergences superseded — DB realigned to + immich's official combo, which upstream main itself chose 2026-08-07). +- **immich-server v3.2.1 (2026-09-14) + v3.2.2 (2026-09-15) pins** (from `docker/docker-compose.yml` + @ the v3.2.2 tag): `postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357…` and + `valkey:9@sha256:70739f85ad…` — **BOTH identical to the recipe's v3.2.0-era pins** (DB and + redis stay put; live valkey `9` tag has drifted to `c123e371…` = 9.1.2 but immich's tested + pin `70739f85…` = 9.1.1 is kept per the 2026-07-17 precedent; live postgres tag digest + re-verified = the pin, not stale). `example.env` @ v3.2.2 byte-identical to v3.2.0. Both + releases are bug-fix-only (v3.2.1: sync connection-pool exhaustion, search modal, partner + assets on people page, person merge, server auto-VACUUM after TypeORM migrations; v3.2.2: + reassign-faces fix) — no breaking changes, no config additions, no operator action. Done in + the 2026-09-18 upgrade (v3.2.0→v3.2.2, app+ML only; fresh PR — PR #4 was closed when its + v3.2.0 content merged upstream as #19 / `1.11.0+v3.2.0`). - **2026-08-07 INFRA note: cc-ci runner's gitea clone-token is STALE (HTTP 401).** `!testme` build #1210 died at the recipe `git clone` step (~5s, before any deploy) with `could not read Username for 'https://git.autonomic.zone'` — the runner's mounted gitea token `13e299f2…` is rejected (verified diff --git a/cc-ci-plan/upstream/lasuite-docs.md b/cc-ci-plan/upstream/lasuite-docs.md index dcbe1e0..f781c03 100644 --- a/cc-ci-plan/upstream/lasuite-docs.md +++ b/cc-ci-plan/upstream/lasuite-docs.md @@ -6,10 +6,10 @@ | backend | lasuite/impress-backend | https://github.com/suitenumerique/docs | https://github.com/suitenumerique/docs/releases | | celery | lasuite/impress-backend | https://github.com/suitenumerique/docs | https://github.com/suitenumerique/docs/releases | | y-provider | lasuite/impress-y-provider | https://github.com/suitenumerique/docs | https://github.com/suitenumerique/docs/releases | -| docspec | ghcr.io/docspecio/api | https://github.com/docspecIO/api | https://github.com/docspecIO/api/releases | +| docspec | ghcr.io/docspec/api | https://github.com/docspec/docspec | https://github.com/docspec/docspec/releases | | db | pgautoupgrade/pgautoupgrade | https://github.com/pgautoupgrade/docker-pgautoupgrade | https://github.com/pgautoupgrade/docker-pgautoupgrade/releases | | redis | redis | https://github.com/redis/redis | https://hub.docker.com/_/redis/tags | -| minio | minio/minio | https://github.com/minio/minio | https://github.com/minio/minio/releases | +| minio | quay.io/minio/minio | https://github.com/minio/minio | https://github.com/minio/minio/releases | | web | nginx | https://github.com/nginx/nginx | https://nginx.org/en/CHANGES | ## Standing notes @@ -36,3 +36,17 @@ - redis sidecar (cache/broker for celery+backend): prefer the patch line (8.8.1 over 8.10.0) unless a feature is needed — 8.8.1 is a security patch (RedisBloom/TDigest RESTORE RCE); 8.10.0 is a minor with many new features. +- **2026-09-18: docspec image switched with impress v5.7.0** (upstream PR suitenumerique/docs#2553): + `ghcr.io/docspecio/api:3.0.x` (Elixir, repo ARCHIVED) → `ghcr.io/docspec/api:1.21.4` (Rust rewrite, + github.com/docspec/docspec). New image listens on **3000** (was 4000): recipe sets + `DOCSPEC_API_URL=http://docspec:3000/conversion` and healthcheck `http://127.0.0.1:3000/health` + (path `/conversion` and the raw-body request contract unchanged). +- ⚠️ When checking GHCR tags by script: the `/tags/list` endpoint CAPS AT 100 tags — a missing tag + there means nothing. Verify the **manifest** endpoint instead (e.g. `ghcr.io/docspec/api:1.21.4` + is absent from tags/list but pulls fine). +- **2026-09-13ish: upstream #25 moved the minio pin to `quay.io/minio/minio`** (same + RELEASE.2025-09-07T16-13-09Z tag — manifest-verified pullable on quay; quay's own tags/list is + paginated and misleading, use manifest checks there too). +- **v5.7.0 moved the favorites API endpoint to `/documents/favorites/`** — breaking only for + external API consumers, not the recipe/cc-ci tests (which use /authenticate/, /users/me/, + /documents/). diff --git a/cc-ci-plan/upstream/lasuite-drive.md b/cc-ci-plan/upstream/lasuite-drive.md index 51aaacf..6266b86 100644 --- a/cc-ci-plan/upstream/lasuite-drive.md +++ b/cc-ci-plan/upstream/lasuite-drive.md @@ -9,8 +9,8 @@ | db | pgautoupgrade/pgautoupgrade | https://github.com/pgautoupgrade/docker-pgautoupgrade | https://github.com/pgautoupgrade/docker-pgautoupgrade/releases | | redis | redis | https://github.com/redis/redis | https://github.com/redis/redis/releases | | mailcatcher | sj26/mailcatcher | https://github.com/sj26/mailcatcher | https://github.com/sj26/mailcatcher/releases | -| minio | minio/minio | https://github.com/minio/minio | https://github.com/minio/minio/releases | -| minio-createbuckets | minio/minio | https://github.com/minio/minio | https://github.com/minio/minio/releases | +| minio | quay.io/minio/minio | https://github.com/minio/minio | https://github.com/minio/minio/releases | +| minio-createbuckets | quay.io/minio/minio | https://github.com/minio/minio | https://github.com/minio/minio/releases | | collabora | collabora/code | https://github.com/CollaboraOnline/online | https://www.collaboraonline.com/release-notes/ (per-version: .../collabora-online-25-04-release-notes/) | | onlyoffice | onlyoffice/documentserver-de | https://github.com/ONLYOFFICE/DocumentServer | https://github.com/ONLYOFFICE/DocumentServer/blob/master/CHANGELOG.md | | web | nginx | https://github.com/nginx/nginx | https://nginx.org/en/CHANGES | @@ -19,9 +19,12 @@ - lasuite/drive-frontend and lasuite/drive-backend share the same version tag (drive monorepo). - minio and onlyoffice tags use non-semver formats; abra cannot auto-detect upgrades for them. - collabora/code uses a 5-part version scheme; abra cannot auto-detect upgrades for it. -- minio: Docker Hub `minio/minio` `latest` is frozen at `RELEASE.2025-09-07T16-13-09Z` (the - `RELEASE.2025-10-15T17-29-55Z` GitHub security release is NOT on Docker Hub `minio/minio`). - Newer minio releases moved to the **AIStor** product line (`quay.io/minio/aistor/minio`). - A future minio bump requires a registry+product switch — operator decision, not a routine tag bump. +- minio: Docker Hub `minio/minio` was **REMOVED** (verified 2026-09-18: Hub repo API 404, registry + manifest 401 on anonymous pull; last successful docker.io pull was the 2026-09-11 run). The same + release tag lives on **`quay.io/minio/minio`** (MinIO's official distribution channel) — recipe + pins switched there in lasuite-drive PR #7 (2026-09-18), same release `RELEASE.2025-09-07T16-13-09Z`, + mirroring the identical fix merged upstream for lasuite-docs ("fix: pull MinIO image from Quay", + PR #25, 2026-09-15). A future minio **version** bump still requires the AIStor product-line + decision — quay.io only re-hosts the frozen community line. - nginx 1.31.x (1.31.0, 1.31.1) contains multiple security CVE fixes; upgrade is recommended. - nginx 1.31.0 breaking change: HTTP/2 and HTTP/3 requests with Connection/Proxy-Connection/Keep-Alive/Transfer-Encoding/Upgrade headers are now rejected (affects proxied HTTP/2 backends). diff --git a/cc-ci-plan/upstream/mattermost-lts.md b/cc-ci-plan/upstream/mattermost-lts.md index be8c356..b87f8c6 100644 --- a/cc-ci-plan/upstream/mattermost-lts.md +++ b/cc-ci-plan/upstream/mattermost-lts.md @@ -6,6 +6,37 @@ | postgres | postgres | https://github.com/postgres/postgres | https://www.postgresql.org/docs/release/ | ## Standing notes + - **2026-09-18 re-check** (Mattermost release-policy docs; GitHub release; Docker Hub; vendor + security feed): **11.7 ESR line UNCHANGED** — newest patch **11.7.11** (released 2026-09-15, + "Low to High severity level security fixes"; Docker Hub tag pushed 2026-09-15). v10.11 ESR + remains EXPIRED (2026-08-15); 10.11.24 sits on the dead line — do NOT target. This run + extended open PR #2 **11.7.10 → 11.7.11** (branch tip 5696f0b4). Direct `--chaos` deploy on + cc-ci exercised the full operator path (deploy at 10.11.22 → seed admin user via API → + upgrade to 11.7.11): schema migrations ran on boot, **seeded user data survived** (same + id/roles, login 200), ping 200 with x-version-id 11.7.11; `!testme` **GREEN**, drone 1373 + (level 5: install/upgrade/backup/restore/custom all pass, clean teardown). Vendor feed: **6 + server advisories fixed in 11.7.11** (MMSA-2026-00711 Low; -00761/-00763/-00764/-00765/-00773 + Medium, all affecting 11.7.x <= 11.7.10; CVE ids pending 30-day disclosure, due 2026-10-15). + **CVE-2026-13426 re-adjudicated FIXED** for the 10.11.22 → 11.7.11 window (fresh GitHub + compare: fix commit 3321db82 is an ancestor of v11.7.11 — ahead/behind 0 — and diverged from + v10.11.22) → window CVE count **14** (13 deterministic + 1 adjudicated). postgres **15-alpine + HELD** (DB-major out of scope). Gotchas learned: `abra app new` RESETS the recipe checkout to + main (re-checkout the branch before a `--chaos` deploy), and `abra app new --domain` does NOT + write DOMAIN into the app .env (edit the .env + redeploy to fix traefik routing). + - **2026-09-11 re-check** (Mattermost Server Releases docs; endoflife.date/api/mattermost.json; + GitHub releases + module tags): **11.7 ESR line UNCHANGED** — newest patch still **11.7.10** + (2026-08-26; no 11.7.11; v11.11.0 is still a GitHub **prerelease**, innovation). **10.11 ESR + expired 2026-08-15; 10.11.23 (2026-08-13) was its FINAL patch** — the "10.x is LTS, never + 11.x" survey hint appeared a THIRD time (2026-06-26, 2026-08-07, 2026-09-11) and was again + NOT followed: no supported 10.x LTS exists (10.12 = innovation, expired 2025-12-15). PR #2 + re-verified unchanged (tree 59e8c2c0; direct `--chaos` deploy converged + ping 200; + `!testme` GREEN, drone 1355). **Adjudicated CVE-2026-13426 (MMSA-2025-00532, medium) FIXED by + the 10.11.22 → 11.7.10 window**: module fix `server/public` v0.1.22 (commit 3321db82, tag cut + 2026-01-30 from the 11.4.0 cycle) is an ancestor of v11.7.10 but NOT of v10.11.22/v10.11.23 + (GitHub compare evidence) → CVE count for the ESR move = **14** (13 deterministic + 1 + adjudicated). postgres **15-alpine HELD** (recipe has no compose overlays beyond compose.yml; + no pg_upgrade/pgautoupgrade support in recipe or cc-ci tests; the floating 15-alpine tag picks + up 15.x patches on re-pull — pg 15.19 fixes CVE-2026-14662/14663/14664). - **2026-08-28 re-check** (endoflife.date/api/mattermost.json 2026-08-28; Docker Hub; GitHub releases): **11.7.10** (released 2026-08-26, "various bug fixes", not prerelease) is the newest 11.7.x ESR patch — the ESR/LTS line has NOT moved, still **11.7** (EOL **2027-05-15**). This run diff --git a/cc-ci-plan/upstream/n8n.md b/cc-ci-plan/upstream/n8n.md index e897ba6..258334d 100644 --- a/cc-ci-plan/upstream/n8n.md +++ b/cc-ci-plan/upstream/n8n.md @@ -171,3 +171,71 @@ flag (unchanged from 2.37.0): the two API behavior changes only affect HTTP API callers. Recommended release: `-y` (feature minor across 2.34.4 → 2.38.4). CVE-2026-73088 was adjudicated NOT-FIXED in the 2.x window by the 2026-09-04 parent run; not blocking. +- 2.38.5 (2026-09-09, patch): 1 core fix (keep reporting the original job error when the job key is + gone). +- 2.38.6 (2026-09-10, patch; **Stable/Latest badge**): 1 perf fix (bound peak memory during source + control push). +- 2.39.0 (2026-09-08, Pre-release): the 2.39 feature minor — large feature/bugfix release. Features: + source-control push/pull + status endpoints in the public API, **workflow version endpoint + (deprecates the old two-variable path — API-level deprecation)**, Instance reporting module, opt-in + concurrency limits for Instance AI runs, OTLP gRPC support for OTel, Confluence/Jira Atlassian + Service Account (2LO) auth, MS Teams channel-message + Online Meeting ops, Databricks partner UA + + hidden chat model, Instance AI folder exploration / past-conversation search / credential-selection + guidance, **encryption-key module rework** (always load the encryption module and seed keys safely + on startup; cache keys in memory; pluggable write path with rotation moved into the module; signing + deployment keys stored in the same wrapped form as data-encryption keys), deprecate + hide the Ask + AI tab in the Code node. Many core/editor/node fixes (webhook Set-Cookie response headers; OTel + header redaction in settings; Azure OpenAI classic/Foundry credential fix; OIDC failed-login + redirect; Qdrant client update for Node 26; AMQP trigger receiver reattach; Anthropic disabled + thinking; etc.). No breaking compose/env/migration changes; no `N8N_*` env renames. +- 2.39.1 (2026-09-09, Pre-release): 1 core fix — **repair legacy-format data-encryption keys during + bootstrap** (upgrade-relevant: old-format keys are repaired, not rejected). +- 2.39.2 (2026-09-10, Pre-release): 2 core fixes (memoize pairedItem ancestry traversal; reuse an + existing consent grant after the visitor authenticates mid-flow) + 1 perf (stop copying user + workflows into the Instance AI sandbox). Docker Hub `2.39.2` manifest verified multi-arch + (amd64 + arm64, active, 2026-09-10). +- 2026-09-11 run: survey framed 2.34.4 → 2.39.2, but the morning reconcile closed PR #7 as + merged-upstream and upstream main released **3.5.0+2.38.4** — so the effective compose window is + **2.38.4 → 2.39.2** on a FRESH branch (no open upgrade PR to extend). 2.38.6 holds the + Stable/Latest badge; 2.39.x is Pre-release (consistent tracking-the-newest-tag precedent: + 2.34.2/2.35.2/2.36.3/2.37.3/2.37.6/2.38.4 were all Pre-release when taken). Rolling upgrade safe + (sqlite mode, TypeORM auto-migrate on boot). Live-deploy watch item: the 2.39.0 encryption-key + module rework + 2.39.1 legacy-key repair (recipe injects N8N_ENCRYPTION_KEY as a secret). + Operator flags, all HTTP-API-caller-level only: 2.33.0 workflow activate/deactivate deprecation; + 2.36.0 Array.merge→mergeIntoObject + workflow-tags API migration; 2.37.0 "Any workflow" caller-policy + deprecation + JSON content-type on decorator body routes + binary-data endpoint adapt; 2.39.0 + workflow-version endpoint deprecation. Recommended release: `-y` (feature minor). +- 2.38.7 (2026-09-11, patch): 3 core fixes (MCP toolkit members on workers; consent-grant reuse; + trust the CA from GIT_SSL_CAINFO for source-control HTTPS remotes). +- 2.39.3 (2026-09-11, Pre-release): 1 core fix (GIT_SSL_CAINFO trust — 2.38.7 backport). +- 2.39.4 (2026-09-11, Pre-release): 1 core fix (sandbox npm install flags by provider). +- 2.39.5 (2026-09-14, stable line): editor fix (revoke end-user credentials) + feature + (instance-report log streaming events). +- 2.39.6 (2026-09-16, stable line): core fix (Gateway credits eligibility) + features (**v3 + breaking-change rule for the storage-directory rename**; **warn on deprecated N8N_DB_PING_TIMEOUT + env var** — recipe does not set it). +- 2.39.7 (2026-09-17, stable line; **`stable` tag points here** — the stable badge moved to the + 2.39.x line): core breaking-change-detection resilience; editor numeric credential inputs. +- 2.40.0 (2026-09-15, Pre-release, feature minor — large): Microsoft Dataverse core node; AI Agent + forced first-iteration tool calls; Git-based promotion model (selective promote, timestamped + branches, promotion entities/DTOs); **workflow publication service enabled by default**; + ephemeral sandboxes; agent episodic memory; ai_preference table; HTTP Request + PROPFIND/MKCOL/MOVE/COPY/REPORT; Grist OAuth2; MS Teams online-meeting/channel-message ops; v3 + migration-report rules (storage-directory rename, preExecute) + "chat hub goes away" warning; + N8N_DB_PING_TIMEOUT deprecation warning; secret-redaction consolidation; queue-mode execution + timeout for stuck jobs; ~150 core/editor/node bugfixes. **No breaking compose/env/migration + changes; no N8N_* renames.** +- 2.40.1 (2026-09-16, Pre-release): GitHub release body EMPTY (maintenance republish, no listed + changes). +- 2.40.2 (2026-09-17, Pre-release): core breaking-change-detection resilience + editor numeric + credential inputs (the 2.39.7 pair on the 2.40 line). Docker Hub manifest verified active. +- 2.40.3 (2026-09-18, Pre-release): core "repair data-encryption keys stored as the raw instance + key" + editor blank-workflow-preview fix. Published the morning of the 2026-09-18 run — newer + than the operator's snapshot (which named 2.40.2 as newest); NOT taken that run (operator's + explicit target 2.40.2), flagged in the plan/PR as a trivial follow-up bump. +- 2026-09-18 run: PR #8 (branch upgrade-ef0dd56, at 2.39.2 since 2026-09-11, !testme GREEN then) + extended **2.38.4 → 2.40.2** on upstream main tip 0b436ec (still 3.5.0+2.38.4 — upstream has not + started the bump). Stable badge on 2.39.7; 2.40.x pre-release per precedent. Rolling upgrade + safe (sqlite, TypeORM auto-migrate). Operator flags unchanged (API-caller-level deprecations, + encryption-key rework informational, N8N_DB_PING_TIMEOUT warn-only, recipe doesn't set it). + 2.40.3 exists (see above) — flagged, not taken. Recommended release: `-y`. diff --git a/flake.lock b/flake.lock index 7529cd0..0d4e490 100644 --- a/flake.lock +++ b/flake.lock @@ -10,11 +10,11 @@ ] }, "locked": { - "lastModified": 1788820034, - "narHash": "sha256-mkktTMeGKutfPp3pn3AhoPoGSCm4iRSnPvpmbEpMmwA=", + "lastModified": 1790010051, + "narHash": "sha256-0dz8fg4zqg54tk+1A7CHZAw+lK9vWxre6Tioi5B0ocU=", "ref": "refs/heads/main", - "rev": "b11cc0b7385aee9fb89bd9a7ed23896401d683e5", - "revCount": 1543, + "rev": "2f6787f0433e7ff3053116039f10c00cde34128f", + "revCount": 1558, "type": "git", "url": "https://git.autonomic.zone/recipe-maintainers/cc-ci.git" }, diff --git a/memory/MEMORY.md b/memory/MEMORY.md index 0f35100..a46baae 100644 --- a/memory/MEMORY.md +++ b/memory/MEMORY.md @@ -3,7 +3,8 @@ - [Orchestrator host: Hetzner](orchestrator-host-hetzner.md) — runs on Hetzner cpx22; rebuild cmd, loops-service bounce, git-identity gotcha - [Push commits to remote](push-commits-to-remote.md) — push to git.autonomic.zone right after every commit in this repo - [Regression canary cadence](regression-canary-cadence.md) — server E2E canaries run on polish/review/release, not every commit -- [Recipe-mirrors public / org blocker](recipe-mirrors-public-org-blocker.md) — mirrors public but recipe-maintainers ORG is private → live PR-STATUS column dark until operator flips org public +- [Recipe-mirrors MUST be public](recipe-mirrors-public-org-blocker.md) — org public since ~2026-06, but /recipe-enroll created late mirrors (gitea, wordpress) private-from-birth, darkening report STATUS cells; flipped public + skill fixed 2026-09-14 +- [Gitea Anubis UA challenge](gitea-anubis-ua-challenge.md) — git.autonomic.zone 307-challenges browser UAs to an un-CORS-able origin; server-side proxies representing browser JS must pin a non-browser UA (cc-ci PR #38, 2026-09-14) - [abra chaos-deploy checkout gotcha](abra-chaos-deploy-checkout-gotcha.md) — `abra app new` moves recipe checkout to release tag; checkout PR branch after, or chaos deploys wrong tree - [Shared recipe-checkout race](shared-recipe-checkout-race.md) — never git-checkout ~/.abra/recipes/ on cc-ci while its CI build runs; harness deploys from that tree - [immich pgvecto.rs DROP DATABASE panic](immich-pgvectors-drop-database-panic.md) — DROP DATABASE crashes immich's postgres image; use pg_dump --clean --if-exists + search_path rewrite diff --git a/memory/gitea-anubis-ua-challenge.md b/memory/gitea-anubis-ua-challenge.md new file mode 100644 index 0000000..57d2393 --- /dev/null +++ b/memory/gitea-anubis-ua-challenge.md @@ -0,0 +1,26 @@ +--- +name: gitea-anubis-ua-challenge +description: "Gitea (git.autonomic.zone) sits behind Anubis, which 307-challenges browser-like User-Agents to an un-CORS-able counter-domain — any server-side proxy that forwards a browser UA on behalf of client JS breaks; pin a non-browser UA" +metadata: + node_type: memory + type: project +--- + +`git.autonomic.zone` sits behind **Anubis** (`anubis.swarm.autonomic.zone`). Anubis +307-challenges requests whose User-Agent looks like a real browser to +`/.within.website/?redir=…` — a JS proof-of-work challenge page on a *different* origin with +**no CORS headers**. + +Impact: any server-side proxy that serves client-side JS but forwards the *end browser's* UA +hits this asymmetry — `curl` (non-browser UA) passes through cleanly, a real Firefox/Chrome UA +gets the 307. Concretely: the Recipe Report's same-origin PR-STATUS proxy +(`report./pr//`, cc-ci `nix/modules/reports.nix`) forwarded the browser UA, so +every live cell in the browser rendered `?` (week-2026-09-11, operator-CORS console report). + +Fix (cc-ci PR #38, merged 2026-09-14): `proxy_set_header User-Agent "ccci-reports-proxy/1.0";` +in the `/pr/` nginx location — a stable non-browser UA passes Anubis unmolested. Deployed via +`nix flake update cc-ci` → `nixos-rebuild test` → health → `switch` (flake.lock commit). + +Rule of thumb: server-side callers of the Gitea API (scripts, proxied fetches, harness code +running in a browser's name) must pin a non-browser User-Agent or cookie-carry; browser-visit +flows keep the JS challenge and that's fine. Related: [[recipe-mirrors-public-org-blocker]]. diff --git a/memory/recipe-mirrors-public-org-blocker.md b/memory/recipe-mirrors-public-org-blocker.md index cb167a1..cca0e30 100644 --- a/memory/recipe-mirrors-public-org-blocker.md +++ b/memory/recipe-mirrors-public-org-blocker.md @@ -1,29 +1,26 @@ --- name: recipe-mirrors-public-org-blocker -description: "Recipe mirrors are public repos but the recipe-maintainers ORG is private-visibility, so anon reads 404; bot can't flip the org" -metadata: +description: "Recipe mirrors MUST be public (fleet invariant: the report's tokenless PR-STATUS proxy only sees public repos); org is public since ~2026-06, but late-enrolled mirrors (gitea, wordpress) were created private-from-birth by /recipe-enroll — fixed 2026-09-14" +metadata: node_type: memory type: project originSessionId: f7960036-d990-4a21-a81e-f7c486d97fea --- -As of 2026-06-09 all 21 recipe mirrors under `recipe-maintainers` were flipped `private=false` -(secret-scanned first), to power the Recipe Report's live PR-STATUS column via the tokenless -same-origin proxy `report.ci.commoninternet.net/pr//` (shipped in cc-ci -`nix/modules/reports.nix`). BUT the **org itself is `visibility: private`**, which makes Gitea 404 -all its repos for anonymous users — so the live STATUS column shows a muted "?" instead of open/✓. +The Recipe Report's live PR-STATUS column (`report.ci.commoninternet.net/pr//`, +shipped in cc-ci `nix/modules/reports.nix`) is a tokenless same-origin nginx proxy to the Gitea +API — it can ONLY see public repos. So **`recipe-maintainers/` mirrors MUST be public** +(fleet invariant; cc-ci/cc-ci-orchestrator/archived repos stay deliberately private). -**Blocker:** `autonomic-bot` cannot flip the org (PATCH `/orgs/recipe-maintainers` → 403 "Must be an -organization owner"; `is_admin=false`; the basic-auth credential lacks `write:organization` scope, -even though the bot is in the Owners team). Confirmed model: `autonomic-cooperative` is a public org -and its repos ARE anonymously visible; `recipe-maintainers` is private and they are not. +History: originally ALL mirrors were private; on 2026-06-09 the 21 then-existing mirrors were +flipped `private=false` after a secret-scan, and the `recipe-maintainers` org was later flipped +public by the operator (the old 'org is private' blocker is RESOLVED). -**Why:** the whole live-status feature is dark until this is resolved. Private repos stay hidden even -in a public org, so flipping the org public does NOT expose the four locked-private repos (`cc-ci`, -`cc-ci-secrets`, `cc-ci-orchestrator`, `archived-cc-ci-orchestrator`). - -**How to apply:** operator (an org owner) must set `recipe-maintainers` org visibility to **public** -in the Gitea UI (Settings → make org public), OR provision a token with `write:organization` scope. -The instant that happens, the proxy returns 200 PR JSON and the column lights up — no redeploy needed. -Verify: `curl https://report.ci.commoninternet.net/pr/cryptpad/5` should return PR JSON, not a 404. +Recurrence (2026-09-14): /recipe-enroll still carried the old instruction to create mirrors with +`"private":true` — so the later-enrolled mirrors `gitea` (2026-06-11) and `wordpress` (2026-08-03) +were **private from birth**, leaving the report STATUS cells for those rows permanently "?" +(gitea/wordpress rows in week-2026-09-11). Fixed: secret-scanned both, flipped `private=false` +via PATCH `/repos/recipe-maintainers/`, patched the skill to create `private:false`. The +report now lights up for every row — verify any new PR row with +`curl https://report.ci.commoninternet.net/pr//` returning PR JSON, not 404. Related: [[push-commits-to-remote]]. diff --git a/memory/weekly-upgrader-opencode-go-glm.md b/memory/weekly-upgrader-opencode-go-glm.md new file mode 100644 index 0000000..0bb3ed5 --- /dev/null +++ b/memory/weekly-upgrader-opencode-go-glm.md @@ -0,0 +1,111 @@ +--- +name: weekly-upgrader-opencode-go-glm +description: weekly upgrade run now defaults to opencode-go/glm-5.3-flash (PR #20, 2026-09-14); zen tier dead on this host; how backend/model + monitoring are wired +metadata: + node_type: memory + type: project + originSessionId: 85355980-5e4f-4f90-b1ca-d0e4fe82f04b +--- + +The weekly recipe-upgrade run (`cc-ci-plan/launch-upgrader.py`, fired by the +`cc-ci-upgrade-all` systemd timer Thu 22:00 Boston) now **defaults to backend=opencode, +model=opencode-go/glm-5.3-flash** when no env is set (operator choice 2026-09-08 via +`/srv/cc-ci/upgrader.env`; PR #20 hardcoded the same defaults into the launchers 2026-09-14 — +for all three launchers: upgrader, report, supervisor). Model default tracks the +backend — an explicit `LOOP_BACKEND=claude` (no model) falls back to `sonnet`. + +**2026-09-14 — ZEN IS DEAD ON THIS HOST.** The ZEN provider models (`opencode/glm-5.2`) are +gone from the provider catalogue AND the host holds no ZEN key, so any launch that bypassed +upgrader.env died instantly with `ProviderModelNotFoundError: opencode/glm-5.2` — this is what +killed the 2026-09-11 report agent at launch and every manual retry (all valid ZEN credentials +mean nothing without the ZEN models existing server-side). If a launcher log shows +`ProviderModelNotFoundError`, the model id does not resolve on this host: run +`opencode models | grep glm` for what does, then fix the tier/model default. See PR +recipe-maintainers/cc-ci-orchestrator#20. + +**Provider:** "OpenCode **Go**" — a flat $10/mo subscription, DISTINCT from "opencode Zen" +(pay-as-you-go). Endpoint `https://opencode.ai/zen/go/v1/` (OpenAI-compatible), model id +**`opencode-go/glm-5.2`** (Zen tops out at glm-5.1 — `opencode.ai/zen/v1` does NOT have 5.2; +updating the opencode CLI does NOT change this, the catalog is server-side). API key lives in +`~/.local/share/opencode/auth.json` (mode 600, entry `opencode-go` type api) — **never in the +repo**. opencode CLI installed: 1.15.13. + +**Override per run without a rebuild:** drop `/srv/cc-ci/upgrader.env` (read via +`EnvironmentFile=-` on the unit) with e.g. `LOOP_BACKEND=claude` / `LOOP_MODEL=sonnet`. Absent +file → the glm-5.2 default. + +**Monitoring (opencode backend):** the launcher does `opencode run -m --share --attach +http://127.0.0.1:4096`. `opencode-web.service` (=`opencode serve` :4096, nginx → +http://oc.commoninternet.net, tailnet-only) must be active. `OPENCODE_SHARE=1` (default) also +mints a public `https://opncd.ai/share/` link (printed in the TUI / readable from the server +API `GET /session/` → `.share.url`). Note: launch-upgrader.py's `session_busy()` regex is +claude-tuned, so it misreports an active opencode run as "idle/finishing" — check the process +(`opencode run`) or web UI instead. + +Gotcha fixed: opencode `-m/--model`/`--attach`/`--share`/`--title` are flags on the `run` +SUBCOMMAND — they must come AFTER `run` (a global `opencode --model X run` is silently ignored). + +`launch-report.py` also defaults to opencode-go/glm-5.2 now (overridable via REPORT_BACKEND/REPORT_MODEL). +In the weekly flow the upgrader writes the summary inline AND auto-launches the report (upgrade-all §6). + +**Usage limit + auto-resume watchdog (added 2026-06-23):** opencode-go enforces a ROLLING usage window +(~5h) capped by dollar value. A full glm-5.2 upgrade run is heavy — ~8 recipes (≈2h20m wall) exhausted +it mid-run, returning HTTP 429 + `retry-after` (seconds) on the go endpoint. When limited, the +`opencode run` agent loop ENDS and does NOT self-resume (process lingers but idle). Fix: `launch-upgrader.py` +now has `resume` (continue the SAME session, context preserved, via `opencode run -s --continue` — +finds the session from the web server, kills the idle proc via /proc scan NOT `pkill -f` which self-matches) +and `watchdog` (polls the session log; on >15min stall, waits out the 429 retry-after then auto-resumes). +An opencode `start` auto-spawns the watchdog; logs to `cc-ci-upgrader-watchdog.log`. To resume by hand: +`python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py resume`. (Subcommand `watchdog`, matching agents.py — +renamed from `babysit` 2026-06-23.) + +**Watchdog covers ALL parts of the weekly run (2026-06-23):** (1) it runs in a tmux session +(`-watchdog`) under the persistent tmux server, NOT a Popen child — so it survives the +`cc-ci-upgrade-all` systemd Type=oneshot whose cgroup is reaped on exit (a Popen child would be killed). +(2) The watchdog is generic (env: `UPGRADER_SESSION/_MODEL/_DONE_MARKER/_RESUME_FILE`), so `launch-report.py` +reuses it for the REPORT job (`cc-ci-report` session, marker `RECIPE REPORT COMPLETE`) — the report runs on +glm-5.2 sharing the same opencode-go budget the upgrade run drains, so it can 429-stall too. `_run_pids()` +is scoped to the managed session (title or `-s `) so the two watchdogs never kill each other's process. +Weekly timer: Thu 22:00 America/New_York; no `LOOP_BACKEND`/`upgrader.env` override ⇒ glm-5.2 defaults. +See [[orchestrator-backend-switch-gotcha]]. + +**2026-07-04 hardening (after the 2026-07-03 run stalled on a HOST DISK-FULL and never resumed — +the log-idle/429 watchdog does NOT cover an environmental wedge).** Three changes (commits 1bd156e, +399e999), host rebuilt: +- **Step-0 stale-image pre-reclaim** (`launch-upgrader.py` `prereclaim_cc_ci()`): before each weekly + run, `ssh cc-ci docker image prune -af --filter until=168h` — only images unused AND >1 week old, so + this week's likely-reused images stay. Best-effort; env `UPGRADER_PRERECLAIM*`. A full run's images + pile up week-over-week and had run root to 100%. (The 2026-07-03 pile was 93 GB / 194 images; pruning + took it 88%→17%.) +- **watchdog proc-death detection**: an `opencode run` proc EXITS when the model ends its turn, so a + long /upgrade-all dies+restarts many times before completing (and the log mtime freezes on death, + making log-idle useless). `watchdog()` now resumes on "no live run proc for the session + not + `_completed()`", not just log-idle. Capped at `UPGRADER_MAX_RESUMES` (20) → hands off if stuck. + `resume()` now auto-spawns a watchdog if none is alive (skips when the watchdog itself called it). +- **Hourly glm-5.2 SUPERVISOR** (`launch-supervisor.py`, systemd `cc-ci-upgrade-supervisor` .service + + .timer, hourly at :07): cheap deterministic gate — no-ops (0 tokens) if the run is complete / + progressing / a per-run watchdog is alive; only when a run stalled+died AND the watchdog gave up does + it launch a short-lived glm-5.2 agent to diagnose (disk/wedge) + drive to completion. `force`/`status`/ + `stop` subcommands. Progress judged by `lu._run_pids(sid)` + log mtime (session_busy() is claude-tuned, + misreads a headless opencode run as idle — do NOT use it for opencode). Window guard: won't + auto-resurrect a session >96h old. Overrides via `/srv/cc-ci/upgrader.env` (e.g. `SUPERVISOR_MODEL=`). + Ownership: **watchdog = prompt recovery (single writer while alive); supervisor = backstop when the + watchdog is absent/gave up.** They're mutually exclusive (gate defers while watchdog tmux alive). + +**2026-07-04 finish + more watchdog fixes (commits 69dd178, ddedea8, b6a95c4).** Finishing the stalled +2026-07-03 run exposed that a glm-5.2 `opencode run` is FLAKY for long autonomous jobs — its turn ENDS +(proc exits) repeatedly, `--continue` of a bloated session often doesn't advance, and it can wedge on a +hung subagent. Bugs fixed in `launch-upgrader.py`: (1) `_session_idle_min()` now measures staleness +across the whole session TREE (parent + descendant subagents) via the opencode server time.updated — +NOT the tmux log mtime (freezes on `--continue`) and NOT the parent alone (a 20-40min recipe subagent +made the parent look idle → false-resume killed the productive run). (2) watchdog loop is wrapped in +try/except so a transient blip can't kill it. (3) **`_completed()` false-positive**: it matched +DONE_MARKER inside TOOL parts (a subagent `task` prompt / bash echoing "print UPGRADE RUN COMPLETE") → +watchdog exited early, run wedged. Now requires the marker in the LAST assistant TEXT (prose) message. +(4) supervisor gate also treats the run as done once the weekly summary file exists (a run can finish +work+summary+report yet never print a clean marker). **Operational fallback that WORKED:** when the +report agent wedged twice, published the report by hand from the deterministic path — +`recipe-report.py survey ` → build the spec JSON from survey.json → `render` → `publish`. This +is reliable and bypasses the flaky agent entirely; keep it as the go-to when an agent-driven report +stalls. The 2026-07-03 report is live at report.ci.commoninternet.net/week-2026-07-03.html (16 GREEN, +plausible + lasuite-drive RED, discourse 5-PR reconcile). diff --git a/nix/hosts/cc-ci/configuration.nix b/nix/hosts/cc-ci/configuration.nix index 3f171c2..0b64691 100644 --- a/nix/hosts/cc-ci/configuration.nix +++ b/nix/hosts/cc-ci/configuration.nix @@ -30,8 +30,10 @@ # Weekly self-update (Tue 03:00 UTC; skips itself while CI is busy; see nix/modules/auto-update.nix). cc-ci-orchestrator.autoUpdate.enable = true; - # The opencode UI: traefik (public 443, the *.ci.commoninternet.net cert) → nginx basic auth. - cc-ci-orchestrator.opencodeUiHost = "oc.ci.commoninternet.net"; + # The opencode UI: traefik (public 443, the dual-zone wildcard cert) → nginx basic auth. + # Domain cutover 2026-09: new name primary; legacy name kept answering during the bake window. + cc-ci-orchestrator.opencodeUiHost = "oc.ci.autonomic.zone"; + cc-ci-orchestrator.opencodeUiExtraHosts = [ "oc.ci.commoninternet.net" ]; cc-ci-orchestrator.opencodeUiTraefikNetwork = "proxy"; # ---- no tailscale on this host (operator 2026-09-07) -------------------------------------- @@ -124,7 +126,13 @@ # This host's own public names resolve to itself regardless of external DNS state (host # processes: the drone runner, the harness, the orchestrator; containers use the resolvers # above). Per-run recipe domains are random and cannot be pinned — those follow public DNS. + # Domain cutover 2026-09: new names pinned; legacy names kept during the bake window. networking.hosts."195.201.88.249" = [ + "ci.autonomic.zone" + "drone.ci.autonomic.zone" + "report.ci.autonomic.zone" + "traefik.ci.autonomic.zone" + "oc.ci.autonomic.zone" "ci.commoninternet.net" "drone.ci.commoninternet.net" "report.ci.commoninternet.net" diff --git a/nix/modules/cc-ci.nix b/nix/modules/cc-ci.nix index a170ff4..d42936e 100644 --- a/nix/modules/cc-ci.nix +++ b/nix/modules/cc-ci.nix @@ -92,15 +92,13 @@ User = "loops"; Group = "users"; WorkingDirectory = "/srv/cc-ci"; # Optional per-run overrides for backend/model (LOOP_BACKEND, LOOP_MODEL, OPENCODE_SHARE, - # UPGRADER_ARGS, …). The leading "-" makes it optional: absent file → claude/sonnet defaults. - # Current config (as of 2026-08-16): the upgrader + report run on tinfoil/deepseek-v4-pro - # (LOOP_MODEL + REPORT_MODEL in the env file); the hourly SUPERVISOR stays on glm-5.2 - # (SUPERVISOR_MODEL defaults to opencode-go/glm-5.2 in launch-supervisor.py, NOT overridden - # here). Subagents bind deepseek via the cc-ci repo's opencode config. LOOP_TIER=zen is kept - # so the tier check passes; the watchdog's usage-limit probe sends the deepseek model name to - # the zen endpoint, which returns 200 (not 429) → resume immediately (correct: tinfoil has no - # rolling usage limit to wait out). No rebuild needed to switch — the env file is read at each - # timer fire. Holds no secrets (the tinfoil API key lives in the opencode config / auth.json). + # UPGRADER_ARGS, …). The leading "-" makes it optional: absent file → the in-code defaults, + # which are opencode + LOOP_TIER=go (the only tier with a key on this host; ZEN models are + # dead here — the in-code default is 'go' for exactly that reason, see launch-upgrader.py). + # Current config: LOOP_TIER=go with LOOP_MODEL / REPORT_MODEL / SUPERVISOR_MODEL all + # opencode-go/glm-5.3-flash (upgrader, report, supervisor). Subagents bind deepseek via the + # cc-ci repo's opencode config. No rebuild needed to switch — the env file is read at each + # timer fire. Holds no secrets (the API key lives in the opencode config / auth.json). EnvironmentFile = "-/srv/cc-ci/upgrader.env"; }; environment = { HOME = "/home/loops"; }; diff --git a/nix/modules/orchestrator-host.nix b/nix/modules/orchestrator-host.nix index 6e4e806..ab6ed03 100644 --- a/nix/modules/orchestrator-host.nix +++ b/nix/modules/orchestrator-host.nix @@ -62,6 +62,16 @@ in description = "nginx server_name for the opencode web UI (TLS + basic auth)."; }; + opencodeUiExtraHosts = lib.mkOption { + type = lib.types.listOf lib.types.str; + default = [ ]; + description = '' + Extra hostnames routed to the opencode UI alongside opencodeUiHost, rendered as + `||`-joined Host conditions on the same router. Domain cutover 2026-09: the + legacy oc.ci.commoninternet.net keeps answering here during the bake window. + ''; + }; + opencodeUiHtpasswdFile = lib.mkOption { type = lib.types.str; default = "/secrets/nginx/oc-htpasswd"; @@ -216,6 +226,7 @@ SSHCFG recommendedProxySettings = true; virtualHosts.${cfg.opencodeUiHost} = { listen = [ { addr = "0.0.0.0"; port = cfg.opencodeUiBackendPort; } ]; + serverAliases = cfg.opencodeUiExtraHosts; basicAuthFile = cfg.opencodeUiHtpasswdFile; extraConfig = '' # traefik sits on the docker networks (ingress 10.0.0.0/24, gwbridge 172.18.0.0/16) @@ -256,7 +267,7 @@ SSHCFG replicas: 1 labels: - "traefik.enable=true" - - "traefik.http.routers.opencode-ui.rule=Host(`${cfg.opencodeUiHost}`)" + - "traefik.http.routers.opencode-ui.rule=${lib.concatStringsSep " || " (map (h: "Host(`${h}`)") ([ cfg.opencodeUiHost ] ++ cfg.opencodeUiExtraHosts))}" - "traefik.http.routers.opencode-ui.entrypoints=web-secure" - "traefik.http.routers.opencode-ui.tls=true" - "traefik.http.services.opencode-ui.loadbalancer.server.port=${toString cfg.opencodeUiBackendPort}" diff --git a/references/recipe-maintainer b/references/recipe-maintainer index acd5cfb..6698723 160000 --- a/references/recipe-maintainer +++ b/references/recipe-maintainer @@ -1 +1 @@ -Subproject commit acd5cfbd9581ea096382aaff38859aed4d4ba0c9 +Subproject commit 6698723a77a8d51301e947fa131b4f3931ce3883 diff --git a/scripts/gen-cctest-skills.py b/scripts/gen-cctest-skills.py index a01d3e7..4a94ffa 100644 --- a/scripts/gen-cctest-skills.py +++ b/scripts/gen-cctest-skills.py @@ -50,6 +50,76 @@ run `/cctest-intro` / `/cctest-setup-sandbox` first. a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration. """ +# Per-skill extra body sections appended after BODY_TEMPLATE (survive regeneration). +PER_SKILL_NOTES: dict[str, str] = { + "recipe-upstream": """ +**Sandboxed vs non-sandboxed mode.** This skill needs only git + (optionally) the Gitea API — +not the test server — so it can run either way. Probe, then follow that branch: + +- **Sandboxed** (ARM env configured: `test-ssh/.testenv` with `GITEA_USERNAME`/`GITEA_PASSWORD`/ + `GITEA_URL`, sandbox/test instances): the canonical `recipe-upstream` script in + `references/recipe-maintainer/.claude/commands/recipe-upstream.md` runs as written. Note its + WORKSPACE probing expects `/workspace` or `~/Documents/recipe-maintainer`; on a bare host pass + the submodule dir explicitly instead. +- **Non-sandboxed** (no ARM env on the host — e.g. the orchestrator, where the recipe-maintainer + checkout is only a pinned submodule): no sandbox/test instances are needed and NONE of the + setup skills are. Recipe-maintainer mirrors on `git.autonomic.zone` are publicly readable, so: + 1. Check out the recipe if missing: `abra recipe fetch ` (lands in `~/.abra/recipes/`) + — or a plain anonymous `git clone https://git.autonomic.zone/recipe-maintainers/.git` + if abra is unavailable. + 2. Fetch the PR head branch from the mirror **anonymously** — no credentials in the remote URL: + `git remote add gitea https://git.autonomic.zone/recipe-maintainers/.git` + (remote update rather than re-add if it exists), then + `git fetch gitea +refs/pull//head:refs/heads/`. + 3. Fetch PR metadata (head/base refs, merged flag, release bump line) from + `https://git.autonomic.zone/api/v1/repos/recipe-maintainers//pulls/` — + unauthenticated; use bot creds only if the repo turns out to be private (orchestrator hosts + can read them from `/srv/cc-ci-orch/.testenv` — never written anywhere else). + 4. Everything else in the canonical script (origin/dev remote setup, release recommendation, + emitted next-steps) is identical. + +Every time the branch was prepared **here**, remember it exists only on this host — the +operator's machine must fetch it first. Always emit this **step 0** before the push step +(anonymous public fetch, no credentials needed): + +``` + # 0. On a machine WITHOUT the branch pre-fetched, get it from the autonomic mirror + # (fetch by URL — works regardless of what the local remotes are named): + cd + git fetch https://git.autonomic.zone/recipe-maintainers/.git +refs/pull//head:refs/heads/ + git checkout +``` + +If the operator's checkout does NOT yet have the mirror remote, emit once before the fetch: + +``` + git remote add gitea https://git.autonomic.zone/recipe-maintainers/.git +``` + +In both modes the final output is a set of commands for the operator to run on a machine **with +push access to `git.coopcloud.tech`** — always print them, even when everything local is +already prepared. + +**Verify the merge-base BEFORE rebasing — bases can be traps.** Upstream repos can +carry a stray, divergent `main` ALONGSIDE the real `master` base branch. Rebase targets must be +chosen by evidence: `git ls-remote origin | grep -E 'refs/heads/(main|master)$'`, then +`git merge-base origin/` — the correct base is the one where the upgrade +branch's merge-base is its tip's parent (i.e. `` is already directly on that line — +then NO cherry-pick is needed, the raw branch compares clean). Cherry-picking onto the wrong +base produces the very messy 3-dot compare this guard exists to prevent (real case: gitea — +first "rebase" landed on a stray `main` and had to be redone onto `master`). + +**Branch-name mismatch mirror vs upstream.** The mirror and upstream can use different base-branch +names, and a mirror can even carry a stray, divergent `main` alongside the real base (gitea: mirror +PR against `main`, upstream `master`; keycloak PR #9: base `main`, both mirror and upstream on +`master`). The canonical script now detects the upstream default branch by evidence (`git +ls-remote --symref origin HEAD`, then the git.coopcloud.tech API, then the PR base with a warning) +and uses it for the compare URL AND the post-merge release steps — never hardcode `main`, and when +it prints the base-mismatch warning, confirm the upgrade branch actually sits on that line +(`git merge-base` against both candidates) before emitting the push step. +""", +} + WRAPPER_TEMPLATE = """# {wrapped} (thin wrapper) The canonical definition of this skill lives in the **opencode** position: @@ -99,7 +169,7 @@ def main() -> int: ) canon = ROOT / ".opencode/skills" / wrapped / "SKILL.md" canon.parent.mkdir(parents=True, exist_ok=True) - canon.write_text(frontmatter + BODY_TEMPLATE.format(name=name, wrapped=wrapped)) + canon.write_text(frontmatter + BODY_TEMPLATE.format(name=name, wrapped=wrapped) + PER_SKILL_NOTES.get(name, "")) thin = ROOT / ".claude/skills" / wrapped / "SKILL.md" thin.parent.mkdir(parents=True, exist_ok=True) thin.write_text(frontmatter + WRAPPER_TEMPLATE.format(name=name, wrapped=wrapped))