Compare commits

..
Author SHA1 Message Date
notplants bd7196bf9e Improve recipe advisory coverage and report non-CVE fixes 2026-11-02 19:40:46 +00:00
autonomic-bot 508d6bb0de Merge pull request 'fix: restore CVEs report column heading' (#35) from fix/report-cves-heading into main 2026-11-02 19:16:17 +00:00
notplants 72b3e1223a fix: restore CVEs report column heading 2026-11-02 19:16:01 +00:00
autonomic-bot aa6a3f095c Merge pull request 'fix: verify GHCR releases when image tag list is huge' (#34) from fix/ghcr-large-tag-fallback into main 2026-11-02 19:13:03 +00:00
notplants 9f103c3516 fix: verify GHCR release manifests when tag listing is huge 2026-11-02 19:12:28 +00:00
autonomic-bot 371546d1f8 Merge pull request 'fix: fail closed on incomplete weekly image and CVE surveys' (#33) from fix/weekly-update-security-survey into main 2026-11-02 19:09:48 +00:00
notplants b41eeaed75 fix: fail closed on incomplete weekly image and CVE surveys 2026-11-02 19:08:54 +00:00
autonomic-bot e5cfa67e6e Merge pull request 'chore: update cc-ci runner input after test fixes' (#32) from chore/update-cc-ci-runner-input into main 2026-11-02 18:37:24 +00:00
notplants a1357e6866 chore: update cc-ci runner input after test fixes 2026-11-02 18:37:08 +00:00
autonomic-bot f59eb1480f Merge pull request 'chore: update cc-ci input with disk pressure cleanup' (#31) from fix/cc-ci-disk-prune-input into main 2026-11-02 18:17:15 +00:00
notplants babb83debf chore: update cc-ci input with disk pressure cleanup 2026-11-02 18:16:56 +00:00
autono-bot 93fe2b6f1a journal: rustdesk-server one-off upgrade + enrollment (PR #1 GREEN !testme, cc-ci #48 merged, host switch); memory: gate fixtures must fail loudly + --cc-ci-only deploy path 2026-10-30 21:24:06 +00:00
auto-update 4e286ba777 flake.lock: cc-ci -> main 2026-10-30 21:16:05 +00:00
trav cd1ec01b23 used-recipes: rustdesk-server weekly (enrolled 2026-10-30, cc-ci PR #48) 2026-10-30 18:59:10 +00:00
trav 0fe2827889 upstream(rustdesk-server): release-notes sources (one-off upgrade; not enrolled) 2026-10-30 07:55:19 +00:00
autonomic-bot bb76edde9f journal(upgrade-all): weekly run 2026-10-30 — 12 green, 1 pending canonical v6 seed, 1 failed (mailu backup defect) 2026-10-30 04:58:53 +00:00
autonomic-bot 7f3627e76c upstream(immich): postgres pgvectors 0.2.0->0.3.0 window notes 2026-10-30 03:25:38 +00:00
autonomic-bot 6234e36261 upstream(cryptpad): version-2026.10.0 now on Docker Hub; 2026.10 recipe notes 2026-10-30 02:21:52 +00:00
autonomic-bot 78bbf763ff journal(upgrade-all): weekly run 2026-10-23 — 9 green, 2 pending warm-canonical minio fix
Only this event entry committed; pre-existing uncommitted orchestrator WIP left in the worktree untouched.
2026-10-23 04:05:49 +00:00
autonomic-bot c57ddfd7f1 upstream(n8n): 2.43.7-2.45.2 notes + 2026-10-23 run window 2026-10-23 03:58:32 +00:00
autonomic-bot 5d422207f0 upstream(lasuite-docs): v6 yhub stack, runtime RSA keys, docspec 1.x 2026-10-23 03:31:15 +00:00
autonomic-bot 4b8b617d8c upstream(immich): release-notes sources (v3.3.3-v3.3.5) 2026-10-23 03:13:04 +00:00
cc-ci bot 53b1a76f70 upstream(gitea): add 28.x release/security notes 2026-10-23 02:43:34 +00:00
autonomic-bot f4ed2e3fbe upstream(cryptpad): Docker Hub tag pointer + 2026.10.0/opendesk notes 2026-10-23 02:24:24 +00:00
autonomic-bot 7df1792dc9 Merge pull request 'config: drop mattermost-lts from weekly upgrade set (-> external)' (#30) from chore/external-mattermost-lts into main 2026-10-19 16:56:55 +00:00
autonomic-bot afa6922093 config: drop mattermost-lts from weekly upgrade set (-> external)
Move mattermost-lts from weekly to external in used-recipes.md, matching the
plausible precedent. The weekly /upgrade-all survey, upgrade PRs and their
!testme verification no longer run for it; cc-ci still deploys/tests it and
the recipe + tests stay enrolled.
2026-10-19 16:56:43 +00:00
autonomic-bot 50776ecb0c upstream(n8n): 2.42.3-2.44.2 release notes + 2026-10-09/16 run windows 2026-10-16 03:26:50 +00:00
autonomic-bot 1b3ae31091 upstream(custom-html): nginx 1.31.7 security-window notes 2026-10-16 02:19:30 +00:00
autonomic-bot c97798ff8f upstream(immich): v3.3.0/v3.3.1 pins + release-window notes 2026-10-09 02:55:01 +00:00
autonomic-bot 4a043b12ca Merge pull request 'orchestrator-host: weekly opencode auto-upgrade (latest-release check, reinstall, restart web)' (#29) from opencode-auto-upgrade into main 2026-10-05 18:40:31 +00:00
autonomic-bot 88e5a93f94 orchestrator-host: weekly opencode auto-upgrade (latest-release check, reinstall, restart web)
opencode-install only installs when the binary is missing, so the standalone
CLI aged in place (1.18.29 for weeks while 1.18.33+ was out); opencode's
built-in autoupdate never fires here because every agent runs inside the
long-lived opencode serve. New opencode-upgrade.service + weekly timer
(Tue 02:00 UTC, an hour before the host auto-update) compares the installed
version with the latest GitHub release, reinstalls via the official
installer when they differ, restarts opencode-web so the new binary takes
effect, and verifies the UI answers its 401 challenge. The auto-update.nix
busy gate is replicated so a mid-flight CI run / weekly upgrade / sweep is
never cut (skipped runs retry next week). Deploying this module does not
itself bump opencode: boot installs stay install-if-missing and the upgrade
is timer-driven only. State per run: .cc-ci-logs/opencode-update-state.
2026-10-05 18:39:35 +00:00
autonomic-bot b6bfbe2a3d journal: upgrader session 2026-10-02 (8 green PRs; gitea 28.0.0 era; mirror master-branch repair; lasuite-docs base blocker persists) 2026-10-02 04:19:53 +00:00
autonomic-bot 5c6a1dbcf7 upstream(n8n): 2.40.6-2.42.2 release notes + 2026-10-02 run window 2026-10-02 04:12:34 +00:00
autonomic-bot 36019ed8c8 upstream(immich): v3.2.4 pins + release-window notes 2026-10-02 03:43:41 +00:00
autonomic-bot 004c9bcdd3 journal: upgrader session 2026-09-28 (7 green PRs; subagent model bump deepseek-v4.1-flash; quay minio 401 watch) 2026-09-28 21:54:41 +00:00
autonomic-bot 075c9d356d upstream(n8n): release notes 2026-09-21..28 window 2026-09-28 21:42:04 +00:00
23 changed files with 956 additions and 89 deletions
+16 -6
View File
@@ -95,8 +95,9 @@ For each recipe produce **one window per image**: `current pinned tag → newest
the sidecars (redis, postgres, nginx …), not just the app — a sidecar bump is where discourse's only the sidecars (redis, postgres, nginx …), not just the app — a sidecar bump is where discourse's only
CRITICAL came from, and an image with no window is not counted at all. CRITICAL came from, and an image with no window is not counted at all.
- **No upgrade available** → the recipe is `UPTODATE`; its CVE count is **`0`**, not `?`. There is - **No upgrade available** → the recipe is `UPTODATE`; its **CVEs fixed by an available upgrade**
nothing an upgrade could fix. Record it and move on. count is `0`. This does not assert that the running image has no known CVEs. If advisory sources
failed or are blind, use `?` and say why.
- **No output at all is NOT "no upgrade".** An abra call that times out, FATAs, or prints nothing - **No output at all is NOT "no upgrade".** An abra call that times out, FATAs, or prints nothing
leaves the recipe **unverified** — treat it as a distinct outcome, never fold it into up-to-date. leaves the recipe **unverified** — treat it as a distinct outcome, never fold it into up-to-date.
Re-run it, and if it still yields nothing, resolve the versions by direct registry check (box item 4). Re-run it, and if it still yields nothing, resolve the versions by direct registry check (box item 4).
@@ -111,9 +112,17 @@ python3 cc-ci-plan/audit-sources.py --security-sources
A recipe whose sources yield **no CVE data at all** cannot produce a meaningful `0` — nothing was A recipe whose sources yield **no CVE data at all** cannot produce a meaningful `0` — nothing was
measured, the same way a missing registry file cannot. Render those as **`?`**, not `0`. measured, the same way a missing registry file cannot. Render those as **`?`**, not `0`.
**The fleet is currently at zero such recipes.** The last two — `mattermost-lts` (empty advisory Do not assume that the fleet has zero blind recipes. Run the audit every sweep and inspect source
feed, client-side-rendered bulletins) and `mumble` (nothing published anywhere) — were fixed by statuses; an HTTP error or rate limit is not evidence of no CVEs. `rustdesk-server` has an NVD CPE
declaring an NVD CPE in their registry: scoped to the OSS server edition; a successful zero-result query is a checked zero for that source,
not a reason to count Client or Server Pro vulnerabilities against it.
Check `test -s /srv/cc-ci/.github-token` before a fleet scan. Without that read-only token,
GitHub's anonymous API limit can be exhausted partway through the fleet; HTTP 403 results are
failed sources, so the affected CVE counts remain unknown.
Two older blind spots — `mattermost-lts` and `mumble` — were addressed by declaring NVD CPEs in
their registries:
``` ```
- nvd-cpe: mattermost-team-edition = cpe:2.3:a:mattermost:mattermost_server:*:*:*:*:*:*:*:* - nvd-cpe: mattermost-team-edition = cpe:2.3:a:mattermost:mattermost_server:*:*:*:*:*:*:*:*
``` ```
@@ -169,7 +178,8 @@ Severity outranks raw count: 2 CVSS-9.8 RCEs matter more than 120 medium plugin
- `table[]` — every recipe swept. `recipe`; `change` = the window you scanned, e.g. - `table[]` — every recipe swept. `recipe`; `change` = the window you scanned, e.g.
`1.27.0 → 1.27.1 · redis 7.4 → 8.10`; `status` = `UPTODATE` when nothing is available, else `1.27.0 → 1.27.1 · redis 7.4 → 8.10`; `status` = `UPTODATE` when nothing is available, else
`PENDING` (an upgrade exists and is not yet taken); **`cve`** = the count (integer, `?` only per the `PENDING` (an upgrade exists and is not yet taken); **`cve`** = the count (integer, `?` only per the
rules below); `notes` = severity mix, whether the number is a floor, and `maintained elsewhere` for rules below, or `"YES"` when a confirmed security advisory has no CVE ID and no numbered CVEs
were found); `notes` = the advisory ID and severity, whether the number is a floor, and `maintained elsewhere` for
`external` rows. **Leave `ci`/`pr` empty — nothing was built and no PR exists.** `external` rows. **Leave `ci`/`pr` empty — nothing was built and no PR exists.**
- `addendum[]` — real anomalies only: registry URLs that failed, recipes whose window could not be - `addendum[]` — real anomalies only: registry URLs that failed, recipes whose window could not be
established, a scan whose count is a floor with many undetermined advisories. established, a scan whose count is a floor with many undetermined advisories.
+13 -4
View File
@@ -30,7 +30,9 @@ keeps every weekly edition looking the same regardless of which model writes the
2. **Survey.** `python3 /srv/cc-ci/cc-ci-plan/recipe-report.py survey <DATE> > /tmp/survey.json`. This 2. **Survey.** `python3 /srv/cc-ci/cc-ci-plan/recipe-report.py survey <DATE> > /tmp/survey.json`. This
gives you: the raw `/upgrade-all` summary; for every recipe mirror its open PRs + the `cc-ci/testme` gives you: the raw `/upgrade-all` summary; for every recipe mirror its open PRs + the `cc-ci/testme`
verdict; and each recipe's **per-recipe upgrade notes** (`upgrade_notes_md` — the breaking-change / verdict; and each recipe's **per-recipe upgrade notes** (`upgrade_notes_md` — the breaking-change /
**CVE / security** analysis the upgrade did). Read it all. **CVE / security** analysis the upgrade did). Read it all. Also run
`python3 /srv/cc-ci/cc-ci-plan/audit-sources.py --security-sources`; a recipe with no verified
security source or a failed source must be called out, not presented as a clean CVE scan.
3. **Review & classify.** The page order is: **short lead → the full wire table (priority-sorted) → 3. **Review & classify.** The page order is: **short lead → the full wire table (priority-sorted) →
Addendum → Security Bulletin → per-recipe "What changed".** Same newspaper style; this order. Addendum → Security Bulletin → per-recipe "What changed".** Same newspaper style; this order.
@@ -46,9 +48,16 @@ keeps every weekly edition looking the same regardless of which model writes the
lists **failed sources**, the count is **not** authoritative: render the cve cell as `?` lists **failed sources**, the count is **not** authoritative: render the cve cell as `?`
(unknown), never `none` — a blank that reads as "clean" is exactly how two CVSS-9.8 gitea (unknown), never `none` — a blank that reads as "clean" is exactly how two CVSS-9.8 gitea
RCEs were reported as "none" on 2026-08-07. RCEs were reported as "none" on 2026-08-07.
- **`?` must stay RARE — it means "we tried and could not tell", not "we didn't look".** Use it - The table's **CVEs** column counts known CVEs addressed by the available recipe upgrade.
ONLY when a scan ran and reported genuinely failed sources, **or when the scan block says A zero for an up-to-date recipe means there is no upgrade window to credit; it does not
COUNT UNKNOWN**. In that case the scan's `0` means *not determined*: publish `?` and say so in certify that the running images are free of vulnerabilities. If an upgrade exists but its
advisory scan is absent, a source failed, or an image window was unmatched, use `?`.
If a confirmed security advisory is fixed but has no CVE ID, put `"YES"` in `cve` rather
than `0`; describe its advisory ID and severity in Notes. If CVE IDs are also fixed, show
the numeric CVE count and mention the additional non-CVE advisories in Notes.
- **`?` means the count is not verified.** Use it when an upgrade has no scan, a source failed,
an image window was unmatched, or the scan block says COUNT UNKNOWN. In that case publish
`?` and say why in
the notes; publishing `0` would assert a clean bill of health nothing supports. Note a the notes; publishing `0` would assert a clean bill of health nothing supports. Note a
**version-scheme change is no longer a reason for `?`** — the scan resolves semver→calver jumps **version-scheme change is no longer a reason for `?`** — the scan resolves semver→calver jumps
(discourse 3.5.3 → 2026.7.1) by falling back to advisory publish dates and reports a real number. (discourse 3.5.3 → 2026.7.1) by falling back to advisory publish dates and reports a real number.
+25 -6
View File
@@ -143,12 +143,31 @@ ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; R=<r>; \
script -qec "abra recipe upgrade $R -m -n" /dev/null; \ script -qec "abra recipe upgrade $R -m -n" /dev/null; \
git -C ~/.abra/recipes/$R stash pop >/dev/null 2>&1 || true' git -C ~/.abra/recipes/$R stash pop >/dev/null 2>&1 || true'
``` ```
Build `RECIPES_TO_UPGRADE` = recipes with **≥1 available upgrade** (after the stash, so the untracked **Mandatory registry cross-check for EVERY recipe and EVERY image, even if abra says current:** after
overlay doesn't count as dirty). Others go to `SKIPPED_UPFRONT` with a reason (`dirty-worktree` only for reconciling the mirror and refreshing the recipe checkout, run
real TRACKED edits, `up-to-date`, `not-fetchable`). **A `tag and digest … not supported` parse FATA is `python3 /srv/cc-ci/cc-ci-plan/resolve-images.py <recipe> --recipe-dir /root/.abra/recipes`
NOT `not-fetchable`** — per box item 4, catch it and resolve availability via the direct upstream-tag on cc-ci. Read the JSON `images`, `upgrades_available`, `unresolved`, and `all_resolved` fields.
check; only `not-fetchable` when the mirror genuinely can't be fetched (network/auth), never for a The candidate list is the **union** of abra's upgrades and the resolver's upgrades. CryptPad's
digest-pinned image. `version-YYYY.M.D` app tag is a known counterexample: abra only reports a parse warning, while the
resolver finds the newer image. A warning is never evidence that an image is current. Inspect both
`newest_same_shape` and `newest_within_major`; a newer major is still a candidate for planning,
though it must not be applied without a compatibility check.
Build `RECIPES_TO_UPGRADE` from every recipe with at least one candidate. Only label a recipe
`up-to-date` if abra completed successfully **and** the resolver checked every relevant image
(`all_resolved: true`) **and** neither found a candidate. If an image is unresolved, templated,
floating, or lacks comparable tags, investigate it directly and record the evidence. Otherwise
record `unverified` with the image and reason; never fold it into `up-to-date`. `dirty-worktree`
applies only to real tracked edits. A tag+digest parse failure is not `not-fetchable`.
Run `python3 /srv/cc-ci/cc-ci-plan/audit-sources.py --security-sources` once for the full inventory
and include any `NO VERIFIED CVE SOURCE` recipes or failed advisory sources in the weekly summary.
Do not describe their CVE count as clean. This audit is a coverage check, not a substitute for
`advisory-scan.py` on each actual upgrade window.
Before the audit, check `test -s /srv/cc-ci/.github-token`. If it is missing, record that GitHub
advisory checks are subject to the anonymous 60-request/hour limit; any resulting HTTP 403 is a
failed source and must become `?` in the report. Never replace it with `0` or silently retry from
model memory. The file is a read-only GitHub token location; never print its contents.
**Reconcile every candidate's mirror during the survey** (even the up-to-date ones), so merged-upstream **Reconcile every candidate's mirror during the survey** (even the up-to-date ones), so merged-upstream
PRs are closed and every mirror `main` tracks true upstream — fleet-wide, not just where there's a new PRs are closed and every mirror `main` tracks true upstream — fleet-wide, not just where there's a new
+99
View File
@@ -1311,3 +1311,102 @@ harness → `results.json` + PR card `✅ passed` linking `ci.autonomic.zone/run
- NOTE: no tailscale on this host (`tailscale: command not found`) — the AGENTS.md "ssh cc-ci" - NOTE: no tailscale on this host (`tailscale: command not found`) — the AGENTS.md "ssh cc-ci"
alias + 100.90.116.4 peer notes are stale post-rebuild; public-IP SSH is the access path. alias + 100.90.116.4 peer notes are stale post-rebuild; public-IP SSH is the access path.
Recovery scripts in scripts/recovery/ still reference old server id 134485294 — worth updating. Recovery scripts in scripts/recovery/ still reference old server id 134485294 — worth updating.
---
## 2026-09-28 — upgrader session (weekly /upgrade-all)
Weekly run 2026-09-28 complete: 19 considered · 7 GREEN PRs · 1 externally-CI-blocked (lasuite-docs, quay minio 401) · 0 failed · 11 skipped. Summary (PR list): `<logs>/upgrades/upgrade-all-2026-09-28.md` — repo ignores .cc-ci-logs, so the PR list lives in the file + the weekly report.
**Operational config change (no PR):** subagent `general` model `deepseek-v4-flash` was killed upstream ("Model access is disabled", zen endpoint; glm-5.2 also dead) — pointed to `deepseek-v4.1-flash` in `/srv/cc-ci-orch/cc-ci/opencode.json`, `/srv/cc-ci-orch/opencode.json` (uncommitted) + live `/etc/cc-ci/opencode.json`. Prior committed setting still in the cc-ci repo's main. Operator: confirm + PR the bump (config change precedent #37/#42).
**minio watch:** docker.io removal (2026-09-18) → now quay.io/minio/minio 401s since ~09-24; lasuite-docs CI-blocked on it; lasuite-drive passed (image cached).
**abandoned worktree note:** `/srv/cc-ci-orch/cc-ci-conc/` (worktree of cc-ci, untracked in orch repo) has a stale worktree at 2f6787f; no writes this week; recorded so the next sweep notices.
Key logs: .cc-ci-logs/upgrades/*-2026-09-28.md (8 per-recipe + summary).
## 2026-10-02 — upgrader session (weekly /upgrade-all)
Weekly run 2026-10-02 complete: 20 considered · 9 upgraded (PRs opened/extended) · 8 GREEN (!testme) · 1 live-verified but CI-blocked externally (lasuite-docs base canonical, commented) + 1 RED on a genuinely stale gitea LFS test (commented; operator re-runs --with-tests) · 0 failed · 9 skipped. Summary (PR list): <logs>/upgrades/upgrade-all-2026-10-02.md — repo ignores .cc-ci-logs, so the PR list lives in the file + the weekly report.
Key highlights: gitea upstream shipped its first 2-digit stable (28.0.0-rootless, 2026-09-29 — BREAKING git-egress/actions-retention notes documented in PR #10); ghost 6.67.0 fixes HIGH CVE-2026-84383; nulls held at mysql 8.4 / mariadb 12.3 / postgres 13/15 lines per precedent; matrix-synapse extended to synapse 1.162.0 + MAS 1.26.0; immich bumped v3.2.4 (rc.3.3 declined). Operational: recipe-maintainers/gitea mirror had LOST its master branch (abra whole-recipe FATA) — repaired both refs to upstream HEAD f167b73; reconcile-upstream.sh should learn to recreate a missing upstream-default branch ref, not only sync main. 2 targets (gitea, n8n 2.42.2) were caught in follow-up passes after the initial worklist mis-derivation — nothing left unprocessed. Sweep start: 4 leaked volumes + 46.3 GB images reclaimed (disk 70%->40%); end-of-run reap 0 leaked dev deploys; disk 65% (51G free).
Per-recipe logs: .cc-ci-logs/upgrades/<recipe>-upgrade-2026-10-02.md (10 files incl. survey).
NEVER MERGED anything; all PRs await operator review + the operator's `abra recipe release <recipe> -x/y/z` after each merge.
### Event 2026-10-05 ~17:45 UTC — gitea PR #10 stale test fixed (cc-ci PR #44, verified GREEN, NOT merged)
Operator invoked --with-tests for gitea PR #10 (28.0.0-rootless, RED on test_lfs_roundtrip run #31).
Diagnosis: stale test — `_api()` caught only HTTPError so one urlopen(timeout=20) socket timeout
during the 28.0.0 post-restart migration window escaped the poll loop; poll also hard-coded 120s
deadline vs recipe_meta HTTP_TIMEOUT=600 (STYLE.md §5). Fix: never-raising _api (mirrors
lifecycle.http_fetch) + deadline from recipe_meta.HTTP_TIMEOUT; assertions untouched.
cc-ci PR #44 (branch fix/gitea-lfs-restart-wait) also carries the two PRE-EXISTING main lint
failures (warm.py ruff format; acme-dns.nix statix W201, eval-identical restructure) that had made
push build #33 RED and would have blocked merge — build #34 now success, lint.sh PASS.
Verified: full cold suite on host worktree (e83cd46) vs recipe head 91e42c8 → 5/5 tiers pass,
test_lfs_roundtrip PASSED; unit sample 315 pass / 4 fail = identical pre-existing failures on main.
Pair note posted on both PRs: merge #44, then re-!testme #10. Nothing merged.
### Event 2026-10-05 ~18:00 UTC — wrong-recipe card screenshot root-caused; runner fix #45 merged; gitea !testme re-run GREEN
Operator's !testme on gitea PR #10 (drone build 35) showed a custom-html-tiny screenshot in the card.
Root cause: drone build numbers restarted when its DB was re-created (2026-09-27) while
/var/lib/cc-ci-runs still holds old-era dirs (numbered to 1377) — build 35 collided with a May-31
run 35 whose fossil screenshot.png (12950 B = exactly custom-html-tiny's real shot) was embedded
because screenshot.capture()'s hook branch SKIPPED the snap when out_path existed (no hook saves
itself). Run itself was genuinely gitea level 5. Fix (cosmetics-only, R7): results.fresh_artifact_dir()
wipes owned artifacts at run start + capture() always snaps; 4 unit tests (non-vacuous: capture test
fails on main). Adversary verification PASS (independent fresh clones + own E2E collision test).
cc-ci PR #45 merged (squash, 35ba9976); push self-tests 38/39 green; lint gate now green on main
(#44 had already fixed the two pre-existing lint failures).
Merged #44 first (operator informed; verified earlier via full cold harness run).
Re-ran !testme on gitea PR #10 → drone build 40: GREEN, level 5, card links /runs/40/, screenshot
is a fresh PNG of the run's own gitea (footer 1.24.2 = the upgrade-tier BASE deploy, the last-green
warm canonical — pre-existing capture timing, correct behavior, only freshness was broken before).
Gitea PR #10 now fully green incl. the fixed LFS test (uses merged #44 test). NOTHING else merged;
recipe PR #10 + unrelated PR #4 still await operator merge + `abra recipe release gitea -x`.
Cleanup: host /tmp/v /tmp/r, local worktrees, test-file mis-commit in orch cc-ci clone reverted.
### Event 2026-10-23 04:15 UTC — weekly /upgrade-all run complete (DEFAULT mode, nothing merged)
Run 2026-10-23: 19 considered · 9 upgraded with !testme GREEN · 2 PRs open pending the warm-canonical
minio-prepull fix (lasuite-docs PR #11, lasuite-drive PR #8 — !testme RED at the upgrade BASE prepull:
warm canonicals 0.4.0+v5.4.1 / 0.12.0+v0.21.0 pin the REMOVED docker.io/minio/minio; PR comments
explain; operator: advance those canonicals then re-post !testme) · 0 failed · 8 skipped up-to-date
(cryptpad discourse drone hedgedoc keycloak mailu mumble wordpress — several had stale survey data;
each re-verified against true upstream before skipping). GREEN PRs: bluesky-pds #6 · custom-html #10 ·
custom-html-tiny #9 · ghost #8 · gitea #11 · immich #6 · lasuite-meet #12 · matrix-synapse #8 · n8n #9.
Summary: .cc-ci-logs/upgrades/upgrade-all-2026-10-23.md (per-recipe logs alongside). Step-0 sweep
reclaimed 19GB images (disk 61%->49%); end-of-run reap 0 leaked dev deploys.
NEVER MERGED anything; PRs await operator review + `abra recipe release` after each merge.
### Event 2026-10-30 02:00 UTC — weekly /upgrade-all run complete (DEFAULT mode, nothing merged)
Run 2026-10-30: 19 considered · 12 upgraded with !testme GREEN · 1 PR open pending its stale warm-canonical
base (lasuite-docs PR #11 — !testme RED at the base INSTALL rung: canonical 0.4.0+v5.4.1 references the
removed collab_ss secret; last week's minio blocker is already fixed via #47; head linted + live 12/12;
operator: seed a v6-cold canonical, re-post !testme) · 1 failed (mailu PR #9 — restore tier RED 3x on a
PRE-EXISTING upstream defect: main's new backup support lacks a restart post-hook so /data/main.db is
never restored; PR + operator comment with options) · 5 skipped (gitea + lasuite-meet were already merged
upstream by their turn; custom-html-tiny drone mumble up-to-date). GREEN PRs: bluesky-pds #6 · cryptpad #6
(2026.5.1→2026.10.0 breaking server rewrite handled) · custom-html #10→11 (stale nginx digest re-pin) ·
discourse #12 · ghost #8 · hedgedoc #3 (pgautoupgrade 16→18 in-place) · immich #7 (pgvectors 0.3.0
same-scheme re-tag) · keycloak #13 (mariadb 12.3→13.0 + AUTO_UPGRADE) · lasuite-drive #8 (+collabora +
onlyoffice direct checks) · matrix-synapse #8 · n8n #10 · wordpress #3.
Summary: .cc-ci-logs/upgrades/upgrade-all-2026-10-30.md (per-recipe logs alongside). Step-0 sweep
reclaimed 66.76GB images + orphan stacks (disk 92%->49%); end-of-run reap 0 leaked dev deploys.
NEVER MERGED anything; PRs await operator review + `abra recipe release` after each merge.
### Event 2026-10-30 19:00–21:30 UTC — rustdesk-server one-off upgrade + FULL ENROLLMENT (orchestrator)
One-off upgrade requested from scratch: public mirror `recipe-maintainers/rustdesk-server` created +
synced; recipe PR https://git.autonomic.zone/recipe-maintainers/rustdesk-server/pulls/1 (1.1.12→1.1.16,
branch upgrade-2e65135, unmerged — operator merges). Upstream checked (low-churn, stray 1.4.5 docker
tags = aborted push; CVE fix POV: UDP punch-hole amplification GH #670 + mio bump; scan 0 CVE-ids,
client-side CVEs fixed in client 1.4.9). Dev-deploy converged clean twice, then !== enrollment: tests
authored (status-key health sidecar overlay compose.ccci.yml serving the private-tool-unreachable
real hbbs public key; ports 21115-21119 + UDP; ws 101 on 21118/19; identity-key cross-source assert;
punch-hole-amplification regression test w/ version-gate), committed d71a53c, cc-ci PR #48.
Verify: run 1 RED (custom tier) — hand-rolled version read returned None so the punch-hole gate
didn't skip on the 1.1.12 head; fixed via harness lifecycle.deployed_identity + loud fail on
unparsable tag (1653582, f5f11cd); runs 2+3 GREEN level 5/5 (recipe main with gate skipping, AND
the upgrade head with the punch-hole test passing actively). PR #48 evidence-commented + MERGED
(200); --cc-ci-only deploy: flake.lock cc-ci 2f6787f→a8b836e, lock commit 4e286ba pushed,
/etc/cc-ci ff to a8b836e, detached nixos-rebuild test → health (0 failed, swarm 1/1) → switch
(21:17 UTC). Bridge env confirms POLL_REPOS += recipe-maintainers/rustdesk-server (1/1, no auth
errors). !testme posted on recipe PR #1 via testme-on-pr.sh → build 119 VERDICT=GREEN (level 5,
lint pass, clean_teardown, no_secret_leak; card embedded in PR by bridge). Card saved:
.cc-ci-logs/upgrades/rustdesk-server-testme-20261030.png. Next weekly /upgrade-all picks
rustdesk-server up automatically (used-recipes.md row committed cd1ec01). /root/cc-ci-test-verify
removed. Operator merges PR #1 then `abra recipe release`.
+1
View File
@@ -23,3 +23,4 @@ restarts the loops on boot. Count the lines below to see how often it's happenin
- 2026-05-30 17:03:05 BST — reboot detected; loops auto-started by systemd (resuming phase index 6). boot_id=f565f752-0463-42db-b787-9e0db35a5e3f - 2026-05-30 17:03:05 BST — reboot detected; loops auto-started by systemd (resuming phase index 6). boot_id=f565f752-0463-42db-b787-9e0db35a5e3f
- 2026-05-31 03:38:29 UTC — reboot detected; loops auto-started by systemd (resuming phase index 5). boot_id=51c17fc3-8391-4109-bce2-413fbee6f26d - 2026-05-31 03:38:29 UTC — reboot detected; loops auto-started by systemd (resuming phase index 5). boot_id=51c17fc3-8391-4109-bce2-413fbee6f26d
- 2026-08-23 03:11:24 UTC — reboot detected; loops auto-started by systemd (resuming phase index 14). boot_id=8215fc5e-8032-4844-b2b8-e0f8c0d9b13e - 2026-08-23 03:11:24 UTC — reboot detected; loops auto-started by systemd (resuming phase index 14). boot_id=8215fc5e-8032-4844-b2b8-e0f8c0d9b13e
- 2026-09-28 19:54:59 UTC — reboot detected; loops auto-started by systemd (resuming phase index 14). boot_id=66dadc4e-7f44-493a-aee9-fa76952d78c3
+171 -27
View File
@@ -40,6 +40,7 @@ from __future__ import annotations
import argparse import argparse
import gzip import gzip
import ipaddress
import json import json
import os import os
import re import re
@@ -166,6 +167,43 @@ def _vkey(v: str | None) -> tuple:
return tuple(out) return tuple(out)
def _nvd_affects(version: str | None, ranges: list[dict]) -> bool | None:
"""Whether NVD marks a concrete image version affected; None means tag lacks precision."""
current = _vkey(version)
if not current:
return None
uncertain = False
for row in ranges:
exact = _vkey(row.get("version")) if row.get("version") not in (None, "*") else ()
bounds = [(name, _vkey(row.get(name))) for name in (
"versionStartIncluding", "versionStartExcluding",
"versionEndIncluding", "versionEndExcluding") if row.get(name)]
versions = ([exact] if exact else []) + [value for _, value in bounds]
if any(not value for value in versions):
uncertain = True
continue
if any(len(value) > len(current) and value[:len(current)] == current for value in versions):
uncertain = True # e.g. postgres:13-alpine does not say which 13.x patch is running
continue
def compare(other):
length = max(len(current), len(other))
return (current + (0,) * (length - len(current))) > (other + (0,) * (length - len(other)))
if exact and current != exact:
continue
def outside(name, value):
if name == "versionStartExcluding":
return not compare(value)
if name == "versionStartIncluding":
return not compare(value) and current != value
if name == "versionEndIncluding":
return compare(value)
return compare(value) or current == value # versionEndExcluding
if any(outside(name, value) for name, value in bounds):
continue
return True
return None if uncertain else False
def _already_fixed_on_from_line(kf: tuple, cands: list[tuple]) -> bool: def _already_fixed_on_from_line(kf: tuple, cands: list[tuple]) -> bool:
"""Was it ALREADY fixed on the line we are upgrading FROM? """Was it ALREADY fixed on the line we are upgrading FROM?
@@ -231,6 +269,16 @@ def registry_urls(recipe: str, registry_dir: str) -> tuple[list[str], str | None
# recipe render '?' for no real reason — that is what put immich and n8n in the unknown # recipe render '?' for no real reason — that is what put immich and n8n in the unknown
# column on 2026-08-07 (https://docs.n8n.io/release-notes/` ← note the backtick). # column on 2026-08-07 (https://docs.n8n.io/release-notes/` ← note the backtick).
u = u.rstrip("`'\"*.,;:>)") u = u.rstrip("`'\"*.,;:>)")
if "…" in u or "..." in u:
continue # prose placeholder, not a fetchable security source
host = urllib.parse.urlsplit(u).hostname
if not host or host == "localhost" or "." not in host:
continue # a container/loopback URL in operational notes, not a public advisory source
try:
if not ipaddress.ip_address(host).is_global:
continue
except ValueError:
pass
if u and u not in urls: if u and u not in urls:
urls.append(u) urls.append(u)
return urls, path return urls, path
@@ -369,15 +417,57 @@ def vendor_pages(urls: list[str]) -> list[dict]:
NVD_API = "https://services.nvd.nist.gov/rest/json/cves/2.0" NVD_API = "https://services.nvd.nist.gov/rest/json/cves/2.0"
NVD_CPE_RE = re.compile(r"^\s*[-*]?\s*nvd-cpe:\s*(\S+)\s*=\s*(cpe:2\.3:[^\s`]+)", re.M | re.I) NVD_CPE_RE = re.compile(r"^\s*[-*]?\s*nvd-cpe:\s*(\S+)\s*=\s*(cpe:2\.3:[^\s`]+)", re.M | re.I)
_NVD_CACHE: dict[str, dict] = {}
COMMON_IMAGE_CPES = {
"postgres": "cpe:2.3:a:postgresql:postgresql:*:*:*:*:*:*:*:*",
"mariadb": "cpe:2.3:a:mariadb:mariadb:*:*:*:*:*:*:*:*",
"mysql": "cpe:2.3:a:oracle:mysql:*:*:*:*:*:*:*:*",
}
def registry_cpes(recipe: str, registry_dir: str) -> list[tuple[str, str]]: def registry_cpes(recipe: str, registry_dir: str) -> list[tuple[str, str]]:
"""[(image-key, cpe)] declared in the recipe's registry as `nvd-cpe: <key> = <cpe>`.""" """Explicit CPEs plus common database CPEs for images listed in the registry table."""
path = os.path.join(registry_dir, f"{recipe}.md") path = os.path.join(registry_dir, f"{recipe}.md")
try: try:
return [(m.group(1), m.group(2)) for m in NVD_CPE_RE.finditer(open(path).read())] text = open(path).read()
except OSError: except OSError:
return [] return []
out = [(m.group(1), m.group(2)) for m in NVD_CPE_RE.finditer(text)]
explicit = {key for key, _ in out}
for line in text.splitlines():
if not line.startswith("|"):
continue
cells = [cell.strip().strip("`") for cell in line.strip().strip("|").split("|")]
if len(cells) < 2:
continue
image = cells[1].split("/")[-1].lower()
if image in COMMON_IMAGE_CPES and image not in explicit:
out.append((image, COMMON_IMAGE_CPES[image]))
explicit.add(image)
return out
def registry_image_sources(recipe: str, registry_dir: str) -> dict[str, set[str]]:
"""Map image repo basenames to GitHub advisory sources from the registry table."""
path = os.path.join(registry_dir, f"{recipe}.md")
try:
lines = open(path).read().splitlines()
except OSError:
return {}
out: dict[str, set[str]] = {}
for line in lines:
if not line.startswith("|"):
continue
cells = [cell.strip().strip("`") for cell in line.strip().strip("|").split("|")]
if len(cells) < 3:
continue
image = cells[1].split("/")[-1].lower()
match = re.search(r"https?://github\.com/([^/\s|]+)/([^/\s|)]+)", cells[2])
if not match or image in ("image", "---", ""):
continue
source = f"github-advisories:{match.group(1)}/{match.group(2).removesuffix('.git')}"
out.setdefault(image, set()).add(source)
return out
def nvd_advisories(cpe: str, key: str) -> dict: def nvd_advisories(cpe: str, key: str) -> dict:
@@ -395,6 +485,8 @@ def nvd_advisories(cpe: str, key: str) -> dict:
NVD LAGS the vendor (it had neither gitea CVSS-9.8 RCE at publication), so this is a fallback, NVD LAGS the vendor (it had neither gitea CVSS-9.8 RCE at publication), so this is a fallback,
never a replacement for 2a/2b. Unauthenticated calls are rate-limited to ~5/30s, hence the retry. never a replacement for 2a/2b. Unauthenticated calls are rate-limited to ~5/30s, hence the retry.
""" """
if cpe in _NVD_CACHE:
return {**_NVD_CACHE[cpe], "source": f"nvd:{key}"}
entry = {"source": f"nvd:{key}", "status": "ok", "advisories": []} entry = {"source": f"nvd:{key}", "status": "ok", "advisories": []}
url = f"{NVD_API}?resultsPerPage=2000&virtualMatchString={urllib.parse.quote(cpe)}" url = f"{NVD_API}?resultsPerPage=2000&virtualMatchString={urllib.parse.quote(cpe)}"
data = None data = None
@@ -407,15 +499,31 @@ def nvd_advisories(cpe: str, key: str) -> dict:
entry["status"] = f"error: {type(e).__name__}" entry["status"] = f"error: {type(e).__name__}"
return entry return entry
time.sleep(8) time.sleep(8)
for v in (data or {}).get("vulnerabilities", []): vulnerabilities = (data or {}).get("vulnerabilities", [])
if (data or {}).get("totalResults", len(vulnerabilities)) > len(vulnerabilities):
entry["status"] = "error: NVD result set truncated"
return entry
for v in vulnerabilities:
c = v.get("cve") or {} c = v.get("cve") or {}
cid = c.get("id") cid = c.get("id")
if not cid: if not cid or c.get("vulnStatus") == "Rejected":
continue continue
fixed, affected_max = set(), set() fixed, affected_max, affected_ranges = set(), set(), []
cpe_product = ":".join(cpe.split(":")[:5]) + ":"
cpe_edition = cpe.split(":")[9] if len(cpe.split(":")) > 9 else "*"
for cfg in c.get("configurations", []): for cfg in c.get("configurations", []):
for node in cfg.get("nodes", []): for node in cfg.get("nodes", []):
for m in node.get("cpeMatch", []): for m in node.get("cpeMatch", []):
criteria = m.get("criteria") or ""
if not m.get("vulnerable") or not criteria.startswith(cpe_product):
continue
edition = criteria.split(":")[9] if len(criteria.split(":")) > 9 else "*"
if cpe_edition != "*" and edition not in (cpe_edition, "*"):
continue
affected_ranges.append({k: m.get(k) for k in (
"versionStartIncluding", "versionStartExcluding",
"versionEndIncluding", "versionEndExcluding") if m.get(k)})
affected_ranges[-1]["version"] = criteria.split(":")[5]
if m.get("versionEndExcluding"): if m.get("versionEndExcluding"):
fixed.add(m["versionEndExcluding"]) fixed.add(m["versionEndExcluding"])
elif m.get("versionEndIncluding"): elif m.get("versionEndIncluding"):
@@ -433,9 +541,11 @@ def nvd_advisories(cpe: str, key: str) -> dict:
"vulnerable_range": None, "vulnerable_range": None,
"patched": "; ".join(sorted(fixed)) or None, "patched": "; ".join(sorted(fixed)) or None,
"affected_max": "; ".join(sorted(affected_max)) or None, "affected_max": "; ".join(sorted(affected_max)) or None,
"affected_ranges": affected_ranges or None,
"url": f"https://nvd.nist.gov/vuln/detail/{cid}", "url": f"https://nvd.nist.gov/vuln/detail/{cid}",
"published_at": c.get("published"), "description": None, "cvss": None, "published_at": c.get("published"), "description": None, "cvss": None,
}) })
_NVD_CACHE[cpe] = entry
return entry return entry
@@ -737,7 +847,8 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
"vulnerable_range": None, "patched": None, "vulnerable_range": None, "patched": None,
"context": None, "published_at": None, "context": None, "published_at": None,
"description": None, "url": None, "cvss": None, "description": None, "url": None, "cvss": None,
"changelog_fixed_in": None, "affected_max": None}) "changelog_fixed_in": None, "affected_max": None,
"affected_ranges": None})
if src not in e["sources"]: if src not in e["sources"]:
e["sources"].append(src) e["sources"].append(src)
for k, v in extra.items(): for k, v in extra.items():
@@ -748,8 +859,9 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
report["sources"].append({"source": entry["source"], "status": entry["status"], report["sources"].append({"source": entry["source"], "status": entry["status"],
"found": len(entry.get("advisories", []))}) "found": len(entry.get("advisories", []))})
for a in entry.get("advisories", []): for a in entry.get("advisories", []):
if a.get("cve"): advisory_id = a.get("cve") or a.get("ghsa")
record(a["cve"], entry["source"], severity=a.get("severity"), ghsa=a.get("ghsa"), if advisory_id:
record(advisory_id, entry["source"], severity=a.get("severity"), ghsa=a.get("ghsa"),
vulnerable_range=a.get("vulnerable_range"), patched=a.get("patched"), vulnerable_range=a.get("vulnerable_range"), patched=a.get("patched"),
context=a.get("summary"), published_at=a.get("published_at"), context=a.get("summary"), published_at=a.get("published_at"),
description=a.get("description"), url=a.get("url"), cvss=a.get("cvss")) description=a.get("description"), url=a.get("url"), cvss=a.get("cvss"))
@@ -761,6 +873,7 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
for a in entry.get("advisories", []): for a in entry.get("advisories", []):
record(a["cve"], entry["source"], severity=a.get("severity"), record(a["cve"], entry["source"], severity=a.get("severity"),
patched=a.get("patched"), affected_max=a.get("affected_max"), patched=a.get("patched"), affected_max=a.get("affected_max"),
affected_ranges=a.get("affected_ranges"),
context=a.get("summary"), published_at=a.get("published_at"), context=a.get("summary"), published_at=a.get("published_at"),
url=a.get("url")) url=a.get("url"))
@@ -825,18 +938,25 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
if pname in k or k in pname: if pname in k or k in pname:
windows[src] = (v_from, v_to) windows[src] = (v_from, v_to)
window_key[src] = k window_key[src] = k
image_sources = registry_image_sources(recipe, registry_dir)
for key, wf, wt in (images or []): for key, wf, wt in (images or []):
matched = False
explicit = image_sources.get(key.lower(), set())
for src in gh_sources: for src in gh_sources:
if src in windows:
continue
# Match BOTH ways: an image name is often longer than its source repo # Match BOTH ways: an image name is often longer than its source repo
# (`clickhouse/clickhouse-server` vs source `ClickHouse/ClickHouse`) and sometimes # (`clickhouse/clickhouse-server` vs source `ClickHouse/ClickHouse`) and sometimes
# shorter (`redis` vs `redis/redis`). One-directional matching silently dropped the # shorter (`redis` vs `redis/redis`). One-directional matching silently dropped the
# clickhouse window when the key was derived from a compose file. # clickhouse window when the key was derived from a compose file.
k, name = key.lower(), src.split("/")[-1].lower() k, name = key.lower(), src.split("/")[-1].lower()
if k in src.lower() or name in k: if src in explicit or k in src.lower() or name in k:
if src in windows:
matched |= windows[src] == (wf, wt)
continue # two images in one monorepo share a source and version window
windows[src] = (wf, wt) windows[src] = (wf, wt)
window_key[src] = key window_key[src] = key
matched = True
if not matched:
report.setdefault("unmatched_image_windows", []).append(key)
report["windows"] = {k: {"from": f, "to": t} for k, (f, t) in windows.items()} report["windows"] = {k: {"from": f, "to": t} for k, (f, t) in windows.items()}
def _classify_window(src, wf, wt): def _classify_window(src, wf, wt):
@@ -844,6 +964,21 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
`indeterminate` = advisories from this source that the method COULD NOT JUDGE (no usable `indeterminate` = advisories from this source that the method COULD NOT JUDGE (no usable
patched version, or no publish date). They are neither counted nor dismissed.""" patched version, or no publish date). They are neither counted nor dismissed."""
if src.startswith("nvd:"):
got, undecidable = set(), set()
for advisory, evidence in report["cves"].items():
if src not in evidence["sources"]:
continue
if not evidence.get("affected_ranges"):
undecidable.add(advisory)
continue
old = _nvd_affects(wf, evidence["affected_ranges"])
new = _nvd_affects(wt, evidence["affected_ranges"])
if old is True and new is False:
got.add(advisory)
elif old is None or (old is True and new is None):
undecidable.add(advisory)
return got, "NVD affected-version ranges", None, False, undecidable
kf, kt = _vkey(wf), _vkey(wt) kf, kt = _vkey(wf), _vkey(wt)
# A version-SCHEME change (semver 3.5.3 → calver 2026.7.1) makes numeric ordering # A version-SCHEME change (semver 3.5.3 → calver 2026.7.1) makes numeric ordering
# meaningless: 2025.12.2 compares "newer" than 3.5.3 while shipping earlier. # meaningless: 2025.12.2 compares "newer" than 3.5.3 while shipping earlier.
@@ -1012,6 +1147,7 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
report["cves"][cve]["classification"] = "indeterminate: no fix version published" report["cves"][cve]["classification"] = "indeterminate: no fix version published"
unknown = [c for c in unknown if c not in fixed_set] unknown = [c for c in unknown if c not in fixed_set]
report["fixed_by_this_upgrade"] = sorted(fixed_set) report["fixed_by_this_upgrade"] = sorted(fixed_set)
report["fixed_non_cve_advisories"] = sorted(x for x in fixed_set if not x.startswith("CVE-"))
report["indeterminate"] = sorted(indeterminate) report["indeterminate"] = sorted(indeterminate)
report["cve_count_indeterminate"] = len(indeterminate) report["cve_count_indeterminate"] = len(indeterminate)
report["unclassified"] = sorted(unknown) report["unclassified"] = sorted(unknown)
@@ -1020,24 +1156,22 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
# A scan with NO usable source has not measured anything, so it must not report a number — # A scan with NO usable source has not measured anything, so it must not report a number —
# least of all 0, which asserts safety. mumble had no cc-ci-plan/upstream/mumble.md at all and # least of all 0, which asserts safety. mumble had no cc-ci-plan/upstream/mumble.md at all and
# still produced "0 identified", which was then published as a clean 0 in a CVE report. # still produced "0 identified", which was then published as a clean 0 in a CVE report.
usable_sources = [ usable_sources = [s for s in report["sources"] if s["status"] == "ok"]
s for s in report["sources"]
if s["status"] == "ok" or s["status"].startswith("no-advisories-published")
]
no_sources = not usable_sources no_sources = not usable_sources
if no_sources: if no_sources:
report["no_usable_sources"] = True report["no_usable_sources"] = True
report["count_known"] = not unresolved_any and not no_sources
report["cve_count_fixed"] = (len(fixed_set)
if (not unresolved_any and not no_sources) else None)
report["cve_count_total_seen"] = len(report["cves"])
# Only GENUINE failures make a count unreliable. "no-advisories-published" (404: the repo has
# no advisory feed) and "skipped: template URL" are benign and must not degrade the verdict.
report["sources_failed"] = [ report["sources_failed"] = [
s["source"] s["source"]
for s in report["sources"] for s in report["sources"]
if not (s["status"] == "ok" or s["status"].startswith(("no-advisories-published", "skipped:"))) if not (s["status"] == "ok" or s["status"].startswith(("no-advisories-published", "skipped:")))
] ]
coverage_failed = bool(report.get("unmatched_image_windows")) or bool(v_from and not primary)
report["count_known"] = not unresolved_any and not no_sources and not coverage_failed and not report["sources_failed"]
report["cve_count_fixed"] = (sum(x.startswith("CVE-") for x in fixed_set)
if report["count_known"] else None)
report["cve_count_total_seen"] = sum(x.startswith("CVE-") for x in report["cves"])
# Only GENUINE failures make a count unreliable. "no-advisories-published" (404: the repo has
# no advisory feed) and "skipped: template URL" are benign and must not degrade the verdict.
report["sources_benign"] = [ report["sources_benign"] = [
s["source"] s["source"]
for s in report["sources"] for s in report["sources"]
@@ -1052,16 +1186,20 @@ def markdown(rep: dict) -> str:
f"{rep.get('from') or '?'} → {rep.get('to') or '?'}"] f"{rep.get('from') or '?'} → {rep.get('to') or '?'}"]
if not rep.get("count_known", True): if not rep.get("count_known", True):
L.append("\n**CVEs fixed by this upgrade: UNKNOWN — the scan could NOT determine a count.**") L.append("\n**CVEs fixed by this upgrade: UNKNOWN — the scan could NOT determine a count.**")
if rep.get("fixed_by_this_upgrade"):
L.append(f"\nAt least {len(rep['fixed_by_this_upgrade'])} fixed advisory/advisories were identified: "
+ ", ".join(rep["fixed_by_this_upgrade"][:15]))
if rep.get("no_usable_sources"): if rep.get("no_usable_sources"):
L.append("\n⚠ This is NOT zero. **No usable source was checked at all** — the registry " L.append("\n⚠ This is NOT zero. **No usable source was checked at all** — the registry "
"file `cc-ci-plan/upstream/<recipe>.md` is missing or every source failed, so " "file `cc-ci-plan/upstream/<recipe>.md` is missing or every source failed, so "
"nothing was measured. Render this recipe's cve cell as `?`, never `0`, and add " "nothing was measured. Render this recipe's cve cell as `?`, never `0`, and add "
"the registry file.") "the registry file.")
else: else:
L.append("\n⚠ This is NOT zero. A version-scheme change (e.g. semver → calver) makes " L.append("\n⚠ This is NOT zero. At least one version window or advisory source could "
"numeric ordering meaningless across this jump, so no advisory could be " "not be checked completely. Render this recipe's cve cell as `?`, never `0`.")
"classified. Render this recipe's cve cell as `?`, never `0`. Read the vendor's " if rep.get("unmatched_image_windows"):
"release notes for the jump and count by hand.") L.append("\n⚠ image windows with no matching advisory source: "
+ ", ".join(rep["unmatched_image_windows"]))
if rep["unclassified"]: if rep["unclassified"]:
L.append(f"\nAdvisories seen but unclassifiable ({len(rep['unclassified'])}) — includes " L.append(f"\nAdvisories seen but unclassifiable ({len(rep['unclassified'])}) — includes "
f"other images in this recipe: " + ", ".join(rep["unclassified"][:12])) f"other images in this recipe: " + ", ".join(rep["unclassified"][:12]))
@@ -1072,7 +1210,8 @@ def markdown(rep: dict) -> str:
f"reading in the upgrade step._") f"reading in the upgrade step._")
return "\n".join(L) return "\n".join(L)
ind = rep.get("cve_count_indeterminate") or 0 ind = rep.get("cve_count_indeterminate") or 0
if rep["fixed_by_this_upgrade"]: fixed_cves = [x for x in rep["fixed_by_this_upgrade"] if x.startswith("CVE-")]
if fixed_cves:
floor = " (at least — see undetermined below)" if ind else "" floor = " (at least — see undetermined below)" if ind else ""
L.append(f"\n**CVEs fixed by this upgrade: {rep['cve_count_fixed']}**{floor}\n") L.append(f"\n**CVEs fixed by this upgrade: {rep['cve_count_fixed']}**{floor}\n")
cb = rep.get("classified_by") or {} cb = rep.get("classified_by") or {}
@@ -1086,12 +1225,17 @@ def markdown(rep: dict) -> str:
L.append("") L.append("")
L.append("| CVE | severity | fixed in | advisory | source |") L.append("| CVE | severity | fixed in | advisory | source |")
L.append("|---|---|---|---|---|") L.append("|---|---|---|---|---|")
for cve in rep["fixed_by_this_upgrade"]: for cve in fixed_cves:
e = rep["cves"][cve] e = rep["cves"][cve]
L.append(f"| {cve} | {e.get('severity') or '?'} | {e.get('patched') or '?'} | " L.append(f"| {cve} | {e.get('severity') or '?'} | {e.get('patched') or '?'} | "
f"{e.get('ghsa') or '-'} | {e['sources'][0]} |") f"{e.get('ghsa') or '-'} | {e['sources'][0]} |")
else: else:
L.append("\n**CVEs fixed by this upgrade: 0 identified by the deterministic scan.**") L.append("\n**CVEs fixed by this upgrade: 0 identified by the deterministic scan.**")
if rep.get("fixed_non_cve_advisories"):
L.append("\n**Security advisories fixed without CVE IDs: "
+ str(len(rep["fixed_non_cve_advisories"])) + "** — "
+ ", ".join(rep["fixed_non_cve_advisories"][:15])
+ ". In the report CVEs column, use YES when the CVE count is zero.")
if ind: if ind:
L.append(f"\n⚠ **{ind} advisory/advisories could NOT be judged** — the vendor published no fix " L.append(f"\n⚠ **{ind} advisory/advisories could NOT be judged** — the vendor published no fix "
f"version (GitHub carries `TBD` or a placeholder like `7.4.X`) and the vulnerable " f"version (GitHub carries `TBD` or a placeholder like `7.4.X`) and the vulnerable "
+15 -18
View File
@@ -134,7 +134,7 @@ def security_source_audit(recipe: str) -> list[dict]:
e = A.nvd_advisories(cpe, key) e = A.nvd_advisories(cpe, key)
n = len(e.get("advisories") or []) n = len(e.get("advisories") or [])
out.append({"source": e["source"] + f" ({cpe.split(':')[4]}/{cpe.split(':')[3]})", out.append({"source": e["source"] + f" ({cpe.split(':')[4]}/{cpe.split(':')[3]})",
"kind": "advisory-feed" if n else "no-cve-data", "kind": "advisory-feed", "verified_empty": n == 0,
"status": e["status"], "cves": n, "usable": n}) "status": e["status"], "cves": n, "usable": n})
for entry in A.github_advisories(urls): for entry in A.github_advisories(urls):
out.append({"source": entry["source"], "kind": "advisory-feed", out.append({"source": entry["source"], "kind": "advisory-feed",
@@ -227,16 +227,7 @@ def audit_recipe(recipe: str, ssh: str | None, quiet_days: int) -> dict:
def _registry_urls(recipe: str): def _registry_urls(recipe: str):
path = os.path.join(REGISTRY_DIR, f"{recipe}.md") return A.registry_urls(recipe, REGISTRY_DIR)
if not os.path.exists(path):
return [], None
text = open(path).read()
urls = []
for u in re.findall(r"https?://[^\s)|\]]+", text):
u = u.rstrip("`'\"*.,;:>)")
if u and u not in urls:
urls.append(u)
return urls, path
def all_recipes() -> list[str]: def all_recipes() -> list[str]:
@@ -272,23 +263,29 @@ def main() -> int:
blind_recipes, noisy = [], 0 blind_recipes, noisy = [], 0
for r in recipes: for r in recipes:
rows = security_source_audit(r) rows = security_source_audit(r)
feeds = [x for x in rows if x["kind"] == "advisory-feed" and x["cves"] > 0] feeds = [x for x in rows if x["kind"] == "advisory-feed"
logs = [x for x in rows if x["kind"] == "changelog"] and (x["cves"] > 0 or x.get("verified_empty"))
and x["status"] == "ok"]
logs = [x for x in rows if x["kind"] == "changelog" and x["status"] == "ok"]
unattr = [x for x in rows if x["kind"] == "unattributable"] unattr = [x for x in rows if x["kind"] == "unattributable"]
failed = [x for x in rows if x["status"].startswith("error:")]
noisy += len(unattr) noisy += len(unattr)
usable = len(feeds) + len(logs) usable = len(feeds) + len(logs)
if usable == 0: if usable == 0:
blind_recipes.append(r) blind_recipes.append(r)
print(f"!! {r}: NO USABLE CVE SOURCE — {len(unattr)} unparseable page(s), " print(f"!! {r}: NO VERIFIED CVE SOURCE — {len(unattr)} unparseable page(s), "
f"0 advisory feeds, 0 attributable changelogs") f"0 populated advisory feeds, 0 attributable changelogs, {len(failed)} failed source(s)")
for x in rows: for x in rows:
print(f" {x['kind']:15} {x['source'][:64]} ({x['cves']} CVEs)") unit = "advisories" if x["kind"] == "advisory-feed" else "CVE IDs"
print(f" {x['kind']:15} {x['source'][:64]} ({x['cves']} {unit}; {x['status']})")
else: else:
print(f"OK {r}: {len(feeds)} advisory-feed(s), {len(logs)} changelog(s)" print(f"OK {r}: {len(feeds)} advisory-feed(s), {len(logs)} changelog(s)"
+ (f", {len(unattr)} unparseable page(s) (redundant — covered by a feed)" + (f", {len(unattr)} unparseable page(s)" if unattr else "")
if unattr else "")) + (f", {len(failed)} failed source(s) — counts incomplete" if failed else ""))
for x in logs: for x in logs:
print(f" changelog {x['source'][:62]} ({x['usable']}/{x['cves']})") print(f" changelog {x['source'][:62]} ({x['usable']}/{x['cves']})")
for x in failed:
print(f" FAILED {x['source'][:90]} ({x['status']})")
print(f"\n{len(recipes)} recipes · {len(blind_recipes)} with NO usable CVE source" print(f"\n{len(recipes)} recipes · {len(blind_recipes)} with NO usable CVE source"
+ (f": {', '.join(blind_recipes)}" if blind_recipes else "") + (f": {', '.join(blind_recipes)}" if blind_recipes else "")
+ f" · {noisy} unparseable page(s) elsewhere (harmless where a feed covers them)") + f" · {noisy} unparseable page(s) elsewhere (harmless where a feed covers them)")
+3 -1
View File
@@ -30,7 +30,7 @@ SPEC SHAPE (the agent writes this JSON):
{"date":"YYYY-MM-DD","subtitle":"Week of <human date>", {"date":"YYYY-MM-DD","subtitle":"Week of <human date>",
"lead":"<ONE short paragraph>", "lead":"<ONE short paragraph>",
"table":[{"recipe":"x","change":"a → b","status":"GREEN|FAILED|STALE|SKIPPED|UPTODATE", # -> TESTS col "table":[{"recipe":"x","change":"a → b","status":"GREEN|FAILED|STALE|SKIPPED|UPTODATE", # -> TESTS col
"cve":16, # number of CVEs this PR fixes; 0/omit for none "cve":16, # CVE count; "YES" for advisory without CVE ID; "?" unknown
"ci":"build 154 ✓","ci_url":"…","pr":"#4","pr_url":"…","notes":"…"}], "ci":"build 154 ✓","ci_url":"…","pr":"#4","pr_url":"…","notes":"…"}],
# ROWS SORTED by recommended priority to address — recipes WITH CVEs first, then failures, # ROWS SORTED by recommended priority to address — recipes WITH CVEs first, then failures,
# then stale-tests, then routine green, then up-to-date/skipped. # then stale-tests, then routine green, then up-to-date/skipped.
@@ -232,6 +232,8 @@ def _table(rows, repo_url=None):
# int is the confirmed CVE count; 0/omit is a confirmed-clean scan. # int is the confirmed CVE count; 0/omit is a confirmed-clean scan.
if isinstance(cve, str) and cve.strip() == "?": if isinstance(cve, str) and cve.strip() == "?":
cve_cell = '<span class="muted" title="advisory scan incomplete or absent — CVE count unknown">?</span>' cve_cell = '<span class="muted" title="advisory scan incomplete or absent — CVE count unknown">?</span>'
elif isinstance(cve, str) and cve.strip().upper() == "YES":
cve_cell = '<span class="cve">YES</span>'
elif isinstance(cve, (int, float)) and cve: elif isinstance(cve, (int, float)) and cve:
cve_cell = f'<span class="cve">{int(cve)}</span>' cve_cell = f'<span class="cve">{int(cve)}</span>'
else: else:
+76 -16
View File
@@ -163,7 +163,8 @@ def parse_ref(ref: str) -> dict:
return {"registry": host, "repo": repo, "tag": tag, "digest": digest} return {"registry": host, "repo": repo, "tag": tag, "digest": digest}
HUB_RECENT_PAGES = int(os.environ.get("RESOLVE_IMAGES_HUB_PAGES", "10")) HUB_RECENT_PAGES = int(os.environ.get("RESOLVE_IMAGES_HUB_PAGES", "40"))
_HUB_TRUNCATED: set[str] = set()
def _hub_tag_exists(repo: str, tag: str) -> bool: def _hub_tag_exists(repo: str, tag: str) -> bool:
@@ -177,10 +178,10 @@ def _hub_tag_exists(repo: str, tag: str) -> bool:
def _hub_tags(repo: str) -> list[str]: def _hub_tags(repo: str) -> list[str]:
"""Recently-pushed tags, newest first. """Recently-pushed tags, newest first.
Popular Docker Hub repos carry many thousands of tags, so a full enumeration is impractical — Popular Docker Hub repos carry many thousands of tags, so enumerate a bounded recent window.
but it is also unnecessary: a tag NEWER than the one we run must have been pushed AFTER it, so A rebuilt old tag can displace a newer version from that window, so callers MUST treat a
ordering by last_updated and reading a bounded recent window is sufficient to find any upgrade. truncated listing with no detected upgrade as unverified, not current. (ghcr offers no recency
(ghcr offers no ordering, which is why that path needs a different strategy.) ordering, which is why that path needs a different strategy.)
""" """
tags, url = [], (f"https://hub.docker.com/v2/repositories/{repo}/tags" tags, url = [], (f"https://hub.docker.com/v2/repositories/{repo}/tags"
f"?page_size=100&ordering=last_updated") f"?page_size=100&ordering=last_updated")
@@ -191,6 +192,8 @@ def _hub_tags(repo: str) -> list[str]:
url = d.get("next") url = d.get("next")
if not url: if not url:
break break
if url:
_HUB_TRUNCATED.add(repo)
return tags return tags
@@ -236,15 +239,26 @@ def _oci_tags(host: str, repo: str) -> list[str]:
hdrs = _oci_bearer(host, repo) hdrs = _oci_bearer(host, repo)
tags, url = [], f"https://{host}/v2/{repo}/tags/list?n=1000" tags, url = [], f"https://{host}/v2/{repo}/tags/list?n=1000"
for _ in range(MAX_TAG_PAGES): for _ in range(MAX_TAG_PAGES):
req = urllib.request.Request(url, headers={"User-Agent": UA, **hdrs}) for attempt in range(RETRIES):
with urllib.request.urlopen(req, timeout=TIMEOUT) as r: req = urllib.request.Request(url, headers={"User-Agent": UA, **hdrs})
tags += (json.load(r) or {}).get("tags") or [] try:
link = r.headers.get("Link", "") or "" with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
tags += (json.load(r) or {}).get("tags") or []
link = r.headers.get("Link", "") or ""
break
except urllib.error.HTTPError as e:
if e.code not in (429, 503) or attempt == RETRIES - 1:
raise
retry_after = e.headers.get("Retry-After") or ""
delay = min(int(retry_after) if retry_after.isdigit() else 2 ** (attempt + 1), 60)
time.sleep(delay)
m = re.search(r'<([^>]+)>;\s*rel="next"', link) m = re.search(r'<([^>]+)>;\s*rel="next"', link)
if not m: if not m:
break break
nxt = m.group(1) nxt = m.group(1)
url = f"https://{host}{nxt}" if nxt.startswith("/") else nxt url = f"https://{host}{nxt}" if nxt.startswith("/") else nxt
else:
raise RuntimeError(f"tag listing truncated after {MAX_TAG_PAGES} pages")
return tags return tags
@@ -299,6 +313,39 @@ def _release_fallback_repos(registry: str, repo: str) -> list[tuple[str, str]]:
return cands return cands
def _oci_tag_exists(host: str, repo: str, tag: str) -> bool:
"""Verify a release candidate is actually published as an image manifest."""
hdrs = {"User-Agent": UA,
"Accept": "application/vnd.oci.image.index.v1+json, "
"application/vnd.docker.distribution.manifest.list.v2+json, "
"application/vnd.oci.image.manifest.v1+json",
**_oci_bearer(host, repo)}
url = f"https://{host}/v2/{repo}/manifests/{urllib.parse.quote(tag, safe='')}"
try:
with urllib.request.urlopen(urllib.request.Request(url, headers=hdrs, method="HEAD"),
timeout=TIMEOUT) as response:
return response.status == 200
except urllib.error.HTTPError as e:
if e.code == 404:
return False
raise
def _release_manifest_fallback(registry: str, repo: str, current: str) -> tuple[list[str], str] | None:
"""For huge GHCR registries, use release tags only after checking actual manifests."""
for owner, name in _release_fallback_repos(registry, repo):
releases = github_release_tags(owner, name)
if current not in releases:
continue # incomplete release window; it cannot establish a current baseline
candidates = sorted((tag for tag in set(releases)
if shape(tag) == shape(current) and vkey(tag) >= vkey(current)),
key=vkey, reverse=True)
for tag in candidates:
if _oci_tag_exists(registry, repo, tag):
return [current, tag], f"github-releases+registry-manifest:{owner}/{name}"
return None
_TAG_CACHE: dict[tuple[str, str], tuple[list[str], str | None]] = {} _TAG_CACHE: dict[tuple[str, str], tuple[list[str], str | None]] = {}
@@ -335,8 +382,16 @@ def resolve(ref: str) -> dict:
"upgrade_available": False} "upgrade_available": False}
tags, err = list_tags(info["registry"], info["repo"]) tags, err = list_tags(info["registry"], info["repo"])
if err: if err:
out["status"] = f"error: {err}" try:
return out fallback = (_release_manifest_fallback(info["registry"], info["repo"], info["tag"])
if "tag listing truncated" in err else None)
except Exception as e: # noqa: BLE001 — a failed fallback must stay unresolved
out["status"] = f"error: registry {err}; release fallback {type(e).__name__}: {e}"
return out
if not fallback:
out["status"] = f"error: {err}"
return out
tags, out["source"] = fallback
out["tags_seen"] = len(set(tags)) out["tags_seen"] = len(set(tags))
# INTEGRITY CHECK: the tag we are currently running MUST appear in the listing. If it does not, # INTEGRITY CHECK: the tag we are currently running MUST appear in the listing. If it does not,
# the listing is incomplete and any "newest" derived from it is a guess — ghcr paginates to tens # the listing is incomplete and any "newest" derived from it is a guess — ghcr paginates to tens
@@ -367,8 +422,9 @@ def resolve(ref: str) -> dict:
same = [t for t in set(tags) if shape(t) == want and vkey(t)] same = [t for t in set(tags) if shape(t) == want and vkey(t)]
out["candidates"] = len(same) out["candidates"] = len(same)
if not same: if not same:
# Not a failure: digest-only pins and `latest`/`stable` have no comparable siblings. # A floating or unusual tag cannot prove that an image is current. Keep it visible to
out["status"] = "no comparable tags (shape has no numeric siblings)" # callers rather than letting an empty candidate set become a false UPTODATE verdict.
out["status"] = "unverifiable: no comparable numeric tags"
return out return out
newest = max(same, key=vkey) newest = max(same, key=vkey)
out["newest_same_shape"] = newest out["newest_same_shape"] = newest
@@ -377,6 +433,10 @@ def resolve(ref: str) -> dict:
if within: if within:
out["newest_within_major"] = max(within, key=vkey) out["newest_within_major"] = max(within, key=vkey)
out["upgrade_available"] = bool(cur and vkey(newest) > cur) out["upgrade_available"] = bool(cur and vkey(newest) > cur)
if (info["registry"] in ("docker.io", "registry-1.docker.io")
and info["repo"] in _HUB_TRUNCATED and not out["upgrade_available"]):
out["status"] = (f"unverifiable: Docker Hub listing truncated after "
f"{HUB_RECENT_PAGES * 100} recent tags")
return out return out
@@ -436,16 +496,16 @@ def main() -> int:
"recipe": a.recipe, "recipe": a.recipe,
"images": results, "images": results,
"upgrades_available": [r["ref"] for r in results if r["upgrade_available"]], "upgrades_available": [r["ref"] for r in results if r["upgrade_available"]],
"unresolved": [r["ref"] for r in results if r["status"].startswith("error")], "unresolved": [r["ref"] for r in results if r["status"].startswith(("error", "skipped", "unverifiable"))],
# The whole point: distinguish "checked, current" from "could not check". # The whole point: distinguish "checked, current" from "could not check".
"all_resolved": not any(r["status"].startswith("error") for r in results), "all_resolved": bool(results) and all(r["status"] == "ok" for r in results),
} }
if not a.table: if not a.table:
print(json.dumps(report, indent=2)) print(json.dumps(report, indent=2))
return 0 return 0
print(f"{a.recipe} — {len(results)} images") print(f"{a.recipe} — {len(results)} images")
for r in results: for r in results:
flag = "UPGRADE" if r["upgrade_available"] else ("ERROR" if r["status"].startswith("error") else "current") flag = "UPGRADE" if r["upgrade_available"] else ("UNKNOWN" if r["status"] != "ok" else "current")
print(f" [{flag:7}] {r['repo']}:{r['tag']}" + (" (digest-pinned)" if r["digest"] else "")) print(f" [{flag:7}] {r['repo']}:{r['tag']}" + (" (digest-pinned)" if r["digest"] else ""))
print(f" shape={r['shape']} candidates={r['candidates']}" print(f" shape={r['shape']} candidates={r['candidates']}"
f" newest_same_shape={r['newest_same_shape']}" f" newest_same_shape={r['newest_same_shape']}"
+80 -3
View File
@@ -22,6 +22,7 @@ import io
import json import json
import os import os
import pathlib import pathlib
import tempfile
import sys import sys
import unittest import unittest
import unittest.mock import unittest.mock
@@ -51,7 +52,7 @@ def vendor(url, cves=(), status="ok"):
def run_scan(gh_entries=(), vendor_entries=(), tag_dates=None, *, v_from=None, v_to=None, def run_scan(gh_entries=(), vendor_entries=(), tag_dates=None, *, v_from=None, v_to=None,
images=None, recipe="fixture", urls=None, releases=None): images=None, recipe="fixture", urls=None, releases=None, image_sources=None):
"""scan() with every network call replaced by fixtures. """scan() with every network call replaced by fixtures.
`releases` maps CVE id -> tags whose release notes name it (the third, release-note method).""" `releases` maps CVE id -> tags whose release notes name it (the third, release-note method)."""
@@ -61,6 +62,7 @@ def run_scan(gh_entries=(), vendor_entries=(), tag_dates=None, *, v_from=None, v
with unittest.mock.patch.object(A, "registry_urls", lambda r, d: (list(urls), "/fake/reg.md")), \ with unittest.mock.patch.object(A, "registry_urls", lambda r, d: (list(urls), "/fake/reg.md")), \
unittest.mock.patch.object(A, "github_advisories", lambda u: list(gh_entries)), \ unittest.mock.patch.object(A, "github_advisories", lambda u: list(gh_entries)), \
unittest.mock.patch.object(A, "vendor_pages", lambda u: list(vendor_entries)), \ unittest.mock.patch.object(A, "vendor_pages", lambda u: list(vendor_entries)), \
unittest.mock.patch.object(A, "registry_image_sources", lambda r, d: image_sources or {}), \
unittest.mock.patch.object(A, "osv", lambda r, v: None), \ unittest.mock.patch.object(A, "osv", lambda r, v: None), \
unittest.mock.patch.object(A, "_tag_date", lambda o, r, v: tag_dates.get(v)), \ unittest.mock.patch.object(A, "_tag_date", lambda o, r, v: tag_dates.get(v)), \
unittest.mock.patch.object(A, "release_fix_versions", lambda src, cve: list(releases.get(cve, []))): unittest.mock.patch.object(A, "release_fix_versions", lambda src, cve: list(releases.get(cve, []))):
@@ -153,6 +155,26 @@ class TestRegistryUrls(unittest.TestCase):
p.write_text(text) p.write_text(text)
return A.registry_urls("r", tmp) return A.registry_urls("r", tmp)
def test_prose_ellipsis_is_not_a_security_source(self):
with tempfile.TemporaryDirectory() as directory:
urls, _ = self._write(directory,
"https://github.com/cryptpad/cryptpad and example `https://…` URLs "
"http://docspec:3000/conversion http://127.0.0.1:3000/health")
self.assertEqual(urls, ["https://github.com/cryptpad/cryptpad"])
def test_image_source_table_maps_monorepo_images(self):
mapping = A.registry_image_sources("lasuite-drive", str(HERE / "upstream"))
self.assertIn("github-advisories:suitenumerique/drive", mapping["drive-backend"])
self.assertIn("github-advisories:CollaboraOnline/online", mapping["code"])
def test_common_database_cpes_follow_registry_images(self):
self.assertIn(("postgres", A.COMMON_IMAGE_CPES["postgres"]),
A.registry_cpes("immich", str(HERE / "upstream")))
self.assertIn(("mariadb", A.COMMON_IMAGE_CPES["mariadb"]),
A.registry_cpes("gitea", str(HERE / "upstream")))
self.assertIn(("rustdesk-server", "cpe:2.3:a:rustdesk:rustdesk_server:*:*:*:*:oss:*:*:*"),
A.registry_cpes("rustdesk-server", str(HERE / "upstream")))
def test_strips_trailing_markdown_punctuation(self): def test_strips_trailing_markdown_punctuation(self):
# Production defect: a captured backtick 404'd the fetch and rendered n8n/immich as '?'. # Production defect: a captured backtick 404'd the fetch and rendered n8n/immich as '?'.
import tempfile import tempfile
@@ -174,6 +196,25 @@ class TestRegistryUrls(unittest.TestCase):
self.assertEqual((urls, path), ([], None)) self.assertEqual((urls, path), ([], None))
class TestNvdCompleteness(unittest.TestCase):
def test_truncated_results_are_not_a_clean_zero(self):
payload = json.dumps({"totalResults": 2, "vulnerabilities": [{"cve": {"id": "CVE-2026-0001"}}]})
with unittest.mock.patch.object(A, "_fetch", return_value=payload):
row = A.nvd_advisories("cpe:2.3:a:example:server:*:*:*:*:*:*:*:*", "server")
self.assertEqual(row["status"], "error: NVD result set truncated")
self.assertEqual(row["advisories"], [])
def test_database_backport_is_not_credited_again_on_major_upgrade(self):
ranges = [
{"version": "*", "versionStartIncluding": "15.0", "versionEndExcluding": "15.14"},
{"version": "*", "versionStartIncluding": "16.0", "versionEndExcluding": "16.5"},
]
self.assertIs(A._nvd_affects("15.13", ranges), True)
self.assertIs(A._nvd_affects("15.19", ranges), False)
self.assertIs(A._nvd_affects("18.6", ranges), False)
self.assertIsNone(A._nvd_affects("15-alpine", ranges))
# ── C. --image argument parsing ─────────────────────────────────────────────────────────────────── # ── C. --image argument parsing ───────────────────────────────────────────────────────────────────
class TestImageArgParsing(unittest.TestCase): class TestImageArgParsing(unittest.TestCase):
@@ -290,12 +331,32 @@ class TestPerImageWindows(unittest.TestCase):
self.assertEqual(rep["windows"]["github-advisories:discourse/discourse"], self.assertEqual(rep["windows"]["github-advisories:discourse/discourse"],
{"from": "3.5.3", "to": "3.5.4"}) {"from": "3.5.3", "to": "3.5.4"})
def test_unmatched_image_name_is_silently_ignored(self): def test_unmatched_image_name_makes_count_unknown(self):
# Documents CURRENT behaviour: a typo'd name costs coverage without warning. # A typo'd sidecar name must not silently turn its CVEs into a clean zero.
rep = run_scan([self.APP], v_from="3.5.3", v_to="3.5.4", rep = run_scan([self.APP], v_from="3.5.3", v_to="3.5.4",
images=[("nosuchimage", "1", "2")], urls=self.URLS[:1]) images=[("nosuchimage", "1", "2")], urls=self.URLS[:1])
self.assertEqual(list(rep["windows"]), ["github-advisories:discourse/discourse"]) self.assertEqual(list(rep["windows"]), ["github-advisories:discourse/discourse"])
self.assertFalse(rep["count_known"])
self.assertEqual(rep["unmatched_image_windows"], ["nosuchimage"])
self.assertIsNone(rep["cve_count_fixed"])
def test_registry_maps_image_to_differently_named_source(self):
rep = run_scan([self.APP, gh("suitenumerique/drive", [adv("CVE-DRIVE-1", patched="0.25.3")])],
v_from="3.5.3", v_to="3.5.4", images=[("drive-backend", "0.22.0", "0.25.3")],
urls=self.URLS,
image_sources={"drive-backend": {"github-advisories:suitenumerique/drive"}})
self.assertEqual(rep.get("unmatched_image_windows"), None)
self.assertIn("CVE-DRIVE-1", rep["fixed_by_this_upgrade"])
def test_monorepo_images_share_identical_window(self):
source = "github-advisories:suitenumerique/drive"
rep = run_scan([gh("suitenumerique/drive", [adv("CVE-DRIVE-1", patched="0.25.3")])],
v_from="0.22.0", v_to="0.25.3",
images=[("drive-backend", "0.22.0", "0.25.3")],
image_sources={"drive-backend": {source}})
self.assertTrue(rep["count_known"]) self.assertTrue(rep["count_known"])
self.assertEqual(rep.get("unmatched_image_windows"), None)
self.assertEqual(rep["cve_count_fixed"], 1)
class TestSchemeChangeDateFallback(unittest.TestCase): class TestSchemeChangeDateFallback(unittest.TestCase):
@@ -358,6 +419,15 @@ class TestCountGuarantees(unittest.TestCase):
self.assertTrue(rep["count_known"]) self.assertTrue(rep["count_known"])
self.assertIn("0 identified", A.markdown(rep)) self.assertIn("0 identified", A.markdown(rep))
def test_ghsa_without_cve_is_preserved_as_security_advisory(self):
entry = {"cve": None, "ghsa": "GHSA-aaaa-bbbb-cccc", "severity": "low",
"patched": "1.1", "summary": "security fix", "published_at": None}
rep = run_scan([gh("app/app", [entry])], v_from="1.0", v_to="1.1")
self.assertTrue(rep["count_known"])
self.assertEqual(rep["cve_count_fixed"], 0)
self.assertEqual(rep["fixed_non_cve_advisories"], ["GHSA-aaaa-bbbb-cccc"])
self.assertIn("use YES", A.markdown(rep))
def test_404_advisory_feed_is_benign_not_a_failure(self): def test_404_advisory_feed_is_benign_not_a_failure(self):
rep = run_scan([gh("app/app", [adv("CVE-1", patched="1.1")]), rep = run_scan([gh("app/app", [adv("CVE-1", patched="1.1")]),
gh("side/car", [], status="no-advisories-published")], gh("side/car", [], status="no-advisories-published")],
@@ -377,6 +447,13 @@ class TestCountGuarantees(unittest.TestCase):
v_from="1.0", v_to="1.1") v_from="1.0", v_to="1.1")
self.assertIn("github-advisories:app/app", rep["sources_failed"]) self.assertIn("github-advisories:app/app", rep["sources_failed"])
self.assertIn("FAILED", A.markdown(rep)) self.assertIn("FAILED", A.markdown(rep))
self.assertFalse(rep["count_known"])
def test_only_404_feed_is_not_evidence_of_clean_scan(self):
rep = run_scan([gh("app/app", [], status="no-advisories-published")],
v_from="1.0", v_to="1.1")
self.assertTrue(rep["no_usable_sources"])
self.assertFalse(rep["count_known"])
def test_no_window_given_classifies_nothing(self): def test_no_window_given_classifies_nothing(self):
rep = run_scan([gh("app/app", [adv("CVE-1", patched="1.1")])]) rep = run_scan([gh("app/app", [adv("CVE-1", patched="1.1")])])
+55
View File
@@ -0,0 +1,55 @@
#!/usr/bin/env python3
"""Offline regressions for the weekly image survey's false-current cases."""
import importlib.util
import pathlib
import unittest
from unittest import mock
HERE = pathlib.Path(__file__).resolve().parent
spec = importlib.util.spec_from_file_location("resolve_images", HERE / "resolve-images.py")
resolver = importlib.util.module_from_spec(spec)
spec.loader.exec_module(resolver)
class TestResolverCoverage(unittest.TestCase):
def test_cryptpad_calver_tag_is_found_independently_of_abra(self):
with mock.patch.object(resolver, "list_tags", return_value=(
["version-2026.5.1", "version-2026.10.0", "opendesk-2026.08.04"], None)):
row = resolver.resolve("cryptpad/cryptpad:version-2026.5.1")
self.assertTrue(row["upgrade_available"])
self.assertEqual(row["newest_same_shape"], "version-2026.10.0")
def test_floating_and_templated_tags_cannot_prove_current(self):
with mock.patch.object(resolver, "list_tags", return_value=(["alpine", "1.2.3"], None)):
floating = resolver.resolve("atmoz/sftp:alpine")
templated = resolver.resolve("ghost:${IMAGE_VERSION}-alpine")
self.assertTrue(floating["status"].startswith("unverifiable:"))
self.assertTrue(templated["status"].startswith("skipped:"))
def test_truncated_registry_listing_cannot_prove_current(self):
with mock.patch.object(resolver, "list_tags", return_value=(["1.2.3"], None)):
with mock.patch.object(resolver, "_HUB_TRUNCATED", {"library/example"}):
row = resolver.resolve("example:1.2.3")
self.assertTrue(row["status"].startswith("unverifiable:"))
def test_huge_ghcr_listing_uses_only_published_release_manifests(self):
ref = "ghcr.io/immich-app/immich-machine-learning:v3.3.5"
with mock.patch.object(resolver, "list_tags", return_value=([], "RuntimeError: tag listing truncated")), \
mock.patch.object(resolver, "github_release_tags",
return_value=["v3.3.6", "v3.3.5"]), \
mock.patch.object(resolver, "_oci_tag_exists", side_effect=lambda h, r, t: t == "v3.3.5"):
row = resolver.resolve(ref)
self.assertEqual(row["status"], "ok")
self.assertFalse(row["upgrade_available"]) # release exists; image does not yet
self.assertIn("registry-manifest", row["source"])
with mock.patch.object(resolver, "list_tags", return_value=([], "RuntimeError: tag listing truncated")), \
mock.patch.object(resolver, "github_release_tags",
return_value=["v3.3.6", "v3.3.5"]), \
mock.patch.object(resolver, "_oci_tag_exists", return_value=True):
upgraded = resolver.resolve(ref)
self.assertTrue(upgraded["upgrade_available"])
if __name__ == "__main__":
unittest.main()
+16 -1
View File
@@ -2,11 +2,26 @@
| service | image | source repo | releases / changelog | | service | image | source repo | releases / changelog |
|---------|-------|-------------|----------------------| |---------|-------|-------------|----------------------|
| app | cryptpad/cryptpad | https://github.com/cryptpad/cryptpad | https://github.com/cryptpad/cryptpad/releases | | app | cryptpad/cryptpad | https://github.com/cryptpad/cryptpad | https://github.com/cryptpad/cryptpad/releases · https://hub.docker.com/r/cryptpad/cryptpad/tags |
| web | nginx | https://github.com/nginx/nginx | https://nginx.org/en/CHANGES | | web | nginx | https://github.com/nginx/nginx | https://nginx.org/en/CHANGES |
## Standing notes ## Standing notes
- `abra recipe upgrade` cannot parse the `version-YYYY.M.D` tag format of the cryptpad/cryptpad image, - `abra recipe upgrade` cannot parse the `version-YYYY.M.D` tag format of the cryptpad/cryptpad image,
so the app image is bumped by hand; the nginx sidecar is what `abra recipe upgrade` actually moves. so the app image is bumped by hand; the nginx sidecar is what `abra recipe upgrade` actually moves.
- Only use tags in the canonical `version-YYYY.M.x` scheme. The `opendesk-YYYY.MM.DD` tags on
Docker Hub are openDesk-specific builds, not newer CryptPad releases — e.g. `opendesk-2026.08.04`
(built 2026-08-04) still contains CryptPad `2026.5.1` in `/cryptpad/package.json`.
- CryptPad's GitHub releases and Docker Hub tags are not simultaneous: `2026.10.0` was released
2026-10-21; `version-2026.10.0` appeared on Docker Hub on 2026-10-23 (first seen by the weekly
upgrade run on 2026-10-30). Re-check the Docker Hub tags page before treating a GitHub release as
available for the recipe.
- The 2026.10.0 release is a breaking one for the recipe: full server rewrite (separate
`cryptpad-server` package), split `config.js` → `config/infra.js` (public origin/ports move),
one proxy port for HTTP + websocket (no separate `3003` websocket port), SSO plugin v1.0.0
required, and Euro-Office 9.3 replacing the OnlyOffice editors. Recipe handling in the
`upgrade-2026.10.0` PR: `CPAD_MAIN_DOMAIN`/`CPAD_SANDBOX_DOMAIN` must be full `https://…` URLs
(the new server assigns them verbatim to `infra.public.origin`/`sandboxOrigin`; the image's
entrypoint auto-creates `config/infra.js`), nginx `/cryptpad_websocket` proxies to `:3000`,
`abra.sh` config/nginx/SSO-entrypoint version labels bumped, SSO default 1.0.0.
- nginx is HTTP/1.1-only here (sidecar on :8083, `proxy_http_version 1.1`), so HTTP/2/3 CVE changes in - nginx is HTTP/1.1-only here (sidecar on :8083, `proxy_http_version 1.1`), so HTTP/2/3 CVE changes in
nginx releases generally don't affect this recipe — but still note them. nginx releases generally don't affect this recipe — but still note them.
+1
View File
@@ -7,6 +7,7 @@
## Standing notes ## Standing notes
- nginx even-numbered minor versions (1.28.x, 1.30.x) are mainline; odd (1.27.x, 1.29.x) are stable. The recipe tracks mainline. - nginx even-numbered minor versions (1.28.x, 1.30.x) are mainline; odd (1.27.x, 1.29.x) are stable. The recipe tracks mainline.
- nginx 1.31.7 (14 Oct 2026) is a security patch release with FIVE security fixes: CVE-2026-15176 (buffer overflow in a worker using `alias` when the URI was changed by `rewrite`), CVE-2026-93479 (buffer overflow evaluating string expressions with variables — affects document-root calculation and request logging in http/stream), CVE-2026-95101 (HTTP/3: part of an unknown frame treated as request body — bypasses `client_max_body_size` / injects into proxied request), CVE-2026-97629 (TLS session-ticket key rotation left one key uninitialized — session forgery), CVE-2026-97205 (use-after-free with async file I/O when an SSI page has `include virtual`). Features: geoip MMDB support, backend SSL cert verification by IP, QUIC reconfiguration + binary upgrade with `quic_bpf`. Change: `ngx_http_v2_module` now uses RFC 9218 prioritisation (RFC 7540 priority signals ignored). Bugfixes (X-Accel-Redirect cached response segfault, dav COPY/MOVE with alias segfault, QUIC flood false positives, SSL close_notify handling). NONE affect the custom-html static-file-serving use case (no HTTP/3/QUIC, no AIO/SSI, no `rewrite`+`alias` path mapping, no session-ticket rotation config) — but the 5 CVEs are why this bump is warranted. No breaking config changes; no operator action beyond the image pull.
- compose.git-pull.yml is an optional overlay for git-pull functionality; alpine/git version lives there. - compose.git-pull.yml is an optional overlay for git-pull functionality; alpine/git version lives there.
- compose.sftp.yml and compose.sso.yml are other optional overlays; linuxserver/openssh-server uses `latest` tag and is not version-pinned. - compose.sftp.yml and compose.sso.yml are other optional overlays; linuxserver/openssh-server uses `latest` tag and is not version-pinned.
- nginx 1.31.6 (15 Sep 2026) is a security patch release: CVE-2026-90439 (heap buffer overflow in a worker under certain configurations when using HTTP/3 with OpenSSL 3.5.0 and earlier). Change: QUIC transport parameters extension received in an SSL connection is now always ignored. Bugfixes: binary upgrade with control API socket + perl module; predicate-location evaluation error ignored; nested location lookup with regex/predicate locations; segfault reading config with `geo` … `ranges` + corrupted binary base file. None affect the custom-html static-file-serving use case (no HTTP/3/QUIC, no predicate locations, no perl, no `geo`). - nginx 1.31.6 (15 Sep 2026) is a security patch release: CVE-2026-90439 (heap buffer overflow in a worker under certain configurations when using HTTP/3 with OpenSSL 3.5.0 and earlier). Change: QUIC transport parameters extension received in an SSL connection is now always ignored. Bugfixes: binary upgrade with control API socket + perl module; predicate-location evaluation error ignored; nested location lookup with regex/predicate locations; segfault reading config with `geo` … `ranges` + corrupted binary base file. None affect the custom-html static-file-serving use case (no HTTP/3/QUIC, no predicate locations, no perl, no `geo`).
+4
View File
@@ -41,3 +41,7 @@ in this file, so keep vendor security pages listed here.
- **1.26.3**: carries a regression (#38177 "context deadline exceeded" opening repo code pages) — upstream says upgrade straight to 1.26.4, skip 1.26.3. - **1.26.3**: carries a regression (#38177 "context deadline exceeded" opening repo code pages) — upstream says upgrade straight to 1.26.4, skip 1.26.3.
- **1.27.0** (released 2026-07-13, MINOR with BREAKING changes): `Feat(actions)!: improve support for reusable workflows` (#37478) and `Use Content-Security-Policy: script nonce` (#37232, may break custom inline JS / reverse-proxy CSP). Many security + feature additions. A dedicated minor-bump run should evaluate CSP / reusable-workflow impact before adopting. - **1.27.0** (released 2026-07-13, MINOR with BREAKING changes): `Feat(actions)!: improve support for reusable workflows` (#37478) and `Use Content-Security-Policy: script nonce` (#37232, may break custom inline JS / reverse-proxy CSP). Many security + feature additions. A dedicated minor-bump run should evaluate CSP / reusable-workflow impact before adopting.
- **1.27.1** (released 2026-07-27, PATCH): security fix (oauth2 mandatory 2FA enforcement on authorize/grant endpoints, #38606), API swagger alignment, diff-contrast enhancement, and many bugfixes (actions reusable-workflow/job-stranding, OIDC end-session, repo-deletion cleanup, webhook/mail). No breaking changes; safe patch bump from 1.27.0. Release notes: https://github.com/go-gitea/gitea/releases/tag/v1.27.1 - **1.27.1** (released 2026-07-27, PATCH): security fix (oauth2 mandatory 2FA enforcement on authorize/grant endpoints, #38606), API swagger alignment, diff-contrast enhancement, and many bugfixes (actions reusable-workflow/job-stranding, OIDC end-session, repo-deletion cleanup, webhook/mail). No breaking changes; safe patch bump from 1.27.0. Release notes: https://github.com/go-gitea/gitea/releases/tag/v1.27.1
- **28.0.0** (released 2026-09-29, NEW MAJOR, BREAKING): git network operations now route through an internal proxy and egress settings changed (`Fix(git)!`, #39426) — outbound Git may need `[egress]` allow-listing; Actions adds `RUN_RETENTION_DAYS` (#38855). Also a large SECURITY batch (SSH key fingerprint identification, rejected invalid/duplicate Git objects, repo-scoped team authz, x/crypto DoS). GitHub notes: https://github.com/go-gitea/gitea/releases/tag/v28.0.0 — vendor post: https://blog.gitea.com/release-of-28.0.0/
- **28.1.0** (released 2026-10-06, MINOR): 9 security fixes (push-mirror permission/`DISABLE_NEW_PUSH`, local-path import authz, repository media stored XSS, OAuth2 refresh-token kind, issue-attachment authz, `FORCE_PRIVATE` push-to-create, private RSS/Atom feed leak, tag-delete deleting releases, migration page-size DoS), plus MySQL/MariaDB `READ COMMITTED` and go-git repack fixes. No breaking changes. GitHub notes: https://github.com/go-gitea/gitea/releases/tag/v28.1.0 — vendor post (CVE list): https://blog.gitea.com/release-of-28.1.0/
- **28.2.0** (released 2026-10-18, MINOR): 9 newly-fixed Gitea CVEs (SSH cert principal confusion, OpenID linking `state`, locked-issue reactions authz, release-attachment metadata leak, `/{username}/-/repositories` panic, draft-release attachment disclosure, Actions artifact v4 body limit, CSRF on repo topics, cross-org label deletion) plus dependency CVE-2026-78659 (`golang.org/x/net` v0.60.0). Three more CVEs are disclosed here but were already fixed in ≤28.1 (CVE-2026-110324, CVE-2026-109844, CVE-2026-97859). No breaking changes and no Postgres requirement. GitHub notes: https://github.com/go-gitea/gitea/releases/tag/v28.2.0 — vendor post (CVE list): https://blog.gitea.com/release-of-28.2.0/
- **app.ini writability**: Gitea 1.24+ writes runtime config (e.g. `WORK_PATH`) back to `app.ini`; mounting it directly from a read-only swarm config makes that write fail (non-fatal but warned). The unmerged mirror PR #4 (`fix-app-ini-writable`) seeds a writable copy in the `config` volume. Image bumps must not change the config mount / `docker-setup.sh` handling on their own.
+55
View File
@@ -79,6 +79,61 @@
reassign-faces fix) — no breaking changes, no config additions, no operator action. Done in reassign-faces fix) — no breaking changes, no config additions, no operator action. Done in
the 2026-09-18 upgrade (v3.2.0→v3.2.2, app+ML only; fresh PR — PR #4 was closed when its the 2026-09-18 upgrade (v3.2.0→v3.2.2, app+ML only; fresh PR — PR #4 was closed when its
v3.2.0 content merged upstream as #19 / `1.11.0+v3.2.0`). v3.2.0 content merged upstream as #19 / `1.11.0+v3.2.0`).
- **immich-server v3.2.4 (2026-09-28, latest stable) pins** (from `docker/docker-compose.yml` @ the
v3.2.4 tag): `postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357…` and
`valkey:9@sha256:70739f85ad…` — **BOTH identical to the recipe's v3.2.2-era pins** (DB + redis stay
put; live `valkey:9` tag has now moved to `ac9c858b…` = **9.1.2**, but immich's tested
`70739f85…` = 9.1.1 is kept per the 2026-07-17 precedent). `example.env` @ v3.2.4 byte-identical to
v3.2.2. Window v3.2.2→v3.2.4 is 4 commits: mobile sync-status fix (#31644), an internal `v3.2.3`
version commit (no Release object), a base-image dep bump to `202609281550` (#31854), and `v3.2.4`.
The v3.2.4 release body: "small patch that primarily fixes the memory leak people have observed
through a dependency update." **No breaking changes, no config additions, no operator action.**
The base-image bump (`202608300913`→`202609281550`, immich-app/base-images) adds a hardened
ImageMagick security policy (#390), reduces server image size (#388), and restores `curl` in the
prod image so `immich-healthcheck` works (#391) — all internal to the server image. `v3.3.0-rc.0/1`
are pre-releases (2026-09-29/30) — do NOT bump to rc. Done in the 2026-10-02 upgrade
(v3.2.2→v3.2.4, app+ML only, fresh PR). Direct registry check confirmed
`immich-server/ml:v3.2.4` exist; `abra recipe upgrade` still FATAs on the tag+digest DB pin.
Valkey CVE note: CVE-2026-56684 / CVE-2026-63639 (UAF→RCE) are fixed in 9.1.1 and are therefore
** already closed** by the recipe's pinned 9.1.1 — not by this app bump; ignore the live-`9`-tag
drift. CVE-2026-25243 (valkey RESTORE zipmap) is vendor-page-only and undecided.
- **immich-server v3.3.0 (2026-10-07) + v3.3.1 (2026-10-08, latest stable) pins** (from
`docker/docker-compose.yml` @ the v3.3.1 tag): `postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357…`
(UNCHANGED vs v3.2.4) and `valkey:9@sha256:c123e3715db63d06d4ad6964884037aa0d5d4d703939b9929954112889708e1d`
— the **valkey pin moved** 9.1.1 (`70739f85…`) → **9.1.2 (`c123e371…`)**, which is now immich's own
official tested pin (the live `9` tag has since drifted to `418652cf…`; keep immich's `c123e371…`
per the 2026-07-17 precedent). `example.env` @ v3.3.1 is **byte-identical** to v3.2.4. v3.3.0 is a
feature release (people sharing / cluster groups, birthday memories, auto-stacked mobile edits,
higher-quality thumbnails via linear-light resampling, OAuth claims synced on every login, PP-OCRv6
OCR models); v3.3.1 is a patch (people-sharing + mobile scroll fixes). **No breaking server-side
changes, no required config, no operator action.** New TypeORM migrations auto-run
(`ConvertUserOAuthIdEmptyStringToNull`, `RenameGeoNamesCountries`, `PersonSharing`,
`AddPersonUserTableSharedBySharedWithConstraint`) — all succeeded in a live v3.2.4→v3.3.1 deploy.
Opt-in only: `MACHINE_LEARNING_MODEL_REVISION=v2` (optimized models, numerically equivalent, no job
re-run) and the people-sharing feature (user-triggered button, not a deploy migration). Done in the
2026-10-09 upgrade (v3.2.4→v3.3.1, app+ML+valkey, extending PR #6; fresh `!testme` GREEN — build #45).
- **immich-server v3.3.3 (2026-10-14) + v3.3.4 (2026-10-20) + v3.3.5 (2026-10-21, latest stable) pins** (from
`docker/docker-compose.yml` @ the v3.3.5 tag): `postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357…`
(UNCHANGED since v3.3.0) and `valkey:9@sha256:c123e371…` (immich's tested 9.1.2, also unchanged). Upstream
coopcloud published v3.3.4 as `1.12.0+v3.3.4` (`6b294fce`) and moved the recipe's `redis` pin to the then-live
`418652cf…`; that divergence is left as upstream has it (no reason to churn a sidecar in an app bump).
`docker/docker-compose.yml` and `docker/example.env` are byte-identical between v3.3.4 and v3.3.5.
v3.3.3 = bug fixes (mobile memories date, people-query perf); v3.3.4 = further people-query perf + translation;
**v3.3.5 = security patch** (certifi CA-trust fix + October 2026 Node.js security patches). No schema
migration, no config/env change, no breaking server-side change. Done in the 2026-10-23 upgrade
(v3.3.4→v3.3.5, app+ML only, extending PR #6; live v3.3.4→v3.3.5 dev deploy healthy — migrations finished,
`vector`/`vchord` present, no pgvectors drop; `!testme` GREEN first run — build #78).
- **2026-10-30: postgres sidecar pgvectors 0.2.0 → 0.3.0 (same PG14 + VectorChord 0.4.3 scheme).** With
app/ML already at v3.3.5, the only newer same-scheme tag is
`ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.3.0` (multi-arch OCI index digest
`sha256:87c050465fb969a68c7ac23e375e21f4c95cfacd0edce5fa1bc31e63b7891891`; re-verified via ghcr +
`docker buildx imagetools inspect`). immich v3.3.5's own compose still names 0.2.0, but immich only
supports `vchord`/`vector` now — `vectors` (pgvecto.rs) is not in its `VECTOR_EXTENSIONS`, so the
sidecar library is unused; 0.3.0 is backward-compatible (`ALTER EXTENSION vectors UPDATE`), immich
added 0.3.0 support in immich-app/immich#11624, and this recipe previously shipped 0.3.0 with
v3.0.1/v2.7.5. Direct dev deploy: 0.2.0-created volume reused by the 0.3.0 task, PG14 clean start,
`vchord 0.4.3`/`vector 0.8.1` installed, `/api/server/version` = 3.3.5, `!testme` GREEN (build #105).
Recipe PR extends upstream main `ac03d23`; publish with `abra recipe release immich -z`.
- **2026-08-07 INFRA note: cc-ci runner's gitea clone-token is STALE (HTTP 401).** `!testme` build - **2026-08-07 INFRA note: cc-ci runner's gitea clone-token is STALE (HTTP 401).** `!testme` build
#1210 died at the recipe `git clone` step (~5s, before any deploy) with `could not read Username for #1210 died at the recipe `git clone` step (~5s, before any deploy) with `could not read Username for
'https://git.autonomic.zone'` — the runner's mounted gitea token `13e299f2…` is rejected (verified 'https://git.autonomic.zone'` — the runner's mounted gitea token `13e299f2…` is rejected (verified
+17 -2
View File
@@ -6,6 +6,9 @@
| backend | lasuite/impress-backend | https://github.com/suitenumerique/docs | https://github.com/suitenumerique/docs/releases | | backend | lasuite/impress-backend | https://github.com/suitenumerique/docs | https://github.com/suitenumerique/docs/releases |
| celery | lasuite/impress-backend | https://github.com/suitenumerique/docs | https://github.com/suitenumerique/docs/releases | | celery | lasuite/impress-backend | https://github.com/suitenumerique/docs | https://github.com/suitenumerique/docs/releases |
| y-provider | lasuite/impress-y-provider | https://github.com/suitenumerique/docs | https://github.com/suitenumerique/docs/releases | | y-provider | lasuite/impress-y-provider | https://github.com/suitenumerique/docs | https://github.com/suitenumerique/docs/releases |
| yhub | lasuite/impress-yhub | https://github.com/suitenumerique/docs | https://github.com/suitenumerique/docs/releases |
| yhub-valkey | valkey/valkey | https://github.com/valkey-io/valkey | https://github.com/valkey-io/valkey/releases |
| jwt-keys | lasuite/impress-yhub (Node crypto sidecar) | https://github.com/suitenumerique/docs | https://github.com/suitenumerique/docs/blob/main/UPGRADE.md |
| docspec | ghcr.io/docspec/api | https://github.com/docspec/docspec | https://github.com/docspec/docspec/releases | | docspec | ghcr.io/docspec/api | https://github.com/docspec/docspec | https://github.com/docspec/docspec/releases |
| db | pgautoupgrade/pgautoupgrade | https://github.com/pgautoupgrade/docker-pgautoupgrade | https://github.com/pgautoupgrade/docker-pgautoupgrade/releases | | db | pgautoupgrade/pgautoupgrade | https://github.com/pgautoupgrade/docker-pgautoupgrade | https://github.com/pgautoupgrade/docker-pgautoupgrade/releases |
| redis | redis | https://github.com/redis/redis | https://hub.docker.com/_/redis/tags | | redis | redis | https://github.com/redis/redis | https://hub.docker.com/_/redis/tags |
@@ -13,8 +16,20 @@
| web | nginx | https://github.com/nginx/nginx | https://nginx.org/en/CHANGES | | web | nginx | https://github.com/nginx/nginx | https://nginx.org/en/CHANGES |
## Standing notes ## Standing notes
- impress-frontend, impress-backend (used for both backend + celery services), impress-y-provider all - **v6.0.x replaces the collaboration server**: hocuspocus/y-provider collaboration is gone; a new
share the same version tag from the suitenumerique/docs monorepo. Upgrade all four in lockstep. `yhub` service (port 3002) serves `/collaboration/*`, with its own `yhub` PostgreSQL database and a
Valkey store. `y-provider` is conversion-only. New RSA keys (`JWT_PRIVATE_KEY_FILE` backend,
`YHUB_JWT_PRIVATE_KEY_FILE` yhub), `COLLABORATION_SERVER_SECRET` removed. Docs must be migrated into
yhub (`SOFT_MIGRATION=true` + `backend migrate_documents`). See UPGRADE.md `[6.0.0]`.
- **v6.0.2** moves yhub to `@y/hub 0.10.1`, adding the `yhub_ydoc_versions_v1` table/indexes: re-run the
yhub schema script (`init-db`) — the recipe does it on every yhub start.
- **The recipe generates the two RSA keys at runtime** into a `jwt_keys` volume (`jwt-keys` sidecar)
because abra cannot generate PEM secrets (`generate=false`) and Swarm secrets must pre-exist.
- **docspec**: since Docs v5.7.0 the Rust `ghcr.io/docspec/api` (port 3000) replaces the archived Elixir
`ghcr.io/docspecio/api` (port 4000); Docs v6.0.2 expects `DOCSPEC_API_URL=http://docspec:3000/conversion`.
- impress-frontend, impress-backend (used for both backend + celery services), impress-y-provider,
impress-yhub (and the jwt-keys sidecar image) all share the same version tag from the
suitenumerique/docs monorepo. Upgrade all of them in lockstep.
- AUTO_MIGRATIONS=true means DB migrations run automatically on backend startup. No manual step needed. - AUTO_MIGRATIONS=true means DB migrations run automatically on backend startup. No manual step needed.
- Minio tag uses a date-based RELEASE.YYYY-MM-DDTHH-MM-SSZ format — abra cannot parse it for upgrades; - Minio tag uses a date-based RELEASE.YYYY-MM-DDTHH-MM-SSZ format — abra cannot parse it for upgrades;
check manually on https://github.com/minio/minio/releases. check manually on https://github.com/minio/minio/releases.
+147
View File
@@ -239,3 +239,150 @@
safe (sqlite, TypeORM auto-migrate). Operator flags unchanged (API-caller-level deprecations, safe (sqlite, TypeORM auto-migrate). Operator flags unchanged (API-caller-level deprecations,
encryption-key rework informational, N8N_DB_PING_TIMEOUT warn-only, recipe doesn't set it). encryption-key rework informational, N8N_DB_PING_TIMEOUT warn-only, recipe doesn't set it).
2.40.3 exists (see above) — flagged, not taken. Recommended release: `-y`. 2.40.3 exists (see above) — flagged, not taken. Recommended release: `-y`.
- 2.40.4 (2026-09-21, Pre-release): core fix (tear down workflow triggers when publication meets a
node type the instance cannot load) + perf (stop loading project members when listing
credentials).
- 2.40.5 (2026-09-21, Pre-release): core fix (limit declarative routing during base URL ownership
checks). Docker Hub `2.40.5` manifest verified active multi-arch (amd64+arm64, digest
sha256:9f693fd5..., 2026-09-21) — not a withdrawn 2.37.5-style partial tag. NOTE: 2.40.x
`POST /rest/login` now takes `emailOrLdapLoginId` instead of `email` (request-shape change; the
cc-ci login-state test only GETs /rest/login so unaffected).
- 2026-09-21 run: the flagged catch-up. PR #8 (branch `upgrade-ef0dd56`, at 2.40.2, !testme GREEN
2026-09-18) extended **2.40.2 → 2.40.5** on the same upstream main tip `0b436ec` (n8n NOT
merged upstream today, unlike discourse/keycloak/matrix-synapse). 2.40.2→2.40.5 is bugfix-only
(encryption-key raw-repair, trigger teardown, declarative-routing limit); no breaking
compose/env changes; no new migrations beyond the 15 already verified at 2.40.2. Stable badge
now on the 2.39.x line (2.39.9/2.39.10 released 2026-09-21); 2.40.x pre-release per precedent.
Recommended release: `-y`.
- 2.40.6 (2026-09-24, patch): core fixes (keep OpenTelemetry export working after a restart when
Sentry is enabled; retry instance reports that cross the UTC midnight boundary) + feature
(authenticate instance reports with the license certificate).
- 2.40.7 (2026-09-25, patch): 1 core fix (propagate project span attributes to node spans).
- 2.41.0 (2026-09-22, Pre-release; the 2.41 feature minor — note the 2.40.x patch line continued
alongside with 2.40.6/2.40.7): a large feature/bugfix release. Features: standalone node
execution as a Workflow Builder / AI Assistant tool, admin permission for node execution in the
assistant, API credential creation with source IDs, push-based reload endpoint for custom-node
development, execution view/delete permissions in project roles, MCP-registry capability
filtering, Databricks Embeddings/Chat-Model node, MS Teams @mentions + a Chat resource, MiniMax
dynamic model list, email-change confirmation flow. Core bugfixes include repair of
data-encryption keys stored as the raw instance key, stalled-job success finalization,
multi-main workflow-publishing fix, stop an unreachable external-secrets provider blocking
startup, undici 7.29.1 + vm2 3.12.2 bumps, RFC 9068 MCP OAuth compliance. **No breaking
compose/env/migration changes; no `N8N_*` env renames.** (2.41.1/2.41.2 = patch bugfixes.)
- 2.41.3 (2026-09-25, patch): 1 core fix (propagate project span attributes to node spans). Last
week's PR #9 target — !testme GREEN (run 21, 2026-09-28); TypeORM migrations clean live.
- 2.41.4 (2026-09-30, patch): API/core/editor fixes (return an execution when its stored trace
context is incomplete; count a database ping as successful when its reply arrives during
event-loop lag; store queue job results only for executions this process enqueued) + features
(n8n Assistant onboarding thread for new Cloud signups; route assistant credit CTAs to top-up).
- 2.41.5 (2026-10-01, patch): atom-feed release body empty (release republish / no listed changes).
- 2.42.0 (2026-09-29, Pre-release; the 2.42 feature minor): a large feature/bugfix release.
Features: AI Agent Tool can use its own tools under a pre-v3 parent agent, workflow descriptions
on create, `extendsCredential` in workflow create/update, OpenAPI path-parameter validation,
credential descriptions behind an instance flag, Anthropic prompt caching in the Message op,
API projects with source IDs, pinned Apply/Apply-Continue CLI commands, MCP-registry feature-flag
removal, Agent Builder/editor improvements. ~60 core/editor/node bugfixes (chat-integration
shutdown, Redis pubsub reconnect + half-open detection, OTel-after-restart-with-Sentry,
instance-report unique index, expression-isolate release, async delete retries). **No breaking
compose/env/migration changes; no `N8N_*` env renames.**
- 2.42.1 (2026-09-30, patch): API/editor fixes (emit valid schemas for untyped values in the Public
API spec; return an execution when its trace context is incomplete) + editor feature (show the
n8n logo on canvas in canvas-only mode).
- 2.42.2 (2026-10-01, patch; **newest numeric tag on Docker Hub — verified 2026-10-02**): 1 core
fix (stop waiting on Bull `job.finished()` for queued executions). Docker Hub `2.42.2` manifest
active; no 2.43.x exists.
- 2026-10-02 run: reconcile confirmed coopcloud upstream main still `9141e23` (3.5.1+2.40.5) and
PR #9 (branch `upgrade-a144baf`, at 2.41.3, !testme GREEN 2026-09-28) still open/unmerged — so
per the one-evolving-PR contract the run extends PR #9 with a fast-forward commit
**2.40.5 → 2.42.2**. abra lists 2.42.2 as the newest candidate and a direct Docker Hub tag
enumeration agrees (no 2.43.x). The delta spans two feature minors (2.41.0, 2.42.0) plus patches
on both lines; **no breaking compose/env/migration changes, no `N8N_*` renames**. Rolling
upgrade safe (sqlite default; TypeORM migrations auto-run on boot). Operator flags remain
HTTP-API-caller-level only (2.33.0 workflow activate/deactivate deprecation; 2.36.0
`Array.merge`→`mergeIntoObject`; 2.37.0 JSON content-type on decorator body routes + binary-data
endpoint adapt; 2.39.0 workflow-version endpoint deprecation; 2.40.x `POST /rest/login`
`emailOrLdapLoginId` rename). Recommended release: `-y`.
- 2.42.3 (2026-10-05, patch): API fix (let global members grant variable scopes to API keys) + core
fix (keep the task runner running on an unhandled promise rejection) + editor feature (Gateway
credits promotions on community nodes).
- 2.42.4 (2026-10-07, patch): release body empty (republish / no listed changes).
- 2.42.5 (2026-10-08, patch): 1 editor fix (show the Agent artifact while the builder creates it).
- 2.42.6 (2026-10-09, patch): core feature (block create/update of workflows with deprecated nodes) +
Gemini node feature (Nano Banana 2.1 + 2 Lite in the image-model list). Backport of the 2.43.3 pair
onto the stable 2.42 line.
- 2.42.7 (2026-10-12, patch): 2 core fixes (concurrent credential tests no longer share one node-type
registry; resume tool calls for deferred tools a skill loaded).
- 2.43.0 (2026-10-06, Pre-release; the 2.43 feature minor): a large feature/bugfix release. Features:
Agent Builder / Instance AI / editor improvements + ~many core/API/editor fixes (valid schemas for
untyped values in the Public API spec; global members can grant variable scopes to API keys; return
an execution when its stored trace context is incomplete; Azure OpenAI Chat Model; etc.). **No
breaking compose/env/migration changes; no `N8N_*` env renames.** (2.43.1/2.43.2 = patch bugfixes:
editor Assistant/Execute UI; Kimi K3 + OpenAI-compatible Assistant providers self-hosted; Agent
artifact preview; agent-builder prompt-caching/background-sandbox perf.)
- 2.43.3 (2026-10-09, Pre-release): core feature (block create/update of workflows with deprecated
nodes) + Gemini node feature (Nano Banana 2.1 + 2 Lite in the image-model list).
- 2.43.4 (2026-10-12, patch, **stable line**): 2 core fixes (handle each Chat Hub segment once in
queue mode; resume tool calls for deferred tools a skill loaded).
- 2.43.5 (2026-10-14, patch, stable line): core fix (validate the `size` arg of the `chunk()`
expression extension) + editor feature (show Gemini models by name / prioritised first).
- 2.43.6 (2026-10-15, patch, **stable line — `stable` floating Docker tag points here**, digest
`sha256:dd580fe1…`): 1 core fix (create an XML parser per webhook request body). Backport of the
2.44.2 fix onto the stable 2.43 line.
- 2.44.0 (2026-10-13, **Pre-release / beta line — `beta` floating Docker tag points to 2.44.2**): the
2.44 feature minor. Large core/editor change set (credential/node-type policy enforcement across
agent tools, credential tests/probes, package import, folder/credential/delete-transfer moves;
block the assistant when AI credits hit zero; per-process crash-journal paths; configurable
healthcheck timeout; Microsoft SharePoint Trigger node; many bugfixes). **No breaking
compose/env/volume/label changes; no `N8N_*` renames.**
- 2.44.1 (2026-10-14, Pre-release/beta): 1 editor feature (Gemini models by name / prioritised first).
- 2.44.2 (2026-10-15, Pre-release/beta; **newest numeric tag abra lists**; Docker Hub `beta` floating
tag = `sha256:b57e7a9c…`): 1 core fix (create an XML parser per webhook request body).
- 2026-10-09 run (not separately logged here): PR #9 (branch `upgrade-a144baf`) extended from the
2.42.2 of the 2026-10-02 run to **2.43.2** (newest numeric tag abra listed that day). 2.43.2 was
still Pre-release on GitHub at the time (tracking-the-newest-tag precedent).
- 2026-10-16 run: reconcile confirmed coopcloud upstream main still `9141e23` (3.5.1+2.40.5) and
PR #9 (branch `upgrade-a144baf`, at 2.43.2) still open/unmerged, so per the one-evolving-PR
contract the run extends PR #9 **2.43.2 → 2.43.6**. Operator guidance this run: verify whether
2.44.x is the newest STABLE line and take the newest SAFE patch. **2.44.x is the beta/pre-release
line** — GitHub marks 2.44.0/2.44.1/2.44.2 `prerelease=true`, and Docker Hub's `beta` floating tag
points at 2.44.2 while `stable` points at 2.43.6 (verified live). So the safe target is **2.43.6**
(newest patch on the now-stable 2.43 line), NOT the beta 2.44.2. 2.43.2→2.43.6 is patch-only
(2.43.3 deprecated-node workflow guard + Gemini image list; 2.43.4 Chat-Hub queue-mode + deferred
tools; 2.43.5 `chunk()` size-arg validation + Gemini picker; 2.43.6 per-request XML parser). No
breaking compose/env/migration changes. Live-deployed on cc-ci `dev-n8n` (2.40.5 → 2.43.6): TypeORM
migrations completed cleanly, editor `/rest/login` 401 (auth-gated, as expected), `/healthz` 200 —
converged. Advisory scan 2.43.2→2.43.6: 0 CVEs fixed (9 sources, none failed). Recommended
release: `-y` (relative to the published label app tag 2.40.5 the delta spans feature minors
2.41/2.42/2.43).
- 2.43.7 (2026-10-16, patch): core suspend in-flight production executions at worker shutdown.
- 2.43.8 (2026-10-19, patch): core turn off expression spans by default (backport of 2.44.4).
- 2.44.3 (2026-10-16, patch): core keep Observer detail lines instead of discarding the response.
- 2.44.4 (2026-10-19, patch): core reuse connections in the log-streaming webhook destination; turn
off expression spans by default.
- 2.44.5 (2026-10-21, patch): core key queue job outcomes by job on shutdown resume; editor load
credential types on startup.
- 2.44.6 (2026-10-22, patch; **`stable` floating tag points here**): Postgres Node prevent global
pgtypes overwrite.
- 2.45.0 (2026-10-20, Pre-release; the 2.45 feature minor): automated migrations for removed nodes;
node and credential type policies enabled by default; durable scheduler enabled by default;
Idempotency-Key on Public API POSTs; workflow migration-findings endpoints/notifications; Kafka v2
default for new nodes; Monday.com V2 default; AI Agent/Assistant, editor and node fixes; UI rebrand.
**No breaking compose/env/volume/label changes; no `N8N_*` renames.**
- 2.45.1 (2026-10-21, Pre-release): core claim an idle worker's next queue job and its lock atomically;
Microsoft Teams Trigger Graph subscription renewal via lifecycle notifications; editors can triage
migration findings.
- 2.45.2 (2026-10-22, Pre-release; **`beta` floating tag points here — newest numeric tag**): core read
type-availability policies from memory at enforcement points; Postgres Node prevent global pgtypes
overwrite; migration report CSV export; OAuth-confirmation/email logo update.
- 2026-10-23 run: reconcile confirmed coopcloud upstream main moved to `c834df2` ("NODE_FUNCTION_ALLOW_EXTERNAL
in env file", on top of 3.5.1+2.40.5) and PR #9 (branch `upgrade-a144baf`, at 2.43.6) still
open/unmerged. Per the one-evolving-PR contract, PR #9 extended **2.40.5 → 2.45.2** (explicit target
= highest tag abra lists). 2.45.x is the current Pre-release/beta line; **stable = 2.44.6** (verified
live via Docker Hub floating tags), flagged to the operator. Window spans feature minors 2.41–2.45,
but release notes show **no breaking compose/env/volume/label changes and no `N8N_*` renames**.
Live-deployed on cc-ci `dev-n8n`: TypeORM migrations completed cleanly through
`CreatePromotionRunAndReviewTables1792139680417`, editor accessible, `/healthz` 200, `/rest/login`
401; runtime deprecation warnings only (`WEBHOOK_URL`→`N8N_WEBHOOK_URL`, internal runner mode,
unverified-packages default, runner-timeout/compression-limit default reductions). Advisory scan
2.40.5→2.45.2: 0 CVEs identified, BUT the GitHub advisory sources failed (unauthenticated rate
limit) — scan incomplete, not to be read as clean. Recommended release: `-y`.
+23
View File
@@ -0,0 +1,23 @@
# Upstream sources — rustdesk-server
| service | image | source repo | releases / changelog |
|---------|-------|-------------|----------------------|
| app | rustdesk/rustdesk-server | https://github.com/rustdesk/rustdesk-server | https://github.com/rustdesk/rustdesk-server/releases |
| signal | rustdesk/rustdesk-server | https://github.com/rustdesk/rustdesk-server | https://github.com/rustdesk/rustdesk-server/releases |
- nvd-cpe: rustdesk-server = cpe:2.3:a:rustdesk:rustdesk_server:*:*:*:*:oss:*:*:*
## Standing notes
- Enrolled in the cc-ci weekly set. The NVD CPE above is scoped to the open source server edition;
do not count RustDesk Client or Server Pro CVEs for this recipe.
- Both services (`app` = hbbs, `signal` = hbbr) use the SAME image — always bump them together.
- Docker Hub has stray `1.4.5-*` arch tags (pushed 2026-01-12, top-level `1.4.5` tag deleted, no
GitHub release) — an aborted/alias push, NOT a real version line. The real line is `1.1.x`;
GitHub "Latest" + docker `latest` agree (1.1.16 as of 2026-07-20). Ignore 1.4.5.
- Recipe uses `-k _` on both hbbs and hbbr (enforces key verification, enforced-key mode since
1.1.11) and publishes host-mode ports 21115-21119 (hbbs) + 21117/21119 (hbbr) — direct
client connections, only 21118/21119 are the web-socket/http endpoints behind nothing; the
traefik labels on `app` are vestigial (hbbs has no HTTP :80 service; healthcheck commented out).
- 1.1.15 raised **default bandwidth limits** and added auth-failure logging (fail2ban-friendly);
1.1.16 fixed **unauthenticated UDP punch-hole reflection/amplification abuse** + critical `mio`
0.8.5→0.8.11 issue + i32 overflow (peers offline 24.9–49.7d shown online).
+2 -1
View File
@@ -28,10 +28,11 @@ lasuite-drive weekly
lasuite-meet weekly lasuite-meet weekly
mailu weekly mailu weekly
matrix-synapse weekly matrix-synapse weekly
mattermost-lts weekly mattermost-lts external # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded
mumble weekly mumble weekly
n8n weekly n8n weekly
plausible external # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded plausible external # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded
rustdesk-server weekly
uptime-kuma external # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded uptime-kuma external # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded
wordpress weekly wordpress weekly
``` ```
Generated
+4 -4
View File
@@ -10,11 +10,11 @@
] ]
}, },
"locked": { "locked": {
"lastModified": 1790010051, "lastModified": 1793644559,
"narHash": "sha256-0dz8fg4zqg54tk+1A7CHZAw+lK9vWxre6Tioi5B0ocU=", "narHash": "sha256-u6xPHneKwIrd1taLVs1X12dUs46RzTzkZ9fJ8d/ZvYI=",
"ref": "refs/heads/main", "ref": "refs/heads/main",
"rev": "2f6787f0433e7ff3053116039f10c00cde34128f", "rev": "d569e6528f2eba4309db4794b091d6a1ecc029bf",
"revCount": 1558, "revCount": 1589,
"type": "git", "type": "git",
"url": "https://git.autonomic.zone/recipe-maintainers/cc-ci.git" "url": "https://git.autonomic.zone/recipe-maintainers/cc-ci.git"
}, },
+2
View File
@@ -1,5 +1,7 @@
# Memory index # Memory index
- [Enroll gate fixtures must fail loudly](enroll-gate-fixtures-fail-loudly.md) — version-gate reads via lifecycle.deployed_identity, never None-into-if; post-host-merge deploy = --cc-ci-only after merge (staging obsolete)
- [Orchestrator host: Hetzner](orchestrator-host-hetzner.md) — runs on Hetzner cpx22; rebuild cmd, loops-service bounce, git-identity gotcha - [Orchestrator host: Hetzner](orchestrator-host-hetzner.md) — runs on Hetzner cpx22; rebuild cmd, loops-service bounce, git-identity gotcha
- [Push commits to remote](push-commits-to-remote.md) — push to git.autonomic.zone right after every commit in this repo - [Push commits to remote](push-commits-to-remote.md) — push to git.autonomic.zone right after every commit in this repo
- [Regression canary cadence](regression-canary-cadence.md) — server E2E canaries run on polish/review/release, not every commit - [Regression canary cadence](regression-canary-cadence.md) — server E2E canaries run on polish/review/release, not every commit
@@ -0,0 +1,43 @@
---
tags: [cc-ci, enrollment, testing, deploy]
---
# Enroll-verify gate fixtures must fail loudly
## Context
During rustdesk-server enrollment (2026-10-30, cc-ci PR #48) the first full-suite verification
run came back RED only because of the *fixture*, not the app: the punch-hole regression test's
version gate read the deployed image with a hand-rolled `docker service inspect`, got None, and
the `if version is not None and version < (1,1,16)` gate silently decided to RUN on a 1.1.12 head
where pre-fix answering is the expected behavior. An expected-RED looked like a real bug.
## Lesson
- A test gate that cannot decide its condition must `pytest.fail` LOUDLY with the raw value —
never default to None-into-an-if (silently runs → wrong RED) or silent skip (vacuous green).
- cc-ci already has a first-class image reader: `runner/harness/lifecycle.py::deployed_identity(domain)`
→ `{"version","image","chaos"}`; parse the tag from `.image` and strip any `@sha256:` digest.
Hand-rolled service inspects in recipe_meta/custom tests are a duplication hazard.
- Version-gated custom tests skip on pre-fix heads by design (STYLE §2) — that is fine as long
as the version read itself cannot fail silently.
# cc-ci host deploy mechanism (post host-merge, 2026-09+)
To run any merged cc-ci main change on the live host now, use the `--cc-ci-only` path from
/cc-ci-orchestrator-update — merge the PR first (enrollment PRs: comment verification evidence,
then merge — invocation = authorization), then:
```
ssh cc-ci 'sudo -H -u loops bash -lc "cd /srv/cc-ci-orch && nix flake update cc-ci &&
git -c user.name=auto-update -c user.email=auto-update@cc-ci commit -qm \"flake.lock: cc-ci -> main\" -- flake.lock &&
git push -q origin HEAD:main"'
ssh cc-ci 'git -C /etc/cc-ci pull --ff-only --recurse-submodules'
systemd-run --unit=ccci-rebuild-test --collect -E HOME=/root -p WorkingDirectory=/srv/cc-ci-orch \
bash -lc "nixos-rebuild test --flake /srv/cc-ci-orch#cc-ci" # := health checks := then switch
```
The old staging procedure (branch clone at /root/cc-ci-deploy, referenced by the retired
/cc-ci-server-update skill and old enroll SKILL.md step 7 lines) is obsolete — the combined-host
flake takes the cc-ci module from the lockfile, so a BRANCH cannot be staged that way; deploy
after merge instead.
+88
View File
@@ -188,6 +188,94 @@ SSHCFG
''; '';
}; };
# ---- weekly opencode CLI auto-upgrade ----------------------------------------------------
# opencode is not in nixpkgs: opencode-install (above) only installs the standalone CLI when
# the binary is MISSING, so the installed version just ages in place (it sat on 1.18.29 for
# weeks while 1.18.33 was out). opencode's built-in autoupdate does not cover this host: it
# auto-applies patch releases only and only fires on a fresh interactive TUI start, and every
# agent here lives inside the long-lived `opencode serve` (opencode-web). This unit is the
# mechanism instead. It runs weekly in the Tuesday maintenance window (an hour BEFORE the
# host auto-update at Tue 03:00 UTC, and clear of the Thursday recipe-upgrade run and the
# Sunday canonical sweep), and:
# 1. compares the installed version with the latest GitHub release (no-op when equal);
# 2. reinstalls via the official installer (same code path as opencode-install) when they
# differ, and re-links ~/.local/bin/opencode;
# 3. restarts opencode-web so the new binary actually takes effect — sessions attached to
# it (orchestrator, upgrader, report) drop and their supervisors re-attach/resume, which
# is why the busy gate below must hold: a mid-flight CI/upgrade run is never cut, and a
# skipped run simply retries next week.
# Deploying this module never itself bumps opencode: boot/activation installs stay
# install-if-missing in opencode-install, and this unit is timer-driven only.
# The outcome of each run lands in .cc-ci-logs/opencode-update-state (read by /cc-ci-status).
systemd.services.opencode-upgrade = {
description = "Weekly opencode CLI auto-upgrade (latest release → reinstall → restart opencode-web)";
after = [ "network-online.target" "opencode-install.service" "opencode-web.service" ];
wants = [ "network-online.target" ];
serviceConfig = { Type = "oneshot"; TimeoutStartSec = "30min"; };
path = with pkgs; [ curl bash coreutils gnugrep gnutar gzip unzip systemd util-linux procps ];
environment = { HOME = "/home/loops"; OPENCODE_UI_HOST = cfg.opencodeUiHost; };
script = ''
set -u
BIN=/home/loops/.local/bin/opencode
STATE=/srv/cc-ci-orch/.cc-ci-logs/opencode-update-state
ocver() { "$BIN" --version 2>/dev/null | tail -1 || true; }
state() { printf '%s result=%s installed=%s note=%s\n' "$(date -u +%FT%TZ)" "$1" "$2" "$3" > "$STATE"; chown loops:users "$STATE" 2>/dev/null || true; }
busy() { echo "BUSY: $1 — skipping this week's opencode upgrade (retries next week)"; state skipped "$(ocver)" "$1"; exit 0; }
# Same busy gate as auto-update.nix: never cut a CI run, the weekly upgrade or the sweep.
pgrep -f run_recipe_ci >/dev/null && busy "a CI run is in flight"
systemctl is-active --quiet nightly-sweep.service && busy "the canonical sweep is running"
runuser -u loops -- tmux has-session -t cc-ci-upgrader 2>/dev/null && busy "the weekly recipe-upgrade run is in flight (tmux cc-ci-upgrader)"
runuser -u loops -- tmux has-session -t cc-ci-report 2>/dev/null && busy "the weekly report is being written (tmux cc-ci-report)"
if [ -r /run/secrets/bridge_drone_token ]; then
running=$(curl -s -m 20 -H "Authorization: Bearer $(cat /run/secrets/bridge_drone_token)" \
"https://drone.ci.commoninternet.net/api/repos/recipe-maintainers/cc-ci/builds?per_page=10" \
| grep -o '"status":"running"' | wc -l)
[ "''${running:-0}" -eq 0 ] || busy "$running Drone build(s) running"
fi
INSTALLED=$(ocver)
LATEST=$(curl -fsSL -m 30 https://api.github.com/repos/anomalyco/opencode/releases/latest \
| grep -Po '"tag_name":\s*"v?\K[0-9][0-9.]*' || true)
[ -n "$LATEST" ] || { echo "could not determine the latest opencode release"; state failed "''${INSTALLED:-none}" "latest-unresolved"; exit 1; }
echo "installed: ''${INSTALLED:-none} latest: $LATEST"
if [ "$INSTALLED" = "$LATEST" ]; then
echo "opencode is up to date"
state ok "$INSTALLED" "up-to-date"
exit 0
fi
echo "upgrading opencode: ''${INSTALLED:-none} -> $LATEST"
runuser -u loops -- env HOME=/home/loops bash -c 'curl -fsSL https://opencode.ai/install | bash' \
|| { echo "install failed — the previously installed version is untouched"; state failed "''${INSTALLED:-none}" "install-failed"; exit 1; }
mkdir -p /home/loops/.local/bin
ln -sfn /home/loops/.opencode/bin/opencode /home/loops/.local/bin/opencode
NEWVER=$(ocver)
[ "$NEWVER" = "$LATEST" ] || { echo "install ran but opencode reports $NEWVER (wanted $LATEST) — not restarting"; state failed "$NEWVER" "install-mismatch"; exit 1; }
echo "restarting opencode-web so the new binary takes effect (attached sessions drop; their supervisors resume)"
systemctl restart opencode-web.service
sleep 10
systemctl is-active --quiet opencode-web.service || { echo "opencode-web did not come back after the upgrade"; state failed "$NEWVER" "web-restart-failed"; exit 1; }
code=$(curl -s -m 20 -o /dev/null -w '%{http_code}' --resolve "$OPENCODE_UI_HOST:443:127.0.0.1" "https://$OPENCODE_UI_HOST/")
[ "$code" = "401" ] || { echo "opencode UI answered $code, not the 401 auth challenge"; state failed "$NEWVER" "ui-check-$code"; exit 1; }
echo "opencode upgraded to $NEWVER; opencode-web restarted and healthy"
state ok "$NEWVER" "upgraded; web restarted"
'';
};
systemd.timers.opencode-upgrade = {
wantedBy = [ "timers.target" ];
timerConfig = {
# Weekly slot in the Tuesday maintenance window, an hour BEFORE the host auto-update
# (cc-ci-auto-update.timer: Tue 03:00 UTC) so a restarted opencode-web has settled before
# that run's health check; Persistent=false, like the auto-update — no catch-up at boot.
OnCalendar = "Tue *-*-* 02:00:00 UTC";
Persistent = false;
RandomizedDelaySec = "10min";
};
};
# ---- opencode web server: one shared instance the opencode-backed agents attach to ------- # ---- opencode web server: one shared instance the opencode-backed agents attach to -------
# Provider creds come from /srv/cc-ci/.testenv (out of band, see README). # Provider creds come from /srv/cc-ci/.testenv (out of band, see README).
systemd.services.opencode-web = { systemd.services.opencode-web = {