The Gitea host was stored in the credentials env file, so a literal
secret scanner treated it as a secret and refused two READMEs in another
repo that link to git.autonomic.zone.
It was never secret: advisory-scan.py, pr-survey.py and recipe-report.py
already default it to git.autonomic.zone, and it is this repo's own
remote host. It is a bare hostname with no scheme, userinfo or path.
Removed from the env file and defaulted here the same way the Python
already does. GITEA_USERNAME and GITEA_PASSWORD keep their :? guard --
those are real credentials and must still fail loudly if unset.
The orchestrator host is a general agent box that also serves several unrelated
projects (atproto-likes, the lichen and project orchestrators, the shared
opencode web server). Everything that exists FOR cc-ci — the loops, the cc-ci
orchestrator session, the weekly recipe-upgrade run and its hourly supervisor —
now lives in nix/modules/cc-ci.nix and is exported as `nixosModules.cc-ci`, so
the two can evolve and be reviewed independently and another host can consume
just the cc-ci half.
No behaviour change. Verified by building .#cc-ci-orchestrator-hetzner after the
split: the unit set is identical to the running system, and all six cc-ci units
are byte-identical modulo nix store hashes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016L6nYYwkCWnrEFKTnKAfet
Move plausible from weekly to external in used-recipes.md. cc-ci still
deploys/tests it (the recipe + tests stay enrolled, !testme bridge
unchanged), but it is no longer surveyed/upgraded by the weekly /upgrade-all
run — maintained elsewhere, like uptime-kuma.
The open plausible PR #5 (clickhouse 23.4.2.11 -> 23.4.6.25) from this week's
run stays open for the operator; no future weekly PRs will be opened for it.
The weekly /upgrade-all parent session and the /recipe-report session now
run on tinfoil/deepseek-v4-pro (LOOP_MODEL + REPORT_MODEL in upgrader.env).
The hourly supervisor stays on opencode-go/glm-5.2 (SUPERVISOR_MODEL default
in launch-supervisor.py, not overridden). Subagents already bind deepseek via
the cc-ci repo's opencode config (fix from 2026-08-10, verified this week:
all 16 subagents across both waves ran deepseek-v4-pro).
LOOP_TIER=zen is kept so the tier check passes; the watchdog's usage-limit
probe sends the deepseek model name to the zen endpoint, which returns 200
(not 429) → resume immediately — correct, since tinfoil has no rolling usage
limit to wait out. Verified the probe behaviour with a direct curl.
Root cause: the 2026-08-14 run stalled mid-recipe on 'Insufficient balance'
(opencode zen workspace balance exhausted), then sat unfinished for 40h while
the supervisor cron spun hourly unable to recover it. Deepseek (pay-per-use
API key) has no rolling balance limit, so this can't recur.
Also documents the session recovery in JOURNAL.md (the stalled run was
completed via a fresh scoped upgrader — the original 2.58M-token session was
unresumable: the inference endpoint silently drops the oversized request).
The skill said to render a sourceless recipe as '?'. It now says how to stop it
being sourceless: declare an NVD CPE in the registry. That is what took the fleet
from two blind recipes to zero, and it is the first thing to try when the sweep
flags another.
Two recipes could not see CVEs at all. mattermost-lts has an empty GitHub advisory
feed and renders its security bulletins client-side, so a text sweep finds nothing;
mumble publishes nothing anywhere the registry points. Both returned '?' - nothing
measured - which is honest but useless.
NVD is CPE-indexed and carries structured version ranges, so it answers where the
vendor does not. Declared per recipe as 'nvd-cpe: <image> = <cpe:2.3:...>'.
mattermost-lts 10.5.0 -> 10.12.4 165 CVEs
mattermost-lts 10.11.22 -> 10.12.4 0 CVEs (measured, not unknown)
mumble 1.3.0 -> 1.6.870 2 CVEs
Both NVD range forms are used: versionEndExcluding is a patched version;
versionEndIncluding means the fix version is unpublished but the upgrade delivers
it whenever it crosses X.
That 0 for the actual mattermost upgrade is the interesting one, and it needed a
new rule to be correct: a fix on the line you upgrade FROM was already yours.
mattermost patches every maintained line at once, so 10.11.22 -> 10.12.4 crosses
10.12.1 while 10.11.22 already had the 10.11.4 backport. Without the rule the scan
claimed 12 CVEs the upgrade did not deliver.
The rule is skipped for placeholders: '7.4.X' parses to a bare 7.4 and would read
as 'already fixed at 7.4', which silently dropped redis CVE-2024-46981 and took
discourse 140 -> 139 before I caught it.
79 tests. discourse 140 / gitea 2 / mailu 2 / keycloak 12 / plausible 6 unchanged.
Fleet sweep: 0 recipes with no usable CVE source, down from 2.
Follow-up to the nginx blind spot. Sweeping all 22 recipes for sources whose CVEs
are USABLE (structured advisory feed, or a changelog attributable to releases)
rather than merely visible.
Before the changelog-attribution fix: 20 unusable sources. After: 5, and all five
are redundant - the same project also publishes an advisory feed (redis, gitea,
minio, clickhouse), so nothing is actually lost.
One real find, same shape as nginx: ONLYOFFICE/DocumentServer publishes NO GitHub
advisories, and the registry pointed its CHANGELOG.md at the GitHub *blob* page -
636KB of markup in which the release headings do not survive HTML-stripping, so 24
CVEs were visible and NONE attributable. The raw URL attributes all 24. Rather than
fix one registry line, advisory-scan now normalises github.com/../blob/.. to
raw.githubusercontent.com, which fixes every entry present and future.
lasuite-drive bumps documentserver, so this was live.
Genuinely blind after all that: mattermost-lts and mumble - no advisory feed, no
attributable changelog, no CVE data anywhere the registry points. mattermost is the
notable one: its bulletins are client-side rendered, so a regex sweep sees nothing.
Their scans can report 0 while nothing was measured, so /cve-check now renders those
recipes as ? and says why.
The audit output distinguishes a blind RECIPE from an unparseable PAGE, because
conflating them made 5 harmless redundancies look like 5 gaps.
nginx publishes NO GitHub security advisories. Every nginx CVE we can see comes
from nginx.org/en/CHANGES, and the scan scraped ids out of it without attributing
them to a release - so they had no patched version, could never be classified, and
every nginx bump in the fleet reported 0 CVEs. nginx is a sidecar in most recipes,
so this was a fleet-wide blind spot.
Measured on the two PRs that prompted the question:
lasuite-docs#7 nginx 1.31.1 -> 1.31.3 0 -> 6 CVEs
lasuite-drive#6 nginx 1.31.2 -> 1.31.3 0 -> 3 CVEs
matching a hand count of the changelog exactly (three fixed in 1.31.2, three in
1.31.3; the narrower window correctly counts only the latter).
How: when a vendor page is organised by release, each CVE is attributed to the
nearest preceding release heading ('Changes with nginx 1.31.3', '## v1.31.3'),
and that becomes its fixed-in version. The CVE is tied to a window by the image
name appearing in the page URL (window 'nginx' <-> nginx.org/...). A changelog
lists the project's whole history, so only releases the window actually crosses
count - asserted by a test that the 2013 entries stay out.
76 tests. discourse 140 / gitea 2 / mailu 2 / keycloak 12 unchanged.
Typing --from/--to/--image by hand means someone has to remember the recipe also
bumped its redis. That is how sidecar CVEs went uncounted for months. Point this
at a PR's compose.yml and it reads the windows off the diff instead.
advisory-scan.py plausible --compose-to <.../branch/<pr>/compose.yml>
-> community-edition: v2.0.0 -> v3.2.1
-> clickhouse-server: 23.4.2.11-alpine -> 24.12-alpine
-> 6 CVEs, identical to the hand-specified args
Details that mattered:
- keyed by SERVICE, not image repo. plausible moved plausible/analytics ->
ghcr.io/plausible/community-edition; keyed by repo that reads as one image
vanishing and an unrelated one appearing, and the app window - the one carrying
the critical - is lost entirely.
- the baseline is the repo's DEFAULT BRANCH resolved from the API, never assumed
to be main, because several recipes keep a stale main beside a live master.
- image names are matched against advisory sources BOTH ways: an image name is
often longer than its source repo (clickhouse/clickhouse-server vs
ClickHouse/ClickHouse) and sometimes shorter (redis vs redis/redis). One
direction silently dropped the clickhouse window.
- credentials go in an Authorization header, never the URL: in-URL creds leak
into shell history and process lists, and urllib mis-parses a password
containing a colon.
--from/--to/--image remain for finer-grained checks (scanning a window that is
not a literal compose diff). 71 tests; discourse 140 / gitea 2 / mailu 2
unchanged.
25 open recipe PRs had accumulated, and the list had stopped being readable: CI
sweep artifacts that were never meant to merge sat next to genuine CVE fixes, and
three PRs the operator had been told were outstanding were in fact already merged
upstream (discourse #6 with 140 CVEs, keycloak #6 with 12, n8n #5) — visible only
once the mirrors were reconciled.
The skill: reconcile every mirror from true upstream FIRST (that step alone closed
those three), survey every open PR deterministically, close the ones that cannot
merge or were never meant to, and report prioritised action items — CVE-carrying
first — for the ones that should land. It never merges a recipe PR.
pr-survey.py gathers the facts and decides nothing: behind_main, mergeable,
diff_files, which images the PR adds vs which are already pinned in main, the
newest !testme verdict, branch kind, age.
One correctness detail worth the extra state: a FAILED diff fetch is reported as
unknown, never as an empty diff. gitea #4 reads that way (force-pushed branch)
while being a verified green fix, and 'empty diff' is a close signal — so the
tool says DIFF-UNREADABLE(do not close on this) instead.
Operator policy (2026-08-11). For cc-ci-orchestrator and cc-ci: branch, open a PR
whose description is written to be read AFTER the fact, merge it yourself once
verified, and let the operator review retrospectively. The PR is not a gate — it
is how the work stays legible — so a description that says 'fix scanner' has
failed at its only job.
Explicitly does NOT extend to recipe repos: those are created and verified but
never agent-merged, because they change what deploys on other people's
infrastructure.
Also records what to do when work has already landed on main without a PR: pin a
branch at the pre-work commit and PR against that, rather than rewriting
published history.
I recorded that recreating the runs directory with a fresh inode preceded
recovery. It recurred afterwards (build 1252), so that was not the fix.
The real signal is that it is INTERMITTENT and tracks concurrent activity: every
failure landed while a second run or manual deploy was in flight, and every build
on a quiet host passed (1243, 1250, 1251, 1253). Free space never moves during a
failing build. Practical guidance is therefore to wait for the host to go quiet
and re-trigger before calling it a recipe failure, and DRONE_RUNNER_CAPACITY=2 is
the obvious knob to test if it becomes disruptive.
Root cause still not established, and the note now says so plainly rather than
presenting a coincidence as understood.
Working against a stale mirror has cost us three different ways:
- mailu #6 was linked as the fix for two internet-facing Roundcube CVEs while
upstream had already merged AND released it (3.1.3+2024.06.57). The work was
done; only our mirror was behind. Reconciling closed the PR automatically.
- a stale mirror makes a survey report 'no upgrades available', so the recipe
silently drops out of the weekly run.
- reading the wrong branch: several coopcloud recipes keep a stale 'main' beside
the real default 'master'. gitea's main is 1.24.2-rootless while master has
1.27.1-rootless and the merged PRs, so reading main manufactures a false
'three releases behind, missing two CVSS-9.8 RCEs' finding.
The reconcile logic already existed inside open-recipe-pr.sh --reconcile-only and
already resolves the default branch itself. What was missing was a single obvious
entry point and a rule saying to run it. reconcile-upstream.sh takes recipes or
--all, and is idempotent — recipe work lives in branches, never on mirror main, so
force-syncing main discards nothing.
/ci-test-review and /cc-ci-tests-update had NO reconcile step at all; both now
require it. /cve-check, /recipe-upgrade and /upgrade-all already reconciled and now
point at the shared script.
Builds 1244-1249 died on mkdir of the run dir with 110GB free and 16% inodes.
Ruled out: actual disk (df sampled every 2s across a failing build never moved),
inodes, quotas, a poisoned parent directory (61/61 stress creations succeeded),
runner sandboxing (namespaces identical to the host), and a wedged runner
(restart changed nothing). The same harness with the same numeric run id, run by
hand outside drone, worked every time.
Recreating the runs directory with a fresh inode preceded recovery; builds have
run normally since. The root cause is NOT established, so the note says so rather
than presenting a fix that might be coincidence.
The CI server filled up and every !testme from build 1236 to 1242 died at harness
startup with ENOSPC on /var/lib/cc-ci-runs/<build>. Because the harness never got
far enough to write results.json, the PR badges just said 'failure' — so it read
as recipe regressions, and plausible's genuinely-fixed suite looked still-broken.
Cause: every run pulls each recipe's images and nothing ever removed the old ones.
72GB of images, 63GB of it unused. Reclaimed 69.8GB; the host went 73% -> 22%.
Two changes so it does not recur:
- sweep-orphans.sh (runs at the start AND end of every /upgrade-all) now prunes
unused images when the disk is >=60% (DISK_PRUNE_PCT). Below that it keeps the
layer cache so runs stay fast. 'docker image prune -a' spares anything a container
references, so infra and warm-* canonicals are safe. Volumes are still NOT
pruned — warm-* canonical volumes are data-warm and legitimately dangling.
- /cc-ci-status flags server disk at >65% rather than >80%, because this is not a
steady-state measure: the host was at 73% when runs started failing. It also now
checks that recent builds actually produced results.json — an empty run dir is
the fingerprint of a host problem masquerading as a recipe failure — and records
how to read a drone step log out of its sqlite when the API token is unreachable.
A recipe tracks an image repo and a set of registry URLs. When upstream moves,
nothing errors — the old repo just stops receiving tags and the recipe looks
'up to date' forever. plausible is the case: it tracked plausible/analytics on
Docker Hub while upstream moved to ghcr.io/plausible/community-edition. Every
survey said 'no upgrades available' while v3 shipped elsewhere.
audit-sources.py reports the signals that catch it, per image and per registry
URL: image gone quiet (newest tag older than --quiet-days), deprecation wording
in the registry description, and GitHub repos that are archived, renamed or
gone. Signals, not verdicts — a stable image can be quiet for good reason — so
each finding says what was measured.
First run over 22 recipes, 11 findings, 4 alerts. It independently re-derived
the plausible case (analytics quiet 1126 days), and found:
- drone: harness/drone now answers as harness/harness (the image is fine)
- lasuite-docs, lasuite-drive: minio/minio is ARCHIVED on GitHub
- lasuite-docs: docspecio/api is ARCHIVED
- matrix-synapse: halfshot/matrix-appservice-discord image quiet 2078 days
- mumble: NO cc-ci-plan/upstream/mumble.md at all
That last one exposed a scanner bug. With no registry file there is no source to
query, yet the scan still printed '0 identified by the deterministic scan' — and
that 0 was published as a clean count in the 2026-08-11 CVE check. A scan with no
usable source has measured nothing and must not report a number, least of all 0.
It now returns UNKNOWN and says the registry file is missing.
upstream/mumble.md added; mumble now scans 6 sources for a genuine 0.
/recipe-upgrade --with-tests, /ci-test-review and /cc-ci-tests-update all author
test changes, and all three had only 'never weaken a test' as guidance. That did
not stop the plausible failure: the fixture INSERTed rows into the app's database,
which was correct for v2 and silently wrong for v3, where a site must belong to a
team. Events were acked 202 and discarded; the recipe sat RED for six weeks.
The rule that would have prevented it — set state up through the app's own
interface, not its database — now lives in tests/STYLE.md in the cc-ci repo, and
each of the three paths is told to read it before editing a test.
immich pins two images with BOTH a tag and a digest, which makes abra FATA and
abandon the WHOLE recipe. It therefore contributed no version data at all and
silently dropped out of every survey — indistinguishable from 'up to date'. The
standing answer was prose in three skills telling an agent to check registries by
hand. This replaces it with a tool.
resolve-images.py reads the compose files and queries registries itself:
- Docker Hub, ghcr, and any OCI registry via its own auth challenge (lscr.io
and dock.mau.dev advertise different realms; assuming ghcr's shape 401'd).
- tag SHAPES (digits -> '#') so -alpine stays on -alpine and 'latest' is never
proposed as an upgrade.
- reports newest_within_major AND newest_same_shape, and refuses to choose:
immich's postgres tag encodes the pg major plus the vectorchord/pgvectors
build immich-server expects, so taking the newest breaks the deploy.
- integrity check: if the CURRENT pin is absent from the listing, the listing
was truncated and any 'newest' is a guess. ghcr caps out past 40k tags, so
that falls back to the project's GitHub releases.
- per-repo cache + backoff + Docker Hub auth: a fleet sweep re-reads nginx,
redis and postgres many times and was getting 429s reported as 'unresolved'.
21/21 recipes now resolve. It found upgrades abra missed entirely in five:
mumble (abra said 'no new versions'; four patches behind), plausible's
clickhouse, lasuite-drive's collabora, gitea's mariadb, immich's postgres.
plausible's carried four CVEs, three high.
Also fixes a real over-count found while validating that: a fix inside the
numeric window is not a fix on the branch you land on. ClickHouse patched
CVE-2023-48704 in 23.9.6.20 AND 23.10.5.20 — landing on 23.10.4.25 crosses the
23.9 fix but sits below its own line's, so it does NOT have it. A fix named on
the target's own line and above the target is now proof of absence.
70 tests (64 offline + 6 live). keycloak's live expectation moves 7 -> 12 and
mailu's 0 -> 2: both are the release-note source finding real fixes that were
never filed as advisories.
Skipping the reconcile to keep the sweep 'read-only' was wrong. It researches a
stale checkout, and on the first real run left two recipes with no survey output
at all — indistinguishable from 'no upgrades available' unless you look. The
reconcile is safe precisely because recipe work lives in branches, not on main.
Also documents the trap that produced a false finding in that run: several
coopcloud recipes keep a stale 'main' beside the real default 'master'. gitea's
main is at 1.24.2-rootless while master has 1.27.1-rootless and the merged PRs,
so reading main reports a recipe three releases behind and missing two CVSS-9.8
RCE fixes. Resolve default_branch from the API before reading any file.
And: no output is not 'no upgrade'. It is a third outcome, and only becomes '?'
after the direct registry check has also failed.
1. Release-note resolution now covers vendor pages on the same repo. It required
a github-advisories: source, so mailu's Roundcube CVEs — announced only on
github.com/Mailu/Mailu/releases — went to pass 2 even though the answer was
sitting in the release notes. mailu now reports 2 deterministically, matching
what previously took an agent reading the notes.
2. 'All known fix versions predate the version we were on' is now a DECISION,
not an unknown. mailu's redis 8.8.0 -> 8.10.0 crosses 12 advisories all fixed
by 8.6.3 or earlier; reporting them as 'could not judge' overstated the
uncertainty. Recorded as outside-window with the naming tags as evidence.
A fix landing ABOVE the window still stays indeterminate on purpose: that is
an open vulnerability and must stay visible.
60 offline tests (was 58). discourse 140 / gitea 2 unchanged.
/cve-check answers 'what are we exposed to that an upgrade would fix?' without
running an upgrade: per-recipe, resolve the available window for EVERY image
(sidecars included), run the advisory scan over it, adjudicate whatever pass 1
could not decide, publish a report. Read-only — no PRs, no CI, no merges.
/cve-check-and-upgrade does that sweep, then runs /recipe-upgrade only on the
recipes whose upgrade actually closes a CVE, worst severity first, and reports
on both. --min-severity high for just the urgent ones; --dry-run prints the
queue and stops. Never merges.
Deliberate choices, each written into the skills:
- externals are SWEPT but never upgraded here — a security sweep that skipped
deployed software would misreport exposure, but we don't maintain them.
- an unknown count never justifies an upgrade AND is never treated as clean;
it goes to the Addendum.
- no upgrade available means 0 CVEs, not '?'.
- subagents are told which CVEs justify their upgrade, so the PR says why it
exists — a PR naming the RCE it closes gets reviewed sooner.
recipe-report.py grows a page kind: 'cve' files as cve-DATE.html so a sweep
can't overwrite a weekly edition, while BOTH appear in the same archive index,
suffixed 'full report' / 'CVE check'.
/help and /cc-ci-status updated to route to them.
Adds test-advisory-scan.py (58 offline tests on fixtures + 6 live regressions
against the week-2026-08-07 report) and audit-advisory-scan.py, which re-derives
every count with a SEPARATE semver implementation and its own release fetch and
diffs against the scanner. Both found real defects:
1. Window membership was compared on ragged tuples, so (18,) < (18,0) — a CVE
patched in 18.0 fell OUTSIDE a window ending at 18. Bare major tags are the
norm for sidecars (postgres:18, redis:8-alpine). Now zero-padded, which also
keeps the upper bound conservative (18.5 stays out of a window ending at 18).
2. Advisories with no knowable fix version were silently counted as 'not fixed'.
Twelve redis advisories say patched_versions 'TBD' or '7.4.X' with an
open-ended range — six of them high severity. They are now INDETERMINATE:
not counted, not dismissed, and surfaced in the output.
All twelve turned out to be genuinely fixed: redis names each in the release
notes of every branch that got the fix (CVE-2025-32023 -> 6.2.19, 7.2.10,
7.4.5, 8.0.3, 8.2.0). So a third deterministic method resolves them from
release notes, with the naming tags recorded as the citation. discourse's
redis contribution goes 5 -> 17, and its total 128 -> 140.
Pass 2 (--adjudicate) is the model-judged stage for what arithmetic cannot
settle: it hands over each open case's full evidence, plus every verdict pass 1
reached, and takes FIXED/NOT-FIXED/STILL-UNKNOWN with a reason citing that
evidence. It may only raise a count. Vendor-page-only CVEs — the shape of both
gitea CVSS-9.8 RCEs — now reach it instead of being dropped.
Tests cover pass 1 only, by design; pass 2's judgement is a model's. What is
tested there is deterministic: which cases it selects, and that truncation is
announced rather than silent.
SPEC.md rewritten around the two passes.
Restores the single-value form (operator preference) under the --image name.
Repeat the flag per image, all in one call. Malformed values warn on stderr and
are skipped rather than aborting the scan, since it is an additive pre-step.
Counts unchanged: discourse 128 with redis / 123 without, gitea 2.
autonomic-bot
autonomic-bot
