Step-by-step specification of cc-ci-plan/advisory-scan.py: inputs, the three source
classes and why each is ranked where it is, the union, both classification paths
(patched-version ranges, and the advisory-publish-date fallback for version-scheme
changes), the output contract, and how /recipe-report must read it.
Each rule records the production wrong answer that motivated it — the false 133 from
cross-image counting, the n8n misclassification from reading only vulnerabilities[0],
the '?' sprawl from url punctuation and benign-404s, and the 'never emit 0 for an
undetermined count' rule. Claims cross-checked against the implementation.
Keep this file in the same commit as any behaviour change.
Answers 'how can the weekly run produce counts like the hand count?' — by doing
exactly what the hand count did, deterministically. Two changes:
1. PAGINATION. The scanner requested per_page=100 and stopped. This endpoint caps at
100 AND ignores ?page= (it re-returns the same rows — which is how a manual count
first produced exact triplicates and a bogus 300). Busy projects were silently
truncated: discourse has 286 advisories, so a single page could not see the window
at all. Now follows the Link rel=next cursor to exhaustion.
2. DATE-BASED FALLBACK. Version strings cannot be ordered across a scheme change
(discourse semver 3.5.3 -> calver 2026.7.1), which is why the scan first reported a
false 133, then correctly refused. Release DATES always order. When the version path
refuses, the scan now resolves both versions to their git tag dates on the primary
repo and counts advisories PUBLISHED in that window, labelling the method in the
output. The version path is still preferred when usable — it is exact rather than
temporal.
Verified: discourse 3.5.3 -> 2026.7.1 now reports 123, matching the hand count
(1 critical, 16 high, 91 medium, 16 low; window 2025-12-30 -> 2026-07-31); gitea
1.27.0 -> 1.27.1 still reports 2 via the version path.
Operator: 'the scanner should not say 0 when it was not able to scan.' Correct — the
previous patch still led with '0 identified' and relegated the caveat to a footnote,
so the headline number was wrong even though the prose was right. A 0 in a security
column is an assertion of safety; it must never be emitted for an undetermined result.
Now: cve_count_fixed is null (not 0) in JSON, a count_known flag distinguishes
'counted zero' from 'could not count', and the markdown headline reads
'CVEs fixed by this upgrade: UNKNOWN — the scan could NOT determine a count' with an
explicit 'This is NOT zero' and instructions to render '?'.
Verified: discourse 3.5.3 -> 2026.7.1 (semver->calver) now reports UNKNOWN; gitea
1.27.0 -> 1.27.1 still reports 2.
Operator disbelieved discourse's '133 CVEs fixed' — correctly. Two defects made it
confidently wrong:
1. ONE WINDOW APPLIED TO EVERY IMAGE. The scan queries all source repos in the
recipe's registry (app + redis/postgres/nginx sidecars) but judged them all with
the APP's version window. 34 of the 133 were redis advisories, including
CVE-2021-21309 — patched in redis 6.0.11 back in 2021 — scored as 'fixed by this
upgrade' purely because 6.0.11 sits numerically inside discourse's 3.5.3 ->
2026.7.1 range. Only the PRIMARY app repo is now classified; other sources are
reported as unclassified so they stay visible without inflating the count.
2. VERSION-SCHEME CHANGES BREAK ORDERING. discourse moved semver -> calver
(3.5.3 -> 2026.7.1), so 2025.12.2 compares 'newer' than 3.5.3 while shipping
earlier. Numeric comparison cannot order that. The scan now detects a leading-
component jump >= 100, refuses to classify, and says so in the block: the count
is '0 by refusal, not by evidence — read the vendor's release notes'.
Refusing to answer beats answering wrongly: a fabricated 133 in a public security
report is worse than an explicit 'cannot determine'.
Verified after the fix: discourse 133 -> 0 (with the refusal caveat), gitea still
exactly 2 (both criticals, patched 1.27.1), keycloak 7 all genuinely from
keycloak/keycloak patched in 26.7.1, plausible 1. No other count changed.
The 2026-08-07 regeneration rendered '?' for 5 of 21 recipes. '?' is meant to be a
rare 'we tried and could not tell'; at that rate it is indistinguishable from noise
and hides the real unknowns. Three causes, none of them genuine uncertainty:
1. URL EXTRACTION BUG (mine). The registry is markdown, so urls appear inside
`backticks` and 'quotes'. The extractor captured the trailing punctuation, so
it fetched https://docs.n8n.io/release-notes/` and https://git.autonomic.zone'`
— both 404 on the malformed url, both 200 when clean. Trailing markdown
punctuation is now stripped. Fixed immich + n8n.
2. STALE REGISTRY URL. mattermost-lts pointed at
docs.mattermost.com/about/mattermost-changelog.html, which 404s; the page moved
to /deploy/. Corrected (same class as the pgautoupgrade fix).
3. WRONG SEMANTICS FOR 'NO UPGRADE'. lasuite-docs and custom-html-tiny were
up-to-date this run, so no scan block existed and the report fell back to '?'.
But a recipe with no upgrade has nothing an upgrade could have fixed — that is
0, not unknown. The report skill now says so explicitly, restricts '?' to scans
that RAN and reported genuinely failed sources, states that benign notes
(no-advisories-published / template URL) never trigger '?', and instructs that
many '?' is itself a bug to raise in the Addendum.
Result across all 16 scanned recipes of that run: 0 failed sources (was 5).
Counts also improved with the classifier fix: discourse 130->133, keycloak ->7.
Exposed by asking whether the scan catches the n8n CVEs (CVE-2026-42231/42232). It
did not — the advisories were fetched correctly but both misclassified as
out-of-window. Two bugs:
1. Only vulnerabilities[0] was read. An advisory carries ONE ENTRY PER PATCHED
RELEASE LINE: n8n patches three (1.123.32, 2.17.4, 2.18.1), so whichever line
the deployment is actually on was silently dropped. gitea passed only because it
patches a single line. Now all entries are kept.
2. patched_versions is a RANGE EXPRESSION ('>= 2.18.1'), not a bare version. Naive
parsing produced (18,1) instead of (2,18,1), so no comparison could ever match.
Version tokens are now extracted with a regex and the advisory counts as
fixed-by-this-upgrade if ANY patched line falls in (from, to].
Verified: n8n 2.17.0 -> 2.18.1 now reports 12 CVEs including both criticals
(CVE-2026-42231 GHSA-q5f4-99jv-pgg5, CVE-2026-42232); gitea 1.27.0 -> 1.27.1 still
reports exactly 2. Note our deployed n8n (2.27.2+) is already past all three patched
lines, so these were never outstanding for us — the bug was in detection, not
exposure.
Two refinements found by running the scan across all 14 recipes of the 2026-08-07 run:
1. A repo with no advisory feed returns HTTP 404 on /security-advisories (e.g. the
pgautoupgrade sidecar image). That is a BENIGN ABSENCE, not a failed check.
Likewise registry entries that are TEMPLATE urls for humans
(…/changelog/v<VERSION>/, …/<vX.Y.Z>/…) are documentation, not fetchable.
Counting either as a failure pushed most recipes to '?', which would make the
unknown-vs-clean distinction meaningless again — the exact signal the ? exists to
preserve. Both are now recorded in sources_benign; only genuine errors (rate
limit, network, 5xx, wrong URL) land in sources_failed.
2. upstream/*.md pointed at github.com/pgautoupgrade/pgautoupgrade, which 404s —
the repo is pgautoupgrade/docker-pgautoupgrade. Corrected in n8n, lasuite-docs,
lasuite-drive, lasuite-meet. A 404ing registry URL means we were not scanning a
source we believed we were.
Effect on the 2026-08-07 data: recipes with genuine failed sources 5 -> 3 (the
remainder are really unreachable vendor pages). CVE counts unchanged where they
were already sound: discourse 130, gitea 2, plausible 1.
Anonymous GitHub API is 60 req/hr — a full weekly sweep across ~20 recipes exhausts
it and the scan then reports sources as failed (visible, but degraded coverage). A
token lifts it to 5000/hr.
_github_token(): GITHUB_TOKEN env wins, else GITHUB_TOKEN_FILE (default
/srv/cc-ci/.github-token, 0600, gitignored). Reading PUBLIC advisories needs NO
scopes — a classic PAT with nothing ticked, or fine-grained limited to 'Public
repositories: read'. The tool only ever GETs advisories; do not grant write scopes.
A missing token is not an error: the scan runs anonymously and surfaces failures.
Also gitignores .github-token and .hcloud-token.
Why: gitea 1.27.1 fixed CVE-2026-60004 + CVE-2026-59774 (both CVSS 9.8). The
2026-08-03 report printed gitea's CVE count as '1', the 2026-08-07 report as
'none'. Cause chain: the upgrade subagent read the GitHub release notes, which
name NEITHER cve (they are announced only in the vendor blog's security section),
so it recorded one unrelated minor item; the report then derived security content
from those notes plus model knowledge, and the model's training predates the CVEs.
Nothing in the pipeline ever queried an advisory source.
cc-ci-plan/advisory-scan.py — deterministic, per recipe, per upgrade window:
1. GitHub Security Advisories API for every source repo in the upstream registry.
PRIMARY: CVE + GHSA + severity + vulnerable/patched ranges, so 'fixed by THIS
upgrade' is computed. Needs no new per-recipe config (134 registry URLs are
already github.com).
2. Vendor release/security pages — every registry URL, fetched + regex-scanned.
This is the source that actually had the gitea CVEs.
3. OSV where a package mapping exists — supplementary.
Each source reports its own status so 'checked, none found' is never confused with
'not checked'. Source selection was measured, not assumed: for these two CVEs OSV
404'd and NVD's API had them by neither CPE, id, nor keyword — advisory DBs lag the
vendor, hence 1+2 lead.
Wiring is strictly ADDITIVE:
- /recipe-upgrade gains step 2a: run the scan, paste the block into the per-recipe
log, and report the UNION of it and the existing release-note reading. The scan
may never lower a count established by reading.
- /recipe-report treats the block as a FURTHER source, prefers its advisory ids /
severities / fixed-in versions for citation, and must render '?' (not 'none')
when a scan is absent or has failed sources — the false-clean 'none' is exactly
what happened on 2026-08-07.
- upstream/gitea.md records blog.gitea.com as the security-announcement URL.
Verified on the real regression: 1.27.0 -> 1.27.1 now yields exactly the 2 missed
criticals with their GHSA ids; the wider 1.26.2 -> 1.27.1 window yields 62.
Regression from the 2026-08-04 session-pinning work: lu._session_id() prefers the
pin file, but launch-report.py never cleared or re-established it. A surviving pin
from a PREVIOUS report run points at a session whose last message already carries
RECIPE REPORT COMPLETE, so the shared watchdog evaluates _completed()=True and
exits within one poll ('run completed — exiting'), leaving the CURRENT run
unwatched. Observed live: the 2026-08-07 finish-run's report step was watched by a
watchdog that quit after 3 minutes against an Aug-4 pin, then the report session
ended early with nothing to resume it.
start() now archives stale titles, clears the pin, snapshots existing ids, and
re-pins the new session after launch — the same contract launch-upgrader.start()
already follows. Scopes the shared helpers via UPGRADER_SESSION=<report session>.
Root cause chain, all confirmed on the live system:
1. _run_pids() substring-matched the WHOLE cmdline for the session name. An agent's
kickoff PROMPT is an argv element, and the supervisor's prompt text contains the
literal 'cc-ci-upgrader' — so the supervisor's OWN billing-hung agent matched as a
live upgrader run. Now matches FLAG VALUES only (--title <SESSION> / -s <sid>).
2. The gate treated 'a live proc exists' as progress. A provider-walled run keeps its
process alive and SPINNING while emitting nothing (verified: 3 days, zero session
output, still burning CPU). Progress now REQUIRES the session tree to have advanced
within STALL_MIN; a live-but-idle proc is explicitly logged as stalled.
Result was ~60 consecutive false 'run progressing — leaving it' no-ops while the weekly
run sat unfinished and unreported.
Billing-walled runs are REPORTED, never killed (operator policy 2026-08-10): they may
resume when the wall lifts and their context is the run's state. New _billing_blocked()
detects the wall from the log tail and the gate surfaces
'run BLOCKED on a provider billing/usage wall — NOT killing; operator action required'.
Verified live: _run_pids no longer matches the hung Aug-7 supervisor (1497561) while
still matching the real finisher; gate now reasons 'session advanced Nm ago'.
- _archive_stale_titles default label is now 'archive-<title> —' (operator
convention: all archived sessions start with archive- so they sort/filter
together in the web UI).
- launch-report.py start() archives older cc-ci-report sessions before launch,
same invariant as upgrader/supervisor (reuses the launch-upgrader helper).
- 33 sessions restyled/archived live; canonical names now unique:
cc-ci-upgrader (idle finisher), cc-ci-report (generating), cc-ci-supervisor
(none — only exists during a rescue).
_archive_stale_titles() generalized to (title, label); launch-supervisor's
spawn_supervisor() archives every older 'cc-ci-supervisor' session before
launching, so exactly one session carries the name. 11 historical supervisor
sessions archive-renamed live ('supervisor archive — <date> <time>').
Bug (2026-08-04 16:00): _session_id() sorted candidates on (s.time.created) which the
/session API rows DON'T carry — every key was 0, 'newest' degraded to server list
order, and the watchdog resumed the old giant unresumable session, kill_session()ing
the healthy fresh run mid-work.
Fixes:
- Pin the managed session id at launch/resume to LOG_DIR/.{SESSION}-session-id;
_session_id() prefers the pin, validated via direct GET /session/<id> (the LIST is
paginated ~100 rows, membership scans lie). Title lookup is only the fallback and
now sorts on authoritative sqlite time_created.
- _archive_stale_titles() at start: every older top-level session titled
cc-ci-upgrader is renamed 'upgrader archive — weekly <date>', so EXACTLY ONE
session ever carries the canonical name in the opencode web UI (easy to find;
finished runs stay browsable under archive names). 11 historical sessions
renamed live today; the in-flight finisher pinned.
Verified live: _session_id() returns the pinned running session; tree-idle 0.0min
while subagents active. Full synthetic-stall watchdog confirmation queued post-run
(task #13).
Covers the 2.32.4 -> 2.33.3 range for the 2026-08-03 /upgrade-all run:
2.33.0 minor features (admin-managed instance creds, workflow review
requests + publish/unpublish API, API deprecation of activate/deactivate
endpoints, optional N8N_SCHEDULER_MAX_ATTEMPTS env) and the 2.33.1-2.33.3
patches; notes 2.34.0 exists but is not this run's target.
The repo already vendored ARM as the references/recipe-maintainer submodule (old
repo name recipe-maintainers/recipe-maintainer, pinned 460eba0). Rather than carry
two copies, drop the just-added vendor/ duplicate and:
- retarget references/recipe-maintainer to
ssh://git.autonomic.zone/recipe-maintainers/autonomic-recipe-maintainer.git
(same lineage — 460eba0 is an ancestor) and bump to latest acd5cfb, which also
refreshes the parity-test SOURCE reference the tests cite.
- gen-cctest-skills.py + all 30 cctest-* wrappers + /help now point at
references/recipe-maintainer.
- JOURNAL.md: pending session entries (server-update policy addendum, tests-update,
orchestrator-update, upgrade-run notes).
- cc-ci-server-update / cc-ci-tests-update / cc-ci-update: operator policy change —
open PRs for visibility/historical record and merge directly once verified
(invocation = authorization); reports must list merged-PR links + change summaries.
(tests-update still never merges the paired recipe upgrade PRs.)
- cc-ci-server-update: mandatory nixos-rebuild test step (5d) before switch, with
detached-activation + transient-unit notes from the 2026-08-03 26.05 deploy.
- AGENTS.md: test-before-switch policy for orchestrator host rebuilds.
- NEW skill cc-ci-orchestrator-update (+ thin .claude wrapper): the /cc-ci-server-update
analogue for this host (flake /srv/cc-ci-orch, .#cc-ci-orchestrator-hetzner, Hetzner
server 134487234), with self-update caveats.
- JOURNAL.md: 2026-08-03 server-update incident/recovery handoff entry.
Switch the cc-ci-orchestrator agent (cc-ci-plan/agents.toml) from the claude
backend to the opencode backend so it attaches to the shared opencode web
server (opencode-web.service, 127.0.0.1:4096) and is reachable for Remote
Control at https://oc.commoninternet.net under the /srv/cc-ci-orch project.
Model = opencode/glm-5.2 (same provider family as the weekly upgrader /
supervisor). resume=false — the opencode backend has no --resume equivalent
(a fresh session per launch, matching the existing launch-opencode.sh shape).
Add a cc-ci-orchestrator.service systemd unit so on boot it runs
'agents.py up orchestrator' after opencode-web.service is up — the same
reboot-resilience shape lichen-orchestrator / project-orchestrator already
have. KillMode=process so a rebuild that merely touches this unit does not
tear down the shared tmux server and every agent session with it.
No secrets: the diff is only config + comments (paths, model names, unit
definition).
GO subscription hit a monthly usage limit (resets in 9 days). Add LOOP_TIER
env (default 'zen') to launch-upgrader.py + launch-report.py so the opencode
backend can use either subscription: zen→opencode/glm-5.2, go→opencode-go/glm-5.2.
The tier selects the default model, the usage-limit probe endpoint, and the
auth.json key. The systemd timer reads LOOP_TIER from /srv/cc-ci/upgrader.env.
A run can finish the recipe work + summary + report yet never print a clean
sign-off marker (it may wedge afterwards, as on 2026-07-03). Without a second
completion signal the supervisor would try to 'resume' the already-finished run
every hour until the 96h window closed. Now the gate also treats the run as done
when a weekly summary file (upgrade-all-<date>.md, the Step-5 output) exists with
an mtime after the session began.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
The 2026-07-03 finish run wedged because _completed() returned True while the run
was still mid-work — so the watchdog exited early and nothing recovered the wedge.
Cause: it scanned part.get('text') across ALL message parts, so DONE_MARKER inside
a TOOL part (a subagent `task` prompt / bash command that referenced 'print
UPGRADE RUN COMPLETE') matched. Now: require the marker in the LAST assistant
TEXT (prose) message — the genuine sign-off — ignoring tool-call args and any
mid-run echo of the instruction (work after the echo disqualifies it).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
More fixes from live-running the finish of the 2026-07-03 run:
- _session_idle_min(): measure staleness across the whole session TREE (top-level
run + all descendant subagents), via the opencode server time.updated, NOT the
tmux log mtime (which freezes when a headless run doesn't stream to the pane).
A per-recipe subagent deploy runs 20-40min during which the PARENT session's
updated time is stale — reading the parent alone looked 'idle' and would false-
resume, killing the productive run. Renamed from _log_idle_min (kept as alias).
- watchdog(): wrap each poll in try/except so a transient server blip/race can
never kill the watchdog (a dead watchdog silently abandons the run).
- watchdog + supervisor now read the tree-aware session idle instead of log mtime.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
autonomic-bot
cc-ci-bot
cc-ci
cc-ci-bot
