Two refinements found by running the scan across all 14 recipes of the 2026-08-07 run:
1. A repo with no advisory feed returns HTTP 404 on /security-advisories (e.g. the
pgautoupgrade sidecar image). That is a BENIGN ABSENCE, not a failed check.
Likewise registry entries that are TEMPLATE urls for humans
(…/changelog/v<VERSION>/, …/<vX.Y.Z>/…) are documentation, not fetchable.
Counting either as a failure pushed most recipes to '?', which would make the
unknown-vs-clean distinction meaningless again — the exact signal the ? exists to
preserve. Both are now recorded in sources_benign; only genuine errors (rate
limit, network, 5xx, wrong URL) land in sources_failed.
2. upstream/*.md pointed at github.com/pgautoupgrade/pgautoupgrade, which 404s —
the repo is pgautoupgrade/docker-pgautoupgrade. Corrected in n8n, lasuite-docs,
lasuite-drive, lasuite-meet. A 404ing registry URL means we were not scanning a
source we believed we were.
Effect on the 2026-08-07 data: recipes with genuine failed sources 5 -> 3 (the
remainder are really unreachable vendor pages). CVE counts unchanged where they
were already sound: discourse 130, gitea 2, plausible 1.
Anonymous GitHub API is 60 req/hr — a full weekly sweep across ~20 recipes exhausts
it and the scan then reports sources as failed (visible, but degraded coverage). A
token lifts it to 5000/hr.
_github_token(): GITHUB_TOKEN env wins, else GITHUB_TOKEN_FILE (default
/srv/cc-ci/.github-token, 0600, gitignored). Reading PUBLIC advisories needs NO
scopes — a classic PAT with nothing ticked, or fine-grained limited to 'Public
repositories: read'. The tool only ever GETs advisories; do not grant write scopes.
A missing token is not an error: the scan runs anonymously and surfaces failures.
Also gitignores .github-token and .hcloud-token.
Why: gitea 1.27.1 fixed CVE-2026-60004 + CVE-2026-59774 (both CVSS 9.8). The
2026-08-03 report printed gitea's CVE count as '1', the 2026-08-07 report as
'none'. Cause chain: the upgrade subagent read the GitHub release notes, which
name NEITHER cve (they are announced only in the vendor blog's security section),
so it recorded one unrelated minor item; the report then derived security content
from those notes plus model knowledge, and the model's training predates the CVEs.
Nothing in the pipeline ever queried an advisory source.
cc-ci-plan/advisory-scan.py — deterministic, per recipe, per upgrade window:
1. GitHub Security Advisories API for every source repo in the upstream registry.
PRIMARY: CVE + GHSA + severity + vulnerable/patched ranges, so 'fixed by THIS
upgrade' is computed. Needs no new per-recipe config (134 registry URLs are
already github.com).
2. Vendor release/security pages — every registry URL, fetched + regex-scanned.
This is the source that actually had the gitea CVEs.
3. OSV where a package mapping exists — supplementary.
Each source reports its own status so 'checked, none found' is never confused with
'not checked'. Source selection was measured, not assumed: for these two CVEs OSV
404'd and NVD's API had them by neither CPE, id, nor keyword — advisory DBs lag the
vendor, hence 1+2 lead.
Wiring is strictly ADDITIVE:
- /recipe-upgrade gains step 2a: run the scan, paste the block into the per-recipe
log, and report the UNION of it and the existing release-note reading. The scan
may never lower a count established by reading.
- /recipe-report treats the block as a FURTHER source, prefers its advisory ids /
severities / fixed-in versions for citation, and must render '?' (not 'none')
when a scan is absent or has failed sources — the false-clean 'none' is exactly
what happened on 2026-08-07.
- upstream/gitea.md records blog.gitea.com as the security-announcement URL.
Verified on the real regression: 1.27.0 -> 1.27.1 now yields exactly the 2 missed
criticals with their GHSA ids; the wider 1.26.2 -> 1.27.1 window yields 62.
The extend path grafts HEAD^{tree} WHOLESALE onto the existing upgrade-* branch
(commit-tree -p <branch tip>). Reconcile force-syncs the MIRROR's main to upstream
but never brought the branch — or the local checkout — forward, so each week the
PR base drifted further back and upstream changes made since the branch was cut
were silently absent from the pushed tree. CI then verified a tree that would
never deploy.
Found on gitea PR #5 (2026-08-10): base 0ab323d predated upstream's
'BREAKING CHANGE: remove forgejo' (37ebd22), so the 1.27.1 bump fixing
CVE-2026-60004 + CVE-2026-59774 was !testme-GREEN against a forgejo-bearing tree.
Two changes:
1. Before pushing, if the local work does not contain the freshly-synced upstream
main, merge upstream in — and FAIL LOUDLY (exit 1, naming the checkout) if that
cannot auto-merge, rather than pushing a tree that omits upstream changes.
2. The extend commit now also parents on upstream main when the branch predated it,
so the recorded history matches the pushed tree. Without it the merge-base stays
stale and a later merge can REVERT upstream's changes. Still no force-push.
Verified against the real gitea drift: detection fires, merge is clean, resulting
tree keeps forgejo removed AND the 1.27.1 pin, history contains upstream.
It lived in the orchestrator repo (the parent session's project) and so never
governed the task-tool subagents, which resolve their parent session's directory —
for launcher-started runs that is /srv/cc-ci-orch/cc-ci. The real config now lives
there (cc-ci repo, operator-approved). Extension was never the problem: .jsonc
parsed and resolved fine; the LOCATION was wrong.
Regression from the 2026-08-04 session-pinning work: lu._session_id() prefers the
pin file, but launch-report.py never cleared or re-established it. A surviving pin
from a PREVIOUS report run points at a session whose last message already carries
RECIPE REPORT COMPLETE, so the shared watchdog evaluates _completed()=True and
exits within one poll ('run completed — exiting'), leaving the CURRENT run
unwatched. Observed live: the 2026-08-07 finish-run's report step was watched by a
watchdog that quit after 3 minutes against an Aug-4 pin, then the report session
ended early with nothing to resume it.
start() now archives stale titles, clears the pin, snapshots existing ids, and
re-pins the new session after launch — the same contract launch-upgrader.start()
already follows. Scopes the shared helpers via UPGRADER_SESSION=<report session>.
Root cause chain, all confirmed on the live system:
1. _run_pids() substring-matched the WHOLE cmdline for the session name. An agent's
kickoff PROMPT is an argv element, and the supervisor's prompt text contains the
literal 'cc-ci-upgrader' — so the supervisor's OWN billing-hung agent matched as a
live upgrader run. Now matches FLAG VALUES only (--title <SESSION> / -s <sid>).
2. The gate treated 'a live proc exists' as progress. A provider-walled run keeps its
process alive and SPINNING while emitting nothing (verified: 3 days, zero session
output, still burning CPU). Progress now REQUIRES the session tree to have advanced
within STALL_MIN; a live-but-idle proc is explicitly logged as stalled.
Result was ~60 consecutive false 'run progressing — leaving it' no-ops while the weekly
run sat unfinished and unreported.
Billing-walled runs are REPORTED, never killed (operator policy 2026-08-10): they may
resume when the wall lifts and their context is the run's state. New _billing_blocked()
detects the wall from the log tail and the gate surfaces
'run BLOCKED on a provider billing/usage wall — NOT killing; operator action required'.
Verified live: _run_pids no longer matches the hung Aug-7 supervisor (1497561) while
still matching the real finisher; gate now reasons 'session advanced Nm ago'.
- _archive_stale_titles default label is now 'archive-<title> —' (operator
convention: all archived sessions start with archive- so they sort/filter
together in the web UI).
- launch-report.py start() archives older cc-ci-report sessions before launch,
same invariant as upgrader/supervisor (reuses the launch-upgrader helper).
- 33 sessions restyled/archived live; canonical names now unique:
cc-ci-upgrader (idle finisher), cc-ci-report (generating), cc-ci-supervisor
(none — only exists during a rescue).
_archive_stale_titles() generalized to (title, label); launch-supervisor's
spawn_supervisor() archives every older 'cc-ci-supervisor' session before
launching, so exactly one session carries the name. 11 historical supervisor
sessions archive-renamed live ('supervisor archive — <date> <time>').
Bug (2026-08-04 16:00): _session_id() sorted candidates on (s.time.created) which the
/session API rows DON'T carry — every key was 0, 'newest' degraded to server list
order, and the watchdog resumed the old giant unresumable session, kill_session()ing
the healthy fresh run mid-work.
Fixes:
- Pin the managed session id at launch/resume to LOG_DIR/.{SESSION}-session-id;
_session_id() prefers the pin, validated via direct GET /session/<id> (the LIST is
paginated ~100 rows, membership scans lie). Title lookup is only the fallback and
now sorts on authoritative sqlite time_created.
- _archive_stale_titles() at start: every older top-level session titled
cc-ci-upgrader is renamed 'upgrader archive — weekly <date>', so EXACTLY ONE
session ever carries the canonical name in the opencode web UI (easy to find;
finished runs stay browsable under archive names). 11 historical sessions
renamed live today; the in-flight finisher pinned.
Verified live: _session_id() returns the pinned running session; tree-idle 0.0min
while subagents active. Full synthetic-stall watchdog confirmation queued post-run
(task #13).
Covers the 2.32.4 -> 2.33.3 range for the 2026-08-03 /upgrade-all run:
2.33.0 minor features (admin-managed instance creds, workflow review
requests + publish/unpublish API, API deprecation of activate/deactivate
endpoints, optional N8N_SCHEDULER_MAX_ATTEMPTS env) and the 2.33.1-2.33.3
patches; notes 2.34.0 exists but is not this run's target.
Operator decision: main driving sessions stay glm-5.2; every task-tool subagent
runs opencode/deepseek-v4-pro (~3-5x cheaper, near-free cache hits) — the weekly
upgrade run burns most of its budget in subagents. Report generator stays glm-5.2
(launch-report.py default). Project-scoped config.
The 2026-08-03 cc-ci outage was recovered with ad-hoc tooling living in /tmp
(leftover from a PREVIOUS incident, half-evaporated). Promoted to the repo:
- scripts/recovery/hetzner.py — Hetzner API helper (status/actions/reboot/reset/
power/rescue-on|off/console), knows cc-ci=134485294 + orchestrator=134487234 by
name; token from HCLOUD_TOKEN or /srv/cc-ci/.hcloud-token (0600, never in git).
- scripts/recovery/hetzner-console.sh — shell-only VGA console: fresh console
session -> websocat bridge -> vncdotool (venv auto-bootstrapped in ~/.cache).
screenshot / key / type subcommands; encodes the reset-invalidates-session and
single-connection-bridge gotchas.
- scripts/recovery/README.md — the condensed 10-minute unreachable-server drill,
incl. the GRUB submenu 1>N ids + clear-grubenv-after-switch rule.
- hetzner-server-recovery skill: console/API sections now point at the repo tools
instead of describing /tmp rebuilds.
Smoke-tested: hetzner.py cc-ci status OK.
Operator decision: no policy difference between cc-ci and recipe-maintainer. On
inspection ARM already agrees (recipe-upgrade-cron-all: 'PRs are reviewed and merged
manually by a human afterwards... never merges anything'; 'no human review in the
middle' = skip the mid-run plan confirmation only). Wrappers previously framed this
as a cc-ci override over ARM auto-merge flows — wrong reading; now stated as ONE
unified rule. /help conventions updated to match.
The repo already vendored ARM as the references/recipe-maintainer submodule (old
repo name recipe-maintainers/recipe-maintainer, pinned 460eba0). Rather than carry
two copies, drop the just-added vendor/ duplicate and:
- retarget references/recipe-maintainer to
ssh://git.autonomic.zone/recipe-maintainers/autonomic-recipe-maintainer.git
(same lineage — 460eba0 is an ancestor) and bump to latest acd5cfb, which also
refreshes the parity-test SOURCE reference the tests cite.
- gen-cctest-skills.py + all 30 cctest-* wrappers + /help now point at
references/recipe-maintainer.
- JOURNAL.md: pending session entries (server-update policy addendum, tests-update,
orchestrator-update, upgrade-run notes).
One operator interface for both toolkits (operator decision 2026-08-04):
- vendor/autonomic-recipe-maintainer: ARM pinned as a submodule at acd5cfb (latest).
- scripts/gen-cctest-skills.py: generates a cctest-<name> wrapper pair
(.opencode canonical + .claude thin) for every ARM skill — frontmatter carries ARM's
own description tagged [recipe-maintainer/cctest]; body points at the canonical
submodule SKILL.md, sets cwd/sandbox context, and states the policy overrides
(auto-merge-style ARM flows need per-run operator opt-in; never touch cc-ci infra
from an ARM skill; submodule is read-only here). Re-run after every submodule bump.
- 30 cctest-* wrappers generated.
- /help: cctest family section + situation-table rows + the cc-ci-vs-cctest rule of thumb.
cctest = the recipe-maintainer test server; these skills run against it + the ARM
sandbox, never against the cc-ci CI server/swarm.
Seven check areas: weekly-run recency+outcome, report publishing, stale recipes/tests,
open recipe PRs (CVE-carrying PRs open >14d flagged high-priority, ready-to-merge PRs
listed as normal), server+orchestrator flake-update recency vs channel tip, host/service
health incl. the bridge !testme path (silent-401 stale-secret check from the 2026-08-03
finding), and maintained-set consistency. Ends with ALL HEALTHY or a prioritized findings
list, each mapped to the skill to invoke. Strictly read-only.
Codifies the full path walked for the 2026-08-03 wordpress enrollment (cc-ci PR #14):
survey -> mirror create+sync from coopcloud -> author test suite (health floor +
non-vacuous recipe-specific tests incl. sec4.3 create-an-object round-trip, recipe-local
setup helper, PARITY.md) -> bridge POLL_REPOS -> used-recipes.md weekly row +
upstream/<recipe>.md registry -> full-suite-green verification with the new tests ->
bridge deploy via test-before-switch -> merge-on-green + report (PR-visibility policy).
Includes the traps hit live: creds injection over stdin for cc-ci-side helpers, fresh-
deploy wizard state in HEALTH_OK, repo-dev-shell ruff, swarm serialization, stale bridge
secret 401s silently dropping !testme.
- cc-ci-server-update / cc-ci-tests-update / cc-ci-update: operator policy change —
open PRs for visibility/historical record and merge directly once verified
(invocation = authorization); reports must list merged-PR links + change summaries.
(tests-update still never merges the paired recipe upgrade PRs.)
- cc-ci-server-update: mandatory nixos-rebuild test step (5d) before switch, with
detached-activation + transient-unit notes from the 2026-08-03 26.05 deploy.
- AGENTS.md: test-before-switch policy for orchestrator host rebuilds.
- NEW skill cc-ci-orchestrator-update (+ thin .claude wrapper): the /cc-ci-server-update
analogue for this host (flake /srv/cc-ci-orch, .#cc-ci-orchestrator-hetzner, Hetzner
server 134487234), with self-update caveats.
- JOURNAL.md: 2026-08-03 server-update incident/recovery handoff entry.
Reverse the skill location: the full definitions now live in
.opencode/skills/<name>/SKILL.md (opencode's native project-skill
location, auto-scanned), and .claude/skills/<name>/SKILL.md are thin
pointer wrappers that keep the frontmatter for discoverability and
redirect to the .opencode/ canonical file for the procedure.
Also drops the 'operator-authorized mid-run' gating from
/cc-ci-server-update: the operator's choice to invoke the skill IS the
authorization for the live nixos-rebuild switch — no second check to
wait for. The quiescent-moment preference stays (don't disrupt an
in-flight !testme), but it's not a gate.
Three new skills for keeping cc-ci itself current (manual triggers for
what should ideally be automatic):
- /cc-ci-server-update — bump the cc-ci SERVER host's nixpkgs/sops-nix
flake inputs (hard rev pin, moves deliberately), open a cc-ci PR, then
deploy to the live host via nixos-rebuild switch (operator-authorized,
no-CI window) + health gate (no failed units, infra up, endpoints 200).
Manual trigger for periodic nixpkgs bumps.
- /cc-ci-tests-update — sweep all maintained recipes for stale cc-ci
tests (carry-over from /upgrade-all's commented stale tests + a fresh
/ci-test-review sweep), author the minimal test update per stale
recipe, open a cc-ci test PR, verify via the standard PR + !testme ->
CI flow. Closes the loop the weekly /upgrade-all deliberately defers
(it comments-only; this fixes the tests). Never weakens a test.
- /cc-ci-update — umbrella: run server-update then tests-update, in
that order, gated on the server health check (don't sweep tests on a
sick/rolled-back host). Server-first because the harness toolchain
rides on the host nixpkgs.
All three: create + verify, NEVER merge. Reuse the existing ci-test-review
helpers (open-cc-ci-pr.sh, run-all-recipes.sh, verify-pr.sh) and the
recipe-upgrade --with-tests discipline.
Switch the cc-ci-orchestrator agent (cc-ci-plan/agents.toml) from the claude
backend to the opencode backend so it attaches to the shared opencode web
server (opencode-web.service, 127.0.0.1:4096) and is reachable for Remote
Control at https://oc.commoninternet.net under the /srv/cc-ci-orch project.
Model = opencode/glm-5.2 (same provider family as the weekly upgrader /
supervisor). resume=false — the opencode backend has no --resume equivalent
(a fresh session per launch, matching the existing launch-opencode.sh shape).
Add a cc-ci-orchestrator.service systemd unit so on boot it runs
'agents.py up orchestrator' after opencode-web.service is up — the same
reboot-resilience shape lichen-orchestrator / project-orchestrator already
have. KillMode=process so a rebuild that merely touches this unit does not
tear down the shared tmux server and every agent session with it.
No secrets: the diff is only config + comments (paths, model names, unit
definition).
Add a 443 listener on the tailscale IP for the opencode vhost, alongside the
existing port 80. The name resolves to a CGNAT tailscale IP so Let's Encrypt
HTTP-01 can't validate it and there's no DNS-01 provider here; the vhost is
tailnet-only, so a self-signed cert (out-of-band at /etc/nginx/oc-selfsigned.*,
like oc-htpasswd) is acceptable. addSSL=true is required so the NixOS nginx
module actually renders ssl_certificate — without it nginx -t fails and takes
the atproto vhost down too. Rationale + regen command documented inline.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016L6nYYwkCWnrEFKTnKAfet
One tmux server hosts every agent session on this box and it lives in
cc-ci-loops.service's cgroup, so any rebuild touching that unit killed ALL sessions
(cc-ci, lichen, PO, pi). With KillMode=process systemd kills only the already-exited
launcher, not the cgroup — added to cc-ci-loops, lichen-orchestrator and
project-orchestrator. Trade-off: `systemctl stop` no longer tears agents down; do it
by hand.
Carries an in-flight nginx basicAuth block from the cc-ci agent that was already
uncommitted in this file and is now deployed with the 26.05 switch — committed here so
it is not lost, not authored by me.
Co-Authored-By: Claude <noreply@anthropic.com>
nixpkgs un-pinned from a frozen rev (50ab7937, 24.11, 2025-06-30) to the nixos-26.05
channel; sops-nix un-pinned to follow it. The old pins claimed to match the cc-ci
server "for ecosystem consistency", but this host runs agents/tmux/nginx/docker, not
recipe CI, so it need not track that server — and a frozen rev only accrues unpatched
CVEs. Built + activated: generation nixos-26.05.20260731.5b4f72e, kernel 6.6.94 ->
6.18.40 (reboot pending for the kernel). Pre-update files: /tmp/flake.{nix,lock}.pre-update.
Co-Authored-By: Claude <noreply@anthropic.com>
Upstream module dropped host-specific comments for its public repo, so the
context that matters here — gateway fronting, why this vhost stays off the
tailscale address group where opencode lives, and the exposure check script —
moves into the copy's header where it belongs.
Functional changes from upstream: the container now runs as uid 10001 and the
unit chowns the bind-mounted cache before start (without it, cache files left
by the earlier root container are unreadable and every cached actor 500s).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmEK2voMnBa23495aLk1Ce
Sync of the canonical module. The atproto vhost listens on 0.0.0.0 only, so it
never shares an address group with oc.commoninternet.net (which is the default
server on 100.84.190.30:80 and has no auth in front of it). A new
public-default-reject vhost owns the public addresses — return 444 on 80,
ssl_reject_handshake on 443 — so unmatched, absent or spoofed Host headers from
the internet get nothing, now and after any future vhost is added.
The tailnet is untouched: oc.commoninternet.net still serves opencode there.
Verified with tools/check-exposure.sh in the project repo — 9/9 ok.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmEK2voMnBa23495aLk1Ce
Sync of the canonical module in the project repo. forceSSL would have bounced
the gateway's plain-HTTP hop back to the gateway; and the oc vhost's explicit
listen on 100.84.190.30:80 was shadowing this vhost on the tailnet, serving the
opencode UI for Host: atproto.commoninternet.net.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmEK2voMnBa23495aLk1Ce
Imports the notplants-atproto module: Docker daemon (this host had none), a
systemd unit running the project's docker-compose stack, and an nginx vhost
proxying to it on 127.0.0.1:8731 with ACME and websocket support.
Opens 80/443 publicly. Until now only 22 was open and everything else was
tailnet-only, so this is a real change in exposure — nginx is now reachable
from the internet. The existing oc.commoninternet.net vhost is untouched and
stays bound to the tailscale IP.
nix/atproto-likes.nix is a COPY; the canonical file lives in the project repo
at /srv/project-orchestrator/projects/notplants-atproto/nix/. Pure evaluation
forbids importing an absolute path outside the flake tree, so it has to be
duplicated here — and a new file must be git-added or nix silently ignores it.
ACME currently FAILS: *.commoninternet.net is a wildcard pointing at
143.244.213.108, so the HTTP-01 challenge is answered by that host (500).
nginx serves a self-signed placeholder and starts fine. Fix is an explicit
A record atproto.commoninternet.net -> 168.119.126.100, then
`systemctl start acme-atproto.commoninternet.net.service`.
Applied with nixos-rebuild switch; container healthy, TLS proxy and wss
verified end to end.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmEK2voMnBa23495aLk1Ce
The PO is the operator's fleet-management Remote Control point; until now nothing
brought it back after a reboot. Mirrors lichen-orchestrator: oneshot + RemainAfterExit
running `agents.py up` as loops, which starts the PO agent and its watchdog. The PO's
agents.toml declares no `wake`, so the watchdog only heals a dead session — it never
sends periodic prompts.
Also lands the /mnt/data hardware.nix mount (by-uuid + nofail) that was sitting
uncommitted; fstab output is unchanged from the running system.
Deployed 2026-08-01: build delta was 5 derivations (the new unit + etc/system-units);
every other unit file byte-identical, dry-activate showed only a dbus reload. The
in-flight pi-coding-agent / launch.py work is parked on wip/pi-orchestrator-2026-08-01.
Co-Authored-By: Claude <noreply@anthropic.com>
autonomic-bot
autonomic-bot
notplantsandClaude
cc-ci-bot