Operator: standard 443, routed by domain. Traefik already owns 443 with the
*.ci.commoninternet.net cert, so opencode-ui-route.service drops a dynamic
config into the traefik stack's watched file-providers volume routing
oc.ci.commoninternet.net -> http://172.18.0.1:8080, where nginx (reachable
only on docker_gwbridge) enforces the basic auth and logs 401s with the
real client IP (real_ip from X-Forwarded-For). The fail2ban nginx jail bans
in DOCKER-USER, since that traffic is forwarded, not INPUT. 8443 and the
ACME-host variant are gone.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
Operator 2026-09-07: the combined host stays off the tailnet. ssh is
key-only (password + keyboard-interactive auth off), the opencode UI is
served as oc.ci.commoninternet.net on 8443 — TLS from the CI server's
*.ci.commoninternet.net ACME cert (new orchestrator-host option
opencodeUiAcmeHost; nginx joins the acme group) behind the existing basic
auth — and fail2ban watches sshd plus nginx's 401s (nginx-http-auth jail).
tailscaled ordering dropped from the orchestrator units. README updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
Found on the first activation of #cc-ci on 195.201.88.249: the opencode
installer lands in ~/.opencode/bin (opencode-web then failed at EXEC), and
nginx's pre-start config check runs as the nginx user, so the staged
cert/htpasswd must be root:nginx 0640, not root:root 0600.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
hardware.nix / networking.nix are now the nixos-infect output of
195.201.88.249 (ESP E079-7D41, eth0 MAC-pinned), with the one edit that
matters: defaultGateway as an attrset with interface = "eth0".
README §2a records what it actually took to get NixOS running on the
Debian 13 image: NO_SWAP=true (tmpfs /tmp), a manual lustration from rescue
mode because 26.05's systemd initrd did not honour NIXOS_LUSTRATE (Debian's
unit files shadowed every NixOS service), and the gateway fix applied via a
chroot rebuild with the nix sandbox off.
flake: cc-ci input back on main (PR #32 merged as f6dbfa3); loops uid pinned
to 1000 so workspace rsyncs by uid line up.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
The orchestrator's flake now builds the machine it shares with the cc-ci CI
server: `nixosConfigurations.cc-ci` composes cc-ci's nixosModules.cc-ci-server
(new flake input, nixpkgs + sops-nix follow ours), this repo's orchestrator
module (nix/modules/cc-ci.nix, exported as cc-ci-orchestrator, `cc-ci` kept
as an alias for notplants-nix) and the new nix/modules/orchestrator-host.nix
— the host contract those units always assumed (loops user, claude/opencode
CLIs, opencode web server + tailnet-only UI on 8443 since traefik owns
80/443, nix-ld, tool set, `ssh cc-ci` → loopback).
nix/hosts/cc-ci/{hardware,networking}.nix are PROVISIONAL copies of the old
server's layout so the flake evaluates; they get replaced by the
nixos-infect output of 195.201.88.249.
README.md is the deploy guide (Hetzner Debian → nixos-infect → this flake →
staging → data restore → cutover). archive/ holds the retired Incus/Hetzner
orchestrator host configs, the old terraform and the migration plans;
references updated. cc-ci-plan/plan-cc-ci-combined-host.md is the working
plan for the move.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
The Gitea host was stored in the credentials env file, so a literal
secret scanner treated it as a secret and refused two READMEs in another
repo that link to git.autonomic.zone.
It was never secret: advisory-scan.py, pr-survey.py and recipe-report.py
already default it to git.autonomic.zone, and it is this repo's own
remote host. It is a bare hostname with no scheme, userinfo or path.
Removed from the env file and defaulted here the same way the Python
already does. GITEA_USERNAME and GITEA_PASSWORD keep their :? guard --
those are real credentials and must still fail loudly if unset.
The orchestrator host is a general agent box that also serves several unrelated
projects (atproto-likes, the lichen and project orchestrators, the shared
opencode web server). Everything that exists FOR cc-ci — the loops, the cc-ci
orchestrator session, the weekly recipe-upgrade run and its hourly supervisor —
now lives in nix/modules/cc-ci.nix and is exported as `nixosModules.cc-ci`, so
the two can evolve and be reviewed independently and another host can consume
just the cc-ci half.
No behaviour change. Verified by building .#cc-ci-orchestrator-hetzner after the
split: the unit set is identical to the running system, and all six cc-ci units
are byte-identical modulo nix store hashes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016L6nYYwkCWnrEFKTnKAfet
Move plausible from weekly to external in used-recipes.md. cc-ci still
deploys/tests it (the recipe + tests stay enrolled, !testme bridge
unchanged), but it is no longer surveyed/upgraded by the weekly /upgrade-all
run — maintained elsewhere, like uptime-kuma.
The open plausible PR #5 (clickhouse 23.4.2.11 -> 23.4.6.25) from this week's
run stays open for the operator; no future weekly PRs will be opened for it.
The weekly /upgrade-all parent session and the /recipe-report session now
run on tinfoil/deepseek-v4-pro (LOOP_MODEL + REPORT_MODEL in upgrader.env).
The hourly supervisor stays on opencode-go/glm-5.2 (SUPERVISOR_MODEL default
in launch-supervisor.py, not overridden). Subagents already bind deepseek via
the cc-ci repo's opencode config (fix from 2026-08-10, verified this week:
all 16 subagents across both waves ran deepseek-v4-pro).
LOOP_TIER=zen is kept so the tier check passes; the watchdog's usage-limit
probe sends the deepseek model name to the zen endpoint, which returns 200
(not 429) → resume immediately — correct, since tinfoil has no rolling usage
limit to wait out. Verified the probe behaviour with a direct curl.
Root cause: the 2026-08-14 run stalled mid-recipe on 'Insufficient balance'
(opencode zen workspace balance exhausted), then sat unfinished for 40h while
the supervisor cron spun hourly unable to recover it. Deepseek (pay-per-use
API key) has no rolling balance limit, so this can't recur.
Also documents the session recovery in JOURNAL.md (the stalled run was
completed via a fresh scoped upgrader — the original 2.58M-token session was
unresumable: the inference endpoint silently drops the oversized request).
The skill said to render a sourceless recipe as '?'. It now says how to stop it
being sourceless: declare an NVD CPE in the registry. That is what took the fleet
from two blind recipes to zero, and it is the first thing to try when the sweep
flags another.
Two recipes could not see CVEs at all. mattermost-lts has an empty GitHub advisory
feed and renders its security bulletins client-side, so a text sweep finds nothing;
mumble publishes nothing anywhere the registry points. Both returned '?' - nothing
measured - which is honest but useless.
NVD is CPE-indexed and carries structured version ranges, so it answers where the
vendor does not. Declared per recipe as 'nvd-cpe: <image> = <cpe:2.3:...>'.
mattermost-lts 10.5.0 -> 10.12.4 165 CVEs
mattermost-lts 10.11.22 -> 10.12.4 0 CVEs (measured, not unknown)
mumble 1.3.0 -> 1.6.870 2 CVEs
Both NVD range forms are used: versionEndExcluding is a patched version;
versionEndIncluding means the fix version is unpublished but the upgrade delivers
it whenever it crosses X.
That 0 for the actual mattermost upgrade is the interesting one, and it needed a
new rule to be correct: a fix on the line you upgrade FROM was already yours.
mattermost patches every maintained line at once, so 10.11.22 -> 10.12.4 crosses
10.12.1 while 10.11.22 already had the 10.11.4 backport. Without the rule the scan
claimed 12 CVEs the upgrade did not deliver.
The rule is skipped for placeholders: '7.4.X' parses to a bare 7.4 and would read
as 'already fixed at 7.4', which silently dropped redis CVE-2024-46981 and took
discourse 140 -> 139 before I caught it.
79 tests. discourse 140 / gitea 2 / mailu 2 / keycloak 12 / plausible 6 unchanged.
Fleet sweep: 0 recipes with no usable CVE source, down from 2.
Follow-up to the nginx blind spot. Sweeping all 22 recipes for sources whose CVEs
are USABLE (structured advisory feed, or a changelog attributable to releases)
rather than merely visible.
Before the changelog-attribution fix: 20 unusable sources. After: 5, and all five
are redundant - the same project also publishes an advisory feed (redis, gitea,
minio, clickhouse), so nothing is actually lost.
One real find, same shape as nginx: ONLYOFFICE/DocumentServer publishes NO GitHub
advisories, and the registry pointed its CHANGELOG.md at the GitHub *blob* page -
636KB of markup in which the release headings do not survive HTML-stripping, so 24
CVEs were visible and NONE attributable. The raw URL attributes all 24. Rather than
fix one registry line, advisory-scan now normalises github.com/../blob/.. to
raw.githubusercontent.com, which fixes every entry present and future.
lasuite-drive bumps documentserver, so this was live.
Genuinely blind after all that: mattermost-lts and mumble - no advisory feed, no
attributable changelog, no CVE data anywhere the registry points. mattermost is the
notable one: its bulletins are client-side rendered, so a regex sweep sees nothing.
Their scans can report 0 while nothing was measured, so /cve-check now renders those
recipes as ? and says why.
The audit output distinguishes a blind RECIPE from an unparseable PAGE, because
conflating them made 5 harmless redundancies look like 5 gaps.
nginx publishes NO GitHub security advisories. Every nginx CVE we can see comes
from nginx.org/en/CHANGES, and the scan scraped ids out of it without attributing
them to a release - so they had no patched version, could never be classified, and
every nginx bump in the fleet reported 0 CVEs. nginx is a sidecar in most recipes,
so this was a fleet-wide blind spot.
Measured on the two PRs that prompted the question:
lasuite-docs#7 nginx 1.31.1 -> 1.31.3 0 -> 6 CVEs
lasuite-drive#6 nginx 1.31.2 -> 1.31.3 0 -> 3 CVEs
matching a hand count of the changelog exactly (three fixed in 1.31.2, three in
1.31.3; the narrower window correctly counts only the latter).
How: when a vendor page is organised by release, each CVE is attributed to the
nearest preceding release heading ('Changes with nginx 1.31.3', '## v1.31.3'),
and that becomes its fixed-in version. The CVE is tied to a window by the image
name appearing in the page URL (window 'nginx' <-> nginx.org/...). A changelog
lists the project's whole history, so only releases the window actually crosses
count - asserted by a test that the 2013 entries stay out.
76 tests. discourse 140 / gitea 2 / mailu 2 / keycloak 12 unchanged.
Typing --from/--to/--image by hand means someone has to remember the recipe also
bumped its redis. That is how sidecar CVEs went uncounted for months. Point this
at a PR's compose.yml and it reads the windows off the diff instead.
advisory-scan.py plausible --compose-to <.../branch/<pr>/compose.yml>
-> community-edition: v2.0.0 -> v3.2.1
-> clickhouse-server: 23.4.2.11-alpine -> 24.12-alpine
-> 6 CVEs, identical to the hand-specified args
Details that mattered:
- keyed by SERVICE, not image repo. plausible moved plausible/analytics ->
ghcr.io/plausible/community-edition; keyed by repo that reads as one image
vanishing and an unrelated one appearing, and the app window - the one carrying
the critical - is lost entirely.
- the baseline is the repo's DEFAULT BRANCH resolved from the API, never assumed
to be main, because several recipes keep a stale main beside a live master.
- image names are matched against advisory sources BOTH ways: an image name is
often longer than its source repo (clickhouse/clickhouse-server vs
ClickHouse/ClickHouse) and sometimes shorter (redis vs redis/redis). One
direction silently dropped the clickhouse window.
- credentials go in an Authorization header, never the URL: in-URL creds leak
into shell history and process lists, and urllib mis-parses a password
containing a colon.
--from/--to/--image remain for finer-grained checks (scanning a window that is
not a literal compose diff). 71 tests; discourse 140 / gitea 2 / mailu 2
unchanged.
25 open recipe PRs had accumulated, and the list had stopped being readable: CI
sweep artifacts that were never meant to merge sat next to genuine CVE fixes, and
three PRs the operator had been told were outstanding were in fact already merged
upstream (discourse #6 with 140 CVEs, keycloak #6 with 12, n8n #5) — visible only
once the mirrors were reconciled.
The skill: reconcile every mirror from true upstream FIRST (that step alone closed
those three), survey every open PR deterministically, close the ones that cannot
merge or were never meant to, and report prioritised action items — CVE-carrying
first — for the ones that should land. It never merges a recipe PR.
pr-survey.py gathers the facts and decides nothing: behind_main, mergeable,
diff_files, which images the PR adds vs which are already pinned in main, the
newest !testme verdict, branch kind, age.
One correctness detail worth the extra state: a FAILED diff fetch is reported as
unknown, never as an empty diff. gitea #4 reads that way (force-pushed branch)
while being a verified green fix, and 'empty diff' is a close signal — so the
tool says DIFF-UNREADABLE(do not close on this) instead.
Operator policy (2026-08-11). For cc-ci-orchestrator and cc-ci: branch, open a PR
whose description is written to be read AFTER the fact, merge it yourself once
verified, and let the operator review retrospectively. The PR is not a gate — it
is how the work stays legible — so a description that says 'fix scanner' has
failed at its only job.
Explicitly does NOT extend to recipe repos: those are created and verified but
never agent-merged, because they change what deploys on other people's
infrastructure.
Also records what to do when work has already landed on main without a PR: pin a
branch at the pre-work commit and PR against that, rather than rewriting
published history.
I recorded that recreating the runs directory with a fresh inode preceded
recovery. It recurred afterwards (build 1252), so that was not the fix.
The real signal is that it is INTERMITTENT and tracks concurrent activity: every
failure landed while a second run or manual deploy was in flight, and every build
on a quiet host passed (1243, 1250, 1251, 1253). Free space never moves during a
failing build. Practical guidance is therefore to wait for the host to go quiet
and re-trigger before calling it a recipe failure, and DRONE_RUNNER_CAPACITY=2 is
the obvious knob to test if it becomes disruptive.
Root cause still not established, and the note now says so plainly rather than
presenting a coincidence as understood.
Working against a stale mirror has cost us three different ways:
- mailu #6 was linked as the fix for two internet-facing Roundcube CVEs while
upstream had already merged AND released it (3.1.3+2024.06.57). The work was
done; only our mirror was behind. Reconciling closed the PR automatically.
- a stale mirror makes a survey report 'no upgrades available', so the recipe
silently drops out of the weekly run.
- reading the wrong branch: several coopcloud recipes keep a stale 'main' beside
the real default 'master'. gitea's main is 1.24.2-rootless while master has
1.27.1-rootless and the merged PRs, so reading main manufactures a false
'three releases behind, missing two CVSS-9.8 RCEs' finding.
The reconcile logic already existed inside open-recipe-pr.sh --reconcile-only and
already resolves the default branch itself. What was missing was a single obvious
entry point and a rule saying to run it. reconcile-upstream.sh takes recipes or
--all, and is idempotent — recipe work lives in branches, never on mirror main, so
force-syncing main discards nothing.
/ci-test-review and /cc-ci-tests-update had NO reconcile step at all; both now
require it. /cve-check, /recipe-upgrade and /upgrade-all already reconciled and now
point at the shared script.
Builds 1244-1249 died on mkdir of the run dir with 110GB free and 16% inodes.
Ruled out: actual disk (df sampled every 2s across a failing build never moved),
inodes, quotas, a poisoned parent directory (61/61 stress creations succeeded),
runner sandboxing (namespaces identical to the host), and a wedged runner
(restart changed nothing). The same harness with the same numeric run id, run by
hand outside drone, worked every time.
Recreating the runs directory with a fresh inode preceded recovery; builds have
run normally since. The root cause is NOT established, so the note says so rather
than presenting a fix that might be coincidence.
notplantsandClaude Fable 5.1
autonomic-bot
autonomic-bot