During the DNS cutover the Hetzner resolvers nixos-infect had left in
networking.nix still served the old address (3h TTL) while 1.1.1.1/8.8.8.8
had the new one, so every other lookup of drone.ci.commoninternet.net from
the box itself went to the old server: runner "cannot request stage 404",
bridge "drone trigger failed 404", traefik probes alternating 200/404.
Drop those resolvers; pin ci/drone/report/traefik/warm-keycloak/oc names to
this host in networking.hosts. README §7 records it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
Enabling FILE_PROVIDER_DIRECTORY_ENABLED in the coop-cloud traefik recipe
REPLACES its single provider file (wildcard cert + `security` middleware),
which took every cc-ci front door down for two minutes on 2026-09-07.
Reverted. The route is now what every recipe does: a one-container swarm
stack (`opencode-ui`, alpine/socat relay to nginx on the docker bridge)
carrying the traefik router labels, deployed by opencode-ui-route.service.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
Operator: standard 443, routed by domain. Traefik already owns 443 with the
*.ci.commoninternet.net cert, so opencode-ui-route.service drops a dynamic
config into the traefik stack's watched file-providers volume routing
oc.ci.commoninternet.net -> http://172.18.0.1:8080, where nginx (reachable
only on docker_gwbridge) enforces the basic auth and logs 401s with the
real client IP (real_ip from X-Forwarded-For). The fail2ban nginx jail bans
in DOCKER-USER, since that traffic is forwarded, not INPUT. 8443 and the
ACME-host variant are gone.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
Operator 2026-09-07: the combined host stays off the tailnet. ssh is
key-only (password + keyboard-interactive auth off), the opencode UI is
served as oc.ci.commoninternet.net on 8443 — TLS from the CI server's
*.ci.commoninternet.net ACME cert (new orchestrator-host option
opencodeUiAcmeHost; nginx joins the acme group) behind the existing basic
auth — and fail2ban watches sshd plus nginx's 401s (nginx-http-auth jail).
tailscaled ordering dropped from the orchestrator units. README updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
Found on the first activation of #cc-ci on 195.201.88.249: the opencode
installer lands in ~/.opencode/bin (opencode-web then failed at EXEC), and
nginx's pre-start config check runs as the nginx user, so the staged
cert/htpasswd must be root:nginx 0640, not root:root 0600.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
hardware.nix / networking.nix are now the nixos-infect output of
195.201.88.249 (ESP E079-7D41, eth0 MAC-pinned), with the one edit that
matters: defaultGateway as an attrset with interface = "eth0".
README §2a records what it actually took to get NixOS running on the
Debian 13 image: NO_SWAP=true (tmpfs /tmp), a manual lustration from rescue
mode because 26.05's systemd initrd did not honour NIXOS_LUSTRATE (Debian's
unit files shadowed every NixOS service), and the gateway fix applied via a
chroot rebuild with the nix sandbox off.
flake: cc-ci input back on main (PR #32 merged as f6dbfa3); loops uid pinned
to 1000 so workspace rsyncs by uid line up.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
The orchestrator's flake now builds the machine it shares with the cc-ci CI
server: `nixosConfigurations.cc-ci` composes cc-ci's nixosModules.cc-ci-server
(new flake input, nixpkgs + sops-nix follow ours), this repo's orchestrator
module (nix/modules/cc-ci.nix, exported as cc-ci-orchestrator, `cc-ci` kept
as an alias for notplants-nix) and the new nix/modules/orchestrator-host.nix
— the host contract those units always assumed (loops user, claude/opencode
CLIs, opencode web server + tailnet-only UI on 8443 since traefik owns
80/443, nix-ld, tool set, `ssh cc-ci` → loopback).
nix/hosts/cc-ci/{hardware,networking}.nix are PROVISIONAL copies of the old
server's layout so the flake evaluates; they get replaced by the
nixos-infect output of 195.201.88.249.
README.md is the deploy guide (Hetzner Debian → nixos-infect → this flake →
staging → data restore → cutover). archive/ holds the retired Incus/Hetzner
orchestrator host configs, the old terraform and the migration plans;
references updated. cc-ci-plan/plan-cc-ci-combined-host.md is the working
plan for the move.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
The Gitea host was stored in the credentials env file, so a literal
secret scanner treated it as a secret and refused two READMEs in another
repo that link to git.autonomic.zone.
It was never secret: advisory-scan.py, pr-survey.py and recipe-report.py
already default it to git.autonomic.zone, and it is this repo's own
remote host. It is a bare hostname with no scheme, userinfo or path.
Removed from the env file and defaulted here the same way the Python
already does. GITEA_USERNAME and GITEA_PASSWORD keep their :? guard --
those are real credentials and must still fail loudly if unset.
The orchestrator host is a general agent box that also serves several unrelated
projects (atproto-likes, the lichen and project orchestrators, the shared
opencode web server). Everything that exists FOR cc-ci — the loops, the cc-ci
orchestrator session, the weekly recipe-upgrade run and its hourly supervisor —
now lives in nix/modules/cc-ci.nix and is exported as `nixosModules.cc-ci`, so
the two can evolve and be reviewed independently and another host can consume
just the cc-ci half.
No behaviour change. Verified by building .#cc-ci-orchestrator-hetzner after the
split: the unit set is identical to the running system, and all six cc-ci units
are byte-identical modulo nix store hashes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016L6nYYwkCWnrEFKTnKAfet
Move plausible from weekly to external in used-recipes.md. cc-ci still
deploys/tests it (the recipe + tests stay enrolled, !testme bridge
unchanged), but it is no longer surveyed/upgraded by the weekly /upgrade-all
run — maintained elsewhere, like uptime-kuma.
The open plausible PR #5 (clickhouse 23.4.2.11 -> 23.4.6.25) from this week's
run stays open for the operator; no future weekly PRs will be opened for it.
The weekly /upgrade-all parent session and the /recipe-report session now
run on tinfoil/deepseek-v4-pro (LOOP_MODEL + REPORT_MODEL in upgrader.env).
The hourly supervisor stays on opencode-go/glm-5.2 (SUPERVISOR_MODEL default
in launch-supervisor.py, not overridden). Subagents already bind deepseek via
the cc-ci repo's opencode config (fix from 2026-08-10, verified this week:
all 16 subagents across both waves ran deepseek-v4-pro).
LOOP_TIER=zen is kept so the tier check passes; the watchdog's usage-limit
probe sends the deepseek model name to the zen endpoint, which returns 200
(not 429) → resume immediately — correct, since tinfoil has no rolling usage
limit to wait out. Verified the probe behaviour with a direct curl.
Root cause: the 2026-08-14 run stalled mid-recipe on 'Insufficient balance'
(opencode zen workspace balance exhausted), then sat unfinished for 40h while
the supervisor cron spun hourly unable to recover it. Deepseek (pay-per-use
API key) has no rolling balance limit, so this can't recur.
Also documents the session recovery in JOURNAL.md (the stalled run was
completed via a fresh scoped upgrader — the original 2.58M-token session was
unresumable: the inference endpoint silently drops the oversized request).
The skill said to render a sourceless recipe as '?'. It now says how to stop it
being sourceless: declare an NVD CPE in the registry. That is what took the fleet
from two blind recipes to zero, and it is the first thing to try when the sweep
flags another.
Two recipes could not see CVEs at all. mattermost-lts has an empty GitHub advisory
feed and renders its security bulletins client-side, so a text sweep finds nothing;
mumble publishes nothing anywhere the registry points. Both returned '?' - nothing
measured - which is honest but useless.
NVD is CPE-indexed and carries structured version ranges, so it answers where the
vendor does not. Declared per recipe as 'nvd-cpe: <image> = <cpe:2.3:...>'.
mattermost-lts 10.5.0 -> 10.12.4 165 CVEs
mattermost-lts 10.11.22 -> 10.12.4 0 CVEs (measured, not unknown)
mumble 1.3.0 -> 1.6.870 2 CVEs
Both NVD range forms are used: versionEndExcluding is a patched version;
versionEndIncluding means the fix version is unpublished but the upgrade delivers
it whenever it crosses X.
That 0 for the actual mattermost upgrade is the interesting one, and it needed a
new rule to be correct: a fix on the line you upgrade FROM was already yours.
mattermost patches every maintained line at once, so 10.11.22 -> 10.12.4 crosses
10.12.1 while 10.11.22 already had the 10.11.4 backport. Without the rule the scan
claimed 12 CVEs the upgrade did not deliver.
The rule is skipped for placeholders: '7.4.X' parses to a bare 7.4 and would read
as 'already fixed at 7.4', which silently dropped redis CVE-2024-46981 and took
discourse 140 -> 139 before I caught it.
79 tests. discourse 140 / gitea 2 / mailu 2 / keycloak 12 / plausible 6 unchanged.
Fleet sweep: 0 recipes with no usable CVE source, down from 2.
Follow-up to the nginx blind spot. Sweeping all 22 recipes for sources whose CVEs
are USABLE (structured advisory feed, or a changelog attributable to releases)
rather than merely visible.
Before the changelog-attribution fix: 20 unusable sources. After: 5, and all five
are redundant - the same project also publishes an advisory feed (redis, gitea,
minio, clickhouse), so nothing is actually lost.
One real find, same shape as nginx: ONLYOFFICE/DocumentServer publishes NO GitHub
advisories, and the registry pointed its CHANGELOG.md at the GitHub *blob* page -
636KB of markup in which the release headings do not survive HTML-stripping, so 24
CVEs were visible and NONE attributable. The raw URL attributes all 24. Rather than
fix one registry line, advisory-scan now normalises github.com/../blob/.. to
raw.githubusercontent.com, which fixes every entry present and future.
lasuite-drive bumps documentserver, so this was live.
Genuinely blind after all that: mattermost-lts and mumble - no advisory feed, no
attributable changelog, no CVE data anywhere the registry points. mattermost is the
notable one: its bulletins are client-side rendered, so a regex sweep sees nothing.
Their scans can report 0 while nothing was measured, so /cve-check now renders those
recipes as ? and says why.
The audit output distinguishes a blind RECIPE from an unparseable PAGE, because
conflating them made 5 harmless redundancies look like 5 gaps.
nginx publishes NO GitHub security advisories. Every nginx CVE we can see comes
from nginx.org/en/CHANGES, and the scan scraped ids out of it without attributing
them to a release - so they had no patched version, could never be classified, and
every nginx bump in the fleet reported 0 CVEs. nginx is a sidecar in most recipes,
so this was a fleet-wide blind spot.
Measured on the two PRs that prompted the question:
lasuite-docs#7 nginx 1.31.1 -> 1.31.3 0 -> 6 CVEs
lasuite-drive#6 nginx 1.31.2 -> 1.31.3 0 -> 3 CVEs
matching a hand count of the changelog exactly (three fixed in 1.31.2, three in
1.31.3; the narrower window correctly counts only the latter).
How: when a vendor page is organised by release, each CVE is attributed to the
nearest preceding release heading ('Changes with nginx 1.31.3', '## v1.31.3'),
and that becomes its fixed-in version. The CVE is tied to a window by the image
name appearing in the page URL (window 'nginx' <-> nginx.org/...). A changelog
lists the project's whole history, so only releases the window actually crosses
count - asserted by a test that the 2013 entries stay out.
76 tests. discourse 140 / gitea 2 / mailu 2 / keycloak 12 unchanged.
Typing --from/--to/--image by hand means someone has to remember the recipe also
bumped its redis. That is how sidecar CVEs went uncounted for months. Point this
at a PR's compose.yml and it reads the windows off the diff instead.
advisory-scan.py plausible --compose-to <.../branch/<pr>/compose.yml>
-> community-edition: v2.0.0 -> v3.2.1
-> clickhouse-server: 23.4.2.11-alpine -> 24.12-alpine
-> 6 CVEs, identical to the hand-specified args
Details that mattered:
- keyed by SERVICE, not image repo. plausible moved plausible/analytics ->
ghcr.io/plausible/community-edition; keyed by repo that reads as one image
vanishing and an unrelated one appearing, and the app window - the one carrying
the critical - is lost entirely.
- the baseline is the repo's DEFAULT BRANCH resolved from the API, never assumed
to be main, because several recipes keep a stale main beside a live master.
- image names are matched against advisory sources BOTH ways: an image name is
often longer than its source repo (clickhouse/clickhouse-server vs
ClickHouse/ClickHouse) and sometimes shorter (redis vs redis/redis). One
direction silently dropped the clickhouse window.
- credentials go in an Authorization header, never the URL: in-URL creds leak
into shell history and process lists, and urllib mis-parses a password
containing a colon.
--from/--to/--image remain for finer-grained checks (scanning a window that is
not a literal compose diff). 71 tests; discourse 140 / gitea 2 / mailu 2
unchanged.
notplantsandClaude Fable 5.1
autonomic-bot
autonomic-bot