recipe-maintainers/cc-ci-orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

weekly health-gated auto-update of the cc-ci host; skills rewritten for the combined host

Browse Source 
nix/modules/auto-update.nix (own module, no notplants-nix dependency): Tuesday
03:00 UTC, busy-gated (CI run, weekly upgrader, report, sweep, running Drone
builds → skip), `nix flake update` → build → switch-to-configuration test →
cc-ci health checks (sshd, 0 failed units, core units, every swarm service at
replica count, sops decrypted, dashboard/reports/drone 200, opencode UI 401)
→ profile + bootloader → flake.lock committed and pushed to main → /etc/cc-ci
fast-forwarded; revert + lock restore on failure; one-line state file for
/cc-ci-status.

Skills (.opencode canonical, .claude pointers' descriptions synced):
- cc-ci-orchestrator-update: THE host update — drives the auto-update unit by
  hand; --cc-ci-only for a cc-ci-main-only move; failure playbook.
- cc-ci-server-update: delegates to it and explains why the old procedure
  (rebuilding the cc-ci repo's standalone #cc-ci) must not be run on this host.
- cc-ci-update: chains orchestrator-update then tests-update.
- cc-ci-status: §5/§6 for one host — auto-update state, generation vs boot
  generation, front doors (oc.ci = 401), sops, fail2ban, timers, orchestrator
  agent session, secrets inventory; verdict updated.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
This commit is contained in:
notplants
2026-09-07 22:22:50 +00:00
co-authored by Claude Fable 5.1
parent 48226dd78f
commit a8af8429f2
11 changed files with 366 additions and 476 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/skills/cc-ci-orchestrator-update/SKILL.md
+1 -1
View File
@@ -1,6 +1,6 @@
---
name: cc-ci-orchestrator-update
description: Update the cc-ci ORCHESTRATOR host (the machine the orchestrator itself runs on) to the latest nixos-26.05 nixpkgs, then nixos-rebuild test + switch + verify, ship the bump as a verified PR on recipe-maintainers/cc-ci-orchestrator and merge it directly (the skill invocation IS the authorization). The orchestrator-host analogue of /cc-ci-server-update. Self-update caveats apply — the switch restarts services on the very host running this session; always `nixos-rebuild test` before `switch` so a reboot recovers to the last-known-good generation. The end-of-run report includes the merged PR link + a summary of what changed. Invoke as /cc-ci-orchestrator-update (also answers to /cc-ci-update-orchestrator).
description: Update the cc-ci HOST — since 2026-09 the ONE Hetzner machine that runs both the cc-ci CI server and the orchestrator (built from this repo's `#cc-ci`, which imports cc-ci's server module) — to the latest nixos-26.05 nixpkgs, sops-nix and cc-ci main, then `nixos-rebuild test` → cc-ci health checks → `switch`, committing the health-checked flake.lock straight to main (the skill invocation IS the authorization; the operator reviews retrospectively). It drives the host's own weekly auto-update unit by hand, so a manual and an automatic update are the same code path. Refuses while CI is busy. `--cc-ci-only` moves only the cc-ci input (harness/tests/modules), for after a cc-ci merge. Answers to /cc-ci-orchestrator-update, /cc-ci-update-orchestrator and (since the hosts merged) /cc-ci-server-update.
---
# cc-ci-orchestrator-update (thin wrapper)
.claude/skills/cc-ci-server-update/SKILL.md
+1 -1
View File
@@ -1,6 +1,6 @@
---
name: cc-ci-server-update
description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main. Opens the cc-ci PR for visibility/historical record and merges it directly once the deploy verifies (the skill invocation IS the authorization); the end-of-run report includes the merged PR link + a summary of what changed. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update.
description: (Delegates) Update the cc-ci CI server host to the latest nixpkgs — which, since 2026-09-07, is the SAME machine as the orchestrator, built from the cc-ci-orchestrator flake. This skill only forwards to /cc-ci-orchestrator-update; it no longer bumps or rebuilds the cc-ci repo's own flake, because that standalone `#cc-ci` target would replace the combined host config. Invoke as /cc-ci-server-update.
---
# cc-ci-server-update (thin wrapper)
.claude/skills/cc-ci-update/SKILL.md
+1 -1
View File
@@ -1,6 +1,6 @@
---
name: cc-ci-update
description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update.
description: Run both cc-ci maintenance updates in one pass — first /cc-ci-orchestrator-update (the cc-ci host: nixpkgs, sops-nix and the cc-ci input, test → health check → switch, lock committed to main), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update.
---
# cc-ci-update (thin wrapper)
.opencode/skills/cc-ci-orchestrator-update/SKILL.md
+66 -143
View File
@@ -1,164 +1,87 @@
---
name: cc-ci-orchestrator-update
description: Update the cc-ci ORCHESTRATOR host (the machine the orchestrator itself runs on) to the latest nixos-26.05 nixpkgs, then nixos-rebuild test + switch + verify, ship the bump as a verified PR on recipe-maintainers/cc-ci-orchestrator and merge it directly (the skill invocation IS the authorization). The orchestrator-host analogue of /cc-ci-server-update. Self-update caveats apply — the switch restarts services on the very host running this session; always `nixos-rebuild test` before `switch` so a reboot recovers to the last-known-good generation. The end-of-run report includes the merged PR link + a summary of what changed. Invoke as /cc-ci-orchestrator-update (also answers to /cc-ci-update-orchestrator).
description: Update the cc-ci HOST — since 2026-09 the ONE Hetzner machine that runs both the cc-ci CI server and the orchestrator (built from this repo's `#cc-ci`, which imports cc-ci's server module) — to the latest nixos-26.05 nixpkgs, sops-nix and cc-ci main, then `nixos-rebuild test` → cc-ci health checks → `switch`, committing the health-checked flake.lock straight to main (the skill invocation IS the authorization; the operator reviews retrospectively). It drives the host's own weekly auto-update unit by hand, so a manual and an automatic update are the same code path. Refuses while CI is busy. `--cc-ci-only` moves only the cc-ci input (harness/tests/modules), for after a cc-ci merge. Answers to /cc-ci-orchestrator-update, /cc-ci-update-orchestrator and (since the hosts merged) /cc-ci-server-update.
---
# cc-ci-orchestrator-update
Manually update the **orchestrator host**'s NixOS dependencies (nixpkgs; sops-nix follows nixpkgs)
to the current `nixos-26.05` channel tip, ship the bump via a verified PR + a live
`nixos-rebuild test` → `switch`, and confirm the host is healthy afterward. This is the
orchestrator-host analogue of `/cc-ci-server-update` — done deliberately, with a reviewable PR and
a health gate, not as silent drift.
Update **the cc-ci host** — one machine since 2026-09-07 (`cc-ci`, 195.201.88.249, no tailnet):
the CI server (docker swarm, traefik, drone, runner, bridge, dashboard, reports, acme-dns) AND the
orchestrator (loops, weekly upgrader, opencode agent) — to the current inputs, verified, committed.
> **Two hosts, two flakes — don't confuse them.** This skill updates the **orchestrator** host:
> the machine this session runs on (`cc-ci-orchestrator-1`, Hetzner cpx22 **server 134487234**,
> tailnet `cc-ci`, public `195.201.88.249` — the SAME host as the cc-ci CI server since 2026-09), flake checkout **`/srv/cc-ci-orch`** (repo
> `recipe-maintainers/cc-ci-orchestrator`), target **`.#cc-ci`** (which now also rebuilds the CI server half, from the cc-ci repo flake input). The **cc-ci
> CI server** (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, target `.#cc-ci`) is a different
> machine — that's `/cc-ci-server-update`, NOT this skill.
> **One flake, one target.** The host is `nixosConfigurations.cc-ci` in
> `recipe-maintainers/cc-ci-orchestrator` (`/srv/cc-ci-orch` on the host). The cc-ci repo's own
> `flake.nix` inputs are IRRELEVANT to the live build — this flake makes the `cc-ci` input follow
> its nixpkgs/sops-nix. Never run `nixos-rebuild` on the cc-ci repo's `#cc-ci` target on this host:
> that is the standalone-server config and would replace the combined one (orchestrator half gone).
> **This is a SELF-update.** The `switch` restarts units on the host running this very session.
> tmux sessions and the orchestrator survive a switch (systemd restarts don't kill user tmux), but
> `tailscaled`/`sshd` restarts can briefly drop Remote Control / SSH — run activation steps
> detached (`setsid nohup … &`) and re-poll. A full **reboot** DOES end the session: the
> auto-launch flow (REBOOTS.md + `cc-ci-loops.service`) restarts the loops, and the operator
> reconnects the orchestrator — announce a planned reboot to the operator first.
## What this skill does NOT do
- **Does NOT touch the cc-ci CI server** — that's `/cc-ci-server-update`.
- **Does NOT bump recipe images or cc-ci tests** — `/upgrade-all` / `/cc-ci-tests-update`.
- **Does NOT change the channel** (`nixos-26.05` branch ref in `flake.nix`) without operator
choice — a channel jump (e.g. → 26.11 when it releases) is a bigger change: ask first, read the
release notes, and expect module/option renames.
The work is done by **`cc-ci-auto-update.service`** (`nix/modules/auto-update.nix`), the same unit
the Tuesday 03:00 UTC timer fires. Running it by hand gives you the identical sequence —
busy gate → `nix flake update` → build → `switch-to-configuration test` → 45 s settle → health
check (sshd, 0 failed units, docker/acme-dns/fail2ban/nginx/opencode-web/runner active, every swarm
service at its replica count, sops decrypted, dashboard/reports/drone 200, opencode UI 401) →
profile + bootloader → `flake.lock` committed as `auto-update` and pushed to `main` →
`/etc/cc-ci` fast-forwarded — with a journal to read and a one-line state file.
## Preconditions
- You are on the orchestrator host; `/srv/cc-ci-orch` is the live checkout (git repo, remote
`recipe-maintainers/cc-ci-orchestrator`).
- Quiescent moment: no `/upgrade-all` in flight, no active build-loop phase, no cc-ci CI run that
would be disturbed if this host blips (`pgrep -af run_recipe_ci` empty on cc-ci is a bonus, but
the orchestrator host restarting mostly risks its own sessions, not the CI server).
- `git status` in `/srv/cc-ci-orch` — note pre-existing dirt; don't sweep unrelated changes into
the bump commit.
- `ssh cc-ci` works from the notplants-orchestrator box (root on the host), or you are on the host.
- **Not busy.** The unit checks itself (CI run, weekly upgrader, report, sweep, running Drone
builds → `result=skipped note=busy`, exit 0). Don't fight it: wait, or `tmux ls` as loops and
`docker service ls` to see what is running. Never kill a run to update a host.
- Note the current generation so you can name the old one in the report:
`ssh cc-ci 'readlink /run/current-system; cat /srv/cc-ci/.cc-ci-logs/auto-update-state'`.
## Procedure
### 1. Snapshot baseline
```bash
# 1. what will move
ssh cc-ci 'cd /srv/cc-ci-orch && sudo -H -u loops nix flake update --dry-run 2>&1 | grep -v "^warning" | tail -8' # informational; the unit does the real update
git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05 | cut -c1-12
```
nixos-version; readlink /run/current-system; uname -r
systemctl --failed --no-legend
df -h / | tail -1
cd /srv/cc-ci-orch && nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
tmux ls
```
Record: current nixpkgs rev, generation, failed units (should be 0), disk, live tmux sessions.
# 2. run the unit and follow it (10-25 min: nixpkgs bumps rebuild the two OCI images too)
ssh cc-ci 'systemctl start --no-block cc-ci-auto-update; journalctl -fu cc-ci-auto-update' # Ctrl-C when it prints "auto-update committed" / "REVERT" / "BUSY"
# or, non-interactively:
ssh cc-ci 'systemctl start cc-ci-auto-update; systemctl is-active cc-ci-auto-update; journalctl -u cc-ci-auto-update --no-pager -n 40'
### 2. Branch + bump the lock
Work directly in `/srv/cc-ci-orch` on a branch (it's the live checkout; the flake edit is inert
until a rebuild):
```
cd /srv/cc-ci-orch
git checkout -b chore/orchestrator-flake-update-$(date -u +%Y%m%d)
nix flake update 2>&1 | tail -5 # nixpkgs tracks nixos-26.05; sops-nix follows nixpkgs
nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
```
Note old→new revs. If the lock didn't move, report "already at channel tip" and stop (nothing to
deploy).
### 3. Build (catch errors before any activation)
```
cd /srv/cc-ci-orch && nixos-rebuild build --flake .#cc-ci 2>&1 | tail -15
readlink -f result
```
Build failure → fix on the branch (option renames etc.) before going further. Never activate a
build that didn't build cleanly.
### 4. Test-activate (MANDATORY before switch — general policy)
`nixos-rebuild test` activates without touching the bootloader or system profile, so if the
activation breaks the host (cf. the cc-ci server's 2026-08-03 no-default-route outage), a plain
reboot — Hetzner API power-cycle on server **134487234** if SSH is gone (see
`hetzner-server-recovery`) — lands back on the last-known-good generation.
```
cd /srv/cc-ci-orch && setsid nohup nixos-rebuild test --flake .#cc-ci \
> /tmp/orchestrator-test-switch.log 2>&1 < /dev/null & echo launched
# after it settles (poll; tailscaled/sshd may blip):
readlink /run/current-system # should be the new store path
ip route show default # default route survived
systemctl --failed --no-legend # 0 failed
tmux ls # sessions survived
tail -5 /tmp/orchestrator-test-switch.log
```
Regression here → you're still on the old boot default: diagnose, fix, or reboot to revert. Do NOT
switch.
### 5. Switch (make permanent — only after 4 is healthy)
```
cd /srv/cc-ci-orch && nixos-rebuild switch --flake .#cc-ci 2>&1 | tail -10
```
(If it fails with "Unit nixos-rebuild-switch-to-configuration.service was already loaded", the
detached test's transient unit is still running — wait or `systemctl stop` it, then retry.)
### 6. Health gate (do not skip)
```
nixos-version; readlink /run/current-system
systemctl --failed --no-legend
tmux ls # orchestrator + loop sessions intact
tailscale status | head -3 # tailnet up
systemctl list-timers cc-ci-upgrade-all.timer --no-pager | head -3
ssh cc-ci 'hostname' 2>&1 # can still reach the CI server
df -h / | tail -1
```
Optional (operator's call — it ends this session): a cold-boot proof reboot. If done, the
REBOOTS.md auto-launch flow restarts the loops; the operator reconnects the orchestrator.
### 7. Commit + PR + merge + report
```
cd /srv/cc-ci-orch
git add flake.lock # flake.nix too if the channel ref changed
git commit -m "flake: bump nixpkgs (nixos-26.05, $(date -u +%Y-%m-%d))
nixpkgs: <old-rev[:8]> -> <new-rev[:8]> (nixos-26.05 tip)
Deployed to the cc-ci host (.#cc-ci): build + test + switch + health gate green."
git push -u origin HEAD
```
Open the PR on `recipe-maintainers/cc-ci-orchestrator` (Gitea API with the `GITEA_*` creds from
`/srv/cc-ci/.testenv`, same as `open-cc-ci-pr.sh` does for cc-ci — the helper is hardcoded to the
cc-ci repo, so use the API directly or a copy with the repo swapped). PR body: old→new revs, what
was verified. Then **merge it directly** — the PR is for visibility/historical record; the skill
invocation is the authorization. On a rolled-back/failed run, leave the PR open with an
explanatory comment.
Back on the host, return the checkout to main so the live tree matches the merged state:
```
git checkout main && git pull
# 3. read the outcome
ssh cc-ci 'cat /srv/cc-ci/.cc-ci-logs/auto-update-state; readlink /run/current-system; readlink /nix/var/nix/profiles/system; systemctl --failed --no-legend; cd /srv/cc-ci-orch && git log --oneline -1 && git status --short'
```
Write `/srv/cc-ci-orch/.cc-ci-logs/orchestrator-update-<YYYY-MM-DD>.md`
(`RESULT: SUCCESS/ROLLED-BACK/FAILED …`), append a line to `cc-ci-plan/JOURNAL.md`, and report to
the operator with the **merged PR link + a summary of what changed**.
`--cc-ci-only` (after merging something in the cc-ci repo that the host should run now):
```bash
ssh cc-ci 'sudo -H -u loops bash -lc "cd /srv/cc-ci-orch && nix flake update cc-ci && git -c user.name=auto-update -c user.email=auto-update@cc-ci commit -qm \"flake.lock: cc-ci -> main\" -- flake.lock && git push -q origin HEAD:main"'
ssh cc-ci 'git -C /etc/cc-ci pull --ff-only --recurse-submodules'
ssh cc-ci 'systemd-run --unit=ccci-rebuild --collect -E HOME=/root -p WorkingDirectory=/srv/cc-ci-orch bash -lc "nixos-rebuild test --flake /srv/cc-ci-orch#cc-ci && nixos-rebuild switch --flake /srv/cc-ci-orch#cc-ci"'
# then the same step-3 checks, plus the §6 health commands from /cc-ci-status
```
## When it fails
`result=failed note=<step>` means the unit already re-activated the previous generation and
restored `flake.lock`; the host is as it was. Read `journalctl -u cc-ci-auto-update` for the
`HEALTH FAIL:` / `REVERT:` line:
- **build failed** — usually an upstream nixpkgs breakage or a cc-ci main that does not evaluate
against the new nixpkgs. Reproduce: `sudo -H -u loops nixos-rebuild build --flake /srv/cc-ci-orch#cc-ci`
in the checkout after `nix flake update`; fix in the right repo (PR, merge), re-run.
- **swarm services not at their replica count** — one of the reconcile oneshots did not converge
in 10 min: `journalctl -u deploy-<proxy|drone|bridge|dashboard|reports>`, `docker service ps <svc>`.
- **NAR hash mismatch** on the cc-ci input — the lock was made on another machine; `nix flake
update cc-ci` on the host and re-run (README day-2).
- **push failed** — bookkeeping only (the generation is committed): push the lock by hand as loops.
- A hung run: the unit has a 3 h timeout; `systemctl status cc-ci-auto-update`. Do NOT restart the
unit or the host mid-switch (see the `stopIfChanged` comment in the module).
## Report (end of run)
- old generation → new generation, the nixpkgs/sops-nix/cc-ci revs before → after, the lock commit
on `main`, the health check result, and anything the journal warned about.
- Then `/cc-ci-status` §5–§6 should read `ok` — say if it does not.
## Guardrails
- **Build before test, test before switch** — never `switch` without a healthy `test` activation
(general policy for this host and the cc-ci server; `test` keeps the bootloader untouched so a
reboot always recovers).
- **Detach activations** (`setsid nohup … &`) — a dropped SSH/Remote-Control connection must not
kill a half-applied activation.
- **Self-update awareness:** announce before any reboot (it ends the operator's session); a
switch is fine but expect brief tailscaled/sshd blips.
- **Single-writer:** dedicated branch; don't sweep unrelated working-tree changes into the bump
commit; return the live checkout to `main` after the merge.
- **Stable channel only** (`nixos-26.05` until the operator chooses a jump).
- **PRs for visibility, merged directly** once verified; failed runs leave the PR open. Always
report merged-PR links + change summaries.
- **Rescue path:** Hetzner server **134487234** — same rescue-mode procedure as
`hetzner-server-recovery` (grub generations live in a submenu: `1>N` ids).
- The unit is the only thing that `switch`es here; do not hand-roll `nixos-rebuild switch` unless
`--cc-ci-only` says so, and then `test` first.
- Never `nix flake update` from the notplants box and push a lock for this host (NAR hashes differ
between Nix versions) — locks for the host are made on the host.
- Nothing recipe-side: no PRs merged, no tests touched; `/cc-ci-tests-update` is separate.
.opencode/skills/cc-ci-server-update/SKILL.md
+17 -290
View File
@@ -1,298 +1,25 @@
---
name: cc-ci-server-update
description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main. Opens the cc-ci PR for visibility/historical record and merges it directly once the deploy verifies (the skill invocation IS the authorization); the end-of-run report includes the merged PR link + a summary of what changed. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update.
description: (Delegates) Update the cc-ci CI server host to the latest nixpkgs — which, since 2026-09-07, is the SAME machine as the orchestrator, built from the cc-ci-orchestrator flake. This skill only forwards to /cc-ci-orchestrator-update; it no longer bumps or rebuilds the cc-ci repo's own flake, because that standalone `#cc-ci` target would replace the combined host config. Invoke as /cc-ci-server-update.
---
# cc-ci-server-update
# cc-ci-server-update → `/cc-ci-orchestrator-update`
Manually update the **cc-ci CI server host**'s NixOS dependencies (nixpkgs + sops-nix) to their
latest pinned revisions, ship the bump via a verified cc-ci PR + a live `nixos-rebuild switch`, and
confirm the server is healthy afterward. This is the cc-ci-server analogue of a `nix flake update`
+ rebuild — done deliberately, with a reviewable PR and a health gate, not as silent drift.
There is no separate CI-server host any more. The CI server runs on the cc-ci host
(195.201.88.249) together with the orchestrator, from **one** flake:
`recipe-maintainers/cc-ci-orchestrator` `#cc-ci`, which imports the cc-ci repo's
`nixosModules.cc-ci-server` and makes that input follow its own nixpkgs and sops-nix.
> **Two hosts, two flakes — don't confuse them.** This skill updates the **cc-ci server** flake
> (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, host target `.#cc-ci` = `nix/hosts/cc-ci-hetzner/`).
> The **orchestrator** host (`/srv/cc-ci-orch`, flake host `.#cc-ci-orchestrator-hetzner`) is a
> different machine with its own flake — it is NOT touched here. If you want to update the
> orchestrator host, that's a separate `nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner`
> from `/srv/cc-ci-orch` (the orchestrator flake already tracks `nixos-26.05` and is rebuilt by the
> operator). This skill is only about the cc-ci **server**.
**Run `/cc-ci-orchestrator-update`.** It updates nixpkgs, sops-nix AND the cc-ci input, tests,
health-checks (the CI server's swarm, front doors, sops, acme-dns — the checks this skill used to
do) and switches, via the host's own weekly auto-update unit.
## What this skill does NOT do
Do **not** do what the old version of this skill did:
- do not bump `flake.lock` in the cc-ci repo expecting it to change the live host (it cannot — the
host follows the orchestrator flake's inputs; cc-ci's lock only matters for the standalone
`#cc-ci` target, which nothing deploys);
- do not run `nixos-rebuild switch --flake /etc/cc-ci#cc-ci` or `...#cc-ci-hetzner` on the host —
that is the standalone-server configuration and would take the orchestrator half down.
- **Does NOT bump recipe image tags** — that's `/upgrade-all` / `/recipe-upgrade`.
- **Does NOT touch cc-ci tests / harness / runner** — that's `/cc-ci-tests-update` or
`/ci-dev-workflow`. This is a flake-input bump only (`flake.nix` input revs + `flake.lock`).
- **Does NOT update the orchestrator host** — only the cc-ci server.
- **Does merge the cc-ci PR itself** — the PR exists for visibility and historical record, and is
merged directly at the end of a verified run (the skill invocation is the authorization; the
operator reviews the merged PR afterwards via the link in the report).
## Preconditions / access
- `ssh cc-ci` works (root, tailnet `100.95.31.88`). If it doesn't, fix access first (Tailscale ACL
/ `cc-ci-tailscaled` proxy) — do NOT proceed.
- **The operator's choice to invoke this skill IS the authorization for the live rebuild** — there
is no mid-run operator check to wait for. A `nixos-rebuild switch` cycles the abra reconcile
oneshots (`deploy-proxy`, `warm-keycloak`, …), so prefer a **quiescent moment** (no `!testme` /
nightly sweep running) to avoid disrupting an in-flight CI run. Check first:
```
ssh cc-ci 'docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"' # any unhealthy task?
ssh cc-ci 'pgrep -af run_recipe_ci | grep -v grep' # a CI run in flight?
```
If either is non-empty, WAIT until the host is quiescent — do not rebuild mid-CI.
- `/srv/cc-ci/.testenv` has the `GITEA_*` creds (for opening the cc-ci PR).
- The **operator-held sops secrets** are on the cc-ci host at `/etc/cc-ci/secrets/secrets.yaml`
(NOT in git). The rebuild needs them — see the rebuild procedure below.
## Procedure
### 1. Snapshot current state (so you can compare + roll back)
```
ssh cc-ci 'hostname; uptime; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \
deploy-bridge deploy-reports deploy-backupbot drone-runner-exec; systemctl --failed --no-legend; \
df -h / | tail -1; nixos-version; readlink /run/current-system'
```
Record: the current `nixos-version`, the current-system store path, the list of active infra
services, and the disk usage. These are your "before" baseline. Any rebuild that leaves a unit
failed or an endpoint down gets compared against this.
Also record the **current flake input revs** (the "before" of what you're bumping):
```
ssh cc-ci 'cd /root/builder-clone && nix flake metadata --json 2>/dev/null | jq -r ".locks.nodes | to_entries[] | \"\(.key) \(.value.locked.rev // \"-\")\""' 2>&1 | head
```
### 2. Prepare a working clone of the cc-ci repo (single-writer)
Work in a **separate clone on the orchestrator** — never push `main`, never touch the build loops'
working clones. Use the existing `/home/loops/work/cc-ci-fix` clone if present and clean, or make a
fresh one:
```
git clone ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /home/loops/work/cc-ci-server-update
cd /home/loops/work/cc-ci-server-update
git fetch origin && git checkout main && git reset --hard origin/main
git checkout -b chore/server-flake-update-$(date -u +%Y%m%d)
```
(If `/home/loops/work/cc-ci-fix` already exists and is on a stale branch, either reuse it on a new
branch or make the fresh clone above — the key rule is the branch is dedicated + off `main`.)
### 3. Bump the flake inputs (deliberate, not drift)
The cc-ci server flake **intentionally hard-pins** `nixpkgs` to a specific revision (the comment in
`flake.nix` says "Bump deliberately, not drift") and `sops-nix` to a commit compatible with that
nixpkgs. So the bump is a **considered move to a newer nixpkgs + a compatible sops-nix**, not a
blind `nix flake update --relock-all`.
**(a) Decide the new nixpkgs.** The cc-ci server runs recipe CI; nixpkgs provides docker, abra's
runtime, traefik, keycloak, drone, the harness python/pytest/playwright toolchain. Pick the newest
**stable channel revision** that's compatible:
- Default: move to the latest revision of the **same channel** the orchestrator host tracks
(`nixos-26.05`), so both hosts stay on a shared nixpkgs and CVEs get patched. Look up the current
tip: `git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05` (the SHA after the ref).
- If a newer stable channel is out and the operator wants it, use that — but treat a **channel
jump** (e.g. 26.05 → 26.11) as a bigger change: scan the release notes for breaking changes
(docker, systemd, python, nginx module option renames) and surface them in the PR body. Default
to staying on the same channel unless the operator asks for the jump.
- **Never** jump to `nixos-unstable` for the cc-ci server — recipe CI reproducibility depends on a
stable nixpkgs. Unstable is for experiments, not production.
**(b) Update `flake.nix`** with the new nixpkgs rev (replace the `rev = …` in the `nixpkgs.url`),
and check whether `sops-nix` needs a matching bump:
- sops-nix `inputs.nixpkgs.follows = "nixpkgs"`, so it tracks nixpkgs automatically — but its own
rev pin may need moving if the pinned sops-nix commit is incompatible with the new nixpkgs (e.g.
the `buildGoModule` → `buildGo125Module` gotcha the current pin's comment calls out). After
editing `flake.nix`, run `nix flake lock --update-input nixpkgs --update-input sops-nix` (or
`nix flake update` if both follow) in the clone and let nix resolve a compatible sops-nix. If the
lock fails (sops-nix master needs a newer nixpkgs than you picked), pick a sops-nix commit that
still builds against your chosen nixpkgs (its `flake.nix` declares the nixpkgs it expects).
- **Keep the `sops-nix.inputs.nixpkgs.follows = "nixpkgs"` line** — that's what keeps them aligned.
- Update the comment in `flake.nix` to reflect the new pin date + reason (the existing comment
style: "Pinned to … (operator YYYY-MM-DD)" or "Follows nixos-26.05 channel").
**(c) Re-lock + sanity-build locally first** (cheap, catches a broken input before touching the
host):
```
cd /home/loops/work/cc-ci-server-update # (or cc-ci-fix on the new branch)
nix flake lock --update-input nixpkgs --update-input sops-nix # rewrites flake.lock
nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
git diff flake.nix flake.lock | head -40
```
Verify the lock moved to the revs you intended. **Do NOT `nixos-rebuild build` from the
orchestrator** — the orchestrator's nixpkgs is different and the cc-ci host target won't build here
(meaningfully). The real build happens on the cc-ci host in step 5.
### 4. Commit + open the cc-ci PR (merged in step 7 after the health gate)
```
git add flake.nix flake.lock
git commit -m "flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))
nixpkgs: <old-rev[:8]> → <new-rev[:8]> (nixos-<channel>)
sops-nix: <old-rev[:8]> → <new-rev[:8]>
<one-line reason — CVE patch / channel bump / sops-nix compat>"
git push -u origin HEAD
```
Open the PR via the helper:
```
set -a; . /srv/cc-ci/.testenv; set +a
TITLE="flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))" \
BODY_FILE=/tmp/cc-ci-server-update-pr-body.md \
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh
```
(Write the PR body to `/tmp/cc-ci-server-update-pr-body.md` first: old→new revs, the channel, any
breaking changes from the release notes, and the rebuild plan. The PR is opened for visibility and
historical record; it gets merged directly in step 7 once the deploy passes the health gate.)
Capture the `PR_URL`. **The cc-ci push-build lint gate will run on the branch** — if it's red on a
flake-lock-only change, it's a pre-existing lint drift (not yours); note it in the PR body and
proceed (the rebuild doesn't depend on the lint gate passing).
### 5. Deploy to the live cc-ci host (no-CI window)
This is the host-rebuild procedure from memory `cc-ci-host-rebuild-procedure` — the cc-ci server
has **no self-service rebuild path**, so the orchestrator stages the repo + rebuilds it. The
deploy clone on the host is `/root/builder-clone`; staging a fresh copy avoids disturbing it.
**(a) Stage the PR branch on the host:**
```
# from the orchestrator, on the PR branch, clean tree:
rsync -a --delete --exclude='.git' /home/loops/work/cc-ci-server-update/ root@cc-ci:/root/cc-ci-deploy/
ssh cc-ci 'chown -R root:root /root/cc-ci-deploy'
```
(A `--delete` rsync from a clean tree = exactly the PR head. Excluding `.git` keeps it a plain path
flake — see (c).)
**(b) Copy the operator-held sops secrets** (NOT in git, required for the build):
```
ssh cc-ci 'mkdir -p /root/cc-ci-deploy/secrets && cp /etc/cc-ci/secrets/secrets.yaml /root/cc-ci-deploy/secrets/secrets.yaml && chmod 600 /root/cc-ci-deploy/secrets/secrets.yaml'
```
(The age key is at `/var/lib/sops-nix/key.txt` on the host — already in place from prior rebuilds.
Without `secrets/secrets.yaml` the build FATAs `secrets/secrets.yaml does not exist`.)
**(c) Build first (catch errors before switching):**
```
ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild build --flake .#cc-ci' 2>&1 | tail -30
```
- Target is `.#cc-ci` (= `.#cc-ci-hetzner` = `nix/hosts/cc-ci-hetzner/configuration.nix`).
- nixpkgs is now your bumped pin, so this **will** rebuild more than a same-pin no-op — expect a
longer build (docker/traefik/keycloak/python toolchain may all rebuild against the new nixpkgs).
That's expected; this is a real nixpkgs bump.
- If the build fails: read the error, fix on the PR branch (e.g. a renamed option, a removed
package), re-push, re-stage, re-build. Do NOT `switch` a failed build. Common breakages: NixOS
module option renames across channels (check the release notes), python package removals, a
sops-nix incompatibility (go back to step 3b and pick a compatible sops-nix rev).
**(d) Test-activate FIRST (general policy for nix deploys to this server AND the orchestrator
host): `nixos-rebuild test` before any `switch`:**
```
# detach it so a dropped SSH can't kill the activation mid-flight:
ssh cc-ci 'cd /root/cc-ci-deploy && setsid nohup nixos-rebuild test --flake .#cc-ci \
> /root/cc-ci-deploy/test-switch.log 2>&1 < /dev/null & echo launched'
# poll until reachable again, then check the essentials survived:
ssh cc-ci 'readlink /run/current-system; ip route show default; systemctl --failed --no-legend'
```
`test` activates the new configuration **without touching the bootloader or the system profile**,
so if the activation kills networking (as the 2026-08-03 26.05 bump did — no default route), a
plain reboot (Hetzner API power-cycle if SSH is gone) lands back on the last-known-good
generation. Only proceed to `switch` once the test activation is up, reachable, and healthy.
Note: the detached test leaves a transient `nixos-rebuild-switch-to-configuration` unit; if the
follow-up `switch` fails with "Unit ... was already loaded", wait for it to finish or
`systemctl stop nixos-rebuild-switch-to-configuration.service` first.
**(e) Switch (make it permanent — only after (d) verified):**
```
ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild switch --flake .#cc-ci' 2>&1 | tail -30
```
This is the moment the server moves to the new nixpkgs. The abra reconcile oneshots
(`deploy-proxy`, `warm-keycloak`, …) re-run as part of the switch — watch the tail for any that
fail. If a reconcile unit fails, do NOT roll back blindly — diagnose first (a warm-keycloak blip
may be transient; a proxy failure is real).
### 6. Verify the server is healthy (health gate — do not skip)
```
ssh cc-ci 'echo "=== version ==="; nixos-version; readlink /run/current-system; \
echo "=== failed units ==="; systemctl --failed --no-legend; \
echo "=== infra services ==="; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \
deploy-bridge deploy-reports deploy-backupbot drone-runner-exec | paste - - - - - - -; \
echo "=== docker stacks ==="; docker stack ls --format "{{.Name}} {{.Services}}"; \
echo "=== service replicas (any 0/N?) ==="; docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"; \
echo "=== endpoints ==="; curl -sS -o /dev/null -w "traefik:%{http_code}\n" https://traefik.ci.commoninternet.net/api/version; \
curl -sS -o /dev/null -w "dashboard:%{http_code}\n" https://ci.commoninternet.net/; \
echo "=== disk ==="; df -h / | tail -1'
```
Compare against the step-1 baseline:
- `nixos-version` moved (or the current-system path changed) — the bump landed.
- **No failed units** (the `--failed` output is empty).
- All 7 infra services `active` (or `active (exited)` for the oneshots).
- `docker service ls` shows no `0/N` replicas (everything converged).
- Endpoints `200` (traefik `/api/version`, dashboard root).
- Disk didn't spike (a nixpkgs bump adds a new system generation; old ones are GC'd eventually).
If **anything regressed**: diagnose. A real regression → roll back to the previous generation:
`ssh cc-ci 'nixos-rebuild switch --rollback'` (this switches to the pre-bump system path; the
flake input change on the PR branch is still your source of truth, but the running host reverts
while you fix). Report the rollback in the PR.
### 7. Merge the PR + report
On a **verified** run (health gate green): post a PR comment summarizing what was deployed (revs,
any incidents, verification results), then **merge the PR directly** — the skill invocation is the
authorization; the PR exists for visibility and historical record. On a rolled-back or failed run,
leave the PR **open** with a comment explaining the state.
Print + record a one-line result + the PR URL. Write a short note to
`/srv/cc-ci/.cc-ci-logs/server-update-<YYYY-MM-DD>.md`:
- `RESULT: SUCCESS — cc-ci server rebuilt on nixpkgs <new-rev[:8]>, no failed units, endpoints 200, PR merged: <url>`
- `RESULT: ROLLED-BACK — nixpkgs <new-rev> rebuild failed/ regressed (<reason>); host reverted to <old-rev>; PR <url> left open`
- `RESULT: FAILED — <step>: <reason>`
The end-of-run report to the operator MUST include **links to every merged PR** plus a summary of
what changed in each, so the operator can review them after the fact if they want to.
### 8. Optional cold-boot proof
A `nixos-rebuild switch` proves the new generation works on the running kernel. A **reboot** proves
it boots cleanly from cold — the stronger guarantee (cf. the pxgate cold-boot proof). Only do this
if the operator wants it, and only outside a CI window:
```
ssh cc-ci 'systemctl reboot'
# wait ~60s, re-check ssh cc-ci + the step-6 health gate
```
Record the new `boot_id` if you do this (it's a genuine reboot — log it to
`cc-ci-plan/REBOOTS.md`? No — that file is for the **orchestrator** host. The cc-ci server's reboots
are tracked in its own journal; a note in the server-update report is enough).
## Guardrails
- **Live rebuild is authorized by invocation.** The operator's choice to run this skill IS the
go-ahead for the `nixos-rebuild switch` — do not stop mid-run to re-confirm. A switch cycles
reconcile oneshots and can disrupt an in-flight `!testme`, so prefer a quiescent moment (the
step-1 check), but don't gate on a second authorization.
- **Single-writer:** dedicated branch in a separate clone; never push `main`; never touch the
build loops' `/cc-ci` / `/cc-ci-adv` clones or `/root/builder-clone` (stage to
`/root/cc-ci-deploy` instead).
- **Open the PR for visibility, merge it directly.** The live rebuild is the deploy; the PR is the
reviewable historical record. Merge only after the health gate passes; a rolled-back/failed run
leaves the PR open with an explanatory comment. Always report merged-PR links + change summaries.
- **Build before test, test before switch.** Never `nixos-rebuild switch` a build that didn't build
cleanly first, and never `switch` without a healthy `nixos-rebuild test` activation first — `test`
leaves the bootloader untouched, so a reboot always recovers to the last-known-good generation.
This is general policy for nix deploys to the cc-ci server **and** the orchestrator host.
- **Health gate is mandatory.** No "it switched, ship it" — run the step-6 verification and compare
to the baseline. Roll back on regression.
- **Stable channel only** for the cc-ci server (never `nixos-unstable`).
- **Keep sops-nix.follows** so the two inputs stay aligned; don't fork them.
- **Secrets never in git.** The operator-held `secrets/secrets.yaml` is copied on the host only.
- **Rollback is always available:** `nixos-rebuild switch --rollback` reverts to the prior
generation without touching the flake input revs.
## Relationship to an automatic update
This skill is the **manual trigger** for what should ideally run on a periodic timer: bump the cc-ci
flake inputs, build, switch, verify. If/when an automatic `cc-ci-server-update.timer` is added
(NixOS systemd timer on the orchestrator, analog of `cc-ci-upgrade-all.timer`), it should invoke
this skill via a dedicated one-shot agent (the way `cc-ci-upgrader` runs `/upgrade-all`). The
skill stays as the authoritative procedure either way — automatic just means "run on a schedule
without a human prompt."
If what you actually wanted is "the host should run cc-ci `main` as of now" (a harness or test
change), that is `/cc-ci-orchestrator-update --cc-ci-only`.
.opencode/skills/cc-ci-status/SKILL.md
+59 -29
View File
@@ -58,35 +58,62 @@ For each open PR: age (now − created_at), and whether the PR title/body/report
- Verified-green PRs awaiting operator merge are normal — list them as "ready to merge", not
as failures.
### 5. Server + orchestrator host update recency
### 5. The cc-ci host: update recency + the weekly auto-update
For BOTH hosts:
```
# cc-ci server: ssh cc-ci 'nixos-version; cd /root/cc-ci-deploy && nix flake metadata --json' (or builder-clone)
# orchestrator: nixos-version; cd /srv/cc-ci-orch && nix flake metadata --json
git ls-remote https://github.com/NixOS/nixpkgs <channel> # current tip
ls -t /srv/cc-ci-orch/.cc-ci-logs/server-update-*.md /srv/cc-ci-orch/.cc-ci-logs/orchestrator-update-*.md | head -2
```
- Report: days since last update log + how far the running nixpkgs rev lags the channel tip.
- **Lagging >30 days** (or a NixOS release behind) → recommend `/cc-ci-server-update` /
`/cc-ci-orchestrator-update`.
### 6. Host + service health (both machines)
Since 2026-09-07 the CI server and the orchestrator are ONE Hetzner host (`cc-ci`,
195.201.88.249, no tailnet), built from `recipe-maintainers/cc-ci-orchestrator`'s
`nixosConfigurations.cc-ci` (which imports cc-ci's `nixosModules.cc-ci-server`). From the
notplants-orchestrator box `ssh cc-ci` reaches it as root; on the host itself `ssh cc-ci` is loopback.
```
ssh cc-ci 'systemctl --failed --no-legend; df -h / | tail -1; docker service ls --format "{{.Name}} {{.Replicas}}"'
systemctl --failed --no-legend; df -h / | tail -1; tmux ls
ssh cc-ci 'nixos-version; readlink /run/current-system; readlink /nix/var/nix/profiles/system'
ssh cc-ci 'cd /srv/cc-ci-orch && nix flake metadata --json | jq -r ".locks.nodes | to_entries[] | \"\(.key) \(.value.locked.rev // \"-\") \(.value.locked.lastModified // \"-\")\""'
git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05 | cut -c1-12 # channel tip
# the weekly self-update (nix/modules/auto-update.nix, Tue 03:00 UTC, busy-gated):
ssh cc-ci 'cat /srv/cc-ci/.cc-ci-logs/auto-update-state; systemctl list-timers --no-pager cc-ci-auto-update.timer | sed -n 2p; journalctl -u cc-ci-auto-update --no-pager -n 3'
```
- Failed units, core swarm services not 1/1 (warm-* spares flapping is a known benign pattern —
note, don't page), disk **>65% (server)** / >85% (orchestrator) → findings. Server unreachable →
HIGH: recommend `hetzner-server-recovery`.
> **65%, not 80%, on the server — it is not a steady-state measure.** Two concurrent recipe runs
> pull images and write volumes worth tens of GB, so a host sitting at 73% still hits 100% mid-run.
> That is exactly what happened on 2026-08-11: 63GB of unused images had accumulated (nothing ever
> pruned them), the filesystem filled during a run, and the harness died at startup with
> `OSError: [Errno 28] No space left on device`. Remedy: `docker image prune -af` on cc-ci — it
> spares anything a container references, so infra and warm-* canonicals are untouched. Do NOT
> `docker volume prune`: warm-* canonical volumes are data-warm and legitimately dangling.
- The state file is one line: `<UTC time> result=ok|unchanged|skipped|failed generation=<store path> note=<why>`.
- `ok` / `unchanged` within the last 8 days → healthy.
- `skipped` (busy) two weeks in a row, `failed`, or no state line / timer inactive for >8 days →
finding: *"host auto-update <state> since <date>: <note>"* → recommend
`/cc-ci-orchestrator-update` (it runs the same unit by hand and reads its journal).
- `failed` also means the host is running the previous generation and `flake.lock` was restored —
say so; the journal (`journalctl -u cc-ci-auto-update`) names the health check that failed.
- Running `/run/current-system` must equal the profile's boot generation (`system` link) — a
mismatch means a `test`-only activation was left behind (finding: switch or reboot decision).
- nixpkgs rev lagging the `nixos-26.05` channel tip by >30 days → finding → `/cc-ci-orchestrator-update`.
- `cc-ci` (the CI server input) lagging cc-ci `main` → note; the auto-update moves it weekly, or
`/cc-ci-orchestrator-update --cc-ci-only`.
### 6. Host + service health (one machine)
```
ssh cc-ci 'systemctl --failed --no-legend; df -h / | tail -1; free -m | sed -n 2p'
ssh cc-ci 'docker service ls --format "{{.Name}} {{.Replicas}}"; docker stack ls --format "{{.Name}}"'
ssh cc-ci 'for h in ci report.ci drone.ci oc.ci; do printf "%s " $h; curl -s -o /dev/null -w "%{http_code} ssl=%{ssl_verify_result}\n" https://$h.commoninternet.net/; done'
ssh cc-ci 'ls -s /run/secrets/test_secret; systemctl is-active acme-dns fail2ban nginx opencode-web drone-runner-exec; fail2ban-client status sshd | grep -E "Currently banned|Total banned"; fail2ban-client status nginx-http-auth | grep -E "Currently banned|Total banned"'
ssh cc-ci 'systemctl list-timers --no-pager cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer cc-ci-auto-update.timer nightly-sweep.timer | tail -n +2 | head -4'
ssh cc-ci 'sudo -iu loops tmux ls' # cc-ci-orchestrator must exist (the opencode orchestrator agent); cc-ci-upgrader only during a run
```
- Failed units, swarm services not at their replica count (`warm-*` spares flapping is benign —
note, don't page), a front door not 200 (`oc.ci` must be **401**: basic auth in front of the
opencode UI; 404 there = the `ccci-opencode-ui` stack is gone, `systemctl restart opencode-ui-route`
and check the weekly sweep's keep-list), `ssl≠0` (cert), `/run/secrets` empty (sops), acme-dns /
fail2ban / nginx / opencode-web / drone-runner-exec inactive, any timer missing, the orchestrator
tmux session missing (`systemctl restart cc-ci-orchestrator`) → findings. Disk **>65%** → finding.
Host unreachable → HIGH: `hetzner-server-recovery` (token in `/srv/cc-ci/.hcloud-token` on the
notplants box while the operator keeps it active).
> **65%, not 80% — it is not a steady-state measure.** Two concurrent recipe runs pull images and
> write volumes worth tens of GB, so a host sitting at 73% still hits 100% mid-run (2026-08-11:
> 63GB of unused images, `OSError: [Errno 28] No space left on device`). Remedy: `docker image
> prune -af` — it spares anything a container references. Do NOT `docker volume prune`: warm-*
> canonical volumes are data-warm and legitimately dangling.
- fail2ban: a non-zero "Currently banned" is information, not a finding (that is the jail
working); the notplants box's own address showing up banned IS a finding (it means a skill
misbehaved) — `fail2ban-client set <jail> unbanip <ip>`.
- Secrets hygiene: `ssh cc-ci 'ls /secrets/files'` must list ONLY cc-ci.testenv, cc-ci-local-ed25519(.pub),
autonomic-bot-cc-ci-ed25519(.pub), opencode-auth.json. Anything else there (a tailscale key, a
master age key, another host's ssh key…) → finding: *"secret sprawl on the cc-ci host"*.
- **!testme actually produces results** (the check that would have caught the above days earlier):
the newest few `/var/lib/cc-ci-runs/<build>/` dirs must each contain `results.json`. A build that
dies before the harness writes one leaves an EMPTY dir — and the PR badge still says "failure", so
@@ -143,14 +170,17 @@ minutes, no PRs). If it is instead that a known CVE is sitting unpatched, recomm
`/cve-check` over waiting for the next weekly run whenever the question is "are we exposed?".
`ALL HEALTHY` requires: recent successful weekly run + published report, no stale tests, no
CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, recent builds all
producing results.json, disk under
thresholds, bridge clean, maintained-set consistent. Anything else is a finding — even minor
CVE PR open >14 days, the host <30 days behind its channel with the weekly auto-update `ok`/`unchanged`
within 8 days and running generation == boot generation, zero failed units, every front door
answering (oc.ci with 401), sops decrypted, fail2ban up, the orchestrator agent session present,
recent builds all producing results.json, disk under threshold, bridge clean, maintained-set
consistent, secrets inventory exact. Anything else is a finding — even minor
ones get a recommended next step. Order findings by priority (CVE/unreachable-host first).
## Guardrails
- **Read-only.** This skill diagnoses and recommends; it invokes nothing and changes nothing.
- Use the pseudo-TTY wrap for any abra call; plain ssh for everything else.
- Use the pseudo-TTY wrap for any abra call; plain ssh for everything else. `ssh cc-ci` from the
notplants box is root on 195.201.88.249 (loops ssh config); there is no tailnet.
- Don't double-count: a finding that explains another (bridge 401 → !testme "failures") gets
reported once, at the root cause.
.opencode/skills/cc-ci-update/SKILL.md
+7 -8
View File
@@ -1,12 +1,11 @@
---
name: cc-ci-update
description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update.
description: Run both cc-ci maintenance updates in one pass — first /cc-ci-orchestrator-update (the cc-ci host: nixpkgs, sops-nix and the cc-ci input, test → health check → switch, lock committed to main), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update.
---
# cc-ci-update
The combined **keep cc-ci itself current** pass: run `/cc-ci-server-update` (host dependency bump +
rebuild) **then** `/cc-ci-tests-update` (stale-test sweep + fixes), in that order, end-to-end. This
The combined **keep cc-ci itself current** pass: run `/cc-ci-orchestrator-update` (host dependency bump + health-checked rebuild) **then** `/cc-ci-tests-update` (stale-test sweep + fixes), in that order, end-to-end. This
is just sequencing the two skills — it doesn't add or change any behavior. Read each sub-skill's
SKILL.md for the full procedure; this file is the orchestration + the why-of-the-order.
@@ -21,7 +20,7 @@ SKILL.md for the full procedure; this file is the orchestration + the why-of-the
## Arguments
None. (If you only want one of the two updates, invoke the sub-skill directly:
`/cc-ci-server-update` or `/cc-ci-tests-update`.)
`/cc-ci-orchestrator-update` or `/cc-ci-tests-update`.)
## Preconditions
@@ -34,12 +33,12 @@ Both sub-skills' preconditions hold:
## Procedure
### 1. Run `/cc-ci-server-update` (host first)
### 1. Run `/cc-ci-orchestrator-update` (host first)
Run the full `/cc-ci-server-update` procedure: snapshot baseline → bump flake inputs (nixpkgs +
Run the full `/cc-ci-orchestrator-update` procedure: snapshot baseline → bump flake inputs (nixpkgs +
sops-nix) → commit + open cc-ci PR → stage on host → build → `nixos-rebuild switch` → **health
gate** (no failed units, infra up, endpoints 200, disk free). See
`.claude/skills/cc-ci-server-update/SKILL.md`.
`.claude/skills/cc-ci-orchestrator-update/SKILL.md`.
**Gate on the result before continuing:**
- `SUCCESS` → the server is on the new nixpkgs, healthy. Proceed to step 2.
@@ -90,7 +89,7 @@ operator review.
Inherited from both sub-skills (read their Guardrails sections). The combined-specific ones:
- **Server-first, gated.** Don't run the tests sweep on a sick/rolled-back host. The server-update
health gate is a hard go/no-go for step 2.
- **No-CI window for the rebuild** (the server half) — same as `/cc-ci-server-update`.
- **No-CI window for the rebuild** (the server half) — same as `/cc-ci-orchestrator-update`.
- **Serialize the tests half on the shared Swarm** — same as `/cc-ci-tests-update`; don't run it
concurrent with `/upgrade-all` or active build loops.
- **Never merge anything** — server PR, test PRs, paired recipe upgrade PRs all operator-merged.
README.md
+9 -3
View File
@@ -333,9 +333,15 @@ stopped on the new host while its directory is copied:
## 8. Day 2
- **Update the host** (nixpkgs bump for both halves): `/cc-ci-orchestrator-update`, which is
`nix flake update` → `nixos-rebuild test` → verify → `switch` → PR. The `cc-ci` input follows
this flake's nixpkgs, so the CI server is rebuilt on the same nixpkgs.
- **The host updates itself weekly**: `cc-ci-auto-update.timer` (Tuesday 03:00 UTC,
`nix/modules/auto-update.nix`) runs `nix flake update` on all inputs, builds, activates with
`test`, runs the cc-ci health checks (swarm replicas, front doors, sops, acme-dns, fail2ban…),
and only then commits the generation + bootloader and pushes the health-checked `flake.lock` to
`main`; on any failure it re-activates the previous generation and restores the lock. It skips
itself while a CI run, the weekly upgrader, the report or the Sunday sweep is in flight. Outcome
in `/srv/cc-ci/.cc-ci-logs/auto-update-state` (one line) and `journalctl -u cc-ci-auto-update`;
`/cc-ci-status` §5 reads it. `/cc-ci-orchestrator-update` runs the same unit by hand.
The `cc-ci` input follows this flake's nixpkgs, so the CI server is rebuilt on the same nixpkgs.
- **Update only cc-ci's code** (harness/tests/modules): merge in the cc-ci repo, then
`nix flake update cc-ci` here and rebuild; also `git -C /etc/cc-ci pull --recurse-submodules`
so the deployed checkout the sweep runs from matches. Run the lock update **on this host**: a
flake.nix
+3
View File
@@ -31,6 +31,8 @@
# The host contract those units assume: loops user, claude/opencode CLIs, opencode web
# server + tailnet UI, nix-ld, tool set, `ssh cc-ci` config.
orchestrator-host = ./nix/modules/orchestrator-host.nix;
# Weekly health-gated self-update of the host (busy-gated around CI runs).
auto-update = ./nix/modules/auto-update.nix;
# Old name of cc-ci-orchestrator, kept while notplants-nix still imports it (2026-09).
cc-ci = ./nix/modules/cc-ci.nix;
};
@@ -44,6 +46,7 @@
cc-ci.nixosModules.cc-ci-server
self.nixosModules.cc-ci-orchestrator
self.nixosModules.orchestrator-host
self.nixosModules.auto-update
./nix/hosts/cc-ci/configuration.nix
];
};
nix/hosts/cc-ci/configuration.nix
+3
View File
@@ -27,6 +27,9 @@
# ---- orchestrator identity --------------------------------------------------------------
# The CI server is this very host, so `ssh cc-ci` goes to loopback (the module default).
cc-ci-orchestrator.ciSshHost = "127.0.0.1";
# Weekly self-update (Tue 03:00 UTC; skips itself while CI is busy; see nix/modules/auto-update.nix).
cc-ci-orchestrator.autoUpdate.enable = true;
# The opencode UI: traefik (public 443, the *.ci.commoninternet.net cert) → nginx basic auth.
cc-ci-orchestrator.opencodeUiHost = "oc.ci.commoninternet.net";
cc-ci-orchestrator.opencodeUiTraefikNetwork = "proxy";
nix/modules/auto-update.nix
+199
View File
@@ -0,0 +1,199 @@
# auto-update.nix — weekly, health-gated self-update of the combined cc-ci host.
#
# Modelled on notplants-nix's modules/auto-update.nix (the nightly updater the old orchestrator
# box had), with the two things the cc-ci host needs on top:
# * a BUSY GATE — a `switch` restarts docker/traefik/drone/the runner, so the update skips
# (and simply retries next week) while a CI run, the weekly recipe-upgrade run or the
# Sunday canonical sweep is in flight;
# * cc-ci HEALTH CHECKS — beyond "sshd + no failed units": every swarm service at its replica
# count, the front doors (dashboard, reports, drone, the opencode UI) answering, sops having
# decrypted, acme-dns and fail2ban up.
#
# Sequence (as in the notplants module — read its comments for the whys):
# flake update (all inputs: nixpkgs, sops-nix, cc-ci) → nixos-rebuild build (as loops) →
# switch-to-configuration test (bootloader untouched) → settle → health check →
# on failure: re-activate the previous generation, restore flake.lock, exit 1
# on success: set the system profile + bootloader, commit flake.lock as "auto-update", push
# main (the repo stays the source of truth), refresh /etc/cc-ci.
# Nothing is committed that did not pass the health check. Never fires a missed run at boot.
# `/cc-ci-status` reads the outcome from the state file this writes.
{ config, lib, pkgs, ... }:
let
cfg = config.cc-ci-orchestrator.autoUpdate;
builderHome = config.users.users.${cfg.buildUser}.home;
stateFile = "${cfg.flakePath}/.cc-ci-logs/auto-update-state";
curl = "${pkgs.curl}/bin/curl";
busyGate = pkgs.writeShellScript "cc-ci-auto-update-busy-gate" ''
set -u
busy() { echo "BUSY: $1 — skipping this week's update"; exit 3; }
${pkgs.procps}/bin/pgrep -f run_recipe_ci >/dev/null && busy "a CI run is in flight"
${pkgs.systemd}/bin/systemctl is-active --quiet nightly-sweep.service && busy "the canonical sweep is running"
${pkgs.util-linux}/bin/runuser -u ${cfg.buildUser} -- ${pkgs.tmux}/bin/tmux has-session -t cc-ci-upgrader 2>/dev/null \
&& busy "the weekly recipe-upgrade run is in flight (tmux cc-ci-upgrader)"
${pkgs.util-linux}/bin/runuser -u ${cfg.buildUser} -- ${pkgs.tmux}/bin/tmux has-session -t cc-ci-report 2>/dev/null \
&& busy "the weekly report is being written (tmux cc-ci-report)"
if [ -r /run/secrets/bridge_drone_token ]; then
running=$(${curl} -s -m 20 -H "Authorization: Bearer $(cat /run/secrets/bridge_drone_token)" \
"https://drone.${cfg.ciDomain}/api/repos/recipe-maintainers/cc-ci/builds?per_page=10" \
| ${pkgs.gnugrep}/bin/grep -o '"status":"running"' | ${pkgs.coreutils}/bin/wc -l)
[ "''${running:-0}" -eq 0 ] || busy "$running Drone build(s) running"
fi
echo "not busy"
'';
healthCheck = pkgs.writeShellScript "cc-ci-auto-update-health-check" ''
set -u
fail() { echo "HEALTH FAIL: $1"; exit 1; }
${pkgs.systemd}/bin/systemctl is-active --quiet sshd || fail "sshd not active"
${pkgs.iproute2}/bin/ss -tlnH | ${pkgs.gnugrep}/bin/grep -q ':22 ' || fail "nothing listening on :22"
n=$(${pkgs.systemd}/bin/systemctl --failed --no-legend | ${pkgs.coreutils}/bin/wc -l)
[ "$n" -eq 0 ] || fail "$n failed unit(s): $(${pkgs.systemd}/bin/systemctl --failed --no-legend --plain | ${pkgs.gawk}/bin/awk '{print $1}' | ${pkgs.coreutils}/bin/tr '\n' ' ')"
for u in docker acme-dns fail2ban nginx opencode-web drone-runner-exec; do
${pkgs.systemd}/bin/systemctl is-active --quiet "$u" || fail "$u not active"
done
# The reconcile oneshots re-run on activation; give the swarm up to 10 minutes to converge.
for i in $(${pkgs.coreutils}/bin/seq 1 40); do
short=$(${pkgs.docker}/bin/docker service ls --format '{{.Name}} {{.Replicas}}' \
| ${pkgs.gawk}/bin/awk '{ split($2,a,"/"); if (a[1] != a[2]) print $1 }')
[ -z "$short" ] && break
${pkgs.coreutils}/bin/sleep 15
done
[ -z "$short" ] || fail "swarm services not at their replica count: $(echo "$short" | ${pkgs.coreutils}/bin/tr '\n' ' ')"
[ -s /run/secrets/test_secret ] || fail "sops did not decrypt (/run/secrets/test_secret missing)"
code() { ${curl} -s -m 20 -o /dev/null -w '%{http_code}' --resolve "$1:443:127.0.0.1" "https://$1/$2"; }
[ "$(code ${cfg.ciDomain} "")" = 200 ] || fail "dashboard not 200"
[ "$(code report.${cfg.ciDomain} "")" = 200 ] || fail "reports not 200"
[ "$(code drone.${cfg.ciDomain} version)" = 200 ] || fail "drone /version not 200"
[ "$(code ${cfg.opencodeUiHost} "")" = 401 ] || fail "opencode UI not answering with its auth challenge"
echo "HEALTH OK"
'';
in
{
options.cc-ci-orchestrator.autoUpdate = {
enable = lib.mkEnableOption "weekly health-gated self-update of the cc-ci host";
flakePath = lib.mkOption {
type = lib.types.str;
default = "/srv/cc-ci-orch";
description = "The cc-ci-orchestrator checkout (owned by buildUser) whose flake.lock is updated, committed and pushed.";
};
flakeRef = lib.mkOption {
type = lib.types.str;
default = "${cfg.flakePath}#cc-ci";
description = "Flake reference to build.";
};
buildUser = lib.mkOption {
type = lib.types.str;
default = "loops";
description = "Owner of the checkout; runs the update, build, commit and push (root activates).";
};
onCalendar = lib.mkOption {
type = lib.types.str;
default = "Tue *-*-* 03:00:00 UTC";
description = ''
When to run. Must stay clear of the weekly recipe-upgrade run (Thu 22:00 America/New_York
= Fri 02:00/03:00 UTC, several hours) and the Sunday 03:00 UTC canonical sweep (up to 6 h);
the busy gate covers the rest.
'';
};
remote = lib.mkOption { type = lib.types.str; default = "origin"; };
branch = lib.mkOption { type = lib.types.str; default = "main"; };
ciDomain = lib.mkOption {
type = lib.types.str;
default = "ci.commoninternet.net";
description = "The CI apex; report./drone. are checked under it.";
};
opencodeUiHost = lib.mkOption {
type = lib.types.str;
default = config.cc-ci-orchestrator.opencodeUiHost;
description = "The opencode UI host name (expected to answer 401).";
};
};
config = lib.mkIf cfg.enable {
systemd.services.cc-ci-auto-update = {
description = "Weekly cc-ci host self-update (flake update → build → test → health check → commit)";
after = [ "network-online.target" "docker.service" ];
wants = [ "network-online.target" ];
path = with pkgs; [ nixos-rebuild nix git openssh coreutils gnugrep gawk systemd util-linux bash python3 docker curl ];
serviceConfig = { Type = "oneshot"; TimeoutStartSec = "3h"; };
# A unit that performs the switch must never be stopped or restarted BY that switch
# (notplants-orchestrator, 2026-09-07: it was, and the host ran half-switched for 12 h).
stopIfChanged = false;
restartIfChanged = false;
script = ''
set -u
cd ${cfg.flakePath}
as_builder() { runuser -u ${cfg.buildUser} -- env HOME=${builderHome} "$@"; }
state() { printf '%s result=%s generation=%s note=%s\n' "$(date -u +%FT%TZ)" "$1" "$2" "$3" > ${stateFile}; chown ${cfg.buildUser} ${stateFile} || true; }
PREV=$(readlink -f /run/current-system)
LOCK_BAK=${cfg.flakePath}/.flake.lock.auto-update-prev
if ! ${busyGate}; then
state skipped "$PREV" busy
exit 0
fi
revert() {
echo "REVERT: $1"
as_builder cp "$LOCK_BAK" flake.lock 2>/dev/null || true
"$PREV"/bin/switch-to-configuration test || echo "re-activation of $PREV failed"
state failed "$PREV" "$1"
exit 1
}
as_builder cp flake.lock "$LOCK_BAK"
echo "=== checkout at: $(as_builder git rev-parse --short HEAD) on $(as_builder git rev-parse --abbrev-ref HEAD)"
echo "=== flake update (all inputs)"
as_builder nix flake update || revert "flake update failed"
if as_builder git diff --quiet -- flake.lock; then
echo "=== nothing moved; done"
state unchanged "$PREV" "lock already current"
exit 0
fi
as_builder git --no-pager diff --stat -- flake.lock
echo "=== build (cannot affect the running system)"
as_builder nixos-rebuild build --flake ${cfg.flakeRef} || revert "build failed"
NEW=$(readlink -f result) || revert "no build result"
as_builder rm -f result
echo "=== built: $NEW"
echo "=== activate WITHOUT touching the bootloader"
"$NEW"/bin/switch-to-configuration test || revert "test activation failed"
echo "=== settle, then health check"
sleep 45
${healthCheck} || revert "health check failed"
echo "=== healthy: committing (system profile + bootloader)"
nix-env -p /nix/var/nix/profiles/system --set "$NEW" || revert "could not set system profile"
"$NEW"/bin/switch-to-configuration boot || revert "bootloader update failed"
state ok "$NEW" "committed"
# Bookkeeping from here on — never revert a healthy, committed generation over it.
if as_builder git -c user.name="auto-update" -c user.email="auto-update@cc-ci" \
commit -q -m "flake.lock: weekly auto-update, health-checked" -- flake.lock; then
echo "=== flake.lock committed: $(as_builder git rev-parse --short HEAD)"
as_builder git push -q ${cfg.remote} HEAD:${cfg.branch} \
&& echo "=== pushed ${cfg.branch}" \
|| echo "WARNING: push failed; the lock is committed locally, push it by hand"
else
echo "WARNING: could not commit flake.lock; the system IS committed and healthy"
fi
# Keep the deployed cc-ci checkout (sweep + sops file) on the same cc-ci rev as the flake input.
git -C /etc/cc-ci pull -q --ff-only --recurse-submodules || echo "WARNING: /etc/cc-ci pull failed"
echo "=== auto-update committed: $NEW"
'';
};
systemd.timers.cc-ci-auto-update = {
wantedBy = [ "timers.target" ];
timerConfig = {
OnCalendar = cfg.onCalendar;
Persistent = false;
RandomizedDelaySec = "10min";
};
};
};
}