From a8af8429f272a1acadb736a98640bdaed20a0a5b Mon Sep 17 00:00:00 2001 From: autonomic-bot Date: Mon, 7 Sep 2026 22:22:50 +0000 Subject: [PATCH] weekly health-gated auto-update of the cc-ci host; skills rewritten for the combined host MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit nix/modules/auto-update.nix (own module, no notplants-nix dependency): Tuesday 03:00 UTC, busy-gated (CI run, weekly upgrader, report, sweep, running Drone builds → skip), `nix flake update` → build → switch-to-configuration test → cc-ci health checks (sshd, 0 failed units, core units, every swarm service at replica count, sops decrypted, dashboard/reports/drone 200, opencode UI 401) → profile + bootloader → flake.lock committed and pushed to main → /etc/cc-ci fast-forwarded; revert + lock restore on failure; one-line state file for /cc-ci-status. Skills (.opencode canonical, .claude pointers' descriptions synced): - cc-ci-orchestrator-update: THE host update — drives the auto-update unit by hand; --cc-ci-only for a cc-ci-main-only move; failure playbook. - cc-ci-server-update: delegates to it and explains why the old procedure (rebuilding the cc-ci repo's standalone #cc-ci) must not be run on this host. - cc-ci-update: chains orchestrator-update then tests-update. - cc-ci-status: §5/§6 for one host — auto-update state, generation vs boot generation, front doors (oc.ci = 401), sops, fail2ban, timers, orchestrator agent session, secrets inventory; verdict updated. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz --- .../skills/cc-ci-orchestrator-update/SKILL.md | 2 +- .claude/skills/cc-ci-server-update/SKILL.md | 2 +- .claude/skills/cc-ci-update/SKILL.md | 2 +- .../skills/cc-ci-orchestrator-update/SKILL.md | 209 ++++-------- .opencode/skills/cc-ci-server-update/SKILL.md | 307 +----------------- .opencode/skills/cc-ci-status/SKILL.md | 88 +++-- .opencode/skills/cc-ci-update/SKILL.md | 15 +- README.md | 12 +- flake.nix | 3 + nix/hosts/cc-ci/configuration.nix | 3 + nix/modules/auto-update.nix | 199 ++++++++++++ 11 files changed, 366 insertions(+), 476 deletions(-) create mode 100644 nix/modules/auto-update.nix diff --git a/.claude/skills/cc-ci-orchestrator-update/SKILL.md b/.claude/skills/cc-ci-orchestrator-update/SKILL.md index a7b2d7e..eeb8463 100644 --- a/.claude/skills/cc-ci-orchestrator-update/SKILL.md +++ b/.claude/skills/cc-ci-orchestrator-update/SKILL.md @@ -1,6 +1,6 @@ --- name: cc-ci-orchestrator-update -description: Update the cc-ci ORCHESTRATOR host (the machine the orchestrator itself runs on) to the latest nixos-26.05 nixpkgs, then nixos-rebuild test + switch + verify, ship the bump as a verified PR on recipe-maintainers/cc-ci-orchestrator and merge it directly (the skill invocation IS the authorization). The orchestrator-host analogue of /cc-ci-server-update. Self-update caveats apply — the switch restarts services on the very host running this session; always `nixos-rebuild test` before `switch` so a reboot recovers to the last-known-good generation. The end-of-run report includes the merged PR link + a summary of what changed. Invoke as /cc-ci-orchestrator-update (also answers to /cc-ci-update-orchestrator). +description: Update the cc-ci HOST — since 2026-09 the ONE Hetzner machine that runs both the cc-ci CI server and the orchestrator (built from this repo's `#cc-ci`, which imports cc-ci's server module) — to the latest nixos-26.05 nixpkgs, sops-nix and cc-ci main, then `nixos-rebuild test` → cc-ci health checks → `switch`, committing the health-checked flake.lock straight to main (the skill invocation IS the authorization; the operator reviews retrospectively). It drives the host's own weekly auto-update unit by hand, so a manual and an automatic update are the same code path. Refuses while CI is busy. `--cc-ci-only` moves only the cc-ci input (harness/tests/modules), for after a cc-ci merge. Answers to /cc-ci-orchestrator-update, /cc-ci-update-orchestrator and (since the hosts merged) /cc-ci-server-update. --- # cc-ci-orchestrator-update (thin wrapper) diff --git a/.claude/skills/cc-ci-server-update/SKILL.md b/.claude/skills/cc-ci-server-update/SKILL.md index 40ba9e5..0128b81 100644 --- a/.claude/skills/cc-ci-server-update/SKILL.md +++ b/.claude/skills/cc-ci-server-update/SKILL.md @@ -1,6 +1,6 @@ --- name: cc-ci-server-update -description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main. Opens the cc-ci PR for visibility/historical record and merges it directly once the deploy verifies (the skill invocation IS the authorization); the end-of-run report includes the merged PR link + a summary of what changed. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update. +description: (Delegates) Update the cc-ci CI server host to the latest nixpkgs — which, since 2026-09-07, is the SAME machine as the orchestrator, built from the cc-ci-orchestrator flake. This skill only forwards to /cc-ci-orchestrator-update; it no longer bumps or rebuilds the cc-ci repo's own flake, because that standalone `#cc-ci` target would replace the combined host config. Invoke as /cc-ci-server-update. --- # cc-ci-server-update (thin wrapper) diff --git a/.claude/skills/cc-ci-update/SKILL.md b/.claude/skills/cc-ci-update/SKILL.md index 5a84b8f..326f42a 100644 --- a/.claude/skills/cc-ci-update/SKILL.md +++ b/.claude/skills/cc-ci-update/SKILL.md @@ -1,6 +1,6 @@ --- name: cc-ci-update -description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update. +description: Run both cc-ci maintenance updates in one pass — first /cc-ci-orchestrator-update (the cc-ci host: nixpkgs, sops-nix and the cc-ci input, test → health check → switch, lock committed to main), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update. --- # cc-ci-update (thin wrapper) diff --git a/.opencode/skills/cc-ci-orchestrator-update/SKILL.md b/.opencode/skills/cc-ci-orchestrator-update/SKILL.md index dc4ac8c..f1f4026 100644 --- a/.opencode/skills/cc-ci-orchestrator-update/SKILL.md +++ b/.opencode/skills/cc-ci-orchestrator-update/SKILL.md @@ -1,164 +1,87 @@ --- name: cc-ci-orchestrator-update -description: Update the cc-ci ORCHESTRATOR host (the machine the orchestrator itself runs on) to the latest nixos-26.05 nixpkgs, then nixos-rebuild test + switch + verify, ship the bump as a verified PR on recipe-maintainers/cc-ci-orchestrator and merge it directly (the skill invocation IS the authorization). The orchestrator-host analogue of /cc-ci-server-update. Self-update caveats apply — the switch restarts services on the very host running this session; always `nixos-rebuild test` before `switch` so a reboot recovers to the last-known-good generation. The end-of-run report includes the merged PR link + a summary of what changed. Invoke as /cc-ci-orchestrator-update (also answers to /cc-ci-update-orchestrator). +description: Update the cc-ci HOST — since 2026-09 the ONE Hetzner machine that runs both the cc-ci CI server and the orchestrator (built from this repo's `#cc-ci`, which imports cc-ci's server module) — to the latest nixos-26.05 nixpkgs, sops-nix and cc-ci main, then `nixos-rebuild test` → cc-ci health checks → `switch`, committing the health-checked flake.lock straight to main (the skill invocation IS the authorization; the operator reviews retrospectively). It drives the host's own weekly auto-update unit by hand, so a manual and an automatic update are the same code path. Refuses while CI is busy. `--cc-ci-only` moves only the cc-ci input (harness/tests/modules), for after a cc-ci merge. Answers to /cc-ci-orchestrator-update, /cc-ci-update-orchestrator and (since the hosts merged) /cc-ci-server-update. --- # cc-ci-orchestrator-update -Manually update the **orchestrator host**'s NixOS dependencies (nixpkgs; sops-nix follows nixpkgs) -to the current `nixos-26.05` channel tip, ship the bump via a verified PR + a live -`nixos-rebuild test` → `switch`, and confirm the host is healthy afterward. This is the -orchestrator-host analogue of `/cc-ci-server-update` — done deliberately, with a reviewable PR and -a health gate, not as silent drift. +Update **the cc-ci host** — one machine since 2026-09-07 (`cc-ci`, 195.201.88.249, no tailnet): +the CI server (docker swarm, traefik, drone, runner, bridge, dashboard, reports, acme-dns) AND the +orchestrator (loops, weekly upgrader, opencode agent) — to the current inputs, verified, committed. -> **Two hosts, two flakes — don't confuse them.** This skill updates the **orchestrator** host: -> the machine this session runs on (`cc-ci-orchestrator-1`, Hetzner cpx22 **server 134487234**, -> tailnet `cc-ci`, public `195.201.88.249` — the SAME host as the cc-ci CI server since 2026-09), flake checkout **`/srv/cc-ci-orch`** (repo -> `recipe-maintainers/cc-ci-orchestrator`), target **`.#cc-ci`** (which now also rebuilds the CI server half, from the cc-ci repo flake input). The **cc-ci -> CI server** (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, target `.#cc-ci`) is a different -> machine — that's `/cc-ci-server-update`, NOT this skill. +> **One flake, one target.** The host is `nixosConfigurations.cc-ci` in +> `recipe-maintainers/cc-ci-orchestrator` (`/srv/cc-ci-orch` on the host). The cc-ci repo's own +> `flake.nix` inputs are IRRELEVANT to the live build — this flake makes the `cc-ci` input follow +> its nixpkgs/sops-nix. Never run `nixos-rebuild` on the cc-ci repo's `#cc-ci` target on this host: +> that is the standalone-server config and would replace the combined one (orchestrator half gone). -> **This is a SELF-update.** The `switch` restarts units on the host running this very session. -> tmux sessions and the orchestrator survive a switch (systemd restarts don't kill user tmux), but -> `tailscaled`/`sshd` restarts can briefly drop Remote Control / SSH — run activation steps -> detached (`setsid nohup … &`) and re-poll. A full **reboot** DOES end the session: the -> auto-launch flow (REBOOTS.md + `cc-ci-loops.service`) restarts the loops, and the operator -> reconnects the orchestrator — announce a planned reboot to the operator first. - -## What this skill does NOT do - -- **Does NOT touch the cc-ci CI server** — that's `/cc-ci-server-update`. -- **Does NOT bump recipe images or cc-ci tests** — `/upgrade-all` / `/cc-ci-tests-update`. -- **Does NOT change the channel** (`nixos-26.05` branch ref in `flake.nix`) without operator - choice — a channel jump (e.g. → 26.11 when it releases) is a bigger change: ask first, read the - release notes, and expect module/option renames. +The work is done by **`cc-ci-auto-update.service`** (`nix/modules/auto-update.nix`), the same unit +the Tuesday 03:00 UTC timer fires. Running it by hand gives you the identical sequence — +busy gate → `nix flake update` → build → `switch-to-configuration test` → 45 s settle → health +check (sshd, 0 failed units, docker/acme-dns/fail2ban/nginx/opencode-web/runner active, every swarm +service at its replica count, sops decrypted, dashboard/reports/drone 200, opencode UI 401) → +profile + bootloader → `flake.lock` committed as `auto-update` and pushed to `main` → +`/etc/cc-ci` fast-forwarded — with a journal to read and a one-line state file. ## Preconditions -- You are on the orchestrator host; `/srv/cc-ci-orch` is the live checkout (git repo, remote - `recipe-maintainers/cc-ci-orchestrator`). -- Quiescent moment: no `/upgrade-all` in flight, no active build-loop phase, no cc-ci CI run that - would be disturbed if this host blips (`pgrep -af run_recipe_ci` empty on cc-ci is a bonus, but - the orchestrator host restarting mostly risks its own sessions, not the CI server). -- `git status` in `/srv/cc-ci-orch` — note pre-existing dirt; don't sweep unrelated changes into - the bump commit. +- `ssh cc-ci` works from the notplants-orchestrator box (root on the host), or you are on the host. +- **Not busy.** The unit checks itself (CI run, weekly upgrader, report, sweep, running Drone + builds → `result=skipped note=busy`, exit 0). Don't fight it: wait, or `tmux ls` as loops and + `docker service ls` to see what is running. Never kill a run to update a host. +- Note the current generation so you can name the old one in the report: + `ssh cc-ci 'readlink /run/current-system; cat /srv/cc-ci/.cc-ci-logs/auto-update-state'`. ## Procedure -### 1. Snapshot baseline +```bash +# 1. what will move +ssh cc-ci 'cd /srv/cc-ci-orch && sudo -H -u loops nix flake update --dry-run 2>&1 | grep -v "^warning" | tail -8' # informational; the unit does the real update +git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05 | cut -c1-12 -``` -nixos-version; readlink /run/current-system; uname -r -systemctl --failed --no-legend -df -h / | tail -1 -cd /srv/cc-ci-orch && nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"' -tmux ls -``` -Record: current nixpkgs rev, generation, failed units (should be 0), disk, live tmux sessions. +# 2. run the unit and follow it (10-25 min: nixpkgs bumps rebuild the two OCI images too) +ssh cc-ci 'systemctl start --no-block cc-ci-auto-update; journalctl -fu cc-ci-auto-update' # Ctrl-C when it prints "auto-update committed" / "REVERT" / "BUSY" +# or, non-interactively: +ssh cc-ci 'systemctl start cc-ci-auto-update; systemctl is-active cc-ci-auto-update; journalctl -u cc-ci-auto-update --no-pager -n 40' -### 2. Branch + bump the lock - -Work directly in `/srv/cc-ci-orch` on a branch (it's the live checkout; the flake edit is inert -until a rebuild): -``` -cd /srv/cc-ci-orch -git checkout -b chore/orchestrator-flake-update-$(date -u +%Y%m%d) -nix flake update 2>&1 | tail -5 # nixpkgs tracks nixos-26.05; sops-nix follows nixpkgs -nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"' -``` -Note old→new revs. If the lock didn't move, report "already at channel tip" and stop (nothing to -deploy). - -### 3. Build (catch errors before any activation) - -``` -cd /srv/cc-ci-orch && nixos-rebuild build --flake .#cc-ci 2>&1 | tail -15 -readlink -f result -``` -Build failure → fix on the branch (option renames etc.) before going further. Never activate a -build that didn't build cleanly. - -### 4. Test-activate (MANDATORY before switch — general policy) - -`nixos-rebuild test` activates without touching the bootloader or system profile, so if the -activation breaks the host (cf. the cc-ci server's 2026-08-03 no-default-route outage), a plain -reboot — Hetzner API power-cycle on server **134487234** if SSH is gone (see -`hetzner-server-recovery`) — lands back on the last-known-good generation. -``` -cd /srv/cc-ci-orch && setsid nohup nixos-rebuild test --flake .#cc-ci \ - > /tmp/orchestrator-test-switch.log 2>&1 < /dev/null & echo launched -# after it settles (poll; tailscaled/sshd may blip): -readlink /run/current-system # should be the new store path -ip route show default # default route survived -systemctl --failed --no-legend # 0 failed -tmux ls # sessions survived -tail -5 /tmp/orchestrator-test-switch.log -``` -Regression here → you're still on the old boot default: diagnose, fix, or reboot to revert. Do NOT -switch. - -### 5. Switch (make permanent — only after 4 is healthy) - -``` -cd /srv/cc-ci-orch && nixos-rebuild switch --flake .#cc-ci 2>&1 | tail -10 -``` -(If it fails with "Unit nixos-rebuild-switch-to-configuration.service was already loaded", the -detached test's transient unit is still running — wait or `systemctl stop` it, then retry.) - -### 6. Health gate (do not skip) - -``` -nixos-version; readlink /run/current-system -systemctl --failed --no-legend -tmux ls # orchestrator + loop sessions intact -tailscale status | head -3 # tailnet up -systemctl list-timers cc-ci-upgrade-all.timer --no-pager | head -3 -ssh cc-ci 'hostname' 2>&1 # can still reach the CI server -df -h / | tail -1 -``` -Optional (operator's call — it ends this session): a cold-boot proof reboot. If done, the -REBOOTS.md auto-launch flow restarts the loops; the operator reconnects the orchestrator. - -### 7. Commit + PR + merge + report - -``` -cd /srv/cc-ci-orch -git add flake.lock # flake.nix too if the channel ref changed -git commit -m "flake: bump nixpkgs (nixos-26.05, $(date -u +%Y-%m-%d)) - -nixpkgs: -> (nixos-26.05 tip) -Deployed to the cc-ci host (.#cc-ci): build + test + switch + health gate green." -git push -u origin HEAD -``` -Open the PR on `recipe-maintainers/cc-ci-orchestrator` (Gitea API with the `GITEA_*` creds from -`/srv/cc-ci/.testenv`, same as `open-cc-ci-pr.sh` does for cc-ci — the helper is hardcoded to the -cc-ci repo, so use the API directly or a copy with the repo swapped). PR body: old→new revs, what -was verified. Then **merge it directly** — the PR is for visibility/historical record; the skill -invocation is the authorization. On a rolled-back/failed run, leave the PR open with an -explanatory comment. - -Back on the host, return the checkout to main so the live tree matches the merged state: -``` -git checkout main && git pull +# 3. read the outcome +ssh cc-ci 'cat /srv/cc-ci/.cc-ci-logs/auto-update-state; readlink /run/current-system; readlink /nix/var/nix/profiles/system; systemctl --failed --no-legend; cd /srv/cc-ci-orch && git log --oneline -1 && git status --short' ``` -Write `/srv/cc-ci-orch/.cc-ci-logs/orchestrator-update-.md` -(`RESULT: SUCCESS/ROLLED-BACK/FAILED …`), append a line to `cc-ci-plan/JOURNAL.md`, and report to -the operator with the **merged PR link + a summary of what changed**. +`--cc-ci-only` (after merging something in the cc-ci repo that the host should run now): +```bash +ssh cc-ci 'sudo -H -u loops bash -lc "cd /srv/cc-ci-orch && nix flake update cc-ci && git -c user.name=auto-update -c user.email=auto-update@cc-ci commit -qm \"flake.lock: cc-ci -> main\" -- flake.lock && git push -q origin HEAD:main"' +ssh cc-ci 'git -C /etc/cc-ci pull --ff-only --recurse-submodules' +ssh cc-ci 'systemd-run --unit=ccci-rebuild --collect -E HOME=/root -p WorkingDirectory=/srv/cc-ci-orch bash -lc "nixos-rebuild test --flake /srv/cc-ci-orch#cc-ci && nixos-rebuild switch --flake /srv/cc-ci-orch#cc-ci"' +# then the same step-3 checks, plus the §6 health commands from /cc-ci-status +``` + +## When it fails + +`result=failed note=` means the unit already re-activated the previous generation and +restored `flake.lock`; the host is as it was. Read `journalctl -u cc-ci-auto-update` for the +`HEALTH FAIL:` / `REVERT:` line: +- **build failed** — usually an upstream nixpkgs breakage or a cc-ci main that does not evaluate + against the new nixpkgs. Reproduce: `sudo -H -u loops nixos-rebuild build --flake /srv/cc-ci-orch#cc-ci` + in the checkout after `nix flake update`; fix in the right repo (PR, merge), re-run. +- **swarm services not at their replica count** — one of the reconcile oneshots did not converge + in 10 min: `journalctl -u deploy-`, `docker service ps `. +- **NAR hash mismatch** on the cc-ci input — the lock was made on another machine; `nix flake + update cc-ci` on the host and re-run (README day-2). +- **push failed** — bookkeeping only (the generation is committed): push the lock by hand as loops. +- A hung run: the unit has a 3 h timeout; `systemctl status cc-ci-auto-update`. Do NOT restart the + unit or the host mid-switch (see the `stopIfChanged` comment in the module). + +## Report (end of run) + +- old generation → new generation, the nixpkgs/sops-nix/cc-ci revs before → after, the lock commit + on `main`, the health check result, and anything the journal warned about. +- Then `/cc-ci-status` §5–§6 should read `ok` — say if it does not. ## Guardrails -- **Build before test, test before switch** — never `switch` without a healthy `test` activation - (general policy for this host and the cc-ci server; `test` keeps the bootloader untouched so a - reboot always recovers). -- **Detach activations** (`setsid nohup … &`) — a dropped SSH/Remote-Control connection must not - kill a half-applied activation. -- **Self-update awareness:** announce before any reboot (it ends the operator's session); a - switch is fine but expect brief tailscaled/sshd blips. -- **Single-writer:** dedicated branch; don't sweep unrelated working-tree changes into the bump - commit; return the live checkout to `main` after the merge. -- **Stable channel only** (`nixos-26.05` until the operator chooses a jump). -- **PRs for visibility, merged directly** once verified; failed runs leave the PR open. Always - report merged-PR links + change summaries. -- **Rescue path:** Hetzner server **134487234** — same rescue-mode procedure as - `hetzner-server-recovery` (grub generations live in a submenu: `1>N` ids). +- The unit is the only thing that `switch`es here; do not hand-roll `nixos-rebuild switch` unless + `--cc-ci-only` says so, and then `test` first. +- Never `nix flake update` from the notplants box and push a lock for this host (NAR hashes differ + between Nix versions) — locks for the host are made on the host. +- Nothing recipe-side: no PRs merged, no tests touched; `/cc-ci-tests-update` is separate. diff --git a/.opencode/skills/cc-ci-server-update/SKILL.md b/.opencode/skills/cc-ci-server-update/SKILL.md index fa20a9c..b119015 100644 --- a/.opencode/skills/cc-ci-server-update/SKILL.md +++ b/.opencode/skills/cc-ci-server-update/SKILL.md @@ -1,298 +1,25 @@ --- name: cc-ci-server-update -description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main. Opens the cc-ci PR for visibility/historical record and merges it directly once the deploy verifies (the skill invocation IS the authorization); the end-of-run report includes the merged PR link + a summary of what changed. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update. +description: (Delegates) Update the cc-ci CI server host to the latest nixpkgs — which, since 2026-09-07, is the SAME machine as the orchestrator, built from the cc-ci-orchestrator flake. This skill only forwards to /cc-ci-orchestrator-update; it no longer bumps or rebuilds the cc-ci repo's own flake, because that standalone `#cc-ci` target would replace the combined host config. Invoke as /cc-ci-server-update. --- -# cc-ci-server-update +# cc-ci-server-update → `/cc-ci-orchestrator-update` -Manually update the **cc-ci CI server host**'s NixOS dependencies (nixpkgs + sops-nix) to their -latest pinned revisions, ship the bump via a verified cc-ci PR + a live `nixos-rebuild switch`, and -confirm the server is healthy afterward. This is the cc-ci-server analogue of a `nix flake update` -+ rebuild — done deliberately, with a reviewable PR and a health gate, not as silent drift. +There is no separate CI-server host any more. The CI server runs on the cc-ci host +(195.201.88.249) together with the orchestrator, from **one** flake: +`recipe-maintainers/cc-ci-orchestrator` `#cc-ci`, which imports the cc-ci repo's +`nixosModules.cc-ci-server` and makes that input follow its own nixpkgs and sops-nix. -> **Two hosts, two flakes — don't confuse them.** This skill updates the **cc-ci server** flake -> (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, host target `.#cc-ci` = `nix/hosts/cc-ci-hetzner/`). -> The **orchestrator** host (`/srv/cc-ci-orch`, flake host `.#cc-ci-orchestrator-hetzner`) is a -> different machine with its own flake — it is NOT touched here. If you want to update the -> orchestrator host, that's a separate `nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner` -> from `/srv/cc-ci-orch` (the orchestrator flake already tracks `nixos-26.05` and is rebuilt by the -> operator). This skill is only about the cc-ci **server**. +**Run `/cc-ci-orchestrator-update`.** It updates nixpkgs, sops-nix AND the cc-ci input, tests, +health-checks (the CI server's swarm, front doors, sops, acme-dns — the checks this skill used to +do) and switches, via the host's own weekly auto-update unit. -## What this skill does NOT do +Do **not** do what the old version of this skill did: +- do not bump `flake.lock` in the cc-ci repo expecting it to change the live host (it cannot — the + host follows the orchestrator flake's inputs; cc-ci's lock only matters for the standalone + `#cc-ci` target, which nothing deploys); +- do not run `nixos-rebuild switch --flake /etc/cc-ci#cc-ci` or `...#cc-ci-hetzner` on the host — + that is the standalone-server configuration and would take the orchestrator half down. -- **Does NOT bump recipe image tags** — that's `/upgrade-all` / `/recipe-upgrade`. -- **Does NOT touch cc-ci tests / harness / runner** — that's `/cc-ci-tests-update` or - `/ci-dev-workflow`. This is a flake-input bump only (`flake.nix` input revs + `flake.lock`). -- **Does NOT update the orchestrator host** — only the cc-ci server. -- **Does merge the cc-ci PR itself** — the PR exists for visibility and historical record, and is - merged directly at the end of a verified run (the skill invocation is the authorization; the - operator reviews the merged PR afterwards via the link in the report). - -## Preconditions / access - -- `ssh cc-ci` works (root, tailnet `100.95.31.88`). If it doesn't, fix access first (Tailscale ACL - / `cc-ci-tailscaled` proxy) — do NOT proceed. -- **The operator's choice to invoke this skill IS the authorization for the live rebuild** — there - is no mid-run operator check to wait for. A `nixos-rebuild switch` cycles the abra reconcile - oneshots (`deploy-proxy`, `warm-keycloak`, …), so prefer a **quiescent moment** (no `!testme` / - nightly sweep running) to avoid disrupting an in-flight CI run. Check first: - ``` - ssh cc-ci 'docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"' # any unhealthy task? - ssh cc-ci 'pgrep -af run_recipe_ci | grep -v grep' # a CI run in flight? - ``` - If either is non-empty, WAIT until the host is quiescent — do not rebuild mid-CI. -- `/srv/cc-ci/.testenv` has the `GITEA_*` creds (for opening the cc-ci PR). -- The **operator-held sops secrets** are on the cc-ci host at `/etc/cc-ci/secrets/secrets.yaml` - (NOT in git). The rebuild needs them — see the rebuild procedure below. - -## Procedure - -### 1. Snapshot current state (so you can compare + roll back) - -``` -ssh cc-ci 'hostname; uptime; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \ - deploy-bridge deploy-reports deploy-backupbot drone-runner-exec; systemctl --failed --no-legend; \ - df -h / | tail -1; nixos-version; readlink /run/current-system' -``` -Record: the current `nixos-version`, the current-system store path, the list of active infra -services, and the disk usage. These are your "before" baseline. Any rebuild that leaves a unit -failed or an endpoint down gets compared against this. - -Also record the **current flake input revs** (the "before" of what you're bumping): -``` -ssh cc-ci 'cd /root/builder-clone && nix flake metadata --json 2>/dev/null | jq -r ".locks.nodes | to_entries[] | \"\(.key) \(.value.locked.rev // \"-\")\""' 2>&1 | head -``` - -### 2. Prepare a working clone of the cc-ci repo (single-writer) - -Work in a **separate clone on the orchestrator** — never push `main`, never touch the build loops' -working clones. Use the existing `/home/loops/work/cc-ci-fix` clone if present and clean, or make a -fresh one: -``` -git clone ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /home/loops/work/cc-ci-server-update -cd /home/loops/work/cc-ci-server-update -git fetch origin && git checkout main && git reset --hard origin/main -git checkout -b chore/server-flake-update-$(date -u +%Y%m%d) -``` -(If `/home/loops/work/cc-ci-fix` already exists and is on a stale branch, either reuse it on a new -branch or make the fresh clone above — the key rule is the branch is dedicated + off `main`.) - -### 3. Bump the flake inputs (deliberate, not drift) - -The cc-ci server flake **intentionally hard-pins** `nixpkgs` to a specific revision (the comment in -`flake.nix` says "Bump deliberately, not drift") and `sops-nix` to a commit compatible with that -nixpkgs. So the bump is a **considered move to a newer nixpkgs + a compatible sops-nix**, not a -blind `nix flake update --relock-all`. - -**(a) Decide the new nixpkgs.** The cc-ci server runs recipe CI; nixpkgs provides docker, abra's -runtime, traefik, keycloak, drone, the harness python/pytest/playwright toolchain. Pick the newest -**stable channel revision** that's compatible: -- Default: move to the latest revision of the **same channel** the orchestrator host tracks - (`nixos-26.05`), so both hosts stay on a shared nixpkgs and CVEs get patched. Look up the current - tip: `git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05` (the SHA after the ref). -- If a newer stable channel is out and the operator wants it, use that — but treat a **channel - jump** (e.g. 26.05 → 26.11) as a bigger change: scan the release notes for breaking changes - (docker, systemd, python, nginx module option renames) and surface them in the PR body. Default - to staying on the same channel unless the operator asks for the jump. -- **Never** jump to `nixos-unstable` for the cc-ci server — recipe CI reproducibility depends on a - stable nixpkgs. Unstable is for experiments, not production. - -**(b) Update `flake.nix`** with the new nixpkgs rev (replace the `rev = …` in the `nixpkgs.url`), -and check whether `sops-nix` needs a matching bump: -- sops-nix `inputs.nixpkgs.follows = "nixpkgs"`, so it tracks nixpkgs automatically — but its own - rev pin may need moving if the pinned sops-nix commit is incompatible with the new nixpkgs (e.g. - the `buildGoModule` → `buildGo125Module` gotcha the current pin's comment calls out). After - editing `flake.nix`, run `nix flake lock --update-input nixpkgs --update-input sops-nix` (or - `nix flake update` if both follow) in the clone and let nix resolve a compatible sops-nix. If the - lock fails (sops-nix master needs a newer nixpkgs than you picked), pick a sops-nix commit that - still builds against your chosen nixpkgs (its `flake.nix` declares the nixpkgs it expects). -- **Keep the `sops-nix.inputs.nixpkgs.follows = "nixpkgs"` line** — that's what keeps them aligned. -- Update the comment in `flake.nix` to reflect the new pin date + reason (the existing comment - style: "Pinned to … (operator YYYY-MM-DD)" or "Follows nixos-26.05 channel"). - -**(c) Re-lock + sanity-build locally first** (cheap, catches a broken input before touching the -host): -``` -cd /home/loops/work/cc-ci-server-update # (or cc-ci-fix on the new branch) -nix flake lock --update-input nixpkgs --update-input sops-nix # rewrites flake.lock -nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"' -git diff flake.nix flake.lock | head -40 -``` -Verify the lock moved to the revs you intended. **Do NOT `nixos-rebuild build` from the -orchestrator** — the orchestrator's nixpkgs is different and the cc-ci host target won't build here -(meaningfully). The real build happens on the cc-ci host in step 5. - -### 4. Commit + open the cc-ci PR (merged in step 7 after the health gate) - -``` -git add flake.nix flake.lock -git commit -m "flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d)) - -nixpkgs: → (nixos-) -sops-nix: → -" -git push -u origin HEAD -``` -Open the PR via the helper: -``` -set -a; . /srv/cc-ci/.testenv; set +a -TITLE="flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))" \ -BODY_FILE=/tmp/cc-ci-server-update-pr-body.md \ -bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh -``` -(Write the PR body to `/tmp/cc-ci-server-update-pr-body.md` first: old→new revs, the channel, any -breaking changes from the release notes, and the rebuild plan. The PR is opened for visibility and -historical record; it gets merged directly in step 7 once the deploy passes the health gate.) - -Capture the `PR_URL`. **The cc-ci push-build lint gate will run on the branch** — if it's red on a -flake-lock-only change, it's a pre-existing lint drift (not yours); note it in the PR body and -proceed (the rebuild doesn't depend on the lint gate passing). - -### 5. Deploy to the live cc-ci host (no-CI window) - -This is the host-rebuild procedure from memory `cc-ci-host-rebuild-procedure` — the cc-ci server -has **no self-service rebuild path**, so the orchestrator stages the repo + rebuilds it. The -deploy clone on the host is `/root/builder-clone`; staging a fresh copy avoids disturbing it. - -**(a) Stage the PR branch on the host:** -``` -# from the orchestrator, on the PR branch, clean tree: -rsync -a --delete --exclude='.git' /home/loops/work/cc-ci-server-update/ root@cc-ci:/root/cc-ci-deploy/ -ssh cc-ci 'chown -R root:root /root/cc-ci-deploy' -``` -(A `--delete` rsync from a clean tree = exactly the PR head. Excluding `.git` keeps it a plain path -flake — see (c).) - -**(b) Copy the operator-held sops secrets** (NOT in git, required for the build): -``` -ssh cc-ci 'mkdir -p /root/cc-ci-deploy/secrets && cp /etc/cc-ci/secrets/secrets.yaml /root/cc-ci-deploy/secrets/secrets.yaml && chmod 600 /root/cc-ci-deploy/secrets/secrets.yaml' -``` -(The age key is at `/var/lib/sops-nix/key.txt` on the host — already in place from prior rebuilds. -Without `secrets/secrets.yaml` the build FATAs `secrets/secrets.yaml does not exist`.) - -**(c) Build first (catch errors before switching):** -``` -ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild build --flake .#cc-ci' 2>&1 | tail -30 -``` -- Target is `.#cc-ci` (= `.#cc-ci-hetzner` = `nix/hosts/cc-ci-hetzner/configuration.nix`). -- nixpkgs is now your bumped pin, so this **will** rebuild more than a same-pin no-op — expect a - longer build (docker/traefik/keycloak/python toolchain may all rebuild against the new nixpkgs). - That's expected; this is a real nixpkgs bump. -- If the build fails: read the error, fix on the PR branch (e.g. a renamed option, a removed - package), re-push, re-stage, re-build. Do NOT `switch` a failed build. Common breakages: NixOS - module option renames across channels (check the release notes), python package removals, a - sops-nix incompatibility (go back to step 3b and pick a compatible sops-nix rev). - -**(d) Test-activate FIRST (general policy for nix deploys to this server AND the orchestrator -host): `nixos-rebuild test` before any `switch`:** -``` -# detach it so a dropped SSH can't kill the activation mid-flight: -ssh cc-ci 'cd /root/cc-ci-deploy && setsid nohup nixos-rebuild test --flake .#cc-ci \ - > /root/cc-ci-deploy/test-switch.log 2>&1 < /dev/null & echo launched' -# poll until reachable again, then check the essentials survived: -ssh cc-ci 'readlink /run/current-system; ip route show default; systemctl --failed --no-legend' -``` -`test` activates the new configuration **without touching the bootloader or the system profile**, -so if the activation kills networking (as the 2026-08-03 26.05 bump did — no default route), a -plain reboot (Hetzner API power-cycle if SSH is gone) lands back on the last-known-good -generation. Only proceed to `switch` once the test activation is up, reachable, and healthy. -Note: the detached test leaves a transient `nixos-rebuild-switch-to-configuration` unit; if the -follow-up `switch` fails with "Unit ... was already loaded", wait for it to finish or -`systemctl stop nixos-rebuild-switch-to-configuration.service` first. - -**(e) Switch (make it permanent — only after (d) verified):** -``` -ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild switch --flake .#cc-ci' 2>&1 | tail -30 -``` -This is the moment the server moves to the new nixpkgs. The abra reconcile oneshots -(`deploy-proxy`, `warm-keycloak`, …) re-run as part of the switch — watch the tail for any that -fail. If a reconcile unit fails, do NOT roll back blindly — diagnose first (a warm-keycloak blip -may be transient; a proxy failure is real). - -### 6. Verify the server is healthy (health gate — do not skip) - -``` -ssh cc-ci 'echo "=== version ==="; nixos-version; readlink /run/current-system; \ - echo "=== failed units ==="; systemctl --failed --no-legend; \ - echo "=== infra services ==="; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \ - deploy-bridge deploy-reports deploy-backupbot drone-runner-exec | paste - - - - - - -; \ - echo "=== docker stacks ==="; docker stack ls --format "{{.Name}} {{.Services}}"; \ - echo "=== service replicas (any 0/N?) ==="; docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"; \ - echo "=== endpoints ==="; curl -sS -o /dev/null -w "traefik:%{http_code}\n" https://traefik.ci.commoninternet.net/api/version; \ - curl -sS -o /dev/null -w "dashboard:%{http_code}\n" https://ci.commoninternet.net/; \ - echo "=== disk ==="; df -h / | tail -1' -``` -Compare against the step-1 baseline: -- `nixos-version` moved (or the current-system path changed) — the bump landed. -- **No failed units** (the `--failed` output is empty). -- All 7 infra services `active` (or `active (exited)` for the oneshots). -- `docker service ls` shows no `0/N` replicas (everything converged). -- Endpoints `200` (traefik `/api/version`, dashboard root). -- Disk didn't spike (a nixpkgs bump adds a new system generation; old ones are GC'd eventually). - -If **anything regressed**: diagnose. A real regression → roll back to the previous generation: -`ssh cc-ci 'nixos-rebuild switch --rollback'` (this switches to the pre-bump system path; the -flake input change on the PR branch is still your source of truth, but the running host reverts -while you fix). Report the rollback in the PR. - -### 7. Merge the PR + report - -On a **verified** run (health gate green): post a PR comment summarizing what was deployed (revs, -any incidents, verification results), then **merge the PR directly** — the skill invocation is the -authorization; the PR exists for visibility and historical record. On a rolled-back or failed run, -leave the PR **open** with a comment explaining the state. - -Print + record a one-line result + the PR URL. Write a short note to -`/srv/cc-ci/.cc-ci-logs/server-update-.md`: -- `RESULT: SUCCESS — cc-ci server rebuilt on nixpkgs , no failed units, endpoints 200, PR merged: ` -- `RESULT: ROLLED-BACK — nixpkgs rebuild failed/ regressed (); host reverted to ; PR left open` -- `RESULT: FAILED — : ` - -The end-of-run report to the operator MUST include **links to every merged PR** plus a summary of -what changed in each, so the operator can review them after the fact if they want to. - -### 8. Optional cold-boot proof - -A `nixos-rebuild switch` proves the new generation works on the running kernel. A **reboot** proves -it boots cleanly from cold — the stronger guarantee (cf. the pxgate cold-boot proof). Only do this -if the operator wants it, and only outside a CI window: -``` -ssh cc-ci 'systemctl reboot' -# wait ~60s, re-check ssh cc-ci + the step-6 health gate -``` -Record the new `boot_id` if you do this (it's a genuine reboot — log it to -`cc-ci-plan/REBOOTS.md`? No — that file is for the **orchestrator** host. The cc-ci server's reboots -are tracked in its own journal; a note in the server-update report is enough). - -## Guardrails - -- **Live rebuild is authorized by invocation.** The operator's choice to run this skill IS the - go-ahead for the `nixos-rebuild switch` — do not stop mid-run to re-confirm. A switch cycles - reconcile oneshots and can disrupt an in-flight `!testme`, so prefer a quiescent moment (the - step-1 check), but don't gate on a second authorization. -- **Single-writer:** dedicated branch in a separate clone; never push `main`; never touch the - build loops' `/cc-ci` / `/cc-ci-adv` clones or `/root/builder-clone` (stage to - `/root/cc-ci-deploy` instead). -- **Open the PR for visibility, merge it directly.** The live rebuild is the deploy; the PR is the - reviewable historical record. Merge only after the health gate passes; a rolled-back/failed run - leaves the PR open with an explanatory comment. Always report merged-PR links + change summaries. -- **Build before test, test before switch.** Never `nixos-rebuild switch` a build that didn't build - cleanly first, and never `switch` without a healthy `nixos-rebuild test` activation first — `test` - leaves the bootloader untouched, so a reboot always recovers to the last-known-good generation. - This is general policy for nix deploys to the cc-ci server **and** the orchestrator host. -- **Health gate is mandatory.** No "it switched, ship it" — run the step-6 verification and compare - to the baseline. Roll back on regression. -- **Stable channel only** for the cc-ci server (never `nixos-unstable`). -- **Keep sops-nix.follows** so the two inputs stay aligned; don't fork them. -- **Secrets never in git.** The operator-held `secrets/secrets.yaml` is copied on the host only. -- **Rollback is always available:** `nixos-rebuild switch --rollback` reverts to the prior - generation without touching the flake input revs. - -## Relationship to an automatic update - -This skill is the **manual trigger** for what should ideally run on a periodic timer: bump the cc-ci -flake inputs, build, switch, verify. If/when an automatic `cc-ci-server-update.timer` is added -(NixOS systemd timer on the orchestrator, analog of `cc-ci-upgrade-all.timer`), it should invoke -this skill via a dedicated one-shot agent (the way `cc-ci-upgrader` runs `/upgrade-all`). The -skill stays as the authoritative procedure either way — automatic just means "run on a schedule -without a human prompt." +If what you actually wanted is "the host should run cc-ci `main` as of now" (a harness or test +change), that is `/cc-ci-orchestrator-update --cc-ci-only`. diff --git a/.opencode/skills/cc-ci-status/SKILL.md b/.opencode/skills/cc-ci-status/SKILL.md index 6c58cd7..e1dff78 100644 --- a/.opencode/skills/cc-ci-status/SKILL.md +++ b/.opencode/skills/cc-ci-status/SKILL.md @@ -58,35 +58,62 @@ For each open PR: age (now − created_at), and whether the PR title/body/report - Verified-green PRs awaiting operator merge are normal — list them as "ready to merge", not as failures. -### 5. Server + orchestrator host update recency +### 5. The cc-ci host: update recency + the weekly auto-update -For BOTH hosts: -``` -# cc-ci server: ssh cc-ci 'nixos-version; cd /root/cc-ci-deploy && nix flake metadata --json' (or builder-clone) -# orchestrator: nixos-version; cd /srv/cc-ci-orch && nix flake metadata --json -git ls-remote https://github.com/NixOS/nixpkgs # current tip -ls -t /srv/cc-ci-orch/.cc-ci-logs/server-update-*.md /srv/cc-ci-orch/.cc-ci-logs/orchestrator-update-*.md | head -2 -``` -- Report: days since last update log + how far the running nixpkgs rev lags the channel tip. -- **Lagging >30 days** (or a NixOS release behind) → recommend `/cc-ci-server-update` / - `/cc-ci-orchestrator-update`. - -### 6. Host + service health (both machines) +Since 2026-09-07 the CI server and the orchestrator are ONE Hetzner host (`cc-ci`, +195.201.88.249, no tailnet), built from `recipe-maintainers/cc-ci-orchestrator`'s +`nixosConfigurations.cc-ci` (which imports cc-ci's `nixosModules.cc-ci-server`). From the +notplants-orchestrator box `ssh cc-ci` reaches it as root; on the host itself `ssh cc-ci` is loopback. ``` -ssh cc-ci 'systemctl --failed --no-legend; df -h / | tail -1; docker service ls --format "{{.Name}} {{.Replicas}}"' -systemctl --failed --no-legend; df -h / | tail -1; tmux ls +ssh cc-ci 'nixos-version; readlink /run/current-system; readlink /nix/var/nix/profiles/system' +ssh cc-ci 'cd /srv/cc-ci-orch && nix flake metadata --json | jq -r ".locks.nodes | to_entries[] | \"\(.key) \(.value.locked.rev // \"-\") \(.value.locked.lastModified // \"-\")\""' +git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05 | cut -c1-12 # channel tip +# the weekly self-update (nix/modules/auto-update.nix, Tue 03:00 UTC, busy-gated): +ssh cc-ci 'cat /srv/cc-ci/.cc-ci-logs/auto-update-state; systemctl list-timers --no-pager cc-ci-auto-update.timer | sed -n 2p; journalctl -u cc-ci-auto-update --no-pager -n 3' ``` -- Failed units, core swarm services not 1/1 (warm-* spares flapping is a known benign pattern — - note, don't page), disk **>65% (server)** / >85% (orchestrator) → findings. Server unreachable → - HIGH: recommend `hetzner-server-recovery`. - > **65%, not 80%, on the server — it is not a steady-state measure.** Two concurrent recipe runs - > pull images and write volumes worth tens of GB, so a host sitting at 73% still hits 100% mid-run. - > That is exactly what happened on 2026-08-11: 63GB of unused images had accumulated (nothing ever - > pruned them), the filesystem filled during a run, and the harness died at startup with - > `OSError: [Errno 28] No space left on device`. Remedy: `docker image prune -af` on cc-ci — it - > spares anything a container references, so infra and warm-* canonicals are untouched. Do NOT - > `docker volume prune`: warm-* canonical volumes are data-warm and legitimately dangling. +- The state file is one line: ` result=ok|unchanged|skipped|failed generation= note=`. + - `ok` / `unchanged` within the last 8 days → healthy. + - `skipped` (busy) two weeks in a row, `failed`, or no state line / timer inactive for >8 days → + finding: *"host auto-update since : "* → recommend + `/cc-ci-orchestrator-update` (it runs the same unit by hand and reads its journal). + - `failed` also means the host is running the previous generation and `flake.lock` was restored — + say so; the journal (`journalctl -u cc-ci-auto-update`) names the health check that failed. +- Running `/run/current-system` must equal the profile's boot generation (`system` link) — a + mismatch means a `test`-only activation was left behind (finding: switch or reboot decision). +- nixpkgs rev lagging the `nixos-26.05` channel tip by >30 days → finding → `/cc-ci-orchestrator-update`. +- `cc-ci` (the CI server input) lagging cc-ci `main` → note; the auto-update moves it weekly, or + `/cc-ci-orchestrator-update --cc-ci-only`. + +### 6. Host + service health (one machine) + +``` +ssh cc-ci 'systemctl --failed --no-legend; df -h / | tail -1; free -m | sed -n 2p' +ssh cc-ci 'docker service ls --format "{{.Name}} {{.Replicas}}"; docker stack ls --format "{{.Name}}"' +ssh cc-ci 'for h in ci report.ci drone.ci oc.ci; do printf "%s " $h; curl -s -o /dev/null -w "%{http_code} ssl=%{ssl_verify_result}\n" https://$h.commoninternet.net/; done' +ssh cc-ci 'ls -s /run/secrets/test_secret; systemctl is-active acme-dns fail2ban nginx opencode-web drone-runner-exec; fail2ban-client status sshd | grep -E "Currently banned|Total banned"; fail2ban-client status nginx-http-auth | grep -E "Currently banned|Total banned"' +ssh cc-ci 'systemctl list-timers --no-pager cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer cc-ci-auto-update.timer nightly-sweep.timer | tail -n +2 | head -4' +ssh cc-ci 'sudo -iu loops tmux ls' # cc-ci-orchestrator must exist (the opencode orchestrator agent); cc-ci-upgrader only during a run +``` +- Failed units, swarm services not at their replica count (`warm-*` spares flapping is benign — + note, don't page), a front door not 200 (`oc.ci` must be **401**: basic auth in front of the + opencode UI; 404 there = the `ccci-opencode-ui` stack is gone, `systemctl restart opencode-ui-route` + and check the weekly sweep's keep-list), `ssl≠0` (cert), `/run/secrets` empty (sops), acme-dns / + fail2ban / nginx / opencode-web / drone-runner-exec inactive, any timer missing, the orchestrator + tmux session missing (`systemctl restart cc-ci-orchestrator`) → findings. Disk **>65%** → finding. + Host unreachable → HIGH: `hetzner-server-recovery` (token in `/srv/cc-ci/.hcloud-token` on the + notplants box while the operator keeps it active). + > **65%, not 80% — it is not a steady-state measure.** Two concurrent recipe runs pull images and + > write volumes worth tens of GB, so a host sitting at 73% still hits 100% mid-run (2026-08-11: + > 63GB of unused images, `OSError: [Errno 28] No space left on device`). Remedy: `docker image + > prune -af` — it spares anything a container references. Do NOT `docker volume prune`: warm-* + > canonical volumes are data-warm and legitimately dangling. +- fail2ban: a non-zero "Currently banned" is information, not a finding (that is the jail + working); the notplants box's own address showing up banned IS a finding (it means a skill + misbehaved) — `fail2ban-client set unbanip `. +- Secrets hygiene: `ssh cc-ci 'ls /secrets/files'` must list ONLY cc-ci.testenv, cc-ci-local-ed25519(.pub), + autonomic-bot-cc-ci-ed25519(.pub), opencode-auth.json. Anything else there (a tailscale key, a + master age key, another host's ssh key…) → finding: *"secret sprawl on the cc-ci host"*. - **!testme actually produces results** (the check that would have caught the above days earlier): the newest few `/var/lib/cc-ci-runs//` dirs must each contain `results.json`. A build that dies before the harness writes one leaves an EMPTY dir — and the PR badge still says "failure", so @@ -143,14 +170,17 @@ minutes, no PRs). If it is instead that a known CVE is sitting unpatched, recomm `/cve-check` over waiting for the next weekly run whenever the question is "are we exposed?". `ALL HEALTHY` requires: recent successful weekly run + published report, no stale tests, no -CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, recent builds all -producing results.json, disk under -thresholds, bridge clean, maintained-set consistent. Anything else is a finding — even minor +CVE PR open >14 days, the host <30 days behind its channel with the weekly auto-update `ok`/`unchanged` +within 8 days and running generation == boot generation, zero failed units, every front door +answering (oc.ci with 401), sops decrypted, fail2ban up, the orchestrator agent session present, +recent builds all producing results.json, disk under threshold, bridge clean, maintained-set +consistent, secrets inventory exact. Anything else is a finding — even minor ones get a recommended next step. Order findings by priority (CVE/unreachable-host first). ## Guardrails - **Read-only.** This skill diagnoses and recommends; it invokes nothing and changes nothing. -- Use the pseudo-TTY wrap for any abra call; plain ssh for everything else. +- Use the pseudo-TTY wrap for any abra call; plain ssh for everything else. `ssh cc-ci` from the + notplants box is root on 195.201.88.249 (loops ssh config); there is no tailnet. - Don't double-count: a finding that explains another (bridge 401 → !testme "failures") gets reported once, at the root cause. diff --git a/.opencode/skills/cc-ci-update/SKILL.md b/.opencode/skills/cc-ci-update/SKILL.md index 23fe271..bf59869 100644 --- a/.opencode/skills/cc-ci-update/SKILL.md +++ b/.opencode/skills/cc-ci-update/SKILL.md @@ -1,12 +1,11 @@ --- name: cc-ci-update -description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update. +description: Run both cc-ci maintenance updates in one pass — first /cc-ci-orchestrator-update (the cc-ci host: nixpkgs, sops-nix and the cc-ci input, test → health check → switch, lock committed to main), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update. --- # cc-ci-update -The combined **keep cc-ci itself current** pass: run `/cc-ci-server-update` (host dependency bump + -rebuild) **then** `/cc-ci-tests-update` (stale-test sweep + fixes), in that order, end-to-end. This +The combined **keep cc-ci itself current** pass: run `/cc-ci-orchestrator-update` (host dependency bump + health-checked rebuild) **then** `/cc-ci-tests-update` (stale-test sweep + fixes), in that order, end-to-end. This is just sequencing the two skills — it doesn't add or change any behavior. Read each sub-skill's SKILL.md for the full procedure; this file is the orchestration + the why-of-the-order. @@ -21,7 +20,7 @@ SKILL.md for the full procedure; this file is the orchestration + the why-of-the ## Arguments None. (If you only want one of the two updates, invoke the sub-skill directly: -`/cc-ci-server-update` or `/cc-ci-tests-update`.) +`/cc-ci-orchestrator-update` or `/cc-ci-tests-update`.) ## Preconditions @@ -34,12 +33,12 @@ Both sub-skills' preconditions hold: ## Procedure -### 1. Run `/cc-ci-server-update` (host first) +### 1. Run `/cc-ci-orchestrator-update` (host first) -Run the full `/cc-ci-server-update` procedure: snapshot baseline → bump flake inputs (nixpkgs + +Run the full `/cc-ci-orchestrator-update` procedure: snapshot baseline → bump flake inputs (nixpkgs + sops-nix) → commit + open cc-ci PR → stage on host → build → `nixos-rebuild switch` → **health gate** (no failed units, infra up, endpoints 200, disk free). See -`.claude/skills/cc-ci-server-update/SKILL.md`. +`.claude/skills/cc-ci-orchestrator-update/SKILL.md`. **Gate on the result before continuing:** - `SUCCESS` → the server is on the new nixpkgs, healthy. Proceed to step 2. @@ -90,7 +89,7 @@ operator review. Inherited from both sub-skills (read their Guardrails sections). The combined-specific ones: - **Server-first, gated.** Don't run the tests sweep on a sick/rolled-back host. The server-update health gate is a hard go/no-go for step 2. -- **No-CI window for the rebuild** (the server half) — same as `/cc-ci-server-update`. +- **No-CI window for the rebuild** (the server half) — same as `/cc-ci-orchestrator-update`. - **Serialize the tests half on the shared Swarm** — same as `/cc-ci-tests-update`; don't run it concurrent with `/upgrade-all` or active build loops. - **Never merge anything** — server PR, test PRs, paired recipe upgrade PRs all operator-merged. diff --git a/README.md b/README.md index e941853..4422c69 100644 --- a/README.md +++ b/README.md @@ -333,9 +333,15 @@ stopped on the new host while its directory is copied: ## 8. Day 2 -- **Update the host** (nixpkgs bump for both halves): `/cc-ci-orchestrator-update`, which is - `nix flake update` → `nixos-rebuild test` → verify → `switch` → PR. The `cc-ci` input follows - this flake's nixpkgs, so the CI server is rebuilt on the same nixpkgs. +- **The host updates itself weekly**: `cc-ci-auto-update.timer` (Tuesday 03:00 UTC, + `nix/modules/auto-update.nix`) runs `nix flake update` on all inputs, builds, activates with + `test`, runs the cc-ci health checks (swarm replicas, front doors, sops, acme-dns, fail2ban…), + and only then commits the generation + bootloader and pushes the health-checked `flake.lock` to + `main`; on any failure it re-activates the previous generation and restores the lock. It skips + itself while a CI run, the weekly upgrader, the report or the Sunday sweep is in flight. Outcome + in `/srv/cc-ci/.cc-ci-logs/auto-update-state` (one line) and `journalctl -u cc-ci-auto-update`; + `/cc-ci-status` §5 reads it. `/cc-ci-orchestrator-update` runs the same unit by hand. + The `cc-ci` input follows this flake's nixpkgs, so the CI server is rebuilt on the same nixpkgs. - **Update only cc-ci's code** (harness/tests/modules): merge in the cc-ci repo, then `nix flake update cc-ci` here and rebuild; also `git -C /etc/cc-ci pull --recurse-submodules` so the deployed checkout the sweep runs from matches. Run the lock update **on this host**: a diff --git a/flake.nix b/flake.nix index 517e491..12d19a1 100644 --- a/flake.nix +++ b/flake.nix @@ -31,6 +31,8 @@ # The host contract those units assume: loops user, claude/opencode CLIs, opencode web # server + tailnet UI, nix-ld, tool set, `ssh cc-ci` config. orchestrator-host = ./nix/modules/orchestrator-host.nix; + # Weekly health-gated self-update of the host (busy-gated around CI runs). + auto-update = ./nix/modules/auto-update.nix; # Old name of cc-ci-orchestrator, kept while notplants-nix still imports it (2026-09). cc-ci = ./nix/modules/cc-ci.nix; }; @@ -44,6 +46,7 @@ cc-ci.nixosModules.cc-ci-server self.nixosModules.cc-ci-orchestrator self.nixosModules.orchestrator-host + self.nixosModules.auto-update ./nix/hosts/cc-ci/configuration.nix ]; }; diff --git a/nix/hosts/cc-ci/configuration.nix b/nix/hosts/cc-ci/configuration.nix index 4a359c1..3bc00fb 100644 --- a/nix/hosts/cc-ci/configuration.nix +++ b/nix/hosts/cc-ci/configuration.nix @@ -27,6 +27,9 @@ # ---- orchestrator identity -------------------------------------------------------------- # The CI server is this very host, so `ssh cc-ci` goes to loopback (the module default). cc-ci-orchestrator.ciSshHost = "127.0.0.1"; + # Weekly self-update (Tue 03:00 UTC; skips itself while CI is busy; see nix/modules/auto-update.nix). + cc-ci-orchestrator.autoUpdate.enable = true; + # The opencode UI: traefik (public 443, the *.ci.commoninternet.net cert) → nginx basic auth. cc-ci-orchestrator.opencodeUiHost = "oc.ci.commoninternet.net"; cc-ci-orchestrator.opencodeUiTraefikNetwork = "proxy"; diff --git a/nix/modules/auto-update.nix b/nix/modules/auto-update.nix new file mode 100644 index 0000000..f1be5c6 --- /dev/null +++ b/nix/modules/auto-update.nix @@ -0,0 +1,199 @@ +# auto-update.nix — weekly, health-gated self-update of the combined cc-ci host. +# +# Modelled on notplants-nix's modules/auto-update.nix (the nightly updater the old orchestrator +# box had), with the two things the cc-ci host needs on top: +# * a BUSY GATE — a `switch` restarts docker/traefik/drone/the runner, so the update skips +# (and simply retries next week) while a CI run, the weekly recipe-upgrade run or the +# Sunday canonical sweep is in flight; +# * cc-ci HEALTH CHECKS — beyond "sshd + no failed units": every swarm service at its replica +# count, the front doors (dashboard, reports, drone, the opencode UI) answering, sops having +# decrypted, acme-dns and fail2ban up. +# +# Sequence (as in the notplants module — read its comments for the whys): +# flake update (all inputs: nixpkgs, sops-nix, cc-ci) → nixos-rebuild build (as loops) → +# switch-to-configuration test (bootloader untouched) → settle → health check → +# on failure: re-activate the previous generation, restore flake.lock, exit 1 +# on success: set the system profile + bootloader, commit flake.lock as "auto-update", push +# main (the repo stays the source of truth), refresh /etc/cc-ci. +# Nothing is committed that did not pass the health check. Never fires a missed run at boot. +# `/cc-ci-status` reads the outcome from the state file this writes. +{ config, lib, pkgs, ... }: +let + cfg = config.cc-ci-orchestrator.autoUpdate; + builderHome = config.users.users.${cfg.buildUser}.home; + stateFile = "${cfg.flakePath}/.cc-ci-logs/auto-update-state"; + curl = "${pkgs.curl}/bin/curl"; + + busyGate = pkgs.writeShellScript "cc-ci-auto-update-busy-gate" '' + set -u + busy() { echo "BUSY: $1 — skipping this week's update"; exit 3; } + ${pkgs.procps}/bin/pgrep -f run_recipe_ci >/dev/null && busy "a CI run is in flight" + ${pkgs.systemd}/bin/systemctl is-active --quiet nightly-sweep.service && busy "the canonical sweep is running" + ${pkgs.util-linux}/bin/runuser -u ${cfg.buildUser} -- ${pkgs.tmux}/bin/tmux has-session -t cc-ci-upgrader 2>/dev/null \ + && busy "the weekly recipe-upgrade run is in flight (tmux cc-ci-upgrader)" + ${pkgs.util-linux}/bin/runuser -u ${cfg.buildUser} -- ${pkgs.tmux}/bin/tmux has-session -t cc-ci-report 2>/dev/null \ + && busy "the weekly report is being written (tmux cc-ci-report)" + if [ -r /run/secrets/bridge_drone_token ]; then + running=$(${curl} -s -m 20 -H "Authorization: Bearer $(cat /run/secrets/bridge_drone_token)" \ + "https://drone.${cfg.ciDomain}/api/repos/recipe-maintainers/cc-ci/builds?per_page=10" \ + | ${pkgs.gnugrep}/bin/grep -o '"status":"running"' | ${pkgs.coreutils}/bin/wc -l) + [ "''${running:-0}" -eq 0 ] || busy "$running Drone build(s) running" + fi + echo "not busy" + ''; + + healthCheck = pkgs.writeShellScript "cc-ci-auto-update-health-check" '' + set -u + fail() { echo "HEALTH FAIL: $1"; exit 1; } + ${pkgs.systemd}/bin/systemctl is-active --quiet sshd || fail "sshd not active" + ${pkgs.iproute2}/bin/ss -tlnH | ${pkgs.gnugrep}/bin/grep -q ':22 ' || fail "nothing listening on :22" + n=$(${pkgs.systemd}/bin/systemctl --failed --no-legend | ${pkgs.coreutils}/bin/wc -l) + [ "$n" -eq 0 ] || fail "$n failed unit(s): $(${pkgs.systemd}/bin/systemctl --failed --no-legend --plain | ${pkgs.gawk}/bin/awk '{print $1}' | ${pkgs.coreutils}/bin/tr '\n' ' ')" + for u in docker acme-dns fail2ban nginx opencode-web drone-runner-exec; do + ${pkgs.systemd}/bin/systemctl is-active --quiet "$u" || fail "$u not active" + done + # The reconcile oneshots re-run on activation; give the swarm up to 10 minutes to converge. + for i in $(${pkgs.coreutils}/bin/seq 1 40); do + short=$(${pkgs.docker}/bin/docker service ls --format '{{.Name}} {{.Replicas}}' \ + | ${pkgs.gawk}/bin/awk '{ split($2,a,"/"); if (a[1] != a[2]) print $1 }') + [ -z "$short" ] && break + ${pkgs.coreutils}/bin/sleep 15 + done + [ -z "$short" ] || fail "swarm services not at their replica count: $(echo "$short" | ${pkgs.coreutils}/bin/tr '\n' ' ')" + [ -s /run/secrets/test_secret ] || fail "sops did not decrypt (/run/secrets/test_secret missing)" + code() { ${curl} -s -m 20 -o /dev/null -w '%{http_code}' --resolve "$1:443:127.0.0.1" "https://$1/$2"; } + [ "$(code ${cfg.ciDomain} "")" = 200 ] || fail "dashboard not 200" + [ "$(code report.${cfg.ciDomain} "")" = 200 ] || fail "reports not 200" + [ "$(code drone.${cfg.ciDomain} version)" = 200 ] || fail "drone /version not 200" + [ "$(code ${cfg.opencodeUiHost} "")" = 401 ] || fail "opencode UI not answering with its auth challenge" + echo "HEALTH OK" + ''; +in +{ + options.cc-ci-orchestrator.autoUpdate = { + enable = lib.mkEnableOption "weekly health-gated self-update of the cc-ci host"; + flakePath = lib.mkOption { + type = lib.types.str; + default = "/srv/cc-ci-orch"; + description = "The cc-ci-orchestrator checkout (owned by buildUser) whose flake.lock is updated, committed and pushed."; + }; + flakeRef = lib.mkOption { + type = lib.types.str; + default = "${cfg.flakePath}#cc-ci"; + description = "Flake reference to build."; + }; + buildUser = lib.mkOption { + type = lib.types.str; + default = "loops"; + description = "Owner of the checkout; runs the update, build, commit and push (root activates)."; + }; + onCalendar = lib.mkOption { + type = lib.types.str; + default = "Tue *-*-* 03:00:00 UTC"; + description = '' + When to run. Must stay clear of the weekly recipe-upgrade run (Thu 22:00 America/New_York + = Fri 02:00/03:00 UTC, several hours) and the Sunday 03:00 UTC canonical sweep (up to 6 h); + the busy gate covers the rest. + ''; + }; + remote = lib.mkOption { type = lib.types.str; default = "origin"; }; + branch = lib.mkOption { type = lib.types.str; default = "main"; }; + ciDomain = lib.mkOption { + type = lib.types.str; + default = "ci.commoninternet.net"; + description = "The CI apex; report./drone. are checked under it."; + }; + opencodeUiHost = lib.mkOption { + type = lib.types.str; + default = config.cc-ci-orchestrator.opencodeUiHost; + description = "The opencode UI host name (expected to answer 401)."; + }; + }; + + config = lib.mkIf cfg.enable { + systemd.services.cc-ci-auto-update = { + description = "Weekly cc-ci host self-update (flake update → build → test → health check → commit)"; + after = [ "network-online.target" "docker.service" ]; + wants = [ "network-online.target" ]; + path = with pkgs; [ nixos-rebuild nix git openssh coreutils gnugrep gawk systemd util-linux bash python3 docker curl ]; + serviceConfig = { Type = "oneshot"; TimeoutStartSec = "3h"; }; + # A unit that performs the switch must never be stopped or restarted BY that switch + # (notplants-orchestrator, 2026-09-07: it was, and the host ran half-switched for 12 h). + stopIfChanged = false; + restartIfChanged = false; + script = '' + set -u + cd ${cfg.flakePath} + as_builder() { runuser -u ${cfg.buildUser} -- env HOME=${builderHome} "$@"; } + state() { printf '%s result=%s generation=%s note=%s\n' "$(date -u +%FT%TZ)" "$1" "$2" "$3" > ${stateFile}; chown ${cfg.buildUser} ${stateFile} || true; } + PREV=$(readlink -f /run/current-system) + LOCK_BAK=${cfg.flakePath}/.flake.lock.auto-update-prev + + if ! ${busyGate}; then + state skipped "$PREV" busy + exit 0 + fi + + revert() { + echo "REVERT: $1" + as_builder cp "$LOCK_BAK" flake.lock 2>/dev/null || true + "$PREV"/bin/switch-to-configuration test || echo "re-activation of $PREV failed" + state failed "$PREV" "$1" + exit 1 + } + + as_builder cp flake.lock "$LOCK_BAK" + echo "=== checkout at: $(as_builder git rev-parse --short HEAD) on $(as_builder git rev-parse --abbrev-ref HEAD)" + echo "=== flake update (all inputs)" + as_builder nix flake update || revert "flake update failed" + if as_builder git diff --quiet -- flake.lock; then + echo "=== nothing moved; done" + state unchanged "$PREV" "lock already current" + exit 0 + fi + as_builder git --no-pager diff --stat -- flake.lock + + echo "=== build (cannot affect the running system)" + as_builder nixos-rebuild build --flake ${cfg.flakeRef} || revert "build failed" + NEW=$(readlink -f result) || revert "no build result" + as_builder rm -f result + echo "=== built: $NEW" + + echo "=== activate WITHOUT touching the bootloader" + "$NEW"/bin/switch-to-configuration test || revert "test activation failed" + + echo "=== settle, then health check" + sleep 45 + ${healthCheck} || revert "health check failed" + + echo "=== healthy: committing (system profile + bootloader)" + nix-env -p /nix/var/nix/profiles/system --set "$NEW" || revert "could not set system profile" + "$NEW"/bin/switch-to-configuration boot || revert "bootloader update failed" + state ok "$NEW" "committed" + + # Bookkeeping from here on — never revert a healthy, committed generation over it. + if as_builder git -c user.name="auto-update" -c user.email="auto-update@cc-ci" \ + commit -q -m "flake.lock: weekly auto-update, health-checked" -- flake.lock; then + echo "=== flake.lock committed: $(as_builder git rev-parse --short HEAD)" + as_builder git push -q ${cfg.remote} HEAD:${cfg.branch} \ + && echo "=== pushed ${cfg.branch}" \ + || echo "WARNING: push failed; the lock is committed locally, push it by hand" + else + echo "WARNING: could not commit flake.lock; the system IS committed and healthy" + fi + # Keep the deployed cc-ci checkout (sweep + sops file) on the same cc-ci rev as the flake input. + git -C /etc/cc-ci pull -q --ff-only --recurse-submodules || echo "WARNING: /etc/cc-ci pull failed" + echo "=== auto-update committed: $NEW" + ''; + }; + + systemd.timers.cc-ci-auto-update = { + wantedBy = [ "timers.target" ]; + timerConfig = { + OnCalendar = cfg.onCalendar; + Persistent = false; + RandomizedDelaySec = "10min"; + }; + }; + }; +}