recipe-maintainers/cc-ci-orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

weekly health-gated auto-update of the cc-ci host; skills rewritten for the combined host

Browse Source 
nix/modules/auto-update.nix (own module, no notplants-nix dependency): Tuesday
03:00 UTC, busy-gated (CI run, weekly upgrader, report, sweep, running Drone
builds → skip), `nix flake update` → build → switch-to-configuration test →
cc-ci health checks (sshd, 0 failed units, core units, every swarm service at
replica count, sops decrypted, dashboard/reports/drone 200, opencode UI 401)
→ profile + bootloader → flake.lock committed and pushed to main → /etc/cc-ci
fast-forwarded; revert + lock restore on failure; one-line state file for
/cc-ci-status.

Skills (.opencode canonical, .claude pointers' descriptions synced):
- cc-ci-orchestrator-update: THE host update — drives the auto-update unit by
  hand; --cc-ci-only for a cc-ci-main-only move; failure playbook.
- cc-ci-server-update: delegates to it and explains why the old procedure
  (rebuilding the cc-ci repo's standalone #cc-ci) must not be run on this host.
- cc-ci-update: chains orchestrator-update then tests-update.
- cc-ci-status: §5/§6 for one host — auto-update state, generation vs boot
  generation, front doors (oc.ci = 401), sops, fail2ban, timers, orchestrator
  agent session, secrets inventory; verdict updated.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
This commit is contained in:
notplants
2026-09-07 22:22:50 +00:00
co-authored by Claude Fable 5.1
parent 48226dd78f
commit a8af8429f2
11 changed files with 366 additions and 476 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.opencode/skills/cc-ci-status/SKILL.md
+59 -29
View File
@@ -58,35 +58,62 @@ For each open PR: age (now − created_at), and whether the PR title/body/report
- Verified-green PRs awaiting operator merge are normal — list them as "ready to merge", not
as failures.
### 5. Server + orchestrator host update recency
### 5. The cc-ci host: update recency + the weekly auto-update
For BOTH hosts:
```
# cc-ci server: ssh cc-ci 'nixos-version; cd /root/cc-ci-deploy && nix flake metadata --json' (or builder-clone)
# orchestrator: nixos-version; cd /srv/cc-ci-orch && nix flake metadata --json
git ls-remote https://github.com/NixOS/nixpkgs <channel> # current tip
ls -t /srv/cc-ci-orch/.cc-ci-logs/server-update-*.md /srv/cc-ci-orch/.cc-ci-logs/orchestrator-update-*.md | head -2
```
- Report: days since last update log + how far the running nixpkgs rev lags the channel tip.
- **Lagging >30 days** (or a NixOS release behind) → recommend `/cc-ci-server-update` /
`/cc-ci-orchestrator-update`.
### 6. Host + service health (both machines)
Since 2026-09-07 the CI server and the orchestrator are ONE Hetzner host (`cc-ci`,
195.201.88.249, no tailnet), built from `recipe-maintainers/cc-ci-orchestrator`'s
`nixosConfigurations.cc-ci` (which imports cc-ci's `nixosModules.cc-ci-server`). From the
notplants-orchestrator box `ssh cc-ci` reaches it as root; on the host itself `ssh cc-ci` is loopback.
```
ssh cc-ci 'systemctl --failed --no-legend; df -h / | tail -1; docker service ls --format "{{.Name}} {{.Replicas}}"'
systemctl --failed --no-legend; df -h / | tail -1; tmux ls
ssh cc-ci 'nixos-version; readlink /run/current-system; readlink /nix/var/nix/profiles/system'
ssh cc-ci 'cd /srv/cc-ci-orch && nix flake metadata --json | jq -r ".locks.nodes | to_entries[] | \"\(.key) \(.value.locked.rev // \"-\") \(.value.locked.lastModified // \"-\")\""'
git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05 | cut -c1-12 # channel tip
# the weekly self-update (nix/modules/auto-update.nix, Tue 03:00 UTC, busy-gated):
ssh cc-ci 'cat /srv/cc-ci/.cc-ci-logs/auto-update-state; systemctl list-timers --no-pager cc-ci-auto-update.timer | sed -n 2p; journalctl -u cc-ci-auto-update --no-pager -n 3'
```
- Failed units, core swarm services not 1/1 (warm-* spares flapping is a known benign pattern —
note, don't page), disk **>65% (server)** / >85% (orchestrator) → findings. Server unreachable →
HIGH: recommend `hetzner-server-recovery`.
> **65%, not 80%, on the server — it is not a steady-state measure.** Two concurrent recipe runs
> pull images and write volumes worth tens of GB, so a host sitting at 73% still hits 100% mid-run.
> That is exactly what happened on 2026-08-11: 63GB of unused images had accumulated (nothing ever
> pruned them), the filesystem filled during a run, and the harness died at startup with
> `OSError: [Errno 28] No space left on device`. Remedy: `docker image prune -af` on cc-ci — it
> spares anything a container references, so infra and warm-* canonicals are untouched. Do NOT
> `docker volume prune`: warm-* canonical volumes are data-warm and legitimately dangling.
- The state file is one line: `<UTC time> result=ok|unchanged|skipped|failed generation=<store path> note=<why>`.
- `ok` / `unchanged` within the last 8 days → healthy.
- `skipped` (busy) two weeks in a row, `failed`, or no state line / timer inactive for >8 days →
finding: *"host auto-update <state> since <date>: <note>"* → recommend
`/cc-ci-orchestrator-update` (it runs the same unit by hand and reads its journal).
- `failed` also means the host is running the previous generation and `flake.lock` was restored —
say so; the journal (`journalctl -u cc-ci-auto-update`) names the health check that failed.
- Running `/run/current-system` must equal the profile's boot generation (`system` link) — a
mismatch means a `test`-only activation was left behind (finding: switch or reboot decision).
- nixpkgs rev lagging the `nixos-26.05` channel tip by >30 days → finding → `/cc-ci-orchestrator-update`.
- `cc-ci` (the CI server input) lagging cc-ci `main` → note; the auto-update moves it weekly, or
`/cc-ci-orchestrator-update --cc-ci-only`.
### 6. Host + service health (one machine)
```
ssh cc-ci 'systemctl --failed --no-legend; df -h / | tail -1; free -m | sed -n 2p'
ssh cc-ci 'docker service ls --format "{{.Name}} {{.Replicas}}"; docker stack ls --format "{{.Name}}"'
ssh cc-ci 'for h in ci report.ci drone.ci oc.ci; do printf "%s " $h; curl -s -o /dev/null -w "%{http_code} ssl=%{ssl_verify_result}\n" https://$h.commoninternet.net/; done'
ssh cc-ci 'ls -s /run/secrets/test_secret; systemctl is-active acme-dns fail2ban nginx opencode-web drone-runner-exec; fail2ban-client status sshd | grep -E "Currently banned|Total banned"; fail2ban-client status nginx-http-auth | grep -E "Currently banned|Total banned"'
ssh cc-ci 'systemctl list-timers --no-pager cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer cc-ci-auto-update.timer nightly-sweep.timer | tail -n +2 | head -4'
ssh cc-ci 'sudo -iu loops tmux ls' # cc-ci-orchestrator must exist (the opencode orchestrator agent); cc-ci-upgrader only during a run
```
- Failed units, swarm services not at their replica count (`warm-*` spares flapping is benign —
note, don't page), a front door not 200 (`oc.ci` must be **401**: basic auth in front of the
opencode UI; 404 there = the `ccci-opencode-ui` stack is gone, `systemctl restart opencode-ui-route`
and check the weekly sweep's keep-list), `ssl≠0` (cert), `/run/secrets` empty (sops), acme-dns /
fail2ban / nginx / opencode-web / drone-runner-exec inactive, any timer missing, the orchestrator
tmux session missing (`systemctl restart cc-ci-orchestrator`) → findings. Disk **>65%** → finding.
Host unreachable → HIGH: `hetzner-server-recovery` (token in `/srv/cc-ci/.hcloud-token` on the
notplants box while the operator keeps it active).
> **65%, not 80% — it is not a steady-state measure.** Two concurrent recipe runs pull images and
> write volumes worth tens of GB, so a host sitting at 73% still hits 100% mid-run (2026-08-11:
> 63GB of unused images, `OSError: [Errno 28] No space left on device`). Remedy: `docker image
> prune -af` — it spares anything a container references. Do NOT `docker volume prune`: warm-*
> canonical volumes are data-warm and legitimately dangling.
- fail2ban: a non-zero "Currently banned" is information, not a finding (that is the jail
working); the notplants box's own address showing up banned IS a finding (it means a skill
misbehaved) — `fail2ban-client set <jail> unbanip <ip>`.
- Secrets hygiene: `ssh cc-ci 'ls /secrets/files'` must list ONLY cc-ci.testenv, cc-ci-local-ed25519(.pub),
autonomic-bot-cc-ci-ed25519(.pub), opencode-auth.json. Anything else there (a tailscale key, a
master age key, another host's ssh key…) → finding: *"secret sprawl on the cc-ci host"*.
- **!testme actually produces results** (the check that would have caught the above days earlier):
the newest few `/var/lib/cc-ci-runs/<build>/` dirs must each contain `results.json`. A build that
dies before the harness writes one leaves an EMPTY dir — and the PR badge still says "failure", so
@@ -143,14 +170,17 @@ minutes, no PRs). If it is instead that a known CVE is sitting unpatched, recomm
`/cve-check` over waiting for the next weekly run whenever the question is "are we exposed?".
`ALL HEALTHY` requires: recent successful weekly run + published report, no stale tests, no
CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, recent builds all
producing results.json, disk under
thresholds, bridge clean, maintained-set consistent. Anything else is a finding — even minor
CVE PR open >14 days, the host <30 days behind its channel with the weekly auto-update `ok`/`unchanged`
within 8 days and running generation == boot generation, zero failed units, every front door
answering (oc.ci with 401), sops decrypted, fail2ban up, the orchestrator agent session present,
recent builds all producing results.json, disk under threshold, bridge clean, maintained-set
consistent, secrets inventory exact. Anything else is a finding — even minor
ones get a recommended next step. Order findings by priority (CVE/unreachable-host first).
## Guardrails
- **Read-only.** This skill diagnoses and recommends; it invokes nothing and changes nothing.
- Use the pseudo-TTY wrap for any abra call; plain ssh for everything else.
- Use the pseudo-TTY wrap for any abra call; plain ssh for everything else. `ssh cc-ci` from the
notplants box is root on 195.201.88.249 (loops ssh config); there is no tailnet.
- Don't double-count: a finding that explains another (bridge 401 → !testme "failures") gets
reported once, at the root cause.