cc-ci-orchestrator/.opencode/skills/cc-ci-status/SKILL.md

name, description

name

description

cc-ci-status

Comprehensive read-only status check of the whole cc-ci system - how the recent weekly upgrade runs went and whether their reports published, which recipes/tests are stale, how long since the server + orchestrator host flake updates, open recipe PRs (flagging CVE-carrying PRs that have been open too long), host health (failed units, disk, timers, bridge/!testme path), ending with a verdict (ALL HEALTHY or a findings list) and recommended next steps mapped to the skills to invoke. Never changes anything - it only reads and reports. Invoke as /cc-ci-status.

cc-ci-status

One comprehensive, read-only status pass over the cc-ci system. Output ends with either ALL HEALTHY or a prioritized findings list, each finding paired with the skill to invoke. Nothing here mutates state — no restarts, no deploys, no merges.

Checks (run all; collect findings, don't stop at the first)

1. Weekly upgrade runs — recency + outcome

ls -t /srv/cc-ci/.cc-ci-logs/upgrades/upgrade-all-*.md | head -3
head -20 <latest>            # the Summary block: Considered/green/stale/Failed/Skipped
systemctl list-timers cc-ci-upgrade-all.timer --no-pager | head -3

• Overdue if the newest report is >8 days old, or the timer is inactive/missing → recommend systemctl start cc-ci-upgrade-all.service (or investigate the timer) / /upgrade-all.
• Failed entries in the latest report → recommend /recipe-upgrade <recipe> per entry (or /ci-test-review if the failure is harness-side).
• A run currently in flight (tmux cc-ci-upgrader session live) is NOT a finding — report it as in-progress and skip staleness checks that depend on its output.

2. Report publishing — report.ci.commoninternet.net

curl -s -o /dev/null -w '%{http_code}' https://report.ci.commoninternet.net/
ls -t /var/lib/cc-ci-reports/week-*.html | head -2      # via ssh cc-ci

• Index must be 200 and there must be a week-*.html at least as new as the last completed upgrade run (a completed run without a matching page = report generation broke → recommend /recipe-report / inspect launch-report.py).

3. Stale recipes / stale tests

• Latest report's "PRs where a test looks stale" section + carry-over notes.
• Any entries → recommend /cc-ci-tests-update (fleet) or /recipe-upgrade <recipe> --with-tests (single).
• Also check /srv/cc-ci/.cc-ci-logs/tests-update-*.md recency — if stale tests were reported weeks ago and no tests-update run since, say so.

4. Open recipe PRs — especially CVE-carrying ones that linger

Enumerate open PRs across recipe-maintainers/* (Gitea API, creds in /srv/cc-ci/.testenv):

GET /repos/recipe-maintainers/<repo>/pulls?state=open   # repos = the used-recipes.md inventory + cc-ci

For each open PR: age (now − created_at), and whether the PR title/body/report row mentions CVE / security patch.

• CVE-carrying PR open >14 days → HIGH-priority finding: name the PR, the CVE context, and the blocker (commonly a stale test — check the report row) → recommend the unblocking skill (/cc-ci-tests-update) plus "operator: review + merge ".
• Non-CVE PRs open >30 days → low-priority note ("operator review backlog: N PRs").
• Verified-green PRs awaiting operator merge are normal — list them as "ready to merge", not as failures.

5. The cc-ci host: update recency + the weekly auto-update

Since 2026-09-07 the CI server and the orchestrator are ONE Hetzner host (cc-ci, 195.201.88.249, no tailnet), built from recipe-maintainers/cc-ci-orchestrator's nixosConfigurations.cc-ci (which imports cc-ci's nixosModules.cc-ci-server). From the notplants-orchestrator box ssh cc-ci reaches it as root; on the host itself ssh cc-ci is loopback.

ssh cc-ci 'nixos-version; readlink /run/current-system; readlink /nix/var/nix/profiles/system'
ssh cc-ci 'cd /srv/cc-ci-orch && nix flake metadata --json | jq -r ".locks.nodes | to_entries[] | \"\(.key) \(.value.locked.rev // \"-\") \(.value.locked.lastModified // \"-\")\""'
git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05 | cut -c1-12   # channel tip
# the weekly self-update (nix/modules/auto-update.nix, Tue 03:00 UTC, busy-gated):
ssh cc-ci 'cat /srv/cc-ci/.cc-ci-logs/auto-update-state; systemctl list-timers --no-pager cc-ci-auto-update.timer | sed -n 2p; journalctl -u cc-ci-auto-update --no-pager -n 3'

• The state file is one line: <UTC time> result=ok|unchanged|skipped|failed generation=<store path> note=<why>.
  • ok / unchanged within the last 8 days → healthy.
  • skipped (busy) two weeks in a row, failed, or no state line / timer inactive for >8 days → finding: "host auto-update since : " → recommend /cc-ci-orchestrator-update (it runs the same unit by hand and reads its journal).
  • failed also means the host is running the previous generation and flake.lock was restored — say so; the journal (journalctl -u cc-ci-auto-update) names the health check that failed.
• Running /run/current-system must equal the profile's boot generation (system link) — a mismatch means a test-only activation was left behind (finding: switch or reboot decision).
• nixpkgs rev lagging the nixos-26.05 channel tip by >30 days → finding → /cc-ci-orchestrator-update.
• cc-ci (the CI server input) lagging cc-ci main → note; the auto-update moves it weekly, or /cc-ci-orchestrator-update --cc-ci-only.

6. Host + service health (one machine)

ssh cc-ci 'systemctl --failed --no-legend; df -h / | tail -1; free -m | sed -n 2p'
ssh cc-ci 'docker service ls --format "{{.Name}} {{.Replicas}}"; docker stack ls --format "{{.Name}}"'
ssh cc-ci 'for h in ci report.ci drone.ci oc.ci; do printf "%s " $h; curl -s -o /dev/null -w "%{http_code} ssl=%{ssl_verify_result}\n" https://$h.commoninternet.net/; done'
ssh cc-ci 'ls -s /run/secrets/test_secret; systemctl is-active acme-dns fail2ban nginx opencode-web drone-runner-exec; fail2ban-client status sshd | grep -E "Currently banned|Total banned"; fail2ban-client status nginx-http-auth | grep -E "Currently banned|Total banned"'
ssh cc-ci 'systemctl list-timers --no-pager cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer cc-ci-auto-update.timer nightly-sweep.timer | tail -n +2 | head -4'
ssh cc-ci 'sudo -iu loops tmux ls'      # cc-ci-orchestrator must exist (the opencode orchestrator agent); cc-ci-upgrader only during a run

• Failed units, swarm services not at their replica count (warm-* spares flapping is benign — note, don't page), a front door not 200 (oc.ci must be 401: basic auth in front of the opencode UI; 404 there = the ccci-opencode-ui stack is gone, systemctl restart opencode-ui-route and check the weekly sweep's keep-list), ssl≠0 (cert), /run/secrets empty (sops), acme-dns / fail2ban / nginx / opencode-web / drone-runner-exec inactive, any timer missing, the orchestrator tmux session missing (systemctl restart cc-ci-orchestrator) → findings. Disk >65% → finding. Host unreachable → HIGH: hetzner-server-recovery (token in /srv/cc-ci/.hcloud-token on the notplants box while the operator keeps it active).

  65%, not 80% — it is not a steady-state measure. Two concurrent recipe runs pull images and write volumes worth tens of GB, so a host sitting at 73% still hits 100% mid-run (2026-08-11: 63GB of unused images, OSError: [Errno 28] No space left on device). Remedy: docker image prune -af — it spares anything a container references. Do NOT docker volume prune: warm-* canonical volumes are data-warm and legitimately dangling.

• fail2ban: a non-zero "Currently banned" is information, not a finding (that is the jail working); the notplants box's own address showing up banned IS a finding (it means a skill misbehaved) — fail2ban-client set <jail> unbanip <ip>.
• Secrets hygiene: ssh cc-ci 'ls /secrets/files' must list ONLY cc-ci.testenv, cc-ci-local-ed25519(.pub), autonomic-bot-cc-ci-ed25519(.pub), opencode-auth.json. Anything else there (a tailscale key, a master age key, another host's ssh key…) → finding: "secret sprawl on the cc-ci host".
• !testme actually produces results (the check that would have caught the above days earlier): the newest few /var/lib/cc-ci-runs/<build>/ dirs must each contain results.json. A build that dies before the harness writes one leaves an EMPTY dir — and the PR badge still says "failure", so it reads as a recipe regression rather than a sick host. Builds 1236–1242 all failed that way. Finding: "N recent builds produced no results.json — the harness is dying at startup, check disk and the drone step log". The step log lives in drone's sqlite (/var/lib/docker/volumes/drone_ci_commoninternet_net_data/_data/database.sqlite) — copy it and read logs.log_data for the failing steps.step_id; the bridge's drone token is not extractable (distroless container, swarm secret).

  If the error is ENOSPC but the disk is fine, it is not disk. Seen 2026-08-11: builds 1244-1249 died on mkdir /var/lib/cc-ci-runs/<build> with 110GB free and 16% inodes, while the identical mkdir succeeded as root over ssh, inside the runner's own mount namespace, and 61/61 times in a stress loop — and the same harness run by hand with a numeric run id worked fine. Restarting drone-runner-exec did NOT help, and neither did recreating the runs directory with a fresh inode (it recurred afterwards — that apparent fix was coincidence).

  It is INTERMITTENT and tracks concurrent activity, which is the useful signal: every failure landed while a second run or a manual deploy was in flight (1252 was triggered while 1251 was still finishing), and every build on a quiet host succeeded (1243, 1250, 1251, 1253). Free space never moved during a failing build. So on ENOSPC-with-free-disk: wait for the host to go quiet and re-trigger before treating it as a recipe failure. Root cause is still NOT established; DRONE_RUNNER_CAPACITY=2 allows the overlap, so lowering it to 1 is the obvious next experiment if it becomes disruptive.

• Bridge / !testme path: docker service ls shows ccci-bridge_app 1/1 AND the bridge log has no auth errors (docker service logs --since 24h ccci-bridge_app 2>&1 | grep -ci "401\|user does not exist" == 0). A silently-401ing bridge drops !testme (seen 2026-08-03, stale rotated Gitea secret) → recommend refreshing the bridge secret + redeploy (test-before-switch).

7. Maintained-set consistency (quick)

• Every weekly row in used-recipes.md has tests/<recipe>/ on cc-ci AND a recipe-maintainers/<recipe> mirror AND is in bridge POLL_REPOS; mismatches → recommend finishing enrollment (/recipe-enroll <recipe> covers all touchpoints).

Output format

# cc-ci status — <date>
## Verdict: ALL HEALTHY | N findings (M high-priority)
## Weekly upgrades: <last run date + one-line outcome; next timer firing>
## Report site: <ok/broken + latest page>
## Stale tests: <none | list>
## Open PRs: <count; CVE-carrying + age flagged FIRST; ready-to-merge list>
## Host updates: server <rev, N days behind tip> · orchestrator <rev, N days>
## Health: server <failed/disk/services> · orchestrator <failed/disk/sessions> · bridge <ok/401s>
## Recommended next steps
1. <finding> → /<skill> (or operator action)
…

When a finding is that the fleet's security exposure is unknown — the last weekly run failed or is stale, so nobody has scanned for CVEs recently — the recommended step is /cve-check (read-only, minutes, no PRs). If it is instead that a known CVE is sitting unpatched, recommend /cve-check-and-upgrade (add --min-severity high when only the urgent ones matter). Prefer /cve-check over waiting for the next weekly run whenever the question is "are we exposed?".

ALL HEALTHY requires: recent successful weekly run + published report, no stale tests, no CVE PR open >14 days, the host <30 days behind its channel with the weekly auto-update ok/unchanged within 8 days and running generation == boot generation, zero failed units, every front door answering (oc.ci with 401), sops decrypted, fail2ban up, the orchestrator agent session present, recent builds all producing results.json, disk under threshold, bridge clean, maintained-set consistent, secrets inventory exact. Anything else is a finding — even minor ones get a recommended next step. Order findings by priority (CVE/unreachable-host first).

Guardrails

• Read-only. This skill diagnoses and recommends; it invokes nothing and changes nothing.
• Use the pseudo-TTY wrap for any abra call; plain ssh for everything else. ssh cc-ci from the notplants box is root on 195.201.88.249 (loops ssh config); there is no tailnet.
• Don't double-count: a finding that explains another (bridge 401 → !testme "failures") gets reported once, at the root cause.