cc-ci-orchestrator/.opencode/skills/cc-ci-status/SKILL.md

name, description

name

description

cc-ci-status

Comprehensive read-only status check of the whole cc-ci system - how the recent weekly upgrade runs went and whether their reports published, which recipes/tests are stale, how long since the server + orchestrator host flake updates, open recipe PRs (flagging CVE-carrying PRs that have been open too long), host health (failed units, disk, timers, bridge/!testme path), ending with a verdict (ALL HEALTHY or a findings list) and recommended next steps mapped to the skills to invoke. Never changes anything - it only reads and reports. Invoke as /cc-ci-status.

cc-ci-status

One comprehensive, read-only status pass over the cc-ci system. Output ends with either ALL HEALTHY or a prioritized findings list, each finding paired with the skill to invoke. Nothing here mutates state — no restarts, no deploys, no merges.

Checks (run all; collect findings, don't stop at the first)

1. Weekly upgrade runs — recency + outcome

ls -t /srv/cc-ci/.cc-ci-logs/upgrades/upgrade-all-*.md | head -3
head -20 <latest>            # the Summary block: Considered/green/stale/Failed/Skipped
systemctl list-timers cc-ci-upgrade-all.timer --no-pager | head -3

• Overdue if the newest report is >8 days old, or the timer is inactive/missing → recommend systemctl start cc-ci-upgrade-all.service (or investigate the timer) / /upgrade-all.
• Failed entries in the latest report → recommend /recipe-upgrade <recipe> per entry (or /ci-test-review if the failure is harness-side).
• A run currently in flight (tmux cc-ci-upgrader session live) is NOT a finding — report it as in-progress and skip staleness checks that depend on its output.

2. Report publishing — report.ci.commoninternet.net

curl -s -o /dev/null -w '%{http_code}' https://report.ci.commoninternet.net/
ls -t /var/lib/cc-ci-reports/week-*.html | head -2      # via ssh cc-ci

• Index must be 200 and there must be a week-*.html at least as new as the last completed upgrade run (a completed run without a matching page = report generation broke → recommend /recipe-report / inspect launch-report.py).

3. Stale recipes / stale tests

• Latest report's "PRs where a test looks stale" section + carry-over notes.
• Any entries → recommend /cc-ci-tests-update (fleet) or /recipe-upgrade <recipe> --with-tests (single).
• Also check /srv/cc-ci/.cc-ci-logs/tests-update-*.md recency — if stale tests were reported weeks ago and no tests-update run since, say so.

4. Open recipe PRs — especially CVE-carrying ones that linger

Enumerate open PRs across recipe-maintainers/* (Gitea API, creds in /srv/cc-ci/.testenv):

GET /repos/recipe-maintainers/<repo>/pulls?state=open   # repos = the used-recipes.md inventory + cc-ci

For each open PR: age (now − created_at), and whether the PR title/body/report row mentions CVE / security patch.

• CVE-carrying PR open >14 days → HIGH-priority finding: name the PR, the CVE context, and the blocker (commonly a stale test — check the report row) → recommend the unblocking skill (/cc-ci-tests-update) plus "operator: review + merge ".
• Non-CVE PRs open >30 days → low-priority note ("operator review backlog: N PRs").
• Verified-green PRs awaiting operator merge are normal — list them as "ready to merge", not as failures.

5. Server + orchestrator host update recency

For BOTH hosts:

# cc-ci server:      ssh cc-ci 'nixos-version; cd /root/cc-ci-deploy && nix flake metadata --json' (or builder-clone)
# orchestrator:      nixos-version; cd /srv/cc-ci-orch && nix flake metadata --json
git ls-remote https://github.com/NixOS/nixpkgs <channel>   # current tip
ls -t /srv/cc-ci-orch/.cc-ci-logs/server-update-*.md /srv/cc-ci-orch/.cc-ci-logs/orchestrator-update-*.md | head -2

• Report: days since last update log + how far the running nixpkgs rev lags the channel tip.
• Lagging >30 days (or a NixOS release behind) → recommend /cc-ci-server-update / /cc-ci-orchestrator-update.

6. Host + service health (both machines)

ssh cc-ci 'systemctl --failed --no-legend; df -h / | tail -1; docker service ls --format "{{.Name}} {{.Replicas}}"'
systemctl --failed --no-legend; df -h / | tail -1; tmux ls

• Failed units, core swarm services not 1/1 (warm-* spares flapping is a known benign pattern — note, don't page), disk >80% (server) / >85% (orchestrator) → findings. Server unreachable → HIGH: recommend hetzner-server-recovery.
• Bridge / !testme path: docker service ls shows ccci-bridge_app 1/1 AND the bridge log has no auth errors (docker service logs --since 24h ccci-bridge_app 2>&1 | grep -ci "401\|user does not exist" == 0). A silently-401ing bridge drops !testme (seen 2026-08-03, stale rotated Gitea secret) → recommend refreshing the bridge secret + redeploy (test-before-switch).

7. Maintained-set consistency (quick)

• Every weekly row in used-recipes.md has tests/<recipe>/ on cc-ci AND a recipe-maintainers/<recipe> mirror AND is in bridge POLL_REPOS; mismatches → recommend finishing enrollment (/recipe-enroll <recipe> covers all touchpoints).

Output format

# cc-ci status — <date>
## Verdict: ALL HEALTHY | N findings (M high-priority)
## Weekly upgrades: <last run date + one-line outcome; next timer firing>
## Report site: <ok/broken + latest page>
## Stale tests: <none | list>
## Open PRs: <count; CVE-carrying + age flagged FIRST; ready-to-merge list>
## Host updates: server <rev, N days behind tip> · orchestrator <rev, N days>
## Health: server <failed/disk/services> · orchestrator <failed/disk/sessions> · bridge <ok/401s>
## Recommended next steps
1. <finding> → /<skill> (or operator action)
…

ALL HEALTHY requires: recent successful weekly run + published report, no stale tests, no CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, disk under thresholds, bridge clean, maintained-set consistent. Anything else is a finding — even minor ones get a recommended next step. Order findings by priority (CVE/unreachable-host first).

Guardrails

• Read-only. This skill diagnoses and recommends; it invokes nothing and changes nothing.
• Use the pseudo-TTY wrap for any abra call; plain ssh for everything else.
• Don't double-count: a finding that explains another (bridge 401 → !testme "failures") gets reported once, at the root cause.