autonomic-bot
94ce5c4de2
I recorded that recreating the runs directory with a fresh inode preceded recovery. It recurred afterwards (build 1252), so that was not the fix. The real signal is that it is INTERMITTENT and tracks concurrent activity: every failure landed while a second run or manual deploy was in flight, and every build on a quiet host passed (1243, 1250, 1251, 1253). Free space never moves during a failing build. Practical guidance is therefore to wait for the host to go quiet and re-trigger before calling it a recipe failure, and DRONE_RUNNER_CAPACITY=2 is the obvious knob to test if it becomes disruptive. Root cause still not established, and the note now says so plainly rather than presenting a coincidence as understood.
157 lines
9.5 KiB
Markdown
157 lines
9.5 KiB
Markdown
---
|
||
name: cc-ci-status
|
||
description: Comprehensive read-only status check of the whole cc-ci system - how the recent weekly upgrade runs went and whether their reports published, which recipes/tests are stale, how long since the server + orchestrator host flake updates, open recipe PRs (flagging CVE-carrying PRs that have been open too long), host health (failed units, disk, timers, bridge/!testme path), ending with a verdict (ALL HEALTHY or a findings list) and recommended next steps mapped to the skills to invoke. Never changes anything - it only reads and reports. Invoke as /cc-ci-status.
|
||
---
|
||
|
||
# cc-ci-status
|
||
|
||
One comprehensive, **read-only** status pass over the cc-ci system. Output ends with either
|
||
**`ALL HEALTHY`** or a prioritized findings list, each finding paired with the skill to invoke.
|
||
Nothing here mutates state — no restarts, no deploys, no merges.
|
||
|
||
## Checks (run all; collect findings, don't stop at the first)
|
||
|
||
### 1. Weekly upgrade runs — recency + outcome
|
||
|
||
```
|
||
ls -t /srv/cc-ci/.cc-ci-logs/upgrades/upgrade-all-*.md | head -3
|
||
head -20 <latest> # the Summary block: Considered/green/stale/Failed/Skipped
|
||
systemctl list-timers cc-ci-upgrade-all.timer --no-pager | head -3
|
||
```
|
||
- **Overdue** if the newest report is >8 days old, or the timer is inactive/missing → recommend
|
||
`systemctl start cc-ci-upgrade-all.service` (or investigate the timer) / `/upgrade-all`.
|
||
- **Failed entries** in the latest report → recommend `/recipe-upgrade <recipe>` per entry (or
|
||
`/ci-test-review` if the failure is harness-side).
|
||
- A run currently in flight (tmux `cc-ci-upgrader` session live) is NOT a finding — report it as
|
||
in-progress and skip staleness checks that depend on its output.
|
||
|
||
### 2. Report publishing — report.ci.commoninternet.net
|
||
|
||
```
|
||
curl -s -o /dev/null -w '%{http_code}' https://report.ci.commoninternet.net/
|
||
ls -t /var/lib/cc-ci-reports/week-*.html | head -2 # via ssh cc-ci
|
||
```
|
||
- Index must be 200 and there must be a `week-*.html` at least as new as the last **completed**
|
||
upgrade run (a completed run without a matching page = report generation broke → recommend
|
||
`/recipe-report` / inspect `launch-report.py`).
|
||
|
||
### 3. Stale recipes / stale tests
|
||
|
||
- Latest report's "PRs where a test looks stale" section + carry-over notes.
|
||
- Any entries → recommend `/cc-ci-tests-update` (fleet) or `/recipe-upgrade <recipe>
|
||
--with-tests` (single).
|
||
- Also check `/srv/cc-ci/.cc-ci-logs/tests-update-*.md` recency — if stale tests were reported
|
||
weeks ago and no tests-update run since, say so.
|
||
|
||
### 4. Open recipe PRs — especially CVE-carrying ones that linger
|
||
|
||
Enumerate open PRs across `recipe-maintainers/*` (Gitea API, creds in `/srv/cc-ci/.testenv`):
|
||
```
|
||
GET /repos/recipe-maintainers/<repo>/pulls?state=open # repos = the used-recipes.md inventory + cc-ci
|
||
```
|
||
For each open PR: age (now − created_at), and whether the PR title/body/report row mentions
|
||
**CVE** / security patch.
|
||
- **CVE-carrying PR open >14 days** → HIGH-priority finding: name the PR, the CVE context, and
|
||
the blocker (commonly a stale test — check the report row) → recommend the unblocking skill
|
||
(`/cc-ci-tests-update`) plus "operator: review + merge <PR>".
|
||
- Non-CVE PRs open >30 days → low-priority note ("operator review backlog: N PRs").
|
||
- Verified-green PRs awaiting operator merge are normal — list them as "ready to merge", not
|
||
as failures.
|
||
|
||
### 5. Server + orchestrator host update recency
|
||
|
||
For BOTH hosts:
|
||
```
|
||
# cc-ci server: ssh cc-ci 'nixos-version; cd /root/cc-ci-deploy && nix flake metadata --json' (or builder-clone)
|
||
# orchestrator: nixos-version; cd /srv/cc-ci-orch && nix flake metadata --json
|
||
git ls-remote https://github.com/NixOS/nixpkgs <channel> # current tip
|
||
ls -t /srv/cc-ci-orch/.cc-ci-logs/server-update-*.md /srv/cc-ci-orch/.cc-ci-logs/orchestrator-update-*.md | head -2
|
||
```
|
||
- Report: days since last update log + how far the running nixpkgs rev lags the channel tip.
|
||
- **Lagging >30 days** (or a NixOS release behind) → recommend `/cc-ci-server-update` /
|
||
`/cc-ci-orchestrator-update`.
|
||
|
||
### 6. Host + service health (both machines)
|
||
|
||
```
|
||
ssh cc-ci 'systemctl --failed --no-legend; df -h / | tail -1; docker service ls --format "{{.Name}} {{.Replicas}}"'
|
||
systemctl --failed --no-legend; df -h / | tail -1; tmux ls
|
||
```
|
||
- Failed units, core swarm services not 1/1 (warm-* spares flapping is a known benign pattern —
|
||
note, don't page), disk **>65% (server)** / >85% (orchestrator) → findings. Server unreachable →
|
||
HIGH: recommend `hetzner-server-recovery`.
|
||
> **65%, not 80%, on the server — it is not a steady-state measure.** Two concurrent recipe runs
|
||
> pull images and write volumes worth tens of GB, so a host sitting at 73% still hits 100% mid-run.
|
||
> That is exactly what happened on 2026-08-11: 63GB of unused images had accumulated (nothing ever
|
||
> pruned them), the filesystem filled during a run, and the harness died at startup with
|
||
> `OSError: [Errno 28] No space left on device`. Remedy: `docker image prune -af` on cc-ci — it
|
||
> spares anything a container references, so infra and warm-* canonicals are untouched. Do NOT
|
||
> `docker volume prune`: warm-* canonical volumes are data-warm and legitimately dangling.
|
||
- **!testme actually produces results** (the check that would have caught the above days earlier):
|
||
the newest few `/var/lib/cc-ci-runs/<build>/` dirs must each contain `results.json`. A build that
|
||
dies before the harness writes one leaves an EMPTY dir — and the PR badge still says "failure", so
|
||
it reads as a recipe regression rather than a sick host. Builds 1236–1242 all failed that way.
|
||
Finding: *"N recent builds produced no results.json — the harness is dying at startup, check disk
|
||
and the drone step log"*. The step log lives in drone's sqlite
|
||
(`/var/lib/docker/volumes/drone_ci_commoninternet_net_data/_data/database.sqlite`) — copy it and
|
||
read `logs.log_data` for the failing `steps.step_id`; the bridge's drone token is not extractable
|
||
(distroless container, swarm secret).
|
||
> **If the error is ENOSPC but the disk is fine**, it is not disk. Seen 2026-08-11: builds 1244-1249
|
||
> died on `mkdir /var/lib/cc-ci-runs/<build>` with **110GB free and 16% inodes**, while the identical
|
||
> mkdir succeeded as root over ssh, inside the runner's own mount namespace, and 61/61 times in a
|
||
> stress loop — and the same harness run by hand with a numeric run id worked fine. Restarting
|
||
> `drone-runner-exec` did NOT help, and neither did recreating the runs directory with a fresh
|
||
> inode (it recurred afterwards — that apparent fix was coincidence).
|
||
>
|
||
> **It is INTERMITTENT and tracks concurrent activity**, which is the useful signal: every failure
|
||
> landed while a second run or a manual deploy was in flight (1252 was triggered while 1251 was
|
||
> still finishing), and every build on a quiet host succeeded (1243, 1250, 1251, 1253). Free space
|
||
> never moved during a failing build. So on ENOSPC-with-free-disk: **wait for the host to go quiet
|
||
> and re-trigger** before treating it as a recipe failure. Root cause is still NOT established;
|
||
> `DRONE_RUNNER_CAPACITY=2` allows the overlap, so lowering it to 1 is the obvious next experiment
|
||
> if it becomes disruptive.
|
||
- **Bridge / !testme path**: `docker service ls` shows `ccci-bridge_app 1/1` AND the bridge log
|
||
has no auth errors (`docker service logs --since 24h ccci-bridge_app 2>&1 | grep -ci "401\|user does not exist"` == 0).
|
||
A silently-401ing bridge drops `!testme` (seen 2026-08-03, stale rotated Gitea secret) →
|
||
recommend refreshing the bridge secret + redeploy (test-before-switch).
|
||
|
||
### 7. Maintained-set consistency (quick)
|
||
|
||
- Every `weekly` row in `used-recipes.md` has `tests/<recipe>/` on cc-ci AND a
|
||
`recipe-maintainers/<recipe>` mirror AND is in bridge `POLL_REPOS`; mismatches → recommend
|
||
finishing enrollment (`/recipe-enroll <recipe>` covers all touchpoints).
|
||
|
||
## Output format
|
||
|
||
```
|
||
# cc-ci status — <date>
|
||
## Verdict: ALL HEALTHY | N findings (M high-priority)
|
||
## Weekly upgrades: <last run date + one-line outcome; next timer firing>
|
||
## Report site: <ok/broken + latest page>
|
||
## Stale tests: <none | list>
|
||
## Open PRs: <count; CVE-carrying + age flagged FIRST; ready-to-merge list>
|
||
## Host updates: server <rev, N days behind tip> · orchestrator <rev, N days>
|
||
## Health: server <failed/disk/services> · orchestrator <failed/disk/sessions> · bridge <ok/401s>
|
||
## Recommended next steps
|
||
1. <finding> → /<skill> (or operator action)
|
||
…
|
||
```
|
||
When a finding is that the fleet's **security exposure is unknown** — the last weekly run failed or
|
||
is stale, so nobody has scanned for CVEs recently — the recommended step is **`/cve-check`** (read-only,
|
||
minutes, no PRs). If it is instead that a known CVE is sitting unpatched, recommend
|
||
**`/cve-check-and-upgrade`** (add `--min-severity high` when only the urgent ones matter). Prefer
|
||
`/cve-check` over waiting for the next weekly run whenever the question is "are we exposed?".
|
||
|
||
`ALL HEALTHY` requires: recent successful weekly run + published report, no stale tests, no
|
||
CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, recent builds all
|
||
producing results.json, disk under
|
||
thresholds, bridge clean, maintained-set consistent. Anything else is a finding — even minor
|
||
ones get a recommended next step. Order findings by priority (CVE/unreachable-host first).
|
||
|
||
## Guardrails
|
||
|
||
- **Read-only.** This skill diagnoses and recommends; it invokes nothing and changes nothing.
|
||
- Use the pseudo-TTY wrap for any abra call; plain ssh for everything else.
|
||
- Don't double-count: a finding that explains another (bridge 401 → !testme "failures") gets
|
||
reported once, at the root cause.
|