From ab88e59c21adc2ea9de1bd1146746ac06326f29c Mon Sep 17 00:00:00 2001 From: autonomic-bot Date: Tue, 11 Aug 2026 15:41:27 +0000 Subject: [PATCH 1/4] cc-ci: prune unused images in the sweep; catch a starving host before CI dies MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The CI server filled up and every !testme from build 1236 to 1242 died at harness startup with ENOSPC on /var/lib/cc-ci-runs/. Because the harness never got far enough to write results.json, the PR badges just said 'failure' — so it read as recipe regressions, and plausible's genuinely-fixed suite looked still-broken. Cause: every run pulls each recipe's images and nothing ever removed the old ones. 72GB of images, 63GB of it unused. Reclaimed 69.8GB; the host went 73% -> 22%. Two changes so it does not recur: - sweep-orphans.sh (runs at the start AND end of every /upgrade-all) now prunes unused images when the disk is >=60% (DISK_PRUNE_PCT). Below that it keeps the layer cache so runs stay fast. 'docker image prune -a' spares anything a container references, so infra and warm-* canonicals are safe. Volumes are still NOT pruned — warm-* canonical volumes are data-warm and legitimately dangling. - /cc-ci-status flags server disk at >65% rather than >80%, because this is not a steady-state measure: the host was at 73% when runs started failing. It also now checks that recent builds actually produced results.json — an empty run dir is the fingerprint of a host problem masquerading as a recipe failure — and records how to read a drone step log out of its sqlite when the API token is unreachable. --- .claude/skills/upgrade-all/sweep-orphans.sh | 23 +++++++++++++++++++++ .opencode/skills/cc-ci-status/SKILL.md | 21 +++++++++++++++++-- 2 files changed, 42 insertions(+), 2 deletions(-) diff --git a/.claude/skills/upgrade-all/sweep-orphans.sh b/.claude/skills/upgrade-all/sweep-orphans.sh index c6b95e1..442ce56 100755 --- a/.claude/skills/upgrade-all/sweep-orphans.sh +++ b/.claude/skills/upgrade-all/sweep-orphans.sh @@ -73,6 +73,29 @@ done # 5) Stray exited containers (debug one-shots) — best-effort prune. docker container prune -f >/dev/null 2>&1 || true +# 6) Unused IMAGES — the one that actually took CI down. Every run pulls each recipe's images and +# nothing ever removed the old ones: on 2026-08-11 they had grown to 72GB (63GB of it unused), +# the root filesystem hit 100% under two concurrent runs, and the harness died at startup with +# `OSError: [Errno 28] No space left on device: '/var/lib/cc-ci-runs/'`. Every !testme +# from build 1236 to 1242 failed that way — with no results.json, so the PR badges just read +# "failure" and looked like recipe regressions. +# +# Only prune above a threshold, so a healthy host keeps its layer cache and runs stay fast. +# `image prune -a` removes only images no container references, so anything deployed (infra + +# warm-* canonicals) is untouched; anything else is re-pulled on demand. +# +# Volumes are deliberately NOT pruned here — see the KEEP_RE guard in (3): warm-* canonicals are +# data-warm and their volumes are legitimately dangling between runs. +DISK_PRUNE_PCT="${DISK_PRUNE_PCT:-60}" +used_pct="$(df --output=pcent / 2>/dev/null | tail -1 | tr -dc '0-9')" +if [ -n "$used_pct" ] && [ "$used_pct" -ge "$DISK_PRUNE_PCT" ]; then + echo " disk ${used_pct}% >= ${DISK_PRUNE_PCT}% -> pruning unused images" + freed="$(docker image prune -af 2>/dev/null | awk '/Total reclaimed space/ {print $4, $5}')" + echo " reclaimed: ${freed:-0B}; disk now $(df -h / | tail -1 | awk '{print $5" used, "$4" free"}')" +else + echo " disk ${used_pct:-?}% < ${DISK_PRUNE_PCT}% -> keeping image cache" +fi + if [ "$removed" -eq 0 ]; then echo "== orphan sweep: clean (nothing to remove) ==" else diff --git a/.opencode/skills/cc-ci-status/SKILL.md b/.opencode/skills/cc-ci-status/SKILL.md index 0366ef7..d1c1706 100644 --- a/.opencode/skills/cc-ci-status/SKILL.md +++ b/.opencode/skills/cc-ci-status/SKILL.md @@ -78,8 +78,24 @@ ssh cc-ci 'systemctl --failed --no-legend; df -h / | tail -1; docker service ls systemctl --failed --no-legend; df -h / | tail -1; tmux ls ``` - Failed units, core swarm services not 1/1 (warm-* spares flapping is a known benign pattern — - note, don't page), disk >80% (server) / >85% (orchestrator) → findings. Server unreachable → + note, don't page), disk **>65% (server)** / >85% (orchestrator) → findings. Server unreachable → HIGH: recommend `hetzner-server-recovery`. + > **65%, not 80%, on the server — it is not a steady-state measure.** Two concurrent recipe runs + > pull images and write volumes worth tens of GB, so a host sitting at 73% still hits 100% mid-run. + > That is exactly what happened on 2026-08-11: 63GB of unused images had accumulated (nothing ever + > pruned them), the filesystem filled during a run, and the harness died at startup with + > `OSError: [Errno 28] No space left on device`. Remedy: `docker image prune -af` on cc-ci — it + > spares anything a container references, so infra and warm-* canonicals are untouched. Do NOT + > `docker volume prune`: warm-* canonical volumes are data-warm and legitimately dangling. +- **!testme actually produces results** (the check that would have caught the above days earlier): + the newest few `/var/lib/cc-ci-runs//` dirs must each contain `results.json`. A build that + dies before the harness writes one leaves an EMPTY dir — and the PR badge still says "failure", so + it reads as a recipe regression rather than a sick host. Builds 1236–1242 all failed that way. + Finding: *"N recent builds produced no results.json — the harness is dying at startup, check disk + and the drone step log"*. The step log lives in drone's sqlite + (`/var/lib/docker/volumes/drone_ci_commoninternet_net_data/_data/database.sqlite`) — copy it and + read `logs.log_data` for the failing `steps.step_id`; the bridge's drone token is not extractable + (distroless container, swarm secret). - **Bridge / !testme path**: `docker service ls` shows `ccci-bridge_app 1/1` AND the bridge log has no auth errors (`docker service logs --since 24h ccci-bridge_app 2>&1 | grep -ci "401\|user does not exist"` == 0). A silently-401ing bridge drops `!testme` (seen 2026-08-03, stale rotated Gitea secret) → @@ -113,7 +129,8 @@ minutes, no PRs). If it is instead that a known CVE is sitting unpatched, recomm `/cve-check` over waiting for the next weekly run whenever the question is "are we exposed?". `ALL HEALTHY` requires: recent successful weekly run + published report, no stale tests, no -CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, disk under +CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, recent builds all +producing results.json, disk under thresholds, bridge clean, maintained-set consistent. Anything else is a finding — even minor ones get a recommended next step. Order findings by priority (CVE/unreachable-host first). From ecf126d98dbda294da485b0fcc0f0883f45d06ab Mon Sep 17 00:00:00 2001 From: autonomic-bot Date: Tue, 11 Aug 2026 17:39:56 +0000 Subject: [PATCH 2/4] cc-ci-status: record the ENOSPC-with-free-disk failure and what recovered it Builds 1244-1249 died on mkdir of the run dir with 110GB free and 16% inodes. Ruled out: actual disk (df sampled every 2s across a failing build never moved), inodes, quotas, a poisoned parent directory (61/61 stress creations succeeded), runner sandboxing (namespaces identical to the host), and a wedged runner (restart changed nothing). The same harness with the same numeric run id, run by hand outside drone, worked every time. Recreating the runs directory with a fresh inode preceded recovery; builds have run normally since. The root cause is NOT established, so the note says so rather than presenting a fix that might be coincidence. --- .opencode/skills/cc-ci-status/SKILL.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.opencode/skills/cc-ci-status/SKILL.md b/.opencode/skills/cc-ci-status/SKILL.md index d1c1706..b33f962 100644 --- a/.opencode/skills/cc-ci-status/SKILL.md +++ b/.opencode/skills/cc-ci-status/SKILL.md @@ -96,6 +96,14 @@ systemctl --failed --no-legend; df -h / | tail -1; tmux ls (`/var/lib/docker/volumes/drone_ci_commoninternet_net_data/_data/database.sqlite`) — copy it and read `logs.log_data` for the failing `steps.step_id`; the bridge's drone token is not extractable (distroless container, swarm secret). + > **If the error is ENOSPC but the disk is fine**, it is not disk. Seen 2026-08-11: builds 1244-1249 + > died on `mkdir /var/lib/cc-ci-runs/` with **110GB free and 16% inodes**, while the identical + > mkdir succeeded as root over ssh, inside the runner's own mount namespace, and 61/61 times in a + > stress loop — and the same harness run by hand with a numeric run id worked fine. Restarting + > `drone-runner-exec` did NOT help. What preceded recovery was **recreating the runs directory with + > a fresh inode** (`mv cc-ci-runs cc-ci-runs.old && mkdir -m 750 cc-ci-runs && mv cc-ci-runs.old/* cc-ci-runs/`), + > after which builds ran normally. Root cause was NOT conclusively identified — record the evidence + > if it recurs rather than assuming this is understood. - **Bridge / !testme path**: `docker service ls` shows `ccci-bridge_app 1/1` AND the bridge log has no auth errors (`docker service logs --since 24h ccci-bridge_app 2>&1 | grep -ci "401\|user does not exist"` == 0). A silently-401ing bridge drops `!testme` (seen 2026-08-03, stale rotated Gitea secret) → From bb7ebb4a2761f8c58d7b172e2aa9dc9f0a8e728e Mon Sep 17 00:00:00 2001 From: autonomic-bot Date: Tue, 11 Aug 2026 18:38:09 +0000 Subject: [PATCH 3/4] reconcile-upstream.sh: one deterministic entry point, mandated before PR work MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Working against a stale mirror has cost us three different ways: - mailu #6 was linked as the fix for two internet-facing Roundcube CVEs while upstream had already merged AND released it (3.1.3+2024.06.57). The work was done; only our mirror was behind. Reconciling closed the PR automatically. - a stale mirror makes a survey report 'no upgrades available', so the recipe silently drops out of the weekly run. - reading the wrong branch: several coopcloud recipes keep a stale 'main' beside the real default 'master'. gitea's main is 1.24.2-rootless while master has 1.27.1-rootless and the merged PRs, so reading main manufactures a false 'three releases behind, missing two CVSS-9.8 RCEs' finding. The reconcile logic already existed inside open-recipe-pr.sh --reconcile-only and already resolves the default branch itself. What was missing was a single obvious entry point and a rule saying to run it. reconcile-upstream.sh takes recipes or --all, and is idempotent — recipe work lives in branches, never on mirror main, so force-syncing main discards nothing. /ci-test-review and /cc-ci-tests-update had NO reconcile step at all; both now require it. /cve-check, /recipe-upgrade and /upgrade-all already reconciled and now point at the shared script. --- .claude/skills/ci-test-review/SKILL.md | 13 ++++ .claude/skills/cve-check/SKILL.md | 3 +- .opencode/skills/cc-ci-tests-update/SKILL.md | 13 ++++ cc-ci-plan/reconcile-upstream.sh | 64 ++++++++++++++++++++ 4 files changed, 92 insertions(+), 1 deletion(-) create mode 100755 cc-ci-plan/reconcile-upstream.sh diff --git a/.claude/skills/ci-test-review/SKILL.md b/.claude/skills/ci-test-review/SKILL.md index de06743..8d9072c 100644 --- a/.claude/skills/ci-test-review/SKILL.md +++ b/.claude/skills/ci-test-review/SKILL.md @@ -89,6 +89,19 @@ For each real (non-flaky) finding, write the actual fix and open a PR. **Never m branch in a SEPARATE clone — **never push `main`, never touch the build loops' working clones** (`/cc-ci`, `/cc-ci-adv`) or their in-flight state. +> ### ⚠️ RECONCILE FROM UPSTREAM FIRST — always, before any PR work or upgrade check +> ``` +> cc-ci-plan/reconcile-upstream.sh ... # or --all +> ``` +> Deterministic, idempotent, and safe (recipe work lives in branches, never on mirror `main`). It +> force-syncs each mirror to coopcloud's **default branch — resolved from the API, `main` OR +> `master`** — and closes any mirror PR whose changes upstream already merged. Skipping it has cost +> us three distinct ways: mailu #6 was reported as the fix for two internet-facing CVEs while +> upstream had already merged AND released it; a stale mirror makes a survey report "no upgrades +> available" so the recipe drops out of the weekly run; and reading the wrong branch on a recipe with +> a stale `main` beside a live `master` (gitea) manufactures a false "three releases behind, missing +> two CVSS-9.8 RCEs" finding. + ### 5. VERIFY each PR on the CI server (deterministic; still never merge) A PR is only "working" once **cc-ci verifies it green** (operator rule) — dogfood the CI that found the bug. Verification is deterministic (the harness), not an AI judgement. diff --git a/.claude/skills/cve-check/SKILL.md b/.claude/skills/cve-check/SKILL.md index 1a64867..999cc16 100644 --- a/.claude/skills/cve-check/SKILL.md +++ b/.claude/skills/cve-check/SKILL.md @@ -57,7 +57,8 @@ This is `/recipe-upgrade` step 1's research, stopping before it implements anyth > vectorchord/pgvectors versions immich-server is built against, so taking the newest would break > the deploy. `all_resolved: false` means an image could NOT be resolved — that is a `?`, never a 0. -**Reconcile the mirror from true upstream FIRST — ALWAYS, no exceptions.** This is the same reconcile +**Reconcile the mirror from true upstream FIRST — ALWAYS, no exceptions** — one command, +`cc-ci-plan/reconcile-upstream.sh ... | --all`. This is the same reconcile `/upgrade-all` does. Do not skip it in the name of keeping the sweep read-only: skipping it makes you research a stale checkout, and on the first real run that produced **two recipes with no survey output at all**, which is indistinguishable from "no upgrades" unless you check. It is safe — recipe work diff --git a/.opencode/skills/cc-ci-tests-update/SKILL.md b/.opencode/skills/cc-ci-tests-update/SKILL.md index d94bbcb..63e636b 100644 --- a/.opencode/skills/cc-ci-tests-update/SKILL.md +++ b/.opencode/skills/cc-ci-tests-update/SKILL.md @@ -83,6 +83,19 @@ failure (AI — this is the `ci-test-review` step-3 diagnosis): changed upstream, what the test currently asserts. - **FLAKY** → re-run once or twice; if it passes, drop it (not stale, just flaky). +> ### ⚠️ RECONCILE FROM UPSTREAM FIRST — always, before any PR work or upgrade check +> ``` +> cc-ci-plan/reconcile-upstream.sh ... # or --all +> ``` +> Deterministic, idempotent, and safe (recipe work lives in branches, never on mirror `main`). It +> force-syncs each mirror to coopcloud's **default branch — resolved from the API, `main` OR +> `master`** — and closes any mirror PR whose changes upstream already merged. Skipping it has cost +> us three distinct ways: mailu #6 was reported as the fix for two internet-facing CVEs while +> upstream had already merged AND released it; a stale mirror makes a survey report "no upgrades +> available" so the recipe drops out of the weekly run; and reading the wrong branch on a recipe with +> a stale `main` beside a live `master` (gitea) manufactures a false "three releases behind, missing +> two CVSS-9.8 RCEs" finding. + ### 2. For each stale test — author the minimal test update (AI; never weaken) > **Read `tests/STYLE.md` in the cc-ci repo before writing the update.** It is the rulebook for test diff --git a/cc-ci-plan/reconcile-upstream.sh b/cc-ci-plan/reconcile-upstream.sh new file mode 100755 index 0000000..0ac88cb --- /dev/null +++ b/cc-ci-plan/reconcile-upstream.sh @@ -0,0 +1,64 @@ +#!/usr/bin/env bash +# reconcile-upstream — sync recipe mirrors from TRUE upstream. Run this FIRST, always. +# ---------------------------------------------------------------------------------- +# Every recipe we maintain is a MIRROR of a coopcloud recipe. Work done against a stale +# mirror is wasted or wrong, in three ways we have actually hit: +# +# 1. A PR whose changes upstream ALREADY MERGED. mailu #6 (2024.06.57 + redis 8.10, +# two internet-facing Roundcube CVEs) sat open and was reported as the fix for +# those CVEs — while upstream had merged and released it as 3.1.3+2024.06.57. The +# work was done; only our mirror was behind. +# 2. A survey that reads the stale mirror and reports "no upgrades available", so a +# recipe silently drops out of the weekly run. +# 3. Reading the WRONG BRANCH. Several coopcloud recipes keep a stale `main` beside +# the real default `master` — gitea's `main` is at 1.24.2-rootless while `master` +# has 1.27.1-rootless plus the merged PRs. Reading `main` there says the recipe is +# three releases behind and missing two CVSS-9.8 RCE fixes, which reads exactly +# like a real finding. open-recipe-pr.sh resolves the default branch itself +# (main OR master) — never hand-pick one. +# +# This is deterministic: it force-syncs each mirror's `main` to upstream's default +# branch and closes any mirror PR whose changes are already upstream. No AI judgement. +# +# reconcile-upstream.sh ... # specific recipes +# reconcile-upstream.sh --all # every recipe in used-recipes.md +# +# Safe to run repeatedly; a mirror already in sync is a no-op. Recipe work lives in +# BRANCHES, never on mirror `main`, so force-syncing `main` discards nothing. +set -o errexit -o nounset -o pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ORCH="$(dirname "$HERE")" +SSH="${SSH:-cc-ci}" +TESTENV="${TESTENV:-/srv/cc-ci/.testenv}" +RECONCILE="${RECONCILE:-$ORCH/.claude/skills/recipe-upgrade/open-recipe-pr.sh}" +USED_RECIPES="${USED_RECIPES:-$HERE/used-recipes.md}" + +[ -f "$RECONCILE" ] || { echo "ERROR: reconcile helper not found: $RECONCILE" >&2; exit 1; } +set -a; . "$TESTENV"; set +a +: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:?}" + +if [ "${1:-}" = "--all" ]; then + mapfile -t RECIPES < <(awk '!/^[[:space:]]*#/ && ($2=="weekly" || $2=="external") {print $1}' "$USED_RECIPES") +else + [ "$#" -gt 0 ] || { echo "usage: reconcile-upstream.sh ... | --all" >&2; exit 2; } + RECIPES=("$@") +fi + +synced=0; closed=0; failed=0 +for r in "${RECIPES[@]}"; do + echo "── $r" + if out="$(ssh "$SSH" "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s $r --reconcile-only" < "$RECONCILE" 2>&1)"; then + printf '%s\n' "$out" | grep -E "Force-syncing|already in sync|closed PR|still open|✓" | sed 's/^/ /' || true + synced=$((synced + 1)) + closed=$((closed + $(printf '%s' "$out" | grep -c "closed PR" || true))) + else + printf '%s\n' "$out" | tail -3 | sed 's/^/ /' + echo " ✗ FAILED — do NOT proceed against this mirror until it reconciles" + failed=$((failed + 1)) + fi +done + +echo +echo "reconcile-upstream: ${synced} mirror(s) synced, ${closed} already-upstream PR(s) closed, ${failed} failed" +[ "$failed" -eq 0 ] From 94ce5c4de20a92b2b9226da145f62c367432017c Mon Sep 17 00:00:00 2001 From: autonomic-bot Date: Tue, 11 Aug 2026 18:54:58 +0000 Subject: [PATCH 4/4] =?UTF-8?q?cc-ci-status:=20correct=20the=20ENOSPC=20no?= =?UTF-8?q?te=20=E2=80=94=20inode=20recreation=20was=20coincidence?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I recorded that recreating the runs directory with a fresh inode preceded recovery. It recurred afterwards (build 1252), so that was not the fix. The real signal is that it is INTERMITTENT and tracks concurrent activity: every failure landed while a second run or manual deploy was in flight, and every build on a quiet host passed (1243, 1250, 1251, 1253). Free space never moves during a failing build. Practical guidance is therefore to wait for the host to go quiet and re-trigger before calling it a recipe failure, and DRONE_RUNNER_CAPACITY=2 is the obvious knob to test if it becomes disruptive. Root cause still not established, and the note now says so plainly rather than presenting a coincidence as understood. --- .opencode/skills/cc-ci-status/SKILL.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/.opencode/skills/cc-ci-status/SKILL.md b/.opencode/skills/cc-ci-status/SKILL.md index b33f962..6c58cd7 100644 --- a/.opencode/skills/cc-ci-status/SKILL.md +++ b/.opencode/skills/cc-ci-status/SKILL.md @@ -100,10 +100,16 @@ systemctl --failed --no-legend; df -h / | tail -1; tmux ls > died on `mkdir /var/lib/cc-ci-runs/` with **110GB free and 16% inodes**, while the identical > mkdir succeeded as root over ssh, inside the runner's own mount namespace, and 61/61 times in a > stress loop — and the same harness run by hand with a numeric run id worked fine. Restarting - > `drone-runner-exec` did NOT help. What preceded recovery was **recreating the runs directory with - > a fresh inode** (`mv cc-ci-runs cc-ci-runs.old && mkdir -m 750 cc-ci-runs && mv cc-ci-runs.old/* cc-ci-runs/`), - > after which builds ran normally. Root cause was NOT conclusively identified — record the evidence - > if it recurs rather than assuming this is understood. + > `drone-runner-exec` did NOT help, and neither did recreating the runs directory with a fresh + > inode (it recurred afterwards — that apparent fix was coincidence). + > + > **It is INTERMITTENT and tracks concurrent activity**, which is the useful signal: every failure + > landed while a second run or a manual deploy was in flight (1252 was triggered while 1251 was + > still finishing), and every build on a quiet host succeeded (1243, 1250, 1251, 1253). Free space + > never moved during a failing build. So on ENOSPC-with-free-disk: **wait for the host to go quiet + > and re-trigger** before treating it as a recipe failure. Root cause is still NOT established; + > `DRONE_RUNNER_CAPACITY=2` allows the overlap, so lowering it to 1 is the obvious next experiment + > if it becomes disruptive. - **Bridge / !testme path**: `docker service ls` shows `ccci-bridge_app 1/1` AND the bridge log has no auth errors (`docker service logs --since 24h ccci-bridge_app 2>&1 | grep -ci "401\|user does not exist"` == 0). A silently-401ing bridge drops `!testme` (seen 2026-08-03, stale rotated Gitea secret) →