recipe-maintainers/cc-ci-orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request 'CI server health: image pruning, disk thresholds, reconcile-upstream' (#5) from review/4-ci-health into review/3-upstream-resolution

Browse Source
This commit was merged in pull request #5.
This commit is contained in:
autonomic-bot
2026-08-11 19:03:54 +00:00
parent e89da2d842 94ce5c4de2
commit a0d1b82869
6 changed files with 148 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/skills/ci-test-review/SKILL.md
+13
View File
@@ -89,6 +89,19 @@ For each real (non-flaky) finding, write the actual fix and open a PR. **Never m
branch in a SEPARATE clone — **never push `main`, never touch the build loops' working clones**
(`/cc-ci`, `/cc-ci-adv`) or their in-flight state.
> ### ⚠️ RECONCILE FROM UPSTREAM FIRST — always, before any PR work or upgrade check
> ```
> cc-ci-plan/reconcile-upstream.sh <recipe>... # or --all
> ```
> Deterministic, idempotent, and safe (recipe work lives in branches, never on mirror `main`). It
> force-syncs each mirror to coopcloud's **default branch — resolved from the API, `main` OR
> `master`** — and closes any mirror PR whose changes upstream already merged. Skipping it has cost
> us three distinct ways: mailu #6 was reported as the fix for two internet-facing CVEs while
> upstream had already merged AND released it; a stale mirror makes a survey report "no upgrades
> available" so the recipe drops out of the weekly run; and reading the wrong branch on a recipe with
> a stale `main` beside a live `master` (gitea) manufactures a false "three releases behind, missing
> two CVSS-9.8 RCEs" finding.
### 5. VERIFY each PR on the CI server (deterministic; still never merge)
A PR is only "working" once **cc-ci verifies it green** (operator rule) — dogfood the CI that found
the bug. Verification is deterministic (the harness), not an AI judgement.
.claude/skills/cve-check/SKILL.md
+2 -1
View File
@@ -57,7 +57,8 @@ This is `/recipe-upgrade` step 1's research, stopping before it implements anyth
> vectorchord/pgvectors versions immich-server is built against, so taking the newest would break
> the deploy. `all_resolved: false` means an image could NOT be resolved — that is a `?`, never a 0.
**Reconcile the mirror from true upstream FIRST — ALWAYS, no exceptions.** This is the same reconcile
**Reconcile the mirror from true upstream FIRST — ALWAYS, no exceptions** — one command,
`cc-ci-plan/reconcile-upstream.sh <recipe>... | --all`. This is the same reconcile
`/upgrade-all` does. Do not skip it in the name of keeping the sweep read-only: skipping it makes you
research a stale checkout, and on the first real run that produced **two recipes with no survey output
at all**, which is indistinguishable from "no upgrades" unless you check. It is safe — recipe work
.claude/skills/upgrade-all/sweep-orphans.sh
+23
View File
@@ -73,6 +73,29 @@ done
# 5) Stray exited containers (debug one-shots) — best-effort prune.
docker container prune -f >/dev/null 2>&1 || true
# 6) Unused IMAGES — the one that actually took CI down. Every run pulls each recipe's images and
# nothing ever removed the old ones: on 2026-08-11 they had grown to 72GB (63GB of it unused),
# the root filesystem hit 100% under two concurrent runs, and the harness died at startup with
# `OSError: [Errno 28] No space left on device: '/var/lib/cc-ci-runs/<build>'`. Every !testme
# from build 1236 to 1242 failed that way — with no results.json, so the PR badges just read
# "failure" and looked like recipe regressions.
#
# Only prune above a threshold, so a healthy host keeps its layer cache and runs stay fast.
# `image prune -a` removes only images no container references, so anything deployed (infra +
# warm-* canonicals) is untouched; anything else is re-pulled on demand.
#
# Volumes are deliberately NOT pruned here — see the KEEP_RE guard in (3): warm-* canonicals are
# data-warm and their volumes are legitimately dangling between runs.
DISK_PRUNE_PCT="${DISK_PRUNE_PCT:-60}"
used_pct="$(df --output=pcent / 2>/dev/null | tail -1 | tr -dc '0-9')"
if [ -n "$used_pct" ] && [ "$used_pct" -ge "$DISK_PRUNE_PCT" ]; then
echo " disk ${used_pct}% >= ${DISK_PRUNE_PCT}% -> pruning unused images"
freed="$(docker image prune -af 2>/dev/null | awk '/Total reclaimed space/ {print $4, $5}')"
echo " reclaimed: ${freed:-0B}; disk now $(df -h / | tail -1 | awk '{print $5" used, "$4" free"}')"
else
echo " disk ${used_pct:-?}% < ${DISK_PRUNE_PCT}% -> keeping image cache"
fi
if [ "$removed" -eq 0 ]; then
echo "== orphan sweep: clean (nothing to remove) =="
else
.opencode/skills/cc-ci-status/SKILL.md
+33 -2
View File
@@ -78,8 +78,38 @@ ssh cc-ci 'systemctl --failed --no-legend; df -h / | tail -1; docker service ls
systemctl --failed --no-legend; df -h / | tail -1; tmux ls
```
- Failed units, core swarm services not 1/1 (warm-* spares flapping is a known benign pattern —
note, don't page), disk >80% (server) / >85% (orchestrator) → findings. Server unreachable →
note, don't page), disk **>65% (server)** / >85% (orchestrator) → findings. Server unreachable →
HIGH: recommend `hetzner-server-recovery`.
> **65%, not 80%, on the server — it is not a steady-state measure.** Two concurrent recipe runs
> pull images and write volumes worth tens of GB, so a host sitting at 73% still hits 100% mid-run.
> That is exactly what happened on 2026-08-11: 63GB of unused images had accumulated (nothing ever
> pruned them), the filesystem filled during a run, and the harness died at startup with
> `OSError: [Errno 28] No space left on device`. Remedy: `docker image prune -af` on cc-ci — it
> spares anything a container references, so infra and warm-* canonicals are untouched. Do NOT
> `docker volume prune`: warm-* canonical volumes are data-warm and legitimately dangling.
- **!testme actually produces results** (the check that would have caught the above days earlier):
the newest few `/var/lib/cc-ci-runs/<build>/` dirs must each contain `results.json`. A build that
dies before the harness writes one leaves an EMPTY dir — and the PR badge still says "failure", so
it reads as a recipe regression rather than a sick host. Builds 1236–1242 all failed that way.
Finding: *"N recent builds produced no results.json — the harness is dying at startup, check disk
and the drone step log"*. The step log lives in drone's sqlite
(`/var/lib/docker/volumes/drone_ci_commoninternet_net_data/_data/database.sqlite`) — copy it and
read `logs.log_data` for the failing `steps.step_id`; the bridge's drone token is not extractable
(distroless container, swarm secret).
> **If the error is ENOSPC but the disk is fine**, it is not disk. Seen 2026-08-11: builds 1244-1249
> died on `mkdir /var/lib/cc-ci-runs/<build>` with **110GB free and 16% inodes**, while the identical
> mkdir succeeded as root over ssh, inside the runner's own mount namespace, and 61/61 times in a
> stress loop — and the same harness run by hand with a numeric run id worked fine. Restarting
> `drone-runner-exec` did NOT help, and neither did recreating the runs directory with a fresh
> inode (it recurred afterwards — that apparent fix was coincidence).
>
> **It is INTERMITTENT and tracks concurrent activity**, which is the useful signal: every failure
> landed while a second run or a manual deploy was in flight (1252 was triggered while 1251 was
> still finishing), and every build on a quiet host succeeded (1243, 1250, 1251, 1253). Free space
> never moved during a failing build. So on ENOSPC-with-free-disk: **wait for the host to go quiet
> and re-trigger** before treating it as a recipe failure. Root cause is still NOT established;
> `DRONE_RUNNER_CAPACITY=2` allows the overlap, so lowering it to 1 is the obvious next experiment
> if it becomes disruptive.
- **Bridge / !testme path**: `docker service ls` shows `ccci-bridge_app 1/1` AND the bridge log
has no auth errors (`docker service logs --since 24h ccci-bridge_app 2>&1 | grep -ci "401\|user does not exist"` == 0).
A silently-401ing bridge drops `!testme` (seen 2026-08-03, stale rotated Gitea secret) →
@@ -113,7 +143,8 @@ minutes, no PRs). If it is instead that a known CVE is sitting unpatched, recomm
`/cve-check` over waiting for the next weekly run whenever the question is "are we exposed?".
`ALL HEALTHY` requires: recent successful weekly run + published report, no stale tests, no
CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, disk under
CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, recent builds all
producing results.json, disk under
thresholds, bridge clean, maintained-set consistent. Anything else is a finding — even minor
ones get a recommended next step. Order findings by priority (CVE/unreachable-host first).
.opencode/skills/cc-ci-tests-update/SKILL.md
+13
View File
@@ -83,6 +83,19 @@ failure (AI — this is the `ci-test-review` step-3 diagnosis):
changed upstream, what the test currently asserts.
- **FLAKY** → re-run once or twice; if it passes, drop it (not stale, just flaky).
> ### ⚠️ RECONCILE FROM UPSTREAM FIRST — always, before any PR work or upgrade check
> ```
> cc-ci-plan/reconcile-upstream.sh <recipe>... # or --all
> ```
> Deterministic, idempotent, and safe (recipe work lives in branches, never on mirror `main`). It
> force-syncs each mirror to coopcloud's **default branch — resolved from the API, `main` OR
> `master`** — and closes any mirror PR whose changes upstream already merged. Skipping it has cost
> us three distinct ways: mailu #6 was reported as the fix for two internet-facing CVEs while
> upstream had already merged AND released it; a stale mirror makes a survey report "no upgrades
> available" so the recipe drops out of the weekly run; and reading the wrong branch on a recipe with
> a stale `main` beside a live `master` (gitea) manufactures a false "three releases behind, missing
> two CVSS-9.8 RCEs" finding.
### 2. For each stale test — author the minimal test update (AI; never weaken)
> **Read `tests/STYLE.md` in the cc-ci repo before writing the update.** It is the rulebook for test
cc-ci-plan/reconcile-upstream.sh
Executable
+64
View File
@@ -0,0 +1,64 @@
#!/usr/bin/env bash
# reconcile-upstream — sync recipe mirrors from TRUE upstream. Run this FIRST, always.
# ----------------------------------------------------------------------------------
# Every recipe we maintain is a MIRROR of a coopcloud recipe. Work done against a stale
# mirror is wasted or wrong, in three ways we have actually hit:
#
# 1. A PR whose changes upstream ALREADY MERGED. mailu #6 (2024.06.57 + redis 8.10,
# two internet-facing Roundcube CVEs) sat open and was reported as the fix for
# those CVEs — while upstream had merged and released it as 3.1.3+2024.06.57. The
# work was done; only our mirror was behind.
# 2. A survey that reads the stale mirror and reports "no upgrades available", so a
# recipe silently drops out of the weekly run.
# 3. Reading the WRONG BRANCH. Several coopcloud recipes keep a stale `main` beside
# the real default `master` — gitea's `main` is at 1.24.2-rootless while `master`
# has 1.27.1-rootless plus the merged PRs. Reading `main` there says the recipe is
# three releases behind and missing two CVSS-9.8 RCE fixes, which reads exactly
# like a real finding. open-recipe-pr.sh resolves the default branch itself
# (main OR master) — never hand-pick one.
#
# This is deterministic: it force-syncs each mirror's `main` to upstream's default
# branch and closes any mirror PR whose changes are already upstream. No AI judgement.
#
# reconcile-upstream.sh <recipe>... # specific recipes
# reconcile-upstream.sh --all # every recipe in used-recipes.md
#
# Safe to run repeatedly; a mirror already in sync is a no-op. Recipe work lives in
# BRANCHES, never on mirror `main`, so force-syncing `main` discards nothing.
set -o errexit -o nounset -o pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ORCH="$(dirname "$HERE")"
SSH="${SSH:-cc-ci}"
TESTENV="${TESTENV:-/srv/cc-ci/.testenv}"
RECONCILE="${RECONCILE:-$ORCH/.claude/skills/recipe-upgrade/open-recipe-pr.sh}"
USED_RECIPES="${USED_RECIPES:-$HERE/used-recipes.md}"
[ -f "$RECONCILE" ] || { echo "ERROR: reconcile helper not found: $RECONCILE" >&2; exit 1; }
set -a; . "$TESTENV"; set +a
: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:?}"
if [ "${1:-}" = "--all" ]; then
mapfile -t RECIPES < <(awk '!/^[[:space:]]*#/ && ($2=="weekly" || $2=="external") {print $1}' "$USED_RECIPES")
else
[ "$#" -gt 0 ] || { echo "usage: reconcile-upstream.sh <recipe>... | --all" >&2; exit 2; }
RECIPES=("$@")
fi
synced=0; closed=0; failed=0
for r in "${RECIPES[@]}"; do
echo "── $r"
if out="$(ssh "$SSH" "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s $r --reconcile-only" < "$RECONCILE" 2>&1)"; then
printf '%s\n' "$out" | grep -E "Force-syncing|already in sync|closed PR|still open|✓" | sed 's/^/ /' || true
synced=$((synced + 1))
closed=$((closed + $(printf '%s' "$out" | grep -c "closed PR" || true)))
else
printf '%s\n' "$out" | tail -3 | sed 's/^/ /'
echo " ✗ FAILED — do NOT proceed against this mirror until it reconciles"
failed=$((failed + 1))
fi
done
echo
echo "reconcile-upstream: ${synced} mirror(s) synced, ${closed} already-upstream PR(s) closed, ${failed} failed"
[ "$failed" -eq 0 ]