recipe-maintainers/cc-ci-orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

upgrade-all: proxy VIP-exhaustion guard in Step 0; runbooks for proxy /16 enlarge + ghost PR debug

Browse Source 
Root-caused (empirically, dockerd logs) the discourse/ghost deploy wedges:
the shared proxy overlay (/24=254 VIPs) exhausts as concurrent stack rm leaks
endpoints over many days -> tasks stuck in Swarm 'New'. Add a per-run safety
net to Step 0 (network prune + docker restart when VIP-allocation failures are
logged). Plans + memory for the durable fix (enlarge proxy to /16 in swarm.nix,
maintenance window) and for debugging/fixing the ghost PR afterward.
This commit is contained in:
autonomic-bot
2026-06-12 03:30:00 +00:00
parent 7ce898e0e4
commit ca02a0dd6f
7 changed files with 184 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/skills/upgrade-all/SKILL.md
+22
View File
@@ -58,6 +58,28 @@ the dedicated reaper too so the start/end cleanup is explicit and symmetric (`TH
ssh cc-ci 'THRESHOLD=0 bash -s' < /srv/cc-ci/.claude/skills/upgrade-all/reap-dev-deploys.sh
```
Then **reclaim leaked overlay IPs and guard against `proxy` VIP exhaustion.** The shared `proxy`
overlay (a `/24` = 254 VIPs that EVERY recipe deploy joins) leaks endpoints under concurrent stack
`rm` (a Swarm endpoint-GC race); over many days of churn the pool exhausts and new test deploys hang
in Swarm `New` state with `could not find an available IP while allocating VIP` — which looks exactly
like a recipe failure but is infra (root-caused 2026-06-12; see cc-ci-plan `plan-proxy-vip-exhaustion-fix.md`).
Reclaim before the run:
```
# 1. reclaim leaked per-stack overlay networks (cheap, always safe)
ssh cc-ci 'docker network prune -f'
# 2. proxy VIP-exhaustion guard: if the allocator recently failed to assign a VIP, the leak has hit
# the ceiling — rebuild the allocator with a docker restart (the box is QUIESCENT at run start, so
# this is a ~30s infra blip that auto-recovers). Only fires when actually needed.
VIPFAIL=$(ssh cc-ci 'journalctl -u docker --since "26 hours ago" --no-pager 2>/dev/null | grep -c "available IP while allocating VIP"')
if [ "${VIPFAIL:-0}" -gt 0 ]; then
echo "!! proxy VIP exhaustion detected ($VIPFAIL recent failures) — restarting docker to reclaim leaked endpoints"
ssh cc-ci 'sudo systemctl restart docker'; sleep 25
ssh cc-ci 'docker node ls && docker service ls --format "{{.Replicas}}" | grep -c "/"' # sanity: node Ready, infra back
fi
```
(The durable fix — enlarging the `proxy` subnet to a /16 so it never exhausts — is tracked in
`plan-proxy-vip-exhaustion-fix.md`; this guard is the per-run safety net until that lands.)
## 1. Build the candidate list
Enrolled recipes = the cc-ci `tests/<recipe>/` dirs (same set `ci-test-review` sweeps):
```