upgrade-all: proxy VIP-exhaustion guard in Step 0; runbooks for proxy /16 enlarge + ghost PR debug

Root-caused (empirically, dockerd logs) the discourse/ghost deploy wedges:
the shared proxy overlay (/24=254 VIPs) exhausts as concurrent stack rm leaks
endpoints over many days -> tasks stuck in Swarm 'New'. Add a per-run safety
net to Step 0 (network prune + docker restart when VIP-allocation failures are
logged). Plans + memory for the durable fix (enlarge proxy to /16 in swarm.nix,
maintenance window) and for debugging/fixing the ghost PR afterward.
This commit is contained in:
autonomic-bot
2026-06-12 03:30:00 +00:00
parent 7ce898e0e4
commit ca02a0dd6f
7 changed files with 184 additions and 0 deletions
+15
View File
@@ -528,3 +528,18 @@ session cc-ci-orchestrator-stale can be killed; recipe-mirrors org still private
it with the diagnosis → "one clean discourse retry then move on regardless; comment+skip
if it re-wedges". Agent recovered, now checking build state before retry. Rest of queue
(ghost/immich/keycloak/lasuite-*/mailu/matrix-synapse) still ahead. cfold still paused.
## 2026-06-12 ~03:30 — ROOT CAUSE: proxy overlay VIP exhaustion (not "tired box")
- Empirically verified from dockerd logs: the shared `proxy` overlay (10.0.1.0/24 = 254 VIPs,
joined by every recipe deploy) exhausted its IP pool. Endpoint-GC race on concurrent stack rm
(`key modified`/`network proxy remove failed`, 45×) leaked IPs over 11 days of dockerd uptime →
13× `could not find an available IP while allocating VIP` from 22:53 → tasks stuck in Swarm `New`
→ discourse + ghost deploys wedged (looked like recipe failures; were infra). 02:50 docker
restart rebuilt the allocator → cleared.
- FIXES: (a) upgrade-all Step 0 now prunes leaked overlays + restarts docker if VIP-failures are in
the journal (per-run safety net, committed). (b) DURABLE: enlarge proxy to /16 in swarm.nix —
runbook plan-proxy-vip-exhaustion-fix.md + memory [[proxy-vip-exhaustion-runbook]], orchestrator
to execute in a maintenance window AFTER the current upgrade (recreating proxy disrupts routing).
(c) ghost PR debug: plan-ghostpr-debug-fix.md + memory [[ghost-pr-debug]].
- NOT switching the upgrade to sequential (operator: concurrency is fine; the leak is the issue).
Duplicate ghost subagent from the interrupt churn — told the upgrader to TaskStop one.