upgrade-all: proxy VIP-exhaustion guard in Step 0; runbooks for proxy /16 enlarge + ghost PR debug
Root-caused (empirically, dockerd logs) the discourse/ghost deploy wedges: the shared proxy overlay (/24=254 VIPs) exhausts as concurrent stack rm leaks endpoints over many days -> tasks stuck in Swarm 'New'. Add a per-run safety net to Step 0 (network prune + docker restart when VIP-allocation failures are logged). Plans + memory for the durable fix (enlarge proxy to /16 in swarm.nix, maintenance window) and for debugging/fixing the ghost PR afterward.
This commit is contained in:
@@ -528,3 +528,18 @@ session cc-ci-orchestrator-stale can be killed; recipe-mirrors org still private
|
||||
it with the diagnosis → "one clean discourse retry then move on regardless; comment+skip
|
||||
if it re-wedges". Agent recovered, now checking build state before retry. Rest of queue
|
||||
(ghost/immich/keycloak/lasuite-*/mailu/matrix-synapse) still ahead. cfold still paused.
|
||||
|
||||
## 2026-06-12 ~03:30 — ROOT CAUSE: proxy overlay VIP exhaustion (not "tired box")
|
||||
- Empirically verified from dockerd logs: the shared `proxy` overlay (10.0.1.0/24 = 254 VIPs,
|
||||
joined by every recipe deploy) exhausted its IP pool. Endpoint-GC race on concurrent stack rm
|
||||
(`key modified`/`network proxy remove failed`, 45×) leaked IPs over 11 days of dockerd uptime →
|
||||
13× `could not find an available IP while allocating VIP` from 22:53 → tasks stuck in Swarm `New`
|
||||
→ discourse + ghost deploys wedged (looked like recipe failures; were infra). 02:50 docker
|
||||
restart rebuilt the allocator → cleared.
|
||||
- FIXES: (a) upgrade-all Step 0 now prunes leaked overlays + restarts docker if VIP-failures are in
|
||||
the journal (per-run safety net, committed). (b) DURABLE: enlarge proxy to /16 in swarm.nix —
|
||||
runbook plan-proxy-vip-exhaustion-fix.md + memory [[proxy-vip-exhaustion-runbook]], orchestrator
|
||||
to execute in a maintenance window AFTER the current upgrade (recreating proxy disrupts routing).
|
||||
(c) ghost PR debug: plan-ghostpr-debug-fix.md + memory [[ghost-pr-debug]].
|
||||
- NOT switching the upgrade to sequential (operator: concurrency is fine; the leak is the issue).
|
||||
Duplicate ghost subagent from the interrupt churn — told the upgrader to TaskStop one.
|
||||
|
||||
Reference in New Issue
Block a user