---
name: proxy-vip-exhaustion-runbook
description: TODO after the weekly upgrade — enlarge the proxy overlay subnet to /16 (it exhausts at /24=254 VIPs); runbook + empirical verify
metadata: 
  node_type: memory
  type: project
  originSessionId: 85355980-5e4f-4f90-b1ca-d0e4fe82f04b
---

**Root cause found 2026-06-12 (empirically, from dockerd logs):** recipe test deploys hung at 0/1 in
Swarm `New` state (looked like discourse/ghost "failing") because the shared **`proxy` overlay
network** (`10.0.1.0/24` = 254 VIPs, joined by every recipe deploy) **exhausted its IP pool**.
Leaked endpoints from concurrent stack `rm` (Swarm endpoint-GC race: `key modified` / `network proxy
remove failed`, 45×) accumulated over 11 days of dockerd uptime → `could not find an available IP
while allocating VIP` (13×). A `docker restart` rebuilds the allocator and reclaims it (proven).

**Per-run safety net (DONE 2026-06-12):** upgrade-all Step 0 now runs `docker network prune -f` + a
guard that restarts docker if recent VIP-allocation failures are in the journal.

**TODO (durable fix, ORCHESTRATOR, in a maintenance window AFTER the current upgrade + when the box
is quiescent — recreating proxy disrupts traefik routing):** enlarge `proxy` to a /16. Edit
`nix/modules/swarm.nix:~43` (`docker network create --driver overlay --attachable proxy` → add
`--subnet 10.10.0.0/16`), recreate the proxy network on the host, `nixos-rebuild`, and empirically
verify (reproduce the leak before/after). Full runbook: `cc-ci-plan/plan-proxy-vip-exhaustion-fix.md`.
Then debug the ghost PR ([[ghost-pr-debug]]). Delete this memory once proxy is /16 + verified.