# Runbook — fix `proxy` overlay VIP exhaustion (durable) + empirical verification

**Owner: ORCHESTRATOR** (host/swarm infra, not a recipe/test change). Execute **after the current
weekly upgrade run finishes** (the box must be quiescent — recreating `proxy` disrupts traefik
routing for every live service). Do NOT run mid-upgrade.

## Root cause (empirically verified 2026-06-12, from dockerd logs)
- The shared **`proxy` overlay network** (ID was `ab54…`) is **`10.0.1.0/24` = 254 VIPs**. EVERY
  recipe deploy joins it (traefik routing).
- Under concurrent stack `rm`, Swarm's endpoint GC races (`Unable to complete atomic operation,
  key modified` / `network proxy remove failed`) and **leaks endpoints → leaks IPs** (45 such
  errors over the day). `dockerd` had **11 days** uptime accumulating leaks.
- The pool exhausted → **13× `could not find an available IP while allocating VIP`** (first 22:53,
  straddling both wedges) → new services' tasks stuck in Swarm **`New`** state (never scheduled).
- The 02:50 docker restart rebuilt the allocator and reclaimed everything → healthy.
- This presents as a recipe FAILURE (discourse, ghost both "failed") but is purely infra.

## The fix (durable): enlarge the `proxy` subnet
`nix/modules/swarm.nix:~43` creates it with no `--subnet` (defaults to a /24):
```
docker network create --driver overlay --attachable proxy
```
Change to a **/16** (≈65,534 VIPs, ~258× headroom — the leak can't reach it before a routine
reboot/`nixos-rebuild` resets the allocator). Pick a block clear of `ingress` (10.0.0.0/24) and the
current proxy (10.0.1.0/24); the default-addr-pool is 10.0.0.0/8, so use e.g. **`10.10.0.0/16`**:
```
docker network create --driver overlay --attachable --subnet 10.10.0.0/16 proxy
```

## Procedure
1. **Pre-req:** weekly upgrade run done; `docker stack ls` shows only infra + `warm-*`.
2. **EMPIRICAL BEFORE — measure the leak.** Baseline `proxy` endpoint/IP count, then deploy +
   *concurrently* `rm` N (~10) throwaway published-port stacks; re-count. Show endpoints/IPs do NOT
   return to baseline (leak), and grep dockerd for fresh `key modified`/`network proxy remove`
   errors. Record the per-cycle leak rate → projects the /24 exhaustion time.
3. **Edit `nix/modules/swarm.nix`** — add `--subnet 10.10.0.0/16` to the proxy create (commit to
   the cc-ci repo; this is infra/nix, orchestrator-authored, push to git.autonomic.zone).
4. **Recreate `proxy` on the host (DISRUPTIVE):** the network can't be resized in place. Either
   `nixos-rebuild` after temporarily removing proxy, or manually: detach services / `docker stack
   rm` the live recipe stacks (none mid-upgrade), `docker network rm proxy`, recreate with the /16,
   then redeploy/reconcile traefik + the `ccci-*` control plane + `warm-*` so they rejoin. Verify
   traefik routing, drone, dashboard, bridge, reports all healthy.
5. **`nixos-rebuild switch`** so the /16 persists across reboots (sync `/root/cc-ci` first, per the
   host-deploy mechanism).
6. **EMPIRICAL AFTER — prove it.** Re-run step 2's reproduction: confirm (a) `proxy` now reports a
   /16 with vast headroom, (b) `docker network prune -f` reclaims the leaked per-stack overlays,
   (c) the leak no longer approaches exhaustion. Confirm a fresh recipe `!testme` deploys clean (no
   `New`-state hang).

## Acceptance
`proxy` is a /16 (pinned in swarm.nix, survives rebuild); reproduction shows the leak is bounded
far below the new ceiling; the upgrade Step-0 guard (prune + VIP-failure docker-restart, already
added to the skill 2026-06-12) remains as the per-run safety net. Then delete the
[[proxy-vip-exhaustion-runbook]] memory.

## Guardrails
- Maintenance window only (recreating proxy = brief routing outage for ALL services). Never during
  a live upgrade or phase run. No secrets in commits. Author `autonomic-bot
  <autonomic-bot@noreply.git.autonomic.zone>`; push after commit.