upgrade-all: proxy VIP-exhaustion guard in Step 0; runbooks for proxy /16 enlarge + ghost PR debug
Root-caused (empirically, dockerd logs) the discourse/ghost deploy wedges: the shared proxy overlay (/24=254 VIPs) exhausts as concurrent stack rm leaks endpoints over many days -> tasks stuck in Swarm 'New'. Add a per-run safety net to Step 0 (network prune + docker restart when VIP-allocation failures are logged). Plans + memory for the durable fix (enlarge proxy to /16 in swarm.nix, maintenance window) and for debugging/fixing the ghost PR afterward.
This commit is contained in:
autonomic-bot
@@ -528,3 +528,18 @@ session cc-ci-orchestrator-stale can be killed; recipe-mirrors org still private
|
||||
it with the diagnosis → "one clean discourse retry then move on regardless; comment+skip
|
||||
if it re-wedges". Agent recovered, now checking build state before retry. Rest of queue
|
||||
(ghost/immich/keycloak/lasuite-*/mailu/matrix-synapse) still ahead. cfold still paused.
|
||||
|
||||
## 2026-06-12 ~03:30 — ROOT CAUSE: proxy overlay VIP exhaustion (not "tired box")
|
||||
- Empirically verified from dockerd logs: the shared `proxy` overlay (10.0.1.0/24 = 254 VIPs,
|
||||
joined by every recipe deploy) exhausted its IP pool. Endpoint-GC race on concurrent stack rm
|
||||
(`key modified`/`network proxy remove failed`, 45×) leaked IPs over 11 days of dockerd uptime →
|
||||
13× `could not find an available IP while allocating VIP` from 22:53 → tasks stuck in Swarm `New`
|
||||
→ discourse + ghost deploys wedged (looked like recipe failures; were infra). 02:50 docker
|
||||
restart rebuilt the allocator → cleared.
|
||||
- FIXES: (a) upgrade-all Step 0 now prunes leaked overlays + restarts docker if VIP-failures are in
|
||||
the journal (per-run safety net, committed). (b) DURABLE: enlarge proxy to /16 in swarm.nix —
|
||||
runbook plan-proxy-vip-exhaustion-fix.md + memory [[proxy-vip-exhaustion-runbook]], orchestrator
|
||||
to execute in a maintenance window AFTER the current upgrade (recreating proxy disrupts routing).
|
||||
(c) ghost PR debug: plan-ghostpr-debug-fix.md + memory [[ghost-pr-debug]].
|
||||
- NOT switching the upgrade to sequential (operator: concurrency is fine; the leak is the issue).
|
||||
Duplicate ghost subagent from the interrupt churn — told the upgrader to TaskStop one.
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
# Plan — debug & fix the ghost recipe upgrade PR
|
||||
|
||||
**Context:** during the 2026-06-12 weekly upgrade, ghost (ghost 6.42.0→6.44.1 + mysql bump) was the
|
||||
recipe whose `!testme` kept wedging. Its test deploys (`ghos-bdd2f3` etc.) hung at 0/1 in Swarm
|
||||
`New` state — which we now know was the **`proxy` VIP exhaustion** (see
|
||||
[[proxy-vip-exhaustion-runbook]] / `plan-proxy-vip-exhaustion-fix.md`), NOT necessarily a ghost
|
||||
defect. It also got run by a DUPLICATE subagent during the interrupt churn, so the PR/branch state
|
||||
may be messy. This plan figures out what actually went wrong and leaves the ghost PR clean + green.
|
||||
|
||||
**Execute AFTER** the proxy VIP fix (so the infra confound is gone) and the current upgrade settles.
|
||||
Owner: orchestrator, or a focused `/recipe-upgrade ghost` re-run.
|
||||
|
||||
## Steps
|
||||
1. **Inventory the ghost PR state.** On recipe-maintainers/ghost: list open PRs — is there ONE
|
||||
upgrade PR or a DUPLICATE (two branches/PRs from the two ghost subagents)? Capture each PR's
|
||||
branch, diff (image tag + version-label bumps), and its `!testme` comment history / build
|
||||
results. Read the upgrader transcript for both ghost subagents to see what each did.
|
||||
2. **Separate infra failure from real failure.** The deploy wedges were proxy-VIP exhaustion
|
||||
(infra). Determine whether ghost ALSO has a genuine upgrade problem: does ghost 6.44.1 + the
|
||||
mysql bump deploy + pass its tests on a HEALTHY swarm? Re-run `!testme` on the ghost PR now that
|
||||
the box is healthy (post docker-restart / post proxy fix) and watch the real result.
|
||||
3. **Dedup.** If two ghost PRs/branches exist, keep the correct one (right version bump, clean
|
||||
diff), close the duplicate with a note, and ensure no leftover `dev-ghost`/`ghos-*` stacks remain
|
||||
(reap).
|
||||
4. **Fix forward to green.** If `!testme` is RED for a REAL reason (e.g. ghost 6.44.1 needs a config
|
||||
change, or the mysql major bump needs a migration step / a genuinely-stale test): apply the
|
||||
minimal recipe fix per `/recipe-upgrade` rules — recipe PR changes only; if a cc-ci TEST is
|
||||
genuinely stale, leave an explanatory PR COMMENT (do NOT edit tests in default mode). Iterate
|
||||
`!testme` ≤3× to green.
|
||||
5. **Leave it operator-ready.** One clean ghost PR, `!testme` GREEN (or a clear comment explaining a
|
||||
legitimately-deferred issue), no duplicate, no leaked deploys. NEVER merge — operator merges.
|
||||
|
||||
## Acceptance
|
||||
The ghost upgrade is represented by exactly one PR with a clear, green (or clearly-explained)
|
||||
`!testme`, the duplicate-subagent mess cleaned, and a one-line note on whether ghost's original
|
||||
failure was purely the proxy-VIP infra issue or a real upgrade problem (and how it was fixed).
|
||||
|
||||
## Guardrails
|
||||
Recipe mirror = PR only, never merge / never push main. Reap any `dev-ghost`/`ghos-*` test stacks on
|
||||
exit. No secrets in logs/commits. Don't run while the proxy recreate (maintenance window) is in
|
||||
progress.
|
||||
@@ -0,0 +1,59 @@
|
||||
# Runbook — fix `proxy` overlay VIP exhaustion (durable) + empirical verification
|
||||
|
||||
**Owner: ORCHESTRATOR** (host/swarm infra, not a recipe/test change). Execute **after the current
|
||||
weekly upgrade run finishes** (the box must be quiescent — recreating `proxy` disrupts traefik
|
||||
routing for every live service). Do NOT run mid-upgrade.
|
||||
|
||||
## Root cause (empirically verified 2026-06-12, from dockerd logs)
|
||||
- The shared **`proxy` overlay network** (ID was `ab54…`) is **`10.0.1.0/24` = 254 VIPs**. EVERY
|
||||
recipe deploy joins it (traefik routing).
|
||||
- Under concurrent stack `rm`, Swarm's endpoint GC races (`Unable to complete atomic operation,
|
||||
key modified` / `network proxy remove failed`) and **leaks endpoints → leaks IPs** (45 such
|
||||
errors over the day). `dockerd` had **11 days** uptime accumulating leaks.
|
||||
- The pool exhausted → **13× `could not find an available IP while allocating VIP`** (first 22:53,
|
||||
straddling both wedges) → new services' tasks stuck in Swarm **`New`** state (never scheduled).
|
||||
- The 02:50 docker restart rebuilt the allocator and reclaimed everything → healthy.
|
||||
- This presents as a recipe FAILURE (discourse, ghost both "failed") but is purely infra.
|
||||
|
||||
## The fix (durable): enlarge the `proxy` subnet
|
||||
`nix/modules/swarm.nix:~43` creates it with no `--subnet` (defaults to a /24):
|
||||
```
|
||||
docker network create --driver overlay --attachable proxy
|
||||
```
|
||||
Change to a **/16** (≈65,534 VIPs, ~258× headroom — the leak can't reach it before a routine
|
||||
reboot/`nixos-rebuild` resets the allocator). Pick a block clear of `ingress` (10.0.0.0/24) and the
|
||||
current proxy (10.0.1.0/24); the default-addr-pool is 10.0.0.0/8, so use e.g. **`10.10.0.0/16`**:
|
||||
```
|
||||
docker network create --driver overlay --attachable --subnet 10.10.0.0/16 proxy
|
||||
```
|
||||
|
||||
## Procedure
|
||||
1. **Pre-req:** weekly upgrade run done; `docker stack ls` shows only infra + `warm-*`.
|
||||
2. **EMPIRICAL BEFORE — measure the leak.** Baseline `proxy` endpoint/IP count, then deploy +
|
||||
*concurrently* `rm` N (~10) throwaway published-port stacks; re-count. Show endpoints/IPs do NOT
|
||||
return to baseline (leak), and grep dockerd for fresh `key modified`/`network proxy remove`
|
||||
errors. Record the per-cycle leak rate → projects the /24 exhaustion time.
|
||||
3. **Edit `nix/modules/swarm.nix`** — add `--subnet 10.10.0.0/16` to the proxy create (commit to
|
||||
the cc-ci repo; this is infra/nix, orchestrator-authored, push to git.autonomic.zone).
|
||||
4. **Recreate `proxy` on the host (DISRUPTIVE):** the network can't be resized in place. Either
|
||||
`nixos-rebuild` after temporarily removing proxy, or manually: detach services / `docker stack
|
||||
rm` the live recipe stacks (none mid-upgrade), `docker network rm proxy`, recreate with the /16,
|
||||
then redeploy/reconcile traefik + the `ccci-*` control plane + `warm-*` so they rejoin. Verify
|
||||
traefik routing, drone, dashboard, bridge, reports all healthy.
|
||||
5. **`nixos-rebuild switch`** so the /16 persists across reboots (sync `/root/cc-ci` first, per the
|
||||
host-deploy mechanism).
|
||||
6. **EMPIRICAL AFTER — prove it.** Re-run step 2's reproduction: confirm (a) `proxy` now reports a
|
||||
/16 with vast headroom, (b) `docker network prune -f` reclaims the leaked per-stack overlays,
|
||||
(c) the leak no longer approaches exhaustion. Confirm a fresh recipe `!testme` deploys clean (no
|
||||
`New`-state hang).
|
||||
|
||||
## Acceptance
|
||||
`proxy` is a /16 (pinned in swarm.nix, survives rebuild); reproduction shows the leak is bounded
|
||||
far below the new ceiling; the upgrade Step-0 guard (prune + VIP-failure docker-restart, already
|
||||
added to the skill 2026-06-12) remains as the per-run safety net. Then delete the
|
||||
[[proxy-vip-exhaustion-runbook]] memory.
|
||||
|
||||
## Guardrails
|
||||
- Maintenance window only (recreating proxy = brief routing outage for ALL services). Never during
|
||||
a live upgrade or phase run. No secrets in commits. Author `autonomic-bot
|
||||
<autonomic-bot@noreply.git.autonomic.zone>`; push after commit.
|
||||
Reference in New Issue
Block a user