recipe-maintainers/cc-ci
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
692e6d2108eebdbaefe3799a501c9abebb6057da
cc-ci/machine-docs/REVIEW-pvfix.md
autonomic-bot ccd93da65c
Some checks failed
continuous-integration/drone/push Build is failing
Details
review(pvfix-M2): M2 PASS + [adversary] A1 health gate deadlock 
M2 PASS: proxy confirmed 10.10.0.0/16 (created 05:38:02Z), all 9 services 1/1,
swarm-init active script has --subnet, ci.commoninternet.net=200,
drone.ci.commoninternet.net=303.

A1 [adversary]: deploy-proxy health gate (ci.commoninternet.net=200) circular
with deploy-dashboard After=deploy-proxy ordering — deadlocks on fresh boot
(TimeoutStartSec=900). Pre-existing; pvfix exposed it. Needs fix before D8 pass.
2026-06-13 05:50:22 +00:00

166 lines
6.9 KiB
Markdown
Raw Blame History

# REVIEW — phase pvfix (Adversary)
Adversary clone: `/srv/cc-ci/cc-ci-adv`
Phase plan: `/srv/cc-ci/cc-ci-plan/plan-phase-pvfix-swarm-proxy.md`
---
## Phase context (initial orientation, 2026-06-13T05:30Z)
Cold check of live host and current repo:
- `docker network inspect proxy` → Subnet: `10.0.1.0/24` (default /24 — the exhaustion vector)
- `docker network ls | grep proxy` → `ab54qfa7gsk5 proxy overlay swarm`
- `nix/modules/swarm.nix` → `swarm-init` creates proxy without `--subnet`, inheriting Docker's
default `/24`. No explicit subnet configured.
- Builder has not started pvfix work yet (no STATUS-pvfix.md in repo).
The fix is needed. Watching for Builder M1 claim (patch + procedure + live inspection proof).
### Break-it probe: live host subnet collision check (2026-06-13T05:31Z)
Existing subnets on host:
- `ingress`: `10.0.0.0/24`
- `proxy` (current): `10.0.1.0/24`
- `docker0`: `172.17.0.0/16`
- `docker_gwbridge`: `172.18.0.0/16`
- Host IP: `91.98.47.73` (public), `100.95.31.88` (tailscale), gateway `172.31.1.1`
**10.10.0.0/16 (proposed):** does NOT collide with any existing subnet. Safe.
Services currently on proxy (will be disrupted during recreation):
- `traefik` → 10.0.1.9
- `ccci-reports` → 10.0.1.7
- `drone` → 10.0.1.12
- `ccci-bridge` → 10.0.1.248
- `ccci-dashboard` → 10.0.1.249
- `warm-keycloak` → 10.0.1.251
Stacks currently running (all will briefly lose routing):
`backups`, `ccci-bridge`, `ccci-dashboard`, `ccci-reports`, `drone`, `traefik`, `warm-keycloak`
**Maintenance window status:** CLEAR — no active recipe test stacks (`*-pr*`), no cfold sweep,
no /upgrade-all visible. A quiet window is available now.
**Key risk to probe when M2 is claimed:** confirm that after proxy recreation, all 6 services
above rejoin with healthy VIP allocations and Traefik routes are reachable end-to-end.
---
## M1: PASS @2026-06-13T05:33Z
**Claim:** `nix/modules/swarm.nix` patched with `--subnet 10.10.0.0/16`; maintenance procedure
documented; chosen /16 proven safe from live host inspection.
**Commit:** `e6349a9` (`claim(pvfix-M1): proxy /16 patch + maintenance plan ready`)
### Cold-run evidence
**1. Patch in repo:**
```
grep -n 'subnet' nix/modules/swarm.nix
→ 47: docker network create --driver overlay --attachable --subnet 10.10.0.0/16 proxy
```
Correct. The `if ! docker network inspect proxy` guard ensures idempotent create. Comment
accurately names the failure mode and runbook. ✓
**2. Subnet safety — live host inspection:**
```
docker network inspect $(docker network ls -q) --format "{{.Name}}: {{range .IPAM.Config}}{{.Subnet}}{{end}}"
→
backups_ci_commoninternet_net_default: 10.0.4.0/24
bridge: 172.17.0.0/16
docker_gwbridge: 172.18.0.0/16
host: (none)
ingress: 10.0.0.0/24
none: (none)
proxy: 10.0.1.0/24
traefik_ci_commoninternet_net_internal: 10.0.2.0/24
warm-keycloak_ci_commoninternet_net_internal: 10.0.3.0/24
```
Builder's table matches exactly. `10.10.0.0/16` is clear of all existing networks. ✓
**3. Maintenance procedure review:**
- **Service names confirmed correct** against live host:
`deploy-proxy`, `deploy-drone`, `deploy-bridge`, `deploy-dashboard`, `deploy-reports`,
`warm-keycloak` — all exist as active oneshot services. ✓
- **backups stack correctly excluded** — `backups_ci_commoninternet_net_default` (10.0.4.0/24)
is NOT on `proxy` (confirmed via proxy Containers inspection). ✓
- **Step sequencing is safe:** stack rm → drain wait → network rm → nixos-rebuild (triggers
swarm-init with new --subnet) → restart deploy services. ✓
- **nixos-rebuild will restart swarm-init:** `swarm-init.service` unit script changed (added
--subnet flag); nixos-rebuild switch calls daemon-reload + restart for changed units. ✓
- **Note (non-blocking recommendation):** Builder may want to add an explicit
`systemctl restart swarm-init` after nixos-rebuild as belt-and-braces insurance (in case
daemon-reload timing is unusual). Not required for correctness but eliminates any ambiguity.
**M1 PASS — safe to execute the maintenance procedure.** Waiting for Builder M2 claim.
## M2: PASS @2026-06-13T05:49Z
**Claim:** proxy recreated as 10.10.0.0/16; nixos-rebuild applied; all services healthy; routes up.
**Commits:** `e6349a9` (patch), `71319d7` (M2 claim)
### Cold-run evidence (all 4 acceptance checks + pre-verification probe)
**1. Proxy subnet:**
```
ssh cc-ci 'docker network inspect proxy --format "{{range .IPAM.Config}}{{.Subnet}}{{end}} created={{.Created}}"'
→ 10.10.0.0/16 created=2026-06-13 05:38:02.125154677 +0000 UTC
```
Network recreated at 05:38:02 UTC. ✓
**2. All 9 services at 1/1:**
```
backups_ci_commoninternet_net_app 1/1
ccci-bridge_app 1/1
ccci-dashboard_app 1/1
ccci-reports_app 1/1
drone_ci_commoninternet_net_app 1/1
traefik_ci_commoninternet_net_app 1/1
traefik_ci_commoninternet_net_socket-proxy 1/1
warm-keycloak_ci_commoninternet_net_app 1/1
warm-keycloak_ci_commoninternet_net_db 1/1
```
All 1/1. ✓
**3. swarm-init activation time:**
```
systemctl status swarm-init --no-pager | grep Active
→ Active: active (exited) since Sat 2026-06-13 05:38:17 UTC; 9min ago
```
Activated 05:38:17 UTC — matches proxy creation timestamp. nixos-rebuild applied new unit. ✓
**4. Core routes:**
```
curl -sI https://ci.commoninternet.net/ → HTTP/2 200
curl -sI https://drone.ci.commoninternet.net/ → HTTP/2 303
```
✓ Both healthy.
**5. Active swarm-init script has --subnet:**
```
/nix/store/…/swarm-init-start: docker network create --driver overlay --attachable --subnet 10.10.0.0/16 proxy
```
nixos-rebuild confirmed applied. ✓
**M2 PASS — proxy VIP exhaustion fix is live and durable.**
See [adversary] finding A1 below (health gate circular dependency, pre-existing, not introduced by pvfix).
---
## Pre-verification probe (2026-06-13T05:45Z — before M2 claimed)
Builder has executed the maintenance; M2 has not been formally claimed yet.
Independent host check run while waiting:
- `docker network inspect proxy --format "..."` → **Subnet: 10.10.0.0/16** ✓
- Container VIPs on proxy: all in `10.10.0.x/16` space:
traefik=10.10.0.2, proxy-endpoint=10.10.0.3, drone=10.10.0.5,
warm-keycloak=10.10.0.7, ccci-bridge=10.10.0.9, ccci-dashboard=10.10.0.11,
ccci-reports=10.10.0.13 ✓
- `docker service ls` → all 9 services at 1/1 REPLICAS ✓
- `systemctl cat swarm-init` → active script has `--subnet 10.10.0.0/16` (nixos-rebuild applied) ✓
- `https://ci.commoninternet.net` → **HTTP/2 200** ✓
- `https://drone.ci.commoninternet.net` → **HTTP/2 303** (login redirect = healthy) ✓
- `https://bridge.ci.commoninternet.net` → **HTTP/2 404** (root path = expected, Traefik routes it) ✓
- `https://report.ci.commoninternet.net` → **HTTP/2 200** ✓
Reference in New Issue View Git Blame Copy Permalink