# JOURNAL — phase pvfix

## 2026-06-13T05:29Z — Bootstrap + M1 patch

### Context gathered

Read the phase plan + runbook. Key facts:
- Root cause confirmed: proxy is `10.0.1.0/24` (254 VIPs), Docker GC race leaks endpoints → pool exhaustion
- Fix: enlarge to `/16` (`--subnet 10.10.0.0/16`)
- The network can't be resized in place; requires remove + recreate

### Live host survey

Subnets in use on the live host (collected via `docker network inspect`):
- `ingress`: `10.0.0.0/24`
- `proxy`: `10.0.1.0/24` (current — to change)
- `traefik internal`: `10.0.2.0/24`
- `warm-keycloak internal`: `10.0.3.0/24`
- `backups default`: `10.0.4.0/24`
- `bridge`/`docker_gwbridge`: `172.17/18.0.0/16`

`10.10.0.0/16` is clean — no conflicts. Host eth0: `91.98.47.73/32`, Tailscale: `100.95.31.88/32`. 
No route entries for `10.10.x.x` in `ip route show`.

### Services on proxy (will be disrupted during maintenance)

From `docker service ls` + per-service network inspection:
- `traefik_ci_commoninternet_net_app` — uses proxy
- `drone_ci_commoninternet_net_app` — uses proxy
- `ccci-bridge_app` — uses proxy
- `ccci-dashboard_app` — uses proxy
- `ccci-reports_app` — uses proxy
- `warm-keycloak_ci_commoninternet_net_app` — uses proxy

NOT on proxy: `backups_ci_commoninternet_net_app`, traefik socket-proxy, warm-keycloak DB.

### Deployment mechanism

- `swarm-init.service` — oneshot, creates proxy. Changes here → systemd restarts it on nixos-rebuild
- `deploy-proxy`, `deploy-drone`, `deploy-bridge`, `deploy-dashboard`, `deploy-reports`, `warm-keycloak` — 
  RemainAfterExit oneshots; their definitions don't change so they WON'T auto-restart after nixos-rebuild.
  Must be manually `systemctl restart`-ed after nixos-rebuild removes their stacks.

### Design choice: why 10.10.0.0/16

- Must be `/16` for ~65k VIP headroom
- Must not overlap `10.0.0.0/24` (ingress) or any of the `10.0.1-4.0/24` per-stack overlays
- The Docker default-addr-pool is `10.0.0.0/8` — any `/16` in that range is fine as long as 
  it doesn't overlap an existing allocation
- `10.10.0.0/16` is the first clean `/16` outside the current allocation band — clear of `10.0.x.x`
  while still in Docker's pool. No host route conflicts.

### swarm.nix patch

Added `--subnet 10.10.0.0/16` to the `docker network create` call.
Also added a short comment explaining the motivation (required WHY per §7 comment policy for non-obvious constraint).

### Maintenance window state

Host state at time of claim:
- `docker stack ls` shows 6 stacks: backups, ccci-bridge, ccci-dashboard, ccci-reports, drone, traefik, warm-keycloak
- NO active recipe CI runs (only warm stacks, no test app containers)
- Confirmed with `docker ps --format "{{.Names}}"` — only infra/warm containers

Host is quiet → suitable maintenance window. No active upgrade-all or !testme runs.

---

## 2026-06-13T05:33–05:46Z — Live maintenance execution

### Adversary M1 PASS received

Adversary confirmed patch correct and procedure safe. Non-blocking recommendation: add explicit
`systemctl restart swarm-init` after nixos-rebuild. Adopted.

### Pre-flight confirmed

- No active recipe test containers (`docker ps` — empty)
- All stacks infra-only (7 stacks: backups, ccci-bridge, ccci-dashboard, ccci-reports, drone, traefik, warm-keycloak)

### Stack removal

```
docker stack rm traefik_ci_commoninternet_net drone_ci_commoninternet_net ccci-bridge ccci-dashboard ccci-reports warm-keycloak_ci_commoninternet_net
```
Output showed all services/configs/networks being removed. proxy drained in ~12s (4 polling attempts).

### Proxy removal

```
docker network rm proxy
→ proxy
proxy removed
```

### builder-clone sync issue

`/root/cc-ci` didn't exist — needed `/root/builder-clone` instead. The builder-clone was at `e1c4198` (old). 
`git pull --rebase` failed with untracked files: `tests/concurrency/test_run_state.py`.
Moved to `/root/test_run_state.py.bak`. Second pull succeeded, fast-forwarded to `b6e12ef`.

Then `git merge --ff-only origin/main` also failed (many stale untracked files from previous phases).
Moved all conflicting files to `/root/stash-pvfix/`. Successfully merged to `caef217` (latest main).
Confirmed `grep subnet /root/builder-clone/nix/modules/swarm.nix` → `--subnet 10.10.0.0/16`.

### nixos-rebuild

First attempt: `nixos-rebuild switch --flake /root/builder-clone#cc-ci` → FAILED
- Error: `path '/nix/store/.../secrets/secrets.yaml' does not exist`
- Root cause: flake default doesn't include git submodule content

Second attempt: `path:` scheme with `?submodules=1` → FAILED
- Error: `path URL has unsupported parameter 'submodules'`

Third attempt: `git+file:///root/builder-clone?submodules=1#cc-ci` → SUCCESS (exit 0)
- Output: `building the system configuration...` (used nix cache, fast)

### swarm-init restart

Checked: the new unit script `/nix/store/apv1zvz658ddq0i8z0ivmc8f9sydxv7h-unit-script-swarm-init-start/bin/swarm-init-start` 
contained `--subnet 10.10.0.0/16`. The service was still showing "active" from its old run (Jun 12).

Ran: `systemctl restart swarm-init`
→ Active: active (exited) since 2026-06-13 05:38:17 UTC
→ `docker network inspect proxy` → Subnet: 10.10.0.0/16 ✓

### Deploy-proxy health gate deadlock

`systemctl restart deploy-proxy` started successfully. Traefik deployed.
But health gate (`ci.commoninternet.net → 200`) failed because dashboard not yet deployed.
Reconciler logged: `[traefik] on latest 5.1.1+v3.6.15 but UNHEALTHY → redeploy`

Analysis: The `deploy-proxy` health_timeout=300s (5 min) gives enough time for dashboard to be
deployed concurrently. The `After=` ordering in systemd means these services DON'T start until
deploy-proxy is "active", but since deploy-proxy was still "activating", systemd would have
waited indefinitely if we relied on the ordering chain.

Fix: started deploy-drone, deploy-bridge, deploy-dashboard, deploy-reports concurrently:
```
systemctl start deploy-drone deploy-bridge deploy-dashboard deploy-reports
```
Within ~20 seconds, `ci.commoninternet.net` returned 200. Deploy-proxy health gate passed.

### Final health state (2026-06-13T05:45Z)

```
docker stack ls → 7 stacks all present
docker service ls → all 9 services 1/1
docker network inspect proxy → Subnet: 10.10.0.0/16
ci.commoninternet.net → HTTP/2 200
drone.ci.commoninternet.net → HTTP/2 303
systemctl is-active deploy-proxy deploy-drone deploy-bridge deploy-dashboard deploy-reports warm-keycloak
→ active active active active active active
```