claim(pvfix-M2): proxy recreated as /16 — all routes healthy
Some checks failed
continuous-integration/drone/push Build is failing
Some checks failed
continuous-integration/drone/push Build is failing
Live maintenance executed 2026-06-13T05:33–05:46Z: - Removed 6 stacks from proxy (traefik, drone, bridge, dashboard, reports, warm-keycloak) - Waited for proxy to drain, removed old 10.0.1.0/24 network - nixos-rebuild switch with git+file:///?submodules=1 → swarm-init restarted - proxy recreated: Subnet 10.10.0.0/16, gateway 10.10.0.1 - All 9 swarm services running 1/1 - ci.commoninternet.net → HTTP/2 200; drone → 303 Adversary: verify from host that proxy subnet is /16 and routes healthy. Full evidence in STATUS-pvfix.md. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@ -63,3 +63,92 @@ Host state at time of claim:
|
||||
- Confirmed with `docker ps --format "{{.Names}}"` — only infra/warm containers
|
||||
|
||||
Host is quiet → suitable maintenance window. No active upgrade-all or !testme runs.
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-13T05:33–05:46Z — Live maintenance execution
|
||||
|
||||
### Adversary M1 PASS received
|
||||
|
||||
Adversary confirmed patch correct and procedure safe. Non-blocking recommendation: add explicit
|
||||
`systemctl restart swarm-init` after nixos-rebuild. Adopted.
|
||||
|
||||
### Pre-flight confirmed
|
||||
|
||||
- No active recipe test containers (`docker ps` — empty)
|
||||
- All stacks infra-only (7 stacks: backups, ccci-bridge, ccci-dashboard, ccci-reports, drone, traefik, warm-keycloak)
|
||||
|
||||
### Stack removal
|
||||
|
||||
```
|
||||
docker stack rm traefik_ci_commoninternet_net drone_ci_commoninternet_net ccci-bridge ccci-dashboard ccci-reports warm-keycloak_ci_commoninternet_net
|
||||
```
|
||||
Output showed all services/configs/networks being removed. proxy drained in ~12s (4 polling attempts).
|
||||
|
||||
### Proxy removal
|
||||
|
||||
```
|
||||
docker network rm proxy
|
||||
→ proxy
|
||||
proxy removed
|
||||
```
|
||||
|
||||
### builder-clone sync issue
|
||||
|
||||
`/root/cc-ci` didn't exist — needed `/root/builder-clone` instead. The builder-clone was at `e1c4198` (old).
|
||||
`git pull --rebase` failed with untracked files: `tests/concurrency/test_run_state.py`.
|
||||
Moved to `/root/test_run_state.py.bak`. Second pull succeeded, fast-forwarded to `b6e12ef`.
|
||||
|
||||
Then `git merge --ff-only origin/main` also failed (many stale untracked files from previous phases).
|
||||
Moved all conflicting files to `/root/stash-pvfix/`. Successfully merged to `caef217` (latest main).
|
||||
Confirmed `grep subnet /root/builder-clone/nix/modules/swarm.nix` → `--subnet 10.10.0.0/16`.
|
||||
|
||||
### nixos-rebuild
|
||||
|
||||
First attempt: `nixos-rebuild switch --flake /root/builder-clone#cc-ci` → FAILED
|
||||
- Error: `path '/nix/store/.../secrets/secrets.yaml' does not exist`
|
||||
- Root cause: flake default doesn't include git submodule content
|
||||
|
||||
Second attempt: `path:` scheme with `?submodules=1` → FAILED
|
||||
- Error: `path URL has unsupported parameter 'submodules'`
|
||||
|
||||
Third attempt: `git+file:///root/builder-clone?submodules=1#cc-ci` → SUCCESS (exit 0)
|
||||
- Output: `building the system configuration...` (used nix cache, fast)
|
||||
|
||||
### swarm-init restart
|
||||
|
||||
Checked: the new unit script `/nix/store/apv1zvz658ddq0i8z0ivmc8f9sydxv7h-unit-script-swarm-init-start/bin/swarm-init-start`
|
||||
contained `--subnet 10.10.0.0/16`. The service was still showing "active" from its old run (Jun 12).
|
||||
|
||||
Ran: `systemctl restart swarm-init`
|
||||
→ Active: active (exited) since 2026-06-13 05:38:17 UTC
|
||||
→ `docker network inspect proxy` → Subnet: 10.10.0.0/16 ✓
|
||||
|
||||
### Deploy-proxy health gate deadlock
|
||||
|
||||
`systemctl restart deploy-proxy` started successfully. Traefik deployed.
|
||||
But health gate (`ci.commoninternet.net → 200`) failed because dashboard not yet deployed.
|
||||
Reconciler logged: `[traefik] on latest 5.1.1+v3.6.15 but UNHEALTHY → redeploy`
|
||||
|
||||
Analysis: The `deploy-proxy` health_timeout=300s (5 min) gives enough time for dashboard to be
|
||||
deployed concurrently. The `After=` ordering in systemd means these services DON'T start until
|
||||
deploy-proxy is "active", but since deploy-proxy was still "activating", systemd would have
|
||||
waited indefinitely if we relied on the ordering chain.
|
||||
|
||||
Fix: started deploy-drone, deploy-bridge, deploy-dashboard, deploy-reports concurrently:
|
||||
```
|
||||
systemctl start deploy-drone deploy-bridge deploy-dashboard deploy-reports
|
||||
```
|
||||
Within ~20 seconds, `ci.commoninternet.net` returned 200. Deploy-proxy health gate passed.
|
||||
|
||||
### Final health state (2026-06-13T05:45Z)
|
||||
|
||||
```
|
||||
docker stack ls → 7 stacks all present
|
||||
docker service ls → all 9 services 1/1
|
||||
docker network inspect proxy → Subnet: 10.10.0.0/16
|
||||
ci.commoninternet.net → HTTP/2 200
|
||||
drone.ci.commoninternet.net → HTTP/2 303
|
||||
systemctl is-active deploy-proxy deploy-drone deploy-bridge deploy-dashboard deploy-reports warm-keycloak
|
||||
→ active active active active active active
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user