cc-ci/machine-docs/JOURNAL-pvcheck.md

JOURNAL — phase pvcheck (post-proxy verification)

Builder-private reasoning and working notes. Anti-anchoring: Adversary reads STATUS for claims, not this file.

---

2026-06-13T05:55–06:02Z — Phase orientation and M1 data collection

Phase pvfix is DONE. Entered pvcheck. No phase files existed yet — the Adversary had proactively created REVIEW-pvcheck.md and BACKLOG-pvcheck.md with a baseline probe at 05:56Z.

Adversary baseline findings (from REVIEW-pvcheck.md):

• All preconditions verified cold (pvfix DONE, proxy /16 live, all services 1/1, all routes 200/303)
• [A2]: stale text in upgrade-all SKILL.md — "per-run safety net until that lands" (fix: proxy /16 HAS landed)

My verification runs:

$ ssh cc-ci 'docker network inspect proxy --format "{{range .IPAM.Config}}{{.Subnet}}{{end}}, Endpoints: {{len .Containers}}"'
10.10.0.0/16, Endpoints: 7

$ curl -sk -o /dev/null -w "%{http_code}" https://ci.commoninternet.net/ → 200
$ curl -sk -o /dev/null -w "%{http_code}" https://drone.ci.commoninternet.net/ → 303
$ curl -sk -o /dev/null -w "%{http_code}" https://report.ci.commoninternet.net/ → 200

$ ssh cc-ci 'journalctl -u docker --since "2026-06-13 05:38:00" | grep -c "available IP while allocating VIP"'
0

The "could not find network allocator STATE" errors in the 05:35Z window are expected transient noise: they occur when swarm tries to allocate VIPs for the old deleted /24 network IDs (mlxau8…, 85p3aq…) during the recreation — not the "available IP while allocating VIP" signature of actual exhaustion.

A2 fix applied:

• Edited /srv/cc-ci-orch/.claude/skills/upgrade-all/SKILL.md line 80-81
• Committed to orchestrator repo as 84e13a7
• Guard logic unchanged — only the description now reflects reality (durable fix has landed)

Decision on bridge /hook: bridge is exposed at PathPrefix(/hook) and only accepts POST (webhook). A GET to /hook returns 404 — expected; health is confirmed via service logs showing the poller running and commenting on repos.

M1 claim: All control-plane facts documented. Claiming M1 now. Will work on M2 while awaiting verdict.

---

2026-06-13T06:02Z — M2 planning

M2 requires:

1. Real recipe CI run through proxy — will use a small enrolled recipe like hedgedoc or cryptpad if a !testme PR exists, or trigger via the harness directly
2. Allocator headroom proof — deploy/remove 3-5 throwaway stacks with published ports (simulating concurrent deploys), confirm endpoint count stays small and no VIP exhaustion

Will check what enrolled recipes have open PRs available for !testme first.

---

2026-06-13T06:02–06:10Z — M2 execution

Allocator headroom proof (Builder):

# Baseline
ssh cc-ci 'docker network inspect proxy --format "{{len .Containers}}"' → 8

# Deploy 5 throwaway nginx stacks concurrently, each joining proxy with published ports
for i in 1..5: docker stack deploy pvcheck-throw-$i (background)
wait; sleep 5
→ AFTER DEPLOY: 13 (+5)

# Concurrent removal (same pattern as original GC race)
for i in 1..5: docker stack rm pvcheck-throw-$i (background)
wait; sleep 8
→ AFTER concurrent rm: 8 (back to baseline)
→ VIP exhaustion errors since 06:00Z: 0
→ docker network prune → empty (no residue)
→ docker stack ls | grep pvcheck → empty (all removed)

Real recipe CI run:

# Posted !testme on recipe-maintainers/hedgedoc PR#1 at 06:02:48Z (post-proxy-fix)
curl POST /repos/recipe-maintainers/hedgedoc/issues/1/comments body="!testme"
→ comment id: 14505

# Bridge picked up in 4 seconds (06:02:52Z)
# Started Drone build #608 for hedgedoc @ 441c411c

# Monitored: runner process PID 3016375 with RECIPE=hedgedoc, CI_BUILD_NUMBER=608

# Build #608 completed at 06:04:22Z → ✅ passed, level 5
# Proxy endpoint count after run: 7 (same as M1 baseline, clean teardown)

Key confirmation: the build was triggered at 06:02Z which is 24 minutes AFTER the proxy recreation at 05:38Z. Recipe containers deployed into and cleaned up from the /16 proxy network without issue.