cc-ci/machine-docs/JOURNAL-2pc.md

JOURNAL — Phase 2pc (sane image-prune policy)

Append-only reasoning log. Facts/verification for the Adversary live in STATUS-2pc.md.

2026-05-29 — Orientation + scope correction

Read SSOT plan-phase2pc-image-cache.md + plan.md §6.1/§7/§9. Operator issued a scope correction mid-orientation: drop the registry:2 pull-through cache. Rationale (operator): single host → Docker's own local image store already IS the cache; re-deploys reuse local layers with no re-download; the daemon is PAT-authenticated so residual manifest checks sit under 200/6h. The churn was caused by over-pruning (docker image prune -af wiping the store), not a missing cache. A separate registry only pays off multi-node / separate-survivable storage, which we are not. I had not yet written any registry code (still orienting) → nothing to revert.

Phase 2pc is now PC1 (prune policy) + PC2/PC3 (confirm + verify local-store retention/auth).

Findings from orientation (why the fix is one module)

• The ONLY automated image pruner in the whole repo is virtualisation.docker.autoPrune = { flags = ["--all" "--filter" "until=24h"]; } in nix/modules/swarm.nix. NixOS renders this as docker system prune --force --all --filter until=24h daily. --all removes every image not used by a running container — between runs there are no test apps running, so it evicts the cached recipe base images → cold re-pull on the next run. That is exactly the prune→re-pull→rate-limit churn documented in JOURNAL-2 (lines 507/542/690-693).
• runner/harness/lifecycle.py::teardown_app removes services (abra undeploy / docker stack rm), volumes, secrets, and the .env — and no images (grep for rmi/image rm/image prune in runner/ + tests/conftest.py is empty). So PC1's "teardown must NOT remove images" already holds.
• janitor, warm_reconcile.py, nightly-sweep.nix, drone*.nix, .drone.yml — none prune images.
• Daemon is already PAT-authenticated: docker info → Username: nptest2; sops dockerhub_auth (base64 nptest2:<PAT>) → sops.templates."docker-config.json" → /root/.docker/config.json (nix/modules/secrets.nix). PC2 needs no change — confirm + document.
• Disk on cc-ci: / is 64G, 19G used, 43G free (31%) — bounded; aggressive --all is unnecessary, which is the whole premise.

PC1 design

Replace autoPrune with a dedicated nix/modules/docker-prune.nix: a daily systemd.timer + oneshot systemd.service running a surgical, triple-gated prune:

1. Disk-pressure gate — do nothing unless / usage ≥ 80% (Docker's local store IS our cache; keep it warm; reclaim only under genuine pressure).
2. No-run gate — skip if any run-app stack (<=4char>-<6hex>_ci_commoninternet_net_*) is live (mid-pull layers can look prunable; "never prune mid-run").
3. No-converge gate — skip if any swarm service has unmet replicas (a deploy/pull in flight, incl. infra warm redeploys). When all gates pass: docker {container,image,builder} prune -f --filter until=24h — dangling + age-gated only. NEVER --all (keeps tagged base/in-use images), NEVER --volumes (warm canonical data, per swarm.nix's existing comment).

2026-05-29 — Implemented + deployed + verified on cc-ci

Implementation. nix/modules/docker-prune.nix (NEW) + swarm.nix (dropped autoPrune block) + configuration.nix import. Unit renamed docker-prune → ci-docker-prune because the NixOS docker module reserves systemd.services.docker-prune (build conflict caught by nixos-rebuild build: "conflicting definition values for systemd.services.docker-prune.description"). Renamed, rebuilt clean.

Deploy. Synced the 3 changed nix files to /root/cc-ci (tar over ssh; isolated change — host tree otherwise unchanged), nixos-rebuild build (clean, shellcheck on the writeShellApplication passed), then systemd-run --unit=ccci-sw ... nixos-rebuild switch path:/root/cc-ci#cc-ci. Switch finished (22.5s CPU), systemctl is-system-running → running.

Verification (real host).

• Old NixOS docker-prune.timer → is-enabled = not-found (autoPrune gone). ci-docker-prune.timer → enabled + active; list-timers NEXT = Sat 2026-05-30 00:00 UTC (daily).
• Manual systemctl start ci-docker-prune.service at /=31%: log → docker-prune: / at 31% (< 80%) — keeping local image cache, nothing to do. No images removed (21 → 21). Gate works.
• PC2: docker info | grep Username → nptest2 (PAT auth retained after rebuild). /var/lib/docker persistent (21 recipe images retained across the rebuild).
• PC3 layer-reuse proof (real swarm deploy→teardown→redeploy, redis:7-alpine, docker.io via authed daemon):

  COLD pull: 897d... Already exists; c14c.. f546.. a300.. 941e.. 4f4f.. 677c.. Pull complete  (6 downloaded)
             Status: Downloaded newer image for redis:7-alpine        COLD_PULL_MS=5303
  service create pc3b -> 1/1
  service rm pc3b      -> retained_after_teardown: redis:7-alpine 487efc061638   (image REMAINS)
  WARM pull: Status: Image is up to date for redis:7-alpine          WARM_PULL_MS=674   (no bytes)
  redeploy create pc3b -> redeploy_ok (reused local layers)
  

  Cold 5303ms (6 layer downloads) → warm 674ms (authenticated manifest check only, 0 layers re-downloaded). The alpine base layer 897d... showed "Already exists" even on the cold pull = cross-image base-layer reuse, a bonus cache win. Teardown (service rm) retained the image — matches teardown_app (no rmi).

Docs/decisions. docs/runbook.md (new "Image cache & prune policy" + updated rate-limit note), docs/warm.md (autoPrune→ci-docker-prune), DECISIONS.md (Phase-2pc entry), cc-ci-plan/IDEAS.md (deferred registry cache + revisit trigger). Gate claimed.

2026-05-29 — Probe-5 evidence: surgical prune reclaims, keeps tagged/recent

Ran the exact active-path command the gated unit uses (docker image prune -f --filter until=24h

• container/builder variants) on the host to demonstrate surgical reclaim (the daily timer only reaches this under ≥80% disk, but the command's effect is the same):

• all images 23→17, dangling 10→4 (the 4 remaining are <24h old — the until=24h age gate kept them), 2.341 GB reclaimed, disk 31%→27% (19G→17G used).
• ALL tagged/in-use images survived (keycloak:26.6.2, mariadb:12.2, nginx:1.30.0, redis:8.6.3, …) — no --all, so nothing tagged or container-referenced was touched. Confirms: disk stays bounded WITHOUT -af; the policy reclaims real space from old orphaned layers while keeping the warm cache intact.

2026-05-29 — F2pc-1 (committed≠host) resolution + claim discipline

Adversary FAILed gate 2pc on F2pc-1: at claim commit de6103d the committed docker-prune.nix still named units docker-prune while the verified host runs ci-docker-prune → git wouldn't reproduce the verified system (D8). Root cause: I renamed the units locally (sed) + synced to host + verified, but the rename rode in a SEPARATE commit (b9bbd25) pushed AFTER the claim( commit — and the Adversary cold-verified the claim commit's tree. Behavior was GREEN; only the artifact lagged.

b9bbd25 already committed the rename (git == host == ci-docker-prune), which is the Adversary's own endorsed fix. Confirmed current HEAD: grep systemd.(services|timers) → ci-docker-prune; host module matches; host runs ci-docker-prune.timer enabled+active; builtin docker-prune.service inactive/linked (inert NixOS default, never triggered with autoPrune off). Re-claimed.

Lesson (now a standing rule, orchestrator): before ANY gate claim, git status must be clean — everything committed AND pushed — because the Adversary cold-verifies from a fresh clone. A fix built locally but uncommitted (or trailing the claim commit) is a guaranteed cold-build mismatch. The claim commit must be the LAST thing, with the verified artifact already in it.