cc-ci/machine-docs/JOURNAL-2pc.md

# JOURNAL — Phase 2pc (sane image-prune policy)

Append-only reasoning log. Facts/verification for the Adversary live in STATUS-2pc.md.

## 2026-05-29 — Orientation + scope correction

Read SSOT `plan-phase2pc-image-cache.md` + plan.md §6.1/§7/§9. Operator issued a **scope
correction** mid-orientation: **drop the registry:2 pull-through cache.** Rationale (operator):
single host → Docker's own local image store already IS the cache; re-deploys reuse local layers
with no re-download; the daemon is PAT-authenticated so residual manifest checks sit under 200/6h.
The churn was caused by **over-pruning** (`docker image prune -af` wiping the store), not a missing
cache. A separate registry only pays off multi-node / separate-survivable storage, which we are not.
**I had not yet written any registry code** (still orienting) → nothing to revert.

Phase 2pc is now **PC1 (prune policy) + PC2/PC3 (confirm + verify local-store retention/auth).**

### Findings from orientation (why the fix is one module)

- The ONLY automated image pruner in the whole repo is
  `virtualisation.docker.autoPrune = { flags = ["--all" "--filter" "until=24h"]; }` in
  `nix/modules/swarm.nix`. NixOS renders this as `docker system prune --force --all --filter until=24h`
  daily. `--all` removes every image **not used by a running container** — between runs there are no
  test apps running, so it evicts the cached recipe base images → cold re-pull on the next run. That
  is exactly the prune→re-pull→rate-limit churn documented in JOURNAL-2 (lines 507/542/690-693).
- `runner/harness/lifecycle.py::teardown_app` removes services (abra undeploy / `docker stack rm`),
  volumes, secrets, and the `.env` — and **no images** (`grep` for `rmi`/`image rm`/`image prune` in
  `runner/` + `tests/conftest.py` is empty). So PC1's "teardown must NOT remove images" already holds.
- `janitor`, `warm_reconcile.py`, `nightly-sweep.nix`, `drone*.nix`, `.drone.yml` — none prune images.
- Daemon is already PAT-authenticated: `docker info` → `Username: nptest2`; sops `dockerhub_auth`
  (base64 `nptest2:<PAT>`) → `sops.templates."docker-config.json"` → `/root/.docker/config.json`
  (`nix/modules/secrets.nix`). PC2 needs no change — confirm + document.
- Disk on cc-ci: `/` is 64G, 19G used, **43G free (31%)** — bounded; aggressive `--all` is
  unnecessary, which is the whole premise.

### PC1 design

Replace `autoPrune` with a dedicated `nix/modules/docker-prune.nix`: a daily `systemd.timer` +
oneshot `systemd.service` running a surgical, **triple-gated** prune:
1. **Disk-pressure gate** — do nothing unless `/` usage ≥ 80% (Docker's local store IS our cache;
   keep it warm; reclaim only under genuine pressure).
2. **No-run gate** — skip if any run-app stack (`<=4char>-<6hex>_ci_commoninternet_net_*`) is live
   (mid-pull layers can look prunable; "never prune mid-run").
3. **No-converge gate** — skip if any swarm service has unmet replicas (a deploy/pull in flight,
   incl. infra warm redeploys).
When all gates pass: `docker {container,image,builder} prune -f --filter until=24h` — dangling +
age-gated only. NEVER `--all` (keeps tagged base/in-use images), NEVER `--volumes` (warm canonical
data, per swarm.nix's existing comment).

## 2026-05-29 — Implemented + deployed + verified on cc-ci

**Implementation.** `nix/modules/docker-prune.nix` (NEW) + `swarm.nix` (dropped autoPrune block) +
`configuration.nix` import. Unit renamed `docker-prune` → **`ci-docker-prune`** because the NixOS
docker module reserves `systemd.services.docker-prune` (build conflict caught by `nixos-rebuild
build`: "conflicting definition values for systemd.services.docker-prune.description"). Renamed,
rebuilt clean.

**Deploy.** Synced the 3 changed nix files to `/root/cc-ci` (tar over ssh; isolated change — host
tree otherwise unchanged), `nixos-rebuild build` (clean, shellcheck on the writeShellApplication
passed), then `systemd-run --unit=ccci-sw ... nixos-rebuild switch path:/root/cc-ci#cc-ci`. Switch
finished (22.5s CPU), `systemctl is-system-running` → `running`.

**Verification (real host).**
- Old NixOS `docker-prune.timer` → `is-enabled` = **not-found** (autoPrune gone). `ci-docker-prune.timer`
  → enabled + active; `list-timers` NEXT = Sat 2026-05-30 00:00 UTC (daily).
- Manual `systemctl start ci-docker-prune.service` at `/`=31%: log →
  `docker-prune: / at 31% (< 80%) — keeping local image cache, nothing to do`. No images removed
  (21 → 21). Gate works.
- PC2: `docker info | grep Username` → `nptest2` (PAT auth retained after rebuild). `/var/lib/docker`
  persistent (21 recipe images retained across the rebuild).
- PC3 layer-reuse proof (real swarm deploy→teardown→redeploy, redis:7-alpine, docker.io via authed daemon):
  ```
  COLD pull: 897d... Already exists; c14c.. f546.. a300.. 941e.. 4f4f.. 677c.. Pull complete  (6 downloaded)
             Status: Downloaded newer image for redis:7-alpine        COLD_PULL_MS=5303
  service create pc3b -> 1/1
  service rm pc3b      -> retained_after_teardown: redis:7-alpine 487efc061638   (image REMAINS)
  WARM pull: Status: Image is up to date for redis:7-alpine          WARM_PULL_MS=674   (no bytes)
  redeploy create pc3b -> redeploy_ok (reused local layers)
  ```
  Cold 5303ms (6 layer downloads) → warm 674ms (authenticated manifest check only, 0 layers
  re-downloaded). The alpine base layer `897d...` showed "Already exists" even on the cold pull =
  cross-image base-layer reuse, a bonus cache win. Teardown (`service rm`) retained the image —
  matches `teardown_app` (no rmi).

**Docs/decisions.** `docs/runbook.md` (new "Image cache & prune policy" + updated rate-limit note),
`docs/warm.md` (autoPrune→ci-docker-prune), `DECISIONS.md` (Phase-2pc entry), `cc-ci-plan/IDEAS.md`
(deferred registry cache + revisit trigger). Gate claimed.