claim(M1-nixenv): single-source harness runtime env — ccciPyEnv+ccciRuntimeTools+cc-ci-run in packages.nix, referenced by harness/sweep/both hosts; sweep execs cc-ci-run (no dup pyEnv, no DEFECT-3 PATH patch); cc-ci host gains git-lfs+openssl; both #cc-ci and #cc-ci-hetzner build; awaiting Adversary

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

machine-docs/BACKLOG-nixenv.md

@@ -0,0 +1,19 @@
+# BACKLOG — phase `nixenv`
+
+## Build backlog
+
+- [x] M1: define shared harness/recipe-test runtime env once (overlay in `packages.nix`):
+      `ccciPyEnv` + `ccciRuntimeTools` (the union tool set) + `cc-ci-run`.
+- [x] M1: `harness.nix` references `pkgs.cc-ci-run` (no local pyEnv/runtimeInputs).
+- [x] M1: `nightly-sweep.nix` invokes `cc-ci-run` (no duplicate pyEnv, no own tool list, DEFECT-3 patch gone).
+- [x] M1: both host `configuration.nix` `systemPackages` reference `pkgs.ccciRuntimeTools` (+ openssh); end identical.
+- [x] M1: grep proof — exactly one `withPackages`/`pytest playwright` in nix/ (packages.nix); no module declares its own harness tool list.
+- [x] M1: `nixos-rebuild build` succeeds for both `#cc-ci` and `#cc-ci-hetzner`.
+- [x] M1: CLAIM, await Adversary PASS.
+- [ ] M2: deploy via `nixos-rebuild switch`; verify host health (systemctl --failed, oneshots, timer, endpoints).
+- [ ] M2: live parity — gitea `test_lfs_roundtrip` green under BOTH Drone path and a real timer fire from the unified env.
+- [ ] M2: canon-style sweep still promotes/SKIPs correctly (no regression).
+- [ ] M2: CLAIM, await Adversary PASS → `## DONE`.
+
+## Adversary findings
+<!-- Adversary-owned section. Builder does not edit. -->

machine-docs/JOURNAL-nixenv.md

@@ -0,0 +1,59 @@
+# JOURNAL — phase `nixenv` (Builder)
+
+## 2026-06-17 — M1: single-source the harness runtime env
+
+### Why this design
+The phase plan §2 wants ONE definition of "what's needed to run a recipe test", referenced from
+three places, so DEFECT-3 (a dep present for one path, missing for another) becomes structurally
+impossible. I put the single source in `nix/modules/packages.nix` because it is the existing
+"shared pkgs" overlay module already imported by both host configs — so `pkgs.ccciRuntimeTools`
+and `pkgs.cc-ci-run` are reachable from every module/host without a fragile cross-module `let`.
+
+Three overlay defs:
+- `ccciPyEnv` (let-bound, internal) — `python3.withPackages [pytest playwright]`, the ONLY pyEnv now.
+- `ccciRuntimeTools` (overlay attr) — the union tool set.
+- `cc-ci-run` (overlay attr) — `writeShellApplication` with `runtimeInputs = [ccciPyEnv] ++ ccciRuntimeTools`.
+
+Consumers:
+- `harness.nix` → `environment.systemPackages = [ pkgs.cc-ci-run ]` (installs the entrypoint).
+- `nightly-sweep.nix` → wrapper execs `cc-ci-run` (same binary the Drone pipeline runs), so pyEnv +
+  tooling + PLAYWRIGHT env are identical to the Drone path by construction. Dropped: the duplicate
+  pyEnv, the parallel `runtimeInputs` tool list, and the DEFECT-3 `export PATH=/run/current-system/sw/bin…`
+  prepend — git-lfs/bash/util-linux/openssl now come from cc-ci-run's runtimeInputs.
+- both host `configuration.nix` → `systemPackages = pkgs.ccciRuntimeTools ++ [ pkgs.openssh ]`.
+
+### Why the union is a superset (nothing dropped)
+- old cc-ci-run: `abra docker git coreutils util-linux` ⊂ set.
+- old sweep: `bash abra docker git curl jq gnused gnugrep gnutar coreutils util-linux procps` ⊂ set;
+  its host-PATH-derived git-lfs/openssl are now EXPLICIT in the set.
+- old host PATH: `curl git jq` (+ git-lfs on hetzner only) ⊂ set; `openssh` kept as host-only add.
+- pyEnv (python3+pytest+playwright) + playwright browsers (via PLAYWRIGHT_BROWSERS_PATH) preserved.
+Additions vs any single prior list: `git-lfs`, `openssl` (plan §2). The `cc-ci` host GAINS git-lfs,
+killing the one-off hetzner-only divergence — both host configs now byte-identical.
+
+### Why writeShellApplication makes this work
+`writeShellApplication` emits `export PATH="<runtimeInputs>:$PATH"` (confirmed on the live wrapper).
+So cc-ci-run's full tool set is the PATH *prefix* regardless of caller. Under Drone the inherited
+suffix is `/run/current-system/sw/bin:/run/wrappers/bin`; under the sweep it's the systemd-minimal
+PATH — but the harness tools all resolve from the shared prefix either way, which is the parity the
+plan wants. The host `systemPackages` reference is the belt-and-suspenders path for direct
+`.drone.yml` shell-outs (`abra --version`, `docker info`) that don't go through cc-ci-run.
+
+### buildEnv collision watch (resolved)
+Worry: adding coreutils/util-linux/procps/bash/gnu* to host `systemPackages` could collide with the
+NixOS base `requiredPackages`. It did not — base requiredPackages are `lowPrio`, so the normal-prio
+additions override cleanly. Both `#cc-ci` and `#cc-ci-hetzner` built with no collision error.
+
+### Note on other modules' tool lists
+`backupbot/docker-prune/drone/proxy/warm-keycloak.nix` still list gnused/gnugrep/etc. in their OWN
+`runtimeInputs` — those are independent reconcile-service scripts, never part of the harness/recipe
+-test env, never part of the DEFECT-3 divergence. Single-sourcing is scoped to the harness env
+(pyEnv + recipe-test tooling consumed by cc-ci-run / sweep / host PATH), which is now packages.nix only.
+
+### Verification (local, dirty tree needs `?submodules=1` — `secrets/` is a submodule)
+- `nixos-rebuild build --flake '.?submodules=1#cc-ci-hetzner'` → built `nixos-system-…dhmpm232…`.
+- `nixos-rebuild build --flake '.?submodules=1#cc-ci'` → built OK.
+- cc-ci-run store `zxlx9jnylh7la5m48bsqb1wfm5l9r0bd`; PATH carries all 15 tools incl git-lfs-3.6.1 + openssl-3.3.3.
+- sweep wrapper `gh02w1kc…` execs the SAME `zxlx9j…/bin/cc-ci-run`.
+- cc-ci host sw/bin now lists git-lfs + openssl (was missing git-lfs pre-refactor).
+- `grep -rn withPackages nix/` → 1 hit (packages.nix:17).

machine-docs/STATUS-nixenv.md

@@ -0,0 +1,65 @@
+# STATUS — phase `nixenv` (Builder)
+
+Phase plan: `/srv/cc-ci/cc-ci-plan/plan-phase-nixenv-shared-runtime-env.md`
+
+## Phase
+Single-source the harness/recipe-test runtime env so the Drone runner, the nightly/weekly sweep
+timer, and host `systemPackages` share ONE declaration (no duplicate `pyEnv`, no divergent
+`runtimeInputs`, DEFECT-3 host-PATH patch removed/subsumed).
+
+## Gate: M1 — CLAIMED, awaiting Adversary
+
+**WHAT (M1 DoD).** The harness/recipe-test runtime env is declared ONCE and referenced by all
+consumers; `nixos-rebuild build` succeeds for both hosts; the shared set is superset-or-equal of
+every prior list (nothing dropped); the sweep and the Drone runner resolve the same tooling; a
+future dep added to the shared set reaches all consumers.
+
+**WHERE (inputs).** All changes at the tip of `main` (commit pushed with this claim).
+- Single source: `nix/modules/packages.nix` — overlay defines `ccciPyEnv` (let), `ccciRuntimeTools`
+  (overlay attr), `cc-ci-run` (overlay attr, `runtimeInputs = [ccciPyEnv] ++ ccciRuntimeTools`).
+- Consumers: `nix/modules/harness.nix` (`systemPackages = [ pkgs.cc-ci-run ]`),
+  `nix/modules/nightly-sweep.nix` (wrapper execs `cc-ci-run`),
+  `nix/hosts/cc-ci/configuration.nix` + `nix/hosts/cc-ci-hetzner/configuration.nix`
+  (`systemPackages = pkgs.ccciRuntimeTools ++ [ pkgs.openssh ]`).
+- `nix/modules/drone-runner.nix` unchanged (still `PATH=/run/current-system/sw/bin:/run/wrappers/bin`;
+  it consumes the host PATH, which now references the shared set).
+
+**HOW + EXPECTED (cold-verifiable; `secrets/` is a git submodule → use `?submodules=1` for a dirty
+tree, or build from a `git clone --recursive`).**
+
+1. Builds succeed (both hosts):
+   - `nixos-rebuild build --flake '.?submodules=1#cc-ci-hetzner'` → builds
+     `nixos-system-nixos-24.11.…` (locally: `/nix/store/dhmpm232r6m0sq3s7y5r5jpyv5kxgzwi-nixos-system-nixos-24.11.20250630.50ab793`;
+     store hash may differ on a fresh clone if paths differ, but it MUST build with no collision error).
+   - `nixos-rebuild build --flake '.?submodules=1#cc-ci'` → builds OK (no collision error).
+
+2. Single source (grep proofs):
+   - `grep -rn withPackages nix/` → EXACTLY 1 hit: `nix/modules/packages.nix` (`ccciPyEnv`).
+   - `grep -rn "pytest playwright" nix/` → EXACTLY 1 hit: same line. (No duplicate pyEnv.)
+   - `grep -rn ccciRuntimeTools nix/` → defined once (packages.nix), referenced by both host configs.
+   - `nightly-sweep.nix` contains NO `withPackages`, NO `python3`, NO `/run/current-system/sw/bin`
+     PATH prepend, and its `runtimeInputs = [ pkgs.cc-ci-run ]` only; it `exec cc-ci-run …`.
+
+3. Superset-or-equal — `cc-ci-run` carries every tool (inspect the built wrapper's PATH):
+   - `CCRUN=$(nix eval --raw '.?submodules=1#nixosConfigurations.cc-ci-hetzner.pkgs.cc-ci-run'); grep '^export PATH' "$CCRUN/bin/cc-ci-run"`
+   - EXPECTED store dirs on PATH (15): python3-3.12.8-env, abra-0.13.0-beta, docker-27.5.1,
+     git-2.47.2, **git-lfs-3.6.1**, bash-5.2p37, coreutils-9.5, util-linux-2.39.4, curl-8.12.1,
+     jq-1.7.1, gnused-4.9, gnugrep-3.11, gnutar-1.35, **openssl-3.3.3**, procps-4.0.4.
+   - git-lfs + openssl are the additions vs prior lists; nothing from any prior list is dropped.
+
+4. Sweep ≡ Drone entrypoint (parity by construction):
+   - The built `cc-ci-nightly-sweep` wrapper `exec cc-ci-run …` resolves the BYTE-IDENTICAL
+     cc-ci-run store path that the `.drone.yml` `cc-ci-run runner/run_recipe_ci.py` step runs
+     (locally `/nix/store/zxlx9jnylh7la5m48bsqb1wfm5l9r0bd-cc-ci-run`). Same store path ⇒ same
+     pyEnv, same tooling, same PLAYWRIGHT_BROWSERS_PATH.
+
+5. Host divergence removed:
+   - Both host `configuration.nix` `systemPackages` lines are textually identical
+     (`pkgs.ccciRuntimeTools ++ [ pkgs.openssh ]`). The `cc-ci` host now GAINS `git-lfs`+`openssl`
+     on its system PATH (`ls $(nix eval --raw '.?submodules=1#nixosConfigurations.cc-ci.config.system.build.toplevel')/sw/bin/ | grep -E '^(git-lfs|openssl)$'` → both present; pre-refactor cc-ci lacked git-lfs).
+
+6. Future-dep propagation: adding a pkg to `ccciRuntimeTools` in packages.nix lands in cc-ci-run's
+   runtimeInputs (Drone + sweep) AND both hosts' systemPackages from the single edit.
+
+## Build backlog
+See `BACKLOG-nixenv.md`. M2 (deploy + live parity witness) is gated behind the M1 PASS.

nix/hosts/cc-ci-hetzner/configuration.nix

@@ -63,13 +63,13 @@
     allowedTCPPorts = [ 22 80 443 ];
   };
 
-  environment.systemPackages = with pkgs; [
+  # Phase `nixenv`: the Drone exec runner resolves recipe shell-outs from this host PATH
-    curl
+  # (PATH=/run/current-system/sw/bin). Reference the SINGLE shared harness tool set
-    git
+  # (pkgs.ccciRuntimeTools — includes git-lfs, openssl, etc.) instead of a hand-maintained list,
-    git-lfs
+  # so the Drone path and the harness env (cc-ci-run / sweep) can never diverge. `openssh` is a
-    jq
+  # host-only addition (ssh client), not part of the recipe-test tool set. Identical to the
-    openssh
+  # `cc-ci` host config — the prior one-off `git-lfs` divergence is gone.
-  ];
+  environment.systemPackages = pkgs.ccciRuntimeTools ++ [ pkgs.openssh ];
 
   nix.settings.experimental-features = [ "nix-command" "flakes" ];
 

nix/hosts/cc-ci/configuration.nix

@@ -53,12 +53,12 @@
     allowedTCPPorts = [ 22 ];
   };
 
-  environment.systemPackages = with pkgs; [
+  # Phase `nixenv`: the Drone exec runner resolves recipe shell-outs from this host PATH
-    curl
+  # (PATH=/run/current-system/sw/bin). Reference the SINGLE shared harness tool set
-    git
+  # (pkgs.ccciRuntimeTools — includes git-lfs, openssl, etc.) instead of a hand-maintained list,
-    jq
+  # so the Drone path and the harness env (cc-ci-run / sweep) can never diverge. `openssh` is a
-    openssh
+  # host-only addition (ssh client), not part of the recipe-test tool set.
-  ];
+  environment.systemPackages = pkgs.ccciRuntimeTools ++ [ pkgs.openssh ];
 
   nix.settings.experimental-features = [ "nix-command" "flakes" ];
 

nix/modules/harness.nix

@@ -1,20 +1,11 @@
-# CI harness runtime (M4): a reproducible Python env with pytest + Playwright and the
+# CI harness runtime (M4): `cc-ci-run` exposes a reproducible Python env (pytest + Playwright,
-# Nix-provided browsers, exposed as `cc-ci-run` on the host so the Drone exec pipeline (and
+# Nix-provided browsers) plus the recipe-test tooling, so the Drone exec pipeline (and manual dev)
-# manual dev) can run the harness with `cc-ci-run runner/run_recipe_ci.py`. Playwright on NixOS
+# can run the harness with `cc-ci-run runner/run_recipe_ci.py`.
-# needs the browsers from nixpkgs (not a downloaded copy) via PLAYWRIGHT_BROWSERS_PATH.
+#
+# Phase `nixenv`: `cc-ci-run` (and the env it carries) is now defined ONCE in modules/packages.nix
+# as `pkgs.cc-ci-run` — the SAME definition the nightly sweep execs and the same tool set the host
+# `systemPackages` reference. This module just installs it on the host.
 { pkgs, ... }:
-let
-  pyEnv = pkgs.python3.withPackages (ps: with ps; [ pytest playwright ]);
-  ccciRun = pkgs.writeShellApplication {
-    name = "cc-ci-run";
-    runtimeInputs = [ pyEnv pkgs.abra pkgs.docker pkgs.git pkgs.coreutils pkgs.util-linux ];
-    text = ''
-      export PLAYWRIGHT_BROWSERS_PATH=${pkgs.playwright-driver.browsers}
-      export PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1
-      exec ${pyEnv}/bin/python3 "$@"
-    '';
-  };
-in
 {
-  environment.systemPackages = [ ccciRun ];
+  environment.systemPackages = [ pkgs.cc-ci-run ];
 }

nix/modules/nightly-sweep.nix

@@ -13,30 +13,22 @@
 # and lets sweep-logic changes ship via a checkout pull without a store rebuild.
 { pkgs, ... }:
 let
-  # The sweep drives run_recipe_ci.py (pytest/playwright) — needs the full harness env like cc-ci-run.
+  # Phase `nixenv`: the sweep drives nightly_sweep.py (pytest/playwright) through the SAME
-  pyEnv = pkgs.python3.withPackages (ps: with ps; [ pytest playwright ]);
+  # entrypoint the Drone runner uses — `cc-ci-run` (pkgs.cc-ci-run, defined once in packages.nix).
+  # So the python env + recipe-test tooling are IDENTICAL to the Drone path by construction: no
+  # duplicate pyEnv, no parallel runtimeInputs list, and no host-PATH-prepend patch (the old
+  # DEFECT-3 fix) — git-lfs/bash/util-linux/openssl/etc. now come from cc-ci-run's runtimeInputs,
+  # which is the single shared `ccciRuntimeTools`. This wrapper only sets the sweep-specific
+  # environment (HOME, the deployed-checkout repo) before handing off.
   sweep = pkgs.writeShellApplication {
     name = "cc-ci-nightly-sweep";
-    # util-linux provides `script` (abra's PTY wrapper for backup/restore TTY ops) — same as cc-ci-run.
+    runtimeInputs = [ pkgs.cc-ci-run ];
-    # bash: the sweep's mirror_sync shells out to `bash scripts/recipe-mirror-sync.sh`; writeShellApplication
-    # sets a clean PATH limited to runtimeInputs, so bash must be listed (a real timer fire caught its
-    # absence — manual ssh runs had bash on PATH and masked it).
-    runtimeInputs = with pkgs; [ bash abra docker git curl jq gnused gnugrep gnutar coreutils util-linux procps ];
     text = ''
       export HOME=/root
-      export PLAYWRIGHT_BROWSERS_PATH=${pkgs.playwright-driver.browsers}
-      export PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1
-      # ENV PARITY with the Drone recipe-CI runner (canon DEFECT-3): the recipes + their tests shell
-      # out to host tooling (git-lfs for gitea, openssl, etc.). Drone's exec runner runs them with
-      # PATH=/run/current-system/sw/bin:/run/wrappers/bin; writeShellApplication otherwise gives a
-      # clean nix-only PATH, so the timer sweep silently lacked tools the recipes assume (a real fire
-      # caught git-lfs + bash gaps that manual ssh runs, with a login PATH, masked). Prepend the host
-      # system PATH so the sweep validates recipes in the SAME environment Drone does.
-      export PATH="/run/current-system/sw/bin:/run/wrappers/bin:$PATH"
       # canon M1.4: read enrollment + run the harness from the deployed checkout (has tests/).
       export CCCI_REPO=/etc/cc-ci
       cd "$CCCI_REPO"
-      exec ${pyEnv}/bin/python3 "$CCCI_REPO/runner/nightly_sweep.py"
+      exec cc-ci-run "$CCCI_REPO/runner/nightly_sweep.py"
     '';
   };
 in

nix/modules/packages.nix

@@ -1,25 +1,77 @@
 # Project package overlay. `abra` (the Co-op Cloud CLI) is exposed as `pkgs.abra` so every
 # module (systemPackages, the proxy/drone reconcile oneshots) can use the same pinned build.
+#
+# Phase `nixenv` — SINGLE SOURCE OF TRUTH for the harness/recipe-test runtime env. The Drone
+# runner entrypoint (`cc-ci-run`), the nightly/weekly sweep timer, and host `systemPackages` ALL
+# reference the definitions here, so a dependency can never be present for one path and missing
+# for another (that was DEFECT-3, caught in `canon`: the sweep's tool list had drifted from what
+# the Drone path provided). Adding the next dependency to `ccciRuntimeTools` propagates atomically
+# to every consumer.
 _:
 {
   nixpkgs.overlays = [
-    (_: prev: {
+    (final: prev:
-      abra = prev.stdenv.mkDerivation rec {
+      let
-        pname = "abra";
+        # The harness drives run_recipe_ci.py / nightly_sweep.py (pytest + Playwright). Defined
-        version = "0.13.0-beta";
+        # ONCE here; consumed only via `cc-ci-run` below (no module builds its own pyEnv anymore).
-        src = prev.fetchurl {
+        ccciPyEnv = final.python3.withPackages (ps: with ps; [ pytest playwright ]);
-          url = "https://git.coopcloud.tech/toolshed/abra/releases/download/${version}/abra_${version}_linux_amd64.tar.gz";
+      in
-          sha256 = "12csk6wp1pk9cspzqfl4a6h5jdz8p055sf0ggxw9k7ljhpd5qvc6";
+      {
+        abra = prev.stdenv.mkDerivation rec {
+          pname = "abra";
+          version = "0.13.0-beta";
+          src = prev.fetchurl {
+            url = "https://git.coopcloud.tech/toolshed/abra/releases/download/${version}/abra_${version}_linux_amd64.tar.gz";
+            sha256 = "12csk6wp1pk9cspzqfl4a6h5jdz8p055sf0ggxw9k7ljhpd5qvc6";
+          };
+          sourceRoot = ".";
+          nativeBuildInputs = [ prev.autoPatchelfHook ];
+          buildInputs = [ prev.stdenv.cc.cc.lib ];
+          installPhase = ''
+            runHook preInstall
+            install -Dm755 abra "$out/bin/abra"
+            runHook postInstall
+          '';
         };
-        sourceRoot = ".";
+
-        nativeBuildInputs = [ prev.autoPatchelfHook ];
+        # === Single source of truth: the recipe-test runtime tooling ===
-        buildInputs = [ prev.stdenv.cc.cc.lib ];
+        # The union of everything a recipe test + the harness shell out to. This is a
-        installPhase = ''
+        # superset-or-equal of every prior list (cc-ci-run's old runtimeInputs, the sweep's old
-          runHook preInstall
+        # runtimeInputs, and the host PATH's hand-maintained `curl git git-lfs jq`):
-          install -Dm755 abra "$out/bin/abra"
+        #   abra docker git coreutils util-linux        (old cc-ci-run)
-          runHook postInstall
+        # + bash curl jq gnused gnugrep gnutar procps    (old sweep)
-        '';
+        # + git-lfs openssl                              (formerly only on the host PATH / plan §2)
-      };
+        # `util-linux` provides `script` (abra's PTY wrapper for backup/restore TTY ops).
-    })
+        ccciRuntimeTools = with final; [
+          abra
+          docker
+          git
+          git-lfs
+          bash
+          coreutils
+          util-linux
+          curl
+          jq
+          gnused
+          gnugrep
+          gnutar
+          openssl
+          procps
+        ];
+
+        # === The harness entrypoint, used by BOTH the Drone exec pipeline (.drone.yml:
+        # `cc-ci-run runner/run_recipe_ci.py`) and the nightly sweep (which execs this same
+        # binary). Carries the full tool set in runtimeInputs so the recipe shell-outs resolve
+        # the same tooling on every path, independent of the caller's inherited PATH. ===
+        cc-ci-run = final.writeShellApplication {
+          name = "cc-ci-run";
+          runtimeInputs = [ ccciPyEnv ] ++ final.ccciRuntimeTools;
+          text = ''
+            export PLAYWRIGHT_BROWSERS_PATH=${final.playwright-driver.browsers}
+            export PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1
+            exec ${ccciPyEnv}/bin/python3 "$@"
+          '';
+        };
+      })
   ];
 }