upstream(n8n): release-notes sources 2.36.4..2.37.3

The Gitea host was stored in the credentials env file, so a literal
secret scanner treated it as a secret and refused two READMEs in another
repo that link to git.autonomic.zone.

It was never secret: advisory-scan.py, pr-survey.py and recipe-report.py
already default it to git.autonomic.zone, and it is this repo's own
remote host. It is a bare hostname with no scheme, userinfo or path.

Removed from the env file and defaulted here the same way the Python
already does. GITEA_USERNAME and GITEA_PASSWORD keep their :? guard --
those are real credentials and must still fail loudly if unset.

cc-ci-plan/JOURNAL.md

@@ -867,6 +867,65 @@ session cc-ci-orchestrator-stale can be killed; recipe-mirrors org still private
   (/srv/cc-ci-orch/cc-ci), and task-tool subagents inherit their parent session's directory. The
   config now lives in the cc-ci repo at that path. VERIFIED end-to-end with the launcher's exact
   invocation: parent=glm-5.2, subagent=deepseek-v4-pro read back from the session DB.
-  LESSON: `opencode debug config` proves resolution, NOT binding — only a live subagent's recorded
-  modelID proves binding. First attempt was a false pass because the probe passed --dir (unlike the
-  real launcher) and landed in a different project.
+   LESSON: `opencode debug config` proves resolution, NOT binding — only a live subagent's recorded
+   modelID proves binding. First attempt was a false pass because the probe passed --dir (unlike the
+   real launcher) and landed in a different project.
+
+## Session 2026-08-15 19:25 UTC — opencode glm-5.2
+
+**Left off:** Recovered the stalled 2026-08-14 weekly /upgrade-all run. Killed a supervisor that had
+been relaunching hourly for ~40h (balance exhausted), then started a FRESH scoped upgrader. Run is now
+progressing (surveying the 9 remaining recipes). Watching it through to completion.
+
+**What happened (the stall):**
+- The 2026-08-14 /upgrade-all run (session ses_00200382fffeYIGl2sc3mO9JId) stalled at 03:18 Aug 14
+  mid-`lasuite-drive` with `Error: Insufficient balance` (opencode zen workspace balance ran out). It
+  had already done bluesky-pds, ghost, gitea, hedgedoc (PRs) + immich, lasuite-docs (SKIPPED up-to-date)
+  alphabetically; lasuite-drive had a plan + partial PR #6 but no RESULT/verify.
+- The supervisor cron (glm-5.2, opencode-go tier) relaunched an hourly one-shot supervisor ~40 times
+  to "drive it to completion", but each was also balance-walled (and later, just spinning). The run sat
+  INCOMPLETE + not progressing for 40h. No weekly summary, no report published for week of Aug 14.
+
+**What I did this session:**
+- Diagnosed: the opencode zen endpoint is NOW healthy (direct probe `say OK` → HTTP 200 in 1.35s —
+  balance is restored). But resuming the ORIGINAL giant session is impossible: it's 2.58M tokens
+  (267K input + 2.3M cache) and `opencode run -s … --continue` sits idle on `do_epoll_wait` with zero
+  I/O — the inference endpoint silently drops the oversized request (matches the supervisor's
+  `socket connection was closed unexpectedly` errors). A fresh small `opencode run` works fine. So the
+  giant session is unresumable; a fresh start is the only path.
+- Killed the stuck supervisor (tmux `cc-ci-supervisor`, proc 377329).
+- `UPGRADER_ARGS="lasuite-drive lasuite-meet mailu matrix-synapse mattermost-lts mumble n8n plausible
+  wordpress --sequential" python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py fresh` — this killed the
+  stuck resume, archived the old giant session (`archive-cc-ci-upgrader — 2026-08-14`), reclaimed 10GB
+  stale images on cc-ci (disk 29%), and started a FRESH small session
+  `ses_ff920cf39ffeoogwXHTajp94cr` (zen/glm-5.2) scoped to the 9 recipes not yet done this week
+  (positions 13-21 alphabetically; positions 1-12 were already surveyed — 6 PRs + 6 up-to-date). A
+  fresh watchdog is watching the new session. The skill is idempotent (reuses existing PRs incl.
+  lasuite-drive #6, never duplicates), so scoping is safe.
+- Confirmed the fresh run is progressing: pane shows it surveying the 9 recipes (verified all present
+  in abra + all `weekly` tier; currently probing plausible/wordpress tags). Proc alive, log growing.
+
+**Phase / loop state:**
+- Build/adversary loops: STOPPED (whole sequence completed 2026-08-01; phase ghost DONE).
+- Weekly upgrader: RUNNING (fresh session ses_ff920cf39, scoped 9 recipes, --sequential, watchdog up).
+- cc-ci server: healthy (disk 29%, runner active).
+
+**Open items for next session:**
+- **Monitor the fresh upgrader to completion.** It will survey the 9 recipes, /recipe-upgrade the
+  upgradeable ones (subagents, !testme verify, open/extend PRs — NEVER merge), write the weekly summary
+  to `/srv/cc-ci/.cc-ci-logs/upgrades/`, then `launch-report.py fresh` (the upgrade-all skill does this
+  itself per SKILL.md §5), print `UPGRADE RUN COMPLETE`, and go idle. If it stalls on a usage limit,
+  the watchdog auto-resumes the SAME (small) session — that works now.
+- **Do NOT try to resume the archived giant session ses_00200382** — it's unresumable (endpoint drops
+  the 2.58M-token request). It's archived; leave it.
+- After the run completes + report publishes, operator review queue = this week's recipe PRs.
+- The supervisor cron (hourly at XX:07) should now leave the run alone once it's progressing; if a
+  supervisor fires while the run is mid-flight, its guardrails say to hand back to the resumed run, not
+  double-write. No action needed unless it interferes.
+
+**Notes:**
+- Root cause of the 40h silence was the same BUG 1 from 2026-08-10 (supervisor progress gate) partly:
+  the supervisor kept firing because the run never reached "progressing". Now that balance is restored
+  and a fresh small session is running, the gate should see progress and stand down.
+- Lesson: when a weekly run dies mid-flight on a giant context, do NOT resume the original session —
+  start fresh and scope to the remaining recipes. The /upgrade-all skill is idempotent so this is safe.

cc-ci-plan/reconcile-upstream.sh

@@ -36,7 +36,7 @@ USED_RECIPES="${USED_RECIPES:-$HERE/used-recipes.md}"
 
 [ -f "$RECONCILE" ] || { echo "ERROR: reconcile helper not found: $RECONCILE" >&2; exit 1; }
 set -a; . "$TESTENV"; set +a
-: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:?}"
+: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:=git.autonomic.zone}"
 
 if [ "${1:-}" = "--all" ]; then
   mapfile -t RECIPES < <(awk '!/^[[:space:]]*#/ && ($2=="weekly" || $2=="external") {print $1}' "$USED_RECIPES")

cc-ci-plan/upstream/mattermost-lts.md

@@ -6,6 +6,14 @@
 | postgres | postgres                                  | https://github.com/postgres/postgres              | https://www.postgresql.org/docs/release/                          |
 
 ## Standing notes
+  - **2026-08-28 re-check** (endoflife.date/api/mattermost.json 2026-08-28; Docker Hub; GitHub
+    releases): **11.7.10** (released 2026-08-26, "various bug fixes", not prerelease) is the newest
+    11.7.x ESR patch — the ESR/LTS line has NOT moved, still **11.7** (EOL **2027-05-15**). This run
+    bumps open PR #2 **11.7.9 → 11.7.10** (ESR patch within the same line). 11.8/11.9/11.10 remain
+    innovation releases (EOL 2026-09-15 / 10-15 / 11-15), NOT ESR — do NOT target; 10.12 expired.
+    postgres **15-alpine HELD** (DB-major 16/17/18 out of scope, plain-postgres recipe w/o
+    pgautoupgrade: operator dump/restore needed). git.autonomic.zone mirror main reconciled to
+    coopcloud (86e0d0b5 = 10.11.22, upstream still pins the EXPIRED 10.11 ESR).
 - mattermost-lts tracks the **11.7 ESR (Extended Support Release)** = Mattermost's LTS line. The "LTS"
   concept in Mattermost is **ESR**; there is NO "10.x LTS line". Release calendar (endoflife.date/mattermost):
   - **10.11** = previous ESR, security support **ends 2026-08-15** (~7 weeks) — migrated away 2026-06-19
@@ -76,6 +84,16 @@
   The previous inline-label approach (no restore hook) was a defect: raw PGDATA restore without a reload
   was a silent no-op. Fixed in PR #2 (restore fix cherry-picked from PR #1 ci/pg-restore).
 
+  - **2026-08-21 re-check** (endoflife.date/api/mattermost.json 2026-08-17; Mattermost docs; GitHub):
+    **11.7 ESR is the only current ESR/LTS** — latest patch **11.7.9** (2026-08-13, "Low to High
+    security fixes"), EOL 2027-05-15. **10.11 ESR ENDED 2026-08-15** — upstream main still pins
+    10.11.22 (now on an EXPIRED ESR). 11.10/11.9/11.8 are innovation (EOL 2026-11-15/10-15/09-15),
+    NOT LTS. The `-lts` recipe must track 11.7. This run **redirects open PR #2 away from the
+    innovation 11.10.0 onto 11.7.9** (the correct LTS/ESR; ESR→ESR 10.11→11.7 is "fully supported
+    and tested" per Mattermost docs — no staging hop). Refer to the 2026-08-15 report where PR #2
+    was on 11.10.0 (operator chose innovation line that week); operator should now decide the
+    `-lts` line. postgres 15-alpine still HELD (DB-major out of scope).
+
 ## NVD CPE fallback
 This project publishes nothing machine-readable we can reach — no GitHub advisory feed,
 no release-attributable changelog — so its CVE count was `?` (nothing measured). NVD is

cc-ci-plan/upstream/n8n.md

@@ -65,3 +65,49 @@
 - 2.35.3 (2026-08-14, Pre-release): bugfixes (Google Ads v21→v25 API migration, MS Teams OAuth scope
   restore, workflow publication outbox abort deadline) + feature (skip update approval for workflows
   from same Instance AI session). Not deployed (2.35.2 was the survey target).
+- 2.36.0 (2026-08-18, Pre-release; 2.36.1 was briefly tagged then withdrawn — 2.36.2 compares
+  2.36.0...2.36.2): a large feature minor. Core/ai-builder/editor bugfixes + new features (Agent
+  sandboxes + writable workspaces, MCP folder/move/read-data-table tools, Schedule Trigger missed-
+  execution grace period + catch-up coalescing, Confluence Cloud OAuth2 + page get sub-tree, MiniMax
+  M3/H3, LangSmith session debug export, role-mapping/roles API endpoints, n8n User Auth GA, etc.).
+  No breaking compose/env/migration changes, no `N8N_*` env renames; the deprecation notices
+  (Array.merge→Array.mergeIntoObject in expressions, workflow tags API migration) are API-level only.
+  2.36.2 (2026-08-19, patch): 3 fixes (decorator-route request schema in /discover, end-user
+  credential resolution in node parameters, editor agent-preview alignment). 2.36.3 (2026-08-20,
+  patch): 4 core/editor fixes (task-runners only restart when slow, expression engine init on
+  expression commands, test-webhook isolate release after teardown, Edit Fields type selector styles).
+- 2026-08-21 run: planning 2.34.4 → 2.36.3 (newest tag abra lists). Whole 2.36.x line still
+  Pre-release on GitHub (2.33.5/Latest was superseded by 2.35.x line). Rolling upgrade safe, sqlite
+  mode, TypeORM migrations auto-run on boot. No operator action required.
+- 2.36.4 (2026-08-21, Pre-release): 3 core fixes (env normalization before schema parsing; chat and
+  MCP trigger auth-mode validation for end-user credentials; wrap trigger closeFunction in an
+  expression isolate).
+- 2.36.5 (2026-08-21, Pre-release): 1 core fix (raise AI Assistant model verification token limit).
+- 2.36.6 (2026-08-24): 1 editor fix (personalization survey / community registration modal on
+  Instance AI landing page).
+- 2.36.7 (2026-08-25, **current Stable / Latest badge**; `stable` tag points here): API fix (adapt
+  binary data endpoint), API cross-backport (require a JSON content type on decorator routes that
+  take a body), core fixes (reconcile license auto-renewal on startup leader takeover; route OAuth2
+  token requests through environment proxies), feature (remove offset query param from list workflow
+  history endpoint). This is the newest stable-line release; the 2.37.x line is still pre-release.
+- 2.37.0 (2026-08-25, Pre-release): a large feature minor. Agent Builder / AIA / core/editor
+  bugfixes + features (Slack Agent view, prompt caching for Anthropic, SharePoint node v2 = default,
+  form-trigger OAuth2 (added then reverted), role-mapping rule CRUD endpoints, execution
+  startedAfter/startedBefore filters, Agent/community-package hardening: serialize community
+  package installs + non-destructive downloads + cold-boot storm reduction, MCP tool schemas as JSON
+  Schema 2020-12, distroless n8n image variant). **Deprecation: the "Any workflow" caller policy**
+  (API/config-level — flag for operators using execution-caller policies). Two enforced API behavior
+  changes: **require JSON content type on decorator body routes** and **"adapt binary data endpoint"**
+  (HTTP clients sending a body to those routes must send `Content-Type: application/json`). No
+  breaking compose/env/migration changes, no `N8N_*` env renames.
+- 2.37.1 (2026-08-25, Pre-release): 1 core fix (route OAuth2 token requests through env proxies in
+  every process).
+- 2.37.3 (2026-08-27, Pre-release; 2.37.2 was withdrawn): 2 runner-image fixes (copy glibc into
+  distroless runners image as a matched pair; stage libatomic into the distroless runners image) —
+  relevant if using n8n's community/distroless runner image.
+- 2026-08-28 run: PR #7 extended 2.34.4 → **2.37.3** (newest tag abra lists; 2.37.2 withdrawn, that
+  tag does not exist). 2.36.7 holds the Stable/Latest badge; 2.37.x remains Pre-release on GitHub,
+  consistent with the tracking-the-newest-tag precedent (2.34.2/2.35.2/2.36.3 were pre-release when
+  taken). Rolling upgrade safe (sqlite, TypeORM auto-migrate on boot). Operator flag: the two 2.37.0
+  API behavior changes (JSON content-type on decorator body routes; binary-data endpoint adapt) only
+  affect HTTP API callers. Recommended release: `-y` (feature minor).

cc-ci-plan/upstream/wordpress.md

@@ -4,6 +4,7 @@
 |---------|-------|-------------|----------------------|
 | app | wordpress | https://github.com/WordPress/WordPress | https://wordpress.org/news/category/releases/ (image: https://hub.docker.com/_/wordpress) |
 | db | mariadb | https://github.com/MariaDB/server | https://mariadb.com/kb/en/release-notes/ (image: https://hub.docker.com/_/mariadb) |
+| ftp (optional overlay) | atmoz/sftp | https://github.com/atmoz/sftp | https://github.com/atmoz/sftp/releases (image: https://hub.docker.com/r/atmoz/sftp) |
 
 ## Standing notes
 - Enrolled 2026-08-03 (operator request). Recipe = coopcloud `wordpress`; mirror
@@ -22,3 +23,7 @@
   image). mariadb major bumps (e.g. 12.x→13.x) need the usual dump/restore caution if the
   recipe ever pins a non-`mariadb`-auto-upgrading setup; the official image handles minor
   bumps in place.
+- `atmoz/sftp` is pinned as the floating `alpine` tag (updated 2026-07-28, current as of
+  2026-08-21) — not a semver abra can evaluate, so it never contributes an upgrade candidate;
+  verified current by direct Docker Hub tag check. Only bump if a real newer supported tag
+  exists and the app needs it.

cc-ci-plan/used-recipes.md

@@ -31,7 +31,7 @@ matrix-synapse    weekly
 mattermost-lts    weekly
 mumble            weekly
 n8n               weekly
-plausible         weekly
+plausible         external   # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded
 uptime-kuma       external   # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded
 wordpress         weekly
 ```

flake.nix

@@ -18,6 +18,11 @@
       system = "x86_64-linux";
     in
     {
+      # The cc-ci part of a host, on its own, so a host that runs cc-ci can import just this and
+      # keep its own (unrelated) configuration separate. Split out 2026-08-20; consumed by
+      # notplants-nix's `notplants-orchestrator` host.
+      nixosModules.cc-ci = ./nix/modules/cc-ci.nix;
+
       # Hetzner cpx11 host (nixos-infect generated hardware.nix + orchestrator config).
       # Provision with terraform/ then run Stage 2 per terraform/README.md.
       nixosConfigurations.cc-ci-orchestrator-hetzner = nixpkgs.lib.nixosSystem {

nix/hosts/cc-ci-orchestrator-hetzner/configuration.nix

@@ -14,7 +14,11 @@
   # with ACME, and opens 80/443 (previously only 22 was public). Canonical source of
   # the module is the project repo; ../atproto-likes.nix is a copy kept in this tree
   # because pure evaluation cannot import a path outside the flake.
-  imports = [ ../../atproto-likes.nix ];
+  # cc-ci.nix holds everything that exists FOR cc-ci (the loops, the cc-ci orchestrator session,
+  # the weekly recipe-upgrade run and its hourly supervisor). Split out 2026-08-20 so this host —
+  # a general agent box that also serves unrelated projects — can be reasoned about separately
+  # from cc-ci. Also exported as `nixosModules.cc-ci` from this repo's flake for other hosts.
+  imports = [ ../../atproto-likes.nix ../../modules/cc-ci.nix ];
 
   services.openssh = {
     enable = true;
@@ -225,72 +229,7 @@ SSHCFG
     };
   };
 
-  # cc-ci-loops supervisor — workspace staged 2026-05-31, so ENABLED for reboot-resilience.
-  systemd.services.cc-ci-loops = {
-    description = "cc-ci Builder/Adversary loops + watchdog (launch.sh start)";
-    wantedBy = [ "multi-user.target" ];  # enabled after workspace staged (Hetzner cutover)
-    after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
-    wants = [ "network-online.target" ];
-    serviceConfig = {
-      # KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the
-      # default (control-group) systemd kills every leftover process in the cgroup when the unit
-      # stops — and since one tmux server hosts every agent session on this host, a rebuild that
-      # merely touched this unit wiped all of them (operator 2026-08-01). Only the (already
-      # exited) main process is killed now; `systemctl stop` therefore does NOT tear down agents.
-      KillMode = "process";
-      Type = "oneshot"; RemainAfterExit = true;
-      User = "loops"; Group = "users";
-      WorkingDirectory = "/srv/cc-ci/cc-ci";
-      # Append one line to REBOOTS.md per genuine reboot (boot_id-gated; not on manual restart).
-      ExecStartPre = "${pkgs.bash}/bin/bash /srv/cc-ci/cc-ci-plan/reboot-log.sh";
-    };
-    # CLAUDE_BIN points at the standalone CLI installed by claude-install.service; the loops
-    # backend defaults to claude (persisted in .loop-backend). Without this, launch.py's preflight
-    # `which(claude)` fails because the systemd `path` below has no /home/loops/.local/bin.
-    environment = { RESUME_PHASE = "1"; HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
-    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
-    script = ''
-      # Put the standalone claude/opencode binaries on PATH. On a cold boot this is the env the
-      # tmux server (and thus every agent session) inherits, so bare `claude` resolves everywhere.
-      export PATH="/home/loops/.local/bin:$PATH"
-      [ -x /srv/cc-ci/cc-ci-plan/launch.sh ] && /srv/cc-ci/cc-ci-plan/launch.sh start || \
-        echo "workspace not staged yet — skipping loop start"
-    '';
-  };
 
-  # cc-ci-orchestrator supervisor — the operator's steering session. Same shape as
-  # lichen-orchestrator / project-orchestrator above: this unit only LAUNCHES the orchestrator's
-  # tmux session via the agent-orchestrator harness (cc-ci-plan/agents.py); it does not own the
-  # session or the tmux server. The orchestrator agent is declared in cc-ci-plan/agents.toml on
-  # the OPencode backend (backend = "opencode", model = "opencode/glm-5.2"), so on boot it
-  # attaches to the shared opencode web server (opencode-web.service below) and is reachable for
-  # Remote Control at https://oc.commoninternet.net under the /srv/cc-ci-orch project. The harness
-  # watchdog (started by `agents.py up`) keeps it alive: heal-only (no stall reboots — a persistent
-  # supervisor must not be killed just for idling). Added 2026-08-03 to give the cc-ci orchestrator
-  # the same reboot-resilience the other two orchestrators already have.
-  systemd.services.cc-ci-orchestrator = {
-    description = "cc-ci orchestrator (operator steering session) — agents.py up orchestrator, opencode backend";
-    wantedBy = [ "multi-user.target" ];
-    after = [ "network-online.target" "tailscaled.service" "opencode-web.service" ];
-    wants = [ "network-online.target" ];
-    serviceConfig = {
-      # KillMode=process: see the note on cc-ci-loops — a rebuild that merely touches this unit
-      # must not tear down the (shared) tmux server and every agent session with it.
-      KillMode = "process";
-      Type = "oneshot"; RemainAfterExit = true;
-      User = "loops"; Group = "users";
-      WorkingDirectory = "/srv/cc-ci-orch";
-    };
-    environment = { HOME = "/home/loops"; };
-    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
-    script = ''
-      export PATH="/home/loops/.local/bin:$PATH"
-      proj="/srv/cc-ci-orch"
-      echo "$(cat /proc/sys/kernel/random/boot_id) boot $(date -u +%FT%TZ) — cc-ci-orchestrator up" \
-        >> "$proj/cc-ci-plan/.ao-boot.log" 2>/dev/null || true
-      cd "$proj" && python3 cc-ci-plan/agents.py up orchestrator || echo "cc-ci orchestrator agents.py up failed"
-    '';
-  };
 
   # p-lichen-orchestrator supervisor — the SEPARATE lichen.page testing/hardening orchestrator
   # (distinct from cc-ci-loops above). Reboot-resilience: on boot, resume the orchestrator's Remote
@@ -356,76 +295,7 @@ SSHCFG
     '';
   };
 
-  # Weekly recipe upgrade — runs /upgrade-all over every enrolled recipe (opens recipe PRs
-  # verified by !testme, never merges). Replaces the boot-fragile busybox-crond-in-tmux from
-  # phase 5 §4 with a reboot-safe systemd timer. The service is timer-triggered only (NOT
-  # wantedBy multi-user.target) so it never runs on boot/activation — only on the schedule.
-  systemd.services.cc-ci-upgrade-all = {
-    description = "cc-ci weekly /upgrade-all run (recipe upgrade survey + PRs, never merges)";
-    after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
-    wants = [ "network-online.target" ];
-    serviceConfig = {
-      Type = "oneshot";  # launch-upgrader.py spawns the cc-ci-upgrader tmux session and returns
-      User = "loops"; Group = "users";
-      WorkingDirectory = "/srv/cc-ci";
-      # Optional per-run overrides for backend/model (LOOP_BACKEND, LOOP_MODEL, OPENCODE_SHARE,
-      # UPGRADER_ARGS, …). The leading "-" makes it optional: absent file → claude/sonnet defaults
-      # (current behavior). To run the weekly job on e.g. opencode-go/glm-5.2, drop a file with
-      #   LOOP_BACKEND=opencode
-      #   LOOP_MODEL=opencode-go/glm-5.2
-      # No rebuild needed to switch — the env file is read at each timer fire. Holds no secrets
-      # (the opencode-go API key lives in ~/.local/share/opencode/auth.json, mode 600).
-      EnvironmentFile = "-/srv/cc-ci/upgrader.env";
-    };
-    environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
-    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
-    script = ''
-      export PATH="/home/loops/.local/bin:$PATH"
-      python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py start >> /srv/cc-ci/.cc-ci-logs/upgrader-cron.log 2>&1
-    '';
-  };
 
-  systemd.timers.cc-ci-upgrade-all = {
-    description = "Weekly trigger for cc-ci-upgrade-all (Thursdays 22:00 America/New_York — Boston 10pm)";
-    wantedBy = [ "timers.target" ];
-    timerConfig = {
-      # 10pm Thursday Boston time — DST-aware (EDT→02:00 UTC, EST→03:00 UTC) via the tz in OnCalendar.
-      OnCalendar = "Thu *-*-* 22:00:00 America/New_York";
-      Persistent = true;  # if the box was down at the scheduled time, run once on next boot
-    };
-  };
 
-  # Hourly SUPERVISOR — a glm-5.2 orchestrator wake-up that keeps the weekly run on track. The
-  # log-idle/429 watchdog only handles opencode-go usage-limit stalls; it does NOT cover a host
-  # disk-full crash (which killed the 2026-07-03 run) or any other environmental wedge. This is a
-  # CHEAP deterministic gate: if the weekly run is complete or actively progressing it does NOTHING
-  # (zero model tokens). Only when a run has stalled/died before completing does it launch a
-  # short-lived glm-5.2 agent that diagnoses the blockage and drives the run to a clean DONE.
-  systemd.services.cc-ci-upgrade-supervisor = {
-    description = "cc-ci hourly weekly-run supervisor (glm-5.2 — drives a stalled /upgrade-all to completion)";
-    after = [ "network-online.target" "tailscaled.service" ];
-    wants = [ "network-online.target" ];
-    serviceConfig = {
-      Type = "oneshot";  # launch-supervisor.py check: gate now, spawn the agent into tmux, return
-      User = "loops"; Group = "users";
-      WorkingDirectory = "/srv/cc-ci";
-      # Shares the weekly run's optional override file (e.g. SUPERVISOR_MODEL=…); "-" = optional.
-      EnvironmentFile = "-/srv/cc-ci/upgrader.env";
-    };
-    environment = { HOME = "/home/loops"; };
-    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
-    script = ''
-      export PATH="/home/loops/.local/bin:$PATH"
-      python3 /srv/cc-ci/cc-ci-plan/launch-supervisor.py check >> /srv/cc-ci/.cc-ci-logs/supervisor-cron.log 2>&1
-    '';
-  };
 
-  systemd.timers.cc-ci-upgrade-supervisor = {
-    description = "Hourly trigger for cc-ci-upgrade-supervisor (weekly-run health check + drive)";
-    wantedBy = [ "timers.target" ];
-    timerConfig = {
-      OnCalendar = "*-*-* *:07:00";  # every hour at :07 (offset from the weekly :00 fire)
-      Persistent = false;            # a missed hourly check is moot — the next hour re-checks
-    };
-  };
 }

nix/modules/cc-ci.nix

@@ -0,0 +1,157 @@
+# cc-ci.nix — everything on this host that exists FOR cc-ci, and nothing else.
+#
+# Split out of the orchestrator host config on 2026-08-20. The host it runs on is a general
+# agent/orchestration box that also serves several unrelated projects; this module is the cc-ci
+# part of it, so that the two can evolve (and be reviewed) independently. It is exported from this
+# repo's flake as `nixosModules.cc-ci` and imported by whichever host runs cc-ci.
+#
+# All of it assumes the cc-ci workspaces exist on the host:
+#   /srv/cc-ci        the loops workspace (+ .cc-ci-logs, upgrader.env)
+#   /srv/cc-ci-orch   this repo (the orchestrator's own working dir)
+# and that a `loops` user, tmux, python3 and the standalone claude/opencode CLIs are present —
+# those are host concerns, provided by the host config, not by this module.
+{ config, pkgs, lib, ... }:
+{
+  # cc-ci-loops supervisor — workspace staged 2026-05-31, so ENABLED for reboot-resilience.
+  systemd.services.cc-ci-loops = {
+    description = "cc-ci Builder/Adversary loops + watchdog (launch.sh start)";
+    wantedBy = [ "multi-user.target" ];  # enabled after workspace staged (Hetzner cutover)
+    after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
+    wants = [ "network-online.target" ];
+    serviceConfig = {
+      # KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the
+      # default (control-group) systemd kills every leftover process in the cgroup when the unit
+      # stops — and since one tmux server hosts every agent session on this host, a rebuild that
+      # merely touched this unit wiped all of them (operator 2026-08-01). Only the (already
+      # exited) main process is killed now; `systemctl stop` therefore does NOT tear down agents.
+      KillMode = "process";
+      Type = "oneshot"; RemainAfterExit = true;
+      User = "loops"; Group = "users";
+      WorkingDirectory = "/srv/cc-ci/cc-ci";
+      # Append one line to REBOOTS.md per genuine reboot (boot_id-gated; not on manual restart).
+      ExecStartPre = "${pkgs.bash}/bin/bash /srv/cc-ci/cc-ci-plan/reboot-log.sh";
+    };
+    # CLAUDE_BIN points at the standalone CLI installed by claude-install.service; the loops
+    # backend defaults to claude (persisted in .loop-backend). Without this, launch.py's preflight
+    # `which(claude)` fails because the systemd `path` below has no /home/loops/.local/bin.
+    environment = { RESUME_PHASE = "1"; HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
+    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
+    script = ''
+      # Put the standalone claude/opencode binaries on PATH. On a cold boot this is the env the
+      # tmux server (and thus every agent session) inherits, so bare `claude` resolves everywhere.
+      export PATH="/home/loops/.local/bin:$PATH"
+      [ -x /srv/cc-ci/cc-ci-plan/launch.sh ] && /srv/cc-ci/cc-ci-plan/launch.sh start || \
+        echo "workspace not staged yet — skipping loop start"
+    '';
+  };
+
+  # cc-ci-orchestrator supervisor — the operator's steering session. Same shape as
+  # lichen-orchestrator / project-orchestrator above: this unit only LAUNCHES the orchestrator's
+  # tmux session via the agent-orchestrator harness (cc-ci-plan/agents.py); it does not own the
+  # session or the tmux server. The orchestrator agent is declared in cc-ci-plan/agents.toml on
+  # the OPencode backend (backend = "opencode", model = "opencode/glm-5.2"), so on boot it
+  # attaches to the shared opencode web server (opencode-web.service below) and is reachable for
+  # Remote Control at https://oc.commoninternet.net under the /srv/cc-ci-orch project. The harness
+  # watchdog (started by `agents.py up`) keeps it alive: heal-only (no stall reboots — a persistent
+  # supervisor must not be killed just for idling). Added 2026-08-03 to give the cc-ci orchestrator
+  # the same reboot-resilience the other two orchestrators already have.
+  systemd.services.cc-ci-orchestrator = {
+    description = "cc-ci orchestrator (operator steering session) — agents.py up orchestrator, opencode backend";
+    wantedBy = [ "multi-user.target" ];
+    after = [ "network-online.target" "tailscaled.service" "opencode-web.service" ];
+    wants = [ "network-online.target" ];
+    serviceConfig = {
+      # KillMode=process: see the note on cc-ci-loops — a rebuild that merely touches this unit
+      # must not tear down the (shared) tmux server and every agent session with it.
+      KillMode = "process";
+      Type = "oneshot"; RemainAfterExit = true;
+      User = "loops"; Group = "users";
+      WorkingDirectory = "/srv/cc-ci-orch";
+    };
+    environment = { HOME = "/home/loops"; };
+    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
+    script = ''
+      export PATH="/home/loops/.local/bin:$PATH"
+      proj="/srv/cc-ci-orch"
+      echo "$(cat /proc/sys/kernel/random/boot_id) boot $(date -u +%FT%TZ) — cc-ci-orchestrator up" \
+        >> "$proj/cc-ci-plan/.ao-boot.log" 2>/dev/null || true
+      cd "$proj" && python3 cc-ci-plan/agents.py up orchestrator || echo "cc-ci orchestrator agents.py up failed"
+    '';
+  };
+
+  # Weekly recipe upgrade — runs /upgrade-all over every enrolled recipe (opens recipe PRs
+  # verified by !testme, never merges). Replaces the boot-fragile busybox-crond-in-tmux from
+  # phase 5 §4 with a reboot-safe systemd timer. The service is timer-triggered only (NOT
+  # wantedBy multi-user.target) so it never runs on boot/activation — only on the schedule.
+  systemd.services.cc-ci-upgrade-all = {
+    description = "cc-ci weekly /upgrade-all run (recipe upgrade survey + PRs, never merges)";
+    after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
+    wants = [ "network-online.target" ];
+    serviceConfig = {
+      Type = "oneshot";  # launch-upgrader.py spawns the cc-ci-upgrader tmux session and returns
+      User = "loops"; Group = "users";
+      WorkingDirectory = "/srv/cc-ci";
+      # Optional per-run overrides for backend/model (LOOP_BACKEND, LOOP_MODEL, OPENCODE_SHARE,
+      # UPGRADER_ARGS, …). The leading "-" makes it optional: absent file → claude/sonnet defaults.
+      # Current config (as of 2026-08-16): the upgrader + report run on tinfoil/deepseek-v4-pro
+      # (LOOP_MODEL + REPORT_MODEL in the env file); the hourly SUPERVISOR stays on glm-5.2
+      # (SUPERVISOR_MODEL defaults to opencode-go/glm-5.2 in launch-supervisor.py, NOT overridden
+      # here). Subagents bind deepseek via the cc-ci repo's opencode config. LOOP_TIER=zen is kept
+      # so the tier check passes; the watchdog's usage-limit probe sends the deepseek model name to
+      # the zen endpoint, which returns 200 (not 429) → resume immediately (correct: tinfoil has no
+      # rolling usage limit to wait out). No rebuild needed to switch — the env file is read at each
+      # timer fire. Holds no secrets (the tinfoil API key lives in the opencode config / auth.json).
+      EnvironmentFile = "-/srv/cc-ci/upgrader.env";
+    };
+    environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
+    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
+    script = ''
+      export PATH="/home/loops/.local/bin:$PATH"
+      python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py start >> /srv/cc-ci/.cc-ci-logs/upgrader-cron.log 2>&1
+    '';
+  };
+
+  systemd.timers.cc-ci-upgrade-all = {
+    description = "Weekly trigger for cc-ci-upgrade-all (Thursdays 22:00 America/New_York — Boston 10pm)";
+    wantedBy = [ "timers.target" ];
+    timerConfig = {
+      # 10pm Thursday Boston time — DST-aware (EDT→02:00 UTC, EST→03:00 UTC) via the tz in OnCalendar.
+      OnCalendar = "Thu *-*-* 22:00:00 America/New_York";
+      Persistent = true;  # if the box was down at the scheduled time, run once on next boot
+    };
+  };
+
+  # Hourly SUPERVISOR — a glm-5.2 orchestrator wake-up that keeps the weekly run on track. The
+  # log-idle/429 watchdog only handles opencode-go usage-limit stalls; it does NOT cover a host
+  # disk-full crash (which killed the 2026-07-03 run) or any other environmental wedge. This is a
+  # CHEAP deterministic gate: if the weekly run is complete or actively progressing it does NOTHING
+  # (zero model tokens). Only when a run has stalled/died before completing does it launch a
+  # short-lived glm-5.2 agent that diagnoses the blockage and drives the run to a clean DONE.
+  systemd.services.cc-ci-upgrade-supervisor = {
+    description = "cc-ci hourly weekly-run supervisor (glm-5.2 — drives a stalled /upgrade-all to completion)";
+    after = [ "network-online.target" "tailscaled.service" ];
+    wants = [ "network-online.target" ];
+    serviceConfig = {
+      Type = "oneshot";  # launch-supervisor.py check: gate now, spawn the agent into tmux, return
+      User = "loops"; Group = "users";
+      WorkingDirectory = "/srv/cc-ci";
+      # Shares the weekly run's optional override file (e.g. SUPERVISOR_MODEL=…); "-" = optional.
+      EnvironmentFile = "-/srv/cc-ci/upgrader.env";
+    };
+    environment = { HOME = "/home/loops"; };
+    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
+    script = ''
+      export PATH="/home/loops/.local/bin:$PATH"
+      python3 /srv/cc-ci/cc-ci-plan/launch-supervisor.py check >> /srv/cc-ci/.cc-ci-logs/supervisor-cron.log 2>&1
+    '';
+  };
+
+  systemd.timers.cc-ci-upgrade-supervisor = {
+    description = "Hourly trigger for cc-ci-upgrade-supervisor (weekly-run health check + drive)";
+    wantedBy = [ "timers.target" ];
+    timerConfig = {
+      OnCalendar = "*-*-* *:07:00";  # every hour at :07 (offset from the weekly :00 fire)
+      Persistent = false;            # a missed hourly check is moot — the next hour re-checks
+    };
+  };
+}