orchestrator-host: weekly opencode auto-upgrade (latest-release check, reinstall, restart web)

opencode-install only installs when the binary is missing, so the standalone
CLI aged in place (1.18.29 for weeks while 1.18.33+ was out); opencode's
built-in autoupdate never fires here because every agent runs inside the
long-lived opencode serve. New opencode-upgrade.service + weekly timer
(Tue 02:00 UTC, an hour before the host auto-update) compares the installed
version with the latest GitHub release, reinstalls via the official
installer when they differ, restarts opencode-web so the new binary takes
effect, and verifies the UI answers its 401 challenge. The auto-update.nix
busy gate is replicated so a mid-flight CI run / weekly upgrade / sweep is
never cut (skipped runs retry next week). Deploying this module does not
itself bump opencode: boot installs stay install-if-missing and the upgrade
is timer-driven only. State per run: .cc-ci-logs/opencode-update-state.

cc-ci-plan/JOURNAL.md

@@ -1311,3 +1311,23 @@ harness → `results.json` + PR card `✅ passed` linking `ci.autonomic.zone/run
 - NOTE: no tailscale on this host (`tailscale: command not found`) — the AGENTS.md "ssh cc-ci"
   alias + 100.90.116.4 peer notes are stale post-rebuild; public-IP SSH is the access path.
   Recovery scripts in scripts/recovery/ still reference old server id 134485294 — worth updating.
+
+---
+## 2026-09-28 — upgrader session (weekly /upgrade-all)
+
+Weekly run 2026-09-28 complete: 19 considered · 7 GREEN PRs · 1 externally-CI-blocked (lasuite-docs, quay minio 401) · 0 failed · 11 skipped. Summary (PR list): `<logs>/upgrades/upgrade-all-2026-09-28.md` — repo ignores .cc-ci-logs, so the PR list lives in the file + the weekly report.
+
+**Operational config change (no PR):** subagent `general` model `deepseek-v4-flash` was killed upstream ("Model access is disabled", zen endpoint; glm-5.2 also dead) — pointed to `deepseek-v4.1-flash` in `/srv/cc-ci-orch/cc-ci/opencode.json`, `/srv/cc-ci-orch/opencode.json` (uncommitted) + live `/etc/cc-ci/opencode.json`. Prior committed setting still in the cc-ci repo's main. Operator: confirm + PR the bump (config change precedent #37/#42).
+
+**minio watch:** docker.io removal (2026-09-18) → now quay.io/minio/minio 401s since ~09-24; lasuite-docs CI-blocked on it; lasuite-drive passed (image cached).
+
+**abandoned worktree note:** `/srv/cc-ci-orch/cc-ci-conc/` (worktree of cc-ci, untracked in orch repo) has a stale worktree at 2f6787f; no writes this week; recorded so the next sweep notices.
+
+Key logs: .cc-ci-logs/upgrades/*-2026-09-28.md (8 per-recipe + summary).
+
+## 2026-10-02 — upgrader session (weekly /upgrade-all)
+
+Weekly run 2026-10-02 complete: 20 considered · 9 upgraded (PRs opened/extended) · 8 GREEN (!testme) · 1 live-verified but CI-blocked externally (lasuite-docs base canonical, commented) + 1 RED on a genuinely stale gitea LFS test (commented; operator re-runs --with-tests) · 0 failed · 9 skipped. Summary (PR list): <logs>/upgrades/upgrade-all-2026-10-02.md — repo ignores .cc-ci-logs, so the PR list lives in the file + the weekly report.
+Key highlights: gitea upstream shipped its first 2-digit stable (28.0.0-rootless, 2026-09-29 — BREAKING git-egress/actions-retention notes documented in PR #10); ghost 6.67.0 fixes HIGH CVE-2026-84383; nulls held at mysql 8.4 / mariadb 12.3 / postgres 13/15 lines per precedent; matrix-synapse extended to synapse 1.162.0 + MAS 1.26.0; immich bumped v3.2.4 (rc.3.3 declined). Operational: recipe-maintainers/gitea mirror had LOST its master branch (abra whole-recipe FATA) — repaired both refs to upstream HEAD f167b73; reconcile-upstream.sh should learn to recreate a missing upstream-default branch ref, not only sync main. 2 targets (gitea, n8n 2.42.2) were caught in follow-up passes after the initial worklist mis-derivation — nothing left unprocessed. Sweep start: 4 leaked volumes + 46.3 GB images reclaimed (disk 70%->40%); end-of-run reap 0 leaked dev deploys; disk 65% (51G free).
+Per-recipe logs: .cc-ci-logs/upgrades/<recipe>-upgrade-2026-10-02.md (10 files incl. survey).
+NEVER MERGED anything; all PRs await operator review + the operator's `abra recipe release <recipe> -x/y/z` after each merge.

cc-ci-plan/upstream/immich.md

@@ -79,6 +79,24 @@
   reassign-faces fix) — no breaking changes, no config additions, no operator action. Done in
   the 2026-09-18 upgrade (v3.2.0→v3.2.2, app+ML only; fresh PR — PR #4 was closed when its
   v3.2.0 content merged upstream as #19 / `1.11.0+v3.2.0`).
+- **immich-server v3.2.4 (2026-09-28, latest stable) pins** (from `docker/docker-compose.yml` @ the
+  v3.2.4 tag): `postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357…` and
+  `valkey:9@sha256:70739f85ad…` — **BOTH identical to the recipe's v3.2.2-era pins** (DB + redis stay
+  put; live `valkey:9` tag has now moved to `ac9c858b…` = **9.1.2**, but immich's tested
+  `70739f85…` = 9.1.1 is kept per the 2026-07-17 precedent). `example.env` @ v3.2.4 byte-identical to
+  v3.2.2. Window v3.2.2→v3.2.4 is 4 commits: mobile sync-status fix (#31644), an internal `v3.2.3`
+  version commit (no Release object), a base-image dep bump to `202609281550` (#31854), and `v3.2.4`.
+  The v3.2.4 release body: "small patch that primarily fixes the memory leak people have observed
+  through a dependency update." **No breaking changes, no config additions, no operator action.**
+  The base-image bump (`202608300913`→`202609281550`, immich-app/base-images) adds a hardened
+  ImageMagick security policy (#390), reduces server image size (#388), and restores `curl` in the
+  prod image so `immich-healthcheck` works (#391) — all internal to the server image. `v3.3.0-rc.0/1`
+  are pre-releases (2026-09-29/30) — do NOT bump to rc. Done in the 2026-10-02 upgrade
+  (v3.2.2→v3.2.4, app+ML only, fresh PR). Direct registry check confirmed
+  `immich-server/ml:v3.2.4` exist; `abra recipe upgrade` still FATAs on the tag+digest DB pin.
+  Valkey CVE note: CVE-2026-56684 / CVE-2026-63639 (UAF→RCE) are fixed in 9.1.1 and are therefore
+  **already closed** by the recipe's pinned 9.1.1 — not by this app bump; ignore the live-`9`-tag
+  drift. CVE-2026-25243 (valkey RESTORE zipmap) is vendor-page-only and undecided.
 - **2026-08-07 INFRA note: cc-ci runner's gitea clone-token is STALE (HTTP 401).** `!testme` build
   #1210 died at the recipe `git clone` step (~5s, before any deploy) with `could not read Username for
   'https://git.autonomic.zone'` — the runner's mounted gitea token `13e299f2…` is rejected (verified

cc-ci-plan/upstream/n8n.md

@@ -239,3 +239,66 @@
   safe (sqlite, TypeORM auto-migrate). Operator flags unchanged (API-caller-level deprecations,
   encryption-key rework informational, N8N_DB_PING_TIMEOUT warn-only, recipe doesn't set it).
   2.40.3 exists (see above) — flagged, not taken. Recommended release: `-y`.
+- 2.40.4 (2026-09-21, Pre-release): core fix (tear down workflow triggers when publication meets a
+  node type the instance cannot load) + perf (stop loading project members when listing
+  credentials).
+- 2.40.5 (2026-09-21, Pre-release): core fix (limit declarative routing during base URL ownership
+  checks). Docker Hub `2.40.5` manifest verified active multi-arch (amd64+arm64, digest
+  sha256:9f693fd5..., 2026-09-21) — not a withdrawn 2.37.5-style partial tag. NOTE: 2.40.x
+  `POST /rest/login` now takes `emailOrLdapLoginId` instead of `email` (request-shape change; the
+  cc-ci login-state test only GETs /rest/login so unaffected).
+- 2026-09-21 run: the flagged catch-up. PR #8 (branch `upgrade-ef0dd56`, at 2.40.2, !testme GREEN
+  2026-09-18) extended **2.40.2 → 2.40.5** on the same upstream main tip `0b436ec` (n8n NOT
+  merged upstream today, unlike discourse/keycloak/matrix-synapse). 2.40.2→2.40.5 is bugfix-only
+  (encryption-key raw-repair, trigger teardown, declarative-routing limit); no breaking
+  compose/env changes; no new migrations beyond the 15 already verified at 2.40.2. Stable badge
+  now on the 2.39.x line (2.39.9/2.39.10 released 2026-09-21); 2.40.x pre-release per precedent.
+  Recommended release: `-y`.
+- 2.40.6 (2026-09-24, patch): core fixes (keep OpenTelemetry export working after a restart when
+  Sentry is enabled; retry instance reports that cross the UTC midnight boundary) + feature
+  (authenticate instance reports with the license certificate).
+- 2.40.7 (2026-09-25, patch): 1 core fix (propagate project span attributes to node spans).
+- 2.41.0 (2026-09-22, Pre-release; the 2.41 feature minor — note the 2.40.x patch line continued
+  alongside with 2.40.6/2.40.7): a large feature/bugfix release. Features: standalone node
+  execution as a Workflow Builder / AI Assistant tool, admin permission for node execution in the
+  assistant, API credential creation with source IDs, push-based reload endpoint for custom-node
+  development, execution view/delete permissions in project roles, MCP-registry capability
+  filtering, Databricks Embeddings/Chat-Model node, MS Teams @mentions + a Chat resource, MiniMax
+  dynamic model list, email-change confirmation flow. Core bugfixes include repair of
+  data-encryption keys stored as the raw instance key, stalled-job success finalization,
+  multi-main workflow-publishing fix, stop an unreachable external-secrets provider blocking
+  startup, undici 7.29.1 + vm2 3.12.2 bumps, RFC 9068 MCP OAuth compliance. **No breaking
+  compose/env/migration changes; no `N8N_*` env renames.** (2.41.1/2.41.2 = patch bugfixes.)
+- 2.41.3 (2026-09-25, patch): 1 core fix (propagate project span attributes to node spans). Last
+  week's PR #9 target — !testme GREEN (run 21, 2026-09-28); TypeORM migrations clean live.
+- 2.41.4 (2026-09-30, patch): API/core/editor fixes (return an execution when its stored trace
+  context is incomplete; count a database ping as successful when its reply arrives during
+  event-loop lag; store queue job results only for executions this process enqueued) + features
+  (n8n Assistant onboarding thread for new Cloud signups; route assistant credit CTAs to top-up).
+- 2.41.5 (2026-10-01, patch): atom-feed release body empty (release republish / no listed changes).
+- 2.42.0 (2026-09-29, Pre-release; the 2.42 feature minor): a large feature/bugfix release.
+  Features: AI Agent Tool can use its own tools under a pre-v3 parent agent, workflow descriptions
+  on create, `extendsCredential` in workflow create/update, OpenAPI path-parameter validation,
+  credential descriptions behind an instance flag, Anthropic prompt caching in the Message op,
+  API projects with source IDs, pinned Apply/Apply-Continue CLI commands, MCP-registry feature-flag
+  removal, Agent Builder/editor improvements. ~60 core/editor/node bugfixes (chat-integration
+  shutdown, Redis pubsub reconnect + half-open detection, OTel-after-restart-with-Sentry,
+  instance-report unique index, expression-isolate release, async delete retries). **No breaking
+  compose/env/migration changes; no `N8N_*` env renames.**
+- 2.42.1 (2026-09-30, patch): API/editor fixes (emit valid schemas for untyped values in the Public
+  API spec; return an execution when its trace context is incomplete) + editor feature (show the
+  n8n logo on canvas in canvas-only mode).
+- 2.42.2 (2026-10-01, patch; **newest numeric tag on Docker Hub — verified 2026-10-02**): 1 core
+  fix (stop waiting on Bull `job.finished()` for queued executions). Docker Hub `2.42.2` manifest
+  active; no 2.43.x exists.
+- 2026-10-02 run: reconcile confirmed coopcloud upstream main still `9141e23` (3.5.1+2.40.5) and
+  PR #9 (branch `upgrade-a144baf`, at 2.41.3, !testme GREEN 2026-09-28) still open/unmerged — so
+  per the one-evolving-PR contract the run extends PR #9 with a fast-forward commit
+  **2.40.5 → 2.42.2**. abra lists 2.42.2 as the newest candidate and a direct Docker Hub tag
+  enumeration agrees (no 2.43.x). The delta spans two feature minors (2.41.0, 2.42.0) plus patches
+  on both lines; **no breaking compose/env/migration changes, no `N8N_*` renames**. Rolling
+  upgrade safe (sqlite default; TypeORM migrations auto-run on boot). Operator flags remain
+  HTTP-API-caller-level only (2.33.0 workflow activate/deactivate deprecation; 2.36.0
+  `Array.merge`→`mergeIntoObject`; 2.37.0 JSON content-type on decorator body routes + binary-data
+  endpoint adapt; 2.39.0 workflow-version endpoint deprecation; 2.40.x `POST /rest/login`
+  `emailOrLdapLoginId` rename). Recommended release: `-y`.

nix/modules/orchestrator-host.nix

@@ -188,6 +188,94 @@ SSHCFG
       '';
     };
 
+    # ---- weekly opencode CLI auto-upgrade ----------------------------------------------------
+    # opencode is not in nixpkgs: opencode-install (above) only installs the standalone CLI when
+    # the binary is MISSING, so the installed version just ages in place (it sat on 1.18.29 for
+    # weeks while 1.18.33 was out). opencode's built-in autoupdate does not cover this host: it
+    # auto-applies patch releases only and only fires on a fresh interactive TUI start, and every
+    # agent here lives inside the long-lived `opencode serve` (opencode-web). This unit is the
+    # mechanism instead. It runs weekly in the Tuesday maintenance window (an hour BEFORE the
+    # host auto-update at Tue 03:00 UTC, and clear of the Thursday recipe-upgrade run and the
+    # Sunday canonical sweep), and:
+    #   1. compares the installed version with the latest GitHub release (no-op when equal);
+    #   2. reinstalls via the official installer (same code path as opencode-install) when they
+    #      differ, and re-links ~/.local/bin/opencode;
+    #   3. restarts opencode-web so the new binary actually takes effect — sessions attached to
+    #      it (orchestrator, upgrader, report) drop and their supervisors re-attach/resume, which
+    #      is why the busy gate below must hold: a mid-flight CI/upgrade run is never cut, and a
+    #      skipped run simply retries next week.
+    # Deploying this module never itself bumps opencode: boot/activation installs stay
+    # install-if-missing in opencode-install, and this unit is timer-driven only.
+    # The outcome of each run lands in .cc-ci-logs/opencode-update-state (read by /cc-ci-status).
+    systemd.services.opencode-upgrade = {
+      description = "Weekly opencode CLI auto-upgrade (latest release → reinstall → restart opencode-web)";
+      after = [ "network-online.target" "opencode-install.service" "opencode-web.service" ];
+      wants = [ "network-online.target" ];
+      serviceConfig = { Type = "oneshot"; TimeoutStartSec = "30min"; };
+      path = with pkgs; [ curl bash coreutils gnugrep gnutar gzip unzip systemd util-linux procps ];
+      environment = { HOME = "/home/loops"; OPENCODE_UI_HOST = cfg.opencodeUiHost; };
+      script = ''
+        set -u
+        BIN=/home/loops/.local/bin/opencode
+        STATE=/srv/cc-ci-orch/.cc-ci-logs/opencode-update-state
+        ocver() { "$BIN" --version 2>/dev/null | tail -1 || true; }
+        state() { printf '%s result=%s installed=%s note=%s\n' "$(date -u +%FT%TZ)" "$1" "$2" "$3" > "$STATE"; chown loops:users "$STATE" 2>/dev/null || true; }
+        busy() { echo "BUSY: $1 — skipping this week's opencode upgrade (retries next week)"; state skipped "$(ocver)" "$1"; exit 0; }
+
+        # Same busy gate as auto-update.nix: never cut a CI run, the weekly upgrade or the sweep.
+        pgrep -f run_recipe_ci >/dev/null && busy "a CI run is in flight"
+        systemctl is-active --quiet nightly-sweep.service && busy "the canonical sweep is running"
+        runuser -u loops -- tmux has-session -t cc-ci-upgrader 2>/dev/null && busy "the weekly recipe-upgrade run is in flight (tmux cc-ci-upgrader)"
+        runuser -u loops -- tmux has-session -t cc-ci-report 2>/dev/null && busy "the weekly report is being written (tmux cc-ci-report)"
+        if [ -r /run/secrets/bridge_drone_token ]; then
+          running=$(curl -s -m 20 -H "Authorization: Bearer $(cat /run/secrets/bridge_drone_token)" \
+            "https://drone.ci.commoninternet.net/api/repos/recipe-maintainers/cc-ci/builds?per_page=10" \
+            | grep -o '"status":"running"' | wc -l)
+          [ "''${running:-0}" -eq 0 ] || busy "$running Drone build(s) running"
+        fi
+
+        INSTALLED=$(ocver)
+        LATEST=$(curl -fsSL -m 30 https://api.github.com/repos/anomalyco/opencode/releases/latest \
+          | grep -Po '"tag_name":\s*"v?\K[0-9][0-9.]*' || true)
+        [ -n "$LATEST" ] || { echo "could not determine the latest opencode release"; state failed "''${INSTALLED:-none}" "latest-unresolved"; exit 1; }
+        echo "installed: ''${INSTALLED:-none}   latest: $LATEST"
+        if [ "$INSTALLED" = "$LATEST" ]; then
+          echo "opencode is up to date"
+          state ok "$INSTALLED" "up-to-date"
+          exit 0
+        fi
+
+        echo "upgrading opencode: ''${INSTALLED:-none} -> $LATEST"
+        runuser -u loops -- env HOME=/home/loops bash -c 'curl -fsSL https://opencode.ai/install | bash' \
+          || { echo "install failed — the previously installed version is untouched"; state failed "''${INSTALLED:-none}" "install-failed"; exit 1; }
+        mkdir -p /home/loops/.local/bin
+        ln -sfn /home/loops/.opencode/bin/opencode /home/loops/.local/bin/opencode
+        NEWVER=$(ocver)
+        [ "$NEWVER" = "$LATEST" ] || { echo "install ran but opencode reports $NEWVER (wanted $LATEST) — not restarting"; state failed "$NEWVER" "install-mismatch"; exit 1; }
+
+        echo "restarting opencode-web so the new binary takes effect (attached sessions drop; their supervisors resume)"
+        systemctl restart opencode-web.service
+        sleep 10
+        systemctl is-active --quiet opencode-web.service || { echo "opencode-web did not come back after the upgrade"; state failed "$NEWVER" "web-restart-failed"; exit 1; }
+        code=$(curl -s -m 20 -o /dev/null -w '%{http_code}' --resolve "$OPENCODE_UI_HOST:443:127.0.0.1" "https://$OPENCODE_UI_HOST/")
+        [ "$code" = "401" ] || { echo "opencode UI answered $code, not the 401 auth challenge"; state failed "$NEWVER" "ui-check-$code"; exit 1; }
+        echo "opencode upgraded to $NEWVER; opencode-web restarted and healthy"
+        state ok "$NEWVER" "upgraded; web restarted"
+      '';
+    };
+
+    systemd.timers.opencode-upgrade = {
+      wantedBy = [ "timers.target" ];
+      timerConfig = {
+        # Weekly slot in the Tuesday maintenance window, an hour BEFORE the host auto-update
+        # (cc-ci-auto-update.timer: Tue 03:00 UTC) so a restarted opencode-web has settled before
+        # that run's health check; Persistent=false, like the auto-update — no catch-up at boot.
+        OnCalendar = "Tue *-*-* 02:00:00 UTC";
+        Persistent = false;
+        RandomizedDelaySec = "10min";
+      };
+    };
+
     # ---- opencode web server: one shared instance the opencode-backed agents attach to -------
     # Provider creds come from /srv/cc-ci/.testenv (out of band, see README).
     systemd.services.opencode-web = {