Compare commits
9
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c472aa3642 | ||
|
|
7252f7ee35 | ||
|
|
75386fc23b | ||
|
|
19abebb312 | ||
|
|
565e021185 | ||
|
|
929faeb055 | ||
|
|
a29b993160 | ||
|
|
a0d6fc9417 | ||
|
|
ab5d11ea2a |
+62
-3
@@ -867,6 +867,65 @@ session cc-ci-orchestrator-stale can be killed; recipe-mirrors org still private
|
||||
(/srv/cc-ci-orch/cc-ci), and task-tool subagents inherit their parent session's directory. The
|
||||
config now lives in the cc-ci repo at that path. VERIFIED end-to-end with the launcher's exact
|
||||
invocation: parent=glm-5.2, subagent=deepseek-v4-pro read back from the session DB.
|
||||
LESSON: `opencode debug config` proves resolution, NOT binding — only a live subagent's recorded
|
||||
modelID proves binding. First attempt was a false pass because the probe passed --dir (unlike the
|
||||
real launcher) and landed in a different project.
|
||||
LESSON: `opencode debug config` proves resolution, NOT binding — only a live subagent's recorded
|
||||
modelID proves binding. First attempt was a false pass because the probe passed --dir (unlike the
|
||||
real launcher) and landed in a different project.
|
||||
|
||||
## Session 2026-08-15 19:25 UTC — opencode glm-5.2
|
||||
|
||||
**Left off:** Recovered the stalled 2026-08-14 weekly /upgrade-all run. Killed a supervisor that had
|
||||
been relaunching hourly for ~40h (balance exhausted), then started a FRESH scoped upgrader. Run is now
|
||||
progressing (surveying the 9 remaining recipes). Watching it through to completion.
|
||||
|
||||
**What happened (the stall):**
|
||||
- The 2026-08-14 /upgrade-all run (session ses_00200382fffeYIGl2sc3mO9JId) stalled at 03:18 Aug 14
|
||||
mid-`lasuite-drive` with `Error: Insufficient balance` (opencode zen workspace balance ran out). It
|
||||
had already done bluesky-pds, ghost, gitea, hedgedoc (PRs) + immich, lasuite-docs (SKIPPED up-to-date)
|
||||
alphabetically; lasuite-drive had a plan + partial PR #6 but no RESULT/verify.
|
||||
- The supervisor cron (glm-5.2, opencode-go tier) relaunched an hourly one-shot supervisor ~40 times
|
||||
to "drive it to completion", but each was also balance-walled (and later, just spinning). The run sat
|
||||
INCOMPLETE + not progressing for 40h. No weekly summary, no report published for week of Aug 14.
|
||||
|
||||
**What I did this session:**
|
||||
- Diagnosed: the opencode zen endpoint is NOW healthy (direct probe `say OK` → HTTP 200 in 1.35s —
|
||||
balance is restored). But resuming the ORIGINAL giant session is impossible: it's 2.58M tokens
|
||||
(267K input + 2.3M cache) and `opencode run -s … --continue` sits idle on `do_epoll_wait` with zero
|
||||
I/O — the inference endpoint silently drops the oversized request (matches the supervisor's
|
||||
`socket connection was closed unexpectedly` errors). A fresh small `opencode run` works fine. So the
|
||||
giant session is unresumable; a fresh start is the only path.
|
||||
- Killed the stuck supervisor (tmux `cc-ci-supervisor`, proc 377329).
|
||||
- `UPGRADER_ARGS="lasuite-drive lasuite-meet mailu matrix-synapse mattermost-lts mumble n8n plausible
|
||||
wordpress --sequential" python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py fresh` — this killed the
|
||||
stuck resume, archived the old giant session (`archive-cc-ci-upgrader — 2026-08-14`), reclaimed 10GB
|
||||
stale images on cc-ci (disk 29%), and started a FRESH small session
|
||||
`ses_ff920cf39ffeoogwXHTajp94cr` (zen/glm-5.2) scoped to the 9 recipes not yet done this week
|
||||
(positions 13-21 alphabetically; positions 1-12 were already surveyed — 6 PRs + 6 up-to-date). A
|
||||
fresh watchdog is watching the new session. The skill is idempotent (reuses existing PRs incl.
|
||||
lasuite-drive #6, never duplicates), so scoping is safe.
|
||||
- Confirmed the fresh run is progressing: pane shows it surveying the 9 recipes (verified all present
|
||||
in abra + all `weekly` tier; currently probing plausible/wordpress tags). Proc alive, log growing.
|
||||
|
||||
**Phase / loop state:**
|
||||
- Build/adversary loops: STOPPED (whole sequence completed 2026-08-01; phase ghost DONE).
|
||||
- Weekly upgrader: RUNNING (fresh session ses_ff920cf39, scoped 9 recipes, --sequential, watchdog up).
|
||||
- cc-ci server: healthy (disk 29%, runner active).
|
||||
|
||||
**Open items for next session:**
|
||||
- **Monitor the fresh upgrader to completion.** It will survey the 9 recipes, /recipe-upgrade the
|
||||
upgradeable ones (subagents, !testme verify, open/extend PRs — NEVER merge), write the weekly summary
|
||||
to `/srv/cc-ci/.cc-ci-logs/upgrades/`, then `launch-report.py fresh` (the upgrade-all skill does this
|
||||
itself per SKILL.md §5), print `UPGRADE RUN COMPLETE`, and go idle. If it stalls on a usage limit,
|
||||
the watchdog auto-resumes the SAME (small) session — that works now.
|
||||
- **Do NOT try to resume the archived giant session ses_00200382** — it's unresumable (endpoint drops
|
||||
the 2.58M-token request). It's archived; leave it.
|
||||
- After the run completes + report publishes, operator review queue = this week's recipe PRs.
|
||||
- The supervisor cron (hourly at XX:07) should now leave the run alone once it's progressing; if a
|
||||
supervisor fires while the run is mid-flight, its guardrails say to hand back to the resumed run, not
|
||||
double-write. No action needed unless it interferes.
|
||||
|
||||
**Notes:**
|
||||
- Root cause of the 40h silence was the same BUG 1 from 2026-08-10 (supervisor progress gate) partly:
|
||||
the supervisor kept firing because the run never reached "progressing". Now that balance is restored
|
||||
and a fresh small session is running, the gate should see progress and stand down.
|
||||
- Lesson: when a weekly run dies mid-flight on a giant context, do NOT resume the original session —
|
||||
start fresh and scope to the remaining recipes. The /upgrade-all skill is idempotent so this is safe.
|
||||
|
||||
@@ -76,6 +76,16 @@
|
||||
The previous inline-label approach (no restore hook) was a defect: raw PGDATA restore without a reload
|
||||
was a silent no-op. Fixed in PR #2 (restore fix cherry-picked from PR #1 ci/pg-restore).
|
||||
|
||||
- **2026-08-21 re-check** (endoflife.date/api/mattermost.json 2026-08-17; Mattermost docs; GitHub):
|
||||
**11.7 ESR is the only current ESR/LTS** — latest patch **11.7.9** (2026-08-13, "Low to High
|
||||
security fixes"), EOL 2027-05-15. **10.11 ESR ENDED 2026-08-15** — upstream main still pins
|
||||
10.11.22 (now on an EXPIRED ESR). 11.10/11.9/11.8 are innovation (EOL 2026-11-15/10-15/09-15),
|
||||
NOT LTS. The `-lts` recipe must track 11.7. This run **redirects open PR #2 away from the
|
||||
innovation 11.10.0 onto 11.7.9** (the correct LTS/ESR; ESR→ESR 10.11→11.7 is "fully supported
|
||||
and tested" per Mattermost docs — no staging hop). Refer to the 2026-08-15 report where PR #2
|
||||
was on 11.10.0 (operator chose innovation line that week); operator should now decide the
|
||||
`-lts` line. postgres 15-alpine still HELD (DB-major out of scope).
|
||||
|
||||
## NVD CPE fallback
|
||||
This project publishes nothing machine-readable we can reach — no GitHub advisory feed,
|
||||
no release-attributable changelog — so its CVE count was `?` (nothing measured). NVD is
|
||||
|
||||
@@ -65,3 +65,17 @@
|
||||
- 2.35.3 (2026-08-14, Pre-release): bugfixes (Google Ads v21→v25 API migration, MS Teams OAuth scope
|
||||
restore, workflow publication outbox abort deadline) + feature (skip update approval for workflows
|
||||
from same Instance AI session). Not deployed (2.35.2 was the survey target).
|
||||
- 2.36.0 (2026-08-18, Pre-release; 2.36.1 was briefly tagged then withdrawn — 2.36.2 compares
|
||||
2.36.0...2.36.2): a large feature minor. Core/ai-builder/editor bugfixes + new features (Agent
|
||||
sandboxes + writable workspaces, MCP folder/move/read-data-table tools, Schedule Trigger missed-
|
||||
execution grace period + catch-up coalescing, Confluence Cloud OAuth2 + page get sub-tree, MiniMax
|
||||
M3/H3, LangSmith session debug export, role-mapping/roles API endpoints, n8n User Auth GA, etc.).
|
||||
No breaking compose/env/migration changes, no `N8N_*` env renames; the deprecation notices
|
||||
(Array.merge→Array.mergeIntoObject in expressions, workflow tags API migration) are API-level only.
|
||||
2.36.2 (2026-08-19, patch): 3 fixes (decorator-route request schema in /discover, end-user
|
||||
credential resolution in node parameters, editor agent-preview alignment). 2.36.3 (2026-08-20,
|
||||
patch): 4 core/editor fixes (task-runners only restart when slow, expression engine init on
|
||||
expression commands, test-webhook isolate release after teardown, Edit Fields type selector styles).
|
||||
- 2026-08-21 run: planning 2.34.4 → 2.36.3 (newest tag abra lists). Whole 2.36.x line still
|
||||
Pre-release on GitHub (2.33.5/Latest was superseded by 2.35.x line). Rolling upgrade safe, sqlite
|
||||
mode, TypeORM migrations auto-run on boot. No operator action required.
|
||||
|
||||
@@ -4,6 +4,7 @@
|
||||
|---------|-------|-------------|----------------------|
|
||||
| app | wordpress | https://github.com/WordPress/WordPress | https://wordpress.org/news/category/releases/ (image: https://hub.docker.com/_/wordpress) |
|
||||
| db | mariadb | https://github.com/MariaDB/server | https://mariadb.com/kb/en/release-notes/ (image: https://hub.docker.com/_/mariadb) |
|
||||
| ftp (optional overlay) | atmoz/sftp | https://github.com/atmoz/sftp | https://github.com/atmoz/sftp/releases (image: https://hub.docker.com/r/atmoz/sftp) |
|
||||
|
||||
## Standing notes
|
||||
- Enrolled 2026-08-03 (operator request). Recipe = coopcloud `wordpress`; mirror
|
||||
@@ -22,3 +23,7 @@
|
||||
image). mariadb major bumps (e.g. 12.x→13.x) need the usual dump/restore caution if the
|
||||
recipe ever pins a non-`mariadb`-auto-upgrading setup; the official image handles minor
|
||||
bumps in place.
|
||||
- `atmoz/sftp` is pinned as the floating `alpine` tag (updated 2026-07-28, current as of
|
||||
2026-08-21) — not a semver abra can evaluate, so it never contributes an upgrade candidate;
|
||||
verified current by direct Docker Hub tag check. Only bump if a real newer supported tag
|
||||
exists and the app needs it.
|
||||
|
||||
@@ -31,7 +31,7 @@ matrix-synapse weekly
|
||||
mattermost-lts weekly
|
||||
mumble weekly
|
||||
n8n weekly
|
||||
plausible weekly
|
||||
plausible external # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded
|
||||
uptime-kuma external # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded
|
||||
wordpress weekly
|
||||
```
|
||||
|
||||
@@ -18,6 +18,11 @@
|
||||
system = "x86_64-linux";
|
||||
in
|
||||
{
|
||||
# The cc-ci part of a host, on its own, so a host that runs cc-ci can import just this and
|
||||
# keep its own (unrelated) configuration separate. Split out 2026-08-20; consumed by
|
||||
# notplants-nix's `notplants-orchestrator` host.
|
||||
nixosModules.cc-ci = ./nix/modules/cc-ci.nix;
|
||||
|
||||
# Hetzner cpx11 host (nixos-infect generated hardware.nix + orchestrator config).
|
||||
# Provision with terraform/ then run Stage 2 per terraform/README.md.
|
||||
nixosConfigurations.cc-ci-orchestrator-hetzner = nixpkgs.lib.nixosSystem {
|
||||
|
||||
@@ -14,7 +14,11 @@
|
||||
# with ACME, and opens 80/443 (previously only 22 was public). Canonical source of
|
||||
# the module is the project repo; ../atproto-likes.nix is a copy kept in this tree
|
||||
# because pure evaluation cannot import a path outside the flake.
|
||||
imports = [ ../../atproto-likes.nix ];
|
||||
# cc-ci.nix holds everything that exists FOR cc-ci (the loops, the cc-ci orchestrator session,
|
||||
# the weekly recipe-upgrade run and its hourly supervisor). Split out 2026-08-20 so this host —
|
||||
# a general agent box that also serves unrelated projects — can be reasoned about separately
|
||||
# from cc-ci. Also exported as `nixosModules.cc-ci` from this repo's flake for other hosts.
|
||||
imports = [ ../../atproto-likes.nix ../../modules/cc-ci.nix ];
|
||||
|
||||
services.openssh = {
|
||||
enable = true;
|
||||
@@ -225,72 +229,7 @@ SSHCFG
|
||||
};
|
||||
};
|
||||
|
||||
# cc-ci-loops supervisor — workspace staged 2026-05-31, so ENABLED for reboot-resilience.
|
||||
systemd.services.cc-ci-loops = {
|
||||
description = "cc-ci Builder/Adversary loops + watchdog (launch.sh start)";
|
||||
wantedBy = [ "multi-user.target" ]; # enabled after workspace staged (Hetzner cutover)
|
||||
after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
# KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the
|
||||
# default (control-group) systemd kills every leftover process in the cgroup when the unit
|
||||
# stops — and since one tmux server hosts every agent session on this host, a rebuild that
|
||||
# merely touched this unit wiped all of them (operator 2026-08-01). Only the (already
|
||||
# exited) main process is killed now; `systemctl stop` therefore does NOT tear down agents.
|
||||
KillMode = "process";
|
||||
Type = "oneshot"; RemainAfterExit = true;
|
||||
User = "loops"; Group = "users";
|
||||
WorkingDirectory = "/srv/cc-ci/cc-ci";
|
||||
# Append one line to REBOOTS.md per genuine reboot (boot_id-gated; not on manual restart).
|
||||
ExecStartPre = "${pkgs.bash}/bin/bash /srv/cc-ci/cc-ci-plan/reboot-log.sh";
|
||||
};
|
||||
# CLAUDE_BIN points at the standalone CLI installed by claude-install.service; the loops
|
||||
# backend defaults to claude (persisted in .loop-backend). Without this, launch.py's preflight
|
||||
# `which(claude)` fails because the systemd `path` below has no /home/loops/.local/bin.
|
||||
environment = { RESUME_PHASE = "1"; HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
|
||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||
script = ''
|
||||
# Put the standalone claude/opencode binaries on PATH. On a cold boot this is the env the
|
||||
# tmux server (and thus every agent session) inherits, so bare `claude` resolves everywhere.
|
||||
export PATH="/home/loops/.local/bin:$PATH"
|
||||
[ -x /srv/cc-ci/cc-ci-plan/launch.sh ] && /srv/cc-ci/cc-ci-plan/launch.sh start || \
|
||||
echo "workspace not staged yet — skipping loop start"
|
||||
'';
|
||||
};
|
||||
|
||||
# cc-ci-orchestrator supervisor — the operator's steering session. Same shape as
|
||||
# lichen-orchestrator / project-orchestrator above: this unit only LAUNCHES the orchestrator's
|
||||
# tmux session via the agent-orchestrator harness (cc-ci-plan/agents.py); it does not own the
|
||||
# session or the tmux server. The orchestrator agent is declared in cc-ci-plan/agents.toml on
|
||||
# the OPencode backend (backend = "opencode", model = "opencode/glm-5.2"), so on boot it
|
||||
# attaches to the shared opencode web server (opencode-web.service below) and is reachable for
|
||||
# Remote Control at https://oc.commoninternet.net under the /srv/cc-ci-orch project. The harness
|
||||
# watchdog (started by `agents.py up`) keeps it alive: heal-only (no stall reboots — a persistent
|
||||
# supervisor must not be killed just for idling). Added 2026-08-03 to give the cc-ci orchestrator
|
||||
# the same reboot-resilience the other two orchestrators already have.
|
||||
systemd.services.cc-ci-orchestrator = {
|
||||
description = "cc-ci orchestrator (operator steering session) — agents.py up orchestrator, opencode backend";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network-online.target" "tailscaled.service" "opencode-web.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
# KillMode=process: see the note on cc-ci-loops — a rebuild that merely touches this unit
|
||||
# must not tear down the (shared) tmux server and every agent session with it.
|
||||
KillMode = "process";
|
||||
Type = "oneshot"; RemainAfterExit = true;
|
||||
User = "loops"; Group = "users";
|
||||
WorkingDirectory = "/srv/cc-ci-orch";
|
||||
};
|
||||
environment = { HOME = "/home/loops"; };
|
||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||
script = ''
|
||||
export PATH="/home/loops/.local/bin:$PATH"
|
||||
proj="/srv/cc-ci-orch"
|
||||
echo "$(cat /proc/sys/kernel/random/boot_id) boot $(date -u +%FT%TZ) — cc-ci-orchestrator up" \
|
||||
>> "$proj/cc-ci-plan/.ao-boot.log" 2>/dev/null || true
|
||||
cd "$proj" && python3 cc-ci-plan/agents.py up orchestrator || echo "cc-ci orchestrator agents.py up failed"
|
||||
'';
|
||||
};
|
||||
|
||||
# p-lichen-orchestrator supervisor — the SEPARATE lichen.page testing/hardening orchestrator
|
||||
# (distinct from cc-ci-loops above). Reboot-resilience: on boot, resume the orchestrator's Remote
|
||||
@@ -356,76 +295,7 @@ SSHCFG
|
||||
'';
|
||||
};
|
||||
|
||||
# Weekly recipe upgrade — runs /upgrade-all over every enrolled recipe (opens recipe PRs
|
||||
# verified by !testme, never merges). Replaces the boot-fragile busybox-crond-in-tmux from
|
||||
# phase 5 §4 with a reboot-safe systemd timer. The service is timer-triggered only (NOT
|
||||
# wantedBy multi-user.target) so it never runs on boot/activation — only on the schedule.
|
||||
systemd.services.cc-ci-upgrade-all = {
|
||||
description = "cc-ci weekly /upgrade-all run (recipe upgrade survey + PRs, never merges)";
|
||||
after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
Type = "oneshot"; # launch-upgrader.py spawns the cc-ci-upgrader tmux session and returns
|
||||
User = "loops"; Group = "users";
|
||||
WorkingDirectory = "/srv/cc-ci";
|
||||
# Optional per-run overrides for backend/model (LOOP_BACKEND, LOOP_MODEL, OPENCODE_SHARE,
|
||||
# UPGRADER_ARGS, …). The leading "-" makes it optional: absent file → claude/sonnet defaults
|
||||
# (current behavior). To run the weekly job on e.g. opencode-go/glm-5.2, drop a file with
|
||||
# LOOP_BACKEND=opencode
|
||||
# LOOP_MODEL=opencode-go/glm-5.2
|
||||
# No rebuild needed to switch — the env file is read at each timer fire. Holds no secrets
|
||||
# (the opencode-go API key lives in ~/.local/share/opencode/auth.json, mode 600).
|
||||
EnvironmentFile = "-/srv/cc-ci/upgrader.env";
|
||||
};
|
||||
environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
|
||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||
script = ''
|
||||
export PATH="/home/loops/.local/bin:$PATH"
|
||||
python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py start >> /srv/cc-ci/.cc-ci-logs/upgrader-cron.log 2>&1
|
||||
'';
|
||||
};
|
||||
|
||||
systemd.timers.cc-ci-upgrade-all = {
|
||||
description = "Weekly trigger for cc-ci-upgrade-all (Thursdays 22:00 America/New_York — Boston 10pm)";
|
||||
wantedBy = [ "timers.target" ];
|
||||
timerConfig = {
|
||||
# 10pm Thursday Boston time — DST-aware (EDT→02:00 UTC, EST→03:00 UTC) via the tz in OnCalendar.
|
||||
OnCalendar = "Thu *-*-* 22:00:00 America/New_York";
|
||||
Persistent = true; # if the box was down at the scheduled time, run once on next boot
|
||||
};
|
||||
};
|
||||
|
||||
# Hourly SUPERVISOR — a glm-5.2 orchestrator wake-up that keeps the weekly run on track. The
|
||||
# log-idle/429 watchdog only handles opencode-go usage-limit stalls; it does NOT cover a host
|
||||
# disk-full crash (which killed the 2026-07-03 run) or any other environmental wedge. This is a
|
||||
# CHEAP deterministic gate: if the weekly run is complete or actively progressing it does NOTHING
|
||||
# (zero model tokens). Only when a run has stalled/died before completing does it launch a
|
||||
# short-lived glm-5.2 agent that diagnoses the blockage and drives the run to a clean DONE.
|
||||
systemd.services.cc-ci-upgrade-supervisor = {
|
||||
description = "cc-ci hourly weekly-run supervisor (glm-5.2 — drives a stalled /upgrade-all to completion)";
|
||||
after = [ "network-online.target" "tailscaled.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
Type = "oneshot"; # launch-supervisor.py check: gate now, spawn the agent into tmux, return
|
||||
User = "loops"; Group = "users";
|
||||
WorkingDirectory = "/srv/cc-ci";
|
||||
# Shares the weekly run's optional override file (e.g. SUPERVISOR_MODEL=…); "-" = optional.
|
||||
EnvironmentFile = "-/srv/cc-ci/upgrader.env";
|
||||
};
|
||||
environment = { HOME = "/home/loops"; };
|
||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||
script = ''
|
||||
export PATH="/home/loops/.local/bin:$PATH"
|
||||
python3 /srv/cc-ci/cc-ci-plan/launch-supervisor.py check >> /srv/cc-ci/.cc-ci-logs/supervisor-cron.log 2>&1
|
||||
'';
|
||||
};
|
||||
|
||||
systemd.timers.cc-ci-upgrade-supervisor = {
|
||||
description = "Hourly trigger for cc-ci-upgrade-supervisor (weekly-run health check + drive)";
|
||||
wantedBy = [ "timers.target" ];
|
||||
timerConfig = {
|
||||
OnCalendar = "*-*-* *:07:00"; # every hour at :07 (offset from the weekly :00 fire)
|
||||
Persistent = false; # a missed hourly check is moot — the next hour re-checks
|
||||
};
|
||||
};
|
||||
}
|
||||
|
||||
@@ -0,0 +1,157 @@
|
||||
# cc-ci.nix — everything on this host that exists FOR cc-ci, and nothing else.
|
||||
#
|
||||
# Split out of the orchestrator host config on 2026-08-20. The host it runs on is a general
|
||||
# agent/orchestration box that also serves several unrelated projects; this module is the cc-ci
|
||||
# part of it, so that the two can evolve (and be reviewed) independently. It is exported from this
|
||||
# repo's flake as `nixosModules.cc-ci` and imported by whichever host runs cc-ci.
|
||||
#
|
||||
# All of it assumes the cc-ci workspaces exist on the host:
|
||||
# /srv/cc-ci the loops workspace (+ .cc-ci-logs, upgrader.env)
|
||||
# /srv/cc-ci-orch this repo (the orchestrator's own working dir)
|
||||
# and that a `loops` user, tmux, python3 and the standalone claude/opencode CLIs are present —
|
||||
# those are host concerns, provided by the host config, not by this module.
|
||||
{ config, pkgs, lib, ... }:
|
||||
{
|
||||
# cc-ci-loops supervisor — workspace staged 2026-05-31, so ENABLED for reboot-resilience.
|
||||
systemd.services.cc-ci-loops = {
|
||||
description = "cc-ci Builder/Adversary loops + watchdog (launch.sh start)";
|
||||
wantedBy = [ "multi-user.target" ]; # enabled after workspace staged (Hetzner cutover)
|
||||
after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
# KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the
|
||||
# default (control-group) systemd kills every leftover process in the cgroup when the unit
|
||||
# stops — and since one tmux server hosts every agent session on this host, a rebuild that
|
||||
# merely touched this unit wiped all of them (operator 2026-08-01). Only the (already
|
||||
# exited) main process is killed now; `systemctl stop` therefore does NOT tear down agents.
|
||||
KillMode = "process";
|
||||
Type = "oneshot"; RemainAfterExit = true;
|
||||
User = "loops"; Group = "users";
|
||||
WorkingDirectory = "/srv/cc-ci/cc-ci";
|
||||
# Append one line to REBOOTS.md per genuine reboot (boot_id-gated; not on manual restart).
|
||||
ExecStartPre = "${pkgs.bash}/bin/bash /srv/cc-ci/cc-ci-plan/reboot-log.sh";
|
||||
};
|
||||
# CLAUDE_BIN points at the standalone CLI installed by claude-install.service; the loops
|
||||
# backend defaults to claude (persisted in .loop-backend). Without this, launch.py's preflight
|
||||
# `which(claude)` fails because the systemd `path` below has no /home/loops/.local/bin.
|
||||
environment = { RESUME_PHASE = "1"; HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
|
||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||
script = ''
|
||||
# Put the standalone claude/opencode binaries on PATH. On a cold boot this is the env the
|
||||
# tmux server (and thus every agent session) inherits, so bare `claude` resolves everywhere.
|
||||
export PATH="/home/loops/.local/bin:$PATH"
|
||||
[ -x /srv/cc-ci/cc-ci-plan/launch.sh ] && /srv/cc-ci/cc-ci-plan/launch.sh start || \
|
||||
echo "workspace not staged yet — skipping loop start"
|
||||
'';
|
||||
};
|
||||
|
||||
# cc-ci-orchestrator supervisor — the operator's steering session. Same shape as
|
||||
# lichen-orchestrator / project-orchestrator above: this unit only LAUNCHES the orchestrator's
|
||||
# tmux session via the agent-orchestrator harness (cc-ci-plan/agents.py); it does not own the
|
||||
# session or the tmux server. The orchestrator agent is declared in cc-ci-plan/agents.toml on
|
||||
# the OPencode backend (backend = "opencode", model = "opencode/glm-5.2"), so on boot it
|
||||
# attaches to the shared opencode web server (opencode-web.service below) and is reachable for
|
||||
# Remote Control at https://oc.commoninternet.net under the /srv/cc-ci-orch project. The harness
|
||||
# watchdog (started by `agents.py up`) keeps it alive: heal-only (no stall reboots — a persistent
|
||||
# supervisor must not be killed just for idling). Added 2026-08-03 to give the cc-ci orchestrator
|
||||
# the same reboot-resilience the other two orchestrators already have.
|
||||
systemd.services.cc-ci-orchestrator = {
|
||||
description = "cc-ci orchestrator (operator steering session) — agents.py up orchestrator, opencode backend";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network-online.target" "tailscaled.service" "opencode-web.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
# KillMode=process: see the note on cc-ci-loops — a rebuild that merely touches this unit
|
||||
# must not tear down the (shared) tmux server and every agent session with it.
|
||||
KillMode = "process";
|
||||
Type = "oneshot"; RemainAfterExit = true;
|
||||
User = "loops"; Group = "users";
|
||||
WorkingDirectory = "/srv/cc-ci-orch";
|
||||
};
|
||||
environment = { HOME = "/home/loops"; };
|
||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||
script = ''
|
||||
export PATH="/home/loops/.local/bin:$PATH"
|
||||
proj="/srv/cc-ci-orch"
|
||||
echo "$(cat /proc/sys/kernel/random/boot_id) boot $(date -u +%FT%TZ) — cc-ci-orchestrator up" \
|
||||
>> "$proj/cc-ci-plan/.ao-boot.log" 2>/dev/null || true
|
||||
cd "$proj" && python3 cc-ci-plan/agents.py up orchestrator || echo "cc-ci orchestrator agents.py up failed"
|
||||
'';
|
||||
};
|
||||
|
||||
# Weekly recipe upgrade — runs /upgrade-all over every enrolled recipe (opens recipe PRs
|
||||
# verified by !testme, never merges). Replaces the boot-fragile busybox-crond-in-tmux from
|
||||
# phase 5 §4 with a reboot-safe systemd timer. The service is timer-triggered only (NOT
|
||||
# wantedBy multi-user.target) so it never runs on boot/activation — only on the schedule.
|
||||
systemd.services.cc-ci-upgrade-all = {
|
||||
description = "cc-ci weekly /upgrade-all run (recipe upgrade survey + PRs, never merges)";
|
||||
after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
Type = "oneshot"; # launch-upgrader.py spawns the cc-ci-upgrader tmux session and returns
|
||||
User = "loops"; Group = "users";
|
||||
WorkingDirectory = "/srv/cc-ci";
|
||||
# Optional per-run overrides for backend/model (LOOP_BACKEND, LOOP_MODEL, OPENCODE_SHARE,
|
||||
# UPGRADER_ARGS, …). The leading "-" makes it optional: absent file → claude/sonnet defaults.
|
||||
# Current config (as of 2026-08-16): the upgrader + report run on tinfoil/deepseek-v4-pro
|
||||
# (LOOP_MODEL + REPORT_MODEL in the env file); the hourly SUPERVISOR stays on glm-5.2
|
||||
# (SUPERVISOR_MODEL defaults to opencode-go/glm-5.2 in launch-supervisor.py, NOT overridden
|
||||
# here). Subagents bind deepseek via the cc-ci repo's opencode config. LOOP_TIER=zen is kept
|
||||
# so the tier check passes; the watchdog's usage-limit probe sends the deepseek model name to
|
||||
# the zen endpoint, which returns 200 (not 429) → resume immediately (correct: tinfoil has no
|
||||
# rolling usage limit to wait out). No rebuild needed to switch — the env file is read at each
|
||||
# timer fire. Holds no secrets (the tinfoil API key lives in the opencode config / auth.json).
|
||||
EnvironmentFile = "-/srv/cc-ci/upgrader.env";
|
||||
};
|
||||
environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
|
||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||
script = ''
|
||||
export PATH="/home/loops/.local/bin:$PATH"
|
||||
python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py start >> /srv/cc-ci/.cc-ci-logs/upgrader-cron.log 2>&1
|
||||
'';
|
||||
};
|
||||
|
||||
systemd.timers.cc-ci-upgrade-all = {
|
||||
description = "Weekly trigger for cc-ci-upgrade-all (Thursdays 22:00 America/New_York — Boston 10pm)";
|
||||
wantedBy = [ "timers.target" ];
|
||||
timerConfig = {
|
||||
# 10pm Thursday Boston time — DST-aware (EDT→02:00 UTC, EST→03:00 UTC) via the tz in OnCalendar.
|
||||
OnCalendar = "Thu *-*-* 22:00:00 America/New_York";
|
||||
Persistent = true; # if the box was down at the scheduled time, run once on next boot
|
||||
};
|
||||
};
|
||||
|
||||
# Hourly SUPERVISOR — a glm-5.2 orchestrator wake-up that keeps the weekly run on track. The
|
||||
# log-idle/429 watchdog only handles opencode-go usage-limit stalls; it does NOT cover a host
|
||||
# disk-full crash (which killed the 2026-07-03 run) or any other environmental wedge. This is a
|
||||
# CHEAP deterministic gate: if the weekly run is complete or actively progressing it does NOTHING
|
||||
# (zero model tokens). Only when a run has stalled/died before completing does it launch a
|
||||
# short-lived glm-5.2 agent that diagnoses the blockage and drives the run to a clean DONE.
|
||||
systemd.services.cc-ci-upgrade-supervisor = {
|
||||
description = "cc-ci hourly weekly-run supervisor (glm-5.2 — drives a stalled /upgrade-all to completion)";
|
||||
after = [ "network-online.target" "tailscaled.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
Type = "oneshot"; # launch-supervisor.py check: gate now, spawn the agent into tmux, return
|
||||
User = "loops"; Group = "users";
|
||||
WorkingDirectory = "/srv/cc-ci";
|
||||
# Shares the weekly run's optional override file (e.g. SUPERVISOR_MODEL=…); "-" = optional.
|
||||
EnvironmentFile = "-/srv/cc-ci/upgrader.env";
|
||||
};
|
||||
environment = { HOME = "/home/loops"; };
|
||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||
script = ''
|
||||
export PATH="/home/loops/.local/bin:$PATH"
|
||||
python3 /srv/cc-ci/cc-ci-plan/launch-supervisor.py check >> /srv/cc-ci/.cc-ci-logs/supervisor-cron.log 2>&1
|
||||
'';
|
||||
};
|
||||
|
||||
systemd.timers.cc-ci-upgrade-supervisor = {
|
||||
description = "Hourly trigger for cc-ci-upgrade-supervisor (weekly-run health check + drive)";
|
||||
wantedBy = [ "timers.target" ];
|
||||
timerConfig = {
|
||||
OnCalendar = "*-*-* *:07:00"; # every hour at :07 (offset from the weekly :00 fire)
|
||||
Persistent = false; # a missed hourly check is moot — the next hour re-checks
|
||||
};
|
||||
};
|
||||
}
|
||||
Reference in New Issue
Block a user