Compare commits

..
Author SHA1 Message Date
autonomic-bot 82d3127c2a docs: record weekly upgrade report 2026-08-31 20:46:55 +00:00
autonomic-bot d824800f8a upstream(n8n): 2026-08-31 release-notes (2.34.5/2.34.6, 2.35.4-7, 2.36.8/2.36.9, 2.37.4, 2.37.5 withdrawn, 2.37.6) 2026-08-31 20:31:59 +00:00
autonomic-bot e7f861ec9e upstream(mattermost-lts): 2026-08-31 re-check note (ESR still 11.7.10, PR #2 re-verify) 2026-08-31 20:20:09 +00:00
autonomic-bot e72fbdb79c fix: default testme Gitea URL 2026-08-31 19:29:31 +00:00
autonomic-bot f6c6b39b18 docs: record acme-dns production issuance 2026-08-31 19:04:54 +00:00
autonomic-bot 02ff90777f docs: record acme-dns production issuance 2026-08-31 19:04:41 +00:00
autonomic-bot 012212498d docs: record acme-dns handoff 2026-08-31 17:17:09 +00:00
autonomic-bot 521f6e6934 docs: record acme-dns handoff 2026-08-31 17:16:51 +00:00
autonomic-bot 07bc0a15d5 Merge pull request 'config: load public cc-ci runtime environment' (#14) from chore/public-runtime-config into main 2026-08-31 16:50:25 +00:00
autonomic-bot 00a4ee6752 config: load public cc-ci runtime environment 2026-08-31 16:49:15 +00:00
autonomic-bot 9c619abd6c upstream(n8n): release-notes sources 2.36.4..2.37.3 2026-08-28 04:15:13 +00:00
autonomic-bot 980530f505 upstream(mattermost-lts): 2026-08-28 re-check — 11.7.10 newest ESR patch 2026-08-28 04:00:02 +00:00
autonomic-bot 3047ee7724 config: GITEA_URL is public config, not a credential
The Gitea host was stored in the credentials env file, so a literal
secret scanner treated it as a secret and refused two READMEs in another
repo that link to git.autonomic.zone.

It was never secret: advisory-scan.py, pr-survey.py and recipe-report.py
already default it to git.autonomic.zone, and it is this repo's own
remote host. It is a bare hostname with no scheme, userinfo or path.

Removed from the env file and defaulted here the same way the Python
already does. GITEA_USERNAME and GITEA_PASSWORD keep their :? guard --
those are real credentials and must still fail loudly if unset.
2026-08-26 17:46:58 +00:00
autonomic-bot c472aa3642 upstream(wordpress): release-notes sources (sftp/atmoz + standing note) 2026-08-21 03:45:57 +00:00
autonomic-bot 7252f7ee35 upstream(mattermost-lts): 2026-08-21 re-check — 11.7.9 ESR is the only current LTS; PR #2 redirected to ESR 2026-08-21 03:34:36 +00:00
autonomic-bot 75386fc23b upstream(n8n): 2.36.x release-notes coverage (2.36.0 feature minor, 2.36.2/2.36.3 patches) 2026-08-21 03:28:47 +00:00
autonomic-botandClaude Opus 4.8 19abebb312 nix: split the cc-ci-specific config into its own module + flake output
The orchestrator host is a general agent box that also serves several unrelated
projects (atproto-likes, the lichen and project orchestrators, the shared
opencode web server). Everything that exists FOR cc-ci — the loops, the cc-ci
orchestrator session, the weekly recipe-upgrade run and its hourly supervisor —
now lives in nix/modules/cc-ci.nix and is exported as `nixosModules.cc-ci`, so
the two can evolve and be reviewed independently and another host can consume
just the cc-ci half.

No behaviour change. Verified by building .#cc-ci-orchestrator-hetzner after the
split: the unit set is identical to the running system, and all six cc-ci units
are byte-identical modulo nix store hashes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016L6nYYwkCWnrEFKTnKAfet
2026-08-20 16:29:02 +00:00
autonomic-bot 565e021185 Merge pull request 'config: drop plausible from weekly upgrade set (-> external)' (#13) from config/drop-plausible-weekly into main 2026-08-17 16:01:38 +00:00
autonomic-bot 929faeb055 config: drop plausible from weekly upgrade set (-> external)
Move plausible from weekly to external in used-recipes.md. cc-ci still
deploys/tests it (the recipe + tests stay enrolled, !testme bridge
unchanged), but it is no longer surveyed/upgraded by the weekly /upgrade-all
run — maintained elsewhere, like uptime-kuma.

The open plausible PR #5 (clickhouse 23.4.2.11 -> 23.4.6.25) from this week's
run stays open for the operator; no future weekly PRs will be opened for it.
2026-08-17 16:01:26 +00:00
autonomic-bot a29b993160 Merge pull request 'config: switch upgrader + report to deepseek, keep supervisor on glm' (#12) from config/deepseek-upgrader-model into main 2026-08-16 02:29:20 +00:00
autonomic-bot ab5d11ea2a Merge pull request 'advisory-scan: NVD by CPE, so mattermost and mumble stop scanning as '?'' (#11) from feat/nvd-cpe-source into main 2026-08-11 22:17:13 +00:00
23 changed files with 399 additions and 169 deletions
@@ -18,7 +18,7 @@ RECIPE="${1:?usage: testme-on-pr.sh <recipe> <pr-index>}"
PRIDX="${2:?usage: testme-on-pr.sh <recipe> <pr-index>}"
TESTENV="${TESTENV:-/srv/cc-ci/.testenv}"
set -a; . "$TESTENV"; set +a
: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:?}"
: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:=git.autonomic.zone}"
NS="${GITEA_NAMESPACE:-recipe-maintainers}"
API="https://${GITEA_URL}/api/v1"; AUTH=(-u "${GITEA_USERNAME}:${GITEA_PASSWORD}")
POST="${POST:-1}"; MAX_WAIT="${MAX_WAIT:-480}"; INTERVAL="${INTERVAL:-30}"
+64
View File
@@ -929,3 +929,67 @@ progressing (surveying the 9 remaining recipes). Watching it through to completi
and a fresh small session is running, the gate should see progress and stand down.
- Lesson: when a weekly run dies mid-flight on a giant context, do NOT resume the original session —
start fresh and scope to the remaining recipes. The /upgrade-all skill is idempotent so this is safe.
## Session 2026-08-31 17:17 UTC — restricted acme-dns renewal staged
**What happened:** Implemented and deployed the cc-ci restricted ACME DNS-01 design in cc-ci PRs
#24#27 (all merged). The initial `sqlite3` backend setting was detected during staging bootstrap
and corrected to acme-dns 2.x `sqlite` before any account registration succeeded.
**Live state:**
- `acme-dns` is authoritative for `acme.commoninternet.net` on `91.98.47.73`, with public UDP/TCP
53 verified directly; it serves the apex NS and `ns-acme.commoninternet.net A 91.98.47.73`.
- Its HTTP update API is bound solely to `127.0.0.1:8080`; registration is disabled (local POST
returns 403). The one restricted Lego account is stored mode `0600` under `/var/lib/acme/`, outside
Git and the Nix store. No Gandi credentials were used or stored.
- Nix ACME is in Let's Encrypt staging mode for `ci.commoninternet.net` and its wildcard; its renewal
timer is enabled. Traefik handoff remains production-gated so staging cannot replace the live cert.
- Every server deployment used `nixos-rebuild test`, reachability/service verification, then
`nixos-rebuild switch` on `#cc-ci-hetzner`.
**Manual Gandi handoff (only blocker):** Add the parent-zone A+NS delegation and permanent CNAME:
`ns-acme.commoninternet.net. A 91.98.47.73`,
`acme.commoninternet.net. NS ns-acme.commoninternet.net.`, and
`_acme-challenge.ci.commoninternet.net. CNAME
9d3f0428-eb97-4b14-bb07-da2f0f6c2073.acme.commoninternet.net.` Once propagated, resume with a
staging issuance, then production cutover/handoff verification. Do not obtain or add a Gandi token.
## Session 2026-08-31 19:05 UTC — ACME production issuance complete
**Completed:** The operator added the narrow Gandi delegation and permanent CNAME. Staging then
validated both `ci.commoninternet.net` and `*.ci.commoninternet.net`; production issuance validated
the same SAN pair and produced a Let's Encrypt certificate valid through 2026-11-29. cc-ci PRs #28#30
were merged to promote production, make Traefik roll certificate-secret changes even when recipe
upgrades are held, and remove the retired SOPS wildcard-file injection.
**Live cc-ci state:**
- `/var/lib/acme/ci.commoninternet.net` is the sole source of the production cert/key; the runtime
pair matches it, is root-only for the key, and Traefik now references matching versioned Swarm
secrets. Reboot-safe `nixos-rebuild switch` completed at cc-ci `7147d77` after the required test
activations and health checks.
- acme-dns remains authoritative only for the delegated subzone, its update API is localhost-only,
registration is disabled, and the restricted-account JSON is mode `0600`. The Nix ACME renewal
timer is active. No Gandi credential was added.
**External blocker, not cc-ci-owned:** Direct TLS to `91.98.47.73:443` now presents the new valid
certificate. The operator-managed public gateway at `143.244.213.108` currently accepts TCP then
closes TLS before presenting a certificate, so its passthrough to cc-ci needs repair/verification.
The cc-ci plan explicitly marks the gateway as operator infrastructure; do not change the DNS
delegation or add a Gandi token to work around it.
## Session 2026-08-31 20:45 UTC — weekly DeepSeek Flash run and report complete
**Completed:** Started the weekly upgrader explicitly on `opencode/deepseek-v4-flash`; its parent
and every recipe subagent were confirmed with that persisted model. All nine eligible recipe PRs
completed `!testme` successfully. The public report was then restarted before publication when the
first handoff was found to use the old GLM model; the replacement report session was confirmed
`opencode/deepseek-v4-flash` and published successfully at
`https://report.ci.commoninternet.net/week-2026-08-31.html` (HTTP 200).
**Follow-up fixes:** Orchestrator PR #17 (`1b75d98`) made `testme-on-pr.sh` apply the documented
public Gitea-host default. cc-ci PR #31 (`769fd29`) adds that public hostname to `.env.public`.
Both commits were scanned clean and contain no coauthor trailers. No recipe PR was merged.
**Security note:** A subagent briefly enabled shell tracing while debugging the verifier, exposing
runtime credentials in its private agent trace. No values were committed or put in this journal,
but rotate the affected `/srv/cc-ci/.testenv` credentials as a precaution.
+10 -5
View File
@@ -1114,19 +1114,24 @@ def markdown(rep: dict) -> str:
def _gitea_auth(url: str) -> dict:
"""Basic auth for the private mirror, from /srv/cc-ci/.testenv.
"""Basic auth for the private mirror, using public config plus .testenv.
Sent as a HEADER, never embedded in the URL: in-URL credentials leak into shell history, process
lists and error messages, and urllib mis-parses a password containing a colon."""
host = re.sub(r"^https?://", "", url).split("/")[0]
env = {}
try:
for ln in open(os.environ.get("CCCI_TESTENV", "/srv/cc-ci/.testenv")):
for path in (
"/srv/cc-ci/cc-ci/.env.public",
os.environ.get("CCCI_TESTENV", "/srv/cc-ci/.testenv"),
):
try:
lines = open(path)
except OSError:
continue
for ln in lines:
if "=" in ln and not ln.strip().startswith("#"):
k, v = ln.strip().split("=", 1)
env[k] = v.strip().strip("\"'")
except OSError:
return {}
if host != env.get("GITEA_URL", "git.autonomic.zone"):
return {}
u, pw = env.get("GITEA_USERNAME"), env.get("GITEA_PASSWORD")
+1 -1
View File
@@ -34,7 +34,7 @@ supports_resume = false
prompt_delivery = "ping" # send after the TUI connects
connect_delay = 12
submit_key = "C-m"
preamble = "set -a; . /srv/cc-ci/.testenv; set +a"
preamble = "set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a"
stall_idle = 900
active_re = "esc interrupt|thinking|inferring|running tool|tool call|preparing patch|reading|searching"
limit_re = "spend limit|usage limit|limit reached|reached your .*limit|out of (credits|tokens)"
+1 -1
View File
@@ -112,7 +112,7 @@ def start(mode="resume"):
if not Path(OPENCODE_BIN).exists():
die(f"opencode not found at {OPENCODE_BIN}")
cmd = (
f"set -a; . /srv/cc-ci/.testenv; set +a; "
f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; "
f"NO_COLOR=1 {OPENCODE_BIN} attach {OPENCODE_SERVER} --dir {WORKDIR}"
)
log(f"starting {SESSION} (backend=opencode, model={LOOP_MODEL or 'default'})")
+1 -1
View File
@@ -154,7 +154,7 @@ def start(mode="resume"):
f"OPENCODE_CONFIG_CONTENT={shlex.quote(json.dumps({'model': LOOP_MODEL}))} "
)
cmd = (
f"set -a; . /srv/cc-ci/.testenv; set +a; "
f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; "
f"{model_env}NO_COLOR=1 {OPENCODE_BIN} attach {OPENCODE_SERVER} --dir {shlex.quote(WORKDIR)}"
)
log(f"starting {SESSION} (backend=opencode, model={LOOP_MODEL or 'default'})")
+1 -1
View File
@@ -109,7 +109,7 @@ def start(mode, date):
cwd = "/srv/cc-ci-orch/cc-ci"
share_flag = "--share" if OPENCODE_SHARE else ""
cmd = (
f"set -a; . /srv/cc-ci/.testenv; set +a; "
f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; "
f"{OPENCODE_BIN} run {model_flag} {share_flag} --attach '{OPENCODE_SERVER}' "
f"--title '{SESSION}' --dir {cwd} \"$(cat '{kf}')\""
)
+1 -1
View File
@@ -107,7 +107,7 @@ def spawn_supervisor(sid, reason):
kf = Path(LOG_DIR) / f".kickoff-{SUP_SESSION}.txt"
kf.write_text(build_kickoff(sid, reason))
share = "--share" if OPENCODE_SHARE else ""
cmd = (f"set -a; . {WORKDIR}/.testenv; set +a; {OPENCODE_BIN} run --model '{MODEL}' {share} "
cmd = (f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; {OPENCODE_BIN} run --model '{MODEL}' {share} "
f"--attach '{OPENCODE_SERVER}' --title '{SUP_SESSION}' --dir {WORKDIR} \"$(cat '{kf}')\"")
_sh(["tmux", "new-session", "-d", "-s", SUP_SESSION, "-c", WORKDIR, cmd])
_sh(["tmux", "pipe-pane", "-o", "-t", SUP_SESSION, f"cat >> '{LOG_DIR}/{SUP_SESSION}.log'"])
+2 -2
View File
@@ -216,7 +216,7 @@ def start(mode="use-or-create"):
# so they must come AFTER `run` (a global `opencode --model X run` is ignored).
share_flag = "--share" if OPENCODE_SHARE else ""
cmd = (
f"set -a; . /srv/cc-ci/.testenv; set +a; "
f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; "
f"{OPENCODE_BIN} run {model_flag} {share_flag} --attach '{OPENCODE_SERVER}' "
f"--title '{SESSION}' \"$(cat '{kf}')\""
)
@@ -521,7 +521,7 @@ def resume(reason="manual"):
"write+push the weekly summary, then `python3 /srv/cc-ci/cc-ci-plan/launch-report.py fresh`, print "
"'" + DONE_MARKER + "', and go idle.")
share = "--share" if OPENCODE_SHARE else ""
cmd = (f"set -a; . /srv/cc-ci/.testenv; set +a; {OPENCODE_BIN} run -s {sid} --continue "
cmd = (f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; {OPENCODE_BIN} run -s {sid} --continue "
f"--model '{MODEL}' {share} --attach '{OPENCODE_SERVER}' --dir '{WORKDIR}' \"$(cat '{kf}')\"")
subprocess.run(["tmux", "new-session", "-d", "-s", SESSION, "-c", WORKDIR, cmd])
subprocess.run(["tmux", "pipe-pane", "-o", "-t", SESSION, f"cat >> '{LOG_FILE}'"])
+1 -1
View File
@@ -323,7 +323,7 @@ def start_agent(role, session, workdir):
# absolute repo paths for builder/adversary work.
session_cwd = "/srv/cc-ci-orch/cc-ci"
cmd = (
f"set -a; . /srv/cc-ci/.testenv; set +a; "
f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; "
f"NO_COLOR=1 {OPENCODE_BIN} attach {OPENCODE_SERVER} --dir {session_cwd}"
)
log(f"starting {session} (backend=opencode, phase={pid}, model={model or 'default'})")
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env bash
# Load cc-ci runtime configuration. Source this file; do not execute it.
#
# Callers enable `set -a` so both files become exported environment variables.
# Public configuration is loaded first, then the secret file. This preserves a
# local secret override only where an emergency migration explicitly needs it.
if [ -r /srv/cc-ci/cc-ci/.env.public ]; then
. /srv/cc-ci/cc-ci/.env.public
fi
if [ ! -r /srv/cc-ci/.testenv ]; then
echo "cc-ci secret environment is missing: /srv/cc-ci/.testenv" >&2
return 1
fi
. /srv/cc-ci/.testenv
+1 -1
View File
@@ -71,7 +71,7 @@ log "writing report $REPORT ..."
if [ -f "$SUMMARY" ]; then cat "$SUMMARY"; else echo "(no dated summary at $SUMMARY — /upgrade-all may have stalled on the usage limit or errored; check the cc-ci-upgrader session + $LOG)"; fi
echo
echo "## Open PRs per recipe (post-run)"
set -a; . /srv/cc-ci/.testenv 2>/dev/null; set +a
set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a
G="https://${GITEA_USERNAME}:${GITEA_PASSWORD}@${GITEA_URL}/api/v1"
for r in $(curl -fsS "$G/orgs/recipe-maintainers/repos?limit=100" 2>/dev/null | python3 -c "import sys,json;print('\n'.join(sorted(x['name'] for x in json.load(sys.stdin))))" 2>/dev/null); do
case "$r" in cc-ci|cc-ci-orchestrator|cc-ci-secrets|archived-*) continue;; esac
+3 -2
View File
@@ -83,9 +83,10 @@ repo, a commit, a log, or the dashboard** (§9) — reference locations only.
| What | Where | How to use |
|---|---|---|
| **cc-ci SSH (root)** | private key `~/.ssh/cc-ci-root-ed25519`; `Host cc-ci` in `~/.ssh/config` (HostName `100.90.116.4`, no ProxyCommand) | Just run `ssh cc-ci` (logs in as **root**). The orchestrator VM is a direct tailnet peer — direct route, no proxy. Pubkey already in cc-ci's `/root/.ssh/authorized_keys`. |
| **Gitea bot account** | `/srv/cc-ci/.testenv``GITEA_USERNAME` (`autonomic-bot`), `GITEA_PASSWORD`, `GITEA_URL` (`git.autonomic.zone`) | Basic-auth to the Gitea API, or mint a scoped token: `POST https://$GITEA_URL/api/v1/users/$GITEA_USERNAME/tokens`. Used to push the `cc-ci` project repo, read recipe repos, comment on PRs, and poll for `!testme` (read-level; the bot does not register webhooks). |
| **Gitea bot account** | `/srv/cc-ci/cc-ci/.env.public``GITEA_USERNAME` (`autonomic-bot`); `/srv/cc-ci/.testenv` `GITEA_PASSWORD`; `GITEA_URL` defaults to `git.autonomic.zone` | Basic-auth to the Gitea API, or mint a scoped token: `POST https://$GITEA_URL/api/v1/users/$GITEA_USERNAME/tokens`. Used to push the `cc-ci` project repo, read recipe repos, comment on PRs, and poll for `!testme` (read-level; the bot does not register webhooks). |
Load them in a shell with: `set -a; . /srv/cc-ci/.testenv; set +a` (don't echo the values).
Load runtime configuration in a shell with: `set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a`
(don't echo secret values). The helper loads tracked public configuration first and then `.testenv`.
### The Tailscale connection (how `ssh cc-ci` works)
+7 -4
View File
@@ -40,19 +40,22 @@ from datetime import datetime, timezone
HERE = os.path.dirname(os.path.abspath(__file__))
USED_RECIPES = os.path.join(HERE, "used-recipes.md")
TESTENV = os.environ.get("CCCI_TESTENV", "/srv/cc-ci/.testenv")
PUBLIC_ENV = "/srv/cc-ci/cc-ci/.env.public"
NS = "recipe-maintainers"
def _env() -> dict:
e = {}
try:
for ln in open(TESTENV):
for path in (PUBLIC_ENV, TESTENV):
try:
lines = open(path)
except OSError:
continue
for ln in lines:
ln = ln.strip()
if "=" in ln and not ln.startswith("#"):
k, v = ln.split("=", 1)
e[k] = v.strip().strip('"').strip("'")
except OSError:
pass
return e
+7 -4
View File
@@ -46,6 +46,7 @@ from datetime import datetime, timezone
LOGDIR = "/srv/cc-ci/.cc-ci-logs"
TESTENV = "/srv/cc-ci/.testenv"
PUBLIC_ENV = "/srv/cc-ci/cc-ci/.env.public"
INFRA = {"cc-ci", "cc-ci-orchestrator", "cc-ci-secrets"}
HOST_REPORTS = "/var/lib/cc-ci-reports"
# Both kinds live in ONE archive, distinguished by a suffix on a common title.
@@ -56,14 +57,16 @@ KINDS = {"week": ("The Recipe Report", "Week of {d} — full"),
def _env():
e = {}
try:
for ln in open(TESTENV):
for path in (PUBLIC_ENV, TESTENV):
try:
lines = open(path)
except FileNotFoundError:
continue
for ln in lines:
ln = ln.strip()
if "=" in ln and not ln.startswith("#"):
k, v = ln.split("=", 1)
e[k] = v.strip().strip('"').strip("'")
except FileNotFoundError:
pass
return e
+2 -3
View File
@@ -30,13 +30,12 @@ set -o errexit -o nounset -o pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ORCH="$(dirname "$HERE")"
SSH="${SSH:-cc-ci}"
TESTENV="${TESTENV:-/srv/cc-ci/.testenv}"
RECONCILE="${RECONCILE:-$ORCH/.claude/skills/recipe-upgrade/open-recipe-pr.sh}"
USED_RECIPES="${USED_RECIPES:-$HERE/used-recipes.md}"
[ -f "$RECONCILE" ] || { echo "ERROR: reconcile helper not found: $RECONCILE" >&2; exit 1; }
set -a; . "$TESTENV"; set +a
: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:?}"
set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a
: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:=git.autonomic.zone}"
if [ "${1:-}" = "--all" ]; then
mapfile -t RECIPES < <(awk '!/^[[:space:]]*#/ && ($2=="weekly" || $2=="external") {print $1}' "$USED_RECIPES")
+28
View File
@@ -6,6 +6,14 @@
| postgres | postgres | https://github.com/postgres/postgres | https://www.postgresql.org/docs/release/ |
## Standing notes
- **2026-08-28 re-check** (endoflife.date/api/mattermost.json 2026-08-28; Docker Hub; GitHub
releases): **11.7.10** (released 2026-08-26, "various bug fixes", not prerelease) is the newest
11.7.x ESR patch — the ESR/LTS line has NOT moved, still **11.7** (EOL **2027-05-15**). This run
bumps open PR #2 **11.7.9 → 11.7.10** (ESR patch within the same line). 11.8/11.9/11.10 remain
innovation releases (EOL 2026-09-15 / 10-15 / 11-15), NOT ESR — do NOT target; 10.12 expired.
postgres **15-alpine HELD** (DB-major 16/17/18 out of scope, plain-postgres recipe w/o
pgautoupgrade: operator dump/restore needed). git.autonomic.zone mirror main reconciled to
coopcloud (86e0d0b5 = 10.11.22, upstream still pins the EXPIRED 10.11 ESR).
- mattermost-lts tracks the **11.7 ESR (Extended Support Release)** = Mattermost's LTS line. The "LTS"
concept in Mattermost is **ESR**; there is NO "10.x LTS line". Release calendar (endoflife.date/mattermost):
- **10.11** = previous ESR, security support **ends 2026-08-15** (~7 weeks) — migrated away 2026-06-19
@@ -76,6 +84,26 @@
The previous inline-label approach (no restore hook) was a defect: raw PGDATA restore without a reload
was a silent no-op. Fixed in PR #2 (restore fix cherry-picked from PR #1 ci/pg-restore).
- **2026-08-21 re-check** (endoflife.date/api/mattermost.json 2026-08-17; Mattermost docs; GitHub):
**11.7 ESR is the only current ESR/LTS** — latest patch **11.7.9** (2026-08-13, "Low to High
security fixes"), EOL 2027-05-15. **10.11 ESR ENDED 2026-08-15** — upstream main still pins
10.11.22 (now on an EXPIRED ESR). 11.10/11.9/11.8 are innovation (EOL 2026-11-15/10-15/09-15),
NOT LTS. The `-lts` recipe must track 11.7. This run **redirects open PR #2 away from the
innovation 11.10.0 onto 11.7.9** (the correct LTS/ESR; ESR→ESR 10.11→11.7 is "fully supported
and tested" per Mattermost docs — no staging hop). Refer to the 2026-08-15 report where PR #2
was on 11.10.0 (operator chose innovation line that week); operator should now decide the
`-lts` line. postgres 15-alpine still HELD (DB-major out of scope).
- **2026-08-31 re-check** (endoflife.date/api/mattermost.json 2026-08-31; Docker Hub tag list;
GitHub release `v11.7.10`): **11.7 ESR line UNCHANGED** — latest patch still **11.7.10**
(released 2026-08-26, "various bug fixes"), EOL 2027-05-15. PR #2 already carries 11.7.9 →
11.7.10; the wildcard cert is RENEWED (valid to 2026-11-29), so this run just RE-VERIFIES PR #2
via `!testme` (the 2026-08-28 run's `!testme` was infra-blocked). 11.8.5 / 11.9.1 / 11.10.1
remain innovation (EOL 2026-09-15 / 10-15 / 11-15), NOT ESR — do NOT target; 10.11 ESR ended
2026-08-15 (upstream main still pins 10.11.22 = EXPIRED ESR → the 10→11 ESR move PR #2 carries
remains required; Mattermost docs: ESR→ESR is "fully supported and tested"). postgres 15-alpine
still HELD (DB-major out of scope, operator dump/pg_upgrade).
## NVD CPE fallback
This project publishes nothing machine-readable we can reach — no GitHub advisory feed,
no release-attributable changelog — so its CVE count was `?` (nothing measured). NVD is
+72
View File
@@ -65,3 +65,75 @@
- 2.35.3 (2026-08-14, Pre-release): bugfixes (Google Ads v21→v25 API migration, MS Teams OAuth scope
restore, workflow publication outbox abort deadline) + feature (skip update approval for workflows
from same Instance AI session). Not deployed (2.35.2 was the survey target).
- 2.36.0 (2026-08-18, Pre-release; 2.36.1 was briefly tagged then withdrawn — 2.36.2 compares
2.36.0...2.36.2): a large feature minor. Core/ai-builder/editor bugfixes + new features (Agent
sandboxes + writable workspaces, MCP folder/move/read-data-table tools, Schedule Trigger missed-
execution grace period + catch-up coalescing, Confluence Cloud OAuth2 + page get sub-tree, MiniMax
M3/H3, LangSmith session debug export, role-mapping/roles API endpoints, n8n User Auth GA, etc.).
No breaking compose/env/migration changes, no `N8N_*` env renames; the deprecation notices
(Array.merge→Array.mergeIntoObject in expressions, workflow tags API migration) are API-level only.
2.36.2 (2026-08-19, patch): 3 fixes (decorator-route request schema in /discover, end-user
credential resolution in node parameters, editor agent-preview alignment). 2.36.3 (2026-08-20,
patch): 4 core/editor fixes (task-runners only restart when slow, expression engine init on
expression commands, test-webhook isolate release after teardown, Edit Fields type selector styles).
- 2026-08-21 run: planning 2.34.4 → 2.36.3 (newest tag abra lists). Whole 2.36.x line still
Pre-release on GitHub (2.33.5/Latest was superseded by 2.35.x line). Rolling upgrade safe, sqlite
mode, TypeORM migrations auto-run on boot. No operator action required.
- 2.36.4 (2026-08-21, Pre-release): 3 core fixes (env normalization before schema parsing; chat and
MCP trigger auth-mode validation for end-user credentials; wrap trigger closeFunction in an
expression isolate).
- 2.36.5 (2026-08-21, Pre-release): 1 core fix (raise AI Assistant model verification token limit).
- 2.36.6 (2026-08-24): 1 editor fix (personalization survey / community registration modal on
Instance AI landing page).
- 2.36.7 (2026-08-25, **current Stable / Latest badge**; `stable` tag points here): API fix (adapt
binary data endpoint), API cross-backport (require a JSON content type on decorator routes that
take a body), core fixes (reconcile license auto-renewal on startup leader takeover; route OAuth2
token requests through environment proxies), feature (remove offset query param from list workflow
history endpoint). This is the newest stable-line release; the 2.37.x line is still pre-release.
- 2.37.0 (2026-08-25, Pre-release): a large feature minor. Agent Builder / AIA / core/editor
bugfixes + features (Slack Agent view, prompt caching for Anthropic, SharePoint node v2 = default,
form-trigger OAuth2 (added then reverted), role-mapping rule CRUD endpoints, execution
startedAfter/startedBefore filters, Agent/community-package hardening: serialize community
package installs + non-destructive downloads + cold-boot storm reduction, MCP tool schemas as JSON
Schema 2020-12, distroless n8n image variant). **Deprecation: the "Any workflow" caller policy**
(API/config-level — flag for operators using execution-caller policies). Two enforced API behavior
changes: **require JSON content type on decorator body routes** and **"adapt binary data endpoint"**
(HTTP clients sending a body to those routes must send `Content-Type: application/json`). No
breaking compose/env/migration changes, no `N8N_*` env renames.
- 2.37.1 (2026-08-25, Pre-release): 1 core fix (route OAuth2 token requests through env proxies in
every process).
- 2.37.3 (2026-08-27, Pre-release; 2.37.2 was withdrawn): 2 runner-image fixes (copy glibc into
distroless runners image as a matched pair; stage libatomic into the distroless runners image) —
relevant if using n8n's community/distroless runner image.
- 2026-08-28 run: PR #7 extended 2.34.4 → **2.37.3** (newest tag abra lists; 2.37.2 withdrawn, that
tag does not exist). 2.36.7 holds the Stable/Latest badge; 2.37.x remains Pre-release on GitHub,
consistent with the tracking-the-newest-tag precedent (2.34.2/2.35.2/2.36.3 were pre-release when
taken). Rolling upgrade safe (sqlite, TypeORM auto-migrate on boot). Operator flag: the two 2.37.0
API behavior changes (JSON content-type on decorator body routes; binary-data endpoint adapt) only
affect HTTP API callers. Recommended release: `-y` (feature minor).
NOTE: PR #7's `!testme` on this date was BLOCKED — `*.ci.commoninternet.net` wildcard cert expired
(bridge: `CERTIFICATE_VERIFY_FAILED certificate has expired`). UNVERIFIED; operator renewed the cert
(now valid to 2026-11-29). The 2.37.3 work must be re-verified by a fresh `!testme`.
- 2.34.5 (2026-08-12, patch): 1 core fix (apply TLS options per hop through a proxy).
- 2.34.6 (2026-08-14, patch): 3 fixes (core `continueErrorOutput` `details` field; Google Ads sunset
v21→v25 API migration; MS Teams Restore `Group.ReadWrite.All` OAuth2 scope — matches the 2.35.3
in-progress backports).
- 2.35.4/2.35.5/2.35.6/2.35.7 (2026-08-19..21, patches): API schema for decorator routes in /discover;
Google Ads v25 view metric rename; task-runners not restarted when only slow; expression engine
init on expression commands; test-webhook isolate release after teardown; env normalization before
schema parsing; end-user credential resolution; trigger closeFunction isolate; AI Assistant token
limit raise. All patch bugfixes, no breaking changes.
- 2.36.8 (2026-08-28, patch): 1 core fix (domain-restricted credential usable in its own node) —
cross-backport of the 2.37.4 fix into the stable line.
- 2.36.9 (2026-08-31, patch; **now the Stable/Latest badge**`stable` tag): 1 core fix (apply proxy
environment variables consistently across packages and processes, cross-backport of 2.37.1/2.37.3).
- 2.37.4 (2026-08-28, Pre-release): 1 core fix (allow a domain-restricted credential to work in its
own node).
- 2.37.5 (2026-08-31): **withdrawn** — no plain `2.37.5` tag on Docker Hub (only partial
`2.37.5-amd64/-arm64/-pc` artifact tags, no manifest). Skip entirely, like 2.37.2/2.36.1.
- 2.37.6 (2026-08-31, Pre-release; **newest 2.37.x tag**): 1 core fix (thread execution id through
dynamic-credential storage).
- 2026-08-31 run: PR #7 extended 2.34.4 → **2.37.6** (newest tag abra lists = 2.37.6/2.37.4/2.37.3/…;
2.37.5 withdrawn). 2.36.9 holds the Stable/Latest badge; 2.37.x remains Pre-release on GitHub
(consistent precedent). Re-verified 2.37.3→2.37.6 (pure core bugfixes), no breaking changes beyond
the already-flagged 2.37.0 API behavior pair. Rolling upgrade safe. Recommended release: `-y`.
+5
View File
@@ -4,6 +4,7 @@
|---------|-------|-------------|----------------------|
| app | wordpress | https://github.com/WordPress/WordPress | https://wordpress.org/news/category/releases/ (image: https://hub.docker.com/_/wordpress) |
| db | mariadb | https://github.com/MariaDB/server | https://mariadb.com/kb/en/release-notes/ (image: https://hub.docker.com/_/mariadb) |
| ftp (optional overlay) | atmoz/sftp | https://github.com/atmoz/sftp | https://github.com/atmoz/sftp/releases (image: https://hub.docker.com/r/atmoz/sftp) |
## Standing notes
- Enrolled 2026-08-03 (operator request). Recipe = coopcloud `wordpress`; mirror
@@ -22,3 +23,7 @@
image). mariadb major bumps (e.g. 12.x→13.x) need the usual dump/restore caution if the
recipe ever pins a non-`mariadb`-auto-upgrading setup; the official image handles minor
bumps in place.
- `atmoz/sftp` is pinned as the floating `alpine` tag (updated 2026-07-28, current as of
2026-08-21) — not a semver abra can evaluate, so it never contributes an upgrade candidate;
verified current by direct Docker Hub tag check. Only bump if a real newer supported tag
exists and the app needs it.
+1 -1
View File
@@ -31,7 +31,7 @@ matrix-synapse weekly
mattermost-lts weekly
mumble weekly
n8n weekly
plausible weekly
plausible external # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded
uptime-kuma external # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded
wordpress weekly
```
+5
View File
@@ -18,6 +18,11 @@
system = "x86_64-linux";
in
{
# The cc-ci part of a host, on its own, so a host that runs cc-ci can import just this and
# keep its own (unrelated) configuration separate. Split out 2026-08-20; consumed by
# notplants-nix's `notplants-orchestrator` host.
nixosModules.cc-ci = ./nix/modules/cc-ci.nix;
# Hetzner cpx11 host (nixos-infect generated hardware.nix + orchestrator config).
# Provision with terraform/ then run Stage 2 per terraform/README.md.
nixosConfigurations.cc-ci-orchestrator-hetzner = nixpkgs.lib.nixosSystem {
@@ -14,7 +14,11 @@
# with ACME, and opens 80/443 (previously only 22 was public). Canonical source of
# the module is the project repo; ../atproto-likes.nix is a copy kept in this tree
# because pure evaluation cannot import a path outside the flake.
imports = [ ../../atproto-likes.nix ];
# cc-ci.nix holds everything that exists FOR cc-ci (the loops, the cc-ci orchestrator session,
# the weekly recipe-upgrade run and its hourly supervisor). Split out 2026-08-20 so this host —
# a general agent box that also serves unrelated projects — can be reasoned about separately
# from cc-ci. Also exported as `nixosModules.cc-ci` from this repo's flake for other hosts.
imports = [ ../../atproto-likes.nix ../../modules/cc-ci.nix ];
services.openssh = {
enable = true;
@@ -127,7 +131,8 @@ SSHCFG
# opencode web server — one shared instance; agent sessions attach to it for web visibility.
# Serves the web UI at http://oc.commoninternet.net (via nginx below, tailscale-only).
# Provider creds are read from /srv/cc-ci/.testenv at startup.
# Public provider configuration is tracked in the cc-ci repo; credentials
# remain in /srv/cc-ci/.testenv.
systemd.services.opencode-web = {
description = "opencode web server for cc-ci agents";
wantedBy = [ "multi-user.target" ];
@@ -137,7 +142,10 @@ SSHCFG
Type = "simple";
User = "loops"; Group = "users";
WorkingDirectory = "/srv/cc-ci-orch/cc-ci";
EnvironmentFile = "/srv/cc-ci/.testenv";
EnvironmentFile = [
"-/srv/cc-ci/cc-ci/.env.public"
"/srv/cc-ci/.testenv"
];
ExecStartPre = "${pkgs.coreutils}/bin/rm -rf /tmp/opencode";
ExecStart = "/home/loops/.local/bin/opencode serve --hostname 127.0.0.1 --port 4096";
Restart = "on-failure";
@@ -225,72 +233,7 @@ SSHCFG
};
};
# cc-ci-loops supervisor — workspace staged 2026-05-31, so ENABLED for reboot-resilience.
systemd.services.cc-ci-loops = {
description = "cc-ci Builder/Adversary loops + watchdog (launch.sh start)";
wantedBy = [ "multi-user.target" ]; # enabled after workspace staged (Hetzner cutover)
after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
wants = [ "network-online.target" ];
serviceConfig = {
# KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the
# default (control-group) systemd kills every leftover process in the cgroup when the unit
# stops — and since one tmux server hosts every agent session on this host, a rebuild that
# merely touched this unit wiped all of them (operator 2026-08-01). Only the (already
# exited) main process is killed now; `systemctl stop` therefore does NOT tear down agents.
KillMode = "process";
Type = "oneshot"; RemainAfterExit = true;
User = "loops"; Group = "users";
WorkingDirectory = "/srv/cc-ci/cc-ci";
# Append one line to REBOOTS.md per genuine reboot (boot_id-gated; not on manual restart).
ExecStartPre = "${pkgs.bash}/bin/bash /srv/cc-ci/cc-ci-plan/reboot-log.sh";
};
# CLAUDE_BIN points at the standalone CLI installed by claude-install.service; the loops
# backend defaults to claude (persisted in .loop-backend). Without this, launch.py's preflight
# `which(claude)` fails because the systemd `path` below has no /home/loops/.local/bin.
environment = { RESUME_PHASE = "1"; HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
script = ''
# Put the standalone claude/opencode binaries on PATH. On a cold boot this is the env the
# tmux server (and thus every agent session) inherits, so bare `claude` resolves everywhere.
export PATH="/home/loops/.local/bin:$PATH"
[ -x /srv/cc-ci/cc-ci-plan/launch.sh ] && /srv/cc-ci/cc-ci-plan/launch.sh start || \
echo "workspace not staged yet skipping loop start"
'';
};
# cc-ci-orchestrator supervisor — the operator's steering session. Same shape as
# lichen-orchestrator / project-orchestrator above: this unit only LAUNCHES the orchestrator's
# tmux session via the agent-orchestrator harness (cc-ci-plan/agents.py); it does not own the
# session or the tmux server. The orchestrator agent is declared in cc-ci-plan/agents.toml on
# the OPencode backend (backend = "opencode", model = "opencode/glm-5.2"), so on boot it
# attaches to the shared opencode web server (opencode-web.service below) and is reachable for
# Remote Control at https://oc.commoninternet.net under the /srv/cc-ci-orch project. The harness
# watchdog (started by `agents.py up`) keeps it alive: heal-only (no stall reboots — a persistent
# supervisor must not be killed just for idling). Added 2026-08-03 to give the cc-ci orchestrator
# the same reboot-resilience the other two orchestrators already have.
systemd.services.cc-ci-orchestrator = {
description = "cc-ci orchestrator (operator steering session) agents.py up orchestrator, opencode backend";
wantedBy = [ "multi-user.target" ];
after = [ "network-online.target" "tailscaled.service" "opencode-web.service" ];
wants = [ "network-online.target" ];
serviceConfig = {
# KillMode=process: see the note on cc-ci-loops — a rebuild that merely touches this unit
# must not tear down the (shared) tmux server and every agent session with it.
KillMode = "process";
Type = "oneshot"; RemainAfterExit = true;
User = "loops"; Group = "users";
WorkingDirectory = "/srv/cc-ci-orch";
};
environment = { HOME = "/home/loops"; };
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
script = ''
export PATH="/home/loops/.local/bin:$PATH"
proj="/srv/cc-ci-orch"
echo "$(cat /proc/sys/kernel/random/boot_id) boot $(date -u +%FT%TZ) cc-ci-orchestrator up" \
>> "$proj/cc-ci-plan/.ao-boot.log" 2>/dev/null || true
cd "$proj" && python3 cc-ci-plan/agents.py up orchestrator || echo "cc-ci orchestrator agents.py up failed"
'';
};
# p-lichen-orchestrator supervisor — the SEPARATE lichen.page testing/hardening orchestrator
# (distinct from cc-ci-loops above). Reboot-resilience: on boot, resume the orchestrator's Remote
@@ -356,79 +299,7 @@ SSHCFG
'';
};
# Weekly recipe upgrade — runs /upgrade-all over every enrolled recipe (opens recipe PRs
# verified by !testme, never merges). Replaces the boot-fragile busybox-crond-in-tmux from
# phase 5 §4 with a reboot-safe systemd timer. The service is timer-triggered only (NOT
# wantedBy multi-user.target) so it never runs on boot/activation — only on the schedule.
systemd.services.cc-ci-upgrade-all = {
description = "cc-ci weekly /upgrade-all run (recipe upgrade survey + PRs, never merges)";
after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
wants = [ "network-online.target" ];
serviceConfig = {
Type = "oneshot"; # launch-upgrader.py spawns the cc-ci-upgrader tmux session and returns
User = "loops"; Group = "users";
WorkingDirectory = "/srv/cc-ci";
# Optional per-run overrides for backend/model (LOOP_BACKEND, LOOP_MODEL, OPENCODE_SHARE,
# UPGRADER_ARGS, …). The leading "-" makes it optional: absent file → claude/sonnet defaults.
# Current config (as of 2026-08-16): the upgrader + report run on tinfoil/deepseek-v4-pro
# (LOOP_MODEL + REPORT_MODEL in the env file); the hourly SUPERVISOR stays on glm-5.2
# (SUPERVISOR_MODEL defaults to opencode-go/glm-5.2 in launch-supervisor.py, NOT overridden
# here). Subagents bind deepseek via the cc-ci repo's opencode config. LOOP_TIER=zen is kept
# so the tier check passes; the watchdog's usage-limit probe sends the deepseek model name to
# the zen endpoint, which returns 200 (not 429) → resume immediately (correct: tinfoil has no
# rolling usage limit to wait out). No rebuild needed to switch — the env file is read at each
# timer fire. Holds no secrets (the tinfoil API key lives in the opencode config / auth.json).
EnvironmentFile = "-/srv/cc-ci/upgrader.env";
};
environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
script = ''
export PATH="/home/loops/.local/bin:$PATH"
python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py start >> /srv/cc-ci/.cc-ci-logs/upgrader-cron.log 2>&1
'';
};
systemd.timers.cc-ci-upgrade-all = {
description = "Weekly trigger for cc-ci-upgrade-all (Thursdays 22:00 America/New_York Boston 10pm)";
wantedBy = [ "timers.target" ];
timerConfig = {
# 10pm Thursday Boston time — DST-aware (EDT→02:00 UTC, EST→03:00 UTC) via the tz in OnCalendar.
OnCalendar = "Thu *-*-* 22:00:00 America/New_York";
Persistent = true; # if the box was down at the scheduled time, run once on next boot
};
};
# Hourly SUPERVISOR — a glm-5.2 orchestrator wake-up that keeps the weekly run on track. The
# log-idle/429 watchdog only handles opencode-go usage-limit stalls; it does NOT cover a host
# disk-full crash (which killed the 2026-07-03 run) or any other environmental wedge. This is a
# CHEAP deterministic gate: if the weekly run is complete or actively progressing it does NOTHING
# (zero model tokens). Only when a run has stalled/died before completing does it launch a
# short-lived glm-5.2 agent that diagnoses the blockage and drives the run to a clean DONE.
systemd.services.cc-ci-upgrade-supervisor = {
description = "cc-ci hourly weekly-run supervisor (glm-5.2 drives a stalled /upgrade-all to completion)";
after = [ "network-online.target" "tailscaled.service" ];
wants = [ "network-online.target" ];
serviceConfig = {
Type = "oneshot"; # launch-supervisor.py check: gate now, spawn the agent into tmux, return
User = "loops"; Group = "users";
WorkingDirectory = "/srv/cc-ci";
# Shares the weekly run's optional override file (e.g. SUPERVISOR_MODEL=…); "-" = optional.
EnvironmentFile = "-/srv/cc-ci/upgrader.env";
};
environment = { HOME = "/home/loops"; };
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
script = ''
export PATH="/home/loops/.local/bin:$PATH"
python3 /srv/cc-ci/cc-ci-plan/launch-supervisor.py check >> /srv/cc-ci/.cc-ci-logs/supervisor-cron.log 2>&1
'';
};
systemd.timers.cc-ci-upgrade-supervisor = {
description = "Hourly trigger for cc-ci-upgrade-supervisor (weekly-run health check + drive)";
wantedBy = [ "timers.target" ];
timerConfig = {
OnCalendar = "*-*-* *:07:00"; # every hour at :07 (offset from the weekly :00 fire)
Persistent = false; # a missed hourly check is moot — the next hour re-checks
};
};
}
+157
View File
@@ -0,0 +1,157 @@
# cc-ci.nix — everything on this host that exists FOR cc-ci, and nothing else.
#
# Split out of the orchestrator host config on 2026-08-20. The host it runs on is a general
# agent/orchestration box that also serves several unrelated projects; this module is the cc-ci
# part of it, so that the two can evolve (and be reviewed) independently. It is exported from this
# repo's flake as `nixosModules.cc-ci` and imported by whichever host runs cc-ci.
#
# All of it assumes the cc-ci workspaces exist on the host:
# /srv/cc-ci the loops workspace (+ .cc-ci-logs, upgrader.env)
# /srv/cc-ci-orch this repo (the orchestrator's own working dir)
# and that a `loops` user, tmux, python3 and the standalone claude/opencode CLIs are present —
# those are host concerns, provided by the host config, not by this module.
{ config, pkgs, lib, ... }:
{
# cc-ci-loops supervisor — workspace staged 2026-05-31, so ENABLED for reboot-resilience.
systemd.services.cc-ci-loops = {
description = "cc-ci Builder/Adversary loops + watchdog (launch.sh start)";
wantedBy = [ "multi-user.target" ]; # enabled after workspace staged (Hetzner cutover)
after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
wants = [ "network-online.target" ];
serviceConfig = {
# KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the
# default (control-group) systemd kills every leftover process in the cgroup when the unit
# stops — and since one tmux server hosts every agent session on this host, a rebuild that
# merely touched this unit wiped all of them (operator 2026-08-01). Only the (already
# exited) main process is killed now; `systemctl stop` therefore does NOT tear down agents.
KillMode = "process";
Type = "oneshot"; RemainAfterExit = true;
User = "loops"; Group = "users";
WorkingDirectory = "/srv/cc-ci/cc-ci";
# Append one line to REBOOTS.md per genuine reboot (boot_id-gated; not on manual restart).
ExecStartPre = "${pkgs.bash}/bin/bash /srv/cc-ci/cc-ci-plan/reboot-log.sh";
};
# CLAUDE_BIN points at the standalone CLI installed by claude-install.service; the loops
# backend defaults to claude (persisted in .loop-backend). Without this, launch.py's preflight
# `which(claude)` fails because the systemd `path` below has no /home/loops/.local/bin.
environment = { RESUME_PHASE = "1"; HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
script = ''
# Put the standalone claude/opencode binaries on PATH. On a cold boot this is the env the
# tmux server (and thus every agent session) inherits, so bare `claude` resolves everywhere.
export PATH="/home/loops/.local/bin:$PATH"
[ -x /srv/cc-ci/cc-ci-plan/launch.sh ] && /srv/cc-ci/cc-ci-plan/launch.sh start || \
echo "workspace not staged yet skipping loop start"
'';
};
# cc-ci-orchestrator supervisor — the operator's steering session. Same shape as
# lichen-orchestrator / project-orchestrator above: this unit only LAUNCHES the orchestrator's
# tmux session via the agent-orchestrator harness (cc-ci-plan/agents.py); it does not own the
# session or the tmux server. The orchestrator agent is declared in cc-ci-plan/agents.toml on
# the OPencode backend (backend = "opencode", model = "opencode/glm-5.2"), so on boot it
# attaches to the shared opencode web server (opencode-web.service below) and is reachable for
# Remote Control at https://oc.commoninternet.net under the /srv/cc-ci-orch project. The harness
# watchdog (started by `agents.py up`) keeps it alive: heal-only (no stall reboots — a persistent
# supervisor must not be killed just for idling). Added 2026-08-03 to give the cc-ci orchestrator
# the same reboot-resilience the other two orchestrators already have.
systemd.services.cc-ci-orchestrator = {
description = "cc-ci orchestrator (operator steering session) agents.py up orchestrator, opencode backend";
wantedBy = [ "multi-user.target" ];
after = [ "network-online.target" "tailscaled.service" "opencode-web.service" ];
wants = [ "network-online.target" ];
serviceConfig = {
# KillMode=process: see the note on cc-ci-loops — a rebuild that merely touches this unit
# must not tear down the (shared) tmux server and every agent session with it.
KillMode = "process";
Type = "oneshot"; RemainAfterExit = true;
User = "loops"; Group = "users";
WorkingDirectory = "/srv/cc-ci-orch";
};
environment = { HOME = "/home/loops"; };
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
script = ''
export PATH="/home/loops/.local/bin:$PATH"
proj="/srv/cc-ci-orch"
echo "$(cat /proc/sys/kernel/random/boot_id) boot $(date -u +%FT%TZ) cc-ci-orchestrator up" \
>> "$proj/cc-ci-plan/.ao-boot.log" 2>/dev/null || true
cd "$proj" && python3 cc-ci-plan/agents.py up orchestrator || echo "cc-ci orchestrator agents.py up failed"
'';
};
# Weekly recipe upgrade — runs /upgrade-all over every enrolled recipe (opens recipe PRs
# verified by !testme, never merges). Replaces the boot-fragile busybox-crond-in-tmux from
# phase 5 §4 with a reboot-safe systemd timer. The service is timer-triggered only (NOT
# wantedBy multi-user.target) so it never runs on boot/activation — only on the schedule.
systemd.services.cc-ci-upgrade-all = {
description = "cc-ci weekly /upgrade-all run (recipe upgrade survey + PRs, never merges)";
after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
wants = [ "network-online.target" ];
serviceConfig = {
Type = "oneshot"; # launch-upgrader.py spawns the cc-ci-upgrader tmux session and returns
User = "loops"; Group = "users";
WorkingDirectory = "/srv/cc-ci";
# Optional per-run overrides for backend/model (LOOP_BACKEND, LOOP_MODEL, OPENCODE_SHARE,
# UPGRADER_ARGS, …). The leading "-" makes it optional: absent file → claude/sonnet defaults.
# Current config (as of 2026-08-16): the upgrader + report run on tinfoil/deepseek-v4-pro
# (LOOP_MODEL + REPORT_MODEL in the env file); the hourly SUPERVISOR stays on glm-5.2
# (SUPERVISOR_MODEL defaults to opencode-go/glm-5.2 in launch-supervisor.py, NOT overridden
# here). Subagents bind deepseek via the cc-ci repo's opencode config. LOOP_TIER=zen is kept
# so the tier check passes; the watchdog's usage-limit probe sends the deepseek model name to
# the zen endpoint, which returns 200 (not 429) → resume immediately (correct: tinfoil has no
# rolling usage limit to wait out). No rebuild needed to switch — the env file is read at each
# timer fire. Holds no secrets (the tinfoil API key lives in the opencode config / auth.json).
EnvironmentFile = "-/srv/cc-ci/upgrader.env";
};
environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
script = ''
export PATH="/home/loops/.local/bin:$PATH"
python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py start >> /srv/cc-ci/.cc-ci-logs/upgrader-cron.log 2>&1
'';
};
systemd.timers.cc-ci-upgrade-all = {
description = "Weekly trigger for cc-ci-upgrade-all (Thursdays 22:00 America/New_York Boston 10pm)";
wantedBy = [ "timers.target" ];
timerConfig = {
# 10pm Thursday Boston time — DST-aware (EDT→02:00 UTC, EST→03:00 UTC) via the tz in OnCalendar.
OnCalendar = "Thu *-*-* 22:00:00 America/New_York";
Persistent = true; # if the box was down at the scheduled time, run once on next boot
};
};
# Hourly SUPERVISOR — a glm-5.2 orchestrator wake-up that keeps the weekly run on track. The
# log-idle/429 watchdog only handles opencode-go usage-limit stalls; it does NOT cover a host
# disk-full crash (which killed the 2026-07-03 run) or any other environmental wedge. This is a
# CHEAP deterministic gate: if the weekly run is complete or actively progressing it does NOTHING
# (zero model tokens). Only when a run has stalled/died before completing does it launch a
# short-lived glm-5.2 agent that diagnoses the blockage and drives the run to a clean DONE.
systemd.services.cc-ci-upgrade-supervisor = {
description = "cc-ci hourly weekly-run supervisor (glm-5.2 drives a stalled /upgrade-all to completion)";
after = [ "network-online.target" "tailscaled.service" ];
wants = [ "network-online.target" ];
serviceConfig = {
Type = "oneshot"; # launch-supervisor.py check: gate now, spawn the agent into tmux, return
User = "loops"; Group = "users";
WorkingDirectory = "/srv/cc-ci";
# Shares the weekly run's optional override file (e.g. SUPERVISOR_MODEL=…); "-" = optional.
EnvironmentFile = "-/srv/cc-ci/upgrader.env";
};
environment = { HOME = "/home/loops"; };
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
script = ''
export PATH="/home/loops/.local/bin:$PATH"
python3 /srv/cc-ci/cc-ci-plan/launch-supervisor.py check >> /srv/cc-ci/.cc-ci-logs/supervisor-cron.log 2>&1
'';
};
systemd.timers.cc-ci-upgrade-supervisor = {
description = "Hourly trigger for cc-ci-upgrade-supervisor (weekly-run health check + drive)";
wantedBy = [ "timers.target" ];
timerConfig = {
OnCalendar = "*-*-* *:07:00"; # every hour at :07 (offset from the weekly :00 fire)
Persistent = false; # a missed hourly check is moot — the next hour re-checks
};
};
}