The extend path grafts HEAD^{tree} WHOLESALE onto the existing upgrade-* branch
(commit-tree -p <branch tip>). Reconcile force-syncs the MIRROR's main to upstream
but never brought the branch — or the local checkout — forward, so each week the
PR base drifted further back and upstream changes made since the branch was cut
were silently absent from the pushed tree. CI then verified a tree that would
never deploy.
Found on gitea PR #5 (2026-08-10): base 0ab323d predated upstream's
'BREAKING CHANGE: remove forgejo' (37ebd22), so the 1.27.1 bump fixing
CVE-2026-60004 + CVE-2026-59774 was !testme-GREEN against a forgejo-bearing tree.
Two changes:
1. Before pushing, if the local work does not contain the freshly-synced upstream
main, merge upstream in — and FAIL LOUDLY (exit 1, naming the checkout) if that
cannot auto-merge, rather than pushing a tree that omits upstream changes.
2. The extend commit now also parents on upstream main when the branch predated it,
so the recorded history matches the pushed tree. Without it the merge-base stays
stale and a later merge can REVERT upstream's changes. Still no force-push.
Verified against the real gitea drift: detection fires, merge is clean, resulting
tree keeps forgejo removed AND the 1.27.1 pin, history contains upstream.
It lived in the orchestrator repo (the parent session's project) and so never
governed the task-tool subagents, which resolve their parent session's directory —
for launcher-started runs that is /srv/cc-ci-orch/cc-ci. The real config now lives
there (cc-ci repo, operator-approved). Extension was never the problem: .jsonc
parsed and resolved fine; the LOCATION was wrong.
Regression from the 2026-08-04 session-pinning work: lu._session_id() prefers the
pin file, but launch-report.py never cleared or re-established it. A surviving pin
from a PREVIOUS report run points at a session whose last message already carries
RECIPE REPORT COMPLETE, so the shared watchdog evaluates _completed()=True and
exits within one poll ('run completed — exiting'), leaving the CURRENT run
unwatched. Observed live: the 2026-08-07 finish-run's report step was watched by a
watchdog that quit after 3 minutes against an Aug-4 pin, then the report session
ended early with nothing to resume it.
start() now archives stale titles, clears the pin, snapshots existing ids, and
re-pins the new session after launch — the same contract launch-upgrader.start()
already follows. Scopes the shared helpers via UPGRADER_SESSION=<report session>.
Root cause chain, all confirmed on the live system:
1. _run_pids() substring-matched the WHOLE cmdline for the session name. An agent's
kickoff PROMPT is an argv element, and the supervisor's prompt text contains the
literal 'cc-ci-upgrader' — so the supervisor's OWN billing-hung agent matched as a
live upgrader run. Now matches FLAG VALUES only (--title <SESSION> / -s <sid>).
2. The gate treated 'a live proc exists' as progress. A provider-walled run keeps its
process alive and SPINNING while emitting nothing (verified: 3 days, zero session
output, still burning CPU). Progress now REQUIRES the session tree to have advanced
within STALL_MIN; a live-but-idle proc is explicitly logged as stalled.
Result was ~60 consecutive false 'run progressing — leaving it' no-ops while the weekly
run sat unfinished and unreported.
Billing-walled runs are REPORTED, never killed (operator policy 2026-08-10): they may
resume when the wall lifts and their context is the run's state. New _billing_blocked()
detects the wall from the log tail and the gate surfaces
'run BLOCKED on a provider billing/usage wall — NOT killing; operator action required'.
Verified live: _run_pids no longer matches the hung Aug-7 supervisor (1497561) while
still matching the real finisher; gate now reasons 'session advanced Nm ago'.
- _archive_stale_titles default label is now 'archive-<title> —' (operator
convention: all archived sessions start with archive- so they sort/filter
together in the web UI).
- launch-report.py start() archives older cc-ci-report sessions before launch,
same invariant as upgrader/supervisor (reuses the launch-upgrader helper).
- 33 sessions restyled/archived live; canonical names now unique:
cc-ci-upgrader (idle finisher), cc-ci-report (generating), cc-ci-supervisor
(none — only exists during a rescue).
_archive_stale_titles() generalized to (title, label); launch-supervisor's
spawn_supervisor() archives every older 'cc-ci-supervisor' session before
launching, so exactly one session carries the name. 11 historical supervisor
sessions archive-renamed live ('supervisor archive — <date> <time>').
Bug (2026-08-04 16:00): _session_id() sorted candidates on (s.time.created) which the
/session API rows DON'T carry — every key was 0, 'newest' degraded to server list
order, and the watchdog resumed the old giant unresumable session, kill_session()ing
the healthy fresh run mid-work.
Fixes:
- Pin the managed session id at launch/resume to LOG_DIR/.{SESSION}-session-id;
_session_id() prefers the pin, validated via direct GET /session/<id> (the LIST is
paginated ~100 rows, membership scans lie). Title lookup is only the fallback and
now sorts on authoritative sqlite time_created.
- _archive_stale_titles() at start: every older top-level session titled
cc-ci-upgrader is renamed 'upgrader archive — weekly <date>', so EXACTLY ONE
session ever carries the canonical name in the opencode web UI (easy to find;
finished runs stay browsable under archive names). 11 historical sessions
renamed live today; the in-flight finisher pinned.
Verified live: _session_id() returns the pinned running session; tree-idle 0.0min
while subagents active. Full synthetic-stall watchdog confirmation queued post-run
(task #13).
Covers the 2.32.4 -> 2.33.3 range for the 2026-08-03 /upgrade-all run:
2.33.0 minor features (admin-managed instance creds, workflow review
requests + publish/unpublish API, API deprecation of activate/deactivate
endpoints, optional N8N_SCHEDULER_MAX_ATTEMPTS env) and the 2.33.1-2.33.3
patches; notes 2.34.0 exists but is not this run's target.
Operator decision: main driving sessions stay glm-5.2; every task-tool subagent
runs opencode/deepseek-v4-pro (~3-5x cheaper, near-free cache hits) — the weekly
upgrade run burns most of its budget in subagents. Report generator stays glm-5.2
(launch-report.py default). Project-scoped config.
The 2026-08-03 cc-ci outage was recovered with ad-hoc tooling living in /tmp
(leftover from a PREVIOUS incident, half-evaporated). Promoted to the repo:
- scripts/recovery/hetzner.py — Hetzner API helper (status/actions/reboot/reset/
power/rescue-on|off/console), knows cc-ci=134485294 + orchestrator=134487234 by
name; token from HCLOUD_TOKEN or /srv/cc-ci/.hcloud-token (0600, never in git).
- scripts/recovery/hetzner-console.sh — shell-only VGA console: fresh console
session -> websocat bridge -> vncdotool (venv auto-bootstrapped in ~/.cache).
screenshot / key / type subcommands; encodes the reset-invalidates-session and
single-connection-bridge gotchas.
- scripts/recovery/README.md — the condensed 10-minute unreachable-server drill,
incl. the GRUB submenu 1>N ids + clear-grubenv-after-switch rule.
- hetzner-server-recovery skill: console/API sections now point at the repo tools
instead of describing /tmp rebuilds.
Smoke-tested: hetzner.py cc-ci status OK.
Operator decision: no policy difference between cc-ci and recipe-maintainer. On
inspection ARM already agrees (recipe-upgrade-cron-all: 'PRs are reviewed and merged
manually by a human afterwards... never merges anything'; 'no human review in the
middle' = skip the mid-run plan confirmation only). Wrappers previously framed this
as a cc-ci override over ARM auto-merge flows — wrong reading; now stated as ONE
unified rule. /help conventions updated to match.
The repo already vendored ARM as the references/recipe-maintainer submodule (old
repo name recipe-maintainers/recipe-maintainer, pinned 460eba0). Rather than carry
two copies, drop the just-added vendor/ duplicate and:
- retarget references/recipe-maintainer to
ssh://git.autonomic.zone/recipe-maintainers/autonomic-recipe-maintainer.git
(same lineage — 460eba0 is an ancestor) and bump to latest acd5cfb, which also
refreshes the parity-test SOURCE reference the tests cite.
- gen-cctest-skills.py + all 30 cctest-* wrappers + /help now point at
references/recipe-maintainer.
- JOURNAL.md: pending session entries (server-update policy addendum, tests-update,
orchestrator-update, upgrade-run notes).
One operator interface for both toolkits (operator decision 2026-08-04):
- vendor/autonomic-recipe-maintainer: ARM pinned as a submodule at acd5cfb (latest).
- scripts/gen-cctest-skills.py: generates a cctest-<name> wrapper pair
(.opencode canonical + .claude thin) for every ARM skill — frontmatter carries ARM's
own description tagged [recipe-maintainer/cctest]; body points at the canonical
submodule SKILL.md, sets cwd/sandbox context, and states the policy overrides
(auto-merge-style ARM flows need per-run operator opt-in; never touch cc-ci infra
from an ARM skill; submodule is read-only here). Re-run after every submodule bump.
- 30 cctest-* wrappers generated.
- /help: cctest family section + situation-table rows + the cc-ci-vs-cctest rule of thumb.
cctest = the recipe-maintainer test server; these skills run against it + the ARM
sandbox, never against the cc-ci CI server/swarm.
Seven check areas: weekly-run recency+outcome, report publishing, stale recipes/tests,
open recipe PRs (CVE-carrying PRs open >14d flagged high-priority, ready-to-merge PRs
listed as normal), server+orchestrator flake-update recency vs channel tip, host/service
health incl. the bridge !testme path (silent-401 stale-secret check from the 2026-08-03
finding), and maintained-set consistency. Ends with ALL HEALTHY or a prioritized findings
list, each mapped to the skill to invoke. Strictly read-only.
Codifies the full path walked for the 2026-08-03 wordpress enrollment (cc-ci PR #14):
survey -> mirror create+sync from coopcloud -> author test suite (health floor +
non-vacuous recipe-specific tests incl. sec4.3 create-an-object round-trip, recipe-local
setup helper, PARITY.md) -> bridge POLL_REPOS -> used-recipes.md weekly row +
upstream/<recipe>.md registry -> full-suite-green verification with the new tests ->
bridge deploy via test-before-switch -> merge-on-green + report (PR-visibility policy).
Includes the traps hit live: creds injection over stdin for cc-ci-side helpers, fresh-
deploy wizard state in HEALTH_OK, repo-dev-shell ruff, swarm serialization, stale bridge
secret 401s silently dropping !testme.
- cc-ci-server-update / cc-ci-tests-update / cc-ci-update: operator policy change —
open PRs for visibility/historical record and merge directly once verified
(invocation = authorization); reports must list merged-PR links + change summaries.
(tests-update still never merges the paired recipe upgrade PRs.)
- cc-ci-server-update: mandatory nixos-rebuild test step (5d) before switch, with
detached-activation + transient-unit notes from the 2026-08-03 26.05 deploy.
- AGENTS.md: test-before-switch policy for orchestrator host rebuilds.
- NEW skill cc-ci-orchestrator-update (+ thin .claude wrapper): the /cc-ci-server-update
analogue for this host (flake /srv/cc-ci-orch, .#cc-ci-orchestrator-hetzner, Hetzner
server 134487234), with self-update caveats.
- JOURNAL.md: 2026-08-03 server-update incident/recovery handoff entry.
Reverse the skill location: the full definitions now live in
.opencode/skills/<name>/SKILL.md (opencode's native project-skill
location, auto-scanned), and .claude/skills/<name>/SKILL.md are thin
pointer wrappers that keep the frontmatter for discoverability and
redirect to the .opencode/ canonical file for the procedure.
Also drops the 'operator-authorized mid-run' gating from
/cc-ci-server-update: the operator's choice to invoke the skill IS the
authorization for the live nixos-rebuild switch — no second check to
wait for. The quiescent-moment preference stays (don't disrupt an
in-flight !testme), but it's not a gate.
Three new skills for keeping cc-ci itself current (manual triggers for
what should ideally be automatic):
- /cc-ci-server-update — bump the cc-ci SERVER host's nixpkgs/sops-nix
flake inputs (hard rev pin, moves deliberately), open a cc-ci PR, then
deploy to the live host via nixos-rebuild switch (operator-authorized,
no-CI window) + health gate (no failed units, infra up, endpoints 200).
Manual trigger for periodic nixpkgs bumps.
- /cc-ci-tests-update — sweep all maintained recipes for stale cc-ci
tests (carry-over from /upgrade-all's commented stale tests + a fresh
/ci-test-review sweep), author the minimal test update per stale
recipe, open a cc-ci test PR, verify via the standard PR + !testme ->
CI flow. Closes the loop the weekly /upgrade-all deliberately defers
(it comments-only; this fixes the tests). Never weakens a test.
- /cc-ci-update — umbrella: run server-update then tests-update, in
that order, gated on the server health check (don't sweep tests on a
sick/rolled-back host). Server-first because the harness toolchain
rides on the host nixpkgs.
All three: create + verify, NEVER merge. Reuse the existing ci-test-review
helpers (open-cc-ci-pr.sh, run-all-recipes.sh, verify-pr.sh) and the
recipe-upgrade --with-tests discipline.
Switch the cc-ci-orchestrator agent (cc-ci-plan/agents.toml) from the claude
backend to the opencode backend so it attaches to the shared opencode web
server (opencode-web.service, 127.0.0.1:4096) and is reachable for Remote
Control at https://oc.commoninternet.net under the /srv/cc-ci-orch project.
Model = opencode/glm-5.2 (same provider family as the weekly upgrader /
supervisor). resume=false — the opencode backend has no --resume equivalent
(a fresh session per launch, matching the existing launch-opencode.sh shape).
Add a cc-ci-orchestrator.service systemd unit so on boot it runs
'agents.py up orchestrator' after opencode-web.service is up — the same
reboot-resilience shape lichen-orchestrator / project-orchestrator already
have. KillMode=process so a rebuild that merely touches this unit does not
tear down the shared tmux server and every agent session with it.
No secrets: the diff is only config + comments (paths, model names, unit
definition).
Add a 443 listener on the tailscale IP for the opencode vhost, alongside the
existing port 80. The name resolves to a CGNAT tailscale IP so Let's Encrypt
HTTP-01 can't validate it and there's no DNS-01 provider here; the vhost is
tailnet-only, so a self-signed cert (out-of-band at /etc/nginx/oc-selfsigned.*,
like oc-htpasswd) is acceptable. addSSL=true is required so the NixOS nginx
module actually renders ssl_certificate — without it nginx -t fails and takes
the atproto vhost down too. Rationale + regen command documented inline.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016L6nYYwkCWnrEFKTnKAfet
Upstream module dropped host-specific comments for its public repo, so the
context that matters here — gateway fronting, why this vhost stays off the
tailscale address group where opencode lives, and the exposure check script —
moves into the copy's header where it belongs.
Functional changes from upstream: the container now runs as uid 10001 and the
unit chowns the bind-mounted cache before start (without it, cache files left
by the earlier root container are unreadable and every cached actor 500s).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmEK2voMnBa23495aLk1Ce
Sync of the canonical module. The atproto vhost listens on 0.0.0.0 only, so it
never shares an address group with oc.commoninternet.net (which is the default
server on 100.84.190.30:80 and has no auth in front of it). A new
public-default-reject vhost owns the public addresses — return 444 on 80,
ssl_reject_handshake on 443 — so unmatched, absent or spoofed Host headers from
the internet get nothing, now and after any future vhost is added.
The tailnet is untouched: oc.commoninternet.net still serves opencode there.
Verified with tools/check-exposure.sh in the project repo — 9/9 ok.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmEK2voMnBa23495aLk1Ce
Sync of the canonical module in the project repo. forceSSL would have bounced
the gateway's plain-HTTP hop back to the gateway; and the oc vhost's explicit
listen on 100.84.190.30:80 was shadowing this vhost on the tailnet, serving the
opencode UI for Host: atproto.commoninternet.net.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmEK2voMnBa23495aLk1Ce
Imports the notplants-atproto module: Docker daemon (this host had none), a
systemd unit running the project's docker-compose stack, and an nginx vhost
proxying to it on 127.0.0.1:8731 with ACME and websocket support.
Opens 80/443 publicly. Until now only 22 was open and everything else was
tailnet-only, so this is a real change in exposure — nginx is now reachable
from the internet. The existing oc.commoninternet.net vhost is untouched and
stays bound to the tailscale IP.
nix/atproto-likes.nix is a COPY; the canonical file lives in the project repo
at /srv/project-orchestrator/projects/notplants-atproto/nix/. Pure evaluation
forbids importing an absolute path outside the flake tree, so it has to be
duplicated here — and a new file must be git-added or nix silently ignores it.
ACME currently FAILS: *.commoninternet.net is a wildcard pointing at
143.244.213.108, so the HTTP-01 challenge is answered by that host (500).
nginx serves a self-signed placeholder and starts fine. Fix is an explicit
A record atproto.commoninternet.net -> 168.119.126.100, then
`systemctl start acme-atproto.commoninternet.net.service`.
Applied with nixos-rebuild switch; container healthy, TLS proxy and wss
verified end to end.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmEK2voMnBa23495aLk1Ce
GO subscription hit a monthly usage limit (resets in 9 days). Add LOOP_TIER
env (default 'zen') to launch-upgrader.py + launch-report.py so the opencode
backend can use either subscription: zen→opencode/glm-5.2, go→opencode-go/glm-5.2.
The tier selects the default model, the usage-limit probe endpoint, and the
auth.json key. The systemd timer reads LOOP_TIER from /srv/cc-ci/upgrader.env.
Mirrors cc-ci-loops.service for the SEPARATE p-lichen-orchestrator (lichen.page testing). On boot,
resumes its Remote Control session + watchdog + pipeline via engine/agents.py up. Added after the
2026-07-08 reboot (Hetzner rollback of a bad staging nixos-rebuild) left this orchestrator down while
cc-ci-loops auto-recovered. Points at the /home path for now; re-point to /srv at that migration.
Verified: diff-closures shows only this unit added; sshd/dhcpcd/tailscaled/network unchanged.
A run can finish the recipe work + summary + report yet never print a clean
sign-off marker (it may wedge afterwards, as on 2026-07-03). Without a second
completion signal the supervisor would try to 'resume' the already-finished run
every hour until the 96h window closed. Now the gate also treats the run as done
when a weekly summary file (upgrade-all-<date>.md, the Step-5 output) exists with
an mtime after the session began.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
The 2026-07-03 finish run wedged because _completed() returned True while the run
was still mid-work — so the watchdog exited early and nothing recovered the wedge.
Cause: it scanned part.get('text') across ALL message parts, so DONE_MARKER inside
a TOOL part (a subagent `task` prompt / bash command that referenced 'print
UPGRADE RUN COMPLETE') matched. Now: require the marker in the LAST assistant
TEXT (prose) message — the genuine sign-off — ignoring tool-call args and any
mid-run echo of the instruction (work after the echo disqualifies it).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
More fixes from live-running the finish of the 2026-07-03 run:
- _session_idle_min(): measure staleness across the whole session TREE (top-level
run + all descendant subagents), via the opencode server time.updated, NOT the
tmux log mtime (which freezes when a headless run doesn't stream to the pane).
A per-recipe subagent deploy runs 20-40min during which the PARENT session's
updated time is stale — reading the parent alone looked 'idle' and would false-
resume, killing the productive run. Renamed from _log_idle_min (kept as alias).
- watchdog(): wrap each poll in try/except so a transient server blip/race can
never kill the watchdog (a dead watchdog silently abandons the run).
- watchdog + supervisor now read the tree-aware session idle instead of log mtime.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
Live-testing the resume path surfaced two gaps: (1) an `opencode run` proc
EXITS when the model ends its turn, so a long /upgrade-all run's process dies
repeatedly before the whole run completes — and the log mtime freezes on death,
so the watchdog's log-idle>15min signal is both too slow and unreliable. (2) A
resumed run had no watchdog, so nothing re-continued it.
- watchdog(): detect PROC-DEATH (no live `opencode run` proc for the session +
not completed) and resume promptly, in addition to log-idle. Guarded by
MAX_RESUMES (default 20) so a no-progress loop (e.g. disk-full) eventually hands
off to the supervisor/operator instead of spinning forever.
- resume(): auto-spawn a watchdog if none is alive (skips when the watchdog itself
called resume — it lives in {SESSION}-watchdog — so no duplicate).
- launch-supervisor.py gate: defer while the per-run watchdog is alive (it is the
single writer for prompt-recovery). The supervisor only takes over once the
watchdog gives up (MAX_RESUMES) — i.e. a wedge a bare resume can't fix. Removes
the supervisor/watchdog double-resume race.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
Root-cause fix for the 2026-07-03 run stalling: the cc-ci host disk filled to
100% (ENOSPC) mid-run (Wave 6, lasuite-drive), the agent stopped to reclaim
space, and nothing resumed it — the log-idle/429 watchdog only covers opencode-go
usage-limit stalls, not an environmental wedge.
- launch-upgrader.py: step-0 prereclaim_cc_ci() prunes STALE cc-ci docker images
(unused AND older than a week, so this week's likely-reused images stay) before
each weekly run. Best-effort; env-tunable (UPGRADER_PRERECLAIM*).
- launch-supervisor.py (new): hourly glm-5.2 orchestrator wake-up. Cheap
deterministic gate — no-ops (zero tokens) when the run is complete or
progressing; only when a run stalled/died before completing does it launch a
short-lived glm-5.2 agent to diagnose + drive it to a clean DONE. Progress is
judged by live run-proc + log mtime (session_busy() is claude-tuned and misreads
a headless opencode run as idle).
- configuration.nix: cc-ci-upgrade-supervisor service + hourly timer (:07).
- upgrade-all SKILL §0: note the stale-image reclaim for manual runs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
Two gaps for the scheduled Thursday glm-5.2 run:
1. Survival: the watchdog was a Popen child of the Type=oneshot service, which
systemd's cgroup cleanup kills on exit. Spawn it under the persistent tmux
server instead (_spawn_watchdog), like the run sessions — survives the oneshot.
2. The report runs on glm-5.2 sharing the same opencode-go budget the upgrade run
drains, so it can 429-stall with no recovery. launch-report.py now spawns the
SAME watchdog pointed at the cc-ci-report session (generic via UPGRADER_SESSION/
_MODEL/_DONE_MARKER/_RESUME_FILE), with a report-specific resume prompt.
Also: _run_pids() is now scoped to the managed session (title or -s <sid>) so the
report watchdog can't kill the idle upgrader process and vice-versa; resume() adds
--dir and honors a custom resume prompt file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_completed() grepped the log for UPGRADE RUN COMPLETE, but the kickoff/resume
PROMPT (a user message) contains that string verbatim, so it false-positived
'done' while the run was still going. Check the model's ASSISTANT message output
via the web server API instead (log grep only as an offline, prompt-excluding
fallback).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Subcommand, function, env (UPGRADER_WATCHDOG), and log file renamed; behavior
unchanged. Only the opencode upgrader 'start' auto-spawns it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The opencode-go subscription's rolling usage-limit (429) ends the 'opencode run'
agent loop mid-run; it does NOT self-resume. Add:
- resume: continue the SAME session (context preserved) via 'opencode run -s <id>
--continue' — finds the session from the web server, kills the idle proc safely
(via /proc scan, never pkill -f self-match), relaunches in the tmux session.
- babysit: poll the session log; on a stall (>15min idle) wait out any 429
retry-after then auto-resume. Spawned automatically by an opencode 'start'.
So a usage-limit pause now self-heals instead of needing a manual nudge.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make explicit that ALL formatting/HTML is owned by recipe-report.py render() and
the model's only artifact is the spec JSON — never hand-write/edit HTML. Matters
now that glm-5.2 drives the report. Also fix stale 'default opus' refs (report
now defaults to opencode-go/glm-5.2, overridable via REPORT_BACKEND/REPORT_MODEL).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Report launcher now defaults backend=opencode, model=opencode-go/glm-5.2 (claude
override → opus). Replaced the broken opencode 'attach + send-keys' path with the
same 'opencode run -m … --share --attach --title' pattern as the upgrader (the
old path passed no model and injected the prompt via keystrokes).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Weekly upgrade run now defaults backend=opencode, model=opencode-go/glm-5.2 with
no env set. Model default tracks backend (claude override → sonnet). Override via
LOOP_BACKEND/LOOP_MODEL or /srv/cc-ci/upgrader.env.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cc-ci-upgrade-all now reads an optional EnvironmentFile so the weekly run can
switch backend/model (e.g. LOOP_BACKEND=opencode LOOP_MODEL=opencode-go/glm-5.2)
without a rebuild. Absent file → claude/sonnet (unchanged). Built+switched on
cc-ci-orchestrator-hetzner, host verified healthy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The opencode backend emitted 'opencode --model X run ...' but -m/--model is a
flag on the run subcommand, so the model was being ignored. Move it after run.
Add OPENCODE_SHARE (default on): attach the session to the shared opencode web
server (oc.commoninternet.net) AND create a public --share link for monitoring.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Trust abra fully for any image whose tag it can read — a normal semver/calver
tag with no newer version is genuinely up-to-date. Only cross-check upstream for
images abra physically can't parse (tag+digest pins, digest-only pins), which is
the actual immich blind spot. Avoids redundant upstream checks on every recipe.
abra recipe upgrade is the first approach, but it silently contributes no
candidate for tag+digest pins (FATA: tag and digest not supported), digest-only
pins, and non-semver tags. immich kept getting skipped this way. Before
concluding SKIPPED — up-to-date, do a direct upstream tag check for every image
abra could not cleanly evaluate; only skip when BOTH agree nothing is newer.
cc-ci recipe-upgrade skill now computes the version via 'abra recipe release --dry-run'
(not a hand-edit) and requires the PR body to link upstream release notes per service.
Bumps the recipe-maintainer submodule pointer to the matching change.
Operator: uptime-kuma is maintained elsewhere — drop it from the weekly upgrade
but keep it in the used-recipes inventory. New cc-ci-plan/used-recipes.md is the
canonical list of every recipe cc-ci deploys/tests, with a weekly|external tier;
upgrade-all §1 now excludes 'external' rows from the candidate list (explicit
--args still override). uptime-kuma = external; all others weekly.
Re-target the traefik health gate off ci.commoninternet.net (the dashboard,
which is After=deploy-proxy) onto a traefik-self endpoint, breaking the
fresh-boot deadlock while keeping health-gated rollback. M1 controlled repro by
the loops; M2 from-scratch cold-boot proof owned by the orchestrator.
The unified agents.py watchdog kept firing the hourly orchestrator supervision
ping even after SEQUENCE-COMPLETE (the old launch.py watchdog exited on
completion, which stopped them). Gate the wake loop on the SEQUENCE-COMPLETE
marker so a finished build stays fully at rest — no pings. Resumes
automatically when new work is queued (that clears the marker, line 631).
The durable /16 proxy fix landed in phase pvfix (2026-06-13).
Update the guard description from "safety net until that lands"
to "belt-and-suspenders even after the /16 fix" — guard logic
unchanged, description now accurate.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The unification transcribed phases from .phases-spec before cf48 was added,
so the operator's just-requested opus 4.8 cfold review got dropped. Re-append
it after ghost (system is past cf55/on pvfix, so can't insert before pvfix
without shifting the live phase index). agents.py re-reads config each tick.
Second independent review of the cfold custom-folder collapse, by Opus 4.8
instead of GPT-5.5, inserted after cf55 (queue ...cfold;cf55;cf48;pvfix;...).
Per-phase overrides .loop-model[-adv]-cf48=claude-opus-4-8 on the claude backend.
autonomic-bot
