cc-ci-orchestrator/memory/weekly-upgrader-opencode-go-glm.md

name, description, metadata

name	description	metadata
weekly-upgrader-opencode-go-glm	weekly upgrade run now defaults to opencode-go/glm-5.3-flash (PR	node_type type originSessionId memory project 85355980-5e4f-4f90-b1ca-d0e4fe82f04b

The weekly recipe-upgrade run (cc-ci-plan/launch-upgrader.py, fired by the cc-ci-upgrade-all systemd timer Thu 22:00 Boston) now defaults to backend=opencode, model=opencode-go/glm-5.3-flash when no env is set (operator choice 2026-09-08 via /srv/cc-ci/upgrader.env; PR #20 hardcoded the same defaults into the launchers 2026-09-14 — for all three launchers: upgrader, report, supervisor). Model default tracks the backend — an explicit LOOP_BACKEND=claude (no model) falls back to sonnet.

2026-09-14 — ZEN IS DEAD ON THIS HOST. The ZEN provider models (opencode/glm-5.2) are gone from the provider catalogue AND the host holds no ZEN key, so any launch that bypassed upgrader.env died instantly with ProviderModelNotFoundError: opencode/glm-5.2 — this is what killed the 2026-09-11 report agent at launch and every manual retry (all valid ZEN credentials mean nothing without the ZEN models existing server-side). If a launcher log shows ProviderModelNotFoundError, the model id does not resolve on this host: run opencode models | grep glm for what does, then fix the tier/model default. See PR recipe-maintainers/cc-ci-orchestrator#20.

Provider: "OpenCode Go" — a flat $10/mo subscription, DISTINCT from "opencode Zen" (pay-as-you-go). Endpoint https://opencode.ai/zen/go/v1/ (OpenAI-compatible), model id opencode-go/glm-5.2 (Zen tops out at glm-5.1 — opencode.ai/zen/v1 does NOT have 5.2; updating the opencode CLI does NOT change this, the catalog is server-side). API key lives in ~/.local/share/opencode/auth.json (mode 600, entry opencode-go type api) — never in the repo. opencode CLI installed: 1.15.13.

Override per run without a rebuild: drop /srv/cc-ci/upgrader.env (read via EnvironmentFile=- on the unit) with e.g. LOOP_BACKEND=claude / LOOP_MODEL=sonnet. Absent file → the glm-5.2 default.

Monitoring (opencode backend): the launcher does opencode run -m <model> --share --attach http://127.0.0.1:4096. opencode-web.service (=opencode serve :4096, nginx → http://oc.commoninternet.net, tailnet-only) must be active. OPENCODE_SHARE=1 (default) also mints a public https://opncd.ai/share/<id> link (printed in the TUI / readable from the server API GET /session/<id> → .share.url). Note: launch-upgrader.py's session_busy() regex is claude-tuned, so it misreports an active opencode run as "idle/finishing" — check the process (opencode run) or web UI instead.

Gotcha fixed: opencode -m/--model/--attach/--share/--title are flags on the run SUBCOMMAND — they must come AFTER run (a global opencode --model X run is silently ignored).

launch-report.py also defaults to opencode-go/glm-5.2 now (overridable via REPORT_BACKEND/REPORT_MODEL). In the weekly flow the upgrader writes the summary inline AND auto-launches the report (upgrade-all §6).

Usage limit + auto-resume watchdog (added 2026-06-23): opencode-go enforces a ROLLING usage window (~5h) capped by dollar value. A full glm-5.2 upgrade run is heavy — ~8 recipes (≈2h20m wall) exhausted it mid-run, returning HTTP 429 + retry-after (seconds) on the go endpoint. When limited, the opencode run agent loop ENDS and does NOT self-resume (process lingers but idle). Fix: launch-upgrader.py now has resume (continue the SAME session, context preserved, via opencode run -s <id> --continue — finds the session from the web server, kills the idle proc via /proc scan NOT pkill -f which self-matches) and watchdog (polls the session log; on >15min stall, waits out the 429 retry-after then auto-resumes). An opencode start auto-spawns the watchdog; logs to cc-ci-upgrader-watchdog.log. To resume by hand: python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py resume. (Subcommand watchdog, matching agents.py — renamed from babysit 2026-06-23.)

Watchdog covers ALL parts of the weekly run (2026-06-23): (1) it runs in a tmux session (<session>-watchdog) under the persistent tmux server, NOT a Popen child — so it survives the cc-ci-upgrade-all systemd Type=oneshot whose cgroup is reaped on exit (a Popen child would be killed). (2) The watchdog is generic (env: UPGRADER_SESSION/_MODEL/_DONE_MARKER/_RESUME_FILE), so launch-report.py reuses it for the REPORT job (cc-ci-report session, marker RECIPE REPORT COMPLETE) — the report runs on glm-5.2 sharing the same opencode-go budget the upgrade run drains, so it can 429-stall too. _run_pids() is scoped to the managed session (title or -s <sid>) so the two watchdogs never kill each other's process. Weekly timer: Thu 22:00 America/New_York; no LOOP_BACKEND/upgrader.env override ⇒ glm-5.2 defaults. See orchestrator-backend-switch-gotcha.

2026-07-04 hardening (after the 2026-07-03 run stalled on a HOST DISK-FULL and never resumed — the log-idle/429 watchdog does NOT cover an environmental wedge). Three changes (commits 1bd156e, 399e999), host rebuilt:

• Step-0 stale-image pre-reclaim (launch-upgrader.py prereclaim_cc_ci()): before each weekly run, ssh cc-ci docker image prune -af --filter until=168h — only images unused AND >1 week old, so this week's likely-reused images stay. Best-effort; env UPGRADER_PRERECLAIM*. A full run's images pile up week-over-week and had run root to 100%. (The 2026-07-03 pile was 93 GB / 194 images; pruning took it 88%→17%.)
• watchdog proc-death detection: an opencode run proc EXITS when the model ends its turn, so a long /upgrade-all dies+restarts many times before completing (and the log mtime freezes on death, making log-idle useless). watchdog() now resumes on "no live run proc for the session + not _completed()", not just log-idle. Capped at UPGRADER_MAX_RESUMES (20) → hands off if stuck. resume() now auto-spawns a watchdog if none is alive (skips when the watchdog itself called it).
• Hourly glm-5.2 SUPERVISOR (launch-supervisor.py, systemd cc-ci-upgrade-supervisor .service + .timer, hourly at :07): cheap deterministic gate — no-ops (0 tokens) if the run is complete / progressing / a per-run watchdog is alive; only when a run stalled+died AND the watchdog gave up does it launch a short-lived glm-5.2 agent to diagnose (disk/wedge) + drive to completion. force/status/ stop subcommands. Progress judged by lu._run_pids(sid) + log mtime (session_busy() is claude-tuned, misreads a headless opencode run as idle — do NOT use it for opencode). Window guard: won't auto-resurrect a session >96h old. Overrides via /srv/cc-ci/upgrader.env (e.g. SUPERVISOR_MODEL=). Ownership: watchdog = prompt recovery (single writer while alive); supervisor = backstop when the watchdog is absent/gave up. They're mutually exclusive (gate defers while watchdog tmux alive).

2026-07-04 finish + more watchdog fixes (commits 69dd178, ddedea8, b6a95c4). Finishing the stalled 2026-07-03 run exposed that a glm-5.2 opencode run is FLAKY for long autonomous jobs — its turn ENDS (proc exits) repeatedly, --continue of a bloated session often doesn't advance, and it can wedge on a hung subagent. Bugs fixed in launch-upgrader.py: (1) _session_idle_min() now measures staleness across the whole session TREE (parent + descendant subagents) via the opencode server time.updated — NOT the tmux log mtime (freezes on --continue) and NOT the parent alone (a 20-40min recipe subagent made the parent look idle → false-resume killed the productive run). (2) watchdog loop is wrapped in try/except so a transient blip can't kill it. (3) _completed() false-positive: it matched DONE_MARKER inside TOOL parts (a subagent task prompt / bash echoing "print UPGRADE RUN COMPLETE") → watchdog exited early, run wedged. Now requires the marker in the LAST assistant TEXT (prose) message. (4) supervisor gate also treats the run as done once the weekly summary file exists (a run can finish work+summary+report yet never print a clean marker). Operational fallback that WORKED: when the report agent wedged twice, published the report by hand from the deterministic path — recipe-report.py survey <date> → build the spec JSON from survey.json → render → publish. This is reliable and bypasses the flaky agent entirely; keep it as the go-to when an agent-driven report stalls. The 2026-07-03 report is live at report.ci.commoninternet.net/week-2026-07-03.html (16 GREEN, plausible + lasuite-drive RED, discourse 5-PR reconcile).