memory: weekly upgrader — go tier + glm-5.3-flash defaults, ZEN dead on this host (PR #20 context)

memory/weekly-upgrader-opencode-go-glm.md

@@ -0,0 +1,111 @@
+---
+name: weekly-upgrader-opencode-go-glm
+description: weekly upgrade run now defaults to opencode-go/glm-5.3-flash (PR #20, 2026-09-14); zen tier dead on this host; how backend/model + monitoring are wired
+metadata: 
+  node_type: memory
+  type: project
+  originSessionId: 85355980-5e4f-4f90-b1ca-d0e4fe82f04b
+---
+
+The weekly recipe-upgrade run (`cc-ci-plan/launch-upgrader.py`, fired by the
+`cc-ci-upgrade-all` systemd timer Thu 22:00 Boston) now **defaults to backend=opencode,
+model=opencode-go/glm-5.3-flash** when no env is set (operator choice 2026-09-08 via
+`/srv/cc-ci/upgrader.env`; PR #20 hardcoded the same defaults into the launchers 2026-09-14 —
+for all three launchers: upgrader, report, supervisor). Model default tracks the
+backend — an explicit `LOOP_BACKEND=claude` (no model) falls back to `sonnet`.
+
+**2026-09-14 — ZEN IS DEAD ON THIS HOST.** The ZEN provider models (`opencode/glm-5.2`) are
+gone from the provider catalogue AND the host holds no ZEN key, so any launch that bypassed
+upgrader.env died instantly with `ProviderModelNotFoundError: opencode/glm-5.2` — this is what
+killed the 2026-09-11 report agent at launch and every manual retry (all valid ZEN credentials
+mean nothing without the ZEN models existing server-side). If a launcher log shows
+`ProviderModelNotFoundError`, the model id does not resolve on this host: run
+`opencode models | grep glm` for what does, then fix the tier/model default. See PR
+recipe-maintainers/cc-ci-orchestrator#20.
+
+**Provider:** "OpenCode **Go**" — a flat $10/mo subscription, DISTINCT from "opencode Zen"
+(pay-as-you-go). Endpoint `https://opencode.ai/zen/go/v1/` (OpenAI-compatible), model id
+**`opencode-go/glm-5.2`** (Zen tops out at glm-5.1 — `opencode.ai/zen/v1` does NOT have 5.2;
+updating the opencode CLI does NOT change this, the catalog is server-side). API key lives in
+`~/.local/share/opencode/auth.json` (mode 600, entry `opencode-go` type api) — **never in the
+repo**. opencode CLI installed: 1.15.13.
+
+**Override per run without a rebuild:** drop `/srv/cc-ci/upgrader.env` (read via
+`EnvironmentFile=-` on the unit) with e.g. `LOOP_BACKEND=claude` / `LOOP_MODEL=sonnet`. Absent
+file → the glm-5.2 default.
+
+**Monitoring (opencode backend):** the launcher does `opencode run -m <model> --share --attach
+http://127.0.0.1:4096`. `opencode-web.service` (=`opencode serve` :4096, nginx → 
+http://oc.commoninternet.net, tailnet-only) must be active. `OPENCODE_SHARE=1` (default) also
+mints a public `https://opncd.ai/share/<id>` link (printed in the TUI / readable from the server
+API `GET /session/<id>` → `.share.url`). Note: launch-upgrader.py's `session_busy()` regex is
+claude-tuned, so it misreports an active opencode run as "idle/finishing" — check the process
+(`opencode run`) or web UI instead.
+
+Gotcha fixed: opencode `-m/--model`/`--attach`/`--share`/`--title` are flags on the `run`
+SUBCOMMAND — they must come AFTER `run` (a global `opencode --model X run` is silently ignored).
+
+`launch-report.py` also defaults to opencode-go/glm-5.2 now (overridable via REPORT_BACKEND/REPORT_MODEL).
+In the weekly flow the upgrader writes the summary inline AND auto-launches the report (upgrade-all §6).
+
+**Usage limit + auto-resume watchdog (added 2026-06-23):** opencode-go enforces a ROLLING usage window
+(~5h) capped by dollar value. A full glm-5.2 upgrade run is heavy — ~8 recipes (≈2h20m wall) exhausted
+it mid-run, returning HTTP 429 + `retry-after` (seconds) on the go endpoint. When limited, the
+`opencode run` agent loop ENDS and does NOT self-resume (process lingers but idle). Fix: `launch-upgrader.py`
+now has `resume` (continue the SAME session, context preserved, via `opencode run -s <id> --continue` —
+finds the session from the web server, kills the idle proc via /proc scan NOT `pkill -f` which self-matches)
+and `watchdog` (polls the session log; on >15min stall, waits out the 429 retry-after then auto-resumes).
+An opencode `start` auto-spawns the watchdog; logs to `cc-ci-upgrader-watchdog.log`. To resume by hand:
+`python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py resume`. (Subcommand `watchdog`, matching agents.py —
+renamed from `babysit` 2026-06-23.)
+
+**Watchdog covers ALL parts of the weekly run (2026-06-23):** (1) it runs in a tmux session
+(`<session>-watchdog`) under the persistent tmux server, NOT a Popen child — so it survives the
+`cc-ci-upgrade-all` systemd Type=oneshot whose cgroup is reaped on exit (a Popen child would be killed).
+(2) The watchdog is generic (env: `UPGRADER_SESSION/_MODEL/_DONE_MARKER/_RESUME_FILE`), so `launch-report.py`
+reuses it for the REPORT job (`cc-ci-report` session, marker `RECIPE REPORT COMPLETE`) — the report runs on
+glm-5.2 sharing the same opencode-go budget the upgrade run drains, so it can 429-stall too. `_run_pids()`
+is scoped to the managed session (title or `-s <sid>`) so the two watchdogs never kill each other's process.
+Weekly timer: Thu 22:00 America/New_York; no `LOOP_BACKEND`/`upgrader.env` override ⇒ glm-5.2 defaults.
+See [[orchestrator-backend-switch-gotcha]].
+
+**2026-07-04 hardening (after the 2026-07-03 run stalled on a HOST DISK-FULL and never resumed —
+the log-idle/429 watchdog does NOT cover an environmental wedge).** Three changes (commits 1bd156e,
+399e999), host rebuilt:
+- **Step-0 stale-image pre-reclaim** (`launch-upgrader.py` `prereclaim_cc_ci()`): before each weekly
+  run, `ssh cc-ci docker image prune -af --filter until=168h` — only images unused AND >1 week old, so
+  this week's likely-reused images stay. Best-effort; env `UPGRADER_PRERECLAIM*`. A full run's images
+  pile up week-over-week and had run root to 100%. (The 2026-07-03 pile was 93 GB / 194 images; pruning
+  took it 88%→17%.)
+- **watchdog proc-death detection**: an `opencode run` proc EXITS when the model ends its turn, so a
+  long /upgrade-all dies+restarts many times before completing (and the log mtime freezes on death,
+  making log-idle useless). `watchdog()` now resumes on "no live run proc for the session + not
+  `_completed()`", not just log-idle. Capped at `UPGRADER_MAX_RESUMES` (20) → hands off if stuck.
+  `resume()` now auto-spawns a watchdog if none is alive (skips when the watchdog itself called it).
+- **Hourly glm-5.2 SUPERVISOR** (`launch-supervisor.py`, systemd `cc-ci-upgrade-supervisor` .service +
+  .timer, hourly at :07): cheap deterministic gate — no-ops (0 tokens) if the run is complete /
+  progressing / a per-run watchdog is alive; only when a run stalled+died AND the watchdog gave up does
+  it launch a short-lived glm-5.2 agent to diagnose (disk/wedge) + drive to completion. `force`/`status`/
+  `stop` subcommands. Progress judged by `lu._run_pids(sid)` + log mtime (session_busy() is claude-tuned,
+  misreads a headless opencode run as idle — do NOT use it for opencode). Window guard: won't
+  auto-resurrect a session >96h old. Overrides via `/srv/cc-ci/upgrader.env` (e.g. `SUPERVISOR_MODEL=`).
+  Ownership: **watchdog = prompt recovery (single writer while alive); supervisor = backstop when the
+  watchdog is absent/gave up.** They're mutually exclusive (gate defers while watchdog tmux alive).
+
+**2026-07-04 finish + more watchdog fixes (commits 69dd178, ddedea8, b6a95c4).** Finishing the stalled
+2026-07-03 run exposed that a glm-5.2 `opencode run` is FLAKY for long autonomous jobs — its turn ENDS
+(proc exits) repeatedly, `--continue` of a bloated session often doesn't advance, and it can wedge on a
+hung subagent. Bugs fixed in `launch-upgrader.py`: (1) `_session_idle_min()` now measures staleness
+across the whole session TREE (parent + descendant subagents) via the opencode server time.updated —
+NOT the tmux log mtime (freezes on `--continue`) and NOT the parent alone (a 20-40min recipe subagent
+made the parent look idle → false-resume killed the productive run). (2) watchdog loop is wrapped in
+try/except so a transient blip can't kill it. (3) **`_completed()` false-positive**: it matched
+DONE_MARKER inside TOOL parts (a subagent `task` prompt / bash echoing "print UPGRADE RUN COMPLETE") →
+watchdog exited early, run wedged. Now requires the marker in the LAST assistant TEXT (prose) message.
+(4) supervisor gate also treats the run as done once the weekly summary file exists (a run can finish
+work+summary+report yet never print a clean marker). **Operational fallback that WORKED:** when the
+report agent wedged twice, published the report by hand from the deterministic path —
+`recipe-report.py survey <date>` → build the spec JSON from survey.json → `render` → `publish`. This
+is reliable and bypasses the flaky agent entirely; keep it as the go-to when an agent-driven report
+stalls. The 2026-07-03 report is live at report.ci.commoninternet.net/week-2026-07-03.html (16 GREEN,
+plausible + lasuite-drive RED, discourse 5-PR reconcile).