112 lines
8.6 KiB
Markdown
112 lines
8.6 KiB
Markdown
---
|
|
name: weekly-upgrader-opencode-go-glm
|
|
description: weekly upgrade run now defaults to opencode-go/glm-5.3-flash (PR #20, 2026-09-14); zen tier dead on this host; how backend/model + monitoring are wired
|
|
metadata:
|
|
node_type: memory
|
|
type: project
|
|
originSessionId: 85355980-5e4f-4f90-b1ca-d0e4fe82f04b
|
|
---
|
|
|
|
The weekly recipe-upgrade run (`cc-ci-plan/launch-upgrader.py`, fired by the
|
|
`cc-ci-upgrade-all` systemd timer Thu 22:00 Boston) now **defaults to backend=opencode,
|
|
model=opencode-go/glm-5.3-flash** when no env is set (operator choice 2026-09-08 via
|
|
`/srv/cc-ci/upgrader.env`; PR #20 hardcoded the same defaults into the launchers 2026-09-14 —
|
|
for all three launchers: upgrader, report, supervisor). Model default tracks the
|
|
backend — an explicit `LOOP_BACKEND=claude` (no model) falls back to `sonnet`.
|
|
|
|
**2026-09-14 — ZEN IS DEAD ON THIS HOST.** The ZEN provider models (`opencode/glm-5.2`) are
|
|
gone from the provider catalogue AND the host holds no ZEN key, so any launch that bypassed
|
|
upgrader.env died instantly with `ProviderModelNotFoundError: opencode/glm-5.2` — this is what
|
|
killed the 2026-09-11 report agent at launch and every manual retry (all valid ZEN credentials
|
|
mean nothing without the ZEN models existing server-side). If a launcher log shows
|
|
`ProviderModelNotFoundError`, the model id does not resolve on this host: run
|
|
`opencode models | grep glm` for what does, then fix the tier/model default. See PR
|
|
recipe-maintainers/cc-ci-orchestrator#20.
|
|
|
|
**Provider:** "OpenCode **Go**" — a flat $10/mo subscription, DISTINCT from "opencode Zen"
|
|
(pay-as-you-go). Endpoint `https://opencode.ai/zen/go/v1/` (OpenAI-compatible), model id
|
|
**`opencode-go/glm-5.2`** (Zen tops out at glm-5.1 — `opencode.ai/zen/v1` does NOT have 5.2;
|
|
updating the opencode CLI does NOT change this, the catalog is server-side). API key lives in
|
|
`~/.local/share/opencode/auth.json` (mode 600, entry `opencode-go` type api) — **never in the
|
|
repo**. opencode CLI installed: 1.15.13.
|
|
|
|
**Override per run without a rebuild:** drop `/srv/cc-ci/upgrader.env` (read via
|
|
`EnvironmentFile=-` on the unit) with e.g. `LOOP_BACKEND=claude` / `LOOP_MODEL=sonnet`. Absent
|
|
file → the glm-5.2 default.
|
|
|
|
**Monitoring (opencode backend):** the launcher does `opencode run -m <model> --share --attach
|
|
http://127.0.0.1:4096`. `opencode-web.service` (=`opencode serve` :4096, nginx →
|
|
http://oc.commoninternet.net, tailnet-only) must be active. `OPENCODE_SHARE=1` (default) also
|
|
mints a public `https://opncd.ai/share/<id>` link (printed in the TUI / readable from the server
|
|
API `GET /session/<id>` → `.share.url`). Note: launch-upgrader.py's `session_busy()` regex is
|
|
claude-tuned, so it misreports an active opencode run as "idle/finishing" — check the process
|
|
(`opencode run`) or web UI instead.
|
|
|
|
Gotcha fixed: opencode `-m/--model`/`--attach`/`--share`/`--title` are flags on the `run`
|
|
SUBCOMMAND — they must come AFTER `run` (a global `opencode --model X run` is silently ignored).
|
|
|
|
`launch-report.py` also defaults to opencode-go/glm-5.2 now (overridable via REPORT_BACKEND/REPORT_MODEL).
|
|
In the weekly flow the upgrader writes the summary inline AND auto-launches the report (upgrade-all §6).
|
|
|
|
**Usage limit + auto-resume watchdog (added 2026-06-23):** opencode-go enforces a ROLLING usage window
|
|
(~5h) capped by dollar value. A full glm-5.2 upgrade run is heavy — ~8 recipes (≈2h20m wall) exhausted
|
|
it mid-run, returning HTTP 429 + `retry-after` (seconds) on the go endpoint. When limited, the
|
|
`opencode run` agent loop ENDS and does NOT self-resume (process lingers but idle). Fix: `launch-upgrader.py`
|
|
now has `resume` (continue the SAME session, context preserved, via `opencode run -s <id> --continue` —
|
|
finds the session from the web server, kills the idle proc via /proc scan NOT `pkill -f` which self-matches)
|
|
and `watchdog` (polls the session log; on >15min stall, waits out the 429 retry-after then auto-resumes).
|
|
An opencode `start` auto-spawns the watchdog; logs to `cc-ci-upgrader-watchdog.log`. To resume by hand:
|
|
`python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py resume`. (Subcommand `watchdog`, matching agents.py —
|
|
renamed from `babysit` 2026-06-23.)
|
|
|
|
**Watchdog covers ALL parts of the weekly run (2026-06-23):** (1) it runs in a tmux session
|
|
(`<session>-watchdog`) under the persistent tmux server, NOT a Popen child — so it survives the
|
|
`cc-ci-upgrade-all` systemd Type=oneshot whose cgroup is reaped on exit (a Popen child would be killed).
|
|
(2) The watchdog is generic (env: `UPGRADER_SESSION/_MODEL/_DONE_MARKER/_RESUME_FILE`), so `launch-report.py`
|
|
reuses it for the REPORT job (`cc-ci-report` session, marker `RECIPE REPORT COMPLETE`) — the report runs on
|
|
glm-5.2 sharing the same opencode-go budget the upgrade run drains, so it can 429-stall too. `_run_pids()`
|
|
is scoped to the managed session (title or `-s <sid>`) so the two watchdogs never kill each other's process.
|
|
Weekly timer: Thu 22:00 America/New_York; no `LOOP_BACKEND`/`upgrader.env` override ⇒ glm-5.2 defaults.
|
|
See [[orchestrator-backend-switch-gotcha]].
|
|
|
|
**2026-07-04 hardening (after the 2026-07-03 run stalled on a HOST DISK-FULL and never resumed —
|
|
the log-idle/429 watchdog does NOT cover an environmental wedge).** Three changes (commits 1bd156e,
|
|
399e999), host rebuilt:
|
|
- **Step-0 stale-image pre-reclaim** (`launch-upgrader.py` `prereclaim_cc_ci()`): before each weekly
|
|
run, `ssh cc-ci docker image prune -af --filter until=168h` — only images unused AND >1 week old, so
|
|
this week's likely-reused images stay. Best-effort; env `UPGRADER_PRERECLAIM*`. A full run's images
|
|
pile up week-over-week and had run root to 100%. (The 2026-07-03 pile was 93 GB / 194 images; pruning
|
|
took it 88%→17%.)
|
|
- **watchdog proc-death detection**: an `opencode run` proc EXITS when the model ends its turn, so a
|
|
long /upgrade-all dies+restarts many times before completing (and the log mtime freezes on death,
|
|
making log-idle useless). `watchdog()` now resumes on "no live run proc for the session + not
|
|
`_completed()`", not just log-idle. Capped at `UPGRADER_MAX_RESUMES` (20) → hands off if stuck.
|
|
`resume()` now auto-spawns a watchdog if none is alive (skips when the watchdog itself called it).
|
|
- **Hourly glm-5.2 SUPERVISOR** (`launch-supervisor.py`, systemd `cc-ci-upgrade-supervisor` .service +
|
|
.timer, hourly at :07): cheap deterministic gate — no-ops (0 tokens) if the run is complete /
|
|
progressing / a per-run watchdog is alive; only when a run stalled+died AND the watchdog gave up does
|
|
it launch a short-lived glm-5.2 agent to diagnose (disk/wedge) + drive to completion. `force`/`status`/
|
|
`stop` subcommands. Progress judged by `lu._run_pids(sid)` + log mtime (session_busy() is claude-tuned,
|
|
misreads a headless opencode run as idle — do NOT use it for opencode). Window guard: won't
|
|
auto-resurrect a session >96h old. Overrides via `/srv/cc-ci/upgrader.env` (e.g. `SUPERVISOR_MODEL=`).
|
|
Ownership: **watchdog = prompt recovery (single writer while alive); supervisor = backstop when the
|
|
watchdog is absent/gave up.** They're mutually exclusive (gate defers while watchdog tmux alive).
|
|
|
|
**2026-07-04 finish + more watchdog fixes (commits 69dd178, ddedea8, b6a95c4).** Finishing the stalled
|
|
2026-07-03 run exposed that a glm-5.2 `opencode run` is FLAKY for long autonomous jobs — its turn ENDS
|
|
(proc exits) repeatedly, `--continue` of a bloated session often doesn't advance, and it can wedge on a
|
|
hung subagent. Bugs fixed in `launch-upgrader.py`: (1) `_session_idle_min()` now measures staleness
|
|
across the whole session TREE (parent + descendant subagents) via the opencode server time.updated —
|
|
NOT the tmux log mtime (freezes on `--continue`) and NOT the parent alone (a 20-40min recipe subagent
|
|
made the parent look idle → false-resume killed the productive run). (2) watchdog loop is wrapped in
|
|
try/except so a transient blip can't kill it. (3) **`_completed()` false-positive**: it matched
|
|
DONE_MARKER inside TOOL parts (a subagent `task` prompt / bash echoing "print UPGRADE RUN COMPLETE") →
|
|
watchdog exited early, run wedged. Now requires the marker in the LAST assistant TEXT (prose) message.
|
|
(4) supervisor gate also treats the run as done once the weekly summary file exists (a run can finish
|
|
work+summary+report yet never print a clean marker). **Operational fallback that WORKED:** when the
|
|
report agent wedged twice, published the report by hand from the deterministic path —
|
|
`recipe-report.py survey <date>` → build the spec JSON from survey.json → `render` → `publish`. This
|
|
is reliable and bypasses the flaky agent entirely; keep it as the go-to when an agent-driven report
|
|
stalls. The 2026-07-03 report is live at report.ci.commoninternet.net/week-2026-07-03.html (16 GREEN,
|
|
plausible + lasuite-drive RED, discourse 5-PR reconcile).
|