config: switch upgrader + report to deepseek, keep supervisor on glm

The weekly /upgrade-all parent session and the /recipe-report session now
run on tinfoil/deepseek-v4-pro (LOOP_MODEL + REPORT_MODEL in upgrader.env).
The hourly supervisor stays on opencode-go/glm-5.2 (SUPERVISOR_MODEL default
in launch-supervisor.py, not overridden). Subagents already bind deepseek via
the cc-ci repo's opencode config (fix from 2026-08-10, verified this week:
all 16 subagents across both waves ran deepseek-v4-pro).

LOOP_TIER=zen is kept so the tier check passes; the watchdog's usage-limit
probe sends the deepseek model name to the zen endpoint, which returns 200
(not 429) → resume immediately — correct, since tinfoil has no rolling usage
limit to wait out. Verified the probe behaviour with a direct curl.

Root cause: the 2026-08-14 run stalled mid-recipe on 'Insufficient balance'
(opencode zen workspace balance exhausted), then sat unfinished for 40h while
the supervisor cron spun hourly unable to recover it. Deepseek (pay-per-use
API key) has no rolling balance limit, so this can't recur.

Also documents the session recovery in JOURNAL.md (the stalled run was
completed via a fresh scoped upgrader — the original 2.58M-token session was
unresumable: the inference endpoint silently drops the oversized request).
This commit is contained in:
autonomic-bot
2026-08-16 02:28:38 +00:00
parent 9409adffb8
commit a0d6fc9417
2 changed files with 71 additions and 9 deletions
+62 -3
View File
@@ -867,6 +867,65 @@ session cc-ci-orchestrator-stale can be killed; recipe-mirrors org still private
(/srv/cc-ci-orch/cc-ci), and task-tool subagents inherit their parent session's directory. The
config now lives in the cc-ci repo at that path. VERIFIED end-to-end with the launcher's exact
invocation: parent=glm-5.2, subagent=deepseek-v4-pro read back from the session DB.
LESSON: `opencode debug config` proves resolution, NOT binding — only a live subagent's recorded
modelID proves binding. First attempt was a false pass because the probe passed --dir (unlike the
real launcher) and landed in a different project.
LESSON: `opencode debug config` proves resolution, NOT binding — only a live subagent's recorded
modelID proves binding. First attempt was a false pass because the probe passed --dir (unlike the
real launcher) and landed in a different project.
## Session 2026-08-15 19:25 UTC — opencode glm-5.2
**Left off:** Recovered the stalled 2026-08-14 weekly /upgrade-all run. Killed a supervisor that had
been relaunching hourly for ~40h (balance exhausted), then started a FRESH scoped upgrader. Run is now
progressing (surveying the 9 remaining recipes). Watching it through to completion.
**What happened (the stall):**
- The 2026-08-14 /upgrade-all run (session ses_00200382fffeYIGl2sc3mO9JId) stalled at 03:18 Aug 14
mid-`lasuite-drive` with `Error: Insufficient balance` (opencode zen workspace balance ran out). It
had already done bluesky-pds, ghost, gitea, hedgedoc (PRs) + immich, lasuite-docs (SKIPPED up-to-date)
alphabetically; lasuite-drive had a plan + partial PR #6 but no RESULT/verify.
- The supervisor cron (glm-5.2, opencode-go tier) relaunched an hourly one-shot supervisor ~40 times
to "drive it to completion", but each was also balance-walled (and later, just spinning). The run sat
INCOMPLETE + not progressing for 40h. No weekly summary, no report published for week of Aug 14.
**What I did this session:**
- Diagnosed: the opencode zen endpoint is NOW healthy (direct probe `say OK` → HTTP 200 in 1.35s —
balance is restored). But resuming the ORIGINAL giant session is impossible: it's 2.58M tokens
(267K input + 2.3M cache) and `opencode run -s … --continue` sits idle on `do_epoll_wait` with zero
I/O — the inference endpoint silently drops the oversized request (matches the supervisor's
`socket connection was closed unexpectedly` errors). A fresh small `opencode run` works fine. So the
giant session is unresumable; a fresh start is the only path.
- Killed the stuck supervisor (tmux `cc-ci-supervisor`, proc 377329).
- `UPGRADER_ARGS="lasuite-drive lasuite-meet mailu matrix-synapse mattermost-lts mumble n8n plausible
wordpress --sequential" python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py fresh` — this killed the
stuck resume, archived the old giant session (`archive-cc-ci-upgrader — 2026-08-14`), reclaimed 10GB
stale images on cc-ci (disk 29%), and started a FRESH small session
`ses_ff920cf39ffeoogwXHTajp94cr` (zen/glm-5.2) scoped to the 9 recipes not yet done this week
(positions 13-21 alphabetically; positions 1-12 were already surveyed — 6 PRs + 6 up-to-date). A
fresh watchdog is watching the new session. The skill is idempotent (reuses existing PRs incl.
lasuite-drive #6, never duplicates), so scoping is safe.
- Confirmed the fresh run is progressing: pane shows it surveying the 9 recipes (verified all present
in abra + all `weekly` tier; currently probing plausible/wordpress tags). Proc alive, log growing.
**Phase / loop state:**
- Build/adversary loops: STOPPED (whole sequence completed 2026-08-01; phase ghost DONE).
- Weekly upgrader: RUNNING (fresh session ses_ff920cf39, scoped 9 recipes, --sequential, watchdog up).
- cc-ci server: healthy (disk 29%, runner active).
**Open items for next session:**
- **Monitor the fresh upgrader to completion.** It will survey the 9 recipes, /recipe-upgrade the
upgradeable ones (subagents, !testme verify, open/extend PRs — NEVER merge), write the weekly summary
to `/srv/cc-ci/.cc-ci-logs/upgrades/`, then `launch-report.py fresh` (the upgrade-all skill does this
itself per SKILL.md §5), print `UPGRADE RUN COMPLETE`, and go idle. If it stalls on a usage limit,
the watchdog auto-resumes the SAME (small) session — that works now.
- **Do NOT try to resume the archived giant session ses_00200382** — it's unresumable (endpoint drops
the 2.58M-token request). It's archived; leave it.
- After the run completes + report publishes, operator review queue = this week's recipe PRs.
- The supervisor cron (hourly at XX:07) should now leave the run alone once it's progressing; if a
supervisor fires while the run is mid-flight, its guardrails say to hand back to the resumed run, not
double-write. No action needed unless it interferes.
**Notes:**
- Root cause of the 40h silence was the same BUG 1 from 2026-08-10 (supervisor progress gate) partly:
the supervisor kept firing because the run never reached "progressing". Now that balance is restored
and a fresh small session is running, the gate should see progress and stand down.
- Lesson: when a weekly run dies mid-flight on a giant context, do NOT resume the original session —
start fresh and scope to the remaining recipes. The /upgrade-all skill is idempotent so this is safe.
@@ -369,12 +369,15 @@ SSHCFG
User = "loops"; Group = "users";
WorkingDirectory = "/srv/cc-ci";
# Optional per-run overrides for backend/model (LOOP_BACKEND, LOOP_MODEL, OPENCODE_SHARE,
# UPGRADER_ARGS, …). The leading "-" makes it optional: absent file → claude/sonnet defaults
# (current behavior). To run the weekly job on e.g. opencode-go/glm-5.2, drop a file with
# LOOP_BACKEND=opencode
# LOOP_MODEL=opencode-go/glm-5.2
# No rebuild needed to switch — the env file is read at each timer fire. Holds no secrets
# (the opencode-go API key lives in ~/.local/share/opencode/auth.json, mode 600).
# UPGRADER_ARGS, …). The leading "-" makes it optional: absent file → claude/sonnet defaults.
# Current config (as of 2026-08-16): the upgrader + report run on tinfoil/deepseek-v4-pro
# (LOOP_MODEL + REPORT_MODEL in the env file); the hourly SUPERVISOR stays on glm-5.2
# (SUPERVISOR_MODEL defaults to opencode-go/glm-5.2 in launch-supervisor.py, NOT overridden
# here). Subagents bind deepseek via the cc-ci repo's opencode config. LOOP_TIER=zen is kept
# so the tier check passes; the watchdog's usage-limit probe sends the deepseek model name to
# the zen endpoint, which returns 200 (not 429) → resume immediately (correct: tinfoil has no
# rolling usage limit to wait out). No rebuild needed to switch — the env file is read at each
# timer fire. Holds no secrets (the tinfoil API key lives in the opencode config / auth.json).
EnvironmentFile = "-/srv/cc-ci/upgrader.env";
};
environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };