|
|
@@ -870,3 +870,62 @@ session cc-ci-orchestrator-stale can be killed; recipe-mirrors org still private
|
|
|
|
LESSON: `opencode debug config` proves resolution, NOT binding — only a live subagent's recorded
|
|
|
|
LESSON: `opencode debug config` proves resolution, NOT binding — only a live subagent's recorded
|
|
|
|
modelID proves binding. First attempt was a false pass because the probe passed --dir (unlike the
|
|
|
|
modelID proves binding. First attempt was a false pass because the probe passed --dir (unlike the
|
|
|
|
real launcher) and landed in a different project.
|
|
|
|
real launcher) and landed in a different project.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Session 2026-08-15 19:25 UTC — opencode glm-5.2
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Left off:** Recovered the stalled 2026-08-14 weekly /upgrade-all run. Killed a supervisor that had
|
|
|
|
|
|
|
|
been relaunching hourly for ~40h (balance exhausted), then started a FRESH scoped upgrader. Run is now
|
|
|
|
|
|
|
|
progressing (surveying the 9 remaining recipes). Watching it through to completion.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**What happened (the stall):**
|
|
|
|
|
|
|
|
- The 2026-08-14 /upgrade-all run (session ses_00200382fffeYIGl2sc3mO9JId) stalled at 03:18 Aug 14
|
|
|
|
|
|
|
|
mid-`lasuite-drive` with `Error: Insufficient balance` (opencode zen workspace balance ran out). It
|
|
|
|
|
|
|
|
had already done bluesky-pds, ghost, gitea, hedgedoc (PRs) + immich, lasuite-docs (SKIPPED up-to-date)
|
|
|
|
|
|
|
|
alphabetically; lasuite-drive had a plan + partial PR #6 but no RESULT/verify.
|
|
|
|
|
|
|
|
- The supervisor cron (glm-5.2, opencode-go tier) relaunched an hourly one-shot supervisor ~40 times
|
|
|
|
|
|
|
|
to "drive it to completion", but each was also balance-walled (and later, just spinning). The run sat
|
|
|
|
|
|
|
|
INCOMPLETE + not progressing for 40h. No weekly summary, no report published for week of Aug 14.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**What I did this session:**
|
|
|
|
|
|
|
|
- Diagnosed: the opencode zen endpoint is NOW healthy (direct probe `say OK` → HTTP 200 in 1.35s —
|
|
|
|
|
|
|
|
balance is restored). But resuming the ORIGINAL giant session is impossible: it's 2.58M tokens
|
|
|
|
|
|
|
|
(267K input + 2.3M cache) and `opencode run -s … --continue` sits idle on `do_epoll_wait` with zero
|
|
|
|
|
|
|
|
I/O — the inference endpoint silently drops the oversized request (matches the supervisor's
|
|
|
|
|
|
|
|
`socket connection was closed unexpectedly` errors). A fresh small `opencode run` works fine. So the
|
|
|
|
|
|
|
|
giant session is unresumable; a fresh start is the only path.
|
|
|
|
|
|
|
|
- Killed the stuck supervisor (tmux `cc-ci-supervisor`, proc 377329).
|
|
|
|
|
|
|
|
- `UPGRADER_ARGS="lasuite-drive lasuite-meet mailu matrix-synapse mattermost-lts mumble n8n plausible
|
|
|
|
|
|
|
|
wordpress --sequential" python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py fresh` — this killed the
|
|
|
|
|
|
|
|
stuck resume, archived the old giant session (`archive-cc-ci-upgrader — 2026-08-14`), reclaimed 10GB
|
|
|
|
|
|
|
|
stale images on cc-ci (disk 29%), and started a FRESH small session
|
|
|
|
|
|
|
|
`ses_ff920cf39ffeoogwXHTajp94cr` (zen/glm-5.2) scoped to the 9 recipes not yet done this week
|
|
|
|
|
|
|
|
(positions 13-21 alphabetically; positions 1-12 were already surveyed — 6 PRs + 6 up-to-date). A
|
|
|
|
|
|
|
|
fresh watchdog is watching the new session. The skill is idempotent (reuses existing PRs incl.
|
|
|
|
|
|
|
|
lasuite-drive #6, never duplicates), so scoping is safe.
|
|
|
|
|
|
|
|
- Confirmed the fresh run is progressing: pane shows it surveying the 9 recipes (verified all present
|
|
|
|
|
|
|
|
in abra + all `weekly` tier; currently probing plausible/wordpress tags). Proc alive, log growing.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Phase / loop state:**
|
|
|
|
|
|
|
|
- Build/adversary loops: STOPPED (whole sequence completed 2026-08-01; phase ghost DONE).
|
|
|
|
|
|
|
|
- Weekly upgrader: RUNNING (fresh session ses_ff920cf39, scoped 9 recipes, --sequential, watchdog up).
|
|
|
|
|
|
|
|
- cc-ci server: healthy (disk 29%, runner active).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Open items for next session:**
|
|
|
|
|
|
|
|
- **Monitor the fresh upgrader to completion.** It will survey the 9 recipes, /recipe-upgrade the
|
|
|
|
|
|
|
|
upgradeable ones (subagents, !testme verify, open/extend PRs — NEVER merge), write the weekly summary
|
|
|
|
|
|
|
|
to `/srv/cc-ci/.cc-ci-logs/upgrades/`, then `launch-report.py fresh` (the upgrade-all skill does this
|
|
|
|
|
|
|
|
itself per SKILL.md §5), print `UPGRADE RUN COMPLETE`, and go idle. If it stalls on a usage limit,
|
|
|
|
|
|
|
|
the watchdog auto-resumes the SAME (small) session — that works now.
|
|
|
|
|
|
|
|
- **Do NOT try to resume the archived giant session ses_00200382** — it's unresumable (endpoint drops
|
|
|
|
|
|
|
|
the 2.58M-token request). It's archived; leave it.
|
|
|
|
|
|
|
|
- After the run completes + report publishes, operator review queue = this week's recipe PRs.
|
|
|
|
|
|
|
|
- The supervisor cron (hourly at XX:07) should now leave the run alone once it's progressing; if a
|
|
|
|
|
|
|
|
supervisor fires while the run is mid-flight, its guardrails say to hand back to the resumed run, not
|
|
|
|
|
|
|
|
double-write. No action needed unless it interferes.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Notes:**
|
|
|
|
|
|
|
|
- Root cause of the 40h silence was the same BUG 1 from 2026-08-10 (supervisor progress gate) partly:
|
|
|
|
|
|
|
|
the supervisor kept firing because the run never reached "progressing". Now that balance is restored
|
|
|
|
|
|
|
|
and a fresh small session is running, the gate should see progress and stand down.
|
|
|
|
|
|
|
|
- Lesson: when a weekly run dies mid-flight on a giant context, do NOT resume the original session —
|
|
|
|
|
|
|
|
start fresh and scope to the remaining recipes. The /upgrade-all skill is idempotent so this is safe.
|
|
|
|