Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a0d6fc9417 |
+62
-3
@@ -867,6 +867,65 @@ session cc-ci-orchestrator-stale can be killed; recipe-mirrors org still private
|
||||
(/srv/cc-ci-orch/cc-ci), and task-tool subagents inherit their parent session's directory. The
|
||||
config now lives in the cc-ci repo at that path. VERIFIED end-to-end with the launcher's exact
|
||||
invocation: parent=glm-5.2, subagent=deepseek-v4-pro read back from the session DB.
|
||||
LESSON: `opencode debug config` proves resolution, NOT binding — only a live subagent's recorded
|
||||
modelID proves binding. First attempt was a false pass because the probe passed --dir (unlike the
|
||||
real launcher) and landed in a different project.
|
||||
LESSON: `opencode debug config` proves resolution, NOT binding — only a live subagent's recorded
|
||||
modelID proves binding. First attempt was a false pass because the probe passed --dir (unlike the
|
||||
real launcher) and landed in a different project.
|
||||
|
||||
## Session 2026-08-15 19:25 UTC — opencode glm-5.2
|
||||
|
||||
**Left off:** Recovered the stalled 2026-08-14 weekly /upgrade-all run. Killed a supervisor that had
|
||||
been relaunching hourly for ~40h (balance exhausted), then started a FRESH scoped upgrader. Run is now
|
||||
progressing (surveying the 9 remaining recipes). Watching it through to completion.
|
||||
|
||||
**What happened (the stall):**
|
||||
- The 2026-08-14 /upgrade-all run (session ses_00200382fffeYIGl2sc3mO9JId) stalled at 03:18 Aug 14
|
||||
mid-`lasuite-drive` with `Error: Insufficient balance` (opencode zen workspace balance ran out). It
|
||||
had already done bluesky-pds, ghost, gitea, hedgedoc (PRs) + immich, lasuite-docs (SKIPPED up-to-date)
|
||||
alphabetically; lasuite-drive had a plan + partial PR #6 but no RESULT/verify.
|
||||
- The supervisor cron (glm-5.2, opencode-go tier) relaunched an hourly one-shot supervisor ~40 times
|
||||
to "drive it to completion", but each was also balance-walled (and later, just spinning). The run sat
|
||||
INCOMPLETE + not progressing for 40h. No weekly summary, no report published for week of Aug 14.
|
||||
|
||||
**What I did this session:**
|
||||
- Diagnosed: the opencode zen endpoint is NOW healthy (direct probe `say OK` → HTTP 200 in 1.35s —
|
||||
balance is restored). But resuming the ORIGINAL giant session is impossible: it's 2.58M tokens
|
||||
(267K input + 2.3M cache) and `opencode run -s … --continue` sits idle on `do_epoll_wait` with zero
|
||||
I/O — the inference endpoint silently drops the oversized request (matches the supervisor's
|
||||
`socket connection was closed unexpectedly` errors). A fresh small `opencode run` works fine. So the
|
||||
giant session is unresumable; a fresh start is the only path.
|
||||
- Killed the stuck supervisor (tmux `cc-ci-supervisor`, proc 377329).
|
||||
- `UPGRADER_ARGS="lasuite-drive lasuite-meet mailu matrix-synapse mattermost-lts mumble n8n plausible
|
||||
wordpress --sequential" python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py fresh` — this killed the
|
||||
stuck resume, archived the old giant session (`archive-cc-ci-upgrader — 2026-08-14`), reclaimed 10GB
|
||||
stale images on cc-ci (disk 29%), and started a FRESH small session
|
||||
`ses_ff920cf39ffeoogwXHTajp94cr` (zen/glm-5.2) scoped to the 9 recipes not yet done this week
|
||||
(positions 13-21 alphabetically; positions 1-12 were already surveyed — 6 PRs + 6 up-to-date). A
|
||||
fresh watchdog is watching the new session. The skill is idempotent (reuses existing PRs incl.
|
||||
lasuite-drive #6, never duplicates), so scoping is safe.
|
||||
- Confirmed the fresh run is progressing: pane shows it surveying the 9 recipes (verified all present
|
||||
in abra + all `weekly` tier; currently probing plausible/wordpress tags). Proc alive, log growing.
|
||||
|
||||
**Phase / loop state:**
|
||||
- Build/adversary loops: STOPPED (whole sequence completed 2026-08-01; phase ghost DONE).
|
||||
- Weekly upgrader: RUNNING (fresh session ses_ff920cf39, scoped 9 recipes, --sequential, watchdog up).
|
||||
- cc-ci server: healthy (disk 29%, runner active).
|
||||
|
||||
**Open items for next session:**
|
||||
- **Monitor the fresh upgrader to completion.** It will survey the 9 recipes, /recipe-upgrade the
|
||||
upgradeable ones (subagents, !testme verify, open/extend PRs — NEVER merge), write the weekly summary
|
||||
to `/srv/cc-ci/.cc-ci-logs/upgrades/`, then `launch-report.py fresh` (the upgrade-all skill does this
|
||||
itself per SKILL.md §5), print `UPGRADE RUN COMPLETE`, and go idle. If it stalls on a usage limit,
|
||||
the watchdog auto-resumes the SAME (small) session — that works now.
|
||||
- **Do NOT try to resume the archived giant session ses_00200382** — it's unresumable (endpoint drops
|
||||
the 2.58M-token request). It's archived; leave it.
|
||||
- After the run completes + report publishes, operator review queue = this week's recipe PRs.
|
||||
- The supervisor cron (hourly at XX:07) should now leave the run alone once it's progressing; if a
|
||||
supervisor fires while the run is mid-flight, its guardrails say to hand back to the resumed run, not
|
||||
double-write. No action needed unless it interferes.
|
||||
|
||||
**Notes:**
|
||||
- Root cause of the 40h silence was the same BUG 1 from 2026-08-10 (supervisor progress gate) partly:
|
||||
the supervisor kept firing because the run never reached "progressing". Now that balance is restored
|
||||
and a fresh small session is running, the gate should see progress and stand down.
|
||||
- Lesson: when a weekly run dies mid-flight on a giant context, do NOT resume the original session —
|
||||
start fresh and scope to the remaining recipes. The /upgrade-all skill is idempotent so this is safe.
|
||||
|
||||
@@ -369,12 +369,15 @@ SSHCFG
|
||||
User = "loops"; Group = "users";
|
||||
WorkingDirectory = "/srv/cc-ci";
|
||||
# Optional per-run overrides for backend/model (LOOP_BACKEND, LOOP_MODEL, OPENCODE_SHARE,
|
||||
# UPGRADER_ARGS, …). The leading "-" makes it optional: absent file → claude/sonnet defaults
|
||||
# (current behavior). To run the weekly job on e.g. opencode-go/glm-5.2, drop a file with
|
||||
# LOOP_BACKEND=opencode
|
||||
# LOOP_MODEL=opencode-go/glm-5.2
|
||||
# No rebuild needed to switch — the env file is read at each timer fire. Holds no secrets
|
||||
# (the opencode-go API key lives in ~/.local/share/opencode/auth.json, mode 600).
|
||||
# UPGRADER_ARGS, …). The leading "-" makes it optional: absent file → claude/sonnet defaults.
|
||||
# Current config (as of 2026-08-16): the upgrader + report run on tinfoil/deepseek-v4-pro
|
||||
# (LOOP_MODEL + REPORT_MODEL in the env file); the hourly SUPERVISOR stays on glm-5.2
|
||||
# (SUPERVISOR_MODEL defaults to opencode-go/glm-5.2 in launch-supervisor.py, NOT overridden
|
||||
# here). Subagents bind deepseek via the cc-ci repo's opencode config. LOOP_TIER=zen is kept
|
||||
# so the tier check passes; the watchdog's usage-limit probe sends the deepseek model name to
|
||||
# the zen endpoint, which returns 200 (not 429) → resume immediately (correct: tinfoil has no
|
||||
# rolling usage limit to wait out). No rebuild needed to switch — the env file is read at each
|
||||
# timer fire. Holds no secrets (the tinfoil API key lives in the opencode config / auth.json).
|
||||
EnvironmentFile = "-/srv/cc-ci/upgrader.env";
|
||||
};
|
||||
environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
|
||||
|
||||
Reference in New Issue
Block a user