Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
929faeb055 | ||
|
|
a29b993160 | ||
|
|
a0d6fc9417 | ||
|
|
ab5d11ea2a |
+62
-3
@@ -867,6 +867,65 @@ session cc-ci-orchestrator-stale can be killed; recipe-mirrors org still private
|
|||||||
(/srv/cc-ci-orch/cc-ci), and task-tool subagents inherit their parent session's directory. The
|
(/srv/cc-ci-orch/cc-ci), and task-tool subagents inherit their parent session's directory. The
|
||||||
config now lives in the cc-ci repo at that path. VERIFIED end-to-end with the launcher's exact
|
config now lives in the cc-ci repo at that path. VERIFIED end-to-end with the launcher's exact
|
||||||
invocation: parent=glm-5.2, subagent=deepseek-v4-pro read back from the session DB.
|
invocation: parent=glm-5.2, subagent=deepseek-v4-pro read back from the session DB.
|
||||||
LESSON: `opencode debug config` proves resolution, NOT binding — only a live subagent's recorded
|
LESSON: `opencode debug config` proves resolution, NOT binding — only a live subagent's recorded
|
||||||
modelID proves binding. First attempt was a false pass because the probe passed --dir (unlike the
|
modelID proves binding. First attempt was a false pass because the probe passed --dir (unlike the
|
||||||
real launcher) and landed in a different project.
|
real launcher) and landed in a different project.
|
||||||
|
|
||||||
|
## Session 2026-08-15 19:25 UTC — opencode glm-5.2
|
||||||
|
|
||||||
|
**Left off:** Recovered the stalled 2026-08-14 weekly /upgrade-all run. Killed a supervisor that had
|
||||||
|
been relaunching hourly for ~40h (balance exhausted), then started a FRESH scoped upgrader. Run is now
|
||||||
|
progressing (surveying the 9 remaining recipes). Watching it through to completion.
|
||||||
|
|
||||||
|
**What happened (the stall):**
|
||||||
|
- The 2026-08-14 /upgrade-all run (session ses_00200382fffeYIGl2sc3mO9JId) stalled at 03:18 Aug 14
|
||||||
|
mid-`lasuite-drive` with `Error: Insufficient balance` (opencode zen workspace balance ran out). It
|
||||||
|
had already done bluesky-pds, ghost, gitea, hedgedoc (PRs) + immich, lasuite-docs (SKIPPED up-to-date)
|
||||||
|
alphabetically; lasuite-drive had a plan + partial PR #6 but no RESULT/verify.
|
||||||
|
- The supervisor cron (glm-5.2, opencode-go tier) relaunched an hourly one-shot supervisor ~40 times
|
||||||
|
to "drive it to completion", but each was also balance-walled (and later, just spinning). The run sat
|
||||||
|
INCOMPLETE + not progressing for 40h. No weekly summary, no report published for week of Aug 14.
|
||||||
|
|
||||||
|
**What I did this session:**
|
||||||
|
- Diagnosed: the opencode zen endpoint is NOW healthy (direct probe `say OK` → HTTP 200 in 1.35s —
|
||||||
|
balance is restored). But resuming the ORIGINAL giant session is impossible: it's 2.58M tokens
|
||||||
|
(267K input + 2.3M cache) and `opencode run -s … --continue` sits idle on `do_epoll_wait` with zero
|
||||||
|
I/O — the inference endpoint silently drops the oversized request (matches the supervisor's
|
||||||
|
`socket connection was closed unexpectedly` errors). A fresh small `opencode run` works fine. So the
|
||||||
|
giant session is unresumable; a fresh start is the only path.
|
||||||
|
- Killed the stuck supervisor (tmux `cc-ci-supervisor`, proc 377329).
|
||||||
|
- `UPGRADER_ARGS="lasuite-drive lasuite-meet mailu matrix-synapse mattermost-lts mumble n8n plausible
|
||||||
|
wordpress --sequential" python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py fresh` — this killed the
|
||||||
|
stuck resume, archived the old giant session (`archive-cc-ci-upgrader — 2026-08-14`), reclaimed 10GB
|
||||||
|
stale images on cc-ci (disk 29%), and started a FRESH small session
|
||||||
|
`ses_ff920cf39ffeoogwXHTajp94cr` (zen/glm-5.2) scoped to the 9 recipes not yet done this week
|
||||||
|
(positions 13-21 alphabetically; positions 1-12 were already surveyed — 6 PRs + 6 up-to-date). A
|
||||||
|
fresh watchdog is watching the new session. The skill is idempotent (reuses existing PRs incl.
|
||||||
|
lasuite-drive #6, never duplicates), so scoping is safe.
|
||||||
|
- Confirmed the fresh run is progressing: pane shows it surveying the 9 recipes (verified all present
|
||||||
|
in abra + all `weekly` tier; currently probing plausible/wordpress tags). Proc alive, log growing.
|
||||||
|
|
||||||
|
**Phase / loop state:**
|
||||||
|
- Build/adversary loops: STOPPED (whole sequence completed 2026-08-01; phase ghost DONE).
|
||||||
|
- Weekly upgrader: RUNNING (fresh session ses_ff920cf39, scoped 9 recipes, --sequential, watchdog up).
|
||||||
|
- cc-ci server: healthy (disk 29%, runner active).
|
||||||
|
|
||||||
|
**Open items for next session:**
|
||||||
|
- **Monitor the fresh upgrader to completion.** It will survey the 9 recipes, /recipe-upgrade the
|
||||||
|
upgradeable ones (subagents, !testme verify, open/extend PRs — NEVER merge), write the weekly summary
|
||||||
|
to `/srv/cc-ci/.cc-ci-logs/upgrades/`, then `launch-report.py fresh` (the upgrade-all skill does this
|
||||||
|
itself per SKILL.md §5), print `UPGRADE RUN COMPLETE`, and go idle. If it stalls on a usage limit,
|
||||||
|
the watchdog auto-resumes the SAME (small) session — that works now.
|
||||||
|
- **Do NOT try to resume the archived giant session ses_00200382** — it's unresumable (endpoint drops
|
||||||
|
the 2.58M-token request). It's archived; leave it.
|
||||||
|
- After the run completes + report publishes, operator review queue = this week's recipe PRs.
|
||||||
|
- The supervisor cron (hourly at XX:07) should now leave the run alone once it's progressing; if a
|
||||||
|
supervisor fires while the run is mid-flight, its guardrails say to hand back to the resumed run, not
|
||||||
|
double-write. No action needed unless it interferes.
|
||||||
|
|
||||||
|
**Notes:**
|
||||||
|
- Root cause of the 40h silence was the same BUG 1 from 2026-08-10 (supervisor progress gate) partly:
|
||||||
|
the supervisor kept firing because the run never reached "progressing". Now that balance is restored
|
||||||
|
and a fresh small session is running, the gate should see progress and stand down.
|
||||||
|
- Lesson: when a weekly run dies mid-flight on a giant context, do NOT resume the original session —
|
||||||
|
start fresh and scope to the remaining recipes. The /upgrade-all skill is idempotent so this is safe.
|
||||||
|
|||||||
@@ -31,7 +31,7 @@ matrix-synapse weekly
|
|||||||
mattermost-lts weekly
|
mattermost-lts weekly
|
||||||
mumble weekly
|
mumble weekly
|
||||||
n8n weekly
|
n8n weekly
|
||||||
plausible weekly
|
plausible external # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded
|
||||||
uptime-kuma external # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded
|
uptime-kuma external # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded
|
||||||
wordpress weekly
|
wordpress weekly
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -369,12 +369,15 @@ SSHCFG
|
|||||||
User = "loops"; Group = "users";
|
User = "loops"; Group = "users";
|
||||||
WorkingDirectory = "/srv/cc-ci";
|
WorkingDirectory = "/srv/cc-ci";
|
||||||
# Optional per-run overrides for backend/model (LOOP_BACKEND, LOOP_MODEL, OPENCODE_SHARE,
|
# Optional per-run overrides for backend/model (LOOP_BACKEND, LOOP_MODEL, OPENCODE_SHARE,
|
||||||
# UPGRADER_ARGS, …). The leading "-" makes it optional: absent file → claude/sonnet defaults
|
# UPGRADER_ARGS, …). The leading "-" makes it optional: absent file → claude/sonnet defaults.
|
||||||
# (current behavior). To run the weekly job on e.g. opencode-go/glm-5.2, drop a file with
|
# Current config (as of 2026-08-16): the upgrader + report run on tinfoil/deepseek-v4-pro
|
||||||
# LOOP_BACKEND=opencode
|
# (LOOP_MODEL + REPORT_MODEL in the env file); the hourly SUPERVISOR stays on glm-5.2
|
||||||
# LOOP_MODEL=opencode-go/glm-5.2
|
# (SUPERVISOR_MODEL defaults to opencode-go/glm-5.2 in launch-supervisor.py, NOT overridden
|
||||||
# No rebuild needed to switch — the env file is read at each timer fire. Holds no secrets
|
# here). Subagents bind deepseek via the cc-ci repo's opencode config. LOOP_TIER=zen is kept
|
||||||
# (the opencode-go API key lives in ~/.local/share/opencode/auth.json, mode 600).
|
# so the tier check passes; the watchdog's usage-limit probe sends the deepseek model name to
|
||||||
|
# the zen endpoint, which returns 200 (not 429) → resume immediately (correct: tinfoil has no
|
||||||
|
# rolling usage limit to wait out). No rebuild needed to switch — the env file is read at each
|
||||||
|
# timer fire. Holds no secrets (the tinfoil API key lives in the opencode config / auth.json).
|
||||||
EnvironmentFile = "-/srv/cc-ci/upgrader.env";
|
EnvironmentFile = "-/srv/cc-ci/upgrader.env";
|
||||||
};
|
};
|
||||||
environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
|
environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
|
||||||
|
|||||||
Reference in New Issue
Block a user