recipe-maintainers/cc-ci-orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

watchdog: cover all parts of the weekly run + survive the systemd oneshot

Browse Source 
Two gaps for the scheduled Thursday glm-5.2 run:
1. Survival: the watchdog was a Popen child of the Type=oneshot service, which
   systemd's cgroup cleanup kills on exit. Spawn it under the persistent tmux
   server instead (_spawn_watchdog), like the run sessions — survives the oneshot.
2. The report runs on glm-5.2 sharing the same opencode-go budget the upgrade run
   drains, so it can 429-stall with no recovery. launch-report.py now spawns the
   SAME watchdog pointed at the cc-ci-report session (generic via UPGRADER_SESSION/
   _MODEL/_DONE_MARKER/_RESUME_FILE), with a report-specific resume prompt.

Also: _run_pids() is now scoped to the managed session (title or -s <sid>) so the
report watchdog can't kill the idle upgrader process and vice-versa; resume() adds
--dir and honors a custom resume prompt file.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
autonomic-bot
2026-06-23 02:42:50 +00:00
co-authored by Claude Opus 4.8
parent 5a6c62e36c
commit f94be45f9c
2 changed files with 83 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cc-ci-plan/launch-report.py
+29 -2
View File
@@ -87,8 +87,35 @@ def start(mode, date):
log(f"starting {SESSION} (backend={BACKEND}, model={MODEL}, date={date or 'today'})")
subprocess.run(["tmux", "new-session", "-d", "-s", SESSION, "-c", cwd, cmd])
subprocess.run(["tmux", "pipe-pane", "-o", "-t", SESSION, f"cat >> '{LOG_DIR}/{SESSION}.log'"])
if BACKEND == "opencode" and OPENCODE_SHARE:
log(f" attached to {OPENCODE_SERVER} → http://oc.commoninternet.net +public --share link")
if BACKEND == "opencode":
if OPENCODE_SHARE:
log(f" attached to {OPENCODE_SERVER} → http://oc.commoninternet.net +public --share link")
# Watchdog for the report too: it runs on glm-5.2 sharing the same opencode-go budget the
# upgrade run just drained, so a 429 stall is likely. Reuse launch-upgrader.py's watchdog,
# pointed at THIS (cc-ci-report) session with a report-specific marker + resume prompt. It
# runs under the tmux server (survives the systemd oneshot like the run sessions).
if os.environ.get("REPORT_WATCHDOG", "1") == "1":
import shlex
LU = "/srv/cc-ci/cc-ci-plan/launch-upgrader.py"
rf = Path(LOG_DIR) / f".kickoff-{SESSION}-resume.txt"
rf.write_text(
"The opencode-go usage limit has reset (or the report stalled). CONTINUE generating and "
"publishing this week's public Recipe Report per /recipe-report: survey → write the spec "
"JSON → render with recipe-report.py → publish. Do NOT hand-write HTML (render() owns all "
"formatting). When the page is live, print 'RECIPE REPORT COMPLETE' and go idle.")
wenv = {"HOME": os.environ.get("HOME") or "/home/loops",
"UPGRADER_SESSION": SESSION, "UPGRADER_DIR": cwd, "LOG_DIR": LOG_DIR,
"UPGRADER_BACKEND": "opencode", "UPGRADER_MODEL": MODEL,
"OPENCODE_BIN": OPENCODE_BIN, "OPENCODE_SERVER": OPENCODE_SERVER,
"OPENCODE_SHARE": "1" if OPENCODE_SHARE else "0",
"UPGRADER_DONE_MARKER": "RECIPE REPORT COMPLETE",
"UPGRADER_RESUME_FILE": str(rf)}
envstr = " ".join(f"{k}={shlex.quote(str(v))}" for k, v in wenv.items())
wlog = f"{LOG_DIR}/{SESSION}-watchdog.log"
wcmd = f"env {envstr} python3 {shlex.quote(LU)} watchdog >> {shlex.quote(wlog)} 2>&1"
subprocess.run(["tmux", "kill-session", "-t", f"{SESSION}-watchdog"], capture_output=True)
subprocess.run(["tmux", "new-session", "-d", "-s", f"{SESSION}-watchdog", "-c", cwd, wcmd])
log(f" report watchdog spawned in tmux '{SESSION}-watchdog' — auto-resume on usage-limit stalls")
log(f"started. attach: tmux attach -t {SESSION}")