orchestrator: reboot-resilience + session auto-resume + full session plan/tooling
Reboot survival for the Pi orchestrator host: - systemd unit cc-ci-plan/systemd/cc-ci-loops.service (installed + enabled): on boot records the reboot, starts loops+watchdog (RESUME_PHASE=1), and resumes the orchestrator session. - reboot-log.sh: boot_id-gated reboot record -> REBOOTS.md (manual restarts don't count). - launch-orchestrator.sh: injects an AGENTS.md startup nudge so an auto-resumed orchestrator announces itself (PushNotification) + reports reboots. - AGENTS.md: on-startup notify routine documented. Plans/tooling accumulated this session: - plan-phase1d (generic suite), 1e (harness corrections), phase4 (final review), sso-dep-testing, orchestrator-migration (parked), test-e2e-testme-acceptance. - launch.sh: 1d/1e/2/2b/3/4 phase sequence, machine-docs-aware state resolution, limit-stall re-nudge, INBOX side-channel detection. - plan.md §6.1/§7: artifact-layer isolation, INBOX, 5-min long-run polling, DEFERRED. - prompts: isolation discipline + INBOX + pacing. - .gitignore: harden (.sops/, cc-ci-secrets/, .claude/, *.tmp.*). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -16,6 +16,25 @@ project (NixOS config, test runner, recipe tests) lives in a **separate** repo t
|
||||
The two loops coordinate **only** through the cc-ci git repo (see `plan.md` §6.1). The orchestrator
|
||||
watches from outside.
|
||||
|
||||
## On startup: announce yourself + report reboots
|
||||
|
||||
**Every time you (the orchestrator) start or resume, send a `PushNotification`** that you are online —
|
||||
the operator wants to know the supervising session is back (especially after a reboot, which kills
|
||||
this session along with the Pi). Include the current phase and the reboot count. Steps on startup:
|
||||
1. Read `cc-ci-plan/REBOOTS.md` (count the `## Reboots` entries) and `cc-ci-plan/launch.sh status`
|
||||
(current phase + whether the loops/watchdog are running).
|
||||
2. `PushNotification` (proactive), e.g.: *"cc-ci orchestrator online — phase 2, loops+watchdog
|
||||
running; N reboots logged (last <date>)."*
|
||||
3. If a reboot happened while you were away (a new line in REBOOTS.md since you last looked, or the
|
||||
loops are down), check that `cc-ci-loops.service` brought the loops back; if not, relaunch with
|
||||
`RESUME_PHASE=1 cc-ci-plan/launch.sh start`.
|
||||
|
||||
Reboot resilience is handled by **`cc-ci-loops.service`** (system unit): on boot it logs the reboot
|
||||
to `REBOOTS.md` (boot_id-gated) and runs `launch.sh start` with `RESUME_PHASE=1`, so the loops +
|
||||
watchdog auto-resume the saved phase. The orchestrator session itself is NOT auto-started — the
|
||||
operator reconnects to it (that's why the startup notification matters). The fuller "move the
|
||||
orchestrator onto its own VM" plan is parked at `cc-ci-plan/plan-orchestrator-migration.md`.
|
||||
|
||||
## Keep the orchestrator open, under remote-control
|
||||
|
||||
Run this session as a long-lived **interactive** session with `--remote-control` so the operator can
|
||||
|
||||
Reference in New Issue
Block a user