orchestrator: reboot-resilience + session auto-resume + full session plan/tooling

Reboot survival for the Pi orchestrator host:
- systemd unit cc-ci-plan/systemd/cc-ci-loops.service (installed + enabled): on boot
  records the reboot, starts loops+watchdog (RESUME_PHASE=1), and resumes the
  orchestrator session.
- reboot-log.sh: boot_id-gated reboot record -> REBOOTS.md (manual restarts don't count).
- launch-orchestrator.sh: injects an AGENTS.md startup nudge so an auto-resumed
  orchestrator announces itself (PushNotification) + reports reboots.
- AGENTS.md: on-startup notify routine documented.

Plans/tooling accumulated this session:
- plan-phase1d (generic suite), 1e (harness corrections), phase4 (final review),
  sso-dep-testing, orchestrator-migration (parked), test-e2e-testme-acceptance.
- launch.sh: 1d/1e/2/2b/3/4 phase sequence, machine-docs-aware state resolution,
  limit-stall re-nudge, INBOX side-channel detection.
- plan.md §6.1/§7: artifact-layer isolation, INBOX, 5-min long-run polling, DEFERRED.
- prompts: isolation discipline + INBOX + pacing.
- .gitignore: harden (.sops/, cc-ci-secrets/, .claude/, *.tmp.*).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-28 20:28:10 +01:00
co-authored by Claude Opus 4.8
parent 5681438b0f
commit 36a6c9872a
20 changed files with 1395 additions and 19 deletions
+19
View File
@@ -16,6 +16,25 @@ project (NixOS config, test runner, recipe tests) lives in a **separate** repo t
The two loops coordinate **only** through the cc-ci git repo (see `plan.md` §6.1). The orchestrator
watches from outside.
## On startup: announce yourself + report reboots
**Every time you (the orchestrator) start or resume, send a `PushNotification`** that you are online —
the operator wants to know the supervising session is back (especially after a reboot, which kills
this session along with the Pi). Include the current phase and the reboot count. Steps on startup:
1. Read `cc-ci-plan/REBOOTS.md` (count the `## Reboots` entries) and `cc-ci-plan/launch.sh status`
(current phase + whether the loops/watchdog are running).
2. `PushNotification` (proactive), e.g.: *"cc-ci orchestrator online — phase 2, loops+watchdog
running; N reboots logged (last <date>)."*
3. If a reboot happened while you were away (a new line in REBOOTS.md since you last looked, or the
loops are down), check that `cc-ci-loops.service` brought the loops back; if not, relaunch with
`RESUME_PHASE=1 cc-ci-plan/launch.sh start`.
Reboot resilience is handled by **`cc-ci-loops.service`** (system unit): on boot it logs the reboot
to `REBOOTS.md` (boot_id-gated) and runs `launch.sh start` with `RESUME_PHASE=1`, so the loops +
watchdog auto-resume the saved phase. The orchestrator session itself is NOT auto-started — the
operator reconnects to it (that's why the startup notification matters). The fuller "move the
orchestrator onto its own VM" plan is parked at `cc-ci-plan/plan-orchestrator-migration.md`.
## Keep the orchestrator open, under remote-control
Run this session as a long-lived **interactive** session with `--remote-control` so the operator can