orchestrator: reboot-resilience + session auto-resume + full session plan/tooling

Reboot survival for the Pi orchestrator host:
- systemd unit cc-ci-plan/systemd/cc-ci-loops.service (installed + enabled): on boot
  records the reboot, starts loops+watchdog (RESUME_PHASE=1), and resumes the
  orchestrator session.
- reboot-log.sh: boot_id-gated reboot record -> REBOOTS.md (manual restarts don't count).
- launch-orchestrator.sh: injects an AGENTS.md startup nudge so an auto-resumed
  orchestrator announces itself (PushNotification) + reports reboots.
- AGENTS.md: on-startup notify routine documented.

Plans/tooling accumulated this session:
- plan-phase1d (generic suite), 1e (harness corrections), phase4 (final review),
  sso-dep-testing, orchestrator-migration (parked), test-e2e-testme-acceptance.
- launch.sh: 1d/1e/2/2b/3/4 phase sequence, machine-docs-aware state resolution,
  limit-stall re-nudge, INBOX side-channel detection.
- plan.md §6.1/§7: artifact-layer isolation, INBOX, 5-min long-run polling, DEFERRED.
- prompts: isolation discipline + INBOX + pacing.
- .gitignore: harden (.sops/, cc-ci-secrets/, .claude/, *.tmp.*).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-28 20:28:10 +01:00
co-authored by Claude Opus 4.8
parent 5681438b0f
commit 36a6c9872a
20 changed files with 1395 additions and 19 deletions
+46 -3
View File
@@ -1,8 +1,13 @@
# cc-ci Phase 2 — Comprehensive per-recipe test authoring (Autonomous Build Plan)
**Status:** QUEUED — starts after Phase 1 (`plan.md`) and the Phase-1b review/lint pass
(`plan-phase1b-review-lint.md`) reach `## DONE`.
**Builds on:** the Phase-1 cc-ci CI server (`plan.md`). This phase adds **test content**, not infra.
**Status:** QUEUED — starts after Phase 1b (`plan-phase1b-review-lint.md`) **and Phase 1d**
(`plan-phase1d-generic-test-suite.md`) reach `## DONE`.
**Builds on:** the Phase-1 cc-ci CI server (`plan.md`) **and Phase 1d's generic test suite** — every
recipe already gets generic install/upgrade/backup/restore for free, reusing one shared deployment.
So this phase is **not** "write every test from scratch": it's **authoring the recipe overlays**
(`test_<op>.py` that override/extend the generic, per 1d's model) + **defining custom install steps**
for recipes that fail the generic form, porting the recipe-maintainer corpus as overlays. This phase
adds **test content**, not infra.
**Reference corpus:** `references/recipe-maintainer/` → `/srv/recipe-maintainer/` (the existing,
human-maintained recipe tests — the canonical source to port from).
**Owner agents:** same Builder + Adversary loops + coordination protocol as Phase 1 (`plan.md` §6/§7).
@@ -10,6 +15,44 @@ human-maintained recipe tests — the canonical source to port from).
---
## 0c. OIDC / SSO-dep recipes — follow the SSO-dep plan (NOT operator-pending)
Recipes that authenticate via OIDC (lasuite-docs, cryptpad, lasuite-drive, lasuite-meet, future)
do **not** need operator input to wire OIDC. The canonical pattern lives at
**`plan-sso-dep-testing.md`** — declared `DEPS = ["keycloak"]` in `recipe_meta.py`, orchestrator
co-deploys the dep, recipe `install_steps.sh` reads `$CCCI_DEPS_FILE` and writes the OIDC env vars
+ injects the client secret via abra. Authenticated tests use `harness.sso.oidc_password_grant` or
Playwright on the dep's login page. **`DEFERRED.md` entries that cite "operator input needed for
OIDC" are mis-categorised** — re-open and execute them per this plan.
## 0b. Auto-mirror missing recipes (NOT a blocker — autonomous loop work)
A recipe is not on `git.autonomic.zone/recipe-maintainers/<recipe>` is **not** an operator-pending
blocker. The bot is admin on `recipe-maintainers` (see memory `cc-ci-gitea-recipes`) and can create
private mirror repos. **For any recipe you want to enroll/test that isn't mirrored yet, mirror it
yourself** before enrolling, based on the **`recipe-create-pr` skill** —
`/srv/recipe-maintainer/.opencode/skills/recipe-create-pr/SKILL.md` (which references
`/srv/recipe-maintainer/.claude/commands/recipe-create-pr.md` for the full procedure).
The flow (adapt the skill's command for the new-mirror case):
1. Create the **private** mirror repo on `git.autonomic.zone/recipe-maintainers/<recipe>` (Gitea API
POST `/orgs/recipe-maintainers/repos`, bot creds from `.testenv`/§1.5).
2. Mirror the upstream `git.coopcloud.tech/coop-cloud/<recipe>` (clone --mirror → push, including
tags) so the mirror's `main` is upstream-synced and tags carry over.
3. Then proceed with normal enrollment + the lifecycle suite (1d generic + your overlays from this
phase).
Treat this as standard loop work — don't sit idle waiting on the operator for missing recipes.
## 0a. Prerequisite: Phase 1e harness corrections (must be DONE first)
The 1d/1b operator review produced three shared-harness corrections, now their own phase
**`plan-phase1e-harness-corrections.md`** (runs **before** this phase). Do **not** author overlays
until 1e is `## DONE`: it changes the foundation every overlay sits on — (HC1) upgrade goes
prev-release → **PR head** via `deploy --chaos`; (HC2) **repo-local PR code runs only for approved
recipes** (default cc-ci-overlays + generic only); (HC3) the **generic runs by default alongside an
overlay**, skipped only via an explicit opt-out. See that plan for detail.
## 0. Relationship to Phase 1 (read first)
Phase 1 built the **machine**: the Drone pipeline, the `!testme` trigger (polling-primary), the