skills: PR-merge policy + test-before-switch + new /cc-ci-orchestrator-update

- cc-ci-server-update / cc-ci-tests-update / cc-ci-update: operator policy change —
  open PRs for visibility/historical record and merge directly once verified
  (invocation = authorization); reports must list merged-PR links + change summaries.
  (tests-update still never merges the paired recipe upgrade PRs.)
- cc-ci-server-update: mandatory nixos-rebuild test step (5d) before switch, with
  detached-activation + transient-unit notes from the 2026-08-03 26.05 deploy.
- AGENTS.md: test-before-switch policy for orchestrator host rebuilds.
- NEW skill cc-ci-orchestrator-update (+ thin .claude wrapper): the /cc-ci-server-update
  analogue for this host (flake /srv/cc-ci-orch, .#cc-ci-orchestrator-hetzner, Hetzner
  server 134487234), with self-update caveats.
- JOURNAL.md: 2026-08-03 server-update incident/recovery handoff entry.
This commit is contained in:
autonomic-bot
2026-08-03 20:35:51 +00:00
parent 2f22d42d34
commit d95afd934b
10 changed files with 282 additions and 35 deletions
+28
View File
@@ -763,3 +763,31 @@ session cc-ci-orchestrator-stale can be killed; recipe-mirrors org still private
session; re-oriented from STATUS/journals via a nudge). Relayed the operator's `${STACK_NAME}_app`
bluesky guidance. 4 recipe PRs + 2 harness fixes; **nothing merged** (operator reviews/merges).
- Queue drained again → hourly wake stops.
## 2026-08-03 ~20:30 UTC — /cc-ci-server-update: 24.11 → 26.05 (outage + recovery + success)
- Session: primary opencode orchestrator started the run; its session was stopped mid-incident and
the **cc-ci-orchestrator-backup** (Claude Code / Fable, session `cc-ci-backup-orchestrator`) was
operator-promoted to take over and finish.
- Flake bump: nixpkgs 50ab793 (nixos-24.11 tip) → 531670d (nixos-26.05), sops-nix 77c423a →
f140661 (master). PR #11, **merged** (new policy: PRs for visibility, merged directly, links in
report): https://git.autonomic.zone/recipe-maintainers/cc-ci/pulls/11
- **Outage:** first switch left the host with no default route (~19:3020:12 UTC). Root cause: NixOS
25.05+ scripted networking installs the default route from the gateway iface's
network-addresses unit; bare-string `defaultGateway` + off-subnet /32 Hetzner gateway → no iface
matched → no route. Fix: explicit `defaultGateway.interface = "eth0"` (commit c541cb1).
- Recovery detail worth remembering: NixOS grub generations live in a **submenu** — grubenv
`next_entry`/`default` need `1>N` ids; a bare index falls back to the (broken) default entry.
Clear any grubenv override after the next switch regenerates grub.cfg (indices shift).
- Final: gen 24, 26.05.20260803.531670d, 0 failed units, services 1/1 (warm spares flapping as
pre-update), report site 200, cold-boot proven (40s). Log:
.cc-ci-logs/server-update-2026-08-03.md
- Policy change (operator): /cc-ci-server-update, /cc-ci-tests-update, /cc-ci-update skills updated
in both .opencode and .claude positions — open PRs for visibility, merge directly once verified
(invocation = authorization), always report merged-PR links + change summaries.
- TODO for operator: rotate the Hetzner Cloud API token pasted into the orchestrator conversation
during recovery.
- Addendum (operator policy, 2026-08-03): **always `nixos-rebuild test` before `switch`** for nix
deploys to the cc-ci server and the orchestrator host — `test` leaves the bootloader/profile
untouched so a reboot always recovers. Codified in the cc-ci-server-update skill (step 5d) and
AGENTS.md (orchestrator rebuild instructions).