skills: PR-merge policy + test-before-switch + new /cc-ci-orchestrator-update
- cc-ci-server-update / cc-ci-tests-update / cc-ci-update: operator policy change — open PRs for visibility/historical record and merge directly once verified (invocation = authorization); reports must list merged-PR links + change summaries. (tests-update still never merges the paired recipe upgrade PRs.) - cc-ci-server-update: mandatory nixos-rebuild test step (5d) before switch, with detached-activation + transient-unit notes from the 2026-08-03 26.05 deploy. - AGENTS.md: test-before-switch policy for orchestrator host rebuilds. - NEW skill cc-ci-orchestrator-update (+ thin .claude wrapper): the /cc-ci-server-update analogue for this host (flake /srv/cc-ci-orch, .#cc-ci-orchestrator-hetzner, Hetzner server 134487234), with self-update caveats. - JOURNAL.md: 2026-08-03 server-update incident/recovery handoff entry.
This commit is contained in:
@@ -763,3 +763,31 @@ session cc-ci-orchestrator-stale can be killed; recipe-mirrors org still private
|
||||
session; re-oriented from STATUS/journals via a nudge). Relayed the operator's `${STACK_NAME}_app`
|
||||
bluesky guidance. 4 recipe PRs + 2 harness fixes; **nothing merged** (operator reviews/merges).
|
||||
- Queue drained again → hourly wake stops.
|
||||
|
||||
## 2026-08-03 ~20:30 UTC — /cc-ci-server-update: 24.11 → 26.05 (outage + recovery + success)
|
||||
|
||||
- Session: primary opencode orchestrator started the run; its session was stopped mid-incident and
|
||||
the **cc-ci-orchestrator-backup** (Claude Code / Fable, session `cc-ci-backup-orchestrator`) was
|
||||
operator-promoted to take over and finish.
|
||||
- Flake bump: nixpkgs 50ab793 (nixos-24.11 tip) → 531670d (nixos-26.05), sops-nix 77c423a →
|
||||
f140661 (master). PR #11, **merged** (new policy: PRs for visibility, merged directly, links in
|
||||
report): https://git.autonomic.zone/recipe-maintainers/cc-ci/pulls/11
|
||||
- **Outage:** first switch left the host with no default route (~19:30–20:12 UTC). Root cause: NixOS
|
||||
25.05+ scripted networking installs the default route from the gateway iface's
|
||||
network-addresses unit; bare-string `defaultGateway` + off-subnet /32 Hetzner gateway → no iface
|
||||
matched → no route. Fix: explicit `defaultGateway.interface = "eth0"` (commit c541cb1).
|
||||
- Recovery detail worth remembering: NixOS grub generations live in a **submenu** — grubenv
|
||||
`next_entry`/`default` need `1>N` ids; a bare index falls back to the (broken) default entry.
|
||||
Clear any grubenv override after the next switch regenerates grub.cfg (indices shift).
|
||||
- Final: gen 24, 26.05.20260803.531670d, 0 failed units, services 1/1 (warm spares flapping as
|
||||
pre-update), report site 200, cold-boot proven (40s). Log:
|
||||
.cc-ci-logs/server-update-2026-08-03.md
|
||||
- Policy change (operator): /cc-ci-server-update, /cc-ci-tests-update, /cc-ci-update skills updated
|
||||
in both .opencode and .claude positions — open PRs for visibility, merge directly once verified
|
||||
(invocation = authorization), always report merged-PR links + change summaries.
|
||||
- TODO for operator: rotate the Hetzner Cloud API token pasted into the orchestrator conversation
|
||||
during recovery.
|
||||
- Addendum (operator policy, 2026-08-03): **always `nixos-rebuild test` before `switch`** for nix
|
||||
deploys to the cc-ci server and the orchestrator host — `test` leaves the bootloader/profile
|
||||
untouched so a reboot always recovers. Codified in the cc-ci-server-update skill (step 5d) and
|
||||
AGENTS.md (orchestrator rebuild instructions).
|
||||
|
||||
Reference in New Issue
Block a user