skills: PR-merge policy + test-before-switch + new /cc-ci-orchestrator-update
- cc-ci-server-update / cc-ci-tests-update / cc-ci-update: operator policy change — open PRs for visibility/historical record and merge directly once verified (invocation = authorization); reports must list merged-PR links + change summaries. (tests-update still never merges the paired recipe upgrade PRs.) - cc-ci-server-update: mandatory nixos-rebuild test step (5d) before switch, with detached-activation + transient-unit notes from the 2026-08-03 26.05 deploy. - AGENTS.md: test-before-switch policy for orchestrator host rebuilds. - NEW skill cc-ci-orchestrator-update (+ thin .claude wrapper): the /cc-ci-server-update analogue for this host (flake /srv/cc-ci-orch, .#cc-ci-orchestrator-hetzner, Hetzner server 134487234), with self-update caveats. - JOURNAL.md: 2026-08-03 server-update incident/recovery handoff entry.
This commit is contained in:
autonomic-bot
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: cc-ci-server-update
|
||||
description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main; never merges the cc-ci PR. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update.
|
||||
description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main. Opens the cc-ci PR for visibility/historical record and merges it directly once the deploy verifies (the skill invocation IS the authorization); the end-of-run report includes the merged PR link + a summary of what changed. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update.
|
||||
---
|
||||
|
||||
# cc-ci-server-update
|
||||
@@ -24,7 +24,9 @@ confirm the server is healthy afterward. This is the cc-ci-server analogue of a
|
||||
- **Does NOT touch cc-ci tests / harness / runner** — that's `/cc-ci-tests-update` or
|
||||
`/ci-dev-workflow`. This is a flake-input bump only (`flake.nix` input revs + `flake.lock`).
|
||||
- **Does NOT update the orchestrator host** — only the cc-ci server.
|
||||
- **Does NOT merge** the cc-ci PR — operator merges after review.
|
||||
- **Does merge the cc-ci PR itself** — the PR exists for visibility and historical record, and is
|
||||
merged directly at the end of a verified run (the skill invocation is the authorization; the
|
||||
operator reviews the merged PR afterwards via the link in the report).
|
||||
|
||||
## Preconditions / access
|
||||
|
||||
@@ -120,7 +122,7 @@ Verify the lock moved to the revs you intended. **Do NOT `nixos-rebuild build` f
|
||||
orchestrator** — the orchestrator's nixpkgs is different and the cc-ci host target won't build here
|
||||
(meaningfully). The real build happens on the cc-ci host in step 5.
|
||||
|
||||
### 4. Commit + open the cc-ci PR (never merge)
|
||||
### 4. Commit + open the cc-ci PR (merged in step 7 after the health gate)
|
||||
|
||||
```
|
||||
git add flake.nix flake.lock
|
||||
@@ -139,7 +141,8 @@ BODY_FILE=/tmp/cc-ci-server-update-pr-body.md \
|
||||
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh
|
||||
```
|
||||
(Write the PR body to `/tmp/cc-ci-server-update-pr-body.md` first: old→new revs, the channel, any
|
||||
breaking changes from the release notes, the rebuild plan, and "NOT merged — for operator review.")
|
||||
breaking changes from the release notes, and the rebuild plan. The PR is opened for visibility and
|
||||
historical record; it gets merged directly in step 7 once the deploy passes the health gate.)
|
||||
|
||||
Capture the `PR_URL`. **The cc-ci push-build lint gate will run on the branch** — if it's red on a
|
||||
flake-lock-only change, it's a pre-existing lint drift (not yours); note it in the PR body and
|
||||
@@ -180,7 +183,24 @@ ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild build --flake .#cc-ci' 2>&1 |
|
||||
module option renames across channels (check the release notes), python package removals, a
|
||||
sops-nix incompatibility (go back to step 3b and pick a compatible sops-nix rev).
|
||||
|
||||
**(d) Switch (the live deploy):**
|
||||
**(d) Test-activate FIRST (general policy for nix deploys to this server AND the orchestrator
|
||||
host): `nixos-rebuild test` before any `switch`:**
|
||||
```
|
||||
# detach it so a dropped SSH can't kill the activation mid-flight:
|
||||
ssh cc-ci 'cd /root/cc-ci-deploy && setsid nohup nixos-rebuild test --flake .#cc-ci \
|
||||
> /root/cc-ci-deploy/test-switch.log 2>&1 < /dev/null & echo launched'
|
||||
# poll until reachable again, then check the essentials survived:
|
||||
ssh cc-ci 'readlink /run/current-system; ip route show default; systemctl --failed --no-legend'
|
||||
```
|
||||
`test` activates the new configuration **without touching the bootloader or the system profile**,
|
||||
so if the activation kills networking (as the 2026-08-03 26.05 bump did — no default route), a
|
||||
plain reboot (Hetzner API power-cycle if SSH is gone) lands back on the last-known-good
|
||||
generation. Only proceed to `switch` once the test activation is up, reachable, and healthy.
|
||||
Note: the detached test leaves a transient `nixos-rebuild-switch-to-configuration` unit; if the
|
||||
follow-up `switch` fails with "Unit ... was already loaded", wait for it to finish or
|
||||
`systemctl stop nixos-rebuild-switch-to-configuration.service` first.
|
||||
|
||||
**(e) Switch (make it permanent — only after (d) verified):**
|
||||
```
|
||||
ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild switch --flake .#cc-ci' 2>&1 | tail -30
|
||||
```
|
||||
@@ -215,17 +235,21 @@ If **anything regressed**: diagnose. A real regression → roll back to the prev
|
||||
flake input change on the PR branch is still your source of truth, but the running host reverts
|
||||
while you fix). Report the rollback in the PR.
|
||||
|
||||
### 7. Report
|
||||
### 7. Merge the PR + report
|
||||
|
||||
On a **verified** run (health gate green): post a PR comment summarizing what was deployed (revs,
|
||||
any incidents, verification results), then **merge the PR directly** — the skill invocation is the
|
||||
authorization; the PR exists for visibility and historical record. On a rolled-back or failed run,
|
||||
leave the PR **open** with a comment explaining the state.
|
||||
|
||||
Print + record a one-line result + the PR URL. Write a short note to
|
||||
`/srv/cc-ci/.cc-ci-logs/server-update-<YYYY-MM-DD>.md`:
|
||||
- `RESULT: SUCCESS — cc-ci server rebuilt on nixpkgs <new-rev[:8]>, no failed units, endpoints 200, PR: <url>`
|
||||
- `RESULT: SUCCESS — cc-ci server rebuilt on nixpkgs <new-rev[:8]>, no failed units, endpoints 200, PR merged: <url>`
|
||||
- `RESULT: ROLLED-BACK — nixpkgs <new-rev> rebuild failed/ regressed (<reason>); host reverted to <old-rev>; PR <url> left open`
|
||||
- `RESULT: FAILED — <step>: <reason>`
|
||||
|
||||
Always state that **the cc-ci PR is NOT merged** — it's the record of the bump; the operator
|
||||
reviews/merges at their discretion (the live rebuild already happened, so the PR is documentation +
|
||||
the path to re-apply, not a gate on the deploy).
|
||||
The end-of-run report to the operator MUST include **links to every merged PR** plus a summary of
|
||||
what changed in each, so the operator can review them after the fact if they want to.
|
||||
|
||||
### 8. Optional cold-boot proof
|
||||
|
||||
@@ -249,8 +273,13 @@ are tracked in its own journal; a note in the server-update report is enough).
|
||||
- **Single-writer:** dedicated branch in a separate clone; never push `main`; never touch the
|
||||
build loops' `/cc-ci` / `/cc-ci-adv` clones or `/root/builder-clone` (stage to
|
||||
`/root/cc-ci-deploy` instead).
|
||||
- **Never merge the cc-ci PR.** The live rebuild is the deploy; the PR is the reviewable record.
|
||||
- **Build before switch.** Never `nixos-rebuild switch` a build that didn't build cleanly first.
|
||||
- **Open the PR for visibility, merge it directly.** The live rebuild is the deploy; the PR is the
|
||||
reviewable historical record. Merge only after the health gate passes; a rolled-back/failed run
|
||||
leaves the PR open with an explanatory comment. Always report merged-PR links + change summaries.
|
||||
- **Build before test, test before switch.** Never `nixos-rebuild switch` a build that didn't build
|
||||
cleanly first, and never `switch` without a healthy `nixos-rebuild test` activation first — `test`
|
||||
leaves the bootloader untouched, so a reboot always recovers to the last-known-good generation.
|
||||
This is general policy for nix deploys to the cc-ci server **and** the orchestrator host.
|
||||
- **Health gate is mandatory.** No "it switched, ship it" — run the step-6 verification and compare
|
||||
to the baseline. Roll back on regression.
|
||||
- **Stable channel only** for the cc-ci server (never `nixos-unstable`).
|
||||
|
||||
Reference in New Issue
Block a user