skills: PR-merge policy + test-before-switch + new /cc-ci-orchestrator-update
- cc-ci-server-update / cc-ci-tests-update / cc-ci-update: operator policy change — open PRs for visibility/historical record and merge directly once verified (invocation = authorization); reports must list merged-PR links + change summaries. (tests-update still never merges the paired recipe upgrade PRs.) - cc-ci-server-update: mandatory nixos-rebuild test step (5d) before switch, with detached-activation + transient-unit notes from the 2026-08-03 26.05 deploy. - AGENTS.md: test-before-switch policy for orchestrator host rebuilds. - NEW skill cc-ci-orchestrator-update (+ thin .claude wrapper): the /cc-ci-server-update analogue for this host (flake /srv/cc-ci-orch, .#cc-ci-orchestrator-hetzner, Hetzner server 134487234), with self-update caveats. - JOURNAL.md: 2026-08-03 server-update incident/recovery handoff entry.
This commit is contained in:
autonomic-bot
@@ -0,0 +1,14 @@
|
||||
---
|
||||
name: cc-ci-orchestrator-update
|
||||
description: Update the cc-ci ORCHESTRATOR host (the machine the orchestrator itself runs on) to the latest nixos-26.05 nixpkgs, then nixos-rebuild test + switch + verify, ship the bump as a verified PR on recipe-maintainers/cc-ci-orchestrator and merge it directly (the skill invocation IS the authorization). The orchestrator-host analogue of /cc-ci-server-update. Self-update caveats apply — the switch restarts services on the very host running this session; always `nixos-rebuild test` before `switch` so a reboot recovers to the last-known-good generation. The end-of-run report includes the merged PR link + a summary of what changed. Invoke as /cc-ci-orchestrator-update (also answers to /cc-ci-update-orchestrator).
|
||||
---
|
||||
|
||||
# cc-ci-orchestrator-update (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cc-ci-orchestrator-update/SKILL.md`**
|
||||
|
||||
Read that file for the full procedure. This `.claude/skills/` copy is kept as a
|
||||
thin pointer for Claude Code compatibility; opencode loads the canonical
|
||||
definition from `.opencode/skills/` directly.
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: cc-ci-server-update
|
||||
description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main; never merges the cc-ci PR. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update.
|
||||
description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main. Opens the cc-ci PR for visibility/historical record and merges it directly once the deploy verifies (the skill invocation IS the authorization); the end-of-run report includes the merged PR link + a summary of what changed. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update.
|
||||
---
|
||||
|
||||
# cc-ci-server-update (thin wrapper)
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: cc-ci-tests-update
|
||||
description: Sweep all maintained recipes for stale cc-ci tests and update the stale ones so they pass again, verified via the standard PR + !testme -> CI flow. Runs the deterministic test suite across every enrolled recipe, identifies failures caused by a genuinely-stale test (the recipe legitimately changed upstream and the cc-ci test/overlay still asserts the old behavior), authors the minimal test update, opens a cc-ci test PR per stale recipe, and verifies each by running !testme on the corresponding recipe upgrade PR with the test change applied (or a direct harness run with the cc-ci branch checked out). NEVER weakens a test to force green; never merges. The fleet-wide stale-test cleanup that the weekly /upgrade-all deliberately defers (it comments-only on stale tests; this skill closes the loop). Invoke as /cc-ci-tests-update.
|
||||
description: Sweep all maintained recipes for stale cc-ci tests and update the stale ones so they pass again, verified via the standard PR + !testme -> CI flow. Runs the deterministic test suite across every enrolled recipe, identifies failures caused by a genuinely-stale test (the recipe legitimately changed upstream and the cc-ci test/overlay still asserts the old behavior), authors the minimal test update, opens a cc-ci test PR per stale recipe, and verifies each by running !testme on the corresponding recipe upgrade PR with the test change applied (or a direct harness run with the cc-ci branch checked out). NEVER weakens a test to force green. Opens each cc-ci test PR for visibility/historical record and merges it directly once verified green (the skill invocation IS the authorization); the end-of-run report includes every merged PR link + a summary of what changed. The fleet-wide stale-test cleanup that the weekly /upgrade-all deliberately defers (it comments-only on stale tests; this skill closes the loop). Invoke as /cc-ci-tests-update.
|
||||
---
|
||||
|
||||
# cc-ci-tests-update (thin wrapper)
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: cc-ci-update
|
||||
description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with verified, ready-to-merge PRs and a healthy server at the end. NEVER merges anything. Invoke as /cc-ci-update.
|
||||
description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update.
|
||||
---
|
||||
|
||||
# cc-ci-update (thin wrapper)
|
||||
|
||||
@@ -0,0 +1,164 @@
|
||||
---
|
||||
name: cc-ci-orchestrator-update
|
||||
description: Update the cc-ci ORCHESTRATOR host (the machine the orchestrator itself runs on) to the latest nixos-26.05 nixpkgs, then nixos-rebuild test + switch + verify, ship the bump as a verified PR on recipe-maintainers/cc-ci-orchestrator and merge it directly (the skill invocation IS the authorization). The orchestrator-host analogue of /cc-ci-server-update. Self-update caveats apply — the switch restarts services on the very host running this session; always `nixos-rebuild test` before `switch` so a reboot recovers to the last-known-good generation. The end-of-run report includes the merged PR link + a summary of what changed. Invoke as /cc-ci-orchestrator-update (also answers to /cc-ci-update-orchestrator).
|
||||
---
|
||||
|
||||
# cc-ci-orchestrator-update
|
||||
|
||||
Manually update the **orchestrator host**'s NixOS dependencies (nixpkgs; sops-nix follows nixpkgs)
|
||||
to the current `nixos-26.05` channel tip, ship the bump via a verified PR + a live
|
||||
`nixos-rebuild test` → `switch`, and confirm the host is healthy afterward. This is the
|
||||
orchestrator-host analogue of `/cc-ci-server-update` — done deliberately, with a reviewable PR and
|
||||
a health gate, not as silent drift.
|
||||
|
||||
> **Two hosts, two flakes — don't confuse them.** This skill updates the **orchestrator** host:
|
||||
> the machine this session runs on (`cc-ci-orchestrator-1`, Hetzner cpx22 **server 134487234**,
|
||||
> tailnet `100.84.190.30`, public `168.119.126.100`), flake checkout **`/srv/cc-ci-orch`** (repo
|
||||
> `recipe-maintainers/cc-ci-orchestrator`), target **`.#cc-ci-orchestrator-hetzner`**. The **cc-ci
|
||||
> CI server** (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, target `.#cc-ci`) is a different
|
||||
> machine — that's `/cc-ci-server-update`, NOT this skill.
|
||||
|
||||
> **This is a SELF-update.** The `switch` restarts units on the host running this very session.
|
||||
> tmux sessions and the orchestrator survive a switch (systemd restarts don't kill user tmux), but
|
||||
> `tailscaled`/`sshd` restarts can briefly drop Remote Control / SSH — run activation steps
|
||||
> detached (`setsid nohup … &`) and re-poll. A full **reboot** DOES end the session: the
|
||||
> auto-launch flow (REBOOTS.md + `cc-ci-loops.service`) restarts the loops, and the operator
|
||||
> reconnects the orchestrator — announce a planned reboot to the operator first.
|
||||
|
||||
## What this skill does NOT do
|
||||
|
||||
- **Does NOT touch the cc-ci CI server** — that's `/cc-ci-server-update`.
|
||||
- **Does NOT bump recipe images or cc-ci tests** — `/upgrade-all` / `/cc-ci-tests-update`.
|
||||
- **Does NOT change the channel** (`nixos-26.05` branch ref in `flake.nix`) without operator
|
||||
choice — a channel jump (e.g. → 26.11 when it releases) is a bigger change: ask first, read the
|
||||
release notes, and expect module/option renames.
|
||||
|
||||
## Preconditions
|
||||
|
||||
- You are on the orchestrator host; `/srv/cc-ci-orch` is the live checkout (git repo, remote
|
||||
`recipe-maintainers/cc-ci-orchestrator`).
|
||||
- Quiescent moment: no `/upgrade-all` in flight, no active build-loop phase, no cc-ci CI run that
|
||||
would be disturbed if this host blips (`pgrep -af run_recipe_ci` empty on cc-ci is a bonus, but
|
||||
the orchestrator host restarting mostly risks its own sessions, not the CI server).
|
||||
- `git status` in `/srv/cc-ci-orch` — note pre-existing dirt; don't sweep unrelated changes into
|
||||
the bump commit.
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Snapshot baseline
|
||||
|
||||
```
|
||||
nixos-version; readlink /run/current-system; uname -r
|
||||
systemctl --failed --no-legend
|
||||
df -h / | tail -1
|
||||
cd /srv/cc-ci-orch && nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
|
||||
tmux ls
|
||||
```
|
||||
Record: current nixpkgs rev, generation, failed units (should be 0), disk, live tmux sessions.
|
||||
|
||||
### 2. Branch + bump the lock
|
||||
|
||||
Work directly in `/srv/cc-ci-orch` on a branch (it's the live checkout; the flake edit is inert
|
||||
until a rebuild):
|
||||
```
|
||||
cd /srv/cc-ci-orch
|
||||
git checkout -b chore/orchestrator-flake-update-$(date -u +%Y%m%d)
|
||||
nix flake update 2>&1 | tail -5 # nixpkgs tracks nixos-26.05; sops-nix follows nixpkgs
|
||||
nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
|
||||
```
|
||||
Note old→new revs. If the lock didn't move, report "already at channel tip" and stop (nothing to
|
||||
deploy).
|
||||
|
||||
### 3. Build (catch errors before any activation)
|
||||
|
||||
```
|
||||
cd /srv/cc-ci-orch && nixos-rebuild build --flake .#cc-ci-orchestrator-hetzner 2>&1 | tail -15
|
||||
readlink -f result
|
||||
```
|
||||
Build failure → fix on the branch (option renames etc.) before going further. Never activate a
|
||||
build that didn't build cleanly.
|
||||
|
||||
### 4. Test-activate (MANDATORY before switch — general policy)
|
||||
|
||||
`nixos-rebuild test` activates without touching the bootloader or system profile, so if the
|
||||
activation breaks the host (cf. the cc-ci server's 2026-08-03 no-default-route outage), a plain
|
||||
reboot — Hetzner API power-cycle on server **134487234** if SSH is gone (see
|
||||
`hetzner-server-recovery`) — lands back on the last-known-good generation.
|
||||
```
|
||||
cd /srv/cc-ci-orch && setsid nohup nixos-rebuild test --flake .#cc-ci-orchestrator-hetzner \
|
||||
> /tmp/orchestrator-test-switch.log 2>&1 < /dev/null & echo launched
|
||||
# after it settles (poll; tailscaled/sshd may blip):
|
||||
readlink /run/current-system # should be the new store path
|
||||
ip route show default # default route survived
|
||||
systemctl --failed --no-legend # 0 failed
|
||||
tmux ls # sessions survived
|
||||
tail -5 /tmp/orchestrator-test-switch.log
|
||||
```
|
||||
Regression here → you're still on the old boot default: diagnose, fix, or reboot to revert. Do NOT
|
||||
switch.
|
||||
|
||||
### 5. Switch (make permanent — only after 4 is healthy)
|
||||
|
||||
```
|
||||
cd /srv/cc-ci-orch && nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner 2>&1 | tail -10
|
||||
```
|
||||
(If it fails with "Unit nixos-rebuild-switch-to-configuration.service was already loaded", the
|
||||
detached test's transient unit is still running — wait or `systemctl stop` it, then retry.)
|
||||
|
||||
### 6. Health gate (do not skip)
|
||||
|
||||
```
|
||||
nixos-version; readlink /run/current-system
|
||||
systemctl --failed --no-legend
|
||||
tmux ls # orchestrator + loop sessions intact
|
||||
tailscale status | head -3 # tailnet up
|
||||
systemctl list-timers cc-ci-upgrade-all.timer --no-pager | head -3
|
||||
ssh cc-ci 'hostname' 2>&1 # can still reach the CI server
|
||||
df -h / | tail -1
|
||||
```
|
||||
Optional (operator's call — it ends this session): a cold-boot proof reboot. If done, the
|
||||
REBOOTS.md auto-launch flow restarts the loops; the operator reconnects the orchestrator.
|
||||
|
||||
### 7. Commit + PR + merge + report
|
||||
|
||||
```
|
||||
cd /srv/cc-ci-orch
|
||||
git add flake.lock # flake.nix too if the channel ref changed
|
||||
git commit -m "flake: bump nixpkgs (nixos-26.05, $(date -u +%Y-%m-%d))
|
||||
|
||||
nixpkgs: <old-rev[:8]> -> <new-rev[:8]> (nixos-26.05 tip)
|
||||
Deployed to cc-ci-orchestrator-hetzner: build + test + switch + health gate green."
|
||||
git push -u origin HEAD
|
||||
```
|
||||
Open the PR on `recipe-maintainers/cc-ci-orchestrator` (Gitea API with the `GITEA_*` creds from
|
||||
`/srv/cc-ci/.testenv`, same as `open-cc-ci-pr.sh` does for cc-ci — the helper is hardcoded to the
|
||||
cc-ci repo, so use the API directly or a copy with the repo swapped). PR body: old→new revs, what
|
||||
was verified. Then **merge it directly** — the PR is for visibility/historical record; the skill
|
||||
invocation is the authorization. On a rolled-back/failed run, leave the PR open with an
|
||||
explanatory comment.
|
||||
|
||||
Back on the host, return the checkout to main so the live tree matches the merged state:
|
||||
```
|
||||
git checkout main && git pull
|
||||
```
|
||||
|
||||
Write `/srv/cc-ci-orch/.cc-ci-logs/orchestrator-update-<YYYY-MM-DD>.md`
|
||||
(`RESULT: SUCCESS/ROLLED-BACK/FAILED …`), append a line to `cc-ci-plan/JOURNAL.md`, and report to
|
||||
the operator with the **merged PR link + a summary of what changed**.
|
||||
|
||||
## Guardrails
|
||||
|
||||
- **Build before test, test before switch** — never `switch` without a healthy `test` activation
|
||||
(general policy for this host and the cc-ci server; `test` keeps the bootloader untouched so a
|
||||
reboot always recovers).
|
||||
- **Detach activations** (`setsid nohup … &`) — a dropped SSH/Remote-Control connection must not
|
||||
kill a half-applied activation.
|
||||
- **Self-update awareness:** announce before any reboot (it ends the operator's session); a
|
||||
switch is fine but expect brief tailscaled/sshd blips.
|
||||
- **Single-writer:** dedicated branch; don't sweep unrelated working-tree changes into the bump
|
||||
commit; return the live checkout to `main` after the merge.
|
||||
- **Stable channel only** (`nixos-26.05` until the operator chooses a jump).
|
||||
- **PRs for visibility, merged directly** once verified; failed runs leave the PR open. Always
|
||||
report merged-PR links + change summaries.
|
||||
- **Rescue path:** Hetzner server **134487234** — same rescue-mode procedure as
|
||||
`hetzner-server-recovery` (grub generations live in a submenu: `1>N` ids).
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: cc-ci-server-update
|
||||
description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main; never merges the cc-ci PR. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update.
|
||||
description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main. Opens the cc-ci PR for visibility/historical record and merges it directly once the deploy verifies (the skill invocation IS the authorization); the end-of-run report includes the merged PR link + a summary of what changed. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update.
|
||||
---
|
||||
|
||||
# cc-ci-server-update
|
||||
@@ -24,7 +24,9 @@ confirm the server is healthy afterward. This is the cc-ci-server analogue of a
|
||||
- **Does NOT touch cc-ci tests / harness / runner** — that's `/cc-ci-tests-update` or
|
||||
`/ci-dev-workflow`. This is a flake-input bump only (`flake.nix` input revs + `flake.lock`).
|
||||
- **Does NOT update the orchestrator host** — only the cc-ci server.
|
||||
- **Does NOT merge** the cc-ci PR — operator merges after review.
|
||||
- **Does merge the cc-ci PR itself** — the PR exists for visibility and historical record, and is
|
||||
merged directly at the end of a verified run (the skill invocation is the authorization; the
|
||||
operator reviews the merged PR afterwards via the link in the report).
|
||||
|
||||
## Preconditions / access
|
||||
|
||||
@@ -120,7 +122,7 @@ Verify the lock moved to the revs you intended. **Do NOT `nixos-rebuild build` f
|
||||
orchestrator** — the orchestrator's nixpkgs is different and the cc-ci host target won't build here
|
||||
(meaningfully). The real build happens on the cc-ci host in step 5.
|
||||
|
||||
### 4. Commit + open the cc-ci PR (never merge)
|
||||
### 4. Commit + open the cc-ci PR (merged in step 7 after the health gate)
|
||||
|
||||
```
|
||||
git add flake.nix flake.lock
|
||||
@@ -139,7 +141,8 @@ BODY_FILE=/tmp/cc-ci-server-update-pr-body.md \
|
||||
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh
|
||||
```
|
||||
(Write the PR body to `/tmp/cc-ci-server-update-pr-body.md` first: old→new revs, the channel, any
|
||||
breaking changes from the release notes, the rebuild plan, and "NOT merged — for operator review.")
|
||||
breaking changes from the release notes, and the rebuild plan. The PR is opened for visibility and
|
||||
historical record; it gets merged directly in step 7 once the deploy passes the health gate.)
|
||||
|
||||
Capture the `PR_URL`. **The cc-ci push-build lint gate will run on the branch** — if it's red on a
|
||||
flake-lock-only change, it's a pre-existing lint drift (not yours); note it in the PR body and
|
||||
@@ -180,7 +183,24 @@ ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild build --flake .#cc-ci' 2>&1 |
|
||||
module option renames across channels (check the release notes), python package removals, a
|
||||
sops-nix incompatibility (go back to step 3b and pick a compatible sops-nix rev).
|
||||
|
||||
**(d) Switch (the live deploy):**
|
||||
**(d) Test-activate FIRST (general policy for nix deploys to this server AND the orchestrator
|
||||
host): `nixos-rebuild test` before any `switch`:**
|
||||
```
|
||||
# detach it so a dropped SSH can't kill the activation mid-flight:
|
||||
ssh cc-ci 'cd /root/cc-ci-deploy && setsid nohup nixos-rebuild test --flake .#cc-ci \
|
||||
> /root/cc-ci-deploy/test-switch.log 2>&1 < /dev/null & echo launched'
|
||||
# poll until reachable again, then check the essentials survived:
|
||||
ssh cc-ci 'readlink /run/current-system; ip route show default; systemctl --failed --no-legend'
|
||||
```
|
||||
`test` activates the new configuration **without touching the bootloader or the system profile**,
|
||||
so if the activation kills networking (as the 2026-08-03 26.05 bump did — no default route), a
|
||||
plain reboot (Hetzner API power-cycle if SSH is gone) lands back on the last-known-good
|
||||
generation. Only proceed to `switch` once the test activation is up, reachable, and healthy.
|
||||
Note: the detached test leaves a transient `nixos-rebuild-switch-to-configuration` unit; if the
|
||||
follow-up `switch` fails with "Unit ... was already loaded", wait for it to finish or
|
||||
`systemctl stop nixos-rebuild-switch-to-configuration.service` first.
|
||||
|
||||
**(e) Switch (make it permanent — only after (d) verified):**
|
||||
```
|
||||
ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild switch --flake .#cc-ci' 2>&1 | tail -30
|
||||
```
|
||||
@@ -215,17 +235,21 @@ If **anything regressed**: diagnose. A real regression → roll back to the prev
|
||||
flake input change on the PR branch is still your source of truth, but the running host reverts
|
||||
while you fix). Report the rollback in the PR.
|
||||
|
||||
### 7. Report
|
||||
### 7. Merge the PR + report
|
||||
|
||||
On a **verified** run (health gate green): post a PR comment summarizing what was deployed (revs,
|
||||
any incidents, verification results), then **merge the PR directly** — the skill invocation is the
|
||||
authorization; the PR exists for visibility and historical record. On a rolled-back or failed run,
|
||||
leave the PR **open** with a comment explaining the state.
|
||||
|
||||
Print + record a one-line result + the PR URL. Write a short note to
|
||||
`/srv/cc-ci/.cc-ci-logs/server-update-<YYYY-MM-DD>.md`:
|
||||
- `RESULT: SUCCESS — cc-ci server rebuilt on nixpkgs <new-rev[:8]>, no failed units, endpoints 200, PR: <url>`
|
||||
- `RESULT: SUCCESS — cc-ci server rebuilt on nixpkgs <new-rev[:8]>, no failed units, endpoints 200, PR merged: <url>`
|
||||
- `RESULT: ROLLED-BACK — nixpkgs <new-rev> rebuild failed/ regressed (<reason>); host reverted to <old-rev>; PR <url> left open`
|
||||
- `RESULT: FAILED — <step>: <reason>`
|
||||
|
||||
Always state that **the cc-ci PR is NOT merged** — it's the record of the bump; the operator
|
||||
reviews/merges at their discretion (the live rebuild already happened, so the PR is documentation +
|
||||
the path to re-apply, not a gate on the deploy).
|
||||
The end-of-run report to the operator MUST include **links to every merged PR** plus a summary of
|
||||
what changed in each, so the operator can review them after the fact if they want to.
|
||||
|
||||
### 8. Optional cold-boot proof
|
||||
|
||||
@@ -249,8 +273,13 @@ are tracked in its own journal; a note in the server-update report is enough).
|
||||
- **Single-writer:** dedicated branch in a separate clone; never push `main`; never touch the
|
||||
build loops' `/cc-ci` / `/cc-ci-adv` clones or `/root/builder-clone` (stage to
|
||||
`/root/cc-ci-deploy` instead).
|
||||
- **Never merge the cc-ci PR.** The live rebuild is the deploy; the PR is the reviewable record.
|
||||
- **Build before switch.** Never `nixos-rebuild switch` a build that didn't build cleanly first.
|
||||
- **Open the PR for visibility, merge it directly.** The live rebuild is the deploy; the PR is the
|
||||
reviewable historical record. Merge only after the health gate passes; a rolled-back/failed run
|
||||
leaves the PR open with an explanatory comment. Always report merged-PR links + change summaries.
|
||||
- **Build before test, test before switch.** Never `nixos-rebuild switch` a build that didn't build
|
||||
cleanly first, and never `switch` without a healthy `nixos-rebuild test` activation first — `test`
|
||||
leaves the bootloader untouched, so a reboot always recovers to the last-known-good generation.
|
||||
This is general policy for nix deploys to the cc-ci server **and** the orchestrator host.
|
||||
- **Health gate is mandatory.** No "it switched, ship it" — run the step-6 verification and compare
|
||||
to the baseline. Roll back on regression.
|
||||
- **Stable channel only** for the cc-ci server (never `nixos-unstable`).
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: cc-ci-tests-update
|
||||
description: Sweep all maintained recipes for stale cc-ci tests and update the stale ones so they pass again, verified via the standard PR + !testme -> CI flow. Runs the deterministic test suite across every enrolled recipe, identifies failures caused by a genuinely-stale test (the recipe legitimately changed upstream and the cc-ci test/overlay still asserts the old behavior), authors the minimal test update, opens a cc-ci test PR per stale recipe, and verifies each by running !testme on the corresponding recipe upgrade PR with the test change applied (or a direct harness run with the cc-ci branch checked out). NEVER weakens a test to force green; never merges. The fleet-wide stale-test cleanup that the weekly /upgrade-all deliberately defers (it comments-only on stale tests; this skill closes the loop). Invoke as /cc-ci-tests-update.
|
||||
description: Sweep all maintained recipes for stale cc-ci tests and update the stale ones so they pass again, verified via the standard PR + !testme -> CI flow. Runs the deterministic test suite across every enrolled recipe, identifies failures caused by a genuinely-stale test (the recipe legitimately changed upstream and the cc-ci test/overlay still asserts the old behavior), authors the minimal test update, opens a cc-ci test PR per stale recipe, and verifies each by running !testme on the corresponding recipe upgrade PR with the test change applied (or a direct harness run with the cc-ci branch checked out). NEVER weakens a test to force green. Opens each cc-ci test PR for visibility/historical record and merges it directly once verified green (the skill invocation IS the authorization); the end-of-run report includes every merged PR link + a summary of what changed. The fleet-wide stale-test cleanup that the weekly /upgrade-all deliberately defers (it comments-only on stale tests; this skill closes the loop). Invoke as /cc-ci-tests-update.
|
||||
---
|
||||
|
||||
# cc-ci-tests-update
|
||||
@@ -12,7 +12,8 @@ PR and stops (operator decides). Over time those commented stale tests accumulat
|
||||
**closes that loop**: it sweeps every maintained recipe, finds the ones whose cc-ci test is stale,
|
||||
authors the minimal test update, opens a cc-ci test PR per stale recipe, and **verifies** each via
|
||||
the standard `!testme` → CI flow (or a direct harness run with the test branch applied). It never
|
||||
weakens a test and never merges.
|
||||
weakens a test. Verified test PRs are merged directly — the PR is the visible, reviewable record;
|
||||
the invocation is the authorization — and the report lists every merged PR + what changed.
|
||||
|
||||
> **Relationship to the other skills.** This is the fleet-wide version of the per-recipe
|
||||
> `--with-tests` path in `/recipe-upgrade` (step 5b), and the "TEST out-of-date" classification
|
||||
@@ -29,7 +30,10 @@ weakens a test and never merges.
|
||||
- **Does NOT fix real upgrade regressions** — if a recipe's `!testme` is red because the upgrade is
|
||||
broken (not because the test is stale), that's a `/recipe-upgrade` (step 5a) or `/ci-test-review`
|
||||
job. This skill only touches a test when the **upgrade is correct** and the **test is wrong**.
|
||||
- **Does NOT merge** any PR — operator merges after review.
|
||||
- **Does merge its own cc-ci test PRs** (once verified green) — they're opened for visibility and
|
||||
historical record; the invocation is the authorization, and the operator reviews the merged PRs
|
||||
via the links in the report. It does NOT merge the paired **recipe upgrade PRs** — those stay
|
||||
operator-owned (`/upgrade-all` scope).
|
||||
|
||||
## Preconditions / access
|
||||
|
||||
@@ -125,7 +129,7 @@ ruff check tests/<recipe>/ && ruff format --check tests/<recipe>/
|
||||
```
|
||||
Fix any lint errors you introduce (don't commit red lint).
|
||||
|
||||
### 3. Open the cc-ci test PR (never merge)
|
||||
### 3. Open the cc-ci test PR (merged in step 4 once verified)
|
||||
|
||||
```
|
||||
git add tests/<recipe>/
|
||||
@@ -145,8 +149,9 @@ BODY_FILE=/tmp/cc-ci-tests-<recipe>-pr-body.md \
|
||||
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh
|
||||
```
|
||||
The PR body must: name the stale test, explain what changed upstream, state that no assertion was
|
||||
weakened, link the **dependent recipe upgrade PR** (the test change only goes green once the recipe
|
||||
upgrade is also applied — they're a pair), and say "NOT merged — for operator review."
|
||||
weakened, and link the **dependent recipe upgrade PR** (the test change only goes green once the
|
||||
recipe upgrade is also applied — they're a pair). The PR is opened for visibility and historical
|
||||
record; it gets merged directly once verification is green (step 4).
|
||||
|
||||
Capture the `TEST_PR_URL`.
|
||||
|
||||
@@ -189,11 +194,11 @@ ssh cc-ci 'rm -rf /root/cc-ci-test-verify'
|
||||
re-verify. If still red after 3, leave the test PR open and report `FAILED — test update not green
|
||||
after 3 verify runs` with the failing evidence. Don't weaken the test to force green.
|
||||
|
||||
> **Re-verify with a real `!testme` on the recipe PR too?** If the recipe upgrade PR is still open,
|
||||
> posting a fresh `!testme` on it after the test PR merges would re-run with the merged tests —
|
||||
> but you can't merge the test PR (operator does). So the harness-branch verification in (a) is the
|
||||
> authoritative pre-merge proof. Leave a comment on the recipe upgrade PR noting the test PR is
|
||||
> open and verified-paired, so the operator knows merging both → green.
|
||||
> **Re-verify with a real `!testme` on the recipe PR too?** The harness-branch verification in (a)
|
||||
> is the authoritative pre-merge proof. After it's green, merge the test PR; if the paired recipe
|
||||
> upgrade PR is still open, optionally post a fresh `!testme` on it (now running with the merged
|
||||
> tests) as a belt-and-braces confirmation, and leave a comment noting the test PR is merged so
|
||||
> the operator knows merging the recipe PR → green.
|
||||
|
||||
### 5. Tear down any dev deploys (defence-in-depth)
|
||||
|
||||
@@ -212,24 +217,26 @@ Write `/srv/cc-ci/.cc-ci-logs/tests-update-<YYYY-MM-DD>.md` and print, leading w
|
||||
# cc-ci Stale-Test Update — <YYYY-MM-DD>
|
||||
## Summary
|
||||
- Swept: N recipes · Stale tests found: N · Test PRs opened + verified: N · Failed: N · Not-stale (dropped): N
|
||||
## Test PRs to review (NOT merged)
|
||||
| recipe | stale test | upstream change | test PR | verified |
|
||||
|--------|-----------|-----------------|---------|----------|
|
||||
## Test PRs (merged — links for after-the-fact review)
|
||||
| recipe | stale test | upstream change | test PR (merged) | verified |
|
||||
|--------|-----------|-----------------|------------------|----------|
|
||||
| <recipe> | <test-name> | <what changed> | <url> | GREEN (paired with recipe PR <url>) |
|
||||
## Failed (needs work)
|
||||
- <recipe> — <test-name>: <reason> (test PR <url> left open, red)
|
||||
## Not-stale (dropped — was flaky / real recipe bug / CI-server bug)
|
||||
- <recipe> — <test-name>: <classification> (handed off / noted)
|
||||
```
|
||||
End with the report path and a reminder that **nothing was merged** — the test PRs (and their
|
||||
paired recipe upgrade PRs) await operator review.
|
||||
End with the report path and **links to every merged test PR + a summary of what changed in each**
|
||||
(so the operator can review after the fact). Failed/red test PRs are left open, not merged; the
|
||||
paired recipe upgrade PRs remain operator-owned and unmerged.
|
||||
|
||||
## Guardrails
|
||||
|
||||
- **Never weaken a test** to turn a red green. The fix makes the test correct for the new upstream
|
||||
behavior; if that's not possible without weakening, stop and report.
|
||||
- **Create + verify, NEVER merge.** Each test PR (and any paired recipe upgrade PR) is
|
||||
operator-merged after review.
|
||||
- **Create for visibility, verify, then merge directly.** Each test PR is merged only after its
|
||||
verification is green (the invocation is the authorization); red/unverified PRs stay open. The
|
||||
paired recipe upgrade PRs are NOT yours to merge. Always report merged-PR links + summaries.
|
||||
- **Single-writer:** dedicated branch in a separate clone; never push `main`; never touch the
|
||||
build loops' `/cc-ci` `/cc-ci-adv` clones or `/root/builder-clone`. Use `/root/cc-ci-test-verify`
|
||||
on the host and `rm -rf` it after.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: cc-ci-update
|
||||
description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with verified, ready-to-merge PRs and a healthy server at the end. NEVER merges anything. Invoke as /cc-ci-update.
|
||||
description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update.
|
||||
---
|
||||
|
||||
# cc-ci-update
|
||||
|
||||
@@ -36,7 +36,12 @@ a **Hetzner `cpx22`** cloud server (`cc-ci-orchestrator-1`, tailnet `100.84.190.
|
||||
`168.119.126.100`, flake host `cc-ci-orchestrator-hetzner`) — see
|
||||
`cc-ci-plan/plan-orchestrator-hetzner-migration.md`. The earlier Pi→Incus-VM move is the historical
|
||||
`cc-ci-plan/plan-orchestrator-migration.md`. Rebuild this host with
|
||||
`nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner` from `/srv/cc-ci-orch`.
|
||||
`nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner` from `/srv/cc-ci-orch` — but **always
|
||||
`nixos-rebuild test` the same flake target first and verify the host is still healthy/reachable
|
||||
before the `switch`** (general policy for nix deploys to this host and the cc-ci server: `test`
|
||||
leaves the bootloader and system profile untouched, so a reboot always recovers to the
|
||||
last-known-good generation; the 2026-08-03 cc-ci 26.05 bump outage is the cautionary tale, see
|
||||
`.cc-ci-logs/server-update-2026-08-03.md`).
|
||||
|
||||
## Keep the orchestrator open, under remote-control
|
||||
|
||||
|
||||
@@ -763,3 +763,31 @@ session cc-ci-orchestrator-stale can be killed; recipe-mirrors org still private
|
||||
session; re-oriented from STATUS/journals via a nudge). Relayed the operator's `${STACK_NAME}_app`
|
||||
bluesky guidance. 4 recipe PRs + 2 harness fixes; **nothing merged** (operator reviews/merges).
|
||||
- Queue drained again → hourly wake stops.
|
||||
|
||||
## 2026-08-03 ~20:30 UTC — /cc-ci-server-update: 24.11 → 26.05 (outage + recovery + success)
|
||||
|
||||
- Session: primary opencode orchestrator started the run; its session was stopped mid-incident and
|
||||
the **cc-ci-orchestrator-backup** (Claude Code / Fable, session `cc-ci-backup-orchestrator`) was
|
||||
operator-promoted to take over and finish.
|
||||
- Flake bump: nixpkgs 50ab793 (nixos-24.11 tip) → 531670d (nixos-26.05), sops-nix 77c423a →
|
||||
f140661 (master). PR #11, **merged** (new policy: PRs for visibility, merged directly, links in
|
||||
report): https://git.autonomic.zone/recipe-maintainers/cc-ci/pulls/11
|
||||
- **Outage:** first switch left the host with no default route (~19:30–20:12 UTC). Root cause: NixOS
|
||||
25.05+ scripted networking installs the default route from the gateway iface's
|
||||
network-addresses unit; bare-string `defaultGateway` + off-subnet /32 Hetzner gateway → no iface
|
||||
matched → no route. Fix: explicit `defaultGateway.interface = "eth0"` (commit c541cb1).
|
||||
- Recovery detail worth remembering: NixOS grub generations live in a **submenu** — grubenv
|
||||
`next_entry`/`default` need `1>N` ids; a bare index falls back to the (broken) default entry.
|
||||
Clear any grubenv override after the next switch regenerates grub.cfg (indices shift).
|
||||
- Final: gen 24, 26.05.20260803.531670d, 0 failed units, services 1/1 (warm spares flapping as
|
||||
pre-update), report site 200, cold-boot proven (40s). Log:
|
||||
.cc-ci-logs/server-update-2026-08-03.md
|
||||
- Policy change (operator): /cc-ci-server-update, /cc-ci-tests-update, /cc-ci-update skills updated
|
||||
in both .opencode and .claude positions — open PRs for visibility, merge directly once verified
|
||||
(invocation = authorization), always report merged-PR links + change summaries.
|
||||
- TODO for operator: rotate the Hetzner Cloud API token pasted into the orchestrator conversation
|
||||
during recovery.
|
||||
- Addendum (operator policy, 2026-08-03): **always `nixos-rebuild test` before `switch`** for nix
|
||||
deploys to the cc-ci server and the orchestrator host — `test` leaves the bootloader/profile
|
||||
untouched so a reboot always recovers. Codified in the cc-ci-server-update skill (step 5d) and
|
||||
AGENTS.md (orchestrator rebuild instructions).
|
||||
|
||||
Reference in New Issue
Block a user