diff --git a/.claude/skills/cc-ci-orchestrator-update/SKILL.md b/.claude/skills/cc-ci-orchestrator-update/SKILL.md new file mode 100644 index 0000000..a7b2d7e --- /dev/null +++ b/.claude/skills/cc-ci-orchestrator-update/SKILL.md @@ -0,0 +1,14 @@ +--- +name: cc-ci-orchestrator-update +description: Update the cc-ci ORCHESTRATOR host (the machine the orchestrator itself runs on) to the latest nixos-26.05 nixpkgs, then nixos-rebuild test + switch + verify, ship the bump as a verified PR on recipe-maintainers/cc-ci-orchestrator and merge it directly (the skill invocation IS the authorization). The orchestrator-host analogue of /cc-ci-server-update. Self-update caveats apply — the switch restarts services on the very host running this session; always `nixos-rebuild test` before `switch` so a reboot recovers to the last-known-good generation. The end-of-run report includes the merged PR link + a summary of what changed. Invoke as /cc-ci-orchestrator-update (also answers to /cc-ci-update-orchestrator). +--- + +# cc-ci-orchestrator-update (thin wrapper) + +The canonical definition of this skill lives in the **opencode** position: + +**`.opencode/skills/cc-ci-orchestrator-update/SKILL.md`** + +Read that file for the full procedure. This `.claude/skills/` copy is kept as a +thin pointer for Claude Code compatibility; opencode loads the canonical +definition from `.opencode/skills/` directly. diff --git a/.claude/skills/cc-ci-server-update/SKILL.md b/.claude/skills/cc-ci-server-update/SKILL.md index 7fbba78..40ba9e5 100644 --- a/.claude/skills/cc-ci-server-update/SKILL.md +++ b/.claude/skills/cc-ci-server-update/SKILL.md @@ -1,6 +1,6 @@ --- name: cc-ci-server-update -description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main; never merges the cc-ci PR. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update. +description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main. Opens the cc-ci PR for visibility/historical record and merges it directly once the deploy verifies (the skill invocation IS the authorization); the end-of-run report includes the merged PR link + a summary of what changed. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update. --- # cc-ci-server-update (thin wrapper) diff --git a/.claude/skills/cc-ci-tests-update/SKILL.md b/.claude/skills/cc-ci-tests-update/SKILL.md index 2e8f71d..0ca3410 100644 --- a/.claude/skills/cc-ci-tests-update/SKILL.md +++ b/.claude/skills/cc-ci-tests-update/SKILL.md @@ -1,6 +1,6 @@ --- name: cc-ci-tests-update -description: Sweep all maintained recipes for stale cc-ci tests and update the stale ones so they pass again, verified via the standard PR + !testme -> CI flow. Runs the deterministic test suite across every enrolled recipe, identifies failures caused by a genuinely-stale test (the recipe legitimately changed upstream and the cc-ci test/overlay still asserts the old behavior), authors the minimal test update, opens a cc-ci test PR per stale recipe, and verifies each by running !testme on the corresponding recipe upgrade PR with the test change applied (or a direct harness run with the cc-ci branch checked out). NEVER weakens a test to force green; never merges. The fleet-wide stale-test cleanup that the weekly /upgrade-all deliberately defers (it comments-only on stale tests; this skill closes the loop). Invoke as /cc-ci-tests-update. +description: Sweep all maintained recipes for stale cc-ci tests and update the stale ones so they pass again, verified via the standard PR + !testme -> CI flow. Runs the deterministic test suite across every enrolled recipe, identifies failures caused by a genuinely-stale test (the recipe legitimately changed upstream and the cc-ci test/overlay still asserts the old behavior), authors the minimal test update, opens a cc-ci test PR per stale recipe, and verifies each by running !testme on the corresponding recipe upgrade PR with the test change applied (or a direct harness run with the cc-ci branch checked out). NEVER weakens a test to force green. Opens each cc-ci test PR for visibility/historical record and merges it directly once verified green (the skill invocation IS the authorization); the end-of-run report includes every merged PR link + a summary of what changed. The fleet-wide stale-test cleanup that the weekly /upgrade-all deliberately defers (it comments-only on stale tests; this skill closes the loop). Invoke as /cc-ci-tests-update. --- # cc-ci-tests-update (thin wrapper) diff --git a/.claude/skills/cc-ci-update/SKILL.md b/.claude/skills/cc-ci-update/SKILL.md index 67a6ced..5a84b8f 100644 --- a/.claude/skills/cc-ci-update/SKILL.md +++ b/.claude/skills/cc-ci-update/SKILL.md @@ -1,6 +1,6 @@ --- name: cc-ci-update -description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with verified, ready-to-merge PRs and a healthy server at the end. NEVER merges anything. Invoke as /cc-ci-update. +description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update. --- # cc-ci-update (thin wrapper) diff --git a/.opencode/skills/cc-ci-orchestrator-update/SKILL.md b/.opencode/skills/cc-ci-orchestrator-update/SKILL.md new file mode 100644 index 0000000..4b9a2e0 --- /dev/null +++ b/.opencode/skills/cc-ci-orchestrator-update/SKILL.md @@ -0,0 +1,164 @@ +--- +name: cc-ci-orchestrator-update +description: Update the cc-ci ORCHESTRATOR host (the machine the orchestrator itself runs on) to the latest nixos-26.05 nixpkgs, then nixos-rebuild test + switch + verify, ship the bump as a verified PR on recipe-maintainers/cc-ci-orchestrator and merge it directly (the skill invocation IS the authorization). The orchestrator-host analogue of /cc-ci-server-update. Self-update caveats apply — the switch restarts services on the very host running this session; always `nixos-rebuild test` before `switch` so a reboot recovers to the last-known-good generation. The end-of-run report includes the merged PR link + a summary of what changed. Invoke as /cc-ci-orchestrator-update (also answers to /cc-ci-update-orchestrator). +--- + +# cc-ci-orchestrator-update + +Manually update the **orchestrator host**'s NixOS dependencies (nixpkgs; sops-nix follows nixpkgs) +to the current `nixos-26.05` channel tip, ship the bump via a verified PR + a live +`nixos-rebuild test` → `switch`, and confirm the host is healthy afterward. This is the +orchestrator-host analogue of `/cc-ci-server-update` — done deliberately, with a reviewable PR and +a health gate, not as silent drift. + +> **Two hosts, two flakes — don't confuse them.** This skill updates the **orchestrator** host: +> the machine this session runs on (`cc-ci-orchestrator-1`, Hetzner cpx22 **server 134487234**, +> tailnet `100.84.190.30`, public `168.119.126.100`), flake checkout **`/srv/cc-ci-orch`** (repo +> `recipe-maintainers/cc-ci-orchestrator`), target **`.#cc-ci-orchestrator-hetzner`**. The **cc-ci +> CI server** (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, target `.#cc-ci`) is a different +> machine — that's `/cc-ci-server-update`, NOT this skill. + +> **This is a SELF-update.** The `switch` restarts units on the host running this very session. +> tmux sessions and the orchestrator survive a switch (systemd restarts don't kill user tmux), but +> `tailscaled`/`sshd` restarts can briefly drop Remote Control / SSH — run activation steps +> detached (`setsid nohup … &`) and re-poll. A full **reboot** DOES end the session: the +> auto-launch flow (REBOOTS.md + `cc-ci-loops.service`) restarts the loops, and the operator +> reconnects the orchestrator — announce a planned reboot to the operator first. + +## What this skill does NOT do + +- **Does NOT touch the cc-ci CI server** — that's `/cc-ci-server-update`. +- **Does NOT bump recipe images or cc-ci tests** — `/upgrade-all` / `/cc-ci-tests-update`. +- **Does NOT change the channel** (`nixos-26.05` branch ref in `flake.nix`) without operator + choice — a channel jump (e.g. → 26.11 when it releases) is a bigger change: ask first, read the + release notes, and expect module/option renames. + +## Preconditions + +- You are on the orchestrator host; `/srv/cc-ci-orch` is the live checkout (git repo, remote + `recipe-maintainers/cc-ci-orchestrator`). +- Quiescent moment: no `/upgrade-all` in flight, no active build-loop phase, no cc-ci CI run that + would be disturbed if this host blips (`pgrep -af run_recipe_ci` empty on cc-ci is a bonus, but + the orchestrator host restarting mostly risks its own sessions, not the CI server). +- `git status` in `/srv/cc-ci-orch` — note pre-existing dirt; don't sweep unrelated changes into + the bump commit. + +## Procedure + +### 1. Snapshot baseline + +``` +nixos-version; readlink /run/current-system; uname -r +systemctl --failed --no-legend +df -h / | tail -1 +cd /srv/cc-ci-orch && nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"' +tmux ls +``` +Record: current nixpkgs rev, generation, failed units (should be 0), disk, live tmux sessions. + +### 2. Branch + bump the lock + +Work directly in `/srv/cc-ci-orch` on a branch (it's the live checkout; the flake edit is inert +until a rebuild): +``` +cd /srv/cc-ci-orch +git checkout -b chore/orchestrator-flake-update-$(date -u +%Y%m%d) +nix flake update 2>&1 | tail -5 # nixpkgs tracks nixos-26.05; sops-nix follows nixpkgs +nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"' +``` +Note old→new revs. If the lock didn't move, report "already at channel tip" and stop (nothing to +deploy). + +### 3. Build (catch errors before any activation) + +``` +cd /srv/cc-ci-orch && nixos-rebuild build --flake .#cc-ci-orchestrator-hetzner 2>&1 | tail -15 +readlink -f result +``` +Build failure → fix on the branch (option renames etc.) before going further. Never activate a +build that didn't build cleanly. + +### 4. Test-activate (MANDATORY before switch — general policy) + +`nixos-rebuild test` activates without touching the bootloader or system profile, so if the +activation breaks the host (cf. the cc-ci server's 2026-08-03 no-default-route outage), a plain +reboot — Hetzner API power-cycle on server **134487234** if SSH is gone (see +`hetzner-server-recovery`) — lands back on the last-known-good generation. +``` +cd /srv/cc-ci-orch && setsid nohup nixos-rebuild test --flake .#cc-ci-orchestrator-hetzner \ + > /tmp/orchestrator-test-switch.log 2>&1 < /dev/null & echo launched +# after it settles (poll; tailscaled/sshd may blip): +readlink /run/current-system # should be the new store path +ip route show default # default route survived +systemctl --failed --no-legend # 0 failed +tmux ls # sessions survived +tail -5 /tmp/orchestrator-test-switch.log +``` +Regression here → you're still on the old boot default: diagnose, fix, or reboot to revert. Do NOT +switch. + +### 5. Switch (make permanent — only after 4 is healthy) + +``` +cd /srv/cc-ci-orch && nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner 2>&1 | tail -10 +``` +(If it fails with "Unit nixos-rebuild-switch-to-configuration.service was already loaded", the +detached test's transient unit is still running — wait or `systemctl stop` it, then retry.) + +### 6. Health gate (do not skip) + +``` +nixos-version; readlink /run/current-system +systemctl --failed --no-legend +tmux ls # orchestrator + loop sessions intact +tailscale status | head -3 # tailnet up +systemctl list-timers cc-ci-upgrade-all.timer --no-pager | head -3 +ssh cc-ci 'hostname' 2>&1 # can still reach the CI server +df -h / | tail -1 +``` +Optional (operator's call — it ends this session): a cold-boot proof reboot. If done, the +REBOOTS.md auto-launch flow restarts the loops; the operator reconnects the orchestrator. + +### 7. Commit + PR + merge + report + +``` +cd /srv/cc-ci-orch +git add flake.lock # flake.nix too if the channel ref changed +git commit -m "flake: bump nixpkgs (nixos-26.05, $(date -u +%Y-%m-%d)) + +nixpkgs: -> (nixos-26.05 tip) +Deployed to cc-ci-orchestrator-hetzner: build + test + switch + health gate green." +git push -u origin HEAD +``` +Open the PR on `recipe-maintainers/cc-ci-orchestrator` (Gitea API with the `GITEA_*` creds from +`/srv/cc-ci/.testenv`, same as `open-cc-ci-pr.sh` does for cc-ci — the helper is hardcoded to the +cc-ci repo, so use the API directly or a copy with the repo swapped). PR body: old→new revs, what +was verified. Then **merge it directly** — the PR is for visibility/historical record; the skill +invocation is the authorization. On a rolled-back/failed run, leave the PR open with an +explanatory comment. + +Back on the host, return the checkout to main so the live tree matches the merged state: +``` +git checkout main && git pull +``` + +Write `/srv/cc-ci-orch/.cc-ci-logs/orchestrator-update-.md` +(`RESULT: SUCCESS/ROLLED-BACK/FAILED …`), append a line to `cc-ci-plan/JOURNAL.md`, and report to +the operator with the **merged PR link + a summary of what changed**. + +## Guardrails + +- **Build before test, test before switch** — never `switch` without a healthy `test` activation + (general policy for this host and the cc-ci server; `test` keeps the bootloader untouched so a + reboot always recovers). +- **Detach activations** (`setsid nohup … &`) — a dropped SSH/Remote-Control connection must not + kill a half-applied activation. +- **Self-update awareness:** announce before any reboot (it ends the operator's session); a + switch is fine but expect brief tailscaled/sshd blips. +- **Single-writer:** dedicated branch; don't sweep unrelated working-tree changes into the bump + commit; return the live checkout to `main` after the merge. +- **Stable channel only** (`nixos-26.05` until the operator chooses a jump). +- **PRs for visibility, merged directly** once verified; failed runs leave the PR open. Always + report merged-PR links + change summaries. +- **Rescue path:** Hetzner server **134487234** — same rescue-mode procedure as + `hetzner-server-recovery` (grub generations live in a submenu: `1>N` ids). diff --git a/.opencode/skills/cc-ci-server-update/SKILL.md b/.opencode/skills/cc-ci-server-update/SKILL.md index 16da17e..fa20a9c 100644 --- a/.opencode/skills/cc-ci-server-update/SKILL.md +++ b/.opencode/skills/cc-ci-server-update/SKILL.md @@ -1,6 +1,6 @@ --- name: cc-ci-server-update -description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main; never merges the cc-ci PR. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update. +description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main. Opens the cc-ci PR for visibility/historical record and merges it directly once the deploy verifies (the skill invocation IS the authorization); the end-of-run report includes the merged PR link + a summary of what changed. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update. --- # cc-ci-server-update @@ -24,7 +24,9 @@ confirm the server is healthy afterward. This is the cc-ci-server analogue of a - **Does NOT touch cc-ci tests / harness / runner** — that's `/cc-ci-tests-update` or `/ci-dev-workflow`. This is a flake-input bump only (`flake.nix` input revs + `flake.lock`). - **Does NOT update the orchestrator host** — only the cc-ci server. -- **Does NOT merge** the cc-ci PR — operator merges after review. +- **Does merge the cc-ci PR itself** — the PR exists for visibility and historical record, and is + merged directly at the end of a verified run (the skill invocation is the authorization; the + operator reviews the merged PR afterwards via the link in the report). ## Preconditions / access @@ -120,7 +122,7 @@ Verify the lock moved to the revs you intended. **Do NOT `nixos-rebuild build` f orchestrator** — the orchestrator's nixpkgs is different and the cc-ci host target won't build here (meaningfully). The real build happens on the cc-ci host in step 5. -### 4. Commit + open the cc-ci PR (never merge) +### 4. Commit + open the cc-ci PR (merged in step 7 after the health gate) ``` git add flake.nix flake.lock @@ -139,7 +141,8 @@ BODY_FILE=/tmp/cc-ci-server-update-pr-body.md \ bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh ``` (Write the PR body to `/tmp/cc-ci-server-update-pr-body.md` first: old→new revs, the channel, any -breaking changes from the release notes, the rebuild plan, and "NOT merged — for operator review.") +breaking changes from the release notes, and the rebuild plan. The PR is opened for visibility and +historical record; it gets merged directly in step 7 once the deploy passes the health gate.) Capture the `PR_URL`. **The cc-ci push-build lint gate will run on the branch** — if it's red on a flake-lock-only change, it's a pre-existing lint drift (not yours); note it in the PR body and @@ -180,7 +183,24 @@ ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild build --flake .#cc-ci' 2>&1 | module option renames across channels (check the release notes), python package removals, a sops-nix incompatibility (go back to step 3b and pick a compatible sops-nix rev). -**(d) Switch (the live deploy):** +**(d) Test-activate FIRST (general policy for nix deploys to this server AND the orchestrator +host): `nixos-rebuild test` before any `switch`:** +``` +# detach it so a dropped SSH can't kill the activation mid-flight: +ssh cc-ci 'cd /root/cc-ci-deploy && setsid nohup nixos-rebuild test --flake .#cc-ci \ + > /root/cc-ci-deploy/test-switch.log 2>&1 < /dev/null & echo launched' +# poll until reachable again, then check the essentials survived: +ssh cc-ci 'readlink /run/current-system; ip route show default; systemctl --failed --no-legend' +``` +`test` activates the new configuration **without touching the bootloader or the system profile**, +so if the activation kills networking (as the 2026-08-03 26.05 bump did — no default route), a +plain reboot (Hetzner API power-cycle if SSH is gone) lands back on the last-known-good +generation. Only proceed to `switch` once the test activation is up, reachable, and healthy. +Note: the detached test leaves a transient `nixos-rebuild-switch-to-configuration` unit; if the +follow-up `switch` fails with "Unit ... was already loaded", wait for it to finish or +`systemctl stop nixos-rebuild-switch-to-configuration.service` first. + +**(e) Switch (make it permanent — only after (d) verified):** ``` ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild switch --flake .#cc-ci' 2>&1 | tail -30 ``` @@ -215,17 +235,21 @@ If **anything regressed**: diagnose. A real regression → roll back to the prev flake input change on the PR branch is still your source of truth, but the running host reverts while you fix). Report the rollback in the PR. -### 7. Report +### 7. Merge the PR + report + +On a **verified** run (health gate green): post a PR comment summarizing what was deployed (revs, +any incidents, verification results), then **merge the PR directly** — the skill invocation is the +authorization; the PR exists for visibility and historical record. On a rolled-back or failed run, +leave the PR **open** with a comment explaining the state. Print + record a one-line result + the PR URL. Write a short note to `/srv/cc-ci/.cc-ci-logs/server-update-.md`: -- `RESULT: SUCCESS — cc-ci server rebuilt on nixpkgs , no failed units, endpoints 200, PR: ` +- `RESULT: SUCCESS — cc-ci server rebuilt on nixpkgs , no failed units, endpoints 200, PR merged: ` - `RESULT: ROLLED-BACK — nixpkgs rebuild failed/ regressed (); host reverted to ; PR left open` - `RESULT: FAILED — : ` -Always state that **the cc-ci PR is NOT merged** — it's the record of the bump; the operator -reviews/merges at their discretion (the live rebuild already happened, so the PR is documentation + -the path to re-apply, not a gate on the deploy). +The end-of-run report to the operator MUST include **links to every merged PR** plus a summary of +what changed in each, so the operator can review them after the fact if they want to. ### 8. Optional cold-boot proof @@ -249,8 +273,13 @@ are tracked in its own journal; a note in the server-update report is enough). - **Single-writer:** dedicated branch in a separate clone; never push `main`; never touch the build loops' `/cc-ci` / `/cc-ci-adv` clones or `/root/builder-clone` (stage to `/root/cc-ci-deploy` instead). -- **Never merge the cc-ci PR.** The live rebuild is the deploy; the PR is the reviewable record. -- **Build before switch.** Never `nixos-rebuild switch` a build that didn't build cleanly first. +- **Open the PR for visibility, merge it directly.** The live rebuild is the deploy; the PR is the + reviewable historical record. Merge only after the health gate passes; a rolled-back/failed run + leaves the PR open with an explanatory comment. Always report merged-PR links + change summaries. +- **Build before test, test before switch.** Never `nixos-rebuild switch` a build that didn't build + cleanly first, and never `switch` without a healthy `nixos-rebuild test` activation first — `test` + leaves the bootloader untouched, so a reboot always recovers to the last-known-good generation. + This is general policy for nix deploys to the cc-ci server **and** the orchestrator host. - **Health gate is mandatory.** No "it switched, ship it" — run the step-6 verification and compare to the baseline. Roll back on regression. - **Stable channel only** for the cc-ci server (never `nixos-unstable`). diff --git a/.opencode/skills/cc-ci-tests-update/SKILL.md b/.opencode/skills/cc-ci-tests-update/SKILL.md index 3addcd5..cd0a211 100644 --- a/.opencode/skills/cc-ci-tests-update/SKILL.md +++ b/.opencode/skills/cc-ci-tests-update/SKILL.md @@ -1,6 +1,6 @@ --- name: cc-ci-tests-update -description: Sweep all maintained recipes for stale cc-ci tests and update the stale ones so they pass again, verified via the standard PR + !testme -> CI flow. Runs the deterministic test suite across every enrolled recipe, identifies failures caused by a genuinely-stale test (the recipe legitimately changed upstream and the cc-ci test/overlay still asserts the old behavior), authors the minimal test update, opens a cc-ci test PR per stale recipe, and verifies each by running !testme on the corresponding recipe upgrade PR with the test change applied (or a direct harness run with the cc-ci branch checked out). NEVER weakens a test to force green; never merges. The fleet-wide stale-test cleanup that the weekly /upgrade-all deliberately defers (it comments-only on stale tests; this skill closes the loop). Invoke as /cc-ci-tests-update. +description: Sweep all maintained recipes for stale cc-ci tests and update the stale ones so they pass again, verified via the standard PR + !testme -> CI flow. Runs the deterministic test suite across every enrolled recipe, identifies failures caused by a genuinely-stale test (the recipe legitimately changed upstream and the cc-ci test/overlay still asserts the old behavior), authors the minimal test update, opens a cc-ci test PR per stale recipe, and verifies each by running !testme on the corresponding recipe upgrade PR with the test change applied (or a direct harness run with the cc-ci branch checked out). NEVER weakens a test to force green. Opens each cc-ci test PR for visibility/historical record and merges it directly once verified green (the skill invocation IS the authorization); the end-of-run report includes every merged PR link + a summary of what changed. The fleet-wide stale-test cleanup that the weekly /upgrade-all deliberately defers (it comments-only on stale tests; this skill closes the loop). Invoke as /cc-ci-tests-update. --- # cc-ci-tests-update @@ -12,7 +12,8 @@ PR and stops (operator decides). Over time those commented stale tests accumulat **closes that loop**: it sweeps every maintained recipe, finds the ones whose cc-ci test is stale, authors the minimal test update, opens a cc-ci test PR per stale recipe, and **verifies** each via the standard `!testme` → CI flow (or a direct harness run with the test branch applied). It never -weakens a test and never merges. +weakens a test. Verified test PRs are merged directly — the PR is the visible, reviewable record; +the invocation is the authorization — and the report lists every merged PR + what changed. > **Relationship to the other skills.** This is the fleet-wide version of the per-recipe > `--with-tests` path in `/recipe-upgrade` (step 5b), and the "TEST out-of-date" classification @@ -29,7 +30,10 @@ weakens a test and never merges. - **Does NOT fix real upgrade regressions** — if a recipe's `!testme` is red because the upgrade is broken (not because the test is stale), that's a `/recipe-upgrade` (step 5a) or `/ci-test-review` job. This skill only touches a test when the **upgrade is correct** and the **test is wrong**. -- **Does NOT merge** any PR — operator merges after review. +- **Does merge its own cc-ci test PRs** (once verified green) — they're opened for visibility and + historical record; the invocation is the authorization, and the operator reviews the merged PRs + via the links in the report. It does NOT merge the paired **recipe upgrade PRs** — those stay + operator-owned (`/upgrade-all` scope). ## Preconditions / access @@ -125,7 +129,7 @@ ruff check tests// && ruff format --check tests// ``` Fix any lint errors you introduce (don't commit red lint). -### 3. Open the cc-ci test PR (never merge) +### 3. Open the cc-ci test PR (merged in step 4 once verified) ``` git add tests// @@ -145,8 +149,9 @@ BODY_FILE=/tmp/cc-ci-tests--pr-body.md \ bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh ``` The PR body must: name the stale test, explain what changed upstream, state that no assertion was -weakened, link the **dependent recipe upgrade PR** (the test change only goes green once the recipe -upgrade is also applied — they're a pair), and say "NOT merged — for operator review." +weakened, and link the **dependent recipe upgrade PR** (the test change only goes green once the +recipe upgrade is also applied — they're a pair). The PR is opened for visibility and historical +record; it gets merged directly once verification is green (step 4). Capture the `TEST_PR_URL`. @@ -189,11 +194,11 @@ ssh cc-ci 'rm -rf /root/cc-ci-test-verify' re-verify. If still red after 3, leave the test PR open and report `FAILED — test update not green after 3 verify runs` with the failing evidence. Don't weaken the test to force green. -> **Re-verify with a real `!testme` on the recipe PR too?** If the recipe upgrade PR is still open, -> posting a fresh `!testme` on it after the test PR merges would re-run with the merged tests — -> but you can't merge the test PR (operator does). So the harness-branch verification in (a) is the -> authoritative pre-merge proof. Leave a comment on the recipe upgrade PR noting the test PR is -> open and verified-paired, so the operator knows merging both → green. +> **Re-verify with a real `!testme` on the recipe PR too?** The harness-branch verification in (a) +> is the authoritative pre-merge proof. After it's green, merge the test PR; if the paired recipe +> upgrade PR is still open, optionally post a fresh `!testme` on it (now running with the merged +> tests) as a belt-and-braces confirmation, and leave a comment noting the test PR is merged so +> the operator knows merging the recipe PR → green. ### 5. Tear down any dev deploys (defence-in-depth) @@ -212,24 +217,26 @@ Write `/srv/cc-ci/.cc-ci-logs/tests-update-.md` and print, leading w # cc-ci Stale-Test Update — ## Summary - Swept: N recipes · Stale tests found: N · Test PRs opened + verified: N · Failed: N · Not-stale (dropped): N -## Test PRs to review (NOT merged) -| recipe | stale test | upstream change | test PR | verified | -|--------|-----------|-----------------|---------|----------| +## Test PRs (merged — links for after-the-fact review) +| recipe | stale test | upstream change | test PR (merged) | verified | +|--------|-----------|-----------------|------------------|----------| | | | | | GREEN (paired with recipe PR ) | ## Failed (needs work) - — : (test PR left open, red) ## Not-stale (dropped — was flaky / real recipe bug / CI-server bug) - — : (handed off / noted) ``` -End with the report path and a reminder that **nothing was merged** — the test PRs (and their -paired recipe upgrade PRs) await operator review. +End with the report path and **links to every merged test PR + a summary of what changed in each** +(so the operator can review after the fact). Failed/red test PRs are left open, not merged; the +paired recipe upgrade PRs remain operator-owned and unmerged. ## Guardrails - **Never weaken a test** to turn a red green. The fix makes the test correct for the new upstream behavior; if that's not possible without weakening, stop and report. -- **Create + verify, NEVER merge.** Each test PR (and any paired recipe upgrade PR) is - operator-merged after review. +- **Create for visibility, verify, then merge directly.** Each test PR is merged only after its + verification is green (the invocation is the authorization); red/unverified PRs stay open. The + paired recipe upgrade PRs are NOT yours to merge. Always report merged-PR links + summaries. - **Single-writer:** dedicated branch in a separate clone; never push `main`; never touch the build loops' `/cc-ci` `/cc-ci-adv` clones or `/root/builder-clone`. Use `/root/cc-ci-test-verify` on the host and `rm -rf` it after. diff --git a/.opencode/skills/cc-ci-update/SKILL.md b/.opencode/skills/cc-ci-update/SKILL.md index fedf491..23fe271 100644 --- a/.opencode/skills/cc-ci-update/SKILL.md +++ b/.opencode/skills/cc-ci-update/SKILL.md @@ -1,6 +1,6 @@ --- name: cc-ci-update -description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with verified, ready-to-merge PRs and a healthy server at the end. NEVER merges anything. Invoke as /cc-ci-update. +description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update. --- # cc-ci-update diff --git a/AGENTS.md b/AGENTS.md index c5d0f0b..955090f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -36,7 +36,12 @@ a **Hetzner `cpx22`** cloud server (`cc-ci-orchestrator-1`, tailnet `100.84.190. `168.119.126.100`, flake host `cc-ci-orchestrator-hetzner`) — see `cc-ci-plan/plan-orchestrator-hetzner-migration.md`. The earlier Pi→Incus-VM move is the historical `cc-ci-plan/plan-orchestrator-migration.md`. Rebuild this host with -`nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner` from `/srv/cc-ci-orch`. +`nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner` from `/srv/cc-ci-orch` — but **always +`nixos-rebuild test` the same flake target first and verify the host is still healthy/reachable +before the `switch`** (general policy for nix deploys to this host and the cc-ci server: `test` +leaves the bootloader and system profile untouched, so a reboot always recovers to the +last-known-good generation; the 2026-08-03 cc-ci 26.05 bump outage is the cautionary tale, see +`.cc-ci-logs/server-update-2026-08-03.md`). ## Keep the orchestrator open, under remote-control diff --git a/cc-ci-plan/JOURNAL.md b/cc-ci-plan/JOURNAL.md index 3b0b42f..9d5bb4e 100644 --- a/cc-ci-plan/JOURNAL.md +++ b/cc-ci-plan/JOURNAL.md @@ -763,3 +763,31 @@ session cc-ci-orchestrator-stale can be killed; recipe-mirrors org still private session; re-oriented from STATUS/journals via a nudge). Relayed the operator's `${STACK_NAME}_app` bluesky guidance. 4 recipe PRs + 2 harness fixes; **nothing merged** (operator reviews/merges). - Queue drained again → hourly wake stops. + +## 2026-08-03 ~20:30 UTC — /cc-ci-server-update: 24.11 → 26.05 (outage + recovery + success) + +- Session: primary opencode orchestrator started the run; its session was stopped mid-incident and + the **cc-ci-orchestrator-backup** (Claude Code / Fable, session `cc-ci-backup-orchestrator`) was + operator-promoted to take over and finish. +- Flake bump: nixpkgs 50ab793 (nixos-24.11 tip) → 531670d (nixos-26.05), sops-nix 77c423a → + f140661 (master). PR #11, **merged** (new policy: PRs for visibility, merged directly, links in + report): https://git.autonomic.zone/recipe-maintainers/cc-ci/pulls/11 +- **Outage:** first switch left the host with no default route (~19:30–20:12 UTC). Root cause: NixOS + 25.05+ scripted networking installs the default route from the gateway iface's + network-addresses unit; bare-string `defaultGateway` + off-subnet /32 Hetzner gateway → no iface + matched → no route. Fix: explicit `defaultGateway.interface = "eth0"` (commit c541cb1). +- Recovery detail worth remembering: NixOS grub generations live in a **submenu** — grubenv + `next_entry`/`default` need `1>N` ids; a bare index falls back to the (broken) default entry. + Clear any grubenv override after the next switch regenerates grub.cfg (indices shift). +- Final: gen 24, 26.05.20260803.531670d, 0 failed units, services 1/1 (warm spares flapping as + pre-update), report site 200, cold-boot proven (40s). Log: + .cc-ci-logs/server-update-2026-08-03.md +- Policy change (operator): /cc-ci-server-update, /cc-ci-tests-update, /cc-ci-update skills updated + in both .opencode and .claude positions — open PRs for visibility, merge directly once verified + (invocation = authorization), always report merged-PR links + change summaries. +- TODO for operator: rotate the Hetzner Cloud API token pasted into the orchestrator conversation + during recovery. +- Addendum (operator policy, 2026-08-03): **always `nixos-rebuild test` before `switch`** for nix + deploys to the cc-ci server and the orchestrator host — `test` leaves the bootloader/profile + untouched so a reboot always recovers. Codified in the cc-ci-server-update skill (step 5d) and + AGENTS.md (orchestrator rebuild instructions).