Compare commits
21
Commits
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: cc-ci-orchestrator-update
|
||||
description: Update the cc-ci ORCHESTRATOR host (the machine the orchestrator itself runs on) to the latest nixos-26.05 nixpkgs, then nixos-rebuild test + switch + verify, ship the bump as a verified PR on recipe-maintainers/cc-ci-orchestrator and merge it directly (the skill invocation IS the authorization). The orchestrator-host analogue of /cc-ci-server-update. Self-update caveats apply — the switch restarts services on the very host running this session; always `nixos-rebuild test` before `switch` so a reboot recovers to the last-known-good generation. The end-of-run report includes the merged PR link + a summary of what changed. Invoke as /cc-ci-orchestrator-update (also answers to /cc-ci-update-orchestrator).
|
||||
description: Update the cc-ci HOST — since 2026-09 the ONE Hetzner machine that runs both the cc-ci CI server and the orchestrator (built from this repo's `#cc-ci`, which imports cc-ci's server module) — to the latest nixos-26.05 nixpkgs, sops-nix and cc-ci main, then `nixos-rebuild test` → cc-ci health checks → `switch`, committing the health-checked flake.lock straight to main (the skill invocation IS the authorization; the operator reviews retrospectively). It drives the host's own weekly auto-update unit by hand, so a manual and an automatic update are the same code path. Refuses while CI is busy. `--cc-ci-only` moves only the cc-ci input (harness/tests/modules), for after a cc-ci merge. Answers to /cc-ci-orchestrator-update, /cc-ci-update-orchestrator and (since the hosts merged) /cc-ci-server-update.
|
||||
---
|
||||
|
||||
# cc-ci-orchestrator-update (thin wrapper)
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: cc-ci-server-update
|
||||
description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main. Opens the cc-ci PR for visibility/historical record and merges it directly once the deploy verifies (the skill invocation IS the authorization); the end-of-run report includes the merged PR link + a summary of what changed. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update.
|
||||
description: (Delegates) Update the cc-ci CI server host to the latest nixpkgs — which, since 2026-09-07, is the SAME machine as the orchestrator, built from the cc-ci-orchestrator flake. This skill only forwards to /cc-ci-orchestrator-update; it no longer bumps or rebuilds the cc-ci repo's own flake, because that standalone `#cc-ci` target would replace the combined host config. Invoke as /cc-ci-server-update.
|
||||
---
|
||||
|
||||
# cc-ci-server-update (thin wrapper)
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: cc-ci-update
|
||||
description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update.
|
||||
description: Run both cc-ci maintenance updates in one pass — first /cc-ci-orchestrator-update (the cc-ci host: nixpkgs, sops-nix and the cc-ci input, test → health check → switch, lock committed to main), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update.
|
||||
---
|
||||
|
||||
# cc-ci-update (thin wrapper)
|
||||
|
||||
@@ -9,14 +9,16 @@
|
||||
# SAFE BY ALLOWLIST. It removes ONLY things NOT on the keep-list, so it can never take down infra or
|
||||
# the warm canonicals. The keep-list (leading name prefix) is:
|
||||
# - traefik, drone, backups : Swarm + CI infra
|
||||
# - ccci-bridge / -dashboard / -reports: the cc-ci control plane
|
||||
# - ccci-bridge / -dashboard / -reports / -opencode-ui: the cc-ci control plane (opencode-ui is
|
||||
# the traefik route to the orchestrator's opencode web UI, deployed by the host's nix config —
|
||||
# the 2026-09-07 run reaped it as an "orphan" before it was on this list)
|
||||
# - warm-* : warm canonicals (idle persistent deps reused across runs;
|
||||
# their retained volumes are spared too)
|
||||
# Everything else deployed on the Swarm is a per-run test stack and is fair game.
|
||||
set -uo pipefail
|
||||
export PATH=/run/current-system/sw/bin:$PATH
|
||||
|
||||
KEEP_RE='^(traefik|drone|backups|ccci-(bridge|dashboard|reports)|warm-)'
|
||||
KEEP_RE='^(traefik|drone|backups|ccci-(bridge|dashboard|reports|opencode-ui)|warm-)'
|
||||
removed=0
|
||||
|
||||
echo "== orphan sweep: scanning (keep-list: infra + warm-* canonicals) =="
|
||||
|
||||
@@ -32,3 +32,6 @@ __pycache__/
|
||||
# Local API tokens — never committed (advisory-scan / hetzner recovery)
|
||||
.github-token
|
||||
.hcloud-token
|
||||
|
||||
# auto-update.nix keeps the pre-update lock here so a failed run can restore it (never committed)
|
||||
.flake.lock.auto-update-prev
|
||||
|
||||
@@ -1,164 +1,87 @@
|
||||
---
|
||||
name: cc-ci-orchestrator-update
|
||||
description: Update the cc-ci ORCHESTRATOR host (the machine the orchestrator itself runs on) to the latest nixos-26.05 nixpkgs, then nixos-rebuild test + switch + verify, ship the bump as a verified PR on recipe-maintainers/cc-ci-orchestrator and merge it directly (the skill invocation IS the authorization). The orchestrator-host analogue of /cc-ci-server-update. Self-update caveats apply — the switch restarts services on the very host running this session; always `nixos-rebuild test` before `switch` so a reboot recovers to the last-known-good generation. The end-of-run report includes the merged PR link + a summary of what changed. Invoke as /cc-ci-orchestrator-update (also answers to /cc-ci-update-orchestrator).
|
||||
description: Update the cc-ci HOST — since 2026-09 the ONE Hetzner machine that runs both the cc-ci CI server and the orchestrator (built from this repo's `#cc-ci`, which imports cc-ci's server module) — to the latest nixos-26.05 nixpkgs, sops-nix and cc-ci main, then `nixos-rebuild test` → cc-ci health checks → `switch`, committing the health-checked flake.lock straight to main (the skill invocation IS the authorization; the operator reviews retrospectively). It drives the host's own weekly auto-update unit by hand, so a manual and an automatic update are the same code path. Refuses while CI is busy. `--cc-ci-only` moves only the cc-ci input (harness/tests/modules), for after a cc-ci merge. Answers to /cc-ci-orchestrator-update, /cc-ci-update-orchestrator and (since the hosts merged) /cc-ci-server-update.
|
||||
---
|
||||
|
||||
# cc-ci-orchestrator-update
|
||||
|
||||
Manually update the **orchestrator host**'s NixOS dependencies (nixpkgs; sops-nix follows nixpkgs)
|
||||
to the current `nixos-26.05` channel tip, ship the bump via a verified PR + a live
|
||||
`nixos-rebuild test` → `switch`, and confirm the host is healthy afterward. This is the
|
||||
orchestrator-host analogue of `/cc-ci-server-update` — done deliberately, with a reviewable PR and
|
||||
a health gate, not as silent drift.
|
||||
Update **the cc-ci host** — one machine since 2026-09-07 (`cc-ci`, 195.201.88.249, no tailnet):
|
||||
the CI server (docker swarm, traefik, drone, runner, bridge, dashboard, reports, acme-dns) AND the
|
||||
orchestrator (loops, weekly upgrader, opencode agent) — to the current inputs, verified, committed.
|
||||
|
||||
> **Two hosts, two flakes — don't confuse them.** This skill updates the **orchestrator** host:
|
||||
> the machine this session runs on (`cc-ci-orchestrator-1`, Hetzner cpx22 **server 134487234**,
|
||||
> tailnet `cc-ci`, public `195.201.88.249` — the SAME host as the cc-ci CI server since 2026-09), flake checkout **`/srv/cc-ci-orch`** (repo
|
||||
> `recipe-maintainers/cc-ci-orchestrator`), target **`.#cc-ci`** (which now also rebuilds the CI server half, from the cc-ci repo flake input). The **cc-ci
|
||||
> CI server** (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, target `.#cc-ci`) is a different
|
||||
> machine — that's `/cc-ci-server-update`, NOT this skill.
|
||||
> **One flake, one target.** The host is `nixosConfigurations.cc-ci` in
|
||||
> `recipe-maintainers/cc-ci-orchestrator` (`/srv/cc-ci-orch` on the host). The cc-ci repo's own
|
||||
> `flake.nix` inputs are IRRELEVANT to the live build — this flake makes the `cc-ci` input follow
|
||||
> its nixpkgs/sops-nix. Never run `nixos-rebuild` on the cc-ci repo's `#cc-ci` target on this host:
|
||||
> that is the standalone-server config and would replace the combined one (orchestrator half gone).
|
||||
|
||||
> **This is a SELF-update.** The `switch` restarts units on the host running this very session.
|
||||
> tmux sessions and the orchestrator survive a switch (systemd restarts don't kill user tmux), but
|
||||
> `tailscaled`/`sshd` restarts can briefly drop Remote Control / SSH — run activation steps
|
||||
> detached (`setsid nohup … &`) and re-poll. A full **reboot** DOES end the session: the
|
||||
> auto-launch flow (REBOOTS.md + `cc-ci-loops.service`) restarts the loops, and the operator
|
||||
> reconnects the orchestrator — announce a planned reboot to the operator first.
|
||||
|
||||
## What this skill does NOT do
|
||||
|
||||
- **Does NOT touch the cc-ci CI server** — that's `/cc-ci-server-update`.
|
||||
- **Does NOT bump recipe images or cc-ci tests** — `/upgrade-all` / `/cc-ci-tests-update`.
|
||||
- **Does NOT change the channel** (`nixos-26.05` branch ref in `flake.nix`) without operator
|
||||
choice — a channel jump (e.g. → 26.11 when it releases) is a bigger change: ask first, read the
|
||||
release notes, and expect module/option renames.
|
||||
The work is done by **`cc-ci-auto-update.service`** (`nix/modules/auto-update.nix`), the same unit
|
||||
the Tuesday 03:00 UTC timer fires. Running it by hand gives you the identical sequence —
|
||||
busy gate → `nix flake update` → build → `switch-to-configuration test` → 45 s settle → health
|
||||
check (sshd, 0 failed units, docker/acme-dns/fail2ban/nginx/opencode-web/runner active, every swarm
|
||||
service at its replica count, sops decrypted, dashboard/reports/drone 200, opencode UI 401) →
|
||||
profile + bootloader → `flake.lock` committed as `auto-update` and pushed to `main` →
|
||||
`/etc/cc-ci` fast-forwarded — with a journal to read and a one-line state file.
|
||||
|
||||
## Preconditions
|
||||
|
||||
- You are on the orchestrator host; `/srv/cc-ci-orch` is the live checkout (git repo, remote
|
||||
`recipe-maintainers/cc-ci-orchestrator`).
|
||||
- Quiescent moment: no `/upgrade-all` in flight, no active build-loop phase, no cc-ci CI run that
|
||||
would be disturbed if this host blips (`pgrep -af run_recipe_ci` empty on cc-ci is a bonus, but
|
||||
the orchestrator host restarting mostly risks its own sessions, not the CI server).
|
||||
- `git status` in `/srv/cc-ci-orch` — note pre-existing dirt; don't sweep unrelated changes into
|
||||
the bump commit.
|
||||
- `ssh cc-ci` works from the notplants-orchestrator box (root on the host), or you are on the host.
|
||||
- **Not busy.** The unit checks itself (CI run, weekly upgrader, report, sweep, running Drone
|
||||
builds → `result=skipped note=busy`, exit 0). Don't fight it: wait, or `tmux ls` as loops and
|
||||
`docker service ls` to see what is running. Never kill a run to update a host.
|
||||
- Note the current generation so you can name the old one in the report:
|
||||
`ssh cc-ci 'readlink /run/current-system; cat /srv/cc-ci/.cc-ci-logs/auto-update-state'`.
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Snapshot baseline
|
||||
```bash
|
||||
# 1. what will move
|
||||
ssh cc-ci 'cd /srv/cc-ci-orch && sudo -H -u loops nix flake update --dry-run 2>&1 | grep -v "^warning" | tail -8' # informational; the unit does the real update
|
||||
git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05 | cut -c1-12
|
||||
|
||||
```
|
||||
nixos-version; readlink /run/current-system; uname -r
|
||||
systemctl --failed --no-legend
|
||||
df -h / | tail -1
|
||||
cd /srv/cc-ci-orch && nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
|
||||
tmux ls
|
||||
```
|
||||
Record: current nixpkgs rev, generation, failed units (should be 0), disk, live tmux sessions.
|
||||
# 2. run the unit and follow it (10-25 min: nixpkgs bumps rebuild the two OCI images too)
|
||||
ssh cc-ci 'systemctl start --no-block cc-ci-auto-update; journalctl -fu cc-ci-auto-update' # Ctrl-C when it prints "auto-update committed" / "REVERT" / "BUSY"
|
||||
# or, non-interactively:
|
||||
ssh cc-ci 'systemctl start cc-ci-auto-update; systemctl is-active cc-ci-auto-update; journalctl -u cc-ci-auto-update --no-pager -n 40'
|
||||
|
||||
### 2. Branch + bump the lock
|
||||
|
||||
Work directly in `/srv/cc-ci-orch` on a branch (it's the live checkout; the flake edit is inert
|
||||
until a rebuild):
|
||||
```
|
||||
cd /srv/cc-ci-orch
|
||||
git checkout -b chore/orchestrator-flake-update-$(date -u +%Y%m%d)
|
||||
nix flake update 2>&1 | tail -5 # nixpkgs tracks nixos-26.05; sops-nix follows nixpkgs
|
||||
nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
|
||||
```
|
||||
Note old→new revs. If the lock didn't move, report "already at channel tip" and stop (nothing to
|
||||
deploy).
|
||||
|
||||
### 3. Build (catch errors before any activation)
|
||||
|
||||
```
|
||||
cd /srv/cc-ci-orch && nixos-rebuild build --flake .#cc-ci 2>&1 | tail -15
|
||||
readlink -f result
|
||||
```
|
||||
Build failure → fix on the branch (option renames etc.) before going further. Never activate a
|
||||
build that didn't build cleanly.
|
||||
|
||||
### 4. Test-activate (MANDATORY before switch — general policy)
|
||||
|
||||
`nixos-rebuild test` activates without touching the bootloader or system profile, so if the
|
||||
activation breaks the host (cf. the cc-ci server's 2026-08-03 no-default-route outage), a plain
|
||||
reboot — Hetzner API power-cycle on server **134487234** if SSH is gone (see
|
||||
`hetzner-server-recovery`) — lands back on the last-known-good generation.
|
||||
```
|
||||
cd /srv/cc-ci-orch && setsid nohup nixos-rebuild test --flake .#cc-ci \
|
||||
> /tmp/orchestrator-test-switch.log 2>&1 < /dev/null & echo launched
|
||||
# after it settles (poll; tailscaled/sshd may blip):
|
||||
readlink /run/current-system # should be the new store path
|
||||
ip route show default # default route survived
|
||||
systemctl --failed --no-legend # 0 failed
|
||||
tmux ls # sessions survived
|
||||
tail -5 /tmp/orchestrator-test-switch.log
|
||||
```
|
||||
Regression here → you're still on the old boot default: diagnose, fix, or reboot to revert. Do NOT
|
||||
switch.
|
||||
|
||||
### 5. Switch (make permanent — only after 4 is healthy)
|
||||
|
||||
```
|
||||
cd /srv/cc-ci-orch && nixos-rebuild switch --flake .#cc-ci 2>&1 | tail -10
|
||||
```
|
||||
(If it fails with "Unit nixos-rebuild-switch-to-configuration.service was already loaded", the
|
||||
detached test's transient unit is still running — wait or `systemctl stop` it, then retry.)
|
||||
|
||||
### 6. Health gate (do not skip)
|
||||
|
||||
```
|
||||
nixos-version; readlink /run/current-system
|
||||
systemctl --failed --no-legend
|
||||
tmux ls # orchestrator + loop sessions intact
|
||||
tailscale status | head -3 # tailnet up
|
||||
systemctl list-timers cc-ci-upgrade-all.timer --no-pager | head -3
|
||||
ssh cc-ci 'hostname' 2>&1 # can still reach the CI server
|
||||
df -h / | tail -1
|
||||
```
|
||||
Optional (operator's call — it ends this session): a cold-boot proof reboot. If done, the
|
||||
REBOOTS.md auto-launch flow restarts the loops; the operator reconnects the orchestrator.
|
||||
|
||||
### 7. Commit + PR + merge + report
|
||||
|
||||
```
|
||||
cd /srv/cc-ci-orch
|
||||
git add flake.lock # flake.nix too if the channel ref changed
|
||||
git commit -m "flake: bump nixpkgs (nixos-26.05, $(date -u +%Y-%m-%d))
|
||||
|
||||
nixpkgs: <old-rev[:8]> -> <new-rev[:8]> (nixos-26.05 tip)
|
||||
Deployed to the cc-ci host (.#cc-ci): build + test + switch + health gate green."
|
||||
git push -u origin HEAD
|
||||
```
|
||||
Open the PR on `recipe-maintainers/cc-ci-orchestrator` (Gitea API with the `GITEA_*` creds from
|
||||
`/srv/cc-ci/.testenv`, same as `open-cc-ci-pr.sh` does for cc-ci — the helper is hardcoded to the
|
||||
cc-ci repo, so use the API directly or a copy with the repo swapped). PR body: old→new revs, what
|
||||
was verified. Then **merge it directly** — the PR is for visibility/historical record; the skill
|
||||
invocation is the authorization. On a rolled-back/failed run, leave the PR open with an
|
||||
explanatory comment.
|
||||
|
||||
Back on the host, return the checkout to main so the live tree matches the merged state:
|
||||
```
|
||||
git checkout main && git pull
|
||||
# 3. read the outcome
|
||||
ssh cc-ci 'cat /srv/cc-ci/.cc-ci-logs/auto-update-state; readlink /run/current-system; readlink /nix/var/nix/profiles/system; systemctl --failed --no-legend; cd /srv/cc-ci-orch && git log --oneline -1 && git status --short'
|
||||
```
|
||||
|
||||
Write `/srv/cc-ci-orch/.cc-ci-logs/orchestrator-update-<YYYY-MM-DD>.md`
|
||||
(`RESULT: SUCCESS/ROLLED-BACK/FAILED …`), append a line to `cc-ci-plan/JOURNAL.md`, and report to
|
||||
the operator with the **merged PR link + a summary of what changed**.
|
||||
`--cc-ci-only` (after merging something in the cc-ci repo that the host should run now):
|
||||
```bash
|
||||
ssh cc-ci 'sudo -H -u loops bash -lc "cd /srv/cc-ci-orch && nix flake update cc-ci && git -c user.name=auto-update -c user.email=auto-update@cc-ci commit -qm \"flake.lock: cc-ci -> main\" -- flake.lock && git push -q origin HEAD:main"'
|
||||
ssh cc-ci 'git -C /etc/cc-ci pull --ff-only --recurse-submodules'
|
||||
ssh cc-ci 'systemd-run --unit=ccci-rebuild --collect -E HOME=/root -p WorkingDirectory=/srv/cc-ci-orch bash -lc "nixos-rebuild test --flake /srv/cc-ci-orch#cc-ci && nixos-rebuild switch --flake /srv/cc-ci-orch#cc-ci"'
|
||||
# then the same step-3 checks, plus the §6 health commands from /cc-ci-status
|
||||
```
|
||||
|
||||
## When it fails
|
||||
|
||||
`result=failed note=<step>` means the unit already re-activated the previous generation and
|
||||
restored `flake.lock`; the host is as it was. Read `journalctl -u cc-ci-auto-update` for the
|
||||
`HEALTH FAIL:` / `REVERT:` line:
|
||||
- **build failed** — usually an upstream nixpkgs breakage or a cc-ci main that does not evaluate
|
||||
against the new nixpkgs. Reproduce: `sudo -H -u loops nixos-rebuild build --flake /srv/cc-ci-orch#cc-ci`
|
||||
in the checkout after `nix flake update`; fix in the right repo (PR, merge), re-run.
|
||||
- **swarm services not at their replica count** — one of the reconcile oneshots did not converge
|
||||
in 10 min: `journalctl -u deploy-<proxy|drone|bridge|dashboard|reports>`, `docker service ps <svc>`.
|
||||
- **NAR hash mismatch** on the cc-ci input — the lock was made on another machine; `nix flake
|
||||
update cc-ci` on the host and re-run (README day-2).
|
||||
- **push failed** — bookkeeping only (the generation is committed): push the lock by hand as loops.
|
||||
- A hung run: the unit has a 3 h timeout; `systemctl status cc-ci-auto-update`. Do NOT restart the
|
||||
unit or the host mid-switch (see the `stopIfChanged` comment in the module).
|
||||
|
||||
## Report (end of run)
|
||||
|
||||
- old generation → new generation, the nixpkgs/sops-nix/cc-ci revs before → after, the lock commit
|
||||
on `main`, the health check result, and anything the journal warned about.
|
||||
- Then `/cc-ci-status` §5–§6 should read `ok` — say if it does not.
|
||||
|
||||
## Guardrails
|
||||
|
||||
- **Build before test, test before switch** — never `switch` without a healthy `test` activation
|
||||
(general policy for this host and the cc-ci server; `test` keeps the bootloader untouched so a
|
||||
reboot always recovers).
|
||||
- **Detach activations** (`setsid nohup … &`) — a dropped SSH/Remote-Control connection must not
|
||||
kill a half-applied activation.
|
||||
- **Self-update awareness:** announce before any reboot (it ends the operator's session); a
|
||||
switch is fine but expect brief tailscaled/sshd blips.
|
||||
- **Single-writer:** dedicated branch; don't sweep unrelated working-tree changes into the bump
|
||||
commit; return the live checkout to `main` after the merge.
|
||||
- **Stable channel only** (`nixos-26.05` until the operator chooses a jump).
|
||||
- **PRs for visibility, merged directly** once verified; failed runs leave the PR open. Always
|
||||
report merged-PR links + change summaries.
|
||||
- **Rescue path:** Hetzner server **134487234** — same rescue-mode procedure as
|
||||
`hetzner-server-recovery` (grub generations live in a submenu: `1>N` ids).
|
||||
- The unit is the only thing that `switch`es here; do not hand-roll `nixos-rebuild switch` unless
|
||||
`--cc-ci-only` says so, and then `test` first.
|
||||
- Never `nix flake update` from the notplants box and push a lock for this host (NAR hashes differ
|
||||
between Nix versions) — locks for the host are made on the host.
|
||||
- Nothing recipe-side: no PRs merged, no tests touched; `/cc-ci-tests-update` is separate.
|
||||
|
||||
@@ -1,298 +1,25 @@
|
||||
---
|
||||
name: cc-ci-server-update
|
||||
description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main. Opens the cc-ci PR for visibility/historical record and merges it directly once the deploy verifies (the skill invocation IS the authorization); the end-of-run report includes the merged PR link + a summary of what changed. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update.
|
||||
description: (Delegates) Update the cc-ci CI server host to the latest nixpkgs — which, since 2026-09-07, is the SAME machine as the orchestrator, built from the cc-ci-orchestrator flake. This skill only forwards to /cc-ci-orchestrator-update; it no longer bumps or rebuilds the cc-ci repo's own flake, because that standalone `#cc-ci` target would replace the combined host config. Invoke as /cc-ci-server-update.
|
||||
---
|
||||
|
||||
# cc-ci-server-update
|
||||
# cc-ci-server-update → `/cc-ci-orchestrator-update`
|
||||
|
||||
Manually update the **cc-ci CI server host**'s NixOS dependencies (nixpkgs + sops-nix) to their
|
||||
latest pinned revisions, ship the bump via a verified cc-ci PR + a live `nixos-rebuild switch`, and
|
||||
confirm the server is healthy afterward. This is the cc-ci-server analogue of a `nix flake update`
|
||||
+ rebuild — done deliberately, with a reviewable PR and a health gate, not as silent drift.
|
||||
There is no separate CI-server host any more. The CI server runs on the cc-ci host
|
||||
(195.201.88.249) together with the orchestrator, from **one** flake:
|
||||
`recipe-maintainers/cc-ci-orchestrator` `#cc-ci`, which imports the cc-ci repo's
|
||||
`nixosModules.cc-ci-server` and makes that input follow its own nixpkgs and sops-nix.
|
||||
|
||||
> **Two hosts, two flakes — don't confuse them.** This skill updates the **cc-ci server** flake
|
||||
> (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, host target `.#cc-ci` = `nix/hosts/cc-ci-hetzner/`).
|
||||
> The **orchestrator** host (`/srv/cc-ci-orch`, flake host `.#cc-ci-orchestrator-hetzner`) is a
|
||||
> different machine with its own flake — it is NOT touched here. If you want to update the
|
||||
> orchestrator host, that's a separate `nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner`
|
||||
> from `/srv/cc-ci-orch` (the orchestrator flake already tracks `nixos-26.05` and is rebuilt by the
|
||||
> operator). This skill is only about the cc-ci **server**.
|
||||
**Run `/cc-ci-orchestrator-update`.** It updates nixpkgs, sops-nix AND the cc-ci input, tests,
|
||||
health-checks (the CI server's swarm, front doors, sops, acme-dns — the checks this skill used to
|
||||
do) and switches, via the host's own weekly auto-update unit.
|
||||
|
||||
## What this skill does NOT do
|
||||
Do **not** do what the old version of this skill did:
|
||||
- do not bump `flake.lock` in the cc-ci repo expecting it to change the live host (it cannot — the
|
||||
host follows the orchestrator flake's inputs; cc-ci's lock only matters for the standalone
|
||||
`#cc-ci` target, which nothing deploys);
|
||||
- do not run `nixos-rebuild switch --flake /etc/cc-ci#cc-ci` or `...#cc-ci-hetzner` on the host —
|
||||
that is the standalone-server configuration and would take the orchestrator half down.
|
||||
|
||||
- **Does NOT bump recipe image tags** — that's `/upgrade-all` / `/recipe-upgrade`.
|
||||
- **Does NOT touch cc-ci tests / harness / runner** — that's `/cc-ci-tests-update` or
|
||||
`/ci-dev-workflow`. This is a flake-input bump only (`flake.nix` input revs + `flake.lock`).
|
||||
- **Does NOT update the orchestrator host** — only the cc-ci server.
|
||||
- **Does merge the cc-ci PR itself** — the PR exists for visibility and historical record, and is
|
||||
merged directly at the end of a verified run (the skill invocation is the authorization; the
|
||||
operator reviews the merged PR afterwards via the link in the report).
|
||||
|
||||
## Preconditions / access
|
||||
|
||||
- `ssh cc-ci` works (root, tailnet `100.95.31.88`). If it doesn't, fix access first (Tailscale ACL
|
||||
/ `cc-ci-tailscaled` proxy) — do NOT proceed.
|
||||
- **The operator's choice to invoke this skill IS the authorization for the live rebuild** — there
|
||||
is no mid-run operator check to wait for. A `nixos-rebuild switch` cycles the abra reconcile
|
||||
oneshots (`deploy-proxy`, `warm-keycloak`, …), so prefer a **quiescent moment** (no `!testme` /
|
||||
nightly sweep running) to avoid disrupting an in-flight CI run. Check first:
|
||||
```
|
||||
ssh cc-ci 'docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"' # any unhealthy task?
|
||||
ssh cc-ci 'pgrep -af run_recipe_ci | grep -v grep' # a CI run in flight?
|
||||
```
|
||||
If either is non-empty, WAIT until the host is quiescent — do not rebuild mid-CI.
|
||||
- `/srv/cc-ci/.testenv` has the `GITEA_*` creds (for opening the cc-ci PR).
|
||||
- The **operator-held sops secrets** are on the cc-ci host at `/etc/cc-ci/secrets/secrets.yaml`
|
||||
(NOT in git). The rebuild needs them — see the rebuild procedure below.
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Snapshot current state (so you can compare + roll back)
|
||||
|
||||
```
|
||||
ssh cc-ci 'hostname; uptime; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \
|
||||
deploy-bridge deploy-reports deploy-backupbot drone-runner-exec; systemctl --failed --no-legend; \
|
||||
df -h / | tail -1; nixos-version; readlink /run/current-system'
|
||||
```
|
||||
Record: the current `nixos-version`, the current-system store path, the list of active infra
|
||||
services, and the disk usage. These are your "before" baseline. Any rebuild that leaves a unit
|
||||
failed or an endpoint down gets compared against this.
|
||||
|
||||
Also record the **current flake input revs** (the "before" of what you're bumping):
|
||||
```
|
||||
ssh cc-ci 'cd /root/builder-clone && nix flake metadata --json 2>/dev/null | jq -r ".locks.nodes | to_entries[] | \"\(.key) \(.value.locked.rev // \"-\")\""' 2>&1 | head
|
||||
```
|
||||
|
||||
### 2. Prepare a working clone of the cc-ci repo (single-writer)
|
||||
|
||||
Work in a **separate clone on the orchestrator** — never push `main`, never touch the build loops'
|
||||
working clones. Use the existing `/home/loops/work/cc-ci-fix` clone if present and clean, or make a
|
||||
fresh one:
|
||||
```
|
||||
git clone ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /home/loops/work/cc-ci-server-update
|
||||
cd /home/loops/work/cc-ci-server-update
|
||||
git fetch origin && git checkout main && git reset --hard origin/main
|
||||
git checkout -b chore/server-flake-update-$(date -u +%Y%m%d)
|
||||
```
|
||||
(If `/home/loops/work/cc-ci-fix` already exists and is on a stale branch, either reuse it on a new
|
||||
branch or make the fresh clone above — the key rule is the branch is dedicated + off `main`.)
|
||||
|
||||
### 3. Bump the flake inputs (deliberate, not drift)
|
||||
|
||||
The cc-ci server flake **intentionally hard-pins** `nixpkgs` to a specific revision (the comment in
|
||||
`flake.nix` says "Bump deliberately, not drift") and `sops-nix` to a commit compatible with that
|
||||
nixpkgs. So the bump is a **considered move to a newer nixpkgs + a compatible sops-nix**, not a
|
||||
blind `nix flake update --relock-all`.
|
||||
|
||||
**(a) Decide the new nixpkgs.** The cc-ci server runs recipe CI; nixpkgs provides docker, abra's
|
||||
runtime, traefik, keycloak, drone, the harness python/pytest/playwright toolchain. Pick the newest
|
||||
**stable channel revision** that's compatible:
|
||||
- Default: move to the latest revision of the **same channel** the orchestrator host tracks
|
||||
(`nixos-26.05`), so both hosts stay on a shared nixpkgs and CVEs get patched. Look up the current
|
||||
tip: `git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05` (the SHA after the ref).
|
||||
- If a newer stable channel is out and the operator wants it, use that — but treat a **channel
|
||||
jump** (e.g. 26.05 → 26.11) as a bigger change: scan the release notes for breaking changes
|
||||
(docker, systemd, python, nginx module option renames) and surface them in the PR body. Default
|
||||
to staying on the same channel unless the operator asks for the jump.
|
||||
- **Never** jump to `nixos-unstable` for the cc-ci server — recipe CI reproducibility depends on a
|
||||
stable nixpkgs. Unstable is for experiments, not production.
|
||||
|
||||
**(b) Update `flake.nix`** with the new nixpkgs rev (replace the `rev = …` in the `nixpkgs.url`),
|
||||
and check whether `sops-nix` needs a matching bump:
|
||||
- sops-nix `inputs.nixpkgs.follows = "nixpkgs"`, so it tracks nixpkgs automatically — but its own
|
||||
rev pin may need moving if the pinned sops-nix commit is incompatible with the new nixpkgs (e.g.
|
||||
the `buildGoModule` → `buildGo125Module` gotcha the current pin's comment calls out). After
|
||||
editing `flake.nix`, run `nix flake lock --update-input nixpkgs --update-input sops-nix` (or
|
||||
`nix flake update` if both follow) in the clone and let nix resolve a compatible sops-nix. If the
|
||||
lock fails (sops-nix master needs a newer nixpkgs than you picked), pick a sops-nix commit that
|
||||
still builds against your chosen nixpkgs (its `flake.nix` declares the nixpkgs it expects).
|
||||
- **Keep the `sops-nix.inputs.nixpkgs.follows = "nixpkgs"` line** — that's what keeps them aligned.
|
||||
- Update the comment in `flake.nix` to reflect the new pin date + reason (the existing comment
|
||||
style: "Pinned to … (operator YYYY-MM-DD)" or "Follows nixos-26.05 channel").
|
||||
|
||||
**(c) Re-lock + sanity-build locally first** (cheap, catches a broken input before touching the
|
||||
host):
|
||||
```
|
||||
cd /home/loops/work/cc-ci-server-update # (or cc-ci-fix on the new branch)
|
||||
nix flake lock --update-input nixpkgs --update-input sops-nix # rewrites flake.lock
|
||||
nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
|
||||
git diff flake.nix flake.lock | head -40
|
||||
```
|
||||
Verify the lock moved to the revs you intended. **Do NOT `nixos-rebuild build` from the
|
||||
orchestrator** — the orchestrator's nixpkgs is different and the cc-ci host target won't build here
|
||||
(meaningfully). The real build happens on the cc-ci host in step 5.
|
||||
|
||||
### 4. Commit + open the cc-ci PR (merged in step 7 after the health gate)
|
||||
|
||||
```
|
||||
git add flake.nix flake.lock
|
||||
git commit -m "flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))
|
||||
|
||||
nixpkgs: <old-rev[:8]> → <new-rev[:8]> (nixos-<channel>)
|
||||
sops-nix: <old-rev[:8]> → <new-rev[:8]>
|
||||
<one-line reason — CVE patch / channel bump / sops-nix compat>"
|
||||
git push -u origin HEAD
|
||||
```
|
||||
Open the PR via the helper:
|
||||
```
|
||||
set -a; . /srv/cc-ci/.testenv; set +a
|
||||
TITLE="flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))" \
|
||||
BODY_FILE=/tmp/cc-ci-server-update-pr-body.md \
|
||||
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh
|
||||
```
|
||||
(Write the PR body to `/tmp/cc-ci-server-update-pr-body.md` first: old→new revs, the channel, any
|
||||
breaking changes from the release notes, and the rebuild plan. The PR is opened for visibility and
|
||||
historical record; it gets merged directly in step 7 once the deploy passes the health gate.)
|
||||
|
||||
Capture the `PR_URL`. **The cc-ci push-build lint gate will run on the branch** — if it's red on a
|
||||
flake-lock-only change, it's a pre-existing lint drift (not yours); note it in the PR body and
|
||||
proceed (the rebuild doesn't depend on the lint gate passing).
|
||||
|
||||
### 5. Deploy to the live cc-ci host (no-CI window)
|
||||
|
||||
This is the host-rebuild procedure from memory `cc-ci-host-rebuild-procedure` — the cc-ci server
|
||||
has **no self-service rebuild path**, so the orchestrator stages the repo + rebuilds it. The
|
||||
deploy clone on the host is `/root/builder-clone`; staging a fresh copy avoids disturbing it.
|
||||
|
||||
**(a) Stage the PR branch on the host:**
|
||||
```
|
||||
# from the orchestrator, on the PR branch, clean tree:
|
||||
rsync -a --delete --exclude='.git' /home/loops/work/cc-ci-server-update/ root@cc-ci:/root/cc-ci-deploy/
|
||||
ssh cc-ci 'chown -R root:root /root/cc-ci-deploy'
|
||||
```
|
||||
(A `--delete` rsync from a clean tree = exactly the PR head. Excluding `.git` keeps it a plain path
|
||||
flake — see (c).)
|
||||
|
||||
**(b) Copy the operator-held sops secrets** (NOT in git, required for the build):
|
||||
```
|
||||
ssh cc-ci 'mkdir -p /root/cc-ci-deploy/secrets && cp /etc/cc-ci/secrets/secrets.yaml /root/cc-ci-deploy/secrets/secrets.yaml && chmod 600 /root/cc-ci-deploy/secrets/secrets.yaml'
|
||||
```
|
||||
(The age key is at `/var/lib/sops-nix/key.txt` on the host — already in place from prior rebuilds.
|
||||
Without `secrets/secrets.yaml` the build FATAs `secrets/secrets.yaml does not exist`.)
|
||||
|
||||
**(c) Build first (catch errors before switching):**
|
||||
```
|
||||
ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild build --flake .#cc-ci' 2>&1 | tail -30
|
||||
```
|
||||
- Target is `.#cc-ci` (= `.#cc-ci-hetzner` = `nix/hosts/cc-ci-hetzner/configuration.nix`).
|
||||
- nixpkgs is now your bumped pin, so this **will** rebuild more than a same-pin no-op — expect a
|
||||
longer build (docker/traefik/keycloak/python toolchain may all rebuild against the new nixpkgs).
|
||||
That's expected; this is a real nixpkgs bump.
|
||||
- If the build fails: read the error, fix on the PR branch (e.g. a renamed option, a removed
|
||||
package), re-push, re-stage, re-build. Do NOT `switch` a failed build. Common breakages: NixOS
|
||||
module option renames across channels (check the release notes), python package removals, a
|
||||
sops-nix incompatibility (go back to step 3b and pick a compatible sops-nix rev).
|
||||
|
||||
**(d) Test-activate FIRST (general policy for nix deploys to this server AND the orchestrator
|
||||
host): `nixos-rebuild test` before any `switch`:**
|
||||
```
|
||||
# detach it so a dropped SSH can't kill the activation mid-flight:
|
||||
ssh cc-ci 'cd /root/cc-ci-deploy && setsid nohup nixos-rebuild test --flake .#cc-ci \
|
||||
> /root/cc-ci-deploy/test-switch.log 2>&1 < /dev/null & echo launched'
|
||||
# poll until reachable again, then check the essentials survived:
|
||||
ssh cc-ci 'readlink /run/current-system; ip route show default; systemctl --failed --no-legend'
|
||||
```
|
||||
`test` activates the new configuration **without touching the bootloader or the system profile**,
|
||||
so if the activation kills networking (as the 2026-08-03 26.05 bump did — no default route), a
|
||||
plain reboot (Hetzner API power-cycle if SSH is gone) lands back on the last-known-good
|
||||
generation. Only proceed to `switch` once the test activation is up, reachable, and healthy.
|
||||
Note: the detached test leaves a transient `nixos-rebuild-switch-to-configuration` unit; if the
|
||||
follow-up `switch` fails with "Unit ... was already loaded", wait for it to finish or
|
||||
`systemctl stop nixos-rebuild-switch-to-configuration.service` first.
|
||||
|
||||
**(e) Switch (make it permanent — only after (d) verified):**
|
||||
```
|
||||
ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild switch --flake .#cc-ci' 2>&1 | tail -30
|
||||
```
|
||||
This is the moment the server moves to the new nixpkgs. The abra reconcile oneshots
|
||||
(`deploy-proxy`, `warm-keycloak`, …) re-run as part of the switch — watch the tail for any that
|
||||
fail. If a reconcile unit fails, do NOT roll back blindly — diagnose first (a warm-keycloak blip
|
||||
may be transient; a proxy failure is real).
|
||||
|
||||
### 6. Verify the server is healthy (health gate — do not skip)
|
||||
|
||||
```
|
||||
ssh cc-ci 'echo "=== version ==="; nixos-version; readlink /run/current-system; \
|
||||
echo "=== failed units ==="; systemctl --failed --no-legend; \
|
||||
echo "=== infra services ==="; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \
|
||||
deploy-bridge deploy-reports deploy-backupbot drone-runner-exec | paste - - - - - - -; \
|
||||
echo "=== docker stacks ==="; docker stack ls --format "{{.Name}} {{.Services}}"; \
|
||||
echo "=== service replicas (any 0/N?) ==="; docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"; \
|
||||
echo "=== endpoints ==="; curl -sS -o /dev/null -w "traefik:%{http_code}\n" https://traefik.ci.commoninternet.net/api/version; \
|
||||
curl -sS -o /dev/null -w "dashboard:%{http_code}\n" https://ci.commoninternet.net/; \
|
||||
echo "=== disk ==="; df -h / | tail -1'
|
||||
```
|
||||
Compare against the step-1 baseline:
|
||||
- `nixos-version` moved (or the current-system path changed) — the bump landed.
|
||||
- **No failed units** (the `--failed` output is empty).
|
||||
- All 7 infra services `active` (or `active (exited)` for the oneshots).
|
||||
- `docker service ls` shows no `0/N` replicas (everything converged).
|
||||
- Endpoints `200` (traefik `/api/version`, dashboard root).
|
||||
- Disk didn't spike (a nixpkgs bump adds a new system generation; old ones are GC'd eventually).
|
||||
|
||||
If **anything regressed**: diagnose. A real regression → roll back to the previous generation:
|
||||
`ssh cc-ci 'nixos-rebuild switch --rollback'` (this switches to the pre-bump system path; the
|
||||
flake input change on the PR branch is still your source of truth, but the running host reverts
|
||||
while you fix). Report the rollback in the PR.
|
||||
|
||||
### 7. Merge the PR + report
|
||||
|
||||
On a **verified** run (health gate green): post a PR comment summarizing what was deployed (revs,
|
||||
any incidents, verification results), then **merge the PR directly** — the skill invocation is the
|
||||
authorization; the PR exists for visibility and historical record. On a rolled-back or failed run,
|
||||
leave the PR **open** with a comment explaining the state.
|
||||
|
||||
Print + record a one-line result + the PR URL. Write a short note to
|
||||
`/srv/cc-ci/.cc-ci-logs/server-update-<YYYY-MM-DD>.md`:
|
||||
- `RESULT: SUCCESS — cc-ci server rebuilt on nixpkgs <new-rev[:8]>, no failed units, endpoints 200, PR merged: <url>`
|
||||
- `RESULT: ROLLED-BACK — nixpkgs <new-rev> rebuild failed/ regressed (<reason>); host reverted to <old-rev>; PR <url> left open`
|
||||
- `RESULT: FAILED — <step>: <reason>`
|
||||
|
||||
The end-of-run report to the operator MUST include **links to every merged PR** plus a summary of
|
||||
what changed in each, so the operator can review them after the fact if they want to.
|
||||
|
||||
### 8. Optional cold-boot proof
|
||||
|
||||
A `nixos-rebuild switch` proves the new generation works on the running kernel. A **reboot** proves
|
||||
it boots cleanly from cold — the stronger guarantee (cf. the pxgate cold-boot proof). Only do this
|
||||
if the operator wants it, and only outside a CI window:
|
||||
```
|
||||
ssh cc-ci 'systemctl reboot'
|
||||
# wait ~60s, re-check ssh cc-ci + the step-6 health gate
|
||||
```
|
||||
Record the new `boot_id` if you do this (it's a genuine reboot — log it to
|
||||
`cc-ci-plan/REBOOTS.md`? No — that file is for the **orchestrator** host. The cc-ci server's reboots
|
||||
are tracked in its own journal; a note in the server-update report is enough).
|
||||
|
||||
## Guardrails
|
||||
|
||||
- **Live rebuild is authorized by invocation.** The operator's choice to run this skill IS the
|
||||
go-ahead for the `nixos-rebuild switch` — do not stop mid-run to re-confirm. A switch cycles
|
||||
reconcile oneshots and can disrupt an in-flight `!testme`, so prefer a quiescent moment (the
|
||||
step-1 check), but don't gate on a second authorization.
|
||||
- **Single-writer:** dedicated branch in a separate clone; never push `main`; never touch the
|
||||
build loops' `/cc-ci` / `/cc-ci-adv` clones or `/root/builder-clone` (stage to
|
||||
`/root/cc-ci-deploy` instead).
|
||||
- **Open the PR for visibility, merge it directly.** The live rebuild is the deploy; the PR is the
|
||||
reviewable historical record. Merge only after the health gate passes; a rolled-back/failed run
|
||||
leaves the PR open with an explanatory comment. Always report merged-PR links + change summaries.
|
||||
- **Build before test, test before switch.** Never `nixos-rebuild switch` a build that didn't build
|
||||
cleanly first, and never `switch` without a healthy `nixos-rebuild test` activation first — `test`
|
||||
leaves the bootloader untouched, so a reboot always recovers to the last-known-good generation.
|
||||
This is general policy for nix deploys to the cc-ci server **and** the orchestrator host.
|
||||
- **Health gate is mandatory.** No "it switched, ship it" — run the step-6 verification and compare
|
||||
to the baseline. Roll back on regression.
|
||||
- **Stable channel only** for the cc-ci server (never `nixos-unstable`).
|
||||
- **Keep sops-nix.follows** so the two inputs stay aligned; don't fork them.
|
||||
- **Secrets never in git.** The operator-held `secrets/secrets.yaml` is copied on the host only.
|
||||
- **Rollback is always available:** `nixos-rebuild switch --rollback` reverts to the prior
|
||||
generation without touching the flake input revs.
|
||||
|
||||
## Relationship to an automatic update
|
||||
|
||||
This skill is the **manual trigger** for what should ideally run on a periodic timer: bump the cc-ci
|
||||
flake inputs, build, switch, verify. If/when an automatic `cc-ci-server-update.timer` is added
|
||||
(NixOS systemd timer on the orchestrator, analog of `cc-ci-upgrade-all.timer`), it should invoke
|
||||
this skill via a dedicated one-shot agent (the way `cc-ci-upgrader` runs `/upgrade-all`). The
|
||||
skill stays as the authoritative procedure either way — automatic just means "run on a schedule
|
||||
without a human prompt."
|
||||
If what you actually wanted is "the host should run cc-ci `main` as of now" (a harness or test
|
||||
change), that is `/cc-ci-orchestrator-update --cc-ci-only`.
|
||||
|
||||
@@ -58,35 +58,62 @@ For each open PR: age (now − created_at), and whether the PR title/body/report
|
||||
- Verified-green PRs awaiting operator merge are normal — list them as "ready to merge", not
|
||||
as failures.
|
||||
|
||||
### 5. Server + orchestrator host update recency
|
||||
### 5. The cc-ci host: update recency + the weekly auto-update
|
||||
|
||||
For BOTH hosts:
|
||||
```
|
||||
# cc-ci server: ssh cc-ci 'nixos-version; cd /root/cc-ci-deploy && nix flake metadata --json' (or builder-clone)
|
||||
# orchestrator: nixos-version; cd /srv/cc-ci-orch && nix flake metadata --json
|
||||
git ls-remote https://github.com/NixOS/nixpkgs <channel> # current tip
|
||||
ls -t /srv/cc-ci-orch/.cc-ci-logs/server-update-*.md /srv/cc-ci-orch/.cc-ci-logs/orchestrator-update-*.md | head -2
|
||||
```
|
||||
- Report: days since last update log + how far the running nixpkgs rev lags the channel tip.
|
||||
- **Lagging >30 days** (or a NixOS release behind) → recommend `/cc-ci-server-update` /
|
||||
`/cc-ci-orchestrator-update`.
|
||||
|
||||
### 6. Host + service health (both machines)
|
||||
Since 2026-09-07 the CI server and the orchestrator are ONE Hetzner host (`cc-ci`,
|
||||
195.201.88.249, no tailnet), built from `recipe-maintainers/cc-ci-orchestrator`'s
|
||||
`nixosConfigurations.cc-ci` (which imports cc-ci's `nixosModules.cc-ci-server`). From the
|
||||
notplants-orchestrator box `ssh cc-ci` reaches it as root; on the host itself `ssh cc-ci` is loopback.
|
||||
|
||||
```
|
||||
ssh cc-ci 'systemctl --failed --no-legend; df -h / | tail -1; docker service ls --format "{{.Name}} {{.Replicas}}"'
|
||||
systemctl --failed --no-legend; df -h / | tail -1; tmux ls
|
||||
ssh cc-ci 'nixos-version; readlink /run/current-system; readlink /nix/var/nix/profiles/system'
|
||||
ssh cc-ci 'cd /srv/cc-ci-orch && nix flake metadata --json | jq -r ".locks.nodes | to_entries[] | \"\(.key) \(.value.locked.rev // \"-\") \(.value.locked.lastModified // \"-\")\""'
|
||||
git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05 | cut -c1-12 # channel tip
|
||||
# the weekly self-update (nix/modules/auto-update.nix, Tue 03:00 UTC, busy-gated):
|
||||
ssh cc-ci 'cat /srv/cc-ci/.cc-ci-logs/auto-update-state; systemctl list-timers --no-pager cc-ci-auto-update.timer | sed -n 2p; journalctl -u cc-ci-auto-update --no-pager -n 3'
|
||||
```
|
||||
- Failed units, core swarm services not 1/1 (warm-* spares flapping is a known benign pattern —
|
||||
note, don't page), disk **>65% (server)** / >85% (orchestrator) → findings. Server unreachable →
|
||||
HIGH: recommend `hetzner-server-recovery`.
|
||||
> **65%, not 80%, on the server — it is not a steady-state measure.** Two concurrent recipe runs
|
||||
> pull images and write volumes worth tens of GB, so a host sitting at 73% still hits 100% mid-run.
|
||||
> That is exactly what happened on 2026-08-11: 63GB of unused images had accumulated (nothing ever
|
||||
> pruned them), the filesystem filled during a run, and the harness died at startup with
|
||||
> `OSError: [Errno 28] No space left on device`. Remedy: `docker image prune -af` on cc-ci — it
|
||||
> spares anything a container references, so infra and warm-* canonicals are untouched. Do NOT
|
||||
> `docker volume prune`: warm-* canonical volumes are data-warm and legitimately dangling.
|
||||
- The state file is one line: `<UTC time> result=ok|unchanged|skipped|failed generation=<store path> note=<why>`.
|
||||
- `ok` / `unchanged` within the last 8 days → healthy.
|
||||
- `skipped` (busy) two weeks in a row, `failed`, or no state line / timer inactive for >8 days →
|
||||
finding: *"host auto-update <state> since <date>: <note>"* → recommend
|
||||
`/cc-ci-orchestrator-update` (it runs the same unit by hand and reads its journal).
|
||||
- `failed` also means the host is running the previous generation and `flake.lock` was restored —
|
||||
say so; the journal (`journalctl -u cc-ci-auto-update`) names the health check that failed.
|
||||
- Running `/run/current-system` must equal the profile's boot generation (`system` link) — a
|
||||
mismatch means a `test`-only activation was left behind (finding: switch or reboot decision).
|
||||
- nixpkgs rev lagging the `nixos-26.05` channel tip by >30 days → finding → `/cc-ci-orchestrator-update`.
|
||||
- `cc-ci` (the CI server input) lagging cc-ci `main` → note; the auto-update moves it weekly, or
|
||||
`/cc-ci-orchestrator-update --cc-ci-only`.
|
||||
|
||||
### 6. Host + service health (one machine)
|
||||
|
||||
```
|
||||
ssh cc-ci 'systemctl --failed --no-legend; df -h / | tail -1; free -m | sed -n 2p'
|
||||
ssh cc-ci 'docker service ls --format "{{.Name}} {{.Replicas}}"; docker stack ls --format "{{.Name}}"'
|
||||
ssh cc-ci 'for h in ci report.ci drone.ci oc.ci; do printf "%s " $h; curl -s -o /dev/null -w "%{http_code} ssl=%{ssl_verify_result}\n" https://$h.commoninternet.net/; done'
|
||||
ssh cc-ci 'ls -s /run/secrets/test_secret; systemctl is-active acme-dns fail2ban nginx opencode-web drone-runner-exec; fail2ban-client status sshd | grep -E "Currently banned|Total banned"; fail2ban-client status nginx-http-auth | grep -E "Currently banned|Total banned"'
|
||||
ssh cc-ci 'systemctl list-timers --no-pager cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer cc-ci-auto-update.timer nightly-sweep.timer | tail -n +2 | head -4'
|
||||
ssh cc-ci 'sudo -iu loops tmux ls' # cc-ci-orchestrator must exist (the opencode orchestrator agent); cc-ci-upgrader only during a run
|
||||
```
|
||||
- Failed units, swarm services not at their replica count (`warm-*` spares flapping is benign —
|
||||
note, don't page), a front door not 200 (`oc.ci` must be **401**: basic auth in front of the
|
||||
opencode UI; 404 there = the `ccci-opencode-ui` stack is gone, `systemctl restart opencode-ui-route`
|
||||
and check the weekly sweep's keep-list), `ssl≠0` (cert), `/run/secrets` empty (sops), acme-dns /
|
||||
fail2ban / nginx / opencode-web / drone-runner-exec inactive, any timer missing, the orchestrator
|
||||
tmux session missing (`systemctl restart cc-ci-orchestrator`) → findings. Disk **>65%** → finding.
|
||||
Host unreachable → HIGH: `hetzner-server-recovery` (token in `/srv/cc-ci/.hcloud-token` on the
|
||||
notplants box while the operator keeps it active).
|
||||
> **65%, not 80% — it is not a steady-state measure.** Two concurrent recipe runs pull images and
|
||||
> write volumes worth tens of GB, so a host sitting at 73% still hits 100% mid-run (2026-08-11:
|
||||
> 63GB of unused images, `OSError: [Errno 28] No space left on device`). Remedy: `docker image
|
||||
> prune -af` — it spares anything a container references. Do NOT `docker volume prune`: warm-*
|
||||
> canonical volumes are data-warm and legitimately dangling.
|
||||
- fail2ban: a non-zero "Currently banned" is information, not a finding (that is the jail
|
||||
working); the notplants box's own address showing up banned IS a finding (it means a skill
|
||||
misbehaved) — `fail2ban-client set <jail> unbanip <ip>`.
|
||||
- Secrets hygiene: `ssh cc-ci 'ls /secrets/files'` must list ONLY cc-ci.testenv, cc-ci-local-ed25519(.pub),
|
||||
autonomic-bot-cc-ci-ed25519(.pub), opencode-auth.json. Anything else there (a tailscale key, a
|
||||
master age key, another host's ssh key…) → finding: *"secret sprawl on the cc-ci host"*.
|
||||
- **!testme actually produces results** (the check that would have caught the above days earlier):
|
||||
the newest few `/var/lib/cc-ci-runs/<build>/` dirs must each contain `results.json`. A build that
|
||||
dies before the harness writes one leaves an EMPTY dir — and the PR badge still says "failure", so
|
||||
@@ -143,14 +170,17 @@ minutes, no PRs). If it is instead that a known CVE is sitting unpatched, recomm
|
||||
`/cve-check` over waiting for the next weekly run whenever the question is "are we exposed?".
|
||||
|
||||
`ALL HEALTHY` requires: recent successful weekly run + published report, no stale tests, no
|
||||
CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, recent builds all
|
||||
producing results.json, disk under
|
||||
thresholds, bridge clean, maintained-set consistent. Anything else is a finding — even minor
|
||||
CVE PR open >14 days, the host <30 days behind its channel with the weekly auto-update `ok`/`unchanged`
|
||||
within 8 days and running generation == boot generation, zero failed units, every front door
|
||||
answering (oc.ci with 401), sops decrypted, fail2ban up, the orchestrator agent session present,
|
||||
recent builds all producing results.json, disk under threshold, bridge clean, maintained-set
|
||||
consistent, secrets inventory exact. Anything else is a finding — even minor
|
||||
ones get a recommended next step. Order findings by priority (CVE/unreachable-host first).
|
||||
|
||||
## Guardrails
|
||||
|
||||
- **Read-only.** This skill diagnoses and recommends; it invokes nothing and changes nothing.
|
||||
- Use the pseudo-TTY wrap for any abra call; plain ssh for everything else.
|
||||
- Use the pseudo-TTY wrap for any abra call; plain ssh for everything else. `ssh cc-ci` from the
|
||||
notplants box is root on 195.201.88.249 (loops ssh config); there is no tailnet.
|
||||
- Don't double-count: a finding that explains another (bridge 401 → !testme "failures") gets
|
||||
reported once, at the root cause.
|
||||
|
||||
@@ -1,12 +1,11 @@
|
||||
---
|
||||
name: cc-ci-update
|
||||
description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update.
|
||||
description: Run both cc-ci maintenance updates in one pass — first /cc-ci-orchestrator-update (the cc-ci host: nixpkgs, sops-nix and the cc-ci input, test → health check → switch, lock committed to main), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update.
|
||||
---
|
||||
|
||||
# cc-ci-update
|
||||
|
||||
The combined **keep cc-ci itself current** pass: run `/cc-ci-server-update` (host dependency bump +
|
||||
rebuild) **then** `/cc-ci-tests-update` (stale-test sweep + fixes), in that order, end-to-end. This
|
||||
The combined **keep cc-ci itself current** pass: run `/cc-ci-orchestrator-update` (host dependency bump + health-checked rebuild) **then** `/cc-ci-tests-update` (stale-test sweep + fixes), in that order, end-to-end. This
|
||||
is just sequencing the two skills — it doesn't add or change any behavior. Read each sub-skill's
|
||||
SKILL.md for the full procedure; this file is the orchestration + the why-of-the-order.
|
||||
|
||||
@@ -21,7 +20,7 @@ SKILL.md for the full procedure; this file is the orchestration + the why-of-the
|
||||
## Arguments
|
||||
|
||||
None. (If you only want one of the two updates, invoke the sub-skill directly:
|
||||
`/cc-ci-server-update` or `/cc-ci-tests-update`.)
|
||||
`/cc-ci-orchestrator-update` or `/cc-ci-tests-update`.)
|
||||
|
||||
## Preconditions
|
||||
|
||||
@@ -34,12 +33,12 @@ Both sub-skills' preconditions hold:
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Run `/cc-ci-server-update` (host first)
|
||||
### 1. Run `/cc-ci-orchestrator-update` (host first)
|
||||
|
||||
Run the full `/cc-ci-server-update` procedure: snapshot baseline → bump flake inputs (nixpkgs +
|
||||
Run the full `/cc-ci-orchestrator-update` procedure: snapshot baseline → bump flake inputs (nixpkgs +
|
||||
sops-nix) → commit + open cc-ci PR → stage on host → build → `nixos-rebuild switch` → **health
|
||||
gate** (no failed units, infra up, endpoints 200, disk free). See
|
||||
`.claude/skills/cc-ci-server-update/SKILL.md`.
|
||||
`.claude/skills/cc-ci-orchestrator-update/SKILL.md`.
|
||||
|
||||
**Gate on the result before continuing:**
|
||||
- `SUCCESS` → the server is on the new nixpkgs, healthy. Proceed to step 2.
|
||||
@@ -90,7 +89,7 @@ operator review.
|
||||
Inherited from both sub-skills (read their Guardrails sections). The combined-specific ones:
|
||||
- **Server-first, gated.** Don't run the tests sweep on a sick/rolled-back host. The server-update
|
||||
health gate is a hard go/no-go for step 2.
|
||||
- **No-CI window for the rebuild** (the server half) — same as `/cc-ci-server-update`.
|
||||
- **No-CI window for the rebuild** (the server half) — same as `/cc-ci-orchestrator-update`.
|
||||
- **Serialize the tests half on the shared Swarm** — same as `/cc-ci-tests-update`; don't run it
|
||||
concurrent with `/upgrade-all` or active build loops.
|
||||
- **Never merge anything** — server PR, test PRs, paired recipe upgrade PRs all operator-merged.
|
||||
|
||||
@@ -31,8 +31,9 @@ the orchestrator watches from outside.
|
||||
Reboot resilience is handled by **`cc-ci-loops.service`** (system unit): on boot it logs the reboot
|
||||
to `REBOOTS.md` (boot_id-gated) and runs `launch.sh start` with `RESUME_PHASE=1`, so the loops +
|
||||
watchdog auto-resume the saved phase. The orchestrator session itself is relaunched by
|
||||
`cc-ci-orchestrator.service` (`agents.py up orchestrator`) — the operator reconnects to it (that's
|
||||
why the startup notification matters). Since 2026-09 the orchestrator runs on the **same Hetzner
|
||||
`cc-ci-orchestrator.service` (`agents.py up orchestrator`) as an **opencode agent** the operator
|
||||
steers from https://oc.ci.commoninternet.net (no Claude on the cc-ci host; Claude sessions run on
|
||||
the notplants-orchestrator box and reach cc-ci over ssh). Since 2026-09 the orchestrator runs on the **same Hetzner
|
||||
host as the cc-ci CI server** (`cc-ci`, public `195.201.88.249`, tailnet `cc-ci`), declared by
|
||||
`nixosConfigurations.cc-ci` in this repo's `flake.nix`, which imports the CI server from the cc-ci
|
||||
repo's `nixosModules.cc-ci-server`. `ssh cc-ci` from the loops user therefore goes to loopback.
|
||||
|
||||
@@ -9,9 +9,9 @@ module from the [cc-ci](https://git.autonomic.zone/recipe-maintainers/cc-ci) rep
|
||||
| | where |
|
||||
|---|---|
|
||||
| Orchestrator loops, timers (weekly upgrader, hourly supervisor) | `nix/modules/cc-ci.nix` → `nixosModules.cc-ci-orchestrator` |
|
||||
| The host contract those need (loops user, claude/opencode CLIs, opencode web UI) | `nix/modules/orchestrator-host.nix` → `nixosModules.orchestrator-host` |
|
||||
| The host contract those need (loops user, opencode CLI, opencode web UI) | `nix/modules/orchestrator-host.nix` → `nixosModules.orchestrator-host` |
|
||||
| The CI server (swarm, traefik, drone, runner, `!testme` bridge, dashboard, reports, acme-dns) | cc-ci repo `nix/modules/` → `nixosModules.cc-ci-server` (flake input `cc-ci`) |
|
||||
| The machine: hardware, networking, tailscale, root keys | `nix/hosts/cc-ci/` → `nixosConfigurations.cc-ci` |
|
||||
| The machine: hardware, networking, root keys, firewall + fail2ban | `nix/hosts/cc-ci/` → `nixosConfigurations.cc-ci` |
|
||||
| Plans, launch tooling, loop prompts, journal | `cc-ci-plan/` (see `AGENTS.md` for roles) |
|
||||
| Skills the orchestrator runs (`/upgrade-all`, `/recipe-upgrade`, `/cc-ci-status`, …) | `.claude/skills/`, `.opencode/skills/` |
|
||||
| How it used to be built (Pi → Incus VM → shared Hetzner box) | `archive/` |
|
||||
@@ -31,10 +31,12 @@ can repeat it. Read it once before starting; the order matters.
|
||||
|
||||
- A **Hetzner Cloud** project you can create servers in (console login or an API token).
|
||||
- **SSH keys**: yours, and the orchestrator's own key so the automation can reach the box. The
|
||||
public keys that get root are tracked in `nix/hosts/cc-ci/ssh-keys` (one per line).
|
||||
public keys that get root are tracked in `nix/hosts/cc-ci/ssh-keys` (one per line). Password
|
||||
login is disabled and fail2ban watches sshd, so a key is the only way in.
|
||||
- Read access to `recipe-maintainers/cc-ci`, `recipe-maintainers/cc-ci-orchestrator` (both public
|
||||
read) and the **private** `recipe-maintainers/cc-ci-secrets` (the `autonomic-bot` deploy key,
|
||||
`autonomic-bot-gitea-ed25519`, has it).
|
||||
read) and the **private** `recipe-maintainers/cc-ci-secrets` (the `autonomic-bot` account has
|
||||
it; the host gets its own deploy key on that account, §4b/§4c — and the bot password, in
|
||||
`.testenv`, is what registers it).
|
||||
- The out-of-band secrets listed in §4. If you are migrating, they come from the old host; if
|
||||
you are starting fresh you create them (each row says how).
|
||||
- Control of the DNS zone (Gandi for `commoninternet.net`) for the cutover in §7.
|
||||
@@ -153,37 +155,50 @@ Commit on a branch; the rebuild in §5 can use the local checkout before the PR
|
||||
|
||||
Everything in this section is **outside git**. Do it as root over SSH, in this order.
|
||||
|
||||
### 4a. Tailscale
|
||||
### 4a. No tailscale
|
||||
|
||||
```bash
|
||||
# a reusable (or fresh) tailnet auth key from the tailscale admin console
|
||||
install -m600 /dev/stdin /etc/ts-auth-key <<<'tskey-auth-…'
|
||||
```
|
||||
The combined host is NOT on the tailnet (operator decision 2026-09-07): ssh is key-only on the
|
||||
public IP, the CI front doors are public via traefik, and the opencode UI is public on the same
|
||||
443 — traefik routes `oc.ci.commoninternet.net` to an nginx vhost on the docker bridge that
|
||||
enforces basic auth, via traefik labels on a tiny swarm relay stack (`ccci-opencode-ui`, on the weekly sweep's keep-list) that
|
||||
`opencode-ui-route.service` deploys. (Not via traefik's file *directory* provider: enabling
|
||||
that in the coop-cloud traefik recipe REPLACES its provider file, which holds the wildcard cert
|
||||
and the `security` middleware, and every front door goes down — it did, for two minutes, on
|
||||
2026-09-07.) fail2ban guards sshd and that login
|
||||
(`nix/hosts/cc-ci/configuration.nix`; the nginx jail bans in the DOCKER-USER chain because the
|
||||
traffic is docker-forwarded, not host INPUT).
|
||||
|
||||
### 4b. The CI server's checkout and its one out-of-band secret
|
||||
|
||||
```bash
|
||||
# root's deploy key for the private cc-ci-secrets submodule
|
||||
# the bot deploy key for the private cc-ci-secrets submodule — generated ON this host by loops
|
||||
# (§4c) and registered on the autonomic-bot Gitea account; root only points at it:
|
||||
install -d -m700 /root/.ssh
|
||||
install -m600 <autonomic-bot-gitea-ed25519> /root/.ssh/autonomic-bot-gitea-ed25519
|
||||
cat > /root/.ssh/config <<'EOF'
|
||||
cat > /root/.ssh/config <<'SSHCFG'
|
||||
Host git.autonomic.zone
|
||||
Port 2222
|
||||
User git
|
||||
IdentityFile /root/.ssh/autonomic-bot-gitea-ed25519
|
||||
IdentityFile /secrets/files/autonomic-bot-cc-ci-ed25519
|
||||
IdentitiesOnly yes
|
||||
EOF
|
||||
SSHCFG
|
||||
# the deployed checkout: nightly-sweep runs from it, sops reads secrets/secrets.yaml from it
|
||||
git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci.git /etc/cc-ci
|
||||
# the master (recovery) age key — the only sops recipient a fresh host can be
|
||||
# the age identity sops-nix decrypts with. FIRST deploy of a host that is not yet a recipient:
|
||||
# the master (recovery) key, temporarily. Once the host is a recipient (below): its own
|
||||
# ssh-host-key-derived identity, and the master key leaves the box.
|
||||
install -d -m700 /var/lib/sops-nix
|
||||
install -m600 <master-age.txt> /var/lib/sops-nix/key.txt
|
||||
```
|
||||
|
||||
`/etc/cc-ci/secrets/secrets.yaml` is encrypted to the master key and the *old* host's SSH host
|
||||
key. That is enough to deploy. Afterwards (optional, tidier) add the new host as a recipient:
|
||||
`ssh-to-age < /etc/ssh/ssh_host_ed25519_key.pub`, add it to `secrets/.sops.yaml` in cc-ci-secrets,
|
||||
`sops updatekeys secrets.yaml`, push, `git -C /etc/cc-ci submodule update --remote`.
|
||||
`/etc/cc-ci/secrets/secrets.yaml` is encrypted to the master key and the *previous* hosts' SSH
|
||||
host keys. The master key is enough for the FIRST deploy. Then make the new host a recipient so
|
||||
the master key can leave the box again (it belongs off-box, with the operator):
|
||||
`ssh-to-age < /etc/ssh/ssh_host_ed25519_key.pub` → add it to `.sops.yaml` in cc-ci-secrets AND in
|
||||
cc-ci, `sops updatekeys secrets.yaml` (needs the master key, so do it where that lives), push,
|
||||
bump the submodule in cc-ci, `git -C /etc/cc-ci pull --recurse-submodules` on the host, then
|
||||
replace `/var/lib/sops-nix/key.txt` with the host-derived identity
|
||||
(`ssh-to-age -private-key -i /etc/ssh/ssh_host_ed25519_key`), rebuild, and delete the master key
|
||||
from the host.
|
||||
|
||||
### 4c. The orchestrator's workspace (as the `loops` user — it exists after the first rebuild, so
|
||||
run §5 once first if this is a fresh host, then come back)
|
||||
@@ -198,29 +213,30 @@ git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci-adv #
|
||||
mkdir -p .cc-ci-logs .sops
|
||||
```
|
||||
|
||||
Then the files below (`install -m600 -o loops -g users`):
|
||||
Secrets live in **`/secrets/files/`** (owned by loops, mode 700 on the directory), the same
|
||||
convention as the notplants-orchestrator box, with the runtime paths symlinked to them so one
|
||||
`ls /secrets/files` is the complete inventory. **Only cc-ci's secrets go there** — nothing from
|
||||
other projects on the box they were copied from (no tailscale, lichen, tangled, b1, borg, …):
|
||||
secrets do not get spread around. `/secrets/README.txt` lists each file and its purpose.
|
||||
|
||||
| file | what | source |
|
||||
| runtime path → `/secrets/files/…` | what | source |
|
||||
|---|---|---|
|
||||
| `/srv/cc-ci/.testenv` | `TS_AUTH_KEY`, `GITEA_PASSWORD` (autonomic-bot), `DOCKERHUB_USERNAME/TOKEN`, model API keys | old host `/secrets/files/cc-ci.testenv`; fresh: create each credential |
|
||||
| `/srv/cc-ci/upgrader.env` | `LOOP_TIER`, `LOOP_MODEL`, `REPORT_MODEL` for the weekly run (no secrets) | old host, or copy the example in `AGENTS.md` |
|
||||
| `/srv/cc-ci/.sops/master-age.txt` | the same master age key as 4b (skills that re-key secrets use it) | old host |
|
||||
| `~loops/.ssh/cc-ci-root-ed25519` (+`.pub`) | `ssh cc-ci` as root — to loopback on this host | old host; fresh: `ssh-keygen -t ed25519` and add the pub to `nix/hosts/cc-ci/ssh-keys` |
|
||||
| `~loops/.ssh/autonomic-bot-gitea-ed25519` (+`.pub`) | pushes recipe branches / PRs as `autonomic-bot` | old host; fresh: new key added to the bot's Gitea account |
|
||||
| `~loops/.ssh/tangled-ed25519` | optional, tangled.org mirrors | old host |
|
||||
| `~loops/.claude/` | Claude Code auth + settings + the orchestrator session history | old host (`rsync -a`); fresh: `claude auth login` as loops (device code, interactive) |
|
||||
| `~loops/.local/share/opencode/auth.json`, `~loops/.config/opencode/` | opencode provider auth (the weekly upgrader runs on opencode) | old host; fresh: `opencode auth login` |
|
||||
| `/etc/nginx/oc-selfsigned.{crt,key}`, `/etc/nginx/oc-htpasswd` | the tailnet-only opencode UI; **nginx refuses to start without them**, and its config check runs as the `nginx` user, so: `root:nginx`, crt `0644`, key + htpasswd `0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host, or generate (commands in `nix/modules/orchestrator-host.nix`) |
|
||||
| `/srv/cc-ci/.testenv` → `cc-ci.testenv` | `GITEA_PASSWORD` (autonomic-bot: PR/API calls), `DOCKERHUB_USERNAME/TOKEN` (harness image pulls). Nothing else: no tailscale key, no third-party model API keys (opencode's own auth is `opencode-auth.json`; add `TINFOIL_API_KEY` back only if `LOOP_MODEL` moves to a tinfoil model). | old host's copy minus the tailscale line; fresh: create each credential |
|
||||
| `/srv/cc-ci/upgrader.env` (not a secret, lives in the checkout) | `LOOP_TIER`, `LOOP_MODEL`, `REPORT_MODEL` for the weekly run | old host, or copy the example in `AGENTS.md` |
|
||||
| `~loops/.ssh/cc-ci-local-ed25519` (+`.pub`) → same names | `ssh cc-ci` as root — to loopback on this host. **Generated on the host**, its pub in `nix/hosts/cc-ci/ssh-keys` | `ssh-keygen -t ed25519 -C cc-ci-loops-to-root@cc-ci` as loops |
|
||||
| `~loops/.ssh/autonomic-bot-cc-ci-ed25519` (+`.pub`) → same names | pushes recipe branches / PRs as `autonomic-bot`; root's ssh config points at the same file to clone the private cc-ci-secrets submodule. **Generated on the host** and registered on the bot's Gitea account (`POST /api/v1/user/keys` with the bot password) | `ssh-keygen -t ed25519 -C autonomic-bot@cc-ci-host` as loops |
|
||||
| `~loops/.local/share/opencode/auth.json` → `opencode-auth.json`; `~loops/.config/opencode/opencode.jsonc` (config, not secret) | opencode provider auth + config — the orchestrator AND the weekly upgrader are opencode agents; there is no Claude on this host | old host; fresh: `opencode auth login` as loops |
|
||||
| `/etc/nginx/oc-htpasswd` (root:nginx; the bcrypt line only — the plaintext stays with the operator, not on the host) | basic auth for the opencode UI (`https://oc.ci.commoninternet.net`, via traefik); **nginx refuses to start without it**, and its config check runs as the `nginx` user, so `root:nginx 0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host (`/secrets/files/oc-basic-auth.txt` has the plaintext), or a new `oc:<bcrypt>` line via `mkpasswd -m bcrypt` |
|
||||
|
||||
`~loops/.ssh/config` is written by the activation script on first rebuild (`Host cc-ci` →
|
||||
`127.0.0.1`, `git.autonomic.zone`, `tangled.org`); it is not overwritten if present.
|
||||
`127.0.0.1`, `git.autonomic.zone`); it is not overwritten if present.
|
||||
|
||||
## 5. Build and activate
|
||||
|
||||
From the checkout with the §3 commit (root can build from the loops-owned checkout via sudo):
|
||||
|
||||
```bash
|
||||
# as root, detached (the activation restarts sshd/tailscale; a dropped session must not kill it).
|
||||
# as root, detached (the activation restarts sshd; a dropped session must not kill it).
|
||||
# Three things the FIRST rebuild on a bare infect system needs, none of which the converged
|
||||
# host needs afterwards: `git` on PATH (nix's flake fetcher shells out to it and the infect
|
||||
# system has none — hence nix-shell), HOME=/root (so root's `git config --global
|
||||
@@ -234,6 +250,9 @@ journalctl -fu ccci-rebuild # ~10 min the first time (image pulls + two
|
||||
|
||||
`test` first, always: it activates WITHOUT touching the bootloader, so if the activation breaks
|
||||
networking or sshd a reboot from the Hetzner console lands on the last known-good generation.
|
||||
(Earlier hosts were on the tailnet, and their rebuilds had to be detached because activation
|
||||
restarted tailscale under the session; this one is plain public ssh, but detached is still the
|
||||
safe habit.)
|
||||
Later rebuilds are simply `sudo nixos-rebuild test|switch --flake .#cc-ci` from the checkout.
|
||||
|
||||
The first activation takes a while: it pulls the traefik/drone/keycloak images, builds the bridge
|
||||
@@ -243,9 +262,9 @@ deploy-reports`, `deploy-backupbot`, `warm-keycloak`). Verify:
|
||||
|
||||
```bash
|
||||
systemctl is-system-running # running — or list-units --failed and read journalctl -u <unit>
|
||||
tailscale status | head -3
|
||||
docker service ls # traefik app+socket-proxy, drone, bridge, dashboard, reports, backups: 1/1
|
||||
systemctl status cc-ci-loops cc-ci-orchestrator opencode-web nginx acme-dns
|
||||
systemctl status cc-ci-loops cc-ci-orchestrator opencode-web nginx acme-dns fail2ban
|
||||
fail2ban-client status sshd; fail2ban-client status nginx-http-auth
|
||||
systemctl list-timers 'cc-ci-*' nightly-sweep
|
||||
sudo -iu loops tmux ls # cc-ci-orchestrator (+ loops sessions if a phase is active)
|
||||
# the CI front doors, before DNS points here (expect 200 / 200 / 303 and ssl_verify=0 once
|
||||
@@ -256,10 +275,10 @@ curl -s --resolve drone.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{h
|
||||
dig +short @<ip> ns-acme.commoninternet.net # acme-dns answering on the public 53
|
||||
```
|
||||
|
||||
Seen on 2026-09-07: `tailscaled-autoconnect` failed with `invalid key: API key does not exist` —
|
||||
the reusable auth key had been revoked. Generate a fresh one in the tailscale admin console, put
|
||||
it in `/etc/ts-auth-key`, `systemctl restart tailscaled-autoconnect`. Nothing else depends on it
|
||||
during the install; the box is reachable on its public IP throughout.
|
||||
The opencode UI: `curl -s --resolve oc.ci.commoninternet.net:443:<ip> -o /dev/null -w '%{http_code}' https://oc.ci.commoninternet.net/`
|
||||
→ 401 without credentials, 200 with `-u oc:<password>`; `fail2ban-client status nginx-http-auth`
|
||||
counts the failures (it reads nginx's journal — NixOS nginx logs to stderr, not to
|
||||
/var/log/nginx/error.log).
|
||||
|
||||
When it is healthy: `sudo nixos-rebuild switch --flake .#cc-ci` (same config, now also the boot
|
||||
default). **If you are migrating from another host, do §6 before letting it serve anything**: right
|
||||
@@ -273,7 +292,7 @@ systemctl mask --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer
|
||||
|
||||
## 6. Migrating: restore state from the previous host
|
||||
|
||||
Over tailscale (`rsync -aHAX --numeric-ids root@<old>:<path> <path>`), with the matching service
|
||||
Over ssh between the hosts (`ssh root@<old> 'tar -C / -cf - <paths>' | ssh root@<new> 'tar -C / -xpf - --numeric-owner'`, or rsync), with the matching service
|
||||
stopped on the new host while its directory is copied:
|
||||
|
||||
| path | holds | notes |
|
||||
@@ -292,7 +311,11 @@ stopped on the new host while its directory is copied:
|
||||
|
||||
1. **DNS** (operator, Gandi zone `commoninternet.net`): A records `ci`, `*.ci` and `ns-acme` →
|
||||
the new IPv4. `acme NS ns-acme` and `_acme-challenge.ci CNAME <account>.acme…` stay as they
|
||||
are. Wait for propagation (`dig +short ci.commoninternet.net`).
|
||||
are. Wait for propagation (`dig +short ci.commoninternet.net`) — the records carry a 3 h TTL,
|
||||
and resolvers that cached the old answer (the Hetzner ones in particular) keep serving it that
|
||||
long. The new host must not list such a resolver: with a mixed set, every other lookup of its
|
||||
own names went to the old server on 2026-09-07 (runner/bridge 404s). `networking.nix` drops
|
||||
the Hetzner resolvers and `configuration.nix` pins the host's own names.
|
||||
2. Check the new host answers on the new IP before DNS moves: `dig @<new-ip> ns-acme.commoninternet.net`
|
||||
(acme-dns), `curl --resolve ci.commoninternet.net:443:<new-ip> https://ci.commoninternet.net/`
|
||||
(dashboard, valid cert), same for `report.ci` and `drone.ci`.
|
||||
@@ -305,16 +328,26 @@ stopped on the new host while its directory is copied:
|
||||
5. The orchestrator: as loops on the new host `cd /srv/cc-ci-orch && python3 cc-ci-plan/agents.py up orchestrator`
|
||||
(or just `systemctl restart cc-ci-orchestrator`), attach with `claude --resume` or from
|
||||
claude.ai/code. Its startup routine (AGENTS.md) reports phase + reboot count.
|
||||
6. Keep the old host as a cold standby for a week, then delete it and its tailnet node.
|
||||
6. Keep the old host as a cold standby for a week, then delete it (and its tailnet node — the
|
||||
old servers were on the tailnet; the new one is not).
|
||||
|
||||
## 8. Day 2
|
||||
|
||||
- **Update the host** (nixpkgs bump for both halves): `/cc-ci-orchestrator-update`, which is
|
||||
`nix flake update` → `nixos-rebuild test` → verify → `switch` → PR. The `cc-ci` input follows
|
||||
this flake's nixpkgs, so the CI server is rebuilt on the same nixpkgs.
|
||||
- **The host updates itself weekly**: `cc-ci-auto-update.timer` (Tuesday 03:00 UTC,
|
||||
`nix/modules/auto-update.nix`) runs `nix flake update` on all inputs, builds, activates with
|
||||
`test`, runs the cc-ci health checks (swarm replicas, front doors, sops, acme-dns, fail2ban…),
|
||||
and only then commits the generation + bootloader and pushes the health-checked `flake.lock` to
|
||||
`main`; on any failure it re-activates the previous generation and restores the lock. It skips
|
||||
itself while a CI run, the weekly upgrader, the report or the Sunday sweep is in flight. Outcome
|
||||
in `/srv/cc-ci/.cc-ci-logs/auto-update-state` (one line) and `journalctl -u cc-ci-auto-update`;
|
||||
`/cc-ci-status` §5 reads it. `/cc-ci-orchestrator-update` runs the same unit by hand.
|
||||
The `cc-ci` input follows this flake's nixpkgs, so the CI server is rebuilt on the same nixpkgs.
|
||||
- **Update only cc-ci's code** (harness/tests/modules): merge in the cc-ci repo, then
|
||||
`nix flake update cc-ci` here and rebuild; also `git -C /etc/cc-ci pull --recurse-submodules`
|
||||
so the deployed checkout the sweep runs from matches.
|
||||
so the deployed checkout the sweep runs from matches. Run the lock update **on this host**: a
|
||||
lock written by a newer Nix elsewhere once failed here with `NAR hash mismatch in input
|
||||
git+https://…cc-ci.git…&shallow=1`; re-running `nix flake update cc-ci` on the host fetched the
|
||||
input properly and the rebuild went through.
|
||||
- **Something is down**: `systemctl --failed`, `journalctl -u deploy-<x>`, `docker service ps <svc>`;
|
||||
the cc-ci repo's `docs/runbook.md`. Host unreachable: Hetzner console → reboot lands on the last
|
||||
`switch`ed generation; rescue mode + `nixos-enter` for anything worse (skill
|
||||
@@ -322,29 +355,38 @@ stopped on the new host while its directory is copied:
|
||||
|
||||
---
|
||||
|
||||
# Operating the orchestrator session
|
||||
# Operating the orchestrator
|
||||
|
||||
The steering session is a long-lived interactive Claude Code session under tmux with
|
||||
`--remote-control`, so it can be watched and steered from **claude.ai/code** (or the mobile app).
|
||||
`cc-ci-orchestrator.service` relaunches it on boot via `cc-ci-plan/agents.py up orchestrator`
|
||||
(backend + model in `cc-ci-plan/agents.toml`).
|
||||
The cc-ci orchestrator on this host is an **opencode agent** (`cc-ci-plan/agents.toml`:
|
||||
backend `opencode`, model `opencode/glm-5.2`), launched on boot by `cc-ci-orchestrator.service`
|
||||
via `cc-ci-plan/agents.py up orchestrator` into tmux session `cc-ci-orchestrator`, attached to
|
||||
the shared opencode web server. Steer it from the web UI, **https://oc.ci.commoninternet.net**
|
||||
(basic auth, user `oc`), where the weekly upgrader's and report's sessions show up too.
|
||||
There is no Claude Code on this host: Claude sessions run on the notplants-orchestrator box and
|
||||
operate the CI from there over ssh (`Host cc-ci` in that box's loops ssh config → this host).
|
||||
|
||||
```bash
|
||||
# attach on the host
|
||||
sudo -iu loops tmux attach -t cc-ci-orchestrator
|
||||
# or resume the conversation by hand in a fresh tmux pane
|
||||
claude --resume autonomous-orchestrator --remote-control "autonomous-orchestrator" --dangerously-skip-permissions
|
||||
# already inside a live session and just want the web surface? /remote-control
|
||||
# on the host
|
||||
sudo -iu loops tmux attach -t cc-ci-orchestrator # the agent's terminal
|
||||
systemctl restart cc-ci-orchestrator # relaunch it (fresh opencode session)
|
||||
sudo -iu loops tmux ls # cc-ci-orchestrator, cc-ci-upgrader (+watchdog) during the weekly run
|
||||
```
|
||||
|
||||
`--resume <name|id>` selects the *conversation* to restore; the `--remote-control "<name>"` value is
|
||||
only the web display label. Don't pass `--fork-session` unless you mean to branch.
|
||||
## Weekly upgrade run
|
||||
|
||||
`cc-ci-upgrade-all.timer` (Thu 22:00 America/New_York) starts `launch-upgrader.py`, which runs
|
||||
`/upgrade-all` as an opencode agent (`upgrader.env`: `LOOP_MODEL`/`REPORT_MODEL`) in tmux
|
||||
`cc-ci-upgrader` with a usage-limit watchdog; the hourly `cc-ci-upgrade-supervisor.timer` drives a
|
||||
stalled run to completion. Start one by hand with `systemctl start cc-ci-upgrade-all`; follow
|
||||
`/srv/cc-ci/.cc-ci-logs/cc-ci-upgrader.log`; it ends with `UPGRADE RUN COMPLETE`, a summary in
|
||||
`.cc-ci-logs/upgrades/upgrade-all-<date>.md`, and a report published to
|
||||
`https://report.ci.commoninternet.net/week-<date>.html`.
|
||||
|
||||
## Kick off / supervise the loops
|
||||
|
||||
```bash
|
||||
cd /srv/cc-ci/cc-ci-plan
|
||||
./launch.sh start # Builder + Adversary loops (interactive --remote-control in tmux) + watchdog
|
||||
./launch.sh start # Builder + Adversary loops + watchdog (all phases are DONE; this just confirms)
|
||||
./launch.sh status # session + DONE state
|
||||
./launch.sh logs builder|adversary|watchdog
|
||||
./launch.sh stop
|
||||
|
||||
@@ -993,3 +993,111 @@ Both commits were scanned clean and contain no coauthor trailers. No recipe PR w
|
||||
**Security note:** A subagent briefly enabled shell tracing while debugging the verifier, exposing
|
||||
runtime credentials in its private agent trace. No values were committed or put in this journal,
|
||||
but rotate the affected `/srv/cc-ci/.testenv` credentials as a precaution.
|
||||
|
||||
## Session 2026-09-07 19:30 UTC — Claude Fable 5.1 orchestrator (re)launch, startup check
|
||||
|
||||
**What happened:** Orchestrator relaunched on the `claude` backend (`agents.toml` now says
|
||||
`backend = "claude"`, `model = "claude-fable-5-1"`, operator change today, uncommitted). Ran the
|
||||
AGENTS.md on-startup routine. NOT a reboot: host uptime 15 days, REBOOTS.md still shows 5 reboots
|
||||
(last 2026-08-23 03:11 UTC). `cc-ci-loops.service` was restarted at 14:50 and 15:14 UTC today by a
|
||||
`nixos-rebuild test --flake /srv/notplants-nix#notplants-orchestrator`, which re-ran `launch.sh start`;
|
||||
the phase sequence immediately re-concluded (all 15 phases DONE, "entire build finished"), so
|
||||
builder/adversary/watchdog being stopped is the expected terminal state. Did NOT relaunch the loops.
|
||||
|
||||
**Current state:**
|
||||
- Weekly `/upgrade-all` 2026-09-04 completed: 8 upgrade PRs extended (custom-html, ghost,
|
||||
lasuite-docs/drive/meet, matrix-synapse, mattermost-lts, n8n), 0 failed, nothing merged. Report
|
||||
`week-2026-09-04.html` returns 200. Next timer run Fri 2026-09-11 02:00 UTC.
|
||||
- Hourly supervisor (XX:07) fires and stands down in ~1s — nothing to drive.
|
||||
- Open operator items from the 09-04 run: review/merge the 8 PRs; `warm-gitea` canonical
|
||||
crash-looping on read-only `/etc/gitea` (pre-existing); deployed `/root/cc-ci/tests` on the CI host
|
||||
lags server-repo `main` (missing `tests/wordpress`).
|
||||
- Uncommitted in this checkout (left alone, operator WIP): `agents.toml` backend switch,
|
||||
auto-appended 2026-08-23 line in `REBOOTS.md`, and the untracked `plan-agent-orchestrator.md` /
|
||||
`plan-phase-ao*.md` / `cc-ci-conc/` set.
|
||||
|
||||
## Session 2026-09-07 20:00 UTC — start of the cc-ci + orchestrator consolidation onto one Hetzner host
|
||||
|
||||
**Operator request:** move the cc-ci CI server AND the orchestrator to a new Hetzner box
|
||||
(`195.201.88.249`, 8 GB), leave everything notplants-side on this host, keep cc-ci's nix in the
|
||||
cc-ci repo and the orchestrator's in cc-ci-orchestrator with the latter including the former,
|
||||
add `archive/` + a from-scratch deploy README, and (last) move to `autonomic.zone` subdomains.
|
||||
Plan + live log: `cc-ci-plan/plan-cc-ci-combined-host.md` (on the branch; copy here).
|
||||
|
||||
**Done this session:**
|
||||
- New ssh key `notplants-orchestrator` (`/secrets/files/notplants-orchestrator-ed25519`), on the new box.
|
||||
- nixos-infect on the new box (Debian 13 → NixOS 26.05). Gotcha: `/tmp` is tmpfs on that image,
|
||||
nixos-infect's temp swapfile fails → `NO_SWAP=true`. It built and rebooted ~19:50 UTC and had
|
||||
NOT come back by 20:00 (no ping) — operator to check the Hetzner console / give an API token.
|
||||
- cc-ci branch `feat/nixos-module-export` (9b99f81, pushed): `nixosModules.cc-ci-server`
|
||||
(`nix/modules/default.nix`), options `cc-ci.publicIPv4` + `cc-ci.sopsFile`; standalone `#cc-ci`
|
||||
drv byte-identical before/after.
|
||||
- cc-ci-orchestrator branch `feat/combined-cc-ci-host` (31af820, pushed): flake input `cc-ci`
|
||||
(follows), `nixosConfigurations.cc-ci`, `nix/modules/orchestrator-host.nix`, `nix/hosts/cc-ci/`
|
||||
(hardware/networking PROVISIONAL until the infect output is captured), README deploy guide,
|
||||
`archive/` (old host configs, terraform, migration plans), AGENTS.md + update-skill refs.
|
||||
`#cc-ci` evaluates. Work is in git worktrees under the session scratchpad, not in this checkout.
|
||||
|
||||
**Next:** box reachable → capture hardware/networking → stage secrets → `nixos-rebuild test`
|
||||
→ data copy → DNS cutover → move the orchestrator → notplants-nix PR dropping cc-ci → autonomic.zone.
|
||||
|
||||
## Session 2026-09-07 20:30 UTC — new combined host is UP, pre-cutover
|
||||
|
||||
- nixos-infect trouble root-caused from Hetzner rescue mode (operator gave an API token, stored
|
||||
at `/srv/cc-ci/.hcloud-token`, server id 165014541, cpx32 nbg1): (1) `NO_SWAP=true` for tmpfs
|
||||
/tmp; (2) 26.05's systemd initrd did NOT lustrate — Debian's units shadowed NixOS's, every
|
||||
service failed; fixed by moving the old root to `/old-root` by hand; (3) bare-string
|
||||
`defaultGateway` → no default route; fixed + chroot `nixos-rebuild boot --option sandbox false`.
|
||||
All documented in the new README §2a.
|
||||
- cc-ci PR #32 merged (module export). cc-ci-orchestrator PR #19 merged (combined host). Both
|
||||
branches scanned clean by the commit hook.
|
||||
- New box: `nixos-rebuild test` → verified → `switch`; reboot test OK. Data restored: acme (+
|
||||
acme-dns account), acme-dns, ci-certs, reports, runs, ci-warm, /root/.abra, Drone volume (with
|
||||
drone scaled to 0 during the copy). Dashboard/reports/drone answer on the new IP with the valid
|
||||
LE cert; acme-dns answers on public 53.
|
||||
- Pre-cutover quarantine on the new box: `ccci-bridge_app` scaled to 0, both cc-ci timers
|
||||
`mask --runtime`, cc-ci-orchestrator/loops units stopped (these do NOT survive a reboot — redo).
|
||||
- Staged for loops: ~/.claude, opencode config+state, ssh keys, .testenv, upgrader.env,
|
||||
.sops/master-age.txt, .cc-ci-logs; nginx oc-* files (root:nginx 0640).
|
||||
- Open: tailscale auth key revoked (`invalid key: API key does not exist`) → operator issues a
|
||||
new one. DNS cutover at Gandi (ci, *.ci, ns-acme → 195.201.88.249) → operator.
|
||||
|
||||
## 2026-09-07 22:15 UTC — first weekly upgrade run on the new host: GREEN, report published
|
||||
|
||||
Started by hand 21:23 UTC (`systemctl start cc-ci-upgrade-all` on 195.201.88.249, opencode /
|
||||
deepseek-v4-flash); `UPGRADE RUN COMPLETE` 22:02 (39 min). Everything ran on the new host — old
|
||||
server's Drone/bridge at 0/0, no new run dirs or report there. 20 recipes surveyed, 2 upgrade PRs
|
||||
extended and `!testme` GREEN on the new Drone (lasuite-docs #8 → v5.6.1, build 1338; n8n #7 →
|
||||
2.38.4, build 1339), 1 PR closed as merged upstream (custom-html #7), 18 skipped as up-to-date or
|
||||
covered. Summary: `.cc-ci-logs/upgrades/upgrade-all-2026-09-07.md`. Report agent published
|
||||
https://report.ci.commoninternet.net/week-2026-09-07.html (200, 42 KB, indexed) at 22:11.
|
||||
One side effect: the run's orphan sweep removed the `opencode-ui` swarm stack (traefik route to
|
||||
the opencode web UI) — redeployed, renamed `ccci-opencode-ui`, added to the sweep keep-list.
|
||||
|
||||
## 2026-09-08 02:05 UTC — cc-ci host auto-update + skills rewritten for the combined host
|
||||
|
||||
**Auto-update (own module, no notplants-nix dependency):** `nix/modules/auto-update.nix`,
|
||||
`cc-ci-auto-update.timer` Tue 03:00 UTC. Busy gate (CI run / weekly upgrader / report / sweep /
|
||||
running Drone builds) → `nix flake update` (all inputs) → build as loops → `switch-to-configuration
|
||||
test` → 45 s settle → health check (sshd, 0 failed units, docker+acme-dns+fail2ban+nginx+
|
||||
opencode-web+runner active, every swarm service at replica count, sops decrypted,
|
||||
dashboard/reports/drone 200, opencode UI 401) → profile + bootloader → flake.lock committed as
|
||||
`auto-update` and pushed to main → `/etc/cc-ci` fast-forwarded. Failure re-activates the previous
|
||||
generation and restores the lock. One-line outcome in `.cc-ci-logs/auto-update-state`.
|
||||
|
||||
**Tested for real, both paths:**
|
||||
- Full run 22:23–22:27 UTC: nixpkgs 20260803.531670d → 20260906.c257840 and sops-nix moved,
|
||||
built, activated, `HEALTH OK`, committed (`d1f5e91`, pushed), running == boot generation.
|
||||
- Busy gate: with a fake `cc-ci-upgrader` tmux session the unit exited 0 with
|
||||
`result=skipped note=busy` and changed nothing.
|
||||
|
||||
**Skills (canonical in `.opencode/skills`, `.claude` pointer descriptions synced):**
|
||||
- `/cc-ci-orchestrator-update` — THE host update; drives the auto-update unit by hand, has the
|
||||
failure playbook and `--cc-ci-only` for a cc-ci-main-only move.
|
||||
- `/cc-ci-server-update` — now delegates to it, and says why the old procedure (rebuilding the
|
||||
cc-ci repo's standalone `#cc-ci`) must never run on this host.
|
||||
- `/cc-ci-update` — chains orchestrator-update then tests-update.
|
||||
- `/cc-ci-status` — §5/§6 rewritten for one host: auto-update state + staleness, running vs boot
|
||||
generation, front doors (oc.ci must be 401), sops, fail2ban, all four timers, the orchestrator
|
||||
agent session, and a secrets-inventory check that flags sprawl. Verdict updated.
|
||||
Verified against the box: 0 failed units, 10/10 swarm services, disk 33%, all timers armed.
|
||||
|
||||
@@ -22,3 +22,4 @@ restarts the loops on boot. Count the lines below to see how often it's happenin
|
||||
auto-logging. Auto-logging is live from the next reboot onward.
|
||||
- 2026-05-30 17:03:05 BST — reboot detected; loops auto-started by systemd (resuming phase index 6). boot_id=f565f752-0463-42db-b787-9e0db35a5e3f
|
||||
- 2026-05-31 03:38:29 UTC — reboot detected; loops auto-started by systemd (resuming phase index 5). boot_id=51c17fc3-8391-4109-bce2-413fbee6f26d
|
||||
- 2026-08-23 03:11:24 UTC — reboot detected; loops auto-started by systemd (resuming phase index 14). boot_id=8215fc5e-8032-4844-b2b8-e0f8c0d9b13e
|
||||
|
||||
@@ -53,8 +53,9 @@ log_dir = "/srv/cc-ci/.cc-ci-logs"
|
||||
[[agent]]
|
||||
name = "orchestrator" # tmux session: cc-ci-orchestrator
|
||||
kind = "persistent"
|
||||
backend = "opencode" # attached to the shared opencode web server (oc.commoninternet.net)
|
||||
model = "opencode/glm-5.2" # opencode backend; visible at https://oc.commoninternet.net
|
||||
backend = "opencode" # operator 2026-09-07 (evening): NO Claude on the cc-ci host — the
|
||||
model = "opencode/glm-5.2" # orchestrator is an opencode agent, steered via the opencode web UI
|
||||
# (https://oc.ci.commoninternet.net). Claude sessions live on notplants-orchestrator.
|
||||
resume = false # opencode backend has no --resume equivalent (fresh session each launch)
|
||||
watch = "heal" # restart if dead / FATAL / backend-mismatch; never stall-reboot
|
||||
wake = { interval = 3600, prompt_file = "ai-progress-monitor-prompt.txt" }
|
||||
|
||||
@@ -124,7 +124,7 @@ def start(mode, date):
|
||||
_lu._pin_new_session(_prev_ids)
|
||||
if BACKEND == "opencode":
|
||||
if OPENCODE_SHARE:
|
||||
log(f" attached to {OPENCODE_SERVER} → http://oc.commoninternet.net +public --share link")
|
||||
log(f" attached to {OPENCODE_SERVER} → https://oc.ci.commoninternet.net +public --share link")
|
||||
# Watchdog for the report too: it runs on glm-5.2 sharing the same opencode-go budget the
|
||||
# upgrade run just drained, so a 429 stall is likely. Reuse launch-upgrader.py's watchdog,
|
||||
# pointed at THIS (cc-ci-report) session with a report-specific marker + resume prompt. It
|
||||
|
||||
@@ -67,7 +67,7 @@ REMOTE_CONTROL = os.environ.get("REMOTE_CONTROL", "1") == "1"
|
||||
OPENCODE_BIN = os.environ.get("OPENCODE_BIN", "/home/loops/.local/bin/opencode")
|
||||
OPENCODE_SERVER = os.environ.get("OPENCODE_SERVER", "http://127.0.0.1:4096")
|
||||
# Web visibility for the opencode backend: attach the session to the shared opencode
|
||||
# web server (viewable at http://oc.commoninternet.net, tailnet-only) AND optionally
|
||||
# web server (viewable at https://oc.ci.commoninternet.net (basic auth)) AND optionally
|
||||
# create a public opencode.ai --share link. Default both on so the run is monitorable.
|
||||
OPENCODE_SHARE = os.environ.get("OPENCODE_SHARE", "1") == "1"
|
||||
|
||||
@@ -220,7 +220,7 @@ def start(mode="use-or-create"):
|
||||
f"{OPENCODE_BIN} run {model_flag} {share_flag} --attach '{OPENCODE_SERVER}' "
|
||||
f"--title '{SESSION}' \"$(cat '{kf}')\""
|
||||
)
|
||||
log(f" attached to {OPENCODE_SERVER} → http://oc.commoninternet.net (tailnet only)"
|
||||
log(f" attached to {OPENCODE_SERVER} → https://oc.ci.commoninternet.net"
|
||||
+ (" +public --share link (printed in the session)" if OPENCODE_SHARE else ""))
|
||||
else:
|
||||
die(f"unknown LOOP_BACKEND '{BACKEND}' — use 'claude' or 'opencode'")
|
||||
@@ -655,7 +655,7 @@ Model: {MODEL} (LOOP_MODEL or UPGRADER_MODEL env var)
|
||||
Args: {UPGRADER_ARGS or '<none>'} (UPGRADER_ARGS env var, passed to /upgrade-all)
|
||||
|
||||
claude: viewable at claude.ai/code
|
||||
opencode: viewable at http://oc.commoninternet.net server={OPENCODE_SERVER}
|
||||
opencode: viewable at https://oc.ci.commoninternet.net server={OPENCODE_SERVER}
|
||||
""")
|
||||
|
||||
|
||||
|
||||
@@ -68,21 +68,21 @@ notplants-side uses it), tailscale hostname → `notplants-orchestrator`.
|
||||
|
||||
## Steps
|
||||
|
||||
1. [ ] nixos-infect the new box (`NIX_CHANNEL=nixos-26.05 PROVIDER=hetzner`); capture
|
||||
1. [x] nixos-infect the new box (`NIX_CHANNEL=nixos-26.05 PROVIDER=hetzner`); capture
|
||||
`hardware-configuration.nix` + `networking.nix`.
|
||||
2. [ ] cc-ci: module export + options; verify `#cc-ci` still evaluates; PR.
|
||||
3. [ ] cc-ci-orchestrator: input + host + modules + archive/ + README + terraform refresh; verify
|
||||
2. [x] cc-ci: module export + options; verify `#cc-ci` still evaluates; PR.
|
||||
3. [x] cc-ci-orchestrator: input + host + modules + archive/ + README + terraform refresh; verify
|
||||
`#cc-ci` evaluates; PR.
|
||||
4. [ ] Stage secrets + clones on the new host; `nixos-rebuild test` → verify → `switch`.
|
||||
4. [x] Stage secrets + clones on the new host; `nixos-rebuild test` → verify → `switch`.
|
||||
Immediately after: scale the new `ccci-bridge_app` to 0 and mask the two cc-ci timers so the
|
||||
new host does not double-process `!testme` or run a second weekly upgrade before cutover.
|
||||
5. [ ] Copy data (rsync over tailscale): reports, runs, ci-warm, acme, acme-dns, ci-certs,
|
||||
5. [x] Copy data (rsync over tailscale): reports, runs, ci-warm, acme, acme-dns, ci-certs,
|
||||
/root/.abra, /etc/cc-ci; Drone volume with Drone scaled to 0 during the copy.
|
||||
6. [ ] Pre-cutover verification on the new IP (`curl --resolve`, port 53, dashboard, reports,
|
||||
6. [x] Pre-cutover verification on the new IP (`curl --resolve`, port 53, dashboard, reports,
|
||||
drone, one direct `cc-ci-run` on custom-html-tiny).
|
||||
7. [ ] Operator: Gandi A records `ci`, `*.ci`, `ns-acme` → 195.201.88.249. Then: old bridge +
|
||||
7. [x] Operator: Gandi A records `ci`, `*.ci`, `ns-acme` → 195.201.88.249. Then: old bridge +
|
||||
drone + timers off, new bridge up, one real `!testme` end-to-end, a `!testme`-driven report page.
|
||||
8. [ ] Move the orchestrator: stop cc-ci units here, final rsync of `/srv/cc-ci-orch` + agent
|
||||
8. [x] Move the orchestrator: stop cc-ci units here, final rsync of `/srv/cc-ci-orch` + agent
|
||||
state, enable on the new host, operator reconnects there; notplants-nix PR removing cc-ci.
|
||||
9. [ ] Old cc-ci server: cold standby ~1 week, then operator deletes it and the stale tailnet node.
|
||||
10. [ ] Domain move to `autonomic.zone` — separate plan, after 1–9 are proven.
|
||||
@@ -100,3 +100,28 @@ notplants-side uses it), tailscale hostname → `notplants-orchestrator`.
|
||||
`#cc-ci` drv is byte-identical before/after. Orchestrator branch `feat/combined-cc-ci-host`:
|
||||
`#cc-ci` evaluates (gcnwq4fy…-nixos-system-cc-ci-26.05.20260803.531670d.drv) with PROVISIONAL
|
||||
hardware/networking copied from the old CI server — to be replaced by the infect output.
|
||||
- 2026-09-07 20:30 UTC — rescue-mode diagnosis + fixes (README §2a); `#cc-ci` test → switch →
|
||||
reboot OK on the new box; data restored incl. Drone volume; PRs cc-ci #32 + cc-ci-orchestrator
|
||||
#19 merged. Quarantine on (bridge 0, timers runtime-masked, orchestrator units stopped).
|
||||
Waiting on operator: new tailscale auth key; Gandi DNS cutover.
|
||||
- 2026-09-07 21:10 UTC — operator: NO tailscale on the new host; opencode UI behind its basic
|
||||
auth on 443 (traefik labels on the `opencode-ui` socat relay → nginx on docker_gwbridge:8090);
|
||||
fail2ban (sshd + nginx-http-auth in DOCKER-USER). Operator repointed Gandi (ci, *.ci, ns-acme)
|
||||
→ 195.201.88.249 ~20:40; propagated at once at public resolvers. CUTOVER DONE: old bridge +
|
||||
drone scaled to 0, old-host timers masked, new bridge up; `!testme` on custom-html #7 = Drone
|
||||
build 1326 on the new box. Two self-inflicted incidents, both documented in README: traefik
|
||||
FILE_PROVIDER_DIRECTORY_ENABLED took the front doors down 2 min; the Hetzner resolvers in
|
||||
networking.nix served the stale address (3 h TTL) → runner/bridge 404s until dropped.
|
||||
`nixos-rebuild switch` = generation 4. The `switch` re-enabled cc-ci-orchestrator on the new
|
||||
box: a fresh Claude orchestrator session runs there (tmux `cc-ci-orchestrator`, remote-control
|
||||
label `cc-ci-orchestrator`) since 20:46 UTC; the old host's cc-ci units are runtime-masked and
|
||||
the notplants-nix branch `chore/drop-cc-ci` (pushed, evaluates) removes them for good —
|
||||
operator merges (that repo auto-deploys nightly).
|
||||
- Remaining: (9) delete old CI server 134485294 + its tailnet node after ~1 week; revoke the
|
||||
Hetzner token; (10) autonomic.zone domains. Also re-key cc-ci-secrets to the new host's ssh host
|
||||
key (optional; master key works) and re-run the cc-ci self-test build (1324/1325 failed during
|
||||
the resolver mess).
|
||||
- 2026-09-07 21:20 UTC — notplants-nix `chore/drop-cc-ci` merged (9c785c1) and applied on the old
|
||||
host (test → switch, generation 57, healthy, cc-ci units gone). cc-ci push self-test green again
|
||||
after #33 (ruff format, pre-existing since 2026-08-31) + #34 (statix: acme-dns.nix systemd
|
||||
attrset): builds 1330/1331 success. New host on generation 6 with cc-ci main 0437210.
|
||||
|
||||
@@ -137,3 +137,37 @@
|
||||
2.37.5 withdrawn). 2.36.9 holds the Stable/Latest badge; 2.37.x remains Pre-release on GitHub
|
||||
(consistent precedent). Re-verified 2.37.3→2.37.6 (pure core bugfixes), no breaking changes beyond
|
||||
the already-flagged 2.37.0 API behavior pair. Rolling upgrade safe. Recommended release: `-y`.
|
||||
- 2.37.7 (2026-09-01, patch): core/editor bugfixes (distroless runners glibc/libatomic copy; thread
|
||||
execution id through dynamic-credential storage; scope active workflow IDs to user's projects;
|
||||
route secrets pasted in AI chat to real credential forms; gate autosave on document hydration).
|
||||
- 2.37.8/2.37.9/2.37.10/2.37.11 (2026-09-02..04, patches): mostly core/editor/ai-builder bugfixes.
|
||||
2.37.9 and 2.37.10 are noted on the Releases page; 2.37.11 = 1 core fix (thread execution id through
|
||||
dynamic-credential storage backport). No breaking compose/env/migration changes.
|
||||
- 2.38.0 (2026-09-01, Pre-release): the 2.38 feature minor — a large feature/bugfix release (Confluence
|
||||
page/space operations, Databricks user-delegated OAuth2, Moonshot/MiniMax/Qwen Cloud model providers
|
||||
in Agent Builder, background sub-agent delegation, nonce-based Content-Security-Policy on HTML pages,
|
||||
chat trigger publish validation + workflow:execute access control, restore mutating array methods on
|
||||
$json in expressions, MongoDB bulkWrite, etc.). No breaking compose/env/migration changes; no
|
||||
`N8N_*` env renames; the 2.37.0 API behavior pair (JSON content-type on decorator body routes;
|
||||
binary-data endpoint adapt) still applies to HTTP API callers only.
|
||||
- 2.38.1 (2026-09-01, Pre-release): a large patch — bugfixes (mongodb sort/projection parameter
|
||||
binding; S3 startup connection timeout; domain-restricted credential in own node; OAuth token grant
|
||||
binding; restore mutating array methods on $json in expressions; etc.) + features (n8n credits
|
||||
section in agent tools modal, Confluence page get comments/labels/space, Moonshot/MiniMax/Qwen Cloud
|
||||
model providers, nonce-based CSP, split hosted chat page into trusted shell + sandboxed frame).
|
||||
No breaking compose/config changes.
|
||||
- 2.38.2 (2026-09-02, Pre-release): core bugfix (restore ...) — see Releases page.
|
||||
- 2.38.3 (2026-09-0?, Pre-release): patch bugfixes (bundle backports to 2.38.x release track —
|
||||
egress filtering, credential-destination grants, proxy-agent behavior, shared-workflow/user queries;
|
||||
nonce-based CSP on HTML pages). No breaking compose/config changes.
|
||||
- 2.38.4 (2026-09-0?, Pre-release; **newest 2.38.x tag** — abra lists it as the top upgrade candidate):
|
||||
patch bugfixes. No breaking compose/env/migration changes; rolling upgrade safe (sqlite, TypeORM
|
||||
auto-migrate on boot). The 2.37.0 API behavior pair (JSON content-type on decorator body routes;
|
||||
binary-data endpoint adapt) still applies — HTTP API callers only, no recipe impact.
|
||||
- 2026-09-07 run: PR #7 extended 2.34.4 → **2.38.4** (newest tag abra lists; 2.37.5/2.38.0-era withdrawn
|
||||
tags skipped). 2.38.x line remains Pre-release on GitHub (consistent tracking-the-newest-tag
|
||||
precedent: 2.34.2/2.35.2/2.36.3/2.37.3/2.37.6 were all Pre-release when taken). Rolling upgrade safe
|
||||
(sqlite, TypeORM auto-migrate on boot). No breaking compose/config changes for the recipe. Operator
|
||||
flag (unchanged from 2.37.0): the two API behavior changes only affect HTTP API callers. Recommended
|
||||
release: `-y` (feature minor across 2.34.4 → 2.38.4). CVE-2026-73088 was adjudicated NOT-FIXED in the
|
||||
2.x window by the 2026-09-04 parent run; not blocking.
|
||||
|
||||
Generated
+10
-10
@@ -10,11 +10,11 @@
|
||||
]
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1788812004,
|
||||
"narHash": "sha256-Vc7RSeqFHCwlVRhnEEjavuJoyIID+RQdJSygHNA8s8Y=",
|
||||
"lastModified": 1788817027,
|
||||
"narHash": "sha256-EJFM3GMCoSeL3daV/20hS/lkjx2BLETtqL//eWnBdgc=",
|
||||
"ref": "refs/heads/main",
|
||||
"rev": "f6dbfa368995f4d45de09f4052631fd433c87d5b",
|
||||
"revCount": 1533,
|
||||
"rev": "b18da1e4bf0c0285e033a93f3f650421abe54daf",
|
||||
"revCount": 1542,
|
||||
"type": "git",
|
||||
"url": "https://git.autonomic.zone/recipe-maintainers/cc-ci.git"
|
||||
},
|
||||
@@ -25,11 +25,11 @@
|
||||
},
|
||||
"nixpkgs": {
|
||||
"locked": {
|
||||
"lastModified": 1785734586,
|
||||
"narHash": "sha256-ODZkEK9Gy50yg6h98u7KkitZ3oc/uuTFK00bh1CRdNA=",
|
||||
"lastModified": 1788690626,
|
||||
"narHash": "sha256-+v4I4LawmRD/mVxO7QIAerRrCkElp3YImzWkkUnvOTg=",
|
||||
"owner": "NixOS",
|
||||
"repo": "nixpkgs",
|
||||
"rev": "531670d871c0e29724a02f3cbcac170adc65b58c",
|
||||
"rev": "c25784012c9982bca5b3e0de87e90bbdac8927d3",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
@@ -53,11 +53,11 @@
|
||||
]
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1783174389,
|
||||
"narHash": "sha256-aCWC8ngycU7OdJrU2+Je3qf+1a2ykuBvpPhZT/9tXMc=",
|
||||
"lastModified": 1788337237,
|
||||
"narHash": "sha256-gkSH8VUtCo6hnysNmb9DbTuDepH2t5pv+QWjP75xKAk=",
|
||||
"owner": "Mic92",
|
||||
"repo": "sops-nix",
|
||||
"rev": "f1406619a3884cd5c47992a70b8b35c9c0fcb4c9",
|
||||
"rev": "fbf759290e0cb0a98dfc813a4eb7d53ad1dacb57",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
|
||||
@@ -31,6 +31,8 @@
|
||||
# The host contract those units assume: loops user, claude/opencode CLIs, opencode web
|
||||
# server + tailnet UI, nix-ld, tool set, `ssh cc-ci` config.
|
||||
orchestrator-host = ./nix/modules/orchestrator-host.nix;
|
||||
# Weekly health-gated self-update of the host (busy-gated around CI runs).
|
||||
auto-update = ./nix/modules/auto-update.nix;
|
||||
# Old name of cc-ci-orchestrator, kept while notplants-nix still imports it (2026-09).
|
||||
cc-ci = ./nix/modules/cc-ci.nix;
|
||||
};
|
||||
@@ -44,6 +46,7 @@
|
||||
cc-ci.nixosModules.cc-ci-server
|
||||
self.nixosModules.cc-ci-orchestrator
|
||||
self.nixosModules.orchestrator-host
|
||||
self.nixosModules.auto-update
|
||||
./nix/hosts/cc-ci/configuration.nix
|
||||
];
|
||||
};
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# cc-ci — ONE Hetzner Cloud host running both the cc-ci CI server and the cc-ci orchestrator.
|
||||
#
|
||||
# This file is only what is physical or identity about the machine: hardware, networking, the
|
||||
# tailscale node, root SSH keys, swap, stateVersion. Everything functional comes from modules:
|
||||
# This file is only what is physical or identity about the machine: hardware, networking,
|
||||
# root SSH keys, firewall + fail2ban, swap, stateVersion. Everything functional comes from modules:
|
||||
# cc-ci.nixosModules.cc-ci-server recipe-maintainers/cc-ci — swarm, traefik, drone,
|
||||
# runner, bridge, dashboard, reports, acme-dns, harness
|
||||
# self.nixosModules.cc-ci-orchestrator nix/modules/cc-ci.nix — loops, orchestrator, timers
|
||||
@@ -27,13 +27,16 @@
|
||||
# ---- orchestrator identity --------------------------------------------------------------
|
||||
# The CI server is this very host, so `ssh cc-ci` goes to loopback (the module default).
|
||||
cc-ci-orchestrator.ciSshHost = "127.0.0.1";
|
||||
# Weekly self-update (Tue 03:00 UTC; skips itself while CI is busy; see nix/modules/auto-update.nix).
|
||||
cc-ci-orchestrator.autoUpdate.enable = true;
|
||||
|
||||
# ---- tailscale — auth key staged out of band at /etc/ts-auth-key -----------------------
|
||||
services.tailscale = {
|
||||
enable = true;
|
||||
authKeyFile = "/etc/ts-auth-key";
|
||||
extraUpFlags = [ "--hostname=cc-ci" ];
|
||||
};
|
||||
# The opencode UI: traefik (public 443, the *.ci.commoninternet.net cert) → nginx basic auth.
|
||||
cc-ci-orchestrator.opencodeUiHost = "oc.ci.commoninternet.net";
|
||||
cc-ci-orchestrator.opencodeUiTraefikNetwork = "proxy";
|
||||
|
||||
# ---- no tailscale on this host (operator 2026-09-07) --------------------------------------
|
||||
# Everything is reached over the public IP: ssh (keys only), the CI front doors via traefik,
|
||||
# and the opencode UI on 443 (traefik → nginx basic auth). fail2ban below guards the two logins.
|
||||
|
||||
# ---- ssh ----------------------------------------------------------------------------------
|
||||
services.openssh = {
|
||||
@@ -46,16 +49,52 @@
|
||||
# The loops user can also be reached directly (same keys) — handy for rsync of its workspace.
|
||||
users.users.loops.openssh.authorizedKeys.keys =
|
||||
builtins.filter (s: s != "") (lib.splitString "\n" (builtins.readFile ./ssh-keys));
|
||||
services.openssh.settings.PasswordAuthentication = false;
|
||||
services.openssh.settings.KbdInteractiveAuthentication = false;
|
||||
|
||||
# ---- firewall -------------------------------------------------------------------------------
|
||||
# 80/443 (traefik) and 53 (acme-dns) are opened by the cc-ci-server module. The tailscale
|
||||
# interface is trusted, which is what makes the opencode UI on 8443 tailnet-only.
|
||||
# 80/443 (traefik) and 53 (acme-dns) are opened by the cc-ci-server module; the opencode UI
|
||||
# rides 443 through traefik (orchestrator-host.nix), so only ssh is opened here.
|
||||
networking.firewall = {
|
||||
enable = true;
|
||||
trustedInterfaces = [ "tailscale0" ];
|
||||
allowedTCPPorts = [ 22 ];
|
||||
};
|
||||
|
||||
# ---- fail2ban: sshd (password auth is off, this stops the log noise and slow brute force) and
|
||||
# the opencode UI's basic auth (nginx logs 401s with the real client IP to the journal; the
|
||||
# built-in nginx-http-auth filter matches them). Those clients arrive through traefik's
|
||||
# docker-published 443, which iptables FORWARDs rather than INPUTs, so the ban for that jail
|
||||
# goes into the DOCKER-USER chain — an INPUT rule would never see the traffic.
|
||||
services.fail2ban = {
|
||||
enable = true;
|
||||
maxretry = 5;
|
||||
bantime = "1h";
|
||||
bantime-increment = { enable = true; maxtime = "48h"; factor = "4"; };
|
||||
ignoreIP = [ "127.0.0.0/8" "::1" ];
|
||||
jails.nginx-http-auth.settings = {
|
||||
enabled = true;
|
||||
filter = "nginx-http-auth";
|
||||
# NixOS nginx logs errors to stderr → the journal, not /var/log/nginx/error.log (which
|
||||
# exists but stays empty). Read the unit's journal instead.
|
||||
backend = "systemd";
|
||||
journalmatch = "_SYSTEMD_UNIT=nginx.service";
|
||||
banaction = "iptables-allports";
|
||||
chain = "DOCKER-USER";
|
||||
};
|
||||
};
|
||||
|
||||
networking.nameservers = [ "1.1.1.1" "8.8.8.8" ];
|
||||
# This host's own public names resolve to itself regardless of external DNS state (host
|
||||
# processes: the drone runner, the harness, the orchestrator; containers use the resolvers
|
||||
# above). Per-run recipe domains are random and cannot be pinned — those follow public DNS.
|
||||
networking.hosts."195.201.88.249" = [
|
||||
"ci.commoninternet.net"
|
||||
"drone.ci.commoninternet.net"
|
||||
"report.ci.commoninternet.net"
|
||||
"traefik.ci.commoninternet.net"
|
||||
"warm-keycloak.ci.commoninternet.net"
|
||||
"oc.ci.commoninternet.net"
|
||||
];
|
||||
|
||||
# ---- memory: 8 GB RAM shared by the swarm (recipe deploys) and 3–6 agent sessions ---------
|
||||
swapDevices = [ { device = "/swapfile"; size = 8192; } ];
|
||||
|
||||
@@ -5,10 +5,12 @@
|
||||
# This file was populated at runtime with the networking
|
||||
# details gathered from the active system.
|
||||
networking = {
|
||||
nameservers = [ "2a01:4ff:ff00::add:2"
|
||||
"2a01:4ff:ff00::add:1"
|
||||
"185.12.64.2"
|
||||
];
|
||||
# The Hetzner resolvers nixos-infect wrote here are deliberately gone: configuration.nix sets
|
||||
# 1.1.1.1/8.8.8.8. During the 2026-09-07 cutover the Hetzner caches kept the OLD address for
|
||||
# the full 3h TTL while the public resolvers already had the new one, and every other lookup
|
||||
# of drone.ci.commoninternet.net from this box went to the old server (404s in the runner and
|
||||
# the bridge). One consistent resolver set, and the host's own names pinned in
|
||||
# configuration.nix, keep the CI independent of that.
|
||||
defaultGateway = { address = "172.31.1.1"; interface = "eth0"; };
|
||||
defaultGateway6 = {
|
||||
address = "fe80::1";
|
||||
|
||||
@@ -8,3 +8,4 @@ ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHOcLo0YBa0UYi7i/l8K/Y/7cF2OclmDqSTlAsHM0dOS
|
||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMniNzAzuI527bfk/EipqFILFayUCwYXDoZ3R7+QgYq6
|
||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOk8NaeBdPbS2gfUvbny8h0AkZlVjGYHzx4QPXSJ38gd claude@claude-vm
|
||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAcyTGb/wVgdhg5oBCZZvBaR1RuUQRY/3WHnOQpNDCsp claude-cc-ci-sandbox@20260526
|
||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKmGDZC6wrOQNJAW5PPDpxgEXXrcsnIU4b3QJLtq05RQ cc-ci-loops-to-root@cc-ci
|
||||
|
||||
@@ -0,0 +1,199 @@
|
||||
# auto-update.nix — weekly, health-gated self-update of the combined cc-ci host.
|
||||
#
|
||||
# Modelled on notplants-nix's modules/auto-update.nix (the nightly updater the old orchestrator
|
||||
# box had), with the two things the cc-ci host needs on top:
|
||||
# * a BUSY GATE — a `switch` restarts docker/traefik/drone/the runner, so the update skips
|
||||
# (and simply retries next week) while a CI run, the weekly recipe-upgrade run or the
|
||||
# Sunday canonical sweep is in flight;
|
||||
# * cc-ci HEALTH CHECKS — beyond "sshd + no failed units": every swarm service at its replica
|
||||
# count, the front doors (dashboard, reports, drone, the opencode UI) answering, sops having
|
||||
# decrypted, acme-dns and fail2ban up.
|
||||
#
|
||||
# Sequence (as in the notplants module — read its comments for the whys):
|
||||
# flake update (all inputs: nixpkgs, sops-nix, cc-ci) → nixos-rebuild build (as loops) →
|
||||
# switch-to-configuration test (bootloader untouched) → settle → health check →
|
||||
# on failure: re-activate the previous generation, restore flake.lock, exit 1
|
||||
# on success: set the system profile + bootloader, commit flake.lock as "auto-update", push
|
||||
# main (the repo stays the source of truth), refresh /etc/cc-ci.
|
||||
# Nothing is committed that did not pass the health check. Never fires a missed run at boot.
|
||||
# `/cc-ci-status` reads the outcome from the state file this writes.
|
||||
{ config, lib, pkgs, ... }:
|
||||
let
|
||||
cfg = config.cc-ci-orchestrator.autoUpdate;
|
||||
builderHome = config.users.users.${cfg.buildUser}.home;
|
||||
stateFile = "${cfg.flakePath}/.cc-ci-logs/auto-update-state";
|
||||
curl = "${pkgs.curl}/bin/curl";
|
||||
|
||||
busyGate = pkgs.writeShellScript "cc-ci-auto-update-busy-gate" ''
|
||||
set -u
|
||||
busy() { echo "BUSY: $1 — skipping this week's update"; exit 3; }
|
||||
${pkgs.procps}/bin/pgrep -f run_recipe_ci >/dev/null && busy "a CI run is in flight"
|
||||
${pkgs.systemd}/bin/systemctl is-active --quiet nightly-sweep.service && busy "the canonical sweep is running"
|
||||
${pkgs.util-linux}/bin/runuser -u ${cfg.buildUser} -- ${pkgs.tmux}/bin/tmux has-session -t cc-ci-upgrader 2>/dev/null \
|
||||
&& busy "the weekly recipe-upgrade run is in flight (tmux cc-ci-upgrader)"
|
||||
${pkgs.util-linux}/bin/runuser -u ${cfg.buildUser} -- ${pkgs.tmux}/bin/tmux has-session -t cc-ci-report 2>/dev/null \
|
||||
&& busy "the weekly report is being written (tmux cc-ci-report)"
|
||||
if [ -r /run/secrets/bridge_drone_token ]; then
|
||||
running=$(${curl} -s -m 20 -H "Authorization: Bearer $(cat /run/secrets/bridge_drone_token)" \
|
||||
"https://drone.${cfg.ciDomain}/api/repos/recipe-maintainers/cc-ci/builds?per_page=10" \
|
||||
| ${pkgs.gnugrep}/bin/grep -o '"status":"running"' | ${pkgs.coreutils}/bin/wc -l)
|
||||
[ "''${running:-0}" -eq 0 ] || busy "$running Drone build(s) running"
|
||||
fi
|
||||
echo "not busy"
|
||||
'';
|
||||
|
||||
healthCheck = pkgs.writeShellScript "cc-ci-auto-update-health-check" ''
|
||||
set -u
|
||||
fail() { echo "HEALTH FAIL: $1"; exit 1; }
|
||||
${pkgs.systemd}/bin/systemctl is-active --quiet sshd || fail "sshd not active"
|
||||
${pkgs.iproute2}/bin/ss -tlnH | ${pkgs.gnugrep}/bin/grep -q ':22 ' || fail "nothing listening on :22"
|
||||
n=$(${pkgs.systemd}/bin/systemctl --failed --no-legend | ${pkgs.coreutils}/bin/wc -l)
|
||||
[ "$n" -eq 0 ] || fail "$n failed unit(s): $(${pkgs.systemd}/bin/systemctl --failed --no-legend --plain | ${pkgs.gawk}/bin/awk '{print $1}' | ${pkgs.coreutils}/bin/tr '\n' ' ')"
|
||||
for u in docker acme-dns fail2ban nginx opencode-web drone-runner-exec; do
|
||||
${pkgs.systemd}/bin/systemctl is-active --quiet "$u" || fail "$u not active"
|
||||
done
|
||||
# The reconcile oneshots re-run on activation; give the swarm up to 10 minutes to converge.
|
||||
for i in $(${pkgs.coreutils}/bin/seq 1 40); do
|
||||
short=$(${pkgs.docker}/bin/docker service ls --format '{{.Name}} {{.Replicas}}' \
|
||||
| ${pkgs.gawk}/bin/awk '{ split($2,a,"/"); if (a[1] != a[2]) print $1 }')
|
||||
[ -z "$short" ] && break
|
||||
${pkgs.coreutils}/bin/sleep 15
|
||||
done
|
||||
[ -z "$short" ] || fail "swarm services not at their replica count: $(echo "$short" | ${pkgs.coreutils}/bin/tr '\n' ' ')"
|
||||
[ -s /run/secrets/test_secret ] || fail "sops did not decrypt (/run/secrets/test_secret missing)"
|
||||
code() { ${curl} -s -m 20 -o /dev/null -w '%{http_code}' --resolve "$1:443:127.0.0.1" "https://$1/$2"; }
|
||||
[ "$(code ${cfg.ciDomain} "")" = 200 ] || fail "dashboard not 200"
|
||||
[ "$(code report.${cfg.ciDomain} "")" = 200 ] || fail "reports not 200"
|
||||
[ "$(code drone.${cfg.ciDomain} version)" = 200 ] || fail "drone /version not 200"
|
||||
[ "$(code ${cfg.opencodeUiHost} "")" = 401 ] || fail "opencode UI not answering with its auth challenge"
|
||||
echo "HEALTH OK"
|
||||
'';
|
||||
in
|
||||
{
|
||||
options.cc-ci-orchestrator.autoUpdate = {
|
||||
enable = lib.mkEnableOption "weekly health-gated self-update of the cc-ci host";
|
||||
flakePath = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "/srv/cc-ci-orch";
|
||||
description = "The cc-ci-orchestrator checkout (owned by buildUser) whose flake.lock is updated, committed and pushed.";
|
||||
};
|
||||
flakeRef = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "${cfg.flakePath}#cc-ci";
|
||||
description = "Flake reference to build.";
|
||||
};
|
||||
buildUser = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "loops";
|
||||
description = "Owner of the checkout; runs the update, build, commit and push (root activates).";
|
||||
};
|
||||
onCalendar = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "Tue *-*-* 03:00:00 UTC";
|
||||
description = ''
|
||||
When to run. Must stay clear of the weekly recipe-upgrade run (Thu 22:00 America/New_York
|
||||
= Fri 02:00/03:00 UTC, several hours) and the Sunday 03:00 UTC canonical sweep (up to 6 h);
|
||||
the busy gate covers the rest.
|
||||
'';
|
||||
};
|
||||
remote = lib.mkOption { type = lib.types.str; default = "origin"; };
|
||||
branch = lib.mkOption { type = lib.types.str; default = "main"; };
|
||||
ciDomain = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "ci.commoninternet.net";
|
||||
description = "The CI apex; report./drone. are checked under it.";
|
||||
};
|
||||
opencodeUiHost = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = config.cc-ci-orchestrator.opencodeUiHost;
|
||||
description = "The opencode UI host name (expected to answer 401).";
|
||||
};
|
||||
};
|
||||
|
||||
config = lib.mkIf cfg.enable {
|
||||
systemd.services.cc-ci-auto-update = {
|
||||
description = "Weekly cc-ci host self-update (flake update → build → test → health check → commit)";
|
||||
after = [ "network-online.target" "docker.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
path = with pkgs; [ nixos-rebuild nix git openssh coreutils gnugrep gawk systemd util-linux bash python3 docker curl ];
|
||||
serviceConfig = { Type = "oneshot"; TimeoutStartSec = "3h"; };
|
||||
# A unit that performs the switch must never be stopped or restarted BY that switch
|
||||
# (notplants-orchestrator, 2026-09-07: it was, and the host ran half-switched for 12 h).
|
||||
stopIfChanged = false;
|
||||
restartIfChanged = false;
|
||||
script = ''
|
||||
set -u
|
||||
cd ${cfg.flakePath}
|
||||
as_builder() { runuser -u ${cfg.buildUser} -- env HOME=${builderHome} "$@"; }
|
||||
state() { printf '%s result=%s generation=%s note=%s\n' "$(date -u +%FT%TZ)" "$1" "$2" "$3" > ${stateFile}; chown ${cfg.buildUser} ${stateFile} || true; }
|
||||
PREV=$(readlink -f /run/current-system)
|
||||
LOCK_BAK=${cfg.flakePath}/.flake.lock.auto-update-prev
|
||||
|
||||
if ! ${busyGate}; then
|
||||
state skipped "$PREV" busy
|
||||
exit 0
|
||||
fi
|
||||
|
||||
revert() {
|
||||
echo "REVERT: $1"
|
||||
as_builder cp "$LOCK_BAK" flake.lock 2>/dev/null || true
|
||||
"$PREV"/bin/switch-to-configuration test || echo "re-activation of $PREV failed"
|
||||
state failed "$PREV" "$1"
|
||||
exit 1
|
||||
}
|
||||
|
||||
as_builder cp flake.lock "$LOCK_BAK"
|
||||
echo "=== checkout at: $(as_builder git rev-parse --short HEAD) on $(as_builder git rev-parse --abbrev-ref HEAD)"
|
||||
echo "=== flake update (all inputs)"
|
||||
as_builder nix flake update || revert "flake update failed"
|
||||
if as_builder git diff --quiet -- flake.lock; then
|
||||
echo "=== nothing moved; done"
|
||||
state unchanged "$PREV" "lock already current"
|
||||
exit 0
|
||||
fi
|
||||
as_builder git --no-pager diff --stat -- flake.lock
|
||||
|
||||
echo "=== build (cannot affect the running system)"
|
||||
as_builder nixos-rebuild build --flake ${cfg.flakeRef} || revert "build failed"
|
||||
NEW=$(readlink -f result) || revert "no build result"
|
||||
as_builder rm -f result
|
||||
echo "=== built: $NEW"
|
||||
|
||||
echo "=== activate WITHOUT touching the bootloader"
|
||||
"$NEW"/bin/switch-to-configuration test || revert "test activation failed"
|
||||
|
||||
echo "=== settle, then health check"
|
||||
sleep 45
|
||||
${healthCheck} || revert "health check failed"
|
||||
|
||||
echo "=== healthy: committing (system profile + bootloader)"
|
||||
nix-env -p /nix/var/nix/profiles/system --set "$NEW" || revert "could not set system profile"
|
||||
"$NEW"/bin/switch-to-configuration boot || revert "bootloader update failed"
|
||||
state ok "$NEW" "committed"
|
||||
|
||||
# Bookkeeping from here on — never revert a healthy, committed generation over it.
|
||||
if as_builder git -c user.name="auto-update" -c user.email="auto-update@cc-ci" \
|
||||
commit -q -m "flake.lock: weekly auto-update, health-checked" -- flake.lock; then
|
||||
echo "=== flake.lock committed: $(as_builder git rev-parse --short HEAD)"
|
||||
as_builder git push -q ${cfg.remote} HEAD:${cfg.branch} \
|
||||
&& echo "=== pushed ${cfg.branch}" \
|
||||
|| echo "WARNING: push failed; the lock is committed locally, push it by hand"
|
||||
else
|
||||
echo "WARNING: could not commit flake.lock; the system IS committed and healthy"
|
||||
fi
|
||||
# Keep the deployed cc-ci checkout (sweep + sops file) on the same cc-ci rev as the flake input.
|
||||
git -C /etc/cc-ci pull -q --ff-only --recurse-submodules || echo "WARNING: /etc/cc-ci pull failed"
|
||||
echo "=== auto-update committed: $NEW"
|
||||
'';
|
||||
};
|
||||
|
||||
systemd.timers.cc-ci-auto-update = {
|
||||
wantedBy = [ "timers.target" ];
|
||||
timerConfig = {
|
||||
OnCalendar = cfg.onCalendar;
|
||||
Persistent = false;
|
||||
RandomizedDelaySec = "10min";
|
||||
};
|
||||
};
|
||||
};
|
||||
}
|
||||
+14
-14
@@ -17,7 +17,7 @@
|
||||
systemd.services.cc-ci-loops = {
|
||||
description = "cc-ci Builder/Adversary loops + watchdog (launch.sh start)";
|
||||
wantedBy = [ "multi-user.target" ]; # enabled after workspace staged (Hetzner cutover)
|
||||
after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
|
||||
after = [ "network-online.target" "opencode-install.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
# KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the
|
||||
@@ -32,14 +32,14 @@
|
||||
# Append one line to REBOOTS.md per genuine reboot (boot_id-gated; not on manual restart).
|
||||
ExecStartPre = "${pkgs.bash}/bin/bash /srv/cc-ci/cc-ci-plan/reboot-log.sh";
|
||||
};
|
||||
# CLAUDE_BIN points at the standalone CLI installed by claude-install.service; the loops
|
||||
# backend defaults to claude (persisted in .loop-backend). Without this, launch.py's preflight
|
||||
# `which(claude)` fails because the systemd `path` below has no /home/loops/.local/bin.
|
||||
environment = { RESUME_PHASE = "1"; HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
|
||||
# The loops backend is persisted in .cc-ci-logs/.loop-backend (opencode on the cc-ci host —
|
||||
# no Claude there since 2026-09-07). The build phases are all DONE; on boot launch.sh just
|
||||
# re-confirms that and exits.
|
||||
environment = { RESUME_PHASE = "1"; HOME = "/home/loops"; };
|
||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||
script = ''
|
||||
# Put the standalone claude/opencode binaries on PATH. On a cold boot this is the env the
|
||||
# tmux server (and thus every agent session) inherits, so bare `claude` resolves everywhere.
|
||||
# Put the standalone opencode binary on PATH. On a cold boot this is the env the tmux
|
||||
# server (and thus every agent session) inherits.
|
||||
export PATH="/home/loops/.local/bin:$PATH"
|
||||
[ -x /srv/cc-ci/cc-ci-plan/launch.sh ] && /srv/cc-ci/cc-ci-plan/launch.sh start || \
|
||||
echo "workspace not staged yet — skipping loop start"
|
||||
@@ -50,15 +50,15 @@
|
||||
# lichen-orchestrator / project-orchestrator above: this unit only LAUNCHES the orchestrator's
|
||||
# tmux session via the agent-orchestrator harness (cc-ci-plan/agents.py); it does not own the
|
||||
# session or the tmux server. The orchestrator agent is declared in cc-ci-plan/agents.toml
|
||||
# (backend/model chosen there — Claude Code under Remote Control since 2026-09-07; before that
|
||||
# opencode/glm-5.2 attached to the shared opencode web server, opencode-web.service in
|
||||
# orchestrator-host.nix, which the upgrader still uses). The harness watchdog (started by
|
||||
# (backend/model chosen there: an opencode agent, opencode/glm-5.2, attached to the shared
|
||||
# opencode web server — opencode-web.service in orchestrator-host.nix — and steered from its
|
||||
# web UI; no Claude on this host, operator 2026-09-07). The harness watchdog (started by
|
||||
# `agents.py up`) keeps it alive: heal-only (no stall reboots — a persistent supervisor must not
|
||||
# be killed just for idling). Added 2026-08-03 for reboot-resilience.
|
||||
systemd.services.cc-ci-orchestrator = {
|
||||
description = "cc-ci orchestrator (operator steering session) — agents.py up orchestrator";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network-online.target" "tailscaled.service" "opencode-web.service" ];
|
||||
after = [ "network-online.target" "opencode-web.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
# KillMode=process: see the note on cc-ci-loops — a rebuild that merely touches this unit
|
||||
@@ -85,7 +85,7 @@
|
||||
# wantedBy multi-user.target) so it never runs on boot/activation — only on the schedule.
|
||||
systemd.services.cc-ci-upgrade-all = {
|
||||
description = "cc-ci weekly /upgrade-all run (recipe upgrade survey + PRs, never merges)";
|
||||
after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
|
||||
after = [ "network-online.target" "opencode-install.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
Type = "oneshot"; # launch-upgrader.py spawns the cc-ci-upgrader tmux session and returns
|
||||
@@ -103,7 +103,7 @@
|
||||
# timer fire. Holds no secrets (the tinfoil API key lives in the opencode config / auth.json).
|
||||
EnvironmentFile = "-/srv/cc-ci/upgrader.env";
|
||||
};
|
||||
environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
|
||||
environment = { HOME = "/home/loops"; };
|
||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||
script = ''
|
||||
export PATH="/home/loops/.local/bin:$PATH"
|
||||
@@ -129,7 +129,7 @@
|
||||
# short-lived glm-5.2 agent that diagnoses the blockage and drives the run to a clean DONE.
|
||||
systemd.services.cc-ci-upgrade-supervisor = {
|
||||
description = "cc-ci hourly weekly-run supervisor (glm-5.2 — drives a stalled /upgrade-all to completion)";
|
||||
after = [ "network-online.target" "tailscaled.service" ];
|
||||
after = [ "network-online.target" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
Type = "oneshot"; # launch-supervisor.py check: gate now, spawn the agent into tmux, return
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# orchestrator-host.nix — the host contract that nix/modules/cc-ci.nix (the orchestrator's
|
||||
# loops/timers) silently assumes, made explicit and reusable: the `loops` user the agents run as,
|
||||
# the standalone claude/opencode CLIs, the shared opencode web server and its tailnet-only UI,
|
||||
# the standalone opencode CLI, the shared opencode web server and its basic-auth web UI,
|
||||
# nix-ld so foreign binaries run on NixOS, and the tool set agents reach for.
|
||||
#
|
||||
# Exported from flake.nix as `nixosModules.orchestrator-host`. A host imports this together with
|
||||
@@ -23,27 +23,45 @@ in
|
||||
example = "100.95.31.88";
|
||||
description = ''
|
||||
Where `ssh cc-ci` (used by every skill and script that drives the CI server) connects to,
|
||||
as root with ~loops/.ssh/cc-ci-root-ed25519. On the combined host the CI server IS this
|
||||
as root with ~loops/.ssh/cc-ci-local-ed25519 (a key generated ON the host — nothing
|
||||
copied from another machine). On the combined host the CI server IS this
|
||||
machine, so the default is loopback; a standalone orchestrator points it at the CI
|
||||
server's tailnet address.
|
||||
'';
|
||||
};
|
||||
|
||||
opencodeUiPort = lib.mkOption {
|
||||
opencodeUiBackendPort = lib.mkOption {
|
||||
type = lib.types.port;
|
||||
default = 8443;
|
||||
default = 8090; # not 8080: acme-dns's local API has it on the combined host
|
||||
description = ''
|
||||
TLS port of the nginx front door for the opencode web UI. Not 443: on the combined host
|
||||
Traefik (docker swarm) owns 80/443. The port is not opened in the firewall, so it is
|
||||
reachable only over the trusted tailscale interface.
|
||||
Plain-HTTP port nginx listens on for the opencode UI, reachable ONLY from the docker
|
||||
bridge (firewall rule on docker_gwbridge). Traefik — which owns the public 443 on the
|
||||
combined host — terminates TLS for opencodeUiHost and forwards here; nginx adds the basic
|
||||
auth and logs failures for fail2ban with the real client IP.
|
||||
'';
|
||||
};
|
||||
|
||||
opencodeUiTraefikNetwork = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
example = "proxy";
|
||||
description = ''
|
||||
Name of the swarm overlay network the cc-ci traefik watches (cc-ci's swarm.nix creates
|
||||
`proxy`). When set, a one-container swarm stack `ccci-opencode-ui` (a socat TCP relay to
|
||||
nginx on the docker bridge) is deployed with traefik labels routing opencodeUiHost on
|
||||
the `web-secure` entrypoint — the same label mechanism every cc-ci service and recipe
|
||||
uses, so it coexists with the traefik recipe's own file provider (the wildcard cert and
|
||||
the `security` middleware live there; switching traefik to a file *directory* replaces
|
||||
that file and takes every front door down — learned 2026-09-07). null = no route.
|
||||
'';
|
||||
};
|
||||
|
||||
opencodeUiHost = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "oc.commoninternet.net";
|
||||
description = "nginx server_name for the opencode web UI (self-signed, basic auth).";
|
||||
description = "nginx server_name for the opencode web UI (TLS + basic auth).";
|
||||
};
|
||||
|
||||
};
|
||||
|
||||
config = {
|
||||
@@ -62,12 +80,12 @@ in
|
||||
commands = [{ command = "ALL"; options = [ "NOPASSWD" ]; }];
|
||||
}];
|
||||
|
||||
# /home/loops/.local/bin holds the standalone claude + opencode binaries; it must be first on
|
||||
# every PATH (interactive shells, tmux, the systemd units in cc-ci.nix prepend it too).
|
||||
# /home/loops/.local/bin holds the standalone opencode binary; it must be first on every PATH
|
||||
# (interactive shells, tmux, the systemd units in cc-ci.nix prepend it too).
|
||||
environment.variables.PATH = lib.mkForce
|
||||
"/home/loops/.local/bin:/run/current-system/sw/bin:/run/wrappers/bin:/usr/bin:/bin";
|
||||
|
||||
# ---- nix-ld: the standalone Claude Code / opencode CLIs are foreign dynamic ELF binaries ---
|
||||
# ---- nix-ld: the standalone opencode CLI is a foreign dynamic ELF binary --------------------
|
||||
programs.nix-ld.enable = true;
|
||||
programs.nix-ld.libraries = with pkgs; [ stdenv.cc.cc.lib zlib openssl curl glibc ];
|
||||
|
||||
@@ -95,7 +113,7 @@ in
|
||||
Host cc-ci
|
||||
HostName ${cfg.ciSshHost}
|
||||
User root
|
||||
IdentityFile /home/loops/.ssh/cc-ci-root-ed25519
|
||||
IdentityFile /home/loops/.ssh/cc-ci-local-ed25519
|
||||
IdentitiesOnly yes
|
||||
StrictHostKeyChecking accept-new
|
||||
ServerAliveInterval 30
|
||||
@@ -104,11 +122,7 @@ Host git.autonomic.zone
|
||||
HostName git.autonomic.zone
|
||||
Port 2222
|
||||
User git
|
||||
IdentityFile /home/loops/.ssh/autonomic-bot-gitea-ed25519
|
||||
IdentitiesOnly yes
|
||||
|
||||
Host tangled.org
|
||||
IdentityFile /home/loops/.ssh/tangled-ed25519
|
||||
IdentityFile /home/loops/.ssh/autonomic-bot-cc-ci-ed25519
|
||||
IdentitiesOnly yes
|
||||
SSHCFG
|
||||
chmod 600 /home/loops/.ssh/config
|
||||
@@ -117,22 +131,9 @@ SSHCFG
|
||||
'';
|
||||
|
||||
# ---- standalone CLIs (idempotent installers; re-run on every activation, no-op if present) --
|
||||
systemd.services.claude-install = {
|
||||
description = "Install Claude Code CLI for loops user (idempotent)";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network-online.target" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = { Type = "oneshot"; RemainAfterExit = true; User = "loops"; Group = "users"; };
|
||||
environment = { HOME = "/home/loops"; };
|
||||
path = [ pkgs.curl pkgs.bash pkgs.coreutils pkgs.gnutar pkgs.gzip ];
|
||||
script = ''
|
||||
if [ ! -x "$HOME/.local/bin/claude" ]; then
|
||||
echo "installing Claude Code CLI for loops user..."
|
||||
curl -fsSL https://claude.ai/install.sh | bash || echo "install failed — retry on next activation"
|
||||
fi
|
||||
'';
|
||||
};
|
||||
|
||||
# No Claude Code on this host (operator 2026-09-07): the orchestrator and the weekly upgrader
|
||||
# are opencode agents; Claude sessions run on the notplants-orchestrator box and reach this
|
||||
# host over ssh.
|
||||
systemd.services.opencode-install = {
|
||||
description = "Install opencode CLI for loops user (idempotent)";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
@@ -158,7 +159,7 @@ SSHCFG
|
||||
systemd.services.opencode-web = {
|
||||
description = "opencode web server for cc-ci agents";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network-online.target" "tailscaled.service" "opencode-install.service" ];
|
||||
after = [ "network-online.target" "opencode-install.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
Type = "simple";
|
||||
@@ -177,27 +178,69 @@ SSHCFG
|
||||
path = [ pkgs.bash pkgs.coreutils pkgs.git pkgs.python3 pkgs.openssh pkgs.tmux pkgs.nettools ];
|
||||
};
|
||||
|
||||
# ---- tailnet-only nginx front door for the opencode UI -------------------------------
|
||||
# Self-signed cert + basic auth, both created out of band (a store path would be world
|
||||
# readable) — see README "Secrets to stage". nginx FAILS TO START if they are missing.
|
||||
# /etc/nginx/oc-selfsigned.crt root:nginx 0644
|
||||
# /etc/nginx/oc-selfsigned.key root:nginx 0640
|
||||
# /etc/nginx/oc-htpasswd root:nginx 0640 (`oc:<bcrypt>`; plaintext in /secrets)
|
||||
# ---- nginx: basic auth for the opencode UI, behind traefik -----------------------------
|
||||
# Traefik (public 443, the CI wildcard cert) routes opencodeUiHost to this plain-HTTP vhost
|
||||
# on the docker bridge address. nginx enforces HTTP basic auth (the opencode web UI has no
|
||||
# auth of its own and can drive agent sessions), and — via real_ip from traefik's
|
||||
# X-Forwarded-For — logs the CLIENT address on a 401, which is what the fail2ban jail bans.
|
||||
# The htpasswd is created out of band (a store path would be world readable); nginx FAILS TO
|
||||
# START without it, and its config check runs as the nginx user:
|
||||
# /etc/nginx/oc-htpasswd root:nginx 0640 (`oc:<bcrypt>`; plaintext kept in /secrets)
|
||||
# Rotate with: printf 'oc:%s\n' "$(mkpasswd -m bcrypt "$P")" > /etc/nginx/oc-htpasswd && systemctl reload nginx
|
||||
services.nginx = {
|
||||
enable = true;
|
||||
recommendedProxySettings = true;
|
||||
virtualHosts.${cfg.opencodeUiHost} = {
|
||||
listen = [ { addr = "0.0.0.0"; port = cfg.opencodeUiPort; ssl = true; } ];
|
||||
# onlySSL flags the vhost as SSL so the module renders ssl_certificate for the listener.
|
||||
onlySSL = true;
|
||||
sslCertificate = "/etc/nginx/oc-selfsigned.crt";
|
||||
sslCertificateKey = "/etc/nginx/oc-selfsigned.key";
|
||||
listen = [ { addr = "0.0.0.0"; port = cfg.opencodeUiBackendPort; } ];
|
||||
basicAuthFile = "/etc/nginx/oc-htpasswd";
|
||||
extraConfig = ''
|
||||
# traefik sits on the docker networks (ingress 10.0.0.0/24, gwbridge 172.18.0.0/16)
|
||||
set_real_ip_from 172.16.0.0/12;
|
||||
set_real_ip_from 10.0.0.0/8;
|
||||
real_ip_header X-Forwarded-For;
|
||||
'';
|
||||
locations."/" = {
|
||||
proxyPass = "http://127.0.0.1:4096";
|
||||
proxyWebsockets = true;
|
||||
};
|
||||
};
|
||||
};
|
||||
# Only docker's bridge may reach the plain-HTTP backend; the public interface stays closed.
|
||||
networking.firewall.interfaces.docker_gwbridge.allowedTCPPorts = [ cfg.opencodeUiBackendPort ];
|
||||
|
||||
# The traefik side of the route: a swarm service carrying the router labels. Named ccci-* so
|
||||
# the weekly run's orphan sweep (skills/upgrade-all/sweep-orphans.sh keep-list) leaves it be. A plain
|
||||
# TCP relay (socat) from the overlay network to nginx on the docker bridge; traefik's
|
||||
# X-Forwarded-For passes through untouched, which is what nginx's real_ip reads.
|
||||
systemd.services.opencode-ui-route = lib.mkIf (cfg.opencodeUiTraefikNetwork != null) {
|
||||
description = "swarm stack ccci-opencode-ui: traefik labels ${cfg.opencodeUiHost} -> nginx basic auth";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "deploy-proxy.service" "docker.service" "nginx.service" ];
|
||||
wants = [ "deploy-proxy.service" ];
|
||||
path = [ pkgs.docker ];
|
||||
serviceConfig = { Type = "oneshot"; RemainAfterExit = true; };
|
||||
script = ''
|
||||
docker stack deploy --detach=true -c ${pkgs.writeText "opencode-ui-stack.yml" ''
|
||||
# Deployed by opencode-ui-route.service (nix/modules/orchestrator-host.nix). Do not edit.
|
||||
version: "3.8"
|
||||
services:
|
||||
relay:
|
||||
image: alpine/socat:1.8.0.3
|
||||
command: ["TCP-LISTEN:${toString cfg.opencodeUiBackendPort},fork,reuseaddr", "TCP:172.18.0.1:${toString cfg.opencodeUiBackendPort}"]
|
||||
networks: [ ${cfg.opencodeUiTraefikNetwork} ]
|
||||
deploy:
|
||||
replicas: 1
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.opencode-ui.rule=Host(`${cfg.opencodeUiHost}`)"
|
||||
- "traefik.http.routers.opencode-ui.entrypoints=web-secure"
|
||||
- "traefik.http.routers.opencode-ui.tls=true"
|
||||
- "traefik.http.services.opencode-ui.loadbalancer.server.port=${toString cfg.opencodeUiBackendPort}"
|
||||
networks:
|
||||
${cfg.opencodeUiTraefikNetwork}:
|
||||
external: true
|
||||
''} ccci-opencode-ui
|
||||
'';
|
||||
};
|
||||
};
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user