Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 autonomic-bot
|
10840263aa |
@@ -1,97 +0,0 @@
|
|||||||
---
|
|
||||||
name: cc-ci-cleanup
|
|
||||||
description: Tidy the fleet's open recipe PRs. Reconciles every mirror from TRUE upstream first (which alone closes PRs upstream already merged), then surveys every open PR deterministically, CLOSES the ones that can no longer be merged or were never meant to be (CI sweep artifacts, obsolete bumps, superseded duplicates) with a reason, and reports prioritised action items for the ones that SHOULD merge — what specifically is blocking each. NEVER merges a recipe PR. Invoke as /cc-ci-cleanup [recipe ...] [--dry-run].
|
|
||||||
---
|
|
||||||
|
|
||||||
# cc-ci-cleanup
|
|
||||||
|
|
||||||
Open recipe PRs accumulate and rot. Some were never meant to merge (CI sweep artifacts), some were
|
|
||||||
overtaken (upstream merged the same change, or a newer PR supersedes them), and some genuinely should
|
|
||||||
land but are quietly blocked. Left alone the list becomes noise, and a real CVE fix hides in it.
|
|
||||||
|
|
||||||
This skill separates those three, acts on the first two, and hands you a short list for the third.
|
|
||||||
|
|
||||||
**Boundaries.** It **CLOSES** irrelevant PRs and **NEVER MERGES** any recipe PR — those change what
|
|
||||||
deploys on other people's infrastructure, so a human merges them (see AGENTS.md). Closing is the only
|
|
||||||
write it performs, always with a comment saying why.
|
|
||||||
|
|
||||||
## Arguments
|
|
||||||
- `<recipe> …` — limit to these recipes (else every recipe in `cc-ci-plan/used-recipes.md`).
|
|
||||||
- `--dry-run` — classify and report, close nothing.
|
|
||||||
|
|
||||||
## Procedure
|
|
||||||
|
|
||||||
### 1. Reconcile every mirror from TRUE upstream — MANDATORY, FIRST
|
|
||||||
```
|
|
||||||
cc-ci-plan/reconcile-upstream.sh --all # or: reconcile-upstream.sh <recipe>...
|
|
||||||
```
|
|
||||||
**Do not skip this and do not reorder it.** Every signal in step 2 is measured against the mirror's
|
|
||||||
`main`; against a stale mirror they are all wrong. This step also does a chunk of the cleanup by
|
|
||||||
itself — it closes any PR whose changes upstream has already merged.
|
|
||||||
|
|
||||||
> On the first real run (2026-08-11) this alone closed **three** PRs that looked pending and were
|
|
||||||
> already merged upstream: discourse #6 (carrying **140 CVEs**), keycloak #6 (**12 CVEs**), n8n #5.
|
|
||||||
> All three had been reported to the operator as outstanding work. mailu #6 went the same way earlier
|
|
||||||
> the same day. Reconciling is not hygiene, it is how you avoid recommending work that is already done.
|
|
||||||
|
|
||||||
### 2. Survey every open PR (deterministic — no judgement yet)
|
|
||||||
```
|
|
||||||
python3 cc-ci-plan/pr-survey.py [recipe ...] # add --json for the raw facts
|
|
||||||
```
|
|
||||||
Per PR it measures: `behind_main`, `ahead`, `mergeable`, `diff_files`, the images it **adds**, which
|
|
||||||
of those are **already in main**, `obsolete`, the newest `!testme` verdict + build, `branch_kind`,
|
|
||||||
and age/idle days. It decides nothing — that is this skill's job.
|
|
||||||
|
|
||||||
### 3. Classify
|
|
||||||
|
|
||||||
**CLOSE — cannot merge, or was never meant to.** Each needs a *positive* reason, not an absence:
|
|
||||||
|
|
||||||
| signal | why it is closeable |
|
|
||||||
|---|---|
|
|
||||||
| `branch_kind: ci-artifact` (`ci/*`) | regall/cfold sweeps and `!testme` probes — harness artifacts, never intended to merge |
|
|
||||||
| `obsolete: true` | every image it adds is **already pinned in main** — it has nothing left to contribute |
|
|
||||||
| superseded | a newer PR on the same recipe makes the same bump (name both numbers in the comment) |
|
|
||||||
| `diff_files: 0` | genuinely empty diff — nothing to merge |
|
|
||||||
|
|
||||||
**NEVER close on:**
|
|
||||||
- `DIFF-UNREADABLE` — the diff could not be fetched, which is NOT an empty diff. gitea #4 reads that
|
|
||||||
way (force-pushed branch) while being a verified, green, needed fix.
|
|
||||||
- any field that came back `null`/unknown.
|
|
||||||
- a PR that carries a **CVE fix** and is the only thing carrying it, even if it looks stale — report it
|
|
||||||
instead. Losing a security fix to tidiness is far worse than a long PR list.
|
|
||||||
- `--dry-run`.
|
|
||||||
|
|
||||||
**NEEDS WORK — should merge, something blocks it.** Give the *specific* next action:
|
|
||||||
| signal | action item |
|
|
||||||
|---|---|
|
|
||||||
| `mergeable: false` | conflicts — rebase the branch on `main` and re-run `!testme` |
|
|
||||||
| `behind_main > 0` | out of date — rebase, then re-verify (a green from before main moved proves nothing) |
|
|
||||||
| `ci: failed` | diagnose via `/ci-test-review`; classify recipe-bug vs stale test |
|
|
||||||
| `ci: never-run` | run `!testme` |
|
|
||||||
| blocked on the operator | say exactly what is needed (a secret, an upstream release, a decision) |
|
|
||||||
|
|
||||||
**READY — green, current, no conflicts.** Action item is simply: review and merge.
|
|
||||||
|
|
||||||
### 4. Close the CLOSE set (skip entirely under `--dry-run`)
|
|
||||||
Comment first, then close. The comment must say **which signal** made it closeable and **what to do
|
|
||||||
if that is wrong** ("reopen if …"), so a wrong call is cheap to undo. Never close silently.
|
|
||||||
|
|
||||||
### 5. Report
|
|
||||||
Order by what deserves attention, not by recipe name:
|
|
||||||
|
|
||||||
1. **CVE-carrying PRs that should merge** — most severe first, with the CVE ids.
|
|
||||||
2. Other **READY** PRs (green + current).
|
|
||||||
3. **NEEDS WORK**, each with its one specific action.
|
|
||||||
4. **CLOSED this run**, with the reason for each.
|
|
||||||
5. Anything **deliberately left alone** despite looking stale, and why.
|
|
||||||
|
|
||||||
End with a one-line summary: `N open → C closed, R ready to merge, W need work`.
|
|
||||||
|
|
||||||
## Guardrails
|
|
||||||
- **Never merge a recipe PR.** Create/verify/close only; the operator merges.
|
|
||||||
- **Reconcile first, always.** Judging a PR against a stale mirror is how you close good work or
|
|
||||||
recommend work that is already done.
|
|
||||||
- **Close only on a positive signal**, never on "looks old". Age alone is not a reason — several
|
|
||||||
60-day-old PRs here are green and mergeable.
|
|
||||||
- **Never close a lone CVE fix.** Report it, however stale.
|
|
||||||
- Every close gets a comment with its reason and a reopen hint.
|
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
name: cc-ci-orchestrator-update
|
name: cc-ci-orchestrator-update
|
||||||
description: Update the cc-ci HOST — since 2026-09 the ONE Hetzner machine that runs both the cc-ci CI server and the orchestrator (built from this repo's `#cc-ci`, which imports cc-ci's server module) — to the latest nixos-26.05 nixpkgs, sops-nix and cc-ci main, then `nixos-rebuild test` → cc-ci health checks → `switch`, committing the health-checked flake.lock straight to main (the skill invocation IS the authorization; the operator reviews retrospectively). It drives the host's own weekly auto-update unit by hand, so a manual and an automatic update are the same code path. Refuses while CI is busy. `--cc-ci-only` moves only the cc-ci input (harness/tests/modules), for after a cc-ci merge. Answers to /cc-ci-orchestrator-update, /cc-ci-update-orchestrator and (since the hosts merged) /cc-ci-server-update.
|
description: Update the cc-ci ORCHESTRATOR host (the machine the orchestrator itself runs on) to the latest nixos-26.05 nixpkgs, then nixos-rebuild test + switch + verify, ship the bump as a verified PR on recipe-maintainers/cc-ci-orchestrator and merge it directly (the skill invocation IS the authorization). The orchestrator-host analogue of /cc-ci-server-update. Self-update caveats apply — the switch restarts services on the very host running this session; always `nixos-rebuild test` before `switch` so a reboot recovers to the last-known-good generation. The end-of-run report includes the merged PR link + a summary of what changed. Invoke as /cc-ci-orchestrator-update (also answers to /cc-ci-update-orchestrator).
|
||||||
---
|
---
|
||||||
|
|
||||||
# cc-ci-orchestrator-update (thin wrapper)
|
# cc-ci-orchestrator-update (thin wrapper)
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
name: cc-ci-server-update
|
name: cc-ci-server-update
|
||||||
description: (Delegates) Update the cc-ci CI server host to the latest nixpkgs — which, since 2026-09-07, is the SAME machine as the orchestrator, built from the cc-ci-orchestrator flake. This skill only forwards to /cc-ci-orchestrator-update; it no longer bumps or rebuilds the cc-ci repo's own flake, because that standalone `#cc-ci` target would replace the combined host config. Invoke as /cc-ci-server-update.
|
description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main. Opens the cc-ci PR for visibility/historical record and merges it directly once the deploy verifies (the skill invocation IS the authorization); the end-of-run report includes the merged PR link + a summary of what changed. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update.
|
||||||
---
|
---
|
||||||
|
|
||||||
# cc-ci-server-update (thin wrapper)
|
# cc-ci-server-update (thin wrapper)
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
name: cc-ci-update
|
name: cc-ci-update
|
||||||
description: Run both cc-ci maintenance updates in one pass — first /cc-ci-orchestrator-update (the cc-ci host: nixpkgs, sops-nix and the cc-ci input, test → health check → switch, lock committed to main), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update.
|
description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update.
|
||||||
---
|
---
|
||||||
|
|
||||||
# cc-ci-update (thin wrapper)
|
# cc-ci-update (thin wrapper)
|
||||||
|
|||||||
@@ -79,29 +79,12 @@ For each real (non-flaky) finding, write the actual fix and open a PR. **Never m
|
|||||||
it handles the mirror to `git.autonomic.zone/recipe-maintainers/<recipe>` (upstream
|
it handles the mirror to `git.autonomic.zone/recipe-maintainers/<recipe>` (upstream
|
||||||
`git.coopcloud.tech`). Keep the change **bounded** to the diagnosed root cause; don't rewrite the
|
`git.coopcloud.tech`). Keep the change **bounded** to the diagnosed root cause; don't rewrite the
|
||||||
recipe.
|
recipe.
|
||||||
- **Before editing any test, read `tests/STYLE.md` in the cc-ci repo.** It encodes the rules a test
|
|
||||||
change must satisfy — set state up through the app's interface rather than its database, gate on
|
|
||||||
version instead of branching, correct the fixture/wait but NEVER the assertion, and diagnose from
|
|
||||||
the app's own telemetry before concluding a test is stale.
|
|
||||||
- **CI-server-side fix → cc-ci PR.** Branch the cc-ci product repo
|
- **CI-server-side fix → cc-ci PR.** Branch the cc-ci product repo
|
||||||
(`recipe-maintainers/cc-ci`), apply the fix, and open the PR via the Gitea API (use the
|
(`recipe-maintainers/cc-ci`), apply the fix, and open the PR via the Gitea API (use the
|
||||||
`GITEA_*` creds from `/srv/cc-ci/.testenv`). **Single-writer discipline:** work on a dedicated
|
`GITEA_*` creds from `/srv/cc-ci/.testenv`). **Single-writer discipline:** work on a dedicated
|
||||||
branch in a SEPARATE clone — **never push `main`, never touch the build loops' working clones**
|
branch in a SEPARATE clone — **never push `main`, never touch the build loops' working clones**
|
||||||
(`/cc-ci`, `/cc-ci-adv`) or their in-flight state.
|
(`/cc-ci`, `/cc-ci-adv`) or their in-flight state.
|
||||||
|
|
||||||
> ### ⚠️ RECONCILE FROM UPSTREAM FIRST — always, before any PR work or upgrade check
|
|
||||||
> ```
|
|
||||||
> cc-ci-plan/reconcile-upstream.sh <recipe>... # or --all
|
|
||||||
> ```
|
|
||||||
> Deterministic, idempotent, and safe (recipe work lives in branches, never on mirror `main`). It
|
|
||||||
> force-syncs each mirror to coopcloud's **default branch — resolved from the API, `main` OR
|
|
||||||
> `master`** — and closes any mirror PR whose changes upstream already merged. Skipping it has cost
|
|
||||||
> us three distinct ways: mailu #6 was reported as the fix for two internet-facing CVEs while
|
|
||||||
> upstream had already merged AND released it; a stale mirror makes a survey report "no upgrades
|
|
||||||
> available" so the recipe drops out of the weekly run; and reading the wrong branch on a recipe with
|
|
||||||
> a stale `main` beside a live `master` (gitea) manufactures a false "three releases behind, missing
|
|
||||||
> two CVSS-9.8 RCEs" finding.
|
|
||||||
|
|
||||||
### 5. VERIFY each PR on the CI server (deterministic; still never merge)
|
### 5. VERIFY each PR on the CI server (deterministic; still never merge)
|
||||||
A PR is only "working" once **cc-ci verifies it green** (operator rule) — dogfood the CI that found
|
A PR is only "working" once **cc-ci verifies it green** (operator rule) — dogfood the CI that found
|
||||||
the bug. Verification is deterministic (the harness), not an AI judgement.
|
the bug. Verification is deterministic (the harness), not an AI judgement.
|
||||||
|
|||||||
@@ -46,19 +46,12 @@ This is `/recipe-upgrade` step 1's research, stopping before it implements anyth
|
|||||||
> `git -C ~/.abra/recipes/<r> remote set-url origin "https://$GITEA_USERNAME:$GITEA_PASSWORD@git.autonomic.zone/recipe-maintainers/<r>.git"`
|
> `git -C ~/.abra/recipes/<r> remote set-url origin "https://$GITEA_USERNAME:$GITEA_PASSWORD@git.autonomic.zone/recipe-maintainers/<r>.git"`
|
||||||
> 3. **dirty worktree** — usually just the untracked cc-ci overlay; `git stash -u` before, `stash pop`
|
> 3. **dirty worktree** — usually just the untracked cc-ci overlay; `git stash -u` before, `stash pop`
|
||||||
> after. Only a genuinely dirty TRACKED tree is a skip.
|
> after. Only a genuinely dirty TRACKED tree is a skip.
|
||||||
> 4. **tag+digest pins abra cannot parse** — abra FATAs and aborts the WHOLE recipe (immich). Do not
|
> 4. **tag+digest pins abra cannot parse** — abra FATAs and aborts the WHOLE recipe (immich). This is
|
||||||
> hand-check the registry; run the resolver, which is abra-independent and covers every image:
|
> **not** "not fetchable": enumerate the compose's `image:` refs yourself and check the upstream
|
||||||
> ```
|
> registry directly for the ones abra could not read, picking the newest tag the app version
|
||||||
> python3 /srv/cc-ci/cc-ci-plan/resolve-images.py <recipe> --ssh cc-ci --table
|
> supports rather than the numerically highest.
|
||||||
> ```
|
|
||||||
> It reports, per image, `newest_within_major` (the compatibility-safe pick) and
|
|
||||||
> `newest_same_shape` (the newest of that tag's form). **Use `newest_within_major` unless you have
|
|
||||||
> checked the app supports the major jump** — immich's postgres tag encodes the pg major plus the
|
|
||||||
> vectorchord/pgvectors versions immich-server is built against, so taking the newest would break
|
|
||||||
> the deploy. `all_resolved: false` means an image could NOT be resolved — that is a `?`, never a 0.
|
|
||||||
|
|
||||||
**Reconcile the mirror from true upstream FIRST — ALWAYS, no exceptions** — one command,
|
**Reconcile the mirror from true upstream FIRST — ALWAYS, no exceptions.** This is the same reconcile
|
||||||
`cc-ci-plan/reconcile-upstream.sh <recipe>... | --all`. This is the same reconcile
|
|
||||||
`/upgrade-all` does. Do not skip it in the name of keeping the sweep read-only: skipping it makes you
|
`/upgrade-all` does. Do not skip it in the name of keeping the sweep read-only: skipping it makes you
|
||||||
research a stale checkout, and on the first real run that produced **two recipes with no survey output
|
research a stale checkout, and on the first real run that produced **two recipes with no survey output
|
||||||
at all**, which is indistinguishable from "no upgrades" unless you check. It is safe — recipe work
|
at all**, which is indistinguishable from "no upgrades" unless you check. It is safe — recipe work
|
||||||
@@ -73,12 +66,6 @@ nothing; it also auto-closes mirror PRs whose changes upstream has already merge
|
|||||||
> API (`/api/v1/repos/coop-cloud/<recipe>` → `default_branch`) before reading any file, and never
|
> API (`/api/v1/repos/coop-cloud/<recipe>` → `default_branch`) before reading any file, and never
|
||||||
> `git reset --hard origin/main` on a checkout that tracks `master`.
|
> `git reset --hard origin/main` on a checkout that tracks `master`.
|
||||||
|
|
||||||
**Cross-check abra with the resolver.** abra is the primary source, but it silently contributes
|
|
||||||
nothing for images it cannot parse, and it reported "no new versions" for images that did have them
|
|
||||||
(mumble v1.6.870-0 → -4). Run `resolve-images.py` for every recipe and take the UNION of the two: on
|
|
||||||
the first real sweep the resolver found upgrades abra missed entirely in five recipes, one of which
|
|
||||||
(plausible's clickhouse) carried four CVEs.
|
|
||||||
|
|
||||||
Then read versions:
|
Then read versions:
|
||||||
```
|
```
|
||||||
set -a; . /srv/cc-ci/.testenv; set +a
|
set -a; . /srv/cc-ci/.testenv; set +a
|
||||||
@@ -104,27 +91,6 @@ CRITICAL came from, and an image with no window is not counted at all.
|
|||||||
distinction was the difference between two false zeros and the truth (both recipes turned out fine,
|
distinction was the difference between two false zeros and the truth (both recipes turned out fine,
|
||||||
but nothing in the survey said so).
|
but nothing in the survey said so).
|
||||||
|
|
||||||
### 2c. Know which recipes CANNOT see CVEs at all
|
|
||||||
```
|
|
||||||
python3 cc-ci-plan/audit-sources.py --security-sources
|
|
||||||
```
|
|
||||||
A recipe whose sources yield **no CVE data at all** cannot produce a meaningful `0` — nothing was
|
|
||||||
measured, the same way a missing registry file cannot. Render those as **`?`**, not `0`.
|
|
||||||
|
|
||||||
**The fleet is currently at zero such recipes.** The last two — `mattermost-lts` (empty advisory
|
|
||||||
feed, client-side-rendered bulletins) and `mumble` (nothing published anywhere) — were fixed by
|
|
||||||
declaring an NVD CPE in their registry:
|
|
||||||
```
|
|
||||||
- nvd-cpe: mattermost-team-edition = cpe:2.3:a:mattermost:mattermost_server:*:*:*:*:*:*:*:*
|
|
||||||
```
|
|
||||||
**If this sweep ever reports a blind recipe again, that is the fix**: find the product's CPE at
|
|
||||||
nvd.nist.gov and add the line. Prefer a real advisory feed or an attributable changelog when one
|
|
||||||
exists — NVD lags the vendor — but a lagging source beats no source, and it turns a `?` into a
|
|
||||||
number.
|
|
||||||
|
|
||||||
An *unparseable page* is NOT the same thing: it is harmless when the same project also publishes an
|
|
||||||
advisory feed (redis, gitea, minio, clickhouse all do). Only "no usable source for this image" counts.
|
|
||||||
|
|
||||||
### 3. Run the advisory scan over that window
|
### 3. Run the advisory scan over that window
|
||||||
```
|
```
|
||||||
python3 /srv/cc-ci/cc-ci-plan/advisory-scan.py <recipe> --from <old-app> --to <new-app> \
|
python3 /srv/cc-ci/cc-ci-plan/advisory-scan.py <recipe> --from <old-app> --to <new-app> \
|
||||||
|
|||||||
@@ -312,12 +312,6 @@ test change, and a test change is **gated by `--with-tests`**:
|
|||||||
Do **NOT** modify any test. Report `SUCCESS-PENDING-TESTS` (recipe PR open; `!testme` red on a
|
Do **NOT** modify any test. Report `SUCCESS-PENDING-TESTS` (recipe PR open; `!testme` red on a
|
||||||
stale test; operator to decide).
|
stale test; operator to decide).
|
||||||
- **`--with-tests` — open + verify a cc-ci test PR.** Make it the `ci-test-review` way:
|
- **`--with-tests` — open + verify a cc-ci test PR.** Make it the `ci-test-review` way:
|
||||||
0. **READ `tests/STYLE.md` in the cc-ci repo FIRST.** It is the rulebook for changing a test, and
|
|
||||||
it is written against the failures this pipeline has actually produced. The two that matter most
|
|
||||||
here: **set state up through the app's own interface, never its database** (a plausible fixture
|
|
||||||
that INSERTed rows passed on v2 and silently broke on v3, holding the recipe RED for six weeks),
|
|
||||||
and **gate on version rather than writing a fixture that supports both** — old-version tests can
|
|
||||||
simply be deleted, since the older version is only exercised through the upgrade tier.
|
|
||||||
1. Branch `recipe-maintainers/cc-ci` in a **separate clone** (single-writer: never push `main`,
|
1. Branch `recipe-maintainers/cc-ci` in a **separate clone** (single-writer: never push `main`,
|
||||||
never touch the build loops' `/cc-ci` `/cc-ci-adv` clones); update the test/overlay.
|
never touch the build loops' `/cc-ci` `/cc-ci-adv` clones); update the test/overlay.
|
||||||
2. **Verify the recipe upgrade WITH the updated test applied.** `!testme` on the recipe PR uses the
|
2. **Verify the recipe upgrade WITH the updated test applied.** `!testme` on the recipe PR uses the
|
||||||
|
|||||||
@@ -18,7 +18,7 @@ RECIPE="${1:?usage: testme-on-pr.sh <recipe> <pr-index>}"
|
|||||||
PRIDX="${2:?usage: testme-on-pr.sh <recipe> <pr-index>}"
|
PRIDX="${2:?usage: testme-on-pr.sh <recipe> <pr-index>}"
|
||||||
TESTENV="${TESTENV:-/srv/cc-ci/.testenv}"
|
TESTENV="${TESTENV:-/srv/cc-ci/.testenv}"
|
||||||
set -a; . "$TESTENV"; set +a
|
set -a; . "$TESTENV"; set +a
|
||||||
: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:=git.autonomic.zone}"
|
: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:?}"
|
||||||
NS="${GITEA_NAMESPACE:-recipe-maintainers}"
|
NS="${GITEA_NAMESPACE:-recipe-maintainers}"
|
||||||
API="https://${GITEA_URL}/api/v1"; AUTH=(-u "${GITEA_USERNAME}:${GITEA_PASSWORD}")
|
API="https://${GITEA_URL}/api/v1"; AUTH=(-u "${GITEA_USERNAME}:${GITEA_PASSWORD}")
|
||||||
POST="${POST:-1}"; MAX_WAIT="${MAX_WAIT:-480}"; INTERVAL="${INTERVAL:-30}"
|
POST="${POST:-1}"; MAX_WAIT="${MAX_WAIT:-480}"; INTERVAL="${INTERVAL:-30}"
|
||||||
|
|||||||
@@ -9,16 +9,14 @@
|
|||||||
# SAFE BY ALLOWLIST. It removes ONLY things NOT on the keep-list, so it can never take down infra or
|
# SAFE BY ALLOWLIST. It removes ONLY things NOT on the keep-list, so it can never take down infra or
|
||||||
# the warm canonicals. The keep-list (leading name prefix) is:
|
# the warm canonicals. The keep-list (leading name prefix) is:
|
||||||
# - traefik, drone, backups : Swarm + CI infra
|
# - traefik, drone, backups : Swarm + CI infra
|
||||||
# - ccci-bridge / -dashboard / -reports / -opencode-ui: the cc-ci control plane (opencode-ui is
|
# - ccci-bridge / -dashboard / -reports: the cc-ci control plane
|
||||||
# the traefik route to the orchestrator's opencode web UI, deployed by the host's nix config —
|
|
||||||
# the 2026-09-07 run reaped it as an "orphan" before it was on this list)
|
|
||||||
# - warm-* : warm canonicals (idle persistent deps reused across runs;
|
# - warm-* : warm canonicals (idle persistent deps reused across runs;
|
||||||
# their retained volumes are spared too)
|
# their retained volumes are spared too)
|
||||||
# Everything else deployed on the Swarm is a per-run test stack and is fair game.
|
# Everything else deployed on the Swarm is a per-run test stack and is fair game.
|
||||||
set -uo pipefail
|
set -uo pipefail
|
||||||
export PATH=/run/current-system/sw/bin:$PATH
|
export PATH=/run/current-system/sw/bin:$PATH
|
||||||
|
|
||||||
KEEP_RE='^(traefik|drone|backups|ccci-(bridge|dashboard|reports|opencode-ui)|warm-)'
|
KEEP_RE='^(traefik|drone|backups|ccci-(bridge|dashboard|reports)|warm-)'
|
||||||
removed=0
|
removed=0
|
||||||
|
|
||||||
echo "== orphan sweep: scanning (keep-list: infra + warm-* canonicals) =="
|
echo "== orphan sweep: scanning (keep-list: infra + warm-* canonicals) =="
|
||||||
@@ -75,29 +73,6 @@ done
|
|||||||
# 5) Stray exited containers (debug one-shots) — best-effort prune.
|
# 5) Stray exited containers (debug one-shots) — best-effort prune.
|
||||||
docker container prune -f >/dev/null 2>&1 || true
|
docker container prune -f >/dev/null 2>&1 || true
|
||||||
|
|
||||||
# 6) Unused IMAGES — the one that actually took CI down. Every run pulls each recipe's images and
|
|
||||||
# nothing ever removed the old ones: on 2026-08-11 they had grown to 72GB (63GB of it unused),
|
|
||||||
# the root filesystem hit 100% under two concurrent runs, and the harness died at startup with
|
|
||||||
# `OSError: [Errno 28] No space left on device: '/var/lib/cc-ci-runs/<build>'`. Every !testme
|
|
||||||
# from build 1236 to 1242 failed that way — with no results.json, so the PR badges just read
|
|
||||||
# "failure" and looked like recipe regressions.
|
|
||||||
#
|
|
||||||
# Only prune above a threshold, so a healthy host keeps its layer cache and runs stay fast.
|
|
||||||
# `image prune -a` removes only images no container references, so anything deployed (infra +
|
|
||||||
# warm-* canonicals) is untouched; anything else is re-pulled on demand.
|
|
||||||
#
|
|
||||||
# Volumes are deliberately NOT pruned here — see the KEEP_RE guard in (3): warm-* canonicals are
|
|
||||||
# data-warm and their volumes are legitimately dangling between runs.
|
|
||||||
DISK_PRUNE_PCT="${DISK_PRUNE_PCT:-60}"
|
|
||||||
used_pct="$(df --output=pcent / 2>/dev/null | tail -1 | tr -dc '0-9')"
|
|
||||||
if [ -n "$used_pct" ] && [ "$used_pct" -ge "$DISK_PRUNE_PCT" ]; then
|
|
||||||
echo " disk ${used_pct}% >= ${DISK_PRUNE_PCT}% -> pruning unused images"
|
|
||||||
freed="$(docker image prune -af 2>/dev/null | awk '/Total reclaimed space/ {print $4, $5}')"
|
|
||||||
echo " reclaimed: ${freed:-0B}; disk now $(df -h / | tail -1 | awk '{print $5" used, "$4" free"}')"
|
|
||||||
else
|
|
||||||
echo " disk ${used_pct:-?}% < ${DISK_PRUNE_PCT}% -> keeping image cache"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [ "$removed" -eq 0 ]; then
|
if [ "$removed" -eq 0 ]; then
|
||||||
echo "== orphan sweep: clean (nothing to remove) =="
|
echo "== orphan sweep: clean (nothing to remove) =="
|
||||||
else
|
else
|
||||||
|
|||||||
@@ -32,6 +32,3 @@ __pycache__/
|
|||||||
# Local API tokens — never committed (advisory-scan / hetzner recovery)
|
# Local API tokens — never committed (advisory-scan / hetzner recovery)
|
||||||
.github-token
|
.github-token
|
||||||
.hcloud-token
|
.hcloud-token
|
||||||
|
|
||||||
# auto-update.nix keeps the pre-update lock here so a failed run can restore it (never committed)
|
|
||||||
.flake.lock.auto-update-prev
|
|
||||||
|
|||||||
@@ -1,87 +1,164 @@
|
|||||||
---
|
---
|
||||||
name: cc-ci-orchestrator-update
|
name: cc-ci-orchestrator-update
|
||||||
description: Update the cc-ci HOST — since 2026-09 the ONE Hetzner machine that runs both the cc-ci CI server and the orchestrator (built from this repo's `#cc-ci`, which imports cc-ci's server module) — to the latest nixos-26.05 nixpkgs, sops-nix and cc-ci main, then `nixos-rebuild test` → cc-ci health checks → `switch`, committing the health-checked flake.lock straight to main (the skill invocation IS the authorization; the operator reviews retrospectively). It drives the host's own weekly auto-update unit by hand, so a manual and an automatic update are the same code path. Refuses while CI is busy. `--cc-ci-only` moves only the cc-ci input (harness/tests/modules), for after a cc-ci merge. Answers to /cc-ci-orchestrator-update, /cc-ci-update-orchestrator and (since the hosts merged) /cc-ci-server-update.
|
description: Update the cc-ci ORCHESTRATOR host (the machine the orchestrator itself runs on) to the latest nixos-26.05 nixpkgs, then nixos-rebuild test + switch + verify, ship the bump as a verified PR on recipe-maintainers/cc-ci-orchestrator and merge it directly (the skill invocation IS the authorization). The orchestrator-host analogue of /cc-ci-server-update. Self-update caveats apply — the switch restarts services on the very host running this session; always `nixos-rebuild test` before `switch` so a reboot recovers to the last-known-good generation. The end-of-run report includes the merged PR link + a summary of what changed. Invoke as /cc-ci-orchestrator-update (also answers to /cc-ci-update-orchestrator).
|
||||||
---
|
---
|
||||||
|
|
||||||
# cc-ci-orchestrator-update
|
# cc-ci-orchestrator-update
|
||||||
|
|
||||||
Update **the cc-ci host** — one machine since 2026-09-07 (`cc-ci`, 195.201.88.249, no tailnet):
|
Manually update the **orchestrator host**'s NixOS dependencies (nixpkgs; sops-nix follows nixpkgs)
|
||||||
the CI server (docker swarm, traefik, drone, runner, bridge, dashboard, reports, acme-dns) AND the
|
to the current `nixos-26.05` channel tip, ship the bump via a verified PR + a live
|
||||||
orchestrator (loops, weekly upgrader, opencode agent) — to the current inputs, verified, committed.
|
`nixos-rebuild test` → `switch`, and confirm the host is healthy afterward. This is the
|
||||||
|
orchestrator-host analogue of `/cc-ci-server-update` — done deliberately, with a reviewable PR and
|
||||||
|
a health gate, not as silent drift.
|
||||||
|
|
||||||
> **One flake, one target.** The host is `nixosConfigurations.cc-ci` in
|
> **Two hosts, two flakes — don't confuse them.** This skill updates the **orchestrator** host:
|
||||||
> `recipe-maintainers/cc-ci-orchestrator` (`/srv/cc-ci-orch` on the host). The cc-ci repo's own
|
> the machine this session runs on (`cc-ci-orchestrator-1`, Hetzner cpx22 **server 134487234**,
|
||||||
> `flake.nix` inputs are IRRELEVANT to the live build — this flake makes the `cc-ci` input follow
|
> tailnet `100.84.190.30`, public `168.119.126.100`), flake checkout **`/srv/cc-ci-orch`** (repo
|
||||||
> its nixpkgs/sops-nix. Never run `nixos-rebuild` on the cc-ci repo's `#cc-ci` target on this host:
|
> `recipe-maintainers/cc-ci-orchestrator`), target **`.#cc-ci-orchestrator-hetzner`**. The **cc-ci
|
||||||
> that is the standalone-server config and would replace the combined one (orchestrator half gone).
|
> CI server** (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, target `.#cc-ci`) is a different
|
||||||
|
> machine — that's `/cc-ci-server-update`, NOT this skill.
|
||||||
|
|
||||||
The work is done by **`cc-ci-auto-update.service`** (`nix/modules/auto-update.nix`), the same unit
|
> **This is a SELF-update.** The `switch` restarts units on the host running this very session.
|
||||||
the Tuesday 03:00 UTC timer fires. Running it by hand gives you the identical sequence —
|
> tmux sessions and the orchestrator survive a switch (systemd restarts don't kill user tmux), but
|
||||||
busy gate → `nix flake update` → build → `switch-to-configuration test` → 45 s settle → health
|
> `tailscaled`/`sshd` restarts can briefly drop Remote Control / SSH — run activation steps
|
||||||
check (sshd, 0 failed units, docker/acme-dns/fail2ban/nginx/opencode-web/runner active, every swarm
|
> detached (`setsid nohup … &`) and re-poll. A full **reboot** DOES end the session: the
|
||||||
service at its replica count, sops decrypted, dashboard/reports/drone 200, opencode UI 401) →
|
> auto-launch flow (REBOOTS.md + `cc-ci-loops.service`) restarts the loops, and the operator
|
||||||
profile + bootloader → `flake.lock` committed as `auto-update` and pushed to `main` →
|
> reconnects the orchestrator — announce a planned reboot to the operator first.
|
||||||
`/etc/cc-ci` fast-forwarded — with a journal to read and a one-line state file.
|
|
||||||
|
## What this skill does NOT do
|
||||||
|
|
||||||
|
- **Does NOT touch the cc-ci CI server** — that's `/cc-ci-server-update`.
|
||||||
|
- **Does NOT bump recipe images or cc-ci tests** — `/upgrade-all` / `/cc-ci-tests-update`.
|
||||||
|
- **Does NOT change the channel** (`nixos-26.05` branch ref in `flake.nix`) without operator
|
||||||
|
choice — a channel jump (e.g. → 26.11 when it releases) is a bigger change: ask first, read the
|
||||||
|
release notes, and expect module/option renames.
|
||||||
|
|
||||||
## Preconditions
|
## Preconditions
|
||||||
|
|
||||||
- `ssh cc-ci` works from the notplants-orchestrator box (root on the host), or you are on the host.
|
- You are on the orchestrator host; `/srv/cc-ci-orch` is the live checkout (git repo, remote
|
||||||
- **Not busy.** The unit checks itself (CI run, weekly upgrader, report, sweep, running Drone
|
`recipe-maintainers/cc-ci-orchestrator`).
|
||||||
builds → `result=skipped note=busy`, exit 0). Don't fight it: wait, or `tmux ls` as loops and
|
- Quiescent moment: no `/upgrade-all` in flight, no active build-loop phase, no cc-ci CI run that
|
||||||
`docker service ls` to see what is running. Never kill a run to update a host.
|
would be disturbed if this host blips (`pgrep -af run_recipe_ci` empty on cc-ci is a bonus, but
|
||||||
- Note the current generation so you can name the old one in the report:
|
the orchestrator host restarting mostly risks its own sessions, not the CI server).
|
||||||
`ssh cc-ci 'readlink /run/current-system; cat /srv/cc-ci/.cc-ci-logs/auto-update-state'`.
|
- `git status` in `/srv/cc-ci-orch` — note pre-existing dirt; don't sweep unrelated changes into
|
||||||
|
the bump commit.
|
||||||
|
|
||||||
## Procedure
|
## Procedure
|
||||||
|
|
||||||
```bash
|
### 1. Snapshot baseline
|
||||||
# 1. what will move
|
|
||||||
ssh cc-ci 'cd /srv/cc-ci-orch && sudo -H -u loops nix flake update --dry-run 2>&1 | grep -v "^warning" | tail -8' # informational; the unit does the real update
|
|
||||||
git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05 | cut -c1-12
|
|
||||||
|
|
||||||
# 2. run the unit and follow it (10-25 min: nixpkgs bumps rebuild the two OCI images too)
|
```
|
||||||
ssh cc-ci 'systemctl start --no-block cc-ci-auto-update; journalctl -fu cc-ci-auto-update' # Ctrl-C when it prints "auto-update committed" / "REVERT" / "BUSY"
|
nixos-version; readlink /run/current-system; uname -r
|
||||||
# or, non-interactively:
|
systemctl --failed --no-legend
|
||||||
ssh cc-ci 'systemctl start cc-ci-auto-update; systemctl is-active cc-ci-auto-update; journalctl -u cc-ci-auto-update --no-pager -n 40'
|
df -h / | tail -1
|
||||||
|
cd /srv/cc-ci-orch && nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
|
||||||
|
tmux ls
|
||||||
|
```
|
||||||
|
Record: current nixpkgs rev, generation, failed units (should be 0), disk, live tmux sessions.
|
||||||
|
|
||||||
# 3. read the outcome
|
### 2. Branch + bump the lock
|
||||||
ssh cc-ci 'cat /srv/cc-ci/.cc-ci-logs/auto-update-state; readlink /run/current-system; readlink /nix/var/nix/profiles/system; systemctl --failed --no-legend; cd /srv/cc-ci-orch && git log --oneline -1 && git status --short'
|
|
||||||
|
Work directly in `/srv/cc-ci-orch` on a branch (it's the live checkout; the flake edit is inert
|
||||||
|
until a rebuild):
|
||||||
|
```
|
||||||
|
cd /srv/cc-ci-orch
|
||||||
|
git checkout -b chore/orchestrator-flake-update-$(date -u +%Y%m%d)
|
||||||
|
nix flake update 2>&1 | tail -5 # nixpkgs tracks nixos-26.05; sops-nix follows nixpkgs
|
||||||
|
nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
|
||||||
|
```
|
||||||
|
Note old→new revs. If the lock didn't move, report "already at channel tip" and stop (nothing to
|
||||||
|
deploy).
|
||||||
|
|
||||||
|
### 3. Build (catch errors before any activation)
|
||||||
|
|
||||||
|
```
|
||||||
|
cd /srv/cc-ci-orch && nixos-rebuild build --flake .#cc-ci-orchestrator-hetzner 2>&1 | tail -15
|
||||||
|
readlink -f result
|
||||||
|
```
|
||||||
|
Build failure → fix on the branch (option renames etc.) before going further. Never activate a
|
||||||
|
build that didn't build cleanly.
|
||||||
|
|
||||||
|
### 4. Test-activate (MANDATORY before switch — general policy)
|
||||||
|
|
||||||
|
`nixos-rebuild test` activates without touching the bootloader or system profile, so if the
|
||||||
|
activation breaks the host (cf. the cc-ci server's 2026-08-03 no-default-route outage), a plain
|
||||||
|
reboot — Hetzner API power-cycle on server **134487234** if SSH is gone (see
|
||||||
|
`hetzner-server-recovery`) — lands back on the last-known-good generation.
|
||||||
|
```
|
||||||
|
cd /srv/cc-ci-orch && setsid nohup nixos-rebuild test --flake .#cc-ci-orchestrator-hetzner \
|
||||||
|
> /tmp/orchestrator-test-switch.log 2>&1 < /dev/null & echo launched
|
||||||
|
# after it settles (poll; tailscaled/sshd may blip):
|
||||||
|
readlink /run/current-system # should be the new store path
|
||||||
|
ip route show default # default route survived
|
||||||
|
systemctl --failed --no-legend # 0 failed
|
||||||
|
tmux ls # sessions survived
|
||||||
|
tail -5 /tmp/orchestrator-test-switch.log
|
||||||
|
```
|
||||||
|
Regression here → you're still on the old boot default: diagnose, fix, or reboot to revert. Do NOT
|
||||||
|
switch.
|
||||||
|
|
||||||
|
### 5. Switch (make permanent — only after 4 is healthy)
|
||||||
|
|
||||||
|
```
|
||||||
|
cd /srv/cc-ci-orch && nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner 2>&1 | tail -10
|
||||||
|
```
|
||||||
|
(If it fails with "Unit nixos-rebuild-switch-to-configuration.service was already loaded", the
|
||||||
|
detached test's transient unit is still running — wait or `systemctl stop` it, then retry.)
|
||||||
|
|
||||||
|
### 6. Health gate (do not skip)
|
||||||
|
|
||||||
|
```
|
||||||
|
nixos-version; readlink /run/current-system
|
||||||
|
systemctl --failed --no-legend
|
||||||
|
tmux ls # orchestrator + loop sessions intact
|
||||||
|
tailscale status | head -3 # tailnet up
|
||||||
|
systemctl list-timers cc-ci-upgrade-all.timer --no-pager | head -3
|
||||||
|
ssh cc-ci 'hostname' 2>&1 # can still reach the CI server
|
||||||
|
df -h / | tail -1
|
||||||
|
```
|
||||||
|
Optional (operator's call — it ends this session): a cold-boot proof reboot. If done, the
|
||||||
|
REBOOTS.md auto-launch flow restarts the loops; the operator reconnects the orchestrator.
|
||||||
|
|
||||||
|
### 7. Commit + PR + merge + report
|
||||||
|
|
||||||
|
```
|
||||||
|
cd /srv/cc-ci-orch
|
||||||
|
git add flake.lock # flake.nix too if the channel ref changed
|
||||||
|
git commit -m "flake: bump nixpkgs (nixos-26.05, $(date -u +%Y-%m-%d))
|
||||||
|
|
||||||
|
nixpkgs: <old-rev[:8]> -> <new-rev[:8]> (nixos-26.05 tip)
|
||||||
|
Deployed to cc-ci-orchestrator-hetzner: build + test + switch + health gate green."
|
||||||
|
git push -u origin HEAD
|
||||||
|
```
|
||||||
|
Open the PR on `recipe-maintainers/cc-ci-orchestrator` (Gitea API with the `GITEA_*` creds from
|
||||||
|
`/srv/cc-ci/.testenv`, same as `open-cc-ci-pr.sh` does for cc-ci — the helper is hardcoded to the
|
||||||
|
cc-ci repo, so use the API directly or a copy with the repo swapped). PR body: old→new revs, what
|
||||||
|
was verified. Then **merge it directly** — the PR is for visibility/historical record; the skill
|
||||||
|
invocation is the authorization. On a rolled-back/failed run, leave the PR open with an
|
||||||
|
explanatory comment.
|
||||||
|
|
||||||
|
Back on the host, return the checkout to main so the live tree matches the merged state:
|
||||||
|
```
|
||||||
|
git checkout main && git pull
|
||||||
```
|
```
|
||||||
|
|
||||||
`--cc-ci-only` (after merging something in the cc-ci repo that the host should run now):
|
Write `/srv/cc-ci-orch/.cc-ci-logs/orchestrator-update-<YYYY-MM-DD>.md`
|
||||||
```bash
|
(`RESULT: SUCCESS/ROLLED-BACK/FAILED …`), append a line to `cc-ci-plan/JOURNAL.md`, and report to
|
||||||
ssh cc-ci 'sudo -H -u loops bash -lc "cd /srv/cc-ci-orch && nix flake update cc-ci && git -c user.name=auto-update -c user.email=auto-update@cc-ci commit -qm \"flake.lock: cc-ci -> main\" -- flake.lock && git push -q origin HEAD:main"'
|
the operator with the **merged PR link + a summary of what changed**.
|
||||||
ssh cc-ci 'git -C /etc/cc-ci pull --ff-only --recurse-submodules'
|
|
||||||
ssh cc-ci 'systemd-run --unit=ccci-rebuild --collect -E HOME=/root -p WorkingDirectory=/srv/cc-ci-orch bash -lc "nixos-rebuild test --flake /srv/cc-ci-orch#cc-ci && nixos-rebuild switch --flake /srv/cc-ci-orch#cc-ci"'
|
|
||||||
# then the same step-3 checks, plus the §6 health commands from /cc-ci-status
|
|
||||||
```
|
|
||||||
|
|
||||||
## When it fails
|
|
||||||
|
|
||||||
`result=failed note=<step>` means the unit already re-activated the previous generation and
|
|
||||||
restored `flake.lock`; the host is as it was. Read `journalctl -u cc-ci-auto-update` for the
|
|
||||||
`HEALTH FAIL:` / `REVERT:` line:
|
|
||||||
- **build failed** — usually an upstream nixpkgs breakage or a cc-ci main that does not evaluate
|
|
||||||
against the new nixpkgs. Reproduce: `sudo -H -u loops nixos-rebuild build --flake /srv/cc-ci-orch#cc-ci`
|
|
||||||
in the checkout after `nix flake update`; fix in the right repo (PR, merge), re-run.
|
|
||||||
- **swarm services not at their replica count** — one of the reconcile oneshots did not converge
|
|
||||||
in 10 min: `journalctl -u deploy-<proxy|drone|bridge|dashboard|reports>`, `docker service ps <svc>`.
|
|
||||||
- **NAR hash mismatch** on the cc-ci input — the lock was made on another machine; `nix flake
|
|
||||||
update cc-ci` on the host and re-run (README day-2).
|
|
||||||
- **push failed** — bookkeeping only (the generation is committed): push the lock by hand as loops.
|
|
||||||
- A hung run: the unit has a 3 h timeout; `systemctl status cc-ci-auto-update`. Do NOT restart the
|
|
||||||
unit or the host mid-switch (see the `stopIfChanged` comment in the module).
|
|
||||||
|
|
||||||
## Report (end of run)
|
|
||||||
|
|
||||||
- old generation → new generation, the nixpkgs/sops-nix/cc-ci revs before → after, the lock commit
|
|
||||||
on `main`, the health check result, and anything the journal warned about.
|
|
||||||
- Then `/cc-ci-status` §5–§6 should read `ok` — say if it does not.
|
|
||||||
|
|
||||||
## Guardrails
|
## Guardrails
|
||||||
|
|
||||||
- The unit is the only thing that `switch`es here; do not hand-roll `nixos-rebuild switch` unless
|
- **Build before test, test before switch** — never `switch` without a healthy `test` activation
|
||||||
`--cc-ci-only` says so, and then `test` first.
|
(general policy for this host and the cc-ci server; `test` keeps the bootloader untouched so a
|
||||||
- Never `nix flake update` from the notplants box and push a lock for this host (NAR hashes differ
|
reboot always recovers).
|
||||||
between Nix versions) — locks for the host are made on the host.
|
- **Detach activations** (`setsid nohup … &`) — a dropped SSH/Remote-Control connection must not
|
||||||
- Nothing recipe-side: no PRs merged, no tests touched; `/cc-ci-tests-update` is separate.
|
kill a half-applied activation.
|
||||||
|
- **Self-update awareness:** announce before any reboot (it ends the operator's session); a
|
||||||
|
switch is fine but expect brief tailscaled/sshd blips.
|
||||||
|
- **Single-writer:** dedicated branch; don't sweep unrelated working-tree changes into the bump
|
||||||
|
commit; return the live checkout to `main` after the merge.
|
||||||
|
- **Stable channel only** (`nixos-26.05` until the operator chooses a jump).
|
||||||
|
- **PRs for visibility, merged directly** once verified; failed runs leave the PR open. Always
|
||||||
|
report merged-PR links + change summaries.
|
||||||
|
- **Rescue path:** Hetzner server **134487234** — same rescue-mode procedure as
|
||||||
|
`hetzner-server-recovery` (grub generations live in a submenu: `1>N` ids).
|
||||||
|
|||||||
@@ -1,25 +1,298 @@
|
|||||||
---
|
---
|
||||||
name: cc-ci-server-update
|
name: cc-ci-server-update
|
||||||
description: (Delegates) Update the cc-ci CI server host to the latest nixpkgs — which, since 2026-09-07, is the SAME machine as the orchestrator, built from the cc-ci-orchestrator flake. This skill only forwards to /cc-ci-orchestrator-update; it no longer bumps or rebuilds the cc-ci repo's own flake, because that standalone `#cc-ci` target would replace the combined host config. Invoke as /cc-ci-server-update.
|
description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main. Opens the cc-ci PR for visibility/historical record and merges it directly once the deploy verifies (the skill invocation IS the authorization); the end-of-run report includes the merged PR link + a summary of what changed. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update.
|
||||||
---
|
---
|
||||||
|
|
||||||
# cc-ci-server-update → `/cc-ci-orchestrator-update`
|
# cc-ci-server-update
|
||||||
|
|
||||||
There is no separate CI-server host any more. The CI server runs on the cc-ci host
|
Manually update the **cc-ci CI server host**'s NixOS dependencies (nixpkgs + sops-nix) to their
|
||||||
(195.201.88.249) together with the orchestrator, from **one** flake:
|
latest pinned revisions, ship the bump via a verified cc-ci PR + a live `nixos-rebuild switch`, and
|
||||||
`recipe-maintainers/cc-ci-orchestrator` `#cc-ci`, which imports the cc-ci repo's
|
confirm the server is healthy afterward. This is the cc-ci-server analogue of a `nix flake update`
|
||||||
`nixosModules.cc-ci-server` and makes that input follow its own nixpkgs and sops-nix.
|
+ rebuild — done deliberately, with a reviewable PR and a health gate, not as silent drift.
|
||||||
|
|
||||||
**Run `/cc-ci-orchestrator-update`.** It updates nixpkgs, sops-nix AND the cc-ci input, tests,
|
> **Two hosts, two flakes — don't confuse them.** This skill updates the **cc-ci server** flake
|
||||||
health-checks (the CI server's swarm, front doors, sops, acme-dns — the checks this skill used to
|
> (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, host target `.#cc-ci` = `nix/hosts/cc-ci-hetzner/`).
|
||||||
do) and switches, via the host's own weekly auto-update unit.
|
> The **orchestrator** host (`/srv/cc-ci-orch`, flake host `.#cc-ci-orchestrator-hetzner`) is a
|
||||||
|
> different machine with its own flake — it is NOT touched here. If you want to update the
|
||||||
|
> orchestrator host, that's a separate `nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner`
|
||||||
|
> from `/srv/cc-ci-orch` (the orchestrator flake already tracks `nixos-26.05` and is rebuilt by the
|
||||||
|
> operator). This skill is only about the cc-ci **server**.
|
||||||
|
|
||||||
Do **not** do what the old version of this skill did:
|
## What this skill does NOT do
|
||||||
- do not bump `flake.lock` in the cc-ci repo expecting it to change the live host (it cannot — the
|
|
||||||
host follows the orchestrator flake's inputs; cc-ci's lock only matters for the standalone
|
|
||||||
`#cc-ci` target, which nothing deploys);
|
|
||||||
- do not run `nixos-rebuild switch --flake /etc/cc-ci#cc-ci` or `...#cc-ci-hetzner` on the host —
|
|
||||||
that is the standalone-server configuration and would take the orchestrator half down.
|
|
||||||
|
|
||||||
If what you actually wanted is "the host should run cc-ci `main` as of now" (a harness or test
|
- **Does NOT bump recipe image tags** — that's `/upgrade-all` / `/recipe-upgrade`.
|
||||||
change), that is `/cc-ci-orchestrator-update --cc-ci-only`.
|
- **Does NOT touch cc-ci tests / harness / runner** — that's `/cc-ci-tests-update` or
|
||||||
|
`/ci-dev-workflow`. This is a flake-input bump only (`flake.nix` input revs + `flake.lock`).
|
||||||
|
- **Does NOT update the orchestrator host** — only the cc-ci server.
|
||||||
|
- **Does merge the cc-ci PR itself** — the PR exists for visibility and historical record, and is
|
||||||
|
merged directly at the end of a verified run (the skill invocation is the authorization; the
|
||||||
|
operator reviews the merged PR afterwards via the link in the report).
|
||||||
|
|
||||||
|
## Preconditions / access
|
||||||
|
|
||||||
|
- `ssh cc-ci` works (root, tailnet `100.95.31.88`). If it doesn't, fix access first (Tailscale ACL
|
||||||
|
/ `cc-ci-tailscaled` proxy) — do NOT proceed.
|
||||||
|
- **The operator's choice to invoke this skill IS the authorization for the live rebuild** — there
|
||||||
|
is no mid-run operator check to wait for. A `nixos-rebuild switch` cycles the abra reconcile
|
||||||
|
oneshots (`deploy-proxy`, `warm-keycloak`, …), so prefer a **quiescent moment** (no `!testme` /
|
||||||
|
nightly sweep running) to avoid disrupting an in-flight CI run. Check first:
|
||||||
|
```
|
||||||
|
ssh cc-ci 'docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"' # any unhealthy task?
|
||||||
|
ssh cc-ci 'pgrep -af run_recipe_ci | grep -v grep' # a CI run in flight?
|
||||||
|
```
|
||||||
|
If either is non-empty, WAIT until the host is quiescent — do not rebuild mid-CI.
|
||||||
|
- `/srv/cc-ci/.testenv` has the `GITEA_*` creds (for opening the cc-ci PR).
|
||||||
|
- The **operator-held sops secrets** are on the cc-ci host at `/etc/cc-ci/secrets/secrets.yaml`
|
||||||
|
(NOT in git). The rebuild needs them — see the rebuild procedure below.
|
||||||
|
|
||||||
|
## Procedure
|
||||||
|
|
||||||
|
### 1. Snapshot current state (so you can compare + roll back)
|
||||||
|
|
||||||
|
```
|
||||||
|
ssh cc-ci 'hostname; uptime; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \
|
||||||
|
deploy-bridge deploy-reports deploy-backupbot drone-runner-exec; systemctl --failed --no-legend; \
|
||||||
|
df -h / | tail -1; nixos-version; readlink /run/current-system'
|
||||||
|
```
|
||||||
|
Record: the current `nixos-version`, the current-system store path, the list of active infra
|
||||||
|
services, and the disk usage. These are your "before" baseline. Any rebuild that leaves a unit
|
||||||
|
failed or an endpoint down gets compared against this.
|
||||||
|
|
||||||
|
Also record the **current flake input revs** (the "before" of what you're bumping):
|
||||||
|
```
|
||||||
|
ssh cc-ci 'cd /root/builder-clone && nix flake metadata --json 2>/dev/null | jq -r ".locks.nodes | to_entries[] | \"\(.key) \(.value.locked.rev // \"-\")\""' 2>&1 | head
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Prepare a working clone of the cc-ci repo (single-writer)
|
||||||
|
|
||||||
|
Work in a **separate clone on the orchestrator** — never push `main`, never touch the build loops'
|
||||||
|
working clones. Use the existing `/home/loops/work/cc-ci-fix` clone if present and clean, or make a
|
||||||
|
fresh one:
|
||||||
|
```
|
||||||
|
git clone ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /home/loops/work/cc-ci-server-update
|
||||||
|
cd /home/loops/work/cc-ci-server-update
|
||||||
|
git fetch origin && git checkout main && git reset --hard origin/main
|
||||||
|
git checkout -b chore/server-flake-update-$(date -u +%Y%m%d)
|
||||||
|
```
|
||||||
|
(If `/home/loops/work/cc-ci-fix` already exists and is on a stale branch, either reuse it on a new
|
||||||
|
branch or make the fresh clone above — the key rule is the branch is dedicated + off `main`.)
|
||||||
|
|
||||||
|
### 3. Bump the flake inputs (deliberate, not drift)
|
||||||
|
|
||||||
|
The cc-ci server flake **intentionally hard-pins** `nixpkgs` to a specific revision (the comment in
|
||||||
|
`flake.nix` says "Bump deliberately, not drift") and `sops-nix` to a commit compatible with that
|
||||||
|
nixpkgs. So the bump is a **considered move to a newer nixpkgs + a compatible sops-nix**, not a
|
||||||
|
blind `nix flake update --relock-all`.
|
||||||
|
|
||||||
|
**(a) Decide the new nixpkgs.** The cc-ci server runs recipe CI; nixpkgs provides docker, abra's
|
||||||
|
runtime, traefik, keycloak, drone, the harness python/pytest/playwright toolchain. Pick the newest
|
||||||
|
**stable channel revision** that's compatible:
|
||||||
|
- Default: move to the latest revision of the **same channel** the orchestrator host tracks
|
||||||
|
(`nixos-26.05`), so both hosts stay on a shared nixpkgs and CVEs get patched. Look up the current
|
||||||
|
tip: `git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05` (the SHA after the ref).
|
||||||
|
- If a newer stable channel is out and the operator wants it, use that — but treat a **channel
|
||||||
|
jump** (e.g. 26.05 → 26.11) as a bigger change: scan the release notes for breaking changes
|
||||||
|
(docker, systemd, python, nginx module option renames) and surface them in the PR body. Default
|
||||||
|
to staying on the same channel unless the operator asks for the jump.
|
||||||
|
- **Never** jump to `nixos-unstable` for the cc-ci server — recipe CI reproducibility depends on a
|
||||||
|
stable nixpkgs. Unstable is for experiments, not production.
|
||||||
|
|
||||||
|
**(b) Update `flake.nix`** with the new nixpkgs rev (replace the `rev = …` in the `nixpkgs.url`),
|
||||||
|
and check whether `sops-nix` needs a matching bump:
|
||||||
|
- sops-nix `inputs.nixpkgs.follows = "nixpkgs"`, so it tracks nixpkgs automatically — but its own
|
||||||
|
rev pin may need moving if the pinned sops-nix commit is incompatible with the new nixpkgs (e.g.
|
||||||
|
the `buildGoModule` → `buildGo125Module` gotcha the current pin's comment calls out). After
|
||||||
|
editing `flake.nix`, run `nix flake lock --update-input nixpkgs --update-input sops-nix` (or
|
||||||
|
`nix flake update` if both follow) in the clone and let nix resolve a compatible sops-nix. If the
|
||||||
|
lock fails (sops-nix master needs a newer nixpkgs than you picked), pick a sops-nix commit that
|
||||||
|
still builds against your chosen nixpkgs (its `flake.nix` declares the nixpkgs it expects).
|
||||||
|
- **Keep the `sops-nix.inputs.nixpkgs.follows = "nixpkgs"` line** — that's what keeps them aligned.
|
||||||
|
- Update the comment in `flake.nix` to reflect the new pin date + reason (the existing comment
|
||||||
|
style: "Pinned to … (operator YYYY-MM-DD)" or "Follows nixos-26.05 channel").
|
||||||
|
|
||||||
|
**(c) Re-lock + sanity-build locally first** (cheap, catches a broken input before touching the
|
||||||
|
host):
|
||||||
|
```
|
||||||
|
cd /home/loops/work/cc-ci-server-update # (or cc-ci-fix on the new branch)
|
||||||
|
nix flake lock --update-input nixpkgs --update-input sops-nix # rewrites flake.lock
|
||||||
|
nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
|
||||||
|
git diff flake.nix flake.lock | head -40
|
||||||
|
```
|
||||||
|
Verify the lock moved to the revs you intended. **Do NOT `nixos-rebuild build` from the
|
||||||
|
orchestrator** — the orchestrator's nixpkgs is different and the cc-ci host target won't build here
|
||||||
|
(meaningfully). The real build happens on the cc-ci host in step 5.
|
||||||
|
|
||||||
|
### 4. Commit + open the cc-ci PR (merged in step 7 after the health gate)
|
||||||
|
|
||||||
|
```
|
||||||
|
git add flake.nix flake.lock
|
||||||
|
git commit -m "flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))
|
||||||
|
|
||||||
|
nixpkgs: <old-rev[:8]> → <new-rev[:8]> (nixos-<channel>)
|
||||||
|
sops-nix: <old-rev[:8]> → <new-rev[:8]>
|
||||||
|
<one-line reason — CVE patch / channel bump / sops-nix compat>"
|
||||||
|
git push -u origin HEAD
|
||||||
|
```
|
||||||
|
Open the PR via the helper:
|
||||||
|
```
|
||||||
|
set -a; . /srv/cc-ci/.testenv; set +a
|
||||||
|
TITLE="flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))" \
|
||||||
|
BODY_FILE=/tmp/cc-ci-server-update-pr-body.md \
|
||||||
|
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh
|
||||||
|
```
|
||||||
|
(Write the PR body to `/tmp/cc-ci-server-update-pr-body.md` first: old→new revs, the channel, any
|
||||||
|
breaking changes from the release notes, and the rebuild plan. The PR is opened for visibility and
|
||||||
|
historical record; it gets merged directly in step 7 once the deploy passes the health gate.)
|
||||||
|
|
||||||
|
Capture the `PR_URL`. **The cc-ci push-build lint gate will run on the branch** — if it's red on a
|
||||||
|
flake-lock-only change, it's a pre-existing lint drift (not yours); note it in the PR body and
|
||||||
|
proceed (the rebuild doesn't depend on the lint gate passing).
|
||||||
|
|
||||||
|
### 5. Deploy to the live cc-ci host (no-CI window)
|
||||||
|
|
||||||
|
This is the host-rebuild procedure from memory `cc-ci-host-rebuild-procedure` — the cc-ci server
|
||||||
|
has **no self-service rebuild path**, so the orchestrator stages the repo + rebuilds it. The
|
||||||
|
deploy clone on the host is `/root/builder-clone`; staging a fresh copy avoids disturbing it.
|
||||||
|
|
||||||
|
**(a) Stage the PR branch on the host:**
|
||||||
|
```
|
||||||
|
# from the orchestrator, on the PR branch, clean tree:
|
||||||
|
rsync -a --delete --exclude='.git' /home/loops/work/cc-ci-server-update/ root@cc-ci:/root/cc-ci-deploy/
|
||||||
|
ssh cc-ci 'chown -R root:root /root/cc-ci-deploy'
|
||||||
|
```
|
||||||
|
(A `--delete` rsync from a clean tree = exactly the PR head. Excluding `.git` keeps it a plain path
|
||||||
|
flake — see (c).)
|
||||||
|
|
||||||
|
**(b) Copy the operator-held sops secrets** (NOT in git, required for the build):
|
||||||
|
```
|
||||||
|
ssh cc-ci 'mkdir -p /root/cc-ci-deploy/secrets && cp /etc/cc-ci/secrets/secrets.yaml /root/cc-ci-deploy/secrets/secrets.yaml && chmod 600 /root/cc-ci-deploy/secrets/secrets.yaml'
|
||||||
|
```
|
||||||
|
(The age key is at `/var/lib/sops-nix/key.txt` on the host — already in place from prior rebuilds.
|
||||||
|
Without `secrets/secrets.yaml` the build FATAs `secrets/secrets.yaml does not exist`.)
|
||||||
|
|
||||||
|
**(c) Build first (catch errors before switching):**
|
||||||
|
```
|
||||||
|
ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild build --flake .#cc-ci' 2>&1 | tail -30
|
||||||
|
```
|
||||||
|
- Target is `.#cc-ci` (= `.#cc-ci-hetzner` = `nix/hosts/cc-ci-hetzner/configuration.nix`).
|
||||||
|
- nixpkgs is now your bumped pin, so this **will** rebuild more than a same-pin no-op — expect a
|
||||||
|
longer build (docker/traefik/keycloak/python toolchain may all rebuild against the new nixpkgs).
|
||||||
|
That's expected; this is a real nixpkgs bump.
|
||||||
|
- If the build fails: read the error, fix on the PR branch (e.g. a renamed option, a removed
|
||||||
|
package), re-push, re-stage, re-build. Do NOT `switch` a failed build. Common breakages: NixOS
|
||||||
|
module option renames across channels (check the release notes), python package removals, a
|
||||||
|
sops-nix incompatibility (go back to step 3b and pick a compatible sops-nix rev).
|
||||||
|
|
||||||
|
**(d) Test-activate FIRST (general policy for nix deploys to this server AND the orchestrator
|
||||||
|
host): `nixos-rebuild test` before any `switch`:**
|
||||||
|
```
|
||||||
|
# detach it so a dropped SSH can't kill the activation mid-flight:
|
||||||
|
ssh cc-ci 'cd /root/cc-ci-deploy && setsid nohup nixos-rebuild test --flake .#cc-ci \
|
||||||
|
> /root/cc-ci-deploy/test-switch.log 2>&1 < /dev/null & echo launched'
|
||||||
|
# poll until reachable again, then check the essentials survived:
|
||||||
|
ssh cc-ci 'readlink /run/current-system; ip route show default; systemctl --failed --no-legend'
|
||||||
|
```
|
||||||
|
`test` activates the new configuration **without touching the bootloader or the system profile**,
|
||||||
|
so if the activation kills networking (as the 2026-08-03 26.05 bump did — no default route), a
|
||||||
|
plain reboot (Hetzner API power-cycle if SSH is gone) lands back on the last-known-good
|
||||||
|
generation. Only proceed to `switch` once the test activation is up, reachable, and healthy.
|
||||||
|
Note: the detached test leaves a transient `nixos-rebuild-switch-to-configuration` unit; if the
|
||||||
|
follow-up `switch` fails with "Unit ... was already loaded", wait for it to finish or
|
||||||
|
`systemctl stop nixos-rebuild-switch-to-configuration.service` first.
|
||||||
|
|
||||||
|
**(e) Switch (make it permanent — only after (d) verified):**
|
||||||
|
```
|
||||||
|
ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild switch --flake .#cc-ci' 2>&1 | tail -30
|
||||||
|
```
|
||||||
|
This is the moment the server moves to the new nixpkgs. The abra reconcile oneshots
|
||||||
|
(`deploy-proxy`, `warm-keycloak`, …) re-run as part of the switch — watch the tail for any that
|
||||||
|
fail. If a reconcile unit fails, do NOT roll back blindly — diagnose first (a warm-keycloak blip
|
||||||
|
may be transient; a proxy failure is real).
|
||||||
|
|
||||||
|
### 6. Verify the server is healthy (health gate — do not skip)
|
||||||
|
|
||||||
|
```
|
||||||
|
ssh cc-ci 'echo "=== version ==="; nixos-version; readlink /run/current-system; \
|
||||||
|
echo "=== failed units ==="; systemctl --failed --no-legend; \
|
||||||
|
echo "=== infra services ==="; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \
|
||||||
|
deploy-bridge deploy-reports deploy-backupbot drone-runner-exec | paste - - - - - - -; \
|
||||||
|
echo "=== docker stacks ==="; docker stack ls --format "{{.Name}} {{.Services}}"; \
|
||||||
|
echo "=== service replicas (any 0/N?) ==="; docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"; \
|
||||||
|
echo "=== endpoints ==="; curl -sS -o /dev/null -w "traefik:%{http_code}\n" https://traefik.ci.commoninternet.net/api/version; \
|
||||||
|
curl -sS -o /dev/null -w "dashboard:%{http_code}\n" https://ci.commoninternet.net/; \
|
||||||
|
echo "=== disk ==="; df -h / | tail -1'
|
||||||
|
```
|
||||||
|
Compare against the step-1 baseline:
|
||||||
|
- `nixos-version` moved (or the current-system path changed) — the bump landed.
|
||||||
|
- **No failed units** (the `--failed` output is empty).
|
||||||
|
- All 7 infra services `active` (or `active (exited)` for the oneshots).
|
||||||
|
- `docker service ls` shows no `0/N` replicas (everything converged).
|
||||||
|
- Endpoints `200` (traefik `/api/version`, dashboard root).
|
||||||
|
- Disk didn't spike (a nixpkgs bump adds a new system generation; old ones are GC'd eventually).
|
||||||
|
|
||||||
|
If **anything regressed**: diagnose. A real regression → roll back to the previous generation:
|
||||||
|
`ssh cc-ci 'nixos-rebuild switch --rollback'` (this switches to the pre-bump system path; the
|
||||||
|
flake input change on the PR branch is still your source of truth, but the running host reverts
|
||||||
|
while you fix). Report the rollback in the PR.
|
||||||
|
|
||||||
|
### 7. Merge the PR + report
|
||||||
|
|
||||||
|
On a **verified** run (health gate green): post a PR comment summarizing what was deployed (revs,
|
||||||
|
any incidents, verification results), then **merge the PR directly** — the skill invocation is the
|
||||||
|
authorization; the PR exists for visibility and historical record. On a rolled-back or failed run,
|
||||||
|
leave the PR **open** with a comment explaining the state.
|
||||||
|
|
||||||
|
Print + record a one-line result + the PR URL. Write a short note to
|
||||||
|
`/srv/cc-ci/.cc-ci-logs/server-update-<YYYY-MM-DD>.md`:
|
||||||
|
- `RESULT: SUCCESS — cc-ci server rebuilt on nixpkgs <new-rev[:8]>, no failed units, endpoints 200, PR merged: <url>`
|
||||||
|
- `RESULT: ROLLED-BACK — nixpkgs <new-rev> rebuild failed/ regressed (<reason>); host reverted to <old-rev>; PR <url> left open`
|
||||||
|
- `RESULT: FAILED — <step>: <reason>`
|
||||||
|
|
||||||
|
The end-of-run report to the operator MUST include **links to every merged PR** plus a summary of
|
||||||
|
what changed in each, so the operator can review them after the fact if they want to.
|
||||||
|
|
||||||
|
### 8. Optional cold-boot proof
|
||||||
|
|
||||||
|
A `nixos-rebuild switch` proves the new generation works on the running kernel. A **reboot** proves
|
||||||
|
it boots cleanly from cold — the stronger guarantee (cf. the pxgate cold-boot proof). Only do this
|
||||||
|
if the operator wants it, and only outside a CI window:
|
||||||
|
```
|
||||||
|
ssh cc-ci 'systemctl reboot'
|
||||||
|
# wait ~60s, re-check ssh cc-ci + the step-6 health gate
|
||||||
|
```
|
||||||
|
Record the new `boot_id` if you do this (it's a genuine reboot — log it to
|
||||||
|
`cc-ci-plan/REBOOTS.md`? No — that file is for the **orchestrator** host. The cc-ci server's reboots
|
||||||
|
are tracked in its own journal; a note in the server-update report is enough).
|
||||||
|
|
||||||
|
## Guardrails
|
||||||
|
|
||||||
|
- **Live rebuild is authorized by invocation.** The operator's choice to run this skill IS the
|
||||||
|
go-ahead for the `nixos-rebuild switch` — do not stop mid-run to re-confirm. A switch cycles
|
||||||
|
reconcile oneshots and can disrupt an in-flight `!testme`, so prefer a quiescent moment (the
|
||||||
|
step-1 check), but don't gate on a second authorization.
|
||||||
|
- **Single-writer:** dedicated branch in a separate clone; never push `main`; never touch the
|
||||||
|
build loops' `/cc-ci` / `/cc-ci-adv` clones or `/root/builder-clone` (stage to
|
||||||
|
`/root/cc-ci-deploy` instead).
|
||||||
|
- **Open the PR for visibility, merge it directly.** The live rebuild is the deploy; the PR is the
|
||||||
|
reviewable historical record. Merge only after the health gate passes; a rolled-back/failed run
|
||||||
|
leaves the PR open with an explanatory comment. Always report merged-PR links + change summaries.
|
||||||
|
- **Build before test, test before switch.** Never `nixos-rebuild switch` a build that didn't build
|
||||||
|
cleanly first, and never `switch` without a healthy `nixos-rebuild test` activation first — `test`
|
||||||
|
leaves the bootloader untouched, so a reboot always recovers to the last-known-good generation.
|
||||||
|
This is general policy for nix deploys to the cc-ci server **and** the orchestrator host.
|
||||||
|
- **Health gate is mandatory.** No "it switched, ship it" — run the step-6 verification and compare
|
||||||
|
to the baseline. Roll back on regression.
|
||||||
|
- **Stable channel only** for the cc-ci server (never `nixos-unstable`).
|
||||||
|
- **Keep sops-nix.follows** so the two inputs stay aligned; don't fork them.
|
||||||
|
- **Secrets never in git.** The operator-held `secrets/secrets.yaml` is copied on the host only.
|
||||||
|
- **Rollback is always available:** `nixos-rebuild switch --rollback` reverts to the prior
|
||||||
|
generation without touching the flake input revs.
|
||||||
|
|
||||||
|
## Relationship to an automatic update
|
||||||
|
|
||||||
|
This skill is the **manual trigger** for what should ideally run on a periodic timer: bump the cc-ci
|
||||||
|
flake inputs, build, switch, verify. If/when an automatic `cc-ci-server-update.timer` is added
|
||||||
|
(NixOS systemd timer on the orchestrator, analog of `cc-ci-upgrade-all.timer`), it should invoke
|
||||||
|
this skill via a dedicated one-shot agent (the way `cc-ci-upgrader` runs `/upgrade-all`). The
|
||||||
|
skill stays as the authoritative procedure either way — automatic just means "run on a schedule
|
||||||
|
without a human prompt."
|
||||||
|
|||||||
@@ -58,85 +58,28 @@ For each open PR: age (now − created_at), and whether the PR title/body/report
|
|||||||
- Verified-green PRs awaiting operator merge are normal — list them as "ready to merge", not
|
- Verified-green PRs awaiting operator merge are normal — list them as "ready to merge", not
|
||||||
as failures.
|
as failures.
|
||||||
|
|
||||||
### 5. The cc-ci host: update recency + the weekly auto-update
|
### 5. Server + orchestrator host update recency
|
||||||
|
|
||||||
Since 2026-09-07 the CI server and the orchestrator are ONE Hetzner host (`cc-ci`,
|
For BOTH hosts:
|
||||||
195.201.88.249, no tailnet), built from `recipe-maintainers/cc-ci-orchestrator`'s
|
```
|
||||||
`nixosConfigurations.cc-ci` (which imports cc-ci's `nixosModules.cc-ci-server`). From the
|
# cc-ci server: ssh cc-ci 'nixos-version; cd /root/cc-ci-deploy && nix flake metadata --json' (or builder-clone)
|
||||||
notplants-orchestrator box `ssh cc-ci` reaches it as root; on the host itself `ssh cc-ci` is loopback.
|
# orchestrator: nixos-version; cd /srv/cc-ci-orch && nix flake metadata --json
|
||||||
|
git ls-remote https://github.com/NixOS/nixpkgs <channel> # current tip
|
||||||
|
ls -t /srv/cc-ci-orch/.cc-ci-logs/server-update-*.md /srv/cc-ci-orch/.cc-ci-logs/orchestrator-update-*.md | head -2
|
||||||
|
```
|
||||||
|
- Report: days since last update log + how far the running nixpkgs rev lags the channel tip.
|
||||||
|
- **Lagging >30 days** (or a NixOS release behind) → recommend `/cc-ci-server-update` /
|
||||||
|
`/cc-ci-orchestrator-update`.
|
||||||
|
|
||||||
|
### 6. Host + service health (both machines)
|
||||||
|
|
||||||
```
|
```
|
||||||
ssh cc-ci 'nixos-version; readlink /run/current-system; readlink /nix/var/nix/profiles/system'
|
ssh cc-ci 'systemctl --failed --no-legend; df -h / | tail -1; docker service ls --format "{{.Name}} {{.Replicas}}"'
|
||||||
ssh cc-ci 'cd /srv/cc-ci-orch && nix flake metadata --json | jq -r ".locks.nodes | to_entries[] | \"\(.key) \(.value.locked.rev // \"-\") \(.value.locked.lastModified // \"-\")\""'
|
systemctl --failed --no-legend; df -h / | tail -1; tmux ls
|
||||||
git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05 | cut -c1-12 # channel tip
|
|
||||||
# the weekly self-update (nix/modules/auto-update.nix, Tue 03:00 UTC, busy-gated):
|
|
||||||
ssh cc-ci 'cat /srv/cc-ci/.cc-ci-logs/auto-update-state; systemctl list-timers --no-pager cc-ci-auto-update.timer | sed -n 2p; journalctl -u cc-ci-auto-update --no-pager -n 3'
|
|
||||||
```
|
```
|
||||||
- The state file is one line: `<UTC time> result=ok|unchanged|skipped|failed generation=<store path> note=<why>`.
|
- Failed units, core swarm services not 1/1 (warm-* spares flapping is a known benign pattern —
|
||||||
- `ok` / `unchanged` within the last 8 days → healthy.
|
note, don't page), disk >80% (server) / >85% (orchestrator) → findings. Server unreachable →
|
||||||
- `skipped` (busy) two weeks in a row, `failed`, or no state line / timer inactive for >8 days →
|
HIGH: recommend `hetzner-server-recovery`.
|
||||||
finding: *"host auto-update <state> since <date>: <note>"* → recommend
|
|
||||||
`/cc-ci-orchestrator-update` (it runs the same unit by hand and reads its journal).
|
|
||||||
- `failed` also means the host is running the previous generation and `flake.lock` was restored —
|
|
||||||
say so; the journal (`journalctl -u cc-ci-auto-update`) names the health check that failed.
|
|
||||||
- Running `/run/current-system` must equal the profile's boot generation (`system` link) — a
|
|
||||||
mismatch means a `test`-only activation was left behind (finding: switch or reboot decision).
|
|
||||||
- nixpkgs rev lagging the `nixos-26.05` channel tip by >30 days → finding → `/cc-ci-orchestrator-update`.
|
|
||||||
- `cc-ci` (the CI server input) lagging cc-ci `main` → note; the auto-update moves it weekly, or
|
|
||||||
`/cc-ci-orchestrator-update --cc-ci-only`.
|
|
||||||
|
|
||||||
### 6. Host + service health (one machine)
|
|
||||||
|
|
||||||
```
|
|
||||||
ssh cc-ci 'systemctl --failed --no-legend; df -h / | tail -1; free -m | sed -n 2p'
|
|
||||||
ssh cc-ci 'docker service ls --format "{{.Name}} {{.Replicas}}"; docker stack ls --format "{{.Name}}"'
|
|
||||||
ssh cc-ci 'for h in ci report.ci drone.ci oc.ci; do printf "%s " $h; curl -s -o /dev/null -w "%{http_code} ssl=%{ssl_verify_result}\n" https://$h.commoninternet.net/; done'
|
|
||||||
ssh cc-ci 'ls -s /run/secrets/test_secret; systemctl is-active acme-dns fail2ban nginx opencode-web drone-runner-exec; fail2ban-client status sshd | grep -E "Currently banned|Total banned"; fail2ban-client status nginx-http-auth | grep -E "Currently banned|Total banned"'
|
|
||||||
ssh cc-ci 'systemctl list-timers --no-pager cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer cc-ci-auto-update.timer nightly-sweep.timer | tail -n +2 | head -4'
|
|
||||||
ssh cc-ci 'sudo -iu loops tmux ls' # cc-ci-orchestrator must exist (the opencode orchestrator agent); cc-ci-upgrader only during a run
|
|
||||||
```
|
|
||||||
- Failed units, swarm services not at their replica count (`warm-*` spares flapping is benign —
|
|
||||||
note, don't page), a front door not 200 (`oc.ci` must be **401**: basic auth in front of the
|
|
||||||
opencode UI; 404 there = the `ccci-opencode-ui` stack is gone, `systemctl restart opencode-ui-route`
|
|
||||||
and check the weekly sweep's keep-list), `ssl≠0` (cert), `/run/secrets` empty (sops), acme-dns /
|
|
||||||
fail2ban / nginx / opencode-web / drone-runner-exec inactive, any timer missing, the orchestrator
|
|
||||||
tmux session missing (`systemctl restart cc-ci-orchestrator`) → findings. Disk **>65%** → finding.
|
|
||||||
Host unreachable → HIGH: `hetzner-server-recovery` (token in `/srv/cc-ci/.hcloud-token` on the
|
|
||||||
notplants box while the operator keeps it active).
|
|
||||||
> **65%, not 80% — it is not a steady-state measure.** Two concurrent recipe runs pull images and
|
|
||||||
> write volumes worth tens of GB, so a host sitting at 73% still hits 100% mid-run (2026-08-11:
|
|
||||||
> 63GB of unused images, `OSError: [Errno 28] No space left on device`). Remedy: `docker image
|
|
||||||
> prune -af` — it spares anything a container references. Do NOT `docker volume prune`: warm-*
|
|
||||||
> canonical volumes are data-warm and legitimately dangling.
|
|
||||||
- fail2ban: a non-zero "Currently banned" is information, not a finding (that is the jail
|
|
||||||
working); the notplants box's own address showing up banned IS a finding (it means a skill
|
|
||||||
misbehaved) — `fail2ban-client set <jail> unbanip <ip>`.
|
|
||||||
- Secrets hygiene: `ssh cc-ci 'ls /secrets/files'` must list ONLY cc-ci.testenv, cc-ci-local-ed25519(.pub),
|
|
||||||
autonomic-bot-cc-ci-ed25519(.pub), opencode-auth.json. Anything else there (a tailscale key, a
|
|
||||||
master age key, another host's ssh key…) → finding: *"secret sprawl on the cc-ci host"*.
|
|
||||||
- **!testme actually produces results** (the check that would have caught the above days earlier):
|
|
||||||
the newest few `/var/lib/cc-ci-runs/<build>/` dirs must each contain `results.json`. A build that
|
|
||||||
dies before the harness writes one leaves an EMPTY dir — and the PR badge still says "failure", so
|
|
||||||
it reads as a recipe regression rather than a sick host. Builds 1236–1242 all failed that way.
|
|
||||||
Finding: *"N recent builds produced no results.json — the harness is dying at startup, check disk
|
|
||||||
and the drone step log"*. The step log lives in drone's sqlite
|
|
||||||
(`/var/lib/docker/volumes/drone_ci_commoninternet_net_data/_data/database.sqlite`) — copy it and
|
|
||||||
read `logs.log_data` for the failing `steps.step_id`; the bridge's drone token is not extractable
|
|
||||||
(distroless container, swarm secret).
|
|
||||||
> **If the error is ENOSPC but the disk is fine**, it is not disk. Seen 2026-08-11: builds 1244-1249
|
|
||||||
> died on `mkdir /var/lib/cc-ci-runs/<build>` with **110GB free and 16% inodes**, while the identical
|
|
||||||
> mkdir succeeded as root over ssh, inside the runner's own mount namespace, and 61/61 times in a
|
|
||||||
> stress loop — and the same harness run by hand with a numeric run id worked fine. Restarting
|
|
||||||
> `drone-runner-exec` did NOT help, and neither did recreating the runs directory with a fresh
|
|
||||||
> inode (it recurred afterwards — that apparent fix was coincidence).
|
|
||||||
>
|
|
||||||
> **It is INTERMITTENT and tracks concurrent activity**, which is the useful signal: every failure
|
|
||||||
> landed while a second run or a manual deploy was in flight (1252 was triggered while 1251 was
|
|
||||||
> still finishing), and every build on a quiet host succeeded (1243, 1250, 1251, 1253). Free space
|
|
||||||
> never moved during a failing build. So on ENOSPC-with-free-disk: **wait for the host to go quiet
|
|
||||||
> and re-trigger** before treating it as a recipe failure. Root cause is still NOT established;
|
|
||||||
> `DRONE_RUNNER_CAPACITY=2` allows the overlap, so lowering it to 1 is the obvious next experiment
|
|
||||||
> if it becomes disruptive.
|
|
||||||
- **Bridge / !testme path**: `docker service ls` shows `ccci-bridge_app 1/1` AND the bridge log
|
- **Bridge / !testme path**: `docker service ls` shows `ccci-bridge_app 1/1` AND the bridge log
|
||||||
has no auth errors (`docker service logs --since 24h ccci-bridge_app 2>&1 | grep -ci "401\|user does not exist"` == 0).
|
has no auth errors (`docker service logs --since 24h ccci-bridge_app 2>&1 | grep -ci "401\|user does not exist"` == 0).
|
||||||
A silently-401ing bridge drops `!testme` (seen 2026-08-03, stale rotated Gitea secret) →
|
A silently-401ing bridge drops `!testme` (seen 2026-08-03, stale rotated Gitea secret) →
|
||||||
@@ -170,17 +113,13 @@ minutes, no PRs). If it is instead that a known CVE is sitting unpatched, recomm
|
|||||||
`/cve-check` over waiting for the next weekly run whenever the question is "are we exposed?".
|
`/cve-check` over waiting for the next weekly run whenever the question is "are we exposed?".
|
||||||
|
|
||||||
`ALL HEALTHY` requires: recent successful weekly run + published report, no stale tests, no
|
`ALL HEALTHY` requires: recent successful weekly run + published report, no stale tests, no
|
||||||
CVE PR open >14 days, the host <30 days behind its channel with the weekly auto-update `ok`/`unchanged`
|
CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, disk under
|
||||||
within 8 days and running generation == boot generation, zero failed units, every front door
|
thresholds, bridge clean, maintained-set consistent. Anything else is a finding — even minor
|
||||||
answering (oc.ci with 401), sops decrypted, fail2ban up, the orchestrator agent session present,
|
|
||||||
recent builds all producing results.json, disk under threshold, bridge clean, maintained-set
|
|
||||||
consistent, secrets inventory exact. Anything else is a finding — even minor
|
|
||||||
ones get a recommended next step. Order findings by priority (CVE/unreachable-host first).
|
ones get a recommended next step. Order findings by priority (CVE/unreachable-host first).
|
||||||
|
|
||||||
## Guardrails
|
## Guardrails
|
||||||
|
|
||||||
- **Read-only.** This skill diagnoses and recommends; it invokes nothing and changes nothing.
|
- **Read-only.** This skill diagnoses and recommends; it invokes nothing and changes nothing.
|
||||||
- Use the pseudo-TTY wrap for any abra call; plain ssh for everything else. `ssh cc-ci` from the
|
- Use the pseudo-TTY wrap for any abra call; plain ssh for everything else.
|
||||||
notplants box is root on 195.201.88.249 (loops ssh config); there is no tailnet.
|
|
||||||
- Don't double-count: a finding that explains another (bridge 401 → !testme "failures") gets
|
- Don't double-count: a finding that explains another (bridge 401 → !testme "failures") gets
|
||||||
reported once, at the root cause.
|
reported once, at the root cause.
|
||||||
|
|||||||
@@ -83,29 +83,8 @@ failure (AI — this is the `ci-test-review` step-3 diagnosis):
|
|||||||
changed upstream, what the test currently asserts.
|
changed upstream, what the test currently asserts.
|
||||||
- **FLAKY** → re-run once or twice; if it passes, drop it (not stale, just flaky).
|
- **FLAKY** → re-run once or twice; if it passes, drop it (not stale, just flaky).
|
||||||
|
|
||||||
> ### ⚠️ RECONCILE FROM UPSTREAM FIRST — always, before any PR work or upgrade check
|
|
||||||
> ```
|
|
||||||
> cc-ci-plan/reconcile-upstream.sh <recipe>... # or --all
|
|
||||||
> ```
|
|
||||||
> Deterministic, idempotent, and safe (recipe work lives in branches, never on mirror `main`). It
|
|
||||||
> force-syncs each mirror to coopcloud's **default branch — resolved from the API, `main` OR
|
|
||||||
> `master`** — and closes any mirror PR whose changes upstream already merged. Skipping it has cost
|
|
||||||
> us three distinct ways: mailu #6 was reported as the fix for two internet-facing CVEs while
|
|
||||||
> upstream had already merged AND released it; a stale mirror makes a survey report "no upgrades
|
|
||||||
> available" so the recipe drops out of the weekly run; and reading the wrong branch on a recipe with
|
|
||||||
> a stale `main` beside a live `master` (gitea) manufactures a false "three releases behind, missing
|
|
||||||
> two CVSS-9.8 RCEs" finding.
|
|
||||||
|
|
||||||
### 2. For each stale test — author the minimal test update (AI; never weaken)
|
### 2. For each stale test — author the minimal test update (AI; never weaken)
|
||||||
|
|
||||||
> **Read `tests/STYLE.md` in the cc-ci repo before writing the update.** It is the rulebook for test
|
|
||||||
> changes, written from failures this pipeline actually produced. Most load-bearing: set state up
|
|
||||||
> through the app's **own interface, never its database** (a plausible fixture that INSERTed rows
|
|
||||||
> passed on v2 and silently broke on v3 — 202 acks, rows in postgres, nothing ingested — and held the
|
|
||||||
> recipe RED for six weeks), **gate on version rather than supporting both** (old-version tests can be
|
|
||||||
> deleted; the older version is only exercised via the upgrade tier), and correct the fixture or the
|
|
||||||
> wait but **never the assertion**.
|
|
||||||
|
|
||||||
Work on **one recipe at a time** (serialize — each verification deploys a recipe on the shared
|
Work on **one recipe at a time** (serialize — each verification deploys a recipe on the shared
|
||||||
Swarm). For each `STALE_TESTS` entry:
|
Swarm). For each `STALE_TESTS` entry:
|
||||||
|
|
||||||
|
|||||||
@@ -1,11 +1,12 @@
|
|||||||
---
|
---
|
||||||
name: cc-ci-update
|
name: cc-ci-update
|
||||||
description: Run both cc-ci maintenance updates in one pass — first /cc-ci-orchestrator-update (the cc-ci host: nixpkgs, sops-nix and the cc-ci input, test → health check → switch, lock committed to main), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update.
|
description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update.
|
||||||
---
|
---
|
||||||
|
|
||||||
# cc-ci-update
|
# cc-ci-update
|
||||||
|
|
||||||
The combined **keep cc-ci itself current** pass: run `/cc-ci-orchestrator-update` (host dependency bump + health-checked rebuild) **then** `/cc-ci-tests-update` (stale-test sweep + fixes), in that order, end-to-end. This
|
The combined **keep cc-ci itself current** pass: run `/cc-ci-server-update` (host dependency bump +
|
||||||
|
rebuild) **then** `/cc-ci-tests-update` (stale-test sweep + fixes), in that order, end-to-end. This
|
||||||
is just sequencing the two skills — it doesn't add or change any behavior. Read each sub-skill's
|
is just sequencing the two skills — it doesn't add or change any behavior. Read each sub-skill's
|
||||||
SKILL.md for the full procedure; this file is the orchestration + the why-of-the-order.
|
SKILL.md for the full procedure; this file is the orchestration + the why-of-the-order.
|
||||||
|
|
||||||
@@ -20,7 +21,7 @@ SKILL.md for the full procedure; this file is the orchestration + the why-of-the
|
|||||||
## Arguments
|
## Arguments
|
||||||
|
|
||||||
None. (If you only want one of the two updates, invoke the sub-skill directly:
|
None. (If you only want one of the two updates, invoke the sub-skill directly:
|
||||||
`/cc-ci-orchestrator-update` or `/cc-ci-tests-update`.)
|
`/cc-ci-server-update` or `/cc-ci-tests-update`.)
|
||||||
|
|
||||||
## Preconditions
|
## Preconditions
|
||||||
|
|
||||||
@@ -33,12 +34,12 @@ Both sub-skills' preconditions hold:
|
|||||||
|
|
||||||
## Procedure
|
## Procedure
|
||||||
|
|
||||||
### 1. Run `/cc-ci-orchestrator-update` (host first)
|
### 1. Run `/cc-ci-server-update` (host first)
|
||||||
|
|
||||||
Run the full `/cc-ci-orchestrator-update` procedure: snapshot baseline → bump flake inputs (nixpkgs +
|
Run the full `/cc-ci-server-update` procedure: snapshot baseline → bump flake inputs (nixpkgs +
|
||||||
sops-nix) → commit + open cc-ci PR → stage on host → build → `nixos-rebuild switch` → **health
|
sops-nix) → commit + open cc-ci PR → stage on host → build → `nixos-rebuild switch` → **health
|
||||||
gate** (no failed units, infra up, endpoints 200, disk free). See
|
gate** (no failed units, infra up, endpoints 200, disk free). See
|
||||||
`.claude/skills/cc-ci-orchestrator-update/SKILL.md`.
|
`.claude/skills/cc-ci-server-update/SKILL.md`.
|
||||||
|
|
||||||
**Gate on the result before continuing:**
|
**Gate on the result before continuing:**
|
||||||
- `SUCCESS` → the server is on the new nixpkgs, healthy. Proceed to step 2.
|
- `SUCCESS` → the server is on the new nixpkgs, healthy. Proceed to step 2.
|
||||||
@@ -89,7 +90,7 @@ operator review.
|
|||||||
Inherited from both sub-skills (read their Guardrails sections). The combined-specific ones:
|
Inherited from both sub-skills (read their Guardrails sections). The combined-specific ones:
|
||||||
- **Server-first, gated.** Don't run the tests sweep on a sick/rolled-back host. The server-update
|
- **Server-first, gated.** Don't run the tests sweep on a sick/rolled-back host. The server-update
|
||||||
health gate is a hard go/no-go for step 2.
|
health gate is a hard go/no-go for step 2.
|
||||||
- **No-CI window for the rebuild** (the server half) — same as `/cc-ci-orchestrator-update`.
|
- **No-CI window for the rebuild** (the server half) — same as `/cc-ci-server-update`.
|
||||||
- **Serialize the tests half on the shared Swarm** — same as `/cc-ci-tests-update`; don't run it
|
- **Serialize the tests half on the shared Swarm** — same as `/cc-ci-tests-update`; don't run it
|
||||||
concurrent with `/upgrade-all` or active build loops.
|
concurrent with `/upgrade-all` or active build loops.
|
||||||
- **Never merge anything** — server PR, test PRs, paired recipe upgrade PRs all operator-merged.
|
- **Never merge anything** — server PR, test PRs, paired recipe upgrade PRs all operator-merged.
|
||||||
|
|||||||
@@ -31,12 +31,6 @@ Then present the roster grouped as follows, and close with the situation guide.
|
|||||||
PR). `--with-tests` also fixes that recipe's stale test.
|
PR). `--with-tests` also fixes that recipe's stale test.
|
||||||
- **/recipe-report** — (re)generate the weekly report page for report.ci.commoninternet.net.
|
- **/recipe-report** — (re)generate the weekly report page for report.ci.commoninternet.net.
|
||||||
|
|
||||||
**Keeping the PR list honest**
|
|
||||||
- **/cc-ci-cleanup** — reconciles every mirror from true upstream (which alone closes PRs upstream
|
|
||||||
already merged), then closes the open recipe PRs that can no longer merge or were never meant to
|
|
||||||
(CI sweep artifacts, obsolete bumps, superseded duplicates) and reports what is actually blocking
|
|
||||||
the ones that should land. Never merges.
|
|
||||||
|
|
||||||
**Security (CVEs)**
|
**Security (CVEs)**
|
||||||
- **/cve-check** — fleet-wide CVE sweep with **no upgrading**: for every recipe, work out what
|
- **/cve-check** — fleet-wide CVE sweep with **no upgrading**: for every recipe, work out what
|
||||||
upgrade is available (per image, sidecars included), scan it for CVEs, and publish a CVE report.
|
upgrade is available (per image, sidecars included), scan it for CVEs, and publish a CVE report.
|
||||||
@@ -87,7 +81,6 @@ ARM skills never touch cc-ci infra. After a submodule bump run `scripts/gen-ccte
|
|||||||
| "Run the weekly upgrades now" | `/upgrade-all` (or `systemctl start cc-ci-upgrade-all.service`) |
|
| "Run the weekly upgrades now" | `/upgrade-all` (or `systemctl start cc-ci-upgrade-all.service`) |
|
||||||
| "Upgrade just <recipe>" | `/recipe-upgrade <recipe>` |
|
| "Upgrade just <recipe>" | `/recipe-upgrade <recipe>` |
|
||||||
| "The report site is stale/missing a week" | `/recipe-report` |
|
| "The report site is stale/missing a week" | `/recipe-report` |
|
||||||
| "The open PR list is a mess / what should I merge?" | `/cc-ci-cleanup` |
|
|
||||||
| "What CVEs are we exposed to right now?" | `/cve-check` (read-only, no PRs) |
|
| "What CVEs are we exposed to right now?" | `/cve-check` (read-only, no PRs) |
|
||||||
| "A CVE just dropped — check and patch it" | `/cve-check-and-upgrade` (add `--min-severity high` to skip the noise) |
|
| "A CVE just dropped — check and patch it" | `/cve-check-and-upgrade` (add `--min-severity high` to skip the noise) |
|
||||||
| "Is <recipe> vulnerable?" | `/cve-check <recipe>` |
|
| "Is <recipe> vulnerable?" | `/cve-check <recipe>` |
|
||||||
|
|||||||
@@ -30,19 +30,15 @@ the orchestrator watches from outside.
|
|||||||
|
|
||||||
Reboot resilience is handled by **`cc-ci-loops.service`** (system unit): on boot it logs the reboot
|
Reboot resilience is handled by **`cc-ci-loops.service`** (system unit): on boot it logs the reboot
|
||||||
to `REBOOTS.md` (boot_id-gated) and runs `launch.sh start` with `RESUME_PHASE=1`, so the loops +
|
to `REBOOTS.md` (boot_id-gated) and runs `launch.sh start` with `RESUME_PHASE=1`, so the loops +
|
||||||
watchdog auto-resume the saved phase. The orchestrator session itself is relaunched by
|
watchdog auto-resume the saved phase. The orchestrator session itself is NOT auto-started — the
|
||||||
`cc-ci-orchestrator.service` (`agents.py up orchestrator`) as an **opencode agent** the operator
|
operator reconnects to it (that's why the startup notification matters). The orchestrator now runs on
|
||||||
steers from https://oc.ci.commoninternet.net (no Claude on the cc-ci host; Claude sessions run on
|
a **Hetzner `cpx22`** cloud server (`cc-ci-orchestrator-1`, tailnet `100.84.190.30`, public
|
||||||
the notplants-orchestrator box and reach cc-ci over ssh). Since 2026-09 the orchestrator runs on the **same Hetzner
|
`168.119.126.100`, flake host `cc-ci-orchestrator-hetzner`) — see
|
||||||
host as the cc-ci CI server** (`cc-ci`, public `195.201.88.249`, tailnet `cc-ci`), declared by
|
`cc-ci-plan/plan-orchestrator-hetzner-migration.md`. The earlier Pi→Incus-VM move is the historical
|
||||||
`nixosConfigurations.cc-ci` in this repo's `flake.nix`, which imports the CI server from the cc-ci
|
`cc-ci-plan/plan-orchestrator-migration.md`. Rebuild this host with
|
||||||
repo's `nixosModules.cc-ci-server`. `ssh cc-ci` from the loops user therefore goes to loopback.
|
`nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner` from `/srv/cc-ci-orch` — but **always
|
||||||
The full provisioning + deploy guide is `README.md`; the move is recorded in
|
|
||||||
`cc-ci-plan/plan-cc-ci-combined-host.md`; the previous hosts (Pi → Incus VM → Hetzner `cpx22`
|
|
||||||
shared with notplants) are in `archive/`. Rebuild this host with
|
|
||||||
`nixos-rebuild switch --flake .#cc-ci` from `/srv/cc-ci-orch` — but **always
|
|
||||||
`nixos-rebuild test` the same flake target first and verify the host is still healthy/reachable
|
`nixos-rebuild test` the same flake target first and verify the host is still healthy/reachable
|
||||||
before the `switch`** (general policy for nix deploys to this host: `test`
|
before the `switch`** (general policy for nix deploys to this host and the cc-ci server: `test`
|
||||||
leaves the bootloader and system profile untouched, so a reboot always recovers to the
|
leaves the bootloader and system profile untouched, so a reboot always recovers to the
|
||||||
last-known-good generation; the 2026-08-03 cc-ci 26.05 bump outage is the cautionary tale, see
|
last-known-good generation; the 2026-08-03 cc-ci 26.05 bump outage is the cautionary tale, see
|
||||||
`.cc-ci-logs/server-update-2026-08-03.md`).
|
`.cc-ci-logs/server-update-2026-08-03.md`).
|
||||||
@@ -119,29 +115,3 @@ When the orchestrator, Builder, or assistant makes intentional repository change
|
|||||||
promptly and push them to `git.autonomic.zone` in append-only fashion (never force-push). Match the
|
promptly and push them to `git.autonomic.zone` in append-only fashion (never force-push). Match the
|
||||||
existing commit author and message style in this repo. Do not bundle unrelated worktree changes you
|
existing commit author and message style in this repo. Do not bundle unrelated worktree changes you
|
||||||
did not make; stage only the intended files.
|
did not make; stage only the intended files.
|
||||||
|
|
||||||
## Ship as PRs, merge them yourself, operator reviews retrospectively
|
|
||||||
|
|
||||||
**This applies to the two INFRASTRUCTURE repos — `recipe-maintainers/cc-ci-orchestrator` (here) and
|
|
||||||
`recipe-maintainers/cc-ci` (the CI product).** For work in either:
|
|
||||||
|
|
||||||
1. Branch, don't commit straight to `main`.
|
|
||||||
2. Open a PR with a description written to be read **after** the fact: what changed, why, and what
|
|
||||||
evidence says it works (test output, a verified run, a before/after number). The PR *is* the
|
|
||||||
review artifact and the historical record.
|
|
||||||
3. **Merge it yourself once it is verified** — do not wait for review. The invocation is the
|
|
||||||
authorization; blocking on review would stall the pipeline these repos exist to run.
|
|
||||||
4. The operator reviews **retrospectively**, from the PR.
|
|
||||||
|
|
||||||
So the PR is not a gate — it is how the work stays legible. A PR that merely says "fix scanner" has
|
|
||||||
failed at its only job.
|
|
||||||
|
|
||||||
> ### This does NOT extend to RECIPE repos
|
|
||||||
> Recipe PRs — any `coop-cloud/<recipe>` or its `recipe-maintainers/<recipe>` mirror — are
|
|
||||||
> **created and verified but NEVER merged by an agent**. Those change what deploys on other people's
|
|
||||||
> infrastructure, so a human merges them. The split is deliberate: agents own the tooling, the
|
|
||||||
> operator owns the recipes.
|
|
||||||
|
|
||||||
If work has already landed on `main` without a PR, do not rewrite published history to fix it.
|
|
||||||
Create a branch pinned at the pre-work commit and open the PR against that, so the diff is still
|
|
||||||
reviewable and merging only advances the pointer (see PRs #2-#5, 2026-08-11).
|
|
||||||
|
|||||||
@@ -1,414 +1,57 @@
|
|||||||
# cc-ci-orchestrator
|
# cc-ci-orchestrator
|
||||||
|
|
||||||
The **cc-ci orchestrator**: the agent loops that built the cc-ci Co-op Cloud recipe CI server,
|
Orchestrator workspace for building the **cc-ci** Co-op Cloud recipe CI server. The plan, launch
|
||||||
the operator's steering session, and the weekly autonomous recipe-upgrade run — plus the NixOS
|
tooling, and loop prompts live in [`cc-ci-plan/`](cc-ci-plan/); see [`AGENTS.md`](AGENTS.md) for the
|
||||||
host they run on. Since 2026-09 that host is **the same Hetzner server as the CI server itself**:
|
roles and operating model. Secrets (`.testenv`) are gitignored — never commit them.
|
||||||
one `nixos-rebuild` from this repo builds both, because this flake imports the CI server as a
|
|
||||||
module from the [cc-ci](https://git.autonomic.zone/recipe-maintainers/cc-ci) repo.
|
|
||||||
|
|
||||||
| | where |
|
## Run the orchestrator in tmux (survives disconnects + closing your laptop)
|
||||||
|---|---|
|
|
||||||
| Orchestrator loops, timers (weekly upgrader, hourly supervisor) | `nix/modules/cc-ci.nix` → `nixosModules.cc-ci-orchestrator` |
|
|
||||||
| The host contract those need (loops user, opencode CLI, opencode web UI) | `nix/modules/orchestrator-host.nix` → `nixosModules.orchestrator-host` |
|
|
||||||
| The CI server (swarm, traefik, drone, runner, `!testme` bridge, dashboard, reports, acme-dns) | cc-ci repo `nix/modules/` → `nixosModules.cc-ci-server` (flake input `cc-ci`) |
|
|
||||||
| The machine: hardware, networking, root keys, firewall + fail2ban | `nix/hosts/cc-ci/` → `nixosConfigurations.cc-ci` |
|
|
||||||
| Plans, launch tooling, loop prompts, journal | `cc-ci-plan/` (see `AGENTS.md` for roles) |
|
|
||||||
| Skills the orchestrator runs (`/upgrade-all`, `/recipe-upgrade`, `/cc-ci-status`, …) | `.claude/skills/`, `.opencode/skills/` |
|
|
||||||
| How it used to be built (Pi → Incus VM → shared Hetzner box) | `archive/` |
|
|
||||||
|
|
||||||
Secrets (`.testenv`, `upgrader.env`, `.sops/`, everything under `/secrets`) are gitignored — never
|
Keep this supervising session alive on the host with tmux, and use `--remote-control` so you can
|
||||||
commit them.
|
watch/steer it from **claude.ai/code** (or the mobile app).
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
# Deploying a cc-ci host from scratch
|
|
||||||
|
|
||||||
This is the whole path from "nothing" to a working CI server + orchestrator on one Hetzner
|
|
||||||
server. It was last done on 2026-09-07 for `195.201.88.249` and is written so a person or an LLM
|
|
||||||
can repeat it. Read it once before starting; the order matters.
|
|
||||||
|
|
||||||
## 0. What you need in hand
|
|
||||||
|
|
||||||
- A **Hetzner Cloud** project you can create servers in (console login or an API token).
|
|
||||||
- **SSH keys**: yours, and the orchestrator's own key so the automation can reach the box. The
|
|
||||||
public keys that get root are tracked in `nix/hosts/cc-ci/ssh-keys` (one per line). Password
|
|
||||||
login is disabled and fail2ban watches sshd, so a key is the only way in.
|
|
||||||
- Read access to `recipe-maintainers/cc-ci`, `recipe-maintainers/cc-ci-orchestrator` (both public
|
|
||||||
read) and the **private** `recipe-maintainers/cc-ci-secrets` (the `autonomic-bot` account has
|
|
||||||
it; the host gets its own deploy key on that account, §4b/§4c — and the bot password, in
|
|
||||||
`.testenv`, is what registers it).
|
|
||||||
- The out-of-band secrets listed in §4. If you are migrating, they come from the old host; if
|
|
||||||
you are starting fresh you create them (each row says how).
|
|
||||||
- Control of the DNS zone (Gandi for `commoninternet.net`) for the cutover in §7.
|
|
||||||
|
|
||||||
## 1. Provision the server on Hetzner (Debian image)
|
|
||||||
|
|
||||||
In the Hetzner Cloud console (or with `hcloud server create`):
|
|
||||||
|
|
||||||
| setting | value | why |
|
|
||||||
|---|---|---|
|
|
||||||
| Image | **Debian 13** (any recent Debian/Ubuntu works with nixos-infect) | it is replaced by NixOS in §2 |
|
|
||||||
| Type | **x86**, **8 GB RAM**, 4 vCPU — e.g. `cpx32` (dedicated AMD) or `cx33`. **Never `cax*`** (ARM): the flakes are `x86_64-linux`. | swarm + recipe deploys + 3–6 agent sessions; 4 GB is too small |
|
|
||||||
| Disk | the type's default 150+ GB NVMe | docker layers alone are ~60 GB after a few weeks |
|
|
||||||
| Network | public **IPv4** required; IPv6 optional (leave enabled or not, NixOS config ignores it) | cc-ci serves 80/443 and DNS on 53 publicly |
|
|
||||||
| SSH keys | add every key from `nix/hosts/cc-ci/ssh-keys` you want to log in with, at least the orchestrator's | nixos-infect carries `/root/.ssh/authorized_keys` over |
|
|
||||||
| Name | `cc-ci` | becomes the hostname |
|
|
||||||
| Firewall | if a Hetzner Cloud Firewall is attached it must allow **22/tcp, 80/tcp, 443/tcp, 53/tcp, 53/udp** in, and ICMP | the NixOS firewall is separate and is configured by the flake |
|
|
||||||
|
|
||||||
Check you can log in: `ssh root@<ip> hostname`.
|
|
||||||
|
|
||||||
## 2. Convert Debian → NixOS with nixos-infect
|
|
||||||
|
|
||||||
[nixos-infect](https://github.com/elitak/nixos-infect) installs NixOS over the running Debian and
|
|
||||||
reboots. Run it detached so the SSH session dropping does not kill it:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh root@<ip> 'cat > /root/infect.sh <<"EOF"
|
# 0. Exit any running orchestrator session first — a conversation can't be resumed while it's live:
|
||||||
#!/usr/bin/env bash
|
# /exit (inside Claude) or Ctrl-D
|
||||||
set -euo pipefail
|
|
||||||
# Pinned nixos-infect revision (same one that built the previous cc-ci hosts).
|
# 1. Start a detachable tmux session on this host
|
||||||
INFECT_SHA="40f62a680bb0e8f2f607d79abfaaecd99d59401c"
|
tmux new -s orchestrator
|
||||||
export NIX_CHANNEL="nixos-26.05" # must match the nixpkgs channel in flake.nix
|
|
||||||
export PROVIDER="hetznercloud" # GRUB + Hetzner networking
|
# 2. Inside tmux, resume the orchestrator conversation WITH remote control:
|
||||||
export NIXOS_IMPORT="" # the real config comes from the flake in §5
|
claude --resume autonomous-orchestrator \
|
||||||
# The Debian 13 cloud image mounts /tmp as tmpfs; nixos-infect makes a temporary swapfile
|
--remote-control "autonomous-orchestrator" \
|
||||||
# there and swapon fails with "Invalid argument". 8 GB RAM needs no extra swap: skip it.
|
--dangerously-skip-permissions
|
||||||
export NO_SWAP=true
|
# - If name-resume opens a picker instead of resuming directly, choose "autonomous-orchestrator".
|
||||||
curl -fsSL "https://raw.githubusercontent.com/elitak/nixos-infect/${INFECT_SHA}/nixos-infect" | bash -x
|
# - Or resume by the stable session id (more deterministic in a fresh pane):
|
||||||
EOF
|
# claude --resume 34a80a99-b37e-4809-b8da-ccc9fafe785e \
|
||||||
chmod +x /root/infect.sh
|
# --remote-control "autonomous-orchestrator" --dangerously-skip-permissions
|
||||||
nohup /root/infect.sh > /var/log/nixos-infect.log 2>&1 &'
|
|
||||||
|
# 3. Detach — the process keeps running: press Ctrl-b, then d
|
||||||
```
|
```
|
||||||
|
|
||||||
It downloads Nix, builds a NixOS system (5–10 min; follow with
|
**Reconnect later**
|
||||||
`ssh root@<ip> tail -f /var/log/nixos-infect.log`), then reboots. The SSH host key changes:
|
- On this host: `tmux attach -t orchestrator`
|
||||||
`ssh-keygen -R <ip>` and confirm `ssh root@<ip> nixos-version` prints a 26.05 version.
|
- From anywhere: **claude.ai/code** → the `autonomous-orchestrator` session
|
||||||
|
|
||||||
### 2a. What went wrong on 2026-09-07, and the fixes (Debian 13 image, NixOS 26.05)
|
**Why it survives:** tmux keeps the `claude` process alive across SSH disconnects and your laptop
|
||||||
|
closing; remote-control runs *outbound* from this host to Anthropic, so it stays connected
|
||||||
|
regardless of the viewer. After a host reboot, re-run steps 1–2.
|
||||||
|
|
||||||
All three bit on the first attempt; the script above and §3 already include the fixes, this is
|
> Two different "names": `--resume <name|id>` selects the *conversation* to restore (shown in the
|
||||||
so you recognise them if they come back in another form.
|
> `/resume` picker); the `--remote-control "<name>"` value is only the web display label and resumes
|
||||||
|
> nothing. Resuming reuses the same session id each time (stays `34a8…`) — don't pass
|
||||||
1. **`swapon: /tmp/nixos-infect.XXXX.swp: Invalid argument`** right at the start, script exits.
|
> `--fork-session` unless you intend to branch a new conversation.
|
||||||
The Debian 13 cloud image mounts `/tmp` as tmpfs and a swapfile cannot live there.
|
>
|
||||||
Fix: `NO_SWAP=true` (in the script above). An 8 GB box does not need the temporary swap.
|
> Already inside a live session and just want the web surface? Run `/remote-control` — no exit/resume.
|
||||||
2. **The box never comes back after the reboot: it boots NixOS, but nearly every unit fails**
|
|
||||||
(`dbus`, `systemd-logind`, `sshd`, networking …) with
|
|
||||||
`Could not start dynamically linked executable: /usr/bin/dbus-daemon` in the journal.
|
|
||||||
nixos-infect leaves the old Debian root in place and relies on NixOS's first boot to move it
|
|
||||||
to `/old-root` (`/etc/NIXOS_LUSTRATE`). With NixOS 26.05's systemd-based initrd that
|
|
||||||
lustration did not happen, so Debian's `/etc/systemd/system/*.service` files shadowed the
|
|
||||||
NixOS units and started Debian binaries. Fix, from Hetzner **rescue mode**
|
|
||||||
(`enable_rescue` + `reset` in the API/console, ssh in, `mount /dev/sda1 /mnt/root`):
|
|
||||||
move everything except `nix`, `boot`, `swapfile`, `lost+found`, `var/log`, `var/empty`,
|
|
||||||
`etc/nixos`, `etc/resolv.conf`, `etc/NIXOS`, `etc/machine-id`, `etc/ssh/ssh_host_*`,
|
|
||||||
`root/.nix-*`, `root/.ssh` into `/mnt/root/old-root`, delete `etc/NIXOS_LUSTRATE`, unmount,
|
|
||||||
`disable_rescue`, `reset`. (`/old-root`, ~1 GB, can be deleted once the host is in service.)
|
|
||||||
3. **Boots, units fine, but no network.** The generated `networking.nix` has
|
|
||||||
`defaultGateway = "172.31.1.1";` — a bare string. Since NixOS 25.05 that yields no default
|
|
||||||
route. Fix: `defaultGateway = { address = "172.31.1.1"; interface = "eth0"; };` (this is what
|
|
||||||
`nix/hosts/cc-ci/networking.nix` carries). To apply it from rescue mode, chroot into the
|
|
||||||
mounted root and rebuild the boot entry — the nix sandbox cannot `pivot_root` inside a chroot,
|
|
||||||
so turn it off for that one build:
|
|
||||||
```bash
|
|
||||||
for d in proc sys dev dev/pts; do mount --bind /$d /mnt/root/$d; done
|
|
||||||
mount -t tmpfs tmpfs /mnt/root/run; cp -L /etc/resolv.conf /mnt/root/etc/resolv.conf
|
|
||||||
chroot /mnt/root /nix/var/nix/profiles/system/sw/bin/bash -c '
|
|
||||||
export PATH=/nix/var/nix/profiles/system/sw/bin NIX_REMOTE= HOME=/root
|
|
||||||
export NIX_PATH=nixos-config=/etc/nixos/configuration.nix:nixpkgs=/root/.nix-defexpr/channels/nixos
|
|
||||||
ln -sfn /nix/var/nix/profiles/system /run/current-system
|
|
||||||
nixos-rebuild boot --option sandbox false'
|
|
||||||
```
|
|
||||||
The `journalctl -D /mnt/root/var/log/journal -b 0` trick (reading the dead system's journal
|
|
||||||
from rescue mode) is what told these apart.
|
|
||||||
|
|
||||||
> Rescue mode without a console: `POST /servers/<id>/actions/enable_rescue` with your ssh key
|
|
||||||
> id, then `…/actions/reset`; afterwards `disable_rescue` **and check `rescue_enabled` is false
|
|
||||||
> before** the next `reset`, or it boots the rescue image again. `scripts/recovery/hetzner.py`
|
|
||||||
> wraps these (token in `/srv/cc-ci/.hcloud-token`).
|
|
||||||
|
|
||||||
## 3. Capture the machine-specific config into this repo
|
|
||||||
|
|
||||||
nixos-infect wrote `/etc/nixos/{hardware-configuration,networking,configuration}.nix`. Only the
|
|
||||||
first two matter; the flake replaces `configuration.nix`.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
scp root@<ip>:/etc/nixos/hardware-configuration.nix nix/hosts/cc-ci/hardware.nix
|
|
||||||
scp root@<ip>:/etc/nixos/networking.nix nix/hosts/cc-ci/networking.nix
|
|
||||||
```
|
|
||||||
|
|
||||||
Then in `nix/hosts/cc-ci/`:
|
|
||||||
|
|
||||||
- `hardware.nix`: keep as generated (GRUB EFI with `efiInstallAsRemovable`, `/boot/efi` by UUID,
|
|
||||||
`/dev/sda1` root). Do not copy another host's file — the UUIDs are per machine.
|
|
||||||
- `networking.nix`: keep the static IPv4 + Hetzner gateway `172.31.1.1`. Make sure
|
|
||||||
`networking.defaultGateway` has **both** `address` and `interface = "eth0"` (§2a item 3). If
|
|
||||||
the generated IPv6 block has an empty address, delete the IPv6 parts; a real global address
|
|
||||||
(as on the 2026-09 box) can stay.
|
|
||||||
- `configuration.nix`: set `cc-ci.publicIPv4` to the server's IPv4 and check `system.stateVersion`
|
|
||||||
is the release you installed (never change it later).
|
|
||||||
- `ssh-keys`: the root keys.
|
|
||||||
|
|
||||||
Commit on a branch; the rebuild in §5 can use the local checkout before the PR merges.
|
|
||||||
|
|
||||||
## 4. Stage the workspace and secrets on the new host
|
|
||||||
|
|
||||||
Everything in this section is **outside git**. Do it as root over SSH, in this order.
|
|
||||||
|
|
||||||
### 4a. No tailscale
|
|
||||||
|
|
||||||
The combined host is NOT on the tailnet (operator decision 2026-09-07): ssh is key-only on the
|
|
||||||
public IP, the CI front doors are public via traefik, and the opencode UI is public on the same
|
|
||||||
443 — traefik routes `oc.ci.commoninternet.net` to an nginx vhost on the docker bridge that
|
|
||||||
enforces basic auth, via traefik labels on a tiny swarm relay stack (`ccci-opencode-ui`, on the weekly sweep's keep-list) that
|
|
||||||
`opencode-ui-route.service` deploys. (Not via traefik's file *directory* provider: enabling
|
|
||||||
that in the coop-cloud traefik recipe REPLACES its provider file, which holds the wildcard cert
|
|
||||||
and the `security` middleware, and every front door goes down — it did, for two minutes, on
|
|
||||||
2026-09-07.) fail2ban guards sshd and that login
|
|
||||||
(`nix/hosts/cc-ci/configuration.nix`; the nginx jail bans in the DOCKER-USER chain because the
|
|
||||||
traffic is docker-forwarded, not host INPUT).
|
|
||||||
|
|
||||||
### 4b. The CI server's checkout and its one out-of-band secret
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# the bot deploy key for the private cc-ci-secrets submodule — generated ON this host by loops
|
|
||||||
# (§4c) and registered on the autonomic-bot Gitea account; root only points at it:
|
|
||||||
install -d -m700 /root/.ssh
|
|
||||||
cat > /root/.ssh/config <<'SSHCFG'
|
|
||||||
Host git.autonomic.zone
|
|
||||||
Port 2222
|
|
||||||
User git
|
|
||||||
IdentityFile /secrets/files/autonomic-bot-cc-ci-ed25519
|
|
||||||
IdentitiesOnly yes
|
|
||||||
SSHCFG
|
|
||||||
# the deployed checkout: nightly-sweep runs from it, sops reads secrets/secrets.yaml from it
|
|
||||||
git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci.git /etc/cc-ci
|
|
||||||
# the age identity sops-nix decrypts with. FIRST deploy of a host that is not yet a recipient:
|
|
||||||
# the master (recovery) key, temporarily. Once the host is a recipient (below): its own
|
|
||||||
# ssh-host-key-derived identity, and the master key leaves the box.
|
|
||||||
install -d -m0700 /secrets/host
|
|
||||||
install -m600 <master-age.txt> /secrets/host/sops-age-key.txt
|
|
||||||
```
|
|
||||||
|
|
||||||
`/etc/cc-ci/secrets/secrets.yaml` is encrypted to the master key and the *previous* hosts' SSH
|
|
||||||
host keys. The master key is enough for the FIRST deploy. Then make the new host a recipient so
|
|
||||||
the master key can leave the box again (it belongs off-box, with the operator):
|
|
||||||
`ssh-to-age < /etc/ssh/ssh_host_ed25519_key.pub` → add it to `.sops.yaml` in cc-ci-secrets AND in
|
|
||||||
cc-ci, `sops updatekeys secrets.yaml` (needs the master key, so do it where that lives), push,
|
|
||||||
bump the submodule in cc-ci, `git -C /etc/cc-ci pull --recurse-submodules` on the host, then
|
|
||||||
replace `/secrets/host/sops-age-key.txt` with the host-derived identity
|
|
||||||
(`ssh-to-age -private-key -i /secrets/host/ssh_host_ed25519_key`), rebuild, and delete the master
|
|
||||||
key from the host.
|
|
||||||
|
|
||||||
### 4c. The orchestrator's workspace (as the `loops` user — it exists after the first rebuild, so
|
|
||||||
run §5 once first if this is a fresh host, then come back)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sudo -iu loops
|
|
||||||
git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci-orchestrator.git /srv/cc-ci-orch
|
|
||||||
sudo ln -sfn /srv/cc-ci-orch /srv/cc-ci # every script and unit says /srv/cc-ci
|
|
||||||
cd /srv/cc-ci-orch
|
|
||||||
git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci # Builder clone
|
|
||||||
git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci-adv # Adversary clone
|
|
||||||
mkdir -p .cc-ci-logs .sops
|
|
||||||
```
|
|
||||||
|
|
||||||
**`/secrets` is the authoritative location for every secret on this host.** Consumers either read
|
|
||||||
from it directly (where the nix config owns the path) or reach it by a symlink (where the software
|
|
||||||
fixes the path), so `find /secrets -type f` is the complete inventory — including the ssh host
|
|
||||||
keys. The directory is `0711` (traversable, not listable) with three subdirectories, each owned by
|
|
||||||
whoever must read it; the layout is declared in `nix/hosts/cc-ci/configuration.nix`:
|
|
||||||
|
|
||||||
| dir | mode | holds |
|
|
||||||
|---|---|---|
|
|
||||||
| `/secrets/files` | `loops:users 0700` | the agent's secrets: `.testenv`, opencode auth, its two ssh keys |
|
|
||||||
| `/secrets/host` | `root:root 0700` | host identity: the ssh **host** keys and the sops age identity |
|
|
||||||
| `/secrets/nginx` | `root:nginx 0750` | `oc-htpasswd` for the opencode UI (nginx must read it) |
|
|
||||||
|
|
||||||
**Only cc-ci's secrets go there** — nothing from other projects (no tailscale, lichen, tangled,
|
|
||||||
b1, borg…), and every private key is generated **on the host**, never copied from another machine.
|
|
||||||
`/secrets/README.txt` lists each file, its purpose, and the derived material that legitimately
|
|
||||||
lives elsewhere (`/run/secrets`, `/root/.docker/config.json`, the ACME cert) so a later sweep does
|
|
||||||
not "clean" it away.
|
|
||||||
|
|
||||||
> **The ed25519 host key is load-bearing beyond ssh.** Its age identity is a recipient of
|
|
||||||
> `cc-ci-secrets`, so regenerating it makes every cc-ci secret undecryptable. sshd is pointed at
|
|
||||||
> `/secrets/host/...` directly (not through the `/etc/ssh` symlinks) precisely so it can never
|
|
||||||
> write a fresh key through a dangling link. To move a host: copy the key, verify
|
|
||||||
> `ssh-to-age` yields the same identity, then activate.
|
|
||||||
|
|
||||||
| runtime path → `/secrets/files/…` | what | source |
|
|
||||||
|---|---|---|
|
|
||||||
| `/srv/cc-ci/.testenv` → `cc-ci.testenv` | `GITEA_PASSWORD` (autonomic-bot: PR/API calls), `DOCKERHUB_USERNAME/TOKEN` (harness image pulls). Nothing else: no tailscale key, no third-party model API keys (opencode's own auth is `opencode-auth.json`; add `TINFOIL_API_KEY` back only if `LOOP_MODEL` moves to a tinfoil model). | old host's copy minus the tailscale line; fresh: create each credential |
|
|
||||||
| `/srv/cc-ci/upgrader.env` (not a secret, lives in the checkout) | `LOOP_TIER`, `LOOP_MODEL`, `REPORT_MODEL`, `SUPERVISOR_MODEL` for the weekly run. This host is on the **OpenCode Go subscription** (`LOOP_TIER=go`, `opencode-go/…` models); the ZEN tier is pay-as-you-go credit and its key is not on this host | old host, or copy the example in `AGENTS.md` |
|
|
||||||
| `~loops/.ssh/cc-ci-local-ed25519` (+`.pub`) → same names | `ssh cc-ci` as root — to loopback on this host. **Generated on the host**, its pub in `nix/hosts/cc-ci/ssh-keys` | `ssh-keygen -t ed25519 -C cc-ci-loops-to-root@cc-ci` as loops |
|
|
||||||
| `~loops/.ssh/autonomic-bot-cc-ci-ed25519` (+`.pub`) → same names | pushes recipe branches / PRs as `autonomic-bot`; root's ssh config points at the same file to clone the private cc-ci-secrets submodule. **Generated on the host** and registered on the bot's Gitea account (`POST /api/v1/user/keys` with the bot password) | `ssh-keygen -t ed25519 -C autonomic-bot@cc-ci-host` as loops |
|
|
||||||
| `~loops/.local/share/opencode/auth.json` → `opencode-auth.json`; `~loops/.config/opencode/opencode.jsonc` (config, not secret — it must contain NO inline `apiKey`) | opencode provider auth + config. Exactly ONE provider, `opencode-go` — the orchestrator AND the weekly upgrader are opencode agents; there is no Claude on this host | old host; fresh: `opencode auth login` as loops |
|
|
||||||
| `/secrets/nginx/oc-htpasswd` (root:nginx 0640; the bcrypt line only — the plaintext stays with the operator, not on the host) | basic auth for the opencode UI (`https://oc.ci.commoninternet.net`, via traefik); **nginx refuses to start without it**, and its config check runs as the `nginx` user, so `root:nginx 0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host (`/secrets/files/oc-basic-auth.txt` has the plaintext), or a new `oc:<bcrypt>` line via `mkpasswd -m bcrypt` |
|
|
||||||
|
|
||||||
`~loops/.ssh/config` is written by the activation script on first rebuild (`Host cc-ci` →
|
|
||||||
`127.0.0.1`, `git.autonomic.zone`); it is not overwritten if present.
|
|
||||||
|
|
||||||
## 5. Build and activate
|
|
||||||
|
|
||||||
From the checkout with the §3 commit (root can build from the loops-owned checkout via sudo):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# as root, detached (the activation restarts sshd; a dropped session must not kill it).
|
|
||||||
# Three things the FIRST rebuild on a bare infect system needs, none of which the converged
|
|
||||||
# host needs afterwards: `git` on PATH (nix's flake fetcher shells out to it and the infect
|
|
||||||
# system has none — hence nix-shell), HOME=/root (so root's `git config --global
|
|
||||||
# safe.directory '*'` applies to the loops-owned checkout), and a login shell (`bash -l`, for
|
|
||||||
# NIX_SSL_CERT_FILE and friends from /etc/set-environment).
|
|
||||||
git config --global --add safe.directory '*'
|
|
||||||
systemd-run --unit=ccci-rebuild --collect -E HOME=/root -p WorkingDirectory=/srv/cc-ci-orch \
|
|
||||||
bash -lc 'nix-shell -p git --run "nixos-rebuild test --flake /srv/cc-ci-orch#cc-ci"'
|
|
||||||
journalctl -fu ccci-rebuild # ~10 min the first time (image pulls + two OCI image builds)
|
|
||||||
```
|
|
||||||
|
|
||||||
`test` first, always: it activates WITHOUT touching the bootloader, so if the activation breaks
|
|
||||||
networking or sshd a reboot from the Hetzner console lands on the last known-good generation.
|
|
||||||
(Earlier hosts were on the tailnet, and their rebuilds had to be detached because activation
|
|
||||||
restarted tailscale under the session; this one is plain public ssh, but detached is still the
|
|
||||||
safe habit.)
|
|
||||||
Later rebuilds are simply `sudo nixos-rebuild test|switch --flake .#cc-ci` from the checkout.
|
|
||||||
|
|
||||||
The first activation takes a while: it pulls the traefik/drone/keycloak images, builds the bridge
|
|
||||||
and dashboard OCI images with Nix, initialises the swarm and runs the serialized reconcile
|
|
||||||
oneshots (`swarm-init → deploy-proxy → deploy-drone → deploy-bridge → deploy-dashboard →
|
|
||||||
deploy-reports`, `deploy-backupbot`, `warm-keycloak`). Verify:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
systemctl is-system-running # running — or list-units --failed and read journalctl -u <unit>
|
|
||||||
docker service ls # traefik app+socket-proxy, drone, bridge, dashboard, reports, backups: 1/1
|
|
||||||
systemctl status cc-ci-loops cc-ci-orchestrator opencode-web nginx acme-dns fail2ban
|
|
||||||
fail2ban-client status sshd; fail2ban-client status nginx-http-auth
|
|
||||||
systemctl list-timers 'cc-ci-*' nightly-sweep
|
|
||||||
sudo -iu loops tmux ls # cc-ci-orchestrator (+ loops sessions if a phase is active)
|
|
||||||
# the CI front doors, before DNS points here (expect 200 / 200 / 303 and ssl_verify=0 once
|
|
||||||
# /var/lib/acme is restored or a cert has been issued):
|
|
||||||
curl -s --resolve ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://ci.commoninternet.net/
|
|
||||||
curl -s --resolve report.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code}\n' https://report.ci.commoninternet.net/
|
|
||||||
curl -s --resolve drone.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code}\n' https://drone.ci.commoninternet.net/
|
|
||||||
dig +short @<ip> ns-acme.commoninternet.net # acme-dns answering on the public 53
|
|
||||||
```
|
|
||||||
|
|
||||||
The opencode UI: `curl -s --resolve oc.ci.commoninternet.net:443:<ip> -o /dev/null -w '%{http_code}' https://oc.ci.commoninternet.net/`
|
|
||||||
→ 401 without credentials, 200 with `-u oc:<password>`; `fail2ban-client status nginx-http-auth`
|
|
||||||
counts the failures (it reads nginx's journal — NixOS nginx logs to stderr, not to
|
|
||||||
/var/log/nginx/error.log).
|
|
||||||
|
|
||||||
When it is healthy: `sudo nixos-rebuild switch --flake .#cc-ci` (same config, now also the boot
|
|
||||||
default). **If you are migrating from another host, do §6 before letting it serve anything**: right
|
|
||||||
after the first activation scale the `!testme` bridge to 0 and mask the two orchestrator timers so
|
|
||||||
the new box does not process PR comments or start a second weekly run while the old host is live:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker service scale ccci-bridge_app=0
|
|
||||||
systemctl mask --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer
|
|
||||||
```
|
|
||||||
|
|
||||||
## 6. Migrating: restore state from the previous host
|
|
||||||
|
|
||||||
Over ssh between the hosts (`ssh root@<old> 'tar -C / -cf - <paths>' | ssh root@<new> 'tar -C / -xpf - --numeric-owner'`, or rsync), with the matching service
|
|
||||||
stopped on the new host while its directory is copied:
|
|
||||||
|
|
||||||
| path | holds | notes |
|
|
||||||
|---|---|---|
|
|
||||||
| `/var/lib/cc-ci-reports` | the published weekly report pages (`report.ci…`) | |
|
|
||||||
| `/var/lib/cc-ci-runs` | per-run artifacts the dashboard shows | |
|
|
||||||
| `/var/lib/ci-warm` | warm-canonical state + alerts | recipe warm *volumes* are caches: not copied, rebuilt by the Sunday sweep / first use |
|
|
||||||
| `/var/lib/acme` | the Let's Encrypt cert + account **and `acme-dns-accounts.json`** — the account the permanent `_acme-challenge` CNAME points at | without it a fresh registration + a new CNAME at Gandi is needed (registration is disabled in `acme-dns.nix`) |
|
|
||||||
| `/var/lib/acme-dns` | the acme-dns zone DB | |
|
|
||||||
| `/var/lib/ci-certs` | the copy traefik is handed | then `systemctl restart cc-ci-acme-traefik-handoff` |
|
|
||||||
| `/root/.abra` | abra's per-app env files for the deployed stacks | |
|
|
||||||
| Drone data volume `/var/lib/docker/volumes/drone_ci_commoninternet_net_data` | Drone's DB: the Gitea OAuth grant, repo activation, build history | `docker service scale drone_ci_commoninternet_net_app=0` on the new host, copy, scale back to 1. Otherwise run `scripts/bootstrap-drone-oauth.sh` (cc-ci repo) with the bot password and re-activate repos |
|
|
||||||
| `/srv/cc-ci-orch/.cc-ci-logs`, `/srv/cc-ci-orch/cc-ci-plan/upstream/`, `REBOOTS.md`, `JOURNAL.md` | orchestrator history, the upgrader's per-recipe release-note registry | as loops; do the final sync after stopping the orchestrator on the old host |
|
|
||||||
|
|
||||||
## 7. Cutover and verification
|
|
||||||
|
|
||||||
1. **DNS** (operator, Gandi zone `commoninternet.net`): A records `ci`, `*.ci` and `ns-acme` →
|
|
||||||
the new IPv4. `acme NS ns-acme` and `_acme-challenge.ci CNAME <account>.acme…` stay as they
|
|
||||||
are. Wait for propagation (`dig +short ci.commoninternet.net`) — the records carry a 3 h TTL,
|
|
||||||
and resolvers that cached the old answer (the Hetzner ones in particular) keep serving it that
|
|
||||||
long. The new host must not list such a resolver: with a mixed set, every other lookup of its
|
|
||||||
own names went to the old server on 2026-09-07 (runner/bridge 404s). `networking.nix` drops
|
|
||||||
the Hetzner resolvers and `configuration.nix` pins the host's own names.
|
|
||||||
2. Check the new host answers on the new IP before DNS moves: `dig @<new-ip> ns-acme.commoninternet.net`
|
|
||||||
(acme-dns), `curl --resolve ci.commoninternet.net:443:<new-ip> https://ci.commoninternet.net/`
|
|
||||||
(dashboard, valid cert), same for `report.ci` and `drone.ci`.
|
|
||||||
3. Old host: `docker service scale ccci-bridge_app=0 drone_ci_commoninternet_net_app=0`;
|
|
||||||
`systemctl disable --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer` on the old
|
|
||||||
orchestrator. New host: `docker service scale ccci-bridge_app=1`;
|
|
||||||
`systemctl unmask cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer && systemctl start` both.
|
|
||||||
4. End to end: post `!testme` on an open recipe PR and watch it turn green on the new Drone;
|
|
||||||
open `https://ci.commoninternet.net` and `https://report.ci.commoninternet.net`.
|
|
||||||
5. The orchestrator: as loops on the new host `cd /srv/cc-ci-orch && python3 cc-ci-plan/agents.py up orchestrator`
|
|
||||||
(or just `systemctl restart cc-ci-orchestrator`), attach with `claude --resume` or from
|
|
||||||
claude.ai/code. Its startup routine (AGENTS.md) reports phase + reboot count.
|
|
||||||
6. Keep the old host as a cold standby for a week, then delete it (and its tailnet node — the
|
|
||||||
old servers were on the tailnet; the new one is not).
|
|
||||||
|
|
||||||
## 8. Day 2
|
|
||||||
|
|
||||||
- **The host updates itself weekly**: `cc-ci-auto-update.timer` (Tuesday 03:00 UTC,
|
|
||||||
`nix/modules/auto-update.nix`) runs `nix flake update` on all inputs, builds, activates with
|
|
||||||
`test`, runs the cc-ci health checks (swarm replicas, front doors, sops, acme-dns, fail2ban…),
|
|
||||||
and only then commits the generation + bootloader and pushes the health-checked `flake.lock` to
|
|
||||||
`main`; on any failure it re-activates the previous generation and restores the lock. It skips
|
|
||||||
itself while a CI run, the weekly upgrader, the report or the Sunday sweep is in flight. Outcome
|
|
||||||
in `/srv/cc-ci/.cc-ci-logs/auto-update-state` (one line) and `journalctl -u cc-ci-auto-update`;
|
|
||||||
`/cc-ci-status` §5 reads it. `/cc-ci-orchestrator-update` runs the same unit by hand.
|
|
||||||
The `cc-ci` input follows this flake's nixpkgs, so the CI server is rebuilt on the same nixpkgs.
|
|
||||||
- **Update only cc-ci's code** (harness/tests/modules): merge in the cc-ci repo, then
|
|
||||||
`nix flake update cc-ci` here and rebuild; also `git -C /etc/cc-ci pull --recurse-submodules`
|
|
||||||
so the deployed checkout the sweep runs from matches. Run the lock update **on this host**: a
|
|
||||||
lock written by a newer Nix elsewhere once failed here with `NAR hash mismatch in input
|
|
||||||
git+https://…cc-ci.git…&shallow=1`; re-running `nix flake update cc-ci` on the host fetched the
|
|
||||||
input properly and the rebuild went through.
|
|
||||||
- **Something is down**: `systemctl --failed`, `journalctl -u deploy-<x>`, `docker service ps <svc>`;
|
|
||||||
the cc-ci repo's `docs/runbook.md`. Host unreachable: Hetzner console → reboot lands on the last
|
|
||||||
`switch`ed generation; rescue mode + `nixos-enter` for anything worse (skill
|
|
||||||
`/hetzner-server-recovery`).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
# Operating the orchestrator
|
|
||||||
|
|
||||||
The cc-ci orchestrator on this host is an **opencode agent** (`cc-ci-plan/agents.toml`:
|
|
||||||
backend `opencode`, model `opencode/glm-5.2`), launched on boot by `cc-ci-orchestrator.service`
|
|
||||||
via `cc-ci-plan/agents.py up orchestrator` into tmux session `cc-ci-orchestrator`, attached to
|
|
||||||
the shared opencode web server. Steer it from the web UI, **https://oc.ci.commoninternet.net**
|
|
||||||
(basic auth, user `oc`), where the weekly upgrader's and report's sessions show up too.
|
|
||||||
There is no Claude Code on this host: Claude sessions run on the notplants-orchestrator box and
|
|
||||||
operate the CI from there over ssh (`Host cc-ci` in that box's loops ssh config → this host).
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# on the host
|
|
||||||
sudo -iu loops tmux attach -t cc-ci-orchestrator # the agent's terminal
|
|
||||||
systemctl restart cc-ci-orchestrator # relaunch it (fresh opencode session)
|
|
||||||
sudo -iu loops tmux ls # cc-ci-orchestrator, cc-ci-upgrader (+watchdog) during the weekly run
|
|
||||||
```
|
|
||||||
|
|
||||||
## Weekly upgrade run
|
|
||||||
|
|
||||||
`cc-ci-upgrade-all.timer` (Thu 22:00 America/New_York) starts `launch-upgrader.py`, which runs
|
|
||||||
`/upgrade-all` as an opencode agent (`upgrader.env`: `LOOP_MODEL`/`REPORT_MODEL`) in tmux
|
|
||||||
`cc-ci-upgrader` with a usage-limit watchdog; the hourly `cc-ci-upgrade-supervisor.timer` drives a
|
|
||||||
stalled run to completion. Start one by hand with `systemctl start cc-ci-upgrade-all`; follow
|
|
||||||
`/srv/cc-ci/.cc-ci-logs/cc-ci-upgrader.log`; it ends with `UPGRADE RUN COMPLETE`, a summary in
|
|
||||||
`.cc-ci-logs/upgrades/upgrade-all-<date>.md`, and a report published to
|
|
||||||
`https://report.ci.commoninternet.net/week-<date>.html`.
|
|
||||||
|
|
||||||
## Kick off / supervise the loops
|
## Kick off / supervise the loops
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd /srv/cc-ci/cc-ci-plan
|
cd /srv/cc-ci/cc-ci-plan
|
||||||
./launch.sh start # Builder + Adversary loops + watchdog (all phases are DONE; this just confirms)
|
./launch.sh start # Builder + Adversary loops (interactive --remote-control in tmux) + watchdog
|
||||||
./launch.sh status # session + DONE state
|
./launch.sh status # session + DONE state
|
||||||
./launch.sh logs builder|adversary|watchdog
|
./launch.sh logs builder|adversary|watchdog
|
||||||
./launch.sh stop
|
./launch.sh stop
|
||||||
```
|
```
|
||||||
|
|
||||||
Full supervision guide, credential map and history are in `cc-ci-plan/kickoff.md`,
|
Full supervision guide, credential map, and the Incus VM fallback are in
|
||||||
`cc-ci-plan/plan.md` §1.5 and `cc-ci-plan/JOURNAL.md`.
|
[`cc-ci-plan/kickoff.md`](cc-ci-plan/kickoff.md) and [`cc-ci-plan/plan.md`](cc-ci-plan/plan.md) §1.5.
|
||||||
|
|||||||
@@ -1,22 +0,0 @@
|
|||||||
# archive/ — how cc-ci and its orchestrator were built and moved, before the combined host
|
|
||||||
|
|
||||||
Historical record only. Nothing in here is deployed or evaluated. It was moved out of the live
|
|
||||||
tree on 2026-09-07 when the CI server and the orchestrator were consolidated onto one Hetzner
|
|
||||||
host (`nixosConfigurations.cc-ci` in `../flake.nix`; deploy guide in `../README.md`; the plan
|
|
||||||
that did it is `../cc-ci-plan/plan-cc-ci-combined-host.md`).
|
|
||||||
|
|
||||||
| path | what it was |
|
|
||||||
|---|---|
|
|
||||||
| `nix/configuration-incus-vm.nix` | Channel-based NixOS config of the first orchestrator VM on b1 (Incus, 2 GB). Ran the loops as root; hard-coded the dead Incus cc-ci IP. Replaced by the Hetzner host 2026-05-31. |
|
|
||||||
| `nix/README.md` | The README for that Incus VM config. |
|
|
||||||
| `nix/cc-ci-orchestrator-hetzner/` | The orchestrator's own Hetzner `cpx22` host (`168.119.126.100`, tailnet `cc-ci-orchestrator-1`), 2026-05-31 → 2026-09. From 2026-08-20 the live copy of this config was `notplants-nix`'s `notplants-orchestrator` host (the box became a shared agent host for several projects); this one had drifted and still carried lichen/project-orchestrator units. Superseded by `../nix/hosts/cc-ci` + `../nix/modules/orchestrator-host.nix`. |
|
|
||||||
| `nix/atproto-likes.nix` | A notplants (not cc-ci) service that lived on the shared box; kept by notplants-nix. |
|
|
||||||
| `terraform/` | OpenTofu for the `cpx22` orchestrator server (Debian 12 → nixos-infect at `nixos-24.11`). The combined host was provisioned by hand instead; the README documents that path. Note its `user-data.sh` would fail on the Debian 13 image (nixos-infect's temp swapfile on a tmpfs `/tmp`) — see the README's `NO_SWAP=true` note. |
|
|
||||||
| `plans/plan-orchestrator-migration.md` | Pi → Incus VM move of the orchestrator (2026-05). |
|
|
||||||
| `plans/plan-orchestrator-hetzner-migration.md` | Incus VM → Hetzner `cpx22` move of the orchestrator (2026-05-31). Has the reboot-resilience design (`cc-ci-loops.service`). |
|
|
||||||
| `plans/plan-migrate-cc-ci-to-hetzner.md`, `plans/plan-cc-ci-hetzner-migration.md`, `plans/plan-cc-ci-hetzner-terraform.md` | The CI server's own move from the `cc-nix-test` Incus VM to Hetzner `cpx32` (`91.98.47.73`, 2026-05-31), and the terraform that provisioned it (lives in the cc-ci repo). |
|
|
||||||
| `plans/plan-repo-consolidation.md` | The earlier repo layout consolidation. |
|
|
||||||
|
|
||||||
The cc-ci server's own history (machine-docs, decisions, the clean-room rebuild that proved
|
|
||||||
"two repos + one age key + one `nixos-rebuild switch`") is in the cc-ci repo under
|
|
||||||
`machine-docs/` and `docs/`.
|
|
||||||
@@ -870,335 +870,3 @@ session cc-ci-orchestrator-stale can be killed; recipe-mirrors org still private
|
|||||||
LESSON: `opencode debug config` proves resolution, NOT binding — only a live subagent's recorded
|
LESSON: `opencode debug config` proves resolution, NOT binding — only a live subagent's recorded
|
||||||
modelID proves binding. First attempt was a false pass because the probe passed --dir (unlike the
|
modelID proves binding. First attempt was a false pass because the probe passed --dir (unlike the
|
||||||
real launcher) and landed in a different project.
|
real launcher) and landed in a different project.
|
||||||
|
|
||||||
## Session 2026-08-15 19:25 UTC — opencode glm-5.2
|
|
||||||
|
|
||||||
**Left off:** Recovered the stalled 2026-08-14 weekly /upgrade-all run. Killed a supervisor that had
|
|
||||||
been relaunching hourly for ~40h (balance exhausted), then started a FRESH scoped upgrader. Run is now
|
|
||||||
progressing (surveying the 9 remaining recipes). Watching it through to completion.
|
|
||||||
|
|
||||||
**What happened (the stall):**
|
|
||||||
- The 2026-08-14 /upgrade-all run (session ses_00200382fffeYIGl2sc3mO9JId) stalled at 03:18 Aug 14
|
|
||||||
mid-`lasuite-drive` with `Error: Insufficient balance` (opencode zen workspace balance ran out). It
|
|
||||||
had already done bluesky-pds, ghost, gitea, hedgedoc (PRs) + immich, lasuite-docs (SKIPPED up-to-date)
|
|
||||||
alphabetically; lasuite-drive had a plan + partial PR #6 but no RESULT/verify.
|
|
||||||
- The supervisor cron (glm-5.2, opencode-go tier) relaunched an hourly one-shot supervisor ~40 times
|
|
||||||
to "drive it to completion", but each was also balance-walled (and later, just spinning). The run sat
|
|
||||||
INCOMPLETE + not progressing for 40h. No weekly summary, no report published for week of Aug 14.
|
|
||||||
|
|
||||||
**What I did this session:**
|
|
||||||
- Diagnosed: the opencode zen endpoint is NOW healthy (direct probe `say OK` → HTTP 200 in 1.35s —
|
|
||||||
balance is restored). But resuming the ORIGINAL giant session is impossible: it's 2.58M tokens
|
|
||||||
(267K input + 2.3M cache) and `opencode run -s … --continue` sits idle on `do_epoll_wait` with zero
|
|
||||||
I/O — the inference endpoint silently drops the oversized request (matches the supervisor's
|
|
||||||
`socket connection was closed unexpectedly` errors). A fresh small `opencode run` works fine. So the
|
|
||||||
giant session is unresumable; a fresh start is the only path.
|
|
||||||
- Killed the stuck supervisor (tmux `cc-ci-supervisor`, proc 377329).
|
|
||||||
- `UPGRADER_ARGS="lasuite-drive lasuite-meet mailu matrix-synapse mattermost-lts mumble n8n plausible
|
|
||||||
wordpress --sequential" python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py fresh` — this killed the
|
|
||||||
stuck resume, archived the old giant session (`archive-cc-ci-upgrader — 2026-08-14`), reclaimed 10GB
|
|
||||||
stale images on cc-ci (disk 29%), and started a FRESH small session
|
|
||||||
`ses_ff920cf39ffeoogwXHTajp94cr` (zen/glm-5.2) scoped to the 9 recipes not yet done this week
|
|
||||||
(positions 13-21 alphabetically; positions 1-12 were already surveyed — 6 PRs + 6 up-to-date). A
|
|
||||||
fresh watchdog is watching the new session. The skill is idempotent (reuses existing PRs incl.
|
|
||||||
lasuite-drive #6, never duplicates), so scoping is safe.
|
|
||||||
- Confirmed the fresh run is progressing: pane shows it surveying the 9 recipes (verified all present
|
|
||||||
in abra + all `weekly` tier; currently probing plausible/wordpress tags). Proc alive, log growing.
|
|
||||||
|
|
||||||
**Phase / loop state:**
|
|
||||||
- Build/adversary loops: STOPPED (whole sequence completed 2026-08-01; phase ghost DONE).
|
|
||||||
- Weekly upgrader: RUNNING (fresh session ses_ff920cf39, scoped 9 recipes, --sequential, watchdog up).
|
|
||||||
- cc-ci server: healthy (disk 29%, runner active).
|
|
||||||
|
|
||||||
**Open items for next session:**
|
|
||||||
- **Monitor the fresh upgrader to completion.** It will survey the 9 recipes, /recipe-upgrade the
|
|
||||||
upgradeable ones (subagents, !testme verify, open/extend PRs — NEVER merge), write the weekly summary
|
|
||||||
to `/srv/cc-ci/.cc-ci-logs/upgrades/`, then `launch-report.py fresh` (the upgrade-all skill does this
|
|
||||||
itself per SKILL.md §5), print `UPGRADE RUN COMPLETE`, and go idle. If it stalls on a usage limit,
|
|
||||||
the watchdog auto-resumes the SAME (small) session — that works now.
|
|
||||||
- **Do NOT try to resume the archived giant session ses_00200382** — it's unresumable (endpoint drops
|
|
||||||
the 2.58M-token request). It's archived; leave it.
|
|
||||||
- After the run completes + report publishes, operator review queue = this week's recipe PRs.
|
|
||||||
- The supervisor cron (hourly at XX:07) should now leave the run alone once it's progressing; if a
|
|
||||||
supervisor fires while the run is mid-flight, its guardrails say to hand back to the resumed run, not
|
|
||||||
double-write. No action needed unless it interferes.
|
|
||||||
|
|
||||||
**Notes:**
|
|
||||||
- Root cause of the 40h silence was the same BUG 1 from 2026-08-10 (supervisor progress gate) partly:
|
|
||||||
the supervisor kept firing because the run never reached "progressing". Now that balance is restored
|
|
||||||
and a fresh small session is running, the gate should see progress and stand down.
|
|
||||||
- Lesson: when a weekly run dies mid-flight on a giant context, do NOT resume the original session —
|
|
||||||
start fresh and scope to the remaining recipes. The /upgrade-all skill is idempotent so this is safe.
|
|
||||||
|
|
||||||
## Session 2026-08-31 17:17 UTC — restricted acme-dns renewal staged
|
|
||||||
|
|
||||||
**What happened:** Implemented and deployed the cc-ci restricted ACME DNS-01 design in cc-ci PRs
|
|
||||||
#24–#27 (all merged). The initial `sqlite3` backend setting was detected during staging bootstrap
|
|
||||||
and corrected to acme-dns 2.x `sqlite` before any account registration succeeded.
|
|
||||||
|
|
||||||
**Live state:**
|
|
||||||
- `acme-dns` is authoritative for `acme.commoninternet.net` on `91.98.47.73`, with public UDP/TCP
|
|
||||||
53 verified directly; it serves the apex NS and `ns-acme.commoninternet.net A 91.98.47.73`.
|
|
||||||
- Its HTTP update API is bound solely to `127.0.0.1:8080`; registration is disabled (local POST
|
|
||||||
returns 403). The one restricted Lego account is stored mode `0600` under `/var/lib/acme/`, outside
|
|
||||||
Git and the Nix store. No Gandi credentials were used or stored.
|
|
||||||
- Nix ACME is in Let's Encrypt staging mode for `ci.commoninternet.net` and its wildcard; its renewal
|
|
||||||
timer is enabled. Traefik handoff remains production-gated so staging cannot replace the live cert.
|
|
||||||
- Every server deployment used `nixos-rebuild test`, reachability/service verification, then
|
|
||||||
`nixos-rebuild switch` on `#cc-ci-hetzner`.
|
|
||||||
|
|
||||||
**Manual Gandi handoff (only blocker):** Add the parent-zone A+NS delegation and permanent CNAME:
|
|
||||||
`ns-acme.commoninternet.net. A 91.98.47.73`,
|
|
||||||
`acme.commoninternet.net. NS ns-acme.commoninternet.net.`, and
|
|
||||||
`_acme-challenge.ci.commoninternet.net. CNAME
|
|
||||||
9d3f0428-eb97-4b14-bb07-da2f0f6c2073.acme.commoninternet.net.` Once propagated, resume with a
|
|
||||||
staging issuance, then production cutover/handoff verification. Do not obtain or add a Gandi token.
|
|
||||||
|
|
||||||
## Session 2026-08-31 19:05 UTC — ACME production issuance complete
|
|
||||||
|
|
||||||
**Completed:** The operator added the narrow Gandi delegation and permanent CNAME. Staging then
|
|
||||||
validated both `ci.commoninternet.net` and `*.ci.commoninternet.net`; production issuance validated
|
|
||||||
the same SAN pair and produced a Let's Encrypt certificate valid through 2026-11-29. cc-ci PRs #28–#30
|
|
||||||
were merged to promote production, make Traefik roll certificate-secret changes even when recipe
|
|
||||||
upgrades are held, and remove the retired SOPS wildcard-file injection.
|
|
||||||
|
|
||||||
**Live cc-ci state:**
|
|
||||||
- `/var/lib/acme/ci.commoninternet.net` is the sole source of the production cert/key; the runtime
|
|
||||||
pair matches it, is root-only for the key, and Traefik now references matching versioned Swarm
|
|
||||||
secrets. Reboot-safe `nixos-rebuild switch` completed at cc-ci `7147d77` after the required test
|
|
||||||
activations and health checks.
|
|
||||||
- acme-dns remains authoritative only for the delegated subzone, its update API is localhost-only,
|
|
||||||
registration is disabled, and the restricted-account JSON is mode `0600`. The Nix ACME renewal
|
|
||||||
timer is active. No Gandi credential was added.
|
|
||||||
|
|
||||||
**External blocker, not cc-ci-owned:** Direct TLS to `91.98.47.73:443` now presents the new valid
|
|
||||||
certificate. The operator-managed public gateway at `143.244.213.108` currently accepts TCP then
|
|
||||||
closes TLS before presenting a certificate, so its passthrough to cc-ci needs repair/verification.
|
|
||||||
The cc-ci plan explicitly marks the gateway as operator infrastructure; do not change the DNS
|
|
||||||
delegation or add a Gandi token to work around it.
|
|
||||||
|
|
||||||
## Session 2026-08-31 20:45 UTC — weekly DeepSeek Flash run and report complete
|
|
||||||
|
|
||||||
**Completed:** Started the weekly upgrader explicitly on `opencode/deepseek-v4-flash`; its parent
|
|
||||||
and every recipe subagent were confirmed with that persisted model. All nine eligible recipe PRs
|
|
||||||
completed `!testme` successfully. The public report was then restarted before publication when the
|
|
||||||
first handoff was found to use the old GLM model; the replacement report session was confirmed
|
|
||||||
`opencode/deepseek-v4-flash` and published successfully at
|
|
||||||
`https://report.ci.commoninternet.net/week-2026-08-31.html` (HTTP 200).
|
|
||||||
|
|
||||||
**Follow-up fixes:** Orchestrator PR #17 (`1b75d98`) made `testme-on-pr.sh` apply the documented
|
|
||||||
public Gitea-host default. cc-ci PR #31 (`769fd29`) adds that public hostname to `.env.public`.
|
|
||||||
Both commits were scanned clean and contain no coauthor trailers. No recipe PR was merged.
|
|
||||||
|
|
||||||
**Security note:** A subagent briefly enabled shell tracing while debugging the verifier, exposing
|
|
||||||
runtime credentials in its private agent trace. No values were committed or put in this journal,
|
|
||||||
but rotate the affected `/srv/cc-ci/.testenv` credentials as a precaution.
|
|
||||||
|
|
||||||
## Session 2026-09-07 19:30 UTC — Claude Fable 5.1 orchestrator (re)launch, startup check
|
|
||||||
|
|
||||||
**What happened:** Orchestrator relaunched on the `claude` backend (`agents.toml` now says
|
|
||||||
`backend = "claude"`, `model = "claude-fable-5-1"`, operator change today, uncommitted). Ran the
|
|
||||||
AGENTS.md on-startup routine. NOT a reboot: host uptime 15 days, REBOOTS.md still shows 5 reboots
|
|
||||||
(last 2026-08-23 03:11 UTC). `cc-ci-loops.service` was restarted at 14:50 and 15:14 UTC today by a
|
|
||||||
`nixos-rebuild test --flake /srv/notplants-nix#notplants-orchestrator`, which re-ran `launch.sh start`;
|
|
||||||
the phase sequence immediately re-concluded (all 15 phases DONE, "entire build finished"), so
|
|
||||||
builder/adversary/watchdog being stopped is the expected terminal state. Did NOT relaunch the loops.
|
|
||||||
|
|
||||||
**Current state:**
|
|
||||||
- Weekly `/upgrade-all` 2026-09-04 completed: 8 upgrade PRs extended (custom-html, ghost,
|
|
||||||
lasuite-docs/drive/meet, matrix-synapse, mattermost-lts, n8n), 0 failed, nothing merged. Report
|
|
||||||
`week-2026-09-04.html` returns 200. Next timer run Fri 2026-09-11 02:00 UTC.
|
|
||||||
- Hourly supervisor (XX:07) fires and stands down in ~1s — nothing to drive.
|
|
||||||
- Open operator items from the 09-04 run: review/merge the 8 PRs; `warm-gitea` canonical
|
|
||||||
crash-looping on read-only `/etc/gitea` (pre-existing); deployed `/root/cc-ci/tests` on the CI host
|
|
||||||
lags server-repo `main` (missing `tests/wordpress`).
|
|
||||||
- Uncommitted in this checkout (left alone, operator WIP): `agents.toml` backend switch,
|
|
||||||
auto-appended 2026-08-23 line in `REBOOTS.md`, and the untracked `plan-agent-orchestrator.md` /
|
|
||||||
`plan-phase-ao*.md` / `cc-ci-conc/` set.
|
|
||||||
|
|
||||||
## Session 2026-09-07 20:00 UTC — start of the cc-ci + orchestrator consolidation onto one Hetzner host
|
|
||||||
|
|
||||||
**Operator request:** move the cc-ci CI server AND the orchestrator to a new Hetzner box
|
|
||||||
(`195.201.88.249`, 8 GB), leave everything notplants-side on this host, keep cc-ci's nix in the
|
|
||||||
cc-ci repo and the orchestrator's in cc-ci-orchestrator with the latter including the former,
|
|
||||||
add `archive/` + a from-scratch deploy README, and (last) move to `autonomic.zone` subdomains.
|
|
||||||
Plan + live log: `cc-ci-plan/plan-cc-ci-combined-host.md` (on the branch; copy here).
|
|
||||||
|
|
||||||
**Done this session:**
|
|
||||||
- New ssh key `notplants-orchestrator` (`/secrets/files/notplants-orchestrator-ed25519`), on the new box.
|
|
||||||
- nixos-infect on the new box (Debian 13 → NixOS 26.05). Gotcha: `/tmp` is tmpfs on that image,
|
|
||||||
nixos-infect's temp swapfile fails → `NO_SWAP=true`. It built and rebooted ~19:50 UTC and had
|
|
||||||
NOT come back by 20:00 (no ping) — operator to check the Hetzner console / give an API token.
|
|
||||||
- cc-ci branch `feat/nixos-module-export` (9b99f81, pushed): `nixosModules.cc-ci-server`
|
|
||||||
(`nix/modules/default.nix`), options `cc-ci.publicIPv4` + `cc-ci.sopsFile`; standalone `#cc-ci`
|
|
||||||
drv byte-identical before/after.
|
|
||||||
- cc-ci-orchestrator branch `feat/combined-cc-ci-host` (31af820, pushed): flake input `cc-ci`
|
|
||||||
(follows), `nixosConfigurations.cc-ci`, `nix/modules/orchestrator-host.nix`, `nix/hosts/cc-ci/`
|
|
||||||
(hardware/networking PROVISIONAL until the infect output is captured), README deploy guide,
|
|
||||||
`archive/` (old host configs, terraform, migration plans), AGENTS.md + update-skill refs.
|
|
||||||
`#cc-ci` evaluates. Work is in git worktrees under the session scratchpad, not in this checkout.
|
|
||||||
|
|
||||||
**Next:** box reachable → capture hardware/networking → stage secrets → `nixos-rebuild test`
|
|
||||||
→ data copy → DNS cutover → move the orchestrator → notplants-nix PR dropping cc-ci → autonomic.zone.
|
|
||||||
|
|
||||||
## Session 2026-09-07 20:30 UTC — new combined host is UP, pre-cutover
|
|
||||||
|
|
||||||
- nixos-infect trouble root-caused from Hetzner rescue mode (operator gave an API token, stored
|
|
||||||
at `/srv/cc-ci/.hcloud-token`, server id 165014541, cpx32 nbg1): (1) `NO_SWAP=true` for tmpfs
|
|
||||||
/tmp; (2) 26.05's systemd initrd did NOT lustrate — Debian's units shadowed NixOS's, every
|
|
||||||
service failed; fixed by moving the old root to `/old-root` by hand; (3) bare-string
|
|
||||||
`defaultGateway` → no default route; fixed + chroot `nixos-rebuild boot --option sandbox false`.
|
|
||||||
All documented in the new README §2a.
|
|
||||||
- cc-ci PR #32 merged (module export). cc-ci-orchestrator PR #19 merged (combined host). Both
|
|
||||||
branches scanned clean by the commit hook.
|
|
||||||
- New box: `nixos-rebuild test` → verified → `switch`; reboot test OK. Data restored: acme (+
|
|
||||||
acme-dns account), acme-dns, ci-certs, reports, runs, ci-warm, /root/.abra, Drone volume (with
|
|
||||||
drone scaled to 0 during the copy). Dashboard/reports/drone answer on the new IP with the valid
|
|
||||||
LE cert; acme-dns answers on public 53.
|
|
||||||
- Pre-cutover quarantine on the new box: `ccci-bridge_app` scaled to 0, both cc-ci timers
|
|
||||||
`mask --runtime`, cc-ci-orchestrator/loops units stopped (these do NOT survive a reboot — redo).
|
|
||||||
- Staged for loops: ~/.claude, opencode config+state, ssh keys, .testenv, upgrader.env,
|
|
||||||
.sops/master-age.txt, .cc-ci-logs; nginx oc-* files (root:nginx 0640).
|
|
||||||
- Open: tailscale auth key revoked (`invalid key: API key does not exist`) → operator issues a
|
|
||||||
new one. DNS cutover at Gandi (ci, *.ci, ns-acme → 195.201.88.249) → operator.
|
|
||||||
|
|
||||||
## 2026-09-07 22:15 UTC — first weekly upgrade run on the new host: GREEN, report published
|
|
||||||
|
|
||||||
Started by hand 21:23 UTC (`systemctl start cc-ci-upgrade-all` on 195.201.88.249, opencode /
|
|
||||||
deepseek-v4-flash); `UPGRADE RUN COMPLETE` 22:02 (39 min). Everything ran on the new host — old
|
|
||||||
server's Drone/bridge at 0/0, no new run dirs or report there. 20 recipes surveyed, 2 upgrade PRs
|
|
||||||
extended and `!testme` GREEN on the new Drone (lasuite-docs #8 → v5.6.1, build 1338; n8n #7 →
|
|
||||||
2.38.4, build 1339), 1 PR closed as merged upstream (custom-html #7), 18 skipped as up-to-date or
|
|
||||||
covered. Summary: `.cc-ci-logs/upgrades/upgrade-all-2026-09-07.md`. Report agent published
|
|
||||||
https://report.ci.commoninternet.net/week-2026-09-07.html (200, 42 KB, indexed) at 22:11.
|
|
||||||
One side effect: the run's orphan sweep removed the `opencode-ui` swarm stack (traefik route to
|
|
||||||
the opencode web UI) — redeployed, renamed `ccci-opencode-ui`, added to the sweep keep-list.
|
|
||||||
|
|
||||||
## 2026-09-08 02:05 UTC — cc-ci host auto-update + skills rewritten for the combined host
|
|
||||||
|
|
||||||
**Auto-update (own module, no notplants-nix dependency):** `nix/modules/auto-update.nix`,
|
|
||||||
`cc-ci-auto-update.timer` Tue 03:00 UTC. Busy gate (CI run / weekly upgrader / report / sweep /
|
|
||||||
running Drone builds) → `nix flake update` (all inputs) → build as loops → `switch-to-configuration
|
|
||||||
test` → 45 s settle → health check (sshd, 0 failed units, docker+acme-dns+fail2ban+nginx+
|
|
||||||
opencode-web+runner active, every swarm service at replica count, sops decrypted,
|
|
||||||
dashboard/reports/drone 200, opencode UI 401) → profile + bootloader → flake.lock committed as
|
|
||||||
`auto-update` and pushed to main → `/etc/cc-ci` fast-forwarded. Failure re-activates the previous
|
|
||||||
generation and restores the lock. One-line outcome in `.cc-ci-logs/auto-update-state`.
|
|
||||||
|
|
||||||
**Tested for real, both paths:**
|
|
||||||
- Full run 22:23–22:27 UTC: nixpkgs 20260803.531670d → 20260906.c257840 and sops-nix moved,
|
|
||||||
built, activated, `HEALTH OK`, committed (`d1f5e91`, pushed), running == boot generation.
|
|
||||||
- Busy gate: with a fake `cc-ci-upgrader` tmux session the unit exited 0 with
|
|
||||||
`result=skipped note=busy` and changed nothing.
|
|
||||||
|
|
||||||
**Skills (canonical in `.opencode/skills`, `.claude` pointer descriptions synced):**
|
|
||||||
- `/cc-ci-orchestrator-update` — THE host update; drives the auto-update unit by hand, has the
|
|
||||||
failure playbook and `--cc-ci-only` for a cc-ci-main-only move.
|
|
||||||
- `/cc-ci-server-update` — now delegates to it, and says why the old procedure (rebuilding the
|
|
||||||
cc-ci repo's standalone `#cc-ci`) must never run on this host.
|
|
||||||
- `/cc-ci-update` — chains orchestrator-update then tests-update.
|
|
||||||
- `/cc-ci-status` — §5/§6 rewritten for one host: auto-update state + staleness, running vs boot
|
|
||||||
generation, front doors (oc.ci must be 401), sops, fail2ban, all four timers, the orchestrator
|
|
||||||
agent session, and a secrets-inventory check that flags sprawl. Verdict updated.
|
|
||||||
Verified against the box: 0 failed units, 10/10 swarm services, disk 33%, all timers armed.
|
|
||||||
|
|
||||||
## 2026-09-08 16:55 UTC — cc-ci host moved to the operator's OpenCode **Go** key; extra credentials purged
|
|
||||||
|
|
||||||
**The key.** `AUTONOMIC_OPENCODE_KEY` (operator, in `/secrets/files/cc-ci.testenv`) is an OpenCode
|
|
||||||
**Go subscription** key, not a ZEN one. Installed into the ZEN slot it authenticates but every
|
|
||||||
request dies `Insufficient balance` (ZEN = pay-as-you-go credit); on the `opencode-go` endpoint the
|
|
||||||
same key answers immediately. Diagnosis was the operator's — worth remembering as the first thing
|
|
||||||
to check when an opencode key "does not work".
|
|
||||||
|
|
||||||
- `auth.json` (→ `/secrets/files/opencode-auth.json`) now holds **exactly one** provider,
|
|
||||||
`opencode-go`, carrying that key. The previous ZEN key is deleted; `opencode/glm-5.2` now errors.
|
|
||||||
- `agents.toml` orchestrator → `opencode-go/glm-5.2`; `upgrader.env` → `LOOP_TIER=go` and
|
|
||||||
LOOP/REPORT/SUPERVISOR models all `opencode-go/glm-5.2`; cc-ci PR #36 moved the subagent model in
|
|
||||||
`cc-ci/opencode.json` the same way.
|
|
||||||
- **Model note:** `opencode-go/deepseek-v4-flash` (what the weekly run used on ZEN) is China-hosted
|
|
||||||
on the Go tier and returns an opt-in error until the operator enables it at
|
|
||||||
`https://opencode.ai/workspace/wrk_01M20RVAY3D3B28XAYY1Z2Q6AZ/go`. `glm-5.2` is used meanwhile.
|
|
||||||
- Verified: `opencode-go/glm-5.2` answers, the orchestrator agent restarted on it, ZEN gone.
|
|
||||||
|
|
||||||
**Purged while there** (same "only what cc-ci strictly needs" rule): `auth.json` had also carried
|
|
||||||
`openai` (oauth), `opencode-go` (old) and `infomaniak` credentials copied from the notplants box,
|
|
||||||
and `~/.config/opencode/opencode.jsonc` carried a **Tinfoil API key inline in plaintext**. All
|
|
||||||
removed; the config is now a 4-line file with no `apiKey` anywhere. That Tinfoil key still lives in
|
|
||||||
the notplants-orchestrator config it came from — **operator: consider rotating it**, since it was
|
|
||||||
briefly present on a second host.
|
|
||||||
|
|
||||||
**Also:** `vim`/`vi` (absent, the operator hit it over ssh), `sqlite`, `bat`, `bc`, `moreutils`,
|
|
||||||
`pv`, `man-pages` added to the host toolbox with `EDITOR=vim`; on PATH for root and loops.
|
|
||||||
Stale `oc.commoninternet.net` URLs in the launchers → `oc.ci.commoninternet.net`.
|
|
||||||
## Session 2026-09-08 16:52 UTC — OpenCode GPT-5.6-luna startup supervision
|
|
||||||
|
|
||||||
**Left off:** Read the kickoff prompt, confirmed the host loop service is active, and performed the
|
|
||||||
required `RESUME_PHASE=1` relaunch; the completed phase sequence immediately stood down again.
|
|
||||||
|
|
||||||
**Phase / loop state:**
|
|
||||||
- Phase **ghost [15/15]**, `STATUS-ghost.md` is `## DONE`; the full phase sequence completed.
|
|
||||||
- `cc-ci-loops.service` is active (exited successfully); builder, adversary, and watchdog are
|
|
||||||
stopped because there is no remaining phase work.
|
|
||||||
- Reboot count remains **5**, latest logged reboot 2026-08-23 03:11 UTC.
|
|
||||||
|
|
||||||
**Open items:** none for loop supervision.
|
|
||||||
|
|
||||||
**Notes:** `cc-ci-plan/launch.py status` is not executable directly in this checkout, so the
|
|
||||||
equivalent `python3 cc-ci-plan/launch.py status` was used. The required launch was run via
|
|
||||||
`RESUME_PHASE=1 cc-ci-plan/launch.sh start`; it re-confirmed completion at 16:52:34 UTC.
|
|
||||||
## 2026-09-08 17:10 UTC — secrets audit of the cc-ci host (operator asked for a re-verify)
|
|
||||||
|
|
||||||
**Result: `/secrets` is exactly what it should be** — 6 files + the inventory README, every one of
|
|
||||||
them cross-checked as actually referenced by something on the host:
|
|
||||||
`cc-ci.testenv` (GITEA_PASSWORD, DOCKERHUB_*, AUTONOMIC_OPENCODE_KEY), `opencode-auth.json`
|
|
||||||
(one provider, `opencode-go`), and two ssh keypairs **generated on this host**
|
|
||||||
(`cc-ci-local-ed25519` → `ssh cc-ci`; `autonomic-bot-cc-ci-ed25519` → Gitea, key id 156).
|
|
||||||
|
|
||||||
**Proved absent, not just "removed":** the sops **master key** is gone and
|
|
||||||
`/var/lib/sops-nix/key.txt` is provably this host's OWN identity (its public half equals
|
|
||||||
`ssh-to-age` of the host ssh key: `age1tmvg…`). No tailscale key, no `.netrc`/`.git-credentials`/
|
|
||||||
`.aws`/`.npmrc`, no third-party model keys, no inline `apiKey` in any opencode config, and the old
|
|
||||||
ZEN key is absent from files, the opencode database and its logs.
|
|
||||||
|
|
||||||
**One real finding, fixed.** The agent transcripts I rsynced from the old orchestrator box
|
|
||||||
(`/srv/cc-ci/.cc-ci-logs/*.log`, ~145 MB) contained **live credential values in plaintext** — a
|
|
||||||
`tskey-auth-…` tailscale key and an older OpenCode ZEN key `sk-iLmK5…` — because earlier sessions
|
|
||||||
had pasted the values into logged `grep` commands while running their own secret scans. 133
|
|
||||||
occurrences across 7 logs, redacted in place (`<REDACTED-BY-SECRET-SWEEP>`); a whole-host rescan is
|
|
||||||
now clean. **Operator: rotate both of those** if either is still live — they sat on a second host.
|
|
||||||
Lesson for future sweeps: scanning `/secrets` is not enough, agent logs are a secret sink.
|
|
||||||
|
|
||||||
**Legitimately outside `/secrets`** (documented in its README so a later sweep does not "clean"
|
|
||||||
them): `/run/secrets/*`, `/root/.docker/config.json`, `/etc/nginx/oc-htpasswd`,
|
|
||||||
`/var/lib/ci-certs/live/*.pem`, `/var/lib/sops-nix/key.txt`, `/etc/ssh/ssh_host_*`.
|
|
||||||
|
|
||||||
## 2026-09-08 17:30 UTC — /secrets made authoritative (incl. ssh host keys); weekly-run models
|
|
||||||
|
|
||||||
**Operator rule:** every secret lives in `/secrets`; consumers read from there or symlink to it.
|
|
||||||
Implemented declaratively in `nix/hosts/cc-ci/configuration.nix` (cc-ci-orchestrator e7aa055):
|
|
||||||
`/secrets` `0711` with `files/` (loops), `host/` (root: ssh host keys + sops age identity) and
|
|
||||||
`nginx/` (root:nginx: the UI htpasswd). `find / -type f` for key material now returns **only**
|
|
||||||
`/secrets`; `/etc/ssh/ssh_host_*`, `~loops/.ssh/*`, `~loops/.local/share/opencode/auth.json` and
|
|
||||||
`/srv/cc-ci/.testenv` are all symlinks into it.
|
|
||||||
|
|
||||||
**The risky part, done safely.** The ed25519 host key is a sops recipient (`age1tmvg…`), so a
|
|
||||||
regenerated key would make every cc-ci secret undecryptable. Therefore: keys **copied** (never
|
|
||||||
moved-then-regenerated), the age identity verified identical before activation, `services.openssh.
|
|
||||||
hostKeys` and `sops.age.{sshKeyPaths,keyFile}` pointed at `/secrets/host` **directly** rather than
|
|
||||||
through the `/etc/ssh` symlinks (a dangling link would let sshd write a NEW key), `test` before
|
|
||||||
`switch`, and the proof was a fresh `StrictHostKeyChecking=yes` connection plus 8/8 sops secrets.
|
|
||||||
Fingerprint unchanged: `SHA256:r1H85Ybif…`. Then the duplicates outside `/secrets` were deleted
|
|
||||||
(`/var/lib/sops-nix/key.txt`, `/etc/nginx/oc-htpasswd`) and the unused ecdsa host key removed.
|
|
||||||
|
|
||||||
**Logs cleared** (operator: "we don't need them anymore"): 30 agent transcripts + 1 archived gz,
|
|
||||||
558 MB → 2 MB. Kept `upgrades/` (233 weekly records), `state/`, the `*-update-*.md` records and the
|
|
||||||
launcher dotfiles.
|
|
||||||
|
|
||||||
**Weekly-run models** — operator enabled China-hosted models on the workspace, so both verified on
|
|
||||||
the host: `opencode-go/deepseek-v4-flash` and `opencode-go/glm-5.3-flash` answer.
|
|
||||||
- **subagents** (per-recipe workers): `opencode-go/deepseek-v4-flash` — cc-ci PR #37, pulled into
|
|
||||||
both loops clones and `/etc/cc-ci`.
|
|
||||||
- **main agents** (weekly parent, report, hourly supervisor): `opencode-go/glm-5.3-flash` —
|
|
||||||
`upgrader.env` (`LOOP_TIER=go` maps to the `opencode-go` auth entry; `LOOP_MODEL` overrides the
|
|
||||||
tier default). Next fire Fri 2026-09-11 02:00 UTC.
|
|
||||||
- The steering orchestrator agent stays on `opencode-go/glm-5.2` (not asked to change).
|
|
||||||
|
|||||||
@@ -22,4 +22,3 @@ restarts the loops on boot. Count the lines below to see how often it's happenin
|
|||||||
auto-logging. Auto-logging is live from the next reboot onward.
|
auto-logging. Auto-logging is live from the next reboot onward.
|
||||||
- 2026-05-30 17:03:05 BST — reboot detected; loops auto-started by systemd (resuming phase index 6). boot_id=f565f752-0463-42db-b787-9e0db35a5e3f
|
- 2026-05-30 17:03:05 BST — reboot detected; loops auto-started by systemd (resuming phase index 6). boot_id=f565f752-0463-42db-b787-9e0db35a5e3f
|
||||||
- 2026-05-31 03:38:29 UTC — reboot detected; loops auto-started by systemd (resuming phase index 5). boot_id=51c17fc3-8391-4109-bce2-413fbee6f26d
|
- 2026-05-31 03:38:29 UTC — reboot detected; loops auto-started by systemd (resuming phase index 5). boot_id=51c17fc3-8391-4109-bce2-413fbee6f26d
|
||||||
- 2026-08-23 03:11:24 UTC — reboot detected; loops auto-started by systemd (resuming phase index 14). boot_id=8215fc5e-8032-4844-b2b8-e0f8c0d9b13e
|
|
||||||
|
|||||||
@@ -39,8 +39,6 @@ keeps landing in pass 2, the fix is a new deterministic method in pass 1. §4c i
|
|||||||
```
|
```
|
||||||
advisory-scan.py <recipe> [--from <version>] [--to <version>]
|
advisory-scan.py <recipe> [--from <version>] [--to <version>]
|
||||||
[--image <name>=<from>:<to>]... [--adjudicate] [--json] [--registry DIR]
|
[--image <name>=<from>:<to>]... [--adjudicate] [--json] [--registry DIR]
|
||||||
|
|
||||||
advisory-scan.py <recipe> --compose-to <URL> [--compose-from <URL>] # windows derived, not typed
|
|
||||||
```
|
```
|
||||||
|
|
||||||
| Input | Meaning |
|
| Input | Meaning |
|
||||||
@@ -48,8 +46,6 @@ advisory-scan.py <recipe> --compose-to <URL> [--compose-from <URL>] # windows
|
|||||||
| `<recipe>` | Recipe name; selects `cc-ci-plan/upstream/<recipe>.md` (the per-recipe URL registry) |
|
| `<recipe>` | Recipe name; selects `cc-ci-plan/upstream/<recipe>.md` (the per-recipe URL registry) |
|
||||||
| `--from` / `--to` | The **primary app image's** version window being upgraded across |
|
| `--from` / `--to` | The **primary app image's** version window being upgraded across |
|
||||||
| `--image NAME=FROM:TO` | A **sidecar image and the versions it moved between** (repeatable, all in ONE call). `NAME` is substring-matched against source repo names. Malformed values warn on stderr and are skipped. Without it that image's advisories stay unclassified. |
|
| `--image NAME=FROM:TO` | A **sidecar image and the versions it moved between** (repeatable, all in ONE call). `NAME` is substring-matched against source repo names. Malformed values warn on stderr and are skipped. Without it that image's advisories stay unclassified. |
|
||||||
| `--compose-to URL` | **Derive every window by diffing this compose against its baseline**, instead of typing `--from/--to/--image`. Point it at a PR's `compose.yml`. |
|
|
||||||
| `--compose-from URL` | Baseline for the above. Default: the same repo's **default branch, resolved from the API** — never assumed to be `main`. |
|
|
||||||
| `--adjudicate` | Run pass 2: append the evidence dossier for judgement |
|
| `--adjudicate` | Run pass 2: append the evidence dossier for judgement |
|
||||||
| `--registry` | Registry dir; also `CCCI_UPSTREAM_REGISTRY` |
|
| `--registry` | Registry dir; also `CCCI_UPSTREAM_REGISTRY` |
|
||||||
| `GITHUB_TOKEN` / `GITHUB_TOKEN_FILE` | Read-only token; **rate limit only** (60/hr anonymous → 5000/hr). Default file `/srv/cc-ci/.github-token`, mode 600. Public advisories need **no scopes**. |
|
| `GITHUB_TOKEN` / `GITHUB_TOKEN_FILE` | Read-only token; **rate limit only** (60/hr anonymous → 5000/hr). Default file `/srv/cc-ci/.github-token`, mode 600. Public advisories need **no scopes**. |
|
||||||
@@ -107,43 +103,10 @@ URLs containing `<`, `>`, `{`, `}`, `VERSION`, or `vX.Y.Z` are **skipped as temp
|
|||||||
human documentation (`…/changelog/v<VERSION>/`), not fetchable, and counting them as failures is wrong.
|
human documentation (`…/changelog/v<VERSION>/`), not fetchable, and counting them as failures is wrong.
|
||||||
|
|
||||||
This is the source that would have caught gitea: the vendor blog names both CVEs, the GitHub release
|
This is the source that would have caught gitea: the vendor blog names both CVEs, the GitHub release
|
||||||
page names neither.
|
page names neither. A CVE found **only** here carries no version data, so pass 1 cannot place it — it
|
||||||
|
goes to pass 2 (§6).
|
||||||
|
|
||||||
**When the page is a changelog organised by release, each CVE is attributed to the release heading it
|
### 2c. OSV.dev — supplementary
|
||||||
appears under** (`Changes with nginx 1.31.3`, `## v1.31.3`, …) and that becomes its fixed-in version.
|
|
||||||
Without this, a project that publishes no advisory feed can never contribute a CVE:
|
|
||||||
|
|
||||||
> **nginx publishes NO GitHub security advisories.** Every nginx CVE we can see comes from
|
|
||||||
> `nginx.org/en/CHANGES`. Scraping ids out of it without attributing them to a release left them with
|
|
||||||
> no patched version, so they were never classifiable — and every nginx bump in the fleet reported
|
|
||||||
> **0** forever. nginx is a sidecar in most recipes. Measured: `1.31.1 → 1.31.3` fixes **six** CVEs
|
|
||||||
> (three in .2, three in .3); lasuite-docs#7 went 0 → 6 and lasuite-drive#6 went 0 → 3 on this alone.
|
|
||||||
|
|
||||||
A changelog CVE is tied to a window by the **image name appearing in the page URL** (window `nginx` ↔
|
|
||||||
`nginx.org/...`). A CVE found on a vendor page with no attributable release still has no version data,
|
|
||||||
so pass 1 cannot place it — it goes to pass 2 (§6).
|
|
||||||
|
|
||||||
### 2c. NVD by CPE — the fallback for projects that publish nothing
|
|
||||||
|
|
||||||
Declared per recipe in the registry as `nvd-cpe: <image-key> = <cpe:2.3:...>`.
|
|
||||||
|
|
||||||
> **Why it exists.** Two recipes could not see CVEs *at all*: `mattermost-lts` (empty GitHub advisory
|
|
||||||
> feed, security bulletins rendered client-side so a text sweep finds nothing) and `mumble` (nothing
|
|
||||||
> published anywhere the registry points). Their scans returned `?` — nothing measured. NVD is
|
|
||||||
> CPE-indexed and carries structured ranges, so it answers where the vendor does not: mattermost
|
|
||||||
> 10.5.0 → 10.12.4 now scores **165**, and mumble finds `CVE-2025-71264` (fixed 1.6.870).
|
|
||||||
|
|
||||||
Two range forms, both used:
|
|
||||||
|
|
||||||
| NVD field | meaning | how it is judged |
|
|
||||||
|---|---|---|
|
|
||||||
| `versionEndExcluding X` | fixed in X exactly | a normal patched version (§4a) |
|
|
||||||
| `versionEndIncluding X` | affected **up to and including** X; fix version unpublished | fixed when the upgrade crosses X, i.e. `from ≤ X < to` |
|
|
||||||
|
|
||||||
**NVD lags the vendor** — it had neither gitea CVSS-9.8 RCE at publication — so this is a fallback,
|
|
||||||
never a replacement for 2a/2b. Unauthenticated calls are rate-limited (~5/30s), hence the retry.
|
|
||||||
|
|
||||||
### 2d. OSV.dev — supplementary
|
|
||||||
|
|
||||||
Only when the recipe has an entry in `OSV_PACKAGES` (ecosystem + package) and a version is given.
|
Only when the recipe has an entry in `OSV_PACKAGES` (ecosystem + package) and a version is given.
|
||||||
|
|
||||||
@@ -177,44 +140,8 @@ Two invariants govern this step, both learned from a wrong answer in production.
|
|||||||
> `null` / `UNKNOWN`, never `0`. A `0` in a security column asserts safety. Equally, an advisory that
|
> `null` / `UNKNOWN`, never `0`. A `0` in a security column asserts safety. Equally, an advisory that
|
||||||
> cannot be judged is **indeterminate** (§4d) — never silently counted as "not fixed".
|
> cannot be judged is **indeterminate** (§4d) — never silently counted as "not fixed".
|
||||||
|
|
||||||
### 3b. Deriving the windows from a compose diff (`--compose-to`)
|
|
||||||
|
|
||||||
Typing `--from/--to/--image` by hand means someone has to remember that the recipe also bumped its
|
|
||||||
redis. That is how sidecar CVEs went uncounted for months. This mode reads the windows off the diff:
|
|
||||||
|
|
||||||
1. Fetch both compose files (baseline = the repo's **default branch from the API**, since several
|
|
||||||
recipes keep a stale `main` beside a live `master`).
|
|
||||||
2. Parse `{service: (image-repo, tag)}` — keyed by **service, not image repo**, because an upgrade
|
|
||||||
may change the repo itself (plausible moved `plausible/analytics` →
|
|
||||||
`ghcr.io/plausible/community-edition`; keyed by repo that reads as one image vanishing and an
|
|
||||||
unrelated one appearing, losing the app window entirely).
|
|
||||||
3. Every service whose tag or repo changed becomes a window. The `app` service drives `--from/--to`
|
|
||||||
(coop-cloud convention: it is the recipe's primary image); the rest become `--image` windows.
|
|
||||||
Unchanged images produce no window — inventing one would be a false count.
|
|
||||||
4. The derived windows are printed to stderr before the scan, so the inputs are auditable.
|
|
||||||
|
|
||||||
Image names are matched against advisory sources **both ways** — an image name is often longer than
|
|
||||||
its source repo (`clickhouse/clickhouse-server` vs `ClickHouse/ClickHouse`) and sometimes shorter
|
|
||||||
(`redis` vs `redis/redis`).
|
|
||||||
|
|
||||||
Verified on plausible PR #5: from the compose URL alone it derives `v2.0.0 → v3.2.1` plus
|
|
||||||
`clickhouse-server 23.4.2.11-alpine → 24.12-alpine`, and reports **6** — identical to the
|
|
||||||
hand-specified args.
|
|
||||||
|
|
||||||
`--from/--to/--image` remain available for finer-grained checks (scanning a window that is not a
|
|
||||||
literal compose diff, e.g. "what would the compatibility-safe target fix?").
|
|
||||||
|
|
||||||
### 4a. By patched version (preferred — exact)
|
### 4a. By patched version (preferred — exact)
|
||||||
|
|
||||||
**A fix on the line you are upgrading FROM was already yours.** Projects that maintain several lines
|
|
||||||
patch them all at once: mattermost fixed `CVE-2025-11794` in 10.11.4, 10.12.1 *and* 10.5.12. An
|
|
||||||
upgrade 10.11.22 → 10.12.4 crosses 10.12.1, so a naive window test counts it — but 10.11.22 is
|
|
||||||
already past 10.11.4, so the deployment had the fix before the upgrade. Counting it credits the
|
|
||||||
upgrade with work it did not do. This check is **skipped for placeholder versions** (`7.4.X` parses
|
|
||||||
to a bare `7.4`, which would read as "already fixed at 7.4" and silently drop a real fix — exactly
|
|
||||||
how redis `CVE-2024-46981` was lost when the rule was first added).
|
|
||||||
|
|
||||||
|
|
||||||
`patched_versions` is a **range expression** (`">= 2.18.1"`), possibly several joined by `;`. Extract
|
`patched_versions` is a **range expression** (`">= 2.18.1"`), possibly several joined by `;`. Extract
|
||||||
every version-looking token; the advisory is **fixed-by-this-upgrade** if **any** patched version `p`
|
every version-looking token; the advisory is **fixed-by-this-upgrade** if **any** patched version `p`
|
||||||
satisfies `from < p <= to` — exclusive lower (a fix already in the version you were on is not this
|
satisfies `from < p <= to` — exclusive lower (a fix already in the version you were on is not this
|
||||||
|
|||||||
+12
-401
@@ -44,9 +44,7 @@ import json
|
|||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
import sys
|
import sys
|
||||||
import time
|
|
||||||
import urllib.error
|
import urllib.error
|
||||||
import urllib.parse
|
|
||||||
import urllib.request
|
import urllib.request
|
||||||
|
|
||||||
REGISTRY_DIR = os.environ.get("CCCI_UPSTREAM_REGISTRY", "/srv/cc-ci/cc-ci-plan/upstream")
|
REGISTRY_DIR = os.environ.get("CCCI_UPSTREAM_REGISTRY", "/srv/cc-ci/cc-ci-plan/upstream")
|
||||||
@@ -166,42 +164,6 @@ def _vkey(v: str | None) -> tuple:
|
|||||||
return tuple(out)
|
return tuple(out)
|
||||||
|
|
||||||
|
|
||||||
def _already_fixed_on_from_line(kf: tuple, cands: list[tuple]) -> bool:
|
|
||||||
"""Was it ALREADY fixed on the line we are upgrading FROM?
|
|
||||||
|
|
||||||
The mirror image of _superseded_on_target_line, and just as necessary. mattermost fixes each CVE
|
|
||||||
across several maintained lines at once — CVE-2025-11794 is patched in 10.11.4, 10.12.1 and
|
|
||||||
10.5.12. Upgrading 10.11.22 -> 10.12.4 crosses 10.12.1, so a naive window test counts it; but
|
|
||||||
10.11.22 is already past 10.11.4, so the deployment HAD the fix before the upgrade. Counting it
|
|
||||||
credits the upgrade with work it did not do."""
|
|
||||||
if len(kf) < 2:
|
|
||||||
return False
|
|
||||||
line = kf[:2]
|
|
||||||
for c in cands:
|
|
||||||
if len(c) < 2 or c[:2] != line:
|
|
||||||
continue
|
|
||||||
n = max(len(kf), len(c))
|
|
||||||
pad = lambda z: z + (0,) * (n - len(z))
|
|
||||||
if pad(c) <= pad(kf):
|
|
||||||
return True
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def _superseded_on_target_line(kt: tuple, cands: list[tuple]) -> bool:
|
|
||||||
"""Does a patched version on the TARGET's own release line sit ABOVE the target?
|
|
||||||
|
|
||||||
Projects maintain several branches at once and backport per branch, so "some patched version is
|
|
||||||
inside the numeric window" is not the same as "the version we land on has the fix". ClickHouse
|
|
||||||
fixed CVE-2023-48704 in 23.9.6.20 AND 23.10.5.20; an upgrade landing on 23.10.4.25 crosses the
|
|
||||||
23.9 fix numerically but is still BELOW its own line's fix, so it does NOT have it. When the
|
|
||||||
advisory names a fix on the target's own line and the target is older than it, that is proof of
|
|
||||||
absence and outranks any other candidate."""
|
|
||||||
if len(kt) < 2:
|
|
||||||
return False
|
|
||||||
line = kt[:2]
|
|
||||||
return any(c[:2] == line and c > kt for c in cands if len(c) >= 2)
|
|
||||||
|
|
||||||
|
|
||||||
def _within(kf: tuple, kt: tuple, c: tuple) -> bool:
|
def _within(kf: tuple, kt: tuple, c: tuple) -> bool:
|
||||||
"""Is patched-version `c` inside the window (kf, kt] — exclusive lower, inclusive upper?
|
"""Is patched-version `c` inside the window (kf, kt] — exclusive lower, inclusive upper?
|
||||||
|
|
||||||
@@ -295,51 +257,6 @@ def github_advisories(urls: list[str]) -> list[dict]:
|
|||||||
return results
|
return results
|
||||||
|
|
||||||
|
|
||||||
# Release headings in a vendor changelog. nginx's CHANGES uses "Changes with nginx 1.31.3", most
|
|
||||||
# markdown changelogs use "## 1.31.3" / "## v1.31.3".
|
|
||||||
_HEADING_RE = re.compile(
|
|
||||||
r"^\s*(?:#{1,4}\s*)?(?:Changes with\s+\S+\s+|Version\s+|Release\s+)?v?(\d+\.\d+(?:\.\d+)*)\s*$"
|
|
||||||
r"|^\s*Changes with\s+\S+\s+(\d+\.\d+(?:\.\d+)*)", re.I)
|
|
||||||
|
|
||||||
|
|
||||||
def _changelog_versions(text: str) -> dict:
|
|
||||||
"""{cve: version} for a changelog that is ORGANISED BY RELEASE.
|
|
||||||
|
|
||||||
Why this exists: nginx publishes NO GitHub security advisories. Every nginx CVE we can see comes
|
|
||||||
from nginx.org/en/CHANGES, and scraping ids out of it without attributing them to a release
|
|
||||||
leaves them with no patched version — so they can never be classified, and an nginx bump reports
|
|
||||||
0 CVEs forever. nginx 1.31.1 -> 1.31.3 in fact fixes SIX (three in .2, three in .3), and nginx is
|
|
||||||
a sidecar in most of the fleet, so that was a fleet-wide blind spot.
|
|
||||||
|
|
||||||
Attributes each CVE to the nearest PRECEDING release heading — the release that fixed it.
|
|
||||||
"""
|
|
||||||
plain = re.sub(r"<[^>]+>", " ", text)
|
|
||||||
out, cur = {}, None
|
|
||||||
for line in plain.splitlines():
|
|
||||||
m = _HEADING_RE.match(line)
|
|
||||||
if m:
|
|
||||||
cur = m.group(1) or m.group(2)
|
|
||||||
continue
|
|
||||||
if cur:
|
|
||||||
for cve in CVE_RE.findall(line):
|
|
||||||
out.setdefault(cve, cur)
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
_BLOB_RE = re.compile(r"^https://github\.com/([^/]+)/([^/]+)/blob/(.+)$")
|
|
||||||
|
|
||||||
|
|
||||||
def _raw_if_blob(url: str) -> str:
|
|
||||||
"""A GitHub *blob* URL is an HTML viewer, not the file.
|
|
||||||
|
|
||||||
The registry pointed ONLYOFFICE's CHANGELOG.md at its blob page. Fetching that returns 636KB of
|
|
||||||
markup in which the release headings do not survive HTML-stripping, so 24 CVEs were visible and
|
|
||||||
NONE attributable to a release — the same shape of blind spot as nginx. The raw URL attributes
|
|
||||||
all 24. Normalising here fixes every registry entry at once, present and future."""
|
|
||||||
m = _BLOB_RE.match(url)
|
|
||||||
return f"https://raw.githubusercontent.com/{m.group(1)}/{m.group(2)}/{m.group(3)}" if m else url
|
|
||||||
|
|
||||||
|
|
||||||
def vendor_pages(urls: list[str]) -> list[dict]:
|
def vendor_pages(urls: list[str]) -> list[dict]:
|
||||||
"""Fetch each registry URL and regex out CVE ids, with a little surrounding context."""
|
"""Fetch each registry URL and regex out CVE ids, with a little surrounding context."""
|
||||||
out = []
|
out = []
|
||||||
@@ -352,93 +269,20 @@ def vendor_pages(urls: list[str]) -> list[dict]:
|
|||||||
# correct; counting them as failures would wrongly mark the recipe's count unreliable.
|
# correct; counting them as failures would wrongly mark the recipe's count unreliable.
|
||||||
out.append({"source": u, "status": "skipped: template URL (not fetchable)", "cves": [], "context": {}})
|
out.append({"source": u, "status": "skipped: template URL (not fetchable)", "cves": [], "context": {}})
|
||||||
continue
|
continue
|
||||||
entry = {"source": u, "status": "ok", "cves": [], "context": {}, "fixed_in": {}}
|
entry = {"source": u, "status": "ok", "cves": [], "context": {}}
|
||||||
try:
|
try:
|
||||||
text = _fetch(_raw_if_blob(u))
|
text = _fetch(u)
|
||||||
plain = re.sub(r"<[^>]+>", " ", text)
|
plain = re.sub(r"<[^>]+>", " ", text)
|
||||||
for cve in sorted(set(CVE_RE.findall(plain))):
|
for cve in sorted(set(CVE_RE.findall(plain))):
|
||||||
entry["cves"].append(cve)
|
entry["cves"].append(cve)
|
||||||
i = plain.find(cve)
|
i = plain.find(cve)
|
||||||
entry["context"][cve] = re.sub(r"\s+", " ", plain[max(0, i - 160) : i + 200]).strip()
|
entry["context"][cve] = re.sub(r"\s+", " ", plain[max(0, i - 160) : i + 200]).strip()
|
||||||
entry["fixed_in"] = _changelog_versions(text)
|
|
||||||
except Exception as e: # noqa: BLE001
|
except Exception as e: # noqa: BLE001
|
||||||
entry["status"] = f"error: {type(e).__name__}: {e}"
|
entry["status"] = f"error: {type(e).__name__}: {e}"
|
||||||
out.append(entry)
|
out.append(entry)
|
||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
NVD_API = "https://services.nvd.nist.gov/rest/json/cves/2.0"
|
|
||||||
NVD_CPE_RE = re.compile(r"^\s*[-*]?\s*nvd-cpe:\s*(\S+)\s*=\s*(cpe:2\.3:[^\s`]+)", re.M | re.I)
|
|
||||||
|
|
||||||
|
|
||||||
def registry_cpes(recipe: str, registry_dir: str) -> list[tuple[str, str]]:
|
|
||||||
"""[(image-key, cpe)] declared in the recipe's registry as `nvd-cpe: <key> = <cpe>`."""
|
|
||||||
path = os.path.join(registry_dir, f"{recipe}.md")
|
|
||||||
try:
|
|
||||||
return [(m.group(1), m.group(2)) for m in NVD_CPE_RE.finditer(open(path).read())]
|
|
||||||
except OSError:
|
|
||||||
return []
|
|
||||||
|
|
||||||
|
|
||||||
def nvd_advisories(cpe: str, key: str) -> dict:
|
|
||||||
"""CVEs for a CPE from NVD, with the version data the classifier needs.
|
|
||||||
|
|
||||||
THE FALLBACK FOR PROJECTS THAT PUBLISH NOTHING MACHINE-READABLE. mattermost's GitHub advisory
|
|
||||||
feed is empty and its security bulletins are client-side rendered; mumble publishes neither. Both
|
|
||||||
scanned as `?` — nothing measured — until here. NVD is CPE-indexed and carries structured ranges:
|
|
||||||
|
|
||||||
versionEndExcluding X -> fixed in X exactly (a patched version)
|
|
||||||
versionEndIncluding X -> affected up to and INCLUDING X, fixed in some later release. The
|
|
||||||
exact fix version is unknown, but the upgrade fixes it whenever it
|
|
||||||
crosses X — recorded as `affected_max` and judged in the classifier.
|
|
||||||
|
|
||||||
NVD LAGS the vendor (it had neither gitea CVSS-9.8 RCE at publication), so this is a fallback,
|
|
||||||
never a replacement for 2a/2b. Unauthenticated calls are rate-limited to ~5/30s, hence the retry.
|
|
||||||
"""
|
|
||||||
entry = {"source": f"nvd:{key}", "status": "ok", "advisories": []}
|
|
||||||
url = f"{NVD_API}?resultsPerPage=2000&virtualMatchString={urllib.parse.quote(cpe)}"
|
|
||||||
data = None
|
|
||||||
for attempt in range(3):
|
|
||||||
try:
|
|
||||||
data = json.loads(_fetch(url))
|
|
||||||
break
|
|
||||||
except Exception as e: # noqa: BLE001
|
|
||||||
if attempt == 2:
|
|
||||||
entry["status"] = f"error: {type(e).__name__}"
|
|
||||||
return entry
|
|
||||||
time.sleep(8)
|
|
||||||
for v in (data or {}).get("vulnerabilities", []):
|
|
||||||
c = v.get("cve") or {}
|
|
||||||
cid = c.get("id")
|
|
||||||
if not cid:
|
|
||||||
continue
|
|
||||||
fixed, affected_max = set(), set()
|
|
||||||
for cfg in c.get("configurations", []):
|
|
||||||
for node in cfg.get("nodes", []):
|
|
||||||
for m in node.get("cpeMatch", []):
|
|
||||||
if m.get("versionEndExcluding"):
|
|
||||||
fixed.add(m["versionEndExcluding"])
|
|
||||||
elif m.get("versionEndIncluding"):
|
|
||||||
affected_max.add(m["versionEndIncluding"])
|
|
||||||
sev = None
|
|
||||||
for mk in ("cvssMetricV31", "cvssMetricV30", "cvssMetricV2"):
|
|
||||||
got = (c.get("metrics") or {}).get(mk) or []
|
|
||||||
if got:
|
|
||||||
sev = (got[0].get("cvssData") or {}).get("baseSeverity")
|
|
||||||
break
|
|
||||||
entry["advisories"].append({
|
|
||||||
"cve": cid, "ghsa": None, "severity": (sev or "").lower() or None,
|
|
||||||
"summary": next((d.get("value") for d in c.get("descriptions", [])
|
|
||||||
if d.get("lang") == "en"), "")[:200],
|
|
||||||
"vulnerable_range": None,
|
|
||||||
"patched": "; ".join(sorted(fixed)) or None,
|
|
||||||
"affected_max": "; ".join(sorted(affected_max)) or None,
|
|
||||||
"url": f"https://nvd.nist.gov/vuln/detail/{cid}",
|
|
||||||
"published_at": c.get("published"), "description": None, "cvss": None,
|
|
||||||
})
|
|
||||||
return entry
|
|
||||||
|
|
||||||
|
|
||||||
def osv(recipe: str, version: str | None) -> dict | None:
|
def osv(recipe: str, version: str | None) -> dict | None:
|
||||||
pkg = OSV_PACKAGES.get(recipe)
|
pkg = OSV_PACKAGES.get(recipe)
|
||||||
if not pkg or not version:
|
if not pkg or not version:
|
||||||
@@ -736,8 +580,7 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
|
|||||||
e = report["cves"].setdefault(cve, {"sources": [], "severity": None, "ghsa": None,
|
e = report["cves"].setdefault(cve, {"sources": [], "severity": None, "ghsa": None,
|
||||||
"vulnerable_range": None, "patched": None,
|
"vulnerable_range": None, "patched": None,
|
||||||
"context": None, "published_at": None,
|
"context": None, "published_at": None,
|
||||||
"description": None, "url": None, "cvss": None,
|
"description": None, "url": None, "cvss": None})
|
||||||
"changelog_fixed_in": None, "affected_max": None})
|
|
||||||
if src not in e["sources"]:
|
if src not in e["sources"]:
|
||||||
e["sources"].append(src)
|
e["sources"].append(src)
|
||||||
for k, v in extra.items():
|
for k, v in extra.items():
|
||||||
@@ -754,22 +597,11 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
|
|||||||
context=a.get("summary"), published_at=a.get("published_at"),
|
context=a.get("summary"), published_at=a.get("published_at"),
|
||||||
description=a.get("description"), url=a.get("url"), cvss=a.get("cvss"))
|
description=a.get("description"), url=a.get("url"), cvss=a.get("cvss"))
|
||||||
|
|
||||||
for key, cpe in registry_cpes(recipe, registry_dir):
|
|
||||||
entry = nvd_advisories(cpe, key)
|
|
||||||
report["sources"].append({"source": entry["source"], "status": entry["status"],
|
|
||||||
"found": len(entry.get("advisories") or [])})
|
|
||||||
for a in entry.get("advisories", []):
|
|
||||||
record(a["cve"], entry["source"], severity=a.get("severity"),
|
|
||||||
patched=a.get("patched"), affected_max=a.get("affected_max"),
|
|
||||||
context=a.get("summary"), published_at=a.get("published_at"),
|
|
||||||
url=a.get("url"))
|
|
||||||
|
|
||||||
for entry in vendor_pages(urls):
|
for entry in vendor_pages(urls):
|
||||||
report["sources"].append({"source": entry["source"], "status": entry["status"],
|
report["sources"].append({"source": entry["source"], "status": entry["status"],
|
||||||
"found": len(entry.get("cves", []))})
|
"found": len(entry.get("cves", []))})
|
||||||
for cve in entry.get("cves", []):
|
for cve in entry.get("cves", []):
|
||||||
record(cve, entry["source"], context=entry["context"].get(cve),
|
record(cve, entry["source"], context=entry["context"].get(cve))
|
||||||
changelog_fixed_in=(entry.get("fixed_in") or {}).get(cve))
|
|
||||||
|
|
||||||
for version in filter(None, (v_from, v_to)):
|
for version in filter(None, (v_from, v_to)):
|
||||||
o = osv(recipe, version)
|
o = osv(recipe, version)
|
||||||
@@ -804,39 +636,17 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
|
|||||||
#
|
#
|
||||||
# A source with no window is not classified: its advisories are listed as unclassified so they
|
# A source with no window is not classified: its advisories are listed as unclassified so they
|
||||||
# stay visible without inflating the count.
|
# stay visible without inflating the count.
|
||||||
gh_sources = [x["source"] for x in report["sources"]
|
gh_sources = [x["source"] for x in report["sources"] if x["source"].startswith("github-advisories:")]
|
||||||
if x["source"].startswith(("github-advisories:", "nvd:"))]
|
|
||||||
primary = gh_sources[0] if (gh_sources and (v_from or v_to)) else None
|
primary = gh_sources[0] if (gh_sources and (v_from or v_to)) else None
|
||||||
report["primary_source"] = primary
|
report["primary_source"] = primary
|
||||||
|
|
||||||
windows = {} # source name -> (from, to)
|
windows = {} # source name -> (from, to)
|
||||||
window_key = {} # source name -> the image name it covers
|
|
||||||
if primary:
|
if primary:
|
||||||
windows[primary] = (v_from, v_to)
|
windows[primary] = (v_from, v_to)
|
||||||
window_key[primary] = primary.split("/")[-1]
|
|
||||||
# The app's window must also cover its NVD entry. NVD sources are keyed by IMAGE name
|
|
||||||
# (`mattermost-team-edition`) while the advisory feed is keyed by REPO (`mattermost/
|
|
||||||
# mattermost`), so without this the fallback source that exists precisely because the feed
|
|
||||||
# is empty would itself go unwindowed — and mumble/mattermost would still report nothing.
|
|
||||||
pname = primary.split("/")[-1].lower()
|
|
||||||
for src in gh_sources:
|
|
||||||
if src.startswith("nvd:") and src not in windows:
|
|
||||||
k = src.split(":", 1)[1].lower()
|
|
||||||
if pname in k or k in pname:
|
|
||||||
windows[src] = (v_from, v_to)
|
|
||||||
window_key[src] = k
|
|
||||||
for key, wf, wt in (images or []):
|
for key, wf, wt in (images or []):
|
||||||
for src in gh_sources:
|
for src in gh_sources:
|
||||||
if src in windows:
|
if key.lower() in src.lower() and src not in windows:
|
||||||
continue
|
|
||||||
# Match BOTH ways: an image name is often longer than its source repo
|
|
||||||
# (`clickhouse/clickhouse-server` vs source `ClickHouse/ClickHouse`) and sometimes
|
|
||||||
# shorter (`redis` vs `redis/redis`). One-directional matching silently dropped the
|
|
||||||
# clickhouse window when the key was derived from a compose file.
|
|
||||||
k, name = key.lower(), src.split("/")[-1].lower()
|
|
||||||
if k in src.lower() or name in k:
|
|
||||||
windows[src] = (wf, wt)
|
windows[src] = (wf, wt)
|
||||||
window_key[src] = key
|
|
||||||
report["windows"] = {k: {"from": f, "to": t} for k, (f, t) in windows.items()}
|
report["windows"] = {k: {"from": f, "to": t} for k, (f, t) in windows.items()}
|
||||||
|
|
||||||
def _classify_window(src, wf, wt):
|
def _classify_window(src, wf, wt):
|
||||||
@@ -855,31 +665,8 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
|
|||||||
continue
|
continue
|
||||||
patched = e.get("patched") or ""
|
patched = e.get("patched") or ""
|
||||||
cands = [_vkey(t) for t in re.findall(r"\d+(?:\.\d+)*", patched)]
|
cands = [_vkey(t) for t in re.findall(r"\d+(?:\.\d+)*", patched)]
|
||||||
# NEVER on a placeholder: "7.4.X" parses to the bare 7.4, which then reads as
|
|
||||||
# "already fixed at 7.4" and silently drops a real fix (redis CVE-2024-46981).
|
|
||||||
# A placeholder means the fix version is unknown — that is the indeterminate path.
|
|
||||||
if (kf and kt and not PLACEHOLDER_RE.search(patched)
|
|
||||||
and _already_fixed_on_from_line(kf, cands)):
|
|
||||||
# already had it before the upgrade
|
|
||||||
e.setdefault("classification", "outside-window")
|
|
||||||
continue
|
|
||||||
if kf and kt and _superseded_on_target_line(kt, cands):
|
|
||||||
# The target's own line got the fix LATER than the target: not fixed here.
|
|
||||||
e.setdefault("classification", "outside-window")
|
|
||||||
continue
|
|
||||||
if kf and kt and any(_within(kf, kt, c) for c in cands):
|
if kf and kt and any(_within(kf, kt, c) for c in cands):
|
||||||
got.add(cve)
|
got.add(cve)
|
||||||
elif kf and kt and e.get("affected_max"):
|
|
||||||
# NVD's `versionEndIncluding X`: affected up to and INCLUDING X, fixed in some
|
|
||||||
# later release. The exact fix version is unpublished, but the upgrade delivers
|
|
||||||
# it whenever it crosses X — i.e. from <= X < to.
|
|
||||||
for t in re.findall(r"\d+(?:\.\d+)*", e["affected_max"]):
|
|
||||||
x = _vkey(t)
|
|
||||||
n = max(len(kf), len(kt), len(x))
|
|
||||||
pad = lambda z: z + (0,) * (n - len(z))
|
|
||||||
if x and pad(kf) <= pad(x) < pad(kt):
|
|
||||||
got.add(cve)
|
|
||||||
break
|
|
||||||
elif not patched or PLACEHOLDER_RE.search(patched):
|
elif not patched or PLACEHOLDER_RE.search(patched):
|
||||||
# No fix version published ("TBD") or only a placeholder ("7.4.X" — which could
|
# No fix version published ("TBD") or only a placeholder ("7.4.X" — which could
|
||||||
# be 7.4.1, inside the window). We cannot say either way, so say so.
|
# be 7.4.1, inside the window). We cannot say either way, so say so.
|
||||||
@@ -913,32 +700,6 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
|
|||||||
report["cves"][cve]["classification"] = f"fixed-by-this-upgrade ({method}) via {src}"
|
report["cves"][cve]["classification"] = f"fixed-by-this-upgrade ({method}) via {src}"
|
||||||
fixed_set.add(cve)
|
fixed_set.add(cve)
|
||||||
|
|
||||||
# A CVE seen only in a vendor CHANGELOG has no advisory feed behind it, but the changelog says
|
|
||||||
# which release fixed it (see _changelog_versions). Tie it to a window by the image name
|
|
||||||
# appearing in the page URL — nginx's window is `nginx`, and its changelog is nginx.org/... .
|
|
||||||
# Without this, projects that publish no GitHub advisories (nginx being the big one) can never
|
|
||||||
# contribute a CVE, and every nginx bump in the fleet silently reports 0.
|
|
||||||
from_changelog = {}
|
|
||||||
for cve, e in report["cves"].items():
|
|
||||||
if cve in fixed_set or not e.get("changelog_fixed_in"):
|
|
||||||
continue
|
|
||||||
for src, (wf, wt) in windows.items():
|
|
||||||
key = (window_key.get(src) or "").lower()
|
|
||||||
if not key:
|
|
||||||
continue
|
|
||||||
if not any(key in s_.lower() for s_ in e["sources"] if s_.startswith("http")):
|
|
||||||
continue
|
|
||||||
kf, kt = _vkey(wf), _vkey(wt)
|
|
||||||
cand = _vkey(e["changelog_fixed_in"])
|
|
||||||
if kf and kt and cand and _within(kf, kt, cand):
|
|
||||||
e["classification"] = (f"fixed-by-this-upgrade (named under {e['changelog_fixed_in']} "
|
|
||||||
f"in the vendor changelog) via {src}")
|
|
||||||
fixed_set.add(cve)
|
|
||||||
from_changelog[cve] = e["changelog_fixed_in"]
|
|
||||||
break
|
|
||||||
if from_changelog:
|
|
||||||
report["resolved_by_changelog"] = from_changelog
|
|
||||||
|
|
||||||
unknown = []
|
unknown = []
|
||||||
for cve, e in report["cves"].items():
|
for cve, e in report["cves"].items():
|
||||||
if cve in fixed_set:
|
if cve in fixed_set:
|
||||||
@@ -1017,19 +778,8 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
|
|||||||
report["unclassified"] = sorted(unknown)
|
report["unclassified"] = sorted(unknown)
|
||||||
# NEVER report 0 for something we could not determine — a 0 asserts safety. If ANY requested
|
# NEVER report 0 for something we could not determine — a 0 asserts safety. If ANY requested
|
||||||
# window could not be ordered at all, the total is UNKNOWN rather than a partial number.
|
# window could not be ordered at all, the total is UNKNOWN rather than a partial number.
|
||||||
# A scan with NO usable source has not measured anything, so it must not report a number —
|
report["count_known"] = not unresolved_any
|
||||||
# least of all 0, which asserts safety. mumble had no cc-ci-plan/upstream/mumble.md at all and
|
report["cve_count_fixed"] = len(fixed_set) if not unresolved_any else None
|
||||||
# still produced "0 identified", which was then published as a clean 0 in a CVE report.
|
|
||||||
usable_sources = [
|
|
||||||
s for s in report["sources"]
|
|
||||||
if s["status"] == "ok" or s["status"].startswith("no-advisories-published")
|
|
||||||
]
|
|
||||||
no_sources = not usable_sources
|
|
||||||
if no_sources:
|
|
||||||
report["no_usable_sources"] = True
|
|
||||||
report["count_known"] = not unresolved_any and not no_sources
|
|
||||||
report["cve_count_fixed"] = (len(fixed_set)
|
|
||||||
if (not unresolved_any and not no_sources) else None)
|
|
||||||
report["cve_count_total_seen"] = len(report["cves"])
|
report["cve_count_total_seen"] = len(report["cves"])
|
||||||
# Only GENUINE failures make a count unreliable. "no-advisories-published" (404: the repo has
|
# Only GENUINE failures make a count unreliable. "no-advisories-published" (404: the repo has
|
||||||
# no advisory feed) and "skipped: template URL" are benign and must not degrade the verdict.
|
# no advisory feed) and "skipped: template URL" are benign and must not degrade the verdict.
|
||||||
@@ -1052,16 +802,10 @@ def markdown(rep: dict) -> str:
|
|||||||
f"{rep.get('from') or '?'} → {rep.get('to') or '?'}"]
|
f"{rep.get('from') or '?'} → {rep.get('to') or '?'}"]
|
||||||
if not rep.get("count_known", True):
|
if not rep.get("count_known", True):
|
||||||
L.append("\n**CVEs fixed by this upgrade: UNKNOWN — the scan could NOT determine a count.**")
|
L.append("\n**CVEs fixed by this upgrade: UNKNOWN — the scan could NOT determine a count.**")
|
||||||
if rep.get("no_usable_sources"):
|
L.append("\n⚠ This is NOT zero. A version-scheme change (e.g. semver → calver) makes numeric "
|
||||||
L.append("\n⚠ This is NOT zero. **No usable source was checked at all** — the registry "
|
"ordering meaningless across this jump, so no advisory could be classified. Render "
|
||||||
"file `cc-ci-plan/upstream/<recipe>.md` is missing or every source failed, so "
|
"this recipe's cve cell as `?`, never `0`. Read the vendor's release notes for the "
|
||||||
"nothing was measured. Render this recipe's cve cell as `?`, never `0`, and add "
|
"jump and count by hand.")
|
||||||
"the registry file.")
|
|
||||||
else:
|
|
||||||
L.append("\n⚠ This is NOT zero. A version-scheme change (e.g. semver → calver) makes "
|
|
||||||
"numeric ordering meaningless across this jump, so no advisory could be "
|
|
||||||
"classified. Render this recipe's cve cell as `?`, never `0`. Read the vendor's "
|
|
||||||
"release notes for the jump and count by hand.")
|
|
||||||
if rep["unclassified"]:
|
if rep["unclassified"]:
|
||||||
L.append(f"\nAdvisories seen but unclassifiable ({len(rep['unclassified'])}) — includes "
|
L.append(f"\nAdvisories seen but unclassifiable ({len(rep['unclassified'])}) — includes "
|
||||||
f"other images in this recipe: " + ", ".join(rep["unclassified"][:12]))
|
f"other images in this recipe: " + ", ".join(rep["unclassified"][:12]))
|
||||||
@@ -1113,118 +857,6 @@ def markdown(rep: dict) -> str:
|
|||||||
return "\n".join(L)
|
return "\n".join(L)
|
||||||
|
|
||||||
|
|
||||||
def _gitea_auth(url: str) -> dict:
|
|
||||||
"""Basic auth for the private mirror, using public config plus .testenv.
|
|
||||||
|
|
||||||
Sent as a HEADER, never embedded in the URL: in-URL credentials leak into shell history, process
|
|
||||||
lists and error messages, and urllib mis-parses a password containing a colon."""
|
|
||||||
host = re.sub(r"^https?://", "", url).split("/")[0]
|
|
||||||
env = {}
|
|
||||||
for path in (
|
|
||||||
"/srv/cc-ci/cc-ci/.env.public",
|
|
||||||
os.environ.get("CCCI_TESTENV", "/srv/cc-ci/.testenv"),
|
|
||||||
):
|
|
||||||
try:
|
|
||||||
lines = open(path)
|
|
||||||
except OSError:
|
|
||||||
continue
|
|
||||||
for ln in lines:
|
|
||||||
if "=" in ln and not ln.strip().startswith("#"):
|
|
||||||
k, v = ln.strip().split("=", 1)
|
|
||||||
env[k] = v.strip().strip("\"'")
|
|
||||||
if host != env.get("GITEA_URL", "git.autonomic.zone"):
|
|
||||||
return {}
|
|
||||||
u, pw = env.get("GITEA_USERNAME"), env.get("GITEA_PASSWORD")
|
|
||||||
if not (u and pw):
|
|
||||||
return {}
|
|
||||||
import base64 as _b64
|
|
||||||
return {"Authorization": "Basic " + _b64.b64encode(f"{u}:{pw}".encode()).decode()}
|
|
||||||
|
|
||||||
|
|
||||||
def _compose_images(url: str) -> dict[str, tuple[str, str]]:
|
|
||||||
"""{service: (image-repo, tag)} for a compose file.
|
|
||||||
|
|
||||||
Keyed by SERVICE, not by image repo, because an upgrade may change the repo itself: plausible
|
|
||||||
moved `plausible/analytics` -> `ghcr.io/plausible/community-edition`. Keyed by repo that reads
|
|
||||||
as one image vanishing and an unrelated one appearing, and the app's version window is lost —
|
|
||||||
which is exactly the upgrade most worth scanning."""
|
|
||||||
txt = _fetch(url, _gitea_auth(url))
|
|
||||||
out, svc = {}, None
|
|
||||||
in_services = False
|
|
||||||
for line in txt.splitlines():
|
|
||||||
if re.match(r"^services:\s*$", line):
|
|
||||||
in_services = True
|
|
||||||
continue
|
|
||||||
if in_services and re.match(r"^\S", line):
|
|
||||||
in_services = False
|
|
||||||
if not in_services:
|
|
||||||
continue
|
|
||||||
m = re.match(r"^ (\S+):\s*$", line)
|
|
||||||
if m:
|
|
||||||
svc = m.group(1)
|
|
||||||
continue
|
|
||||||
m = re.match(r"^\s+image:\s*[\"']?([^\"'\s]+)", line)
|
|
||||||
if m and svc:
|
|
||||||
ref = m.group(1).split("@", 1)[0]
|
|
||||||
if "${" in ref or "$(" in ref:
|
|
||||||
continue
|
|
||||||
repo, _, tag = ref.rpartition(":")
|
|
||||||
if repo and tag:
|
|
||||||
out[svc] = (repo, tag)
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def _default_branch_compose(url: str) -> str | None:
|
|
||||||
"""Same repo as `url`, but its DEFAULT branch — resolved from the API, never assumed.
|
|
||||||
|
|
||||||
Several coopcloud recipes keep a stale `main` beside the real default `master` (gitea's `main`
|
|
||||||
is 1.24.2-rootless while `master` has 1.27.1-rootless), so guessing the branch produces a
|
|
||||||
confidently wrong baseline."""
|
|
||||||
m = re.match(r"(https?://[^/]+)/([^/]+)/([^/]+)/(?:raw|src)/branch/[^/]+/(.*)$", url)
|
|
||||||
if not m:
|
|
||||||
return None
|
|
||||||
host, owner, repo, path = m.groups()
|
|
||||||
try:
|
|
||||||
meta = json.loads(_fetch(f"{host}/api/v1/repos/{owner}/{repo}", _gitea_auth(host)))
|
|
||||||
br = meta.get("default_branch")
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
return None
|
|
||||||
return f"{host}/{owner}/{repo}/raw/branch/{br}/{path}" if br else None
|
|
||||||
|
|
||||||
|
|
||||||
def windows_from_compose(to_url: str, from_url: str | None = None) -> tuple[list, str | None]:
|
|
||||||
"""Derive the scan's version windows by DIFFING two compose files.
|
|
||||||
|
|
||||||
This is the deterministic alternative to a human (or a model) deciding which `--image` args a
|
|
||||||
given upgrade needs. Point it at a PR's compose and it reads the windows straight off the diff:
|
|
||||||
every image whose tag changed becomes a window, every image that did not change is correctly
|
|
||||||
left out, and nothing depends on anyone remembering that the recipe also bumped its redis.
|
|
||||||
|
|
||||||
Returns (windows, note) where windows is [(image-name, from, to)].
|
|
||||||
"""
|
|
||||||
if from_url is None:
|
|
||||||
from_url = _default_branch_compose(to_url)
|
|
||||||
if not from_url:
|
|
||||||
raise SystemExit("could not resolve a baseline compose; pass --compose-from explicitly")
|
|
||||||
new, old = _compose_images(to_url), _compose_images(from_url)
|
|
||||||
app, others = None, []
|
|
||||||
for svc, (repo, tag) in sorted(new.items()):
|
|
||||||
if svc not in old:
|
|
||||||
continue
|
|
||||||
prev_repo, prev_tag = old[svc]
|
|
||||||
if prev_tag == tag and prev_repo == repo:
|
|
||||||
continue
|
|
||||||
# The `app` service is the recipe's primary image by coop-cloud convention; its window drives
|
|
||||||
# --from/--to so the scan's primary advisory source is judged against it. Everything else is
|
|
||||||
# a sidecar window keyed by its image name.
|
|
||||||
if svc == "app":
|
|
||||||
app = (repo.split("/")[-1], prev_tag, tag)
|
|
||||||
else:
|
|
||||||
others.append((repo.split("/")[-1], prev_tag, tag))
|
|
||||||
wins = ([app] if app else []) + others
|
|
||||||
return wins, f"baseline {from_url}"
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> int:
|
def main() -> int:
|
||||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||||
ap.add_argument("recipe")
|
ap.add_argument("recipe")
|
||||||
@@ -1236,13 +868,6 @@ def main() -> int:
|
|||||||
"fix version published), fetch their full text + references and append a "
|
"fix version published), fetch their full text + references and append a "
|
||||||
"block for the agent to judge. Additive: it never changes the count above.")
|
"block for the agent to judge. Additive: it never changes the count above.")
|
||||||
ap.add_argument("--registry", default=REGISTRY_DIR)
|
ap.add_argument("--registry", default=REGISTRY_DIR)
|
||||||
ap.add_argument("--compose-to", default=None, metavar="URL",
|
|
||||||
help="derive the windows by DIFFING this compose against its baseline, instead "
|
|
||||||
"of passing --from/--to/--image by hand. Point it at a PR's compose.yml "
|
|
||||||
"(e.g. .../raw/branch/<pr-branch>/compose.yml).")
|
|
||||||
ap.add_argument("--compose-from", default=None, metavar="URL",
|
|
||||||
help="baseline compose for --compose-to. Default: the same repo's DEFAULT "
|
|
||||||
"branch, resolved from the API (never assumed to be `main`).")
|
|
||||||
ap.add_argument("--image", action="append", default=[], metavar="NAME=FROM:TO",
|
ap.add_argument("--image", action="append", default=[], metavar="NAME=FROM:TO",
|
||||||
help="a sidecar image and the versions it moved between, e.g. "
|
help="a sidecar image and the versions it moved between, e.g. "
|
||||||
"--image redis=7.4:8.10 (repeatable). NAME matches a source repo name; "
|
"--image redis=7.4:8.10 (repeatable). NAME matches a source repo name; "
|
||||||
@@ -1250,20 +875,6 @@ def main() -> int:
|
|||||||
"being left unclassified.")
|
"being left unclassified.")
|
||||||
a = ap.parse_args()
|
a = ap.parse_args()
|
||||||
images = []
|
images = []
|
||||||
if a.compose_to:
|
|
||||||
wins, note = windows_from_compose(a.compose_to, a.compose_from)
|
|
||||||
if not wins:
|
|
||||||
print(f"### Advisory scan — {a.recipe}\n\n**No image versions changed between the two "
|
|
||||||
f"compose files, so this upgrade fixes no CVEs by definition.**\n\n_{note}_")
|
|
||||||
return 0
|
|
||||||
print(f"_derived from compose diff ({note}):_", file=sys.stderr)
|
|
||||||
for n_, f_, t_ in wins:
|
|
||||||
print(f"_ {n_}: {f_} → {t_}_", file=sys.stderr)
|
|
||||||
# The `app` service (first entry when present) drives --from/--to; the rest are --image
|
|
||||||
# windows. Passing every window as --image too is harmless: each is matched by name against
|
|
||||||
# the advisory sources, and an unmatched name is simply ignored.
|
|
||||||
a.v_from, a.v_to = a.v_from or wins[0][1], a.v_to or wins[0][2]
|
|
||||||
images = list(wins[1:])
|
|
||||||
for spec in a.image:
|
for spec in a.image:
|
||||||
name, _, rng = spec.partition('=')
|
name, _, rng = spec.partition('=')
|
||||||
vf, _, vt = rng.partition(':')
|
vf, _, vt = rng.partition(':')
|
||||||
|
|||||||
@@ -34,7 +34,7 @@ supports_resume = false
|
|||||||
prompt_delivery = "ping" # send after the TUI connects
|
prompt_delivery = "ping" # send after the TUI connects
|
||||||
connect_delay = 12
|
connect_delay = 12
|
||||||
submit_key = "C-m"
|
submit_key = "C-m"
|
||||||
preamble = "set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a"
|
preamble = "set -a; . /srv/cc-ci/.testenv; set +a"
|
||||||
stall_idle = 900
|
stall_idle = 900
|
||||||
active_re = "esc interrupt|thinking|inferring|running tool|tool call|preparing patch|reading|searching"
|
active_re = "esc interrupt|thinking|inferring|running tool|tool call|preparing patch|reading|searching"
|
||||||
limit_re = "spend limit|usage limit|limit reached|reached your .*limit|out of (credits|tokens)"
|
limit_re = "spend limit|usage limit|limit reached|reached your .*limit|out of (credits|tokens)"
|
||||||
@@ -53,9 +53,8 @@ log_dir = "/srv/cc-ci/.cc-ci-logs"
|
|||||||
[[agent]]
|
[[agent]]
|
||||||
name = "orchestrator" # tmux session: cc-ci-orchestrator
|
name = "orchestrator" # tmux session: cc-ci-orchestrator
|
||||||
kind = "persistent"
|
kind = "persistent"
|
||||||
backend = "opencode" # operator 2026-09-07 (evening): NO Claude on the cc-ci host — the
|
backend = "opencode" # attached to the shared opencode web server (oc.commoninternet.net)
|
||||||
model = "opencode-go/glm-5.2" # orchestrator is an opencode agent, steered via the opencode web UI
|
model = "opencode/glm-5.2" # opencode backend; visible at https://oc.commoninternet.net
|
||||||
# (https://oc.ci.commoninternet.net). Claude sessions live on notplants-orchestrator.
|
|
||||||
resume = false # opencode backend has no --resume equivalent (fresh session each launch)
|
resume = false # opencode backend has no --resume equivalent (fresh session each launch)
|
||||||
watch = "heal" # restart if dead / FATAL / backend-mismatch; never stall-reboot
|
watch = "heal" # restart if dead / FATAL / backend-mismatch; never stall-reboot
|
||||||
wake = { interval = 3600, prompt_file = "ai-progress-monitor-prompt.txt" }
|
wake = { interval = 3600, prompt_file = "ai-progress-monitor-prompt.txt" }
|
||||||
|
|||||||
@@ -1,314 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""audit-sources — are we still looking in the right place for each recipe's updates?
|
|
||||||
|
|
||||||
A recipe tracks an image repo and a set of registry URLs. Upstreams move: they rename the image,
|
|
||||||
switch registry, archive the GitHub repo, or split a community edition out of the original. When that
|
|
||||||
happens nothing errors — the old repo simply stops receiving tags, and the recipe looks "up to date"
|
|
||||||
forever while real releases happen somewhere else.
|
|
||||||
|
|
||||||
plausible is the worked example. It tracked `plausible/analytics` on Docker Hub; upstream moved to
|
|
||||||
`ghcr.io/plausible/community-edition`. The old repo still exists and still serves v2.0.0, so every
|
|
||||||
survey said "no upgrades available" while v3 shipped elsewhere.
|
|
||||||
|
|
||||||
This reports the signals that catch that, per image and per registry URL:
|
|
||||||
|
|
||||||
* IMAGE GONE QUIET — newest tag is older than --quiet-days (default 365). The single strongest
|
|
||||||
signal that releases moved somewhere else.
|
|
||||||
* DEPRECATION WORDING — the registry description says deprecated / moved / no longer maintained.
|
|
||||||
* GITHUB REPO ARCHIVED — upstream archived it.
|
|
||||||
* GITHUB REPO RENAMED — the API redirects to a different owner/name than we ask for.
|
|
||||||
* GITHUB REPO GONE — 404.
|
|
||||||
|
|
||||||
Everything is a SIGNAL, not a verdict: a genuinely stable image (mumble, custom-html) can be quiet
|
|
||||||
for good reason. The output is for a human to judge, so each finding says what was measured.
|
|
||||||
|
|
||||||
audit-sources.py [recipe ...] [--ssh HOST] [--quiet-days N] [--json]
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import importlib.util
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
import sys
|
|
||||||
import urllib.error
|
|
||||||
import urllib.request
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
|
|
||||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
|
||||||
_spec = importlib.util.spec_from_file_location("resolve_images", os.path.join(HERE, "resolve-images.py"))
|
|
||||||
RI = importlib.util.module_from_spec(_spec)
|
|
||||||
_spec.loader.exec_module(RI)
|
|
||||||
|
|
||||||
# advisory-scan supplies the source-fetching + changelog-attribution used by --security-sources
|
|
||||||
_aspec = importlib.util.spec_from_file_location("advisory_scan", os.path.join(HERE, "advisory-scan.py"))
|
|
||||||
A = importlib.util.module_from_spec(_aspec)
|
|
||||||
_aspec.loader.exec_module(A)
|
|
||||||
|
|
||||||
REGISTRY_DIR = os.environ.get("CCCI_UPSTREAM_REGISTRY", os.path.join(HERE, "upstream"))
|
|
||||||
USED_RECIPES = os.path.join(HERE, "used-recipes.md")
|
|
||||||
DEPRECATION_RE = re.compile(
|
|
||||||
r"\b(deprecat|no longer maintain|unmaintained|superseded|moved to|migrated to|"
|
|
||||||
r"has moved|discontinued|end.of.life|archived)\b", re.I)
|
|
||||||
|
|
||||||
|
|
||||||
def _days_since(iso: str | None) -> int | None:
|
|
||||||
if not iso:
|
|
||||||
return None
|
|
||||||
try:
|
|
||||||
d = datetime.fromisoformat(iso.replace("Z", "+00:00"))
|
|
||||||
except ValueError:
|
|
||||||
return None
|
|
||||||
return (datetime.now(timezone.utc) - d).days
|
|
||||||
|
|
||||||
|
|
||||||
def hub_repo_meta(repo: str) -> dict:
|
|
||||||
"""Docker Hub repo metadata: when it was last pushed to, and how it describes itself."""
|
|
||||||
try:
|
|
||||||
d = RI._json(f"https://hub.docker.com/v2/repositories/{repo}", RI._hub_auth())
|
|
||||||
except urllib.error.HTTPError as e:
|
|
||||||
return {"status": f"HTTP {e.code}"}
|
|
||||||
except Exception as e: # noqa: BLE001
|
|
||||||
return {"status": f"{type(e).__name__}"}
|
|
||||||
text = f"{d.get('description') or ''}\n{d.get('full_description') or ''}"
|
|
||||||
m = DEPRECATION_RE.search(text)
|
|
||||||
return {"status": "ok", "last_updated": d.get("last_updated"),
|
|
||||||
"deprecation_hint": (m.group(0) if m else None),
|
|
||||||
"archived": bool(d.get("is_archived") or d.get("status") == "inactive")}
|
|
||||||
|
|
||||||
|
|
||||||
def github_repo_meta(owner: str, repo: str) -> dict:
|
|
||||||
"""GitHub repo state — archived, renamed (the API answers with the CURRENT full_name), or gone."""
|
|
||||||
hdrs = {"Accept": "application/vnd.github+json"}
|
|
||||||
tok = RI._gh_token()
|
|
||||||
if tok:
|
|
||||||
hdrs["Authorization"] = f"Bearer {tok}"
|
|
||||||
try:
|
|
||||||
d = RI._json(f"https://api.github.com/repos/{owner}/{repo}", hdrs)
|
|
||||||
except urllib.error.HTTPError as e:
|
|
||||||
return {"status": f"HTTP {e.code}"}
|
|
||||||
except Exception as e: # noqa: BLE001
|
|
||||||
return {"status": f"{type(e).__name__}"}
|
|
||||||
asked, got = f"{owner}/{repo}".lower(), (d.get("full_name") or "").lower()
|
|
||||||
return {"status": "ok", "archived": bool(d.get("archived")), "pushed_at": d.get("pushed_at"),
|
|
||||||
"renamed_to": (d.get("full_name") if got and got != asked else None),
|
|
||||||
"description": d.get("description") or ""}
|
|
||||||
|
|
||||||
|
|
||||||
def newest_tag_date(registry: str, repo: str, tag: str) -> str | None:
|
|
||||||
"""When was the repo's newest same-shape tag pushed? Docker Hub only (it dates its tags)."""
|
|
||||||
if registry not in ("docker.io", "registry-1.docker.io"):
|
|
||||||
return None
|
|
||||||
try:
|
|
||||||
d = RI._json(f"https://hub.docker.com/v2/repositories/{repo}/tags"
|
|
||||||
f"?page_size=100&ordering=last_updated", RI._hub_auth())
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
return None
|
|
||||||
want = RI.shape(tag)
|
|
||||||
for row in d.get("results", []):
|
|
||||||
if RI.shape(row.get("name") or "") == want:
|
|
||||||
return row.get("last_updated")
|
|
||||||
return (d.get("results") or [{}])[0].get("last_updated")
|
|
||||||
|
|
||||||
|
|
||||||
def security_source_audit(recipe: str) -> list[dict]:
|
|
||||||
"""Per source: are its CVEs USABLE, or merely visible?
|
|
||||||
|
|
||||||
The nginx lesson. nginx publishes no GitHub advisories; all its CVEs live in nginx.org/en/CHANGES.
|
|
||||||
The scan saw them and could do nothing with them, because nothing said which release fixed which
|
|
||||||
CVE — so every nginx bump in the fleet reported 0. Attribution (advisory-scan §2b) fixed that for
|
|
||||||
changelogs organised by release, but a page that lists CVEs with NO release structure is still a
|
|
||||||
blind spot: visible, uncountable. This finds those.
|
|
||||||
|
|
||||||
Per source: `advisory-feed` (structured, best), `changelog` (CVEs attributable to a release),
|
|
||||||
`unattributable` (CVEs present but no release structure — BLIND), or `no-cve-data`.
|
|
||||||
"""
|
|
||||||
urls, _ = _registry_urls(recipe)
|
|
||||||
out = []
|
|
||||||
# NVD CPE entries are a first-class source: for projects publishing nothing machine-readable
|
|
||||||
# (mattermost, mumble) they are the ONLY structured source, and omitting them here made two
|
|
||||||
# recipes look permanently blind after they had been fixed.
|
|
||||||
for key, cpe in A.registry_cpes(recipe, REGISTRY_DIR):
|
|
||||||
e = A.nvd_advisories(cpe, key)
|
|
||||||
n = len(e.get("advisories") or [])
|
|
||||||
out.append({"source": e["source"] + f" ({cpe.split(':')[4]}/{cpe.split(':')[3]})",
|
|
||||||
"kind": "advisory-feed" if n else "no-cve-data",
|
|
||||||
"status": e["status"], "cves": n, "usable": n})
|
|
||||||
for entry in A.github_advisories(urls):
|
|
||||||
out.append({"source": entry["source"], "kind": "advisory-feed",
|
|
||||||
"status": entry["status"], "cves": len(entry.get("advisories") or []),
|
|
||||||
"usable": len(entry.get("advisories") or [])})
|
|
||||||
for entry in A.vendor_pages(urls):
|
|
||||||
if entry["status"].startswith("skipped"):
|
|
||||||
continue
|
|
||||||
n = len(entry.get("cves") or [])
|
|
||||||
attributed = len(entry.get("fixed_in") or {})
|
|
||||||
kind = ("no-cve-data" if n == 0 else
|
|
||||||
"changelog" if attributed else "unattributable")
|
|
||||||
out.append({"source": entry["source"], "kind": kind, "status": entry["status"],
|
|
||||||
"cves": n, "usable": attributed})
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def audit_recipe(recipe: str, ssh: str | None, quiet_days: int) -> dict:
|
|
||||||
out = {"recipe": recipe, "findings": [], "images": [], "sources": []}
|
|
||||||
try:
|
|
||||||
refs = (RI.compose_images_ssh(recipe, ssh, "~/.abra/recipes") if ssh
|
|
||||||
else RI.compose_images(recipe, RI.RECIPE_DIR))
|
|
||||||
except Exception as e: # noqa: BLE001
|
|
||||||
out["findings"].append({"level": "error", "what": f"could not read compose: {e}"})
|
|
||||||
return out
|
|
||||||
|
|
||||||
for ref in refs:
|
|
||||||
if "${" in ref:
|
|
||||||
continue
|
|
||||||
info = RI.parse_ref(ref)
|
|
||||||
row = {"ref": ref, "registry": info["registry"], "repo": info["repo"], "tag": info["tag"]}
|
|
||||||
if info["registry"] in ("docker.io", "registry-1.docker.io"):
|
|
||||||
meta = hub_repo_meta(info["repo"])
|
|
||||||
row.update(meta)
|
|
||||||
newest = newest_tag_date(info["registry"], info["repo"], info["tag"])
|
|
||||||
row["newest_tag_pushed"] = newest
|
|
||||||
age = _days_since(newest)
|
|
||||||
row["newest_tag_age_days"] = age
|
|
||||||
if age is not None and age > quiet_days:
|
|
||||||
out["findings"].append({
|
|
||||||
"level": "warn", "what": "image has gone quiet",
|
|
||||||
"detail": f"{info['repo']}: newest {RI.shape(info['tag'])}-shaped tag pushed "
|
|
||||||
f"{age} days ago — releases may have moved elsewhere"})
|
|
||||||
if meta.get("deprecation_hint"):
|
|
||||||
out["findings"].append({
|
|
||||||
"level": "warn", "what": "registry text suggests deprecation",
|
|
||||||
"detail": f"{info['repo']}: says {meta['deprecation_hint']!r}"})
|
|
||||||
if meta.get("archived"):
|
|
||||||
out["findings"].append({"level": "warn", "what": "registry repo archived/inactive",
|
|
||||||
"detail": info["repo"]})
|
|
||||||
out["images"].append(row)
|
|
||||||
|
|
||||||
urls, reg_path = ([], None)
|
|
||||||
try:
|
|
||||||
urls, reg_path = _registry_urls(recipe)
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
pass
|
|
||||||
if reg_path is None:
|
|
||||||
out["findings"].append({"level": "warn", "what": "no upstream registry file",
|
|
||||||
"detail": f"cc-ci-plan/upstream/{recipe}.md is missing — the advisory "
|
|
||||||
f"scan has nowhere to look"})
|
|
||||||
seen = set()
|
|
||||||
for u in urls:
|
|
||||||
m = re.match(r"https?://github\.com/([^/]+)/([^/#?]+)", u)
|
|
||||||
if not m:
|
|
||||||
continue
|
|
||||||
owner, repo = m.group(1), m.group(2).removesuffix(".git")
|
|
||||||
if (owner, repo) in seen:
|
|
||||||
continue
|
|
||||||
seen.add((owner, repo))
|
|
||||||
meta = github_repo_meta(owner, repo)
|
|
||||||
row = {"repo": f"{owner}/{repo}", **meta}
|
|
||||||
age = _days_since(meta.get("pushed_at"))
|
|
||||||
row["pushed_age_days"] = age
|
|
||||||
out["sources"].append(row)
|
|
||||||
if meta.get("status") != "ok":
|
|
||||||
out["findings"].append({"level": "warn", "what": "registry source unreachable",
|
|
||||||
"detail": f"{owner}/{repo}: {meta['status']}"})
|
|
||||||
continue
|
|
||||||
if meta.get("renamed_to"):
|
|
||||||
out["findings"].append({"level": "alert", "what": "GitHub repo has MOVED",
|
|
||||||
"detail": f"{owner}/{repo} now answers as {meta['renamed_to']}"})
|
|
||||||
if meta.get("archived"):
|
|
||||||
out["findings"].append({"level": "alert", "what": "GitHub repo is ARCHIVED",
|
|
||||||
"detail": f"{owner}/{repo} — upstream development has stopped here"})
|
|
||||||
if age is not None and age > quiet_days:
|
|
||||||
out["findings"].append({"level": "warn", "what": "GitHub repo quiet",
|
|
||||||
"detail": f"{owner}/{repo}: last push {age} days ago"})
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def _registry_urls(recipe: str):
|
|
||||||
path = os.path.join(REGISTRY_DIR, f"{recipe}.md")
|
|
||||||
if not os.path.exists(path):
|
|
||||||
return [], None
|
|
||||||
text = open(path).read()
|
|
||||||
urls = []
|
|
||||||
for u in re.findall(r"https?://[^\s)|\]]+", text):
|
|
||||||
u = u.rstrip("`'\"*.,;:>)")
|
|
||||||
if u and u not in urls:
|
|
||||||
urls.append(u)
|
|
||||||
return urls, path
|
|
||||||
|
|
||||||
|
|
||||||
def all_recipes() -> list[str]:
|
|
||||||
out = []
|
|
||||||
for ln in open(USED_RECIPES):
|
|
||||||
ln = ln.strip()
|
|
||||||
if not ln or ln.startswith("#") or ln.startswith("`"):
|
|
||||||
continue
|
|
||||||
parts = ln.split()
|
|
||||||
if len(parts) >= 2 and parts[1] in ("weekly", "external"):
|
|
||||||
out.append(parts[0])
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> int:
|
|
||||||
ap = argparse.ArgumentParser(description=__doc__,
|
|
||||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
||||||
ap.add_argument("recipes", nargs="*")
|
|
||||||
ap.add_argument("--ssh", default=None)
|
|
||||||
ap.add_argument("--quiet-days", type=int, default=365)
|
|
||||||
ap.add_argument("--json", action="store_true")
|
|
||||||
ap.add_argument("--security-sources", action="store_true",
|
|
||||||
help="audit whether each recipe's CVE sources are USABLE (structured advisory "
|
|
||||||
"feed / release-attributable changelog) or merely visible")
|
|
||||||
a = ap.parse_args()
|
|
||||||
|
|
||||||
recipes = a.recipes or all_recipes()
|
|
||||||
if a.security_sources:
|
|
||||||
# What matters is whether the RECIPE can see CVEs at all — not whether some individual page
|
|
||||||
# is unparseable. A page with no release structure is harmless when the same project also
|
|
||||||
# publishes an advisory feed (redis, gitea, minio, clickhouse all do); it is only a blind
|
|
||||||
# spot when nothing else covers that project.
|
|
||||||
blind_recipes, noisy = [], 0
|
|
||||||
for r in recipes:
|
|
||||||
rows = security_source_audit(r)
|
|
||||||
feeds = [x for x in rows if x["kind"] == "advisory-feed" and x["cves"] > 0]
|
|
||||||
logs = [x for x in rows if x["kind"] == "changelog"]
|
|
||||||
unattr = [x for x in rows if x["kind"] == "unattributable"]
|
|
||||||
noisy += len(unattr)
|
|
||||||
usable = len(feeds) + len(logs)
|
|
||||||
if usable == 0:
|
|
||||||
blind_recipes.append(r)
|
|
||||||
print(f"!! {r}: NO USABLE CVE SOURCE — {len(unattr)} unparseable page(s), "
|
|
||||||
f"0 advisory feeds, 0 attributable changelogs")
|
|
||||||
for x in rows:
|
|
||||||
print(f" {x['kind']:15} {x['source'][:64]} ({x['cves']} CVEs)")
|
|
||||||
else:
|
|
||||||
print(f"OK {r}: {len(feeds)} advisory-feed(s), {len(logs)} changelog(s)"
|
|
||||||
+ (f", {len(unattr)} unparseable page(s) (redundant — covered by a feed)"
|
|
||||||
if unattr else ""))
|
|
||||||
for x in logs:
|
|
||||||
print(f" changelog {x['source'][:62]} ({x['usable']}/{x['cves']})")
|
|
||||||
print(f"\n{len(recipes)} recipes · {len(blind_recipes)} with NO usable CVE source"
|
|
||||||
+ (f": {', '.join(blind_recipes)}" if blind_recipes else "")
|
|
||||||
+ f" · {noisy} unparseable page(s) elsewhere (harmless where a feed covers them)")
|
|
||||||
return 0
|
|
||||||
reports = [audit_recipe(r, a.ssh, a.quiet_days) for r in recipes]
|
|
||||||
if a.json:
|
|
||||||
print(json.dumps(reports, indent=2))
|
|
||||||
return 0
|
|
||||||
alerts = 0
|
|
||||||
for rep in reports:
|
|
||||||
fs = rep["findings"]
|
|
||||||
mark = "OK " if not fs else ("!! " if any(f["level"] == "alert" for f in fs) else " ? ")
|
|
||||||
print(f"{mark} {rep['recipe']}")
|
|
||||||
for f in fs:
|
|
||||||
alerts += f["level"] == "alert"
|
|
||||||
print(f" [{f['level']}] {f['what']}: {f.get('detail','')}")
|
|
||||||
print(f"\n{len(reports)} recipes audited · "
|
|
||||||
f"{sum(len(r['findings']) for r in reports)} findings · {alerts} alerts")
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
sys.exit(main())
|
|
||||||
@@ -112,11 +112,11 @@ def start(mode="resume"):
|
|||||||
if not Path(OPENCODE_BIN).exists():
|
if not Path(OPENCODE_BIN).exists():
|
||||||
die(f"opencode not found at {OPENCODE_BIN}")
|
die(f"opencode not found at {OPENCODE_BIN}")
|
||||||
cmd = (
|
cmd = (
|
||||||
f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; "
|
f"set -a; . /srv/cc-ci/.testenv; set +a; "
|
||||||
f"NO_COLOR=1 {OPENCODE_BIN} attach {OPENCODE_SERVER} --dir {WORKDIR}"
|
f"NO_COLOR=1 {OPENCODE_BIN} attach {OPENCODE_SERVER} --dir {WORKDIR}"
|
||||||
)
|
)
|
||||||
log(f"starting {SESSION} (backend=opencode, model={LOOP_MODEL or 'default'})")
|
log(f"starting {SESSION} (backend=opencode, model={LOOP_MODEL or 'default'})")
|
||||||
log(" visible at https://oc.ci.commoninternet.net")
|
log(" visible at http://oc.commoninternet.net (tailnet only)")
|
||||||
else:
|
else:
|
||||||
die(f"unknown LOOP_BACKEND '{BACKEND}' — use 'claude' or 'opencode'")
|
die(f"unknown LOOP_BACKEND '{BACKEND}' — use 'claude' or 'opencode'")
|
||||||
|
|
||||||
|
|||||||
@@ -154,11 +154,11 @@ def start(mode="resume"):
|
|||||||
f"OPENCODE_CONFIG_CONTENT={shlex.quote(json.dumps({'model': LOOP_MODEL}))} "
|
f"OPENCODE_CONFIG_CONTENT={shlex.quote(json.dumps({'model': LOOP_MODEL}))} "
|
||||||
)
|
)
|
||||||
cmd = (
|
cmd = (
|
||||||
f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; "
|
f"set -a; . /srv/cc-ci/.testenv; set +a; "
|
||||||
f"{model_env}NO_COLOR=1 {OPENCODE_BIN} attach {OPENCODE_SERVER} --dir {shlex.quote(WORKDIR)}"
|
f"{model_env}NO_COLOR=1 {OPENCODE_BIN} attach {OPENCODE_SERVER} --dir {shlex.quote(WORKDIR)}"
|
||||||
)
|
)
|
||||||
log(f"starting {SESSION} (backend=opencode, model={LOOP_MODEL or 'default'})")
|
log(f"starting {SESSION} (backend=opencode, model={LOOP_MODEL or 'default'})")
|
||||||
log(f" visible at https://oc.ci.commoninternet.net")
|
log(f" visible at http://oc.commoninternet.net (tailnet only)")
|
||||||
else:
|
else:
|
||||||
die(f"unknown LOOP_BACKEND '{BACKEND}' — use 'claude' or 'opencode'")
|
die(f"unknown LOOP_BACKEND '{BACKEND}' — use 'claude' or 'opencode'")
|
||||||
|
|
||||||
@@ -212,7 +212,7 @@ def main():
|
|||||||
else:
|
else:
|
||||||
backend_note = (
|
backend_note = (
|
||||||
"claude: --resume preserves conversation across reboots; viewable at claude.ai/code\n"
|
"claude: --resume preserves conversation across reboots; viewable at claude.ai/code\n"
|
||||||
" opencode: fresh session each launch (no --resume); viewable at https://oc.ci.commoninternet.net"
|
" opencode: fresh session each launch (no --resume); viewable at http://oc.commoninternet.net"
|
||||||
)
|
)
|
||||||
print(f"""cc-ci orchestrator launcher
|
print(f"""cc-ci orchestrator launcher
|
||||||
|
|
||||||
|
|||||||
@@ -109,7 +109,7 @@ def start(mode, date):
|
|||||||
cwd = "/srv/cc-ci-orch/cc-ci"
|
cwd = "/srv/cc-ci-orch/cc-ci"
|
||||||
share_flag = "--share" if OPENCODE_SHARE else ""
|
share_flag = "--share" if OPENCODE_SHARE else ""
|
||||||
cmd = (
|
cmd = (
|
||||||
f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; "
|
f"set -a; . /srv/cc-ci/.testenv; set +a; "
|
||||||
f"{OPENCODE_BIN} run {model_flag} {share_flag} --attach '{OPENCODE_SERVER}' "
|
f"{OPENCODE_BIN} run {model_flag} {share_flag} --attach '{OPENCODE_SERVER}' "
|
||||||
f"--title '{SESSION}' --dir {cwd} \"$(cat '{kf}')\""
|
f"--title '{SESSION}' --dir {cwd} \"$(cat '{kf}')\""
|
||||||
)
|
)
|
||||||
@@ -124,7 +124,7 @@ def start(mode, date):
|
|||||||
_lu._pin_new_session(_prev_ids)
|
_lu._pin_new_session(_prev_ids)
|
||||||
if BACKEND == "opencode":
|
if BACKEND == "opencode":
|
||||||
if OPENCODE_SHARE:
|
if OPENCODE_SHARE:
|
||||||
log(f" attached to {OPENCODE_SERVER} → https://oc.ci.commoninternet.net +public --share link")
|
log(f" attached to {OPENCODE_SERVER} → http://oc.commoninternet.net +public --share link")
|
||||||
# Watchdog for the report too: it runs on glm-5.2 sharing the same opencode-go budget the
|
# Watchdog for the report too: it runs on glm-5.2 sharing the same opencode-go budget the
|
||||||
# upgrade run just drained, so a 429 stall is likely. Reuse launch-upgrader.py's watchdog,
|
# upgrade run just drained, so a 429 stall is likely. Reuse launch-upgrader.py's watchdog,
|
||||||
# pointed at THIS (cc-ci-report) session with a report-specific marker + resume prompt. It
|
# pointed at THIS (cc-ci-report) session with a report-specific marker + resume prompt. It
|
||||||
|
|||||||
@@ -107,7 +107,7 @@ def spawn_supervisor(sid, reason):
|
|||||||
kf = Path(LOG_DIR) / f".kickoff-{SUP_SESSION}.txt"
|
kf = Path(LOG_DIR) / f".kickoff-{SUP_SESSION}.txt"
|
||||||
kf.write_text(build_kickoff(sid, reason))
|
kf.write_text(build_kickoff(sid, reason))
|
||||||
share = "--share" if OPENCODE_SHARE else ""
|
share = "--share" if OPENCODE_SHARE else ""
|
||||||
cmd = (f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; {OPENCODE_BIN} run --model '{MODEL}' {share} "
|
cmd = (f"set -a; . {WORKDIR}/.testenv; set +a; {OPENCODE_BIN} run --model '{MODEL}' {share} "
|
||||||
f"--attach '{OPENCODE_SERVER}' --title '{SUP_SESSION}' --dir {WORKDIR} \"$(cat '{kf}')\"")
|
f"--attach '{OPENCODE_SERVER}' --title '{SUP_SESSION}' --dir {WORKDIR} \"$(cat '{kf}')\"")
|
||||||
_sh(["tmux", "new-session", "-d", "-s", SUP_SESSION, "-c", WORKDIR, cmd])
|
_sh(["tmux", "new-session", "-d", "-s", SUP_SESSION, "-c", WORKDIR, cmd])
|
||||||
_sh(["tmux", "pipe-pane", "-o", "-t", SUP_SESSION, f"cat >> '{LOG_DIR}/{SUP_SESSION}.log'"])
|
_sh(["tmux", "pipe-pane", "-o", "-t", SUP_SESSION, f"cat >> '{LOG_DIR}/{SUP_SESSION}.log'"])
|
||||||
|
|||||||
@@ -67,7 +67,7 @@ REMOTE_CONTROL = os.environ.get("REMOTE_CONTROL", "1") == "1"
|
|||||||
OPENCODE_BIN = os.environ.get("OPENCODE_BIN", "/home/loops/.local/bin/opencode")
|
OPENCODE_BIN = os.environ.get("OPENCODE_BIN", "/home/loops/.local/bin/opencode")
|
||||||
OPENCODE_SERVER = os.environ.get("OPENCODE_SERVER", "http://127.0.0.1:4096")
|
OPENCODE_SERVER = os.environ.get("OPENCODE_SERVER", "http://127.0.0.1:4096")
|
||||||
# Web visibility for the opencode backend: attach the session to the shared opencode
|
# Web visibility for the opencode backend: attach the session to the shared opencode
|
||||||
# web server (viewable at https://oc.ci.commoninternet.net (basic auth)) AND optionally
|
# web server (viewable at http://oc.commoninternet.net, tailnet-only) AND optionally
|
||||||
# create a public opencode.ai --share link. Default both on so the run is monitorable.
|
# create a public opencode.ai --share link. Default both on so the run is monitorable.
|
||||||
OPENCODE_SHARE = os.environ.get("OPENCODE_SHARE", "1") == "1"
|
OPENCODE_SHARE = os.environ.get("OPENCODE_SHARE", "1") == "1"
|
||||||
|
|
||||||
@@ -216,11 +216,11 @@ def start(mode="use-or-create"):
|
|||||||
# so they must come AFTER `run` (a global `opencode --model X run` is ignored).
|
# so they must come AFTER `run` (a global `opencode --model X run` is ignored).
|
||||||
share_flag = "--share" if OPENCODE_SHARE else ""
|
share_flag = "--share" if OPENCODE_SHARE else ""
|
||||||
cmd = (
|
cmd = (
|
||||||
f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; "
|
f"set -a; . /srv/cc-ci/.testenv; set +a; "
|
||||||
f"{OPENCODE_BIN} run {model_flag} {share_flag} --attach '{OPENCODE_SERVER}' "
|
f"{OPENCODE_BIN} run {model_flag} {share_flag} --attach '{OPENCODE_SERVER}' "
|
||||||
f"--title '{SESSION}' \"$(cat '{kf}')\""
|
f"--title '{SESSION}' \"$(cat '{kf}')\""
|
||||||
)
|
)
|
||||||
log(f" attached to {OPENCODE_SERVER} → https://oc.ci.commoninternet.net"
|
log(f" attached to {OPENCODE_SERVER} → http://oc.commoninternet.net (tailnet only)"
|
||||||
+ (" +public --share link (printed in the session)" if OPENCODE_SHARE else ""))
|
+ (" +public --share link (printed in the session)" if OPENCODE_SHARE else ""))
|
||||||
else:
|
else:
|
||||||
die(f"unknown LOOP_BACKEND '{BACKEND}' — use 'claude' or 'opencode'")
|
die(f"unknown LOOP_BACKEND '{BACKEND}' — use 'claude' or 'opencode'")
|
||||||
@@ -521,7 +521,7 @@ def resume(reason="manual"):
|
|||||||
"write+push the weekly summary, then `python3 /srv/cc-ci/cc-ci-plan/launch-report.py fresh`, print "
|
"write+push the weekly summary, then `python3 /srv/cc-ci/cc-ci-plan/launch-report.py fresh`, print "
|
||||||
"'" + DONE_MARKER + "', and go idle.")
|
"'" + DONE_MARKER + "', and go idle.")
|
||||||
share = "--share" if OPENCODE_SHARE else ""
|
share = "--share" if OPENCODE_SHARE else ""
|
||||||
cmd = (f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; {OPENCODE_BIN} run -s {sid} --continue "
|
cmd = (f"set -a; . /srv/cc-ci/.testenv; set +a; {OPENCODE_BIN} run -s {sid} --continue "
|
||||||
f"--model '{MODEL}' {share} --attach '{OPENCODE_SERVER}' --dir '{WORKDIR}' \"$(cat '{kf}')\"")
|
f"--model '{MODEL}' {share} --attach '{OPENCODE_SERVER}' --dir '{WORKDIR}' \"$(cat '{kf}')\"")
|
||||||
subprocess.run(["tmux", "new-session", "-d", "-s", SESSION, "-c", WORKDIR, cmd])
|
subprocess.run(["tmux", "new-session", "-d", "-s", SESSION, "-c", WORKDIR, cmd])
|
||||||
subprocess.run(["tmux", "pipe-pane", "-o", "-t", SESSION, f"cat >> '{LOG_FILE}'"])
|
subprocess.run(["tmux", "pipe-pane", "-o", "-t", SESSION, f"cat >> '{LOG_FILE}'"])
|
||||||
@@ -655,7 +655,7 @@ Model: {MODEL} (LOOP_MODEL or UPGRADER_MODEL env var)
|
|||||||
Args: {UPGRADER_ARGS or '<none>'} (UPGRADER_ARGS env var, passed to /upgrade-all)
|
Args: {UPGRADER_ARGS or '<none>'} (UPGRADER_ARGS env var, passed to /upgrade-all)
|
||||||
|
|
||||||
claude: viewable at claude.ai/code
|
claude: viewable at claude.ai/code
|
||||||
opencode: viewable at https://oc.ci.commoninternet.net server={OPENCODE_SERVER}
|
opencode: viewable at http://oc.commoninternet.net server={OPENCODE_SERVER}
|
||||||
""")
|
""")
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -323,11 +323,11 @@ def start_agent(role, session, workdir):
|
|||||||
# absolute repo paths for builder/adversary work.
|
# absolute repo paths for builder/adversary work.
|
||||||
session_cwd = "/srv/cc-ci-orch/cc-ci"
|
session_cwd = "/srv/cc-ci-orch/cc-ci"
|
||||||
cmd = (
|
cmd = (
|
||||||
f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; "
|
f"set -a; . /srv/cc-ci/.testenv; set +a; "
|
||||||
f"NO_COLOR=1 {OPENCODE_BIN} attach {OPENCODE_SERVER} --dir {session_cwd}"
|
f"NO_COLOR=1 {OPENCODE_BIN} attach {OPENCODE_SERVER} --dir {session_cwd}"
|
||||||
)
|
)
|
||||||
log(f"starting {session} (backend=opencode, phase={pid}, model={model or 'default'})")
|
log(f"starting {session} (backend=opencode, phase={pid}, model={model or 'default'})")
|
||||||
log(f" visible at https://oc.ci.commoninternet.net")
|
log(f" visible at http://oc.commoninternet.net (tailnet only)")
|
||||||
else:
|
else:
|
||||||
die(f"unknown BACKEND '{BACKEND}' — set LOOP_BACKEND=claude or LOOP_BACKEND=opencode")
|
die(f"unknown BACKEND '{BACKEND}' — set LOOP_BACKEND=claude or LOOP_BACKEND=opencode")
|
||||||
|
|
||||||
|
|||||||
@@ -1,17 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
# Load cc-ci runtime configuration. Source this file; do not execute it.
|
|
||||||
#
|
|
||||||
# Callers enable `set -a` so both files become exported environment variables.
|
|
||||||
# Public configuration is loaded first, then the secret file. This preserves a
|
|
||||||
# local secret override only where an emergency migration explicitly needs it.
|
|
||||||
|
|
||||||
if [ -r /srv/cc-ci/cc-ci/.env.public ]; then
|
|
||||||
. /srv/cc-ci/cc-ci/.env.public
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [ ! -r /srv/cc-ci/.testenv ]; then
|
|
||||||
echo "cc-ci secret environment is missing: /srv/cc-ci/.testenv" >&2
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
. /srv/cc-ci/.testenv
|
|
||||||
@@ -71,7 +71,7 @@ log "writing report $REPORT ..."
|
|||||||
if [ -f "$SUMMARY" ]; then cat "$SUMMARY"; else echo "(no dated summary at $SUMMARY — /upgrade-all may have stalled on the usage limit or errored; check the cc-ci-upgrader session + $LOG)"; fi
|
if [ -f "$SUMMARY" ]; then cat "$SUMMARY"; else echo "(no dated summary at $SUMMARY — /upgrade-all may have stalled on the usage limit or errored; check the cc-ci-upgrader session + $LOG)"; fi
|
||||||
echo
|
echo
|
||||||
echo "## Open PRs per recipe (post-run)"
|
echo "## Open PRs per recipe (post-run)"
|
||||||
set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a
|
set -a; . /srv/cc-ci/.testenv 2>/dev/null; set +a
|
||||||
G="https://${GITEA_USERNAME}:${GITEA_PASSWORD}@${GITEA_URL}/api/v1"
|
G="https://${GITEA_USERNAME}:${GITEA_PASSWORD}@${GITEA_URL}/api/v1"
|
||||||
for r in $(curl -fsS "$G/orgs/recipe-maintainers/repos?limit=100" 2>/dev/null | python3 -c "import sys,json;print('\n'.join(sorted(x['name'] for x in json.load(sys.stdin))))" 2>/dev/null); do
|
for r in $(curl -fsS "$G/orgs/recipe-maintainers/repos?limit=100" 2>/dev/null | python3 -c "import sys,json;print('\n'.join(sorted(x['name'] for x in json.load(sys.stdin))))" 2>/dev/null); do
|
||||||
case "$r" in cc-ci|cc-ci-orchestrator|cc-ci-secrets|archived-*) continue;; esac
|
case "$r" in cc-ci|cc-ci-orchestrator|cc-ci-secrets|archived-*) continue;; esac
|
||||||
|
|||||||
@@ -1,127 +0,0 @@
|
|||||||
# Plan — one Hetzner host for cc-ci (CI server) + cc-ci-orchestrator
|
|
||||||
|
|
||||||
**Status:** IN PROGRESS (started 2026-09-07). Operator request: move the cc-ci CI server AND the
|
|
||||||
cc-ci orchestrator onto one new Hetzner server (`195.201.88.249`, 8 GB, 150 GB, Debian 13 image),
|
|
||||||
cleanly split off from the shared `notplants-orchestrator` box, which keeps everything else
|
|
||||||
(lichen, project-orchestrator, notplants agents). Nix config ownership: cc-ci's config in
|
|
||||||
`recipe-maintainers/cc-ci`, the orchestrator's in `recipe-maintainers/cc-ci-orchestrator`, and the
|
|
||||||
orchestrator flake **includes** cc-ci's module so one `nixos-rebuild` produces the combined host.
|
|
||||||
Last step (separate, after everything works on the current names): move both to `autonomic.zone`
|
|
||||||
subdomains.
|
|
||||||
|
|
||||||
## Facts (2026-09-07)
|
|
||||||
|
|
||||||
| | old cc-ci server | old orchestrator host (stays, becomes notplants-only) | **new combined host** |
|
|
||||||
|---|---|---|---|
|
|
||||||
| public IP | 91.98.47.73 (fsn1, Hetzner 134485294) | 168.119.126.100 (nbg1, Hetzner 134487234) | **195.201.88.249** |
|
|
||||||
| tailnet | `cc-ci` 100.95.31.88 | `cc-ci-orchestrator-1` 100.84.190.30 | `cc-ci` (new node) |
|
|
||||||
| RAM / disk | 8 GB / 150 GB (83 GB used, 59 GB docker) | 4 GB + 4 GB swap / 75 GB + 250 GB `/mnt/data` | 8 GB / 150 GB, one disk |
|
|
||||||
| built by | `cc-ci` flake `#cc-ci` (nixpkgs 26.05 rev 531670d) | `notplants-nix` flake `#notplants-orchestrator` (26.05 channel), importing `cc-ci-orchestrator`'s `nixosModules.cc-ci` | `cc-ci-orchestrator` flake `#cc-ci` importing `cc-ci`'s `nixosModules.cc-ci-server` |
|
|
||||||
| DNS | `ci.`, `*.ci.`, `ns-acme.commoninternet.net` → 91.98.47.73 (Gandi, direct, no gateway) | `oc.commoninternet.net` → 100.84.190.30 | operator repoints at cutover |
|
|
||||||
|
|
||||||
Data on the old cc-ci server that must move: `/var/lib/cc-ci-reports` (published reports),
|
|
||||||
`/var/lib/cc-ci-runs` (dashboard artifacts, 1.7 G), `/var/lib/ci-warm` (1.4 G), `/var/lib/acme`
|
|
||||||
(LE cert valid to 2026-11-29 + **acme-dns account json** that the `_acme-challenge` CNAME points at),
|
|
||||||
`/var/lib/acme-dns` (the authoritative zone DB), `/var/lib/ci-certs`, `/root/.abra` (app env files),
|
|
||||||
`/etc/cc-ci` (deployed checkout the Sunday sweep runs from), Drone's `drone_ci_commoninternet_net_data`
|
|
||||||
volume (Gitea OAuth grant + repo activation + build history). Warm recipe volumes are caches and get
|
|
||||||
rebuilt on first use / the Sunday sweep. Docker swarm secrets/configs cannot be copied; the reconcile
|
|
||||||
oneshots recreate them from sops.
|
|
||||||
|
|
||||||
Out-of-band secrets the new host needs (never in git): `/var/lib/sops-nix/key.txt` (= the master age
|
|
||||||
key, `/srv/cc-ci/.sops/master-age.txt` here — the new host's SSH host key is not a sops recipient),
|
|
||||||
`/etc/ts-auth-key`, `/srv/cc-ci/.testenv`, `/srv/cc-ci/upgrader.env`, `/srv/cc-ci/.sops/master-age.txt`,
|
|
||||||
`~loops/.ssh/{cc-ci-root,autonomic-bot-gitea,tangled}-ed25519`, `/etc/nginx/oc-*` (self-signed cert +
|
|
||||||
htpasswd for the opencode UI), claude/opencode/codex auth under `~loops`.
|
|
||||||
|
|
||||||
## Design
|
|
||||||
|
|
||||||
**cc-ci repo** (`feat/nixos-module-export`):
|
|
||||||
- `nixosModules.cc-ci-server` = `nix/modules/default.nix`: imports all service modules + the
|
|
||||||
host-generic cc-ci settings that used to sit in the host file (UTC, docker/swarm firewall 80/443,
|
|
||||||
`environment.systemPackages = ccciRuntimeTools`, allowUnfree). No hardware, no networking, no
|
|
||||||
tailscale, no root keys, no stateVersion — the host supplies those.
|
|
||||||
- New options under `cc-ci.*`: `publicIPv4` (acme-dns listen + the `ns-acme` A record),
|
|
||||||
`sopsFile` (absolute path to the decrypted-at-activation `secrets.yaml`, default the submodule
|
|
||||||
path so `#cc-ci` keeps working), `repoPath` (`/etc/cc-ci`, used by nightly-sweep).
|
|
||||||
- `nixosConfigurations.cc-ci` (old host) keeps building unchanged via the same module.
|
|
||||||
|
|
||||||
**cc-ci-orchestrator repo** (`feat/combined-cc-ci-host`):
|
|
||||||
- flake input `cc-ci` (https, public) with `nixpkgs`/`sops-nix` `follows` so one nixpkgs + one sops-nix.
|
|
||||||
- `nixosModules.cc-ci-orchestrator` (the existing `nix/modules/cc-ci.nix`, kept exported as
|
|
||||||
`nixosModules.cc-ci` too so notplants-nix keeps evaluating until it drops the input) — the loops,
|
|
||||||
orchestrator session and the weekly/hourly timers.
|
|
||||||
- `nix/modules/orchestrator-host.nix`: the host contract the module assumes — `loops` user + sudo,
|
|
||||||
nix-ld, claude/opencode/codex installers, `opencode-web`, the tailnet-only nginx `oc.` vhost
|
|
||||||
(on the tailscale IP, port **8443**, because traefik owns 80/443), tool packages, PATH.
|
|
||||||
- `nixosConfigurations.cc-ci` = `nix/hosts/cc-ci/{configuration,hardware,networking}.nix` importing
|
|
||||||
both modules. `/srv` is a plain directory (no `/mnt/data`), 8 GB swapfile, root keys, tailscale
|
|
||||||
`--hostname=cc-ci`, firewall 22 (+ what cc-ci-server opens: 80, 443, 53).
|
|
||||||
- `loops`' ssh config `Host cc-ci` → `127.0.0.1` so every `ssh cc-ci …` in skills/scripts keeps working.
|
|
||||||
- `archive/`: the retired Incus/Hetzner-orchestrator host config, old terraform, historical plans.
|
|
||||||
- `README.md`: provisioning (Hetzner Debian → nixos-infect → NixOS), secrets staging, the one
|
|
||||||
`nixos-rebuild`, data restore, cutover, verification — written so a person or an LLM can redo it.
|
|
||||||
|
|
||||||
**notplants-nix** (`chore/drop-cc-ci`, after cutover): remove the `cc-ci` input, module import, the
|
|
||||||
four cc-ci units' mount gating, `loopsSshConfig`, `opencode-web` + the `oc.` vhost (unless something
|
|
||||||
notplants-side uses it), tailscale hostname → `notplants-orchestrator`.
|
|
||||||
|
|
||||||
## Steps
|
|
||||||
|
|
||||||
1. [x] nixos-infect the new box (`NIX_CHANNEL=nixos-26.05 PROVIDER=hetzner`); capture
|
|
||||||
`hardware-configuration.nix` + `networking.nix`.
|
|
||||||
2. [x] cc-ci: module export + options; verify `#cc-ci` still evaluates; PR.
|
|
||||||
3. [x] cc-ci-orchestrator: input + host + modules + archive/ + README + terraform refresh; verify
|
|
||||||
`#cc-ci` evaluates; PR.
|
|
||||||
4. [x] Stage secrets + clones on the new host; `nixos-rebuild test` → verify → `switch`.
|
|
||||||
Immediately after: scale the new `ccci-bridge_app` to 0 and mask the two cc-ci timers so the
|
|
||||||
new host does not double-process `!testme` or run a second weekly upgrade before cutover.
|
|
||||||
5. [x] Copy data (rsync over tailscale): reports, runs, ci-warm, acme, acme-dns, ci-certs,
|
|
||||||
/root/.abra, /etc/cc-ci; Drone volume with Drone scaled to 0 during the copy.
|
|
||||||
6. [x] Pre-cutover verification on the new IP (`curl --resolve`, port 53, dashboard, reports,
|
|
||||||
drone, one direct `cc-ci-run` on custom-html-tiny).
|
|
||||||
7. [x] Operator: Gandi A records `ci`, `*.ci`, `ns-acme` → 195.201.88.249. Then: old bridge +
|
|
||||||
drone + timers off, new bridge up, one real `!testme` end-to-end, a `!testme`-driven report page.
|
|
||||||
8. [x] Move the orchestrator: stop cc-ci units here, final rsync of `/srv/cc-ci-orch` + agent
|
|
||||||
state, enable on the new host, operator reconnects there; notplants-nix PR removing cc-ci.
|
|
||||||
9. [ ] Old cc-ci server: cold standby ~1 week, then operator deletes it and the stale tailnet node.
|
|
||||||
10. [ ] Domain move to `autonomic.zone` — separate plan, after 1–9 are proven.
|
|
||||||
|
|
||||||
## Log
|
|
||||||
|
|
||||||
- 2026-09-07 19:40 UTC — recon done, plan written, ssh to the new box verified as root with
|
|
||||||
`notplants-orchestrator-ed25519`.
|
|
||||||
- 2026-09-07 20:05 UTC — nixos-infect started on 195.201.88.249 (rev 40f62a6, nixos-26.05,
|
|
||||||
PROVIDER=hetznercloud). Two false starts: the Debian 13 image has /tmp on tmpfs, so
|
|
||||||
nixos-infect's temp swapfile fails `swapon: Invalid argument`; fixed with `NO_SWAP=true`.
|
|
||||||
Build ran, box rebooted ~20:11 UTC and has not answered ping/ssh since (>25 min) — needs the
|
|
||||||
Hetzner console (no API token for that project on this host).
|
|
||||||
- 2026-09-07 20:40 UTC — cc-ci branch `feat/nixos-module-export` (9b99f81) pushed: the standalone
|
|
||||||
`#cc-ci` drv is byte-identical before/after. Orchestrator branch `feat/combined-cc-ci-host`:
|
|
||||||
`#cc-ci` evaluates (gcnwq4fy…-nixos-system-cc-ci-26.05.20260803.531670d.drv) with PROVISIONAL
|
|
||||||
hardware/networking copied from the old CI server — to be replaced by the infect output.
|
|
||||||
- 2026-09-07 20:30 UTC — rescue-mode diagnosis + fixes (README §2a); `#cc-ci` test → switch →
|
|
||||||
reboot OK on the new box; data restored incl. Drone volume; PRs cc-ci #32 + cc-ci-orchestrator
|
|
||||||
#19 merged. Quarantine on (bridge 0, timers runtime-masked, orchestrator units stopped).
|
|
||||||
Waiting on operator: new tailscale auth key; Gandi DNS cutover.
|
|
||||||
- 2026-09-07 21:10 UTC — operator: NO tailscale on the new host; opencode UI behind its basic
|
|
||||||
auth on 443 (traefik labels on the `opencode-ui` socat relay → nginx on docker_gwbridge:8090);
|
|
||||||
fail2ban (sshd + nginx-http-auth in DOCKER-USER). Operator repointed Gandi (ci, *.ci, ns-acme)
|
|
||||||
→ 195.201.88.249 ~20:40; propagated at once at public resolvers. CUTOVER DONE: old bridge +
|
|
||||||
drone scaled to 0, old-host timers masked, new bridge up; `!testme` on custom-html #7 = Drone
|
|
||||||
build 1326 on the new box. Two self-inflicted incidents, both documented in README: traefik
|
|
||||||
FILE_PROVIDER_DIRECTORY_ENABLED took the front doors down 2 min; the Hetzner resolvers in
|
|
||||||
networking.nix served the stale address (3 h TTL) → runner/bridge 404s until dropped.
|
|
||||||
`nixos-rebuild switch` = generation 4. The `switch` re-enabled cc-ci-orchestrator on the new
|
|
||||||
box: a fresh Claude orchestrator session runs there (tmux `cc-ci-orchestrator`, remote-control
|
|
||||||
label `cc-ci-orchestrator`) since 20:46 UTC; the old host's cc-ci units are runtime-masked and
|
|
||||||
the notplants-nix branch `chore/drop-cc-ci` (pushed, evaluates) removes them for good —
|
|
||||||
operator merges (that repo auto-deploys nightly).
|
|
||||||
- Remaining: (9) delete old CI server 134485294 + its tailnet node after ~1 week; revoke the
|
|
||||||
Hetzner token; (10) autonomic.zone domains. Also re-key cc-ci-secrets to the new host's ssh host
|
|
||||||
key (optional; master key works) and re-run the cc-ci self-test build (1324/1325 failed during
|
|
||||||
the resolver mess).
|
|
||||||
- 2026-09-07 21:20 UTC — notplants-nix `chore/drop-cc-ci` merged (9c785c1) and applied on the old
|
|
||||||
host (test → switch, generation 57, healthy, cc-ci units gone). cc-ci push self-test green again
|
|
||||||
after #33 (ruff format, pre-existing since 2026-08-31) + #34 (statix: acme-dns.nix systemd
|
|
||||||
attrset): builds 1330/1331 success. New host on generation 6 with cc-ci main 0437210.
|
|
||||||
+2
-3
@@ -83,10 +83,9 @@ repo, a commit, a log, or the dashboard** (§9) — reference locations only.
|
|||||||
| What | Where | How to use |
|
| What | Where | How to use |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| **cc-ci SSH (root)** | private key `~/.ssh/cc-ci-root-ed25519`; `Host cc-ci` in `~/.ssh/config` (HostName `100.90.116.4`, no ProxyCommand) | Just run `ssh cc-ci` (logs in as **root**). The orchestrator VM is a direct tailnet peer — direct route, no proxy. Pubkey already in cc-ci's `/root/.ssh/authorized_keys`. |
|
| **cc-ci SSH (root)** | private key `~/.ssh/cc-ci-root-ed25519`; `Host cc-ci` in `~/.ssh/config` (HostName `100.90.116.4`, no ProxyCommand) | Just run `ssh cc-ci` (logs in as **root**). The orchestrator VM is a direct tailnet peer — direct route, no proxy. Pubkey already in cc-ci's `/root/.ssh/authorized_keys`. |
|
||||||
| **Gitea bot account** | `/srv/cc-ci/cc-ci/.env.public` → `GITEA_USERNAME` (`autonomic-bot`); `/srv/cc-ci/.testenv` → `GITEA_PASSWORD`; `GITEA_URL` defaults to `git.autonomic.zone` | Basic-auth to the Gitea API, or mint a scoped token: `POST https://$GITEA_URL/api/v1/users/$GITEA_USERNAME/tokens`. Used to push the `cc-ci` project repo, read recipe repos, comment on PRs, and poll for `!testme` (read-level; the bot does not register webhooks). |
|
| **Gitea bot account** | `/srv/cc-ci/.testenv` → `GITEA_USERNAME` (`autonomic-bot`), `GITEA_PASSWORD`, `GITEA_URL` (`git.autonomic.zone`) | Basic-auth to the Gitea API, or mint a scoped token: `POST https://$GITEA_URL/api/v1/users/$GITEA_USERNAME/tokens`. Used to push the `cc-ci` project repo, read recipe repos, comment on PRs, and poll for `!testme` (read-level; the bot does not register webhooks). |
|
||||||
|
|
||||||
Load runtime configuration in a shell with: `set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a`
|
Load them in a shell with: `set -a; . /srv/cc-ci/.testenv; set +a` (don't echo the values).
|
||||||
(don't echo secret values). The helper loads tracked public configuration first and then `.testenv`.
|
|
||||||
|
|
||||||
### The Tailscale connection (how `ssh cc-ci` works)
|
### The Tailscale connection (how `ssh cc-ci` works)
|
||||||
|
|
||||||
|
|||||||
@@ -1,234 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""pr-survey — deterministic facts about every open recipe PR, for /cc-ci-cleanup to judge.
|
|
||||||
|
|
||||||
Open recipe PRs rot in specific, detectable ways. This gathers the evidence; it does NOT decide
|
|
||||||
anything — closing a PR is a judgement the skill makes, with these facts in hand.
|
|
||||||
|
|
||||||
RUN `reconcile-upstream.sh --all` FIRST. Every signal below is measured against the mirror's `main`,
|
|
||||||
and an unreconciled mirror makes all of them wrong: on 2026-08-11 three PRs (discourse #6 carrying
|
|
||||||
140 CVEs, keycloak #6 carrying 12, n8n #5) looked pending against a stale mirror while upstream had
|
|
||||||
already merged them. This tool refuses to guess about that — see `reconciled_recently`.
|
|
||||||
|
|
||||||
Per PR:
|
|
||||||
behind_main commits on main not in the branch — the "out of date" measure
|
|
||||||
ahead commits on the branch not on main
|
|
||||||
mergeable gitea's own verdict (false = conflicts, needs a rebase)
|
|
||||||
diff_files files the PR touches (0 = nothing left to merge)
|
|
||||||
adds_images the `+ image:` lines it introduces
|
|
||||||
already_in_main those `+ image:` lines ALREADY present in main -> the bump landed another way
|
|
||||||
obsolete true when every image it adds is already in main (nothing to contribute)
|
|
||||||
ci newest `!testme` verdict + build number parsed from the PR comments
|
|
||||||
branch_kind upgrade / fix / ci-artifact (`ci/*` sweep + probe branches) / other
|
|
||||||
age_days, stale_days (since last update)
|
|
||||||
|
|
||||||
pr-survey.py [recipe ...] [--json]
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import base64
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
import sys
|
|
||||||
import urllib.error
|
|
||||||
import urllib.parse
|
|
||||||
import urllib.request
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
|
|
||||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
|
||||||
USED_RECIPES = os.path.join(HERE, "used-recipes.md")
|
|
||||||
TESTENV = os.environ.get("CCCI_TESTENV", "/srv/cc-ci/.testenv")
|
|
||||||
PUBLIC_ENV = "/srv/cc-ci/cc-ci/.env.public"
|
|
||||||
NS = "recipe-maintainers"
|
|
||||||
|
|
||||||
|
|
||||||
def _env() -> dict:
|
|
||||||
e = {}
|
|
||||||
for path in (PUBLIC_ENV, TESTENV):
|
|
||||||
try:
|
|
||||||
lines = open(path)
|
|
||||||
except OSError:
|
|
||||||
continue
|
|
||||||
for ln in lines:
|
|
||||||
ln = ln.strip()
|
|
||||||
if "=" in ln and not ln.startswith("#"):
|
|
||||||
k, v = ln.split("=", 1)
|
|
||||||
e[k] = v.strip().strip('"').strip("'")
|
|
||||||
return e
|
|
||||||
|
|
||||||
|
|
||||||
ENV = _env()
|
|
||||||
GITEA = os.environ.get("GITEA_URL") or ENV.get("GITEA_URL", "git.autonomic.zone")
|
|
||||||
_AUTH = base64.b64encode(
|
|
||||||
f"{os.environ.get('GITEA_USERNAME') or ENV.get('GITEA_USERNAME','')}:"
|
|
||||||
f"{os.environ.get('GITEA_PASSWORD') or ENV.get('GITEA_PASSWORD','')}".encode()
|
|
||||||
).decode()
|
|
||||||
|
|
||||||
|
|
||||||
def _get(path: str, raw: bool = False):
|
|
||||||
req = urllib.request.Request(
|
|
||||||
f"https://{GITEA}{path}",
|
|
||||||
headers={"Authorization": f"Basic {_AUTH}", "User-Agent": "cc-ci-pr-survey"},
|
|
||||||
)
|
|
||||||
with urllib.request.urlopen(req, timeout=60) as r:
|
|
||||||
body = r.read()
|
|
||||||
return body.decode(errors="replace") if raw else json.loads(body)
|
|
||||||
|
|
||||||
|
|
||||||
def _days(iso: str | None) -> int | None:
|
|
||||||
if not iso:
|
|
||||||
return None
|
|
||||||
try:
|
|
||||||
d = datetime.fromisoformat(iso.replace("Z", "+00:00"))
|
|
||||||
except ValueError:
|
|
||||||
return None
|
|
||||||
return (datetime.now(timezone.utc) - d).days
|
|
||||||
|
|
||||||
|
|
||||||
def _branch_kind(ref: str) -> str:
|
|
||||||
if ref.startswith("ci/"):
|
|
||||||
return "ci-artifact" # regall/cfold sweeps + testme probes; never meant to merge
|
|
||||||
if ref.startswith("upgrade"):
|
|
||||||
return "upgrade"
|
|
||||||
if re.match(r"^(fix|feat|chore|revert)", ref):
|
|
||||||
return "fix"
|
|
||||||
return "other"
|
|
||||||
|
|
||||||
|
|
||||||
def _main_images(recipe: str) -> set[str]:
|
|
||||||
"""Image refs pinned on the mirror's main — the baseline a PR is judged against."""
|
|
||||||
out = set()
|
|
||||||
for f in ("compose.yml",):
|
|
||||||
try:
|
|
||||||
txt = _get(f"/{NS}/{recipe}/raw/branch/main/{f}", raw=True)
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
continue
|
|
||||||
for m in re.finditer(r"^\s*image:\s*[\"']?([^\"'\s]+)", txt, re.M):
|
|
||||||
out.add(m.group(1))
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def _ci_verdict(recipe: str, number: int) -> dict:
|
|
||||||
"""Newest cc-ci !testme outcome recorded on the PR."""
|
|
||||||
try:
|
|
||||||
cs = _get(f"/api/v1/repos/{NS}/{recipe}/issues/{number}/comments?limit=100")
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
return {"verdict": "unknown", "build": None}
|
|
||||||
for c in reversed(cs):
|
|
||||||
b = c.get("body") or ""
|
|
||||||
if "cc-ci:testme" not in b:
|
|
||||||
continue
|
|
||||||
m = re.search(r"/cc-ci/(\d+)", b)
|
|
||||||
if "✅" in b or "passed" in b:
|
|
||||||
return {"verdict": "passed", "build": m.group(1) if m else None}
|
|
||||||
if "❌" in b or "failure" in b:
|
|
||||||
return {"verdict": "failed", "build": m.group(1) if m else None}
|
|
||||||
if "⏳" in b or "in progress" in b:
|
|
||||||
return {"verdict": "running", "build": m.group(1) if m else None}
|
|
||||||
return {"verdict": "never-run", "build": None}
|
|
||||||
|
|
||||||
|
|
||||||
def survey_pr(recipe: str, pr: dict, main_images: set[str]) -> dict:
|
|
||||||
n = pr["number"]
|
|
||||||
head = pr["head"]["ref"]
|
|
||||||
row = {
|
|
||||||
"recipe": recipe, "number": n, "title": pr.get("title", ""), "head": head,
|
|
||||||
"url": pr.get("html_url"), "branch_kind": _branch_kind(head),
|
|
||||||
"age_days": _days(pr.get("created_at")), "stale_days": _days(pr.get("updated_at")),
|
|
||||||
"mergeable": pr.get("mergeable"),
|
|
||||||
}
|
|
||||||
try:
|
|
||||||
row["behind_main"] = _get(
|
|
||||||
f"/api/v1/repos/{NS}/{recipe}/compare/{urllib.parse.quote(head, safe='')}...main"
|
|
||||||
).get("total_commits", 0)
|
|
||||||
row["ahead"] = _get(
|
|
||||||
f"/api/v1/repos/{NS}/{recipe}/compare/main...{urllib.parse.quote(head, safe='')}"
|
|
||||||
).get("total_commits", 0)
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
row["behind_main"], row["ahead"] = None, None
|
|
||||||
# A FAILED diff fetch must never look like an empty diff: gitea#4 404s on .diff (force-pushed
|
|
||||||
# branch) and would otherwise be flagged EMPTY-DIFF and closed — while being a verified, green,
|
|
||||||
# needed fix. Unknown is its own state.
|
|
||||||
diff = None
|
|
||||||
try:
|
|
||||||
body = _get(f"/{NS}/{recipe}/pulls/{n}.diff", raw=True)
|
|
||||||
if body.lstrip().startswith(("diff --git", "From ")) or not body.strip():
|
|
||||||
diff = body
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
diff = None
|
|
||||||
row["diff_files"] = None if diff is None else len(re.findall(r"^diff --git ", diff, re.M))
|
|
||||||
adds = re.findall(r"^\+\s*image:\s*[\"']?([^\"'\s]+)", diff or "", re.M)
|
|
||||||
row["adds_images"] = sorted(set(adds))
|
|
||||||
row["already_in_main"] = sorted({i for i in set(adds) if i in main_images})
|
|
||||||
# Nothing left to contribute: it touches files but every image it introduces is already pinned.
|
|
||||||
# Only claim obsolete when the diff was actually READ. No diff, no verdict.
|
|
||||||
row["obsolete"] = diff is not None and bool(adds) and set(adds).issubset(main_images)
|
|
||||||
row["ci"] = _ci_verdict(recipe, n)
|
|
||||||
return row
|
|
||||||
|
|
||||||
|
|
||||||
def all_recipes() -> list[str]:
|
|
||||||
out = []
|
|
||||||
for ln in open(USED_RECIPES):
|
|
||||||
p = ln.split()
|
|
||||||
if len(p) >= 2 and not ln.startswith(("#", "`")) and p[1] in ("weekly", "external"):
|
|
||||||
out.append(p[0])
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> int:
|
|
||||||
ap = argparse.ArgumentParser(description=__doc__,
|
|
||||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
||||||
ap.add_argument("recipes", nargs="*")
|
|
||||||
ap.add_argument("--json", action="store_true")
|
|
||||||
a = ap.parse_args()
|
|
||||||
|
|
||||||
rows = []
|
|
||||||
for r in (a.recipes or all_recipes()):
|
|
||||||
try:
|
|
||||||
prs = _get(f"/api/v1/repos/{NS}/{r}/pulls?state=open&limit=50")
|
|
||||||
except urllib.error.HTTPError as e:
|
|
||||||
rows.append({"recipe": r, "error": f"HTTP {e.code}"})
|
|
||||||
continue
|
|
||||||
if not prs:
|
|
||||||
continue
|
|
||||||
mi = _main_images(r)
|
|
||||||
for pr in prs:
|
|
||||||
rows.append(survey_pr(r, pr, mi))
|
|
||||||
|
|
||||||
if a.json:
|
|
||||||
print(json.dumps(rows, indent=2))
|
|
||||||
return 0
|
|
||||||
|
|
||||||
print(f"{len(rows)} open PR(s)\n")
|
|
||||||
for x in sorted(rows, key=lambda z: (z.get("recipe", ""), z.get("number", 0))):
|
|
||||||
if x.get("error"):
|
|
||||||
print(f" {x['recipe']}: {x['error']}")
|
|
||||||
continue
|
|
||||||
flags = []
|
|
||||||
if x["obsolete"]:
|
|
||||||
flags.append("OBSOLETE(images already in main)")
|
|
||||||
if x["branch_kind"] == "ci-artifact":
|
|
||||||
flags.append("CI-ARTIFACT")
|
|
||||||
if x["diff_files"] == 0:
|
|
||||||
flags.append("EMPTY-DIFF")
|
|
||||||
if x["diff_files"] is None:
|
|
||||||
flags.append("DIFF-UNREADABLE(do not close on this)")
|
|
||||||
if x["mergeable"] is False:
|
|
||||||
flags.append("CONFLICTS")
|
|
||||||
if (x["behind_main"] or 0) > 0:
|
|
||||||
flags.append(f"BEHIND-{x['behind_main']}")
|
|
||||||
print(f" {x['recipe']}#{x['number']:<3} {x['title'][:52]}")
|
|
||||||
print(f" {x['branch_kind']:12} age={x['age_days']}d idle={x['stale_days']}d "
|
|
||||||
f"ci={x['ci']['verdict']}({x['ci']['build'] or '-'}) files={x['diff_files'] if x['diff_files'] is not None else '?'}")
|
|
||||||
if x["adds_images"]:
|
|
||||||
print(f" adds: {', '.join(i.split('/')[-1] for i in x['adds_images'][:4])}")
|
|
||||||
if flags:
|
|
||||||
print(f" >> {' | '.join(flags)}")
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
sys.exit(main())
|
|
||||||
@@ -12,7 +12,7 @@ Subcommands (the /recipe-report agent runs them around its own review/classifica
|
|||||||
publish OUT.html DATE [KIND] copy to cc-ci:/var/lib/cc-ci-reports/<KIND>-DATE.html and regen the
|
publish OUT.html DATE [KIND] copy to cc-ci:/var/lib/cc-ci-reports/<KIND>-DATE.html and regen the
|
||||||
archive index. KIND is `week` (default, the weekly /recipe-report) or `cve`
|
archive index. KIND is `week` (default, the weekly /recipe-report) or `cve`
|
||||||
(a /cve-check advisory sweep). BOTH kinds appear in the SAME archive index,
|
(a /cve-check advisory sweep). BOTH kinds appear in the SAME archive index,
|
||||||
newest first, each row suffixed "full" or "CVE check"; the distinct
|
newest first, each row suffixed "full report" or "CVE check"; the distinct
|
||||||
filename prefix just stops a sweep overwriting a weekly edition.
|
filename prefix just stops a sweep overwriting a weekly edition.
|
||||||
|
|
||||||
Page order: short lead → the full wire table (priority-sorted, CVEs column) → Addendum → Security
|
Page order: short lead → the full wire table (priority-sorted, CVEs column) → Addendum → Security
|
||||||
@@ -46,27 +46,24 @@ from datetime import datetime, timezone
|
|||||||
|
|
||||||
LOGDIR = "/srv/cc-ci/.cc-ci-logs"
|
LOGDIR = "/srv/cc-ci/.cc-ci-logs"
|
||||||
TESTENV = "/srv/cc-ci/.testenv"
|
TESTENV = "/srv/cc-ci/.testenv"
|
||||||
PUBLIC_ENV = "/srv/cc-ci/cc-ci/.env.public"
|
|
||||||
INFRA = {"cc-ci", "cc-ci-orchestrator", "cc-ci-secrets"}
|
INFRA = {"cc-ci", "cc-ci-orchestrator", "cc-ci-secrets"}
|
||||||
HOST_REPORTS = "/var/lib/cc-ci-reports"
|
HOST_REPORTS = "/var/lib/cc-ci-reports"
|
||||||
# Both kinds live in ONE archive, distinguished by a suffix on a common title.
|
# Both kinds live in ONE archive, distinguished by a suffix on a common title.
|
||||||
# prefix -> (page title, index label)
|
# prefix -> (page title, index label)
|
||||||
KINDS = {"week": ("The Recipe Report", "Week of {d} — full"),
|
KINDS = {"week": ("The Recipe Report", "Week of {d} — full report"),
|
||||||
"cve": ("The Recipe Report — CVE check", "{d} — CVE check")}
|
"cve": ("The Recipe Report — CVE check", "{d} — CVE check")}
|
||||||
|
|
||||||
|
|
||||||
def _env():
|
def _env():
|
||||||
e = {}
|
e = {}
|
||||||
for path in (PUBLIC_ENV, TESTENV):
|
|
||||||
try:
|
try:
|
||||||
lines = open(path)
|
for ln in open(TESTENV):
|
||||||
except FileNotFoundError:
|
|
||||||
continue
|
|
||||||
for ln in lines:
|
|
||||||
ln = ln.strip()
|
ln = ln.strip()
|
||||||
if "=" in ln and not ln.startswith("#"):
|
if "=" in ln and not ln.startswith("#"):
|
||||||
k, v = ln.split("=", 1)
|
k, v = ln.split("=", 1)
|
||||||
e[k] = v.strip().strip('"').strip("'")
|
e[k] = v.strip().strip('"').strip("'")
|
||||||
|
except FileNotFoundError:
|
||||||
|
pass
|
||||||
return e
|
return e
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1,63 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
# reconcile-upstream — sync recipe mirrors from TRUE upstream. Run this FIRST, always.
|
|
||||||
# ----------------------------------------------------------------------------------
|
|
||||||
# Every recipe we maintain is a MIRROR of a coopcloud recipe. Work done against a stale
|
|
||||||
# mirror is wasted or wrong, in three ways we have actually hit:
|
|
||||||
#
|
|
||||||
# 1. A PR whose changes upstream ALREADY MERGED. mailu #6 (2024.06.57 + redis 8.10,
|
|
||||||
# two internet-facing Roundcube CVEs) sat open and was reported as the fix for
|
|
||||||
# those CVEs — while upstream had merged and released it as 3.1.3+2024.06.57. The
|
|
||||||
# work was done; only our mirror was behind.
|
|
||||||
# 2. A survey that reads the stale mirror and reports "no upgrades available", so a
|
|
||||||
# recipe silently drops out of the weekly run.
|
|
||||||
# 3. Reading the WRONG BRANCH. Several coopcloud recipes keep a stale `main` beside
|
|
||||||
# the real default `master` — gitea's `main` is at 1.24.2-rootless while `master`
|
|
||||||
# has 1.27.1-rootless plus the merged PRs. Reading `main` there says the recipe is
|
|
||||||
# three releases behind and missing two CVSS-9.8 RCE fixes, which reads exactly
|
|
||||||
# like a real finding. open-recipe-pr.sh resolves the default branch itself
|
|
||||||
# (main OR master) — never hand-pick one.
|
|
||||||
#
|
|
||||||
# This is deterministic: it force-syncs each mirror's `main` to upstream's default
|
|
||||||
# branch and closes any mirror PR whose changes are already upstream. No AI judgement.
|
|
||||||
#
|
|
||||||
# reconcile-upstream.sh <recipe>... # specific recipes
|
|
||||||
# reconcile-upstream.sh --all # every recipe in used-recipes.md
|
|
||||||
#
|
|
||||||
# Safe to run repeatedly; a mirror already in sync is a no-op. Recipe work lives in
|
|
||||||
# BRANCHES, never on mirror `main`, so force-syncing `main` discards nothing.
|
|
||||||
set -o errexit -o nounset -o pipefail
|
|
||||||
|
|
||||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
||||||
ORCH="$(dirname "$HERE")"
|
|
||||||
SSH="${SSH:-cc-ci}"
|
|
||||||
RECONCILE="${RECONCILE:-$ORCH/.claude/skills/recipe-upgrade/open-recipe-pr.sh}"
|
|
||||||
USED_RECIPES="${USED_RECIPES:-$HERE/used-recipes.md}"
|
|
||||||
|
|
||||||
[ -f "$RECONCILE" ] || { echo "ERROR: reconcile helper not found: $RECONCILE" >&2; exit 1; }
|
|
||||||
set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a
|
|
||||||
: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:=git.autonomic.zone}"
|
|
||||||
|
|
||||||
if [ "${1:-}" = "--all" ]; then
|
|
||||||
mapfile -t RECIPES < <(awk '!/^[[:space:]]*#/ && ($2=="weekly" || $2=="external") {print $1}' "$USED_RECIPES")
|
|
||||||
else
|
|
||||||
[ "$#" -gt 0 ] || { echo "usage: reconcile-upstream.sh <recipe>... | --all" >&2; exit 2; }
|
|
||||||
RECIPES=("$@")
|
|
||||||
fi
|
|
||||||
|
|
||||||
synced=0; closed=0; failed=0
|
|
||||||
for r in "${RECIPES[@]}"; do
|
|
||||||
echo "── $r"
|
|
||||||
if out="$(ssh "$SSH" "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s $r --reconcile-only" < "$RECONCILE" 2>&1)"; then
|
|
||||||
printf '%s\n' "$out" | grep -E "Force-syncing|already in sync|closed PR|still open|✓" | sed 's/^/ /' || true
|
|
||||||
synced=$((synced + 1))
|
|
||||||
closed=$((closed + $(printf '%s' "$out" | grep -c "closed PR" || true)))
|
|
||||||
else
|
|
||||||
printf '%s\n' "$out" | tail -3 | sed 's/^/ /'
|
|
||||||
echo " ✗ FAILED — do NOT proceed against this mirror until it reconciles"
|
|
||||||
failed=$((failed + 1))
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
echo
|
|
||||||
echo "reconcile-upstream: ${synced} mirror(s) synced, ${closed} already-upstream PR(s) closed, ${failed} failed"
|
|
||||||
[ "$failed" -eq 0 ]
|
|
||||||
@@ -1,459 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""resolve-images — what version is each of a recipe's images on, and what is newest?
|
|
||||||
|
|
||||||
An abra-independent version resolver. `abra recipe upgrade` is the normal path, but it has a hard
|
|
||||||
failure mode: an image pinned with BOTH a tag and a digest makes it FATA and abandon the WHOLE
|
|
||||||
recipe — even images it already parsed. immich pins two that way:
|
|
||||||
|
|
||||||
ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf6…
|
|
||||||
docker.io/valkey/valkey:9@sha256:3acc…
|
|
||||||
|
|
||||||
so immich contributes NO version data at all and silently drops out of every survey. That is
|
|
||||||
indistinguishable from "up to date" unless a human notices the missing row — which is exactly how it
|
|
||||||
kept getting skipped, and why a CVE sweep reported it as unknown.
|
|
||||||
|
|
||||||
This reads the compose files directly and queries the registries itself, so a digest pin is just a
|
|
||||||
digest pin. Output is JSON (default) or a table.
|
|
||||||
|
|
||||||
resolve-images.py <recipe> [--ssh HOST] [--recipe-dir DIR] [--table] [--only IMAGE]
|
|
||||||
|
|
||||||
The cc-ci host has no python3, so `--ssh cc-ci` reads the compose files from that host's checkout
|
|
||||||
over ssh and does the resolving locally. That keeps the source of truth the SAME tree abra and CI
|
|
||||||
use, rather than a second copy that can drift.
|
|
||||||
|
|
||||||
TAG SHAPES. Registries mix wildly different tag conventions in one repo, so "newest" is meaningless
|
|
||||||
without a shape. Each tag is reduced to a signature by replacing digit runs with '#':
|
|
||||||
|
|
||||||
v3.1.0 -> v#.#.#
|
|
||||||
1.27.1-rootless -> #.#.#-rootless
|
|
||||||
8.10-alpine -> #.#-alpine
|
|
||||||
14-vectorchord0.4.3-pgvectors0.2.0 -> #-vectorchord#.#.#-pgvectors#.#.#
|
|
||||||
|
|
||||||
Only tags sharing the CURRENT pin's shape are candidates. That keeps `-alpine` on `-alpine`, and
|
|
||||||
stops a `latest`/`release`/`sha-…` tag from ever being proposed as an upgrade.
|
|
||||||
|
|
||||||
TWO ANSWERS, NOT ONE. It reports `newest_same_shape` AND `newest_within_major` (same leading number).
|
|
||||||
For a plain app image they usually agree. For a compatibility-pinned sidecar they do not, and taking
|
|
||||||
the max would be wrong: immich's postgres tag encodes the pg major plus the vectorchord/pgvectors
|
|
||||||
versions that immich-server is built against, so jumping pg major because a newer tag exists breaks
|
|
||||||
the deployment. The caller picks; this tool refuses to guess and shows both.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import glob
|
|
||||||
import shlex
|
|
||||||
import subprocess
|
|
||||||
import gzip
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
import sys
|
|
||||||
import time
|
|
||||||
import urllib.error
|
|
||||||
import urllib.parse
|
|
||||||
import urllib.request
|
|
||||||
|
|
||||||
UA = "cc-ci-resolve-images (+https://git.autonomic.zone/recipe-maintainers/cc-ci)"
|
|
||||||
TIMEOUT = int(os.environ.get("RESOLVE_IMAGES_TIMEOUT", "45"))
|
|
||||||
RECIPE_DIR = os.environ.get("ABRA_RECIPE_DIR", os.path.expanduser("~/.abra/recipes"))
|
|
||||||
MAX_TAG_PAGES = int(os.environ.get("RESOLVE_IMAGES_MAX_PAGES", "40"))
|
|
||||||
|
|
||||||
IMAGE_RE = re.compile(r"""^\s*image:\s*["']?([^"'\s]+)["']?\s*$""", re.M)
|
|
||||||
|
|
||||||
|
|
||||||
RETRIES = int(os.environ.get("RESOLVE_IMAGES_RETRIES", "4"))
|
|
||||||
|
|
||||||
|
|
||||||
def _fetch(url: str, headers: dict | None = None) -> bytes:
|
|
||||||
"""GET with backoff on rate limits.
|
|
||||||
|
|
||||||
Docker Hub throttles anonymous clients hard, and a sweep re-reads the same popular repos
|
|
||||||
(nginx, redis, postgres) for recipe after recipe. A 429 mid-sweep used to surface as
|
|
||||||
'unresolved', which is indistinguishable from a real lookup failure — so retry, and let the
|
|
||||||
per-repo cache below remove most of the requests entirely."""
|
|
||||||
h = {"User-Agent": UA, "Accept-Encoding": "gzip"}
|
|
||||||
h.update(headers or {})
|
|
||||||
delay = 2.0
|
|
||||||
for attempt in range(RETRIES):
|
|
||||||
try:
|
|
||||||
with urllib.request.urlopen(urllib.request.Request(url, headers=h), timeout=TIMEOUT) as r:
|
|
||||||
raw = r.read()
|
|
||||||
if r.headers.get("Content-Encoding") == "gzip":
|
|
||||||
raw = gzip.decompress(raw)
|
|
||||||
return raw
|
|
||||||
except urllib.error.HTTPError as e:
|
|
||||||
if e.code in (429, 503) and attempt < RETRIES - 1:
|
|
||||||
time.sleep(delay)
|
|
||||||
delay *= 2
|
|
||||||
continue
|
|
||||||
raise
|
|
||||||
raise RuntimeError("unreachable")
|
|
||||||
|
|
||||||
|
|
||||||
_HUB_JWT: list = []
|
|
||||||
|
|
||||||
|
|
||||||
def _hub_auth() -> dict:
|
|
||||||
"""Authenticated Docker Hub calls get a far higher rate limit than anonymous ones.
|
|
||||||
|
|
||||||
Credentials come from /srv/cc-ci/.testenv (DOCKERHUB_USERNAME / DOCKERHUB_TOKEN), the same pair
|
|
||||||
the CI host already uses. Absent creds are fine — the sweep just runs anonymous and slower."""
|
|
||||||
if _HUB_JWT:
|
|
||||||
return _HUB_JWT[0]
|
|
||||||
env = {}
|
|
||||||
try:
|
|
||||||
for ln in open(os.environ.get("CCCI_TESTENV", "/srv/cc-ci/.testenv")):
|
|
||||||
if "=" in ln and not ln.strip().startswith("#"):
|
|
||||||
k, v = ln.strip().split("=", 1)
|
|
||||||
env[k] = v.strip().strip("\"'")
|
|
||||||
except OSError:
|
|
||||||
pass
|
|
||||||
u = os.environ.get("DOCKERHUB_USERNAME") or env.get("DOCKERHUB_USERNAME")
|
|
||||||
t = os.environ.get("DOCKERHUB_TOKEN") or env.get("DOCKERHUB_TOKEN")
|
|
||||||
hdrs = {}
|
|
||||||
if u and t:
|
|
||||||
try:
|
|
||||||
body = json.dumps({"username": u, "password": t}).encode()
|
|
||||||
req = urllib.request.Request("https://hub.docker.com/v2/users/login",
|
|
||||||
data=body, method="POST",
|
|
||||||
headers={"Content-Type": "application/json", "User-Agent": UA})
|
|
||||||
with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
|
|
||||||
tokj = json.load(r).get("token")
|
|
||||||
if tokj:
|
|
||||||
hdrs = {"Authorization": f"JWT {tokj}"}
|
|
||||||
except Exception: # noqa: BLE001 — anonymous is a valid fallback
|
|
||||||
hdrs = {}
|
|
||||||
_HUB_JWT.append(hdrs)
|
|
||||||
return hdrs
|
|
||||||
|
|
||||||
|
|
||||||
def _json(url: str, headers: dict | None = None):
|
|
||||||
return json.loads(_fetch(url, headers))
|
|
||||||
|
|
||||||
|
|
||||||
def shape(tag: str) -> str:
|
|
||||||
"""Signature of a tag with every digit run replaced by '#'. See module docstring."""
|
|
||||||
return re.sub(r"\d+", "#", tag)
|
|
||||||
|
|
||||||
|
|
||||||
def vkey(tag: str) -> tuple:
|
|
||||||
"""Ordering key: every number in the tag, in order. '1.27.10' > '1.27.9'; text ignored."""
|
|
||||||
return tuple(int(x) for x in re.findall(r"\d+", tag))
|
|
||||||
|
|
||||||
|
|
||||||
def parse_ref(ref: str) -> dict:
|
|
||||||
"""Split an image reference into registry / repo / tag / digest."""
|
|
||||||
digest = None
|
|
||||||
if "@" in ref:
|
|
||||||
ref, _, digest = ref.partition("@")
|
|
||||||
host, repo, tag = "docker.io", ref, "latest"
|
|
||||||
# A leading component is a REGISTRY only when there is a path after it. Without the slash test,
|
|
||||||
# a bare `postgres:15.18` looks like host "postgres:15.18" because of the tag's colon — which
|
|
||||||
# silently sent every library image to a nonexistent registry.
|
|
||||||
if "/" in ref:
|
|
||||||
first = ref.split("/")[0]
|
|
||||||
if "." in first or ":" in first or first == "localhost":
|
|
||||||
host, _, repo = ref.partition("/")
|
|
||||||
if ":" in repo.split("/")[-1]:
|
|
||||||
repo, _, tag = repo.rpartition(":")
|
|
||||||
if host == "docker.io" and "/" not in repo:
|
|
||||||
repo = f"library/{repo}" # bare `redis` is really `library/redis`
|
|
||||||
return {"registry": host, "repo": repo, "tag": tag, "digest": digest}
|
|
||||||
|
|
||||||
|
|
||||||
HUB_RECENT_PAGES = int(os.environ.get("RESOLVE_IMAGES_HUB_PAGES", "10"))
|
|
||||||
|
|
||||||
|
|
||||||
def _hub_tag_exists(repo: str, tag: str) -> bool:
|
|
||||||
try:
|
|
||||||
_json(f"https://hub.docker.com/v2/repositories/{repo}/tags/{tag}", _hub_auth())
|
|
||||||
return True
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def _hub_tags(repo: str) -> list[str]:
|
|
||||||
"""Recently-pushed tags, newest first.
|
|
||||||
|
|
||||||
Popular Docker Hub repos carry many thousands of tags, so a full enumeration is impractical —
|
|
||||||
but it is also unnecessary: a tag NEWER than the one we run must have been pushed AFTER it, so
|
|
||||||
ordering by last_updated and reading a bounded recent window is sufficient to find any upgrade.
|
|
||||||
(ghcr offers no ordering, which is why that path needs a different strategy.)
|
|
||||||
"""
|
|
||||||
tags, url = [], (f"https://hub.docker.com/v2/repositories/{repo}/tags"
|
|
||||||
f"?page_size=100&ordering=last_updated")
|
|
||||||
auth = _hub_auth()
|
|
||||||
for _ in range(HUB_RECENT_PAGES):
|
|
||||||
d = _json(url, auth)
|
|
||||||
tags += [r["name"] for r in d.get("results", [])]
|
|
||||||
url = d.get("next")
|
|
||||||
if not url:
|
|
||||||
break
|
|
||||||
return tags
|
|
||||||
|
|
||||||
|
|
||||||
def _oci_bearer(host: str, repo: str) -> dict:
|
|
||||||
"""Token for an OCI registry, discovered from its own auth challenge.
|
|
||||||
|
|
||||||
Registries do NOT share a token endpoint. ghcr answers at /token?scope=…&service=ghcr.io, but
|
|
||||||
lscr.io and dock.mau.dev advertise different realms, and assuming ghcr's shape made both 401 —
|
|
||||||
which then read as "could not resolve" rather than "asked the wrong URL". The registry tells us
|
|
||||||
where to go in its WWW-Authenticate header; use that."""
|
|
||||||
try:
|
|
||||||
urllib.request.urlopen(
|
|
||||||
urllib.request.Request(f"https://{host}/v2/{repo}/tags/list?n=1",
|
|
||||||
headers={"User-Agent": UA}), timeout=TIMEOUT)
|
|
||||||
return {} # no auth needed
|
|
||||||
except urllib.error.HTTPError as e:
|
|
||||||
if e.code != 401:
|
|
||||||
return {}
|
|
||||||
chal = e.headers.get("WWW-Authenticate", "") or ""
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
return {}
|
|
||||||
if not chal.lower().startswith("bearer"):
|
|
||||||
return {}
|
|
||||||
parts = dict(re.findall(r'(\w+)="([^"]*)"', chal))
|
|
||||||
realm = parts.get("realm")
|
|
||||||
if not realm:
|
|
||||||
return {}
|
|
||||||
q = {"service": parts.get("service", host), "scope": parts.get("scope", f"repository:{repo}:pull")}
|
|
||||||
url = realm + ("&" if "?" in realm else "?") + urllib.parse.urlencode(q)
|
|
||||||
try:
|
|
||||||
tok = (_json(url) or {}).get("token") or (_json(url) or {}).get("access_token")
|
|
||||||
return {"Authorization": f"Bearer {tok}"} if tok else {}
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
return {}
|
|
||||||
|
|
||||||
|
|
||||||
def _oci_tags(host: str, repo: str) -> list[str]:
|
|
||||||
"""Tags from any OCI/v2 registry, with challenge-derived auth and Link pagination.
|
|
||||||
|
|
||||||
ghcr paginates hard — immich-server has >40,000 tags — and a truncated listing silently hides
|
|
||||||
the newest release line, so follow the cursor and let the caller's integrity check catch a read
|
|
||||||
that never reached the current pin."""
|
|
||||||
hdrs = _oci_bearer(host, repo)
|
|
||||||
tags, url = [], f"https://{host}/v2/{repo}/tags/list?n=1000"
|
|
||||||
for _ in range(MAX_TAG_PAGES):
|
|
||||||
req = urllib.request.Request(url, headers={"User-Agent": UA, **hdrs})
|
|
||||||
with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
|
|
||||||
tags += (json.load(r) or {}).get("tags") or []
|
|
||||||
link = r.headers.get("Link", "") or ""
|
|
||||||
m = re.search(r'<([^>]+)>;\s*rel="next"', link)
|
|
||||||
if not m:
|
|
||||||
break
|
|
||||||
nxt = m.group(1)
|
|
||||||
url = f"https://{host}{nxt}" if nxt.startswith("/") else nxt
|
|
||||||
return tags
|
|
||||||
|
|
||||||
|
|
||||||
def _gh_token() -> str | None:
|
|
||||||
tok = os.environ.get("GITHUB_TOKEN")
|
|
||||||
if tok:
|
|
||||||
return tok.strip()
|
|
||||||
try:
|
|
||||||
return open(os.environ.get("GITHUB_TOKEN_FILE", "/srv/cc-ci/.github-token")).read().strip() or None
|
|
||||||
except OSError:
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def github_release_tags(owner: str, repo: str, max_pages: int = 4) -> list[str]:
|
|
||||||
"""Release tag names for a GitHub repo, newest first.
|
|
||||||
|
|
||||||
FALLBACK for registries whose tag listing cannot be enumerated. ghcr has no ordering and no
|
|
||||||
server-side filter, and immich-machine-learning carries >40,000 tags — a full read is impractical
|
|
||||||
and a partial read silently hides the newest release line. The project's RELEASES are ordered,
|
|
||||||
small, and authoritative: container tags track them. (The GitHub Packages API would answer this
|
|
||||||
directly but needs a scoped token; this scan's token deliberately has none.)
|
|
||||||
"""
|
|
||||||
hdrs = {"Accept": "application/vnd.github+json"}
|
|
||||||
tok = _gh_token()
|
|
||||||
if tok:
|
|
||||||
hdrs["Authorization"] = f"Bearer {tok}"
|
|
||||||
out = []
|
|
||||||
for page in range(1, max_pages + 1):
|
|
||||||
try:
|
|
||||||
rows = _json(f"https://api.github.com/repos/{owner}/{repo}/releases"
|
|
||||||
f"?per_page=100&page={page}", hdrs)
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
break
|
|
||||||
if not rows:
|
|
||||||
break
|
|
||||||
out += [r.get("tag_name") or "" for r in rows]
|
|
||||||
return [t for t in out if t]
|
|
||||||
|
|
||||||
|
|
||||||
def _release_fallback_repos(registry: str, repo: str) -> list[tuple[str, str]]:
|
|
||||||
"""Candidate GitHub repos whose releases track this image's tags."""
|
|
||||||
if "ghcr.io" not in registry:
|
|
||||||
return []
|
|
||||||
parts = repo.split("/")
|
|
||||||
if len(parts) < 2:
|
|
||||||
return []
|
|
||||||
owner, name = parts[0], parts[-1]
|
|
||||||
cands = [(owner, name)]
|
|
||||||
# ghcr.io/immich-app/immich-machine-learning is built from immich-app/immich.
|
|
||||||
if name.startswith(owner.split("-")[0]):
|
|
||||||
cands.append((owner, owner.split("-")[0]))
|
|
||||||
return cands
|
|
||||||
|
|
||||||
|
|
||||||
_TAG_CACHE: dict[tuple[str, str], tuple[list[str], str | None]] = {}
|
|
||||||
|
|
||||||
|
|
||||||
def list_tags(registry: str, repo: str) -> tuple[list[str], str | None]:
|
|
||||||
if (registry, repo) in _TAG_CACHE:
|
|
||||||
return _TAG_CACHE[(registry, repo)]
|
|
||||||
res = _list_tags_uncached(registry, repo)
|
|
||||||
_TAG_CACHE[(registry, repo)] = res
|
|
||||||
return res
|
|
||||||
|
|
||||||
|
|
||||||
def _list_tags_uncached(registry: str, repo: str) -> tuple[list[str], str | None]:
|
|
||||||
try:
|
|
||||||
return (_hub_tags(repo) if registry in ("docker.io", "registry-1.docker.io")
|
|
||||||
else _oci_tags(registry, repo)), None
|
|
||||||
except urllib.error.HTTPError as e:
|
|
||||||
return [], f"HTTP {e.code}"
|
|
||||||
except Exception as e: # noqa: BLE001
|
|
||||||
return [], f"{type(e).__name__}: {e}"
|
|
||||||
|
|
||||||
|
|
||||||
def resolve(ref: str) -> dict:
|
|
||||||
"""Current pin -> newest same-shape tag, and newest within the current major."""
|
|
||||||
if "${" in ref or "$(" in ref:
|
|
||||||
# The tag is a compose variable (ghost pins `ghost:${IMAGE_VERSION}-alpine`). Its real value
|
|
||||||
# lives in .env, not here. Report it as skipped, never as a failed lookup.
|
|
||||||
return {**parse_ref(ref), "ref": ref, "shape": None, "candidates": 0,
|
|
||||||
"newest_same_shape": None, "newest_within_major": None,
|
|
||||||
"upgrade_available": False,
|
|
||||||
"status": "skipped: templated ref (tag comes from a compose variable)"}
|
|
||||||
info = parse_ref(ref)
|
|
||||||
out = {**info, "ref": ref, "shape": shape(info["tag"]), "status": "ok",
|
|
||||||
"newest_same_shape": None, "newest_within_major": None, "candidates": 0,
|
|
||||||
"upgrade_available": False}
|
|
||||||
tags, err = list_tags(info["registry"], info["repo"])
|
|
||||||
if err:
|
|
||||||
out["status"] = f"error: {err}"
|
|
||||||
return out
|
|
||||||
out["tags_seen"] = len(set(tags))
|
|
||||||
# INTEGRITY CHECK: the tag we are currently running MUST appear in the listing. If it does not,
|
|
||||||
# the listing is incomplete and any "newest" derived from it is a guess — ghcr paginates to tens
|
|
||||||
# of thousands of tags and a truncated read silently hides whole release lines. immich's
|
|
||||||
# machine-learning image is pinned v3.1.0, which EXISTS, yet a short read reported v1.134.0 as
|
|
||||||
# newest; without this check that becomes a confident, wrong answer.
|
|
||||||
if info["tag"] not in set(tags):
|
|
||||||
# Docker Hub: the window is recency-ordered, so the pin being outside it just means the pin
|
|
||||||
# is old — which is fine, because anything NEWER is necessarily inside the window. Confirm
|
|
||||||
# the pin genuinely exists (so a typo is still caught) and carry on.
|
|
||||||
if info["registry"] in ("docker.io", "registry-1.docker.io") and _hub_tag_exists(info["repo"], info["tag"]):
|
|
||||||
out["source"] = f"docker-hub:recent-{HUB_RECENT_PAGES * 100}"
|
|
||||||
tags = list(tags) + [info["tag"]]
|
|
||||||
else:
|
|
||||||
for owner, name in _release_fallback_repos(info["registry"], info["repo"]):
|
|
||||||
rel = github_release_tags(owner, name)
|
|
||||||
if info["tag"] in rel:
|
|
||||||
tags = rel
|
|
||||||
out["source"] = f"github-releases:{owner}/{name}"
|
|
||||||
out["tags_seen"] = len(set(rel))
|
|
||||||
break
|
|
||||||
else:
|
|
||||||
out["status"] = ("error: tag listing incomplete — the current pin "
|
|
||||||
f"{info['tag']!r} is absent from {len(set(tags))} registry tags "
|
|
||||||
f"and from the project's GitHub releases")
|
|
||||||
return out
|
|
||||||
want, cur = out["shape"], vkey(info["tag"])
|
|
||||||
same = [t for t in set(tags) if shape(t) == want and vkey(t)]
|
|
||||||
out["candidates"] = len(same)
|
|
||||||
if not same:
|
|
||||||
# Not a failure: digest-only pins and `latest`/`stable` have no comparable siblings.
|
|
||||||
out["status"] = "no comparable tags (shape has no numeric siblings)"
|
|
||||||
return out
|
|
||||||
newest = max(same, key=vkey)
|
|
||||||
out["newest_same_shape"] = newest
|
|
||||||
if cur:
|
|
||||||
within = [t for t in same if vkey(t)[:1] == cur[:1]]
|
|
||||||
if within:
|
|
||||||
out["newest_within_major"] = max(within, key=vkey)
|
|
||||||
out["upgrade_available"] = bool(cur and vkey(newest) > cur)
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def compose_images_ssh(recipe: str, host: str, recipe_dir: str) -> list[str]:
|
|
||||||
"""Same as compose_images, but the recipe tree lives on another host (cc-ci has no python3)."""
|
|
||||||
# NB: no shell-quoting of the directory — it may legitimately start with ~ or $HOME, and
|
|
||||||
# quoting it stops the remote shell expanding it, which yields an empty (and silent) result.
|
|
||||||
d = f"{recipe_dir}/{shlex.quote(recipe)}".replace("~", "$HOME")
|
|
||||||
cmd = (f'for f in {d}/compose*.yml; do case "$f" in *compose.ccci.yml) continue;; esac; '
|
|
||||||
f'[ -f "$f" ] && {{ cat "$f"; echo; }}; done; exit 0')
|
|
||||||
out = subprocess.run(["ssh", host, cmd], capture_output=True, text=True, timeout=120)
|
|
||||||
if out.returncode != 0:
|
|
||||||
raise RuntimeError(f"ssh {host}: {(out.stderr.strip() or 'no output')[:200]}")
|
|
||||||
if not out.stdout.strip():
|
|
||||||
raise RuntimeError(f"ssh {host}: no compose files found under {d}")
|
|
||||||
refs = []
|
|
||||||
for m in IMAGE_RE.finditer(out.stdout):
|
|
||||||
if m.group(1) not in refs:
|
|
||||||
refs.append(m.group(1))
|
|
||||||
return refs
|
|
||||||
|
|
||||||
|
|
||||||
def compose_images(recipe: str, recipe_dir: str) -> list[str]:
|
|
||||||
"""Every `image:` ref in the recipe's own compose files (the cc-ci overlay is NOT the recipe)."""
|
|
||||||
refs, base = [], os.path.join(recipe_dir, recipe)
|
|
||||||
for path in sorted(glob.glob(os.path.join(base, "compose*.yml"))):
|
|
||||||
if os.path.basename(path) == "compose.ccci.yml":
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
for m in IMAGE_RE.finditer(open(path).read()):
|
|
||||||
if m.group(1) not in refs:
|
|
||||||
refs.append(m.group(1))
|
|
||||||
except OSError:
|
|
||||||
continue
|
|
||||||
return refs
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> int:
|
|
||||||
ap = argparse.ArgumentParser(description=__doc__,
|
|
||||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
||||||
ap.add_argument("recipe")
|
|
||||||
ap.add_argument("--ssh", default=None, metavar="HOST",
|
|
||||||
help="read the recipe's compose files from HOST over ssh (e.g. --ssh cc-ci); "
|
|
||||||
"resolving still happens locally")
|
|
||||||
ap.add_argument("--recipe-dir", default=RECIPE_DIR)
|
|
||||||
ap.add_argument("--table", action="store_true", help="human-readable table instead of JSON")
|
|
||||||
ap.add_argument("--only", default=None, help="resolve just the images whose ref contains this")
|
|
||||||
a = ap.parse_args()
|
|
||||||
|
|
||||||
rdir = a.recipe_dir if a.recipe_dir != RECIPE_DIR or not a.ssh else "~/.abra/recipes"
|
|
||||||
refs = (compose_images_ssh(a.recipe, a.ssh, rdir) if a.ssh
|
|
||||||
else compose_images(a.recipe, a.recipe_dir))
|
|
||||||
if a.only:
|
|
||||||
refs = [r for r in refs if a.only in r]
|
|
||||||
results = [resolve(r) for r in refs]
|
|
||||||
report = {
|
|
||||||
"recipe": a.recipe,
|
|
||||||
"images": results,
|
|
||||||
"upgrades_available": [r["ref"] for r in results if r["upgrade_available"]],
|
|
||||||
"unresolved": [r["ref"] for r in results if r["status"].startswith("error")],
|
|
||||||
# The whole point: distinguish "checked, current" from "could not check".
|
|
||||||
"all_resolved": not any(r["status"].startswith("error") for r in results),
|
|
||||||
}
|
|
||||||
if not a.table:
|
|
||||||
print(json.dumps(report, indent=2))
|
|
||||||
return 0
|
|
||||||
print(f"{a.recipe} — {len(results)} images")
|
|
||||||
for r in results:
|
|
||||||
flag = "UPGRADE" if r["upgrade_available"] else ("ERROR" if r["status"].startswith("error") else "current")
|
|
||||||
print(f" [{flag:7}] {r['repo']}:{r['tag']}" + (" (digest-pinned)" if r["digest"] else ""))
|
|
||||||
print(f" shape={r['shape']} candidates={r['candidates']}"
|
|
||||||
f" newest_same_shape={r['newest_same_shape']}"
|
|
||||||
f" newest_within_major={r['newest_within_major']}")
|
|
||||||
if r["status"] != "ok":
|
|
||||||
print(f" status: {r['status']}")
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
sys.exit(main())
|
|
||||||
@@ -503,195 +503,6 @@ class TestReleaseNoteResolution(unittest.TestCase):
|
|||||||
self.assertNotIn("CVE-OK", rep.get("resolved_by_release_notes") or {})
|
self.assertNotIn("CVE-OK", rep.get("resolved_by_release_notes") or {})
|
||||||
|
|
||||||
|
|
||||||
class TestReleaseLineSemantics(unittest.TestCase):
|
|
||||||
"""A fix inside the numeric window is not a fix on the branch you actually land on."""
|
|
||||||
|
|
||||||
def test_fix_later_on_the_targets_own_line_is_not_counted(self):
|
|
||||||
# ClickHouse fixed CVE-2023-48704 in 23.9.6.20 AND 23.10.5.20. Landing on 23.10.4.25 crosses
|
|
||||||
# the 23.9 fix numerically but is BELOW its own line's fix, so it does not have it.
|
|
||||||
rep = run_scan([gh("ClickHouse/ClickHouse",
|
|
||||||
[adv("CVE-2023-48704", patched="v23.10.5.20; v23.9.6.20; v23.8.8.20")])],
|
|
||||||
v_from="23.4.2.11", v_to="23.10.4.25",
|
|
||||||
urls=["https://github.com/ClickHouse/ClickHouse"])
|
|
||||||
self.assertEqual(rep["fixed_by_this_upgrade"], [])
|
|
||||||
|
|
||||||
def test_fix_earlier_on_the_targets_own_line_is_counted(self):
|
|
||||||
rep = run_scan([gh("ClickHouse/ClickHouse",
|
|
||||||
[adv("CVE-2023-47118", patched="v23.10.2.13; v23.8.6.16")])],
|
|
||||||
v_from="23.4.2.11", v_to="23.10.4.25",
|
|
||||||
urls=["https://github.com/ClickHouse/ClickHouse"])
|
|
||||||
self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2023-47118"])
|
|
||||||
|
|
||||||
def test_fix_exactly_at_the_target_is_counted(self):
|
|
||||||
rep = run_scan([gh("ClickHouse/ClickHouse",
|
|
||||||
[adv("CVE-2023-48298", patched="v23.10.4.25; v23.9.5.29")])],
|
|
||||||
v_from="23.4.2.11", v_to="23.10.4.25",
|
|
||||||
urls=["https://github.com/ClickHouse/ClickHouse"])
|
|
||||||
self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2023-48298"])
|
|
||||||
|
|
||||||
def test_no_fix_on_the_target_line_falls_back_to_the_window(self):
|
|
||||||
# redis fixes 7.4.6/8.0.4/8.2.2 with no 8.10.x entry; landing on 8.10 still has them,
|
|
||||||
# because nothing on the 8.10 line is named as a LATER fix.
|
|
||||||
rep = run_scan([gh("redis/redis", [adv("CVE-2025-49844", patched="7.4.6; 8.0.4; 8.2.2")])],
|
|
||||||
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"])
|
|
||||||
self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2025-49844"])
|
|
||||||
|
|
||||||
|
|
||||||
class TestAlreadyFixedOnFromLine(unittest.TestCase):
|
|
||||||
"""A fix that landed on the line we upgrade FROM was already ours before the upgrade."""
|
|
||||||
|
|
||||||
def test_backport_to_our_own_line_is_not_credited(self):
|
|
||||||
# mattermost patches every maintained line at once. 10.11.22 -> 10.12.4 crosses 10.12.1, but
|
|
||||||
# 10.11.22 is already past 10.11.4, so the deployment HAD the fix. Counting it credits the
|
|
||||||
# upgrade with work it did not do.
|
|
||||||
rep = run_scan([gh("mattermost/mattermost",
|
|
||||||
[adv("CVE-1", patched="10.11.4; 10.12.1; 10.5.12")])],
|
|
||||||
v_from="10.11.22", v_to="10.12.4",
|
|
||||||
urls=["https://github.com/mattermost/mattermost"])
|
|
||||||
self.assertEqual(rep["fixed_by_this_upgrade"], [])
|
|
||||||
|
|
||||||
def test_a_fix_ABOVE_our_position_on_the_same_line_still_counts(self):
|
|
||||||
rep = run_scan([gh("mattermost/mattermost", [adv("CVE-2", patched="10.11.30; 10.12.1")])],
|
|
||||||
v_from="10.11.22", v_to="10.12.4",
|
|
||||||
urls=["https://github.com/mattermost/mattermost"])
|
|
||||||
self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2"])
|
|
||||||
|
|
||||||
def test_placeholders_never_feed_this_rule(self):
|
|
||||||
# "7.4.X" parses to a bare 7.4, which would read as "already fixed at 7.4" and silently drop
|
|
||||||
# a real fix — this is exactly how redis CVE-2024-46981 was lost when the rule was added.
|
|
||||||
rep = run_scan([gh("redis/redis", [adv("CVE-3", patched="6.2.X, 7.2.X, 7.4.X")])],
|
|
||||||
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"])
|
|
||||||
self.assertIn("CVE-3", rep["indeterminate"])
|
|
||||||
self.assertEqual(rep["fixed_by_this_upgrade"], [])
|
|
||||||
|
|
||||||
|
|
||||||
class TestChangelogAttribution(unittest.TestCase):
|
|
||||||
"""Projects that publish no advisory feed still say which release fixed what — in their changelog."""
|
|
||||||
|
|
||||||
CHANGES = """
|
|
||||||
Changes with nginx 1.31.3 11 Aug 2026
|
|
||||||
*) Security: a flaw ... (CVE-2026-60005)
|
|
||||||
*) Security: another ... (CVE-2026-56434)
|
|
||||||
|
|
||||||
Changes with nginx 1.31.2 04 Aug 2026
|
|
||||||
*) Security: something ... (CVE-2026-48142)
|
|
||||||
|
|
||||||
Changes with nginx 1.31.1 21 Jul 2026
|
|
||||||
*) Security: older ... (CVE-2026-9256)
|
|
||||||
|
|
||||||
Changes with nginx 1.20.0 01 Jan 2021
|
|
||||||
*) Security: ancient ... (CVE-2013-2028)
|
|
||||||
"""
|
|
||||||
|
|
||||||
def test_each_cve_is_attributed_to_the_release_that_fixed_it(self):
|
|
||||||
got = A._changelog_versions(self.CHANGES)
|
|
||||||
self.assertEqual(got["CVE-2026-60005"], "1.31.3")
|
|
||||||
self.assertEqual(got["CVE-2026-48142"], "1.31.2")
|
|
||||||
self.assertEqual(got["CVE-2026-9256"], "1.31.1")
|
|
||||||
self.assertEqual(got["CVE-2013-2028"], "1.20.0")
|
|
||||||
|
|
||||||
def _scan(self, wfrom, wto):
|
|
||||||
# nginx publishes NO GitHub advisories — the feed is empty and the changelog is everything.
|
|
||||||
return run_scan(
|
|
||||||
[gh("nginx/nginx", [])],
|
|
||||||
[{"source": "https://nginx.org/en/CHANGES", "status": "ok",
|
|
||||||
"cves": sorted(A._changelog_versions(self.CHANGES)),
|
|
||||||
"context": {}, "fixed_in": A._changelog_versions(self.CHANGES)}],
|
|
||||||
images=[("nginx", wfrom, wto)], urls=["https://github.com/nginx/nginx"])
|
|
||||||
|
|
||||||
def test_window_counts_only_the_releases_it_crosses(self):
|
|
||||||
rep = self._scan("1.31.1", "1.31.3") # 1.31.1 is the FROM, so its CVE is already fixed
|
|
||||||
self.assertEqual(set(rep["fixed_by_this_upgrade"]),
|
|
||||||
{"CVE-2026-48142", "CVE-2026-56434", "CVE-2026-60005"})
|
|
||||||
|
|
||||||
def test_a_narrower_window_counts_fewer(self):
|
|
||||||
rep = self._scan("1.31.2", "1.31.3")
|
|
||||||
self.assertEqual(set(rep["fixed_by_this_upgrade"]), {"CVE-2026-56434", "CVE-2026-60005"})
|
|
||||||
|
|
||||||
def test_ancient_entries_are_not_swept_in(self):
|
|
||||||
# The changelog lists the project's whole history; only the crossed releases may count.
|
|
||||||
rep = self._scan("1.31.1", "1.31.3")
|
|
||||||
self.assertNotIn("CVE-2013-2028", rep["fixed_by_this_upgrade"])
|
|
||||||
|
|
||||||
def test_evidence_is_recorded(self):
|
|
||||||
rep = self._scan("1.31.1", "1.31.3")
|
|
||||||
self.assertEqual(rep["resolved_by_changelog"]["CVE-2026-60005"], "1.31.3")
|
|
||||||
|
|
||||||
|
|
||||||
class TestComposeDerivedWindows(unittest.TestCase):
|
|
||||||
"""Windows read off a compose diff, so nobody has to remember which --image args an upgrade needs."""
|
|
||||||
|
|
||||||
OLD = """
|
|
||||||
services:
|
|
||||||
app:
|
|
||||||
image: "plausible/analytics:v2.0.0"
|
|
||||||
db:
|
|
||||||
image: pgautoupgrade/pgautoupgrade:18-alpine
|
|
||||||
plausible_events_db:
|
|
||||||
image: clickhouse/clickhouse-server:23.4.2.11-alpine
|
|
||||||
volumes:
|
|
||||||
data:
|
|
||||||
"""
|
|
||||||
NEW = """
|
|
||||||
services:
|
|
||||||
app:
|
|
||||||
image: "ghcr.io/plausible/community-edition:v3.2.1"
|
|
||||||
db:
|
|
||||||
image: pgautoupgrade/pgautoupgrade:18-alpine
|
|
||||||
plausible_events_db:
|
|
||||||
image: clickhouse/clickhouse-server:24.12-alpine
|
|
||||||
volumes:
|
|
||||||
data:
|
|
||||||
"""
|
|
||||||
|
|
||||||
def _windows(self, old=None, new=None):
|
|
||||||
pages = {"to": new if new is not None else self.NEW,
|
|
||||||
"from": old if old is not None else self.OLD}
|
|
||||||
with unittest.mock.patch.object(A, "_fetch", lambda u, h=None: pages["to" if "to" in u else "from"]), \
|
|
||||||
unittest.mock.patch.object(A, "_gitea_auth", lambda u: {}):
|
|
||||||
return A.windows_from_compose("http://x/to", "http://x/from")[0]
|
|
||||||
|
|
||||||
def test_app_service_leads_and_sidecars_follow(self):
|
|
||||||
w = self._windows()
|
|
||||||
self.assertEqual(w[0], ("community-edition", "v2.0.0", "v3.2.1"))
|
|
||||||
self.assertIn(("clickhouse-server", "23.4.2.11-alpine", "24.12-alpine"), w)
|
|
||||||
|
|
||||||
def test_unchanged_images_are_not_windows(self):
|
|
||||||
# pgautoupgrade is identical in both; inventing a window for it would be a false count.
|
|
||||||
self.assertNotIn("pgautoupgrade", [n for n, _, _ in self._windows()])
|
|
||||||
|
|
||||||
def test_a_changed_image_REPO_is_still_the_same_service(self):
|
|
||||||
# plausible/analytics -> ghcr.io/plausible/community-edition. Keyed by image repo this reads
|
|
||||||
# as one image vanishing and another appearing, and the app window is lost entirely.
|
|
||||||
w = self._windows()
|
|
||||||
self.assertTrue(any(n == "community-edition" and f == "v2.0.0" for n, f, _ in w))
|
|
||||||
|
|
||||||
def test_no_change_yields_no_windows(self):
|
|
||||||
self.assertEqual(self._windows(old=self.NEW, new=self.NEW), [])
|
|
||||||
|
|
||||||
def test_templated_tags_are_skipped(self):
|
|
||||||
new = self.NEW.replace('ghcr.io/plausible/community-edition:v3.2.1', 'ghost:${IMAGE_VERSION}')
|
|
||||||
self.assertNotIn("ghost", [n for n, _, _ in self._windows(new=new)])
|
|
||||||
|
|
||||||
|
|
||||||
class TestImageNameMatching(unittest.TestCase):
|
|
||||||
"""An image name and its advisory source rarely spell each other exactly."""
|
|
||||||
|
|
||||||
def test_matches_when_the_image_name_is_LONGER_than_the_source(self):
|
|
||||||
# clickhouse/clickhouse-server vs source ClickHouse/ClickHouse — one-directional matching
|
|
||||||
# dropped this window silently when the key came from a compose file.
|
|
||||||
rep = run_scan([gh("ClickHouse/ClickHouse", [adv("CVE-1", patched="23.10.2.13")])],
|
|
||||||
images=[("clickhouse-server", "23.4.2.11", "24.12")],
|
|
||||||
urls=["https://github.com/ClickHouse/ClickHouse"])
|
|
||||||
self.assertIn("github-advisories:ClickHouse/ClickHouse", rep["windows"])
|
|
||||||
self.assertEqual(rep["cve_count_fixed"], 1)
|
|
||||||
|
|
||||||
def test_matches_when_the_image_name_is_SHORTER_than_the_source(self):
|
|
||||||
rep = run_scan([gh("redis/redis", [adv("CVE-2", patched="7.4.1")])],
|
|
||||||
images=[("redis", "7.4", "8.10")], urls=["https://github.com/redis/redis"])
|
|
||||||
self.assertEqual(rep["cve_count_fixed"], 1)
|
|
||||||
|
|
||||||
|
|
||||||
class TestAdjudicationEvidenceAssembly(unittest.TestCase):
|
class TestAdjudicationEvidenceAssembly(unittest.TestCase):
|
||||||
"""Pass 2's JUDGEMENT is a model's and not testable; what IS testable is what it gets shown."""
|
"""Pass 2's JUDGEMENT is a model's and not testable; what IS testable is what it gets shown."""
|
||||||
|
|
||||||
@@ -813,22 +624,16 @@ class TestHistoricReportNumbers(unittest.TestCase):
|
|||||||
for cve in delta:
|
for cve in delta:
|
||||||
self.assertIn("redis", both["cves"][cve]["sources"][0])
|
self.assertIn("redis", both["cves"][cve]["sources"][0])
|
||||||
|
|
||||||
def test_mailu_finds_the_roundcube_pair_without_an_agent(self):
|
def test_mailu_scan_finds_zero_and_says_so_knowingly(self):
|
||||||
# Published as 2 on 2026-08-07, but only because an agent read the release notes; the scan
|
# Published as 2 via the UNION with release-note reading; the scan's own contribution is 0,
|
||||||
# itself contributed 0. It now reaches 2 deterministically: the CVEs appear only on
|
# and 0 here must mean "checked, none", not "could not check".
|
||||||
# github.com/Mailu/Mailu/releases, and release 2024.06.56 (inside the window) names them.
|
|
||||||
rep = self._count("mailu", "2024.06.55", "2024.06.57", [("redis", "8.8.0", "8.10.0")])
|
rep = self._count("mailu", "2024.06.55", "2024.06.57", [("redis", "8.8.0", "8.10.0")])
|
||||||
self.assertEqual(rep["cve_count_fixed"], 2)
|
self.assertEqual(rep["cve_count_fixed"], 0)
|
||||||
self.assertEqual(set(rep["fixed_by_this_upgrade"]), {"CVE-2026-54432", "CVE-2026-54433"})
|
self.assertTrue(rep["count_known"])
|
||||||
# The redis bump fixes nothing new — every advisory it crosses was fixed at or before 8.6.3.
|
|
||||||
self.assertEqual(rep["cve_count_indeterminate"], 0)
|
|
||||||
|
|
||||||
def test_keycloak_26_7_0_to_26_7_1_is_12(self):
|
def test_keycloak_26_7_0_to_26_7_1_is_7(self):
|
||||||
# Was 7 while only the GHSA feed was consulted. keycloak lists five more CVEs in the 26.7.1
|
|
||||||
# release notes' fixed-issues section that it never filed as advisories — the gitea pattern.
|
|
||||||
rep = self._count("keycloak", "26.7.0", "26.7.1")
|
rep = self._count("keycloak", "26.7.0", "26.7.1")
|
||||||
self.assertEqual(rep["cve_count_fixed"], 12)
|
self.assertEqual(rep["cve_count_fixed"], 7)
|
||||||
self.assertEqual(len(rep.get("resolved_by_release_notes") or {}), 5)
|
|
||||||
|
|
||||||
|
|
||||||
def _main():
|
def _main():
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
| service | image | source repo | releases / changelog |
|
| service | image | source repo | releases / changelog |
|
||||||
|---------|-------|-------------|----------------------|
|
|---------|-------|-------------|----------------------|
|
||||||
| app | ghcr.io/bluesky-social/pds | https://github.com/bluesky-social/pds | git tags `vX.Y.Z` 1:1 with image tags (no GitHub Releases objects — `releases/latest` 404s); app-level changes live in the atproto monorepo changelog: https://github.com/bluesky-social/atproto/blob/main/packages/pds/CHANGELOG.md (image `0.4.5NNN` == `@atproto/pds` `0.5.NN`) |
|
| app | ghcr.io/bluesky-social/pds | https://github.com/bluesky-social/pds | git tags `vX.Y.Z` (no GitHub Releases objects — `releases/latest` 404s) |
|
||||||
| caddy | docker.io/caddy:2 | https://github.com/caddyserver/caddy | https://github.com/caddyserver/caddy/releases |
|
| caddy | docker.io/caddy:2 | https://github.com/caddyserver/caddy | https://github.com/caddyserver/caddy/releases |
|
||||||
|
|
||||||
## Standing notes (researched 2026-06-11, bsky phase)
|
## Standing notes (researched 2026-06-11, bsky phase)
|
||||||
|
|||||||
@@ -10,27 +10,8 @@ This is where CVE-2026-60004 + CVE-2026-59774 (both CVSS 9.8, fixed in 1.27.1) w
|
|||||||
and why the 2026-08-03/07 reports under-counted gitea's CVEs. advisory-scan.py fetches every URL
|
and why the 2026-08-03/07 reports under-counted gitea's CVEs. advisory-scan.py fetches every URL
|
||||||
in this file, so keep vendor security pages listed here.
|
in this file, so keep vendor security pages listed here.
|
||||||
| db | postgres | https://github.com/postgres/postgres | https://www.postgresql.org/docs/release/ |
|
| db | postgres | https://github.com/postgres/postgres | https://www.postgresql.org/docs/release/ |
|
||||||
| db (mariadb overlay) | mariadb | https://github.com/MariaDB/server | https://mariadb.com/docs/release-notes/community-server/10.11 (series index — the CVE tables live on the PER-RELEASE pages, e.g. https://mariadb.com/docs/release-notes/community-server/10.11/10.11.19) |
|
|
||||||
|
|
||||||
## Standing notes
|
## Standing notes
|
||||||
- **mariadb overlay** (compose.mariadb.yml): pinned `mariadb:10.11.x` (10.11 LTS, maintained until
|
|
||||||
Feb 2028). ⚠ BOTH db overlays (postgres + mariadb) define the same service name `db`, and
|
|
||||||
`abra recipe upgrade -m` machine output keeps only ONE `db` entry — the postgres one (files are
|
|
||||||
processed alphabetically; the mariadb result is clobbered). ALWAYS check the mariadb pin against
|
|
||||||
Docker Hub directly; the weekly survey's single "db:" line hides it. Per-release security notes
|
|
||||||
(CVE tables — GitHub release bodies only link out):
|
|
||||||
https://mariadb.com/docs/release-notes/community-server/10.11/10.11.3.md
|
|
||||||
https://mariadb.com/docs/release-notes/community-server/10.11/10.11.6.md
|
|
||||||
https://mariadb.com/docs/release-notes/community-server/10.11/10.11.8.md
|
|
||||||
https://mariadb.com/docs/release-notes/community-server/10.11/10.11.11.md
|
|
||||||
https://mariadb.com/docs/release-notes/community-server/10.11/10.11.12.md
|
|
||||||
https://mariadb.com/docs/release-notes/community-server/10.11/10.11.15.md
|
|
||||||
https://mariadb.com/docs/release-notes/community-server/10.11/10.11.16.md
|
|
||||||
https://mariadb.com/docs/release-notes/community-server/10.11/10.11.17.md
|
|
||||||
https://mariadb.com/docs/release-notes/community-server/10.11/10.11.18.md
|
|
||||||
https://mariadb.com/docs/release-notes/community-server/10.11/10.11.19.md
|
|
||||||
(the 10 CVE-bearing releases of the 10.11.2→10.11.19 window, 2026-09-11; append newer pages for
|
|
||||||
later windows).
|
|
||||||
- **postgres major version**: recipe uses plain postgres (not pgautoupgrade); pg_backup.sh uses pg_dump/psql (logical backup, not pg_upgrade). Major version bump (e.g. 15→16) requires operator to manually restore from logical backup — no auto-migration tooling. Bump only within same major (e.g. 15.13→15.18) unless recipe adds pg_upgrade support.
|
- **postgres major version**: recipe uses plain postgres (not pgautoupgrade); pg_backup.sh uses pg_dump/psql (logical backup, not pg_upgrade). Major version bump (e.g. 15→16) requires operator to manually restore from logical backup — no auto-migration tooling. Bump only within same major (e.g. 15.13→15.18) unless recipe adds pg_upgrade support.
|
||||||
- **compose.postgres.yml**: postgres is an optional overlay (not in the main compose.yml). The cc-ci tests determine whether the postgres overlay is used. `abra recipe upgrade` reads it too and will propose a db bump — hold db unless doing a migration-aware major bump.
|
- **compose.postgres.yml**: postgres is an optional overlay (not in the main compose.yml). The cc-ci tests determine whether the postgres overlay is used. `abra recipe upgrade` reads it too and will propose a db bump — hold db unless doing a migration-aware major bump.
|
||||||
- **1.25.0**: Removed deprecated auth sources — legacy auth setups may need reconfiguration.
|
- **1.25.0**: Removed deprecated auth sources — legacy auth setups may need reconfiguration.
|
||||||
|
|||||||
@@ -8,6 +8,4 @@
|
|||||||
## Standing notes
|
## Standing notes
|
||||||
- hedgedoc 1.11.0 (2026): 4 security CVEs fixed (HTML injection, YAML DoS, CSRF via Gist export, rate-limit bypass). No breaking changes, no migrations, no schema changes. Optional new env var `CMD_RATE_LIMIT_USING_CLOUDFLARE` only needed if running behind Cloudflare — not required for standard deployments.
|
- hedgedoc 1.11.0 (2026): 4 security CVEs fixed (HTML injection, YAML DoS, CSRF via Gist export, rate-limit bypass). No breaking changes, no migrations, no schema changes. Optional new env var `CMD_RATE_LIMIT_USING_CLOUDFLARE` only needed if running behind Cloudflare — not required for standard deployments.
|
||||||
- pgautoupgrade: handles Postgres major-version upgrades automatically on container start. Bump ONE major at a time (16→17, then 17→18 on next cycle). The image tag is `<pg-major>-alpine`.
|
- pgautoupgrade: handles Postgres major-version upgrades automatically on container start. Bump ONE major at a time (16→17, then 17→18 on next cycle). The image tag is `<pg-major>-alpine`.
|
||||||
- pgautoupgrade 18+ (deferred, learned 2026-09-11): PG 18 changes the data-directory layout (docker-library/postgres#1259; PGDATA → `/var/lib/postgresql/<MAJOR>/docker`) — this recipe mounts `postgres:/var/lib/postgresql/data` with no PGDATA, and per the pgautoupgrade README ("Error message when mounting data to /var/lib/postgresql/data on Postgres v18+") that needs a dedicated PGDATA/volume-mount change before bumping to 18, or you can get an empty instance instead of an upgraded one. 16→17 verified live on cc-ci 2026-09-11 (`Upgrade to PostgreSQL 17.11 complete.`, data survived).
|
- cc-ci tests use the sqlite backend (default compose.yml), not the postgresql compose override — so pgautoupgrade bumps do not affect CI test coverage.
|
||||||
- hedgedoc 1.12.0 (2026-08): requires Node 20.17+ (app image bundles its own node — not recipe-relevant); webp uploads; some highlight.js languages removed/changed. No breaking changes or migrations for the recipe.
|
|
||||||
- cc-ci tests use the sqlite backend (default compose.yml), not the postgresql compose override — so pgautoupgrade bumps do not affect CI test coverage (verify pg bumps via a direct --chaos deploy with the postgresql override instead).
|
|
||||||
|
|||||||
@@ -50,24 +50,6 @@
|
|||||||
Immich is now **up-to-date** (v3.1.0 is latest stable per GitHub releases; no v3.1.x patch / v3.2.x).
|
Immich is now **up-to-date** (v3.1.0 is latest stable per GitHub releases; no v3.1.x patch / v3.2.x).
|
||||||
PR #4 now diverges from upstream main only on `database` (pgvectors0.3.0 vs 0.2.0) + `redis`
|
PR #4 now diverges from upstream main only on `database` (pgvectors0.3.0 vs 0.2.0) + `redis`
|
||||||
(valkey:9@8e8d64b4 vs @3acc0687) pins — operator decides merge-vs-close.
|
(valkey:9@8e8d64b4 vs @3acc0687) pins — operator decides merge-vs-close.
|
||||||
- **immich-server v3.2.0 (2026-09-10) pins** (from its `docker/docker-compose.yml` at the v3.2.0 tag):
|
|
||||||
`valkey:9@sha256:70739f85ad2ee01a726a965584a0f94895f01b0c60b3cc8b0aeef11eaa6888cf`
|
|
||||||
(= valkey **9.1.1** per image env; a NEWER build than v3.1.0's `8e8d64b4…` = 9.1.0 — but OLDER
|
|
||||||
than the live `9` tag, which by 2026-09-11 moved to `c123e371…` = 9.1.2; pin immich's tested
|
|
||||||
`70739f85…`, NOT the live tag, per the 2026-07-17 precedent) and
|
|
||||||
`postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357…` (SAME combo as v3.1.0 — DB
|
|
||||||
unchanged). No breaking server-side changes in v3.2.0; `example.env` byte-identical to v3.1.0;
|
|
||||||
TypeORM migrations auto-run. Operator-info only: the new opt-in cluster-groups feature requires
|
|
||||||
a per-group facial-recognition reset (user-triggered button, not a deploy migration). **ghcr
|
|
||||||
pagination gotcha (hit 2026-09-11):** `ghcr.io/v2/immich-app/postgres/tags/list` pages are ~100
|
|
||||||
tags, NOT purely lexicographic, and the `Link:` header returns a RELATIVE url (`/v2/...`) —
|
|
||||||
prefix the host or pagination silently stops at page 1; the pinned pg tag `14-vectorchord0.4.3-
|
|
||||||
pgvectors0.2.0` is on **page 2** of 174 (a first-page probe wrongly suggests it's gone). Upstream
|
|
||||||
coopcloud has open renovate PRs #18 (v3.2.0 monorepo bump = server+ML only), #17 (valkey digest
|
|
||||||
→ live c123e37 — not immich's tested pin), #16 (postgres → pg16 — unsupported by immich v3.2.0,
|
|
||||||
reject). Done in the 2026-09-11 upgrade (v3.1.0→v3.2.0, extending PR #4 from upstream main
|
|
||||||
794560f; PR #4's pgvectors0.3.0/valkey-8e8d64b4 divergences superseded — DB realigned to
|
|
||||||
immich's official combo, which upstream main itself chose 2026-08-07).
|
|
||||||
- **2026-08-07 INFRA note: cc-ci runner's gitea clone-token is STALE (HTTP 401).** `!testme` build
|
- **2026-08-07 INFRA note: cc-ci runner's gitea clone-token is STALE (HTTP 401).** `!testme` build
|
||||||
#1210 died at the recipe `git clone` step (~5s, before any deploy) with `could not read Username for
|
#1210 died at the recipe `git clone` step (~5s, before any deploy) with `could not read Username for
|
||||||
'https://git.autonomic.zone'` — the runner's mounted gitea token `13e299f2…` is rejected (verified
|
'https://git.autonomic.zone'` — the runner's mounted gitea token `13e299f2…` is rejected (verified
|
||||||
|
|||||||
@@ -18,11 +18,6 @@
|
|||||||
- AUTO_MIGRATIONS=true means DB migrations run automatically on backend startup. No manual step needed.
|
- AUTO_MIGRATIONS=true means DB migrations run automatically on backend startup. No manual step needed.
|
||||||
- Minio tag uses a date-based RELEASE.YYYY-MM-DDTHH-MM-SSZ format — abra cannot parse it for upgrades;
|
- Minio tag uses a date-based RELEASE.YYYY-MM-DDTHH-MM-SSZ format — abra cannot parse it for upgrades;
|
||||||
check manually on https://github.com/minio/minio/releases.
|
check manually on https://github.com/minio/minio/releases.
|
||||||
- **2026-08-14: Minio stopped publishing Docker images after RELEASE.2025-09-07T16-13-09Z.**
|
|
||||||
GitHub has a newer release (`RELEASE.2025-10-15T17-29-55Z`, published 2025-10-16, with CVE fix
|
|
||||||
GHSA-jjjj-jwhf-8rgr), but the Docker image was never pushed to Docker Hub (returns 404; release
|
|
||||||
notes say "clone the source and build the latest container"). quay.io checked — only 2022-era
|
|
||||||
tags. As of this date, `RELEASE.2025-09-07T16-13-09Z` IS the newest available Docker image.
|
|
||||||
- v5.2.0 adds two optional new env vars: DOCUMENT_ALL_ENDPOINT_ENABLED and OIDC_OP_USER_ENDPOINT_FORMAT.
|
- v5.2.0 adds two optional new env vars: DOCUMENT_ALL_ENDPOINT_ENABLED and OIDC_OP_USER_ENDPOINT_FORMAT.
|
||||||
Both are backward-compatible (no action required for existing deployments).
|
Both are backward-compatible (no action required for existing deployments).
|
||||||
- Recipe version label convention: 0.X.Y+vA.B.C where A.B.C is the impress version.
|
- Recipe version label convention: 0.X.Y+vA.B.C where A.B.C is the impress version.
|
||||||
|
|||||||
@@ -6,28 +6,6 @@
|
|||||||
| postgres | postgres | https://github.com/postgres/postgres | https://www.postgresql.org/docs/release/ |
|
| postgres | postgres | https://github.com/postgres/postgres | https://www.postgresql.org/docs/release/ |
|
||||||
|
|
||||||
## Standing notes
|
## Standing notes
|
||||||
- **2026-09-11 re-check** (Mattermost Server Releases docs; endoflife.date/api/mattermost.json;
|
|
||||||
GitHub releases + module tags): **11.7 ESR line UNCHANGED** — newest patch still **11.7.10**
|
|
||||||
(2026-08-26; no 11.7.11; v11.11.0 is still a GitHub **prerelease**, innovation). **10.11 ESR
|
|
||||||
expired 2026-08-15; 10.11.23 (2026-08-13) was its FINAL patch** — the "10.x is LTS, never
|
|
||||||
11.x" survey hint appeared a THIRD time (2026-06-26, 2026-08-07, 2026-09-11) and was again
|
|
||||||
NOT followed: no supported 10.x LTS exists (10.12 = innovation, expired 2025-12-15). PR #2
|
|
||||||
re-verified unchanged (tree 59e8c2c0; direct `--chaos` deploy converged + ping 200;
|
|
||||||
`!testme` GREEN, drone 1355). **Adjudicated CVE-2026-13426 (MMSA-2025-00532, medium) FIXED by
|
|
||||||
the 10.11.22 → 11.7.10 window**: module fix `server/public` v0.1.22 (commit 3321db82, tag cut
|
|
||||||
2026-01-30 from the 11.4.0 cycle) is an ancestor of v11.7.10 but NOT of v10.11.22/v10.11.23
|
|
||||||
(GitHub compare evidence) → CVE count for the ESR move = **14** (13 deterministic + 1
|
|
||||||
adjudicated). postgres **15-alpine HELD** (recipe has no compose overlays beyond compose.yml;
|
|
||||||
no pg_upgrade/pgautoupgrade support in recipe or cc-ci tests; the floating 15-alpine tag picks
|
|
||||||
up 15.x patches on re-pull — pg 15.19 fixes CVE-2026-14662/14663/14664).
|
|
||||||
- **2026-08-28 re-check** (endoflife.date/api/mattermost.json 2026-08-28; Docker Hub; GitHub
|
|
||||||
releases): **11.7.10** (released 2026-08-26, "various bug fixes", not prerelease) is the newest
|
|
||||||
11.7.x ESR patch — the ESR/LTS line has NOT moved, still **11.7** (EOL **2027-05-15**). This run
|
|
||||||
bumps open PR #2 **11.7.9 → 11.7.10** (ESR patch within the same line). 11.8/11.9/11.10 remain
|
|
||||||
innovation releases (EOL 2026-09-15 / 10-15 / 11-15), NOT ESR — do NOT target; 10.12 expired.
|
|
||||||
postgres **15-alpine HELD** (DB-major 16/17/18 out of scope, plain-postgres recipe w/o
|
|
||||||
pgautoupgrade: operator dump/restore needed). git.autonomic.zone mirror main reconciled to
|
|
||||||
coopcloud (86e0d0b5 = 10.11.22, upstream still pins the EXPIRED 10.11 ESR).
|
|
||||||
- mattermost-lts tracks the **11.7 ESR (Extended Support Release)** = Mattermost's LTS line. The "LTS"
|
- mattermost-lts tracks the **11.7 ESR (Extended Support Release)** = Mattermost's LTS line. The "LTS"
|
||||||
concept in Mattermost is **ESR**; there is NO "10.x LTS line". Release calendar (endoflife.date/mattermost):
|
concept in Mattermost is **ESR**; there is NO "10.x LTS line". Release calendar (endoflife.date/mattermost):
|
||||||
- **10.11** = previous ESR, security support **ends 2026-08-15** (~7 weeks) — migrated away 2026-06-19
|
- **10.11** = previous ESR, security support **ends 2026-08-15** (~7 weeks) — migrated away 2026-06-19
|
||||||
@@ -97,32 +75,3 @@
|
|||||||
recreate DB, reimport dump. `DROP DATABASE WITH (FORCE)` requires PostgreSQL 13+ — safe on postgres:15-alpine.
|
recreate DB, reimport dump. `DROP DATABASE WITH (FORCE)` requires PostgreSQL 13+ — safe on postgres:15-alpine.
|
||||||
The previous inline-label approach (no restore hook) was a defect: raw PGDATA restore without a reload
|
The previous inline-label approach (no restore hook) was a defect: raw PGDATA restore without a reload
|
||||||
was a silent no-op. Fixed in PR #2 (restore fix cherry-picked from PR #1 ci/pg-restore).
|
was a silent no-op. Fixed in PR #2 (restore fix cherry-picked from PR #1 ci/pg-restore).
|
||||||
|
|
||||||
- **2026-08-21 re-check** (endoflife.date/api/mattermost.json 2026-08-17; Mattermost docs; GitHub):
|
|
||||||
**11.7 ESR is the only current ESR/LTS** — latest patch **11.7.9** (2026-08-13, "Low to High
|
|
||||||
security fixes"), EOL 2027-05-15. **10.11 ESR ENDED 2026-08-15** — upstream main still pins
|
|
||||||
10.11.22 (now on an EXPIRED ESR). 11.10/11.9/11.8 are innovation (EOL 2026-11-15/10-15/09-15),
|
|
||||||
NOT LTS. The `-lts` recipe must track 11.7. This run **redirects open PR #2 away from the
|
|
||||||
innovation 11.10.0 onto 11.7.9** (the correct LTS/ESR; ESR→ESR 10.11→11.7 is "fully supported
|
|
||||||
and tested" per Mattermost docs — no staging hop). Refer to the 2026-08-15 report where PR #2
|
|
||||||
was on 11.10.0 (operator chose innovation line that week); operator should now decide the
|
|
||||||
`-lts` line. postgres 15-alpine still HELD (DB-major out of scope).
|
|
||||||
|
|
||||||
- **2026-08-31 re-check** (endoflife.date/api/mattermost.json 2026-08-31; Docker Hub tag list;
|
|
||||||
GitHub release `v11.7.10`): **11.7 ESR line UNCHANGED** — latest patch still **11.7.10**
|
|
||||||
(released 2026-08-26, "various bug fixes"), EOL 2027-05-15. PR #2 already carries 11.7.9 →
|
|
||||||
11.7.10; the wildcard cert is RENEWED (valid to 2026-11-29), so this run just RE-VERIFIES PR #2
|
|
||||||
via `!testme` (the 2026-08-28 run's `!testme` was infra-blocked). 11.8.5 / 11.9.1 / 11.10.1
|
|
||||||
remain innovation (EOL 2026-09-15 / 10-15 / 11-15), NOT ESR — do NOT target; 10.11 ESR ended
|
|
||||||
2026-08-15 (upstream main still pins 10.11.22 = EXPIRED ESR → the 10→11 ESR move PR #2 carries
|
|
||||||
remains required; Mattermost docs: ESR→ESR is "fully supported and tested"). postgres 15-alpine
|
|
||||||
still HELD (DB-major out of scope, operator dump/pg_upgrade).
|
|
||||||
|
|
||||||
## NVD CPE fallback
|
|
||||||
This project publishes nothing machine-readable we can reach — no GitHub advisory feed,
|
|
||||||
no release-attributable changelog — so its CVE count was `?` (nothing measured). NVD is
|
|
||||||
CPE-indexed and carries structured version ranges, so it can answer where the vendor
|
|
||||||
cannot. It LAGS the vendor, so it is a fallback, never the primary source.
|
|
||||||
|
|
||||||
- nvd-cpe: mattermost-team-edition = cpe:2.3:a:mattermost:mattermost_server:*:*:*:*:*:*:*:*
|
|
||||||
- nvd-cpe: postgres = cpe:2.3:a:postgresql:postgresql:*:*:*:*:*:*:*:*
|
|
||||||
|
|||||||
@@ -1,29 +0,0 @@
|
|||||||
# Upstream sources — mumble
|
|
||||||
|
|
||||||
| service | image | source repo | releases / changelog |
|
|
||||||
|---------|-------|-------------|----------------------|
|
|
||||||
| app | mumblevoip/mumble-server | https://github.com/mumble-voip/mumble | https://github.com/mumble-voip/mumble/releases |
|
|
||||||
| web | rankenstein/mumble-web | https://github.com/rankenstein/mumble-web | https://github.com/rankenstein/mumble-web/releases |
|
|
||||||
|
|
||||||
## Standing notes
|
|
||||||
- This file was **missing entirely** until 2026-08-11. Without it the advisory scan had no source to
|
|
||||||
query, and still printed "0 identified by the deterministic scan" — which was then published as a
|
|
||||||
clean `0` in the 2026-08-11 CVE check. The scan now refuses to emit a count when it has no usable
|
|
||||||
source (it reports UNKNOWN), and `audit-sources.py` flags a missing registry file directly.
|
|
||||||
- `mumblevoip/mumble-server` tracks the upstream server releases and DOES publish GitHub security
|
|
||||||
advisories, so it is the recipe's primary CVE source.
|
|
||||||
- `rankenstein/mumble-web` is a **fork** of the original `Johni0702/mumble-web`, which has been
|
|
||||||
dormant since 2023-05. The fork itself last pushed 2023-07 and its Docker tag `0.5` was last built
|
|
||||||
well over five years ago. Neither is archived, but treat the web client as effectively unmaintained:
|
|
||||||
if a CVE lands there, expect no upstream fix and plan a replacement rather than an upgrade.
|
|
||||||
- The server image tag is `v<version>-<build>` (e.g. `v1.6.870-4`); the trailing number is the image
|
|
||||||
build, not an app version, and moves independently of upstream releases — `abra recipe upgrade`
|
|
||||||
reports "no new versions" for it, so use `resolve-images.py` to see those bumps.
|
|
||||||
|
|
||||||
## NVD CPE fallback
|
|
||||||
This project publishes nothing machine-readable we can reach — no GitHub advisory feed,
|
|
||||||
no release-attributable changelog — so its CVE count was `?` (nothing measured). NVD is
|
|
||||||
CPE-indexed and carries structured version ranges, so it can answer where the vendor
|
|
||||||
cannot. It LAGS the vendor, so it is a fallback, never the primary source.
|
|
||||||
|
|
||||||
- nvd-cpe: mumble-server = cpe:2.3:a:mumble:mumble:*:*:*:*:*:*:*:*
|
|
||||||
@@ -50,158 +50,3 @@
|
|||||||
- 2026-08-07 run: operator directed 2.33.3 -> 2.34.2 (the newest). The whole 2.34.x line is still
|
- 2026-08-07 run: operator directed 2.33.3 -> 2.34.2 (the newest). The whole 2.34.x line is still
|
||||||
marked Pre-release on GitHub (2.33.5 holds the Latest badge); flagged in the PR body. No breaking
|
marked Pre-release on GitHub (2.33.5 holds the Latest badge); flagged in the PR body. No breaking
|
||||||
changes across 2.33.3 -> 2.34.2; rolling upgrade safe (TypeORM migrations auto-run on boot).
|
changes across 2.33.3 -> 2.34.2; rolling upgrade safe (TypeORM migrations auto-run on boot).
|
||||||
- 2.35.0 (2026-08-11, Pre-release): major feature release — self-hosted AI Assistant onboarding,
|
|
||||||
Simplified Custom Auth credentials, Agent Builder test runs + HITL, Discord agent chat channel,
|
|
||||||
local agent token counting, **VM expression engine now the default** (was opt-in), MCP SDK v2
|
|
||||||
migration + MCP 2026-07-28 discovery handshake, Kafka Node v2, Salesforce OAuth2 JWT, GitHub
|
|
||||||
dispatch timeout, X/Twitter Node OAuth2/API migrated to x.com, Azure Key Vault configurable
|
|
||||||
endpoints, Postgres-version startup warning, workflow review improvements (diffs, metadata, version
|
|
||||||
descriptions), and numerous core/editor bugfixes. No breaking compose/config/migration changes.
|
|
||||||
- 2.35.1 (2026-08-12, Pre-release): 2 core bugfixes — data-tables resume scope, TLS options per hop
|
|
||||||
through a proxy.
|
|
||||||
- 2.35.2 (2026-08-13, Pre-release): 1 core bugfix — report real activation mode for triggers via
|
|
||||||
publication outbox. **Deployed on cc-ci 2026-08-15**: 2.34.4→2.35.2, TypeORM migrations clean,
|
|
||||||
editor served HTTP 200. No breaking changes, no N8N_* env renames, no required operator action.
|
|
||||||
- 2.35.3 (2026-08-14, Pre-release): bugfixes (Google Ads v21→v25 API migration, MS Teams OAuth scope
|
|
||||||
restore, workflow publication outbox abort deadline) + feature (skip update approval for workflows
|
|
||||||
from same Instance AI session). Not deployed (2.35.2 was the survey target).
|
|
||||||
- 2.36.0 (2026-08-18, Pre-release; 2.36.1 was briefly tagged then withdrawn — 2.36.2 compares
|
|
||||||
2.36.0...2.36.2): a large feature minor. Core/ai-builder/editor bugfixes + new features (Agent
|
|
||||||
sandboxes + writable workspaces, MCP folder/move/read-data-table tools, Schedule Trigger missed-
|
|
||||||
execution grace period + catch-up coalescing, Confluence Cloud OAuth2 + page get sub-tree, MiniMax
|
|
||||||
M3/H3, LangSmith session debug export, role-mapping/roles API endpoints, n8n User Auth GA, etc.).
|
|
||||||
No breaking compose/env/migration changes, no `N8N_*` env renames; the deprecation notices
|
|
||||||
(Array.merge→Array.mergeIntoObject in expressions, workflow tags API migration) are API-level only.
|
|
||||||
2.36.2 (2026-08-19, patch): 3 fixes (decorator-route request schema in /discover, end-user
|
|
||||||
credential resolution in node parameters, editor agent-preview alignment). 2.36.3 (2026-08-20,
|
|
||||||
patch): 4 core/editor fixes (task-runners only restart when slow, expression engine init on
|
|
||||||
expression commands, test-webhook isolate release after teardown, Edit Fields type selector styles).
|
|
||||||
- 2026-08-21 run: planning 2.34.4 → 2.36.3 (newest tag abra lists). Whole 2.36.x line still
|
|
||||||
Pre-release on GitHub (2.33.5/Latest was superseded by 2.35.x line). Rolling upgrade safe, sqlite
|
|
||||||
mode, TypeORM migrations auto-run on boot. No operator action required.
|
|
||||||
- 2.36.4 (2026-08-21, Pre-release): 3 core fixes (env normalization before schema parsing; chat and
|
|
||||||
MCP trigger auth-mode validation for end-user credentials; wrap trigger closeFunction in an
|
|
||||||
expression isolate).
|
|
||||||
- 2.36.5 (2026-08-21, Pre-release): 1 core fix (raise AI Assistant model verification token limit).
|
|
||||||
- 2.36.6 (2026-08-24): 1 editor fix (personalization survey / community registration modal on
|
|
||||||
Instance AI landing page).
|
|
||||||
- 2.36.7 (2026-08-25, **current Stable / Latest badge**; `stable` tag points here): API fix (adapt
|
|
||||||
binary data endpoint), API cross-backport (require a JSON content type on decorator routes that
|
|
||||||
take a body), core fixes (reconcile license auto-renewal on startup leader takeover; route OAuth2
|
|
||||||
token requests through environment proxies), feature (remove offset query param from list workflow
|
|
||||||
history endpoint). This is the newest stable-line release; the 2.37.x line is still pre-release.
|
|
||||||
- 2.37.0 (2026-08-25, Pre-release): a large feature minor. Agent Builder / AIA / core/editor
|
|
||||||
bugfixes + features (Slack Agent view, prompt caching for Anthropic, SharePoint node v2 = default,
|
|
||||||
form-trigger OAuth2 (added then reverted), role-mapping rule CRUD endpoints, execution
|
|
||||||
startedAfter/startedBefore filters, Agent/community-package hardening: serialize community
|
|
||||||
package installs + non-destructive downloads + cold-boot storm reduction, MCP tool schemas as JSON
|
|
||||||
Schema 2020-12, distroless n8n image variant). **Deprecation: the "Any workflow" caller policy**
|
|
||||||
(API/config-level — flag for operators using execution-caller policies). Two enforced API behavior
|
|
||||||
changes: **require JSON content type on decorator body routes** and **"adapt binary data endpoint"**
|
|
||||||
(HTTP clients sending a body to those routes must send `Content-Type: application/json`). No
|
|
||||||
breaking compose/env/migration changes, no `N8N_*` env renames.
|
|
||||||
- 2.37.1 (2026-08-25, Pre-release): 1 core fix (route OAuth2 token requests through env proxies in
|
|
||||||
every process).
|
|
||||||
- 2.37.3 (2026-08-27, Pre-release; 2.37.2 was withdrawn): 2 runner-image fixes (copy glibc into
|
|
||||||
distroless runners image as a matched pair; stage libatomic into the distroless runners image) —
|
|
||||||
relevant if using n8n's community/distroless runner image.
|
|
||||||
- 2026-08-28 run: PR #7 extended 2.34.4 → **2.37.3** (newest tag abra lists; 2.37.2 withdrawn, that
|
|
||||||
tag does not exist). 2.36.7 holds the Stable/Latest badge; 2.37.x remains Pre-release on GitHub,
|
|
||||||
consistent with the tracking-the-newest-tag precedent (2.34.2/2.35.2/2.36.3 were pre-release when
|
|
||||||
taken). Rolling upgrade safe (sqlite, TypeORM auto-migrate on boot). Operator flag: the two 2.37.0
|
|
||||||
API behavior changes (JSON content-type on decorator body routes; binary-data endpoint adapt) only
|
|
||||||
affect HTTP API callers. Recommended release: `-y` (feature minor).
|
|
||||||
NOTE: PR #7's `!testme` on this date was BLOCKED — `*.ci.commoninternet.net` wildcard cert expired
|
|
||||||
(bridge: `CERTIFICATE_VERIFY_FAILED certificate has expired`). UNVERIFIED; operator renewed the cert
|
|
||||||
(now valid to 2026-11-29). The 2.37.3 work must be re-verified by a fresh `!testme`.
|
|
||||||
- 2.34.5 (2026-08-12, patch): 1 core fix (apply TLS options per hop through a proxy).
|
|
||||||
- 2.34.6 (2026-08-14, patch): 3 fixes (core `continueErrorOutput` `details` field; Google Ads sunset
|
|
||||||
v21→v25 API migration; MS Teams Restore `Group.ReadWrite.All` OAuth2 scope — matches the 2.35.3
|
|
||||||
in-progress backports).
|
|
||||||
- 2.35.4/2.35.5/2.35.6/2.35.7 (2026-08-19..21, patches): API schema for decorator routes in /discover;
|
|
||||||
Google Ads v25 view metric rename; task-runners not restarted when only slow; expression engine
|
|
||||||
init on expression commands; test-webhook isolate release after teardown; env normalization before
|
|
||||||
schema parsing; end-user credential resolution; trigger closeFunction isolate; AI Assistant token
|
|
||||||
limit raise. All patch bugfixes, no breaking changes.
|
|
||||||
- 2.36.8 (2026-08-28, patch): 1 core fix (domain-restricted credential usable in its own node) —
|
|
||||||
cross-backport of the 2.37.4 fix into the stable line.
|
|
||||||
- 2.36.9 (2026-08-31, patch; **now the Stable/Latest badge** — `stable` tag): 1 core fix (apply proxy
|
|
||||||
environment variables consistently across packages and processes, cross-backport of 2.37.1/2.37.3).
|
|
||||||
- 2.37.4 (2026-08-28, Pre-release): 1 core fix (allow a domain-restricted credential to work in its
|
|
||||||
own node).
|
|
||||||
- 2.37.5 (2026-08-31): **withdrawn** — no plain `2.37.5` tag on Docker Hub (only partial
|
|
||||||
`2.37.5-amd64/-arm64/-pc` artifact tags, no manifest). Skip entirely, like 2.37.2/2.36.1.
|
|
||||||
- 2.37.6 (2026-08-31, Pre-release; **newest 2.37.x tag**): 1 core fix (thread execution id through
|
|
||||||
dynamic-credential storage).
|
|
||||||
- 2026-08-31 run: PR #7 extended 2.34.4 → **2.37.6** (newest tag abra lists = 2.37.6/2.37.4/2.37.3/…;
|
|
||||||
2.37.5 withdrawn). 2.36.9 holds the Stable/Latest badge; 2.37.x remains Pre-release on GitHub
|
|
||||||
(consistent precedent). Re-verified 2.37.3→2.37.6 (pure core bugfixes), no breaking changes beyond
|
|
||||||
the already-flagged 2.37.0 API behavior pair. Rolling upgrade safe. Recommended release: `-y`.
|
|
||||||
- 2.37.7 (2026-09-01, patch): core/editor bugfixes (distroless runners glibc/libatomic copy; thread
|
|
||||||
execution id through dynamic-credential storage; scope active workflow IDs to user's projects;
|
|
||||||
route secrets pasted in AI chat to real credential forms; gate autosave on document hydration).
|
|
||||||
- 2.37.8/2.37.9/2.37.10/2.37.11 (2026-09-02..04, patches): mostly core/editor/ai-builder bugfixes.
|
|
||||||
2.37.9 and 2.37.10 are noted on the Releases page; 2.37.11 = 1 core fix (thread execution id through
|
|
||||||
dynamic-credential storage backport). No breaking compose/env/migration changes.
|
|
||||||
- 2.38.0 (2026-09-01, Pre-release): the 2.38 feature minor — a large feature/bugfix release (Confluence
|
|
||||||
page/space operations, Databricks user-delegated OAuth2, Moonshot/MiniMax/Qwen Cloud model providers
|
|
||||||
in Agent Builder, background sub-agent delegation, nonce-based Content-Security-Policy on HTML pages,
|
|
||||||
chat trigger publish validation + workflow:execute access control, restore mutating array methods on
|
|
||||||
$json in expressions, MongoDB bulkWrite, etc.). No breaking compose/env/migration changes; no
|
|
||||||
`N8N_*` env renames; the 2.37.0 API behavior pair (JSON content-type on decorator body routes;
|
|
||||||
binary-data endpoint adapt) still applies to HTTP API callers only.
|
|
||||||
- 2.38.1 (2026-09-01, Pre-release): a large patch — bugfixes (mongodb sort/projection parameter
|
|
||||||
binding; S3 startup connection timeout; domain-restricted credential in own node; OAuth token grant
|
|
||||||
binding; restore mutating array methods on $json in expressions; etc.) + features (n8n credits
|
|
||||||
section in agent tools modal, Confluence page get comments/labels/space, Moonshot/MiniMax/Qwen Cloud
|
|
||||||
model providers, nonce-based CSP, split hosted chat page into trusted shell + sandboxed frame).
|
|
||||||
No breaking compose/config changes.
|
|
||||||
- 2.38.2 (2026-09-02, Pre-release): core bugfix (restore ...) — see Releases page.
|
|
||||||
- 2.38.3 (2026-09-0?, Pre-release): patch bugfixes (bundle backports to 2.38.x release track —
|
|
||||||
egress filtering, credential-destination grants, proxy-agent behavior, shared-workflow/user queries;
|
|
||||||
nonce-based CSP on HTML pages). No breaking compose/config changes.
|
|
||||||
- 2.38.4 (2026-09-0?, Pre-release; **newest 2.38.x tag** — abra lists it as the top upgrade candidate):
|
|
||||||
patch bugfixes. No breaking compose/env/migration changes; rolling upgrade safe (sqlite, TypeORM
|
|
||||||
auto-migrate on boot). The 2.37.0 API behavior pair (JSON content-type on decorator body routes;
|
|
||||||
binary-data endpoint adapt) still applies — HTTP API callers only, no recipe impact.
|
|
||||||
- 2026-09-07 run: PR #7 extended 2.34.4 → **2.38.4** (newest tag abra lists; 2.37.5/2.38.0-era withdrawn
|
|
||||||
tags skipped). 2.38.x line remains Pre-release on GitHub (consistent tracking-the-newest-tag
|
|
||||||
precedent: 2.34.2/2.35.2/2.36.3/2.37.3/2.37.6 were all Pre-release when taken). Rolling upgrade safe
|
|
||||||
(sqlite, TypeORM auto-migrate on boot). No breaking compose/config changes for the recipe. Operator
|
|
||||||
flag (unchanged from 2.37.0): the two API behavior changes only affect HTTP API callers. Recommended
|
|
||||||
release: `-y` (feature minor across 2.34.4 → 2.38.4). CVE-2026-73088 was adjudicated NOT-FIXED in the
|
|
||||||
2.x window by the 2026-09-04 parent run; not blocking.
|
|
||||||
- 2.38.5 (2026-09-09, patch): 1 core fix (keep reporting the original job error when the job key is
|
|
||||||
gone).
|
|
||||||
- 2.38.6 (2026-09-10, patch; **Stable/Latest badge**): 1 perf fix (bound peak memory during source
|
|
||||||
control push).
|
|
||||||
- 2.39.0 (2026-09-08, Pre-release): the 2.39 feature minor — large feature/bugfix release. Features:
|
|
||||||
source-control push/pull + status endpoints in the public API, **workflow version endpoint
|
|
||||||
(deprecates the old two-variable path — API-level deprecation)**, Instance reporting module, opt-in
|
|
||||||
concurrency limits for Instance AI runs, OTLP gRPC support for OTel, Confluence/Jira Atlassian
|
|
||||||
Service Account (2LO) auth, MS Teams channel-message + Online Meeting ops, Databricks partner UA +
|
|
||||||
hidden chat model, Instance AI folder exploration / past-conversation search / credential-selection
|
|
||||||
guidance, **encryption-key module rework** (always load the encryption module and seed keys safely
|
|
||||||
on startup; cache keys in memory; pluggable write path with rotation moved into the module; signing
|
|
||||||
deployment keys stored in the same wrapped form as data-encryption keys), deprecate + hide the Ask
|
|
||||||
AI tab in the Code node. Many core/editor/node fixes (webhook Set-Cookie response headers; OTel
|
|
||||||
header redaction in settings; Azure OpenAI classic/Foundry credential fix; OIDC failed-login
|
|
||||||
redirect; Qdrant client update for Node 26; AMQP trigger receiver reattach; Anthropic disabled
|
|
||||||
thinking; etc.). No breaking compose/env/migration changes; no `N8N_*` env renames.
|
|
||||||
- 2.39.1 (2026-09-09, Pre-release): 1 core fix — **repair legacy-format data-encryption keys during
|
|
||||||
bootstrap** (upgrade-relevant: old-format keys are repaired, not rejected).
|
|
||||||
- 2.39.2 (2026-09-10, Pre-release): 2 core fixes (memoize pairedItem ancestry traversal; reuse an
|
|
||||||
existing consent grant after the visitor authenticates mid-flow) + 1 perf (stop copying user
|
|
||||||
workflows into the Instance AI sandbox). Docker Hub `2.39.2` manifest verified multi-arch
|
|
||||||
(amd64 + arm64, active, 2026-09-10).
|
|
||||||
- 2026-09-11 run: survey framed 2.34.4 → 2.39.2, but the morning reconcile closed PR #7 as
|
|
||||||
merged-upstream and upstream main released **3.5.0+2.38.4** — so the effective compose window is
|
|
||||||
**2.38.4 → 2.39.2** on a FRESH branch (no open upgrade PR to extend). 2.38.6 holds the
|
|
||||||
Stable/Latest badge; 2.39.x is Pre-release (consistent tracking-the-newest-tag precedent:
|
|
||||||
2.34.2/2.35.2/2.36.3/2.37.3/2.37.6/2.38.4 were all Pre-release when taken). Rolling upgrade safe
|
|
||||||
(sqlite mode, TypeORM auto-migrate on boot). Live-deploy watch item: the 2.39.0 encryption-key
|
|
||||||
module rework + 2.39.1 legacy-key repair (recipe injects N8N_ENCRYPTION_KEY as a secret).
|
|
||||||
Operator flags, all HTTP-API-caller-level only: 2.33.0 workflow activate/deactivate deprecation;
|
|
||||||
2.36.0 Array.merge→mergeIntoObject + workflow-tags API migration; 2.37.0 "Any workflow" caller-policy
|
|
||||||
deprecation + JSON content-type on decorator body routes + binary-data endpoint adapt; 2.39.0
|
|
||||||
workflow-version endpoint deprecation. Recommended release: `-y` (feature minor).
|
|
||||||
|
|||||||
@@ -4,7 +4,6 @@
|
|||||||
|---------|-------|-------------|----------------------|
|
|---------|-------|-------------|----------------------|
|
||||||
| app | wordpress | https://github.com/WordPress/WordPress | https://wordpress.org/news/category/releases/ (image: https://hub.docker.com/_/wordpress) |
|
| app | wordpress | https://github.com/WordPress/WordPress | https://wordpress.org/news/category/releases/ (image: https://hub.docker.com/_/wordpress) |
|
||||||
| db | mariadb | https://github.com/MariaDB/server | https://mariadb.com/kb/en/release-notes/ (image: https://hub.docker.com/_/mariadb) |
|
| db | mariadb | https://github.com/MariaDB/server | https://mariadb.com/kb/en/release-notes/ (image: https://hub.docker.com/_/mariadb) |
|
||||||
| ftp (optional overlay) | atmoz/sftp | https://github.com/atmoz/sftp | https://github.com/atmoz/sftp/releases (image: https://hub.docker.com/r/atmoz/sftp) |
|
|
||||||
|
|
||||||
## Standing notes
|
## Standing notes
|
||||||
- Enrolled 2026-08-03 (operator request). Recipe = coopcloud `wordpress`; mirror
|
- Enrolled 2026-08-03 (operator request). Recipe = coopcloud `wordpress`; mirror
|
||||||
@@ -23,7 +22,3 @@
|
|||||||
image). mariadb major bumps (e.g. 12.x→13.x) need the usual dump/restore caution if the
|
image). mariadb major bumps (e.g. 12.x→13.x) need the usual dump/restore caution if the
|
||||||
recipe ever pins a non-`mariadb`-auto-upgrading setup; the official image handles minor
|
recipe ever pins a non-`mariadb`-auto-upgrading setup; the official image handles minor
|
||||||
bumps in place.
|
bumps in place.
|
||||||
- `atmoz/sftp` is pinned as the floating `alpine` tag (updated 2026-07-28, current as of
|
|
||||||
2026-08-21) — not a semver abra can evaluate, so it never contributes an upgrade candidate;
|
|
||||||
verified current by direct Docker Hub tag check. Only bump if a real newer supported tag
|
|
||||||
exists and the app needs it.
|
|
||||||
|
|||||||
@@ -31,7 +31,7 @@ matrix-synapse weekly
|
|||||||
mattermost-lts weekly
|
mattermost-lts weekly
|
||||||
mumble weekly
|
mumble weekly
|
||||||
n8n weekly
|
n8n weekly
|
||||||
plausible external # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded
|
plausible weekly
|
||||||
uptime-kuma external # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded
|
uptime-kuma external # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded
|
||||||
wordpress weekly
|
wordpress weekly
|
||||||
```
|
```
|
||||||
|
|||||||
Generated
+6
-30
@@ -1,35 +1,12 @@
|
|||||||
{
|
{
|
||||||
"nodes": {
|
"nodes": {
|
||||||
"cc-ci": {
|
|
||||||
"inputs": {
|
|
||||||
"nixpkgs": [
|
|
||||||
"nixpkgs"
|
|
||||||
],
|
|
||||||
"sops-nix": [
|
|
||||||
"sops-nix"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"locked": {
|
|
||||||
"lastModified": 1788820034,
|
|
||||||
"narHash": "sha256-mkktTMeGKutfPp3pn3AhoPoGSCm4iRSnPvpmbEpMmwA=",
|
|
||||||
"ref": "refs/heads/main",
|
|
||||||
"rev": "b11cc0b7385aee9fb89bd9a7ed23896401d683e5",
|
|
||||||
"revCount": 1543,
|
|
||||||
"type": "git",
|
|
||||||
"url": "https://git.autonomic.zone/recipe-maintainers/cc-ci.git"
|
|
||||||
},
|
|
||||||
"original": {
|
|
||||||
"type": "git",
|
|
||||||
"url": "https://git.autonomic.zone/recipe-maintainers/cc-ci.git"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nixpkgs": {
|
"nixpkgs": {
|
||||||
"locked": {
|
"locked": {
|
||||||
"lastModified": 1788690626,
|
"lastModified": 1785734586,
|
||||||
"narHash": "sha256-+v4I4LawmRD/mVxO7QIAerRrCkElp3YImzWkkUnvOTg=",
|
"narHash": "sha256-ODZkEK9Gy50yg6h98u7KkitZ3oc/uuTFK00bh1CRdNA=",
|
||||||
"owner": "NixOS",
|
"owner": "NixOS",
|
||||||
"repo": "nixpkgs",
|
"repo": "nixpkgs",
|
||||||
"rev": "c25784012c9982bca5b3e0de87e90bbdac8927d3",
|
"rev": "531670d871c0e29724a02f3cbcac170adc65b58c",
|
||||||
"type": "github"
|
"type": "github"
|
||||||
},
|
},
|
||||||
"original": {
|
"original": {
|
||||||
@@ -41,7 +18,6 @@
|
|||||||
},
|
},
|
||||||
"root": {
|
"root": {
|
||||||
"inputs": {
|
"inputs": {
|
||||||
"cc-ci": "cc-ci",
|
|
||||||
"nixpkgs": "nixpkgs",
|
"nixpkgs": "nixpkgs",
|
||||||
"sops-nix": "sops-nix"
|
"sops-nix": "sops-nix"
|
||||||
}
|
}
|
||||||
@@ -53,11 +29,11 @@
|
|||||||
]
|
]
|
||||||
},
|
},
|
||||||
"locked": {
|
"locked": {
|
||||||
"lastModified": 1788337237,
|
"lastModified": 1783174389,
|
||||||
"narHash": "sha256-gkSH8VUtCo6hnysNmb9DbTuDepH2t5pv+QWjP75xKAk=",
|
"narHash": "sha256-aCWC8ngycU7OdJrU2+Je3qf+1a2ykuBvpPhZT/9tXMc=",
|
||||||
"owner": "Mic92",
|
"owner": "Mic92",
|
||||||
"repo": "sops-nix",
|
"repo": "sops-nix",
|
||||||
"rev": "fbf759290e0cb0a98dfc813a4eb7d53ad1dacb57",
|
"rev": "f1406619a3884cd5c47992a70b8b35c9c0fcb4c9",
|
||||||
"type": "github"
|
"type": "github"
|
||||||
},
|
},
|
||||||
"original": {
|
"original": {
|
||||||
|
|||||||
@@ -1,55 +1,32 @@
|
|||||||
{
|
{
|
||||||
description = "cc-ci-orchestrator — the cc-ci orchestrator (loops, steering session, weekly upgrader) and the NixOS host it shares with the cc-ci CI server";
|
description = "cc-ci-orchestrator — NixOS host for the cc-ci loops runtime (Builder/Adversary/Watchdog)";
|
||||||
|
|
||||||
inputs = {
|
inputs = {
|
||||||
# Stable release channel (operator 2026-08-01). `nix flake update` moves it; the cc-ci input
|
# Follow the current stable release channel (operator 2026-08-01), was a hard rev pin at
|
||||||
# below FOLLOWS it, so one nixpkgs builds the whole combined host and CVEs get patched once.
|
# nixpkgs 24.11 (50ab7937, 2025-06-30) kept "the same as the cc-ci server". This host runs
|
||||||
|
# agents/tmux/nginx/docker, not recipe CI, so it does not need to match that server — and a
|
||||||
|
# frozen rev only accrues unpatched CVEs. `nix flake update` now actually moves.
|
||||||
nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";
|
nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";
|
||||||
|
|
||||||
|
# sops-nix follows nixpkgs below, so it no longer needs its own matching pin.
|
||||||
sops-nix.url = "github:Mic92/sops-nix";
|
sops-nix.url = "github:Mic92/sops-nix";
|
||||||
sops-nix.inputs.nixpkgs.follows = "nixpkgs";
|
sops-nix.inputs.nixpkgs.follows = "nixpkgs";
|
||||||
|
|
||||||
# The cc-ci CI server, as a NixOS module (`nixosModules.cc-ci-server`). HTTPS, anonymous read:
|
|
||||||
# nix evaluates every input for every output, so the input must be fetchable without
|
|
||||||
# credentials. The private secrets submodule is deliberately NOT fetched through this input —
|
|
||||||
# the host reads the deployed --recursive checkout's secrets.yaml at activation instead
|
|
||||||
# (`cc-ci.sopsFile`). Both `follows` are REQUIRED: without them cc-ci's own nixpkgs/sops-nix
|
|
||||||
# pins would produce a second sops-nix module tree and a second nixpkgs in one system.
|
|
||||||
cc-ci.url = "git+https://git.autonomic.zone/recipe-maintainers/cc-ci.git";
|
|
||||||
cc-ci.inputs.nixpkgs.follows = "nixpkgs";
|
|
||||||
cc-ci.inputs.sops-nix.follows = "sops-nix";
|
|
||||||
};
|
};
|
||||||
|
|
||||||
outputs = { self, nixpkgs, sops-nix, cc-ci, ... }:
|
outputs = { nixpkgs, sops-nix, ... }:
|
||||||
let
|
let
|
||||||
system = "x86_64-linux";
|
system = "x86_64-linux";
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
nixosModules = {
|
# Hetzner cpx11 host (nixos-infect generated hardware.nix + orchestrator config).
|
||||||
# The orchestrator itself: loops supervisor, steering session, weekly/hourly timers.
|
# Provision with terraform/ then run Stage 2 per terraform/README.md.
|
||||||
cc-ci-orchestrator = ./nix/modules/cc-ci.nix;
|
nixosConfigurations.cc-ci-orchestrator-hetzner = nixpkgs.lib.nixosSystem {
|
||||||
# The host contract those units assume: loops user, claude/opencode CLIs, opencode web
|
|
||||||
# server + tailnet UI, nix-ld, tool set, `ssh cc-ci` config.
|
|
||||||
orchestrator-host = ./nix/modules/orchestrator-host.nix;
|
|
||||||
# Weekly health-gated self-update of the host (busy-gated around CI runs).
|
|
||||||
auto-update = ./nix/modules/auto-update.nix;
|
|
||||||
# Old name of cc-ci-orchestrator, kept while notplants-nix still imports it (2026-09).
|
|
||||||
cc-ci = ./nix/modules/cc-ci.nix;
|
|
||||||
};
|
|
||||||
|
|
||||||
nixosConfigurations = {
|
|
||||||
# THE live host: cc-ci CI server + cc-ci orchestrator on one Hetzner cpx32-class box
|
|
||||||
# (195.201.88.249, since 2026-09). README.md is the deploy guide.
|
|
||||||
cc-ci = nixpkgs.lib.nixosSystem {
|
|
||||||
inherit system;
|
inherit system;
|
||||||
modules = [
|
modules = [
|
||||||
cc-ci.nixosModules.cc-ci-server
|
sops-nix.nixosModules.sops
|
||||||
self.nixosModules.cc-ci-orchestrator
|
./nix/hosts/cc-ci-orchestrator-hetzner/hardware.nix
|
||||||
self.nixosModules.orchestrator-host
|
./nix/hosts/cc-ci-orchestrator-hetzner/configuration.nix
|
||||||
self.nixosModules.auto-update
|
|
||||||
./nix/hosts/cc-ci/configuration.nix
|
|
||||||
];
|
];
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
};
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -10,10 +10,9 @@ metadata:
|
|||||||
The cc-ci orchestrator (loops + watchdog + this session) runs on a **Hetzner cpx22** as of
|
The cc-ci orchestrator (loops + watchdog + this session) runs on a **Hetzner cpx22** as of
|
||||||
2026-05-31, replacing the Incus VM (100.116.55.106).
|
2026-05-31, replacing the Incus VM (100.116.55.106).
|
||||||
|
|
||||||
- Since 2026-09-07: ONE Hetzner host for CI server + orchestrator, public **195.201.88.249**,
|
- Hetzner server **134487234**, public **168.119.126.100**, tailnet **cc-ci-orchestrator-1** @
|
||||||
tailnet **cc-ci**, flake host **`.#cc-ci`** (this repo). Before: orchestrator on Hetzner
|
**100.84.190.30**. Flake host **cc-ci-orchestrator-hetzner**.
|
||||||
134487234 (168.119.126.100 / 100.84.190.30, `cc-ci-orchestrator-hetzner`), shared with notplants.
|
- Rebuild: `sudo nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner` from `/srv/cc-ci-orch`
|
||||||
- Rebuild: `sudo nixos-rebuild switch --flake .#cc-ci` from `/srv/cc-ci-orch`
|
|
||||||
(`/srv/cc-ci` is a symlink to it). The Bash tool runs as user **loops** (uid 1000, passwordless
|
(`/srv/cc-ci` is a symlink to it). The Bash tool runs as user **loops** (uid 1000, passwordless
|
||||||
sudo) — plain `nixos-rebuild switch` fails on the profile symlink; use `sudo`.
|
sudo) — plain `nixos-rebuild switch` fails on the profile symlink; use `sudo`.
|
||||||
- Reboot-resilience: `cc-ci-loops.service` is **enabled** (wantedBy multi-user.target); ExecStartPre
|
- Reboot-resilience: `cc-ci-loops.service` is **enabled** (wantedBy multi-user.target); ExecStartPre
|
||||||
@@ -24,4 +23,4 @@ The cc-ci orchestrator (loops + watchdog + this session) runs on a **Hetzner cpx
|
|||||||
identity unknown". Set per-repo to match prior commits: `autonomic-bot
|
identity unknown". Set per-repo to match prior commits: `autonomic-bot
|
||||||
<autonomic-bot@git.autonomic.zone>`.
|
<autonomic-bot@git.autonomic.zone>`.
|
||||||
|
|
||||||
Full record: `archive/plans/plan-orchestrator-hetzner-migration.md`.
|
Full record: `cc-ci-plan/plan-orchestrator-hetzner-migration.md`.
|
||||||
|
|||||||
+137
-11
@@ -14,11 +14,7 @@
|
|||||||
# with ACME, and opens 80/443 (previously only 22 was public). Canonical source of
|
# with ACME, and opens 80/443 (previously only 22 was public). Canonical source of
|
||||||
# the module is the project repo; ../atproto-likes.nix is a copy kept in this tree
|
# the module is the project repo; ../atproto-likes.nix is a copy kept in this tree
|
||||||
# because pure evaluation cannot import a path outside the flake.
|
# because pure evaluation cannot import a path outside the flake.
|
||||||
# cc-ci.nix holds everything that exists FOR cc-ci (the loops, the cc-ci orchestrator session,
|
imports = [ ../../atproto-likes.nix ];
|
||||||
# the weekly recipe-upgrade run and its hourly supervisor). Split out 2026-08-20 so this host —
|
|
||||||
# a general agent box that also serves unrelated projects — can be reasoned about separately
|
|
||||||
# from cc-ci. Also exported as `nixosModules.cc-ci` from this repo's flake for other hosts.
|
|
||||||
imports = [ ../../atproto-likes.nix ../../modules/cc-ci.nix ];
|
|
||||||
|
|
||||||
services.openssh = {
|
services.openssh = {
|
||||||
enable = true;
|
enable = true;
|
||||||
@@ -131,8 +127,7 @@ SSHCFG
|
|||||||
|
|
||||||
# opencode web server — one shared instance; agent sessions attach to it for web visibility.
|
# opencode web server — one shared instance; agent sessions attach to it for web visibility.
|
||||||
# Serves the web UI at http://oc.commoninternet.net (via nginx below, tailscale-only).
|
# Serves the web UI at http://oc.commoninternet.net (via nginx below, tailscale-only).
|
||||||
# Public provider configuration is tracked in the cc-ci repo; credentials
|
# Provider creds are read from /srv/cc-ci/.testenv at startup.
|
||||||
# remain in /srv/cc-ci/.testenv.
|
|
||||||
systemd.services.opencode-web = {
|
systemd.services.opencode-web = {
|
||||||
description = "opencode web server for cc-ci agents";
|
description = "opencode web server for cc-ci agents";
|
||||||
wantedBy = [ "multi-user.target" ];
|
wantedBy = [ "multi-user.target" ];
|
||||||
@@ -142,10 +137,7 @@ SSHCFG
|
|||||||
Type = "simple";
|
Type = "simple";
|
||||||
User = "loops"; Group = "users";
|
User = "loops"; Group = "users";
|
||||||
WorkingDirectory = "/srv/cc-ci-orch/cc-ci";
|
WorkingDirectory = "/srv/cc-ci-orch/cc-ci";
|
||||||
EnvironmentFile = [
|
EnvironmentFile = "/srv/cc-ci/.testenv";
|
||||||
"-/srv/cc-ci/cc-ci/.env.public"
|
|
||||||
"/srv/cc-ci/.testenv"
|
|
||||||
];
|
|
||||||
ExecStartPre = "${pkgs.coreutils}/bin/rm -rf /tmp/opencode";
|
ExecStartPre = "${pkgs.coreutils}/bin/rm -rf /tmp/opencode";
|
||||||
ExecStart = "/home/loops/.local/bin/opencode serve --hostname 127.0.0.1 --port 4096";
|
ExecStart = "/home/loops/.local/bin/opencode serve --hostname 127.0.0.1 --port 4096";
|
||||||
Restart = "on-failure";
|
Restart = "on-failure";
|
||||||
@@ -233,7 +225,72 @@ SSHCFG
|
|||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
|
# cc-ci-loops supervisor — workspace staged 2026-05-31, so ENABLED for reboot-resilience.
|
||||||
|
systemd.services.cc-ci-loops = {
|
||||||
|
description = "cc-ci Builder/Adversary loops + watchdog (launch.sh start)";
|
||||||
|
wantedBy = [ "multi-user.target" ]; # enabled after workspace staged (Hetzner cutover)
|
||||||
|
after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
|
||||||
|
wants = [ "network-online.target" ];
|
||||||
|
serviceConfig = {
|
||||||
|
# KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the
|
||||||
|
# default (control-group) systemd kills every leftover process in the cgroup when the unit
|
||||||
|
# stops — and since one tmux server hosts every agent session on this host, a rebuild that
|
||||||
|
# merely touched this unit wiped all of them (operator 2026-08-01). Only the (already
|
||||||
|
# exited) main process is killed now; `systemctl stop` therefore does NOT tear down agents.
|
||||||
|
KillMode = "process";
|
||||||
|
Type = "oneshot"; RemainAfterExit = true;
|
||||||
|
User = "loops"; Group = "users";
|
||||||
|
WorkingDirectory = "/srv/cc-ci/cc-ci";
|
||||||
|
# Append one line to REBOOTS.md per genuine reboot (boot_id-gated; not on manual restart).
|
||||||
|
ExecStartPre = "${pkgs.bash}/bin/bash /srv/cc-ci/cc-ci-plan/reboot-log.sh";
|
||||||
|
};
|
||||||
|
# CLAUDE_BIN points at the standalone CLI installed by claude-install.service; the loops
|
||||||
|
# backend defaults to claude (persisted in .loop-backend). Without this, launch.py's preflight
|
||||||
|
# `which(claude)` fails because the systemd `path` below has no /home/loops/.local/bin.
|
||||||
|
environment = { RESUME_PHASE = "1"; HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
|
||||||
|
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||||
|
script = ''
|
||||||
|
# Put the standalone claude/opencode binaries on PATH. On a cold boot this is the env the
|
||||||
|
# tmux server (and thus every agent session) inherits, so bare `claude` resolves everywhere.
|
||||||
|
export PATH="/home/loops/.local/bin:$PATH"
|
||||||
|
[ -x /srv/cc-ci/cc-ci-plan/launch.sh ] && /srv/cc-ci/cc-ci-plan/launch.sh start || \
|
||||||
|
echo "workspace not staged yet — skipping loop start"
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
# cc-ci-orchestrator supervisor — the operator's steering session. Same shape as
|
||||||
|
# lichen-orchestrator / project-orchestrator above: this unit only LAUNCHES the orchestrator's
|
||||||
|
# tmux session via the agent-orchestrator harness (cc-ci-plan/agents.py); it does not own the
|
||||||
|
# session or the tmux server. The orchestrator agent is declared in cc-ci-plan/agents.toml on
|
||||||
|
# the OPencode backend (backend = "opencode", model = "opencode/glm-5.2"), so on boot it
|
||||||
|
# attaches to the shared opencode web server (opencode-web.service below) and is reachable for
|
||||||
|
# Remote Control at https://oc.commoninternet.net under the /srv/cc-ci-orch project. The harness
|
||||||
|
# watchdog (started by `agents.py up`) keeps it alive: heal-only (no stall reboots — a persistent
|
||||||
|
# supervisor must not be killed just for idling). Added 2026-08-03 to give the cc-ci orchestrator
|
||||||
|
# the same reboot-resilience the other two orchestrators already have.
|
||||||
|
systemd.services.cc-ci-orchestrator = {
|
||||||
|
description = "cc-ci orchestrator (operator steering session) — agents.py up orchestrator, opencode backend";
|
||||||
|
wantedBy = [ "multi-user.target" ];
|
||||||
|
after = [ "network-online.target" "tailscaled.service" "opencode-web.service" ];
|
||||||
|
wants = [ "network-online.target" ];
|
||||||
|
serviceConfig = {
|
||||||
|
# KillMode=process: see the note on cc-ci-loops — a rebuild that merely touches this unit
|
||||||
|
# must not tear down the (shared) tmux server and every agent session with it.
|
||||||
|
KillMode = "process";
|
||||||
|
Type = "oneshot"; RemainAfterExit = true;
|
||||||
|
User = "loops"; Group = "users";
|
||||||
|
WorkingDirectory = "/srv/cc-ci-orch";
|
||||||
|
};
|
||||||
|
environment = { HOME = "/home/loops"; };
|
||||||
|
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||||
|
script = ''
|
||||||
|
export PATH="/home/loops/.local/bin:$PATH"
|
||||||
|
proj="/srv/cc-ci-orch"
|
||||||
|
echo "$(cat /proc/sys/kernel/random/boot_id) boot $(date -u +%FT%TZ) — cc-ci-orchestrator up" \
|
||||||
|
>> "$proj/cc-ci-plan/.ao-boot.log" 2>/dev/null || true
|
||||||
|
cd "$proj" && python3 cc-ci-plan/agents.py up orchestrator || echo "cc-ci orchestrator agents.py up failed"
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
# p-lichen-orchestrator supervisor — the SEPARATE lichen.page testing/hardening orchestrator
|
# p-lichen-orchestrator supervisor — the SEPARATE lichen.page testing/hardening orchestrator
|
||||||
# (distinct from cc-ci-loops above). Reboot-resilience: on boot, resume the orchestrator's Remote
|
# (distinct from cc-ci-loops above). Reboot-resilience: on boot, resume the orchestrator's Remote
|
||||||
@@ -299,7 +356,76 @@ SSHCFG
|
|||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
# Weekly recipe upgrade — runs /upgrade-all over every enrolled recipe (opens recipe PRs
|
||||||
|
# verified by !testme, never merges). Replaces the boot-fragile busybox-crond-in-tmux from
|
||||||
|
# phase 5 §4 with a reboot-safe systemd timer. The service is timer-triggered only (NOT
|
||||||
|
# wantedBy multi-user.target) so it never runs on boot/activation — only on the schedule.
|
||||||
|
systemd.services.cc-ci-upgrade-all = {
|
||||||
|
description = "cc-ci weekly /upgrade-all run (recipe upgrade survey + PRs, never merges)";
|
||||||
|
after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
|
||||||
|
wants = [ "network-online.target" ];
|
||||||
|
serviceConfig = {
|
||||||
|
Type = "oneshot"; # launch-upgrader.py spawns the cc-ci-upgrader tmux session and returns
|
||||||
|
User = "loops"; Group = "users";
|
||||||
|
WorkingDirectory = "/srv/cc-ci";
|
||||||
|
# Optional per-run overrides for backend/model (LOOP_BACKEND, LOOP_MODEL, OPENCODE_SHARE,
|
||||||
|
# UPGRADER_ARGS, …). The leading "-" makes it optional: absent file → claude/sonnet defaults
|
||||||
|
# (current behavior). To run the weekly job on e.g. opencode-go/glm-5.2, drop a file with
|
||||||
|
# LOOP_BACKEND=opencode
|
||||||
|
# LOOP_MODEL=opencode-go/glm-5.2
|
||||||
|
# No rebuild needed to switch — the env file is read at each timer fire. Holds no secrets
|
||||||
|
# (the opencode-go API key lives in ~/.local/share/opencode/auth.json, mode 600).
|
||||||
|
EnvironmentFile = "-/srv/cc-ci/upgrader.env";
|
||||||
|
};
|
||||||
|
environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
|
||||||
|
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||||
|
script = ''
|
||||||
|
export PATH="/home/loops/.local/bin:$PATH"
|
||||||
|
python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py start >> /srv/cc-ci/.cc-ci-logs/upgrader-cron.log 2>&1
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
systemd.timers.cc-ci-upgrade-all = {
|
||||||
|
description = "Weekly trigger for cc-ci-upgrade-all (Thursdays 22:00 America/New_York — Boston 10pm)";
|
||||||
|
wantedBy = [ "timers.target" ];
|
||||||
|
timerConfig = {
|
||||||
|
# 10pm Thursday Boston time — DST-aware (EDT→02:00 UTC, EST→03:00 UTC) via the tz in OnCalendar.
|
||||||
|
OnCalendar = "Thu *-*-* 22:00:00 America/New_York";
|
||||||
|
Persistent = true; # if the box was down at the scheduled time, run once on next boot
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
# Hourly SUPERVISOR — a glm-5.2 orchestrator wake-up that keeps the weekly run on track. The
|
||||||
|
# log-idle/429 watchdog only handles opencode-go usage-limit stalls; it does NOT cover a host
|
||||||
|
# disk-full crash (which killed the 2026-07-03 run) or any other environmental wedge. This is a
|
||||||
|
# CHEAP deterministic gate: if the weekly run is complete or actively progressing it does NOTHING
|
||||||
|
# (zero model tokens). Only when a run has stalled/died before completing does it launch a
|
||||||
|
# short-lived glm-5.2 agent that diagnoses the blockage and drives the run to a clean DONE.
|
||||||
|
systemd.services.cc-ci-upgrade-supervisor = {
|
||||||
|
description = "cc-ci hourly weekly-run supervisor (glm-5.2 — drives a stalled /upgrade-all to completion)";
|
||||||
|
after = [ "network-online.target" "tailscaled.service" ];
|
||||||
|
wants = [ "network-online.target" ];
|
||||||
|
serviceConfig = {
|
||||||
|
Type = "oneshot"; # launch-supervisor.py check: gate now, spawn the agent into tmux, return
|
||||||
|
User = "loops"; Group = "users";
|
||||||
|
WorkingDirectory = "/srv/cc-ci";
|
||||||
|
# Shares the weekly run's optional override file (e.g. SUPERVISOR_MODEL=…); "-" = optional.
|
||||||
|
EnvironmentFile = "-/srv/cc-ci/upgrader.env";
|
||||||
|
};
|
||||||
|
environment = { HOME = "/home/loops"; };
|
||||||
|
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||||
|
script = ''
|
||||||
|
export PATH="/home/loops/.local/bin:$PATH"
|
||||||
|
python3 /srv/cc-ci/cc-ci-plan/launch-supervisor.py check >> /srv/cc-ci/.cc-ci-logs/supervisor-cron.log 2>&1
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
systemd.timers.cc-ci-upgrade-supervisor = {
|
||||||
|
description = "Hourly trigger for cc-ci-upgrade-supervisor (weekly-run health check + drive)";
|
||||||
|
wantedBy = [ "timers.target" ];
|
||||||
|
timerConfig = {
|
||||||
|
OnCalendar = "*-*-* *:07:00"; # every hour at :07 (offset from the weekly :00 fire)
|
||||||
|
Persistent = false; # a missed hourly check is moot — the next hour re-checks
|
||||||
|
};
|
||||||
|
};
|
||||||
}
|
}
|
||||||
@@ -1,144 +0,0 @@
|
|||||||
# cc-ci — ONE Hetzner Cloud host running both the cc-ci CI server and the cc-ci orchestrator.
|
|
||||||
#
|
|
||||||
# This file is only what is physical or identity about the machine: hardware, networking,
|
|
||||||
# root SSH keys, firewall + fail2ban, swap, stateVersion. Everything functional comes from modules:
|
|
||||||
# cc-ci.nixosModules.cc-ci-server recipe-maintainers/cc-ci — swarm, traefik, drone,
|
|
||||||
# runner, bridge, dashboard, reports, acme-dns, harness
|
|
||||||
# self.nixosModules.cc-ci-orchestrator nix/modules/cc-ci.nix — loops, orchestrator, timers
|
|
||||||
# self.nixosModules.orchestrator-host nix/modules/orchestrator-host.nix — loops user, CLIs
|
|
||||||
# See README.md for provisioning (Hetzner Debian → nixos-infect → this flake) and staging.
|
|
||||||
{ lib, pkgs, ... }:
|
|
||||||
{
|
|
||||||
imports = [
|
|
||||||
./hardware.nix
|
|
||||||
./networking.nix
|
|
||||||
];
|
|
||||||
|
|
||||||
networking.hostName = "cc-ci";
|
|
||||||
|
|
||||||
# ---- cc-ci server identity --------------------------------------------------------------
|
|
||||||
# Public address: acme-dns binds to it and publishes it as the `ns-acme` glue record; the
|
|
||||||
# Gandi A records for ci / *.ci / ns-acme .commoninternet.net point here.
|
|
||||||
cc-ci.publicIPv4 = "195.201.88.249";
|
|
||||||
# cc-ci is a plain flake input here (no private submodule), so the sops file is the one in
|
|
||||||
# the deployed --recursive checkout the weekly sweep runs from (README "Stage the workspace").
|
|
||||||
cc-ci.sopsFile = "/etc/cc-ci/secrets/secrets.yaml";
|
|
||||||
|
|
||||||
# ---- orchestrator identity --------------------------------------------------------------
|
|
||||||
# The CI server is this very host, so `ssh cc-ci` goes to loopback (the module default).
|
|
||||||
cc-ci-orchestrator.ciSshHost = "127.0.0.1";
|
|
||||||
# Weekly self-update (Tue 03:00 UTC; skips itself while CI is busy; see nix/modules/auto-update.nix).
|
|
||||||
cc-ci-orchestrator.autoUpdate.enable = true;
|
|
||||||
|
|
||||||
# The opencode UI: traefik (public 443, the *.ci.commoninternet.net cert) → nginx basic auth.
|
|
||||||
cc-ci-orchestrator.opencodeUiHost = "oc.ci.commoninternet.net";
|
|
||||||
cc-ci-orchestrator.opencodeUiTraefikNetwork = "proxy";
|
|
||||||
|
|
||||||
# ---- no tailscale on this host (operator 2026-09-07) --------------------------------------
|
|
||||||
# Everything is reached over the public IP: ssh (keys only), the CI front doors via traefik,
|
|
||||||
# and the opencode UI on 443 (traefik → nginx basic auth). fail2ban below guards the two logins.
|
|
||||||
|
|
||||||
# ---- /secrets is THE authoritative location for this host's secret material ---------------
|
|
||||||
# Operator rule (2026-09-08): every secret lives under /secrets; anything that needs one either
|
|
||||||
# reads it from there directly (where we own the path) or reaches it by a symlink (where the
|
|
||||||
# consuming software fixes the path). One directory to audit, back up, and reason about.
|
|
||||||
#
|
|
||||||
# /secrets/files loops:users 0700 the agent's secrets (testenv, opencode auth, its ssh keys)
|
|
||||||
# /secrets/host root:root 0700 host identity: ssh host keys + the sops age identity
|
|
||||||
# /secrets/nginx root:nginx 0750 the opencode UI htpasswd (nginx must read it)
|
|
||||||
#
|
|
||||||
# /secrets itself is 0711: traversable so nginx can reach its own subdirectory, not listable.
|
|
||||||
systemd.tmpfiles.rules = [
|
|
||||||
"d /secrets 0711 root root -"
|
|
||||||
"d /secrets/host 0700 root root -"
|
|
||||||
"d /secrets/nginx 0750 root nginx -"
|
|
||||||
# Convenience symlinks at the conventional paths, so an operator (or a tool that assumes the
|
|
||||||
# usual location) still finds the host keys. NOT load-bearing: sshd and sops-nix below are
|
|
||||||
# pointed at /secrets directly, precisely so nothing depends on symlink/activation ordering.
|
|
||||||
"L+ /etc/ssh/ssh_host_ed25519_key - - - - /secrets/host/ssh_host_ed25519_key"
|
|
||||||
"L+ /etc/ssh/ssh_host_ed25519_key.pub - - - - /secrets/host/ssh_host_ed25519_key.pub"
|
|
||||||
"L+ /etc/ssh/ssh_host_rsa_key - - - - /secrets/host/ssh_host_rsa_key"
|
|
||||||
"L+ /etc/ssh/ssh_host_rsa_key.pub - - - - /secrets/host/ssh_host_rsa_key.pub"
|
|
||||||
];
|
|
||||||
|
|
||||||
# sops-nix: the cc-ci server module hard-codes /etc/ssh/... and /var/lib/sops-nix/key.txt.
|
|
||||||
# Override both to the authoritative copies. THE ED25519 HOST KEY IS LOAD-BEARING BEYOND SSH:
|
|
||||||
# its age identity (age1tmvg…) is a recipient of cc-ci-secrets, so replacing or regenerating it
|
|
||||||
# makes every cc-ci secret undecryptable. Move it, never re-create it.
|
|
||||||
sops.age.sshKeyPaths = lib.mkForce [ "/secrets/host/ssh_host_ed25519_key" ];
|
|
||||||
sops.age.keyFile = lib.mkForce "/secrets/host/sops-age-key.txt";
|
|
||||||
|
|
||||||
# ---- ssh ----------------------------------------------------------------------------------
|
|
||||||
services.openssh = {
|
|
||||||
enable = true;
|
|
||||||
settings.PermitRootLogin = "yes";
|
|
||||||
# Host keys live in /secrets (above). sshd is pointed here directly rather than through the
|
|
||||||
# /etc/ssh symlinks, so it can never write a NEW key through a dangling link — that would
|
|
||||||
# silently rotate the age identity that decrypts cc-ci-secrets.
|
|
||||||
hostKeys = [
|
|
||||||
{ path = "/secrets/host/ssh_host_ed25519_key"; type = "ed25519"; }
|
|
||||||
{ path = "/secrets/host/ssh_host_rsa_key"; type = "rsa"; bits = 4096; }
|
|
||||||
];
|
|
||||||
};
|
|
||||||
# Root keys: PUBLIC keys, tracked deliberately in ./ssh-keys (one per line, blank lines ok).
|
|
||||||
users.users.root.openssh.authorizedKeys.keys =
|
|
||||||
builtins.filter (s: s != "") (lib.splitString "\n" (builtins.readFile ./ssh-keys));
|
|
||||||
# The loops user can also be reached directly (same keys) — handy for rsync of its workspace.
|
|
||||||
users.users.loops.openssh.authorizedKeys.keys =
|
|
||||||
builtins.filter (s: s != "") (lib.splitString "\n" (builtins.readFile ./ssh-keys));
|
|
||||||
services.openssh.settings.PasswordAuthentication = false;
|
|
||||||
services.openssh.settings.KbdInteractiveAuthentication = false;
|
|
||||||
|
|
||||||
# ---- firewall -------------------------------------------------------------------------------
|
|
||||||
# 80/443 (traefik) and 53 (acme-dns) are opened by the cc-ci-server module; the opencode UI
|
|
||||||
# rides 443 through traefik (orchestrator-host.nix), so only ssh is opened here.
|
|
||||||
networking.firewall = {
|
|
||||||
enable = true;
|
|
||||||
allowedTCPPorts = [ 22 ];
|
|
||||||
};
|
|
||||||
|
|
||||||
# ---- fail2ban: sshd (password auth is off, this stops the log noise and slow brute force) and
|
|
||||||
# the opencode UI's basic auth (nginx logs 401s with the real client IP to the journal; the
|
|
||||||
# built-in nginx-http-auth filter matches them). Those clients arrive through traefik's
|
|
||||||
# docker-published 443, which iptables FORWARDs rather than INPUTs, so the ban for that jail
|
|
||||||
# goes into the DOCKER-USER chain — an INPUT rule would never see the traffic.
|
|
||||||
services.fail2ban = {
|
|
||||||
enable = true;
|
|
||||||
maxretry = 5;
|
|
||||||
bantime = "1h";
|
|
||||||
bantime-increment = { enable = true; maxtime = "48h"; factor = "4"; };
|
|
||||||
ignoreIP = [ "127.0.0.0/8" "::1" ];
|
|
||||||
jails.nginx-http-auth.settings = {
|
|
||||||
enabled = true;
|
|
||||||
filter = "nginx-http-auth";
|
|
||||||
# NixOS nginx logs errors to stderr → the journal, not /var/log/nginx/error.log (which
|
|
||||||
# exists but stays empty). Read the unit's journal instead.
|
|
||||||
backend = "systemd";
|
|
||||||
journalmatch = "_SYSTEMD_UNIT=nginx.service";
|
|
||||||
banaction = "iptables-allports";
|
|
||||||
chain = "DOCKER-USER";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
networking.nameservers = [ "1.1.1.1" "8.8.8.8" ];
|
|
||||||
# This host's own public names resolve to itself regardless of external DNS state (host
|
|
||||||
# processes: the drone runner, the harness, the orchestrator; containers use the resolvers
|
|
||||||
# above). Per-run recipe domains are random and cannot be pinned — those follow public DNS.
|
|
||||||
networking.hosts."195.201.88.249" = [
|
|
||||||
"ci.commoninternet.net"
|
|
||||||
"drone.ci.commoninternet.net"
|
|
||||||
"report.ci.commoninternet.net"
|
|
||||||
"traefik.ci.commoninternet.net"
|
|
||||||
"warm-keycloak.ci.commoninternet.net"
|
|
||||||
"oc.ci.commoninternet.net"
|
|
||||||
];
|
|
||||||
|
|
||||||
# ---- memory: 8 GB RAM shared by the swarm (recipe deploys) and 3–6 agent sessions ---------
|
|
||||||
swapDevices = [ { device = "/swapfile"; size = 8192; } ];
|
|
||||||
|
|
||||||
# ssh client for root (the orchestrator's `ssh cc-ci` goes through the loops user's own config).
|
|
||||||
environment.systemPackages = [ pkgs.openssh ];
|
|
||||||
|
|
||||||
# Fresh NixOS 26.05 install (nixos-infect, 2026-09-07). Never change this on an existing host.
|
|
||||||
system.stateVersion = "26.05";
|
|
||||||
}
|
|
||||||
@@ -1,19 +0,0 @@
|
|||||||
# Generated by nixos-infect on this machine (2026-09-07), captured verbatim per README §3.
|
|
||||||
# The ESP UUID is specific to THIS server; a new server gets a new file.
|
|
||||||
{ modulesPath, ... }:
|
|
||||||
{
|
|
||||||
imports = [ (modulesPath + "/profiles/qemu-guest.nix") ];
|
|
||||||
boot.loader = {
|
|
||||||
efi.efiSysMountPoint = "/boot/efi";
|
|
||||||
grub = {
|
|
||||||
efiSupport = true;
|
|
||||||
efiInstallAsRemovable = true;
|
|
||||||
device = "nodev";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
fileSystems."/boot/efi" = { device = "/dev/disk/by-uuid/E079-7D41"; fsType = "vfat"; };
|
|
||||||
boot.initrd.availableKernelModules = [ "ata_piix" "uhci_hcd" "xen_blkfront" "vmw_pvscsi" ];
|
|
||||||
boot.initrd.kernelModules = [ "nvme" ];
|
|
||||||
fileSystems."/" = { device = "/dev/sda1"; fsType = "ext4"; };
|
|
||||||
|
|
||||||
}
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
# Generated by nixos-infect on this machine (2026-09-07), captured per README §3, with ONE edit:
|
|
||||||
# `defaultGateway` as an attrset WITH `interface = "eth0"`. The generated bare-string form leaves
|
|
||||||
# NixOS ≥25.05 without a default route (the host boots and is unreachable) — see README §2.
|
|
||||||
{ lib, ... }: {
|
|
||||||
# This file was populated at runtime with the networking
|
|
||||||
# details gathered from the active system.
|
|
||||||
networking = {
|
|
||||||
# The Hetzner resolvers nixos-infect wrote here are deliberately gone: configuration.nix sets
|
|
||||||
# 1.1.1.1/8.8.8.8. During the 2026-09-07 cutover the Hetzner caches kept the OLD address for
|
|
||||||
# the full 3h TTL while the public resolvers already had the new one, and every other lookup
|
|
||||||
# of drone.ci.commoninternet.net from this box went to the old server (404s in the runner and
|
|
||||||
# the bridge). One consistent resolver set, and the host's own names pinned in
|
|
||||||
# configuration.nix, keep the CI independent of that.
|
|
||||||
defaultGateway = { address = "172.31.1.1"; interface = "eth0"; };
|
|
||||||
defaultGateway6 = {
|
|
||||||
address = "fe80::1";
|
|
||||||
interface = "eth0";
|
|
||||||
};
|
|
||||||
dhcpcd.enable = false;
|
|
||||||
usePredictableInterfaceNames = lib.mkForce false;
|
|
||||||
interfaces = {
|
|
||||||
eth0 = {
|
|
||||||
ipv4.addresses = [
|
|
||||||
{ address="195.201.88.249"; prefixLength=32; }
|
|
||||||
];
|
|
||||||
ipv6.addresses = [
|
|
||||||
{ address="2a01:4f8:1c1c:a9b::1"; prefixLength=64; }
|
|
||||||
{ address="fe80::2ff8:e3ea:bbb8:aa39"; prefixLength=64; }
|
|
||||||
];
|
|
||||||
ipv4.routes = [ { address = "172.31.1.1"; prefixLength = 32; } ];
|
|
||||||
ipv6.routes = [ { address = "fe80::1"; prefixLength = 128; } ];
|
|
||||||
};
|
|
||||||
|
|
||||||
};
|
|
||||||
};
|
|
||||||
services.udev.extraRules = ''
|
|
||||||
ATTR{address}=="92:00:09:d5:ec:0d", NAME="eth0"
|
|
||||||
|
|
||||||
'';
|
|
||||||
}
|
|
||||||
@@ -1,11 +0,0 @@
|
|||||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGZGp/DQTFuD1GvsyTzCVBUTmoWqcb5T+Z7zZo5nYLXO
|
|
||||||
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhgo41nt8/L+Cr0PKd8jQK45mw/A+h041j6LQ8JWZisEVaQOzr6s9rxPL8VT5ML4P3/4bMblzdDiXWlJxymcb+yk5S5TnVrMavzHEDhWHwEvTRMe6xNTmsU6cmmhRw7PJqqQ+0GTlQalu3I4jkC0kTF7kuPwduUOgUuSpJqxvDTwYiXoyVnOQHAIygh+BmQvYUz0PBfQgIhgcbYmGZ++T0DnMzdGFzW2UB/iy5mymnpmbaZCgLy0w8AoDE+0YLtUc4gwTXc183nvqO1i7LQr+3jBYkv5ZthCCc52vXFHDSw9xZ5ohsOrBvoi5foRbqinmU5/t0aTK7SSrat7xXm/odIOyS+S7PJyeEcsXN6d5zdxbabAy5vLfodEaKGZd4rqQeDCxOTPAS/BlrBV/EV714n4E+fSOAllAuMBO4IibJM/gLJrh2Dql3co50QW9HEDeSC7iqp2lxRBDxvUs3rIEzy7o4HSN8chqBUK1bbBY6B17fuNHIpBAw4akRVVvPnVM= trav@trav480sweet
|
|
||||||
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6jrKj7iZUNRLBTZG0vZM1D/BXtARhhB4+GrvpyuqmPb9iw2ifT9YqRUwgyGrOW9U6nIAR9yFnfp9+FkyhEKWByqEBbe/zYKlGLRGjfsIdDdW29QQ3hvmqNyboCkXLxZGat93poYhnoomqicmGD/xST4s0OUhcK9E494lUmenlD9dcMZW1aKpJ+9O4Dq6A7nk2z1e4KFcZdrZDI2Hgg+gfEdsKZQqd/R3Mls/eVKpzhfv3Y8BiNoHssUChVf8IGESqTOBOR7Dk7FsU5Z2ZcnQ1coxY7VlBn4fPjTWmz/Ac0jLqgcpCLpNyQzFPDVMYZKYrPVoqBeKVhN5YnfwR5OVP8YsakT/obLwC43sx/esXfjhVGcsRoGpiLOfazzNw/eC8s6FlS8cesOubEM37a7F25z4UEG3d487oM7EjQ39gBCCj/KRgUimCKMWsm6yIas4OSctBWEAo/NhZp0gwulSRxleW6eJCNNwzOmWjdzYIVWoVP0EIeM95Tq8PVUN7gpc= aadil@t480
|
|
||||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMyHSi12R0+HCVBz7+d9fyOBnoJi8Nsj5D7vQ9UQO8a5
|
|
||||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJVlfoLBPseQ9fA9534KmRg2KWcksKZGzAJIpHJ2JpsI mfowler.email@protonmail.com
|
|
||||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAQFuqUB2qNZSDNjDsjjhVA/WnnQNVAMmsUscW6OgMDN
|
|
||||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHOcLo0YBa0UYi7i/l8K/Y/7cF2OclmDqSTlAsHM0dOS notplants-orchestrator
|
|
||||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMniNzAzuI527bfk/EipqFILFayUCwYXDoZ3R7+QgYq6
|
|
||||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOk8NaeBdPbS2gfUvbny8h0AkZlVjGYHzx4QPXSJ38gd claude@claude-vm
|
|
||||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAcyTGb/wVgdhg5oBCZZvBaR1RuUQRY/3WHnOQpNDCsp claude-cc-ci-sandbox@20260526
|
|
||||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKmGDZC6wrOQNJAW5PPDpxgEXXrcsnIU4b3QJLtq05RQ cc-ci-loops-to-root@cc-ci
|
|
||||||
@@ -1,199 +0,0 @@
|
|||||||
# auto-update.nix — weekly, health-gated self-update of the combined cc-ci host.
|
|
||||||
#
|
|
||||||
# Modelled on notplants-nix's modules/auto-update.nix (the nightly updater the old orchestrator
|
|
||||||
# box had), with the two things the cc-ci host needs on top:
|
|
||||||
# * a BUSY GATE — a `switch` restarts docker/traefik/drone/the runner, so the update skips
|
|
||||||
# (and simply retries next week) while a CI run, the weekly recipe-upgrade run or the
|
|
||||||
# Sunday canonical sweep is in flight;
|
|
||||||
# * cc-ci HEALTH CHECKS — beyond "sshd + no failed units": every swarm service at its replica
|
|
||||||
# count, the front doors (dashboard, reports, drone, the opencode UI) answering, sops having
|
|
||||||
# decrypted, acme-dns and fail2ban up.
|
|
||||||
#
|
|
||||||
# Sequence (as in the notplants module — read its comments for the whys):
|
|
||||||
# flake update (all inputs: nixpkgs, sops-nix, cc-ci) → nixos-rebuild build (as loops) →
|
|
||||||
# switch-to-configuration test (bootloader untouched) → settle → health check →
|
|
||||||
# on failure: re-activate the previous generation, restore flake.lock, exit 1
|
|
||||||
# on success: set the system profile + bootloader, commit flake.lock as "auto-update", push
|
|
||||||
# main (the repo stays the source of truth), refresh /etc/cc-ci.
|
|
||||||
# Nothing is committed that did not pass the health check. Never fires a missed run at boot.
|
|
||||||
# `/cc-ci-status` reads the outcome from the state file this writes.
|
|
||||||
{ config, lib, pkgs, ... }:
|
|
||||||
let
|
|
||||||
cfg = config.cc-ci-orchestrator.autoUpdate;
|
|
||||||
builderHome = config.users.users.${cfg.buildUser}.home;
|
|
||||||
stateFile = "${cfg.flakePath}/.cc-ci-logs/auto-update-state";
|
|
||||||
curl = "${pkgs.curl}/bin/curl";
|
|
||||||
|
|
||||||
busyGate = pkgs.writeShellScript "cc-ci-auto-update-busy-gate" ''
|
|
||||||
set -u
|
|
||||||
busy() { echo "BUSY: $1 — skipping this week's update"; exit 3; }
|
|
||||||
${pkgs.procps}/bin/pgrep -f run_recipe_ci >/dev/null && busy "a CI run is in flight"
|
|
||||||
${pkgs.systemd}/bin/systemctl is-active --quiet nightly-sweep.service && busy "the canonical sweep is running"
|
|
||||||
${pkgs.util-linux}/bin/runuser -u ${cfg.buildUser} -- ${pkgs.tmux}/bin/tmux has-session -t cc-ci-upgrader 2>/dev/null \
|
|
||||||
&& busy "the weekly recipe-upgrade run is in flight (tmux cc-ci-upgrader)"
|
|
||||||
${pkgs.util-linux}/bin/runuser -u ${cfg.buildUser} -- ${pkgs.tmux}/bin/tmux has-session -t cc-ci-report 2>/dev/null \
|
|
||||||
&& busy "the weekly report is being written (tmux cc-ci-report)"
|
|
||||||
if [ -r /run/secrets/bridge_drone_token ]; then
|
|
||||||
running=$(${curl} -s -m 20 -H "Authorization: Bearer $(cat /run/secrets/bridge_drone_token)" \
|
|
||||||
"https://drone.${cfg.ciDomain}/api/repos/recipe-maintainers/cc-ci/builds?per_page=10" \
|
|
||||||
| ${pkgs.gnugrep}/bin/grep -o '"status":"running"' | ${pkgs.coreutils}/bin/wc -l)
|
|
||||||
[ "''${running:-0}" -eq 0 ] || busy "$running Drone build(s) running"
|
|
||||||
fi
|
|
||||||
echo "not busy"
|
|
||||||
'';
|
|
||||||
|
|
||||||
healthCheck = pkgs.writeShellScript "cc-ci-auto-update-health-check" ''
|
|
||||||
set -u
|
|
||||||
fail() { echo "HEALTH FAIL: $1"; exit 1; }
|
|
||||||
${pkgs.systemd}/bin/systemctl is-active --quiet sshd || fail "sshd not active"
|
|
||||||
${pkgs.iproute2}/bin/ss -tlnH | ${pkgs.gnugrep}/bin/grep -q ':22 ' || fail "nothing listening on :22"
|
|
||||||
n=$(${pkgs.systemd}/bin/systemctl --failed --no-legend | ${pkgs.coreutils}/bin/wc -l)
|
|
||||||
[ "$n" -eq 0 ] || fail "$n failed unit(s): $(${pkgs.systemd}/bin/systemctl --failed --no-legend --plain | ${pkgs.gawk}/bin/awk '{print $1}' | ${pkgs.coreutils}/bin/tr '\n' ' ')"
|
|
||||||
for u in docker acme-dns fail2ban nginx opencode-web drone-runner-exec; do
|
|
||||||
${pkgs.systemd}/bin/systemctl is-active --quiet "$u" || fail "$u not active"
|
|
||||||
done
|
|
||||||
# The reconcile oneshots re-run on activation; give the swarm up to 10 minutes to converge.
|
|
||||||
for i in $(${pkgs.coreutils}/bin/seq 1 40); do
|
|
||||||
short=$(${pkgs.docker}/bin/docker service ls --format '{{.Name}} {{.Replicas}}' \
|
|
||||||
| ${pkgs.gawk}/bin/awk '{ split($2,a,"/"); if (a[1] != a[2]) print $1 }')
|
|
||||||
[ -z "$short" ] && break
|
|
||||||
${pkgs.coreutils}/bin/sleep 15
|
|
||||||
done
|
|
||||||
[ -z "$short" ] || fail "swarm services not at their replica count: $(echo "$short" | ${pkgs.coreutils}/bin/tr '\n' ' ')"
|
|
||||||
[ -s /run/secrets/test_secret ] || fail "sops did not decrypt (/run/secrets/test_secret missing)"
|
|
||||||
code() { ${curl} -s -m 20 -o /dev/null -w '%{http_code}' --resolve "$1:443:127.0.0.1" "https://$1/$2"; }
|
|
||||||
[ "$(code ${cfg.ciDomain} "")" = 200 ] || fail "dashboard not 200"
|
|
||||||
[ "$(code report.${cfg.ciDomain} "")" = 200 ] || fail "reports not 200"
|
|
||||||
[ "$(code drone.${cfg.ciDomain} version)" = 200 ] || fail "drone /version not 200"
|
|
||||||
[ "$(code ${cfg.opencodeUiHost} "")" = 401 ] || fail "opencode UI not answering with its auth challenge"
|
|
||||||
echo "HEALTH OK"
|
|
||||||
'';
|
|
||||||
in
|
|
||||||
{
|
|
||||||
options.cc-ci-orchestrator.autoUpdate = {
|
|
||||||
enable = lib.mkEnableOption "weekly health-gated self-update of the cc-ci host";
|
|
||||||
flakePath = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
default = "/srv/cc-ci-orch";
|
|
||||||
description = "The cc-ci-orchestrator checkout (owned by buildUser) whose flake.lock is updated, committed and pushed.";
|
|
||||||
};
|
|
||||||
flakeRef = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
default = "${cfg.flakePath}#cc-ci";
|
|
||||||
description = "Flake reference to build.";
|
|
||||||
};
|
|
||||||
buildUser = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
default = "loops";
|
|
||||||
description = "Owner of the checkout; runs the update, build, commit and push (root activates).";
|
|
||||||
};
|
|
||||||
onCalendar = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
default = "Tue *-*-* 03:00:00 UTC";
|
|
||||||
description = ''
|
|
||||||
When to run. Must stay clear of the weekly recipe-upgrade run (Thu 22:00 America/New_York
|
|
||||||
= Fri 02:00/03:00 UTC, several hours) and the Sunday 03:00 UTC canonical sweep (up to 6 h);
|
|
||||||
the busy gate covers the rest.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
remote = lib.mkOption { type = lib.types.str; default = "origin"; };
|
|
||||||
branch = lib.mkOption { type = lib.types.str; default = "main"; };
|
|
||||||
ciDomain = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
default = "ci.commoninternet.net";
|
|
||||||
description = "The CI apex; report./drone. are checked under it.";
|
|
||||||
};
|
|
||||||
opencodeUiHost = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
default = config.cc-ci-orchestrator.opencodeUiHost;
|
|
||||||
description = "The opencode UI host name (expected to answer 401).";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
config = lib.mkIf cfg.enable {
|
|
||||||
systemd.services.cc-ci-auto-update = {
|
|
||||||
description = "Weekly cc-ci host self-update (flake update → build → test → health check → commit)";
|
|
||||||
after = [ "network-online.target" "docker.service" ];
|
|
||||||
wants = [ "network-online.target" ];
|
|
||||||
path = with pkgs; [ nixos-rebuild nix git openssh coreutils gnugrep gawk systemd util-linux bash python3 docker curl ];
|
|
||||||
serviceConfig = { Type = "oneshot"; TimeoutStartSec = "3h"; };
|
|
||||||
# A unit that performs the switch must never be stopped or restarted BY that switch
|
|
||||||
# (notplants-orchestrator, 2026-09-07: it was, and the host ran half-switched for 12 h).
|
|
||||||
stopIfChanged = false;
|
|
||||||
restartIfChanged = false;
|
|
||||||
script = ''
|
|
||||||
set -u
|
|
||||||
cd ${cfg.flakePath}
|
|
||||||
as_builder() { runuser -u ${cfg.buildUser} -- env HOME=${builderHome} "$@"; }
|
|
||||||
state() { printf '%s result=%s generation=%s note=%s\n' "$(date -u +%FT%TZ)" "$1" "$2" "$3" > ${stateFile}; chown ${cfg.buildUser} ${stateFile} || true; }
|
|
||||||
PREV=$(readlink -f /run/current-system)
|
|
||||||
LOCK_BAK=${cfg.flakePath}/.flake.lock.auto-update-prev
|
|
||||||
|
|
||||||
if ! ${busyGate}; then
|
|
||||||
state skipped "$PREV" busy
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
revert() {
|
|
||||||
echo "REVERT: $1"
|
|
||||||
as_builder cp "$LOCK_BAK" flake.lock 2>/dev/null || true
|
|
||||||
"$PREV"/bin/switch-to-configuration test || echo "re-activation of $PREV failed"
|
|
||||||
state failed "$PREV" "$1"
|
|
||||||
exit 1
|
|
||||||
}
|
|
||||||
|
|
||||||
as_builder cp flake.lock "$LOCK_BAK"
|
|
||||||
echo "=== checkout at: $(as_builder git rev-parse --short HEAD) on $(as_builder git rev-parse --abbrev-ref HEAD)"
|
|
||||||
echo "=== flake update (all inputs)"
|
|
||||||
as_builder nix flake update || revert "flake update failed"
|
|
||||||
if as_builder git diff --quiet -- flake.lock; then
|
|
||||||
echo "=== nothing moved; done"
|
|
||||||
state unchanged "$PREV" "lock already current"
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
as_builder git --no-pager diff --stat -- flake.lock
|
|
||||||
|
|
||||||
echo "=== build (cannot affect the running system)"
|
|
||||||
as_builder nixos-rebuild build --flake ${cfg.flakeRef} || revert "build failed"
|
|
||||||
NEW=$(readlink -f result) || revert "no build result"
|
|
||||||
as_builder rm -f result
|
|
||||||
echo "=== built: $NEW"
|
|
||||||
|
|
||||||
echo "=== activate WITHOUT touching the bootloader"
|
|
||||||
"$NEW"/bin/switch-to-configuration test || revert "test activation failed"
|
|
||||||
|
|
||||||
echo "=== settle, then health check"
|
|
||||||
sleep 45
|
|
||||||
${healthCheck} || revert "health check failed"
|
|
||||||
|
|
||||||
echo "=== healthy: committing (system profile + bootloader)"
|
|
||||||
nix-env -p /nix/var/nix/profiles/system --set "$NEW" || revert "could not set system profile"
|
|
||||||
"$NEW"/bin/switch-to-configuration boot || revert "bootloader update failed"
|
|
||||||
state ok "$NEW" "committed"
|
|
||||||
|
|
||||||
# Bookkeeping from here on — never revert a healthy, committed generation over it.
|
|
||||||
if as_builder git -c user.name="auto-update" -c user.email="auto-update@cc-ci" \
|
|
||||||
commit -q -m "flake.lock: weekly auto-update, health-checked" -- flake.lock; then
|
|
||||||
echo "=== flake.lock committed: $(as_builder git rev-parse --short HEAD)"
|
|
||||||
as_builder git push -q ${cfg.remote} HEAD:${cfg.branch} \
|
|
||||||
&& echo "=== pushed ${cfg.branch}" \
|
|
||||||
|| echo "WARNING: push failed; the lock is committed locally, push it by hand"
|
|
||||||
else
|
|
||||||
echo "WARNING: could not commit flake.lock; the system IS committed and healthy"
|
|
||||||
fi
|
|
||||||
# Keep the deployed cc-ci checkout (sweep + sops file) on the same cc-ci rev as the flake input.
|
|
||||||
git -C /etc/cc-ci pull -q --ff-only --recurse-submodules || echo "WARNING: /etc/cc-ci pull failed"
|
|
||||||
echo "=== auto-update committed: $NEW"
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
systemd.timers.cc-ci-auto-update = {
|
|
||||||
wantedBy = [ "timers.target" ];
|
|
||||||
timerConfig = {
|
|
||||||
OnCalendar = cfg.onCalendar;
|
|
||||||
Persistent = false;
|
|
||||||
RandomizedDelaySec = "10min";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@@ -1,157 +0,0 @@
|
|||||||
# cc-ci.nix — the cc-ci ORCHESTRATOR: the Builder/Adversary loops supervisor, the operator's
|
|
||||||
# steering session, and the weekly-upgrade + hourly-supervisor timers. Nothing else.
|
|
||||||
#
|
|
||||||
# Exported from this repo's flake as `nixosModules.cc-ci-orchestrator` (and, for the host that
|
|
||||||
# used to import it under the old name, `nixosModules.cc-ci`). Split out of the shared agent
|
|
||||||
# host config on 2026-08-20; since 2026-09 it runs on the same Hetzner host as the CI server
|
|
||||||
# itself (`#cc-ci` in flake.nix), next to recipe-maintainers/cc-ci's `nixosModules.cc-ci-server`.
|
|
||||||
#
|
|
||||||
# All of it assumes the cc-ci workspaces exist on the host:
|
|
||||||
# /srv/cc-ci the loops workspace (+ .cc-ci-logs, upgrader.env) — a symlink to
|
|
||||||
# /srv/cc-ci-orch this repo (the orchestrator's own working dir), with cc-ci/ checked out
|
|
||||||
# and that a `loops` user, tmux, python3 and the standalone claude/opencode CLIs are present —
|
|
||||||
# those are host concerns, provided by nix/modules/orchestrator-host.nix, not by this module.
|
|
||||||
{ config, pkgs, lib, ... }:
|
|
||||||
{
|
|
||||||
# cc-ci-loops supervisor — workspace staged 2026-05-31, so ENABLED for reboot-resilience.
|
|
||||||
systemd.services.cc-ci-loops = {
|
|
||||||
description = "cc-ci Builder/Adversary loops + watchdog (launch.sh start)";
|
|
||||||
wantedBy = [ "multi-user.target" ]; # enabled after workspace staged (Hetzner cutover)
|
|
||||||
after = [ "network-online.target" "opencode-install.service" ];
|
|
||||||
wants = [ "network-online.target" ];
|
|
||||||
serviceConfig = {
|
|
||||||
# KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the
|
|
||||||
# default (control-group) systemd kills every leftover process in the cgroup when the unit
|
|
||||||
# stops — and since one tmux server hosts every agent session on this host, a rebuild that
|
|
||||||
# merely touched this unit wiped all of them (operator 2026-08-01). Only the (already
|
|
||||||
# exited) main process is killed now; `systemctl stop` therefore does NOT tear down agents.
|
|
||||||
KillMode = "process";
|
|
||||||
Type = "oneshot"; RemainAfterExit = true;
|
|
||||||
User = "loops"; Group = "users";
|
|
||||||
WorkingDirectory = "/srv/cc-ci/cc-ci";
|
|
||||||
# Append one line to REBOOTS.md per genuine reboot (boot_id-gated; not on manual restart).
|
|
||||||
ExecStartPre = "${pkgs.bash}/bin/bash /srv/cc-ci/cc-ci-plan/reboot-log.sh";
|
|
||||||
};
|
|
||||||
# The loops backend is persisted in .cc-ci-logs/.loop-backend (opencode on the cc-ci host —
|
|
||||||
# no Claude there since 2026-09-07). The build phases are all DONE; on boot launch.sh just
|
|
||||||
# re-confirms that and exits.
|
|
||||||
environment = { RESUME_PHASE = "1"; HOME = "/home/loops"; };
|
|
||||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
|
||||||
script = ''
|
|
||||||
# Put the standalone opencode binary on PATH. On a cold boot this is the env the tmux
|
|
||||||
# server (and thus every agent session) inherits.
|
|
||||||
export PATH="/home/loops/.local/bin:$PATH"
|
|
||||||
[ -x /srv/cc-ci/cc-ci-plan/launch.sh ] && /srv/cc-ci/cc-ci-plan/launch.sh start || \
|
|
||||||
echo "workspace not staged yet — skipping loop start"
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
# cc-ci-orchestrator supervisor — the operator's steering session. Same shape as
|
|
||||||
# lichen-orchestrator / project-orchestrator above: this unit only LAUNCHES the orchestrator's
|
|
||||||
# tmux session via the agent-orchestrator harness (cc-ci-plan/agents.py); it does not own the
|
|
||||||
# session or the tmux server. The orchestrator agent is declared in cc-ci-plan/agents.toml
|
|
||||||
# (backend/model chosen there: an opencode agent, opencode/glm-5.2, attached to the shared
|
|
||||||
# opencode web server — opencode-web.service in orchestrator-host.nix — and steered from its
|
|
||||||
# web UI; no Claude on this host, operator 2026-09-07). The harness watchdog (started by
|
|
||||||
# `agents.py up`) keeps it alive: heal-only (no stall reboots — a persistent supervisor must not
|
|
||||||
# be killed just for idling). Added 2026-08-03 for reboot-resilience.
|
|
||||||
systemd.services.cc-ci-orchestrator = {
|
|
||||||
description = "cc-ci orchestrator (operator steering session) — agents.py up orchestrator";
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
|
||||||
after = [ "network-online.target" "opencode-web.service" ];
|
|
||||||
wants = [ "network-online.target" ];
|
|
||||||
serviceConfig = {
|
|
||||||
# KillMode=process: see the note on cc-ci-loops — a rebuild that merely touches this unit
|
|
||||||
# must not tear down the (shared) tmux server and every agent session with it.
|
|
||||||
KillMode = "process";
|
|
||||||
Type = "oneshot"; RemainAfterExit = true;
|
|
||||||
User = "loops"; Group = "users";
|
|
||||||
WorkingDirectory = "/srv/cc-ci-orch";
|
|
||||||
};
|
|
||||||
environment = { HOME = "/home/loops"; };
|
|
||||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
|
||||||
script = ''
|
|
||||||
export PATH="/home/loops/.local/bin:$PATH"
|
|
||||||
proj="/srv/cc-ci-orch"
|
|
||||||
echo "$(cat /proc/sys/kernel/random/boot_id) boot $(date -u +%FT%TZ) — cc-ci-orchestrator up" \
|
|
||||||
>> "$proj/cc-ci-plan/.ao-boot.log" 2>/dev/null || true
|
|
||||||
cd "$proj" && python3 cc-ci-plan/agents.py up orchestrator || echo "cc-ci orchestrator agents.py up failed"
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
# Weekly recipe upgrade — runs /upgrade-all over every enrolled recipe (opens recipe PRs
|
|
||||||
# verified by !testme, never merges). Replaces the boot-fragile busybox-crond-in-tmux from
|
|
||||||
# phase 5 §4 with a reboot-safe systemd timer. The service is timer-triggered only (NOT
|
|
||||||
# wantedBy multi-user.target) so it never runs on boot/activation — only on the schedule.
|
|
||||||
systemd.services.cc-ci-upgrade-all = {
|
|
||||||
description = "cc-ci weekly /upgrade-all run (recipe upgrade survey + PRs, never merges)";
|
|
||||||
after = [ "network-online.target" "opencode-install.service" ];
|
|
||||||
wants = [ "network-online.target" ];
|
|
||||||
serviceConfig = {
|
|
||||||
Type = "oneshot"; # launch-upgrader.py spawns the cc-ci-upgrader tmux session and returns
|
|
||||||
User = "loops"; Group = "users";
|
|
||||||
WorkingDirectory = "/srv/cc-ci";
|
|
||||||
# Optional per-run overrides for backend/model (LOOP_BACKEND, LOOP_MODEL, OPENCODE_SHARE,
|
|
||||||
# UPGRADER_ARGS, …). The leading "-" makes it optional: absent file → claude/sonnet defaults.
|
|
||||||
# Current config (as of 2026-08-16): the upgrader + report run on tinfoil/deepseek-v4-pro
|
|
||||||
# (LOOP_MODEL + REPORT_MODEL in the env file); the hourly SUPERVISOR stays on glm-5.2
|
|
||||||
# (SUPERVISOR_MODEL defaults to opencode-go/glm-5.2 in launch-supervisor.py, NOT overridden
|
|
||||||
# here). Subagents bind deepseek via the cc-ci repo's opencode config. LOOP_TIER=zen is kept
|
|
||||||
# so the tier check passes; the watchdog's usage-limit probe sends the deepseek model name to
|
|
||||||
# the zen endpoint, which returns 200 (not 429) → resume immediately (correct: tinfoil has no
|
|
||||||
# rolling usage limit to wait out). No rebuild needed to switch — the env file is read at each
|
|
||||||
# timer fire. Holds no secrets (the tinfoil API key lives in the opencode config / auth.json).
|
|
||||||
EnvironmentFile = "-/srv/cc-ci/upgrader.env";
|
|
||||||
};
|
|
||||||
environment = { HOME = "/home/loops"; };
|
|
||||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
|
||||||
script = ''
|
|
||||||
export PATH="/home/loops/.local/bin:$PATH"
|
|
||||||
python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py start >> /srv/cc-ci/.cc-ci-logs/upgrader-cron.log 2>&1
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
systemd.timers.cc-ci-upgrade-all = {
|
|
||||||
description = "Weekly trigger for cc-ci-upgrade-all (Thursdays 22:00 America/New_York — Boston 10pm)";
|
|
||||||
wantedBy = [ "timers.target" ];
|
|
||||||
timerConfig = {
|
|
||||||
# 10pm Thursday Boston time — DST-aware (EDT→02:00 UTC, EST→03:00 UTC) via the tz in OnCalendar.
|
|
||||||
OnCalendar = "Thu *-*-* 22:00:00 America/New_York";
|
|
||||||
Persistent = true; # if the box was down at the scheduled time, run once on next boot
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
# Hourly SUPERVISOR — a glm-5.2 orchestrator wake-up that keeps the weekly run on track. The
|
|
||||||
# log-idle/429 watchdog only handles opencode-go usage-limit stalls; it does NOT cover a host
|
|
||||||
# disk-full crash (which killed the 2026-07-03 run) or any other environmental wedge. This is a
|
|
||||||
# CHEAP deterministic gate: if the weekly run is complete or actively progressing it does NOTHING
|
|
||||||
# (zero model tokens). Only when a run has stalled/died before completing does it launch a
|
|
||||||
# short-lived glm-5.2 agent that diagnoses the blockage and drives the run to a clean DONE.
|
|
||||||
systemd.services.cc-ci-upgrade-supervisor = {
|
|
||||||
description = "cc-ci hourly weekly-run supervisor (glm-5.2 — drives a stalled /upgrade-all to completion)";
|
|
||||||
after = [ "network-online.target" ];
|
|
||||||
wants = [ "network-online.target" ];
|
|
||||||
serviceConfig = {
|
|
||||||
Type = "oneshot"; # launch-supervisor.py check: gate now, spawn the agent into tmux, return
|
|
||||||
User = "loops"; Group = "users";
|
|
||||||
WorkingDirectory = "/srv/cc-ci";
|
|
||||||
# Shares the weekly run's optional override file (e.g. SUPERVISOR_MODEL=…); "-" = optional.
|
|
||||||
EnvironmentFile = "-/srv/cc-ci/upgrader.env";
|
|
||||||
};
|
|
||||||
environment = { HOME = "/home/loops"; };
|
|
||||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
|
||||||
script = ''
|
|
||||||
export PATH="/home/loops/.local/bin:$PATH"
|
|
||||||
python3 /srv/cc-ci/cc-ci-plan/launch-supervisor.py check >> /srv/cc-ci/.cc-ci-logs/supervisor-cron.log 2>&1
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
systemd.timers.cc-ci-upgrade-supervisor = {
|
|
||||||
description = "Hourly trigger for cc-ci-upgrade-supervisor (weekly-run health check + drive)";
|
|
||||||
wantedBy = [ "timers.target" ];
|
|
||||||
timerConfig = {
|
|
||||||
OnCalendar = "*-*-* *:07:00"; # every hour at :07 (offset from the weekly :00 fire)
|
|
||||||
Persistent = false; # a missed hourly check is moot — the next hour re-checks
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@@ -1,270 +0,0 @@
|
|||||||
# orchestrator-host.nix — the host contract that nix/modules/cc-ci.nix (the orchestrator's
|
|
||||||
# loops/timers) silently assumes, made explicit and reusable: the `loops` user the agents run as,
|
|
||||||
# the standalone opencode CLI, the shared opencode web server and its basic-auth web UI,
|
|
||||||
# nix-ld so foreign binaries run on NixOS, and the tool set agents reach for.
|
|
||||||
#
|
|
||||||
# Exported from flake.nix as `nixosModules.orchestrator-host`. A host imports this together with
|
|
||||||
# `nixosModules.cc-ci-orchestrator`; the combined CI-server + orchestrator host (`#cc-ci`) also
|
|
||||||
# imports recipe-maintainers/cc-ci's `nixosModules.cc-ci-server`.
|
|
||||||
#
|
|
||||||
# History: until 2026-09 this lived (twice, drifting) in nix/hosts/cc-ci-orchestrator-hetzner/
|
|
||||||
# configuration.nix here and in notplants-nix's hosts/notplants-orchestrator/configuration.nix,
|
|
||||||
# the shared agent box that also ran lichen + project-orchestrator. The cc-ci half moved to its
|
|
||||||
# own host; this file is that half.
|
|
||||||
{ config, lib, pkgs, ... }:
|
|
||||||
let
|
|
||||||
cfg = config.cc-ci-orchestrator;
|
|
||||||
in
|
|
||||||
{
|
|
||||||
options.cc-ci-orchestrator = {
|
|
||||||
ciSshHost = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
default = "127.0.0.1";
|
|
||||||
example = "100.95.31.88";
|
|
||||||
description = ''
|
|
||||||
Where `ssh cc-ci` (used by every skill and script that drives the CI server) connects to,
|
|
||||||
as root with ~loops/.ssh/cc-ci-local-ed25519 (a key generated ON the host — nothing
|
|
||||||
copied from another machine). On the combined host the CI server IS this
|
|
||||||
machine, so the default is loopback; a standalone orchestrator points it at the CI
|
|
||||||
server's tailnet address.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
opencodeUiBackendPort = lib.mkOption {
|
|
||||||
type = lib.types.port;
|
|
||||||
default = 8090; # not 8080: acme-dns's local API has it on the combined host
|
|
||||||
description = ''
|
|
||||||
Plain-HTTP port nginx listens on for the opencode UI, reachable ONLY from the docker
|
|
||||||
bridge (firewall rule on docker_gwbridge). Traefik — which owns the public 443 on the
|
|
||||||
combined host — terminates TLS for opencodeUiHost and forwards here; nginx adds the basic
|
|
||||||
auth and logs failures for fail2ban with the real client IP.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
opencodeUiTraefikNetwork = lib.mkOption {
|
|
||||||
type = lib.types.nullOr lib.types.str;
|
|
||||||
default = null;
|
|
||||||
example = "proxy";
|
|
||||||
description = ''
|
|
||||||
Name of the swarm overlay network the cc-ci traefik watches (cc-ci's swarm.nix creates
|
|
||||||
`proxy`). When set, a one-container swarm stack `ccci-opencode-ui` (a socat TCP relay to
|
|
||||||
nginx on the docker bridge) is deployed with traefik labels routing opencodeUiHost on
|
|
||||||
the `web-secure` entrypoint — the same label mechanism every cc-ci service and recipe
|
|
||||||
uses, so it coexists with the traefik recipe's own file provider (the wildcard cert and
|
|
||||||
the `security` middleware live there; switching traefik to a file *directory* replaces
|
|
||||||
that file and takes every front door down — learned 2026-09-07). null = no route.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
opencodeUiHost = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
default = "oc.commoninternet.net";
|
|
||||||
description = "nginx server_name for the opencode web UI (TLS + basic auth).";
|
|
||||||
};
|
|
||||||
|
|
||||||
opencodeUiHtpasswdFile = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
default = "/secrets/nginx/oc-htpasswd";
|
|
||||||
description = ''
|
|
||||||
htpasswd file for the opencode UI (`oc:<bcrypt>`), created out of band — a store path
|
|
||||||
would be world-readable. Default is under /secrets, the authoritative location for this
|
|
||||||
host's secrets; it must be readable by the `nginx` user (root:nginx 0640 in a directory
|
|
||||||
nginx can traverse). **nginx refuses to start if it is missing**, and its config check
|
|
||||||
runs as the nginx user, so a root-only file fails the check even though the path exists.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
};
|
|
||||||
|
|
||||||
config = {
|
|
||||||
# ---- the loops user -------------------------------------------------------------------
|
|
||||||
# claude sessions run as non-root (--dangerously-skip-permissions is refused for root).
|
|
||||||
users.users.loops = {
|
|
||||||
isNormalUser = true;
|
|
||||||
uid = 1000; # fixed: workspace files are rsynced between hosts by uid
|
|
||||||
home = "/home/loops";
|
|
||||||
shell = pkgs.bash;
|
|
||||||
extraGroups = [ "wheel" "docker" ];
|
|
||||||
};
|
|
||||||
security.sudo.wheelNeedsPassword = false;
|
|
||||||
security.sudo.extraRules = [{
|
|
||||||
users = [ "loops" ];
|
|
||||||
commands = [{ command = "ALL"; options = [ "NOPASSWD" ]; }];
|
|
||||||
}];
|
|
||||||
|
|
||||||
# /home/loops/.local/bin holds the standalone opencode binary; it must be first on every PATH
|
|
||||||
# (interactive shells, tmux, the systemd units in cc-ci.nix prepend it too).
|
|
||||||
environment.variables.PATH = lib.mkForce
|
|
||||||
"/home/loops/.local/bin:/run/current-system/sw/bin:/run/wrappers/bin:/usr/bin:/bin";
|
|
||||||
|
|
||||||
# ---- nix-ld: the standalone opencode CLI is a foreign dynamic ELF binary --------------------
|
|
||||||
programs.nix-ld.enable = true;
|
|
||||||
programs.nix-ld.libraries = with pkgs; [ stdenv.cc.cc.lib zlib openssl curl glibc ];
|
|
||||||
|
|
||||||
# ---- the toolbox every agent (and every operator ssh session) on this box gets ----------
|
|
||||||
# Bar for adding something: an agent or an operator doing ordinary work would otherwise waste
|
|
||||||
# a turn discovering it is absent. Installed system-wide, so it is on PATH for BOTH root and
|
|
||||||
# loops via /run/current-system/sw/bin (which the forced PATH below keeps for every user).
|
|
||||||
environment.systemPackages = with pkgs; [
|
|
||||||
git tmux python3 jq curl cacert
|
|
||||||
gnused gawk coreutils gnugrep findutils util-linux nettools openssh
|
|
||||||
age sops ssh-to-age
|
|
||||||
wget gnutar gzip unzip zip xz
|
|
||||||
ripgrep fd tree file less which
|
|
||||||
procps psmisc htop lsof strace ncdu
|
|
||||||
dnsutils socat netcat-gnu iproute2 iputils
|
|
||||||
openssl gnumake gcc pkg-config
|
|
||||||
yq-go diffutils patch rsync bubblewrap
|
|
||||||
# Editors: `vim` ships `vi` too, so both names resolve for anyone who types either.
|
|
||||||
# `nano` comes from the base system. EDITOR is set below so git/systemctl/visudo agree.
|
|
||||||
vim nano
|
|
||||||
# `sqlite` is not a nicety here: Drone's build/step logs live in its sqlite volume and the
|
|
||||||
# runbook (and /cc-ci-status) tell you to read them there; without it every such check
|
|
||||||
# needed an ad-hoc `nix-shell -p sqlite`.
|
|
||||||
sqlite
|
|
||||||
bat bc moreutils pv man-pages
|
|
||||||
];
|
|
||||||
|
|
||||||
# So `git commit`, `systemctl edit`, `visudo` etc. open something that exists on this host.
|
|
||||||
environment.variables.EDITOR = "vim";
|
|
||||||
|
|
||||||
# ---- ssh config for the loops user: `ssh cc-ci` = the CI server (root) -----------------
|
|
||||||
# Written only if absent so a manual customisation survives rebuilds.
|
|
||||||
system.activationScripts.loopsSshConfig = ''
|
|
||||||
mkdir -p /home/loops/.ssh && chown loops:users /home/loops/.ssh && chmod 700 /home/loops/.ssh
|
|
||||||
if [ ! -f /home/loops/.ssh/config ]; then
|
|
||||||
cat > /home/loops/.ssh/config <<'SSHCFG'
|
|
||||||
Host cc-ci
|
|
||||||
HostName ${cfg.ciSshHost}
|
|
||||||
User root
|
|
||||||
IdentityFile /home/loops/.ssh/cc-ci-local-ed25519
|
|
||||||
IdentitiesOnly yes
|
|
||||||
StrictHostKeyChecking accept-new
|
|
||||||
ServerAliveInterval 30
|
|
||||||
|
|
||||||
Host git.autonomic.zone
|
|
||||||
HostName git.autonomic.zone
|
|
||||||
Port 2222
|
|
||||||
User git
|
|
||||||
IdentityFile /home/loops/.ssh/autonomic-bot-cc-ci-ed25519
|
|
||||||
IdentitiesOnly yes
|
|
||||||
SSHCFG
|
|
||||||
chmod 600 /home/loops/.ssh/config
|
|
||||||
chown loops:users /home/loops/.ssh/config
|
|
||||||
fi
|
|
||||||
'';
|
|
||||||
|
|
||||||
# ---- standalone CLIs (idempotent installers; re-run on every activation, no-op if present) --
|
|
||||||
# No Claude Code on this host (operator 2026-09-07): the orchestrator and the weekly upgrader
|
|
||||||
# are opencode agents; Claude sessions run on the notplants-orchestrator box and reach this
|
|
||||||
# host over ssh.
|
|
||||||
systemd.services.opencode-install = {
|
|
||||||
description = "Install opencode CLI for loops user (idempotent)";
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
|
||||||
after = [ "network-online.target" ];
|
|
||||||
wants = [ "network-online.target" ];
|
|
||||||
serviceConfig = { Type = "oneshot"; RemainAfterExit = true; User = "loops"; Group = "users"; };
|
|
||||||
environment = { HOME = "/home/loops"; };
|
|
||||||
path = [ pkgs.curl pkgs.bash pkgs.coreutils pkgs.gnutar pkgs.gzip pkgs.unzip ];
|
|
||||||
script = ''
|
|
||||||
if [ ! -x "$HOME/.local/bin/opencode" ]; then
|
|
||||||
echo "installing opencode CLI for loops user..."
|
|
||||||
curl -fsSL https://opencode.ai/install | bash || echo "install failed — retry on next activation"
|
|
||||||
# The installer puts the binary in ~/.opencode/bin; every unit here expects ~/.local/bin.
|
|
||||||
if [ -x "$HOME/.opencode/bin/opencode" ]; then
|
|
||||||
mkdir -p "$HOME/.local/bin" && ln -sfn "$HOME/.opencode/bin/opencode" "$HOME/.local/bin/opencode"
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
# ---- opencode web server: one shared instance the opencode-backed agents attach to -------
|
|
||||||
# Provider creds come from /srv/cc-ci/.testenv (out of band, see README).
|
|
||||||
systemd.services.opencode-web = {
|
|
||||||
description = "opencode web server for cc-ci agents";
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
|
||||||
after = [ "network-online.target" "opencode-install.service" ];
|
|
||||||
wants = [ "network-online.target" ];
|
|
||||||
serviceConfig = {
|
|
||||||
Type = "simple";
|
|
||||||
User = "loops"; Group = "users";
|
|
||||||
WorkingDirectory = "/srv/cc-ci-orch/cc-ci";
|
|
||||||
EnvironmentFile = [ "-/srv/cc-ci/cc-ci/.env.public" "/srv/cc-ci/.testenv" ];
|
|
||||||
ExecStartPre = "${pkgs.coreutils}/bin/rm -rf /tmp/opencode";
|
|
||||||
ExecStart = "/home/loops/.local/bin/opencode serve --hostname 127.0.0.1 --port 4096";
|
|
||||||
Restart = "on-failure";
|
|
||||||
RestartSec = "5s";
|
|
||||||
};
|
|
||||||
environment = {
|
|
||||||
HOME = "/home/loops";
|
|
||||||
PATH = lib.mkForce "/run/wrappers/bin:/home/loops/.local/bin:/run/current-system/sw/bin:/usr/bin:/bin:/etc/profiles/per-user/loops/bin:/nix/var/nix/profiles/default/bin";
|
|
||||||
};
|
|
||||||
path = [ pkgs.bash pkgs.coreutils pkgs.git pkgs.python3 pkgs.openssh pkgs.tmux pkgs.nettools ];
|
|
||||||
};
|
|
||||||
|
|
||||||
# ---- nginx: basic auth for the opencode UI, behind traefik -----------------------------
|
|
||||||
# Traefik (public 443, the CI wildcard cert) routes opencodeUiHost to this plain-HTTP vhost
|
|
||||||
# on the docker bridge address. nginx enforces HTTP basic auth (the opencode web UI has no
|
|
||||||
# auth of its own and can drive agent sessions), and — via real_ip from traefik's
|
|
||||||
# X-Forwarded-For — logs the CLIENT address on a 401, which is what the fail2ban jail bans.
|
|
||||||
# The htpasswd is created out of band (a store path would be world readable); nginx FAILS TO
|
|
||||||
# START without it, and its config check runs as the nginx user:
|
|
||||||
# /etc/nginx/oc-htpasswd root:nginx 0640 (`oc:<bcrypt>`; plaintext kept in /secrets)
|
|
||||||
# Rotate with: printf 'oc:%s\n' "$(mkpasswd -m bcrypt "$P")" > /etc/nginx/oc-htpasswd && systemctl reload nginx
|
|
||||||
services.nginx = {
|
|
||||||
enable = true;
|
|
||||||
recommendedProxySettings = true;
|
|
||||||
virtualHosts.${cfg.opencodeUiHost} = {
|
|
||||||
listen = [ { addr = "0.0.0.0"; port = cfg.opencodeUiBackendPort; } ];
|
|
||||||
basicAuthFile = cfg.opencodeUiHtpasswdFile;
|
|
||||||
extraConfig = ''
|
|
||||||
# traefik sits on the docker networks (ingress 10.0.0.0/24, gwbridge 172.18.0.0/16)
|
|
||||||
set_real_ip_from 172.16.0.0/12;
|
|
||||||
set_real_ip_from 10.0.0.0/8;
|
|
||||||
real_ip_header X-Forwarded-For;
|
|
||||||
'';
|
|
||||||
locations."/" = {
|
|
||||||
proxyPass = "http://127.0.0.1:4096";
|
|
||||||
proxyWebsockets = true;
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
|
||||||
# Only docker's bridge may reach the plain-HTTP backend; the public interface stays closed.
|
|
||||||
networking.firewall.interfaces.docker_gwbridge.allowedTCPPorts = [ cfg.opencodeUiBackendPort ];
|
|
||||||
|
|
||||||
# The traefik side of the route: a swarm service carrying the router labels. Named ccci-* so
|
|
||||||
# the weekly run's orphan sweep (skills/upgrade-all/sweep-orphans.sh keep-list) leaves it be. A plain
|
|
||||||
# TCP relay (socat) from the overlay network to nginx on the docker bridge; traefik's
|
|
||||||
# X-Forwarded-For passes through untouched, which is what nginx's real_ip reads.
|
|
||||||
systemd.services.opencode-ui-route = lib.mkIf (cfg.opencodeUiTraefikNetwork != null) {
|
|
||||||
description = "swarm stack ccci-opencode-ui: traefik labels ${cfg.opencodeUiHost} -> nginx basic auth";
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
|
||||||
after = [ "deploy-proxy.service" "docker.service" "nginx.service" ];
|
|
||||||
wants = [ "deploy-proxy.service" ];
|
|
||||||
path = [ pkgs.docker ];
|
|
||||||
serviceConfig = { Type = "oneshot"; RemainAfterExit = true; };
|
|
||||||
script = ''
|
|
||||||
docker stack deploy --detach=true -c ${pkgs.writeText "opencode-ui-stack.yml" ''
|
|
||||||
# Deployed by opencode-ui-route.service (nix/modules/orchestrator-host.nix). Do not edit.
|
|
||||||
version: "3.8"
|
|
||||||
services:
|
|
||||||
relay:
|
|
||||||
image: alpine/socat:1.8.0.3
|
|
||||||
command: ["TCP-LISTEN:${toString cfg.opencodeUiBackendPort},fork,reuseaddr", "TCP:172.18.0.1:${toString cfg.opencodeUiBackendPort}"]
|
|
||||||
networks: [ ${cfg.opencodeUiTraefikNetwork} ]
|
|
||||||
deploy:
|
|
||||||
replicas: 1
|
|
||||||
labels:
|
|
||||||
- "traefik.enable=true"
|
|
||||||
- "traefik.http.routers.opencode-ui.rule=Host(`${cfg.opencodeUiHost}`)"
|
|
||||||
- "traefik.http.routers.opencode-ui.entrypoints=web-secure"
|
|
||||||
- "traefik.http.routers.opencode-ui.tls=true"
|
|
||||||
- "traefik.http.services.opencode-ui.loadbalancer.server.port=${toString cfg.opencodeUiBackendPort}"
|
|
||||||
networks:
|
|
||||||
${cfg.opencodeUiTraefikNetwork}:
|
|
||||||
external: true
|
|
||||||
''} ccci-opencode-ui
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
Reference in New Issue
Block a user