Compare commits
97
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4bc483326b | ||
|
|
f105808bed | ||
|
|
31af820079 | ||
|
|
5bebd13e49 | ||
|
|
82d3127c2a | ||
|
|
d824800f8a | ||
|
|
e7f861ec9e | ||
|
|
1b75d986b2 | ||
|
|
e72fbdb79c | ||
|
|
f6c6b39b18 | ||
|
|
02ff90777f | ||
|
|
012212498d | ||
|
|
521f6e6934 | ||
|
|
07bc0a15d5 | ||
|
|
00a4ee6752 | ||
|
|
9c619abd6c | ||
|
|
980530f505 | ||
|
|
3047ee7724 | ||
|
|
c472aa3642 | ||
|
|
7252f7ee35 | ||
|
|
75386fc23b | ||
|
|
19abebb312 | ||
|
|
565e021185 | ||
|
|
929faeb055 | ||
|
|
a29b993160 | ||
|
|
a0d6fc9417 | ||
|
|
9409adffb8 | ||
|
|
69d1840ea5 | ||
|
|
ab5d11ea2a | ||
|
|
74117c2260 | ||
|
|
985dc06e47 | ||
|
|
4b9978ac02 | ||
|
|
46ace30b4d | ||
|
|
dab3edf3c2 | ||
|
|
db37f1618b | ||
|
|
4bad1ea6db | ||
|
|
ef58e33102 | ||
|
|
c352ea9058 | ||
|
|
fae2fbe21b | ||
|
|
6ebc35bc18 | ||
|
|
96c536f543 | ||
|
|
94ce5c4de2 | ||
|
|
bb7ebb4a27 | ||
|
|
ecf126d98d | ||
|
|
ab88e59c21 | ||
|
|
e89da2d842 | ||
|
|
6c91373357 | ||
|
|
1db85a7e77 | ||
|
|
8df32edfcf | ||
|
|
18caf047bf | ||
|
|
b0bdce2c15 | ||
|
|
44cb9b6704 | ||
|
|
46c4fff1a6 | ||
|
|
65bf3c095b | ||
|
|
8d7320f32e | ||
|
|
b5f8543a9b | ||
|
|
78ae2be8ae | ||
|
|
98a624a13a | ||
|
|
fc36d0e10f | ||
|
|
1daf0fa616 | ||
|
|
318d09bdab | ||
|
|
3e59924450 | ||
|
|
3307bdb0fe | ||
|
|
cf26ef863a | ||
|
|
5775fe23f8 | ||
|
|
02dbd71b49 | ||
|
|
8f85a238cc | ||
|
|
80008da80d | ||
|
|
242a6d9659 | ||
|
|
d441c6caaf | ||
|
|
d101147b93 | ||
|
|
fb1dc7af9c | ||
|
|
0347511a84 | ||
|
|
f8888b2082 | ||
|
|
a56734de0b | ||
|
|
5424954b3f | ||
|
|
51b067770c | ||
|
|
34d62fa049 | ||
|
|
995bcf82d7 | ||
|
|
04a04e51ac | ||
|
|
f750622e3d | ||
|
|
0b6cc632d4 | ||
|
|
e8d7d09445 | ||
|
|
b2063b8235 | ||
|
|
0d37a891f7 | ||
|
|
cb20bea7cd | ||
|
|
be7f8bc850 | ||
|
|
15e4e75681 | ||
|
|
49854472b8 | ||
|
|
b462f1f7f1 | ||
|
|
02cc2c29e2 | ||
|
|
388e7f38c9 | ||
|
|
5ade783a50 | ||
|
|
91179f872c | ||
|
|
6b3a3b1934 | ||
|
|
74a57d37b3 | ||
|
|
bab6481171 |
@@ -0,0 +1,97 @@
|
||||
---
|
||||
name: cc-ci-cleanup
|
||||
description: Tidy the fleet's open recipe PRs. Reconciles every mirror from TRUE upstream first (which alone closes PRs upstream already merged), then surveys every open PR deterministically, CLOSES the ones that can no longer be merged or were never meant to be (CI sweep artifacts, obsolete bumps, superseded duplicates) with a reason, and reports prioritised action items for the ones that SHOULD merge — what specifically is blocking each. NEVER merges a recipe PR. Invoke as /cc-ci-cleanup [recipe ...] [--dry-run].
|
||||
---
|
||||
|
||||
# cc-ci-cleanup
|
||||
|
||||
Open recipe PRs accumulate and rot. Some were never meant to merge (CI sweep artifacts), some were
|
||||
overtaken (upstream merged the same change, or a newer PR supersedes them), and some genuinely should
|
||||
land but are quietly blocked. Left alone the list becomes noise, and a real CVE fix hides in it.
|
||||
|
||||
This skill separates those three, acts on the first two, and hands you a short list for the third.
|
||||
|
||||
**Boundaries.** It **CLOSES** irrelevant PRs and **NEVER MERGES** any recipe PR — those change what
|
||||
deploys on other people's infrastructure, so a human merges them (see AGENTS.md). Closing is the only
|
||||
write it performs, always with a comment saying why.
|
||||
|
||||
## Arguments
|
||||
- `<recipe> …` — limit to these recipes (else every recipe in `cc-ci-plan/used-recipes.md`).
|
||||
- `--dry-run` — classify and report, close nothing.
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Reconcile every mirror from TRUE upstream — MANDATORY, FIRST
|
||||
```
|
||||
cc-ci-plan/reconcile-upstream.sh --all # or: reconcile-upstream.sh <recipe>...
|
||||
```
|
||||
**Do not skip this and do not reorder it.** Every signal in step 2 is measured against the mirror's
|
||||
`main`; against a stale mirror they are all wrong. This step also does a chunk of the cleanup by
|
||||
itself — it closes any PR whose changes upstream has already merged.
|
||||
|
||||
> On the first real run (2026-08-11) this alone closed **three** PRs that looked pending and were
|
||||
> already merged upstream: discourse #6 (carrying **140 CVEs**), keycloak #6 (**12 CVEs**), n8n #5.
|
||||
> All three had been reported to the operator as outstanding work. mailu #6 went the same way earlier
|
||||
> the same day. Reconciling is not hygiene, it is how you avoid recommending work that is already done.
|
||||
|
||||
### 2. Survey every open PR (deterministic — no judgement yet)
|
||||
```
|
||||
python3 cc-ci-plan/pr-survey.py [recipe ...] # add --json for the raw facts
|
||||
```
|
||||
Per PR it measures: `behind_main`, `ahead`, `mergeable`, `diff_files`, the images it **adds**, which
|
||||
of those are **already in main**, `obsolete`, the newest `!testme` verdict + build, `branch_kind`,
|
||||
and age/idle days. It decides nothing — that is this skill's job.
|
||||
|
||||
### 3. Classify
|
||||
|
||||
**CLOSE — cannot merge, or was never meant to.** Each needs a *positive* reason, not an absence:
|
||||
|
||||
| signal | why it is closeable |
|
||||
|---|---|
|
||||
| `branch_kind: ci-artifact` (`ci/*`) | regall/cfold sweeps and `!testme` probes — harness artifacts, never intended to merge |
|
||||
| `obsolete: true` | every image it adds is **already pinned in main** — it has nothing left to contribute |
|
||||
| superseded | a newer PR on the same recipe makes the same bump (name both numbers in the comment) |
|
||||
| `diff_files: 0` | genuinely empty diff — nothing to merge |
|
||||
|
||||
**NEVER close on:**
|
||||
- `DIFF-UNREADABLE` — the diff could not be fetched, which is NOT an empty diff. gitea #4 reads that
|
||||
way (force-pushed branch) while being a verified, green, needed fix.
|
||||
- any field that came back `null`/unknown.
|
||||
- a PR that carries a **CVE fix** and is the only thing carrying it, even if it looks stale — report it
|
||||
instead. Losing a security fix to tidiness is far worse than a long PR list.
|
||||
- `--dry-run`.
|
||||
|
||||
**NEEDS WORK — should merge, something blocks it.** Give the *specific* next action:
|
||||
| signal | action item |
|
||||
|---|---|
|
||||
| `mergeable: false` | conflicts — rebase the branch on `main` and re-run `!testme` |
|
||||
| `behind_main > 0` | out of date — rebase, then re-verify (a green from before main moved proves nothing) |
|
||||
| `ci: failed` | diagnose via `/ci-test-review`; classify recipe-bug vs stale test |
|
||||
| `ci: never-run` | run `!testme` |
|
||||
| blocked on the operator | say exactly what is needed (a secret, an upstream release, a decision) |
|
||||
|
||||
**READY — green, current, no conflicts.** Action item is simply: review and merge.
|
||||
|
||||
### 4. Close the CLOSE set (skip entirely under `--dry-run`)
|
||||
Comment first, then close. The comment must say **which signal** made it closeable and **what to do
|
||||
if that is wrong** ("reopen if …"), so a wrong call is cheap to undo. Never close silently.
|
||||
|
||||
### 5. Report
|
||||
Order by what deserves attention, not by recipe name:
|
||||
|
||||
1. **CVE-carrying PRs that should merge** — most severe first, with the CVE ids.
|
||||
2. Other **READY** PRs (green + current).
|
||||
3. **NEEDS WORK**, each with its one specific action.
|
||||
4. **CLOSED this run**, with the reason for each.
|
||||
5. Anything **deliberately left alone** despite looking stale, and why.
|
||||
|
||||
End with a one-line summary: `N open → C closed, R ready to merge, W need work`.
|
||||
|
||||
## Guardrails
|
||||
- **Never merge a recipe PR.** Create/verify/close only; the operator merges.
|
||||
- **Reconcile first, always.** Judging a PR against a stale mirror is how you close good work or
|
||||
recommend work that is already done.
|
||||
- **Close only on a positive signal**, never on "looks old". Age alone is not a reason — several
|
||||
60-day-old PRs here are green and mergeable.
|
||||
- **Never close a lone CVE fix.** Report it, however stale.
|
||||
- Every close gets a comment with its reason and a reopen hint.
|
||||
@@ -0,0 +1,14 @@
|
||||
---
|
||||
name: cc-ci-status
|
||||
description: Comprehensive read-only status check of the whole cc-ci system - how the recent weekly upgrade runs went and whether their reports published, which recipes/tests are stale, how long since the server + orchestrator host flake updates, open recipe PRs (flagging CVE-carrying PRs that have been open too long), host health (failed units, disk, timers, bridge/!testme path), ending with a verdict (ALL HEALTHY or a findings list) and recommended next steps mapped to the skills to invoke. Never changes anything - it only reads and reports. Invoke as /cc-ci-status.
|
||||
---
|
||||
|
||||
# cc-ci-status (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cc-ci-status/SKILL.md`**
|
||||
|
||||
Read that file for the full procedure. This `.claude/skills/` copy is kept as a
|
||||
thin pointer for Claude Code compatibility; opencode loads the canonical
|
||||
definition from `.opencode/skills/` directly.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-init-instance
|
||||
description: "[recipe-maintainer/cctest] Deploy all maintained recipes to the active test instance from scratch (Wraps the autonomic-recipe-maintainer skill /init-instance; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-init-instance.)"
|
||||
---
|
||||
|
||||
# cctest-init-instance (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-init-instance/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/init-instance/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-intro
|
||||
description: "[recipe-maintainer/cctest] Explain what this project is and how to get started (Wraps the autonomic-recipe-maintainer skill /intro; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-intro.)"
|
||||
---
|
||||
|
||||
# cctest-intro (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-intro/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/intro/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-new-recipe-guide
|
||||
description: "[recipe-maintainer/cctest] Guide for developing a new Co-op Cloud recipe from scratch (Wraps the autonomic-recipe-maintainer skill /new-recipe-guide; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-new-recipe-guide.)"
|
||||
---
|
||||
|
||||
# cctest-new-recipe-guide (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-new-recipe-guide/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/new-recipe-guide/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-opencode-sync
|
||||
description: "[recipe-maintainer/cctest] Ensure every Claude skill has a corresponding OpenCode skill alias (Wraps the autonomic-recipe-maintainer skill /opencode-sync; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-opencode-sync.)"
|
||||
---
|
||||
|
||||
# cctest-opencode-sync (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-opencode-sync/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/opencode-sync/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-recipe-check
|
||||
description: "[recipe-maintainer/cctest] Fetch a Co-op Cloud recipe and check for available upgrades (Wraps the autonomic-recipe-maintainer skill /recipe-check; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-check.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-check (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-recipe-check/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/recipe-check/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-recipe-create-pr
|
||||
description: "[recipe-maintainer/cctest] Push local recipe commits to git.autonomic.zone and open a PR against an upstream-synced main branch (Wraps the autonomic-recipe-maintainer skill /recipe-create-pr; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-create-pr.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-create-pr (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-recipe-create-pr/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/recipe-create-pr/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-recipe-deploy
|
||||
description: "[recipe-maintainer/cctest] Deploy the local recipe checkout to the test instance (Wraps the autonomic-recipe-maintainer skill /recipe-deploy; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-deploy.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-deploy (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-recipe-deploy/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/recipe-deploy/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-recipe-guidelines
|
||||
description: "[recipe-maintainer/cctest] Guidelines for all recipe operations including local change preservation, version format, and secrets (Wraps the autonomic-recipe-maintainer skill /recipe-guidelines; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-guidelines.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-guidelines (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-recipe-guidelines/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/recipe-guidelines/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-recipe-init
|
||||
description: "[recipe-maintainer/cctest] Create a new test instance and recipe-info for a recipe (Wraps the autonomic-recipe-maintainer skill /recipe-init; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-init.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-init (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-recipe-init/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/recipe-init/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-recipe-logging
|
||||
description: "[recipe-maintainer/cctest] Logging instructions for maintaining detailed operation logs in the logs directory (Wraps the autonomic-recipe-maintainer skill /recipe-logging; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-logging.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-logging (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-recipe-logging/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/recipe-logging/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-recipe-new-tag
|
||||
description: "[recipe-maintainer/cctest] Bump the recipe version and create an annotated git tag (Wraps the autonomic-recipe-maintainer skill /recipe-new-tag; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-new-tag.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-new-tag (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-recipe-new-tag/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/recipe-new-tag/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-recipe-overview
|
||||
description: "[recipe-maintainer/cctest] Check all maintained recipes and recommend what to upgrade (Wraps the autonomic-recipe-maintainer skill /recipe-overview; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-overview.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-overview (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-recipe-overview/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/recipe-overview/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-recipe-review
|
||||
description: "[recipe-maintainer/cctest] Review a recipe for Co-op Cloud best practices (Wraps the autonomic-recipe-maintainer skill /recipe-review; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-review.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-review (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-recipe-review/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/recipe-review/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-recipe-test-all
|
||||
description: "[recipe-maintainer/cctest] Run tests for all maintained recipes, deploying each one at a time (Wraps the autonomic-recipe-maintainer skill /recipe-test-all; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-test-all.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-test-all (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-recipe-test-all/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/recipe-test-all/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-recipe-test-backup
|
||||
description: "[recipe-maintainer/cctest] Test backing up and restoring a recipe's test instance (Wraps the autonomic-recipe-maintainer skill /recipe-test-backup; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-test-backup.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-test-backup (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-recipe-test-backup/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/recipe-test-backup/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-recipe-test-new
|
||||
description: "[recipe-maintainer/cctest] Test a recipe's first-time initialization from scratch (Wraps the autonomic-recipe-maintainer skill /recipe-test-new; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-test-new.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-test-new (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-recipe-test-new/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/recipe-test-new/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-recipe-test-update
|
||||
description: "[recipe-maintainer/cctest] Test upgrading a recipe's test instance using abra app deploy (Wraps the autonomic-recipe-maintainer skill /recipe-test-update; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-test-update.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-test-update (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-recipe-test-update/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/recipe-test-update/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-recipe-test
|
||||
description: "[recipe-maintainer/cctest] Run all tests for a Co-op Cloud recipe (Wraps the autonomic-recipe-maintainer skill /recipe-test; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-test.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-test (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-recipe-test/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/recipe-test/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-recipe-upgrade-apply
|
||||
description: "[recipe-maintainer/cctest] Execute a planned recipe upgrade — apply changes, deploy, test, commit/tag (Wraps the autonomic-recipe-maintainer skill /recipe-upgrade-apply; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-upgrade-apply.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-upgrade-apply (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-recipe-upgrade-apply/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/recipe-upgrade-apply/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-recipe-upgrade-cron-all
|
||||
description: "[recipe-maintainer/cctest] Autonomous weekly upgrade run — overview all recipes, upgrade each end-to-end (sequentially by default, parallel with --parallel), open PRs (Wraps the autonomic-recipe-maintainer skill /recipe-upgrade-cron-all; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-upgrade-cron-all.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-upgrade-cron-all (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-recipe-upgrade-cron-all/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/recipe-upgrade-cron-all/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-recipe-upgrade-full
|
||||
description: "[recipe-maintainer/cctest] Plan and apply a recipe upgrade end-to-end, no human review in the middle (Wraps the autonomic-recipe-maintainer skill /recipe-upgrade-full; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-upgrade-full.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-upgrade-full (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-recipe-upgrade-full/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/recipe-upgrade-full/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-recipe-upgrade-plan
|
||||
description: "[recipe-maintainer/cctest] Create a detailed upgrade plan for a recipe (Wraps the autonomic-recipe-maintainer skill /recipe-upgrade-plan; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-upgrade-plan.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-upgrade-plan (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-recipe-upgrade-plan/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/recipe-upgrade-plan/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-recipe-upstream
|
||||
description: "[recipe-maintainer/cctest] From a git.autonomic.zone review-PR URL, fetch the branch + tag locally and emit the commands to open the upstream PR on git.coopcloud.tech (Wraps the autonomic-recipe-maintainer skill /recipe-upstream; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-upstream.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-upstream (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-recipe-upstream/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/recipe-upstream/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-setup-sandbox
|
||||
description: "[recipe-maintainer/cctest] Guide for setting up a sandboxed environment to run the agent with recipe-maintainer (Wraps the autonomic-recipe-maintainer skill /setup-sandbox; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-setup-sandbox.)"
|
||||
---
|
||||
|
||||
# cctest-setup-sandbox (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-setup-sandbox/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/setup-sandbox/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-switch-default-instance
|
||||
description: "[recipe-maintainer/cctest] Switch the default test instance (b1cc or t1cc) for all recipe operations (Wraps the autonomic-recipe-maintainer skill /switch-default-instance; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-switch-default-instance.)"
|
||||
---
|
||||
|
||||
# cctest-switch-default-instance (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-switch-default-instance/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/switch-default-instance/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-sync-secrets
|
||||
description: "[recipe-maintainer/cctest] Sync Docker secrets from the test server into recipe-info/testsecrets/ (Wraps the autonomic-recipe-maintainer skill /sync-secrets; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-sync-secrets.)"
|
||||
---
|
||||
|
||||
# cctest-sync-secrets (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-sync-secrets/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/sync-secrets/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-t1cc-start
|
||||
description: "[recipe-maintainer/cctest] Provision the t1cc DigitalOcean test server and deploy Traefik (Wraps the autonomic-recipe-maintainer skill /t1cc-start; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-t1cc-start.)"
|
||||
---
|
||||
|
||||
# cctest-t1cc-start (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-t1cc-start/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/t1cc-start/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-t1cc-stop
|
||||
description: "[recipe-maintainer/cctest] Destroy the t1cc DigitalOcean test server via terraform (Wraps the autonomic-recipe-maintainer skill /t1cc-stop; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-t1cc-stop.)"
|
||||
---
|
||||
|
||||
# cctest-t1cc-stop (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-t1cc-stop/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/t1cc-stop/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-test-context-reset
|
||||
description: "[recipe-maintainer/cctest] Undeploy all apps from the test server except traefik (Wraps the autonomic-recipe-maintainer skill /test-context-reset; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-test-context-reset.)"
|
||||
---
|
||||
|
||||
# cctest-test-context-reset (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-test-context-reset/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/test-context-reset/SKILL.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: cctest-test-setup
|
||||
description: "[recipe-maintainer/cctest] Verify the test environment is configured correctly (Wraps the autonomic-recipe-maintainer skill /test-setup; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-test-setup.)"
|
||||
---
|
||||
|
||||
# cctest-test-setup (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/cctest-test-setup/SKILL.md`**
|
||||
|
||||
Read that file. It in turn wraps the ARM submodule skill
|
||||
`references/recipe-maintainer/.opencode/skills/test-setup/SKILL.md`.
|
||||
@@ -79,12 +79,29 @@ For each real (non-flaky) finding, write the actual fix and open a PR. **Never m
|
||||
it handles the mirror to `git.autonomic.zone/recipe-maintainers/<recipe>` (upstream
|
||||
`git.coopcloud.tech`). Keep the change **bounded** to the diagnosed root cause; don't rewrite the
|
||||
recipe.
|
||||
- **Before editing any test, read `tests/STYLE.md` in the cc-ci repo.** It encodes the rules a test
|
||||
change must satisfy — set state up through the app's interface rather than its database, gate on
|
||||
version instead of branching, correct the fixture/wait but NEVER the assertion, and diagnose from
|
||||
the app's own telemetry before concluding a test is stale.
|
||||
- **CI-server-side fix → cc-ci PR.** Branch the cc-ci product repo
|
||||
(`recipe-maintainers/cc-ci`), apply the fix, and open the PR via the Gitea API (use the
|
||||
`GITEA_*` creds from `/srv/cc-ci/.testenv`). **Single-writer discipline:** work on a dedicated
|
||||
branch in a SEPARATE clone — **never push `main`, never touch the build loops' working clones**
|
||||
(`/cc-ci`, `/cc-ci-adv`) or their in-flight state.
|
||||
|
||||
> ### ⚠️ RECONCILE FROM UPSTREAM FIRST — always, before any PR work or upgrade check
|
||||
> ```
|
||||
> cc-ci-plan/reconcile-upstream.sh <recipe>... # or --all
|
||||
> ```
|
||||
> Deterministic, idempotent, and safe (recipe work lives in branches, never on mirror `main`). It
|
||||
> force-syncs each mirror to coopcloud's **default branch — resolved from the API, `main` OR
|
||||
> `master`** — and closes any mirror PR whose changes upstream already merged. Skipping it has cost
|
||||
> us three distinct ways: mailu #6 was reported as the fix for two internet-facing CVEs while
|
||||
> upstream had already merged AND released it; a stale mirror makes a survey report "no upgrades
|
||||
> available" so the recipe drops out of the weekly run; and reading the wrong branch on a recipe with
|
||||
> a stale `main` beside a live `master` (gitea) manufactures a false "three releases behind, missing
|
||||
> two CVSS-9.8 RCEs" finding.
|
||||
|
||||
### 5. VERIFY each PR on the CI server (deterministic; still never merge)
|
||||
A PR is only "working" once **cc-ci verifies it green** (operator rule) — dogfood the CI that found
|
||||
the bug. Verification is deterministic (the harness), not an AI judgement.
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
---
|
||||
name: cve-check-and-upgrade
|
||||
description: Security-driven upgrade run. Does a full /cve-check sweep first (per-image advisory scan of every recipe's available upgrade, with adjudication), then runs /recipe-upgrade ONLY on the recipes whose upgrade fixes at least one CVE — worst severity first — opening a verified recipe PR for each, and finally publishes one report covering both the sweep and the PRs. Recipes with no CVEs are left alone; that is the point. NEVER merges. Invoke as /cve-check-and-upgrade [recipe ...] [--min-severity high] [--capacity N] [--dry-run].
|
||||
---
|
||||
|
||||
# cve-check-and-upgrade
|
||||
|
||||
`/upgrade-all` upgrades everything that *has* an upgrade. **This upgrades what has a reason.** It runs
|
||||
the `/cve-check` sweep, then spends CI time only on the recipes where an upgrade actually closes a
|
||||
vulnerability, handling the worst first.
|
||||
|
||||
Use it when you want to act on security rather than churn the whole fleet: after a vendor announcement,
|
||||
when CI capacity is short, or between weekly runs. When you only want to *know*, use `/cve-check`. When
|
||||
you want everything current regardless of CVEs, use `/upgrade-all`.
|
||||
|
||||
**Creates PRs. Never merges.** Every PR is verified green on cc-ci and left for a human.
|
||||
|
||||
## Arguments
|
||||
- `<recipe> …` — restrict the whole run to these recipes.
|
||||
- `--min-severity critical|high|medium|low` — only upgrade recipes whose fixed CVEs reach this
|
||||
severity. Default **`low`** (any CVE at all justifies the upgrade). `--min-severity high` is the
|
||||
useful "just the urgent ones" setting.
|
||||
- `--capacity N` — subagent pool size; defaults to the live `DRONE_RUNNER_CAPACITY` (the drone
|
||||
runner's slots), matching `/upgrade-all`'s rolling-pool behaviour.
|
||||
- `--dry-run` — do the whole sweep and print exactly which recipes *would* be upgraded and why, then
|
||||
stop without spawning a single upgrade. **Publishes no report and opens no PR.**
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Sweep — run `/cve-check` in full
|
||||
Follow `.claude/skills/cve-check/SKILL.md` steps 1–5 exactly: candidate list, per-image upgrade windows,
|
||||
the advisory scan per recipe, pass-2 adjudication of anything undecided, and the severity classification.
|
||||
**Do not publish its report** — this run produces one combined report at the end instead.
|
||||
|
||||
Keep, per recipe: the windows scanned, the CVE count, the CVE ids with severities, and whether the count
|
||||
is a floor (undetermined advisories remain) or unknown.
|
||||
|
||||
### 2. Decide what to upgrade
|
||||
`RECIPES_TO_UPGRADE` = recipes where **the scan found ≥1 CVE** at or above `--min-severity`.
|
||||
|
||||
Deliberate exclusions, each recorded in the report with its reason:
|
||||
- **0 CVEs** — an upgrade may exist, but nothing security-relevant. Left alone; that is the point of
|
||||
this skill. `/upgrade-all` is what sweeps those up.
|
||||
- **`external` tier** — swept for visibility, **never upgraded here**; someone else maintains it. Flag
|
||||
it loudly in the report if it has a critical, since the action is to tell them, not to open a PR.
|
||||
- **`UPTODATE`** — nothing available.
|
||||
- **count `?` / UNKNOWN** — do **not** upgrade blind, and do **not** treat it as clean. Put it in the
|
||||
Addendum as needing a look. An unknown is a gap in our knowledge, not evidence of safety.
|
||||
|
||||
Order the queue by **worst severity first** (critical → high → …), count breaking ties. If `--dry-run`,
|
||||
print this queue with each recipe's CVE ids and severities, and STOP here.
|
||||
|
||||
### 3. Upgrade each one — via `/recipe-upgrade` subagents
|
||||
Run `/recipe-upgrade <recipe>` per queued recipe as a **subagent**, in the queue order above, as a
|
||||
**rolling pool** keeping `--capacity` (default `DRONE_RUNNER_CAPACITY`) running at once and starting the
|
||||
next as each finishes — the same concurrency discipline as `/upgrade-all` §3, and safe for the same
|
||||
reason (per-run recipe trees + app-domain locks).
|
||||
|
||||
Each subagent does the full job: plan, implement the bump, verify green on cc-ci with `!testme`, and
|
||||
open a recipe PR. **Default mode — no `--with-tests`**: a genuinely stale test gets an explanatory PR
|
||||
comment, not a test edit.
|
||||
|
||||
**Tell each subagent which CVEs justify its upgrade**, with ids and severities, so the PR description
|
||||
says why it exists. That is most of this skill's value to a reviewer: a PR that names the CVSS-9.8 RCE
|
||||
it closes gets merged today, an unexplained version bump waits a fortnight.
|
||||
|
||||
Collect per recipe: PR url + number, the `!testme` verdict and build number, and any failure.
|
||||
|
||||
### 4. Report — one page covering sweep AND PRs
|
||||
Write `/tmp/cve-spec-<DATE>.json` per `/cve-check` step 6, with these differences:
|
||||
- Rows for upgraded recipes carry the real `ci` (`build N ✓` / `RED N · <stage>`) + `ci_url`, and
|
||||
`pr`/`pr_url`. `status` is the CI verdict (`GREEN`/`FAILED`/`STALE`); the live PR-status column
|
||||
derives itself from `recipe` + `pr`.
|
||||
- Rows for swept-but-not-upgraded recipes keep `PENDING`/`UPTODATE` with empty `ci`/`pr`, and a
|
||||
`notes` reason (`0 CVEs — not upgraded`, `external — maintained elsewhere`, `below --min-severity`).
|
||||
- Include `changes[]` — one entry per recipe that got a PR, describing what the upgrade changes **and
|
||||
the CVEs it closes**.
|
||||
- Keep `"kind": "cve"`: it titles the page "The Recipe Report — CVE check" and files it as
|
||||
`cve-<DATE>.html`, alongside the weekly editions in the same archive index.
|
||||
|
||||
Then render + publish exactly as `/cve-check` step 7, and verify as its step 8. Print the report URL,
|
||||
`N swept · M upgraded · K PRs green · J failed`, and `CVE CHECK AND UPGRADE COMPLETE`.
|
||||
|
||||
## Guardrails
|
||||
- **NEVER merge.** Create and verify; a human merges. Never push to true upstream.
|
||||
- **Never weaken a test** to make a PR green, and never edit a test without `--with-tests`.
|
||||
- **Never upgrade a recipe whose CVE count is unknown** on the assumption it is fine — surface it.
|
||||
- **Never upgrade an `external` recipe** here, even with a critical; report it instead.
|
||||
- **Public-safe report only** — no secrets, tokens, internal hostnames, raw logs, or spend figures.
|
||||
- If the sweep finds **nothing** at or above `--min-severity`, that is a good outcome: publish the
|
||||
report saying so and open no PRs. Do not manufacture work.
|
||||
@@ -0,0 +1,204 @@
|
||||
---
|
||||
name: cve-check
|
||||
description: Fleet-wide CVE sweep WITHOUT upgrading anything. For every recipe cc-ci deploys, works out what upgrade is available (current pinned tag → newest supported tag, per image including sidecars), runs the deterministic advisory scan over that window, adjudicates whatever the scan could not decide, and publishes a CVE report to report.ci.commoninternet.net as cve-<DATE>.html. READ-ONLY — opens no PRs, edits no recipes, runs no CI, merges nothing. Answers "what are we exposed to that an upgrade would fix?" in minutes rather than the hours a full upgrade run takes. Invoke as /cve-check [recipe ...] [--weekly-only].
|
||||
---
|
||||
|
||||
# cve-check
|
||||
|
||||
A **security sweep, not an upgrade run.** It answers one question for every recipe cc-ci deploys:
|
||||
|
||||
> If we upgraded this recipe today, how many CVEs would that fix, and how bad are they?
|
||||
|
||||
It is the cheap, safe half of `/upgrade-all`: the same version research and the same advisory scan,
|
||||
with **no implementation, no CI, and no PRs**. Use it when you want the security picture now — after a
|
||||
vendor announcement, before deciding what to prioritise, or between weekly runs. When you want the PRs
|
||||
too, use **`/cve-check-and-upgrade`**.
|
||||
|
||||
**Read-only, absolutely.** Never edit a recipe, never open or comment on a PR, never merge, never
|
||||
deploy. The only thing it writes is its own log and the published report page.
|
||||
|
||||
## Arguments
|
||||
- `<recipe> …` — sweep only these recipes (else every recipe in `cc-ci-plan/used-recipes.md`).
|
||||
- `--weekly-only` — skip rows tagged `external`. **Off by default on purpose**: an `external` recipe is
|
||||
still deployed and still exposes us, so a security sweep that silently skipped it would misreport the
|
||||
fleet's exposure. Externals are swept and clearly marked "maintained elsewhere" in the report.
|
||||
|
||||
## Procedure
|
||||
|
||||
> ### ⚠️ Run abra over a pseudo-TTY (or it FATAs `inappropriate ioctl for device`)
|
||||
> `abra` needs a TTY. Wrap every abra call: `ssh cc-ci 'script -qec "abra <args> -n" /dev/null'`.
|
||||
> (`git` and other commands do NOT need the wrapper.)
|
||||
|
||||
### 1. Build the candidate list
|
||||
Read `cc-ci-plan/used-recipes.md` — the canonical inventory. Take every row (both tiers), recording the
|
||||
tier per recipe; with `--weekly-only`, drop the `external` rows. An explicit recipe argument overrides
|
||||
any skip.
|
||||
|
||||
### 2. Per recipe — establish the upgrade window WITHOUT upgrading
|
||||
|
||||
This is `/recipe-upgrade` step 1's research, stopping before it implements anything.
|
||||
|
||||
> ⚠️ **The same four things that silently skip recipes apply here — handle ALL FOUR:**
|
||||
> 1. **pseudo-TTY** — per the box above.
|
||||
> 2. **go-git auth to git.autonomic.zone** — recipes on the private mirror FATA
|
||||
> `authentication required: Unauthorized`. Bake creds into origin first (idempotent, only when
|
||||
> origin is on git.autonomic.zone):
|
||||
> `git -C ~/.abra/recipes/<r> remote set-url origin "https://$GITEA_USERNAME:$GITEA_PASSWORD@git.autonomic.zone/recipe-maintainers/<r>.git"`
|
||||
> 3. **dirty worktree** — usually just the untracked cc-ci overlay; `git stash -u` before, `stash pop`
|
||||
> after. Only a genuinely dirty TRACKED tree is a skip.
|
||||
> 4. **tag+digest pins abra cannot parse** — abra FATAs and aborts the WHOLE recipe (immich). Do not
|
||||
> hand-check the registry; run the resolver, which is abra-independent and covers every image:
|
||||
> ```
|
||||
> python3 /srv/cc-ci/cc-ci-plan/resolve-images.py <recipe> --ssh cc-ci --table
|
||||
> ```
|
||||
> It reports, per image, `newest_within_major` (the compatibility-safe pick) and
|
||||
> `newest_same_shape` (the newest of that tag's form). **Use `newest_within_major` unless you have
|
||||
> checked the app supports the major jump** — immich's postgres tag encodes the pg major plus the
|
||||
> vectorchord/pgvectors versions immich-server is built against, so taking the newest would break
|
||||
> the deploy. `all_resolved: false` means an image could NOT be resolved — that is a `?`, never a 0.
|
||||
|
||||
**Reconcile the mirror from true upstream FIRST — ALWAYS, no exceptions** — one command,
|
||||
`cc-ci-plan/reconcile-upstream.sh <recipe>... | --all`. This is the same reconcile
|
||||
`/upgrade-all` does. Do not skip it in the name of keeping the sweep read-only: skipping it makes you
|
||||
research a stale checkout, and on the first real run that produced **two recipes with no survey output
|
||||
at all**, which is indistinguishable from "no upgrades" unless you check. It is safe — recipe work
|
||||
lives in **branches**, never directly on `main`, so a force-sync of `main` to upstream discards
|
||||
nothing; it also auto-closes mirror PRs whose changes upstream has already merged.
|
||||
|
||||
> ### ⚠️ The default branch may be `master`, not `main` — check, do not assume
|
||||
> Several coopcloud recipes keep a **stale `main` alongside the real default `master`**. gitea is one:
|
||||
> `main` sits at 1.24.2-rootless while `master` has 1.27.1-rootless plus the merged PRs and the 3.6.3
|
||||
> release. Reading `main` there tells you the recipe is three releases behind and missing two CVSS-9.8
|
||||
> RCE fixes — a false alarm that reads exactly like a real one. Resolve the default branch from the
|
||||
> API (`/api/v1/repos/coop-cloud/<recipe>` → `default_branch`) before reading any file, and never
|
||||
> `git reset --hard origin/main` on a checkout that tracks `master`.
|
||||
|
||||
**Cross-check abra with the resolver.** abra is the primary source, but it silently contributes
|
||||
nothing for images it cannot parse, and it reported "no new versions" for images that did have them
|
||||
(mumble v1.6.870-0 → -4). Run `resolve-images.py` for every recipe and take the UNION of the two: on
|
||||
the first real sweep the resolver found upgrades abra missed entirely in five recipes, one of which
|
||||
(plausible's clickhouse) carried four CVEs.
|
||||
|
||||
Then read versions:
|
||||
```
|
||||
set -a; . /srv/cc-ci/.testenv; set +a
|
||||
ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe> --reconcile-only" \
|
||||
< /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh
|
||||
ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; R=<recipe>; \
|
||||
git -C ~/.abra/recipes/$R stash -u >/dev/null 2>&1 || true; \
|
||||
script -qec "abra recipe fetch $R --force -n" /dev/null; \
|
||||
script -qec "abra recipe upgrade $R -m -n" /dev/null; \
|
||||
git -C ~/.abra/recipes/$R stash pop >/dev/null 2>&1 || true'
|
||||
```
|
||||
|
||||
For each recipe produce **one window per image**: `current pinned tag → newest supported tag`. You need
|
||||
the sidecars (redis, postgres, nginx …), not just the app — a sidecar bump is where discourse's only
|
||||
CRITICAL came from, and an image with no window is not counted at all.
|
||||
|
||||
- **No upgrade available** → the recipe is `UPTODATE`; its CVE count is **`0`**, not `?`. There is
|
||||
nothing an upgrade could fix. Record it and move on.
|
||||
- **No output at all is NOT "no upgrade".** An abra call that times out, FATAs, or prints nothing
|
||||
leaves the recipe **unverified** — treat it as a distinct outcome, never fold it into up-to-date.
|
||||
Re-run it, and if it still yields nothing, resolve the versions by direct registry check (box item 4).
|
||||
Only report `?` once BOTH the abra check and the direct check have failed. On the first real run this
|
||||
distinction was the difference between two false zeros and the truth (both recipes turned out fine,
|
||||
but nothing in the survey said so).
|
||||
|
||||
### 2c. Know which recipes CANNOT see CVEs at all
|
||||
```
|
||||
python3 cc-ci-plan/audit-sources.py --security-sources
|
||||
```
|
||||
A recipe whose sources yield **no CVE data at all** cannot produce a meaningful `0` — nothing was
|
||||
measured, the same way a missing registry file cannot. Render those as **`?`**, not `0`.
|
||||
|
||||
**The fleet is currently at zero such recipes.** The last two — `mattermost-lts` (empty advisory
|
||||
feed, client-side-rendered bulletins) and `mumble` (nothing published anywhere) — were fixed by
|
||||
declaring an NVD CPE in their registry:
|
||||
```
|
||||
- nvd-cpe: mattermost-team-edition = cpe:2.3:a:mattermost:mattermost_server:*:*:*:*:*:*:*:*
|
||||
```
|
||||
**If this sweep ever reports a blind recipe again, that is the fix**: find the product's CPE at
|
||||
nvd.nist.gov and add the line. Prefer a real advisory feed or an attributable changelog when one
|
||||
exists — NVD lags the vendor — but a lagging source beats no source, and it turns a `?` into a
|
||||
number.
|
||||
|
||||
An *unparseable page* is NOT the same thing: it is harmless when the same project also publishes an
|
||||
advisory feed (redis, gitea, minio, clickhouse all do). Only "no usable source for this image" counts.
|
||||
|
||||
### 3. Run the advisory scan over that window
|
||||
```
|
||||
python3 /srv/cc-ci/cc-ci-plan/advisory-scan.py <recipe> --from <old-app> --to <new-app> \
|
||||
[--image <name>=<old>:<new>]...
|
||||
```
|
||||
**One call per recipe with every image in it** — the count is a union across images, and the
|
||||
UNKNOWN guarantee only holds when a single run sees them all. Paste the markdown block verbatim into
|
||||
the per-recipe log at `/srv/cc-ci/.cc-ci-logs/cve-check/<DATE>/<recipe>.md`.
|
||||
|
||||
### 4. Adjudicate what the scan could not decide (pass 2)
|
||||
If the block reports advisories it **could NOT judge**, or the count is **UNKNOWN**, re-run with
|
||||
`--adjudicate` and decide each open case yourself:
|
||||
|
||||
```
|
||||
python3 /srv/cc-ci/cc-ci-plan/advisory-scan.py <recipe> … --adjudicate
|
||||
```
|
||||
|
||||
Answer **FIXED / NOT-FIXED / STILL-UNKNOWN** per case, each with a one-line reason **citing the
|
||||
evidence shown** — never from memory of the project, which is the exact failure that let two CVSS-9.8
|
||||
gitea RCEs be published as "none". Every FIXED is added to the count; pass 1's number is a floor. The
|
||||
block also lists what pass 1 already decided — if a verdict looks wrong given its evidence, say so.
|
||||
Record your verdicts in the per-recipe log so the number is auditable.
|
||||
|
||||
### 5. Classify severity and priority
|
||||
For each recipe collect the CVE ids with **severities** (the scan gives them, with GHSA ids). Sort the
|
||||
report rows by what an operator should deal with first:
|
||||
1. recipes with a **critical**, then **high**, then anything else with CVEs (more CVEs higher within a band);
|
||||
2. then `?` (a count that could not be established — investigate, do not ignore);
|
||||
3. then recipes with an upgrade available but **0** CVEs;
|
||||
4. then `UPTODATE`.
|
||||
|
||||
Severity outranks raw count: 2 CVSS-9.8 RCEs matter more than 120 medium plugin advisories.
|
||||
|
||||
### 6. Write the report spec
|
||||
`/tmp/cve-spec-<DATE>.json`, same shape as `/recipe-report` (see `recipe-report.py`'s header), with:
|
||||
- **`"kind": "cve"`** — titles the page "The Recipe Report — CVE check" and files it as
|
||||
`cve-<DATE>.html`. It appears in the SAME archive index as the weekly editions, suffixed
|
||||
"— CVE check" so the two are told apart at a glance. Without this field you would overwrite that
|
||||
date's weekly edition.
|
||||
- `date`, `subtitle` "CVE check <human date>",
|
||||
- `lead` — **one short paragraph**: fleet exposure in a sentence and what to do first.
|
||||
- `table[]` — every recipe swept. `recipe`; `change` = the window you scanned, e.g.
|
||||
`1.27.0 → 1.27.1 · redis 7.4 → 8.10`; `status` = `UPTODATE` when nothing is available, else
|
||||
`PENDING` (an upgrade exists and is not yet taken); **`cve`** = the count (integer, `?` only per the
|
||||
rules below); `notes` = severity mix, whether the number is a floor, and `maintained elsewhere` for
|
||||
`external` rows. **Leave `ci`/`pr` empty — nothing was built and no PR exists.**
|
||||
- `addendum[]` — real anomalies only: registry URLs that failed, recipes whose window could not be
|
||||
established, a scan whose count is a floor with many undetermined advisories.
|
||||
- `security[]` — one entry per **critical/high** finding: recipe · CVE id(s) + severity · what it fixes
|
||||
· **which image** it is in. Name the image: `CVE-2025-49844` is a redis flaw, and an operator reading
|
||||
"discourse" needs to know that.
|
||||
- `changes[]` — **omit** (nothing changed; there are no PRs).
|
||||
|
||||
**`?` must stay RARE.** Use it only when a scan ran and reported genuinely failed sources, or the count
|
||||
came back UNKNOWN and adjudication could not settle it. Never `none` for an unknown — a blank reads as
|
||||
clean. A recipe with no upgrade available is `0`, not `?`. Many `?` is a bug for the Addendum.
|
||||
|
||||
### 7. Render and publish — via the script only
|
||||
```
|
||||
python3 /srv/cc-ci/cc-ci-plan/recipe-report.py render /tmp/cve-spec-<DATE>.json /tmp/cve-<DATE>.html
|
||||
python3 /srv/cc-ci/cc-ci-plan/recipe-report.py publish /tmp/cve-<DATE>.html <DATE> cve
|
||||
```
|
||||
All layout is owned by `recipe-report.py`. Never hand-write or post-process HTML; if `render` errors,
|
||||
fix the spec JSON and re-render. **Public page — no secrets, tokens, internal hostnames, raw logs, or
|
||||
any billing/spend figures.**
|
||||
|
||||
### 8. Verify and stop
|
||||
`curl -fsS https://report.ci.commoninternet.net/cve-<DATE>.html` renders and the index lists it. Print
|
||||
the URL, a one-line summary (`N recipes swept · M with CVEs · K critical`), and `CVE CHECK COMPLETE`,
|
||||
then go idle. One-shot — do not loop, and do not start upgrading anything.
|
||||
|
||||
## Guardrails
|
||||
- **Read-only.** No PRs, no edits, no merges, no deploys, no CI runs. If a recipe looks urgent, say so
|
||||
in the report — do not act on it. `/cve-check-and-upgrade` is the skill that acts.
|
||||
- **Never report `0` for something you could not scan.** `0` means checked-and-clean; unknown is `?`.
|
||||
- A count with undetermined advisories is a **floor** — say so in the notes rather than rounding away.
|
||||
- **Public-safe output only.**
|
||||
@@ -0,0 +1,14 @@
|
||||
---
|
||||
name: help
|
||||
description: Operator orientation - lists every skill available on this orchestrator with what it does and when to reach for it, grouped by purpose (status, weekly maintenance, host updates, tests, enrollment, recovery, reporting), plus a "what do you want to do?" guide for common situations. Read-only. Invoke as /help.
|
||||
---
|
||||
|
||||
# help (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/help/SKILL.md`**
|
||||
|
||||
Read that file for the full content. This `.claude/skills/` copy is kept as a
|
||||
thin pointer for Claude Code compatibility; opencode loads the canonical
|
||||
definition from `.opencode/skills/` directly.
|
||||
@@ -57,60 +57,33 @@ tailscale ping -c 3 <host-alias>
|
||||
|
||||
If the host still does not come back, continue.
|
||||
|
||||
## 3. Request the Hetzner console
|
||||
## 3. Use the repo recovery tools (permanent home — do NOT rebuild these in /tmp)
|
||||
|
||||
Request a remote console session:
|
||||
The API and console tooling live in **`scripts/recovery/`** (see its README for the condensed
|
||||
10-minute drill, proven 2026-08-03):
|
||||
|
||||
```bash
|
||||
curl -s -X POST \
|
||||
-H "Authorization: Bearer ${HCLOUD_TOKEN}" \
|
||||
-H "Content-Type: application/json" \
|
||||
"https://api.hetzner.cloud/v1/servers/<SERVER_ID>/actions/request_console"
|
||||
# API: status / actions / reboot / reset / poweroff / poweron / rescue-on / rescue-off / console
|
||||
python3 /srv/cc-ci-orch/scripts/recovery/hetzner.py cc-ci status
|
||||
python3 /srv/cc-ci-orch/scripts/recovery/hetzner.py cc-ci actions 10
|
||||
|
||||
# Shell-only console access (fresh console session + websocat bridge + vncdotool,
|
||||
# venv auto-bootstrapped at ~/.cache/hetzner-console-venv):
|
||||
bash /srv/cc-ci-orch/scripts/recovery/hetzner-console.sh cc-ci screenshot /tmp/console.png
|
||||
bash /srv/cc-ci-orch/scripts/recovery/hetzner-console.sh cc-ci key Down Down Return
|
||||
```
|
||||
|
||||
The API returns:
|
||||
Known server names: `cc-ci` (134485294), `orchestrator` (134487234). Token: `HCLOUD_TOKEN`
|
||||
env or `/srv/cc-ci/.hcloud-token` (0600, not in git; prefer per-incident revocable tokens,
|
||||
and never paste tokens into a chat transcript).
|
||||
|
||||
- `wss_url`
|
||||
- `password`
|
||||
|
||||
If you have a browser, use the Hetzner console directly.
|
||||
|
||||
If you only have shell access, you can still drive it locally because the console is **raw VNC over
|
||||
websocket**.
|
||||
|
||||
## 4. Shell-only console access (websocket VNC bridge)
|
||||
|
||||
Install temporary tools:
|
||||
|
||||
```bash
|
||||
nix shell nixpkgs#websocat -c websocat --version
|
||||
python3 -m venv /tmp/opencode/hetzner-console-venv
|
||||
/tmp/opencode/hetzner-console-venv/bin/pip install --disable-pip-version-check pillow websocket-client vncdotool
|
||||
```
|
||||
|
||||
Bridge the websocket console to a local VNC TCP port:
|
||||
|
||||
```bash
|
||||
nohup nix shell nixpkgs#websocat -c \
|
||||
websocat -b -E tcp-l:127.0.0.1:5905 '<WSS_URL>' \
|
||||
>/tmp/opencode/hetzner-websockify.log 2>&1 &
|
||||
```
|
||||
|
||||
Validate the RFB banner:
|
||||
|
||||
```bash
|
||||
python3 - <<'PY'
|
||||
import socket
|
||||
s=socket.socket(); s.settimeout(5); s.connect(('127.0.0.1',5905))
|
||||
print(repr(s.recv(32)))
|
||||
PY
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
```text
|
||||
b'RFB 003.008\n'
|
||||
```
|
||||
Notes that used to cost time:
|
||||
- Each console command requests a **fresh** console session — old sessions die on hard reset,
|
||||
and the websocat bridge is single-connection anyway.
|
||||
- A GRUB one-shot/default for a NixOS generation needs the **submenu id `1>N`** (top level:
|
||||
0 = default entry, 1 = the "All configurations" submenu). A bare index silently falls back
|
||||
to the default entry. Clear any grubenv override after the next `switch` regenerates
|
||||
grub.cfg — indices shift.
|
||||
|
||||
Capture a screenshot from the console:
|
||||
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
---
|
||||
name: recipe-enroll
|
||||
description: Add a NEW recipe to cc-ci's maintained set, end to end — create + sync the recipe-maintainers mirror from coopcloud upstream, author a real test suite (health floor + non-vacuous recipe-specific tests incl. a create-an-object round-trip), enroll it in the !testme bridge (POLL_REPOS) and the weekly /upgrade-all inventory (used-recipes.md weekly row + upstream registry), then VERIFY the whole thing: full harness suite green with the new tests, bridge deployed (test-before-switch) and healthy. Opens the cc-ci PR for visibility and merges it directly once verification is green (the skill invocation is the authorization); the report lists merged PR links + what changed. Invoke as /recipe-enroll <recipe>.
|
||||
---
|
||||
|
||||
# recipe-enroll (thin wrapper)
|
||||
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
**`.opencode/skills/recipe-enroll/SKILL.md`**
|
||||
|
||||
Read that file for the full procedure. This `.claude/skills/` copy is kept as a
|
||||
thin pointer for Claude Code compatibility; opencode loads the canonical
|
||||
definition from `.opencode/skills/` directly.
|
||||
@@ -37,6 +37,35 @@ keeps every weekly edition looking the same regardless of which model writes the
|
||||
- **Security analysis.** Scan the per-recipe `upgrade_notes_md` + the summary (and use your own
|
||||
knowledge of the version bumps) for upgrades that fix **CVEs / security issues**. For each recipe,
|
||||
**count the CVEs** the PR fixes — this drives both the table's `cve` column and the priority sort.
|
||||
- **ADDITIONALLY, and never instead:** each per-recipe log carries an `### Advisory scan
|
||||
(deterministic pre-step)` block (from `cc-ci-plan/advisory-scan.py` — GitHub Security
|
||||
Advisories + vendor security pages + OSV, with severities and fixed-in versions). Treat its
|
||||
CVE list as a **further source** and report the **UNION** of it and what you found by reading.
|
||||
Its entries are machine-derived with advisory IDs, so prefer them for CVE ids / severities /
|
||||
fixed-in versions, and cite the GHSA where present in the Security Bulletin. If the block
|
||||
lists **failed sources**, the count is **not** authoritative: render the cve cell as `?`
|
||||
(unknown), never `none` — a blank that reads as "clean" is exactly how two CVSS-9.8 gitea
|
||||
RCEs were reported as "none" on 2026-08-07.
|
||||
- **`?` must stay RARE — it means "we tried and could not tell", not "we didn't look".** Use it
|
||||
ONLY when a scan ran and reported genuinely failed sources, **or when the scan block says
|
||||
COUNT UNKNOWN**. In that case the scan's `0` means *not determined*: publish `?` and say so in
|
||||
the notes; publishing `0` would assert a clean bill of health nothing supports. Note a
|
||||
**version-scheme change is no longer a reason for `?`** — the scan resolves semver→calver jumps
|
||||
(discourse 3.5.3 → 2026.7.1) by falling back to advisory publish dates and reports a real number.
|
||||
- **A count with undetermined advisories is a FLOOR.** If the scan block says N advisories
|
||||
"could NOT be judged", report the number but say in the notes that it is a floor — those
|
||||
advisories are neither fixed nor safe, they are unmeasured. Never round them away.
|
||||
- **Counts span every image, each judged by its own window.** A scan block lists one line per
|
||||
image with its version range and classification method; the headline is their union. So a
|
||||
recipe's count legitimately includes **sidecar** CVEs (discourse's 128 = 123 app + 5 redis).
|
||||
When a sidecar contributes a critical/high, name the image in the bulletin — CVE-2025-49844 is
|
||||
a redis flaw, not a discourse one, and an operator reading "discourse" needs to know that.
|
||||
(The scan headline itself now says `UNKNOWN` rather than a number in that case.)
|
||||
In particular: a recipe with **no upgrade this run** (up-to-date/skipped) has nothing an
|
||||
upgrade could have fixed — report `0`, not `?`. A recipe with a clean scan reports its number (including `0`). Benign notes in a scan
|
||||
block (`no-advisories-published`, `skipped: template URL`) are NOT failures and must not
|
||||
trigger `?`. If you find yourself rendering `?` for many recipes, that is a bug to report in
|
||||
the Addendum, not a normal outcome.
|
||||
Anything **critical/high** also gets a `security` bulletin entry (recipe · CVE id(s) + severity ·
|
||||
what it fixes · PR link); be specific about severity and what's exposed if not merged.
|
||||
- **Lead — ONE short paragraph.** A tight, concrete opener in opus's voice: fleet state in a sentence
|
||||
|
||||
@@ -157,6 +157,56 @@ On cc-ci's `~/.abra/recipes/<recipe>` (wrap every abra call per the pseudo-TTY b
|
||||
`open-recipe-pr.sh`). Do **not** push to upstream; the version bump + tag + publish are the operator's
|
||||
final `abra recipe release` step.
|
||||
|
||||
### 2a. Advisory scan (deterministic; ADDITIVE — run it, never skip it)
|
||||
|
||||
Run the deterministic scanner for the exact upgrade window and **paste its markdown block verbatim
|
||||
into the per-recipe log**:
|
||||
|
||||
```
|
||||
python3 /srv/cc-ci/cc-ci-plan/advisory-scan.py <recipe> --from <old-app-version> --to <new-app-version> \
|
||||
[--image <name>=<old>:<new>]...
|
||||
```
|
||||
|
||||
**Pass an `--image` for EVERY sidecar you upgraded** (redis, postgres, nginx …), not just the app —
|
||||
each image is judged by its own versions, and an image you don't name is not counted at all. Repeat
|
||||
the flag for each one and pass them **all in a single call** (the count is a union across images).
|
||||
e.g. discourse moving app 3.5.3→2026.7.1 *and* redis 7.4→8.10:
|
||||
|
||||
```
|
||||
... --from 3.5.3 --to 2026.7.1 --image redis=7.4:8.10
|
||||
```
|
||||
|
||||
→ 140 CVEs (123 app + 17 redis), where the redis seventeen include a **critical** (CVE-2025-49844)
|
||||
that is invisible if the sidecar is left out. `<name>` is substring-matched against source repo names,
|
||||
so make it specific enough to hit exactly one.
|
||||
|
||||
**If the scan reports advisories it could NOT judge, re-run it with `--adjudicate`.** That is a second
|
||||
pass: it collects each open case's full evidence (advisory prose, references, affected ranges, every
|
||||
release naming the CVE) and asks YOU to decide FIXED / NOT-FIXED / STILL-UNKNOWN with a reason citing
|
||||
that evidence. The deterministic number is a **floor** — add every FIXED to the count. Say
|
||||
STILL-UNKNOWN rather than inferring from memory, and never record an undecided CVE as unaffected.
|
||||
It also shows what pass 1 already decided; if a verdict looks wrong given its evidence, say so.
|
||||
|
||||
It queries, per recipe: the **GitHub Security Advisories API** for every source repo in
|
||||
`cc-ci-plan/upstream/<recipe>.md` (CVE + GHSA + severity + vulnerable/patched ranges, so
|
||||
"fixed by THIS upgrade" is computed, not guessed), every **vendor release/security URL** in that
|
||||
registry (fetched + regex-scanned for CVE ids), and **OSV** where a package mapping exists.
|
||||
|
||||
**This does NOT replace your own release-note reading — it is an ADDITIONAL evidence source.** Do
|
||||
exactly what you did before, then union the two: the CVE count you report is the union of the CVEs
|
||||
you found in the notes and the CVEs the scan found. Never let the scan lower a count you established
|
||||
by reading.
|
||||
|
||||
Why it exists: gitea 1.27.1 fixed CVE-2026-60004 and CVE-2026-59774 (both CVSS 9.8). Both are named
|
||||
only in the vendor's blog security section — the GitHub *release notes* mention neither — so the
|
||||
release-note read found one unrelated minor item and the weekly report printed a CVE count of "1",
|
||||
then "none". Advisory databases lagged too (OSV 404'd on both; NVD's API had neither by CPE, id, or
|
||||
keyword), which is why the GitHub advisory API and the vendor pages lead.
|
||||
|
||||
If the scanner reports **failed sources**, say so in the log — an incomplete scan must not read as
|
||||
a clean one. If a vendor publishes security notes at a URL the registry lacks (gitea's
|
||||
`blog.gitea.com`), **add it to `cc-ci-plan/upstream/<recipe>.md`** so the next scan sees it.
|
||||
|
||||
### 2b. Direct deploy + inspect on cc-ci — live feedback BEFORE CI (recipe-maintainer style)
|
||||
Before opening the PR / running `!testme`, deploy the WIP recipe **directly** on the cc-ci server and
|
||||
watch it converge — the way recipe-maintainer tests on `cctest`. This gives you **live logs +
|
||||
@@ -262,6 +312,12 @@ test change, and a test change is **gated by `--with-tests`**:
|
||||
Do **NOT** modify any test. Report `SUCCESS-PENDING-TESTS` (recipe PR open; `!testme` red on a
|
||||
stale test; operator to decide).
|
||||
- **`--with-tests` — open + verify a cc-ci test PR.** Make it the `ci-test-review` way:
|
||||
0. **READ `tests/STYLE.md` in the cc-ci repo FIRST.** It is the rulebook for changing a test, and
|
||||
it is written against the failures this pipeline has actually produced. The two that matter most
|
||||
here: **set state up through the app's own interface, never its database** (a plausible fixture
|
||||
that INSERTed rows passed on v2 and silently broke on v3, holding the recipe RED for six weeks),
|
||||
and **gate on version rather than writing a fixture that supports both** — old-version tests can
|
||||
simply be deleted, since the older version is only exercised through the upgrade tier.
|
||||
1. Branch `recipe-maintainers/cc-ci` in a **separate clone** (single-writer: never push `main`,
|
||||
never touch the build loops' `/cc-ci` `/cc-ci-adv` clones); update the test/overlay.
|
||||
2. **Verify the recipe upgrade WITH the updated test applied.** `!testme` on the recipe PR uses the
|
||||
|
||||
@@ -102,6 +102,27 @@ if [ "${MODE}" != "--reconcile-only" ]; then
|
||||
DIVERGED=$(git log --oneline origin/main..HEAD 2>/dev/null || true)
|
||||
[ -n "${DIVERGED}" ] || { echo "ERROR: HEAD has no commits beyond origin/main. Nothing to PR."; exit 1; }
|
||||
LATEST_MSG=$(git log -1 --pretty=%s HEAD)
|
||||
|
||||
# --- Keep the LOCAL work current with the freshly-synced upstream main (anti-drift) ---
|
||||
# The push path below grafts `HEAD^{tree}` WHOLESALE onto the PR branch. If this checkout is not
|
||||
# based on the upstream main we just synced, every upstream change made since the branch was cut
|
||||
# is silently ABSENT from the pushed tree — the PR (and the CI that verifies it) then describes a
|
||||
# tree that will never deploy. Observed on gitea PR #5 (2026-08-10): its base predated upstream's
|
||||
# "BREAKING CHANGE: remove forgejo", so `!testme` verified a forgejo-bearing tree while main had
|
||||
# dropped it. Merge upstream in FIRST, and fail loudly rather than paper over a conflict.
|
||||
if ! git merge-base --is-ancestor "${NEW_MAIN_SHA}" HEAD; then
|
||||
echo "→ Local work predates upstream main (${NEW_MAIN_SHA:0:8}) — merging upstream in first..."
|
||||
if ! GIT_AUTHOR_NAME="${GITEA_USERNAME}" GIT_AUTHOR_EMAIL="${GITEA_USERNAME}@git.autonomic.zone" \
|
||||
GIT_COMMITTER_NAME="${GITEA_USERNAME}" GIT_COMMITTER_EMAIL="${GITEA_USERNAME}@git.autonomic.zone" \
|
||||
git merge --no-edit "${NEW_MAIN_SHA}" >/dev/null 2>&1; then
|
||||
git merge --abort 2>/dev/null || true
|
||||
echo "ERROR: cannot auto-merge upstream main (${NEW_MAIN_SHA:0:8}) into the local ${RECIPE} work."
|
||||
echo " Upstream changed files this upgrade also touches. Resolve by hand in"
|
||||
echo " ${RECIPE_DIR}, then re-run. Refusing to push a tree that omits upstream changes."
|
||||
exit 1
|
||||
fi
|
||||
echo " ✓ upstream merged into the local work"
|
||||
fi
|
||||
fi
|
||||
|
||||
# --- Reconcile open PRs against the freshly-synced upstream main ---
|
||||
@@ -165,9 +186,19 @@ if git rev-parse --verify --quiet "refs/remotes/gitea/${BRANCH}" >/dev/null; the
|
||||
if [ "$(git rev-parse 'HEAD^{tree}')" = "$(git rev-parse "${EXIST_TIP}^{tree}")" ]; then
|
||||
echo "→ '${BRANCH}' already has this exact tree — nothing new to push (will still re-test)."
|
||||
else
|
||||
# Parent the new commit on the branch tip AND (when the branch predates it) on upstream main, so
|
||||
# the recorded HISTORY matches the tree we are pushing. Without the second parent the merge-base
|
||||
# stays stale: git would later treat upstream's post-branch changes as "removed by this PR" and a
|
||||
# merge could revert them (the gitea #5 / forgejo-removal drift, 2026-08-10). No force-push: this
|
||||
# is still a fast-forward from the branch tip.
|
||||
EXTRA_PARENT=()
|
||||
if ! git merge-base --is-ancestor "${NEW_MAIN_SHA}" "${EXIST_TIP}"; then
|
||||
EXTRA_PARENT=(-p "${NEW_MAIN_SHA}")
|
||||
echo " (also parenting on upstream main ${NEW_MAIN_SHA:0:8} — branch predated it)"
|
||||
fi
|
||||
ONTOP=$(GIT_AUTHOR_NAME="${GITEA_USERNAME}" GIT_AUTHOR_EMAIL="${GITEA_USERNAME}@git.autonomic.zone" \
|
||||
GIT_COMMITTER_NAME="${GITEA_USERNAME}" GIT_COMMITTER_EMAIL="${GITEA_USERNAME}@git.autonomic.zone" \
|
||||
git commit-tree "$(git rev-parse 'HEAD^{tree}')" -p "${EXIST_TIP}" -m "${LATEST_MSG}")
|
||||
git commit-tree "$(git rev-parse 'HEAD^{tree}')" -p "${EXIST_TIP}" "${EXTRA_PARENT[@]}" -m "${LATEST_MSG}")
|
||||
echo "→ Adding the new work on top of '${BRANCH}' (fast-forward, no force-push)..."
|
||||
git push gitea "${ONTOP}:refs/heads/${BRANCH}"
|
||||
fi
|
||||
|
||||
@@ -18,7 +18,7 @@ RECIPE="${1:?usage: testme-on-pr.sh <recipe> <pr-index>}"
|
||||
PRIDX="${2:?usage: testme-on-pr.sh <recipe> <pr-index>}"
|
||||
TESTENV="${TESTENV:-/srv/cc-ci/.testenv}"
|
||||
set -a; . "$TESTENV"; set +a
|
||||
: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:?}"
|
||||
: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:=git.autonomic.zone}"
|
||||
NS="${GITEA_NAMESPACE:-recipe-maintainers}"
|
||||
API="https://${GITEA_URL}/api/v1"; AUTH=(-u "${GITEA_USERNAME}:${GITEA_PASSWORD}")
|
||||
POST="${POST:-1}"; MAX_WAIT="${MAX_WAIT:-480}"; INTERVAL="${INTERVAL:-30}"
|
||||
|
||||
@@ -73,6 +73,29 @@ done
|
||||
# 5) Stray exited containers (debug one-shots) — best-effort prune.
|
||||
docker container prune -f >/dev/null 2>&1 || true
|
||||
|
||||
# 6) Unused IMAGES — the one that actually took CI down. Every run pulls each recipe's images and
|
||||
# nothing ever removed the old ones: on 2026-08-11 they had grown to 72GB (63GB of it unused),
|
||||
# the root filesystem hit 100% under two concurrent runs, and the harness died at startup with
|
||||
# `OSError: [Errno 28] No space left on device: '/var/lib/cc-ci-runs/<build>'`. Every !testme
|
||||
# from build 1236 to 1242 failed that way — with no results.json, so the PR badges just read
|
||||
# "failure" and looked like recipe regressions.
|
||||
#
|
||||
# Only prune above a threshold, so a healthy host keeps its layer cache and runs stay fast.
|
||||
# `image prune -a` removes only images no container references, so anything deployed (infra +
|
||||
# warm-* canonicals) is untouched; anything else is re-pulled on demand.
|
||||
#
|
||||
# Volumes are deliberately NOT pruned here — see the KEEP_RE guard in (3): warm-* canonicals are
|
||||
# data-warm and their volumes are legitimately dangling between runs.
|
||||
DISK_PRUNE_PCT="${DISK_PRUNE_PCT:-60}"
|
||||
used_pct="$(df --output=pcent / 2>/dev/null | tail -1 | tr -dc '0-9')"
|
||||
if [ -n "$used_pct" ] && [ "$used_pct" -ge "$DISK_PRUNE_PCT" ]; then
|
||||
echo " disk ${used_pct}% >= ${DISK_PRUNE_PCT}% -> pruning unused images"
|
||||
freed="$(docker image prune -af 2>/dev/null | awk '/Total reclaimed space/ {print $4, $5}')"
|
||||
echo " reclaimed: ${freed:-0B}; disk now $(df -h / | tail -1 | awk '{print $5" used, "$4" free"}')"
|
||||
else
|
||||
echo " disk ${used_pct:-?}% < ${DISK_PRUNE_PCT}% -> keeping image cache"
|
||||
fi
|
||||
|
||||
if [ "$removed" -eq 0 ]; then
|
||||
echo "== orphan sweep: clean (nothing to remove) =="
|
||||
else
|
||||
|
||||
@@ -28,3 +28,7 @@ master-age.txt
|
||||
# Python bytecode cache
|
||||
__pycache__/
|
||||
*.pyc
|
||||
|
||||
# Local API tokens — never committed (advisory-scan / hetzner recovery)
|
||||
.github-token
|
||||
.hcloud-token
|
||||
|
||||
+1
-1
@@ -1,3 +1,3 @@
|
||||
[submodule "references/recipe-maintainer"]
|
||||
path = references/recipe-maintainer
|
||||
url = https://git.autonomic.zone/recipe-maintainers/recipe-maintainer
|
||||
url = ssh://git@git.autonomic.zone:2222/recipe-maintainers/autonomic-recipe-maintainer.git
|
||||
|
||||
@@ -13,8 +13,8 @@ a health gate, not as silent drift.
|
||||
|
||||
> **Two hosts, two flakes — don't confuse them.** This skill updates the **orchestrator** host:
|
||||
> the machine this session runs on (`cc-ci-orchestrator-1`, Hetzner cpx22 **server 134487234**,
|
||||
> tailnet `100.84.190.30`, public `168.119.126.100`), flake checkout **`/srv/cc-ci-orch`** (repo
|
||||
> `recipe-maintainers/cc-ci-orchestrator`), target **`.#cc-ci-orchestrator-hetzner`**. The **cc-ci
|
||||
> tailnet `cc-ci`, public `195.201.88.249` — the SAME host as the cc-ci CI server since 2026-09), flake checkout **`/srv/cc-ci-orch`** (repo
|
||||
> `recipe-maintainers/cc-ci-orchestrator`), target **`.#cc-ci`** (which now also rebuilds the CI server half, from the cc-ci repo flake input). The **cc-ci
|
||||
> CI server** (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, target `.#cc-ci`) is a different
|
||||
> machine — that's `/cc-ci-server-update`, NOT this skill.
|
||||
|
||||
@@ -72,7 +72,7 @@ deploy).
|
||||
### 3. Build (catch errors before any activation)
|
||||
|
||||
```
|
||||
cd /srv/cc-ci-orch && nixos-rebuild build --flake .#cc-ci-orchestrator-hetzner 2>&1 | tail -15
|
||||
cd /srv/cc-ci-orch && nixos-rebuild build --flake .#cc-ci 2>&1 | tail -15
|
||||
readlink -f result
|
||||
```
|
||||
Build failure → fix on the branch (option renames etc.) before going further. Never activate a
|
||||
@@ -85,7 +85,7 @@ activation breaks the host (cf. the cc-ci server's 2026-08-03 no-default-route o
|
||||
reboot — Hetzner API power-cycle on server **134487234** if SSH is gone (see
|
||||
`hetzner-server-recovery`) — lands back on the last-known-good generation.
|
||||
```
|
||||
cd /srv/cc-ci-orch && setsid nohup nixos-rebuild test --flake .#cc-ci-orchestrator-hetzner \
|
||||
cd /srv/cc-ci-orch && setsid nohup nixos-rebuild test --flake .#cc-ci \
|
||||
> /tmp/orchestrator-test-switch.log 2>&1 < /dev/null & echo launched
|
||||
# after it settles (poll; tailscaled/sshd may blip):
|
||||
readlink /run/current-system # should be the new store path
|
||||
@@ -100,7 +100,7 @@ switch.
|
||||
### 5. Switch (make permanent — only after 4 is healthy)
|
||||
|
||||
```
|
||||
cd /srv/cc-ci-orch && nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner 2>&1 | tail -10
|
||||
cd /srv/cc-ci-orch && nixos-rebuild switch --flake .#cc-ci 2>&1 | tail -10
|
||||
```
|
||||
(If it fails with "Unit nixos-rebuild-switch-to-configuration.service was already loaded", the
|
||||
detached test's transient unit is still running — wait or `systemctl stop` it, then retry.)
|
||||
@@ -127,7 +127,7 @@ git add flake.lock # flake.nix too if the channel
|
||||
git commit -m "flake: bump nixpkgs (nixos-26.05, $(date -u +%Y-%m-%d))
|
||||
|
||||
nixpkgs: <old-rev[:8]> -> <new-rev[:8]> (nixos-26.05 tip)
|
||||
Deployed to cc-ci-orchestrator-hetzner: build + test + switch + health gate green."
|
||||
Deployed to the cc-ci host (.#cc-ci): build + test + switch + health gate green."
|
||||
git push -u origin HEAD
|
||||
```
|
||||
Open the PR on `recipe-maintainers/cc-ci-orchestrator` (Gitea API with the `GITEA_*` creds from
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
---
|
||||
name: cc-ci-status
|
||||
description: Comprehensive read-only status check of the whole cc-ci system - how the recent weekly upgrade runs went and whether their reports published, which recipes/tests are stale, how long since the server + orchestrator host flake updates, open recipe PRs (flagging CVE-carrying PRs that have been open too long), host health (failed units, disk, timers, bridge/!testme path), ending with a verdict (ALL HEALTHY or a findings list) and recommended next steps mapped to the skills to invoke. Never changes anything - it only reads and reports. Invoke as /cc-ci-status.
|
||||
---
|
||||
|
||||
# cc-ci-status
|
||||
|
||||
One comprehensive, **read-only** status pass over the cc-ci system. Output ends with either
|
||||
**`ALL HEALTHY`** or a prioritized findings list, each finding paired with the skill to invoke.
|
||||
Nothing here mutates state — no restarts, no deploys, no merges.
|
||||
|
||||
## Checks (run all; collect findings, don't stop at the first)
|
||||
|
||||
### 1. Weekly upgrade runs — recency + outcome
|
||||
|
||||
```
|
||||
ls -t /srv/cc-ci/.cc-ci-logs/upgrades/upgrade-all-*.md | head -3
|
||||
head -20 <latest> # the Summary block: Considered/green/stale/Failed/Skipped
|
||||
systemctl list-timers cc-ci-upgrade-all.timer --no-pager | head -3
|
||||
```
|
||||
- **Overdue** if the newest report is >8 days old, or the timer is inactive/missing → recommend
|
||||
`systemctl start cc-ci-upgrade-all.service` (or investigate the timer) / `/upgrade-all`.
|
||||
- **Failed entries** in the latest report → recommend `/recipe-upgrade <recipe>` per entry (or
|
||||
`/ci-test-review` if the failure is harness-side).
|
||||
- A run currently in flight (tmux `cc-ci-upgrader` session live) is NOT a finding — report it as
|
||||
in-progress and skip staleness checks that depend on its output.
|
||||
|
||||
### 2. Report publishing — report.ci.commoninternet.net
|
||||
|
||||
```
|
||||
curl -s -o /dev/null -w '%{http_code}' https://report.ci.commoninternet.net/
|
||||
ls -t /var/lib/cc-ci-reports/week-*.html | head -2 # via ssh cc-ci
|
||||
```
|
||||
- Index must be 200 and there must be a `week-*.html` at least as new as the last **completed**
|
||||
upgrade run (a completed run without a matching page = report generation broke → recommend
|
||||
`/recipe-report` / inspect `launch-report.py`).
|
||||
|
||||
### 3. Stale recipes / stale tests
|
||||
|
||||
- Latest report's "PRs where a test looks stale" section + carry-over notes.
|
||||
- Any entries → recommend `/cc-ci-tests-update` (fleet) or `/recipe-upgrade <recipe>
|
||||
--with-tests` (single).
|
||||
- Also check `/srv/cc-ci/.cc-ci-logs/tests-update-*.md` recency — if stale tests were reported
|
||||
weeks ago and no tests-update run since, say so.
|
||||
|
||||
### 4. Open recipe PRs — especially CVE-carrying ones that linger
|
||||
|
||||
Enumerate open PRs across `recipe-maintainers/*` (Gitea API, creds in `/srv/cc-ci/.testenv`):
|
||||
```
|
||||
GET /repos/recipe-maintainers/<repo>/pulls?state=open # repos = the used-recipes.md inventory + cc-ci
|
||||
```
|
||||
For each open PR: age (now − created_at), and whether the PR title/body/report row mentions
|
||||
**CVE** / security patch.
|
||||
- **CVE-carrying PR open >14 days** → HIGH-priority finding: name the PR, the CVE context, and
|
||||
the blocker (commonly a stale test — check the report row) → recommend the unblocking skill
|
||||
(`/cc-ci-tests-update`) plus "operator: review + merge <PR>".
|
||||
- Non-CVE PRs open >30 days → low-priority note ("operator review backlog: N PRs").
|
||||
- Verified-green PRs awaiting operator merge are normal — list them as "ready to merge", not
|
||||
as failures.
|
||||
|
||||
### 5. Server + orchestrator host update recency
|
||||
|
||||
For BOTH hosts:
|
||||
```
|
||||
# cc-ci server: ssh cc-ci 'nixos-version; cd /root/cc-ci-deploy && nix flake metadata --json' (or builder-clone)
|
||||
# orchestrator: nixos-version; cd /srv/cc-ci-orch && nix flake metadata --json
|
||||
git ls-remote https://github.com/NixOS/nixpkgs <channel> # current tip
|
||||
ls -t /srv/cc-ci-orch/.cc-ci-logs/server-update-*.md /srv/cc-ci-orch/.cc-ci-logs/orchestrator-update-*.md | head -2
|
||||
```
|
||||
- Report: days since last update log + how far the running nixpkgs rev lags the channel tip.
|
||||
- **Lagging >30 days** (or a NixOS release behind) → recommend `/cc-ci-server-update` /
|
||||
`/cc-ci-orchestrator-update`.
|
||||
|
||||
### 6. Host + service health (both machines)
|
||||
|
||||
```
|
||||
ssh cc-ci 'systemctl --failed --no-legend; df -h / | tail -1; docker service ls --format "{{.Name}} {{.Replicas}}"'
|
||||
systemctl --failed --no-legend; df -h / | tail -1; tmux ls
|
||||
```
|
||||
- Failed units, core swarm services not 1/1 (warm-* spares flapping is a known benign pattern —
|
||||
note, don't page), disk **>65% (server)** / >85% (orchestrator) → findings. Server unreachable →
|
||||
HIGH: recommend `hetzner-server-recovery`.
|
||||
> **65%, not 80%, on the server — it is not a steady-state measure.** Two concurrent recipe runs
|
||||
> pull images and write volumes worth tens of GB, so a host sitting at 73% still hits 100% mid-run.
|
||||
> That is exactly what happened on 2026-08-11: 63GB of unused images had accumulated (nothing ever
|
||||
> pruned them), the filesystem filled during a run, and the harness died at startup with
|
||||
> `OSError: [Errno 28] No space left on device`. Remedy: `docker image prune -af` on cc-ci — it
|
||||
> spares anything a container references, so infra and warm-* canonicals are untouched. Do NOT
|
||||
> `docker volume prune`: warm-* canonical volumes are data-warm and legitimately dangling.
|
||||
- **!testme actually produces results** (the check that would have caught the above days earlier):
|
||||
the newest few `/var/lib/cc-ci-runs/<build>/` dirs must each contain `results.json`. A build that
|
||||
dies before the harness writes one leaves an EMPTY dir — and the PR badge still says "failure", so
|
||||
it reads as a recipe regression rather than a sick host. Builds 1236–1242 all failed that way.
|
||||
Finding: *"N recent builds produced no results.json — the harness is dying at startup, check disk
|
||||
and the drone step log"*. The step log lives in drone's sqlite
|
||||
(`/var/lib/docker/volumes/drone_ci_commoninternet_net_data/_data/database.sqlite`) — copy it and
|
||||
read `logs.log_data` for the failing `steps.step_id`; the bridge's drone token is not extractable
|
||||
(distroless container, swarm secret).
|
||||
> **If the error is ENOSPC but the disk is fine**, it is not disk. Seen 2026-08-11: builds 1244-1249
|
||||
> died on `mkdir /var/lib/cc-ci-runs/<build>` with **110GB free and 16% inodes**, while the identical
|
||||
> mkdir succeeded as root over ssh, inside the runner's own mount namespace, and 61/61 times in a
|
||||
> stress loop — and the same harness run by hand with a numeric run id worked fine. Restarting
|
||||
> `drone-runner-exec` did NOT help, and neither did recreating the runs directory with a fresh
|
||||
> inode (it recurred afterwards — that apparent fix was coincidence).
|
||||
>
|
||||
> **It is INTERMITTENT and tracks concurrent activity**, which is the useful signal: every failure
|
||||
> landed while a second run or a manual deploy was in flight (1252 was triggered while 1251 was
|
||||
> still finishing), and every build on a quiet host succeeded (1243, 1250, 1251, 1253). Free space
|
||||
> never moved during a failing build. So on ENOSPC-with-free-disk: **wait for the host to go quiet
|
||||
> and re-trigger** before treating it as a recipe failure. Root cause is still NOT established;
|
||||
> `DRONE_RUNNER_CAPACITY=2` allows the overlap, so lowering it to 1 is the obvious next experiment
|
||||
> if it becomes disruptive.
|
||||
- **Bridge / !testme path**: `docker service ls` shows `ccci-bridge_app 1/1` AND the bridge log
|
||||
has no auth errors (`docker service logs --since 24h ccci-bridge_app 2>&1 | grep -ci "401\|user does not exist"` == 0).
|
||||
A silently-401ing bridge drops `!testme` (seen 2026-08-03, stale rotated Gitea secret) →
|
||||
recommend refreshing the bridge secret + redeploy (test-before-switch).
|
||||
|
||||
### 7. Maintained-set consistency (quick)
|
||||
|
||||
- Every `weekly` row in `used-recipes.md` has `tests/<recipe>/` on cc-ci AND a
|
||||
`recipe-maintainers/<recipe>` mirror AND is in bridge `POLL_REPOS`; mismatches → recommend
|
||||
finishing enrollment (`/recipe-enroll <recipe>` covers all touchpoints).
|
||||
|
||||
## Output format
|
||||
|
||||
```
|
||||
# cc-ci status — <date>
|
||||
## Verdict: ALL HEALTHY | N findings (M high-priority)
|
||||
## Weekly upgrades: <last run date + one-line outcome; next timer firing>
|
||||
## Report site: <ok/broken + latest page>
|
||||
## Stale tests: <none | list>
|
||||
## Open PRs: <count; CVE-carrying + age flagged FIRST; ready-to-merge list>
|
||||
## Host updates: server <rev, N days behind tip> · orchestrator <rev, N days>
|
||||
## Health: server <failed/disk/services> · orchestrator <failed/disk/sessions> · bridge <ok/401s>
|
||||
## Recommended next steps
|
||||
1. <finding> → /<skill> (or operator action)
|
||||
…
|
||||
```
|
||||
When a finding is that the fleet's **security exposure is unknown** — the last weekly run failed or
|
||||
is stale, so nobody has scanned for CVEs recently — the recommended step is **`/cve-check`** (read-only,
|
||||
minutes, no PRs). If it is instead that a known CVE is sitting unpatched, recommend
|
||||
**`/cve-check-and-upgrade`** (add `--min-severity high` when only the urgent ones matter). Prefer
|
||||
`/cve-check` over waiting for the next weekly run whenever the question is "are we exposed?".
|
||||
|
||||
`ALL HEALTHY` requires: recent successful weekly run + published report, no stale tests, no
|
||||
CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, recent builds all
|
||||
producing results.json, disk under
|
||||
thresholds, bridge clean, maintained-set consistent. Anything else is a finding — even minor
|
||||
ones get a recommended next step. Order findings by priority (CVE/unreachable-host first).
|
||||
|
||||
## Guardrails
|
||||
|
||||
- **Read-only.** This skill diagnoses and recommends; it invokes nothing and changes nothing.
|
||||
- Use the pseudo-TTY wrap for any abra call; plain ssh for everything else.
|
||||
- Don't double-count: a finding that explains another (bridge 401 → !testme "failures") gets
|
||||
reported once, at the root cause.
|
||||
@@ -83,8 +83,29 @@ failure (AI — this is the `ci-test-review` step-3 diagnosis):
|
||||
changed upstream, what the test currently asserts.
|
||||
- **FLAKY** → re-run once or twice; if it passes, drop it (not stale, just flaky).
|
||||
|
||||
> ### ⚠️ RECONCILE FROM UPSTREAM FIRST — always, before any PR work or upgrade check
|
||||
> ```
|
||||
> cc-ci-plan/reconcile-upstream.sh <recipe>... # or --all
|
||||
> ```
|
||||
> Deterministic, idempotent, and safe (recipe work lives in branches, never on mirror `main`). It
|
||||
> force-syncs each mirror to coopcloud's **default branch — resolved from the API, `main` OR
|
||||
> `master`** — and closes any mirror PR whose changes upstream already merged. Skipping it has cost
|
||||
> us three distinct ways: mailu #6 was reported as the fix for two internet-facing CVEs while
|
||||
> upstream had already merged AND released it; a stale mirror makes a survey report "no upgrades
|
||||
> available" so the recipe drops out of the weekly run; and reading the wrong branch on a recipe with
|
||||
> a stale `main` beside a live `master` (gitea) manufactures a false "three releases behind, missing
|
||||
> two CVSS-9.8 RCEs" finding.
|
||||
|
||||
### 2. For each stale test — author the minimal test update (AI; never weaken)
|
||||
|
||||
> **Read `tests/STYLE.md` in the cc-ci repo before writing the update.** It is the rulebook for test
|
||||
> changes, written from failures this pipeline actually produced. Most load-bearing: set state up
|
||||
> through the app's **own interface, never its database** (a plausible fixture that INSERTed rows
|
||||
> passed on v2 and silently broke on v3 — 202 acks, rows in postgres, nothing ingested — and held the
|
||||
> recipe RED for six weeks), **gate on version rather than supporting both** (old-version tests can be
|
||||
> deleted; the older version is only exercised via the upgrade tier), and correct the fixture or the
|
||||
> wait but **never the assertion**.
|
||||
|
||||
Work on **one recipe at a time** (serialize — each verification deploys a recipe on the shared
|
||||
Swarm). For each `STALE_TESTS` entry:
|
||||
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-init-instance
|
||||
description: "[recipe-maintainer/cctest] Deploy all maintained recipes to the active test instance from scratch (Wraps the autonomic-recipe-maintainer skill /init-instance; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-init-instance.)"
|
||||
---
|
||||
|
||||
# cctest-init-instance (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/init-instance/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-intro
|
||||
description: "[recipe-maintainer/cctest] Explain what this project is and how to get started (Wraps the autonomic-recipe-maintainer skill /intro; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-intro.)"
|
||||
---
|
||||
|
||||
# cctest-intro (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/intro/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-new-recipe-guide
|
||||
description: "[recipe-maintainer/cctest] Guide for developing a new Co-op Cloud recipe from scratch (Wraps the autonomic-recipe-maintainer skill /new-recipe-guide; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-new-recipe-guide.)"
|
||||
---
|
||||
|
||||
# cctest-new-recipe-guide (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/new-recipe-guide/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-opencode-sync
|
||||
description: "[recipe-maintainer/cctest] Ensure every Claude skill has a corresponding OpenCode skill alias (Wraps the autonomic-recipe-maintainer skill /opencode-sync; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-opencode-sync.)"
|
||||
---
|
||||
|
||||
# cctest-opencode-sync (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/opencode-sync/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-recipe-check
|
||||
description: "[recipe-maintainer/cctest] Fetch a Co-op Cloud recipe and check for available upgrades (Wraps the autonomic-recipe-maintainer skill /recipe-check; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-check.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-check (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-check/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-recipe-create-pr
|
||||
description: "[recipe-maintainer/cctest] Push local recipe commits to git.autonomic.zone and open a PR against an upstream-synced main branch (Wraps the autonomic-recipe-maintainer skill /recipe-create-pr; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-create-pr.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-create-pr (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-create-pr/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-recipe-deploy
|
||||
description: "[recipe-maintainer/cctest] Deploy the local recipe checkout to the test instance (Wraps the autonomic-recipe-maintainer skill /recipe-deploy; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-deploy.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-deploy (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-deploy/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-recipe-guidelines
|
||||
description: "[recipe-maintainer/cctest] Guidelines for all recipe operations including local change preservation, version format, and secrets (Wraps the autonomic-recipe-maintainer skill /recipe-guidelines; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-guidelines.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-guidelines (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-guidelines/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-recipe-init
|
||||
description: "[recipe-maintainer/cctest] Create a new test instance and recipe-info for a recipe (Wraps the autonomic-recipe-maintainer skill /recipe-init; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-init.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-init (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-init/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-recipe-logging
|
||||
description: "[recipe-maintainer/cctest] Logging instructions for maintaining detailed operation logs in the logs directory (Wraps the autonomic-recipe-maintainer skill /recipe-logging; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-logging.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-logging (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-logging/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-recipe-new-tag
|
||||
description: "[recipe-maintainer/cctest] Bump the recipe version and create an annotated git tag (Wraps the autonomic-recipe-maintainer skill /recipe-new-tag; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-new-tag.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-new-tag (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-new-tag/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-recipe-overview
|
||||
description: "[recipe-maintainer/cctest] Check all maintained recipes and recommend what to upgrade (Wraps the autonomic-recipe-maintainer skill /recipe-overview; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-overview.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-overview (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-overview/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-recipe-review
|
||||
description: "[recipe-maintainer/cctest] Review a recipe for Co-op Cloud best practices (Wraps the autonomic-recipe-maintainer skill /recipe-review; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-review.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-review (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-review/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-recipe-test-all
|
||||
description: "[recipe-maintainer/cctest] Run tests for all maintained recipes, deploying each one at a time (Wraps the autonomic-recipe-maintainer skill /recipe-test-all; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-test-all.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-test-all (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-test-all/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-recipe-test-backup
|
||||
description: "[recipe-maintainer/cctest] Test backing up and restoring a recipe's test instance (Wraps the autonomic-recipe-maintainer skill /recipe-test-backup; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-test-backup.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-test-backup (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-test-backup/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-recipe-test-new
|
||||
description: "[recipe-maintainer/cctest] Test a recipe's first-time initialization from scratch (Wraps the autonomic-recipe-maintainer skill /recipe-test-new; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-test-new.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-test-new (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-test-new/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-recipe-test-update
|
||||
description: "[recipe-maintainer/cctest] Test upgrading a recipe's test instance using abra app deploy (Wraps the autonomic-recipe-maintainer skill /recipe-test-update; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-test-update.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-test-update (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-test-update/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-recipe-test
|
||||
description: "[recipe-maintainer/cctest] Run all tests for a Co-op Cloud recipe (Wraps the autonomic-recipe-maintainer skill /recipe-test; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-test.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-test (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-test/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-recipe-upgrade-apply
|
||||
description: "[recipe-maintainer/cctest] Execute a planned recipe upgrade — apply changes, deploy, test, commit/tag (Wraps the autonomic-recipe-maintainer skill /recipe-upgrade-apply; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-upgrade-apply.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-upgrade-apply (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-upgrade-apply/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-recipe-upgrade-cron-all
|
||||
description: "[recipe-maintainer/cctest] Autonomous weekly upgrade run — overview all recipes, upgrade each end-to-end (sequentially by default, parallel with --parallel), open PRs (Wraps the autonomic-recipe-maintainer skill /recipe-upgrade-cron-all; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-upgrade-cron-all.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-upgrade-cron-all (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-upgrade-cron-all/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-recipe-upgrade-full
|
||||
description: "[recipe-maintainer/cctest] Plan and apply a recipe upgrade end-to-end, no human review in the middle (Wraps the autonomic-recipe-maintainer skill /recipe-upgrade-full; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-upgrade-full.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-upgrade-full (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-upgrade-full/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-recipe-upgrade-plan
|
||||
description: "[recipe-maintainer/cctest] Create a detailed upgrade plan for a recipe (Wraps the autonomic-recipe-maintainer skill /recipe-upgrade-plan; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-upgrade-plan.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-upgrade-plan (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-upgrade-plan/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-recipe-upstream
|
||||
description: "[recipe-maintainer/cctest] From a git.autonomic.zone review-PR URL, fetch the branch + tag locally and emit the commands to open the upstream PR on git.coopcloud.tech (Wraps the autonomic-recipe-maintainer skill /recipe-upstream; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-upstream.)"
|
||||
---
|
||||
|
||||
# cctest-recipe-upstream (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-upstream/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-setup-sandbox
|
||||
description: "[recipe-maintainer/cctest] Guide for setting up a sandboxed environment to run the agent with recipe-maintainer (Wraps the autonomic-recipe-maintainer skill /setup-sandbox; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-setup-sandbox.)"
|
||||
---
|
||||
|
||||
# cctest-setup-sandbox (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/setup-sandbox/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-switch-default-instance
|
||||
description: "[recipe-maintainer/cctest] Switch the default test instance (b1cc or t1cc) for all recipe operations (Wraps the autonomic-recipe-maintainer skill /switch-default-instance; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-switch-default-instance.)"
|
||||
---
|
||||
|
||||
# cctest-switch-default-instance (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/switch-default-instance/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-sync-secrets
|
||||
description: "[recipe-maintainer/cctest] Sync Docker secrets from the test server into recipe-info/testsecrets/ (Wraps the autonomic-recipe-maintainer skill /sync-secrets; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-sync-secrets.)"
|
||||
---
|
||||
|
||||
# cctest-sync-secrets (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/sync-secrets/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-t1cc-start
|
||||
description: "[recipe-maintainer/cctest] Provision the t1cc DigitalOcean test server and deploy Traefik (Wraps the autonomic-recipe-maintainer skill /t1cc-start; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-t1cc-start.)"
|
||||
---
|
||||
|
||||
# cctest-t1cc-start (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/t1cc-start/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-t1cc-stop
|
||||
description: "[recipe-maintainer/cctest] Destroy the t1cc DigitalOcean test server via terraform (Wraps the autonomic-recipe-maintainer skill /t1cc-stop; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-t1cc-stop.)"
|
||||
---
|
||||
|
||||
# cctest-t1cc-stop (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/t1cc-stop/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-test-context-reset
|
||||
description: "[recipe-maintainer/cctest] Undeploy all apps from the test server except traefik (Wraps the autonomic-recipe-maintainer skill /test-context-reset; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-test-context-reset.)"
|
||||
---
|
||||
|
||||
# cctest-test-context-reset (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/test-context-reset/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: cctest-test-setup
|
||||
description: "[recipe-maintainer/cctest] Verify the test environment is configured correctly (Wraps the autonomic-recipe-maintainer skill /test-setup; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-test-setup.)"
|
||||
---
|
||||
|
||||
# cctest-test-setup (cctest wrapper)
|
||||
|
||||
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/test-setup/SKILL.md`
|
||||
— read it and follow it. This wrapper only sets context + policy.
|
||||
|
||||
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
|
||||
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
|
||||
its shared swarm. Execute with the submodule as your working directory:
|
||||
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
|
||||
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
|
||||
run `/cctest-intro` / `/cctest-setup-sandbox` first.
|
||||
|
||||
**Unified policy (same as cc-ci — no differences):**
|
||||
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
|
||||
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
|
||||
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
|
||||
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
|
||||
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
|
||||
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
|
||||
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
|
||||
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
|
||||
@@ -0,0 +1,110 @@
|
||||
---
|
||||
name: help
|
||||
description: Operator orientation - lists every skill available on this orchestrator with what it does and when to reach for it, grouped by purpose (status, weekly maintenance, host updates, tests, enrollment, recovery, reporting), plus a "what do you want to do?" guide for common situations. Read-only. Invoke as /help.
|
||||
---
|
||||
|
||||
# help
|
||||
|
||||
Orient the operator: what this orchestrator can do, via which skill, and what's sensible next.
|
||||
|
||||
**First, enumerate the live roster** (skills get added — don't trust this file's list blindly):
|
||||
```
|
||||
ls /srv/cc-ci-orch/.opencode/skills/ # canonical definitions (one dir per skill)
|
||||
```
|
||||
For any skill not described below, read its `SKILL.md` frontmatter description and include it.
|
||||
Then present the roster grouped as follows, and close with the situation guide.
|
||||
|
||||
## The roster (curated descriptions — merge with the live listing)
|
||||
|
||||
**Status & orientation**
|
||||
- **/cc-ci-status** — the comprehensive read-only health/status check: weekly-run outcomes,
|
||||
report publishing, stale tests, CVE-PR aging, host update recency, service health, bridge
|
||||
`!testme` path. Ends `ALL HEALTHY` or prioritized findings each mapped to a skill. **Start
|
||||
here when unsure.**
|
||||
- **/help** — this orientation.
|
||||
|
||||
**Weekly maintenance (recipes)**
|
||||
- **/upgrade-all** — the weekly sweep: survey every `weekly` recipe, open verified upgrade PRs,
|
||||
write the summary + report. Runs on a timer (`cc-ci-upgrade-all.timer`, Fri 02:00 UTC);
|
||||
invoke manually to run it now.
|
||||
- **/recipe-upgrade <recipe>** — the same pipeline for ONE recipe (plan → bump → verify green →
|
||||
PR). `--with-tests` also fixes that recipe's stale test.
|
||||
- **/recipe-report** — (re)generate the weekly report page for report.ci.commoninternet.net.
|
||||
|
||||
**Keeping the PR list honest**
|
||||
- **/cc-ci-cleanup** — reconciles every mirror from true upstream (which alone closes PRs upstream
|
||||
already merged), then closes the open recipe PRs that can no longer merge or were never meant to
|
||||
(CI sweep artifacts, obsolete bumps, superseded duplicates) and reports what is actually blocking
|
||||
the ones that should land. Never merges.
|
||||
|
||||
**Security (CVEs)**
|
||||
- **/cve-check** — fleet-wide CVE sweep with **no upgrading**: for every recipe, work out what
|
||||
upgrade is available (per image, sidecars included), scan it for CVEs, and publish a CVE report.
|
||||
Read-only and quick — the "what are we exposed to?" answer without an upgrade run.
|
||||
- **/cve-check-and-upgrade** — the same sweep, then open verified PRs **only** for the recipes whose
|
||||
upgrade actually fixes a CVE, worst severity first. `--min-severity high` for just the urgent ones.
|
||||
Never merges.
|
||||
|
||||
**Tests**
|
||||
- **/cc-ci-tests-update** — fleet-wide stale-test cleanup: find tests broken by legitimate
|
||||
upstream changes, fix without weakening, verify, merge the test PRs.
|
||||
- **/ci-test-review** — diagnose a specific red CI run: classify recipe bug vs stale test vs
|
||||
CI-server bug, then fix on the right side.
|
||||
|
||||
**cc-ci itself**
|
||||
- **/cc-ci-server-update** — bump the CI **server** host's nixpkgs/sops-nix, deploy with
|
||||
build → `nixos-rebuild test` → switch + health gate, PR merged on green.
|
||||
- **/cc-ci-orchestrator-update** — same for **this** orchestrator host (self-update caveats).
|
||||
- **/cc-ci-update** — both of the above plus /cc-ci-tests-update in one pass.
|
||||
- **/ci-dev-workflow** — harness/CI-server development discipline (changing cc-ci itself).
|
||||
|
||||
**Enrollment**
|
||||
- **/recipe-enroll <recipe>** — add a NEW recipe to the maintained set end-to-end: mirror,
|
||||
test suite, bridge + inventory enrollment, full-suite-green verification, bridge deploy.
|
||||
|
||||
**Recovery**
|
||||
- **hetzner-server-recovery** — when a Hetzner host is unreachable over SSH/tailscale: API
|
||||
reboot, rescue mode, GRUB generation selection (submenu ids are `1>N`), console access.
|
||||
|
||||
**Recipe-maintainer toolkit (`/cctest-*` — the ARM sandbox + cctest test server, NOT cc-ci)**
|
||||
The full autonomic-recipe-maintainer skill set, vendored as a pinned submodule
|
||||
(`references/recipe-maintainer`) and exposed with the `cctest-` prefix — ~30 skills for
|
||||
hands-on recipe work against the recipe-maintainer **cctest** test server and local abra
|
||||
sandbox: `/cctest-intro` (start here), `/cctest-recipe-overview`, `/cctest-recipe-init`,
|
||||
`/cctest-recipe-deploy`, `/cctest-recipe-test*`, `/cctest-recipe-upgrade-plan|apply|full`,
|
||||
`/cctest-new-recipe-guide`, sandbox/instance management (`/cctest-setup-sandbox`,
|
||||
`/cctest-t1cc-start|stop`), and more — enumerate with `ls .opencode/skills | grep ^cctest-`.
|
||||
**Rule of thumb:** verifying/shipping against the CI pipeline → the cc-ci skills above;
|
||||
exploratory or hands-on recipe development on a test instance → `/cctest-*`. Policy is
|
||||
unified: recipe PRs are never agent-merged on either side (operator reviews + merges), and
|
||||
ARM skills never touch cc-ci infra. After a submodule bump run `scripts/gen-cctest-skills.py`.
|
||||
|
||||
## "What do you want to do?"
|
||||
|
||||
| Situation | Do this |
|
||||
|---|---|
|
||||
| "How is everything?" | `/cc-ci-status` |
|
||||
| "Run the weekly upgrades now" | `/upgrade-all` (or `systemctl start cc-ci-upgrade-all.service`) |
|
||||
| "Upgrade just <recipe>" | `/recipe-upgrade <recipe>` |
|
||||
| "The report site is stale/missing a week" | `/recipe-report` |
|
||||
| "The open PR list is a mess / what should I merge?" | `/cc-ci-cleanup` |
|
||||
| "What CVEs are we exposed to right now?" | `/cve-check` (read-only, no PRs) |
|
||||
| "A CVE just dropped — check and patch it" | `/cve-check-and-upgrade` (add `--min-severity high` to skip the noise) |
|
||||
| "Is <recipe> vulnerable?" | `/cve-check <recipe>` |
|
||||
| "Tests are red because upstream changed" | `/cc-ci-tests-update` (fleet) or `/recipe-upgrade <r> --with-tests` |
|
||||
| "A CI run failed and I don't know why" | `/ci-test-review` |
|
||||
| "Update the CI server OS/deps" | `/cc-ci-server-update` |
|
||||
| "Update this orchestrator's OS/deps" | `/cc-ci-orchestrator-update` |
|
||||
| "Add <recipe> to what we maintain" | `/recipe-enroll <recipe>` |
|
||||
| "A host is unreachable" | `hetzner-server-recovery` |
|
||||
| "Hack on / bootstrap a recipe in a sandbox" | `/cctest-recipe-init`, `/cctest-recipe-deploy`, `/cctest-recipe-test` |
|
||||
| "New to recipe work, where do I start?" | `/cctest-intro` |
|
||||
| "What needs my review?" | `/cc-ci-status` → its open-PR section lists CVE-urgent + ready-to-merge PRs |
|
||||
|
||||
**Standing conventions** (all skills follow these): PRs are opened for visibility and merged
|
||||
directly once verified (invocation = authorization) — except recipe upgrade PRs, which stay
|
||||
operator-merged; `nixos-rebuild test` before any `switch`; never weaken a test; single-writer
|
||||
branches; serialize deploy-heavy work on the shared swarm.
|
||||
|
||||
If several things need doing, run `/cc-ci-status` first — its findings come pre-prioritized
|
||||
with the skill to invoke for each.
|
||||
@@ -0,0 +1,137 @@
|
||||
---
|
||||
name: recipe-enroll
|
||||
description: Add a NEW recipe to cc-ci's maintained set, end to end — create + sync the recipe-maintainers mirror from coopcloud upstream, author a real test suite (health floor + non-vacuous recipe-specific tests incl. a create-an-object round-trip), enroll it in the !testme bridge (POLL_REPOS) and the weekly /upgrade-all inventory (used-recipes.md weekly row + upstream registry), then VERIFY the whole thing: full harness suite green with the new tests, bridge deployed (test-before-switch) and healthy. Opens the cc-ci PR for visibility and merges it directly once verification is green (the skill invocation is the authorization); the report lists merged PR links + what changed. Invoke as /recipe-enroll <recipe>.
|
||||
---
|
||||
|
||||
# recipe-enroll
|
||||
|
||||
Enroll a coopcloud recipe as a **maintained** recipe: mirrored, test-covered, `!testme`-triggerable,
|
||||
and picked up by the weekly `/upgrade-all`. This is the full end-to-end path — worked example:
|
||||
the 2026-08-03 wordpress enrollment (cc-ci PR #14).
|
||||
|
||||
**"Maintained" = ALL of:**
|
||||
1. Mirror `recipe-maintainers/<recipe>` exists, `main` = coopcloud upstream main, tags synced.
|
||||
2. `tests/<recipe>/` enrolled in the cc-ci repo (this is what `/upgrade-all` + `ci-test-review`
|
||||
enumerate).
|
||||
3. `POLL_REPOS` in `nix/modules/bridge.nix` includes the mirror (the `!testme` bridge).
|
||||
4. `cc-ci-plan/used-recipes.md` has a `<recipe> weekly` row (orchestrator repo — `/upgrade-all`
|
||||
skips recipes without it or tagged `external`).
|
||||
5. `cc-ci-plan/upstream/<recipe>.md` registry entry (release-notes URLs + standing notes).
|
||||
|
||||
## Preconditions
|
||||
|
||||
- The recipe exists upstream: `ssh cc-ci 'script -qec "abra recipe fetch <recipe>" /dev/null'`
|
||||
succeeds (every recipe must have a coop-cloud correspondent).
|
||||
- `GITEA_*` creds in `/srv/cc-ci/.testenv` (orchestrator side; the cc-ci host does NOT have this
|
||||
file — inject creds over stdin when running helper scripts there, see step 2).
|
||||
- The shared Swarm is quiescent for step 6 (verification deploys the recipe) — do NOT verify
|
||||
concurrent with `/upgrade-all` or other verify runs; author everything first, verify when clear.
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Survey the recipe (read-only)
|
||||
|
||||
On cc-ci, after `abra recipe fetch <recipe>`, read `~/.abra/recipes/<recipe>/`:
|
||||
- `compose.yml` — services + images (for the upstream registry), healthcheck (`start_period`
|
||||
informs `DEPLOY_TIMEOUT`), traefik labels.
|
||||
- `.env.sample` — is the app self-initializing, or does a fresh deploy sit in a setup wizard
|
||||
(e.g. wordpress without `POST_DEPLOY_CMDS core_install`)? The tests must handle the state a
|
||||
fresh CI deploy actually lands in.
|
||||
- Auth model — how will a test create an object? (REST + token, session login, XML-RPC, …)
|
||||
Check overlay configs (htaccess/nginx templates) for blocked endpoints before relying on one.
|
||||
|
||||
### 2. Create + sync the mirror
|
||||
|
||||
Create `recipe-maintainers/<recipe>` (Gitea API: `POST /orgs/recipe-maintainers/repos`,
|
||||
`{"name":…,"private":true,"default_branch":"main","auto_init":false}`), then force-sync from
|
||||
coopcloud with the existing helper **run on cc-ci with creds injected via stdin** (the host has
|
||||
no `.testenv`):
|
||||
```
|
||||
set -a; . /srv/cc-ci/.testenv; set +a
|
||||
{ printf 'export GITEA_USERNAME=%q GITEA_PASSWORD=%q GITEA_URL=%q\n' "$GITEA_USERNAME" "$GITEA_PASSWORD" "$GITEA_URL";
|
||||
cat /srv/cc-ci-orch/.claude/skills/recipe-upgrade/open-recipe-pr.sh; } \
|
||||
| ssh cc-ci 'bash -s -- <recipe> --reconcile-only'
|
||||
```
|
||||
Expect: repo created (or exists), `main` force-synced to upstream, published tags pushed.
|
||||
|
||||
### 3. Author the test suite (dedicated cc-ci clone + branch)
|
||||
|
||||
```
|
||||
git clone ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /home/loops/work/cc-ci-tests-<recipe>
|
||||
cd … && git checkout -b test/<recipe>-enroll-$(date -u +%Y%m%d)
|
||||
```
|
||||
`tests/<recipe>/` contents (template: `tests/uptime-kuma/`, `tests/wordpress/`):
|
||||
- **`recipe_meta.py`** — `HEALTH_PATH` / `HEALTH_OK` (accept the fresh-deploy state, e.g. a
|
||||
302 to a setup wizard), `DEPLOY_TIMEOUT` (recipe healthcheck `start_period` + DB init +
|
||||
first-boot copy, be generous), `HTTP_TIMEOUT`, `WARM_CANONICAL = True` (canon §2.B — all
|
||||
recipes enroll as data-warm canonicals, operator 2026-06-17).
|
||||
- **`custom/`** — the health floor + **≥2 recipe-specific, non-vacuous tests**, one of which is
|
||||
the §4.3 **create-an-object + read-it-back** (write through the app's real API, read back via
|
||||
a *different* path where possible — e.g. wordpress: XML-RPC write → REST read → permalink
|
||||
HTML). If the app needs setup/auth, put it in a recipe-local `_<recipe>.py` helper
|
||||
(idempotent `ensure_*` so test ordering doesn't matter; run-scoped class-B credentials —
|
||||
the app is destroyed at teardown). Design assertions to name the broken layer (e.g. assert
|
||||
both the rewrite-dependent and rewrite-independent API routes separately).
|
||||
- **`PARITY.md`** — table of tests × what's verified × why non-vacuous; note there's no
|
||||
recipe-maintainer parity corpus if so.
|
||||
- Lint with the **repo dev-shell ruff**: `nix develop -c ruff check tests/<recipe>/ && nix
|
||||
develop -c ruff format tests/<recipe>/` (pre-existing drift in other files is not yours).
|
||||
|
||||
### 4. Bridge enrollment (same branch)
|
||||
|
||||
`nix/modules/bridge.nix`: append `,recipe-maintainers/<recipe>` to the `POLL_REPOS=` CSV.
|
||||
|
||||
### 5. Inventory + registry (orchestrator repo, direct to main)
|
||||
|
||||
- `cc-ci-plan/used-recipes.md`: add `<recipe> weekly` row (alphabetical).
|
||||
- `cc-ci-plan/upstream/<recipe>.md`: images table (source repo + releases/changelog links) +
|
||||
standing notes (setup-wizard behavior, auth caveats, known upgrade traps).
|
||||
- Commit + push (doc commits go direct to main in this repo).
|
||||
|
||||
### 6. Verify end-to-end — full suite GREEN with the new tests (swarm-serialized)
|
||||
|
||||
Open the cc-ci PR first (visibility): `TITLE=… BODY_FILE=… bash
|
||||
/srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh`. The body: what's enrolled,
|
||||
the test design rationale, the verify + deploy plan.
|
||||
|
||||
Then, when the swarm is clear:
|
||||
```
|
||||
ssh cc-ci 'rm -rf /root/cc-ci-test-verify && git clone --branch test/<recipe>-enroll-<date> \
|
||||
ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /root/cc-ci-test-verify && \
|
||||
cd /root/cc-ci-test-verify && git submodule update --init secrets 2>/dev/null || true'
|
||||
RECIPE=<recipe> REMOTE_ROOT=/root/cc-ci-test-verify \
|
||||
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/verify-pr.sh # no REF — recipe main
|
||||
```
|
||||
Required: **cold full-suite green** (install/upgrade/backup/restore/custom). Iterate the tests
|
||||
(bounded, ≤3 attempts) if red — fix the TESTS to match real app behavior, never weaken. Clean up
|
||||
`/root/cc-ci-test-verify` after.
|
||||
|
||||
### 7. Deploy the bridge change to the cc-ci host (test-before-switch)
|
||||
|
||||
The `POLL_REPOS` change only takes effect via a host rebuild. Per `/cc-ci-server-update` steps
|
||||
5a-e: stage the branch to `/root/cc-ci-deploy` (+ secrets copy), `nixos-rebuild build`, detached
|
||||
`nixos-rebuild test`, verify reachable + healthy, then `switch`. Confirm the bridge is polling
|
||||
the new repo: `ssh cc-ci 'docker service inspect ccci-bridge_app --format "{{json
|
||||
.Spec.TaskTemplate.ContainerSpec.Env}}"' | grep <recipe>` and the bridge task is 1/1 with no
|
||||
auth errors in its logs (a stale Gitea secret 401s silently — see the 2026-08-03 finding).
|
||||
|
||||
### 8. Merge + report
|
||||
|
||||
Merge the cc-ci PR (invocation = authorization; PR is the visible record — comment the
|
||||
verification evidence first). Report to the operator: merged PR link + change summary, the
|
||||
verify log path, and the note that the **next weekly `/upgrade-all` picks the recipe up
|
||||
automatically** (it enumerates `tests/<recipe>/` dirs × `weekly` rows).
|
||||
|
||||
## Guardrails
|
||||
|
||||
- **Full-suite green is the enrollment gate** — an enrolled-but-red recipe poisons every future
|
||||
sweep. Don't merge on partial green.
|
||||
- **Never weaken**: the tests assert the app's real current behavior, incl. asserting removed
|
||||
auth paths are rejected where that's the upstream intent.
|
||||
- **Single-writer**: dedicated clones/branches; never push `main` of cc-ci; never touch
|
||||
`/root/builder-clone` or the loops' clones; `/root/cc-ci-test-verify` is yours — remove after.
|
||||
- **Serialize on the swarm**: authoring is free, verification + bridge deploy wait for
|
||||
`/upgrade-all`/other runs to finish.
|
||||
- **abra over ssh needs the pseudo-TTY wrap**: `ssh cc-ci 'script -qec "abra …" /dev/null'`.
|
||||
- **PRs for visibility, merged directly once verified**; failed enrollment leaves the PR open
|
||||
with an explanatory comment and the report says exactly what's missing.
|
||||
@@ -30,15 +30,18 @@ the orchestrator watches from outside.
|
||||
|
||||
Reboot resilience is handled by **`cc-ci-loops.service`** (system unit): on boot it logs the reboot
|
||||
to `REBOOTS.md` (boot_id-gated) and runs `launch.sh start` with `RESUME_PHASE=1`, so the loops +
|
||||
watchdog auto-resume the saved phase. The orchestrator session itself is NOT auto-started — the
|
||||
operator reconnects to it (that's why the startup notification matters). The orchestrator now runs on
|
||||
a **Hetzner `cpx22`** cloud server (`cc-ci-orchestrator-1`, tailnet `100.84.190.30`, public
|
||||
`168.119.126.100`, flake host `cc-ci-orchestrator-hetzner`) — see
|
||||
`cc-ci-plan/plan-orchestrator-hetzner-migration.md`. The earlier Pi→Incus-VM move is the historical
|
||||
`cc-ci-plan/plan-orchestrator-migration.md`. Rebuild this host with
|
||||
`nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner` from `/srv/cc-ci-orch` — but **always
|
||||
watchdog auto-resume the saved phase. The orchestrator session itself is relaunched by
|
||||
`cc-ci-orchestrator.service` (`agents.py up orchestrator`) — the operator reconnects to it (that's
|
||||
why the startup notification matters). Since 2026-09 the orchestrator runs on the **same Hetzner
|
||||
host as the cc-ci CI server** (`cc-ci`, public `195.201.88.249`, tailnet `cc-ci`), declared by
|
||||
`nixosConfigurations.cc-ci` in this repo's `flake.nix`, which imports the CI server from the cc-ci
|
||||
repo's `nixosModules.cc-ci-server`. `ssh cc-ci` from the loops user therefore goes to loopback.
|
||||
The full provisioning + deploy guide is `README.md`; the move is recorded in
|
||||
`cc-ci-plan/plan-cc-ci-combined-host.md`; the previous hosts (Pi → Incus VM → Hetzner `cpx22`
|
||||
shared with notplants) are in `archive/`. Rebuild this host with
|
||||
`nixos-rebuild switch --flake .#cc-ci` from `/srv/cc-ci-orch` — but **always
|
||||
`nixos-rebuild test` the same flake target first and verify the host is still healthy/reachable
|
||||
before the `switch`** (general policy for nix deploys to this host and the cc-ci server: `test`
|
||||
before the `switch`** (general policy for nix deploys to this host: `test`
|
||||
leaves the bootloader and system profile untouched, so a reboot always recovers to the
|
||||
last-known-good generation; the 2026-08-03 cc-ci 26.05 bump outage is the cautionary tale, see
|
||||
`.cc-ci-logs/server-update-2026-08-03.md`).
|
||||
@@ -115,3 +118,29 @@ When the orchestrator, Builder, or assistant makes intentional repository change
|
||||
promptly and push them to `git.autonomic.zone` in append-only fashion (never force-push). Match the
|
||||
existing commit author and message style in this repo. Do not bundle unrelated worktree changes you
|
||||
did not make; stage only the intended files.
|
||||
|
||||
## Ship as PRs, merge them yourself, operator reviews retrospectively
|
||||
|
||||
**This applies to the two INFRASTRUCTURE repos — `recipe-maintainers/cc-ci-orchestrator` (here) and
|
||||
`recipe-maintainers/cc-ci` (the CI product).** For work in either:
|
||||
|
||||
1. Branch, don't commit straight to `main`.
|
||||
2. Open a PR with a description written to be read **after** the fact: what changed, why, and what
|
||||
evidence says it works (test output, a verified run, a before/after number). The PR *is* the
|
||||
review artifact and the historical record.
|
||||
3. **Merge it yourself once it is verified** — do not wait for review. The invocation is the
|
||||
authorization; blocking on review would stall the pipeline these repos exist to run.
|
||||
4. The operator reviews **retrospectively**, from the PR.
|
||||
|
||||
So the PR is not a gate — it is how the work stays legible. A PR that merely says "fix scanner" has
|
||||
failed at its only job.
|
||||
|
||||
> ### This does NOT extend to RECIPE repos
|
||||
> Recipe PRs — any `coop-cloud/<recipe>` or its `recipe-maintainers/<recipe>` mirror — are
|
||||
> **created and verified but NEVER merged by an agent**. Those change what deploys on other people's
|
||||
> infrastructure, so a human merges them. The split is deliberate: agents own the tooling, the
|
||||
> operator owns the recipes.
|
||||
|
||||
If work has already landed on `main` without a PR, do not rewrite published history to fix it.
|
||||
Create a branch pinned at the pre-work commit and open the PR against that, so the diff is still
|
||||
reviewable and merging only advances the pointer (see PRs #2-#5, 2026-08-11).
|
||||
|
||||
@@ -1,47 +1,344 @@
|
||||
# cc-ci-orchestrator
|
||||
|
||||
Orchestrator workspace for building the **cc-ci** Co-op Cloud recipe CI server. The plan, launch
|
||||
tooling, and loop prompts live in [`cc-ci-plan/`](cc-ci-plan/); see [`AGENTS.md`](AGENTS.md) for the
|
||||
roles and operating model. Secrets (`.testenv`) are gitignored — never commit them.
|
||||
The **cc-ci orchestrator**: the agent loops that built the cc-ci Co-op Cloud recipe CI server,
|
||||
the operator's steering session, and the weekly autonomous recipe-upgrade run — plus the NixOS
|
||||
host they run on. Since 2026-09 that host is **the same Hetzner server as the CI server itself**:
|
||||
one `nixos-rebuild` from this repo builds both, because this flake imports the CI server as a
|
||||
module from the [cc-ci](https://git.autonomic.zone/recipe-maintainers/cc-ci) repo.
|
||||
|
||||
## Run the orchestrator in tmux (survives disconnects + closing your laptop)
|
||||
| | where |
|
||||
|---|---|
|
||||
| Orchestrator loops, timers (weekly upgrader, hourly supervisor) | `nix/modules/cc-ci.nix` → `nixosModules.cc-ci-orchestrator` |
|
||||
| The host contract those need (loops user, claude/opencode CLIs, opencode web UI) | `nix/modules/orchestrator-host.nix` → `nixosModules.orchestrator-host` |
|
||||
| The CI server (swarm, traefik, drone, runner, `!testme` bridge, dashboard, reports, acme-dns) | cc-ci repo `nix/modules/` → `nixosModules.cc-ci-server` (flake input `cc-ci`) |
|
||||
| The machine: hardware, networking, tailscale, root keys | `nix/hosts/cc-ci/` → `nixosConfigurations.cc-ci` |
|
||||
| Plans, launch tooling, loop prompts, journal | `cc-ci-plan/` (see `AGENTS.md` for roles) |
|
||||
| Skills the orchestrator runs (`/upgrade-all`, `/recipe-upgrade`, `/cc-ci-status`, …) | `.claude/skills/`, `.opencode/skills/` |
|
||||
| How it used to be built (Pi → Incus VM → shared Hetzner box) | `archive/` |
|
||||
|
||||
Keep this supervising session alive on the host with tmux, and use `--remote-control` so you can
|
||||
watch/steer it from **claude.ai/code** (or the mobile app).
|
||||
Secrets (`.testenv`, `upgrader.env`, `.sops/`, everything under `/secrets`) are gitignored — never
|
||||
commit them.
|
||||
|
||||
---
|
||||
|
||||
# Deploying a cc-ci host from scratch
|
||||
|
||||
This is the whole path from "nothing" to a working CI server + orchestrator on one Hetzner
|
||||
server. It was last done on 2026-09-07 for `195.201.88.249` and is written so a person or an LLM
|
||||
can repeat it. Read it once before starting; the order matters.
|
||||
|
||||
## 0. What you need in hand
|
||||
|
||||
- A **Hetzner Cloud** project you can create servers in (console login or an API token).
|
||||
- **SSH keys**: yours, and the orchestrator's own key so the automation can reach the box. The
|
||||
public keys that get root are tracked in `nix/hosts/cc-ci/ssh-keys` (one per line).
|
||||
- Read access to `recipe-maintainers/cc-ci`, `recipe-maintainers/cc-ci-orchestrator` (both public
|
||||
read) and the **private** `recipe-maintainers/cc-ci-secrets` (the `autonomic-bot` deploy key,
|
||||
`autonomic-bot-gitea-ed25519`, has it).
|
||||
- The out-of-band secrets listed in §4. If you are migrating, they come from the old host; if
|
||||
you are starting fresh you create them (each row says how).
|
||||
- Control of the DNS zone (Gandi for `commoninternet.net`) for the cutover in §7.
|
||||
|
||||
## 1. Provision the server on Hetzner (Debian image)
|
||||
|
||||
In the Hetzner Cloud console (or with `hcloud server create`):
|
||||
|
||||
| setting | value | why |
|
||||
|---|---|---|
|
||||
| Image | **Debian 13** (any recent Debian/Ubuntu works with nixos-infect) | it is replaced by NixOS in §2 |
|
||||
| Type | **x86**, **8 GB RAM**, 4 vCPU — e.g. `cpx32` (dedicated AMD) or `cx33`. **Never `cax*`** (ARM): the flakes are `x86_64-linux`. | swarm + recipe deploys + 3–6 agent sessions; 4 GB is too small |
|
||||
| Disk | the type's default 150+ GB NVMe | docker layers alone are ~60 GB after a few weeks |
|
||||
| Network | public **IPv4** required; IPv6 optional (leave enabled or not, NixOS config ignores it) | cc-ci serves 80/443 and DNS on 53 publicly |
|
||||
| SSH keys | add every key from `nix/hosts/cc-ci/ssh-keys` you want to log in with, at least the orchestrator's | nixos-infect carries `/root/.ssh/authorized_keys` over |
|
||||
| Name | `cc-ci` | becomes the hostname |
|
||||
| Firewall | if a Hetzner Cloud Firewall is attached it must allow **22/tcp, 80/tcp, 443/tcp, 53/tcp, 53/udp** in, and ICMP | the NixOS firewall is separate and is configured by the flake |
|
||||
|
||||
Check you can log in: `ssh root@<ip> hostname`.
|
||||
|
||||
## 2. Convert Debian → NixOS with nixos-infect
|
||||
|
||||
[nixos-infect](https://github.com/elitak/nixos-infect) installs NixOS over the running Debian and
|
||||
reboots. Run it detached so the SSH session dropping does not kill it:
|
||||
|
||||
```bash
|
||||
# 0. Exit any running orchestrator session first — a conversation can't be resumed while it's live:
|
||||
# /exit (inside Claude) or Ctrl-D
|
||||
|
||||
# 1. Start a detachable tmux session on this host
|
||||
tmux new -s orchestrator
|
||||
|
||||
# 2. Inside tmux, resume the orchestrator conversation WITH remote control:
|
||||
claude --resume autonomous-orchestrator \
|
||||
--remote-control "autonomous-orchestrator" \
|
||||
--dangerously-skip-permissions
|
||||
# - If name-resume opens a picker instead of resuming directly, choose "autonomous-orchestrator".
|
||||
# - Or resume by the stable session id (more deterministic in a fresh pane):
|
||||
# claude --resume 34a80a99-b37e-4809-b8da-ccc9fafe785e \
|
||||
# --remote-control "autonomous-orchestrator" --dangerously-skip-permissions
|
||||
|
||||
# 3. Detach — the process keeps running: press Ctrl-b, then d
|
||||
ssh root@<ip> 'cat > /root/infect.sh <<"EOF"
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
# Pinned nixos-infect revision (same one that built the previous cc-ci hosts).
|
||||
INFECT_SHA="40f62a680bb0e8f2f607d79abfaaecd99d59401c"
|
||||
export NIX_CHANNEL="nixos-26.05" # must match the nixpkgs channel in flake.nix
|
||||
export PROVIDER="hetznercloud" # GRUB + Hetzner networking
|
||||
export NIXOS_IMPORT="" # the real config comes from the flake in §5
|
||||
# The Debian 13 cloud image mounts /tmp as tmpfs; nixos-infect makes a temporary swapfile
|
||||
# there and swapon fails with "Invalid argument". 8 GB RAM needs no extra swap: skip it.
|
||||
export NO_SWAP=true
|
||||
curl -fsSL "https://raw.githubusercontent.com/elitak/nixos-infect/${INFECT_SHA}/nixos-infect" | bash -x
|
||||
EOF
|
||||
chmod +x /root/infect.sh
|
||||
nohup /root/infect.sh > /var/log/nixos-infect.log 2>&1 &'
|
||||
```
|
||||
|
||||
**Reconnect later**
|
||||
- On this host: `tmux attach -t orchestrator`
|
||||
- From anywhere: **claude.ai/code** → the `autonomous-orchestrator` session
|
||||
It downloads Nix, builds a NixOS system (5–10 min; follow with
|
||||
`ssh root@<ip> tail -f /var/log/nixos-infect.log`), then reboots. The SSH host key changes:
|
||||
`ssh-keygen -R <ip>` and confirm `ssh root@<ip> nixos-version` prints a 26.05 version.
|
||||
|
||||
**Why it survives:** tmux keeps the `claude` process alive across SSH disconnects and your laptop
|
||||
closing; remote-control runs *outbound* from this host to Anthropic, so it stays connected
|
||||
regardless of the viewer. After a host reboot, re-run steps 1–2.
|
||||
### 2a. What went wrong on 2026-09-07, and the fixes (Debian 13 image, NixOS 26.05)
|
||||
|
||||
> Two different "names": `--resume <name|id>` selects the *conversation* to restore (shown in the
|
||||
> `/resume` picker); the `--remote-control "<name>"` value is only the web display label and resumes
|
||||
> nothing. Resuming reuses the same session id each time (stays `34a8…`) — don't pass
|
||||
> `--fork-session` unless you intend to branch a new conversation.
|
||||
>
|
||||
> Already inside a live session and just want the web surface? Run `/remote-control` — no exit/resume.
|
||||
All three bit on the first attempt; the script above and §3 already include the fixes, this is
|
||||
so you recognise them if they come back in another form.
|
||||
|
||||
1. **`swapon: /tmp/nixos-infect.XXXX.swp: Invalid argument`** right at the start, script exits.
|
||||
The Debian 13 cloud image mounts `/tmp` as tmpfs and a swapfile cannot live there.
|
||||
Fix: `NO_SWAP=true` (in the script above). An 8 GB box does not need the temporary swap.
|
||||
2. **The box never comes back after the reboot: it boots NixOS, but nearly every unit fails**
|
||||
(`dbus`, `systemd-logind`, `sshd`, networking …) with
|
||||
`Could not start dynamically linked executable: /usr/bin/dbus-daemon` in the journal.
|
||||
nixos-infect leaves the old Debian root in place and relies on NixOS's first boot to move it
|
||||
to `/old-root` (`/etc/NIXOS_LUSTRATE`). With NixOS 26.05's systemd-based initrd that
|
||||
lustration did not happen, so Debian's `/etc/systemd/system/*.service` files shadowed the
|
||||
NixOS units and started Debian binaries. Fix, from Hetzner **rescue mode**
|
||||
(`enable_rescue` + `reset` in the API/console, ssh in, `mount /dev/sda1 /mnt/root`):
|
||||
move everything except `nix`, `boot`, `swapfile`, `lost+found`, `var/log`, `var/empty`,
|
||||
`etc/nixos`, `etc/resolv.conf`, `etc/NIXOS`, `etc/machine-id`, `etc/ssh/ssh_host_*`,
|
||||
`root/.nix-*`, `root/.ssh` into `/mnt/root/old-root`, delete `etc/NIXOS_LUSTRATE`, unmount,
|
||||
`disable_rescue`, `reset`. (`/old-root`, ~1 GB, can be deleted once the host is in service.)
|
||||
3. **Boots, units fine, but no network.** The generated `networking.nix` has
|
||||
`defaultGateway = "172.31.1.1";` — a bare string. Since NixOS 25.05 that yields no default
|
||||
route. Fix: `defaultGateway = { address = "172.31.1.1"; interface = "eth0"; };` (this is what
|
||||
`nix/hosts/cc-ci/networking.nix` carries). To apply it from rescue mode, chroot into the
|
||||
mounted root and rebuild the boot entry — the nix sandbox cannot `pivot_root` inside a chroot,
|
||||
so turn it off for that one build:
|
||||
```bash
|
||||
for d in proc sys dev dev/pts; do mount --bind /$d /mnt/root/$d; done
|
||||
mount -t tmpfs tmpfs /mnt/root/run; cp -L /etc/resolv.conf /mnt/root/etc/resolv.conf
|
||||
chroot /mnt/root /nix/var/nix/profiles/system/sw/bin/bash -c '
|
||||
export PATH=/nix/var/nix/profiles/system/sw/bin NIX_REMOTE= HOME=/root
|
||||
export NIX_PATH=nixos-config=/etc/nixos/configuration.nix:nixpkgs=/root/.nix-defexpr/channels/nixos
|
||||
ln -sfn /nix/var/nix/profiles/system /run/current-system
|
||||
nixos-rebuild boot --option sandbox false'
|
||||
```
|
||||
The `journalctl -D /mnt/root/var/log/journal -b 0` trick (reading the dead system's journal
|
||||
from rescue mode) is what told these apart.
|
||||
|
||||
> Rescue mode without a console: `POST /servers/<id>/actions/enable_rescue` with your ssh key
|
||||
> id, then `…/actions/reset`; afterwards `disable_rescue` **and check `rescue_enabled` is false
|
||||
> before** the next `reset`, or it boots the rescue image again. `scripts/recovery/hetzner.py`
|
||||
> wraps these (token in `/srv/cc-ci/.hcloud-token`).
|
||||
|
||||
## 3. Capture the machine-specific config into this repo
|
||||
|
||||
nixos-infect wrote `/etc/nixos/{hardware-configuration,networking,configuration}.nix`. Only the
|
||||
first two matter; the flake replaces `configuration.nix`.
|
||||
|
||||
```bash
|
||||
scp root@<ip>:/etc/nixos/hardware-configuration.nix nix/hosts/cc-ci/hardware.nix
|
||||
scp root@<ip>:/etc/nixos/networking.nix nix/hosts/cc-ci/networking.nix
|
||||
```
|
||||
|
||||
Then in `nix/hosts/cc-ci/`:
|
||||
|
||||
- `hardware.nix`: keep as generated (GRUB EFI with `efiInstallAsRemovable`, `/boot/efi` by UUID,
|
||||
`/dev/sda1` root). Do not copy another host's file — the UUIDs are per machine.
|
||||
- `networking.nix`: keep the static IPv4 + Hetzner gateway `172.31.1.1`. Make sure
|
||||
`networking.defaultGateway` has **both** `address` and `interface = "eth0"` (§2a item 3). If
|
||||
the generated IPv6 block has an empty address, delete the IPv6 parts; a real global address
|
||||
(as on the 2026-09 box) can stay.
|
||||
- `configuration.nix`: set `cc-ci.publicIPv4` to the server's IPv4 and check `system.stateVersion`
|
||||
is the release you installed (never change it later).
|
||||
- `ssh-keys`: the root keys.
|
||||
|
||||
Commit on a branch; the rebuild in §5 can use the local checkout before the PR merges.
|
||||
|
||||
## 4. Stage the workspace and secrets on the new host
|
||||
|
||||
Everything in this section is **outside git**. Do it as root over SSH, in this order.
|
||||
|
||||
### 4a. Tailscale
|
||||
|
||||
```bash
|
||||
# a reusable (or fresh) tailnet auth key from the tailscale admin console
|
||||
install -m600 /dev/stdin /etc/ts-auth-key <<<'tskey-auth-…'
|
||||
```
|
||||
|
||||
### 4b. The CI server's checkout and its one out-of-band secret
|
||||
|
||||
```bash
|
||||
# root's deploy key for the private cc-ci-secrets submodule
|
||||
install -d -m700 /root/.ssh
|
||||
install -m600 <autonomic-bot-gitea-ed25519> /root/.ssh/autonomic-bot-gitea-ed25519
|
||||
cat > /root/.ssh/config <<'EOF'
|
||||
Host git.autonomic.zone
|
||||
Port 2222
|
||||
User git
|
||||
IdentityFile /root/.ssh/autonomic-bot-gitea-ed25519
|
||||
IdentitiesOnly yes
|
||||
EOF
|
||||
# the deployed checkout: nightly-sweep runs from it, sops reads secrets/secrets.yaml from it
|
||||
git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci.git /etc/cc-ci
|
||||
# the master (recovery) age key — the only sops recipient a fresh host can be
|
||||
install -d -m700 /var/lib/sops-nix
|
||||
install -m600 <master-age.txt> /var/lib/sops-nix/key.txt
|
||||
```
|
||||
|
||||
`/etc/cc-ci/secrets/secrets.yaml` is encrypted to the master key and the *old* host's SSH host
|
||||
key. That is enough to deploy. Afterwards (optional, tidier) add the new host as a recipient:
|
||||
`ssh-to-age < /etc/ssh/ssh_host_ed25519_key.pub`, add it to `secrets/.sops.yaml` in cc-ci-secrets,
|
||||
`sops updatekeys secrets.yaml`, push, `git -C /etc/cc-ci submodule update --remote`.
|
||||
|
||||
### 4c. The orchestrator's workspace (as the `loops` user — it exists after the first rebuild, so
|
||||
run §5 once first if this is a fresh host, then come back)
|
||||
|
||||
```bash
|
||||
sudo -iu loops
|
||||
git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci-orchestrator.git /srv/cc-ci-orch
|
||||
sudo ln -sfn /srv/cc-ci-orch /srv/cc-ci # every script and unit says /srv/cc-ci
|
||||
cd /srv/cc-ci-orch
|
||||
git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci # Builder clone
|
||||
git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci-adv # Adversary clone
|
||||
mkdir -p .cc-ci-logs .sops
|
||||
```
|
||||
|
||||
Then the files below (`install -m600 -o loops -g users`):
|
||||
|
||||
| file | what | source |
|
||||
|---|---|---|
|
||||
| `/srv/cc-ci/.testenv` | `TS_AUTH_KEY`, `GITEA_PASSWORD` (autonomic-bot), `DOCKERHUB_USERNAME/TOKEN`, model API keys | old host `/secrets/files/cc-ci.testenv`; fresh: create each credential |
|
||||
| `/srv/cc-ci/upgrader.env` | `LOOP_TIER`, `LOOP_MODEL`, `REPORT_MODEL` for the weekly run (no secrets) | old host, or copy the example in `AGENTS.md` |
|
||||
| `/srv/cc-ci/.sops/master-age.txt` | the same master age key as 4b (skills that re-key secrets use it) | old host |
|
||||
| `~loops/.ssh/cc-ci-root-ed25519` (+`.pub`) | `ssh cc-ci` as root — to loopback on this host | old host; fresh: `ssh-keygen -t ed25519` and add the pub to `nix/hosts/cc-ci/ssh-keys` |
|
||||
| `~loops/.ssh/autonomic-bot-gitea-ed25519` (+`.pub`) | pushes recipe branches / PRs as `autonomic-bot` | old host; fresh: new key added to the bot's Gitea account |
|
||||
| `~loops/.ssh/tangled-ed25519` | optional, tangled.org mirrors | old host |
|
||||
| `~loops/.claude/` | Claude Code auth + settings + the orchestrator session history | old host (`rsync -a`); fresh: `claude auth login` as loops (device code, interactive) |
|
||||
| `~loops/.local/share/opencode/auth.json`, `~loops/.config/opencode/` | opencode provider auth (the weekly upgrader runs on opencode) | old host; fresh: `opencode auth login` |
|
||||
| `/etc/nginx/oc-selfsigned.{crt,key}`, `/etc/nginx/oc-htpasswd` | the tailnet-only opencode UI; **nginx refuses to start without them**, and its config check runs as the `nginx` user, so: `root:nginx`, crt `0644`, key + htpasswd `0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host, or generate (commands in `nix/modules/orchestrator-host.nix`) |
|
||||
|
||||
`~loops/.ssh/config` is written by the activation script on first rebuild (`Host cc-ci` →
|
||||
`127.0.0.1`, `git.autonomic.zone`, `tangled.org`); it is not overwritten if present.
|
||||
|
||||
## 5. Build and activate
|
||||
|
||||
From the checkout with the §3 commit (root can build from the loops-owned checkout via sudo):
|
||||
|
||||
```bash
|
||||
# as root, detached (the activation restarts sshd/tailscale; a dropped session must not kill it).
|
||||
# Three things the FIRST rebuild on a bare infect system needs, none of which the converged
|
||||
# host needs afterwards: `git` on PATH (nix's flake fetcher shells out to it and the infect
|
||||
# system has none — hence nix-shell), HOME=/root (so root's `git config --global
|
||||
# safe.directory '*'` applies to the loops-owned checkout), and a login shell (`bash -l`, for
|
||||
# NIX_SSL_CERT_FILE and friends from /etc/set-environment).
|
||||
git config --global --add safe.directory '*'
|
||||
systemd-run --unit=ccci-rebuild --collect -E HOME=/root -p WorkingDirectory=/srv/cc-ci-orch \
|
||||
bash -lc 'nix-shell -p git --run "nixos-rebuild test --flake /srv/cc-ci-orch#cc-ci"'
|
||||
journalctl -fu ccci-rebuild # ~10 min the first time (image pulls + two OCI image builds)
|
||||
```
|
||||
|
||||
`test` first, always: it activates WITHOUT touching the bootloader, so if the activation breaks
|
||||
networking or sshd a reboot from the Hetzner console lands on the last known-good generation.
|
||||
Later rebuilds are simply `sudo nixos-rebuild test|switch --flake .#cc-ci` from the checkout.
|
||||
|
||||
The first activation takes a while: it pulls the traefik/drone/keycloak images, builds the bridge
|
||||
and dashboard OCI images with Nix, initialises the swarm and runs the serialized reconcile
|
||||
oneshots (`swarm-init → deploy-proxy → deploy-drone → deploy-bridge → deploy-dashboard →
|
||||
deploy-reports`, `deploy-backupbot`, `warm-keycloak`). Verify:
|
||||
|
||||
```bash
|
||||
systemctl is-system-running # running — or list-units --failed and read journalctl -u <unit>
|
||||
tailscale status | head -3
|
||||
docker service ls # traefik app+socket-proxy, drone, bridge, dashboard, reports, backups: 1/1
|
||||
systemctl status cc-ci-loops cc-ci-orchestrator opencode-web nginx acme-dns
|
||||
systemctl list-timers 'cc-ci-*' nightly-sweep
|
||||
sudo -iu loops tmux ls # cc-ci-orchestrator (+ loops sessions if a phase is active)
|
||||
# the CI front doors, before DNS points here (expect 200 / 200 / 303 and ssl_verify=0 once
|
||||
# /var/lib/acme is restored or a cert has been issued):
|
||||
curl -s --resolve ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://ci.commoninternet.net/
|
||||
curl -s --resolve report.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code}\n' https://report.ci.commoninternet.net/
|
||||
curl -s --resolve drone.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code}\n' https://drone.ci.commoninternet.net/
|
||||
dig +short @<ip> ns-acme.commoninternet.net # acme-dns answering on the public 53
|
||||
```
|
||||
|
||||
Seen on 2026-09-07: `tailscaled-autoconnect` failed with `invalid key: API key does not exist` —
|
||||
the reusable auth key had been revoked. Generate a fresh one in the tailscale admin console, put
|
||||
it in `/etc/ts-auth-key`, `systemctl restart tailscaled-autoconnect`. Nothing else depends on it
|
||||
during the install; the box is reachable on its public IP throughout.
|
||||
|
||||
When it is healthy: `sudo nixos-rebuild switch --flake .#cc-ci` (same config, now also the boot
|
||||
default). **If you are migrating from another host, do §6 before letting it serve anything**: right
|
||||
after the first activation scale the `!testme` bridge to 0 and mask the two orchestrator timers so
|
||||
the new box does not process PR comments or start a second weekly run while the old host is live:
|
||||
|
||||
```bash
|
||||
docker service scale ccci-bridge_app=0
|
||||
systemctl mask --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer
|
||||
```
|
||||
|
||||
## 6. Migrating: restore state from the previous host
|
||||
|
||||
Over tailscale (`rsync -aHAX --numeric-ids root@<old>:<path> <path>`), with the matching service
|
||||
stopped on the new host while its directory is copied:
|
||||
|
||||
| path | holds | notes |
|
||||
|---|---|---|
|
||||
| `/var/lib/cc-ci-reports` | the published weekly report pages (`report.ci…`) | |
|
||||
| `/var/lib/cc-ci-runs` | per-run artifacts the dashboard shows | |
|
||||
| `/var/lib/ci-warm` | warm-canonical state + alerts | recipe warm *volumes* are caches: not copied, rebuilt by the Sunday sweep / first use |
|
||||
| `/var/lib/acme` | the Let's Encrypt cert + account **and `acme-dns-accounts.json`** — the account the permanent `_acme-challenge` CNAME points at | without it a fresh registration + a new CNAME at Gandi is needed (registration is disabled in `acme-dns.nix`) |
|
||||
| `/var/lib/acme-dns` | the acme-dns zone DB | |
|
||||
| `/var/lib/ci-certs` | the copy traefik is handed | then `systemctl restart cc-ci-acme-traefik-handoff` |
|
||||
| `/root/.abra` | abra's per-app env files for the deployed stacks | |
|
||||
| Drone data volume `/var/lib/docker/volumes/drone_ci_commoninternet_net_data` | Drone's DB: the Gitea OAuth grant, repo activation, build history | `docker service scale drone_ci_commoninternet_net_app=0` on the new host, copy, scale back to 1. Otherwise run `scripts/bootstrap-drone-oauth.sh` (cc-ci repo) with the bot password and re-activate repos |
|
||||
| `/srv/cc-ci-orch/.cc-ci-logs`, `/srv/cc-ci-orch/cc-ci-plan/upstream/`, `REBOOTS.md`, `JOURNAL.md` | orchestrator history, the upgrader's per-recipe release-note registry | as loops; do the final sync after stopping the orchestrator on the old host |
|
||||
|
||||
## 7. Cutover and verification
|
||||
|
||||
1. **DNS** (operator, Gandi zone `commoninternet.net`): A records `ci`, `*.ci` and `ns-acme` →
|
||||
the new IPv4. `acme NS ns-acme` and `_acme-challenge.ci CNAME <account>.acme…` stay as they
|
||||
are. Wait for propagation (`dig +short ci.commoninternet.net`).
|
||||
2. Check the new host answers on the new IP before DNS moves: `dig @<new-ip> ns-acme.commoninternet.net`
|
||||
(acme-dns), `curl --resolve ci.commoninternet.net:443:<new-ip> https://ci.commoninternet.net/`
|
||||
(dashboard, valid cert), same for `report.ci` and `drone.ci`.
|
||||
3. Old host: `docker service scale ccci-bridge_app=0 drone_ci_commoninternet_net_app=0`;
|
||||
`systemctl disable --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer` on the old
|
||||
orchestrator. New host: `docker service scale ccci-bridge_app=1`;
|
||||
`systemctl unmask cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer && systemctl start` both.
|
||||
4. End to end: post `!testme` on an open recipe PR and watch it turn green on the new Drone;
|
||||
open `https://ci.commoninternet.net` and `https://report.ci.commoninternet.net`.
|
||||
5. The orchestrator: as loops on the new host `cd /srv/cc-ci-orch && python3 cc-ci-plan/agents.py up orchestrator`
|
||||
(or just `systemctl restart cc-ci-orchestrator`), attach with `claude --resume` or from
|
||||
claude.ai/code. Its startup routine (AGENTS.md) reports phase + reboot count.
|
||||
6. Keep the old host as a cold standby for a week, then delete it and its tailnet node.
|
||||
|
||||
## 8. Day 2
|
||||
|
||||
- **Update the host** (nixpkgs bump for both halves): `/cc-ci-orchestrator-update`, which is
|
||||
`nix flake update` → `nixos-rebuild test` → verify → `switch` → PR. The `cc-ci` input follows
|
||||
this flake's nixpkgs, so the CI server is rebuilt on the same nixpkgs.
|
||||
- **Update only cc-ci's code** (harness/tests/modules): merge in the cc-ci repo, then
|
||||
`nix flake update cc-ci` here and rebuild; also `git -C /etc/cc-ci pull --recurse-submodules`
|
||||
so the deployed checkout the sweep runs from matches.
|
||||
- **Something is down**: `systemctl --failed`, `journalctl -u deploy-<x>`, `docker service ps <svc>`;
|
||||
the cc-ci repo's `docs/runbook.md`. Host unreachable: Hetzner console → reboot lands on the last
|
||||
`switch`ed generation; rescue mode + `nixos-enter` for anything worse (skill
|
||||
`/hetzner-server-recovery`).
|
||||
|
||||
---
|
||||
|
||||
# Operating the orchestrator session
|
||||
|
||||
The steering session is a long-lived interactive Claude Code session under tmux with
|
||||
`--remote-control`, so it can be watched and steered from **claude.ai/code** (or the mobile app).
|
||||
`cc-ci-orchestrator.service` relaunches it on boot via `cc-ci-plan/agents.py up orchestrator`
|
||||
(backend + model in `cc-ci-plan/agents.toml`).
|
||||
|
||||
```bash
|
||||
# attach on the host
|
||||
sudo -iu loops tmux attach -t cc-ci-orchestrator
|
||||
# or resume the conversation by hand in a fresh tmux pane
|
||||
claude --resume autonomous-orchestrator --remote-control "autonomous-orchestrator" --dangerously-skip-permissions
|
||||
# already inside a live session and just want the web surface? /remote-control
|
||||
```
|
||||
|
||||
`--resume <name|id>` selects the *conversation* to restore; the `--remote-control "<name>"` value is
|
||||
only the web display label. Don't pass `--fork-session` unless you mean to branch.
|
||||
|
||||
## Kick off / supervise the loops
|
||||
|
||||
@@ -53,5 +350,5 @@ cd /srv/cc-ci/cc-ci-plan
|
||||
./launch.sh stop
|
||||
```
|
||||
|
||||
Full supervision guide, credential map, and the Incus VM fallback are in
|
||||
[`cc-ci-plan/kickoff.md`](cc-ci-plan/kickoff.md) and [`cc-ci-plan/plan.md`](cc-ci-plan/plan.md) §1.5.
|
||||
Full supervision guide, credential map and history are in `cc-ci-plan/kickoff.md`,
|
||||
`cc-ci-plan/plan.md` §1.5 and `cc-ci-plan/JOURNAL.md`.
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
# archive/ — how cc-ci and its orchestrator were built and moved, before the combined host
|
||||
|
||||
Historical record only. Nothing in here is deployed or evaluated. It was moved out of the live
|
||||
tree on 2026-09-07 when the CI server and the orchestrator were consolidated onto one Hetzner
|
||||
host (`nixosConfigurations.cc-ci` in `../flake.nix`; deploy guide in `../README.md`; the plan
|
||||
that did it is `../cc-ci-plan/plan-cc-ci-combined-host.md`).
|
||||
|
||||
| path | what it was |
|
||||
|---|---|
|
||||
| `nix/configuration-incus-vm.nix` | Channel-based NixOS config of the first orchestrator VM on b1 (Incus, 2 GB). Ran the loops as root; hard-coded the dead Incus cc-ci IP. Replaced by the Hetzner host 2026-05-31. |
|
||||
| `nix/README.md` | The README for that Incus VM config. |
|
||||
| `nix/cc-ci-orchestrator-hetzner/` | The orchestrator's own Hetzner `cpx22` host (`168.119.126.100`, tailnet `cc-ci-orchestrator-1`), 2026-05-31 → 2026-09. From 2026-08-20 the live copy of this config was `notplants-nix`'s `notplants-orchestrator` host (the box became a shared agent host for several projects); this one had drifted and still carried lichen/project-orchestrator units. Superseded by `../nix/hosts/cc-ci` + `../nix/modules/orchestrator-host.nix`. |
|
||||
| `nix/atproto-likes.nix` | A notplants (not cc-ci) service that lived on the shared box; kept by notplants-nix. |
|
||||
| `terraform/` | OpenTofu for the `cpx22` orchestrator server (Debian 12 → nixos-infect at `nixos-24.11`). The combined host was provisioned by hand instead; the README documents that path. Note its `user-data.sh` would fail on the Debian 13 image (nixos-infect's temp swapfile on a tmpfs `/tmp`) — see the README's `NO_SWAP=true` note. |
|
||||
| `plans/plan-orchestrator-migration.md` | Pi → Incus VM move of the orchestrator (2026-05). |
|
||||
| `plans/plan-orchestrator-hetzner-migration.md` | Incus VM → Hetzner `cpx22` move of the orchestrator (2026-05-31). Has the reboot-resilience design (`cc-ci-loops.service`). |
|
||||
| `plans/plan-migrate-cc-ci-to-hetzner.md`, `plans/plan-cc-ci-hetzner-migration.md`, `plans/plan-cc-ci-hetzner-terraform.md` | The CI server's own move from the `cc-nix-test` Incus VM to Hetzner `cpx32` (`91.98.47.73`, 2026-05-31), and the terraform that provisioned it (lives in the cc-ci repo). |
|
||||
| `plans/plan-repo-consolidation.md` | The earlier repo layout consolidation. |
|
||||
|
||||
The cc-ci server's own history (machine-docs, decisions, the clean-room rebuild that proved
|
||||
"two repos + one age key + one `nixos-rebuild switch`") is in the cc-ci repo under
|
||||
`machine-docs/` and `docs/`.
|
||||
+11
-137
@@ -14,7 +14,11 @@
|
||||
# with ACME, and opens 80/443 (previously only 22 was public). Canonical source of
|
||||
# the module is the project repo; ../atproto-likes.nix is a copy kept in this tree
|
||||
# because pure evaluation cannot import a path outside the flake.
|
||||
imports = [ ../../atproto-likes.nix ];
|
||||
# cc-ci.nix holds everything that exists FOR cc-ci (the loops, the cc-ci orchestrator session,
|
||||
# the weekly recipe-upgrade run and its hourly supervisor). Split out 2026-08-20 so this host —
|
||||
# a general agent box that also serves unrelated projects — can be reasoned about separately
|
||||
# from cc-ci. Also exported as `nixosModules.cc-ci` from this repo's flake for other hosts.
|
||||
imports = [ ../../atproto-likes.nix ../../modules/cc-ci.nix ];
|
||||
|
||||
services.openssh = {
|
||||
enable = true;
|
||||
@@ -127,7 +131,8 @@ SSHCFG
|
||||
|
||||
# opencode web server — one shared instance; agent sessions attach to it for web visibility.
|
||||
# Serves the web UI at http://oc.commoninternet.net (via nginx below, tailscale-only).
|
||||
# Provider creds are read from /srv/cc-ci/.testenv at startup.
|
||||
# Public provider configuration is tracked in the cc-ci repo; credentials
|
||||
# remain in /srv/cc-ci/.testenv.
|
||||
systemd.services.opencode-web = {
|
||||
description = "opencode web server for cc-ci agents";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
@@ -137,7 +142,10 @@ SSHCFG
|
||||
Type = "simple";
|
||||
User = "loops"; Group = "users";
|
||||
WorkingDirectory = "/srv/cc-ci-orch/cc-ci";
|
||||
EnvironmentFile = "/srv/cc-ci/.testenv";
|
||||
EnvironmentFile = [
|
||||
"-/srv/cc-ci/cc-ci/.env.public"
|
||||
"/srv/cc-ci/.testenv"
|
||||
];
|
||||
ExecStartPre = "${pkgs.coreutils}/bin/rm -rf /tmp/opencode";
|
||||
ExecStart = "/home/loops/.local/bin/opencode serve --hostname 127.0.0.1 --port 4096";
|
||||
Restart = "on-failure";
|
||||
@@ -225,72 +233,7 @@ SSHCFG
|
||||
};
|
||||
};
|
||||
|
||||
# cc-ci-loops supervisor — workspace staged 2026-05-31, so ENABLED for reboot-resilience.
|
||||
systemd.services.cc-ci-loops = {
|
||||
description = "cc-ci Builder/Adversary loops + watchdog (launch.sh start)";
|
||||
wantedBy = [ "multi-user.target" ]; # enabled after workspace staged (Hetzner cutover)
|
||||
after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
# KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the
|
||||
# default (control-group) systemd kills every leftover process in the cgroup when the unit
|
||||
# stops — and since one tmux server hosts every agent session on this host, a rebuild that
|
||||
# merely touched this unit wiped all of them (operator 2026-08-01). Only the (already
|
||||
# exited) main process is killed now; `systemctl stop` therefore does NOT tear down agents.
|
||||
KillMode = "process";
|
||||
Type = "oneshot"; RemainAfterExit = true;
|
||||
User = "loops"; Group = "users";
|
||||
WorkingDirectory = "/srv/cc-ci/cc-ci";
|
||||
# Append one line to REBOOTS.md per genuine reboot (boot_id-gated; not on manual restart).
|
||||
ExecStartPre = "${pkgs.bash}/bin/bash /srv/cc-ci/cc-ci-plan/reboot-log.sh";
|
||||
};
|
||||
# CLAUDE_BIN points at the standalone CLI installed by claude-install.service; the loops
|
||||
# backend defaults to claude (persisted in .loop-backend). Without this, launch.py's preflight
|
||||
# `which(claude)` fails because the systemd `path` below has no /home/loops/.local/bin.
|
||||
environment = { RESUME_PHASE = "1"; HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
|
||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||
script = ''
|
||||
# Put the standalone claude/opencode binaries on PATH. On a cold boot this is the env the
|
||||
# tmux server (and thus every agent session) inherits, so bare `claude` resolves everywhere.
|
||||
export PATH="/home/loops/.local/bin:$PATH"
|
||||
[ -x /srv/cc-ci/cc-ci-plan/launch.sh ] && /srv/cc-ci/cc-ci-plan/launch.sh start || \
|
||||
echo "workspace not staged yet — skipping loop start"
|
||||
'';
|
||||
};
|
||||
|
||||
# cc-ci-orchestrator supervisor — the operator's steering session. Same shape as
|
||||
# lichen-orchestrator / project-orchestrator above: this unit only LAUNCHES the orchestrator's
|
||||
# tmux session via the agent-orchestrator harness (cc-ci-plan/agents.py); it does not own the
|
||||
# session or the tmux server. The orchestrator agent is declared in cc-ci-plan/agents.toml on
|
||||
# the OPencode backend (backend = "opencode", model = "opencode/glm-5.2"), so on boot it
|
||||
# attaches to the shared opencode web server (opencode-web.service below) and is reachable for
|
||||
# Remote Control at https://oc.commoninternet.net under the /srv/cc-ci-orch project. The harness
|
||||
# watchdog (started by `agents.py up`) keeps it alive: heal-only (no stall reboots — a persistent
|
||||
# supervisor must not be killed just for idling). Added 2026-08-03 to give the cc-ci orchestrator
|
||||
# the same reboot-resilience the other two orchestrators already have.
|
||||
systemd.services.cc-ci-orchestrator = {
|
||||
description = "cc-ci orchestrator (operator steering session) — agents.py up orchestrator, opencode backend";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network-online.target" "tailscaled.service" "opencode-web.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
# KillMode=process: see the note on cc-ci-loops — a rebuild that merely touches this unit
|
||||
# must not tear down the (shared) tmux server and every agent session with it.
|
||||
KillMode = "process";
|
||||
Type = "oneshot"; RemainAfterExit = true;
|
||||
User = "loops"; Group = "users";
|
||||
WorkingDirectory = "/srv/cc-ci-orch";
|
||||
};
|
||||
environment = { HOME = "/home/loops"; };
|
||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||
script = ''
|
||||
export PATH="/home/loops/.local/bin:$PATH"
|
||||
proj="/srv/cc-ci-orch"
|
||||
echo "$(cat /proc/sys/kernel/random/boot_id) boot $(date -u +%FT%TZ) — cc-ci-orchestrator up" \
|
||||
>> "$proj/cc-ci-plan/.ao-boot.log" 2>/dev/null || true
|
||||
cd "$proj" && python3 cc-ci-plan/agents.py up orchestrator || echo "cc-ci orchestrator agents.py up failed"
|
||||
'';
|
||||
};
|
||||
|
||||
# p-lichen-orchestrator supervisor — the SEPARATE lichen.page testing/hardening orchestrator
|
||||
# (distinct from cc-ci-loops above). Reboot-resilience: on boot, resume the orchestrator's Remote
|
||||
@@ -356,76 +299,7 @@ SSHCFG
|
||||
'';
|
||||
};
|
||||
|
||||
# Weekly recipe upgrade — runs /upgrade-all over every enrolled recipe (opens recipe PRs
|
||||
# verified by !testme, never merges). Replaces the boot-fragile busybox-crond-in-tmux from
|
||||
# phase 5 §4 with a reboot-safe systemd timer. The service is timer-triggered only (NOT
|
||||
# wantedBy multi-user.target) so it never runs on boot/activation — only on the schedule.
|
||||
systemd.services.cc-ci-upgrade-all = {
|
||||
description = "cc-ci weekly /upgrade-all run (recipe upgrade survey + PRs, never merges)";
|
||||
after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
Type = "oneshot"; # launch-upgrader.py spawns the cc-ci-upgrader tmux session and returns
|
||||
User = "loops"; Group = "users";
|
||||
WorkingDirectory = "/srv/cc-ci";
|
||||
# Optional per-run overrides for backend/model (LOOP_BACKEND, LOOP_MODEL, OPENCODE_SHARE,
|
||||
# UPGRADER_ARGS, …). The leading "-" makes it optional: absent file → claude/sonnet defaults
|
||||
# (current behavior). To run the weekly job on e.g. opencode-go/glm-5.2, drop a file with
|
||||
# LOOP_BACKEND=opencode
|
||||
# LOOP_MODEL=opencode-go/glm-5.2
|
||||
# No rebuild needed to switch — the env file is read at each timer fire. Holds no secrets
|
||||
# (the opencode-go API key lives in ~/.local/share/opencode/auth.json, mode 600).
|
||||
EnvironmentFile = "-/srv/cc-ci/upgrader.env";
|
||||
};
|
||||
environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
|
||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||
script = ''
|
||||
export PATH="/home/loops/.local/bin:$PATH"
|
||||
python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py start >> /srv/cc-ci/.cc-ci-logs/upgrader-cron.log 2>&1
|
||||
'';
|
||||
};
|
||||
|
||||
systemd.timers.cc-ci-upgrade-all = {
|
||||
description = "Weekly trigger for cc-ci-upgrade-all (Thursdays 22:00 America/New_York — Boston 10pm)";
|
||||
wantedBy = [ "timers.target" ];
|
||||
timerConfig = {
|
||||
# 10pm Thursday Boston time — DST-aware (EDT→02:00 UTC, EST→03:00 UTC) via the tz in OnCalendar.
|
||||
OnCalendar = "Thu *-*-* 22:00:00 America/New_York";
|
||||
Persistent = true; # if the box was down at the scheduled time, run once on next boot
|
||||
};
|
||||
};
|
||||
|
||||
# Hourly SUPERVISOR — a glm-5.2 orchestrator wake-up that keeps the weekly run on track. The
|
||||
# log-idle/429 watchdog only handles opencode-go usage-limit stalls; it does NOT cover a host
|
||||
# disk-full crash (which killed the 2026-07-03 run) or any other environmental wedge. This is a
|
||||
# CHEAP deterministic gate: if the weekly run is complete or actively progressing it does NOTHING
|
||||
# (zero model tokens). Only when a run has stalled/died before completing does it launch a
|
||||
# short-lived glm-5.2 agent that diagnoses the blockage and drives the run to a clean DONE.
|
||||
systemd.services.cc-ci-upgrade-supervisor = {
|
||||
description = "cc-ci hourly weekly-run supervisor (glm-5.2 — drives a stalled /upgrade-all to completion)";
|
||||
after = [ "network-online.target" "tailscaled.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
Type = "oneshot"; # launch-supervisor.py check: gate now, spawn the agent into tmux, return
|
||||
User = "loops"; Group = "users";
|
||||
WorkingDirectory = "/srv/cc-ci";
|
||||
# Shares the weekly run's optional override file (e.g. SUPERVISOR_MODEL=…); "-" = optional.
|
||||
EnvironmentFile = "-/srv/cc-ci/upgrader.env";
|
||||
};
|
||||
environment = { HOME = "/home/loops"; };
|
||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||
script = ''
|
||||
export PATH="/home/loops/.local/bin:$PATH"
|
||||
python3 /srv/cc-ci/cc-ci-plan/launch-supervisor.py check >> /srv/cc-ci/.cc-ci-logs/supervisor-cron.log 2>&1
|
||||
'';
|
||||
};
|
||||
|
||||
systemd.timers.cc-ci-upgrade-supervisor = {
|
||||
description = "Hourly trigger for cc-ci-upgrade-supervisor (weekly-run health check + drive)";
|
||||
wantedBy = [ "timers.target" ];
|
||||
timerConfig = {
|
||||
OnCalendar = "*-*-* *:07:00"; # every hour at :07 (offset from the weekly :00 fire)
|
||||
Persistent = false; # a missed hourly check is moot — the next hour re-checks
|
||||
};
|
||||
};
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user