diff --git a/README.md b/README.md index 79bd8bb..f8b0800 100644 --- a/README.md +++ b/README.md @@ -186,8 +186,8 @@ git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci.git /e # the age identity sops-nix decrypts with. FIRST deploy of a host that is not yet a recipient: # the master (recovery) key, temporarily. Once the host is a recipient (below): its own # ssh-host-key-derived identity, and the master key leaves the box. -install -d -m700 /var/lib/sops-nix -install -m600 /var/lib/sops-nix/key.txt +install -d -m0700 /secrets/host +install -m600 /secrets/host/sops-age-key.txt ``` `/etc/cc-ci/secrets/secrets.yaml` is encrypted to the master key and the *previous* hosts' SSH @@ -196,9 +196,9 @@ the master key can leave the box again (it belongs off-box, with the operator): `ssh-to-age < /etc/ssh/ssh_host_ed25519_key.pub` → add it to `.sops.yaml` in cc-ci-secrets AND in cc-ci, `sops updatekeys secrets.yaml` (needs the master key, so do it where that lives), push, bump the submodule in cc-ci, `git -C /etc/cc-ci pull --recurse-submodules` on the host, then -replace `/var/lib/sops-nix/key.txt` with the host-derived identity -(`ssh-to-age -private-key -i /etc/ssh/ssh_host_ed25519_key`), rebuild, and delete the master key -from the host. +replace `/secrets/host/sops-age-key.txt` with the host-derived identity +(`ssh-to-age -private-key -i /secrets/host/ssh_host_ed25519_key`), rebuild, and delete the master +key from the host. ### 4c. The orchestrator's workspace (as the `loops` user — it exists after the first rebuild, so run §5 once first if this is a fresh host, then come back) @@ -213,11 +213,29 @@ git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci-adv # mkdir -p .cc-ci-logs .sops ``` -Secrets live in **`/secrets/files/`** (owned by loops, mode 700 on the directory), the same -convention as the notplants-orchestrator box, with the runtime paths symlinked to them so one -`ls /secrets/files` is the complete inventory. **Only cc-ci's secrets go there** — nothing from -other projects on the box they were copied from (no tailscale, lichen, tangled, b1, borg, …): -secrets do not get spread around. `/secrets/README.txt` lists each file and its purpose. +**`/secrets` is the authoritative location for every secret on this host.** Consumers either read +from it directly (where the nix config owns the path) or reach it by a symlink (where the software +fixes the path), so `find /secrets -type f` is the complete inventory — including the ssh host +keys. The directory is `0711` (traversable, not listable) with three subdirectories, each owned by +whoever must read it; the layout is declared in `nix/hosts/cc-ci/configuration.nix`: + +| dir | mode | holds | +|---|---|---| +| `/secrets/files` | `loops:users 0700` | the agent's secrets: `.testenv`, opencode auth, its two ssh keys | +| `/secrets/host` | `root:root 0700` | host identity: the ssh **host** keys and the sops age identity | +| `/secrets/nginx` | `root:nginx 0750` | `oc-htpasswd` for the opencode UI (nginx must read it) | + +**Only cc-ci's secrets go there** — nothing from other projects (no tailscale, lichen, tangled, +b1, borg…), and every private key is generated **on the host**, never copied from another machine. +`/secrets/README.txt` lists each file, its purpose, and the derived material that legitimately +lives elsewhere (`/run/secrets`, `/root/.docker/config.json`, the ACME cert) so a later sweep does +not "clean" it away. + +> **The ed25519 host key is load-bearing beyond ssh.** Its age identity is a recipient of +> `cc-ci-secrets`, so regenerating it makes every cc-ci secret undecryptable. sshd is pointed at +> `/secrets/host/...` directly (not through the `/etc/ssh` symlinks) precisely so it can never +> write a fresh key through a dangling link. To move a host: copy the key, verify +> `ssh-to-age` yields the same identity, then activate. | runtime path → `/secrets/files/…` | what | source | |---|---|---| @@ -226,7 +244,7 @@ secrets do not get spread around. `/secrets/README.txt` lists each file and its | `~loops/.ssh/cc-ci-local-ed25519` (+`.pub`) → same names | `ssh cc-ci` as root — to loopback on this host. **Generated on the host**, its pub in `nix/hosts/cc-ci/ssh-keys` | `ssh-keygen -t ed25519 -C cc-ci-loops-to-root@cc-ci` as loops | | `~loops/.ssh/autonomic-bot-cc-ci-ed25519` (+`.pub`) → same names | pushes recipe branches / PRs as `autonomic-bot`; root's ssh config points at the same file to clone the private cc-ci-secrets submodule. **Generated on the host** and registered on the bot's Gitea account (`POST /api/v1/user/keys` with the bot password) | `ssh-keygen -t ed25519 -C autonomic-bot@cc-ci-host` as loops | | `~loops/.local/share/opencode/auth.json` → `opencode-auth.json`; `~loops/.config/opencode/opencode.jsonc` (config, not secret — it must contain NO inline `apiKey`) | opencode provider auth + config. Exactly ONE provider, `opencode-go` — the orchestrator AND the weekly upgrader are opencode agents; there is no Claude on this host | old host; fresh: `opencode auth login` as loops | -| `/etc/nginx/oc-htpasswd` (root:nginx; the bcrypt line only — the plaintext stays with the operator, not on the host) | basic auth for the opencode UI (`https://oc.ci.commoninternet.net`, via traefik); **nginx refuses to start without it**, and its config check runs as the `nginx` user, so `root:nginx 0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host (`/secrets/files/oc-basic-auth.txt` has the plaintext), or a new `oc:` line via `mkpasswd -m bcrypt` | +| `/secrets/nginx/oc-htpasswd` (root:nginx 0640; the bcrypt line only — the plaintext stays with the operator, not on the host) | basic auth for the opencode UI (`https://oc.ci.commoninternet.net`, via traefik); **nginx refuses to start without it**, and its config check runs as the `nginx` user, so `root:nginx 0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host (`/secrets/files/oc-basic-auth.txt` has the plaintext), or a new `oc:` line via `mkpasswd -m bcrypt` | `~loops/.ssh/config` is written by the activation script on first rebuild (`Host cc-ci` → `127.0.0.1`, `git.autonomic.zone`); it is not overwritten if present. diff --git a/cc-ci-plan/JOURNAL.md b/cc-ci-plan/JOURNAL.md index 3c902ed..371bfed 100644 --- a/cc-ci-plan/JOURNAL.md +++ b/cc-ci-plan/JOURNAL.md @@ -1171,3 +1171,34 @@ Lesson for future sweeps: scanning `/secrets` is not enough, agent logs are a se **Legitimately outside `/secrets`** (documented in its README so a later sweep does not "clean" them): `/run/secrets/*`, `/root/.docker/config.json`, `/etc/nginx/oc-htpasswd`, `/var/lib/ci-certs/live/*.pem`, `/var/lib/sops-nix/key.txt`, `/etc/ssh/ssh_host_*`. + +## 2026-09-08 17:30 UTC — /secrets made authoritative (incl. ssh host keys); weekly-run models + +**Operator rule:** every secret lives in `/secrets`; consumers read from there or symlink to it. +Implemented declaratively in `nix/hosts/cc-ci/configuration.nix` (cc-ci-orchestrator e7aa055): +`/secrets` `0711` with `files/` (loops), `host/` (root: ssh host keys + sops age identity) and +`nginx/` (root:nginx: the UI htpasswd). `find / -type f` for key material now returns **only** +`/secrets`; `/etc/ssh/ssh_host_*`, `~loops/.ssh/*`, `~loops/.local/share/opencode/auth.json` and +`/srv/cc-ci/.testenv` are all symlinks into it. + +**The risky part, done safely.** The ed25519 host key is a sops recipient (`age1tmvg…`), so a +regenerated key would make every cc-ci secret undecryptable. Therefore: keys **copied** (never +moved-then-regenerated), the age identity verified identical before activation, `services.openssh. +hostKeys` and `sops.age.{sshKeyPaths,keyFile}` pointed at `/secrets/host` **directly** rather than +through the `/etc/ssh` symlinks (a dangling link would let sshd write a NEW key), `test` before +`switch`, and the proof was a fresh `StrictHostKeyChecking=yes` connection plus 8/8 sops secrets. +Fingerprint unchanged: `SHA256:r1H85Ybif…`. Then the duplicates outside `/secrets` were deleted +(`/var/lib/sops-nix/key.txt`, `/etc/nginx/oc-htpasswd`) and the unused ecdsa host key removed. + +**Logs cleared** (operator: "we don't need them anymore"): 30 agent transcripts + 1 archived gz, +558 MB → 2 MB. Kept `upgrades/` (233 weekly records), `state/`, the `*-update-*.md` records and the +launcher dotfiles. + +**Weekly-run models** — operator enabled China-hosted models on the workspace, so both verified on +the host: `opencode-go/deepseek-v4-flash` and `opencode-go/glm-5.3-flash` answer. +- **subagents** (per-recipe workers): `opencode-go/deepseek-v4-flash` — cc-ci PR #37, pulled into + both loops clones and `/etc/cc-ci`. +- **main agents** (weekly parent, report, hourly supervisor): `opencode-go/glm-5.3-flash` — + `upgrader.env` (`LOOP_TIER=go` maps to the `opencode-go` auth entry; `LOOP_MODEL` overrides the + tier default). Next fire Fri 2026-09-11 02:00 UTC. +- The steering orchestrator agent stays on `opencode-go/glm-5.2` (not asked to change).