README + journal: the /secrets layout, and the weekly run's flash models

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz

README.md

@@ -186,8 +186,8 @@ git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci.git /e
 # the age identity sops-nix decrypts with. FIRST deploy of a host that is not yet a recipient:
 # the master (recovery) key, temporarily. Once the host is a recipient (below): its own
 # ssh-host-key-derived identity, and the master key leaves the box.
-install -d -m700 /var/lib/sops-nix
+install -d -m0700 /secrets/host
-install -m600 <master-age.txt> /var/lib/sops-nix/key.txt
+install -m600 <master-age.txt> /secrets/host/sops-age-key.txt
 ```
 
 `/etc/cc-ci/secrets/secrets.yaml` is encrypted to the master key and the *previous* hosts' SSH
@@ -196,9 +196,9 @@ the master key can leave the box again (it belongs off-box, with the operator):
 `ssh-to-age < /etc/ssh/ssh_host_ed25519_key.pub` → add it to `.sops.yaml` in cc-ci-secrets AND in
 cc-ci, `sops updatekeys secrets.yaml` (needs the master key, so do it where that lives), push,
 bump the submodule in cc-ci, `git -C /etc/cc-ci pull --recurse-submodules` on the host, then
-replace `/var/lib/sops-nix/key.txt` with the host-derived identity
+replace `/secrets/host/sops-age-key.txt` with the host-derived identity
-(`ssh-to-age -private-key -i /etc/ssh/ssh_host_ed25519_key`), rebuild, and delete the master key
+(`ssh-to-age -private-key -i /secrets/host/ssh_host_ed25519_key`), rebuild, and delete the master
-from the host.
+key from the host.
 
 ### 4c. The orchestrator's workspace (as the `loops` user — it exists after the first rebuild, so
 run §5 once first if this is a fresh host, then come back)
@@ -213,11 +213,29 @@ git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci-adv    #
 mkdir -p .cc-ci-logs .sops
 ```
 
-Secrets live in **`/secrets/files/`** (owned by loops, mode 700 on the directory), the same
+**`/secrets` is the authoritative location for every secret on this host.** Consumers either read
-convention as the notplants-orchestrator box, with the runtime paths symlinked to them so one
+from it directly (where the nix config owns the path) or reach it by a symlink (where the software
-`ls /secrets/files` is the complete inventory. **Only cc-ci's secrets go there** — nothing from
+fixes the path), so `find /secrets -type f` is the complete inventory — including the ssh host
-other projects on the box they were copied from (no tailscale, lichen, tangled, b1, borg, …):
+keys. The directory is `0711` (traversable, not listable) with three subdirectories, each owned by
-secrets do not get spread around. `/secrets/README.txt` lists each file and its purpose.
+whoever must read it; the layout is declared in `nix/hosts/cc-ci/configuration.nix`:
+
+| dir | mode | holds |
+|---|---|---|
+| `/secrets/files` | `loops:users 0700` | the agent's secrets: `.testenv`, opencode auth, its two ssh keys |
+| `/secrets/host` | `root:root 0700` | host identity: the ssh **host** keys and the sops age identity |
+| `/secrets/nginx` | `root:nginx 0750` | `oc-htpasswd` for the opencode UI (nginx must read it) |
+
+**Only cc-ci's secrets go there** — nothing from other projects (no tailscale, lichen, tangled,
+b1, borg…), and every private key is generated **on the host**, never copied from another machine.
+`/secrets/README.txt` lists each file, its purpose, and the derived material that legitimately
+lives elsewhere (`/run/secrets`, `/root/.docker/config.json`, the ACME cert) so a later sweep does
+not "clean" it away.
+
+> **The ed25519 host key is load-bearing beyond ssh.** Its age identity is a recipient of
+> `cc-ci-secrets`, so regenerating it makes every cc-ci secret undecryptable. sshd is pointed at
+> `/secrets/host/...` directly (not through the `/etc/ssh` symlinks) precisely so it can never
+> write a fresh key through a dangling link. To move a host: copy the key, verify
+> `ssh-to-age` yields the same identity, then activate.
 
 | runtime path → `/secrets/files/…` | what | source |
 |---|---|---|
@@ -226,7 +244,7 @@ secrets do not get spread around. `/secrets/README.txt` lists each file and its
 | `~loops/.ssh/cc-ci-local-ed25519` (+`.pub`) → same names | `ssh cc-ci` as root — to loopback on this host. **Generated on the host**, its pub in `nix/hosts/cc-ci/ssh-keys` | `ssh-keygen -t ed25519 -C cc-ci-loops-to-root@cc-ci` as loops |
 | `~loops/.ssh/autonomic-bot-cc-ci-ed25519` (+`.pub`) → same names | pushes recipe branches / PRs as `autonomic-bot`; root's ssh config points at the same file to clone the private cc-ci-secrets submodule. **Generated on the host** and registered on the bot's Gitea account (`POST /api/v1/user/keys` with the bot password) | `ssh-keygen -t ed25519 -C autonomic-bot@cc-ci-host` as loops |
 | `~loops/.local/share/opencode/auth.json` → `opencode-auth.json`; `~loops/.config/opencode/opencode.jsonc` (config, not secret — it must contain NO inline `apiKey`) | opencode provider auth + config. Exactly ONE provider, `opencode-go` — the orchestrator AND the weekly upgrader are opencode agents; there is no Claude on this host | old host; fresh: `opencode auth login` as loops |
-| `/etc/nginx/oc-htpasswd` (root:nginx; the bcrypt line only — the plaintext stays with the operator, not on the host) | basic auth for the opencode UI (`https://oc.ci.commoninternet.net`, via traefik); **nginx refuses to start without it**, and its config check runs as the `nginx` user, so `root:nginx 0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host (`/secrets/files/oc-basic-auth.txt` has the plaintext), or a new `oc:<bcrypt>` line via `mkpasswd -m bcrypt` |
+| `/secrets/nginx/oc-htpasswd` (root:nginx 0640; the bcrypt line only — the plaintext stays with the operator, not on the host) | basic auth for the opencode UI (`https://oc.ci.commoninternet.net`, via traefik); **nginx refuses to start without it**, and its config check runs as the `nginx` user, so `root:nginx 0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host (`/secrets/files/oc-basic-auth.txt` has the plaintext), or a new `oc:<bcrypt>` line via `mkpasswd -m bcrypt` |
 
 `~loops/.ssh/config` is written by the activation script on first rebuild (`Host cc-ci` →
 `127.0.0.1`, `git.autonomic.zone`); it is not overwritten if present.

cc-ci-plan/JOURNAL.md

@@ -1171,3 +1171,34 @@ Lesson for future sweeps: scanning `/secrets` is not enough, agent logs are a se
 **Legitimately outside `/secrets`** (documented in its README so a later sweep does not "clean"
 them): `/run/secrets/*`, `/root/.docker/config.json`, `/etc/nginx/oc-htpasswd`,
 `/var/lib/ci-certs/live/*.pem`, `/var/lib/sops-nix/key.txt`, `/etc/ssh/ssh_host_*`.
+
+## 2026-09-08 17:30 UTC — /secrets made authoritative (incl. ssh host keys); weekly-run models
+
+**Operator rule:** every secret lives in `/secrets`; consumers read from there or symlink to it.
+Implemented declaratively in `nix/hosts/cc-ci/configuration.nix` (cc-ci-orchestrator e7aa055):
+`/secrets` `0711` with `files/` (loops), `host/` (root: ssh host keys + sops age identity) and
+`nginx/` (root:nginx: the UI htpasswd). `find / -type f` for key material now returns **only**
+`/secrets`; `/etc/ssh/ssh_host_*`, `~loops/.ssh/*`, `~loops/.local/share/opencode/auth.json` and
+`/srv/cc-ci/.testenv` are all symlinks into it.
+
+**The risky part, done safely.** The ed25519 host key is a sops recipient (`age1tmvg…`), so a
+regenerated key would make every cc-ci secret undecryptable. Therefore: keys **copied** (never
+moved-then-regenerated), the age identity verified identical before activation, `services.openssh.
+hostKeys` and `sops.age.{sshKeyPaths,keyFile}` pointed at `/secrets/host` **directly** rather than
+through the `/etc/ssh` symlinks (a dangling link would let sshd write a NEW key), `test` before
+`switch`, and the proof was a fresh `StrictHostKeyChecking=yes` connection plus 8/8 sops secrets.
+Fingerprint unchanged: `SHA256:r1H85Ybif…`. Then the duplicates outside `/secrets` were deleted
+(`/var/lib/sops-nix/key.txt`, `/etc/nginx/oc-htpasswd`) and the unused ecdsa host key removed.
+
+**Logs cleared** (operator: "we don't need them anymore"): 30 agent transcripts + 1 archived gz,
+558 MB → 2 MB. Kept `upgrades/` (233 weekly records), `state/`, the `*-update-*.md` records and the
+launcher dotfiles.
+
+**Weekly-run models** — operator enabled China-hosted models on the workspace, so both verified on
+the host: `opencode-go/deepseek-v4-flash` and `opencode-go/glm-5.3-flash` answer.
+- **subagents** (per-recipe workers): `opencode-go/deepseek-v4-flash` — cc-ci PR #37, pulled into
+  both loops clones and `/etc/cc-ci`.
+- **main agents** (weekly parent, report, hourly supervisor): `opencode-go/glm-5.3-flash` —
+  `upgrader.env` (`LOOP_TIER=go` maps to the `opencode-go` auth entry; `LOOP_MODEL` overrides the
+  tier default). Next fire Fri 2026-09-11 02:00 UTC.
+- The steering orchestrator agent stays on `opencode-go/glm-5.2` (not asked to change).