Files

autonomic-bot 51b18841bc M1: Traefik swarm stack (wildcard cert via file provider); HTTPS path proven E2E

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 21:55:08 +01:00

9.0 KiB

Raw Blame History

JOURNAL — cc-ci Builder (append-only)

2026-05-26 — Bootstrap (§1)

Access verification (all pass):

ssh cc-ci 'hostname && whoami && nixos-version' → nixos / root / 24.11.719113.50ab793786d9 (Vicuna)
curl https://git.autonomic.zone/api/v1/version → {"version":"1.24.2"}
Gitea bot auth (curl -u $GITEA_USERNAME:$GITEA_PASSWORD .../api/v1/user) → login: autonomic-bot, id 64
getent hosts probe-$RANDOM.ci.commoninternet.net → 143.244.213.108 (the gateway IP, as expected — TLS passthrough)
Cert present: ls /var/lib/ci-certs/live/ → fullchain.pem (2909 b), privkey.pem (227 b, mode 640)
recipe-maintainers org exists (private); recipe-maintainers/cc-ci → 404 (created below)
Mirrored recipes already present: bluesky-pds, lasuite-docs, custom-html, custom-html-tiny, n8n, keycloak, lasuite-meet, matrix-synapse, cryptpad

Baseline (docs/baseline.md): fresh NixOS 24.11 Incus VM, 2 vCPU, 3.5 GiB RAM, 8.9 GiB disk (3.8 GiB free). No docker/swarm/abra. Channel-based /etc/nixos/configuration.nix (no flake).

Actions:

Created repo recipe-maintainers/cc-ci (private) via Gitea API.
git init in /srv/cc-ci/cc-ci; credential helper reads creds from /srv/cc-ci/.testenv (no secrets stored in git config).
Seeded skeleton layout (§3) + loop-state files + docs/baseline.md.

Next: commit + push bootstrap, then M0 (flake + base config + sops test secret).

2026-05-26 — M0: flake + base config rebuilt from repo

Authored flake.nix (pins nixpkgs rev 50ab793786d9…, the exact rev cc-ci ran), hosts/cc-ci/hardware.nix (incus VM module + cloud-init + DHCP/nameservers) and hosts/cc-ci/configuration.nix (faithful baseline repro: tailscale w/ hardcoded --hostname= cc-nix-test since builtins.readFile /etc/ts-hostname is impure under flakes; sshd root; firewall trust tailscale0 + tcp/22; base pkgs).

Disk/inode hiccup → resolved: first nix flake lock/build hit No space left on device — diagnosed as inode exhaustion (df -i → 6005 free of 586336; old 8.9 GiB fs). Operator grew the VM to 28 GiB while I was measuring; ext4 auto-resized → 22 GiB free, 1.21M inodes free. Retried.

Build + switch (commands + output):

ssh cc-ci 'cd /root/cc-ci && nix flake lock && nixos-rebuild build --flake .#cc-ci' → BUILD EXIT 0, produced nixos-system-nixos-24.11.20250630.50ab793.
ssh cc-ci 'systemd-run --unit=ccci-rebuild --collect --property=Type=oneshot nixos-rebuild switch --flake /root/cc-ci#cc-ci' (detached so it survives ssh drop) → unit Result=success ExecMainStatus=0.

Gate verification:

systemctl is-system-running → running
readlink /run/current-system → …-nixos-system-nixos-24.11.20250630.50ab793 (gen 3, from flake)
systemctl is-active tailscaled → active; sshd.socket → active (sshd is socket-activated, so sshd.service reads inactive — live ssh proves it works)
systemctl --failed → none
nixos-rebuild list-generations → gen 3 current @20:23, prior channel gen 2 retained for rollback.

Known warning (tracked, non-blocking): incus module enables systemd.network while we keep networking.useDHCP=true (scripted dhcpcd); Nix warns both may manage interfaces. Inherited from baseline; networking is up. Clean up by choosing one stack later.

Deploy mechanism settled (DECISIONS.md): switch --flake on-host, repo synced via tar | ssh.

Next: sops-nix wiring (host age key from ssh host key + a decrypt-a-test-secret proof), then CLAIM the M0 gate for the Adversary.

2026-05-26 — M0: sops-nix wiring + decrypt-a-test-secret (M0 COMPLETE, gate CLAIMED)

Keys:

Host age recipient from ssh host key: ssh cc-ci 'nix run nixpkgs#ssh-to-age -- -i /etc/ssh/ssh_host_ed25519_key.pub' → age1h90utdztfc23kx8ewrtrtk80mnddvrf8pg4ppej55rwwwupzhfvqhmp3qa.
Master recovery key generated on host (age-keygen), public age1cmk26t…; private moved off-box to /srv/cc-ci/.sops/master-age.txt (mode 600) and shred-ded from the host. Never in repo.

Files: .sops.yaml (both recipients, rule secrets/.*\.(yaml|json|env)$); modules/secrets.nix (sops.age.sshKeyPaths=[/etc/ssh/ssh_host_ed25519_key], secrets.test_secret={}); flake gains sops-nix input + sops-nix.nixosModules.sops; configuration.nix imports the module.

sops-nix version pin (dead-end avoided): master sops-nix wants buildGo125Module (Go 1.25), absent in pinned nixpkgs 24.11 → eval error. Pinned sops-nix to 77c423a… (2025-06-17, last using plain buildGoModule). Verified the file at that rev uses buildGoModule. Build then OK.

Encrypt test secret: on host, printf 'test_secret: cc-ci-m0-<rand>' > secrets/secrets.yaml then nix run nixpkgs#sops -- --encrypt --in-place secrets/secrets.yaml (run inside repo so .sops.yaml resolves) → rc=0, two age recipients in the file.

Build + switch (commands + output):

nixos-rebuild build --flake .#cc-ci → BUILD EXIT 0 (built sops-install-secrets w/ Go 1.23.8).
systemd-run --unit=ccci-rebuild2 ... nixos-rebuild switch --flake /root/cc-ci#cc-ci → Result=success ExecMainStatus=0.

Gate verification (M0):

systemctl is-system-running → running; systemctl --failed → none.
ls -la /run/secrets/test_secret → -r-------- 1 root root 41 ; stat → root:root 400.
head -c9 → cc-ci-m0- (matches generated value), wc -c → 41 (9 + 32 hex). Decrypt path proven.
Pulled encrypted secrets/secrets.yaml + flake.lock back to clone; grep cc-ci-m0 secrets.yaml → no plaintext leak; lock inputs = nixpkgs, sops-nix.

Gate handshake: set Gate: M0 — CLAIMED, awaiting Adversary in STATUS.md. REVIEW.md still empty (no Adversary activity yet). Per §6.1 liveness I won't idle-block: I keep M0 claimed and proceed with M1 (independent infra build), without advancing to M2 until M0 shows PASS.

Next: M1 — Docker + single-node swarm via Nix (modules/swarm.nix), then Traefik (file provider → /var/lib/ci-certs/live/) + abra, then a by-hand HTTPS deploy/teardown of a trivial recipe.

2026-05-26 — M1: Docker + single-node swarm via Nix

modules/swarm.nix: virtualisation.docker.enable + daily autoprune (--all --volumes until=24h to protect the 28 GiB root), docker in systemPackages, and a swarm-init oneshot (docker swarm init --advertise-addr 127.0.0.1 if not active; docker network create --driver overlay --attachable proxy if absent). Imported into configuration.nix.

Build + switch: nixos-rebuild build --flake .#cc-ci → EXIT 0; systemd-run … switch → Result=success.

Verify (commands + output):

systemctl show swarm-init -p Result → Result=success
docker info --format ... → Swarm=active Managers=1 Nodes=1
docker network ls --filter name=proxy → proxy overlay swarm
systemctl is-system-running → running; --failed → none.

Next: Traefik as a swarm stack (Nix-declared compose + docker stack deploy oneshot): docker swarm provider + file provider serving /var/lib/ci-certs/live/{fullchain,privkey}.pem on :443, attached to proxy. Then abra install + by-hand HTTPS deploy/teardown of a trivial recipe (M1 gate). Rationale for swarm-service Traefik over a host services.traefik: a host process isn't on the proxy overlay, so it can't reach swarm service VIPs; coop-cloud recipes assume an on-proxy Traefik watching swarm labels.

2026-05-26 — M1: Traefik swarm stack + HTTPS path proven

modules/traefik.nix: Traefik v3.3 as a swarm service on proxy (so it reaches recipe VIPs). Config via Nix writeText store files bind-mounted into the container (real files, not /etc symlinks): static traefik.yml (entrypoints web/websecure; providers.swarm unix socket, exposedByDefault=false, network=proxy; providers.file dir /etc/traefik/dynamic; ping; no dashboard) and dynamic certs.yml (wildcard at /var/lib/ci-certs/live/* as stores.default. defaultCertificate + certificates — so any *.ci.commoninternet.net router with tls=true is covered, no ACME). Deployed by a traefik-deploy oneshot (docker stack deploy) after swarm-init. Opened firewall 80/443 (gateway forwards over enp5s0).

Build + switch: build EXIT 0; switch Result=success; traefik-deploy Result=success; docker service ls → traefik_traefik traefik:v3.3 1/1.