cc-ci/JOURNAL.md

JOURNAL — cc-ci Builder (append-only)

2026-05-26 — Bootstrap (§1)

Access verification (all pass):

• ssh cc-ci 'hostname && whoami && nixos-version' → nixos / root / 24.11.719113.50ab793786d9 (Vicuna)
• curl https://git.autonomic.zone/api/v1/version → {"version":"1.24.2"}
• Gitea bot auth (curl -u $GITEA_USERNAME:$GITEA_PASSWORD .../api/v1/user) → login: autonomic-bot, id 64
• getent hosts probe-$RANDOM.ci.commoninternet.net → 143.244.213.108 (the gateway IP, as expected — TLS passthrough)
• Cert present: ls /var/lib/ci-certs/live/ → fullchain.pem (2909 b), privkey.pem (227 b, mode 640)
• recipe-maintainers org exists (private); recipe-maintainers/cc-ci → 404 (created below)
• Mirrored recipes already present: bluesky-pds, lasuite-docs, custom-html, custom-html-tiny, n8n, keycloak, lasuite-meet, matrix-synapse, cryptpad

Baseline (docs/baseline.md): fresh NixOS 24.11 Incus VM, 2 vCPU, 3.5 GiB RAM, 8.9 GiB disk (3.8 GiB free). No docker/swarm/abra. Channel-based /etc/nixos/configuration.nix (no flake).

Actions:

• Created repo recipe-maintainers/cc-ci (private) via Gitea API.
• git init in /srv/cc-ci/cc-ci; credential helper reads creds from /srv/cc-ci/.testenv (no secrets stored in git config).
• Seeded skeleton layout (§3) + loop-state files + docs/baseline.md.

Next: commit + push bootstrap, then M0 (flake + base config + sops test secret).

2026-05-26 — M0: flake + base config rebuilt from repo

Authored flake.nix (pins nixpkgs rev 50ab793786d9…, the exact rev cc-ci ran), hosts/cc-ci/hardware.nix (incus VM module + cloud-init + DHCP/nameservers) and hosts/cc-ci/configuration.nix (faithful baseline repro: tailscale w/ hardcoded --hostname= cc-nix-test since builtins.readFile /etc/ts-hostname is impure under flakes; sshd root; firewall trust tailscale0 + tcp/22; base pkgs).

Disk/inode hiccup → resolved: first nix flake lock/build hit No space left on device — diagnosed as inode exhaustion (df -i → 6005 free of 586336; old 8.9 GiB fs). Operator grew the VM to 28 GiB while I was measuring; ext4 auto-resized → 22 GiB free, 1.21M inodes free. Retried.

Build + switch (commands + output):

• ssh cc-ci 'cd /root/cc-ci && nix flake lock && nixos-rebuild build --flake .#cc-ci' → BUILD EXIT 0, produced nixos-system-nixos-24.11.20250630.50ab793.
• ssh cc-ci 'systemd-run --unit=ccci-rebuild --collect --property=Type=oneshot nixos-rebuild switch --flake /root/cc-ci#cc-ci' (detached so it survives ssh drop) → unit Result=success ExecMainStatus=0.

Gate verification:

• systemctl is-system-running → running
• readlink /run/current-system → …-nixos-system-nixos-24.11.20250630.50ab793 (gen 3, from flake)
• systemctl is-active tailscaled → active; sshd.socket → active (sshd is socket-activated, so sshd.service reads inactive — live ssh proves it works)
• systemctl --failed → none
• nixos-rebuild list-generations → gen 3 current @20:23, prior channel gen 2 retained for rollback.

Known warning (tracked, non-blocking): incus module enables systemd.network while we keep networking.useDHCP=true (scripted dhcpcd); Nix warns both may manage interfaces. Inherited from baseline; networking is up. Clean up by choosing one stack later.

Deploy mechanism settled (DECISIONS.md): switch --flake on-host, repo synced via tar | ssh.

Next: sops-nix wiring (host age key from ssh host key + a decrypt-a-test-secret proof), then CLAIM the M0 gate for the Adversary.

2026-05-26 — M0: sops-nix wiring + decrypt-a-test-secret (M0 COMPLETE, gate CLAIMED)

Keys:

• Host age recipient from ssh host key: ssh cc-ci 'nix run nixpkgs#ssh-to-age -- -i /etc/ssh/ssh_host_ed25519_key.pub' → age1h90utdztfc23kx8ewrtrtk80mnddvrf8pg4ppej55rwwwupzhfvqhmp3qa.
• Master recovery key generated on host (age-keygen), public age1cmk26t…; private moved off-box to /srv/cc-ci/.sops/master-age.txt (mode 600) and shred-ded from the host. Never in repo.

Files: .sops.yaml (both recipients, rule secrets/.*\.(yaml|json|env)$); modules/secrets.nix (sops.age.sshKeyPaths=[/etc/ssh/ssh_host_ed25519_key], secrets.test_secret={}); flake gains sops-nix input + sops-nix.nixosModules.sops; configuration.nix imports the module.

sops-nix version pin (dead-end avoided): master sops-nix wants buildGo125Module (Go 1.25), absent in pinned nixpkgs 24.11 → eval error. Pinned sops-nix to 77c423a… (2025-06-17, last using plain buildGoModule). Verified the file at that rev uses buildGoModule. Build then OK.

Encrypt test secret: on host, printf 'test_secret: cc-ci-m0-<rand>' > secrets/secrets.yaml then nix run nixpkgs#sops -- --encrypt --in-place secrets/secrets.yaml (run inside repo so .sops.yaml resolves) → rc=0, two age recipients in the file.

Build + switch (commands + output):

• nixos-rebuild build --flake .#cc-ci → BUILD EXIT 0 (built sops-install-secrets w/ Go 1.23.8).
• systemd-run --unit=ccci-rebuild2 ... nixos-rebuild switch --flake /root/cc-ci#cc-ci → Result=success ExecMainStatus=0.

Gate verification (M0):

• systemctl is-system-running → running; systemctl --failed → none.
• ls -la /run/secrets/test_secret → -r-------- 1 root root 41 ; stat → root:root 400.
• head -c9 → cc-ci-m0- (matches generated value), wc -c → 41 (9 + 32 hex). Decrypt path proven.
• Pulled encrypted secrets/secrets.yaml + flake.lock back to clone; grep cc-ci-m0 secrets.yaml → no plaintext leak; lock inputs = nixpkgs, sops-nix.

Gate handshake: set Gate: M0 — CLAIMED, awaiting Adversary in STATUS.md. REVIEW.md still empty (no Adversary activity yet). Per §6.1 liveness I won't idle-block: I keep M0 claimed and proceed with M1 (independent infra build), without advancing to M2 until M0 shows PASS.

Next: M1 — Docker + single-node swarm via Nix (modules/swarm.nix), then Traefik (file provider → /var/lib/ci-certs/live/) + abra, then a by-hand HTTPS deploy/teardown of a trivial recipe.

2026-05-26 — M1: Docker + single-node swarm via Nix

modules/swarm.nix: virtualisation.docker.enable + daily autoprune (--all --volumes until=24h to protect the 28 GiB root), docker in systemPackages, and a swarm-init oneshot (docker swarm init --advertise-addr 127.0.0.1 if not active; docker network create --driver overlay --attachable proxy if absent). Imported into configuration.nix.

Build + switch: nixos-rebuild build --flake .#cc-ci → EXIT 0; systemd-run … switch → Result=success.

Verify (commands + output):

• systemctl show swarm-init -p Result → Result=success
• docker info --format ... → Swarm=active Managers=1 Nodes=1
• docker network ls --filter name=proxy → proxy overlay swarm
• systemctl is-system-running → running; --failed → none.

Next: Traefik as a swarm stack (Nix-declared compose + docker stack deploy oneshot): docker swarm provider + file provider serving /var/lib/ci-certs/live/{fullchain,privkey}.pem on :443, attached to proxy. Then abra install + by-hand HTTPS deploy/teardown of a trivial recipe (M1 gate). Rationale for swarm-service Traefik over a host services.traefik: a host process isn't on the proxy overlay, so it can't reach swarm service VIPs; coop-cloud recipes assume an on-proxy Traefik watching swarm labels.

2026-05-26 — M1: Traefik swarm stack + HTTPS path proven

modules/traefik.nix: Traefik v3.3 as a swarm service on proxy (so it reaches recipe VIPs). Config via Nix writeText store files bind-mounted into the container (real files, not /etc symlinks): static traefik.yml (entrypoints web/websecure; providers.swarm unix socket, exposedByDefault=false, network=proxy; providers.file dir /etc/traefik/dynamic; ping; no dashboard) and dynamic certs.yml (wildcard at /var/lib/ci-certs/live/* as stores.default. defaultCertificate + certificates — so any *.ci.commoninternet.net router with tls=true is covered, no ACME). Deployed by a traefik-deploy oneshot (docker stack deploy) after swarm-init. Opened firewall 80/443 (gateway forwards over enp5s0).

Build + switch: build EXIT 0; switch Result=success; traefik-deploy Result=success; docker service ls → traefik_traefik traefik:v3.3 1/1.

Verify (commands + output):

• Local: curl -ksv -H 'Host: probe-test.ci.commoninternet.net' https://localhost/ → subject: CN=*.ci.commoninternet.net, issuer: …Let's Encrypt; CN=E8, TLSv1.3, HTTP 404.
• End-to-end via gateway: curl -ksv --resolve probe-test.ci.commoninternet.net:443:143.244.213.108 https://probe-test.ci.commoninternet.net/ → Connected to …(143.244.213.108) port 443, same wildcard cert, HTTP 404. Confirms gateway SNI-passthrough → cc-ci Traefik TLS termination. 404 is correct (no router for that host yet).

Next: install abra (M1 last task), abra app new a trivial recipe (custom-html) → deploy → reach over HTTPS at .ci.commoninternet.net → teardown leaving no volumes. That completes M1 → CLAIM M1 gate.

2026-05-26 — M1: proxy pivot to real coop-cloud/traefik via abra; recipe deploy/teardown (M1 CLAIMED)

Orchestrator decision (mid-M1): replace the hand-rolled Traefik with the canonical Co-op Cloud traefik recipe deployed via abra, wildcard/file-provider mode, no ACME/token. Removed custom modules/traefik.nix; moved firewall 80/443 into modules/swarm.nix. Recorded in DECISIONS.md.

Why the pivot also fixed a real bug: my custom Traefik used entrypoint websecure; coop-cloud recipes label entrypoints=web-secure. While chasing that I also hit a sharp systemd-run gotcha: systemd-run … nixos-rebuild switch --flake .#cc-ci runs with cwd /, so .# → / → "could not find a flake.nix"; the switch silently failed while a post---collect systemctl show returned a stale Result=success. Fix: always use the absolute flake path /root/cc-ci#cc-ci, and read the result before resetting. (rebuild6/7 had silently not applied; rebuild2–5 used the absolute path.)

abra packaged (modules/abra.nix): release binary 0.13.0-beta, pinned by sha256, autoPatchelf'd. abra --version → 0.13.0-beta-06a57de.

scripts/deploy-proxy.sh (idempotent, pure-bash — host has no python3): ensure local abra server, fetch traefik, write wildcard/no-ACME env (WILDCARDS_ENABLED=1, SECRET_WILDCARD_*_VERSION=v1, COMPOSE_FILE=compose.yml:compose.wildcard.yml, LETS_ENCRYPT_ENV= empty), insert cert secrets via abra app secret insert … -f from /var/lib/ci-certs/live, deploy. Bugs fixed en route: multi-line PEM must use -f (not arg); secret-presence must check docker secret ls (abra's recipe list always shows the name with created on server:false).

Traefik deploy: abra app deploy → deploy succeeded 🟢 (traefik v3.6.15 + socket-proxy). Verify: docker service ls → app+socket-proxy 1/1; via gateway curl --resolve probe.*:443: 143.244.213.108 → CN=*.ci.commoninternet.net (LE E8); 0 ACME log lines.

M1 gate (recipe over HTTPS + teardown):

• abra app new custom-html -s default -D cchtml1.ci.commoninternet.net -S -n then set LETS_ENCRYPT_ENV= and abra app deploy -n -C → 🟢 (nginx 1.29.0).
• curl -ks --resolve cchtml1.ci.commoninternet.net:443:143.244.213.108 https://…/ → http_code=200 size=615, served the nginx welcome page over HTTPS with the wildcard cert.
• Teardown: abra app undeploy -n → 🟢; abra app volume remove -f -n → "1 volumes removed"; leak check → services 0 / volumes 0 / secrets 0 / containers 0. Clean.
• Correct teardown syntax confirmed: secret remove <d> --all -n (not --all-secrets).

docs/install.md seeded (flake apply + deploy-proxy + verify). M1 gate CLAIMED in STATUS.md.

Next: M2 — Drone server + exec runner via Nix, Gitea OAuth app, hello-world .drone.yml green.

2026-05-26 — M2 start: CI engine decision + Gitea OAuth app + Drone secrets

Decision (DECISIONS.md): keep Drone per plan. nixpkgs 24.11 has drone server 2.24.0 but only the abandoned drone-runner-exec (unstable-2020) — accepted (stable RPC), Woodpecker is the documented fallback. Deploy shape mirrors traefik: server via coop-cloud drone recipe (abra, swarm, traefik-routed at drone.ci.commoninternet.net, no ACME), exec runner as a host Nix systemd service.

Recipe recon: coop-cloud drone recipe = drone/drone:2.26.0, secrets rpc_secret + CLIENT_SECRET (Gitea OAuth), Gitea SSO via compose.gitea.yml (GITEA_CLIENT_ID, GITEA_DOMAIN). Server env: DRONE_SERVER_HOST/PROTO, DRONE_USER_CREATE.

Done this tick:

• Created Gitea OAuth app cc-ci-drone (bot): client_id ab4cdb9d-…, redirect https://drone.ci.commoninternet.net/login.
• Generated DRONE_RPC_SECRET (openssl-equivalent /dev/urandom hex32) + stored client_secret; both added to secrets/secrets.yaml via sops set (needed SOPS_AGE_KEY from the host ssh key: ssh-to-age -private-key -i /etc/ssh/ssh_host_ed25519_key). Verified: decrypt shows keys test_secret/drone_rpc_secret/drone_gitea_client_secret; file stays encrypted (4× ENC).

Next: scripts/deploy-drone.sh (abra deploy of drone server w/ Gitea SSO + rpc/client secrets), modules/drone-runner.nix (exec runner systemd unit, rpc secret from sops), wire sops secrets for the runner, then push a hello-world .drone.yml and confirm a green build (M2 gate).

2026-05-26 — M2: Drone server + exec runner up; infra as idempotent-reconcile oneshots

Orchestrator steer (2×): collapse install to a single nixos-rebuild switch — convert the manual deploy scripts into idempotent-reconcile systemd oneshots (writeShellApplication, embedded in store; after swarm-init+docker; wants network-online; wantedBy multi-user; reconcile every activation/boot, NO run-once sentinel; fail visibly on missing cert). Applied to proxy + drone.

Refactor done:

• modules/packages.nix: pkgs.abra overlay (shared pinned build).
• modules/proxy.nix: deploy-proxy oneshot — reconciles coop-cloud traefik (wildcard/no-ACME).
• modules/drone.nix: deploy-drone oneshot — reconciles coop-cloud drone (Gitea SSO, secrets from /run/secrets), after deploy-proxy.
• modules/drone-runner.nix: exec runner (fixed PATH conflict via lib.mkForce; allowUnfree for drone-runner-exec — Polyform license).
• modules/secrets.nix: declared drone_rpc_secret + drone_gitea_client_secret + a sops template drone-runner.env (DRONE_RPC_SECRET) as the runner's EnvironmentFile (shared secret).
• Removed scripts/deploy-*.sh. install.md now = clone + nixos-rebuild switch + preconditions.

Build/switch: build EXIT 0 (shellcheck clean via writeShellApplication; runner pkg unfree-allowed). nixos-rebuild switch → all three units active/success:

• deploy-proxy success (reconciled traefik), deploy-drone → deploy succeeded 🟢 (drone/drone 2.26.0, secrets client_secret+rpc_secret v1, drone_env config), drone-runner-exec active.

Verify (commands + output):

• docker service ls → drone_ci_commoninternet_net_app 1/1, traefik app+socket-proxy 1/1.
• Via gateway: …/healthz → 200; / → 303 (login redirect, correct).
• Runner: journal shows a few startup cannot ping the remote server (404) (drone RPC not ready yet) then successfully pinged the remote server + polling the remote server capacity=2 endpoint=https://drone.ci.commoninternet.net kind=pipeline type=exec. Runner connected via RPC.

Remaining for M2 gate: push a hello-world .drone.yml to cc-ci + get a green build. Needs the cc-ci repo activated in Drone, which requires the bot's Gitea OAuth login (browser flow) to grant Drone a Gitea token (to sync repos + set the push webhook). Next tick: script the OAuth login to mint a Drone token, activate cc-ci, push .drone.yml, confirm green. (DRONE_USER_CREATE made autonomic-bot the admin.)

2026-05-26 — M2 GATE MET: green build via push (Drone + exec runner)

Drone↔Gitea OAuth (scripted, the one manual bootstrap): logged the bot into Gitea (CSRF cookie → form), drove Drone /login → Gitea authorize consent (POST /login/oauth/grant with _csrf+state+ granted=true) → code callback → Drone _session_. Captured the whole flow in scripts/bootstrap-drone-oauth.sh (reads bot creds from env; documented in install.md §2; one-time, token persists in Drone's data volume).

Repo activation: GET /api/user → autonomic-bot admin=true; GET /api/user/repos?latest=true synced 12 repos; POST /api/repos/recipe-maintainers/cc-ci → active=true, config_path .drone.yml (sets the Gitea push webhook).

Green build: added .drone.yml (exec pipeline), pushed (0d89e28). Polled /api/repos/recipe-maintainers/cc-ci/builds → build #1 pending→running→success. Steps: clone success exit 0; hello success exit 0 — log shows whoami=root, abra 0.13.0-beta-06a57de, swarm=active (ran on the host via the exec runner). M2 gate met; CLAIMED.

Next: M3 — comment-bridge service: Gitea issue_comment webhook → verify HMAC + !testme exact + collaborator → resolve PR head repo/SHA → trigger a parameterized Drone build; post a PR comment with the run link. Need a Drone API token for the bridge (mint from the bot's Drone account).

2026-05-26 — M3 start: bridge secrets + comment-bridge source

Secrets (sops): minted a Gitea API token (cc-ci-bridge, scopes read:org/user, write:repo/issue), a Drone API token (POST /api/user/token, the stable personal token; rotates on call), and a webhook HMAC (urandom hex64). Stored as bridge_gitea_token / bridge_drone_token / bridge_webhook_hmac via sops set (host age identity). secrets.yaml now holds 6 secrets.

bridge/bridge.py (Python stdlib only, §4.1): POST /hook handler — verifies Gitea HMAC (X-Gitea-Signature sha256), requires X-Gitea-Event: issue_comment, action=created, body trimmed == !testme, issue is a PR; checks commenter is a collaborator (Gitea collaborators endpoint, 204); resolves PR head sha+repo; triggers a parameterized Drone build (POST /api/repos/<CI_REPO>/builds?branch=main&RECIPE&REF&PR&SRC, custom params → pipeline env); posts a PR comment linking the run. Secrets read from mounted files; config via env. /healthz GET.

Next: package the bridge as a swarm service (dockerTools image, no Docker Hub pull) behind traefik at ci.commoninternet.net/hook via a reconcile oneshot (modules/bridge.nix); register a per-repo webhook with the HMAC; demo on a scratch PR (!testme triggers; non-!testme + non-collab rejected). That's the M3 gate.