# JOURNAL — cc-ci Builder (append-only)

## 2026-05-26 — Bootstrap (§1)

**Access verification (all pass):**
- `ssh cc-ci 'hostname && whoami && nixos-version'` → `nixos` / `root` / `24.11.719113.50ab793786d9 (Vicuna)`
- `curl https://git.autonomic.zone/api/v1/version` → `{"version":"1.24.2"}`
- Gitea bot auth (`curl -u $GITEA_USERNAME:$GITEA_PASSWORD .../api/v1/user`) → `login: autonomic-bot`, id 64
- `getent hosts probe-$RANDOM.ci.commoninternet.net` → `143.244.213.108` (the gateway IP, as expected — TLS passthrough)
- Cert present: `ls /var/lib/ci-certs/live/` → `fullchain.pem` (2909 b), `privkey.pem` (227 b, mode 640)
- recipe-maintainers org exists (private); `recipe-maintainers/cc-ci` → 404 (created below)
- Mirrored recipes already present: bluesky-pds, lasuite-docs, custom-html, custom-html-tiny, n8n,
  keycloak, lasuite-meet, matrix-synapse, cryptpad

**Baseline (docs/baseline.md):** fresh NixOS 24.11 Incus VM, 2 vCPU, 3.5 GiB RAM, 8.9 GiB disk
(3.8 GiB free). No docker/swarm/abra. Channel-based `/etc/nixos/configuration.nix` (no flake).

**Actions:**
- Created repo `recipe-maintainers/cc-ci` (private) via Gitea API.
- `git init` in /srv/cc-ci/cc-ci; credential helper reads creds from /srv/cc-ci/.testenv (no
  secrets stored in git config).
- Seeded skeleton layout (§3) + loop-state files + docs/baseline.md.

**Next:** commit + push bootstrap, then M0 (flake + base config + sops test secret).

## 2026-05-26 — M0: flake + base config rebuilt from repo

**Authored** `flake.nix` (pins nixpkgs rev `50ab793786d9…`, the exact rev cc-ci ran),
`hosts/cc-ci/hardware.nix` (incus VM module + cloud-init + DHCP/nameservers) and
`hosts/cc-ci/configuration.nix` (faithful baseline repro: tailscale w/ hardcoded `--hostname=
cc-nix-test` since `builtins.readFile /etc/ts-hostname` is impure under flakes; sshd root; firewall
trust tailscale0 + tcp/22; base pkgs).

**Disk/inode hiccup → resolved:** first `nix flake lock`/build hit `No space left on device` —
diagnosed as **inode** exhaustion (`df -i` → 6005 free of 586336; old 8.9 GiB fs). Operator grew
the VM to 28 GiB while I was measuring; ext4 auto-resized → 22 GiB free, 1.21M inodes free. Retried.

**Build + switch (commands + output):**
- `ssh cc-ci 'cd /root/cc-ci && nix flake lock && nixos-rebuild build --flake .#cc-ci'` → `BUILD EXIT 0`,
  produced `nixos-system-nixos-24.11.20250630.50ab793`.
- `ssh cc-ci 'systemd-run --unit=ccci-rebuild --collect --property=Type=oneshot nixos-rebuild switch
  --flake /root/cc-ci#cc-ci'` (detached so it survives ssh drop) → unit `Result=success
  ExecMainStatus=0`.

**Gate verification:**
- `systemctl is-system-running` → `running`
- `readlink /run/current-system` → `…-nixos-system-nixos-24.11.20250630.50ab793` (gen 3, from flake)
- `systemctl is-active tailscaled` → `active`; `sshd.socket` → `active` (sshd is socket-activated, so
  `sshd.service` reads inactive — live ssh proves it works)
- `systemctl --failed` → none
- `nixos-rebuild list-generations` → gen 3 current @20:23, prior channel gen 2 retained for rollback.

**Known warning (tracked, non-blocking):** incus module enables `systemd.network` while we keep
`networking.useDHCP=true` (scripted dhcpcd); Nix warns both may manage interfaces. Inherited from
baseline; networking is up. Clean up by choosing one stack later.

**Deploy mechanism settled** (DECISIONS.md): `switch --flake` on-host, repo synced via `tar | ssh`.

**Next:** sops-nix wiring (host age key from ssh host key + a decrypt-a-test-secret proof), then
CLAIM the M0 gate for the Adversary.

## 2026-05-26 — M0: sops-nix wiring + decrypt-a-test-secret (M0 COMPLETE, gate CLAIMED)

**Keys:**
- Host age recipient from ssh host key: `ssh cc-ci 'nix run nixpkgs#ssh-to-age -- -i
  /etc/ssh/ssh_host_ed25519_key.pub'` → `age1h90utdztfc23kx8ewrtrtk80mnddvrf8pg4ppej55rwwwupzhfvqhmp3qa`.
- Master recovery key generated on host (`age-keygen`), public `age1cmk26t…`; private moved off-box
  to `/srv/cc-ci/.sops/master-age.txt` (mode 600) and `shred`-ded from the host. Never in repo.

**Files:** `.sops.yaml` (both recipients, rule `secrets/.*\.(yaml|json|env)$`); `modules/secrets.nix`
(`sops.age.sshKeyPaths=[/etc/ssh/ssh_host_ed25519_key]`, `secrets.test_secret={}`); flake gains
`sops-nix` input + `sops-nix.nixosModules.sops`; configuration.nix imports the module.

**sops-nix version pin (dead-end avoided):** master sops-nix wants `buildGo125Module` (Go 1.25),
absent in pinned nixpkgs 24.11 → eval error. Pinned sops-nix to `77c423a…` (2025-06-17, last using
plain `buildGoModule`). Verified the file at that rev uses `buildGoModule`. Build then OK.

**Encrypt test secret:** on host, `printf 'test_secret: cc-ci-m0-<rand>' > secrets/secrets.yaml`
then `nix run nixpkgs#sops -- --encrypt --in-place secrets/secrets.yaml` (run inside repo so
`.sops.yaml` resolves) → rc=0, two age recipients in the file.

**Build + switch (commands + output):**
- `nixos-rebuild build --flake .#cc-ci` → `BUILD EXIT 0` (built sops-install-secrets w/ Go 1.23.8).
- `systemd-run --unit=ccci-rebuild2 ... nixos-rebuild switch --flake /root/cc-ci#cc-ci` →
  `Result=success ExecMainStatus=0`.

**Gate verification (M0):**
- `systemctl is-system-running` → `running`; `systemctl --failed` → none.
- `ls -la /run/secrets/test_secret` → `-r-------- 1 root root 41` ; `stat` → `root:root 400`.
- `head -c9` → `cc-ci-m0-` (matches generated value), `wc -c` → 41 (9 + 32 hex). Decrypt path proven.
- Pulled encrypted `secrets/secrets.yaml` + `flake.lock` back to clone; `grep cc-ci-m0 secrets.yaml`
  → no plaintext leak; lock inputs = nixpkgs, sops-nix.

**Gate handshake:** set `Gate: M0 — CLAIMED, awaiting Adversary` in STATUS.md. REVIEW.md still empty
(no Adversary activity yet). Per §6.1 liveness I won't idle-block: I keep M0 claimed and proceed
with M1 (independent infra build), without advancing to M2 until M0 shows PASS.

**Next:** M1 — Docker + single-node swarm via Nix (modules/swarm.nix), then Traefik (file provider
→ /var/lib/ci-certs/live/) + abra, then a by-hand HTTPS deploy/teardown of a trivial recipe.

## 2026-05-26 — M1: Docker + single-node swarm via Nix

**modules/swarm.nix:** `virtualisation.docker.enable` + daily autoprune (--all --volumes until=24h
to protect the 28 GiB root), `docker` in systemPackages, and a `swarm-init` oneshot
(`docker swarm init --advertise-addr 127.0.0.1` if not active; `docker network create --driver
overlay --attachable proxy` if absent). Imported into configuration.nix.

**Build + switch:** `nixos-rebuild build --flake .#cc-ci` → EXIT 0; `systemd-run … switch` →
`Result=success`.

**Verify (commands + output):**
- `systemctl show swarm-init -p Result` → `Result=success`
- `docker info --format ...` → `Swarm=active Managers=1 Nodes=1`
- `docker network ls --filter name=proxy` → `proxy overlay swarm`
- `systemctl is-system-running` → `running`; `--failed` → none.

**Next:** Traefik as a swarm stack (Nix-declared compose + `docker stack deploy` oneshot): docker
swarm provider + file provider serving /var/lib/ci-certs/live/{fullchain,privkey}.pem on :443,
attached to `proxy`. Then abra install + by-hand HTTPS deploy/teardown of a trivial recipe (M1 gate).
Rationale for swarm-service Traefik over a host `services.traefik`: a host process isn't on the
`proxy` overlay, so it can't reach swarm service VIPs; coop-cloud recipes assume an on-`proxy`
Traefik watching swarm labels.

## 2026-05-26 — M1: Traefik swarm stack + HTTPS path proven

**modules/traefik.nix:** Traefik v3.3 as a swarm service on `proxy` (so it reaches recipe VIPs).
Config via Nix `writeText` store files bind-mounted into the container (real files, not /etc
symlinks): static `traefik.yml` (entrypoints web/websecure; `providers.swarm` unix socket,
exposedByDefault=false, network=proxy; `providers.file` dir /etc/traefik/dynamic; ping; no
dashboard) and dynamic `certs.yml` (wildcard at /var/lib/ci-certs/live/* as `stores.default.
defaultCertificate` + certificates — so any *.ci.commoninternet.net router with tls=true is covered,
no ACME). Deployed by a `traefik-deploy` oneshot (`docker stack deploy`) after swarm-init. Opened
firewall 80/443 (gateway forwards over enp5s0).

**Build + switch:** build EXIT 0; switch `Result=success`; `traefik-deploy` `Result=success`;
`docker service ls` → `traefik_traefik traefik:v3.3 1/1`.

**Verify (commands + output):**
- Local: `curl -ksv -H 'Host: probe-test.ci.commoninternet.net' https://localhost/` →
  `subject: CN=*.ci.commoninternet.net`, `issuer: …Let's Encrypt; CN=E8`, TLSv1.3, HTTP 404.
- **End-to-end via gateway:** `curl -ksv --resolve probe-test.ci.commoninternet.net:443:143.244.213.108
  https://probe-test.ci.commoninternet.net/` → `Connected to …(143.244.213.108) port 443`,
  same wildcard cert, HTTP 404. Confirms gateway SNI-passthrough → cc-ci Traefik TLS termination.
  404 is correct (no router for that host yet).

**Next:** install abra (M1 last task), `abra app new` a trivial recipe (custom-html) → deploy →
reach over HTTPS at <app>.ci.commoninternet.net → teardown leaving no volumes. That completes M1
→ CLAIM M1 gate.

## 2026-05-26 — M1: proxy pivot to real coop-cloud/traefik via abra; recipe deploy/teardown (M1 CLAIMED)

**Orchestrator decision (mid-M1):** replace the hand-rolled Traefik with the canonical Co-op Cloud
`traefik` recipe deployed via abra, wildcard/file-provider mode, no ACME/token. Removed custom
`modules/traefik.nix`; moved firewall 80/443 into `modules/swarm.nix`. Recorded in DECISIONS.md.

**Why the pivot also fixed a real bug:** my custom Traefik used entrypoint `websecure`; coop-cloud
recipes label `entrypoints=web-secure`. While chasing that I also hit a sharp **systemd-run gotcha**:
`systemd-run … nixos-rebuild switch --flake .#cc-ci` runs with cwd `/`, so `.#` → `/` → "could not
find a flake.nix"; the switch silently failed while a post-`--collect` `systemctl show` returned a
stale `Result=success`. Fix: always use the **absolute** flake path `/root/cc-ci#cc-ci`, and read the
result before resetting. (rebuild6/7 had silently not applied; rebuild2–5 used the absolute path.)

**abra packaged** (modules/abra.nix): release binary 0.13.0-beta, pinned by sha256, autoPatchelf'd.
`abra --version` → `0.13.0-beta-06a57de`.

**scripts/deploy-proxy.sh** (idempotent, pure-bash — host has no python3): ensure local abra server,
fetch traefik, write wildcard/no-ACME env (`WILDCARDS_ENABLED=1`, `SECRET_WILDCARD_*_VERSION=v1`,
`COMPOSE_FILE=compose.yml:compose.wildcard.yml`, `LETS_ENCRYPT_ENV=` empty), insert cert secrets via
`abra app secret insert … -f` from /var/lib/ci-certs/live, deploy. Bugs fixed en route: multi-line
PEM must use `-f` (not arg); secret-presence must check `docker secret ls` (abra's recipe list always
shows the name with `created on server:false`).

**Traefik deploy:** `abra app deploy` → `deploy succeeded 🟢` (traefik v3.6.15 + socket-proxy).
Verify: `docker service ls` → app+socket-proxy 1/1; via gateway `curl --resolve probe.*:443:
143.244.213.108` → `CN=*.ci.commoninternet.net` (LE E8); **0 ACME log lines**.

**M1 gate (recipe over HTTPS + teardown):**
- `abra app new custom-html -s default -D cchtml1.ci.commoninternet.net -S -n` then set
  `LETS_ENCRYPT_ENV=` and `abra app deploy -n -C` → `🟢` (nginx 1.29.0).
- `curl -ks --resolve cchtml1.ci.commoninternet.net:443:143.244.213.108 https://…/` →
  `http_code=200 size=615`, served the nginx welcome page over HTTPS with the wildcard cert.
- Teardown: `abra app undeploy -n` → 🟢; `abra app volume remove -f -n` → "1 volumes removed";
  leak check → services 0 / volumes 0 / secrets 0 / containers 0. **Clean.**
- Correct teardown syntax confirmed: `secret remove <d> --all -n` (not `--all-secrets`).

**docs/install.md** seeded (flake apply + deploy-proxy + verify). M1 gate CLAIMED in STATUS.md.

**Next:** M2 — Drone server + exec runner via Nix, Gitea OAuth app, hello-world .drone.yml green.

## 2026-05-26 — M2 start: CI engine decision + Gitea OAuth app + Drone secrets

**Decision (DECISIONS.md):** keep Drone per plan. nixpkgs 24.11 has drone server 2.24.0 but only the
abandoned `drone-runner-exec` (unstable-2020) — accepted (stable RPC), Woodpecker is the documented
fallback. Deploy shape mirrors traefik: server via coop-cloud `drone` recipe (abra, swarm,
traefik-routed at drone.ci.commoninternet.net, no ACME), exec runner as a host Nix systemd service.

**Recipe recon:** coop-cloud `drone` recipe = drone/drone:2.26.0, secrets `rpc_secret` +
`CLIENT_SECRET` (Gitea OAuth), Gitea SSO via `compose.gitea.yml` (`GITEA_CLIENT_ID`, `GITEA_DOMAIN`).
Server env: DRONE_SERVER_HOST/PROTO, DRONE_USER_CREATE.

**Done this tick:**
- Created Gitea OAuth app `cc-ci-drone` (bot): client_id `ab4cdb9d-…`, redirect
  `https://drone.ci.commoninternet.net/login`.
- Generated `DRONE_RPC_SECRET` (openssl-equivalent /dev/urandom hex32) + stored client_secret;
  both added to `secrets/secrets.yaml` via `sops set` (needed `SOPS_AGE_KEY` from the host ssh key:
  `ssh-to-age -private-key -i /etc/ssh/ssh_host_ed25519_key`). Verified: decrypt shows keys
  test_secret/drone_rpc_secret/drone_gitea_client_secret; file stays encrypted (4× ENC).

**Next:** scripts/deploy-drone.sh (abra deploy of drone server w/ Gitea SSO + rpc/client secrets),
modules/drone-runner.nix (exec runner systemd unit, rpc secret from sops), wire sops secrets for the
runner, then push a hello-world .drone.yml and confirm a green build (M2 gate).

## 2026-05-26 — M2: Drone server + exec runner up; infra as idempotent-reconcile oneshots

**Orchestrator steer (2×):** collapse install to a single `nixos-rebuild switch` — convert the
manual deploy scripts into **idempotent-reconcile systemd oneshots** (writeShellApplication, embedded
in store; after swarm-init+docker; wants network-online; wantedBy multi-user; reconcile every
activation/boot, NO run-once sentinel; fail visibly on missing cert). Applied to proxy + drone.

**Refactor done:**
- `modules/packages.nix`: `pkgs.abra` overlay (shared pinned build).
- `modules/proxy.nix`: `deploy-proxy` oneshot — reconciles coop-cloud traefik (wildcard/no-ACME).
- `modules/drone.nix`: `deploy-drone` oneshot — reconciles coop-cloud drone (Gitea SSO, secrets from
  /run/secrets), after deploy-proxy.
- `modules/drone-runner.nix`: exec runner (fixed PATH conflict via `lib.mkForce`; allowUnfree for
  drone-runner-exec — Polyform license).
- `modules/secrets.nix`: declared drone_rpc_secret + drone_gitea_client_secret + a sops *template*
  `drone-runner.env` (DRONE_RPC_SECRET) as the runner's EnvironmentFile (shared secret).
- Removed `scripts/deploy-*.sh`. install.md now = clone + nixos-rebuild switch + preconditions.

**Build/switch:** build EXIT 0 (shellcheck clean via writeShellApplication; runner pkg unfree-allowed).
`nixos-rebuild switch` → all three units `active`/`success`:
- `deploy-proxy` success (reconciled traefik), `deploy-drone` → `deploy succeeded 🟢` (drone/drone
  2.26.0, secrets client_secret+rpc_secret v1, drone_env config), `drone-runner-exec` active.

**Verify (commands + output):**
- `docker service ls` → `drone_ci_commoninternet_net_app 1/1`, traefik app+socket-proxy 1/1.
- Via gateway: `…/healthz` → **200**; `/` → **303** (login redirect, correct).
- Runner: journal shows a few startup `cannot ping the remote server (404)` (drone RPC not ready
  yet) then `successfully pinged the remote server` + `polling the remote server capacity=2
  endpoint=https://drone.ci.commoninternet.net kind=pipeline type=exec`. **Runner connected via RPC.**

**Remaining for M2 gate:** push a hello-world `.drone.yml` to cc-ci + get a green build. Needs the
cc-ci repo activated in Drone, which requires the bot's Gitea OAuth login (browser flow) to grant
Drone a Gitea token (to sync repos + set the push webhook). Next tick: script the OAuth login to mint
a Drone token, activate cc-ci, push .drone.yml, confirm green. (DRONE_USER_CREATE made autonomic-bot
the admin.)

## 2026-05-26 — M2 GATE MET: green build via push (Drone + exec runner)

**Drone↔Gitea OAuth (scripted, the one manual bootstrap):** logged the bot into Gitea (CSRF cookie
→ form), drove Drone `/login` → Gitea authorize consent (POST `/login/oauth/grant` with _csrf+state+
granted=true) → code callback → Drone `_session_`. Captured the whole flow in
`scripts/bootstrap-drone-oauth.sh` (reads bot creds from env; documented in install.md §2; one-time,
token persists in Drone's data volume).

**Repo activation:** `GET /api/user` → autonomic-bot admin=true; `GET /api/user/repos?latest=true`
synced 12 repos; `POST /api/repos/recipe-maintainers/cc-ci` → active=true, config_path .drone.yml
(sets the Gitea push webhook).

**Green build:** added `.drone.yml` (exec pipeline), pushed (0d89e28). Polled
`/api/repos/recipe-maintainers/cc-ci/builds` → build #1 pending→running→**success**. Steps:
clone success exit 0; hello success exit 0 — log shows `whoami=root`, `abra 0.13.0-beta-06a57de`,
`swarm=active` (ran on the host via the exec runner). **M2 gate met; CLAIMED.**

**Next:** M3 — comment-bridge service: Gitea issue_comment webhook → verify HMAC + `!testme` exact +
collaborator → resolve PR head repo/SHA → trigger a parameterized Drone build; post a PR comment with
the run link. Need a Drone API token for the bridge (mint from the bot's Drone account).

## 2026-05-26 — M3 start: bridge secrets + comment-bridge source

**Secrets (sops):** minted a Gitea API token (`cc-ci-bridge`, scopes read:org/user, write:repo/issue),
a Drone API token (`POST /api/user/token`, the stable personal token; rotates on call), and a webhook
HMAC (urandom hex64). Stored as bridge_gitea_token / bridge_drone_token / bridge_webhook_hmac via
`sops set` (host age identity). secrets.yaml now holds 6 secrets.

**bridge/bridge.py** (Python stdlib only, §4.1): POST /hook handler — verifies Gitea HMAC
(`X-Gitea-Signature` sha256), requires `X-Gitea-Event: issue_comment`, action=created, body trimmed
== `!testme`, issue is a PR; checks commenter is a collaborator (Gitea collaborators endpoint, 204);
resolves PR head sha+repo; triggers a parameterized Drone build
(`POST /api/repos/<CI_REPO>/builds?branch=main&RECIPE&REF&PR&SRC`, custom params → pipeline env);
posts a PR comment linking the run. Secrets read from mounted files; config via env. `/healthz` GET.

**Next:** package the bridge as a swarm service (dockerTools image, no Docker Hub pull) behind
traefik at `ci.commoninternet.net/hook` via a reconcile oneshot (modules/bridge.nix); register a
per-repo webhook with the HMAC; demo on a scratch PR (!testme triggers; non-!testme + non-collab
rejected). That's the M3 gate.

## 2026-05-26 — M3: bridge deployed + verified; webhook DELIVERY blocked (Gitea-side)

**Deployed** the comment-bridge as a Nix-built OCI image (no Docker Hub pull) → swarm service on
`proxy`, behind traefik at `ci.commoninternet.net/hook`, via reconcile oneshot `modules/bridge.nix`.
Swarm secrets (webhook_hmac/drone_token/gitea_token) materialised from /run/secrets.

**Verified working (bridge side):**
- `docker service ls` → ccci-bridge_app 1/1.
- `GET /hook/healthz` → 200 **from the sandbox over real public DNS** (ci.commoninternet.net →
  143.244.213.108); also 200 via gateway from cc-ci.
- HMAC logic: bad sig → 401; a manually openssl-HMAC-signed body → 204 (passes sig, ignored as
  non-trigger); wrong event → 204. (Debug log added: `got=/want=/bodylen/seclen`.)
- Registered per-repo `issue_comment` webhook (id 210) on recipe-maintainers/cc-ci → ci.../hook with
  the HMAC. Created scratch PR #1.

**Blocker found:** commenting `!testme` (×several) and Gitea's "Test Delivery" (UI returns 200) yield
ZERO requests at the bridge container. Bridge is publicly reachable by hostname from a 3rd network;
gateway accepts public sources; public DNS correct → Gitea is not *sending* the delivery. Deliveries
panel is AJAX (uninspectable via curl); bot is not Gitea admin (can't read `ALLOWED_HOST_LIST`).
Conclusion: git.autonomic.zone webhook policy (likely `ALLOWED_HOST_LIST`) blocks ci.commoninternet.net.
Recorded in STATUS ## Blocked with operator options (whitelist host, or I pivot bridge to polling).

**Plan:** surface to operator; meanwhile proceed to M4 (harness + install stage) which doesn't depend
on the webhook (dev recipe-CI builds triggerable directly via the Drone API). Revisit M3 gate once the
host is whitelisted or via the polling fallback.

## 2026-05-27 — M4: harness + install stage green (custom-html), guaranteed teardown

**Built the harness:** `runner/harness/abra.py` (abra wrappers w/ gotchas: no --chaos on
undeploy/volume-remove, `-n` everywhere, parse `app ls -S -m` nested {server:{apps}}, timeouts),
`runner/harness/lifecycle.py` (deploy_app forcing `LETS_ENCRYPT_ENV=""` [A1], wait_healthy =
services-converged + HTTPS, teardown_app = undeploy+volume+secret+env-config, janitor for orphans),
`tests/conftest.py` (`deployed_app` session fixture with finalizer teardown; short unique domain),
`tests/custom-html/test_install.py` (HTTP 200 + Playwright/Chromium content assertion),
`runner/run_recipe_ci.py` (orchestrator: fetch recipe@REF, run stage pytest), `modules/harness.nix`
(`cc-ci-run` = Nix python3+pytest+playwright with PLAYWRIGHT_BROWSERS_PATH from nixpkgs).

**Bugs fixed en route (3):**
1. Swarm config name > 64 chars (long domain) → switched to short `<recipe[:4]>-<6hex>` domain
   scheme (DECISIONS.md). 
2. `services_converged` used wrong stack name (replaced hyphens) → abra keeps hyphens, only dots→_.
3. `http_get` connected to the gateway IP (drops SNI, gateway routes by SNI) → use the real URL
   (resolves to gateway on cc-ci, correct SNI). Also teardown now removes the app .env config.

**Green run + teardown (commands + output):**
- `RECIPE=custom-html PR=0 REF=m4demo cc-ci-run runner/run_recipe_ci.py` →
  `tests/custom-html/test_install.py::test_http_reachable PASSED`,
  `::test_playwright_page PASSED` — **2 passed in 57.99s**.
- Leak check after: services 0 / volumes 0 / secrets 0 / containers 0 / env config removed. Clean.

**A1 addressed:** deploy_app forces `LETS_ENCRYPT_ENV=""` (no ACME) on every deploy. M4 CLAIMED.

**M3 still blocked** (Gitea webhook delivery — operator); no response yet. Next: M5 (upgrade +
backup/restore for custom-html), then wire the parameterized Drone pipeline (API-triggerable).

## 2026-05-27 — M5: upgrade + backup/restore stages green (custom-html)

**Upgrade stage** (tests/custom-html/test_upgrade.py): deploy previous published version
(git-tag sort, second-newest), write a data marker into the served volume (nginx serves
/usr/share/nginx/html, so the marker is HTTP-fetchable), `abra app upgrade` to current, assert
healthy + marker survived. Fix: `upgrade` has no `--chaos` flag (used `-f -D -n`).

**backup-bot-two** deployed as reconcile oneshot (modules/backupbot.nix): restic repo in a local
`backups` volume, restic_password abra-generated (only if missing). Fixes: `abra app secret generate`
needs `-m` (machine) to avoid the TTY/ioctl path, and stdout redirected so generated values never
hit the journal (D6). `abra app backup create`/`restore` need a real PTY ('input device is not a
TTY') → run via util-linux `script -qec` (harness `_run_pty`; util-linux added to cc-ci-run).

**Backup stage** (test_backup.py): write "original" → `abra app backup create` → mutate to
"mutated" → `abra app restore` → assert state back to "original".

**Full 3-stage run** (`STAGES=install,upgrade,backup`):
- install: 2 passed (http 200 + playwright)
- upgrade: 1 passed (data survives upgrade)
- backup:  1 passed (restore returns pre-mutation state)
- teardown: 0 orphaned run services/volumes/secrets; infra (traefik/drone/bridge/backupbot) all 1/1.
M5 CLAIMED.

**M3 still blocked** (webhook; no operator response across several ticks). Plan: if still blocked,
pivot the bridge to poll the Gitea API (self-service, Adversary-endorsed) to unblock D1. Next: M6.

## 2026-05-27 — Fix adversary findings A2 (dead janitor) + A3 (unverified teardown)

**A2 (janitor matched dead `-pr` filter):** rewrote `harness.lifecycle.janitor` to match the real
run-app naming (`RUN_APP_RE = ^[a-z0-9]{1,4}-[0-9a-f]{6}\.ci\.commoninternet\.net$`), reap via
docker primitives, AND scan `docker service ls` to catch orphans whose `.env` is already gone
(reconstructs the domain from the service name). Age-gated (default 2h, env `CCCI_JANITOR_MAX_AGE`)
so concurrent in-flight runs are never killed.

**A3 (teardown unverified + unconditional .env removal):** `teardown_app` now (1) `docker stack rm`
fallback if `abra undeploy` leaves services, (2) removes volumes/secrets *before* the `.env` and
only drops the `.env` after the stack is confirmed gone, (3) retries docker volume rm (a stopped
task briefly holds the volume), (4) **verifies** no residual services/volumes/secrets and raises
`TeardownError` otherwise — so a partial teardown FAILS the run instead of silently orphaning.

**Re-test (commands + output):**
- Normal install run → 2 passed, verified teardown clean.
- Orphan (deploy, no teardown) → `janitor(CCCI_JANITOR_MAX_AGE=0)` → services/volumes/secrets/env 0.
- **Env-less orphan** (deploy then `rm` the .env, the A3 bad state) → janitor reaps via docker stack
  rm → services/volumes/secrets 0.
- Full 3-stage run (install/upgrade/backup) still green with verified teardown, no TeardownError.

A2/A3 fixed; left for the Adversary to re-test + close.

## 2026-05-27 — M6 (part 1): harness enhancements for recipe #2 + D4 discovery

Before enrolling recipe #2, made the shared harness recipe-agnostic so enrolling a recipe needs no
harness-code change (D5):
- **Per-recipe meta** (`tests/<recipe>/recipe_meta.py`, optional): HEALTH_PATH, HEALTH_OK,
  DEPLOY_TIMEOUT, HTTP_TIMEOUT. conftest reads it; `wait_healthy` gained a `path` param (e.g.
  keycloak `/realms/master`). Defaults preserve custom-html behaviour (verified: install still green).
- **Shared naming** (`harness/naming.py`): single source for the `<recipe[:4]>-<6hex>` domain, used
  by conftest + the orchestrator.
- **D4 recipe-local discovery** (`run_recipe_ci.run_recipe_local`): if a recipe ships `tests/` with
  `test_*.py`, deploy the app, run those tests against the LIVE deployment (contract: env
  `CCCI_BASE_URL` + `CCCI_APP_DOMAIN`), merge as another reported stage, guaranteed teardown. Real
  recipes ship tests/ committed in their repo (clean checkout) → discovered on clone/fetch. (custom-
  html via catalogue is an awkward case — abra refuses an unstaged recipe and `abra recipe fetch`
  resets local commits — so D4 is demonstrated end-to-end with recipe #2 hedgedoc, which ships
  committed tests/.)

**Next:** mirror hedgedoc (postgres+hedgedoc, DB-backed) via the mirror+PR flow with a committed
tests/ dir, write tests/hedgedoc/ (install/upgrade/backup + recipe_meta), run all stages + D4 green.

## 2026-05-27 — M6 (part 2): recipe #2 keycloak install green (DB-backed, no harness surgery)

Enrolled keycloak (recipe #2): keycloak 26.6.2 **+ mariadb 12.2** — genuinely DB-backed/multi-service
(vs custom-html stateless). Added only `tests/keycloak/recipe_meta.py` (HEALTH_PATH=/realms/master,
HEALTH_OK=(200,), 600s timeouts) + `tests/keycloak/test_install.py` (realm-endpoint health +
Playwright admin-console login). **No change to runner/harness code** — the recipe-agnostic harness
(per-recipe meta) handled it (D5 evidence).

Run: `RECIPE=keycloak STAGES=install cc-ci-run runner/run_recipe_ci.py` → 2 passed in 545s (keycloak
is slow: image pull + JVM + mariadb migration). Teardown clean (0 keyc-* services/volumes after).

**Next:** D4 demo via a mirror shipping committed tests/ (recipe-local run against live app); then
keycloak upgrade + backup/restore (DB data survival via a realm marker through the admin API).

## 2026-05-27 — M6: D4 recipe-local discovery + recipe #2 enrolled (CLAIMED)

**D4 recipe-local discovery working.** Demo: pushed a committed `tests/test_recipe_local.py` to the
mirror on branch `recipe-maintainers/custom-html@ci/d4-recipe-local`; ran
`RECIPE=custom-html SRC=recipe-maintainers/custom-html REF=ci/d4-recipe-local STAGES=install` →
install 2 passed, then `===== STAGE: recipe-local (D4) =====` ran the recipe-shipped test against
the LIVE app (CCCI_BASE_URL) → 1 passed. Clean teardown (0 orphans).

**Hard-won abra behaviour (DECISIONS.md):** private mirror clone needs the bot token (per-command
`http.extraHeader`, not persisted/logged). abra commands (`app ls`, `secret generate`, version
resolution) silently `git checkout <tag>` the recipe, dropping a PR branch's files — so (1) all
harness abra calls use `-C -o` (chaos+offline = current checkout, no remote fetch), and (2) D4
snapshots the recipe's tests/ to a temp dir right after fetch (later abra cmds still reset it).
Traced the drop step-by-step: app_new ok, deploy ok, but `secret generate` (no flags) and `app ls`
each reset the checkout.

**Recipe #2 = keycloak** (keycloak + mariadb, DB-backed) install green with only
`tests/keycloak/recipe_meta.py` + `test_install.py` — **no runner/harness change** (D5). custom-html
remains 3-stage green (M5). docs/enroll-recipe.md written.

**M6 CLAIMED.** keycloak's full 3-stage (DB data survival via a realm marker) folds into M6.5.
**Next:** M6.5 — keycloak upgrade/backup, then recipes 3–6 across the remaining D10 categories.

---
## 2026-05-27 — Trigger redesign (polling primary) + resource safety + M3 verified

Session restarted by watchdog (prior tmux died mid-turn with uncommitted bridge WIP). Re-oriented
from STATUS + plan; two orchestrator design changes landed and are now implemented + verified.

**(1) Trigger: POLLING PRIMARY, webhook optional, org-membership auth** (plan §4.1/§1.5; commit
7addb96). Rewrote `bridge/bridge.py`: a poll thread (`poll_loop`, always-on, primary) scans each
`POLL_REPOS` repo's open PRs every 30s for new `!testme`; the `/hook` webhook stays as an optional
admin-registered push optimization. Both share an in-memory comment-id seen-set → a comment seen by
both fires once. First poll marks pre-existing comments seen (no startup re-fire). Authorization now
`GET /orgs/{owner}/members/{user}` (204=member, read-level) + optional `AUTH_ALLOWLIST`, replacing
the admin-requiring `/collaborators/{user}/permission`. Bot never self-registers webhooks.
- Verified org endpoint at read level (bot basic-auth):
  `members/{autonomic-bot,trav,notplants}` → 204; `members/definitely-not-a-member-xyz` → 404.
- Deployed (nixos-rebuild, deploy-bridge reconcile); new container logs:
  `poller (primary) watching ['recipe-maintainers/cc-ci'] every 30s` + `(poll primary + optional webhook)`.
- **End-to-end M3 trigger (poll path):** posted `!testme` on PR #1 (comment 13705, by bot) →
  Drone build **#26** appeared after **6s** (latest was #25); bridge logged
  `[poll] triggered build 26 for cc-ci@d397720a (PR #1, comment 13705) by autonomic-bot`; bridge
  posted back `cc-ci: started CI run for cc-ci @ d397720a → https://drone.ci.commoninternet.net/...`.
  Satisfies D1 (<60s) over the read-only outbound path — no operator webhook whitelist needed.

**(2) Resource safety: bound live test apps** (plan §4.2/§4.3; commit 72ff8e2). MAX_TESTS =
`DRONE_RUNNER_CAPACITY` = 1 (`modules/drone-runner.nix`) → Drone runs ≤1 build at once, queues the
rest natively. Per-build timeout = 60m, reconciled best-effort in `modules/drone.nix`
(`PATCH /api/repos/.../cc-ci {"timeout":60}`, non-fatal). Janitor remains the backstop for
SIGKILL'd/timed-out builds (reaps orphaned run apps at run-start before each deploy).
- Verified on host after rebuild: `DRONE_RUNNER_CAPACITY=1`; deploy-drone logged
  `set cc-ci build timeout = 60m`; Drone API confirms repo `timeout: 60`.

**Gap noted (next item):** `.drone.yml` still only has the `self-test` pipeline — a bridge-triggered
build runs the self-test, NOT `runner/run_recipe_ci.py`. M4/M5 ran the orchestrator by hand
(`cc-ci-run`). Need a recipe-CI pipeline keyed on the `RECIPE` build param (runs
`cc-ci-run runner/run_recipe_ci.py` with STAGES=install,upgrade,backup, `CCCI_JANITOR_MAX_AGE=0`,
`concurrency:{limit:1}`) to connect bridge→Drone→harness end-to-end (required for D2/D10 via real
`!testme`). Added to Build backlog.

**M3 CLAIMED** (gate). Trigger + auth + comment-back demoed live; the webhook-delivery blocker is
moot now that polling is primary.

---
## 2026-05-27 — Bridge→Drone→harness integration (recipe-ci pipeline) wired & green

Closed the gap where a bridge-triggered build ran only the self-test. Split `.drone.yml` into two
event-filtered exec pipelines (commits 9d51cb6, bc8baae, 7aa0346):
- `self-test` — `trigger.event: [push]` (M2 sanity on pushes).
- `recipe-ci` — `trigger.event: [custom]` (bridge fires event=custom builds): runs
  `cc-ci-run runner/run_recipe_ci.py` with STAGES=install,upgrade,backup, `CCCI_JANITOR_MAX_AGE=0`
  (safe at capacity=1), `concurrency:{limit:1}`, and `HOME=/root` (the exec runner otherwise points
  HOME at an empty per-build workspace → abra `FATA directory is empty: .../.abra/servers`).

Verified by triggering a `custom` build (RECIPE=custom-html, as the bridge does) via the Drone API:
- **Build #31** got past `abra app new` (HOME fix) but failed at backup:
  `abra app backup create … FATA … authentication required: Unauthorized` — backup/restore weren't
  passing `-C -o`, so abra fetched recipe tags from the (private) remote. Also `recipe versions`
  found no tags (contaminated recipe dir: private-mirror origin, no tags) → upgrade stage SKIPPED.
- Fixes: `abra.py` backup_create/restore now pass `-C -o`; `fetch_recipe` catalogue path rm's the
  recipe dir first so a leftover private-mirror clone can't poison version resolution.
- **Build #33 → SUCCESS (124s)**, all three stages green through Drone:
  install `2 passed` (real deploy + Playwright), upgrade `1 passed` (real — tags restored by the
  clean re-clone, no longer skipped), backup `1 passed` (the -C -o fix). Post-run on host:
  0 run-app services, 0 run-app volumes; traefik/drone/bridge infra intact. Event filtering works
  (only recipe-ci ran, not self-test).

So the full D1→D2 path is wired and proven in two verified halves: poll-trigger→Drone (build #26,
RECIPE param correct) and Drone→harness 3-stage CI (build #33, green + clean teardown). Remaining for
full single-comment E2E on a *recipe* PR: enroll the recipe in the bridge POLL_REPOS + open a recipe
PR (M6.5/M10 breadth work).

**Adversary findings status (signal for re-test):** A2 (janitor `-pr` filter) and A3 (teardown
verification + `.env`-last ordering) are both already fixed in the current code
(`lifecycle.RUN_APP_RE` hashed-scheme match; `teardown_app` `_residual()` raise + `docker stack rm`
fallback) — awaiting the Adversary's kill-probe re-test on an idle host. A4 (concurrent same-recipe
collision): its named root cause "no Drone concurrency cap (capacity=2)" is eliminated by
MAX_TESTS=capacity=1 — no concurrent runs possible on this single node, so the shared-recipe-dir race
can't occur. No Builder fix outstanding on findings; next milestone work is M6.5 breadth.

---
## 2026-05-27 — M6.5: keycloak full 3-stage GREEN through the Drone recipe-ci pipeline

Ran keycloak (DB-backed, SSO/identity category) end-to-end via the integrated recipe-ci pipeline
(triggered `custom` build #39, RECIPE=keycloak). **Build #39 → success (~31m)**, all three stages
green as separate reported stages:
- install `2 passed` (8m30s): `test_realm_endpoint_healthy` (/realms/master 200) + Playwright admin
  console login.
- upgrade `1 passed` (10m10s): `test_upgrade_preserves_realm` — realm marker written pre-upgrade
  survives the previous→latest upgrade (DB data survival).
- backup `1 passed` (8m15s): `test_backup_mutate_restore` — backup→mutate→restore returns original.
Clean teardown verified on host: 0 keyc services, 0 keyc volumes. keycloak cold start is slow on
this VM (Quarkus augmentation ~80s + Liquibase schema init), so each deploy is ~5-8m — well within
the 60m build timeout; that's why the run took ~31m. No harness surgery (D5): keycloak runs off
`tests/keycloak/{recipe_meta,test_install,test_upgrade,test_backup}.py` + `kc_admin.py` only.

This both advances M6.5 (first DB-backed recipe full 3-stage) and confirms the recipe-ci integration
works on a heavy DB-backed recipe (Drone→harness→3 stages→teardown). Next M6.5: enroll recipes 3–6
covering the remaining D10 categories (stateful-no-DB, multi-service+S3, large-volume, etc.).

---
## 2026-05-27 — M6.5: cryptpad (recipe #3) enrolled + full 3-stage green; fixed a real backup bug

Enrolled **cryptpad** (stateful, no external DB — the D10 "stateful/no-DB" category). No shared-harness
surgery beyond a *generic* feature: added per-recipe **EXTRA_ENV** (recipe_meta.py dict or
domain-callable) applied in `deploy_app` at every deploy path. cryptpad uses it for its required
distinct `SANDBOX_DOMAIN` (a sibling subdomain under the wildcard, so no cert work). Data-survival
tests write a marker into the backed-up `cryptpad_data` volume and read it via `exec_in_app`
(cryptpad's datastore isn't HTTP-served like custom-html).

Host runs (HOME=/root, cc-ci-run): install **2 passed** (~2m; http 200 + Playwright loads cryptpad),
upgrade **1 passed** (~1m; marker survives previous→current), backup **1 passed** after a fix
(below). Clean teardown (0 cryp services/volumes).

**Real bug found+fixed — backups were silently mis-wired (set_env newline).** cryptpad backup first
failed: `abra app backup create` → backup-bot-two's `/usr/bin/backup` raised
`KeyError: 'RESTIC_REPOSITORY'`. Root cause: backup-bot-two's `.env.sample` ends with a *newline-less*
comment line, and the reconcile's `set_env` did a bare `printf >> .env`, gluing
`RESTIC_REPOSITORY=/backups/restic` onto that comment → commented out. abra `--debug` confirmed the
backupbot env map lacked `RESTIC_REPOSITORY`, and `docker exec backupbot printenv RESTIC_REPOSITORY`
was empty. Fix: `set_env` now ensures a trailing newline before appending (modules/backupbot.nix +
modules/drone.nix, same latent bug). After rebuild: `.env` has a clean `RESTIC_REPOSITORY=` line, the
backupbot container has `RESTIC_REPOSITORY=/backups/restic`, and cryptpad backup→mutate→restore
passes. NOTE: keycloak backup (build #39) passed off an *earlier, non-corrupted* backupbot deploy;
worth a re-verify, but the mechanism is now correct/reproducible. Triggered Drone build #46 (cryptpad)
as the canonical recipe-ci run.

---
## 2026-05-27 — M6.5: matrix-synapse (recipe #4, DB+media/large-volume) full 3-stage green

Enrolled matrix-synapse (synapse `app` + postgres `db` + nginx `web`) — the large-volume/DB+media
D10 category. No harness surgery (server_name = DOMAIN; no EXTRA_ENV needed). Host runs (cc-ci-run):
install **2 passed** (~2.7m; client API 200 + real `/_matrix/client/versions` JSON), upgrade
**1 passed** (~2.3m; postgres marker survives previous→current), backup **1 passed** (~1.5m). Clean
teardown (0 matr services). The data-survival tests use a `ci_marker` postgres row exec'd via
`psql` in the `db` service — this exercises the recipe's real DB-dump backup hook
(`backupbot.backup.pre-hook=/pg_backup.sh backup` / `restore.post-hook`), the meaningful matrix data
path (not a plain volume copy). Worked first try (the set_env/RESTIC fix holds for hook-based
backups too). Triggering the canonical Drone recipe-ci run.

4 of 6 D10 recipes now green: custom-html (simple), keycloak (SSO/DB), cryptpad (stateful/no-DB),
matrix-synapse (DB+media/large-volume). Remaining categories: multi-service+S3 (lasuite-docs) and
TLS-passthrough (bluesky-pds).