cc-ci-orchestrator/cc-ci-plan/test-e2e-testme-acceptance.md

Acceptance test — real end-to-end !testme on the clean-room-rebuilt VM

Owner: the Builder + Adversary loops (they execute and independently verify this). When: after C4/C5 PASS (genuine throwaway-VM clean-room rebuild verified). The Builder then performs the tailnet swap (§1) and runs the e2e; the Adversary independently verifies. It is the functional acceptance of D8/clean-room: proof that the rebuilt-from-git VM doesn't just match byte-for-byte, but actually serves a real CI run end-to-end through the public domain. This file: /srv/cc-ci/cc-ci-plan/test-e2e-testme-acceptance.md

---

0. Why

The reproducibility gates (C1–C5) prove the rebuilt VM is structurally identical and boots clean. This test proves it is operationally a working CI server: a maintainer comment triggers a build, the app deploys and is reachable on its real public URL through the operator's gateway, the test passes, and it tears down — the whole !testme pipeline, on the from-git VM, over the real domain.

---

1. Setup — the Builder performs the tailnet swap (then the e2e)

The rebuilt throwaway must become the live cc-nix-test so that the public gateway routes real ci.commoninternet.net traffic to it (the gateway TLS-passthroughs via MagicDNS to cc-nix-test.taila4a0bf.ts.net and re-resolves every ~10s, so it auto-follows the name). The swap is two reversible tailscale set --hostname commands on VMs you already control — the Builder does it. Do this only after C4/C5 PASS and after the rebuilt VM's full stack (traefik + bridge + drone + dashboard) is up and serving locally.

Order matters (rename the original aside first, or the throwaway will get cc-nix-test-1):

1. Rename the original prod VM aside (it stays running — do NOT destroy it; needed for swap-back):

   ssh cc-ci 'tailscale set --hostname=cc-nix-test-orig'
   

   (ssh cc-ci is pinned to the original's IP 100.90.116.4, so it keeps reaching the original regardless of the name change.)
2. Rename the rebuilt throwaway → cc-nix-test. Re-derive its current tailscale IP (throwaways get a fresh IP each rebuild): pick the ONLINE throwaway node from tailscale status | grep -i throwaway, then:

   ssh -i /srv/incus-terraform-nix-vm-creator/terraform-secrets/vm_ssh_key \
       root@<throwaway-ip> \
       'tailscale set --hostname=cc-nix-test'
   

   (The orchestrator VM is a direct tailnet peer — no ProxyCommand needed.)

Heads-up — tailnet-wide effect: after the swap, cc-nix-test.taila4a0bf.ts.net resolves to the rebuilt VM for everyone on the tailnet, so any of your own tooling that targets cc-nix-test by MagicDNS name will now hit the rebuilt VM (tooling pinned to the raw IP 100.90.116.4 still hits the original). Account for that when you point !testme/deploys.

Verify the swap took (P1+P2) before starting the e2e — must pass:

tailscale status | grep cc-nix-test   # → the throwaway's IP
curl -sS -o /dev/null -w '%{http_code} ssl_verify=%{ssl_verify_result}\n' https://ci.commoninternet.net/
# expect: 200 ssl_verify=0   (real public path now served by the rebuilt VM, valid cert)

Swap-back when testing is done (reversible): rename the throwaway back to its old name, then ssh cc-ci 'tailscale set --hostname=cc-nix-test' to restore the original; the gateway re-follows.

---

2. Procedure

1. Pick one fast, already-enrolled recipe. Prefer the lightest enrolled app (e.g. custom-html) so the run is quick and resource-cheap. Note the recipe + the repo/issue or PR where !testme is recognised (the same place prior runs were triggered).
2. Record the baseline. Capture the recipe's current latest Drone run number and the dashboard row (https://ci.commoninternet.net/ and https://drone.ci.commoninternet.net/...) so you can prove the run you trigger is new.
3. Trigger via the real path. Post !testme as the bot (the normal maintainer-comment trigger) on that recipe — exactly as a real maintainer would. Do not invoke Drone directly or shortcut the bridge; the comment→bridge→Drone path is part of what's under test.
4. Confirm the bridge picked it up. Within the bridge's poll interval, a new Drone build for that recipe starts. Capture the new run number (must be > the baseline from step 2).
5. Confirm the app deploys and is reachable on its PUBLIC URL. While the build runs, the app is deployed to its *.ci.commoninternet.net test domain. From off the VM (external — through the gateway, not localhost/127.0.0.1), confirm a real request succeeds:

   curl -sS -D- -o /dev/null https://<app-test-subdomain>.ci.commoninternet.net/
   # expect: HTTP 200 (or the app's expected status), valid *.ci.commoninternet.net cert,
   #         served content from the deployed app — NOT a Traefik 404 / default-cert.
   

   This is the crux: it proves routing public-DNS → gateway → MagicDNS → rebuilt VM → Traefik → deployed app all works on the rebuilt server.
6. Confirm the test logic passed. The Drone build runs the recipe's real test assertions (app state, not health-only) and finishes success.
7. Confirm teardown. After the run, the app is undeployed (no leftover stack/containers), per the standard post-run cleanup — verify it's gone.
8. Confirm the result was reported. The outcome posts back to the trigger location and the dashboard row updates to the new run with success.

---

3. Pass criteria (all must hold; Adversary verifies independently)

• E1. Self-check §1 passed (ci.commoninternet.net = 200, valid cert, on the rebuilt VM).
• E2. Posting !testme produced a new Drone build (run # > baseline) via the bridge — not a manual Drone trigger.
• E3. The deployed app answered an external request on its real <app>.ci.commoninternet.net URL (through the gateway) with the expected response + valid cert — captured with headers/body evidence.
• E4. The Drone build's real test assertions ran and the build finished success (no skipped/softened tests).
• E5. The app undeployed cleanly afterward (no residual stack).
• E6. Result reported back + dashboard updated to the new successful run.

Evidence (run #, the external curl headers/body, dashboard before/after, undeploy proof) is logged in JOURNAL-1c.md, and the verdict in REVIEW-1c.md / STATUS-1c.md as E2E-TESTME — PASS.

4. If it fails

Treat as a clean-room finding, not a config patch: a failure here means the from-git rebuild is missing something the running server had out-of-band (a secret, a manual step, drift). Capture the failing stage + logs in JOURNAL-1c.md, raise it as a blocker, and fix it in the git source (base or cc-ci-secrets) so the next rebuild includes it — do not hand-fix the live VM. Re-run this test after the fix.

5. Bound

One recipe, one green run. This is a functional smoke test of the rebuilt VM, not a full recipe-test campaign (that's Phase 2). Don't expand scope here.