Adds terraform/ (hcloud provider, cpx32/nbg1/debian-12) and a new
nix/hosts/cc-ci-hetzner/ flake host to provision the cc-ci server on
Hetzner Cloud as an alternative to the Incus cc-nix-test VM.
Stage 1 (Terraform): creates a cpx32 server (4 vCPU / 8 GB / x86 AMD,
Nuremberg), runs nixos-infect (pinned rev 40f62a6, 2026-03-22) to convert
Debian 12 → NixOS 24.11, and reboots into bare NixOS.
Stage 2 (manual, per terraform/README.md): clone cc-ci --recursive,
provision the bootstrap age key, then `nixos-rebuild switch --flake
.#cc-ci-hetzner`.
Verified (throwaway run 2026-05-31, server 134464512, 168.119.126.100):
- terraform apply: cpx32 in nbg1 created in 17 s
- nixos-infect: NixOS 24.11.719113.50ab793786d9 (same nixpkgs pin as flake)
- nixos-rebuild build --flake .#cc-ci-hetzner: exit 0 on server
(131 derivations; all cc-ci modules: tailscale, drone, drone-runner,
bridge, dashboard, harness, swarm, abra, proxy, secrets)
- terraform plan: no changes (idempotent)
- terraform destroy: server + SSH key removed
Age key step (plan §4 Stage 2): operator-pending. Full switch/convergence
requires bootstrap age key at /var/lib/sops-nix/key.txt. Flake builds
without it; activation needs it.
No secrets committed: HCLOUD_TOKEN via env, tfstate gitignored,
networking.nix contains throwaway IP (update per README for production).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
full9: backup tier FAILed with NameError('__file__' not defined) — recipe_meta.py is exec()'d into a
bare namespace so __file__ is undefined. The harness already has runner/ on sys.path + harness imported,
so import lifecycle directly. (restore PASSED on full9 — the data-integrity fix works; this just fixes
the verify probe crashing the backup tier.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root cause (instrumented, DECISIONS 2026-05-30): a DB recipe dumps its data in a backupbot pre-hook,
but if the DB container cycles mid-dump (intermittent on the loaded CI node — full5/6/7 RED, full8
green; NOT OOM/NOT healthcheck) the dump is truncated/absent and restic snapshots an empty path —
abra app backup 'succeeds' yet a later restore silently loses the data (ghost ci_marker).
Fix (additive, recipe-scoped via meta like READY_PROBE): recipe_meta may define BACKUP_VERIFY(domain)
-> bool, a READ-ONLY post-backup integrity probe. When it returns False the harness re-runs the whole
backup (fresh snapshot, re-stabilised db) up to 3x. Recipes without the hook are unaffected. ghost's
BACKUP_VERIFY confirms /var/lib/mysql/backup.sql.gz is a valid non-empty gzip. Weakens no assertion —
it only retries a flaky CAPTURE so P4 restore is RELIABLY exercised, not luck-dependent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
full4 timed out: abra deploy killed at 1200s while the app was at the near-final email_recipients
migration tables (still 0/1). Wall-time = mysql fresh-dir init (~6min, app crash-loops on ECONNREFUSED
until DB ready — no migration progress lost) + ~9-15min schema migration (round-trip-bound, slower
under host load). Not a test weakening — bounded wait (matches discourse), a genuine hang still fails.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per Adversary course-correction (bdef282) + plan-ccci-compose-overlay-policy.md §1: upgrade-to-latest
is MANDATORY. The 0.7.0+3.3.1 from-version pins the Docker-Hub-removed bitnami/discourse:3.3.1 (404)
and ships a too-tight 5m start_period for the 15-25min Rails cold boot. Minimal base overlay
compose.ccci.yml re-pins app+sidekiq to bitnamilegacy/discourse:3.3.1 (namespace-only, identical
image — same re-pin the PR head makes) + widens start_period to 20m (grace-only). install_steps.sh
provides it; CHAOS_BASE_DEPLOY skips the clean-tree gate; UPGRADE_BASE_VERSION=0.7.0+3.3.1 sets the
true predecessor. Neither change weakens a test. Run shape returns to STAGES=install,upgrade,backup,
restore,custom.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
autonomic-botandClaude Sonnet 4.6