cc-ci-orchestrator/memory/MEMORY.md

Memory index

• Orchestrator host: Hetzner — runs on Hetzner cpx22; rebuild cmd, loops-service bounce, git-identity gotcha
• Push commits to remote — push to git.autonomic.zone right after every commit in this repo
• Regression canary cadence — server E2E canaries run on polish/review/release, not every commit
• Recipe-mirrors MUST be public — org public since ~2026-06, but /recipe-enroll created late mirrors (gitea, wordpress) private-from-birth, darkening report STATUS cells; flipped public + skill fixed 2026-09-14
• Gitea Anubis UA challenge — git.autonomic.zone 307-challenges browser UAs to an un-CORS-able origin; server-side proxies representing browser JS must pin a non-browser UA (cc-ci PR #38, 2026-09-14)
• abra chaos-deploy checkout gotcha — abra app new moves recipe checkout to release tag; checkout PR branch after, or chaos deploys wrong tree
• Shared recipe-checkout race — never git-checkout ~/.abra/recipes/ on cc-ci while its CI build runs; harness deploys from that tree
• immich pgvecto.rs DROP DATABASE panic — DROP DATABASE crashes immich's postgres image; use pg_dump --clean --if-exists + search_path rewrite
• Drone sqlite log extraction — copy /data/database.sqlite from drone container, query builds→stages→steps→logs for full step output
• plausible upgrade-base trap — RESOLVED: PR#3 GREEN L4; lessons: check harness base version pre-!testme; backupbot v2 label syntax; TinyLog not FREEZEable; BEAM exit-0 needs restart_policy any
• Swarm UpdateStatus convergence gotchas — N/N is not converged mid stop-first update; paused flag persists forever; only updating/rollback_started are active
• Weekly upgrade queued after phases — 06-12 cron skipped; auto-runs /upgrade-all when phase queue (drone) finishes; don'''t systemctl start the timer
• cfold paused pending upgrade — cfold phase loops+watchdog STOPPED until /upgrade-all (cc-ci-upgrader) finishes; resume = restart watchdog (phase-idx 9)
• proxy VIP exhaustion runbook — TODO after upgrade: enlarge proxy overlay to /16 (exhausts at /24=254 VIPs); root cause of discourse/ghost deploy wedges
• ghost PR debug — TODO after proxy fix: debug+fix the ghost upgrade PR (wedged on proxy VIP exhaustion; possible duplicate PR)
• launch-system unification — replacing 5 launchers + ~15 dotfiles with one agents.toml + agents.py; plan in cc-ci-plan/plan-unified-launch.md
• orchestrator backend-switch gotcha — full claude↔opencode switch: kill watchdog first, fix all control files, env overrides files, pkill -f self-matches
• cc-ci host rebuild procedure — how to nixos-rebuild the live cc-ci server (no self-service path; stage clone + copy sops secrets + drop .git + --flake .#cc-ci)