The harness default base (recipe_versions[-2]) resolves to 3.0.0+v2.0.0 for
the open 3.1.0 upgrade PR. That release predates x86_64 support in the
clickhouse entrypoint (added 3.0.1): on this amd64 host it downloads
clickhouse-backup-linux-x86_64.tar.gz — a deterministic HTTP 404 — and with
set -e + a silenced wget the container exits 1 before logging anything,
crash-looping until the deploy times out. The base therefore can never
converge, regardless of the PR content (the published tag is immutable).
This is exactly the case the harness documents for UPGRADE_BASE_VERSION:
a PR adding its version ABOVE the newest published tag, where the true
predecessor is [-1] (3.0.1+v2.0.0), not [-2]. The upgrade tier then tests
the real operator path 3.0.1 -> 3.1.0.
Pairs with recipe-maintainers/plausible#3 (its !testme can only go green
once this lands).
Declare intentional skips + custom-html-tiny functional test; 4-rung level ladder
- recipe_meta.EXPECTED_NA = {rung: reason} lists intentionally-skipped rungs; any
essential rung skipped and not listed is unintentional. Skips still cap the level
(never inflate). results.json: skips:{intentional,unintentional} + level_cap_rung.
- Level ladder = the four essential rungs (install, upgrade, backup/restore,
functional; top = L4). integration & recipe-local are optional, not leveled
(SSO still enforced for the run verdict, unchanged).
- Card shows skipped rungs as INTENTIONAL SKIP (green, reason below) / UNINTENTIONAL
SKIP (amber); level badge gains an expected/gap? third segment.
- custom-html-tiny: functional serve test (exact-byte round-trip + 404); declares
backup_restore intentionally skipped (stateless static server).
Independently verified by the adversary: 138 unit tests pass cold; live full-stage
run on custom-html-tiny green (upgrade tier ran; level 2; correct skips/badge);
clean teardown.
nginx:alpine swarm service serving /var/lib/cc-ci-reports behind traefik
(Host(report.ci.commoninternet.net) + wildcard TLS), deployed by a reconcile
oneshot mirroring dashboard.nix. The /recipe-report skill writes the weekly
HTML pages there; nginx serves them live. report.ci.* already resolves
(wildcard *.ci DNS) and is covered by the wildcard cert.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator preference: each !testme should get its own comment response so a
re-run is visible in the PR timeline. process_testme now always posts a fresh
⏳ placeholder comment; watch_and_reflect edits THAT comment to the result.
(Was: reuse/edit a single marker comment in place — which made re-runs on an
unchanged head invisible, only updating commit status.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Having test_backup.py in custom-html-bkp-bad caused both bad-backup and bad-restore
to fail at the backup tier. Create custom-html-rst-bad with its own cc-ci test dir
that has ops.py+test_restore.py but NO test_backup.py, so:
- backup: only generic test_backup_artifact → PASS (snapshot exists)
- restore: pre_restore writes 'mutated', marker stays 'mutated' after restore → FAIL
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
backup-bot-two ignores backupbot.backup.path labels and always backs up
the full volume, making path-based restore-RED infeasible.
New approach: custom-html-bkp-bad has no pre_backup → marker never seeded
→ backup snapshot has no ci-marker.txt. pre_restore writes 'mutated'.
After restore: marker is MISSING or 'mutated' → test_restore_returns_state FAILS.
upgrade=skip (no version tags) is acceptable since passing_tiers_before=[install,backup].
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
No ops.py::pre_backup for custom-html-bkp-bad → ci-marker.txt never seeded.
test_backup_captures_state asserts marker=='original' → MISSING → FAIL → backup=RED.
This works regardless of backupbot label behavior.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
backupbot-two ignores nonexistent backup paths and backs up the whole
volume, making the bad-path approach unreliable. New approach:
- Create recipe-maintainers/custom-html-bkp-bad on Gitea (custom-html
without backupbot.backup=true label) — SHA 4e584063a99a
- Add tests/custom-html-bkp-bad/recipe_meta.py with BACKUP_CAPABLE=True
so the harness runs the backup tier despite auto-detect returning False
- Without a labeled container, backup-bot-two produces no snapshot →
parse_snapshot_id=None → test_backup_artifact fails → backup=RED ✓
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both compose.yml uploads had empty files due to a bash encoding bug.
Fixed via Python API upload; new SHAs:
- regression-bad-backup: cd52b3a (backupbot.backup.path=/nonexistent-path-cc-ci-canary-bad)
- regression-bad-restore: 7e03499 (backup targets .backup-data subdir + command creates it)
Adversary confirmed bad-install ✓ and bad-upgrade ✓ from run artifacts.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- tests/regression/test_canaries.py: replace `from .conftest import ...`
(relative import fails when not a package) with sys.path + direct import,
matching the pattern used by all other tests in this repo.
- Delete machine-docs/BUILDER-INBOX.md (Adversary inbox consumed).
- Update STATUS-regression.md + JOURNAL-regression.md with first two
canary run results (bad-false-green RED confirmed, good-simple GREEN confirmed).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three canaries (@pytest.mark.canary) drive the real cold CI lifecycle:
- good-simple: custom-html-tiny @ main (435df8fc) — fast signal, expects GREEN
- good-significant: lasuite-docs @ main (290a8ad7) — multi-service, expects GREEN
- bad-false-green: custom-html @ v5-stale-docroot (71e7326a) — expects RED
Semantic teeth: beyond exit-code, each test asserts that specific named tests
ran in results.json stages (test_serving, test_serving_and_frontend, test_content_type).
If an assertion is removed, the named test disappears → regression test fails.
Includes conftest (run_recipe_ci helper + stage_has_{passing,failing}_test),
README (cadence policy, how to run, how to add), and phase state files.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Server regression canaries (tests/regression/, pytest -m canary) are
expensive — run them at milestones (polish/review/release), NOT every
commit. Per-recipe lifecycle tests keep their normal per-PR !testme
trigger. Plus the standing 'never weaken a test to pass' guardrail.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ph1: 3 mirrors cold-verified — lasuite-drive/mailu/mumble all HTTP 200,
empty=false, default_branch=main, HEAD SHAs match upstream exactly.
Ph3: POLL_REPOS has 20 repos; all 9 new recipes present + all have tests/.
Ph2: tests authored (recipe_meta.py, test_health_check, test_branding, PARITY.md)
but builds #153/#154 predate authoring (2026-05-28 vs 2026-06-02). Plan requires
!testme green AFTER authoring. Filing A-mirror-1. Phase 4 deploy NOT blocked.
Ph4 operator deploy: OK to proceed. A-mirror-1 must close before Phase 5 DONE.
Supersede the CronCreate/busybox notes: the weekly /upgrade-all now runs
via the reboot-safe cc-ci-upgrade-all systemd timer in the orchestrator
flake. Records the T0 PASS and the schedule move (Mon 23:04 -> Sun 02:00 UTC).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CronCreate mechanism cold-verified: upgrader-cron.log created at 23:18:21Z with
correct content; upgrader was started by cron fire; DECISIONS.md updated.
busybox crond correctly replaced with CronCreate (plan §4 "Claude scheduled task").
All V1-V9 + §4 cron now PASS within 24h. No open findings, no VETOs.
Builder may write ## DONE to STATUS-5.md.
