diff --git a/cc-ci-plan/JOURNAL.md b/cc-ci-plan/JOURNAL.md index 90366a0..81caa87 100644 --- a/cc-ci-plan/JOURNAL.md +++ b/cc-ci-plan/JOURNAL.md @@ -1332,6 +1332,37 @@ Key highlights: gitea upstream shipped its first 2-digit stable (28.0.0-rootless Per-recipe logs: .cc-ci-logs/upgrades/-upgrade-2026-10-02.md (10 files incl. survey). NEVER MERGED anything; all PRs await operator review + the operator's `abra recipe release -x/y/z` after each merge. +### Event 2026-10-05 ~17:45 UTC — gitea PR #10 stale test fixed (cc-ci PR #44, verified GREEN, NOT merged) +Operator invoked --with-tests for gitea PR #10 (28.0.0-rootless, RED on test_lfs_roundtrip run #31). +Diagnosis: stale test — `_api()` caught only HTTPError so one urlopen(timeout=20) socket timeout +during the 28.0.0 post-restart migration window escaped the poll loop; poll also hard-coded 120s +deadline vs recipe_meta HTTP_TIMEOUT=600 (STYLE.md §5). Fix: never-raising _api (mirrors +lifecycle.http_fetch) + deadline from recipe_meta.HTTP_TIMEOUT; assertions untouched. +cc-ci PR #44 (branch fix/gitea-lfs-restart-wait) also carries the two PRE-EXISTING main lint +failures (warm.py ruff format; acme-dns.nix statix W201, eval-identical restructure) that had made +push build #33 RED and would have blocked merge — build #34 now success, lint.sh PASS. +Verified: full cold suite on host worktree (e83cd46) vs recipe head 91e42c8 → 5/5 tiers pass, +test_lfs_roundtrip PASSED; unit sample 315 pass / 4 fail = identical pre-existing failures on main. +Pair note posted on both PRs: merge #44, then re-!testme #10. Nothing merged. + +### Event 2026-10-05 ~18:00 UTC — wrong-recipe card screenshot root-caused; runner fix #45 merged; gitea !testme re-run GREEN +Operator's !testme on gitea PR #10 (drone build 35) showed a custom-html-tiny screenshot in the card. +Root cause: drone build numbers restarted when its DB was re-created (2026-09-27) while +/var/lib/cc-ci-runs still holds old-era dirs (numbered to 1377) — build 35 collided with a May-31 +run 35 whose fossil screenshot.png (12950 B = exactly custom-html-tiny's real shot) was embedded +because screenshot.capture()'s hook branch SKIPPED the snap when out_path existed (no hook saves +itself). Run itself was genuinely gitea level 5. Fix (cosmetics-only, R7): results.fresh_artifact_dir() +wipes owned artifacts at run start + capture() always snaps; 4 unit tests (non-vacuous: capture test +fails on main). Adversary verification PASS (independent fresh clones + own E2E collision test). +cc-ci PR #45 merged (squash, 35ba9976); push self-tests 38/39 green; lint gate now green on main +(#44 had already fixed the two pre-existing lint failures). +Merged #44 first (operator informed; verified earlier via full cold harness run). +Re-ran !testme on gitea PR #10 → drone build 40: GREEN, level 5, card links /runs/40/, screenshot +is a fresh PNG of the run's own gitea (footer 1.24.2 = the upgrade-tier BASE deploy, the last-green +warm canonical — pre-existing capture timing, correct behavior, only freshness was broken before). +Gitea PR #10 now fully green incl. the fixed LFS test (uses merged #44 test). NOTHING else merged; +recipe PR #10 + unrelated PR #4 still await operator merge + `abra recipe release gitea -x`. +Cleanup: host /tmp/v /tmp/r, local worktrees, test-file mis-commit in orch cc-ci clone reverted. ### Event 2026-10-23 04:15 UTC — weekly /upgrade-all run complete (DEFAULT mode, nothing merged) Run 2026-10-23: 19 considered · 9 upgraded with !testme GREEN · 2 PRs open pending the warm-canonical minio-prepull fix (lasuite-docs PR #11, lasuite-drive PR #8 — !testme RED at the upgrade BASE prepull: @@ -1358,3 +1389,24 @@ onlyoffice direct checks) · matrix-synapse #8 · n8n #10 · wordpress #3. Summary: .cc-ci-logs/upgrades/upgrade-all-2026-10-30.md (per-recipe logs alongside). Step-0 sweep reclaimed 66.76GB images + orphan stacks (disk 92%->49%); end-of-run reap 0 leaked dev deploys. NEVER MERGED anything; PRs await operator review + `abra recipe release` after each merge. +### Event 2026-10-30 19:00–21:30 UTC — rustdesk-server one-off upgrade + FULL ENROLLMENT (orchestrator) +One-off upgrade requested from scratch: public mirror `recipe-maintainers/rustdesk-server` created + +synced; recipe PR https://git.autonomic.zone/recipe-maintainers/rustdesk-server/pulls/1 (1.1.12→1.1.16, +branch upgrade-2e65135, unmerged — operator merges). Upstream checked (low-churn, stray 1.4.5 docker +tags = aborted push; CVE fix POV: UDP punch-hole amplification GH #670 + mio bump; scan 0 CVE-ids, +client-side CVEs fixed in client 1.4.9). Dev-deploy converged clean twice, then !== enrollment: tests +authored (status-key health sidecar overlay compose.ccci.yml serving the private-tool-unreachable +real hbbs public key; ports 21115-21119 + UDP; ws 101 on 21118/19; identity-key cross-source assert; +punch-hole-amplification regression test w/ version-gate), committed d71a53c, cc-ci PR #48. +Verify: run 1 RED (custom tier) — hand-rolled version read returned None so the punch-hole gate +didn't skip on the 1.1.12 head; fixed via harness lifecycle.deployed_identity + loud fail on +unparsable tag (1653582, f5f11cd); runs 2+3 GREEN level 5/5 (recipe main with gate skipping, AND +the upgrade head with the punch-hole test passing actively). PR #48 evidence-commented + MERGED +(200); --cc-ci-only deploy: flake.lock cc-ci 2f6787f→a8b836e, lock commit 4e286ba pushed, +/etc/cc-ci ff to a8b836e, detached nixos-rebuild test → health (0 failed, swarm 1/1) → switch +(21:17 UTC). Bridge env confirms POLL_REPOS += recipe-maintainers/rustdesk-server (1/1, no auth +errors). !testme posted on recipe PR #1 via testme-on-pr.sh → build 119 VERDICT=GREEN (level 5, +lint pass, clean_teardown, no_secret_leak; card embedded in PR by bridge). Card saved: +.cc-ci-logs/upgrades/rustdesk-server-testme-20261030.png. Next weekly /upgrade-all picks +rustdesk-server up automatically (used-recipes.md row committed cd1ec01). /root/cc-ci-test-verify +removed. Operator merges PR #1 then `abra recipe release`. diff --git a/cc-ci-plan/REBOOTS.md b/cc-ci-plan/REBOOTS.md index ce0845f..b65b838 100644 --- a/cc-ci-plan/REBOOTS.md +++ b/cc-ci-plan/REBOOTS.md @@ -23,3 +23,4 @@ restarts the loops on boot. Count the lines below to see how often it's happenin - 2026-05-30 17:03:05 BST — reboot detected; loops auto-started by systemd (resuming phase index 6). boot_id=f565f752-0463-42db-b787-9e0db35a5e3f - 2026-05-31 03:38:29 UTC — reboot detected; loops auto-started by systemd (resuming phase index 5). boot_id=51c17fc3-8391-4109-bce2-413fbee6f26d - 2026-08-23 03:11:24 UTC — reboot detected; loops auto-started by systemd (resuming phase index 14). boot_id=8215fc5e-8032-4844-b2b8-e0f8c0d9b13e +- 2026-09-28 19:54:59 UTC — reboot detected; loops auto-started by systemd (resuming phase index 14). boot_id=66dadc4e-7f44-493a-aee9-fa76952d78c3 diff --git a/memory/MEMORY.md b/memory/MEMORY.md index a46baae..30f3bf5 100644 --- a/memory/MEMORY.md +++ b/memory/MEMORY.md @@ -1,5 +1,7 @@ # Memory index +- [Enroll gate fixtures must fail loudly](enroll-gate-fixtures-fail-loudly.md) — version-gate reads via lifecycle.deployed_identity, never None-into-if; post-host-merge deploy = --cc-ci-only after merge (staging obsolete) + - [Orchestrator host: Hetzner](orchestrator-host-hetzner.md) — runs on Hetzner cpx22; rebuild cmd, loops-service bounce, git-identity gotcha - [Push commits to remote](push-commits-to-remote.md) — push to git.autonomic.zone right after every commit in this repo - [Regression canary cadence](regression-canary-cadence.md) — server E2E canaries run on polish/review/release, not every commit diff --git a/memory/enroll-gate-fixtures-fail-loudly.md b/memory/enroll-gate-fixtures-fail-loudly.md new file mode 100644 index 0000000..3109c4a --- /dev/null +++ b/memory/enroll-gate-fixtures-fail-loudly.md @@ -0,0 +1,43 @@ +--- +tags: [cc-ci, enrollment, testing, deploy] +--- + +# Enroll-verify gate fixtures must fail loudly + +## Context + +During rustdesk-server enrollment (2026-10-30, cc-ci PR #48) the first full-suite verification +run came back RED only because of the *fixture*, not the app: the punch-hole regression test's +version gate read the deployed image with a hand-rolled `docker service inspect`, got None, and +the `if version is not None and version < (1,1,16)` gate silently decided to RUN on a 1.1.12 head +where pre-fix answering is the expected behavior. An expected-RED looked like a real bug. + +## Lesson + +- A test gate that cannot decide its condition must `pytest.fail` LOUDLY with the raw value — + never default to None-into-an-if (silently runs → wrong RED) or silent skip (vacuous green). +- cc-ci already has a first-class image reader: `runner/harness/lifecycle.py::deployed_identity(domain)` + → `{"version","image","chaos"}`; parse the tag from `.image` and strip any `@sha256:` digest. + Hand-rolled service inspects in recipe_meta/custom tests are a duplication hazard. +- Version-gated custom tests skip on pre-fix heads by design (STYLE §2) — that is fine as long + as the version read itself cannot fail silently. + +# cc-ci host deploy mechanism (post host-merge, 2026-09+) + +To run any merged cc-ci main change on the live host now, use the `--cc-ci-only` path from +/cc-ci-orchestrator-update — merge the PR first (enrollment PRs: comment verification evidence, +then merge — invocation = authorization), then: + +``` +ssh cc-ci 'sudo -H -u loops bash -lc "cd /srv/cc-ci-orch && nix flake update cc-ci && + git -c user.name=auto-update -c user.email=auto-update@cc-ci commit -qm \"flake.lock: cc-ci -> main\" -- flake.lock && + git push -q origin HEAD:main"' +ssh cc-ci 'git -C /etc/cc-ci pull --ff-only --recurse-submodules' +systemd-run --unit=ccci-rebuild-test --collect -E HOME=/root -p WorkingDirectory=/srv/cc-ci-orch \ + bash -lc "nixos-rebuild test --flake /srv/cc-ci-orch#cc-ci" # := health checks := then switch +``` + +The old staging procedure (branch clone at /root/cc-ci-deploy, referenced by the retired +/cc-ci-server-update skill and old enroll SKILL.md step 7 lines) is obsolete — the combined-host +flake takes the cc-ci module from the lockfile, so a BRANCH cannot be staged that way; deploy +after merge instead.