journal: rustdesk-server one-off upgrade + enrollment (PR #1 GREEN !testme, cc-ci #48 merged, host switch); memory: gate fixtures must fail loudly + --cc-ci-only deploy path

This commit is contained in:
2026-10-30 21:24:06 +00:00
parent 4e286ba777
commit 93fe2b6f1a
4 changed files with 98 additions and 0 deletions
+52
View File
@@ -1332,6 +1332,37 @@ Key highlights: gitea upstream shipped its first 2-digit stable (28.0.0-rootless
Per-recipe logs: .cc-ci-logs/upgrades/<recipe>-upgrade-2026-10-02.md (10 files incl. survey).
NEVER MERGED anything; all PRs await operator review + the operator's `abra recipe release <recipe> -x/y/z` after each merge.
### Event 2026-10-05 ~17:45 UTC — gitea PR #10 stale test fixed (cc-ci PR #44, verified GREEN, NOT merged)
Operator invoked --with-tests for gitea PR #10 (28.0.0-rootless, RED on test_lfs_roundtrip run #31).
Diagnosis: stale test — `_api()` caught only HTTPError so one urlopen(timeout=20) socket timeout
during the 28.0.0 post-restart migration window escaped the poll loop; poll also hard-coded 120s
deadline vs recipe_meta HTTP_TIMEOUT=600 (STYLE.md §5). Fix: never-raising _api (mirrors
lifecycle.http_fetch) + deadline from recipe_meta.HTTP_TIMEOUT; assertions untouched.
cc-ci PR #44 (branch fix/gitea-lfs-restart-wait) also carries the two PRE-EXISTING main lint
failures (warm.py ruff format; acme-dns.nix statix W201, eval-identical restructure) that had made
push build #33 RED and would have blocked merge — build #34 now success, lint.sh PASS.
Verified: full cold suite on host worktree (e83cd46) vs recipe head 91e42c8 → 5/5 tiers pass,
test_lfs_roundtrip PASSED; unit sample 315 pass / 4 fail = identical pre-existing failures on main.
Pair note posted on both PRs: merge #44, then re-!testme #10. Nothing merged.
### Event 2026-10-05 ~18:00 UTC — wrong-recipe card screenshot root-caused; runner fix #45 merged; gitea !testme re-run GREEN
Operator's !testme on gitea PR #10 (drone build 35) showed a custom-html-tiny screenshot in the card.
Root cause: drone build numbers restarted when its DB was re-created (2026-09-27) while
/var/lib/cc-ci-runs still holds old-era dirs (numbered to 1377) — build 35 collided with a May-31
run 35 whose fossil screenshot.png (12950 B = exactly custom-html-tiny's real shot) was embedded
because screenshot.capture()'s hook branch SKIPPED the snap when out_path existed (no hook saves
itself). Run itself was genuinely gitea level 5. Fix (cosmetics-only, R7): results.fresh_artifact_dir()
wipes owned artifacts at run start + capture() always snaps; 4 unit tests (non-vacuous: capture test
fails on main). Adversary verification PASS (independent fresh clones + own E2E collision test).
cc-ci PR #45 merged (squash, 35ba9976); push self-tests 38/39 green; lint gate now green on main
(#44 had already fixed the two pre-existing lint failures).
Merged #44 first (operator informed; verified earlier via full cold harness run).
Re-ran !testme on gitea PR #10 → drone build 40: GREEN, level 5, card links /runs/40/, screenshot
is a fresh PNG of the run's own gitea (footer 1.24.2 = the upgrade-tier BASE deploy, the last-green
warm canonical — pre-existing capture timing, correct behavior, only freshness was broken before).
Gitea PR #10 now fully green incl. the fixed LFS test (uses merged #44 test). NOTHING else merged;
recipe PR #10 + unrelated PR #4 still await operator merge + `abra recipe release gitea -x`.
Cleanup: host /tmp/v /tmp/r, local worktrees, test-file mis-commit in orch cc-ci clone reverted.
### Event 2026-10-23 04:15 UTC — weekly /upgrade-all run complete (DEFAULT mode, nothing merged)
Run 2026-10-23: 19 considered · 9 upgraded with !testme GREEN · 2 PRs open pending the warm-canonical
minio-prepull fix (lasuite-docs PR #11, lasuite-drive PR #8 — !testme RED at the upgrade BASE prepull:
@@ -1358,3 +1389,24 @@ onlyoffice direct checks) · matrix-synapse #8 · n8n #10 · wordpress #3.
Summary: .cc-ci-logs/upgrades/upgrade-all-2026-10-30.md (per-recipe logs alongside). Step-0 sweep
reclaimed 66.76GB images + orphan stacks (disk 92%->49%); end-of-run reap 0 leaked dev deploys.
NEVER MERGED anything; PRs await operator review + `abra recipe release` after each merge.
### Event 2026-10-30 19:00–21:30 UTC — rustdesk-server one-off upgrade + FULL ENROLLMENT (orchestrator)
One-off upgrade requested from scratch: public mirror `recipe-maintainers/rustdesk-server` created +
synced; recipe PR https://git.autonomic.zone/recipe-maintainers/rustdesk-server/pulls/1 (1.1.12→1.1.16,
branch upgrade-2e65135, unmerged — operator merges). Upstream checked (low-churn, stray 1.4.5 docker
tags = aborted push; CVE fix POV: UDP punch-hole amplification GH #670 + mio bump; scan 0 CVE-ids,
client-side CVEs fixed in client 1.4.9). Dev-deploy converged clean twice, then !== enrollment: tests
authored (status-key health sidecar overlay compose.ccci.yml serving the private-tool-unreachable
real hbbs public key; ports 21115-21119 + UDP; ws 101 on 21118/19; identity-key cross-source assert;
punch-hole-amplification regression test w/ version-gate), committed d71a53c, cc-ci PR #48.
Verify: run 1 RED (custom tier) — hand-rolled version read returned None so the punch-hole gate
didn't skip on the 1.1.12 head; fixed via harness lifecycle.deployed_identity + loud fail on
unparsable tag (1653582, f5f11cd); runs 2+3 GREEN level 5/5 (recipe main with gate skipping, AND
the upgrade head with the punch-hole test passing actively). PR #48 evidence-commented + MERGED
(200); --cc-ci-only deploy: flake.lock cc-ci 2f6787f→a8b836e, lock commit 4e286ba pushed,
/etc/cc-ci ff to a8b836e, detached nixos-rebuild test → health (0 failed, swarm 1/1) → switch
(21:17 UTC). Bridge env confirms POLL_REPOS += recipe-maintainers/rustdesk-server (1/1, no auth
errors). !testme posted on recipe PR #1 via testme-on-pr.sh → build 119 VERDICT=GREEN (level 5,
lint pass, clean_teardown, no_secret_leak; card embedded in PR by bridge). Card saved:
.cc-ci-logs/upgrades/rustdesk-server-testme-20261030.png. Next weekly /upgrade-all picks
rustdesk-server up automatically (used-recipes.md row committed cd1ec01). /root/cc-ci-test-verify
removed. Operator merges PR #1 then `abra recipe release`.
+1
View File
@@ -23,3 +23,4 @@ restarts the loops on boot. Count the lines below to see how often it's happenin
- 2026-05-30 17:03:05 BST — reboot detected; loops auto-started by systemd (resuming phase index 6). boot_id=f565f752-0463-42db-b787-9e0db35a5e3f
- 2026-05-31 03:38:29 UTC — reboot detected; loops auto-started by systemd (resuming phase index 5). boot_id=51c17fc3-8391-4109-bce2-413fbee6f26d
- 2026-08-23 03:11:24 UTC — reboot detected; loops auto-started by systemd (resuming phase index 14). boot_id=8215fc5e-8032-4844-b2b8-e0f8c0d9b13e
- 2026-09-28 19:54:59 UTC — reboot detected; loops auto-started by systemd (resuming phase index 14). boot_id=66dadc4e-7f44-493a-aee9-fa76952d78c3
+2
View File
@@ -1,5 +1,7 @@
# Memory index
- [Enroll gate fixtures must fail loudly](enroll-gate-fixtures-fail-loudly.md) — version-gate reads via lifecycle.deployed_identity, never None-into-if; post-host-merge deploy = --cc-ci-only after merge (staging obsolete)
- [Orchestrator host: Hetzner](orchestrator-host-hetzner.md) — runs on Hetzner cpx22; rebuild cmd, loops-service bounce, git-identity gotcha
- [Push commits to remote](push-commits-to-remote.md) — push to git.autonomic.zone right after every commit in this repo
- [Regression canary cadence](regression-canary-cadence.md) — server E2E canaries run on polish/review/release, not every commit
@@ -0,0 +1,43 @@
---
tags: [cc-ci, enrollment, testing, deploy]
---
# Enroll-verify gate fixtures must fail loudly
## Context
During rustdesk-server enrollment (2026-10-30, cc-ci PR #48) the first full-suite verification
run came back RED only because of the *fixture*, not the app: the punch-hole regression test's
version gate read the deployed image with a hand-rolled `docker service inspect`, got None, and
the `if version is not None and version < (1,1,16)` gate silently decided to RUN on a 1.1.12 head
where pre-fix answering is the expected behavior. An expected-RED looked like a real bug.
## Lesson
- A test gate that cannot decide its condition must `pytest.fail` LOUDLY with the raw value —
never default to None-into-an-if (silently runs → wrong RED) or silent skip (vacuous green).
- cc-ci already has a first-class image reader: `runner/harness/lifecycle.py::deployed_identity(domain)`
→ `{"version","image","chaos"}`; parse the tag from `.image` and strip any `@sha256:` digest.
Hand-rolled service inspects in recipe_meta/custom tests are a duplication hazard.
- Version-gated custom tests skip on pre-fix heads by design (STYLE §2) — that is fine as long
as the version read itself cannot fail silently.
# cc-ci host deploy mechanism (post host-merge, 2026-09+)
To run any merged cc-ci main change on the live host now, use the `--cc-ci-only` path from
/cc-ci-orchestrator-update — merge the PR first (enrollment PRs: comment verification evidence,
then merge — invocation = authorization), then:
```
ssh cc-ci 'sudo -H -u loops bash -lc "cd /srv/cc-ci-orch && nix flake update cc-ci &&
git -c user.name=auto-update -c user.email=auto-update@cc-ci commit -qm \"flake.lock: cc-ci -> main\" -- flake.lock &&
git push -q origin HEAD:main"'
ssh cc-ci 'git -C /etc/cc-ci pull --ff-only --recurse-submodules'
systemd-run --unit=ccci-rebuild-test --collect -E HOME=/root -p WorkingDirectory=/srv/cc-ci-orch \
bash -lc "nixos-rebuild test --flake /srv/cc-ci-orch#cc-ci" # := health checks := then switch
```
The old staging procedure (branch clone at /root/cc-ci-deploy, referenced by the retired
/cc-ci-server-update skill and old enroll SKILL.md step 7 lines) is obsolete — the combined-host
flake takes the cc-ci module from the lockfile, so a BRANCH cannot be staged that way; deploy
after merge instead.