cc-ci/machine-docs/JOURNAL-5.md

# JOURNAL — cc-ci Phase 5

## 2026-05-31 — Phase 5 boot

Phase 5 starting. System state verified:
- cc-ci: `systemctl is-system-running` → running; 0 failed units
- Docker services: ccci-bridge 1/1, ccci-dashboard 1/1, drone 1/1, traefik 1/1
- Bridge: 1/1 (container-based, logs via `docker service logs ccci-bridge_app`)

**Sandbox recipe chosen:** `custom-html-tiny` (simple static-web-server; short timeouts; existing
install_steps.sh hook; generic harness; ideal for upgrade-flow testing with minimal CI runtime).

**Existing open PRs on custom-html-tiny mirror:**
- #1 `serve-hidden-files` branch — "chore: publish 1.0.2+2.38.0 release" (feature + version bump,
  NOT from upstream main, NOT merged upstream, from 2026-05-25). Will be closed as superseded when
  we open the upgrade PR (expected V7 behavior).

**Available upgrades for custom-html-tiny:**
- `app` service (joseluisq/static-web-server): 2.38.0 → 2.42.0
- `git` service (alpine/git, compose.git-pull.yml): v2.36.3 → v2.52.0
- New version label: 1.1.0+2.42.0

## 2026-05-31 — V3: recipe-upgrade flow starting

Following SKILL.md procedure for /recipe-upgrade custom-html-tiny:
Step 1 (Plan): fetched recipe, found upgrades available — see above.
Step 2 (Implement): upgrading image tags on cc-ci; bumping version label; committing.
Step 3: open-recipe-pr.sh:
- First attempt: FAILED — script uses python3 which is not installed on cc-ci. Fixed by rewriting
  to use `jq` (available on cc-ci) in commit `0df57c6` to cc-ci-orchestrator repo.
- Second attempt: SUCCESS. Closed PR #1 (`serve-hidden-files`) as superseded, pushed branch
  `upgrade-1.1.0+2.42.0`, opened PR #2 at https://git.autonomic.zone/recipe-maintainers/custom-html-tiny/pulls/2
Step 4: testme-on-pr.sh:
- Initial post: posted !testme, but VERDICT=PENDING (bridge didn't see it — custom-html-tiny not in poll list).
- Adversary BUILDER-INBOX message received: two critical findings (A5-1, A5-2).

## 2026-05-31 — Adversary findings A5-1, A5-2 — both FIXED

A5-2 (CRITICAL): testme-on-pr.sh cannot read verdicts — bridge never posts commit statuses.
- Root cause: bridge only posts PR comments; testme-on-pr.sh reads Gitea commit statuses.
- Fix: Added `post_commit_status()` to bridge.py. Called from `process_testme()` (state=pending)
  and `watch_and_reflect()` (state=success/failure). Commit `5d48436`.
- Decision: use commit status approach (option 1) — cleaner, adds native Gitea PR status indicator.
  Recorded in DECISIONS.md.

A5-1: custom-html-tiny not in bridge poll list.
- Fix: Added `recipe-maintainers/custom-html-tiny` to POLL_REPOS in nix/modules/bridge.nix.
  Commit `5d48436`.
- Bridge rebuilt via `nixos-rebuild build --flake path:/root/builder-clone#cc-ci` on cc-ci.
- Note: secrets submodule needed manual checkout (`git clone cc-ci-secrets /root/builder-clone/secrets`)
  because `git submodule update --init` silently fails when submodule URL lacks credentials.
- Bridge redeployed via `/nix/store/asn4.../cc-ci-reconcile-bridge`, new image `cc-ci-bridge:3761c4221042`.
- Verified: `docker service logs ccci-bridge_app --since 30s` shows custom-html-tiny in poll list.

Next: re-post !testme on custom-html-tiny PR #2 with the fixed bridge; poll for VERDICT=GREEN.

## 2026-05-31 — V3 COMPLETE; V1/V2 partial; testme-on-pr.sh fix

testme-on-pr.sh fix committed (orchestrator repo 6910b19): now reads cc-ci/testme context URL.

Build #29 evidence:
- Params: RECIPE=custom-html-tiny REF=156a49acc... PR=2 stages=install,upgrade,backup,restore,custom
- Results: install PASS, upgrade PASS (1.0.0+2.38.0→1.1.0+2.42.0), backup/restore/custom N/A
- Bridge commit status posted: cc-ci/testme state=success url=.../cc-ci/29 @2026-05-31T13:56:19
- PR comment updated with 🌻 success banner

V2 GREEN verified: POST=0 → VERDICT=GREEN BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/29

V7 verified: mirror main = upstream main (435df8fc); PR#1 (serve-hidden-files) closed as superseded.

Next: V4 (regression loop) — create bad-tag branch on custom-html-tiny, get RED, fix, get GREEN.

## 2026-05-31 — Bootstrap/access checks + V4 regression loop complete

Bootstrap probes from the builder clone:
- `ssh cc-ci "hostname && whoami && nixos-version"` → `cc-ci` / `root` / `24.11.20250630.50ab793 (Vicuna)`
- `set -a; . /srv/cc-ci/.testenv; set +a; curl -s https://$GITEA_URL/api/v1/version` → `{"version":"1.24.2"}`
- `getent ahostsv4 probe-12345.ci.commoninternet.net` → `91.98.47.73` (STREAM/DGRAM/RAW)

V4 red side:
- `POST=0 MAX_WAIT=15 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh custom-html-tiny 5`
  → `VERDICT=RED`
  → `BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/34`
- `curl -fsSL https://ci.commoninternet.net/runs/34/results.json` → install=`pass`, upgrade=`fail`, clean_teardown=`true`, no_secret_leak=`true`

V4 fix on cc-ci host (same recipe PR branch):
- `git -C /root/.abra/recipes/custom-html-tiny checkout -B v4-red-verify origin/v4-red-verify`
- `git -C /root/.abra/recipes/custom-html-tiny checkout origin/upgrade-1.1.0+2.42.0 -- compose.yml compose.git-pull.yml`
- `git -C /root/.abra/recipes/custom-html-tiny -c user.name='autonomic-bot' -c user.email='autonomic-bot@git.autonomic.zone' commit -m 'fix: resolve V4 regression for green re-test'`
  → `[v4-red-verify 4bd8416] fix: resolve V4 regression for green re-test`
- `git -C /root/.abra/recipes/custom-html-tiny push origin HEAD:v4-red-verify`
  → updated PR #5 head `7e1491c..4bd8416`

V4 green side:
- `MAX_WAIT=300 INTERVAL=10 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh custom-html-tiny 5`
  → `VERDICT=GREEN`
  → `BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/37`

Adversary follow-up:
- `REVIEW-5.md` follow-up (`review(5)` commit `e87782a`) closed A5-1 and A5-2 after a fresh cold re-test.
- `BUILDER-INBOX.md` noted that `POST=0` must be env-prefixed in `STATUS-5.md`; corrected here and the inbox is being consumed now.

Next: V5 default stale-test case, then V6 `--with-tests`.

## 2026-06-01 — Adversary finding A5-3 fixed; helper paths corrected

Adversary review+inbox reported a real V2 rerun bug: on a re-`!testme` against the same PR head,
`POST=1 testme-on-pr.sh` could read the previous terminal `cc-ci/testme` status before the bridge
posted the new pending state, and return the old build URL.

Fix authored in the orchestration repo helper:
- `testme-on-pr.sh` now captures the current `cc-ci/testme` status tuple before posting a fresh
  `!testme`, then ignores that unchanged tuple while polling. It returns only once the status changes
  to the new run's state/URL.
- `ci-test-review/{verify-pr.sh,run-all-recipes.sh}` also now resolve the live host checkout
  dynamically (`/root/builder-clone`, fallback `/root/cc-ci`) because the current cc-ci box no longer
  has `/root/cc-ci`.

Verification:
- `bash -n /srv/cc-ci-orch/.claude/skills/recipe-upgrade/testme-on-pr.sh && bash -n /srv/cc-ci-orch/.claude/skills/ci-test-review/verify-pr.sh && bash -n /srv/cc-ci-orch/.claude/skills/ci-test-review/run-all-recipes.sh`
  → exit 0
- `cmp -s /srv/cc-ci-orch/.claude/skills/recipe-upgrade/testme-on-pr.sh /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh && echo same`
  → `same`
- `BEFORE=$(...) ; POST=1 MAX_WAIT=80 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh custom-html-tiny 5 ; RC=$? ; AFTER=$(...) ; printf 'RC=%s\nBEFORE=%s\nAFTER=%s\n' "$RC" "$BEFORE" "$AFTER"`
  → `VERDICT=GREEN`
  → `BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/43`
  → `RC=0`
  → `BEFORE=4`
  → `AFTER=5`

Next: consume `BUILDER-INBOX.md` in git, then continue with V5 stale-test candidate selection.