219 lines
13 KiB
Markdown
219 lines
13 KiB
Markdown
# Phase 5 — BACKLOG
|
||
|
||
SSOT: `/srv/cc-ci/cc-ci-plan/plan-phase5-verify-upgrade-flow.md`. DoD = V1–V9.
|
||
Single-writer: `## Build backlog` = Builder-only; `## Adversary findings` = Adversary-only.
|
||
|
||
---
|
||
|
||
## Build backlog
|
||
|
||
- [x] Create phase 5 state files (STATUS-5.md, BACKLOG-5.md, JOURNAL-5.md)
|
||
- [x] Fix A5-2: Add commit status posting to bridge.py (pending on trigger, success/failure on finish)
|
||
- [x] Fix A5-1: Add custom-html-tiny to bridge POLL_REPOS; redeploy bridge (cc-ci-bridge:3761c4221042)
|
||
- [x] V3: /recipe-upgrade custom-html-tiny end-to-end GREEN (!testme PASS; PR #2 open)
|
||
- [x] V7: mirror reconciliation (PR #1 superseded, PR #4 merged-upstream, main force-synced)
|
||
- [x] V1/V2: !testme trigger + testme-on-pr.sh reads verdict (GREEN on PR #2/#35; RED on PR #5/#34)
|
||
- [x] Fix A5-3: make `POST=1 testme-on-pr.sh` ignore stale prior status on same PR head
|
||
- [x] V4: 3-iteration regression loop (seed bad tag → RED → fix → GREEN in 2 runs)
|
||
- [ ] V5: stale-test DEFAULT = comment, no test edit
|
||
- [ ] V6: --with-tests opens + verifies cc-ci test PR (verify-pr.sh run)
|
||
- [ ] V8: /upgrade-all DEFAULT run (--dry-run list + small live run)
|
||
- [ ] V8a: cc-ci-upgrader agent (launch-upgrader.sh start/stop/status cycle)
|
||
- [ ] V9: cleanup all verification PRs + deploys; install weekly cron (Phase 5 §4)
|
||
|
||
---
|
||
|
||
## Adversary findings
|
||
|
||
### [adversary] A5-5 — V5: explanatory comment references wrong build/failures; no RESULT: SUCCESS-PENDING-TESTS
|
||
**Status:** CLOSED — re-tested 2026-06-01T21:49Z; see `REVIEW-5.md` follow-up entry.
|
||
ORIGINALLY OPEN — found 2026-06-01T21:38Z
|
||
|
||
V5 requires the `recipe-upgrade` skill in DEFAULT mode (no `--with-tests`) to: post an explanatory
|
||
comment that accurately identifies which test is stale + why; and report `RESULT: SUCCESS-PENDING-TESTS`.
|
||
The seeded custom-html evidence does not satisfy both requirements.
|
||
|
||
**Finding 1 — Explanatory comment references build #40, not build #75.**
|
||
The explanatory comment #13883 was posted at 2026-06-01T19:41:22 (before the MIME-only commits
|
||
`ee5cb811`/`71e7326a`) and says: "Observed on `!testme` build `#40`". Build #40 had docroot-path
|
||
failures in three test files (`test_backup.py`, `test_content_roundtrip.py`,
|
||
`test_content_type_header.py`). Build #75 (the final seeded case, ref `71e7326a`) has ONE failure:
|
||
`test_content_type_header.py` MIME type assertion (`application/octet-stream` vs `text/plain`).
|
||
The comment describes a different seeded scenario from the final one — wrong build number, wrong root
|
||
cause, extra test failures that don't appear in build #75.
|
||
|
||
**Finding 2 — No `RESULT: SUCCESS-PENDING-TESTS` produced.**
|
||
No `custom-html-upgrade-*.md` exists in `/srv/cc-ci/.cc-ci-logs/upgrades/`. The V5 evidence uses
|
||
`testme-on-pr.sh POST=1` directly; `/recipe-upgrade custom-html` was not run end-to-end on the
|
||
MIME-only seeded case.
|
||
|
||
**Cold repro:**
|
||
1. Check comment #13883 on `recipe-maintainers/custom-html` PR#3: says "build #40" and docroot-path
|
||
failures.
|
||
2. Check `ci.commoninternet.net/runs/75/results.json`: single failure in `test_content_type_header.py`
|
||
(MIME type), no docroot-path failures.
|
||
3. Run `find /srv/cc-ci* -name "*custom-html*upgrade*"` — no log file produced.
|
||
|
||
**Required fix:**
|
||
Re-run `/recipe-upgrade custom-html` in DEFAULT mode against the existing seeded PR #3 (head
|
||
`71e7326a`). The skill should:
|
||
1. See VERDICT=RED from `testme-on-pr.sh`
|
||
2. Read build #75 failures → only `test_content_type_header.py` (MIME type)
|
||
3. Post a new/updated explanatory comment on PR #3 referencing build #75 and the MIME-type root cause
|
||
4. Write `RESULT: SUCCESS-PENDING-TESTS — custom-html ... recipe PR: ...` to
|
||
`/srv/cc-ci/.cc-ci-logs/upgrades/custom-html-upgrade-<date>.md`
|
||
|
||
(Only Adversary closes this, after re-testing with accurate comment and RESULT line.)
|
||
|
||
---
|
||
|
||
### [adversary] A5-6 — V8: `/upgrade-all uptime-kuma` live run is broken — recipe not enrolled in bridge or tests/
|
||
**Status:** OPEN — found 2026-06-01T21:52Z
|
||
|
||
The V8 live run chose `uptime-kuma` as the test recipe. Two enrollment blockers were found via
|
||
cold verification:
|
||
|
||
**Blocker 1 — uptime-kuma NOT in bridge POLL_REPOS:**
|
||
- Live bridge poll list (from `docker service logs`):
|
||
`['cc-ci','custom-html','custom-html-tiny','keycloak','cryptpad','matrix-synapse','lasuite-docs','lasuite-meet','n8n','hedgedoc']`
|
||
- `uptime-kuma` is absent. So when the upgrader posted `!testme` on PR#1 (comment #13902 at
|
||
`2026-06-01T21:48:39Z`), the bridge will NEVER pick it up.
|
||
- `POST=1 testme-on-pr.sh uptime-kuma 1` will eventually time out and return `VERDICT=PENDING BUILD=?`.
|
||
|
||
**Blocker 2 — uptime-kuma has no tests/ directory in cc-ci:**
|
||
- `ls /root/builder-clone/tests/` on cc-ci shows: `bluesky-pds cryptpad custom-html custom-html-tiny
|
||
discourse _generic ghost immich keycloak lasuite-docs lasuite-drive lasuite-meet mailu
|
||
matrix-synapse mattermost-lts mumble n8n plausible`
|
||
- `uptime-kuma` is absent. Even if the bridge were to trigger a CI run, it would fail with no
|
||
recipe-specific tests to run (only generic harness stages).
|
||
|
||
**Impact:** The V8 live run evidence is invalid — `uptime-kuma` is not a properly enrolled recipe.
|
||
The `/upgrade-all` dry-run survey listed it as a candidate because `abra recipe upgrade` found
|
||
available upgrades, but that check is independent of bridge enrollment or test coverage.
|
||
|
||
**Cold repro:**
|
||
1. `ssh cc-ci '/run/current-system/sw/bin/docker service logs ccci-bridge_app 2>&1 | grep "watching\|uptime"'`
|
||
→ only older poll lists, no `uptime-kuma`
|
||
2. `ssh cc-ci 'ls /root/builder-clone/tests/'` → no `uptime-kuma` directory
|
||
3. `grep uptime /srv/cc-ci/cc-ci-adv/nix/modules/bridge.nix` → no match
|
||
4. Check commit status: `GET /repos/recipe-maintainers/uptime-kuma/commits/728618890a2b/status`
|
||
→ `state:'', total_count:0` after the `!testme` comment was already posted
|
||
|
||
**Required fix:**
|
||
Either (a) for V8 re-run: use already-enrolled recipes (n8n, cryptpad, or lasuite-meet have live
|
||
upgrade candidates) and add them both to POLL_REPOS and have tests/ directories, OR
|
||
(b) first enroll `uptime-kuma` properly: add `tests/uptime-kuma/` to cc-ci and add
|
||
`recipe-maintainers/uptime-kuma` to bridge POLL_REPOS, redeploy bridge, THEN run V8.
|
||
|
||
Also: the `/upgrade-all` skill should validate that a recipe is enrolled (bridge + tests) before
|
||
adding it to the candidate list, or at minimum warn when a candidate lacks tests/.
|
||
|
||
(Only Adversary closes this after cold-verify of a successful live V8 run with an enrolled recipe.)
|
||
|
||
---
|
||
|
||
### [adversary] A5-4 — `matrix-synapse` stale-test/default path leaves no recipe commit status
|
||
**Status:** CLOSED — re-tested 2026-06-01T18:53:30Z; see `REVIEW-5.md` follow-up entry.
|
||
|
||
On the live V5 stale-test candidate `recipe-maintainers/matrix-synapse` PR `#1`, the PR comments show a
|
||
terminal failed `!testme` result for build `#53` plus the default-mode explanatory stale-test comment,
|
||
but the recipe PR head has **no** `cc-ci/testme` commit status at all. As a result, the helper cannot
|
||
read the verdict back from the PR and poll-only returns `PENDING` even though the PR already shows the
|
||
terminal outcome.
|
||
|
||
**Cold repro:**
|
||
1. Use `recipe-maintainers/matrix-synapse` PR `#1`, head
|
||
`21e5d84430bdc52f8fa8aa9a40fa5bda8adf06c0`.
|
||
2. Confirm PR comments include:
|
||
- failure result comment for build `#53` (`#13872`), and
|
||
- explanatory stale-test comment (`#13877`).
|
||
3. Run:
|
||
`POST=0 MAX_WAIT=20 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh matrix-synapse 1`
|
||
4. Observe:
|
||
- helper returns `VERDICT=PENDING` and `BUILD=?`;
|
||
- `GET /repos/recipe-maintainers/matrix-synapse/commits/21e5d84430bdc52f8fa8aa9a40fa5bda8adf06c0/status`
|
||
returns `{"state":"","total_count":0,"statuses":null}`.
|
||
|
||
**Impact:** this breaks the Phase-5 requirement that the upgrade tooling read the verdict back from the
|
||
PR on the live stale-test/default path. The comment surface says the run is terminal; the status surface
|
||
still says nothing.
|
||
|
||
**Re-test result:** no longer reproducible on rerun build `#63`. The recipe PR head now shows
|
||
`cc-ci/testme` `pending -> failure` with target URL `.../63`, and poll-only returns
|
||
`VERDICT=PENDING BUILD=.../63` while in flight, then `VERDICT=RED BUILD=.../63` after completion.
|
||
|
||
### [adversary] A5-3 — `POST=1 testme-on-pr.sh` can return a stale prior GREEN on re-runs
|
||
**Status:** CLOSED — re-tested 2026-06-01T03:31:30Z; see `REVIEW-5.md` follow-up entry.
|
||
|
||
The helper currently posts a fresh `!testme`, then polls the recipe PR head's combined commit status.
|
||
If that PR head SHA already has a previous successful `cc-ci/testme` status and the bridge has not yet
|
||
processed the new comment, the helper exits immediately with the **old** GREEN/build URL instead of a
|
||
fresh `PENDING` or the new run's URL.
|
||
|
||
This is a real Phase-5/V2 correctness bug because re-commenting `!testme` on the same PR head is a
|
||
supported path, and the helper is meant to report the verdict for the run it just triggered.
|
||
|
||
**Cold repro:**
|
||
1. Use an open PR whose current head SHA already has `cc-ci/testme: success` from an earlier run.
|
||
2. Record the PR comment count.
|
||
3. Run:
|
||
`POST=1 MAX_WAIT=40 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh custom-html-tiny 5`
|
||
4. Observe:
|
||
- the PR comment count increases by exactly one (`3 -> 4` in the reproducer), so one fresh `!testme`
|
||
was posted;
|
||
- the helper returns `VERDICT=GREEN` with the **old** build URL
|
||
`https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/37`;
|
||
- later, the live system shows a new run was actually triggered and reflected on the PR as build
|
||
`#41` (`cc-ci/testme pending -> success`, target URL `/41`).
|
||
|
||
**Likely fix direction:** after `POST=1`, do not trust a pre-existing terminal status on the same SHA.
|
||
Poll for evidence that belongs to the newly-triggered run (e.g. a newer status timestamp, a pending
|
||
status after the new comment, or a changed build URL/context generation marker) before returning.
|
||
|
||
### [adversary] A5-2 — CRITICAL: testme-on-pr.sh cannot read verdicts (commit status vs comment mismatch)
|
||
**Status:** CLOSED — re-tested 2026-05-31T19:41:12Z; see `REVIEW-5.md` follow-up entry.
|
||
|
||
`testme-on-pr.sh` reads Gitea commit statuses on the recipe PR's head SHA. But the bridge NEVER
|
||
sets Gitea commit statuses on recipe repos — it only posts PR comments (the YunoHost card+badge).
|
||
Drone posts commit statuses on the `cc-ci` repo (its own repo), not on recipe repos.
|
||
|
||
**Evidence:**
|
||
- `GET /repos/recipe-maintainers/custom-html/commits/db9a95024e9d.../status` → `state:'', statuses:0`
|
||
- `POST=0 testme-on-pr.sh custom-html 2` → `VERDICT=PENDING BUILD=?` (always, on any known-green PR)
|
||
- Bridge source `bridge.py`: no call to `POST /repos/{owner}/{recipe}/statuses/{sha}` anywhere
|
||
|
||
**Required fix (one of):**
|
||
1. (Preferred) Bridge: after triggering a Drone build, POST `state=pending` on the recipe PR's head
|
||
SHA; on build completion, POST `state=success` or `state=failure` with the build URL as
|
||
`target_url`. This makes `testme-on-pr.sh` work unmodified, adds a native SCM status indicator.
|
||
2. `testme-on-pr.sh`: scan the recipe PR's comments for the `<!-- cc-ci:testme -->` marker and parse
|
||
the result from the comment body (fragile but avoids bridge changes).
|
||
|
||
**Repro:** `POST=0 MAX_WAIT=60 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh custom-html 2`
|
||
→ always `VERDICT=PENDING` even after a green Drone build.
|
||
|
||
(Only Adversary closes this, after re-testing with a VERDICT=GREEN on a real green build.)
|
||
|
||
### [adversary] A5-1 — custom-html-tiny not in bridge poll list
|
||
**Status:** CLOSED — re-tested 2026-05-31T19:41:12Z; see `REVIEW-5.md` follow-up entry.
|
||
|
||
The Phase 5 plan specifies using `custom-html-tiny` as the sandbox recipe for V3–V8 tests.
|
||
However the bridge's poll list (from live container logs) does NOT include `recipe-maintainers/custom-html-tiny`:
|
||
```
|
||
poller (primary) watching ['recipe-maintainers/cc-ci', 'recipe-maintainers/custom-html',
|
||
'recipe-maintainers/keycloak', 'recipe-maintainers/cryptpad', 'recipe-maintainers/matrix-synapse',
|
||
'recipe-maintainers/lasuite-docs', 'recipe-maintainers/n8n', 'recipe-maintainers/hedgedoc'] every 30s
|
||
```
|
||
|
||
This means `!testme` on a `custom-html-tiny` PR will NOT trigger a Drone build. Either:
|
||
1. The builder must add `custom-html-tiny` to the bridge's enrolled repos list (and enroll its tests), OR
|
||
2. Use `custom-html` (which IS enrolled) as the sandbox recipe instead, OR
|
||
3. The plan's V3–V8 tests must first enroll the sandbox recipe as part of Phase 5 setup
|
||
|
||
**Repro:** `docker logs ccci-bridge_app.1.<id> 2>&1 | head -3` on cc-ci shows the poll list.
|
||
|
||
**Impact:** V3, V4, V5, V8 tests using `custom-html-tiny` as sandbox will fail silently (the `!testme`
|
||
comment is posted but the bridge never sees it → VERDICT stays PENDING forever).
|
||
|
||
(Only Adversary closes this after re-test.)
|