Compare commits

...
3 changed files with 57 additions and 19 deletions
+15 -3
View File
@@ -101,7 +101,12 @@ the real current recipe, not a stale mirror.
3. If any has a newer supported upstream tag than what's pinned → there **is** an upgrade: proceed to
plan/implement it (per §2, preserving any digest pin).
- **Conclude `SKIPPED — up-to-date` only when** abra found no bump for the images it could read **and**
the direct check finds nothing newer for the abra-unparseable images. If every image had a normal tag
the direct check finds nothing newer for the abra-unparseable images **and no open PR still needs
review**. Inspect open recipe PRs even when image tags are current. If a PR's useful change is
already in current upstream, obsolete, or superseded, close it with a specific explanatory comment.
If it is still useful, replay that change on current upstream, preserving the latest image tags and
required configuration, then update and retest the same PR. A former GREEN result on an old head
does not validate the rebased recipe. If every image had a normal tag
abra could read, abra's verdict stands alone — no upstream check needed. A `tag and digest … not
supported` FATA is a **trigger for the upstream check on that image, never a skip on its own.**
- Check `git log HEAD..origin/main` and upstream PRs (`git.coopcloud.tech/coop-cloud/<recipe>/pulls`)
@@ -276,6 +281,13 @@ only. The body must **also** carry the recommended release command — `abra rec
final publish step (bumps the label + tags + pushes) after the upstream PR merges. Re-running with the
same target version updates the existing same-branch PR rather than duplicating it.
For a useful open PR whose branch is not `upgrade-*`, set `RECIPE_PR_NUMBER=<number>` for
`open-recipe-pr.sh` after replaying its change onto current upstream. The helper then rewrites that
specific PR branch using `--force-with-lease`; never select an unrelated PR by inference. If several
open PRs have useful compatible changes for the same recipe, combine them only after reviewing each
diff and close the superseded PRs with links to the surviving PR. If a change cannot be made
mergeable and verified, leave the PR clearly marked unresolved rather than reporting it as ready.
### 4. VERIFY by running `!testme` ON THE PR (results visible in the PR; iterate ≤3×)
Trigger the **real** CI on the recipe PR by posting `!testme` — the bridge runs the harness on cc-ci
and posts the run + its results back to the PR, so the operator sees them in the PR. Use the PR index
@@ -357,8 +369,8 @@ Always state explicitly that **nothing was merged** — the PR(s) await operator
them to turn a `dev-` deploy or `!testme` green. In DEFAULT mode, fix the recipe or leave a PR comment
— never touch the gate. (Interim safeguard while the upgrader runs on the cc-ci host; see `IDEAS.md`.)
- **Mirror reflects reality.** Each run force-syncs the `recipe-maintainers/<recipe>` mirror `main` to
true upstream main and closes open PRs already merged upstream. It does NOT close superseded/unrelated
open PRs. **An existing open upgrade PR is rewritten with one commit on the current upstream tip
true upstream main and closes open PRs already merged upstream. The weekly review also closes stale
PRs with no remaining value after checking their diff. **An existing useful PR is rewritten with one commit on the current upstream tip
using `--force-with-lease`, then re-tested** — one evolving upgrade PR per recipe.
- **Prefer a recipe-only PR.** Only open a cc-ci test PR (under `--with-tests`) when the upgrade is
correct but a test is genuinely stale — never to paper over a real upgrade regression.
@@ -145,6 +145,8 @@ echo "→ Reconciling open PRs on ${NAMESPACE}/${RECIPE}..."
OPEN_PRS=$(curl -s "${AUTH[@]}" "${API}/repos/${NAMESPACE}/${RECIPE}/pulls?state=open&base=main&limit=50")
# Close merged-upstream PRs (no-op merges); leave everything else open. Also pick an existing open
# UPGRADE PR (branch 'upgrade-*' — this tool's convention) to EXTEND, if one exists.
# RECIPE_PR_NUMBER explicitly selects another existing PR (for example a security fix) after the
# caller has reviewed and carried its useful diff onto current main.
EXTEND_PR=""; EXTEND_BRANCH=""
while IFS=$'\t' read -r IDX HEAD_REF; do
[ -n "${IDX:-}" ] || continue
@@ -153,11 +155,18 @@ while IFS=$'\t' read -r IDX HEAD_REF; do
continue
fi
echo " • PR #${IDX} (${HEAD_REF}) still open vs upstream main — leaving open"
if [ "${MODE}" != "--reconcile-only" ] && [ -z "${EXTEND_PR}" ] && printf '%s' "${HEAD_REF}" | grep -q '^upgrade-'; then
if [ "${MODE}" != "--reconcile-only" ] && [ -n "${RECIPE_PR_NUMBER:-}" ] && [ "${IDX}" = "${RECIPE_PR_NUMBER}" ]; then
EXTEND_PR="${IDX}"; EXTEND_BRANCH="${HEAD_REF}"
elif [ "${MODE}" != "--reconcile-only" ] && [ -z "${RECIPE_PR_NUMBER:-}" ] && [ -z "${EXTEND_PR}" ] && printf '%s' "${HEAD_REF}" | grep -q '^upgrade-'; then
EXTEND_PR="${IDX}"; EXTEND_BRANCH="${HEAD_REF}"
fi
done < <(echo "${OPEN_PRS}" | jq -r '.[] | [(.number | tostring), .head.ref] | @tsv' 2>/dev/null || true)
if [ "${MODE}" != "--reconcile-only" ] && [ -n "${RECIPE_PR_NUMBER:-}" ] && [ "${EXTEND_PR}" != "${RECIPE_PR_NUMBER}" ]; then
echo "ERROR: selected PR #${RECIPE_PR_NUMBER} is not an open PR against current main" >&2
exit 1
fi
if [ "${MODE}" = "--reconcile-only" ]; then
echo "✓ reconcile-only done for ${NAMESPACE}/${RECIPE} (main synced; merged-upstream PRs closed)."
exit 0
+32 -15
View File
@@ -103,9 +103,21 @@ ssh cc-ci 'cd /root/cc-ci/tests && ls -d */' | sed 's#/##' \
| grep -vxF -f <(printf '%s\n' "$EXTERNAL") # drop externally-maintained recipes
```
(or the names passed in `$ARGUMENTS` — an explicit recipe name overrides the `external` skip, so you
can still upgrade one on request.) For each candidate, on cc-ci, check availability — skip
dirty/up-to-date. (If `/root/cc-ci` isn't present on the host, stage it first — see
`cc-ci-plan/plan-proxy-vip-exhaustion-fix.md` / the host-rebuild memory for the staging step.)
can still upgrade one on request.)
**First, reconcile EVERY inventoried recipe's mirror with true upstream, before fetching its checkout,
checking versions, reading open PRs, or deciding it is up-to-date.** This is the first per-recipe
survey action, including for recipes that end up skipped. The helper syncs mirror `main` and closes
PRs whose changes are already upstream. It requires a local recipe checkout; if absent, stage that
checkout first (see `cc-ci-plan/plan-proxy-vip-exhaustion-fix.md` / the host-rebuild memory):
```
set -a; . /srv/cc-ci/.testenv; set +a
ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe> --reconcile-only" \
< /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh
```
Record a reconciliation failure as `unverified` and investigate it; never label the recipe
up-to-date from an unsynced mirror. After successful reconciliation, refresh the checkout from that
mirror and check availability. If `/root/cc-ci` is absent, stage the cc-ci checkout first too.
> ⚠️ **Four things that silently skip recipes — handle ALL FOUR per recipe before the version check:**
> 1. **pseudo-TTY:** abra FATAs `inappropriate ioctl for device` under plain ssh — wrap every abra call
@@ -153,9 +165,18 @@ resolver finds the newer image. A warning is never evidence that an image is cur
`newest_same_shape` and `newest_within_major`; a newer major is still a candidate for planning,
though it must not be applied without a compatibility check.
Build `RECIPES_TO_UPGRADE` from every recipe with at least one candidate. Only label a recipe
Build `RECIPES_TO_UPGRADE` from every recipe with at least one candidate **or an open recipe PR
that still differs from current upstream**. The second case is a PR reconciliation candidate even
when all image tags are current: review whether its change still adds value. If the change is
already present, obsolete, or superseded, close the stale PR with a concrete reason. If it remains
useful, carry its change onto current upstream, update the **same PR**, and run fresh `!testme`.
Do not call it GREEN based on CI for an old head, or leave an unmergeable PR as the week's proposed
fix. Count its CVEs only from the useful change still absent from current upstream.
Only label a recipe
`up-to-date` if abra completed successfully **and** the resolver checked every relevant image
(`all_resolved: true`) **and** neither found a candidate. If an image is unresolved, templated,
(`all_resolved: true`) **and** neither found a candidate **and no open PR needs reconciliation**.
If an image is unresolved, templated,
floating, or lacks comparable tags, investigate it directly and record the evidence. Otherwise
record `unverified` with the image and reason; never fold it into `up-to-date`. `dirty-worktree`
applies only to real tracked edits. A tag+digest parse failure is not `not-fetchable`.
@@ -169,16 +190,8 @@ advisory checks are subject to the anonymous 60-request/hour limit; any resultin
failed source and must become `?` in the report. Never replace it with `0` or silently retry from
model memory. The file is a read-only GitHub token location; never print its contents.
**Reconcile every candidate's mirror during the survey** (even the up-to-date ones), so merged-upstream
PRs are closed and every mirror `main` tracks true upstream — fleet-wide, not just where there's a new
upgrade:
```
set -a; . /srv/cc-ci/.testenv; set +a
ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe> --reconcile-only" \
< /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh
```
(The per-recipe `/recipe-upgrade` also reconciles, so this is belt-and-suspenders for skipped recipes —
count closed-merged PRs in the summary.)
Count PRs closed as already merged upstream in the weekly summary. `/recipe-upgrade` reconciles
again before writing a PR, because upstream may advance while the survey and testing run.
## 2. Determine concurrency, then print the plan
Read the live capacity (unless `--capacity N` / `--sequential` / `--parallel` was passed):
@@ -205,6 +218,10 @@ description `"Upgrade <recipe> on cc-ci"`) with a prompt like:
> upstream. Do NOT merge. Print exactly one `RESULT:` line as your final line (SUCCESS /
> SUCCESS-PENDING-TESTS / FAILED / SKIPPED — `SUCCESS+TESTPR` will not occur in default mode).
For a recipe selected solely because it has an open PR, tell the worker which PR number to review.
The worker may close it if no useful change remains; otherwise it must update that PR rather than
opening another. A closure is a completed reconciliation, not an upgrade success.
**Why default (no `--with-tests`):** the weekly cron must never auto-edit cc-ci tests unattended — a
test change deserves a human decision. So the cron opens recipe PRs only; where a recipe's existing
test looks stale, the operator sees the explanation in the PR comment and can re-run that one recipe