Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
10840263aa | ||
|
|
8df32edfcf | ||
|
|
18caf047bf | ||
|
|
b0bdce2c15 |
@@ -0,0 +1,91 @@
|
|||||||
|
---
|
||||||
|
name: cve-check-and-upgrade
|
||||||
|
description: Security-driven upgrade run. Does a full /cve-check sweep first (per-image advisory scan of every recipe's available upgrade, with adjudication), then runs /recipe-upgrade ONLY on the recipes whose upgrade fixes at least one CVE — worst severity first — opening a verified recipe PR for each, and finally publishes one report covering both the sweep and the PRs. Recipes with no CVEs are left alone; that is the point. NEVER merges. Invoke as /cve-check-and-upgrade [recipe ...] [--min-severity high] [--capacity N] [--dry-run].
|
||||||
|
---
|
||||||
|
|
||||||
|
# cve-check-and-upgrade
|
||||||
|
|
||||||
|
`/upgrade-all` upgrades everything that *has* an upgrade. **This upgrades what has a reason.** It runs
|
||||||
|
the `/cve-check` sweep, then spends CI time only on the recipes where an upgrade actually closes a
|
||||||
|
vulnerability, handling the worst first.
|
||||||
|
|
||||||
|
Use it when you want to act on security rather than churn the whole fleet: after a vendor announcement,
|
||||||
|
when CI capacity is short, or between weekly runs. When you only want to *know*, use `/cve-check`. When
|
||||||
|
you want everything current regardless of CVEs, use `/upgrade-all`.
|
||||||
|
|
||||||
|
**Creates PRs. Never merges.** Every PR is verified green on cc-ci and left for a human.
|
||||||
|
|
||||||
|
## Arguments
|
||||||
|
- `<recipe> …` — restrict the whole run to these recipes.
|
||||||
|
- `--min-severity critical|high|medium|low` — only upgrade recipes whose fixed CVEs reach this
|
||||||
|
severity. Default **`low`** (any CVE at all justifies the upgrade). `--min-severity high` is the
|
||||||
|
useful "just the urgent ones" setting.
|
||||||
|
- `--capacity N` — subagent pool size; defaults to the live `DRONE_RUNNER_CAPACITY` (the drone
|
||||||
|
runner's slots), matching `/upgrade-all`'s rolling-pool behaviour.
|
||||||
|
- `--dry-run` — do the whole sweep and print exactly which recipes *would* be upgraded and why, then
|
||||||
|
stop without spawning a single upgrade. **Publishes no report and opens no PR.**
|
||||||
|
|
||||||
|
## Procedure
|
||||||
|
|
||||||
|
### 1. Sweep — run `/cve-check` in full
|
||||||
|
Follow `.claude/skills/cve-check/SKILL.md` steps 1–5 exactly: candidate list, per-image upgrade windows,
|
||||||
|
the advisory scan per recipe, pass-2 adjudication of anything undecided, and the severity classification.
|
||||||
|
**Do not publish its report** — this run produces one combined report at the end instead.
|
||||||
|
|
||||||
|
Keep, per recipe: the windows scanned, the CVE count, the CVE ids with severities, and whether the count
|
||||||
|
is a floor (undetermined advisories remain) or unknown.
|
||||||
|
|
||||||
|
### 2. Decide what to upgrade
|
||||||
|
`RECIPES_TO_UPGRADE` = recipes where **the scan found ≥1 CVE** at or above `--min-severity`.
|
||||||
|
|
||||||
|
Deliberate exclusions, each recorded in the report with its reason:
|
||||||
|
- **0 CVEs** — an upgrade may exist, but nothing security-relevant. Left alone; that is the point of
|
||||||
|
this skill. `/upgrade-all` is what sweeps those up.
|
||||||
|
- **`external` tier** — swept for visibility, **never upgraded here**; someone else maintains it. Flag
|
||||||
|
it loudly in the report if it has a critical, since the action is to tell them, not to open a PR.
|
||||||
|
- **`UPTODATE`** — nothing available.
|
||||||
|
- **count `?` / UNKNOWN** — do **not** upgrade blind, and do **not** treat it as clean. Put it in the
|
||||||
|
Addendum as needing a look. An unknown is a gap in our knowledge, not evidence of safety.
|
||||||
|
|
||||||
|
Order the queue by **worst severity first** (critical → high → …), count breaking ties. If `--dry-run`,
|
||||||
|
print this queue with each recipe's CVE ids and severities, and STOP here.
|
||||||
|
|
||||||
|
### 3. Upgrade each one — via `/recipe-upgrade` subagents
|
||||||
|
Run `/recipe-upgrade <recipe>` per queued recipe as a **subagent**, in the queue order above, as a
|
||||||
|
**rolling pool** keeping `--capacity` (default `DRONE_RUNNER_CAPACITY`) running at once and starting the
|
||||||
|
next as each finishes — the same concurrency discipline as `/upgrade-all` §3, and safe for the same
|
||||||
|
reason (per-run recipe trees + app-domain locks).
|
||||||
|
|
||||||
|
Each subagent does the full job: plan, implement the bump, verify green on cc-ci with `!testme`, and
|
||||||
|
open a recipe PR. **Default mode — no `--with-tests`**: a genuinely stale test gets an explanatory PR
|
||||||
|
comment, not a test edit.
|
||||||
|
|
||||||
|
**Tell each subagent which CVEs justify its upgrade**, with ids and severities, so the PR description
|
||||||
|
says why it exists. That is most of this skill's value to a reviewer: a PR that names the CVSS-9.8 RCE
|
||||||
|
it closes gets merged today, an unexplained version bump waits a fortnight.
|
||||||
|
|
||||||
|
Collect per recipe: PR url + number, the `!testme` verdict and build number, and any failure.
|
||||||
|
|
||||||
|
### 4. Report — one page covering sweep AND PRs
|
||||||
|
Write `/tmp/cve-spec-<DATE>.json` per `/cve-check` step 6, with these differences:
|
||||||
|
- Rows for upgraded recipes carry the real `ci` (`build N ✓` / `RED N · <stage>`) + `ci_url`, and
|
||||||
|
`pr`/`pr_url`. `status` is the CI verdict (`GREEN`/`FAILED`/`STALE`); the live PR-status column
|
||||||
|
derives itself from `recipe` + `pr`.
|
||||||
|
- Rows for swept-but-not-upgraded recipes keep `PENDING`/`UPTODATE` with empty `ci`/`pr`, and a
|
||||||
|
`notes` reason (`0 CVEs — not upgraded`, `external — maintained elsewhere`, `below --min-severity`).
|
||||||
|
- Include `changes[]` — one entry per recipe that got a PR, describing what the upgrade changes **and
|
||||||
|
the CVEs it closes**.
|
||||||
|
- Keep `"kind": "cve"`: it titles the page "The Recipe Report — CVE check" and files it as
|
||||||
|
`cve-<DATE>.html`, alongside the weekly editions in the same archive index.
|
||||||
|
|
||||||
|
Then render + publish exactly as `/cve-check` step 7, and verify as its step 8. Print the report URL,
|
||||||
|
`N swept · M upgraded · K PRs green · J failed`, and `CVE CHECK AND UPGRADE COMPLETE`.
|
||||||
|
|
||||||
|
## Guardrails
|
||||||
|
- **NEVER merge.** Create and verify; a human merges. Never push to true upstream.
|
||||||
|
- **Never weaken a test** to make a PR green, and never edit a test without `--with-tests`.
|
||||||
|
- **Never upgrade a recipe whose CVE count is unknown** on the assumption it is fine — surface it.
|
||||||
|
- **Never upgrade an `external` recipe** here, even with a critical; report it instead.
|
||||||
|
- **Public-safe report only** — no secrets, tokens, internal hostnames, raw logs, or spend figures.
|
||||||
|
- If the sweep finds **nothing** at or above `--min-severity`, that is a good outcome: publish the
|
||||||
|
report saying so and open no PRs. Do not manufacture work.
|
||||||
@@ -0,0 +1,170 @@
|
|||||||
|
---
|
||||||
|
name: cve-check
|
||||||
|
description: Fleet-wide CVE sweep WITHOUT upgrading anything. For every recipe cc-ci deploys, works out what upgrade is available (current pinned tag → newest supported tag, per image including sidecars), runs the deterministic advisory scan over that window, adjudicates whatever the scan could not decide, and publishes a CVE report to report.ci.commoninternet.net as cve-<DATE>.html. READ-ONLY — opens no PRs, edits no recipes, runs no CI, merges nothing. Answers "what are we exposed to that an upgrade would fix?" in minutes rather than the hours a full upgrade run takes. Invoke as /cve-check [recipe ...] [--weekly-only].
|
||||||
|
---
|
||||||
|
|
||||||
|
# cve-check
|
||||||
|
|
||||||
|
A **security sweep, not an upgrade run.** It answers one question for every recipe cc-ci deploys:
|
||||||
|
|
||||||
|
> If we upgraded this recipe today, how many CVEs would that fix, and how bad are they?
|
||||||
|
|
||||||
|
It is the cheap, safe half of `/upgrade-all`: the same version research and the same advisory scan,
|
||||||
|
with **no implementation, no CI, and no PRs**. Use it when you want the security picture now — after a
|
||||||
|
vendor announcement, before deciding what to prioritise, or between weekly runs. When you want the PRs
|
||||||
|
too, use **`/cve-check-and-upgrade`**.
|
||||||
|
|
||||||
|
**Read-only, absolutely.** Never edit a recipe, never open or comment on a PR, never merge, never
|
||||||
|
deploy. The only thing it writes is its own log and the published report page.
|
||||||
|
|
||||||
|
## Arguments
|
||||||
|
- `<recipe> …` — sweep only these recipes (else every recipe in `cc-ci-plan/used-recipes.md`).
|
||||||
|
- `--weekly-only` — skip rows tagged `external`. **Off by default on purpose**: an `external` recipe is
|
||||||
|
still deployed and still exposes us, so a security sweep that silently skipped it would misreport the
|
||||||
|
fleet's exposure. Externals are swept and clearly marked "maintained elsewhere" in the report.
|
||||||
|
|
||||||
|
## Procedure
|
||||||
|
|
||||||
|
> ### ⚠️ Run abra over a pseudo-TTY (or it FATAs `inappropriate ioctl for device`)
|
||||||
|
> `abra` needs a TTY. Wrap every abra call: `ssh cc-ci 'script -qec "abra <args> -n" /dev/null'`.
|
||||||
|
> (`git` and other commands do NOT need the wrapper.)
|
||||||
|
|
||||||
|
### 1. Build the candidate list
|
||||||
|
Read `cc-ci-plan/used-recipes.md` — the canonical inventory. Take every row (both tiers), recording the
|
||||||
|
tier per recipe; with `--weekly-only`, drop the `external` rows. An explicit recipe argument overrides
|
||||||
|
any skip.
|
||||||
|
|
||||||
|
### 2. Per recipe — establish the upgrade window WITHOUT upgrading
|
||||||
|
|
||||||
|
This is `/recipe-upgrade` step 1's research, stopping before it implements anything.
|
||||||
|
|
||||||
|
> ⚠️ **The same four things that silently skip recipes apply here — handle ALL FOUR:**
|
||||||
|
> 1. **pseudo-TTY** — per the box above.
|
||||||
|
> 2. **go-git auth to git.autonomic.zone** — recipes on the private mirror FATA
|
||||||
|
> `authentication required: Unauthorized`. Bake creds into origin first (idempotent, only when
|
||||||
|
> origin is on git.autonomic.zone):
|
||||||
|
> `git -C ~/.abra/recipes/<r> remote set-url origin "https://$GITEA_USERNAME:$GITEA_PASSWORD@git.autonomic.zone/recipe-maintainers/<r>.git"`
|
||||||
|
> 3. **dirty worktree** — usually just the untracked cc-ci overlay; `git stash -u` before, `stash pop`
|
||||||
|
> after. Only a genuinely dirty TRACKED tree is a skip.
|
||||||
|
> 4. **tag+digest pins abra cannot parse** — abra FATAs and aborts the WHOLE recipe (immich). This is
|
||||||
|
> **not** "not fetchable": enumerate the compose's `image:` refs yourself and check the upstream
|
||||||
|
> registry directly for the ones abra could not read, picking the newest tag the app version
|
||||||
|
> supports rather than the numerically highest.
|
||||||
|
|
||||||
|
**Reconcile the mirror from true upstream FIRST — ALWAYS, no exceptions.** This is the same reconcile
|
||||||
|
`/upgrade-all` does. Do not skip it in the name of keeping the sweep read-only: skipping it makes you
|
||||||
|
research a stale checkout, and on the first real run that produced **two recipes with no survey output
|
||||||
|
at all**, which is indistinguishable from "no upgrades" unless you check. It is safe — recipe work
|
||||||
|
lives in **branches**, never directly on `main`, so a force-sync of `main` to upstream discards
|
||||||
|
nothing; it also auto-closes mirror PRs whose changes upstream has already merged.
|
||||||
|
|
||||||
|
> ### ⚠️ The default branch may be `master`, not `main` — check, do not assume
|
||||||
|
> Several coopcloud recipes keep a **stale `main` alongside the real default `master`**. gitea is one:
|
||||||
|
> `main` sits at 1.24.2-rootless while `master` has 1.27.1-rootless plus the merged PRs and the 3.6.3
|
||||||
|
> release. Reading `main` there tells you the recipe is three releases behind and missing two CVSS-9.8
|
||||||
|
> RCE fixes — a false alarm that reads exactly like a real one. Resolve the default branch from the
|
||||||
|
> API (`/api/v1/repos/coop-cloud/<recipe>` → `default_branch`) before reading any file, and never
|
||||||
|
> `git reset --hard origin/main` on a checkout that tracks `master`.
|
||||||
|
|
||||||
|
Then read versions:
|
||||||
|
```
|
||||||
|
set -a; . /srv/cc-ci/.testenv; set +a
|
||||||
|
ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe> --reconcile-only" \
|
||||||
|
< /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh
|
||||||
|
ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; R=<recipe>; \
|
||||||
|
git -C ~/.abra/recipes/$R stash -u >/dev/null 2>&1 || true; \
|
||||||
|
script -qec "abra recipe fetch $R --force -n" /dev/null; \
|
||||||
|
script -qec "abra recipe upgrade $R -m -n" /dev/null; \
|
||||||
|
git -C ~/.abra/recipes/$R stash pop >/dev/null 2>&1 || true'
|
||||||
|
```
|
||||||
|
|
||||||
|
For each recipe produce **one window per image**: `current pinned tag → newest supported tag`. You need
|
||||||
|
the sidecars (redis, postgres, nginx …), not just the app — a sidecar bump is where discourse's only
|
||||||
|
CRITICAL came from, and an image with no window is not counted at all.
|
||||||
|
|
||||||
|
- **No upgrade available** → the recipe is `UPTODATE`; its CVE count is **`0`**, not `?`. There is
|
||||||
|
nothing an upgrade could fix. Record it and move on.
|
||||||
|
- **No output at all is NOT "no upgrade".** An abra call that times out, FATAs, or prints nothing
|
||||||
|
leaves the recipe **unverified** — treat it as a distinct outcome, never fold it into up-to-date.
|
||||||
|
Re-run it, and if it still yields nothing, resolve the versions by direct registry check (box item 4).
|
||||||
|
Only report `?` once BOTH the abra check and the direct check have failed. On the first real run this
|
||||||
|
distinction was the difference between two false zeros and the truth (both recipes turned out fine,
|
||||||
|
but nothing in the survey said so).
|
||||||
|
|
||||||
|
### 3. Run the advisory scan over that window
|
||||||
|
```
|
||||||
|
python3 /srv/cc-ci/cc-ci-plan/advisory-scan.py <recipe> --from <old-app> --to <new-app> \
|
||||||
|
[--image <name>=<old>:<new>]...
|
||||||
|
```
|
||||||
|
**One call per recipe with every image in it** — the count is a union across images, and the
|
||||||
|
UNKNOWN guarantee only holds when a single run sees them all. Paste the markdown block verbatim into
|
||||||
|
the per-recipe log at `/srv/cc-ci/.cc-ci-logs/cve-check/<DATE>/<recipe>.md`.
|
||||||
|
|
||||||
|
### 4. Adjudicate what the scan could not decide (pass 2)
|
||||||
|
If the block reports advisories it **could NOT judge**, or the count is **UNKNOWN**, re-run with
|
||||||
|
`--adjudicate` and decide each open case yourself:
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 /srv/cc-ci/cc-ci-plan/advisory-scan.py <recipe> … --adjudicate
|
||||||
|
```
|
||||||
|
|
||||||
|
Answer **FIXED / NOT-FIXED / STILL-UNKNOWN** per case, each with a one-line reason **citing the
|
||||||
|
evidence shown** — never from memory of the project, which is the exact failure that let two CVSS-9.8
|
||||||
|
gitea RCEs be published as "none". Every FIXED is added to the count; pass 1's number is a floor. The
|
||||||
|
block also lists what pass 1 already decided — if a verdict looks wrong given its evidence, say so.
|
||||||
|
Record your verdicts in the per-recipe log so the number is auditable.
|
||||||
|
|
||||||
|
### 5. Classify severity and priority
|
||||||
|
For each recipe collect the CVE ids with **severities** (the scan gives them, with GHSA ids). Sort the
|
||||||
|
report rows by what an operator should deal with first:
|
||||||
|
1. recipes with a **critical**, then **high**, then anything else with CVEs (more CVEs higher within a band);
|
||||||
|
2. then `?` (a count that could not be established — investigate, do not ignore);
|
||||||
|
3. then recipes with an upgrade available but **0** CVEs;
|
||||||
|
4. then `UPTODATE`.
|
||||||
|
|
||||||
|
Severity outranks raw count: 2 CVSS-9.8 RCEs matter more than 120 medium plugin advisories.
|
||||||
|
|
||||||
|
### 6. Write the report spec
|
||||||
|
`/tmp/cve-spec-<DATE>.json`, same shape as `/recipe-report` (see `recipe-report.py`'s header), with:
|
||||||
|
- **`"kind": "cve"`** — titles the page "The Recipe Report — CVE check" and files it as
|
||||||
|
`cve-<DATE>.html`. It appears in the SAME archive index as the weekly editions, suffixed
|
||||||
|
"— CVE check" so the two are told apart at a glance. Without this field you would overwrite that
|
||||||
|
date's weekly edition.
|
||||||
|
- `date`, `subtitle` "CVE check <human date>",
|
||||||
|
- `lead` — **one short paragraph**: fleet exposure in a sentence and what to do first.
|
||||||
|
- `table[]` — every recipe swept. `recipe`; `change` = the window you scanned, e.g.
|
||||||
|
`1.27.0 → 1.27.1 · redis 7.4 → 8.10`; `status` = `UPTODATE` when nothing is available, else
|
||||||
|
`PENDING` (an upgrade exists and is not yet taken); **`cve`** = the count (integer, `?` only per the
|
||||||
|
rules below); `notes` = severity mix, whether the number is a floor, and `maintained elsewhere` for
|
||||||
|
`external` rows. **Leave `ci`/`pr` empty — nothing was built and no PR exists.**
|
||||||
|
- `addendum[]` — real anomalies only: registry URLs that failed, recipes whose window could not be
|
||||||
|
established, a scan whose count is a floor with many undetermined advisories.
|
||||||
|
- `security[]` — one entry per **critical/high** finding: recipe · CVE id(s) + severity · what it fixes
|
||||||
|
· **which image** it is in. Name the image: `CVE-2025-49844` is a redis flaw, and an operator reading
|
||||||
|
"discourse" needs to know that.
|
||||||
|
- `changes[]` — **omit** (nothing changed; there are no PRs).
|
||||||
|
|
||||||
|
**`?` must stay RARE.** Use it only when a scan ran and reported genuinely failed sources, or the count
|
||||||
|
came back UNKNOWN and adjudication could not settle it. Never `none` for an unknown — a blank reads as
|
||||||
|
clean. A recipe with no upgrade available is `0`, not `?`. Many `?` is a bug for the Addendum.
|
||||||
|
|
||||||
|
### 7. Render and publish — via the script only
|
||||||
|
```
|
||||||
|
python3 /srv/cc-ci/cc-ci-plan/recipe-report.py render /tmp/cve-spec-<DATE>.json /tmp/cve-<DATE>.html
|
||||||
|
python3 /srv/cc-ci/cc-ci-plan/recipe-report.py publish /tmp/cve-<DATE>.html <DATE> cve
|
||||||
|
```
|
||||||
|
All layout is owned by `recipe-report.py`. Never hand-write or post-process HTML; if `render` errors,
|
||||||
|
fix the spec JSON and re-render. **Public page — no secrets, tokens, internal hostnames, raw logs, or
|
||||||
|
any billing/spend figures.**
|
||||||
|
|
||||||
|
### 8. Verify and stop
|
||||||
|
`curl -fsS https://report.ci.commoninternet.net/cve-<DATE>.html` renders and the index lists it. Print
|
||||||
|
the URL, a one-line summary (`N recipes swept · M with CVEs · K critical`), and `CVE CHECK COMPLETE`,
|
||||||
|
then go idle. One-shot — do not loop, and do not start upgrading anything.
|
||||||
|
|
||||||
|
## Guardrails
|
||||||
|
- **Read-only.** No PRs, no edits, no merges, no deploys, no CI runs. If a recipe looks urgent, say so
|
||||||
|
in the report — do not act on it. `/cve-check-and-upgrade` is the skill that acts.
|
||||||
|
- **Never report `0` for something you could not scan.** `0` means checked-and-clean; unknown is `?`.
|
||||||
|
- A count with undetermined advisories is a **floor** — say so in the notes rather than rounding away.
|
||||||
|
- **Public-safe output only.**
|
||||||
@@ -106,6 +106,12 @@ systemctl --failed --no-legend; df -h / | tail -1; tmux ls
|
|||||||
1. <finding> → /<skill> (or operator action)
|
1. <finding> → /<skill> (or operator action)
|
||||||
…
|
…
|
||||||
```
|
```
|
||||||
|
When a finding is that the fleet's **security exposure is unknown** — the last weekly run failed or
|
||||||
|
is stale, so nobody has scanned for CVEs recently — the recommended step is **`/cve-check`** (read-only,
|
||||||
|
minutes, no PRs). If it is instead that a known CVE is sitting unpatched, recommend
|
||||||
|
**`/cve-check-and-upgrade`** (add `--min-severity high` when only the urgent ones matter). Prefer
|
||||||
|
`/cve-check` over waiting for the next weekly run whenever the question is "are we exposed?".
|
||||||
|
|
||||||
`ALL HEALTHY` requires: recent successful weekly run + published report, no stale tests, no
|
`ALL HEALTHY` requires: recent successful weekly run + published report, no stale tests, no
|
||||||
CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, disk under
|
CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, disk under
|
||||||
thresholds, bridge clean, maintained-set consistent. Anything else is a finding — even minor
|
thresholds, bridge clean, maintained-set consistent. Anything else is a finding — even minor
|
||||||
|
|||||||
@@ -31,6 +31,14 @@ Then present the roster grouped as follows, and close with the situation guide.
|
|||||||
PR). `--with-tests` also fixes that recipe's stale test.
|
PR). `--with-tests` also fixes that recipe's stale test.
|
||||||
- **/recipe-report** — (re)generate the weekly report page for report.ci.commoninternet.net.
|
- **/recipe-report** — (re)generate the weekly report page for report.ci.commoninternet.net.
|
||||||
|
|
||||||
|
**Security (CVEs)**
|
||||||
|
- **/cve-check** — fleet-wide CVE sweep with **no upgrading**: for every recipe, work out what
|
||||||
|
upgrade is available (per image, sidecars included), scan it for CVEs, and publish a CVE report.
|
||||||
|
Read-only and quick — the "what are we exposed to?" answer without an upgrade run.
|
||||||
|
- **/cve-check-and-upgrade** — the same sweep, then open verified PRs **only** for the recipes whose
|
||||||
|
upgrade actually fixes a CVE, worst severity first. `--min-severity high` for just the urgent ones.
|
||||||
|
Never merges.
|
||||||
|
|
||||||
**Tests**
|
**Tests**
|
||||||
- **/cc-ci-tests-update** — fleet-wide stale-test cleanup: find tests broken by legitimate
|
- **/cc-ci-tests-update** — fleet-wide stale-test cleanup: find tests broken by legitimate
|
||||||
upstream changes, fix without weakening, verify, merge the test PRs.
|
upstream changes, fix without weakening, verify, merge the test PRs.
|
||||||
@@ -73,6 +81,9 @@ ARM skills never touch cc-ci infra. After a submodule bump run `scripts/gen-ccte
|
|||||||
| "Run the weekly upgrades now" | `/upgrade-all` (or `systemctl start cc-ci-upgrade-all.service`) |
|
| "Run the weekly upgrades now" | `/upgrade-all` (or `systemctl start cc-ci-upgrade-all.service`) |
|
||||||
| "Upgrade just <recipe>" | `/recipe-upgrade <recipe>` |
|
| "Upgrade just <recipe>" | `/recipe-upgrade <recipe>` |
|
||||||
| "The report site is stale/missing a week" | `/recipe-report` |
|
| "The report site is stale/missing a week" | `/recipe-report` |
|
||||||
|
| "What CVEs are we exposed to right now?" | `/cve-check` (read-only, no PRs) |
|
||||||
|
| "A CVE just dropped — check and patch it" | `/cve-check-and-upgrade` (add `--min-severity high` to skip the noise) |
|
||||||
|
| "Is <recipe> vulnerable?" | `/cve-check <recipe>` |
|
||||||
| "Tests are red because upstream changed" | `/cc-ci-tests-update` (fleet) or `/recipe-upgrade <r> --with-tests` |
|
| "Tests are red because upstream changed" | `/cc-ci-tests-update` (fleet) or `/recipe-upgrade <r> --with-tests` |
|
||||||
| "A CI run failed and I don't know why" | `/ci-test-review` |
|
| "A CI run failed and I don't know why" | `/ci-test-review` |
|
||||||
| "Update the CI server OS/deps" | `/cc-ci-server-update` |
|
| "Update the CI server OS/deps" | `/cc-ci-server-update` |
|
||||||
|
|||||||
@@ -333,6 +333,19 @@ def _releases(owner: str, repo: str, max_pages: int = 4) -> list[tuple[str, str]
|
|||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _source_repo(source: str) -> tuple[str, str] | None:
|
||||||
|
"""(owner, repo) for a source, whether it is an advisory feed or a vendor page on GitHub.
|
||||||
|
|
||||||
|
A CVE that appears ONLY on a vendor page still deserves the release-note method when that page
|
||||||
|
lives on GitHub — mailu announces its Roundcube CVEs on github.com/Mailu/Mailu/releases and
|
||||||
|
nowhere structured, so requiring an advisory feed sent a deterministic case to pass 2."""
|
||||||
|
if source.startswith("github-advisories:"):
|
||||||
|
owner, _, repo = source.split(":", 1)[1].partition("/")
|
||||||
|
return (owner, repo) if owner and repo else None
|
||||||
|
m = re.match(r"https?://github\.com/([^/]+)/([^/#?]+)", source)
|
||||||
|
return (m.group(1), m.group(2).removesuffix(".git")) if m else None
|
||||||
|
|
||||||
|
|
||||||
def release_fix_versions(source: str, cve: str) -> list[str]:
|
def release_fix_versions(source: str, cve: str) -> list[str]:
|
||||||
"""Release tags whose notes NAME this CVE — a deterministic fix version when the advisory has none.
|
"""Release tags whose notes NAME this CVE — a deterministic fix version when the advisory has none.
|
||||||
|
|
||||||
@@ -342,10 +355,10 @@ def release_fix_versions(source: str, cve: str) -> list[str]:
|
|||||||
(CVE-2025-32023 → 6.2.19, 7.2.10, 7.4.5, 8.0.3, 8.2.0). Ignoring that evidence undercounted
|
(CVE-2025-32023 → 6.2.19, 7.2.10, 7.4.5, 8.0.3, 8.2.0). Ignoring that evidence undercounted
|
||||||
discourse by 12 CVEs, so this is checked BEFORE giving up on an advisory.
|
discourse by 12 CVEs, so this is checked BEFORE giving up on an advisory.
|
||||||
"""
|
"""
|
||||||
if not source.startswith("github-advisories:"):
|
ref = _source_repo(source)
|
||||||
|
if not ref:
|
||||||
return []
|
return []
|
||||||
owner, _, repo = source.split(":", 1)[1].partition("/")
|
return [tag for tag, body in _releases(*ref) if cve in body]
|
||||||
return [tag for tag, body in _releases(owner, repo) if cve in body]
|
|
||||||
|
|
||||||
|
|
||||||
def advisory_text(ghsa: str, source: str | None = None) -> dict:
|
def advisory_text(ghsa: str, source: str | None = None) -> dict:
|
||||||
@@ -707,17 +720,31 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
|
|||||||
# THIRD METHOD: before declaring an advisory undecidable, look for the CVE id in the project's
|
# THIRD METHOD: before declaring an advisory undecidable, look for the CVE id in the project's
|
||||||
# own release notes. A tag that names it, inside the window, IS the fix version the advisory
|
# own release notes. A tag that names it, inside the window, IS the fix version the advisory
|
||||||
# failed to publish. Deterministic and citable — not a judgement call.
|
# failed to publish. Deterministic and citable — not a judgement call.
|
||||||
resolved_by_release = {}
|
# A window is keyed by advisory-feed source; map it to its repo so a vendor page on the SAME
|
||||||
for cve in sorted(indeterminate - fixed_set):
|
# repo can be judged by the same window.
|
||||||
|
win_by_repo = {}
|
||||||
|
for wsrc, wv in windows.items():
|
||||||
|
ref = _source_repo(wsrc)
|
||||||
|
if ref:
|
||||||
|
win_by_repo[ref] = wv
|
||||||
|
resolved_by_release, already_fixed = {}, set()
|
||||||
|
candidates = set(indeterminate) | {
|
||||||
|
c for c in unknown
|
||||||
|
if not any(s in windows for s in report["cves"][c]["sources"])
|
||||||
|
and any(_source_repo(s) in win_by_repo for s in report["cves"][c]["sources"])
|
||||||
|
}
|
||||||
|
for cve in sorted(candidates - fixed_set):
|
||||||
e = report["cves"][cve]
|
e = report["cves"][cve]
|
||||||
for src in e["sources"]:
|
for src in e["sources"]:
|
||||||
if src not in windows:
|
ref = _source_repo(src)
|
||||||
|
if src not in windows and ref not in win_by_repo:
|
||||||
continue
|
continue
|
||||||
wf, wt = windows[src]
|
wf, wt = windows[src] if src in windows else win_by_repo[ref]
|
||||||
kf, kt = _vkey(wf), _vkey(wt)
|
kf, kt = _vkey(wf), _vkey(wt)
|
||||||
if not (kf and kt):
|
if not (kf and kt):
|
||||||
continue
|
continue
|
||||||
hits = [t for t in release_fix_versions(src, cve) if _within(kf, kt, _vkey(t))]
|
naming = release_fix_versions(src, cve)
|
||||||
|
hits = [t for t in naming if _within(kf, kt, _vkey(t))]
|
||||||
if hits:
|
if hits:
|
||||||
fixed_set.add(cve)
|
fixed_set.add(cve)
|
||||||
resolved_by_release[cve] = sorted(hits)
|
resolved_by_release[cve] = sorted(hits)
|
||||||
@@ -725,10 +752,23 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
|
|||||||
f"{', '.join(sorted(hits))}) via {src}")
|
f"{', '.join(sorted(hits))}) via {src}")
|
||||||
e["fix_versions_from_release_notes"] = sorted(hits)
|
e["fix_versions_from_release_notes"] = sorted(hits)
|
||||||
break
|
break
|
||||||
|
# Naming releases exist but ALL predate the version we were already on: the fix shipped
|
||||||
|
# before this upgrade, so the upgrade did not deliver it. That is a DECISION, not an
|
||||||
|
# unknown — mailu's redis 8.8.0 → 8.10.0 crosses 12 advisories all fixed by 8.6.3 or
|
||||||
|
# earlier, and reporting them as "could not judge" overstates the uncertainty.
|
||||||
|
if naming and all(_vkey(t) and not _within(kf, kt, _vkey(t)) for t in naming) \
|
||||||
|
and max(_vkey(t) for t in naming if _vkey(t)) <= kf:
|
||||||
|
e["classification"] = ("outside-window: fixed in "
|
||||||
|
f"{', '.join(sorted(naming))}, all at or before {wf}")
|
||||||
|
e["fix_versions_from_release_notes"] = sorted(naming)
|
||||||
|
already_fixed.add(cve)
|
||||||
|
break
|
||||||
if resolved_by_release:
|
if resolved_by_release:
|
||||||
report["resolved_by_release_notes"] = resolved_by_release
|
report["resolved_by_release_notes"] = resolved_by_release
|
||||||
|
|
||||||
indeterminate -= fixed_set
|
indeterminate -= fixed_set | already_fixed
|
||||||
|
if already_fixed:
|
||||||
|
report["already_fixed_before_upgrade"] = sorted(already_fixed)
|
||||||
for cve in indeterminate:
|
for cve in indeterminate:
|
||||||
report["cves"][cve]["classification"] = "indeterminate: no fix version published"
|
report["cves"][cve]["classification"] = "indeterminate: no fix version published"
|
||||||
unknown = [c for c in unknown if c not in fixed_set]
|
unknown = [c for c in unknown if c not in fixed_set]
|
||||||
|
|||||||
+45
-14
@@ -9,7 +9,11 @@ Subcommands (the /recipe-report agent runs them around its own review/classifica
|
|||||||
survey [DATE] JSON of the run + every recipe's open PRs + CI verdict + per-recipe upgrade
|
survey [DATE] JSON of the run + every recipe's open PRs + CI verdict + per-recipe upgrade
|
||||||
notes (breaking-change/CVE analysis), and the /upgrade-all summary.
|
notes (breaking-change/CVE analysis), and the /upgrade-all summary.
|
||||||
render SPEC.json OUT.html render the agent's report spec -> a self-contained newspaper HTML page
|
render SPEC.json OUT.html render the agent's report spec -> a self-contained newspaper HTML page
|
||||||
publish OUT.html DATE copy to cc-ci:/var/lib/cc-ci-reports/week-DATE.html and regen the archive index
|
publish OUT.html DATE [KIND] copy to cc-ci:/var/lib/cc-ci-reports/<KIND>-DATE.html and regen the
|
||||||
|
archive index. KIND is `week` (default, the weekly /recipe-report) or `cve`
|
||||||
|
(a /cve-check advisory sweep). BOTH kinds appear in the SAME archive index,
|
||||||
|
newest first, each row suffixed "full report" or "CVE check"; the distinct
|
||||||
|
filename prefix just stops a sweep overwriting a weekly edition.
|
||||||
|
|
||||||
Page order: short lead → the full wire table (priority-sorted, CVEs column) → Addendum → Security
|
Page order: short lead → the full wire table (priority-sorted, CVEs column) → Addendum → Security
|
||||||
Bulletin → per-recipe "What changed".
|
Bulletin → per-recipe "What changed".
|
||||||
@@ -44,6 +48,10 @@ LOGDIR = "/srv/cc-ci/.cc-ci-logs"
|
|||||||
TESTENV = "/srv/cc-ci/.testenv"
|
TESTENV = "/srv/cc-ci/.testenv"
|
||||||
INFRA = {"cc-ci", "cc-ci-orchestrator", "cc-ci-secrets"}
|
INFRA = {"cc-ci", "cc-ci-orchestrator", "cc-ci-secrets"}
|
||||||
HOST_REPORTS = "/var/lib/cc-ci-reports"
|
HOST_REPORTS = "/var/lib/cc-ci-reports"
|
||||||
|
# Both kinds live in ONE archive, distinguished by a suffix on a common title.
|
||||||
|
# prefix -> (page title, index label)
|
||||||
|
KINDS = {"week": ("The Recipe Report", "Week of {d} — full report"),
|
||||||
|
"cve": ("The Recipe Report — CVE check", "{d} — CVE check")}
|
||||||
|
|
||||||
|
|
||||||
def _env():
|
def _env():
|
||||||
@@ -215,8 +223,16 @@ def _table(rows, repo_url=None):
|
|||||||
if repo_url and r.get("recipe") in repo_url:
|
if repo_url and r.get("recipe") in repo_url:
|
||||||
name = f'<a href="{repo_url[r["recipe"]]}">{name}</a>'
|
name = f'<a href="{repo_url[r["recipe"]]}">{name}</a>'
|
||||||
cve = r.get("cve")
|
cve = r.get("cve")
|
||||||
cve_cell = (f'<span class="cve">{int(cve)}</span>' if isinstance(cve, (int, float)) and cve
|
# "?" = advisory scan absent or had failed sources → count NOT authoritative. Per the
|
||||||
else '<span class="muted">none</span>')
|
# /recipe-report guardrail this must NEVER render as "none" (a blank-that-reads-clean is
|
||||||
|
# exactly how two CVSS-9.8 gitea RCEs were misreported as "none" on 2026-08-07). A positive
|
||||||
|
# int is the confirmed CVE count; 0/omit is a confirmed-clean scan.
|
||||||
|
if isinstance(cve, str) and cve.strip() == "?":
|
||||||
|
cve_cell = '<span class="muted" title="advisory scan incomplete or absent — CVE count unknown">?</span>'
|
||||||
|
elif isinstance(cve, (int, float)) and cve:
|
||||||
|
cve_cell = f'<span class="cve">{int(cve)}</span>'
|
||||||
|
else:
|
||||||
|
cve_cell = '<span class="muted">none</span>'
|
||||||
ci = _esc(r.get("ci"))
|
ci = _esc(r.get("ci"))
|
||||||
if r.get("ci_url"):
|
if r.get("ci_url"):
|
||||||
ci = f'<a href="{_esc(r["ci_url"])}">{ci}</a>'
|
ci = f'<a href="{_esc(r["ci_url"])}">{ci}</a>'
|
||||||
@@ -270,8 +286,10 @@ def _mast():
|
|||||||
|
|
||||||
def render(spec_path, out_path):
|
def render(spec_path, out_path):
|
||||||
s = json.load(open(spec_path))
|
s = json.load(open(spec_path))
|
||||||
|
kind = s.get("kind", "week")
|
||||||
|
title = KINDS.get(kind, KINDS["week"])[0]
|
||||||
gen = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
|
gen = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
|
||||||
sub = s.get("subtitle", "Week of " + s["date"])
|
sub = s.get("subtitle", ("Week of " if kind == "week" else "CVE check ") + s["date"])
|
||||||
lead = s.get("lead", "") or ""
|
lead = s.get("lead", "") or ""
|
||||||
# Auto-link recipe-name mentions in the lead to their mirror repos.
|
# Auto-link recipe-name mentions in the lead to their mirror repos.
|
||||||
gitea = _env().get("GITEA_URL", "git.autonomic.zone")
|
gitea = _env().get("GITEA_URL", "git.autonomic.zone")
|
||||||
@@ -285,7 +303,9 @@ def render(spec_path, out_path):
|
|||||||
f'<span>report.ci.commoninternet.net</span><span>{gen}</span></div>'
|
f'<span>report.ci.commoninternet.net</span><span>{gen}</span></div>'
|
||||||
f'<div class="lead">{lead}</div>')
|
f'<div class="lead">{lead}</div>')
|
||||||
# 1) the full wire — every recipe, in the agent's recommended priority order (CVEs first); CVEs column.
|
# 1) the full wire — every recipe, in the agent's recommended priority order (CVEs first); CVEs column.
|
||||||
body += f'<h2>The full wire — every recipe, in priority order</h2>{_table(s.get("table"), repo_url)}'
|
wire = ("The full wire — every recipe, in priority order" if kind == "week"
|
||||||
|
else "Advisory sweep — every recipe, worst first")
|
||||||
|
body += f'<h2>{wire}</h2>{_table(s.get("table"), repo_url)}'
|
||||||
# 2) addendum — special issues to look into (normal-size header); omitted entirely if there are none.
|
# 2) addendum — special issues to look into (normal-size header); omitted entirely if there are none.
|
||||||
add = [a for a in (s.get("addendum") or []) if str(a).strip()]
|
add = [a for a in (s.get("addendum") or []) if str(a).strip()]
|
||||||
if add:
|
if add:
|
||||||
@@ -298,19 +318,30 @@ def render(spec_path, out_path):
|
|||||||
# 4) what changed — a short section per recipe that has a PR
|
# 4) what changed — a short section per recipe that has a PR
|
||||||
if s.get("changes"):
|
if s.get("changes"):
|
||||||
body += f'<h2>What changed</h2>{_changes(s.get("changes"), repo_url)}'
|
body += f'<h2>What changed</h2>{_changes(s.get("changes"), repo_url)}'
|
||||||
body += (f'<footer>The Recipe Report · generated {gen} · '
|
body += (f'<footer>{title} · generated {gen} · '
|
||||||
f'<a href="https://ci.commoninternet.net/">dashboard</a> · <a href="./">archive</a></footer>')
|
f'<a href="https://ci.commoninternet.net/">dashboard</a> · <a href="./">archive</a></footer>')
|
||||||
open(out_path, "w").write(_page("The Recipe Report — " + s["date"], body))
|
open(out_path, "w").write(_page(f"{title} · " + s["date"], body))
|
||||||
print("wrote", out_path)
|
print("wrote", out_path)
|
||||||
|
|
||||||
|
|
||||||
def publish(html_path, date):
|
def publish(html_path, date, kind="week"):
|
||||||
page = f"week-{date}.html"
|
if kind not in KINDS:
|
||||||
|
print(f"unknown kind {kind!r}; expected one of {', '.join(KINDS)}"); sys.exit(2)
|
||||||
|
page = f"{kind}-{date}.html"
|
||||||
subprocess.run(["ssh", "cc-ci", f"cat > {HOST_REPORTS}/{page}"], input=open(html_path, "rb").read(), check=True)
|
subprocess.run(["ssh", "cc-ci", f"cat > {HOST_REPORTS}/{page}"], input=open(html_path, "rb").read(), check=True)
|
||||||
listing = subprocess.run(["ssh", "cc-ci", f"ls -1 {HOST_REPORTS}/week-*.html 2>/dev/null"],
|
# One index over BOTH families, newest first, each row labelled by its kind — an operator looking
|
||||||
capture_output=True, text=True).stdout.split()
|
# for "the latest security picture" should not have to know which skill produced which page.
|
||||||
dates = sorted({os.path.basename(p)[5:-5] for p in listing}, reverse=True)
|
entries = []
|
||||||
lis = "\n".join(f'<li><a href="week-{d}.html">Week of {d}</a><span class="d">{d}</span></li>' for d in dates)
|
for k in KINDS:
|
||||||
|
listing = subprocess.run(["ssh", "cc-ci", f"ls -1 {HOST_REPORTS}/{k}-*.html 2>/dev/null"],
|
||||||
|
capture_output=True, text=True).stdout.split()
|
||||||
|
for pth in listing:
|
||||||
|
d = os.path.basename(pth)[len(k) + 1:-5]
|
||||||
|
if re.fullmatch(r"\d{4}-\d{2}-\d{2}", d):
|
||||||
|
entries.append((d, k))
|
||||||
|
lis = "\n".join(
|
||||||
|
f'<li><a href="{k}-{d}.html">{KINDS[k][1].format(d=d)}</a><span class="d">{d}</span></li>'
|
||||||
|
for d, k in sorted(set(entries), reverse=True))
|
||||||
idx = _page("The Recipe Report — Archive", _mast() +
|
idx = _page("The Recipe Report — Archive", _mast() +
|
||||||
'<div class="dateline"><span>Weekly review of Co-op Cloud recipe upgrades & CI</span>'
|
'<div class="dateline"><span>Weekly review of Co-op Cloud recipe upgrades & CI</span>'
|
||||||
'<span>report.ci.commoninternet.net</span></div>'
|
'<span>report.ci.commoninternet.net</span></div>'
|
||||||
@@ -328,7 +359,7 @@ def main():
|
|||||||
elif cmd == "render":
|
elif cmd == "render":
|
||||||
render(a[1], a[2])
|
render(a[1], a[2])
|
||||||
elif cmd == "publish":
|
elif cmd == "publish":
|
||||||
publish(a[1], a[2])
|
publish(a[1], a[2], a[3] if len(a) > 3 else "week")
|
||||||
else:
|
else:
|
||||||
print(__doc__); sys.exit(2)
|
print(__doc__); sys.exit(2)
|
||||||
|
|
||||||
|
|||||||
@@ -455,13 +455,38 @@ class TestReleaseNoteResolution(unittest.TestCase):
|
|||||||
self.assertEqual(rep["indeterminate"], [])
|
self.assertEqual(rep["indeterminate"], [])
|
||||||
self.assertEqual(rep["resolved_by_release_notes"]["CVE-TBD"], ["7.4.5", "8.0.3"])
|
self.assertEqual(rep["resolved_by_release_notes"]["CVE-TBD"], ["7.4.5", "8.0.3"])
|
||||||
|
|
||||||
def test_release_naming_it_only_outside_the_window_stays_indeterminate(self):
|
def test_naming_releases_all_below_the_window_means_ALREADY_fixed(self):
|
||||||
|
# Every known fix predates the version we were already on, so this upgrade did not deliver
|
||||||
|
# it. That is a DECISION, not an unknown — mailu's redis 8.8.0 → 8.10.0 crosses 12 such
|
||||||
|
# advisories, and calling them "could not judge" overstates the uncertainty.
|
||||||
rep = run_scan([gh("redis/redis", [adv("CVE-TBD", patched="TBD")])],
|
rep = run_scan([gh("redis/redis", [adv("CVE-TBD", patched="TBD")])],
|
||||||
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"],
|
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"],
|
||||||
releases={"CVE-TBD": ["6.2.19"]})
|
releases={"CVE-TBD": ["6.2.19"]})
|
||||||
self.assertEqual(rep["fixed_by_this_upgrade"], [])
|
self.assertEqual(rep["fixed_by_this_upgrade"], [])
|
||||||
|
self.assertEqual(rep["indeterminate"], [])
|
||||||
|
self.assertIn("CVE-TBD", rep["already_fixed_before_upgrade"])
|
||||||
|
self.assertIn("outside-window", rep["cves"]["CVE-TBD"]["classification"])
|
||||||
|
|
||||||
|
def test_naming_releases_only_ABOVE_the_window_stays_indeterminate(self):
|
||||||
|
# The fix landed after our target, so we are still exposed. Deliberately NOT decided as a
|
||||||
|
# tidy "not fixed": it is an open vulnerability and must stay visible to the operator.
|
||||||
|
rep = run_scan([gh("redis/redis", [adv("CVE-TBD", patched="TBD")])],
|
||||||
|
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"],
|
||||||
|
releases={"CVE-TBD": ["9.0.0"]})
|
||||||
|
self.assertEqual(rep["fixed_by_this_upgrade"], [])
|
||||||
self.assertIn("CVE-TBD", rep["indeterminate"])
|
self.assertIn("CVE-TBD", rep["indeterminate"])
|
||||||
|
|
||||||
|
def test_vendor_page_cve_on_the_same_repo_uses_release_notes(self):
|
||||||
|
# mailu announces its Roundcube CVEs only on github.com/Mailu/Mailu/releases. Requiring an
|
||||||
|
# advisory feed sent a deterministic case to pass 2; it is now decided in pass 1.
|
||||||
|
rep = run_scan([gh("Mailu/Mailu", [])],
|
||||||
|
[vendor("https://github.com/Mailu/Mailu/releases", ["CVE-2026-54432"])],
|
||||||
|
v_from="2024.06.55", v_to="2024.06.57",
|
||||||
|
urls=["https://github.com/Mailu/Mailu"],
|
||||||
|
releases={"CVE-2026-54432": ["2024.06.56"]})
|
||||||
|
self.assertIn("CVE-2026-54432", rep["fixed_by_this_upgrade"])
|
||||||
|
self.assertEqual(rep["cve_count_fixed"], 1)
|
||||||
|
|
||||||
def test_release_evidence_is_recorded_for_audit(self):
|
def test_release_evidence_is_recorded_for_audit(self):
|
||||||
rep = run_scan([gh("redis/redis", [adv("CVE-TBD", patched="TBD")])],
|
rep = run_scan([gh("redis/redis", [adv("CVE-TBD", patched="TBD")])],
|
||||||
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"],
|
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"],
|
||||||
|
|||||||
Reference in New Issue
Block a user