From 74725610abc73eb0c051e79c64f686185aee588f Mon Sep 17 00:00:00 2001 From: autonomic-bot Date: Thu, 28 May 2026 04:04:13 +0100 Subject: [PATCH] fix(1e): HC1 upgrade/restore tier calls now pass head_ref (multi-line edit miss) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Earlier perl substitution missed the multi-line upgrade and restore run_lifecycle_tier calls (still passed `target` = VERSION env, None for !testme runs), so perform_upgrade got head_ref=None for upgrade tier → re-checkout skipped → chaos redeploy of leftover prev checkout (vacuous prev→prev that 'passed' via the chaos-label move fallback). Verified e2e on hedgedoc (install,upgrade; commit pending push): upgrade→PR-head: head_ref=09bf4d54 chaos-version=09bf4d54 version=3.0.9+1.10.7→3.0.10+1.10.8 deploy-count=1, install/upgrade=pass, clean teardown. The chaos-version label deterministically matches head_ref — direct proof PR-head code was deployed. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/enroll-recipe.md | 51 +++++---- docs/testing.md | 205 +++++++++++++++++++++++++------------ machine-docs/JOURNAL-1e.md | 14 +++ machine-docs/STATUS-1e.md | 7 +- runner/run_recipe_ci.py | 4 +- 5 files changed, 192 insertions(+), 89 deletions(-) diff --git a/docs/enroll-recipe.md b/docs/enroll-recipe.md index a173b97..18a03f9 100644 --- a/docs/enroll-recipe.md +++ b/docs/enroll-recipe.md @@ -15,24 +15,28 @@ those are discovered and run against the live app (D4 — see below). tests// ├── recipe_meta.py # optional per-recipe harness config (see below) ├── install_steps.sh # optional custom install-steps hook (pre-deploy setup) -├── test_install.py # optional install overlay (else the generic install tier runs) -├── test_upgrade.py # optional upgrade overlay (else the generic upgrade tier runs) -├── test_backup.py # optional backup overlay (else the generic backup tier runs) -└── test_restore.py # optional restore overlay (else the generic restore tier runs) +├── ops.py # optional pre-op seed hooks (pre_install/pre_upgrade/pre_backup/pre_restore) +├── test_install.py # optional install overlay (runs ADDITIVELY alongside generic) +├── test_upgrade.py # optional upgrade overlay (runs ADDITIVELY alongside generic) +├── test_backup.py # optional backup overlay (runs ADDITIVELY alongside generic) +└── test_restore.py # optional restore overlay (runs ADDITIVELY alongside generic) ``` **A recipe is testable with ZERO config:** with no overlay files, the **generic lifecycle suite** runs (install/upgrade/backup/restore) against a single shared deployment — see `docs/testing.md` for -the full model (tiers, deploy-once, override-vs-extend, precedence, the install-steps hook). The -per-recipe dir only holds the bits where the recipe needs *more* than the generic. +the full model (deploy-once, additive generic+overlay, the chaos PR-head upgrade, the HC2 repo-local +allowlist, the install-steps hook). The per-recipe dir only holds the bits where the recipe needs +*more* than the generic. -To add recipe-specific coverage, drop a `tests//test_.py` **overlay** (it OVERRIDES the -generic for that op; absent ⇒ generic runs). Overlays are **assertion-only** against the shared live -deployment (the `live_app` fixture; they never deploy), and reuse the generic op + serving check by -composition (`from harness import generic; generic.do_upgrade(...)` etc.), adding recipe-specific -assertions. Copy an existing overlay (`tests/custom-html/` simple/volume marker; `tests/keycloak/` -admin-API; `tests/matrix-synapse/` `db`-service psql marker). **Do not edit the shared -`tests/conftest.py` / `runner/harness/` to add a recipe** — set per-recipe config in `recipe_meta.py`: +To add recipe-specific coverage, drop a `tests//test_.py` **overlay** — it runs +**ALONGSIDE** the generic for that op (HC3 additive, Phase 1e); the generic floor is never silently +dropped. Overlays are **assertion-only** against the shared live deployment (the `live_app` fixture; +they never perform the op or deploy/teardown — the orchestrator owns those). If the overlay needs to +SEED pre-op state (data-continuity markers, the backup→restore divergence), put `pre_(domain, +meta)` callables in `tests//ops.py` — the orchestrator runs them BEFORE the op. Copy an +existing recipe (`tests/custom-html/` simple/volume marker; `tests/keycloak/` admin-API; `tests/ +matrix-synapse/` `db`-service psql marker). **Do not edit the shared `tests/conftest.py` / +`runner/harness/` to add a recipe** — set per-recipe knobs in `recipe_meta.py`: ```python HEALTH_PATH = "/realms/master" # path that returns a healthy status (default "/") @@ -41,19 +45,24 @@ DEPLOY_TIMEOUT = 600 # seconds for services to converge (default 600 HTTP_TIMEOUT = 600 # seconds for the app to answer (default 300) BACKUP_CAPABLE = True # override backup-capability auto-detect (default: scan compose) EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(domain) -> dict; extra .env keys set at deploy +SKIP_GENERIC = ["upgrade"] # per-recipe opt-out from the generic floor for the listed ops + # ("all"/"*" = every op); rarely needed — generic is the floor ``` Useful `harness.lifecycle` helpers for overlays: `http_get`, `http_fetch`, `http_body`, -`exec_in_app` (use this for data markers — volume/DB, robust to the serving layer); the lifecycle ops -themselves come from `harness.generic` (`assert_serving`, `do_upgrade`, `do_backup`, `do_restore`). -The harness forces `LETS_ENCRYPT_ENV=""` (no ACME), a unique short domain per run, and guarantees -teardown. +`exec_in_app` (use this for data markers — volume/DB, hardened with returncode+retry); the lifecycle +ops themselves are orchestrator-owned (you never call them from an overlay). The harness forces +`LETS_ENCRYPT_ENV=""` (no ACME), a unique short domain per run, and guarantees teardown. -## 3. Recipe-local tests (D4) +## 3. Recipe-local tests (D4) — default-deny (HC2) -If the recipe's own repo contains `tests/test_*.py`, the runner snapshots them right after fetch and -runs them against the **live deployment** as a `recipe-local` stage. Contract: those tests receive -env `CCCI_BASE_URL` (e.g. `https://.ci.commoninternet.net/`) and `CCCI_APP_DOMAIN`. +If the recipe's own repo contains `tests/test_*.py` / `install_steps.sh` / `ops.py`, the runner +snapshots them right after fetch — but per Phase 1e HC2 it executes them **only** for recipes on the +cc-ci approval allowlist `tests/repo-local-approved.txt` (default empty ⇒ default-deny). PR-author +code runs on the CI host with `/run/secrets/*` present, so adding a recipe to the allowlist is a +deliberate cc-ci-maintainer act (in a cc-ci PR, after reviewing that recipe's repo-local tests). +Without approval, only the cc-ci overlays in this repo + the generic floor run. Approved recipe-local +files receive env `CCCI_BASE_URL` (e.g. `https://.ci.commoninternet.net/`) and `CCCI_APP_DOMAIN`. ## 4. Add the repo to the bridge poll list diff --git a/docs/testing.md b/docs/testing.md index 11c179f..0969393 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -1,33 +1,41 @@ -# The cc-ci test architecture — generic suite + layered recipe overlays (Phase 1d) +# The cc-ci test architecture — generic suite + additive recipe overlays (Phase 1d + 1e) -Every recipe gets a **generic lifecycle test suite for free**. Recipe-specific tests *layer on top* -of the generic default rather than being the only thing that runs. So `!testme` is meaningful on -**any** recipe immediately (zero config), and adding recipe-specific coverage is a thin overlay. +Every recipe gets a **generic lifecycle test suite for free** — the floor under every run, always +on by default. Recipe-specific tests *layer additively* on top: when a recipe ships an overlay for an +op, the **generic still runs alongside it** (the floor is never silently lost). So `!testme` is +meaningful on **any** recipe immediately (zero config), and adding recipe-specific coverage is a thin +overlay that adds, it doesn't subtract. ## The model: tiers against one shared deployment A run is a sequence of **tiers**. The orchestrator (`runner/run_recipe_ci.py`) deploys the app **once** and runs each tier against that single live deployment, then tears it down **once** in a -`finally`. Lifecycle ops mutate the deployment **in place** — there is **no redeploy per tier** -(asserted every run: `deploy-count = 1`). +`finally`. The orchestrator **owns** each mutating op (upgrade/backup/restore) and runs it **exactly +once**; the assertion files (generic and overlay) evaluate the *post-op* state and never perform the +op themselves. Asserted every run: **`deploy-count = 1`** (one `abra app new`). ``` deploy ONCE (base version: the previous published version when an upgrade tier will run and one - exists — so upgrade is a real previous→target; else the target / current PR head) - → INSTALL assertions (app already deployed: assert it really serves) - → UPGRADE abra app upgrade in place → target; assert reconverge + serving + the deployment MOVED - → BACKUP abra app backup create; assert a snapshot artifact (backup-capable recipes only) - → RESTORE abra app restore; assert healthy + serving (backup-capable recipes only) - → CUSTOM any non-lifecycle test_*.py (only if defined) + exists — so upgrade is a real previous→PR-head; else the target / current PR head) + → INSTALL [optional pre_install seed] then generic + overlay assertions (no op) + → UPGRADE [optional pre_upgrade seed] then abra app deploy --chaos to PR-head (op once) + then generic + overlay assertions + → BACKUP [optional pre_backup seed] then abra app backup create (op once) + then generic + overlay assertions (backup-capable only) + → RESTORE [optional pre_restore mutate] then abra app restore (op once) + then generic + overlay assertions (backup-capable only) + → CUSTOM any non-lifecycle test_*.py (only if defined) teardown ONCE (in finally) ``` -Each tier is its own `pytest` invocation, so the run reports **per-operation** pass / fail / skip -(`install / upgrade / backup / restore / custom`). The shared live domain is passed to each tier in -`CCCI_APP_DOMAIN` and exposed by the `live_app` fixture; **tiers are assertion-only and never deploy -or tear down** (that is the orchestrator's job). +Each assertion file is its own `pytest` invocation, so the run reports **per-operation** pass / fail +/ skip (`install / upgrade / backup / restore / custom`). The shared live domain is passed in +`CCCI_APP_DOMAIN` and exposed by the `live_app` fixture; **all assertion tiers are assertion-only and +never deploy or tear down** (that is the orchestrator's job). Op results an assertion needs +(pre-upgrade identity, the produced backup `snapshot_id`) pass op→assertion via a run-scoped JSON +state file at `$CCCI_OP_STATE_FILE`, read by `generic.op_state()`. -## The generic default (recipe-agnostic) +## The generic default (recipe-agnostic, the floor — Phase 1e HC3) Lives in the shared harness — `runner/harness/generic.py` + `tests/_generic/test_.py` — so there is no per-recipe copy-paste: @@ -36,81 +44,149 @@ is no per-recipe copy-paste: a real HTTP(S) response in `HEALTH_OK` (which excludes 404, so a Traefik unmatched-router fallback fails) **and** the body isn't Traefik's default 404 page. A bounded poll (no bare `sleep`) so a state-mutating op settles, while a persistent failure still fails within the timeout. A CA-verified - TLS handshake is also run as an **infra cert sanity check** (catches a lapsed/mis-rotated wildcard); + TLS handshake also runs as an **infra cert sanity check** (catches a lapsed/mis-rotated wildcard); it does **not** distinguish app-vs-fallback (Traefik serves the wildcard zone-wide) — that's the converged + non-404 check. -- **upgrade** (`generic.do_upgrade`) — `abra app upgrade` in place to the target, then assert serving - **and that the deployment actually moved** (the `coop-cloud..version` label and/or image - changed). The move-assertion makes a vacuous no-op upgrade impossible to pass. -- **backup** (`generic.do_backup`) — `abra app backup create`; assert a snapshot artifact was produced - (the `snapshot_id` in the create output). Honest limit: the generic verifies the *mechanism*, not - app-specific data integrity (that's an overlay, below). -- **restore** (`generic.do_restore`) — `abra app restore`; assert the app is healthy + serving after. +- **upgrade** (`generic.assert_upgraded`) — assert serving after the orchestrator's chaos upgrade + (HC1: `abra app deploy --chaos` of the PR-head checkout) and that the deployment is genuinely the + code under test: when the intended PR-head commit is known, the deployed + `coop-cloud..chaos-version` label **must match** it — direct, non-vacuous proof. (A stale + prev-checkout chaos redeploy would stamp prev's commit, not the PR-head, and fail here.) When + head_ref is unknown, falls back to a move check (version/image/chaos changed vs pre-upgrade). +- **backup** (`generic.assert_backup_artifact`) — assert a snapshot artifact was produced (the + `snapshot_id` captured by the orchestrator from `abra app backup create`). Honest limit: the + generic verifies the *mechanism*, not app-specific data integrity (that's an overlay, below). +- **restore** (`generic.assert_restore_healthy`) — assert the app is healthy + serving after the + orchestrator's restore op (`assert_serving` polls so the post-restore reconverge settles). **Backup-capability** is auto-detected: a recipe is backup-capable iff a `compose*.yml` carries a truthy `backupbot.backup` label (override with `BACKUP_CAPABLE` in `recipe_meta.py`). For non-backup-capable recipes the backup/restore tiers are a clean **N/A skip** — not a failure. -## Recipe overlays — override or extend (the generic is always the default) +## Recipe overlays — additive (the generic floor is always on by default) Convention: a recipe-specific tier is a file named exactly `test_install.py` / `test_upgrade.py` / -`test_backup.py` / `test_restore.py`. **If present it OVERRIDES the generic for that op; if absent, -the generic runs** (the invariant). Discovery looks in two locations, with this precedence: +`test_backup.py` / `test_restore.py`. **When present it runs ALONGSIDE the generic for that op** +(both evaluate the shared post-op state); when absent, only the generic runs. Overlays are +**assertion-only** — they never perform the op (the orchestrator owns it). + +Overlay sources, in precedence order: ``` -repo-local /tests/test_.py (upstream-authoritative; wins same-name collisions) +repo-local /tests/test_.py (upstream-authoritative; gated by HC2 allowlist) > cc-ci tests//test_.py (CI-curated overlay) - > generic tests/_generic/test_.py (the floor; always present) + + generic tests/_generic/test_.py (the floor; runs alongside by default) ``` -- **Override** — a present `test_.py` replaces the generic assertions for that op. -- **Extend by composition** — an overlay may `from harness import generic` and call - `generic.assert_serving(...)` / `generic.do_upgrade(...)` / `do_backup` / `do_restore`, then add its - own recipe-specific assertions. (This is how every overlay reuses the generic op + serving check and - layers data-continuity on top — no separate "extend" mechanism needed.) -- **Custom (non-lifecycle) `test_*.py`** — any other `test_*.py` (e.g. `test_sso.py`) is **opt-in and - additive**: it has no generic equivalent and runs only when present, discovered from **both** - locations. Lifecycle names are excluded from the custom set. +Only ONE overlay source wins for a given op (repo-local > cc-ci); the generic floor runs **in +addition** unless explicitly opted out. -Overlays are **assertion-only** and run against the shared deployment via the `live_app` fixture (so -deploy-count stays 1). A data-continuity overlay reads/writes the app's *volume/DB* (via -`lifecycle.exec_in_app`, robust to the serving layer), e.g.: +**Custom (non-lifecycle) `test_*.py`** — any other `test_*.py` (e.g. `test_sso.py`) is **opt-in and +additive**: it has no generic equivalent and runs only when present, discovered from both locations +(repo-local gated by the HC2 allowlist). -- `test_upgrade.py`: seed a marker → `generic.do_upgrade(...)` → assert the marker survived. -- `test_backup.py`: seed "original" → `generic.do_backup(...)` → mutate to "mutated". -- `test_restore.py`: `generic.do_restore(...)` → assert the marker is back to "original" (the backup - tier's mutation persists on the shared deployment until the restore tier runs). +### Pre-op seed hooks (per-recipe `ops.py`) -See `tests/custom-html/` (volume marker) and `tests/keycloak/`, `tests/matrix-synapse/`, -`tests/lasuite-docs/` (admin-API / `db`-service markers) for worked examples. +A data-continuity overlay needs to seed state **before** the op (write a marker, create a DB row, +etc.). Since the orchestrator owns the op, overlays place their seed in an optional per-recipe +`tests//ops.py`: + +```python +# tests//ops.py +from harness import lifecycle + +def pre_upgrade(domain, meta): + # seed a marker before the harness performs the upgrade + lifecycle.exec_in_app(domain, ["sh", "-c", "echo upgrade-survives > /path/marker"]) + +def pre_backup(domain, meta): + # establish a known "original" state before the backup op captures it + lifecycle.exec_in_app(domain, ["sh", "-c", "echo original > /path/marker"]) + +def pre_restore(domain, meta): + # diverge from the backed-up state so a successful restore is observable + lifecycle.exec_in_app(domain, ["sh", "-c", "echo mutated > /path/marker"]) +``` + +The orchestrator imports `ops.py` in-process (with the recipe dir on `sys.path`, so it can import +sibling helpers like `kc_admin.py`) and calls `pre_(domain, meta)` immediately before performing +the op. Then `test_.py` asserts the post-op state. See `tests/custom-html/` (volume marker), +`tests/keycloak/` (admin-API/realm), `tests/matrix-synapse/`, `tests/lasuite-docs/` (psql in the `db` +service) for worked examples. + +### Opting out of the generic floor + +The generic runs additively by default. To skip it (e.g. when an overlay's recipe-specific check +fully replaces the generic's mechanism check) set, in increasing specificity: + +- **env `CCCI_SKIP_GENERIC=1`** — skip generic for ALL ops (run-wide). +- **env `CCCI_SKIP_GENERIC_=1`** — e.g. `CCCI_SKIP_GENERIC_UPGRADE=1` — skip generic for that one op. +- **declarative in `recipe_meta.py`** — `SKIP_GENERIC = ["upgrade"]` (per-op) or `SKIP_GENERIC = ["all"]`. + +Opting out is per-recipe and visible in git — not a hidden global. Truthy = `1`/`true`/`yes`/`on`. + +## Repo-local trust gate (HC2) — default-deny + +PR-author-controlled code (a recipe repo's own `tests/test_*.py`, `install_steps.sh`, `ops.py`) runs +on the CI host with `/run/secrets/*` present — an untrusted-code risk. By default the harness runs +**only cc-ci-authored** overlays/hooks (`tests//...`) + the generic. Repo-local code is +**discovered-but-not-executed** unless its recipe appears in **`tests/repo-local-approved.txt`** (a +checked-in, git-auditable allowlist — one recipe name per line; `#` comments + blank lines ignored; +a lone `*` is NOT a wildcard). To approve a recipe a cc-ci maintainer reviews its repo-local tests +and adds the recipe name in a cc-ci PR (override the allowlist location with +`CCCI_REPO_LOCAL_APPROVED_FILE` — used by tests + cold demonstrations). + +The gate is centralized in `runner/harness/discovery.py` (`repo_local_approved` / +`_gated`) so every discovery function (`resolve_overlay_op`, `custom_tests`, `install_steps`, +`pre_op_hook`) honors it identically; unit tests (`tests/unit/test_discovery.py`) pin the behavior +(approved-vs-not for every kind of code). ## Custom install-steps hook (and the graceful-generic rule) Some recipes need setup the generic flow won't do (pre-seed content, set an env/secret, run a one-off command). Provide a shell hook — `tests//install_steps.sh` (cc-ci) or repo-local -`tests/install_steps.sh` (repo-local wins). The orchestrator runs it during the install tier **after -`abra app new` + env defaults, before `abra app deploy`**, with env: +`tests/install_steps.sh` (repo-local wins, gated by the HC2 allowlist). The orchestrator runs it +during the install tier **after `abra app new` + env defaults, before `abra app deploy`**, with env: - `CCCI_APP_DOMAIN` — the run's app domain - `CCCI_RECIPE` — the recipe name - `CCCI_APP_ENV` — path to the app's `.env` (for `abra`-side edits) -**Graceful-generic rule:** a recipe with **no** hook still attempts the generic install. A recipe that -genuinely needs a step will **fail the generic install — and that's the correct, reported outcome** -(per-op `install: fail`); the fix is to add the step, not to special-case the harness. Worked example: -`tests/custom-html-tiny/install_steps.sh` seeds an `index.html` into the static server's content -volume — without it the generic install fails 404, with it it passes. +**Graceful-generic rule:** a recipe with **no** hook still attempts the generic install. A recipe +that genuinely needs a step will **fail the generic install — and that's the correct, reported +outcome** (per-op `install: fail`); the fix is to add the step, not to special-case the harness. +Worked example: `tests/custom-html-tiny/install_steps.sh` seeds an `index.html` into the static +server's content volume — without it the generic install fails 404, with it it passes. + +## The HC1 upgrade path — chaos to the PR-head code under test + +Concretely, the upgrade tier: + +1. base deployment is the **previous published version** (a clean pinned-tag deploy). +2. orchestrator captures `head_ref` (preferring `$REF` — the PR head sha; falls back to the recipe + checkout HEAD for non-PR `!testme`). +3. on the upgrade tier: re-checkout the recipe to `head_ref` (the prev-tag base deploy reset the + working tree), capture the pre-upgrade identity, then **`abra app deploy --chaos`** redeploys the + running app at that checkout — in place, NOT a new install. +4. `assert_upgraded` (generic) asserts serving + that the deployed + `coop-cloud..chaos-version` matches `head_ref` — proving the PR-head code was deployed. + +Reconciliation with the deploy-once guard: `abra.deploy` (chaos) is called directly, not through +`deploy_app`, so `_record_deploy()` does not fire — `deploy-count` counts only `abra app new` +installs and stays 1. ## How to add a recipe overlay (zero → some coverage) -1. The recipe is already testable with **zero config** — enrol it (poll list + mirror) and the generic - suite runs (`docs/enroll-recipe.md`). -2. To add recipe-specific coverage, drop a `tests//test_.py` overlay (copy an existing one, - e.g. `tests/keycloak/test_upgrade.py`). Reuse the generic op via `generic.do_(...)` and add your - assertions. Read/write app state through `lifecycle.exec_in_app` (volume/DB), not HTTP, for data - checks. Set per-recipe knobs (health path, timeouts) in `recipe_meta.py`. -3. If the recipe needs setup before it can serve, add `tests//install_steps.sh`. -4. Never weaken or skip an assertion to make a run pass — a red tier is information. +1. The recipe is already testable with **zero config** — enrol it (poll list + mirror) and the + generic floor runs (`docs/enroll-recipe.md`). +2. To add recipe-specific coverage, drop `tests//test_.py` (copy an existing one, e.g. + `tests/custom-html/test_upgrade.py`). Assert the POST-op state — reading app state through + `lifecycle.exec_in_app` (volume/DB) for data checks, not HTTP. Generic + your overlay both run. +3. If the overlay needs to seed PRE-op state (data-continuity markers, the backup→restore + divergence), drop `tests//ops.py` with `pre_upgrade/pre_backup/pre_restore(domain, meta)`. +4. If the recipe needs install-time setup, add `tests//install_steps.sh`. +5. Set per-recipe knobs (health path, timeouts, opt-out) in `recipe_meta.py`. +6. **Never weaken or skip an assertion to make a run pass** — a red tier is information. Per-recipe config (`tests//recipe_meta.py`, all optional): @@ -121,7 +197,8 @@ DEPLOY_TIMEOUT = 600 # seconds for services to converge (default 600 HTTP_TIMEOUT = 600 # seconds for the app to answer (default 300) BACKUP_CAPABLE = True # override backup-capability auto-detection (default: scan compose) EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(domain) -> dict; extra .env keys set at deploy +SKIP_GENERIC = ["upgrade"] # per-recipe declarative opt-out from generic ops ("all" = every op) ``` -The harness self-tests for discovery/precedence live in `tests/unit/` (run: `cc-ci-run -m pytest -tests/unit`); they are never picked up as overlays/custom tests. +The harness self-tests for discovery / precedence / the HC2 allowlist live in `tests/unit/` (run: +`cc-ci-run -m pytest tests/unit`); they are never picked up as overlays/custom tests. diff --git a/machine-docs/JOURNAL-1e.md b/machine-docs/JOURNAL-1e.md index 9452093..90330bc 100644 --- a/machine-docs/JOURNAL-1e.md +++ b/machine-docs/JOURNAL-1e.md @@ -101,3 +101,17 @@ Next: confirm opt-out result, claim E1/HC3 gate, then E2 (HC1 chaos-to-PR-head). collided (explains my non-deterministic 1.10→1.11 vs 1.10→1.10 and the None head_ref). Manual ad-hoc runs bypass Drone's capacity=1 queue. Going forward I serialize: don't run a recipe manually while a gate is under Adversary verification; verify when `pgrep run_recipe_ci` is clear. + +## 2026-05-28 — E2 head_ref plumbing bug (fixed) +- Debug print at main() head_ref capture showed `head_ref='09bf4d54...'` (correct hash), but + perform_upgrade printed `head_ref=None`. Root cause: my earlier perl regex to swap `target → + head_ref` in the four `run_lifecycle_tier` call sites only matched the SINGLE-LINE form; the + multi-line `upgrade` and `restore` calls (lint-wrapped) still passed `target` (which is the VERSION + env, None for !testme runs). So perform_upgrade got head_ref=None for upgrade tier → re-checkout + skipped → chaos deploy of whatever leftover checkout (prev tag from deploy_app) → vacuous prev→prev + chaos redeploy that "passed" via the chaos-label move fallback. +- Fixed: explicit Edit on the two multi-line calls so they now pass `head_ref` consistently + (`recipe`/`"upgrade"|"backup"|"restore"`, `repo_local`, `domain`, `meta`, `head_ref`, `op_state`). + grep confirms all 4 tier calls pass head_ref. compile OK. +- Net effect now: head_ref reaches perform_upgrade → recipe_checkout_ref(head_ref) restores PR-head + before chaos deploy → after.chaos == head_ref → assert_upgraded match succeeds non-vacuously. diff --git a/machine-docs/STATUS-1e.md b/machine-docs/STATUS-1e.md index 27bb6da..1c11cfb 100644 --- a/machine-docs/STATUS-1e.md +++ b/machine-docs/STATUS-1e.md @@ -18,8 +18,11 @@ Three corrections, each Adversary cold-verified, no test weakened: ## Definition of Done (Phase 1e) — HC1–HC4, each Adversary cold-verified in REVIEW-1e - [ ] **HC1** — PR-head upgrade proven to deploy PR-head; deploy-count guard reconciled (==1). -- [ ] **HC2** — repo-local ignored for a non-approved recipe, run for an approved one. -- [ ] **HC3** — generic runs alongside an overlay by default; skipped only with the opt-out set. +- [x] **HC2** — repo-local ignored for a non-approved recipe, run for an approved one. + Adversary PASS @2026-05-28 (hostile-code probe, no finding; commit c7ae296). +- [x] **HC3** — generic runs alongside an overlay by default; skipped only with the opt-out set. + Adversary PASS @2026-05-28 (re-claim commit e75ec1b; F1e-1 fix commit 6eabfdc; opt-out + default + cold-verified, deploy-count=1, no assertion weakened). - [ ] **HC4** — no regression cold-verified; deploy-once + teardown still sacred. ## Milestones (plan §3) diff --git a/runner/run_recipe_ci.py b/runner/run_recipe_ci.py index 89d13af..b68d097 100644 --- a/runner/run_recipe_ci.py +++ b/runner/run_recipe_ci.py @@ -377,7 +377,7 @@ def main() -> int: if "upgrade" in stages: results["upgrade"] = ( run_lifecycle_tier( - recipe, "upgrade", repo_local, domain, meta, target, op_state + recipe, "upgrade", repo_local, domain, meta, head_ref, op_state ) if prev else "skip" # only one published version → nothing to upgrade from @@ -394,7 +394,7 @@ def main() -> int: if "restore" in stages: results["restore"] = ( run_lifecycle_tier( - recipe, "restore", repo_local, domain, meta, target, op_state + recipe, "restore", repo_local, domain, meta, head_ref, op_state ) if backup_cap else "skip"