fix(1e): HC1 upgrade/restore tier calls now pass head_ref (multi-line edit miss)
Earlier perl substitution missed the multi-line upgrade and restore run_lifecycle_tier calls (still passed `target` = VERSION env, None for !testme runs), so perform_upgrade got head_ref=None for upgrade tier → re-checkout skipped → chaos redeploy of leftover prev checkout (vacuous prev→prev that 'passed' via the chaos-label move fallback). Verified e2e on hedgedoc (install,upgrade; commit pending push): upgrade→PR-head: head_ref=09bf4d54 chaos-version=09bf4d54 version=3.0.9+1.10.7→3.0.10+1.10.8 deploy-count=1, install/upgrade=pass, clean teardown. The chaos-version label deterministically matches head_ref — direct proof PR-head code was deployed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.7
+30
-21
@@ -15,24 +15,28 @@ those are discovered and run against the live app (D4 — see below).
|
|||||||
tests/<recipe>/
|
tests/<recipe>/
|
||||||
├── recipe_meta.py # optional per-recipe harness config (see below)
|
├── recipe_meta.py # optional per-recipe harness config (see below)
|
||||||
├── install_steps.sh # optional custom install-steps hook (pre-deploy setup)
|
├── install_steps.sh # optional custom install-steps hook (pre-deploy setup)
|
||||||
├── test_install.py # optional install overlay (else the generic install tier runs)
|
├── ops.py # optional pre-op seed hooks (pre_install/pre_upgrade/pre_backup/pre_restore)
|
||||||
├── test_upgrade.py # optional upgrade overlay (else the generic upgrade tier runs)
|
├── test_install.py # optional install overlay (runs ADDITIVELY alongside generic)
|
||||||
├── test_backup.py # optional backup overlay (else the generic backup tier runs)
|
├── test_upgrade.py # optional upgrade overlay (runs ADDITIVELY alongside generic)
|
||||||
└── test_restore.py # optional restore overlay (else the generic restore tier runs)
|
├── test_backup.py # optional backup overlay (runs ADDITIVELY alongside generic)
|
||||||
|
└── test_restore.py # optional restore overlay (runs ADDITIVELY alongside generic)
|
||||||
```
|
```
|
||||||
|
|
||||||
**A recipe is testable with ZERO config:** with no overlay files, the **generic lifecycle suite**
|
**A recipe is testable with ZERO config:** with no overlay files, the **generic lifecycle suite**
|
||||||
runs (install/upgrade/backup/restore) against a single shared deployment — see `docs/testing.md` for
|
runs (install/upgrade/backup/restore) against a single shared deployment — see `docs/testing.md` for
|
||||||
the full model (tiers, deploy-once, override-vs-extend, precedence, the install-steps hook). The
|
the full model (deploy-once, additive generic+overlay, the chaos PR-head upgrade, the HC2 repo-local
|
||||||
per-recipe dir only holds the bits where the recipe needs *more* than the generic.
|
allowlist, the install-steps hook). The per-recipe dir only holds the bits where the recipe needs
|
||||||
|
*more* than the generic.
|
||||||
|
|
||||||
To add recipe-specific coverage, drop a `tests/<recipe>/test_<op>.py` **overlay** (it OVERRIDES the
|
To add recipe-specific coverage, drop a `tests/<recipe>/test_<op>.py` **overlay** — it runs
|
||||||
generic for that op; absent ⇒ generic runs). Overlays are **assertion-only** against the shared live
|
**ALONGSIDE** the generic for that op (HC3 additive, Phase 1e); the generic floor is never silently
|
||||||
deployment (the `live_app` fixture; they never deploy), and reuse the generic op + serving check by
|
dropped. Overlays are **assertion-only** against the shared live deployment (the `live_app` fixture;
|
||||||
composition (`from harness import generic; generic.do_upgrade(...)` etc.), adding recipe-specific
|
they never perform the op or deploy/teardown — the orchestrator owns those). If the overlay needs to
|
||||||
assertions. Copy an existing overlay (`tests/custom-html/` simple/volume marker; `tests/keycloak/`
|
SEED pre-op state (data-continuity markers, the backup→restore divergence), put `pre_<op>(domain,
|
||||||
admin-API; `tests/matrix-synapse/` `db`-service psql marker). **Do not edit the shared
|
meta)` callables in `tests/<recipe>/ops.py` — the orchestrator runs them BEFORE the op. Copy an
|
||||||
`tests/conftest.py` / `runner/harness/` to add a recipe** — set per-recipe config in `recipe_meta.py`:
|
existing recipe (`tests/custom-html/` simple/volume marker; `tests/keycloak/` admin-API; `tests/
|
||||||
|
matrix-synapse/` `db`-service psql marker). **Do not edit the shared `tests/conftest.py` /
|
||||||
|
`runner/harness/` to add a recipe** — set per-recipe knobs in `recipe_meta.py`:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
HEALTH_PATH = "/realms/master" # path that returns a healthy status (default "/")
|
HEALTH_PATH = "/realms/master" # path that returns a healthy status (default "/")
|
||||||
@@ -41,19 +45,24 @@ DEPLOY_TIMEOUT = 600 # seconds for services to converge (default 600
|
|||||||
HTTP_TIMEOUT = 600 # seconds for the app to answer (default 300)
|
HTTP_TIMEOUT = 600 # seconds for the app to answer (default 300)
|
||||||
BACKUP_CAPABLE = True # override backup-capability auto-detect (default: scan compose)
|
BACKUP_CAPABLE = True # override backup-capability auto-detect (default: scan compose)
|
||||||
EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(domain) -> dict; extra .env keys set at deploy
|
EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(domain) -> dict; extra .env keys set at deploy
|
||||||
|
SKIP_GENERIC = ["upgrade"] # per-recipe opt-out from the generic floor for the listed ops
|
||||||
|
# ("all"/"*" = every op); rarely needed — generic is the floor
|
||||||
```
|
```
|
||||||
|
|
||||||
Useful `harness.lifecycle` helpers for overlays: `http_get`, `http_fetch`, `http_body`,
|
Useful `harness.lifecycle` helpers for overlays: `http_get`, `http_fetch`, `http_body`,
|
||||||
`exec_in_app` (use this for data markers — volume/DB, robust to the serving layer); the lifecycle ops
|
`exec_in_app` (use this for data markers — volume/DB, hardened with returncode+retry); the lifecycle
|
||||||
themselves come from `harness.generic` (`assert_serving`, `do_upgrade`, `do_backup`, `do_restore`).
|
ops themselves are orchestrator-owned (you never call them from an overlay). The harness forces
|
||||||
The harness forces `LETS_ENCRYPT_ENV=""` (no ACME), a unique short domain per run, and guarantees
|
`LETS_ENCRYPT_ENV=""` (no ACME), a unique short domain per run, and guarantees teardown.
|
||||||
teardown.
|
|
||||||
|
|
||||||
## 3. Recipe-local tests (D4)
|
## 3. Recipe-local tests (D4) — default-deny (HC2)
|
||||||
|
|
||||||
If the recipe's own repo contains `tests/test_*.py`, the runner snapshots them right after fetch and
|
If the recipe's own repo contains `tests/test_*.py` / `install_steps.sh` / `ops.py`, the runner
|
||||||
runs them against the **live deployment** as a `recipe-local` stage. Contract: those tests receive
|
snapshots them right after fetch — but per Phase 1e HC2 it executes them **only** for recipes on the
|
||||||
env `CCCI_BASE_URL` (e.g. `https://<app>.ci.commoninternet.net/`) and `CCCI_APP_DOMAIN`.
|
cc-ci approval allowlist `tests/repo-local-approved.txt` (default empty ⇒ default-deny). PR-author
|
||||||
|
code runs on the CI host with `/run/secrets/*` present, so adding a recipe to the allowlist is a
|
||||||
|
deliberate cc-ci-maintainer act (in a cc-ci PR, after reviewing that recipe's repo-local tests).
|
||||||
|
Without approval, only the cc-ci overlays in this repo + the generic floor run. Approved recipe-local
|
||||||
|
files receive env `CCCI_BASE_URL` (e.g. `https://<app>.ci.commoninternet.net/`) and `CCCI_APP_DOMAIN`.
|
||||||
|
|
||||||
## 4. Add the repo to the bridge poll list
|
## 4. Add the repo to the bridge poll list
|
||||||
|
|
||||||
|
|||||||
+141
-64
@@ -1,33 +1,41 @@
|
|||||||
# The cc-ci test architecture — generic suite + layered recipe overlays (Phase 1d)
|
# The cc-ci test architecture — generic suite + additive recipe overlays (Phase 1d + 1e)
|
||||||
|
|
||||||
Every recipe gets a **generic lifecycle test suite for free**. Recipe-specific tests *layer on top*
|
Every recipe gets a **generic lifecycle test suite for free** — the floor under every run, always
|
||||||
of the generic default rather than being the only thing that runs. So `!testme` is meaningful on
|
on by default. Recipe-specific tests *layer additively* on top: when a recipe ships an overlay for an
|
||||||
**any** recipe immediately (zero config), and adding recipe-specific coverage is a thin overlay.
|
op, the **generic still runs alongside it** (the floor is never silently lost). So `!testme` is
|
||||||
|
meaningful on **any** recipe immediately (zero config), and adding recipe-specific coverage is a thin
|
||||||
|
overlay that adds, it doesn't subtract.
|
||||||
|
|
||||||
## The model: tiers against one shared deployment
|
## The model: tiers against one shared deployment
|
||||||
|
|
||||||
A run is a sequence of **tiers**. The orchestrator (`runner/run_recipe_ci.py`) deploys the app
|
A run is a sequence of **tiers**. The orchestrator (`runner/run_recipe_ci.py`) deploys the app
|
||||||
**once** and runs each tier against that single live deployment, then tears it down **once** in a
|
**once** and runs each tier against that single live deployment, then tears it down **once** in a
|
||||||
`finally`. Lifecycle ops mutate the deployment **in place** — there is **no redeploy per tier**
|
`finally`. The orchestrator **owns** each mutating op (upgrade/backup/restore) and runs it **exactly
|
||||||
(asserted every run: `deploy-count = 1`).
|
once**; the assertion files (generic and overlay) evaluate the *post-op* state and never perform the
|
||||||
|
op themselves. Asserted every run: **`deploy-count = 1`** (one `abra app new`).
|
||||||
|
|
||||||
```
|
```
|
||||||
deploy ONCE (base version: the previous published version when an upgrade tier will run and one
|
deploy ONCE (base version: the previous published version when an upgrade tier will run and one
|
||||||
exists — so upgrade is a real previous→target; else the target / current PR head)
|
exists — so upgrade is a real previous→PR-head; else the target / current PR head)
|
||||||
→ INSTALL assertions (app already deployed: assert it really serves)
|
→ INSTALL [optional pre_install seed] then generic + overlay assertions (no op)
|
||||||
→ UPGRADE abra app upgrade in place → target; assert reconverge + serving + the deployment MOVED
|
→ UPGRADE [optional pre_upgrade seed] then abra app deploy --chaos to PR-head (op once)
|
||||||
→ BACKUP abra app backup create; assert a snapshot artifact (backup-capable recipes only)
|
then generic + overlay assertions
|
||||||
→ RESTORE abra app restore; assert healthy + serving (backup-capable recipes only)
|
→ BACKUP [optional pre_backup seed] then abra app backup create (op once)
|
||||||
→ CUSTOM any non-lifecycle test_*.py (only if defined)
|
then generic + overlay assertions (backup-capable only)
|
||||||
|
→ RESTORE [optional pre_restore mutate] then abra app restore (op once)
|
||||||
|
then generic + overlay assertions (backup-capable only)
|
||||||
|
→ CUSTOM any non-lifecycle test_*.py (only if defined)
|
||||||
teardown ONCE (in finally)
|
teardown ONCE (in finally)
|
||||||
```
|
```
|
||||||
|
|
||||||
Each tier is its own `pytest` invocation, so the run reports **per-operation** pass / fail / skip
|
Each assertion file is its own `pytest` invocation, so the run reports **per-operation** pass / fail
|
||||||
(`install / upgrade / backup / restore / custom`). The shared live domain is passed to each tier in
|
/ skip (`install / upgrade / backup / restore / custom`). The shared live domain is passed in
|
||||||
`CCCI_APP_DOMAIN` and exposed by the `live_app` fixture; **tiers are assertion-only and never deploy
|
`CCCI_APP_DOMAIN` and exposed by the `live_app` fixture; **all assertion tiers are assertion-only and
|
||||||
or tear down** (that is the orchestrator's job).
|
never deploy or tear down** (that is the orchestrator's job). Op results an assertion needs
|
||||||
|
(pre-upgrade identity, the produced backup `snapshot_id`) pass op→assertion via a run-scoped JSON
|
||||||
|
state file at `$CCCI_OP_STATE_FILE`, read by `generic.op_state()`.
|
||||||
|
|
||||||
## The generic default (recipe-agnostic)
|
## The generic default (recipe-agnostic, the floor — Phase 1e HC3)
|
||||||
|
|
||||||
Lives in the shared harness — `runner/harness/generic.py` + `tests/_generic/test_<op>.py` — so there
|
Lives in the shared harness — `runner/harness/generic.py` + `tests/_generic/test_<op>.py` — so there
|
||||||
is no per-recipe copy-paste:
|
is no per-recipe copy-paste:
|
||||||
@@ -36,81 +44,149 @@ is no per-recipe copy-paste:
|
|||||||
a real HTTP(S) response in `HEALTH_OK` (which excludes 404, so a Traefik unmatched-router fallback
|
a real HTTP(S) response in `HEALTH_OK` (which excludes 404, so a Traefik unmatched-router fallback
|
||||||
fails) **and** the body isn't Traefik's default 404 page. A bounded poll (no bare `sleep`) so a
|
fails) **and** the body isn't Traefik's default 404 page. A bounded poll (no bare `sleep`) so a
|
||||||
state-mutating op settles, while a persistent failure still fails within the timeout. A CA-verified
|
state-mutating op settles, while a persistent failure still fails within the timeout. A CA-verified
|
||||||
TLS handshake is also run as an **infra cert sanity check** (catches a lapsed/mis-rotated wildcard);
|
TLS handshake also runs as an **infra cert sanity check** (catches a lapsed/mis-rotated wildcard);
|
||||||
it does **not** distinguish app-vs-fallback (Traefik serves the wildcard zone-wide) — that's the
|
it does **not** distinguish app-vs-fallback (Traefik serves the wildcard zone-wide) — that's the
|
||||||
converged + non-404 check.
|
converged + non-404 check.
|
||||||
- **upgrade** (`generic.do_upgrade`) — `abra app upgrade` in place to the target, then assert serving
|
- **upgrade** (`generic.assert_upgraded`) — assert serving after the orchestrator's chaos upgrade
|
||||||
**and that the deployment actually moved** (the `coop-cloud.<stack>.version` label and/or image
|
(HC1: `abra app deploy --chaos` of the PR-head checkout) and that the deployment is genuinely the
|
||||||
changed). The move-assertion makes a vacuous no-op upgrade impossible to pass.
|
code under test: when the intended PR-head commit is known, the deployed
|
||||||
- **backup** (`generic.do_backup`) — `abra app backup create`; assert a snapshot artifact was produced
|
`coop-cloud.<stack>.chaos-version` label **must match** it — direct, non-vacuous proof. (A stale
|
||||||
(the `snapshot_id` in the create output). Honest limit: the generic verifies the *mechanism*, not
|
prev-checkout chaos redeploy would stamp prev's commit, not the PR-head, and fail here.) When
|
||||||
app-specific data integrity (that's an overlay, below).
|
head_ref is unknown, falls back to a move check (version/image/chaos changed vs pre-upgrade).
|
||||||
- **restore** (`generic.do_restore`) — `abra app restore`; assert the app is healthy + serving after.
|
- **backup** (`generic.assert_backup_artifact`) — assert a snapshot artifact was produced (the
|
||||||
|
`snapshot_id` captured by the orchestrator from `abra app backup create`). Honest limit: the
|
||||||
|
generic verifies the *mechanism*, not app-specific data integrity (that's an overlay, below).
|
||||||
|
- **restore** (`generic.assert_restore_healthy`) — assert the app is healthy + serving after the
|
||||||
|
orchestrator's restore op (`assert_serving` polls so the post-restore reconverge settles).
|
||||||
|
|
||||||
**Backup-capability** is auto-detected: a recipe is backup-capable iff a `compose*.yml` carries a
|
**Backup-capability** is auto-detected: a recipe is backup-capable iff a `compose*.yml` carries a
|
||||||
truthy `backupbot.backup` label (override with `BACKUP_CAPABLE` in `recipe_meta.py`). For
|
truthy `backupbot.backup` label (override with `BACKUP_CAPABLE` in `recipe_meta.py`). For
|
||||||
non-backup-capable recipes the backup/restore tiers are a clean **N/A skip** — not a failure.
|
non-backup-capable recipes the backup/restore tiers are a clean **N/A skip** — not a failure.
|
||||||
|
|
||||||
## Recipe overlays — override or extend (the generic is always the default)
|
## Recipe overlays — additive (the generic floor is always on by default)
|
||||||
|
|
||||||
Convention: a recipe-specific tier is a file named exactly `test_install.py` / `test_upgrade.py` /
|
Convention: a recipe-specific tier is a file named exactly `test_install.py` / `test_upgrade.py` /
|
||||||
`test_backup.py` / `test_restore.py`. **If present it OVERRIDES the generic for that op; if absent,
|
`test_backup.py` / `test_restore.py`. **When present it runs ALONGSIDE the generic for that op**
|
||||||
the generic runs** (the invariant). Discovery looks in two locations, with this precedence:
|
(both evaluate the shared post-op state); when absent, only the generic runs. Overlays are
|
||||||
|
**assertion-only** — they never perform the op (the orchestrator owns it).
|
||||||
|
|
||||||
|
Overlay sources, in precedence order:
|
||||||
|
|
||||||
```
|
```
|
||||||
repo-local <recipe-repo>/tests/test_<op>.py (upstream-authoritative; wins same-name collisions)
|
repo-local <recipe-repo>/tests/test_<op>.py (upstream-authoritative; gated by HC2 allowlist)
|
||||||
> cc-ci tests/<recipe>/test_<op>.py (CI-curated overlay)
|
> cc-ci tests/<recipe>/test_<op>.py (CI-curated overlay)
|
||||||
> generic tests/_generic/test_<op>.py (the floor; always present)
|
+ generic tests/_generic/test_<op>.py (the floor; runs alongside by default)
|
||||||
```
|
```
|
||||||
|
|
||||||
- **Override** — a present `test_<op>.py` replaces the generic assertions for that op.
|
Only ONE overlay source wins for a given op (repo-local > cc-ci); the generic floor runs **in
|
||||||
- **Extend by composition** — an overlay may `from harness import generic` and call
|
addition** unless explicitly opted out.
|
||||||
`generic.assert_serving(...)` / `generic.do_upgrade(...)` / `do_backup` / `do_restore`, then add its
|
|
||||||
own recipe-specific assertions. (This is how every overlay reuses the generic op + serving check and
|
|
||||||
layers data-continuity on top — no separate "extend" mechanism needed.)
|
|
||||||
- **Custom (non-lifecycle) `test_*.py`** — any other `test_*.py` (e.g. `test_sso.py`) is **opt-in and
|
|
||||||
additive**: it has no generic equivalent and runs only when present, discovered from **both**
|
|
||||||
locations. Lifecycle names are excluded from the custom set.
|
|
||||||
|
|
||||||
Overlays are **assertion-only** and run against the shared deployment via the `live_app` fixture (so
|
**Custom (non-lifecycle) `test_*.py`** — any other `test_*.py` (e.g. `test_sso.py`) is **opt-in and
|
||||||
deploy-count stays 1). A data-continuity overlay reads/writes the app's *volume/DB* (via
|
additive**: it has no generic equivalent and runs only when present, discovered from both locations
|
||||||
`lifecycle.exec_in_app`, robust to the serving layer), e.g.:
|
(repo-local gated by the HC2 allowlist).
|
||||||
|
|
||||||
- `test_upgrade.py`: seed a marker → `generic.do_upgrade(...)` → assert the marker survived.
|
### Pre-op seed hooks (per-recipe `ops.py`)
|
||||||
- `test_backup.py`: seed "original" → `generic.do_backup(...)` → mutate to "mutated".
|
|
||||||
- `test_restore.py`: `generic.do_restore(...)` → assert the marker is back to "original" (the backup
|
|
||||||
tier's mutation persists on the shared deployment until the restore tier runs).
|
|
||||||
|
|
||||||
See `tests/custom-html/` (volume marker) and `tests/keycloak/`, `tests/matrix-synapse/`,
|
A data-continuity overlay needs to seed state **before** the op (write a marker, create a DB row,
|
||||||
`tests/lasuite-docs/` (admin-API / `db`-service markers) for worked examples.
|
etc.). Since the orchestrator owns the op, overlays place their seed in an optional per-recipe
|
||||||
|
`tests/<recipe>/ops.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# tests/<recipe>/ops.py
|
||||||
|
from harness import lifecycle
|
||||||
|
|
||||||
|
def pre_upgrade(domain, meta):
|
||||||
|
# seed a marker before the harness performs the upgrade
|
||||||
|
lifecycle.exec_in_app(domain, ["sh", "-c", "echo upgrade-survives > /path/marker"])
|
||||||
|
|
||||||
|
def pre_backup(domain, meta):
|
||||||
|
# establish a known "original" state before the backup op captures it
|
||||||
|
lifecycle.exec_in_app(domain, ["sh", "-c", "echo original > /path/marker"])
|
||||||
|
|
||||||
|
def pre_restore(domain, meta):
|
||||||
|
# diverge from the backed-up state so a successful restore is observable
|
||||||
|
lifecycle.exec_in_app(domain, ["sh", "-c", "echo mutated > /path/marker"])
|
||||||
|
```
|
||||||
|
|
||||||
|
The orchestrator imports `ops.py` in-process (with the recipe dir on `sys.path`, so it can import
|
||||||
|
sibling helpers like `kc_admin.py`) and calls `pre_<op>(domain, meta)` immediately before performing
|
||||||
|
the op. Then `test_<op>.py` asserts the post-op state. See `tests/custom-html/` (volume marker),
|
||||||
|
`tests/keycloak/` (admin-API/realm), `tests/matrix-synapse/`, `tests/lasuite-docs/` (psql in the `db`
|
||||||
|
service) for worked examples.
|
||||||
|
|
||||||
|
### Opting out of the generic floor
|
||||||
|
|
||||||
|
The generic runs additively by default. To skip it (e.g. when an overlay's recipe-specific check
|
||||||
|
fully replaces the generic's mechanism check) set, in increasing specificity:
|
||||||
|
|
||||||
|
- **env `CCCI_SKIP_GENERIC=1`** — skip generic for ALL ops (run-wide).
|
||||||
|
- **env `CCCI_SKIP_GENERIC_<OP>=1`** — e.g. `CCCI_SKIP_GENERIC_UPGRADE=1` — skip generic for that one op.
|
||||||
|
- **declarative in `recipe_meta.py`** — `SKIP_GENERIC = ["upgrade"]` (per-op) or `SKIP_GENERIC = ["all"]`.
|
||||||
|
|
||||||
|
Opting out is per-recipe and visible in git — not a hidden global. Truthy = `1`/`true`/`yes`/`on`.
|
||||||
|
|
||||||
|
## Repo-local trust gate (HC2) — default-deny
|
||||||
|
|
||||||
|
PR-author-controlled code (a recipe repo's own `tests/test_*.py`, `install_steps.sh`, `ops.py`) runs
|
||||||
|
on the CI host with `/run/secrets/*` present — an untrusted-code risk. By default the harness runs
|
||||||
|
**only cc-ci-authored** overlays/hooks (`tests/<recipe>/...`) + the generic. Repo-local code is
|
||||||
|
**discovered-but-not-executed** unless its recipe appears in **`tests/repo-local-approved.txt`** (a
|
||||||
|
checked-in, git-auditable allowlist — one recipe name per line; `#` comments + blank lines ignored;
|
||||||
|
a lone `*` is NOT a wildcard). To approve a recipe a cc-ci maintainer reviews its repo-local tests
|
||||||
|
and adds the recipe name in a cc-ci PR (override the allowlist location with
|
||||||
|
`CCCI_REPO_LOCAL_APPROVED_FILE` — used by tests + cold demonstrations).
|
||||||
|
|
||||||
|
The gate is centralized in `runner/harness/discovery.py` (`repo_local_approved` /
|
||||||
|
`_gated`) so every discovery function (`resolve_overlay_op`, `custom_tests`, `install_steps`,
|
||||||
|
`pre_op_hook`) honors it identically; unit tests (`tests/unit/test_discovery.py`) pin the behavior
|
||||||
|
(approved-vs-not for every kind of code).
|
||||||
|
|
||||||
## Custom install-steps hook (and the graceful-generic rule)
|
## Custom install-steps hook (and the graceful-generic rule)
|
||||||
|
|
||||||
Some recipes need setup the generic flow won't do (pre-seed content, set an env/secret, run a one-off
|
Some recipes need setup the generic flow won't do (pre-seed content, set an env/secret, run a one-off
|
||||||
command). Provide a shell hook — `tests/<recipe>/install_steps.sh` (cc-ci) or repo-local
|
command). Provide a shell hook — `tests/<recipe>/install_steps.sh` (cc-ci) or repo-local
|
||||||
`tests/install_steps.sh` (repo-local wins). The orchestrator runs it during the install tier **after
|
`tests/install_steps.sh` (repo-local wins, gated by the HC2 allowlist). The orchestrator runs it
|
||||||
`abra app new` + env defaults, before `abra app deploy`**, with env:
|
during the install tier **after `abra app new` + env defaults, before `abra app deploy`**, with env:
|
||||||
|
|
||||||
- `CCCI_APP_DOMAIN` — the run's app domain
|
- `CCCI_APP_DOMAIN` — the run's app domain
|
||||||
- `CCCI_RECIPE` — the recipe name
|
- `CCCI_RECIPE` — the recipe name
|
||||||
- `CCCI_APP_ENV` — path to the app's `.env` (for `abra`-side edits)
|
- `CCCI_APP_ENV` — path to the app's `.env` (for `abra`-side edits)
|
||||||
|
|
||||||
**Graceful-generic rule:** a recipe with **no** hook still attempts the generic install. A recipe that
|
**Graceful-generic rule:** a recipe with **no** hook still attempts the generic install. A recipe
|
||||||
genuinely needs a step will **fail the generic install — and that's the correct, reported outcome**
|
that genuinely needs a step will **fail the generic install — and that's the correct, reported
|
||||||
(per-op `install: fail`); the fix is to add the step, not to special-case the harness. Worked example:
|
outcome** (per-op `install: fail`); the fix is to add the step, not to special-case the harness.
|
||||||
`tests/custom-html-tiny/install_steps.sh` seeds an `index.html` into the static server's content
|
Worked example: `tests/custom-html-tiny/install_steps.sh` seeds an `index.html` into the static
|
||||||
volume — without it the generic install fails 404, with it it passes.
|
server's content volume — without it the generic install fails 404, with it it passes.
|
||||||
|
|
||||||
|
## The HC1 upgrade path — chaos to the PR-head code under test
|
||||||
|
|
||||||
|
Concretely, the upgrade tier:
|
||||||
|
|
||||||
|
1. base deployment is the **previous published version** (a clean pinned-tag deploy).
|
||||||
|
2. orchestrator captures `head_ref` (preferring `$REF` — the PR head sha; falls back to the recipe
|
||||||
|
checkout HEAD for non-PR `!testme`).
|
||||||
|
3. on the upgrade tier: re-checkout the recipe to `head_ref` (the prev-tag base deploy reset the
|
||||||
|
working tree), capture the pre-upgrade identity, then **`abra app deploy --chaos`** redeploys the
|
||||||
|
running app at that checkout — in place, NOT a new install.
|
||||||
|
4. `assert_upgraded` (generic) asserts serving + that the deployed
|
||||||
|
`coop-cloud.<stack>.chaos-version` matches `head_ref` — proving the PR-head code was deployed.
|
||||||
|
|
||||||
|
Reconciliation with the deploy-once guard: `abra.deploy` (chaos) is called directly, not through
|
||||||
|
`deploy_app`, so `_record_deploy()` does not fire — `deploy-count` counts only `abra app new`
|
||||||
|
installs and stays 1.
|
||||||
|
|
||||||
## How to add a recipe overlay (zero → some coverage)
|
## How to add a recipe overlay (zero → some coverage)
|
||||||
|
|
||||||
1. The recipe is already testable with **zero config** — enrol it (poll list + mirror) and the generic
|
1. The recipe is already testable with **zero config** — enrol it (poll list + mirror) and the
|
||||||
suite runs (`docs/enroll-recipe.md`).
|
generic floor runs (`docs/enroll-recipe.md`).
|
||||||
2. To add recipe-specific coverage, drop a `tests/<recipe>/test_<op>.py` overlay (copy an existing one,
|
2. To add recipe-specific coverage, drop `tests/<recipe>/test_<op>.py` (copy an existing one, e.g.
|
||||||
e.g. `tests/keycloak/test_upgrade.py`). Reuse the generic op via `generic.do_<op>(...)` and add your
|
`tests/custom-html/test_upgrade.py`). Assert the POST-op state — reading app state through
|
||||||
assertions. Read/write app state through `lifecycle.exec_in_app` (volume/DB), not HTTP, for data
|
`lifecycle.exec_in_app` (volume/DB) for data checks, not HTTP. Generic + your overlay both run.
|
||||||
checks. Set per-recipe knobs (health path, timeouts) in `recipe_meta.py`.
|
3. If the overlay needs to seed PRE-op state (data-continuity markers, the backup→restore
|
||||||
3. If the recipe needs setup before it can serve, add `tests/<recipe>/install_steps.sh`.
|
divergence), drop `tests/<recipe>/ops.py` with `pre_upgrade/pre_backup/pre_restore(domain, meta)`.
|
||||||
4. Never weaken or skip an assertion to make a run pass — a red tier is information.
|
4. If the recipe needs install-time setup, add `tests/<recipe>/install_steps.sh`.
|
||||||
|
5. Set per-recipe knobs (health path, timeouts, opt-out) in `recipe_meta.py`.
|
||||||
|
6. **Never weaken or skip an assertion to make a run pass** — a red tier is information.
|
||||||
|
|
||||||
Per-recipe config (`tests/<recipe>/recipe_meta.py`, all optional):
|
Per-recipe config (`tests/<recipe>/recipe_meta.py`, all optional):
|
||||||
|
|
||||||
@@ -121,7 +197,8 @@ DEPLOY_TIMEOUT = 600 # seconds for services to converge (default 600
|
|||||||
HTTP_TIMEOUT = 600 # seconds for the app to answer (default 300)
|
HTTP_TIMEOUT = 600 # seconds for the app to answer (default 300)
|
||||||
BACKUP_CAPABLE = True # override backup-capability auto-detection (default: scan compose)
|
BACKUP_CAPABLE = True # override backup-capability auto-detection (default: scan compose)
|
||||||
EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(domain) -> dict; extra .env keys set at deploy
|
EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(domain) -> dict; extra .env keys set at deploy
|
||||||
|
SKIP_GENERIC = ["upgrade"] # per-recipe declarative opt-out from generic ops ("all" = every op)
|
||||||
```
|
```
|
||||||
|
|
||||||
The harness self-tests for discovery/precedence live in `tests/unit/` (run: `cc-ci-run -m pytest
|
The harness self-tests for discovery / precedence / the HC2 allowlist live in `tests/unit/` (run:
|
||||||
tests/unit`); they are never picked up as overlays/custom tests.
|
`cc-ci-run -m pytest tests/unit`); they are never picked up as overlays/custom tests.
|
||||||
|
|||||||
@@ -101,3 +101,17 @@ Next: confirm opt-out result, claim E1/HC3 gate, then E2 (HC1 chaos-to-PR-head).
|
|||||||
collided (explains my non-deterministic 1.10→1.11 vs 1.10→1.10 and the None head_ref). Manual ad-hoc
|
collided (explains my non-deterministic 1.10→1.11 vs 1.10→1.10 and the None head_ref). Manual ad-hoc
|
||||||
runs bypass Drone's capacity=1 queue. Going forward I serialize: don't run a recipe manually while a
|
runs bypass Drone's capacity=1 queue. Going forward I serialize: don't run a recipe manually while a
|
||||||
gate is under Adversary verification; verify when `pgrep run_recipe_ci` is clear.
|
gate is under Adversary verification; verify when `pgrep run_recipe_ci` is clear.
|
||||||
|
|
||||||
|
## 2026-05-28 — E2 head_ref plumbing bug (fixed)
|
||||||
|
- Debug print at main() head_ref capture showed `head_ref='09bf4d54...'` (correct hash), but
|
||||||
|
perform_upgrade printed `head_ref=None`. Root cause: my earlier perl regex to swap `target →
|
||||||
|
head_ref` in the four `run_lifecycle_tier` call sites only matched the SINGLE-LINE form; the
|
||||||
|
multi-line `upgrade` and `restore` calls (lint-wrapped) still passed `target` (which is the VERSION
|
||||||
|
env, None for !testme runs). So perform_upgrade got head_ref=None for upgrade tier → re-checkout
|
||||||
|
skipped → chaos deploy of whatever leftover checkout (prev tag from deploy_app) → vacuous prev→prev
|
||||||
|
chaos redeploy that "passed" via the chaos-label move fallback.
|
||||||
|
- Fixed: explicit Edit on the two multi-line calls so they now pass `head_ref` consistently
|
||||||
|
(`recipe`/`"upgrade"|"backup"|"restore"`, `repo_local`, `domain`, `meta`, `head_ref`, `op_state`).
|
||||||
|
grep confirms all 4 tier calls pass head_ref. compile OK.
|
||||||
|
- Net effect now: head_ref reaches perform_upgrade → recipe_checkout_ref(head_ref) restores PR-head
|
||||||
|
before chaos deploy → after.chaos == head_ref → assert_upgraded match succeeds non-vacuously.
|
||||||
|
|||||||
@@ -18,8 +18,11 @@ Three corrections, each Adversary cold-verified, no test weakened:
|
|||||||
|
|
||||||
## Definition of Done (Phase 1e) — HC1–HC4, each Adversary cold-verified in REVIEW-1e
|
## Definition of Done (Phase 1e) — HC1–HC4, each Adversary cold-verified in REVIEW-1e
|
||||||
- [ ] **HC1** — PR-head upgrade proven to deploy PR-head; deploy-count guard reconciled (==1).
|
- [ ] **HC1** — PR-head upgrade proven to deploy PR-head; deploy-count guard reconciled (==1).
|
||||||
- [ ] **HC2** — repo-local ignored for a non-approved recipe, run for an approved one.
|
- [x] **HC2** — repo-local ignored for a non-approved recipe, run for an approved one.
|
||||||
- [ ] **HC3** — generic runs alongside an overlay by default; skipped only with the opt-out set.
|
Adversary PASS @2026-05-28 (hostile-code probe, no finding; commit c7ae296).
|
||||||
|
- [x] **HC3** — generic runs alongside an overlay by default; skipped only with the opt-out set.
|
||||||
|
Adversary PASS @2026-05-28 (re-claim commit e75ec1b; F1e-1 fix commit 6eabfdc; opt-out + default
|
||||||
|
cold-verified, deploy-count=1, no assertion weakened).
|
||||||
- [ ] **HC4** — no regression cold-verified; deploy-once + teardown still sacred.
|
- [ ] **HC4** — no regression cold-verified; deploy-once + teardown still sacred.
|
||||||
|
|
||||||
## Milestones (plan §3)
|
## Milestones (plan §3)
|
||||||
|
|||||||
@@ -377,7 +377,7 @@ def main() -> int:
|
|||||||
if "upgrade" in stages:
|
if "upgrade" in stages:
|
||||||
results["upgrade"] = (
|
results["upgrade"] = (
|
||||||
run_lifecycle_tier(
|
run_lifecycle_tier(
|
||||||
recipe, "upgrade", repo_local, domain, meta, target, op_state
|
recipe, "upgrade", repo_local, domain, meta, head_ref, op_state
|
||||||
)
|
)
|
||||||
if prev
|
if prev
|
||||||
else "skip" # only one published version → nothing to upgrade from
|
else "skip" # only one published version → nothing to upgrade from
|
||||||
@@ -394,7 +394,7 @@ def main() -> int:
|
|||||||
if "restore" in stages:
|
if "restore" in stages:
|
||||||
results["restore"] = (
|
results["restore"] = (
|
||||||
run_lifecycle_tier(
|
run_lifecycle_tier(
|
||||||
recipe, "restore", repo_local, domain, meta, target, op_state
|
recipe, "restore", repo_local, domain, meta, head_ref, op_state
|
||||||
)
|
)
|
||||||
if backup_cap
|
if backup_cap
|
||||||
else "skip"
|
else "skip"
|
||||||
|
|||||||
Reference in New Issue
Block a user