75 lines
5.7 KiB
Markdown
75 lines
5.7 KiB
Markdown
# REVIEW-1e — Adversary verdicts (Phase 1e: generic-harness corrections)
|
||
|
||
Adversary-owned, append-only. Phase plan: `/srv/cc-ci/cc-ci-plan/plan-phase1e-harness-corrections.md`.
|
||
Definition of Done = HC1–HC4 each cold-verified PASS here (handshake per plan.md §6.1).
|
||
|
||
## Definition-of-Done tracker
|
||
- [ ] **HC1** — Upgrade tier upgrades to PR head (prev published → PR-head via `abra app deploy --chaos`), not a published tag; moved-assertion adapted; DG4.1 deploy-count guard reconciled.
|
||
- [x] **HC2** — Repo-local (PR-authored) `test_*.py` / `install_steps.sh` NOT executed unless recipe is on the cc-ci approval allowlist (default-deny). **PASS @2026-05-28 (E0, commit c7ae296).**
|
||
- [ ] **HC3** — Generic runs by default alongside an overlay (additive); skipped only via explicit opt-out; op runs once.
|
||
- [ ] **HC4** — No regression: D1–D10 / DG1–DG8 re-verified cold; deploy-once (DG4.1) holds; teardown sacred; three new behaviors demonstrated.
|
||
|
||
Maps to Builder milestones: E0=HC2, E1=HC3, E2=HC1, E3=HC4+docs.
|
||
|
||
## Cold-start access (re-verified each phase)
|
||
- @2026-05-28 — `ssh cc-ci` OK (NixOS 24.11), dashboard HTTP 200 via SOCKS proxy 127.0.0.1:1055. Proxy/SSH path healthy.
|
||
|
||
## Verdicts
|
||
|
||
### E0 / HC2 — repo-local trust gate (default-deny) — PASS @2026-05-28
|
||
Builder claim (STATUS-1e, commit c7ae296 / feat d38a695): repo-local (PR-authored)
|
||
`test_*.py`/`install_steps.sh`/`ops.py` consulted only for recipes on `tests/repo-local-approved.txt`
|
||
(empty ⇒ deny); centralized `_gated()` in `discovery.py`; 8 unit tests pass.
|
||
|
||
**Cold verification (own clone HEAD=c7ae296, shipped to cc-ci, run via `cc-ci-run`):**
|
||
1. **Unit suite, independent run:** `cd /tmp/adv-1e && cc-ci-run -m pytest tests/unit -v` →
|
||
**8 passed in 0.06s** (incl. repo-local-ignored-when-unapproved / wins-when-approved for
|
||
overlay+custom+install_steps+pre_op, and default-allowlist-is-empty).
|
||
2. **My own break-it probe** (`hc2_probe.py`, planted a HOSTILE repo-local `install_steps.sh`
|
||
`rm -rf /` + `ops.py` `os.system('id')` + `test_install.py`):
|
||
- real checked-in allowlist → `approved_recipes() == set()` (default-deny).
|
||
- `real-default` → `approved=False`, overlay falls back to **cc-ci**, `install_steps=None`,
|
||
`pre_op=None` (hostile repo-local code NOT selected).
|
||
- lone `*` → **DENY** (not a wildcard, as the file header promises).
|
||
- only-comment / whitespace lines → **DENY**.
|
||
- approving a *different* recipe (hedgedoc) → custom-html still **DENY** (no leak).
|
||
- `custom-html` listed → `approved=True`, overlay/install_steps/pre_op all flip to **repo-local**.
|
||
3. **No bypass:** every execution path in `runner/run_recipe_ci.py` routes through gated
|
||
`discovery.*` (`resolve_op`→`resolve_overlay_op`, `custom_tests`, `install_steps`→lifecycle hook).
|
||
`snapshot_recipe_tests` reads the repo-local dir ungated but only **copies** it (discover), never
|
||
executes — matches the plan's "discovered-but-NOT-executed". `pre_op_hook` not yet wired into the
|
||
orchestrator (E1/HC3 work); its discovery fn is already gated.
|
||
|
||
Verdict: **PASS** — default-secure, centralized gate, flips only on explicit per-recipe approval;
|
||
hostile repo-local code provably not executed under the shipped default. No finding.
|
||
**Note (not a defect):** orchestrator still uses single-file override `resolve_op` (1d semantics);
|
||
the additive generic floor (HC3) is E1 in-flight — will re-check the gate survives the HC3 refactor.
|
||
|
||
### E1 / HC3 — additive generic + op/assertion split — FAIL (PASS WITHHELD) @2026-05-28
|
||
Builder claim (STATUS-1e gate, commit b7e6cbd): generic runs additively alongside overlays;
|
||
orchestrator owns each op (once); opt-out via `CCCI_SKIP_GENERIC[_<OP>]`/`recipe_meta.SKIP_GENERIC`;
|
||
deploy-count stays 1; two e2e (default + opt-out) "clean."
|
||
|
||
**Cold verification (own clone HEAD=b7e6cbd shipped to cc-ci `/tmp/adv-1e`, run via `cc-ci-run`):**
|
||
- **Structure (PASS):** read the refactor — `run_lifecycle_tier` performs the op ONCE
|
||
(`_perform_op`→`generic.perform_{upgrade,backup,restore}`, none call `deploy_app`), then runs generic
|
||
(unless `_skip_generic`) + overlay as separate pytests vs the shared post-op state. Generic+overlay
|
||
test files are assertion-only; seeding moved to `ops.py pre_<op>`. `assert_upgraded` keeps the
|
||
non-vacuous move check (F1d-2). `_record_deploy()` lives only in `deploy_app`.
|
||
- **Default e2e** (custom-html, all stages): EVERY tier ran BOTH `assert (generic)` AND
|
||
`assert (cc-ci)`; pre_upgrade/backup/restore seeds fired; **deploy-count=1**; install/upgrade/backup/
|
||
restore all PASS; custom=skip; clean teardown (no leftover stack/volume). ✓ additive confirmed.
|
||
- **Opt-out e2e** (`CCCI_SKIP_GENERIC=1`): generic skipped on every tier (**0** `_generic/` files ran),
|
||
overlay-only, **deploy-count=1** ✓ — **but backup=FAIL**: `test_backup_captures_state` →
|
||
`AssertionError: '' == 'original'`. Same code/recipe; only diff is the opt-out flag.
|
||
|
||
**Verdict: FAIL — opt-out is not behavior-neutral.** Opting out of the generic removes an accidental
|
||
~1s timing buffer (the generic pytest spawn) and surfaces a real race: the backup/restore overlays
|
||
read the marker via `exec_in_app` immediately after a container-cycling op with no readiness/retry, and
|
||
`exec_in_app` silently returns empty stdout on a failed `docker exec` (returncode ignored). A healthy
|
||
recipe can thus be reported RED under opt-out. Filed **F1e-1 [adversary]** (BACKLOG-1e) with root cause
|
||
+ repro + fix direction (check exec returncode + bounded readiness retry; do NOT weaken the assertion).
|
||
Isolated (no-concurrency) reproduction in flight to rule out the parallel-Builder-run confound — which
|
||
would itself be a concurrency-collision finding. **HC3 PASS withheld until F1e-1 is fixed + re-verified
|
||
cold under opt-out.**
|