Compare commits
101 Commits
phase-lvl5
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 39e53d739e | |||
| 4b4d665ede | |||
| e1d623a361 | |||
| 44e02425ab | |||
| 87928a9096 | |||
| 8fba68e27c | |||
| 87566b1c95 | |||
| 574306ea9c | |||
| 720c6584b4 | |||
| 7b4081cb42 | |||
| cdd141841d | |||
| 1be74fb9e1 | |||
| 4f8943d10e | |||
| 3de5925614 | |||
| 7723cfef3d | |||
| 52866602e7 | |||
| 0aa46dbe72 | |||
| 75c46ac5c1 | |||
| b676d61df4 | |||
| 5384f5c13f | |||
| 7d18d6e561 | |||
| 32125c6e65 | |||
| 7e7e84df34 | |||
| d20bffd597 | |||
| eb58f9f053 | |||
| eec29614ae | |||
| 1adfbd70cb | |||
| 51c3280163 | |||
| 8ca5b44186 | |||
| f3c526d9e9 | |||
| 6607d7767f | |||
| be526c8252 | |||
| e37a7df496 | |||
| b17b6f1232 | |||
| 73ea239cfc | |||
| ec5882dd71 | |||
| 85a781368a | |||
| 560e772b5f | |||
| b9352e8313 | |||
| bb1ebd34f6 | |||
| 2fa3f528a6 | |||
| 1fbc4e0b15 | |||
| 36ece30442 | |||
| 4b5051f003 | |||
| ccabad8209 | |||
| 06e1cee47c | |||
| f96a639197 | |||
| 9afdf3de5a | |||
| 48a66b96a1 | |||
| 1d51a7907b | |||
| fe8922c2da | |||
| 8da59cff22 | |||
| 9eb5261c1e | |||
| f46aa05151 | |||
| 43826918ed | |||
| 17c8d29a8f | |||
| 71358da446 | |||
| 1e22f6ea79 | |||
| 7e783368c4 | |||
| fb411b2563 | |||
| 2da1f01849 | |||
| 53db62258e | |||
| e9c26c72af | |||
| a4c0dfcf11 | |||
| d0d762c9c8 | |||
| e9eed8e7b7 | |||
| 0cc31a507e | |||
| 9959ad6a2d | |||
| 866a429a6f | |||
| 9a097d3185 | |||
| 40c321f5f9 | |||
| f6058b9a00 | |||
| ef577c7d60 | |||
| 42eabbaa24 | |||
| 5b0e42adc2 | |||
| 369f4f486b | |||
| cba53b69a4 | |||
| f1500123e7 | |||
| cfda9e72db | |||
| 73889ed860 | |||
| 72b3d6c089 | |||
| e9745c8c74 | |||
| f88c6bc78d | |||
| 823023a19a | |||
| fc16250db2 | |||
| 8d5bf305e8 | |||
| 9ce987188a | |||
| 13cad1f985 | |||
| a521d43a17 | |||
| dc924c679b | |||
| 763f8d1a47 | |||
| 68c3486216 | |||
| 1fb70aafa6 | |||
| 29047a8dec | |||
| 08e6cc8273 | |||
| cfc87fd8d3 | |||
| 5ce813e910 | |||
| 40caaab8fb | |||
| 24baac559c | |||
| cd62743055 | |||
| 589943f46e |
@ -3,6 +3,14 @@
|
||||
Working notes for agents (and humans) modifying the cc-ci server. See `README.md` for what the server
|
||||
does and `machine-docs/` for the build's living state (`DECISIONS.md`, `DEFERRED.md`, `STATUS-*.md`).
|
||||
|
||||
## File-location rule (mandatory)
|
||||
|
||||
ALL coordination / loop-state files live under **`machine-docs/`**, NEVER the repo root. That means
|
||||
the phase-namespaced `STATUS-*.md`, `BACKLOG-*.md`, `REVIEW-*.md`, `JOURNAL-*.md`, the shared
|
||||
`DECISIONS.md` / `DEFERRED.md`, and the `ADVERSARY-INBOX.md` / `BUILDER-INBOX.md` side-channels.
|
||||
Create `machine-docs/` if missing; if you ever find one of these at the root, `git mv` it into
|
||||
`machine-docs/`. (The repo root is for actual server code/config — `runner/`, `tests/`, `nix/`, etc.)
|
||||
|
||||
## Testing cadence
|
||||
|
||||
Two kinds of tests live here — run them on **different** cadences:
|
||||
|
||||
@ -1,18 +0,0 @@
|
||||
# BACKLOG — Phase lvl5
|
||||
|
||||
## Build backlog
|
||||
|
||||
- [ ] B1 (P1) `level.py`: append rung `lint` (L5); new status vocabulary {pass, fail, skip, unver}; `compute_level()` → new formula (level = max i: rung_i pass ∧ ∀j<i status ∈ {pass,skip}); DELETE cap_reason/capped concepts.
|
||||
- [ ] B2 (P1) lint executor (`harness/lint.py`): `abra recipe lint <recipe>` against the exact tested ref; hard ~60s timeout; rc+full output → `lint.txt` artifact; pass/fail/unver classification (missing abra / timeout / exception → unver, never pass, never skip); mirror-context handling per phase-plan §2.3 (probe abra behavior first; any filtering = named + unit-tested + DECISIONS.md).
|
||||
- [ ] B3 (P1) `results.py`: wire lint into `derive_rungs` + explicit intentional-vs-unintentional classification of EVERY N/A source; drop level_cap_reason/level_cap_rung from schema; `skips()` reflects new statuses; orchestrator (`run_recipe_ci.py`) runs lint executor at the tested-ref point + passes result through; verdict-neutral (R7 wrap).
|
||||
- [ ] B4 (P1) unit tests: rewrite test_level.py/test_results.py to new semantics incl. mission worked examples (fail-blocks → L1; intentional-skip climbs → L5; unver-blocks → L2; lint unver → L4; unclassifiable N/A → unver default); lint executor tests; old-artifact rendering compat tests.
|
||||
- [ ] B5 (P2) `card.py`: 0–5 color ramp; cap line removed ("level N of 5" neutral); rung table renders ✔/✘/intentional-skip/unverified; level_badge_svg loses cap_skip third segment (badge = number+color only); tolerate old artifacts.
|
||||
- [ ] B6 (P2) `dashboard.py`: _LEVEL_COLOR 5-scale; _level_pill/badge SVG number-only; legend text; old results.json (cap_reason present, lint absent) render without KeyError.
|
||||
- [ ] B7 (P2) docs: results-ux.md, testing.md, recipe-customization.md §EXPECTED_NA wording — L5 ladder, de-cap semantics.
|
||||
- [ ] B8 (P1) DECISIONS.md: semantics change record (replaces Phase-3 "N/A caps"); N/A classification table (every derive_rungs N/A source → intentional|unintentional); mirror-filter decision for lint (if any filtering).
|
||||
- [ ] B9 — gate M1: claim (branch w/ P1+P2; clean tree; cold-verifiable).
|
||||
- [ ] B10 (P3) lint sweep over ALL enrolled recipes (scratch clones — never touch ~/.abra/recipes during builds); matrix here (pass/fail + rule hits); mechanical fixes → mirror PRs (never push main/never merge); rest → DEFERRED.md.
|
||||
- [ ] B11 (P4) real-CI proofs: ≥1 genuine L5; ≥1 lint-blocked L4 (synth branch ok); ≥1 N/A-skip climb; 2× drone !testme; canary suite at re-derived designed levels; 1 synthesized unver-blocks run; before/after level table for ALL enrolled recipes; card/dashboard PNG/SVG visually verified.
|
||||
- [ ] B12 — gate M2: claim; then ## DONE after fresh PASS.
|
||||
|
||||
## Adversary findings
|
||||
@ -1,19 +0,0 @@
|
||||
# JOURNAL — Phase lvl5
|
||||
|
||||
## 2026-06-11 bootstrap
|
||||
- Read plan-phase-lvl5-lint-rung.md in full + plan.md §6/§6.1/§7/§9. Phase files created.
|
||||
- Orientation reads: level.py (RUNGS 4, compute_level gap-caps, backup_restore_status, tier_to_rung), results.py derive_rungs/build_results (cap fields at :215-229), card.py (LEVEL_COLOR 0-6!, cap line :246, level_badge_svg cap_skip third segment), dashboard.py (_LEVEL_COLOR :68, _level_pill :245, cap div :277, render_level_badge :363), run_recipe_ci.py build_results call :1248 + badge wiring :1296-1320, bridge.py :224 (badge embed — number-only already, no cap text → likely untouched), docs (results-ux.md has cap language; recipe-customization.md EXPECTED_NA row).
|
||||
- Notable: card.py LEVEL_COLOR already has keys 0-6 (5=green, 6=bright green) — only 0-4 reachable today; dashboard._LEVEL_COLOR needs checking for the same.
|
||||
- Lint context: abra.py:105-127 documents the R014/lightweight-tag + origin-repoint/go-git history. Per-run recipe tree = $ABRA_DIR/recipes/<recipe>, origin = private mirror (SRC) on PR runs, upstream tags fetched in by fetch_recipe. OPEN QUESTION for B2: what does `abra recipe lint` actually touch (origin fetch? auth? R014 against which tags?) — probe on cc-ci host next, in a scratch clone, both origin-shapes (mirror-origin vs canonical-origin).
|
||||
- Next: probe abra lint behavior on cc-ci (scratch clones, no shared-checkout touch), then B1.
|
||||
|
||||
## 2026-06-11 abra lint probe (B2 design input) — all on cc-ci, scratch ABRA_DIR=/tmp/lvl5-lint-probe/abra
|
||||
- `abra recipe lint hedgedoc` (fresh canonical clone): FATA "inappropriate ioctl for device" rc=1 — needs a PTY even with `-n`. Under `script -qec "abra recipe lint -n hedgedoc" /dev/null`: rc=0, 21-line unicode table R001–R016 (cols: ref|rule|severity|satisfied ✅/❌|skipped|how-to-fix), maxlen 146 no wrapping, wall time 0.7s.
|
||||
- rc SEMANTICS: rc≠0 ONLY on FATA (cannot lint). Probes:
|
||||
- rm .env.sample + commit → rc=1 FATA "unable to validate recipe: .env.sample ... no such file" (content-attributable FATA).
|
||||
- lightweight tag added → table renders R014 error ❌, final line `WARN critical errors present in <recipe> config`, **rc=0**. So pass/fail MUST be parsed from the table (error-severity ❌ rows), sentinel line as cross-check. Baseline warn-only ❌ (R015) → NO sentinel, rc=0 → pass.
|
||||
- untracked compose.ccci.yml (CI overlay) in tree → FATA "version mismatched between two composefiles" rc=1 — abra lint globs compose*.yml INCLUDING untracked harness overlays ⇒ lint MUST run on a pristine clone of the exact ref, not the deploy tree.
|
||||
- origin repointed to auth-required mirror URL → rc=1 FATA "unable to fetch tags in ...: repository not found" — lint force-fetches tags from origin ⇒ scratch clone's origin must be fetchable without auth. Cloning FROM the per-run tree (local path origin) satisfies this offline and preserves the run's true tag set (fetch_recipe pulls upstream tags into the per-run tree).
|
||||
- run_quick emits no results.json/card (build_results only at run_recipe_ci.py:1248, cold path) → lint rung wiring is full-path only.
|
||||
- Executor design settled (DECISIONS.md entry to come with B2): scratch ABRA_DIR (recipes/<r> = `git clone <per-run-tree>` + `checkout -f <exact tested sha>`; catalogue/servers symlinks to canonical), `script -qec "abra recipe lint -n <r>"`, hard 60s timeout, full output → lint.txt artifact, parse table rows; status = fail iff any error-severity row ❌(not skipped) or content-attributable FATA ("unable to validate recipe"); pass iff table rendered & no error-row ❌; anything else (timeout, abra missing, fetch FATA, unparseable) → unver + loud log. No rule filtering needed (mirror pollution solved by context, not by ignoring rules).
|
||||
- Tier-skip sources mapped for derive_rungs classification (run_recipe_ci.py:1040-1131): upgrade skip ⟺ `prev` falsy ("only one published version", structural-intentional) given install passed; backup/restore skip ⟺ not backup_cap (structural-intentional); install-fail → downstream tiers skip (unintentional); custom skip ⟺ no custom tests (unintentional unless EXPECTED_NA declares functional); tier absent from `stages` (CCCI_STAGES dev escape) → missing key (unintentional).
|
||||
@ -22,7 +22,7 @@ secrets/ sops-encrypted infra secrets (cc-ci-secrets submodule)
|
||||
bridge/ !testme webhook listener source
|
||||
runner/ run_recipe_ci.py + shared pytest harness
|
||||
dashboard/ results overview generator
|
||||
tests/<recipe>/ per-recipe install/upgrade/backup tests + playwright/
|
||||
tests/<recipe>/ per-recipe install/upgrade/backup tests + custom/
|
||||
docs/ install, enroll-recipe, secrets, architecture, runbook, baseline
|
||||
```
|
||||
|
||||
|
||||
@ -1,6 +0,0 @@
|
||||
# STATUS — Phase lvl5 (L5 lint rung + de-cap)
|
||||
|
||||
Phase: lvl5 — OPEN (bootstrapped 2026-06-11)
|
||||
Gate: none claimed yet
|
||||
In flight: P1 — level.py new semantics + lint executor design (abra lint behavior probe on CI host first)
|
||||
Blockers: none
|
||||
@ -22,12 +22,11 @@ tests/<recipe>/
|
||||
├── test_backup.py # optional backup overlay (runs ADDITIVELY alongside generic)
|
||||
├── test_restore.py # optional restore overlay (runs ADDITIVELY alongside generic)
|
||||
├── PARITY.md # Phase 2 P2: mapping table (recipe-maintainer tests → cc-ci tests)
|
||||
├── functional/ # Phase 2 P3: parity ports + ≥2 NEW recipe-specific tests
|
||||
│ ├── test_health_check.py # parity port of recipe-info/<recipe>/tests/health_check.py
|
||||
│ ├── test_<behavior>.py # ≥2 NEW recipe-specific functional tests
|
||||
│ └── …
|
||||
└── playwright/ # Phase 2 P6: browser flows where the app's core UX is a UI
|
||||
└── test_<flow>.py
|
||||
└── custom/ # custom tier: parity ports + recipe-specific tests + browser flows
|
||||
├── test_health_check.py # parity port of recipe-info/<recipe>/tests/health_check.py
|
||||
├── test_<behavior>.py # ≥2 NEW recipe-specific tests
|
||||
├── test_<flow>.py # browser/UI flows where relevant
|
||||
└── …
|
||||
```
|
||||
|
||||
**A recipe is testable with ZERO config:** with no overlay files, the **generic lifecycle suite**
|
||||
@ -68,18 +67,18 @@ ops themselves are orchestrator-owned (you never call them from an overlay). The
|
||||
Beyond the lifecycle overlays, each recipe carries (plan §4.1):
|
||||
|
||||
- **`PARITY.md`** — a mapping table from every `references/recipe-maintainer/recipe-info/<recipe>/
|
||||
tests/*.py` to a comparable cc-ci test under `tests/<recipe>/functional/`, asserting the
|
||||
tests/*.py` to a comparable cc-ci test under `tests/<recipe>/custom/`, asserting the
|
||||
*same thing* (not a renamed file). A deliberate non-port is documented in `DECISIONS.md` with
|
||||
a technical reason — never a silent omission.
|
||||
- **`functional/`** — parity-port tests + **≥2 NEW recipe-specific functional tests** that
|
||||
exercise the app's characteristic behavior (per plan §4.3 — e.g. "create-an-object +
|
||||
read-it-back, and one more that touches a distinctive feature"). Each parity-port file carries
|
||||
a `SOURCE = "recipe-info/<recipe>/tests/<file>"` comment near the top so audit is in-file.
|
||||
- **`playwright/`** — browser flows where the recipe's core UX is a UI (P6).
|
||||
- **`custom/`** — parity-port tests + **≥2 NEW recipe-specific tests** that exercise the app's
|
||||
characteristic behavior (per plan §4.3 — e.g. "create-an-object + read-it-back, and one more
|
||||
that touches a distinctive feature"). Browser/UI flows live in the same folder too. Each
|
||||
parity-port file carries a `SOURCE = "recipe-info/<recipe>/tests/<file>"` comment near the top
|
||||
so audit is in-file.
|
||||
|
||||
The orchestrator's **custom** tier discovers `test_*.py` in `tests/<recipe>/{functional,playwright}/`
|
||||
ONLY (the placement rule, via `runner/harness/discovery.custom_tests` — a top-level `test_*.py`
|
||||
is a lifecycle overlay and nothing else) and runs each as its own pytest against the same
|
||||
The orchestrator's **custom** tier discovers `test_*.py` in canonical `tests/<recipe>/custom/`
|
||||
(plus deprecated `functional/` / `playwright/` aliases during migration; discovery warns when it
|
||||
uses them) and runs each as its own pytest against the same
|
||||
`live_app` shared deployment. Lifecycle-named files (`test_install.py`/etc.) are **excluded**
|
||||
from the custom tier even inside those subdirs (safety net against double-running).
|
||||
|
||||
@ -176,7 +175,7 @@ shapes (proven on mumble, mailu, and the SSO-dependent suite):
|
||||
|
||||
**Non-HTTP protocol tests (mumble).** Reach a TCP service published `mode: host` (via a host-ports
|
||||
overlay) at `127.0.0.1:<port>` — cc-ci runs tests on-host (cc-ci-run). mumble ships a stdlib protocol
|
||||
client (`tests/mumble/functional/_mumble_proto.py`) doing the real TLS handshake → ServerSync; the
|
||||
client (`tests/mumble/custom/_mumble_proto.py`) doing the real TLS handshake → ServerSync; the
|
||||
recipe-specific tests assert channel presence and config round-trips (a deploy-set `WELCOME_TEXT`/
|
||||
`USERS` value surfaces over the protocol — version-independent, non-vacuous).
|
||||
|
||||
@ -244,7 +243,7 @@ tests/lasuite-docs/
|
||||
├── test_backup.py # lifecycle backup overlay (marker captured)
|
||||
├── test_restore.py # lifecycle restore overlay (marker restored to pre-mutation)
|
||||
├── PARITY.md # parity-port mapping (P2)
|
||||
└── functional/
|
||||
└── custom/
|
||||
├── test_health_check.py # parity port (SOURCE comment cites recipe-info file)
|
||||
├── test_auth_required.py # specific: /api/v1.0/users/me/ → 401 without auth
|
||||
└── test_oidc_with_keycloak.py # specific: full OIDC flow against the dep keycloak (uses
|
||||
@ -256,8 +255,8 @@ tests/lasuite-docs/
|
||||
creds to `$CCCI_DEPS_FILE` — BEFORE the recipe deploy.
|
||||
2. Deploy lasuite-docs (`lasu-<6hex>.ci.commoninternet.net`); `install_steps.sh` wires the OIDC
|
||||
env into that one deploy.
|
||||
3. Run install / upgrade / backup / restore + the 3 functional tests against the shared
|
||||
deployment (custom tier).
|
||||
3. Run install / upgrade / backup / restore + the 3 custom tests against the shared
|
||||
deployment (custom tier).
|
||||
4. Teardown lasuite-docs, then the keycloak dep (LAST), both with verify=True.
|
||||
5. Print the run summary; non-zero exit code on any failure (DG4.1 deploy-count mismatch, tier
|
||||
FAIL, dep teardown leak — all surfaced).
|
||||
@ -268,10 +267,10 @@ tests/lasuite-docs/
|
||||
`COMPOSE_FILE=compose.yml:compose.mumbleweb.yml` for the base; `UPGRADE_EXTRA_ENV` adds the
|
||||
native `compose.host-ports.yml` at PR-head so 64738 is host-published on latest; private
|
||||
`_WELCOME_TEXT_MARKER`/`_MAX_USERS` constants; `READY_PROBE(ctx)` TCP 64738 — phase-aware via
|
||||
the live COMPOSE_FILE), `functional/_mumble_proto.py` + the protocol/config-round-trip
|
||||
the live COMPOSE_FILE), `custom/_mumble_proto.py` + the protocol/config-round-trip
|
||||
tests, `ops.py`/`test_backup.py`/`test_restore.py` (sqlite P4). See §2.4.
|
||||
- **Multi-service, dep-less, in-container functional — `tests/mailu/`**: `recipe_meta.py`
|
||||
(`EXTRA_ENV(ctx)` with `TLS_FLAVOR=notls` + `MAIL_DOMAIN`/`HOSTNAMES`/`TRAEFIK_STACK_NAME`),
|
||||
`functional/_mailu.py` (flask-CLI helpers), `test_mailbox.py` (create→config-export read-back),
|
||||
`custom/_mailu.py` (flask-CLI helpers), `test_mailbox.py` (create→config-export read-back),
|
||||
`test_mail_flow.py` (in-container sendmail→doveadm delivery). No backupbot → P4 N/A (PARITY.md +
|
||||
DEFERRED.md). See §2.4.
|
||||
|
||||
@ -22,7 +22,7 @@ A recipe customizes its CI through **three distinct mechanisms**:
|
||||
|---|---|---|
|
||||
| **Declarative settings** | Python assignments in `tests/<recipe>/recipe_meta.py` | `DEPLOY_TIMEOUT = 1500`, `UPGRADE_BASE_VERSION = "2.3.1+..."` |
|
||||
| **Code hooks** | Callables in `recipe_meta.py`, `ops.py` functions, one shell hook | `def READY_PROBE(ctx): ...`, `pre_upgrade(ctx)`, `install_steps.sh` |
|
||||
| **File presence** | A file existing at a discovered path changes behavior | `test_upgrade.py` overlay, `functional/test_*.py`, `compose.ccci.yml` |
|
||||
| **File presence** | A file existing at a discovered path changes behavior | `test_upgrade.py` overlay, `custom/test_*.py`, `compose.ccci.yml` |
|
||||
|
||||
There is additionally a fourth, **operator-facing, local-dev-only** surface: environment variables
|
||||
(`CCCI_SKIP_GENERIC*`) that suppress the generic floor at run time (§7). Whatever a run resolves
|
||||
@ -60,15 +60,15 @@ tests/<recipe>/ # cc-ci side (repo-local mirrors the same s
|
||||
├── recipe_meta.py # THE config file: registry-validated keys + ctx-hooks (§4)
|
||||
├── test_<op>.py # lifecycle overlay assertions, op ∈ install|upgrade|backup|restore (§5.1)
|
||||
├── ops.py # pre_<op>(ctx) seed hooks (§5.2)
|
||||
├── functional/test_*.py # custom tier: parity ports + recipe-specific (§5.3)
|
||||
├── playwright/test_*.py # custom tier: UI flows (§5.3)
|
||||
├── custom/test_*.py # custom tier: parity ports + recipe-specific + UI flows (§5.3)
|
||||
├── install_steps.sh # pre-deploy shell hook (the ONLY shell hook) (§5.4)
|
||||
├── compose.ccci.yml # CI-only compose overlay (first-class) (§5.5)
|
||||
└── PARITY.md # enrollment contract doc (human-read only)
|
||||
```
|
||||
|
||||
**Placement rule (custom tests):** ALL custom-tier tests live under `functional/` or
|
||||
`playwright/`. A top-level `test_*.py` is a lifecycle overlay (`test_<op>.py`) and nothing else —
|
||||
**Placement rule (custom tests):** ALL custom-tier tests live under canonical `custom/`.
|
||||
Deprecated `functional/` and `playwright/` aliases are still discovered with a loud warning so
|
||||
coverage is not silently lost while recipe trees migrate. A top-level `test_*.py` is a lifecycle overlay (`test_<op>.py`) and nothing else —
|
||||
top-level non-lifecycle files are NOT discovered (`discovery.custom_tests`; the lifecycle-name
|
||||
exclusion stays as a safety net so a misfiled `test_<op>.py` can never double-run).
|
||||
|
||||
@ -76,7 +76,8 @@ Precedence (machine-docs/DECISIONS.md, implemented in `discovery.py`):
|
||||
|
||||
- lifecycle overlay `test_<op>.py`: repo-local **wins** over cc-ci (same-name collision); the
|
||||
generic floor still runs additively alongside.
|
||||
- custom tier (`functional/` + `playwright/`): **ALL** run, from both locations (no collision
|
||||
- custom tier (`custom/`, plus deprecated alias dirs during migration): **ALL** run, from both
|
||||
locations (no collision
|
||||
concept).
|
||||
- `install_steps.sh`: repo-local > cc-ci, or none.
|
||||
- `ops.py` pre-op hook: cc-ci wins; repo-local consulted only if approved.
|
||||
@ -116,7 +117,7 @@ _This table is GENERATED from the `runner/harness/meta.py` KEYS registry by `scr
|
||||
| `DEPLOY_TIMEOUT` | `int` | `600` | Max seconds to wait for swarm convergence per deploy. |
|
||||
| `HTTP_TIMEOUT` | `int` | `300` | Max seconds to wait for HTTP health after convergence. |
|
||||
| `BACKUP_CAPABLE` | `bool` | `None` | Override the backup-tier capability auto-detect (compose `backupbot.backup` labels). `False` forces an intentional skip of the backup/restore rung; `True` forces the tier on; unset = auto-detect. |
|
||||
| `EXPECTED_NA` | `dict` | `None` | Declare a non-run rung an INTENTIONAL skip: `{rung: reason}` — the level climbs past it; an undeclared non-run rung is *unverified* and blocks the level above it (classification table: machine-docs/DECISIONS.md phase lvl5). Never overrides an exercised pass/fail; the `lint` rung has no escape hatch. |
|
||||
| `EXPECTED_NA` | `dict` | `None` | Declare a non-run rung an INTENTIONAL skip: `{rung: reason}` — the level climbs past it; an undeclared non-run rung is *unverified* and blocks the level above it (classification table: machine-docs/DECISIONS.md phase lvl5). Never overrides an exercised pass/fail; the `lint` rung has no escape hatch. Declaring `upgrade` also suppresses the upgrade-tier BASE deploy — the single deploy is the PR head itself — for recipes whose published versions exist but are genuinely undeployable (phase bsky). |
|
||||
| `READY_PROBE` | `hook` | `None` | Callable `(ctx) -> [probe, ...]` returning extra readiness probes, run after install AND after upgrade: HTTP `{host, path, ok}` or TCP `{tcp_host, tcp_port, stable}`. |
|
||||
| `UPGRADE_BASE_VERSION` | `str` | `None` | Exact published tag overriding the upgrade tier's base (default: `recipe_versions[-2]`). |
|
||||
| `BACKUP_VERIFY` | `hook` | `None` | Callable `(ctx) -> bool` post-backup data-capture check; `False` re-runs the backup (truncated-dump race guard), retried up to 3 attempts. |
|
||||
@ -181,15 +182,16 @@ def pre_restore(ctx): _psql(ctx.domain, "DROP TABLE ci_marker") # damage, rest
|
||||
Seed → op → assert is the whole pattern: `pre_backup` writes a marker, the orchestrator backs up,
|
||||
`pre_restore` destroys it, the orchestrator restores, `test_restore.py` asserts the marker is back.
|
||||
|
||||
### 5.3 Custom tier — `functional/` and `playwright/` ONLY
|
||||
### 5.3 Custom tier — canonical `custom/`
|
||||
|
||||
All custom-tier tests live under `tests/<recipe>/functional/` or `tests/<recipe>/playwright/`
|
||||
(discovery: `discovery.custom_tests`; the placement rule, §3). Run in the CUSTOM tier, after
|
||||
All custom-tier tests live under `tests/<recipe>/custom/` (discovery: `discovery.custom_tests`;
|
||||
the placement rule, §3). Deprecated `functional/` and `playwright/` dirs are still recognized
|
||||
with a warning during the migration window. Custom tests run in the CUSTOM tier, after
|
||||
restore, against the post-upgrade (PR-head) app. ALL discovered files run — cc-ci's and (if
|
||||
HC2-approved) repo-local's, additively.
|
||||
|
||||
Enrollment contract (`docs/enroll-recipe.md`): ≥2 NEW functional tests beyond ports of existing
|
||||
upstream checks; ported tests carry `SOURCE:` comments. Playwright tests get the shared
|
||||
Enrollment contract (`docs/enroll-recipe.md`): ≥2 NEW custom tests beyond ports of existing
|
||||
upstream checks; ported tests carry `SOURCE:` comments. Browser-driven custom tests get the shared
|
||||
browser/harness helpers (`harness.browser`); SSO recipes get `harness.sso`
|
||||
(`setup_keycloak_realm` — idempotent, `oidc_password_grant` — provider-pluggable). The documented
|
||||
import toolbox for custom tests is `from harness import lifecycle, sso, browser`.
|
||||
@ -268,7 +270,7 @@ deploy BASE (UPGRADE_BASE_VERSION or recipe_versions[-2]; EXTRA_ENV; install_ste
|
||||
→ BACKUP tier
|
||||
→ pre_restore(ctx) → restore
|
||||
→ RESTORE tier
|
||||
→ CUSTOM tier (functional/ + playwright/; deps via the `deps` fixture)
|
||||
→ CUSTOM tier (custom/; deps via the `deps` fixture)
|
||||
→ SCREENSHOT (best-effort, never affects the verdict)
|
||||
→ teardown (deps LAST)
|
||||
```
|
||||
@ -293,7 +295,7 @@ RECIPE=<recipe> PR=<n> REF=<sha> SRC=recipe-maintainers/<recipe> \
|
||||
meta (non-default): DEPLOY_TIMEOUT=1500 DEPS=['keycloak'] EXTRA_ENV='<hook>'
|
||||
hooks: ops.py[pre_backup,pre_upgrade](cc-ci) install_steps.sh(cc-ci) compose.ccci.yml(cc-ci)
|
||||
overlays: test_backup.py(cc-ci) test_restore.py(repo-local)
|
||||
custom tests: functional/=5 playwright/=2 (cc-ci)
|
||||
custom tests: custom/=7 (cc-ci)
|
||||
env overrides: (none)
|
||||
```
|
||||
|
||||
|
||||
@ -114,11 +114,12 @@ repo-local <recipe-repo>/tests/test_<op>.py (upstream-authoritative; gated
|
||||
Only ONE overlay source wins for a given op (repo-local > cc-ci); the generic floor runs **in
|
||||
addition** unless explicitly opted out.
|
||||
|
||||
**Custom (non-lifecycle) tests** — e.g. `functional/test_sso.py` — are **opt-in and additive**:
|
||||
**Custom (non-lifecycle) tests** — e.g. `custom/test_sso.py` — are **opt-in and additive**:
|
||||
they have no generic equivalent and run only when present, discovered from both locations
|
||||
(repo-local gated by the HC2 allowlist). Placement rule: custom tests live ONLY under
|
||||
`functional/` or `playwright/`; a top-level `test_*.py` is a lifecycle overlay and nothing else
|
||||
(top-level non-lifecycle files are not discovered).
|
||||
(repo-local gated by the HC2 allowlist). Placement rule: custom tests live under canonical
|
||||
`custom/`; deprecated `functional/` and `playwright/` aliases are still discovered with a loud
|
||||
warning so old recipe trees are not silently dropped. A top-level `test_*.py` is a lifecycle
|
||||
overlay and nothing else (top-level non-lifecycle files are not discovered).
|
||||
|
||||
### Pre-op seed hooks (per-recipe `ops.py`)
|
||||
|
||||
|
||||
18
machine-docs/BACKLOG-bsky.md
Normal file
18
machine-docs/BACKLOG-bsky.md
Normal file
@ -0,0 +1,18 @@
|
||||
# BACKLOG — phase bsky
|
||||
|
||||
## Build backlog
|
||||
|
||||
- [x] B1: Root-cause diagnosis — inspect recipe compose/entrypoint + actual `:0.4` image vs exact tags on cc-ci (2026-06-11)
|
||||
- [x] B2: Upstream research persisted to cc-ci-plan/upstream/bluesky-pds.md (plan repo f395247)
|
||||
- [x] B3: DECISIONS.md entry — pin choice (exact 0.4.219 over 0.5.1-main / digest pin), version label bump
|
||||
- [x] B4: Mirror PR branch `upgrade-0.3.0+v0.4.219` — compose.yml re-pin + label bump; open PR on recipe-maintainers/bluesky-pds
|
||||
- [x] B5: `!testme` on the PR → full lifecycle green (install/health, upgrade-path status justified, backup/restore, functional, L5 lint); record level under de-capped semantics + reconcile expected baseline
|
||||
- [x] B6: Screenshot on the green PR run — verify PNG real/representative/credential-free (Read it); SCREENSHOT hook only if needed
|
||||
- [x] B7: Claim M1 (root cause + green fix PR + screenshot verified)
|
||||
- [ ] B8: Close DEFERRED bluesky entries with pointers; JOURNAL note updating shot-phase N/A disposition
|
||||
- [ ] B9: Operator handoff summary in STATUS-bsky.md (what was wrong, what the PR changes, post-merge expectations incl. canonical/warm reseed)
|
||||
- [x] B10: Claim M2
|
||||
|
||||
## Adversary findings
|
||||
|
||||
(Adversary-owned)
|
||||
141
machine-docs/BACKLOG-cfold.md
Normal file
141
machine-docs/BACKLOG-cfold.md
Normal file
@ -0,0 +1,141 @@
|
||||
# BACKLOG — phase cfold
|
||||
|
||||
## Build backlog
|
||||
(Builder-only section — read-only to Adversary)
|
||||
|
||||
- [x] Seed `STATUS-cfold.md` + `JOURNAL-cfold.md`; consume Adversary inbox
|
||||
- [x] Record deprecated-folder policy in `DECISIONS.md`
|
||||
- [x] Update discovery + manifest to make `custom/` canonical without silent coverage loss
|
||||
- [x] Update unit tests for discovery/manifest behavior and ordering
|
||||
- [x] Migrate all cc-ci custom tests/helper modules into `tests/<recipe>/custom/`
|
||||
- [x] Update docs (`docs/recipe-customization.md`, `docs/testing.md`, `docs/enroll-recipe.md`)
|
||||
- [x] Produce M1 coverage-diff proof: discovered custom-test set identical before/after
|
||||
- [x] Claim M1 with WHAT/HOW/EXPECTED/WHERE in `STATUS-cfold.md`
|
||||
- [x] Await Adversary M1 verdict
|
||||
- [ ] Build the pre-sweep recipe baseline matrix for M2
|
||||
- [ ] Run the full real-CI `!testme` sweep and capture recipe-by-recipe evidence
|
||||
- [ ] Claim M2 only after the sweep is green and zero leaks are confirmed
|
||||
|
||||
## Adversary findings
|
||||
|
||||
No findings yet. Pre-migration baseline recorded below for reference during M1 verification.
|
||||
|
||||
### Baseline inventory (pre-migration, 2026-06-11T22:54Z)
|
||||
|
||||
**64 custom test files** across 20 recipes, all in `functional/` or `playwright/` subdirs:
|
||||
|
||||
| Recipe | functional/ | playwright/ | Helper modules |
|
||||
|---|---|---|---|
|
||||
| bluesky-pds | 4 | 0 | — |
|
||||
| cryptpad | 2 | 2 | — |
|
||||
| custom-html | 3 | 1 | — |
|
||||
| custom-html-tiny | 1 | 0 | — |
|
||||
| discourse | 3 | 0 | _discourse.py |
|
||||
| drone | 1 | 0 | __init__.py |
|
||||
| ghost | 4 | 0 | _ghost.py |
|
||||
| hedgedoc | 2 | 0 | — |
|
||||
| immich | 3 | 0 | — |
|
||||
| keycloak | 3 | 0 | — |
|
||||
| lasuite-docs | 5 | 0 | — |
|
||||
| lasuite-drive | 3 | 0 | — |
|
||||
| lasuite-meet | 3 | 0 | — |
|
||||
| mailu | 3 | 0 | _mailu.py |
|
||||
| matrix-synapse | 3 | 0 | — |
|
||||
| mattermost-lts | 3 | 0 | _mm.py |
|
||||
| mumble | 5 | 0 | _mumble_proto.py |
|
||||
| n8n | 4 | 0 | — |
|
||||
| plausible | 2 | 0 | — |
|
||||
| uptime-kuma | 3 | 1 | — |
|
||||
| **TOTAL** | **59** | **5** | **6 helper modules** |
|
||||
|
||||
Full file list (64 test files):
|
||||
```
|
||||
tests/bluesky-pds/functional/test_account_and_post.py
|
||||
tests/bluesky-pds/functional/test_describe_server.py
|
||||
tests/bluesky-pds/functional/test_health_check.py
|
||||
tests/bluesky-pds/functional/test_session_auth.py
|
||||
tests/cryptpad/functional/test_health_check.py
|
||||
tests/cryptpad/functional/test_spa_assets.py
|
||||
tests/cryptpad/playwright/test_pad_content_roundtrip.py
|
||||
tests/cryptpad/playwright/test_pad_create.py
|
||||
tests/custom-html/functional/test_content_roundtrip.py
|
||||
tests/custom-html/functional/test_content_type_header.py
|
||||
tests/custom-html/functional/test_health_check.py
|
||||
tests/custom-html/playwright/test_browser_smoke.py
|
||||
tests/custom-html-tiny/functional/test_serves_content.py
|
||||
tests/discourse/functional/test_create_topic.py
|
||||
tests/discourse/functional/test_health_check.py
|
||||
tests/discourse/functional/test_site_basic.py
|
||||
tests/drone/functional/test_scm_configured.py
|
||||
tests/ghost/functional/test_admin_redirect.py
|
||||
tests/ghost/functional/test_content_api.py
|
||||
tests/ghost/functional/test_health_check.py
|
||||
tests/ghost/functional/test_post_roundtrip.py
|
||||
tests/hedgedoc/functional/test_branding.py
|
||||
tests/hedgedoc/functional/test_health_check.py
|
||||
tests/immich/functional/test_asset_processing.py
|
||||
tests/immich/functional/test_asset_upload.py
|
||||
tests/immich/functional/test_health_check.py
|
||||
tests/keycloak/functional/test_create_client_and_use.py
|
||||
tests/keycloak/functional/test_health_check.py
|
||||
tests/keycloak/functional/test_password_grant_token.py
|
||||
tests/lasuite-docs/functional/test_auth_required.py
|
||||
tests/lasuite-docs/functional/test_create_doc.py
|
||||
tests/lasuite-docs/functional/test_health_check.py
|
||||
tests/lasuite-docs/functional/test_oidc_login.py
|
||||
tests/lasuite-docs/functional/test_oidc_with_keycloak.py
|
||||
tests/lasuite-drive/functional/test_health_check.py
|
||||
tests/lasuite-drive/functional/test_minio_storage.py
|
||||
tests/lasuite-drive/functional/test_oidc_with_keycloak.py
|
||||
tests/lasuite-meet/functional/test_health_check.py
|
||||
tests/lasuite-meet/functional/test_meeting_flow.py
|
||||
tests/lasuite-meet/functional/test_oidc_with_keycloak.py
|
||||
tests/mailu/functional/test_health_check.py
|
||||
tests/mailu/functional/test_mailbox.py
|
||||
tests/mailu/functional/test_mail_flow.py
|
||||
tests/matrix-synapse/functional/test_federation_version.py
|
||||
tests/matrix-synapse/functional/test_health_check.py
|
||||
tests/matrix-synapse/functional/test_register_and_message.py
|
||||
tests/mattermost-lts/functional/test_create_message.py
|
||||
tests/mattermost-lts/functional/test_health_check.py
|
||||
tests/mattermost-lts/functional/test_multiuser_message.py
|
||||
tests/mumble/functional/test_protocol_handshake.py
|
||||
tests/mumble/functional/test_server_config_limits.py
|
||||
tests/mumble/functional/test_tcp_health.py
|
||||
tests/mumble/functional/test_web_client.py
|
||||
tests/mumble/functional/test_welcome_text_roundtrip.py
|
||||
tests/n8n/functional/test_health_check.py
|
||||
tests/n8n/functional/test_login_state.py
|
||||
tests/n8n/functional/test_rest_settings.py
|
||||
tests/n8n/functional/test_workflow_roundtrip.py
|
||||
tests/plausible/functional/test_health_check.py
|
||||
tests/plausible/functional/test_event_tracking.py
|
||||
tests/uptime-kuma/functional/test_health_check.py
|
||||
tests/uptime-kuma/functional/test_socketio_handshake.py
|
||||
tests/uptime-kuma/functional/test_spa_branding.py
|
||||
tests/uptime-kuma/playwright/test_monitor_wizard.py
|
||||
```
|
||||
|
||||
Helper modules also in functional/ dirs (must move to custom/ alongside tests):
|
||||
- tests/discourse/functional/_discourse.py
|
||||
- tests/drone/functional/__init__.py
|
||||
- tests/ghost/functional/_ghost.py
|
||||
- tests/mailu/functional/_mailu.py
|
||||
- tests/mattermost-lts/functional/_mm.py
|
||||
- tests/mumble/functional/_mumble_proto.py
|
||||
|
||||
**String literal audit** — all places that name the FOLDER (not the playwright package):
|
||||
- runner/harness/discovery.py:113 — `subdirs = ("functional", "playwright")`
|
||||
- runner/harness/manifest.py:55 — comment `# functional | playwright`
|
||||
- docs/recipe-customization.md — multiple §5.3 references
|
||||
- docs/enroll-recipe.md — multiple references
|
||||
- docs/testing.md:117,120 — placement rule
|
||||
- tests/unit/test_discovery_phase2.py — creates functional/ and playwright/ dirs
|
||||
- tests/unit/test_manifest.py — creates functional/ and playwright/ dirs; asserts `{"functional": 2, "playwright": 1}`
|
||||
- tests/unit/test_discovery.py:83,84 — creates functional/ dirs
|
||||
|
||||
NOT to touch (playwright package references, not folder):
|
||||
- runner/harness/browser.py (playwright package import)
|
||||
- runner/harness/screenshot.py (playwright package import)
|
||||
- runner/harness/card.py:232 (playwright package import)
|
||||
- level.py, results.py (rung name "functional" — NOT a folder name)
|
||||
222
machine-docs/BACKLOG-drone.md
Normal file
222
machine-docs/BACKLOG-drone.md
Normal file
@ -0,0 +1,222 @@
|
||||
# BACKLOG — phase drone (drone enrollment with gitea SCM dep)
|
||||
|
||||
**Phase plan:** `/srv/cc-ci/cc-ci-plan/plan-phase-drone-enroll.md`
|
||||
|
||||
---
|
||||
|
||||
## Build backlog
|
||||
|
||||
_(Builder's section — Adversary read-only)_
|
||||
|
||||
### M1 tasks
|
||||
|
||||
- [x] Read plan + Adversary pre-probes
|
||||
- [x] Create phase state files (STATUS/JOURNAL/BACKLOG/REVIEW init)
|
||||
- [x] Implement `setup_gitea_oauth()` in `runner/harness/sso.py`
|
||||
- [x] Extend `_enrich_deps_with_sso` in `runner/run_recipe_ci.py` for gitea
|
||||
- [x] Create `tests/gitea/recipe_meta.py`
|
||||
- [x] Create `tests/drone/recipe_meta.py`
|
||||
- [x] Create `tests/drone/install_steps.sh`
|
||||
- [x] Create `tests/drone/functional/test_scm_configured.py` (ADV-drone-01 fixed in 7e7e84d)
|
||||
- [x] Create `tests/drone/PARITY.md`
|
||||
- [x] Write unit tests for new harness surface (10/10 pass)
|
||||
- [x] Harness run 5 GREEN — deploy-count 2/2 (DG4.1 PASS), level=5, install+upgrade+custom PASS
|
||||
- [x] Claim M1 — Adversary PASS @2026-06-11T22:22Z (commit `3de5925`)
|
||||
|
||||
### M2 tasks (after M1 PASS)
|
||||
|
||||
- [x] Mirror drone + gitea on git.autonomic.zone (for !testme CI path)
|
||||
- [x] Open !testme PR for drone recipe — PR #1 `testme-1.9.0-cc-ci` @ recipe-maintainers/drone
|
||||
- [x] CI run via !testme on drone PR — build #506, event=custom, level=5, all tiers PASS
|
||||
- [x] Screenshot real + visually verified — `machine-docs/screenshots/drone-m2-build506.png`
|
||||
- [x] Level recorded — level=5
|
||||
- [x] DEFERRED updated — Adversary §7.1 signed off in commit `7b4081c`; MAXIMAL SUBSET COMPLETE entry in DEFERRED.md
|
||||
- [x] Operator summary written — see STATUS-drone.md ## DONE
|
||||
- [x] Claim M2 — Adversary M2 PASS @2026-06-11T22:30Z (commit `7b4081c`). Phase drone DONE.
|
||||
|
||||
---
|
||||
|
||||
## Adversary findings
|
||||
|
||||
### ADV-drone-01 [adversary] test_scm_configured follows all redirects — assertion always fails
|
||||
|
||||
**Filed:** 2026-06-11T21:37Z
|
||||
**Severity:** CRITICAL — SCM-configured test is always failing, even for a correctly wired drone
|
||||
|
||||
**Defect:** `tests/drone/functional/test_scm_configured.py::test_login_redirects_to_gitea_dep`
|
||||
uses `urllib.request.urlopen(req, context=ctx)` which follows ALL redirect hops. The redirect
|
||||
chain for a correctly-wired drone is:
|
||||
|
||||
1. `GET /login` → 303 → `https://<gitea-dep>/login/oauth/authorize?client_id=...&...`
|
||||
2. Gitea (unauthenticated user) → 302 → `https://<gitea-dep>/user/login?redirect_to=...`
|
||||
3. Final: `https://<gitea-dep>/user/login` (200 OK)
|
||||
|
||||
The test asserts `parsed.path == "/login/oauth/authorize"` but `final_url` is `/user/login`.
|
||||
**The assertion ALWAYS fails even when drone is correctly wired.**
|
||||
|
||||
**Verified:** reproduced against the live drone.ci.commoninternet.net:
|
||||
```
|
||||
python3 -c "
|
||||
import ssl, urllib.request, urllib.parse
|
||||
ctx = ssl.create_default_context(); ctx.check_hostname = False; ctx.verify_mode = ssl.CERT_NONE
|
||||
req = urllib.request.Request('https://drone.ci.commoninternet.net/login', method='GET')
|
||||
with urllib.request.urlopen(req, timeout=30, context=ctx) as resp:
|
||||
print(resp.geturl())
|
||||
# → https://git.autonomic.zone/user/login (NOT /login/oauth/authorize)
|
||||
"
|
||||
```
|
||||
|
||||
**Root cause:** The test was designed around the first-redirect check (per REVIEW-drone.md
|
||||
pre-probe) but implemented as a follow-all check. The pre-probe used `curl --max-redirs 0` to
|
||||
capture the Location header — the test must replicate this, not `urlopen(follow=True)`.
|
||||
|
||||
**Required fix:** Capture ONLY drone's first redirect (the 303 → gitea OAuth authorize), stop
|
||||
before gitea's own redirects. One correct pattern:
|
||||
|
||||
```python
|
||||
class _CaptureOneRedirect(urllib.request.HTTPRedirectHandler):
|
||||
def http_error_302(self, req, fp, code, msg, headers):
|
||||
raise urllib.error.HTTPError(req.full_url, code, msg, headers, fp)
|
||||
http_error_303 = http_error_302
|
||||
|
||||
opener = urllib.request.build_opener(
|
||||
_CaptureOneRedirect(),
|
||||
urllib.request.HTTPSHandler(context=ctx),
|
||||
)
|
||||
try:
|
||||
opener.open(f"https://{live_app}/login", timeout=30)
|
||||
pytest.fail("Expected redirect from /login but got 200")
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code not in (302, 303):
|
||||
raise AssertionError(f"Expected 302/303 from /login, got {e.code}")
|
||||
redirect_url = e.headers.get("Location") or e.headers.get("location", "")
|
||||
|
||||
parsed = urllib.parse.urlparse(redirect_url)
|
||||
# now check parsed.netloc == gitea_domain and parsed.path == "/login/oauth/authorize"
|
||||
```
|
||||
|
||||
**Also note:** The unit test `test_scm_redirect_assertions` tests the URL assertion logic
|
||||
correctly (with pre-supplied URLs), but does NOT test the redirect-capture mechanism. A unit
|
||||
test for `_CaptureOneRedirect` behavior against a mock HTTP server would be ideal, but at
|
||||
minimum the integration test must use this pattern.
|
||||
|
||||
**Repro steps:**
|
||||
1. Deploy a correctly-wired drone (with gitea dep, compose.gitea.yml, DRONE_GITEA_CLIENT_ID set)
|
||||
2. Run `test_login_redirects_to_gitea_dep`
|
||||
3. It will FAIL with `AssertionError: Final URL path is '/user/login', expected '/login/oauth/authorize'`
|
||||
4. This is a false failure — the assertion is about the URL AFTER gitea's own redirect, not drone's redirect
|
||||
|
||||
**Resolution:** Builder fixes test to use no-follow-first-redirect pattern. Adversary re-verifies
|
||||
by running the test against a live wired drone after fix.
|
||||
|
||||
- [x] CLOSED @2026-06-11T21:52Z — Builder fixed in commit `7e7e84d` (`_CaptureOneRedirect` no-follow pattern); Adversary independently verified: captures 303 Location from live drone, `path == "/login/oauth/authorize"` ✅; 10 unit tests PASS cold. [Note: Builder ticked this — Adversary owns Adversary findings per §6.1; recording explicit Adversary close here.]
|
||||
|
||||
---
|
||||
|
||||
### ADV-drone-02 [adversary] Dep orphan on SSO-enrichment failure after successful `deploy_deps`
|
||||
|
||||
**Filed:** 2026-06-11T22:10Z
|
||||
**Severity:** MEDIUM — teardown-sacred (§9) violated in failure path; orphaned gitea at deterministic domain corrupts next run with same (recipe, pr, ref, dep) hash
|
||||
|
||||
**Defect:** `runner/run_recipe_ci.py::main()` initialises `deps_state = {}` (line 1015). Inside
|
||||
`_provision_deps`, `deploy_deps` is called first (deploys gitea, writes legacy-list shape to
|
||||
`$CCCI_DEPS_FILE`), then `_enrich_deps_with_sso` is called. If `_enrich_deps_with_sso` raises
|
||||
(e.g. `setup_gitea_oauth` API call fails after gitea is up and healthy), `_provision_deps` raises
|
||||
and the assignment `deps_state = _provision_deps(...)` (line 1034) never completes. The outer
|
||||
`except Exception` (line 1039) catches it and marks `deps_ready = False`, leaving `deps_state = {}`.
|
||||
|
||||
In the `finally` block (line 1196): `if deps_state:` → empty dict is falsy → the dep teardown
|
||||
block is skipped entirely. **The gitea container and its volumes are orphaned.**
|
||||
|
||||
**Failure path:**
|
||||
```
|
||||
deploy_deps(...) # gitea deployed + healthy; writes [{recipe:gitea, domain:gite-...}] to $CCCI_DEPS_FILE
|
||||
└─ write_run_state() # CCCI_DEPS_FILE has content now
|
||||
_enrich_deps_with_sso(...)
|
||||
└─ setup_gitea_oauth() # RAISES (API failure, gitea not ready yet, etc.)
|
||||
_provision_deps() raises
|
||||
deps_state = {} # assignment never completed
|
||||
...
|
||||
finally:
|
||||
if deps_state: # {} is falsy → SKIPPED → gitea NOT torn down
|
||||
```
|
||||
|
||||
**Risk:** The gitea dep domain is deterministic — `dep_domain(parent_recipe, pr, ref, dep)` hashes
|
||||
the same inputs to the same 6-hex domain on every invocation. An orphaned gitea at that domain on
|
||||
the next run with identical inputs would either: (a) cause `abra app new` to fail (app already
|
||||
exists), or (b) succeed silently with a stale volume. `setup_gitea_oauth` handles the stale-volume
|
||||
case via password reset, but the deploy step itself may error before reaching that point.
|
||||
|
||||
**Note:** `deploy_deps` (deps.py:104-109) tears down a dep immediately if its readiness check
|
||||
fails. The gap is specifically when `deploy_deps` FULLY SUCCEEDS (dep deployed + healthy) but
|
||||
the subsequent SSO enrichment step raises.
|
||||
|
||||
**Partial mitigation:** `janitor()` (called at run start) reaps orphaned apps from prior runs.
|
||||
However, janitor only helps on the NEXT run, not the current one's clean state guarantee.
|
||||
|
||||
**Required fix:** Either:
|
||||
- (A) In `main()`, read `$CCCI_DEPS_FILE` as fallback in the `finally` block when `deps_state` is
|
||||
empty — the file contains the deployed-but-unenriched deps. Tear those down via `teardown_deps`.
|
||||
- (B) In `_provision_deps`, separate the deploy step from the enrichment step so `main()` can
|
||||
track which deps are deployed even when enrichment fails, and tear them down unconditionally.
|
||||
- (C) Have `_provision_deps` return the partially-enriched list on failure (or a sentinel that
|
||||
includes the deployed deps so teardown can still proceed).
|
||||
|
||||
- [x] CLOSED @2026-06-11T22:22Z — Builder fixed in commit `0aa46db` (Option A: else-branch fallback in main() finally block reads $CCCI_DEPS_FILE via load_run_state() and calls teardown_deps on cold entries). Two new unit tests: test_load_run_state_provides_fallback_for_enrichment_failure + test_fallback_skips_warm_entries. 19/19 PASS. Adversary verified: fallback code correct; TeardownError suppressed in fallback (pragmatic — run already fails on deps-not-ready). Teardown-sacred §9 satisfied. CLOSED.
|
||||
|
||||
---
|
||||
|
||||
### ADV-drone-03 [adversary] DG4.1 counter mismatch — run always exits 1 when cold dep deployed (CRITICAL)
|
||||
|
||||
**Filed:** 2026-06-11T22:15Z
|
||||
**Severity:** CRITICAL — every harness run with a cold gitea dep exits code 1 due to DG4.1
|
||||
violation, even when all tiers pass and level=5 is achieved.
|
||||
|
||||
**Observed in Builder's run 4 (PID 2105952, /tmp/drone-m1-run4.log):**
|
||||
```
|
||||
!! deploy-count 1 != 2 (DG4.1 violation)
|
||||
deploy-count = 1 (expect 2)
|
||||
deps deployed: ['gitea']
|
||||
results.json written: /var/lib/cc-ci-runs/manual/results.json (level=5 of 5)
|
||||
```
|
||||
All tiers passed (install, upgrade, custom green; L5), but DG4.1 sets `overall = 1` → exit code 1 → CI FAIL.
|
||||
|
||||
**Root cause:** Internal contradiction between two parts of `deps.py`:
|
||||
|
||||
1. **Module docstring (line 19-20):** `"Dep deploys DO count toward the DG4.1 deploy-count
|
||||
invariant. The formula in run_recipe_ci.py is expected_deploy_count = 1 + deps_deployed_count,
|
||||
so each dep deploy increments the counter."`
|
||||
|
||||
2. **`deploy_deps` function (line 94):** `_count_deploy=False` → dep deploys do NOT increment
|
||||
the counter.
|
||||
|
||||
The formula in `run_recipe_ci.py` (line 1252) uses `expected = 1 + deps_deployed_count = 2`.
|
||||
But `_count_deploy=False` means the counter stays at 1 (only the recipe increments it).
|
||||
Result: `actual=1 != expected=2` → DG4.1 fires.
|
||||
|
||||
**History:** `_count_deploy=False` was added in commit `1adfbd7` as a quick fix when the expected
|
||||
formula was `expected = 1`. Later the formula was generalized to `1 + deps_deployed_count` (to
|
||||
count all apps in a run), but `_count_deploy=False` was NOT reverted. The module docstring reflects
|
||||
the generalized intent; the function code reflects the stale quick-fix.
|
||||
|
||||
**Required fix:** In `deps.py:deploy_deps` (line 94), remove or revert `_count_deploy=False`:
|
||||
```python
|
||||
# Before (wrong):
|
||||
lifecycle.deploy_app(dep, domain, ..., _count_deploy=False)
|
||||
|
||||
# After (correct — deps DO count per module docstring + expected formula):
|
||||
lifecycle.deploy_app(dep, domain, ...) # _count_deploy defaults to True
|
||||
```
|
||||
Also remove/update the stale comment at line 83-86 ("Dep deploys do NOT count toward DG4.1...").
|
||||
|
||||
**Also fix:** The comment in `deploy_deps` at lines 83-86:
|
||||
```python
|
||||
# Dep deploys do NOT count toward the DG4.1 "one deploy per run" invariant — that
|
||||
# contract covers the recipe-under-test only; each dep is a supporting service, not the
|
||||
# subject of the test. Pass _count_deploy=False so the main recipe's single-deploy
|
||||
# assertion isn't distorted by the number of deps declared.
|
||||
```
|
||||
This is now wrong. Replace with: "Dep deploys DO count toward DG4.1 (see module docstring);
|
||||
`expected_deploy_count = 1 + n_cold_deps`."
|
||||
|
||||
- [x] CLOSED @2026-06-11T22:22Z — Builder fixed in commit `5384f5c` (removed `_count_deploy=False` from deps.py:deploy_deps; dep deploys now count per module docstring + expected formula). Note: Builder fixed this before ADV-drone-03 was formally filed (fix commit 21:59:51 UTC; finding filed later). Run 5 confirms: deploy-count = 2 (expect 2) → no DG4.1 violation. CLOSED.
|
||||
73
machine-docs/BACKLOG-dstamp.md
Normal file
73
machine-docs/BACKLOG-dstamp.md
Normal file
@ -0,0 +1,73 @@
|
||||
# BACKLOG — phase `dstamp`
|
||||
|
||||
## Build backlog (Builder-owned)
|
||||
|
||||
- [x] Read phase plan + plan.md §6.1/§7/§9 + Adversary prep notes + stamp-relevant harness code.
|
||||
- [x] Establish abra's chaos-version mechanism from abra source @06a57de (= pinned binary).
|
||||
- [x] Rule out abra-version drift (constant store path since nixos system-4, 2026-06-01).
|
||||
- [x] Minimal reproductions of the git/abra chaos-version path (cp-a; go-git base; mirror-faithful)
|
||||
— all stamp the CORRECT head 7ae7b0f7, NO drift in current host state.
|
||||
- [x] Timeline: run 184 (06-05, solo) green @7ae7b0f; clustered 06-10/06-11 runs drift @ same ref.
|
||||
- [x] Identify shared-stack collision vector (`app_domain` = hash(recipe|pr|ref); upgrade
|
||||
chaos_redeploy bypasses app-domain flock).
|
||||
- [x] Isolated real runs (repro1–4) + direct UpdateStatus/PreviousSpec capture → root cause attributed.
|
||||
- [x] Concurrency REFUTED (solo repro1/4 reproduce). Mechanism = swarm `failure_action:rollback`
|
||||
reverts the chaos-version label (direct evidence repro4: Spec=7ae7b0f7+U→PreviousSpec=eb96de9+U).
|
||||
- [x] 06-05→06-10 change = rcust-phase heavier resident host load → start-first new task reliably OOMs → rollback every run (solo 06-05 run 184 didn't; my repro2 didn't either).
|
||||
- [x] Blast-radius: only discourse affected (keycloak/n8n have the policy but upgrade PASS L4 across runs; drone/traefik infra). General harness guard covers all.
|
||||
- [x] Restore discourse to its true level in real CI via the drone `!testme` path (M2): build #450 = LEVEL 5, all tiers PASS (install/upgrade/backup/restore/custom), clean teardown, no leak; PR#2 ✅ passed. fix1+fix2+450 = 3 consecutive green with the fix.
|
||||
- [~] HC1 teeth: code unchanged (generic.py:174-175) + assert_upgrade_converged RED on rollback (repro1/4). Live negative test = Adversary's M2 verification.
|
||||
- [x] Closed the DEFERRED.md dstamp re-entry with pointers (✅ RESOLVED).
|
||||
|
||||
## Adversary findings
|
||||
<!-- Adversary-owned. Do not edit above this line in this section. -->
|
||||
|
||||
**Root cause independently confirmed @2026-06-11T17:3x (JOURNAL not read, anti-anchoring preserved):**
|
||||
|
||||
Docker Swarm `failure_action: rollback` + `order: start-first` in discourse's `compose.yml` app
|
||||
service (BOTH `eb96de94` base AND `7ae7b0f` PR-head). On the upgrade chaos redeploy, `start-first`
|
||||
runs OLD + NEW tasks co-resident (~2× memory); the heavy Rails/precompile app fails swarm's 5s
|
||||
update monitor under host memory pressure → rollback fires → app service spec reverts to
|
||||
PreviousSpec (`chaos-version=eb96de94+U`). Because `start-first` kept the OLD task serving,
|
||||
`wait_healthy` passed; `deployed_identity` read the rolled-back spec; HC1 misreported it as
|
||||
"stamp mismatch" (the real failure was "new task failed the update monitor").
|
||||
|
||||
`services_converged` blind spot: `"rollback_completed"` not in blocking states → returned True.
|
||||
|
||||
Evidence: `docker service inspect disc-ae10f0_..._app` confirmed `UpdateConfig: {On failure:
|
||||
rollback, Order: start-first, Monitoring Period: 5s}`. repro1 (isolated, no concurrency) ALSO
|
||||
showed drift → pure-concurrency hypothesis REFUTED independently before reading Builder evidence.
|
||||
|
||||
abra exonerated: abra reads `git HEAD = 7ae7b0f` and stamps `7ae7b0f7+U` CORRECTLY. Three
|
||||
bail-at-secrets repros + repro2 debug line confirm. The `+U` comes from `compose.ccci.yml` as
|
||||
untracked file in per-run recipe dir (rcust-era overlay absent from run 184's pre-rcust path).
|
||||
|
||||
Fix 0cc31a5 assessed CORRECT: overlay sets `order: stop-first` (eliminates OOM 2×-memory
|
||||
trigger); `lifecycle.assert_upgrade_converged` closes the wait_healthy blind spot by catching
|
||||
`"rollback_completed"|"rollback_paused"|"paused"` and failing HONESTLY. HC1 unchanged.
|
||||
Minor race window in `assert_upgrade_converged` (first poll could see "none" before Docker
|
||||
starts the roll) is covered: with stop-first, a post-race rollback also fails `wait_healthy`.
|
||||
No blocker. Formal verdict awaits Builder's `claim(dstamp)` commit.
|
||||
|
||||
**Blast-radius sweep @2026-06-11T17:4x:**
|
||||
|
||||
All 24 enrolled recipes swept for `failure_action: rollback` + `order: start-first` in `compose.yml`:
|
||||
|
||||
| Recipe | failure_action | order | ccci overlay | upgrade tests | recent upgrade | risk |
|
||||
|-----------|---------------|-------------|--------------|---------------|----------------|------|
|
||||
| discourse | rollback | start-first | YES (fixed) | yes | FIXED | fixed |
|
||||
| drone | rollback | start-first | no | NO tests | n/a | latent, no CI exposure |
|
||||
| keycloak | rollback | start-first | no | yes | PASS L4 | latent, low (JVM, lighter than Rails) |
|
||||
| n8n | rollback | start-first | no | yes | PASS L4 | latent, low (Node.js) |
|
||||
| traefik | rollback | STOP-first | no | no | n/a | SAFE |
|
||||
| all others | none or absent | — | — | — | — | not at risk |
|
||||
|
||||
`assert_upgrade_converged` (added in 0cc31a5) provides a general harness backstop: if any
|
||||
recipe's rolling update rolls back or pauses, the upgrade is failed HONESTLY for all recipes
|
||||
— not just discourse. So keycloak/n8n are already covered by the harness fix even without
|
||||
overlay changes.
|
||||
|
||||
Recommended overlay addition for keycloak if/when OOM symptoms appear:
|
||||
`deploy.update_config.order: stop-first` (same pattern as discourse). Not urgent — current
|
||||
host load shows no rollback symptom for keycloak/n8n and they're lighter apps than discourse.
|
||||
drone has no upgrade tier in cc-ci; no action needed there.
|
||||
28
machine-docs/BACKLOG-kuma.md
Normal file
28
machine-docs/BACKLOG-kuma.md
Normal file
@ -0,0 +1,28 @@
|
||||
# BACKLOG — phase `kuma` (uptime-kuma create-a-monitor functional test)
|
||||
|
||||
## Build backlog
|
||||
|
||||
### DONE
|
||||
- [x] Phase state files created (STATUS-kuma.md, BACKLOG-kuma.md, REVIEW-kuma.md, JOURNAL-kuma.md)
|
||||
- [x] Approach decision: Playwright over python-socketio (recorded in DECISIONS.md)
|
||||
- [x] Inspect uptime-kuma 2.2.1 source for exact DOM selectors
|
||||
- [x] Implement `tests/uptime-kuma/playwright/test_monitor_wizard.py`
|
||||
|
||||
### DONE (continued)
|
||||
- [x] Open recipe-maintainers/uptime-kuma PR #3 + trigger `!testme`
|
||||
- [x] Drone build #460 = LEVEL 5, playwright:1 PASS
|
||||
- [x] Claim M1 gate (fe8922c)
|
||||
|
||||
### IN PROGRESS
|
||||
- [ ] Second `!testme` run (comment #14352, flake check) — polling for build
|
||||
- [ ] M1 Adversary review
|
||||
|
||||
### PENDING (after M1 Adversary PASS)
|
||||
- [ ] Second `!testme` run (flake check — 2 consecutive green)
|
||||
- [ ] Update PARITY.md (note the new playwright/ test)
|
||||
- [ ] Close DEFERRED.md entry "2026-05-28 — uptime-kuma create-a-monitor"
|
||||
- [ ] Claim M2 gate
|
||||
- [ ] Write ## DONE after M2 Adversary PASS
|
||||
|
||||
## Adversary findings
|
||||
(Adversary-owned — no items yet; populated as issues are found)
|
||||
99
machine-docs/BACKLOG-lvl5.md
Normal file
99
machine-docs/BACKLOG-lvl5.md
Normal file
@ -0,0 +1,99 @@
|
||||
# BACKLOG — Phase lvl5
|
||||
|
||||
## Build backlog
|
||||
|
||||
- [x] B1 (P1) `level.py`: append rung `lint` (L5); new status vocabulary {pass, fail, skip, unver}; `compute_level()` → new formula (level = max i: rung_i pass ∧ ∀j<i status ∈ {pass,skip}); DELETE cap_reason/capped concepts.
|
||||
- [x] B2 (P1) lint executor (`harness/lint.py`): `abra recipe lint <recipe>` against the exact tested ref; hard ~60s timeout; rc+full output → `lint.txt` artifact; pass/fail/unver classification (missing abra / timeout / exception → unver, never pass, never skip); mirror-context handling per phase-plan §2.3 (probe abra behavior first; any filtering = named + unit-tested + DECISIONS.md).
|
||||
- [x] B3 (P1) `results.py`: wire lint into `derive_rungs` + explicit intentional-vs-unintentional classification of EVERY N/A source; drop level_cap_reason/level_cap_rung from schema; `skips()` reflects new statuses; orchestrator (`run_recipe_ci.py`) runs lint executor at the tested-ref point + passes result through; verdict-neutral (R7 wrap).
|
||||
- [x] B4 (P1) unit tests: rewrite test_level.py/test_results.py to new semantics incl. mission worked examples (fail-blocks → L1; intentional-skip climbs → L5; unver-blocks → L2; lint unver → L4; unclassifiable N/A → unver default); lint executor tests; old-artifact rendering compat tests.
|
||||
- [x] B5 (P2) `card.py`: 0–5 color ramp; cap line removed ("level N of 5" neutral); rung table renders ✔/✘/intentional-skip/unverified; level_badge_svg loses cap_skip third segment (badge = number+color only); tolerate old artifacts.
|
||||
- [x] B6 (P2) `dashboard.py`: _LEVEL_COLOR 5-scale; _level_pill/badge SVG number-only; legend text; old results.json (cap_reason present, lint absent) render without KeyError.
|
||||
- [x] B7 (P2) docs: results-ux.md, testing.md, recipe-customization.md §EXPECTED_NA wording — L5 ladder, de-cap semantics.
|
||||
- [x] B8 (P1) DECISIONS.md: semantics change record (replaces Phase-3 "N/A caps"); N/A classification table (every derive_rungs N/A source → intentional|unintentional); mirror-filter decision for lint (if any filtering).
|
||||
- [x] B9 — gate M1: claim (branch w/ P1+P2; clean tree; cold-verifiable).
|
||||
- [x] B10 (P3) lint sweep over ALL enrolled recipes (scratch clones — never touch ~/.abra/recipes during builds); matrix here (pass/fail + rule hits); mechanical fixes → mirror PRs (never push main/never merge); rest → DEFERRED.md.
|
||||
- [x] B11 (P4) real-CI proofs: ≥1 genuine L5; ≥1 lint-blocked L4 (synth branch ok); ≥1 N/A-skip climb; 2× drone !testme; canary suite at re-derived designed levels; 1 synthesized unver-blocks run; before/after level table for ALL enrolled recipes; card/dashboard PNG/SVG visually verified.
|
||||
- [x] B12 — gate M2: claim; then ## DONE after fresh PASS.
|
||||
|
||||
## Adversary findings
|
||||
|
||||
## P3 lint sweep matrix (B10) — all 19 enrolled, mirror main HEAD, 2026-06-11
|
||||
|
||||
Method: per recipe, fresh scratch clone of its canonical origin (mirror for the 17
|
||||
recipe-maintainers recipes; coopcloud upstream for bluesky-pds/custom-html-tiny/mumble) +
|
||||
upstream version tags fetched (production fetch_recipe shape), then `harness.lint.run_lint`
|
||||
from phase-lvl5 @ 3d8d286 in a scratch ABRA_DIR (`/tmp/lvl5-sweep` on cc-ci; full outputs in
|
||||
`/tmp/lvl5-sweep/art/<recipe>/lint.txt`). Canonical `~/.abra/recipes` never touched.
|
||||
|
||||
**Result: 19/19 PASS** (no error-severity rule unsatisfied anywhere). No recipe-mirror PRs and
|
||||
no DEFERRED entries needed. Warn-severity misses (informational, do not fail the rung):
|
||||
|
||||
| recipe | lint | warn-rule misses |
|
||||
|---|---|---|
|
||||
| bluesky-pds | pass | R002 R007 R015 |
|
||||
| cryptpad | pass | R002 R005 R007 |
|
||||
| custom-html | pass | R002 R004 R005 |
|
||||
| custom-html-tiny | pass | R002 |
|
||||
| discourse | pass | R002 R007 R015 |
|
||||
| ghost | pass | R015 |
|
||||
| hedgedoc | pass | R015 |
|
||||
| immich | pass | R002 R005 |
|
||||
| keycloak | pass | R002 R015 |
|
||||
| lasuite-docs | pass | R005 |
|
||||
| lasuite-drive | pass | R002 R005 |
|
||||
| lasuite-meet | pass | R002 |
|
||||
| mailu | pass | R002 |
|
||||
| matrix-synapse | pass | R002 R015 |
|
||||
| mattermost-lts | pass | R002 R015 |
|
||||
| mumble | pass | R002 |
|
||||
| n8n | pass | R002 R015 |
|
||||
| plausible | pass | R002 R005 R007 |
|
||||
| uptime-kuma | pass | R015 |
|
||||
|
||||
Note: lasuite-meet's historically-lightweight tag `0.3.0+v1.16.0` is now ANNOTATED upstream
|
||||
(verified `git cat-file -t` = tag on all three version tags) — R014 passes genuinely; the
|
||||
abra.py:105 lightweight-tag deploy fallback simply no longer triggers for it.
|
||||
|
||||
## Before/after level table skeleton (§2.9 — "after" to be filled by P4 real runs)
|
||||
|
||||
Baseline = latest results.json on cc-ci per recipe re-scored under the CURRENT (pre-lvl5,
|
||||
4-rung) rule; ancient 6-rung artifacts (builds ≤205, integration/recipe_local era) re-read on
|
||||
their four essential rungs. Predicted = same tier outcomes + sweep lint result under the new
|
||||
rule (assumption flagged; P4 produces the real values).
|
||||
|
||||
| recipe | baseline rungs (latest artifact) | baseline level | predicted new level | REAL new level (P4 run) | why it shifts |
|
||||
|---|---|---|---|---|---|
|
||||
| bluesky-pds | no artifact (deploy-gated upstream, shot-phase N/A) | — | — | — (still deploy-gated; documented N/A) | still deploy-gated |
|
||||
| cryptpad | I✔ U✔ B✔ F✔ (#181) | 4 | 5 | (not re-run; analytic 5) | + lint pass |
|
||||
| custom-html | I✔ U✔ B✔ F✔ (#182) | 4 | 5 | **4** (#405 PR4 lintdemo: lint fail R011; main analytic 5) | + lint pass |
|
||||
| custom-html-tiny | I✔ U✔ B-na F-na (#205, predates functional/) | 2 | 5 | **5** (#399 — N/A-skip climb, was 2) | de-cap: backup skip declared; functional/ tests exist now; + lint |
|
||||
| discourse | I✔ U✔ B✔ F✔ (#184) | 4 | 5 | (not re-run; analytic 5) | + lint pass |
|
||||
| ghost | I✔ U✔ B✔ F✔ (#185) | 4 | 5 | (not re-run; analytic 5) | + lint pass |
|
||||
| hedgedoc | I✔ U✔ B✔ F✔ (#113) | 4 | 5 | **5** (#398, 100s) | + lint pass |
|
||||
| immich | I✔ U✔ B✔ F✔ (#370) | 4 | 5 | **5** (#406, drone !testme PR2, 199s) | + lint pass |
|
||||
| keycloak | I✔ U✔ B✔ F✔ (#187) | 4 | 5 | (not re-run; analytic 5) | + lint pass |
|
||||
| lasuite-docs | I✔ U✔ B✔ F✔ (#188) | 4 | 5 | (not re-run; analytic 5) | + lint pass |
|
||||
| lasuite-drive | I✔ U✔ B✔ F✔ (#189) | 4 | 5 | (not re-run; analytic 5) | + lint pass |
|
||||
| lasuite-meet | I✔ U✔ B✔ F✔ (#204) | 4 | 5 | (not re-run; analytic 5) | + lint pass |
|
||||
| mailu | I✔ U✔ B-na F✔ (#191) | 2 | 5 | (not re-run; analytic 5 — same de-cap as #399) | de-cap: not backup-capable → skip climbs (the §2.9 N/A-skip demo) |
|
||||
| matrix-synapse | I✔ U✔ B✔ F✔ (#203) | 4 | 5 | (not re-run; analytic 5) | + lint pass |
|
||||
| mattermost-lts | I✔ U✔ B✔ F✔ (#196) | 4 | 5 | (not re-run; analytic 5) | + lint pass |
|
||||
| mumble | no results.json artifact retained | — | — | **5** (#413, 80s — first retained artifact) | P4 run to establish |
|
||||
| n8n | I✔ U✔ B✔ F✔ (#197) | 4 | 5 | (not re-run; analytic 5) | + lint pass |
|
||||
| plausible | I✔ U✔ B✔ F✔ (#371) | 4 | 5 | **5** (#407, drone !testme PR3, 164s) | + lint pass |
|
||||
| uptime-kuma | I✔ U✔ B✔ F✔ (#165) | 4 | 5 | (not re-run; analytic 5) | + lint pass |
|
||||
|
||||
Canaries (designed levels under the NEW formula, re-derived): custom-html-bkp-bad /
|
||||
custom-html-rst-bad — backup-capable with a failing backup/restore tier → backup_restore rung
|
||||
FAIL → level 2 (fail still blocks; run verdict red as today). To be proven in P4.
|
||||
|
||||
### Canary designed-level re-derivation (P4, runs 415/416 — 2026-06-11)
|
||||
|
||||
Under the NEW formula the bad canaries' designed level is **1**, not the old 2: their mirrors
|
||||
carry no published version tags on the SRC+REF path → upgrade = intentional skip (climbs past
|
||||
but never earns), backup_restore = FAIL blocks → level = install = 1. Verified live: 415
|
||||
(bkp-bad) + 416 (rst-bad) both **verdict FAILURE (red)**, rungs
|
||||
{install: pass, upgrade: skip, backup_restore: fail, functional: unver (post-failure abort),
|
||||
lint: pass}, LEVEL 1. Backup/restore fail still blocks; verdict logic untouched.
|
||||
(First attempts 411/412 failed in 1s: canaries are mirror-only, not catalogue recipes — they
|
||||
need SRC+REF params, as prior phases ran them.)
|
||||
32
machine-docs/BACKLOG-mailu.md
Normal file
32
machine-docs/BACKLOG-mailu.md
Normal file
@ -0,0 +1,32 @@
|
||||
# BACKLOG — phase `mailu` (backupbot labels + backup/restore coverage)
|
||||
|
||||
## Build backlog
|
||||
(Builder-owned — read only for Adversary)
|
||||
|
||||
## Adversary findings
|
||||
|
||||
### [ADV-mailu-01] `/mail` Maildir volume restoration not tested — seed too shallow [adversary]
|
||||
|
||||
**Filed**: 2026-06-11T20:58Z
|
||||
**Status**: CLOSED @2026-06-11T21:00Z — fix verified green in build #477 (M1 PASS)
|
||||
|
||||
**Plan requirement** (`plan-phase-mailu-backup.md` §2.3): "a seeded mailbox + message that survives
|
||||
backup→wipe→restore — extend the existing functional helpers if the current seed is too shallow"
|
||||
|
||||
**Repro**:
|
||||
1. Current `ops.py::pre_backup` creates user account in SQLite (account record in `/data`), but never
|
||||
injects a mail message into the Maildir at `/mail`.
|
||||
2. `ops.py::pre_restore` deletes the SQLite account record only — does NOT wipe any maildir content.
|
||||
3. `test_restore.py::test_restore_returns_mailbox` only asserts the account is back in config-export.
|
||||
4. Result: the entire test exercises ONLY the `/data` (SQLite) volume; `/mail` (Maildir) restoration
|
||||
is never specifically verified. If backupbot silently failed to restore `/mail`, this test passes.
|
||||
|
||||
**Fix**:
|
||||
1. `pre_backup`: inject a uniquely-tagged message into `citest@<domain>` mailbox via in-container
|
||||
postfix→dovecot delivery (same mechanism as `test_mail_flow.py::test_send_and_receive_mail`)
|
||||
2. `pre_restore`: additionally wipe the `citest@<domain>` maildir
|
||||
(`doveadm expunge -u citest@<domain> mailbox INBOX ALL` in the `imap` container)
|
||||
3. `test_restore.py`: also assert the seeded message is back
|
||||
(e.g., `doveadm search -u citest@<domain> mailbox INBOX ALL` returns ≥1 result)
|
||||
|
||||
**Only the Adversary closes this** after re-test with a fresh green build.
|
||||
@ -4,6 +4,13 @@ Architecture decisions and dead-ends. One line of rationale each. (§0, §8)
|
||||
|
||||
## Settled
|
||||
|
||||
- **cfold deprecated-folder policy — SETTLED (2026-06-12, phase cfold).** `tests/<recipe>/custom/`
|
||||
is the canonical home for custom tests. Discovery keeps recognizing legacy `functional/` and
|
||||
`playwright/` subdirs for both cc-ci and approved repo-local tests as a temporary compatibility
|
||||
alias, but it emits a one-line warning to stderr whenever it discovers tests there. Rationale:
|
||||
the phase plan forbids silent coverage loss, and recipe repos outside this clone may still be on
|
||||
the old layout during the migration window.
|
||||
|
||||
- **Wildcard TLS:** operator pre-issues wildcard cert at `/var/lib/ci-certs/live/`; Traefik file
|
||||
provider serves it; **no ACME** for commoninternet.net. (Plan §4.0/§8 — fixed.)
|
||||
- **Repo:** `git.autonomic.zone/recipe-maintainers/cc-ci`, private. Bot is org admin. (Bootstrap.)
|
||||
@ -1353,3 +1360,54 @@ recipe"); pass iff the table rendered clean; anything else unver + loud log. Har
|
||||
(observed ~0.7s); executor runs before the tiers (tree at tested ref), double-wrapped, R7
|
||||
verdict-neutral. Full output → run artifact `lint.txt` (dashboard-served); status + failing
|
||||
rule ids → results.json `lint`.
|
||||
|
||||
**bluesky-pds re-pin decision (phase bsky, 2026-06-11).** The recipe pinned the moving tag
|
||||
`ghcr.io/bluesky-social/pds:0.4`, which upstream now republishes with main-branch builds
|
||||
(currently @atproto/pds 0.5.1, Node 24, `/app/index.ts` — no `index.js`), breaking the
|
||||
recipe's entrypoint override (`exec node --enable-source-maps index.js`). Fix: pin the
|
||||
newest RELEASED exact tag `0.4.219` (Node 20.20, `/app/index.js`, CMD identical to the
|
||||
recipe's exec line — entrypoint stays valid unchanged) and bump the version label
|
||||
`0.2.0+v0.4` → `0.3.0+v0.4.219` (minor bump for an upstream pin change, immich-PR#2
|
||||
precedent). REJECTED: tracking 0.5.1 (only exists as moving/sha- tags built from main —
|
||||
no release tag; would also require entrypoint `index.ts` migration against an unreleased
|
||||
version); digest-suffix pinning (abra survey/upgrade tooling chokes on tag@digest — see
|
||||
immich standing note). When upstream cuts real 0.5.x release tags, upgrade properly
|
||||
(entrypoint will then need the index.ts/Node-24 migration — recorded in
|
||||
cc-ci-plan/upstream/bluesky-pds.md). Never re-pin to `:0.4`/`latest`/minor tags.
|
||||
|
||||
**EXPECTED_NA["upgrade"] suppresses the upgrade-tier base deploy (phase bsky, 2026-06-11).**
|
||||
The deploy-once design deploys the upgrade BASE (previous published version) and only the
|
||||
upgrade tier chaos-redeploys the PR head — so a recipe whose published versions ALL became
|
||||
undeployable (bluesky-pds: every tag pins moving `ghcr.io/bluesky-social/pds:0.4`, which
|
||||
upstream republished with incompatible main builds) fails INSTALL at the base before the PR
|
||||
head is ever exercised, and no UPGRADE_BASE_VERSION value can help (it must be a published
|
||||
tag — they're all broken). Decision: declaring the upgrade rung in EXPECTED_NA (the existing
|
||||
intentional-skip mechanism) now ALSO makes upgrade_base() return None → the single deploy is
|
||||
the PR head itself; the upgrade tier records "skip"; derive_rungs classifies it as the
|
||||
DECLARED intentional skip with the recipe's reason (results.json skips.intentional). NOT a
|
||||
gate weakening: the rung is never reported pass, the skip + reason are fully visible, and the
|
||||
declaration is evidence-backed in the recipe_meta comment + upstream registry; it is the only
|
||||
way to exercise a PR at all for a recipe in this state. Re-enable path documented per-recipe
|
||||
(bluesky: drop EXPECTED_NA + set UPGRADE_BASE_VERSION="0.3.0+v0.4.219" once merged+published).
|
||||
Locked by tests/unit/test_upgrade_base.py.
|
||||
|
||||
## 2026-06-11 — uptime-kuma: Playwright (option b) for monitor-wizard test (phase kuma)
|
||||
|
||||
**Decision:** use Playwright (option b from plan-phase-kuma-monitor.md §1) to implement
|
||||
the `tests/uptime-kuma/playwright/test_monitor_wizard.py` test.
|
||||
|
||||
**Why not python-socketio (option a):** python-socketio is NOT installed in the cc-ci
|
||||
Nix Python environment (site-packages has playwright + pytest only; no socketio wheel).
|
||||
Adding it would require modifying `nix/cc-ci.nix` and running `nixos-rebuild switch` on
|
||||
cc-ci — extra Nix overhead when Playwright already handles Socket.IO transparently through
|
||||
the real browser. The option (a) benefit (speed, headless) is outweighed by the absence of
|
||||
the package.
|
||||
|
||||
**Why Playwright works here:** uptime-kuma 2.2.1 has stable `data-cy` attributes on the
|
||||
setup form and `data-testid` attributes on the monitor form + status badge — confirmed
|
||||
present in the compiled bundle (`dist/assets/index-D_mnxLA0.js`). These are the canonical
|
||||
Cypress/testing selectors; they do not change without an intentional test-attribute removal.
|
||||
The Playwright flow is deterministic: wizard → `/add` form → `/dashboard/:id` detail page.
|
||||
|
||||
**Runtime implication:** Playwright adds ~5–10 s overhead vs a headless socketio client,
|
||||
but stays well within the ≤90 s budget. Acceptable.
|
||||
|
||||
@ -118,6 +118,8 @@ before the build is called done) — but does **not** force closure.
|
||||
- **Linked IDEA:** —
|
||||
|
||||
### 2026-05-28 — uptime-kuma create-a-monitor (§4.3 prescribed)
|
||||
- [x] **CLOSED @2026-06-11 (Builder, phase kuma):** `tests/uptime-kuma/playwright/test_monitor_wizard.py` implemented and proven in real CI. Playwright (option b) drives the actual browser; Socket.IO handled transparently. Flow: wizard admin-create → self-probe monitor (→ Up, real heartbeat row) + dead-port monitor (→ Down, proves probe engine). Commits: `8da59cf` (test) + `fe8922c` (M1 claim). Drone builds #460 + #462 both LEVEL 5 with `test_monitor_wizard [pass]`. M1+M2 Adversary PASSes in REVIEW-kuma.md. DEFERRED is closed.
|
||||
- [x] **RE-ENTERED @2026-06-11:** operator approved — executing as phase `kuma` (cc-ci-plan/plan-phase-kuma-monitor.md).
|
||||
- [ ] **What:** Add a test that completes uptime-kuma's first-run setup wizard via Socket.IO,
|
||||
logs in to obtain a JWT, creates a monitor (`monitor add` Socket.IO emit), and asserts the
|
||||
monitor appears in the listed-monitors response.
|
||||
@ -210,6 +212,7 @@ before the build is called done) — but does **not** force closure.
|
||||
(none yet — append `### YYYY-MM-DD — <slug> CLOSED (commit/PR)` here when re-entered.)
|
||||
|
||||
### 2026-05-28 — plausible (Q4.7) recipe enrollment
|
||||
- [x] **CLOSED @2026-06-11 (operator housekeeping):** overtaken — plausible is enrolled and running in CI (§4.3 floor `71af595`); the full-lifecycle remainder is the Q4.7b entry below (recipe PR#3 green, operator merge pending).
|
||||
- [ ] **What:** Enroll plausible in cc-ci with parity health_check + ≥2 specific tests (per
|
||||
plan §4.3: "track a test event, query it back"). `tests/plausible/recipe_meta.py` +
|
||||
`tests/plausible/functional/test_health_check.py` are drafted (commit pending) but the
|
||||
@ -237,6 +240,7 @@ before the build is called done) — but does **not** force closure.
|
||||
Defensible defer; lift when the operator wants the deeper coverage OR Phase-4 reviews.
|
||||
|
||||
### 2026-05-29 — immich recipe needs a pg_dump backup hook for reliable DB restore (P4)
|
||||
- [x] **CLOSED @2026-06-11:** cc-ci-authored immich recipe PR#2 (pg_dump hook) verified green; operator confirmed 2026-06-11 — merge pending, no further loop work.
|
||||
- [ ] **What:** immich's upstream recipe backs up the LIVE postgres data VOLUME via restic
|
||||
(`backupbot.backup=true` on `database`, no pg_dump hook), so a DB row does NOT survive
|
||||
`abra app restore` (diagnosed: seed→backup→drop→restore→row absent; app healthy). Real
|
||||
@ -256,6 +260,7 @@ before the build is called done) — but does **not** force closure.
|
||||
- **Linked IDEA:** —
|
||||
|
||||
### 2026-05-29 — discourse: upstream recipe pins removed bitnami images (undeployable)
|
||||
- [x] **CLOSED @2026-06-11 (operator housekeeping):** superseded — discourse is enrolled and runs the full lifecycle in CI (L4 baseline run 184, 2026-06-05); the bitnami-pin blocker no longer applies.
|
||||
- [ ] **What:** discourse (Q4.6) cannot be enrolled/tested because the recipe pins
|
||||
`image: bitnami/discourse:<tag>` (app + sidekiq) and **Docker Hub no longer serves any
|
||||
`bitnami/discourse:*` tag** (bitnami's 2024/2025 legacy migration). Proven on cc-ci:
|
||||
@ -282,6 +287,14 @@ before the build is called done) — but does **not** force closure.
|
||||
- **Linked IDEA / BACKLOG:** Q4.6.
|
||||
|
||||
### 2026-05-29 — mailu: no backup config (P4 N/A) — recipe-PR to add backupbot
|
||||
- [x] **CLOSED @2026-06-11 (phase mailu, Builder):** Mirror PR#3 (`add-backupbot-labels`, head
|
||||
`edc0201a79d3`) on `git.autonomic.zone/recipe-maintainers/mailu` adds backupbot v2 labels to
|
||||
`admin` service (`/data` SQLite) and `imap` service (`/mail` Maildir). Full lifecycle at PR head
|
||||
= LEVEL 5 (drone build #477): install/upgrade/backup/restore/functional all PASS; both
|
||||
`/data` (SQLite) and `/mail` (Maildir) seeded + wiped + verified restored. Adversary M1 PASS
|
||||
@2026-06-11T21:00Z. PR left open for operator merge. mailu's backup rung is now earned
|
||||
(`backup_capable=True`), not skipped. Phase mailu M1 PASS; M2 claim in progress.
|
||||
- [x] **RE-ENTERED @2026-06-11:** operator approved the backupbot recipe-PR route — executing as phase `mailu` (cc-ci-plan/plan-phase-mailu-backup.md).
|
||||
- [ ] **What:** mailu (Q4.9) ships **no `backupbot.backup` label** on any service, so cc-ci's
|
||||
backup/restore tiers cleanly SKIP (`backup_capable=False`) — P4 (backup data-integrity) is N/A
|
||||
for mailu as published (no backup mechanism to exercise). Durable fix = a recipe-PR adding
|
||||
@ -296,6 +309,9 @@ before the build is called done) — but does **not** force closure.
|
||||
- **Linked IDEA / BACKLOG:** Q4.9.
|
||||
|
||||
### 2026-05-29 — drone (Q4.10) blocked on host /etc/timezone deploy (gitea SCM dep) + scoped integration
|
||||
- [x] **RE-ENTERED @2026-06-11:** operator approved — executing as phase `drone` (cc-ci-plan/plan-phase-drone-enroll.md); P0 host /etc/timezone deploy is orchestrator-owned.
|
||||
- [x] **MAXIMAL SUBSET COMPLETE @2026-06-11T22:30Z — Adversary M2 PASS, build #506 L5.** All mandatory tiers (install+upgrade+functional+lint) pass; backup structural skip justified in PARITY.md; bridge-triggered !testme CI run confirmed `event:custom`. DEFERRED item progressed: (1) P0 host fix: DONE; (2) Integration MAXIMAL SUBSET: DONE. **Build-creation gap (§4.3) remains open** — deferred sub-item per original filing.
|
||||
- **Adversary §7.1 sign-off on build-creation gap @2026-06-11T22:30Z:** The drone API build-creation flow (creating/running CI pipelines via drone's own API — requires drone OAuth token + `.drone.yml` + webhook) is accepted as a genuine, proportionate deferral. It is a harness capability gap, not a recipe gap. Drone boots with gitea SCM wired correctly (proven L5 in build #506); build-creation automation is a follow-on. SIGNED OFF. Remaining DEFERRED: build-creation API automation only.
|
||||
- [ ] **What:** drone (Q4.10, LAST §5 recipe) cannot be enrolled until two things land:
|
||||
(1) **HOST FIX — operator-deploy needed:** drone is a CI server that REQUIRES a git-provider SCM
|
||||
to boot; the only viable dep is **gitea**, which the recipe binds `/etc/timezone:ro` from the
|
||||
@ -322,6 +338,7 @@ before the build is called done) — but does **not** force closure.
|
||||
- **Linked IDEA / BACKLOG:** Q4.10; JOURNAL-2 f86a58a; commit 3bde76f.
|
||||
|
||||
### 2026-05-30 — plausible Q4.7 full (recipe-PR Q4.7b: fix ClickHouse entrypoint wget restart-storm)
|
||||
- [x] **CLOSED @2026-06-11:** recipe PR#3 (ClickHouse entrypoint + backup fixes) verified GREEN at PR head; operator confirmed 2026-06-11 — merge pending. Post-merge follow-up: full lifecycle on main to formally claim Q4.7.
|
||||
- [ ] **What:** Fix the recipe `entrypoint.clickhouse.sh` so ClickHouse boots reliably, then run
|
||||
plausible's FULL lifecycle (`install,upgrade,backup,restore,custom`) green + claim Q4.7. Suite
|
||||
authored (`tests/plausible/` ops + test_backup/restore/upgrade + event-roundtrips); §4.3 floor
|
||||
@ -335,8 +352,29 @@ before the build is called done) — but does **not** force closure.
|
||||
- **Re-entry trigger:** Builder authors recipe-PR Q4.7b (cache tarball on a volume / wget
|
||||
retry+backoff / drop `2>/dev/null` / `set +e` w/ fallback), then runs plausible-full green + claims.
|
||||
- **Linked:** REVIEW-2 `e850281` (root-cause + DENY), `71af595` (§4.3 floor); DECISIONS 2026-05-30.
|
||||
- discourse upgrade-HC1 @7ae7b0f stamps prev-base tag commit (eb96de94+U) on BOTH old+new harness since ~06-10 (baseline 184 was L4 on 06-05); harness-neutral (rcust exonerated, M2-closed) but abra stamp-resolution mechanism UNATTRIBUTED — worth a standalone dig outside rcust. Evidence: /var/lib/cc-ci-runs/{m2p-discourse,ab-discourse-7ae7b0f-oldmain}, JOURNAL-rcust 2026-06-11.
|
||||
- bluesky-pds: UPSTREAM IMAGE BREAKAGE (non-rcust, M2-justified exclusion from baseline match).
|
||||
- [RE-ENTERED @2026-06-11 → phase `dstamp` (cc-ci-plan/plan-phase-dstamp-discourse-drift.md)] discourse upgrade-HC1 @7ae7b0f stamps prev-base tag commit (eb96de94+U) on BOTH old+new harness since ~06-10 (baseline 184 was L4 on 06-05); harness-neutral (rcust exonerated, M2-closed) but abra stamp-resolution mechanism UNATTRIBUTED — worth a standalone dig outside rcust. Evidence: /var/lib/cc-ci-runs/{m2p-discourse,ab-discourse-7ae7b0f-oldmain}, JOURNAL-rcust 2026-06-11.
|
||||
- ✅ **RESOLVED @2026-06-11 (phase `dstamp`, Builder).** NOT an abra stamp-resolution bug — abra
|
||||
stamps the PR head `7ae7b0f7+U` CORRECTLY (proven: repro2 `--debug` line + 3 bail-at-secrets
|
||||
repros; per-run git HEAD=7ae7b0f at deploy, reflog-verified). **Root cause:** discourse
|
||||
`compose.yml` app service `deploy.update_config: { failure_action: rollback, order: start-first,
|
||||
monitor: 5s }`. On the upgrade chaos redeploy, start-first co-resides OLD+NEW (~2× memory) for
|
||||
the precompile/Rails-heavy app; under host memory pressure the NEW task fails swarm's 5s update
|
||||
monitor → `failure_action: rollback` reverts the app service to PreviousSpec, including the
|
||||
`chaos-version` label (head→base `eb96de94+U`). start-first kept the old task serving so
|
||||
`wait_healthy` passed; HC1 then read the reverted base commit and misreported it as a stamp
|
||||
mismatch. **Direct evidence:** `/var/lib/cc-ci-runs/dstamp-repro4.console.log` — post-redeploy
|
||||
`UpdateStatus.State=updating`, `.Spec chaos-version=7ae7b0f7+U` (head applied), `.PreviousSpec
|
||||
chaos-version=eb96de94+U` (base); the read after the rollback = base. **Fix (commits 0cc31a5 +
|
||||
e9c26c7):** (1) `tests/discourse/compose.ccci.yml` app `update_config.order: stop-first` (new
|
||||
task boots with full memory → no OOM → no spurious rollback; `failure_action: rollback` left
|
||||
intact); (2) general `lifecycle.assert_upgrade_converged` (2-phase StartedAt protocol) detects a
|
||||
swarm rollback/pause and fails the upgrade HONESTLY — HC1 commit-match unchanged, unweakened.
|
||||
**Proven in real CI:** drone `!testme` build **#450** (discourse @7ae7b0f, cc-ci main 2da1f01) =
|
||||
**LEVEL 5**, all tiers PASS (install/upgrade/backup/restore/custom), clean_teardown + no_secret_leak
|
||||
true; PR recipe-maintainers/discourse#2 comment shows ✅ passed. **Blast-radius:** only discourse
|
||||
affected (keycloak/n8n have the same policy but upgrade-PASS L4 across runs; drone/traefik infra);
|
||||
the harness guard covers all rollback-policy recipes. M1+M2 evidence: STATUS-/JOURNAL-/REVIEW-dstamp.
|
||||
- [RE-ENTERED @2026-06-11 → phase `bsky`] ✅ **RESOLVED @2026-06-11 (phase bsky, Builder):** root cause = upstream republishes the MOVING tag `:0.4` with main-branch builds (now @atproto/pds 0.5.1, Node 24, `/app/index.ts` — no `index.js`), breaking the recipe's entrypoint override. Fix PR open (operator merges): **recipe-maintainers/bluesky-pds PR #2** (`upgrade-0.3.0+v0.4.219`, head f7b6c8df — exact-pin `0.4.219` + version-label bump). Proven green at PR head via real drone CI: run 427 **level 5** (install/backup_restore/functional/lint PASS; upgrade = declared intentional skip — no deployable published base, both old tags pin the republished `:0.4`; negative control run 423). Screenshot real (PDS landing page). The shot-phase deploy-gated N/A is lifted on the PR runs. Upstream registry: cc-ci-plan/upstream/bluesky-pds.md; decisions: DECISIONS.md 2026-06-11 (pin choice + EXPECTED_NA-upgrade base suppression). Both the re-pin follow-up AND the rcust M2 exclusion note are hereby closed with these pointers. Original entry follows: bluesky-pds: UPSTREAM IMAGE BREAKAGE (non-rcust, M2-justified exclusion from baseline match).
|
||||
The app container crash-loops `Error: Cannot find module '/app/index.js'` (MODULE_NOT_FOUND,
|
||||
Node v24.15.0) under the recipe's pinned tag on EVERY current run — new main @ mirror head
|
||||
(m2r-bluesky-pds), new main serial re-run (m2rr-bluesky-pds), AND old pre-rcust main @ old
|
||||
@ -360,3 +398,13 @@ before the build is called done) — but does **not** force closure.
|
||||
Evidence: /tmp/mumble-probe{2,3,4}.out + /tmp/mumble-orch{4,5}.log on cc-ci (90s DOM/console/
|
||||
network observation; websockify reachable, /ws & /websocket 404 from websockify itself);
|
||||
/var/lib/cc-ci-runs/shot-proof-mumble/screenshot.png (L4 run, loader frame).
|
||||
|
||||
## WC5 promote-on-green-cold ignores stage completeness (filed 2026-06-11, Builder, phase lvl5)
|
||||
|
||||
Observed during the lvl5 unver-blocks proof: a GREEN hand-run with `STAGES=install,upgrade,custom`
|
||||
(backup/restore excluded) on latest still advanced custom-html's warm canonical —
|
||||
`should_promote_canonical` checks green+cold+latest but not that ALL stages ran. Pre-existing
|
||||
behavior (not introduced or worsened by lvl5; Adversary concurs it is not a finding). Only
|
||||
reachable via the operator/dev STAGES escape — production drone runs always run all stages.
|
||||
**Needed from operator:** decide whether promote should additionally require the full stage set
|
||||
(one-line guard in `should_promote_canonical`), or whether dev hand-runs promoting is acceptable.
|
||||
|
||||
120
machine-docs/JOURNAL-bsky.md
Normal file
120
machine-docs/JOURNAL-bsky.md
Normal file
@ -0,0 +1,120 @@
|
||||
# JOURNAL — phase bsky
|
||||
|
||||
## 2026-06-11T11:31Z–11:55Z — bootstrap + root-cause diagnosis (B1, B2)
|
||||
|
||||
Phase start. Read plan-phase-bsky-fix.md + plan.md §6.1/§7/§9. Adversary seeded
|
||||
REVIEW-bsky.md (8d5bf30) with cold baseline recon — same suspects I confirmed below.
|
||||
|
||||
**Diagnosis chain (commands + outputs):**
|
||||
|
||||
1. Mirror clone (b2d86ef): `compose.yml` pins `image: ghcr.io/bluesky-social/pds:0.4`,
|
||||
overrides entrypoint (`dumb-init --` + config-mounted `/entrypoint.sh`);
|
||||
`entrypoint.sh.tmpl` ends `exec node --enable-source-maps index.js` — relative path,
|
||||
resolved against image WORKDIR.
|
||||
|
||||
2. Live image inspection on cc-ci:
|
||||
`docker image inspect ghcr.io/bluesky-social/pds:0.4 --format "{{.Id}} created={{.Created}} workdir={{.Config.WorkingDir}} ... cmd={{.Config.Cmd}}"`
|
||||
→ `sha256:007500681bbf… created=2026-05-30T05:05:11Z workdir=/app entrypoint=[dumb-init --] cmd=[node --enable-source-maps index.ts]`
|
||||
`docker run --rm --entrypoint sh ghcr.io/bluesky-social/pds:0.4 -c 'node --version; ls /app'`
|
||||
→ `v24.15.0` / `index.ts node_modules package.json pnpm-lock.yaml` — **no index.js**.
|
||||
`grep @atproto/pds /app/package.json` → `"@atproto/pds": "0.5.1"`; /usr/local/bin/goat present.
|
||||
So `:0.4` is now a main-branch 0.5.1 build → recipe's `index.js` exec = MODULE_NOT_FOUND.
|
||||
This precisely explains the rcust-era crash-loop evidence (Node v24.15.0 in traceback).
|
||||
|
||||
3. Upstream research:
|
||||
- ghcr tags/list (paginated): exact tags …0.4.158, 0.4.169, 0.4.182, 0.4.188, 0.4.193,
|
||||
0.4.204, 0.4.208, 0.4.219, plus anomalous 0.4.5001. `:0.4` digest `871194d2…` ==
|
||||
`latest`, ≠ `0.4.219` (`e0b756701c92…`) → :0.4 republished past the release line.
|
||||
- Dockerfile@v0.4.219: node:20.20-alpine3.23, WORKDIR /app, CMD index.js, dumb-init.
|
||||
- Dockerfile@main: node:24.15-alpine3.23, CMD index.ts, + goat binary — matches what
|
||||
`:0.4` now contains. GitHub `releases/latest` 404s (they only push git tags).
|
||||
- service/package.json@v0.4.219: `"@atproto/pds": "0.4.219"`.
|
||||
|
||||
4. Candidate-fix image verified on cc-ci:
|
||||
`docker run --rm --entrypoint sh ghcr.io/bluesky-social/pds:0.4.219 -c 'node --version; ls /app; grep @atproto/pds /app/package.json; which dumb-init'`
|
||||
→ `v20.20.2` / index.js present / `"@atproto/pds": "0.4.219"` / `/usr/bin/dumb-init`.
|
||||
Image CMD `[node --enable-source-maps index.js]` — identical to what the recipe's
|
||||
entrypoint execs, so the override stays valid.
|
||||
|
||||
**Why pin 0.4.219 and not chase 0.5.1 (rationale, summarized in DECISIONS.md):** 0.5.1
|
||||
exists only as the moving `:0.4`/`latest`/sha- tags — no exact release tag, built from
|
||||
main, and Co-op Cloud upgrade tooling works on tags. Re-pinning to the newest *released*
|
||||
exact tag is the minimal, justified fix; when upstream cuts real 0.5.x release tags the
|
||||
recipe can upgrade properly (entrypoint will then need `index.ts` + Node 24 — noted in
|
||||
upstream registry).
|
||||
|
||||
Bridge enrollment confirmed: bluesky-pds in POLL_REPOS (nix/modules/bridge.nix:43) →
|
||||
`!testme` works. Mirror has only closed PR#1 (skill smoke test); my fix → PR#2.
|
||||
|
||||
Next: DECISIONS entry (B3), mirror branch + PR (B4), !testme (B5).
|
||||
|
||||
## 2026-06-11T11:40Z–11:55Z — run 423 red: the upgrade-BASE trap (B5 first attempt)
|
||||
|
||||
PR #2 opened (branch upgrade-0.3.0+v0.4.219, head f7b6c8df, 2-line diff) and !testme'd
|
||||
(comment 14340) → drone build/run 423. RESULT: install=fail, level 0 — but NOT the PR:
|
||||
the run never deployed the PR head. The harness deploys ONCE at the upgrade BASE
|
||||
(`previous_version` = vers[-2] = 0.1.1+v0.4 — confirmed: run-423's recipe checkout sat at
|
||||
tag 0.1.1+v0.4) and only the upgrade tier chaos-redeploys the PR head. Both published tags
|
||||
(0.1.1+v0.4, 0.2.0+v0.4) pin the broken moving `:0.4` → the base crash-loops the SAME
|
||||
MODULE_NOT_FOUND (run-423 app log: Node v24.15.0, /app/index.js missing) → install fails
|
||||
before my fix is ever exercised. No published version can EVER deploy again (upstream
|
||||
republished the tag) — so the upgrade path is structurally unverifiable until a fixed
|
||||
version is published post-merge.
|
||||
|
||||
Fix (harness, evidence-backed, not a weakening): EXPECTED_NA["upgrade"] (the EXISTING
|
||||
declared-intentional-skip mechanism, de-capped levels phase lvl5) now also suppresses the
|
||||
base deploy — extracted `upgrade_base()` pure helper in run_recipe_ci.py; single deploy
|
||||
becomes the PR head; upgrade tier records "skip"; derive_rungs classifies it intentional
|
||||
with the declared reason (visible in results.json skips.intentional — never reported as a
|
||||
pass). tests/bluesky-pds/recipe_meta.py declares it with the full reason + the re-enable
|
||||
path (UPGRADE_BASE_VERSION="0.3.0+v0.4.219" once published). 6 new unit tests
|
||||
(tests/unit/test_upgrade_base.py) lock the decision matrix; meta-key doc regenerated.
|
||||
Verified: 253 unit tests pass on cc-ci (was 247), repo lint PASS. Pushed e9745c8.
|
||||
|
||||
Re-triggered !testme (comment 14342) → build/run 427. Monitor armed.
|
||||
|
||||
## 2026-06-11T12:05Z — run 427 GREEN: level 5 at PR head; M1 claimed (B5, B6, B7)
|
||||
|
||||
Run 427 (drone build 427, comment 14342): level 5 — install/backup_restore/functional/
|
||||
lint PASS, upgrade = declared intentional skip (reason verbatim in skips.intentional),
|
||||
clean_teardown + no_secret_leak true, ref f7b6c8dfb81c. Per-run recipe checkout at PR
|
||||
head f7b6c8d with image 0.4.219 (the fix WAS what deployed). Bridge reflected success →
|
||||
PR comment 14343 ✅. Screenshot Read and verified: genuine PDS landing page (ASCII
|
||||
butterfly, "This is an AT Protocol Personal Data Server", /xrpc/ pointer) — exactly the
|
||||
default capture the phase plan predicted would work once deploy works; no hook needed.
|
||||
Card (summary.png): 5/5, upgrade shown INTENTIONAL SKIP with reason; badge "level 5"
|
||||
green. M1 claimed in STATUS-bsky.md.
|
||||
|
||||
## 2026-06-11T12:15Z — records closed (B8) + operator summary drafted (B9)
|
||||
|
||||
DEFERRED bluesky entry marked RESOLVED with pointers (f150012) — covers BOTH the re-pin
|
||||
follow-up and the rcust M2 baseline-exclusion note.
|
||||
|
||||
**Shot-phase N/A disposition update (supersedes the deploy-gated classification):**
|
||||
the shot phase classified bluesky-pds's screenshot "deploy-gated N/A — never capturable
|
||||
because the app never comes up". With the PR#2 fix deployed (run 427, PR head), the
|
||||
DEFAULT landing-page capture works exactly as the phase plan predicted: a real,
|
||||
representative, credential-free PDS landing page (ASCII butterfly + "This is an AT
|
||||
Protocol Personal Data Server" + /xrpc/ pointer). No SCREENSHOT hook was needed. The
|
||||
N/A stands for HISTORICAL runs only; post-merge, bluesky-pds screenshots like any other
|
||||
recipe.
|
||||
|
||||
Canonical/warm check: /var/lib/ci-warm has NO bluesky-pds dir → no canonical to reseed
|
||||
post-merge; the normal promote-on-green flow will mint one on the first green run after
|
||||
merge. Operator summary written to STATUS-bsky.md (B9).
|
||||
|
||||
## 2026-06-11T15:50Z — M1 PASS received; M2 claimed (B10)
|
||||
|
||||
M1 PASS @12:30Z (REVIEW-bsky 369f4f4), no findings, no VETO — every item reproduced cold
|
||||
incl. negative-control teeth and the per-recipe scoping of the EXPECTED_NA change. (Gap
|
||||
12:30→15:45 was a quota window, not work.) All M2 builder-side items were already in
|
||||
place (DEFERRED f150012, operator summary cba53b6); claimed M2 with re-trigger
|
||||
instructions for the fresh cold pass. Phase DoD after M2 PASS → ## DONE with PR open.
|
||||
|
||||
## 2026-06-11T15:55Z — M2 PASS → ## DONE
|
||||
|
||||
M2 PASS @15:48Z (42eabba): Adversary independently re-triggered !testme (comment 14344 →
|
||||
build 435, level 5 at f7b6c8df, identical rung profile + screenshot sha to 427) and
|
||||
corroborated every handoff item — including that 0.5.x has NO release tag, fully settling
|
||||
the §2.2 upgrade-preference question. ## DONE written. Phase ends with PR #2 open for the
|
||||
operator; loop stopped.
|
||||
110
machine-docs/JOURNAL-cfold.md
Normal file
110
machine-docs/JOURNAL-cfold.md
Normal file
@ -0,0 +1,110 @@
|
||||
# JOURNAL — phase cfold
|
||||
|
||||
## 2026-06-11 — Phase cfold start
|
||||
|
||||
### Investigation findings
|
||||
|
||||
Pre-existing test layout:
|
||||
- 60 files in `functional/` subdirs across 20 recipes
|
||||
- 4 files in `playwright/` subdirs (cryptpad, custom-html, uptime-kuma)
|
||||
- Helper modules to move: `_discourse.py`, `_ghost.py`, `_mailu.py`, `_mm.py`, `_mumble_proto.py`, `drone/functional/__init__.py`
|
||||
- `mailu/test_backup.py`, `test_restore.py`, `ops.py` explicitly add `functional/` to sys.path — need updating to `custom/`
|
||||
|
||||
### Decision: deprecated aliases
|
||||
|
||||
Per plan §2 option (RECOMMENDED): keep recognizing `functional/`/`playwright/` as deprecated aliases
|
||||
AND emit a loud one-line warning when a test is found in a deprecated folder. Using `warnings.warn()`
|
||||
at import time of discovery or `print()` directly. Will use `print()` (stderr) so it shows up in CI
|
||||
logs without needing to configure warning filters.
|
||||
|
||||
Implementation: `subdirs = ("custom", "functional", "playwright")` — canonical first — and after
|
||||
finding a test in `functional/` or `playwright/`, emit:
|
||||
`print(f"WARNING [cfold]: test found in deprecated folder '{sub}/' — move to custom/: {path}", flush=True, file=sys.stderr)`
|
||||
|
||||
This way:
|
||||
- `custom/` is canonical and gets discovered first
|
||||
- Old folders still work (zero breakage for repo-local tests) but emit a loud warning
|
||||
- No silent coverage loss possible
|
||||
|
||||
## 2026-06-12 — M1 checkpoint: canonical `custom/` layout landed locally
|
||||
|
||||
Code/work completed:
|
||||
- `runner/harness/discovery.py`: canonical `custom/` discovery, deprecated alias warnings, and
|
||||
`custom_subdir_label()` normalization helper.
|
||||
- `runner/harness/manifest.py`: custom-test counts now normalize to canonical `custom`.
|
||||
- all cc-ci custom tests/helper modules moved from `tests/<recipe>/{functional,playwright}/` into
|
||||
`tests/<recipe>/custom/`.
|
||||
- helper-import fallout fixed where needed (`tests/mailu/{ops.py,test_backup.py,test_restore.py}`).
|
||||
- docs updated to describe `custom/` as the canonical layout and explain the alias-compatibility window.
|
||||
|
||||
Mechanical move summary:
|
||||
- 64 custom test files relocated into `custom/`
|
||||
- helper modules relocated too: `_discourse.py`, `_ghost.py`, `_mailu.py`, `_mm.py`,
|
||||
`_mumble_proto.py`, `tests/drone/custom/__init__.py`
|
||||
|
||||
Verification:
|
||||
```bash
|
||||
nix shell nixpkgs#python312Packages.pytest --command pytest \
|
||||
tests/unit/test_discovery.py tests/unit/test_discovery_phase2.py tests/unit/test_manifest.py -q
|
||||
# ..................
|
||||
# 18 passed in 0.09s
|
||||
```
|
||||
|
||||
Post-move grep state:
|
||||
- remaining `functional/` / `playwright/` matches in live code are intentional: alias-policy docs,
|
||||
deprecated-folder assertions in the unit tests, and discovery comments describing the alias behavior.
|
||||
- the pre-migration inventory in `BACKLOG-cfold.md` is intentionally unchanged because it is the M1
|
||||
baseline record the Adversary will compare against.
|
||||
|
||||
## 2026-06-12 — M1 coverage proof assembled
|
||||
|
||||
Verification commands + observed outputs:
|
||||
|
||||
```bash
|
||||
$ git ls-files "tests/*/custom/test_*.py" | wc -l
|
||||
64
|
||||
|
||||
$ git ls-files "tests/*/functional/*" "tests/*/playwright/*"
|
||||
# no output
|
||||
|
||||
$ for recipe in bluesky-pds cryptpad custom-html custom-html-tiny discourse drone ghost hedgedoc immich keycloak lasuite-docs lasuite-drive lasuite-meet mailu matrix-synapse mattermost-lts mumble n8n plausible uptime-kuma; do count=$(git ls-files "tests/$recipe/custom/test_*.py" | wc -l); printf "%s %s\n" "$recipe" "$count"; done
|
||||
bluesky-pds 4
|
||||
cryptpad 4
|
||||
custom-html 4
|
||||
custom-html-tiny 1
|
||||
discourse 3
|
||||
drone 1
|
||||
ghost 4
|
||||
hedgedoc 2
|
||||
immich 3
|
||||
keycloak 3
|
||||
lasuite-docs 5
|
||||
lasuite-drive 3
|
||||
lasuite-meet 3
|
||||
mailu 3
|
||||
matrix-synapse 3
|
||||
mattermost-lts 3
|
||||
mumble 5
|
||||
n8n 4
|
||||
plausible 2
|
||||
uptime-kuma 4
|
||||
|
||||
$ nix shell nixpkgs#python311Packages.pytest -c pytest tests/unit/test_discovery.py tests/unit/test_discovery_phase2.py tests/unit/test_manifest.py -q
|
||||
..................
|
||||
18 passed in 0.14s
|
||||
```
|
||||
|
||||
Conclusion: the migrated tree still contains the exact same 64 custom test files with the same
|
||||
per-recipe cardinality as the pre-cfold baseline in `BACKLOG-cfold.md`; only the folder paths changed.
|
||||
|
||||
## 2026-06-12 — Adversary M1 PASS received
|
||||
|
||||
Pulled `review(cfold): M1 PASS cold verification` (`4b4d665`). Confirmed in `REVIEW-cfold.md`:
|
||||
- total canonical custom tests = 64
|
||||
- old tracked `functional/` / `playwright/` trees = none
|
||||
- per-recipe counts match the baseline exactly
|
||||
- focused unit suite = `18 passed`
|
||||
- deprecated-alias warning probe works
|
||||
- normalized `(recipe, filename)` before/after set = exact match (`missing []`, `extra []`)
|
||||
|
||||
No fix-forward required. Phase advances to M2 baseline assembly.
|
||||
59
machine-docs/JOURNAL-drone.md
Normal file
59
machine-docs/JOURNAL-drone.md
Normal file
@ -0,0 +1,59 @@
|
||||
# JOURNAL — phase drone (drone enrollment with gitea SCM dep)
|
||||
|
||||
**Phase plan:** `/srv/cc-ci/cc-ci-plan/plan-phase-drone-enroll.md`
|
||||
**Builder:** autonomic-bot / Claude
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-11 — Phase start + design decisions
|
||||
|
||||
### Context read
|
||||
- P0 confirmed: `/etc/timezone` exists (UTC) on cc-ci host — fix from commit 3bde76f is live
|
||||
- Adversary pre-probes read from REVIEW-drone.md:
|
||||
- Confirms P0 satisfied
|
||||
- Confirms drone 1.9.0+2.26.0 (latest), 1.8.0+2.25.0 (previous) — upgrade tier viable
|
||||
- Confirms gitea 3.5.3+1.24.2-rootless (latest), sqlite3 overlay is right choice for dep
|
||||
- Confirms SCM-configured test must exercise actual OAuth flow (not just /healthz)
|
||||
|
||||
### Architecture decisions
|
||||
|
||||
**Gitea as dep:**
|
||||
- Use `compose.sqlite3.yml` overlay — no mariadb needed for a CI dep; lighter resource footprint
|
||||
- `REQUIRE_SIGNIN_VIEW=false` so health check works without login
|
||||
- Admin user created via `gitea admin user create` CLI in container post-deploy
|
||||
- OAuth2 app created via gitea API (basic auth with ci_admin user)
|
||||
|
||||
**SCM-configured test:**
|
||||
- Playwright test completes the full gitea→drone OAuth flow
|
||||
- Navigates to drone's /login → redirects to gitea OAuth authorize page
|
||||
- Fills ci_admin credentials → clicks authorize → lands on drone dashboard
|
||||
- Verifies drone `GET /api/user` returns 200 (session valid)
|
||||
- This proves the full OAuth circuit works (not just health)
|
||||
- Negative teeth: a drone without gitea wiring would not redirect to gitea
|
||||
|
||||
**Drone EXTRA_ENV in install_steps.sh:**
|
||||
- Sets `COMPOSE_FILE=compose.yml:compose.gitea.yml` (activates gitea SCM overlay)
|
||||
- Sets `GITEA_CLIENT_ID`, `GITEA_DOMAIN` from deps creds
|
||||
- Creates `client_secret` Docker secret with gitea OAuth2 client_secret
|
||||
- Sets `DRONE_USER_CREATE=username:ci_admin,admin:true` (ci_admin = gitea admin user)
|
||||
|
||||
**Backup analysis:**
|
||||
- Drone recipe compose.yml has `data` volume but NO backupbot labels
|
||||
- `abra.sh` only exports `DRONE_ENV_VERSION=v2`, no backup functions
|
||||
- Therefore: `backup_capable=False`, backup rung = structural skip (justified in PARITY.md)
|
||||
|
||||
### Implementation sequence
|
||||
1. Add `setup_gitea_oauth()` to `runner/harness/sso.py`
|
||||
2. Update `_enrich_deps_with_sso` in `runner/run_recipe_ci.py` for gitea
|
||||
3. Create `tests/gitea/recipe_meta.py`
|
||||
4. Create `tests/drone/recipe_meta.py`
|
||||
5. Create `tests/drone/install_steps.sh`
|
||||
6. Create `tests/drone/functional/test_scm_configured.py`
|
||||
7. Create `tests/drone/PARITY.md`
|
||||
8. Add unit tests
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-11 — Implementation
|
||||
|
||||
_Evidence of each step logged below as work proceeds._
|
||||
186
machine-docs/JOURNAL-dstamp.md
Normal file
186
machine-docs/JOURNAL-dstamp.md
Normal file
@ -0,0 +1,186 @@
|
||||
# JOURNAL — phase `dstamp` (Builder, reasoning/private)
|
||||
|
||||
## 2026-06-11 — Bootstrap + investigation
|
||||
|
||||
Read the phase plan, plan.md §6.1/§7/§9, the Adversary's REVIEW-dstamp prep notes, and the
|
||||
stamp-relevant harness code (`abra.py`, `lifecycle.py:deployed_identity/recipe_checkout_ref/
|
||||
chaos_redeploy/prepull_images`, `generic.py:perform_upgrade/assert_upgraded`, run_recipe_ci
|
||||
upgrade op + fetch_recipe).
|
||||
|
||||
### Mechanism (from abra source @06a57de = the pinned binary)
|
||||
chaos-version label is set in `cli/app/deploy.go`: for a `-C` deploy, `getDeployVersion` (l.365)
|
||||
returns `Recipe.ChaosVersion()` (l.367-373) and `SetChaosVersionLabel(compose, stack, toDeployVersion)`
|
||||
(l.168). `ChaosVersion` (`pkg/recipe/git.go:300`) = `formatter.SmallSHA(Head().String())` + `+U`
|
||||
if dirty. `Head` (l.483) = go-git `repo.Head()`. Crucially, `app.Recipe.Ensure(ctx)` (deploy.go:86)
|
||||
calls into git.go:38 which **early-returns on `ctx.Chaos`** (l.41-43) — so a chaos deploy does NOT
|
||||
re-checkout the .env version. `GetEnsureContext` (cli/internal/ensure.go) wires `EnsureContext{Chaos,
|
||||
Offline, IgnoreEnvVersion=DeployLatest}` from the CLI flags. So `-C` ⇒ Ensure no-op ⇒ chaos version
|
||||
= whatever git HEAD the harness left checked out.
|
||||
|
||||
### The contradiction that drove the dig
|
||||
The m2p failure message is `chaos commit 'eb96de94+U', not the intended PR-head '7ae7b0f76efb'`.
|
||||
`eb96de9` = tag `0.7.0+3.3.1` (the upgrade base); `7ae7b0f` = PR head (9 commits past that tag,
|
||||
and there is NO 0.8/0.9 tag despite HEAD's "upgrade to 0.9.0+3.5.0" message). The harness
|
||||
`perform_upgrade` does `recipe_checkout_ref(head_ref=7ae7b0f)` then `chaos_redeploy`, with only
|
||||
`env_set` + `prepull_images` (pure docker compose, no git) in between — and the run's recipe
|
||||
**snapshot HEAD = 7ae7b0f**. So at deploy time HEAD *should* be 7ae7b0f ⇒ stamp 7ae7b0f. Yet it
|
||||
stamped eb96de9. abra's source says chaos = Head(); so for eb96de9 to be stamped, HEAD had to be
|
||||
eb96de9 at the chaos deploy — which the isolated flow never produces.
|
||||
|
||||
### Reproductions (all on cc-ci, scratch ABRA_DIR, deploys bail at `secret not generated`
|
||||
### which is deploy.go:140, AFTER the chaos version is computed+logged at deploy.go:372)
|
||||
1. cp -a canonical recipe, checkout head→base(tag)→head, `abra app deploy -C` → `taking chaos
|
||||
version: 7ae7b0f7`. HEAD stays 7ae7b0f. NO drift.
|
||||
2. real non-chaos base deploy (exercises go-git `EnsureVersion` which checks out tag via
|
||||
`Branch: refs/tags/0.7.0+3.3.1`, leaving HEAD=eb96de9), then CLI `git checkout -f head`, then
|
||||
`-C` deploy → `taking chaos version: 7ae7b0f7`. NO drift.
|
||||
3. mirror-faithful: `git clone <recipe-maintainers/discourse>` + `git checkout 7ae7b0f` +
|
||||
`git fetch <coop-cloud/discourse> refs/tags/*:refs/tags/*` (exact `fetch_recipe`), then base
|
||||
deploy → re-checkout head → `-C` deploy → `taking chaos version: 7ae7b0f7`. NO drift.
|
||||
|
||||
Conclusion: the isolated git/abra version-resolution path is **correct** in the current host
|
||||
state. The drift is not in that path.
|
||||
|
||||
### Timeline / differentiator
|
||||
- abra binary: constant since 2026-06-01 (system-4). Not abra.
|
||||
- Same ref 7ae7b0f: run 184 (06-05 02:17, **solo**) was L4 upgrade-PASS. The drift runs
|
||||
(m2b 06-10 20:54, m2p 06-11 00:44, ab 06-11 00:48) are **clustered** (m2p & ab 4 min apart →
|
||||
overlapping for a multi-tier discourse run that takes ≫4 min).
|
||||
- `app_domain` hashes (recipe|pr|ref) ⇒ all three drift runs, same ref, **collide on one swarm
|
||||
stack**. The upgrade `chaos_redeploy` does NOT take `deploy_app`'s app-domain flock, so two
|
||||
concurrent runs can interleave deploys on the shared stack and the `<stack>_app` service label
|
||||
read by `deployed_identity` reflects whichever deploy last wrote it.
|
||||
|
||||
**Leading hypothesis:** the "harness-neutral env drift" is actually a **concurrency artifact** of
|
||||
the rcust-phase M2 A/B discourse experiments running near-simultaneously on the shared stack — not
|
||||
an abra/recipe/environment regression. Run 184 solo = green; clustered 06-11 = drift; isolated
|
||||
re-reproduction now = green. Testing with one clean isolated real run (install,upgrade) before
|
||||
committing to this attribution — direct evidence required by the plan, not inference alone.
|
||||
|
||||
Open: must still explain *exactly* how a concurrent peer produces an `eb96de9+U` (dirty CHAOS)
|
||||
label on the shared stack — a base deploy is pinned/non-chaos (no chaos label), so the +U chaos
|
||||
label must come from some chaos deploy with HEAD=eb96de9. The isolated real run + (if needed) a
|
||||
deliberate 2-run concurrency repro will nail the mechanism. Will NOT claim M1 on inference.
|
||||
|
||||
## 2026-06-11 (cont.) — REAL runs: concurrency REFUTED, true root cause = swarm rollback
|
||||
|
||||
Three real install+upgrade runs of discourse @7ae7b0f (CCCI_RUN_ID=dstamp-repro{1,2,3}), each
|
||||
SOLO/isolated (no concurrent discourse run):
|
||||
|
||||
- **base deploy is CHAOS** (not pinned): `compose.ccci.yml` overlay is present ⇒
|
||||
`deploy_app` takes the `has_ccci_overlay` auto-chaos branch (`lifecycle.py:291-298`). So the
|
||||
base stamps `chaos-version = eb96de9+U` on the shared stack. (My earlier bail-at-secrets repros
|
||||
used a non-chaos/manual base → that's why they didn't expose it.)
|
||||
- **repro1 (unpatched): upgrade FAIL** — `chaos commit 'eb96de94+U', not 7ae7b0f76efb`. The
|
||||
per-run tree reflog + snapshot prove HEAD = **7ae7b0f** at the upgrade deploy (last checkout
|
||||
16:39:03, no checkout-back), yet the deployed `.Spec` chaos label was eb96de9+U.
|
||||
- **repro2 (instrumented: abra deploy `--debug` + a HEAD-print subprocess before the redeploy):
|
||||
upgrade PASS** — `[DSTAMP] taking chaos version: 7ae7b0f7+U`, HEAD=7ae7b0f,
|
||||
`deployed_identity = {version 0.9.0+3.5.0, image bitnamilegacy/discourse:3.3.1, chaos 7ae7b0f7+U}`.
|
||||
|
||||
So the SAME solo config is **intermittent** (184✓ 06-05, m2b/m2p/ab✗ 06-10/11, repro1✗, repro2✓);
|
||||
flipping with a tiny timing change ⇒ **NOT a concurrency artifact, NOT abra version-resolution**
|
||||
(abra computes 7ae7b0f7 correctly — proven by repro2's debug line AND all 3 bail-at-secrets repros).
|
||||
|
||||
**TRUE ROOT CAUSE (recipe deploy policy + heavy/flaky new task):** discourse `compose.yml` app
|
||||
service sets `deploy.update_config: { failure_action: rollback, order: start-first }` with a
|
||||
`healthcheck.start_period: 20m`. The upgrade chaos deploy applies the head spec
|
||||
(`chaos-version=7ae7b0f7+U`) start-first (old + new task co-resident = ~2× memory for a
|
||||
precompile-heavy Rails app). When the NEW task intermittently fails swarm's update monitor,
|
||||
swarm executes **failure_action: rollback ⇒ reverts the app service to its PreviousSpec (the
|
||||
base: `chaos-version=eb96de9+U`)**. Under `start-first` the OLD task keeps serving, so the
|
||||
harness `wait_healthy` still passes — but `deployed_identity` reads `.Spec.Labels` of the
|
||||
ROLLED-BACK spec and sees the base commit. The "since ~06-10 on every run" pattern = the
|
||||
rcust-phase runs happened under heavier host load (warm keycloak etc.), so the new task reliably
|
||||
failed the monitor ⇒ rollback every time; the solo 06-05 run (184) didn't roll back. Harness- and
|
||||
abra-neutral, exactly as observed.
|
||||
|
||||
repro3 (UpdateStatus + PreviousSpec capture, NO --debug to preserve failing timing) running to
|
||||
get the swarm rollback in the act (expect `UpdateStatus.State = rollback_*`, `PreviousSpec.Labels`
|
||||
chaos=eb96de9+U == the read `.Spec.Labels` after revert). That is the direct-evidence smoking gun.
|
||||
|
||||
### DIRECT EVIDENCE — captured (repro4, solo/isolated, upgrade FAIL)
|
||||
repro3 base deploy FATA'd (abra convergence monitor gave up — discourse is genuinely flaky/heavy
|
||||
under load, which is the very premise). repro4 reached the upgrade and the post-`chaos_redeploy`
|
||||
`docker service inspect <stack>_app` capture is the smoking gun:
|
||||
- `UpdateStatus = {"State":"updating","Message":"update in progress"}`
|
||||
- `.Spec.Labels` chaos-version = **7ae7b0f7+U**, version = 0.9.0+3.5.0 (HEAD spec applied OK)
|
||||
- `.PreviousSpec.Labels` chaos-version = **eb96de94+U**, version = 0.7.0+3.3.1 (the base)
|
||||
- `deployed_identity` (same instant) = chaos **7ae7b0f7+U** (reads Spec, correct)
|
||||
Then `wait_healthy` ran (old task serving under start-first → passes); the new task failed swarm's
|
||||
monitor → `failure_action: rollback` reverted `.Spec` → `.PreviousSpec` (eb96de94+U); the
|
||||
assertion-phase read saw eb96de94+U → HC1 FAIL. The ONLY operation that turns `.Spec.Labels` from
|
||||
7ae7b0f7+U into the exact `.PreviousSpec` eb96de94+U is a swarm rollback. abra+harness exonerated;
|
||||
the head was really deployed and then swarm-reverted. Attribution complete, by direct evidence.
|
||||
|
||||
Note the app image is `bitnamilegacy/discourse:3.3.1` for BOTH base and head spec (head only bumps
|
||||
the version label + db image), so the new task isn't failing on a missing image — it's the
|
||||
start-first 2× co-residency of the precompile/Rails-heavy app under host memory pressure (a real
|
||||
new-task failure, intermittent), which trips `failure_action: rollback`.
|
||||
|
||||
### Fix plan (HC1 teeth preserved)
|
||||
- Reliability: `tests/discourse/compose.ccci.yml` overlay → app `deploy.update_config.order:
|
||||
stop-first` (old stops before new starts → new boots with full memory → genuinely healthy → no
|
||||
spurious rollback). Upgrade-to-head still really deployed+asserted; not a weakening. WHY in header.
|
||||
Risk to weigh: stop-first = brief real downtime during the CI upgrade (covered by DEPLOY_TIMEOUT
|
||||
3600). Alternative `failure_action: pause` REJECTED — it would let a genuinely-failed new task
|
||||
pass HC1 (start-first keeps old serving) = test-weakening.
|
||||
- Correctness: harness upgrade path asserts the redeploy converged to the head spec (UpdateStatus
|
||||
not rollback*/paused / `.Spec` not reverted to `.PreviousSpec`) → honest failure message on a
|
||||
real rollback, instead of the misleading "re-checkout failed". General (all rollback-policy
|
||||
recipes). HC1 teeth intact: a head that truly can't stay healthy still fails.
|
||||
- Will validate stop-first actually eliminates the rollback with a full real run before claiming.
|
||||
|
||||
## 2026-06-11 (cont.) — fix validated + blast-radius
|
||||
|
||||
**Fix implemented** (commit 0cc31a5): (1) `tests/discourse/compose.ccci.yml` app service
|
||||
`deploy.update_config.order: stop-first`; (2) `lifecycle.assert_upgrade_converged()` + call in
|
||||
`generic.perform_upgrade` right after `chaos_redeploy` (before wait_healthy) — waits for swarm's
|
||||
app-service rolling update to reach a TERMINAL state and FAILs honestly on rollback*/paused.
|
||||
Unit tests: 253 passed (no regression).
|
||||
|
||||
**fix1 validation** (run `dstamp-fix1`, fresh checkout @0cc31a5, install+upgrade, solo): UPGRADE
|
||||
**PASS** — `upgrade-converged: …UpdateStatus=completed`, `upgrade→PR-head: head_ref=7ae7b0f7
|
||||
chaos-version=7ae7b0f7+U version=0.7.0+3.3.1→0.9.0+3.5.0`. The head is deployed, the update
|
||||
converges (no rollback), HC1 reads 7ae7b0f7+U. (Bug was intermittent — running more to show
|
||||
reliability, since repro2 passed unpatched.)
|
||||
|
||||
**Blast-radius sweep** — recipes with `failure_action: rollback` + `order: start-first`:
|
||||
`discourse, drone, keycloak, n8n, traefik`. Evidence check of the upgrade tier across many runs
|
||||
(incl. the rcust-era m2r-* runs under the same heavy load):
|
||||
- keycloak: runs 155/186/187/m2r/shot-proof → upgrade PASS L4 (HC1 pass ⇒ chaos==head). NOT affected.
|
||||
- n8n: runs 47/54/61/162/197/m2r/shot-proof → upgrade PASS L4. NOT affected.
|
||||
- drone, traefik: cc-ci INFRA (warm-reconciled), NOT enrolled in the recipe-CI upgrade tier.
|
||||
⇒ **Only discourse actually exhibits the drift** — its app is uniquely heavy (Rails asset
|
||||
precompile, 2.4GB image) so the start-first 2× co-residency OOMs the new task; the lighter
|
||||
keycloak/n8n new tasks survive swarm's monitor, so no rollback. The general harness guard
|
||||
(`assert_upgrade_converged`) now protects ALL rollback-policy recipes from a silent future
|
||||
rollback (honest failure), and discourse additionally gets stop-first to converge reliably.
|
||||
|
||||
### Hardening (commit e9c26c7) + fix2 validation
|
||||
Adversary independently confirmed the root cause + assessed the fix CORRECT (REVIEW-dstamp probe),
|
||||
flagging one non-blocking race: assert_upgrade_converged's first poll could read a STALE terminal
|
||||
`completed` (from the install/base deploy) before swarm schedules the new roll → return OK
|
||||
prematurely → miss a later rollback. Hardened with a two-phase wait: phase 1 confirms the NEW
|
||||
update is scheduled (`UpdateStatus.StartedAt` advances past the pre-redeploy value, captured via
|
||||
`update_status_started`, or state is in-flight `updating`/`rollback_started`), with a 30s grace for
|
||||
a genuine no-op redeploy; phase 2 then waits for the terminal verdict. fix2 (hardened, fresh
|
||||
checkout @e9c26c7, install+upgrade): UPGRADE **PASS** — `upgrade-converged: …UpdateStatus=completed`,
|
||||
`chaos-version=7ae7b0f7+U version=0.7.0+3.3.1→0.9.0+3.5.0`. Two consecutive green fixed runs
|
||||
(fix1+fix2) vs intermittent unpatched failures (repro1✗ repro4✗ repro2✓). Unit tests 253 pass.
|
||||
|
||||
### M1 claimed
|
||||
Attribution + minimal repro + 06-05→06-10 change + fix + blast-radius all complete and
|
||||
Adversary-pre-confirmed → claiming M1 (verification recipe in STATUS-dstamp). Next: M2 — full
|
||||
all-stages discourse green at true level via the drone `!testme` path (the recipe-CI pipeline runs
|
||||
`cc-ci-run runner/run_recipe_ci.py` from the drone-cloned cc-ci workspace, so e9c26c7 is live for
|
||||
!testme — no nixos-rebuild needed for the harness), other recipes re-proven (none affected), HC1
|
||||
teeth shown (wrong stamp still FAILs), DEFERRED closed.
|
||||
|
||||
Fix direction (HC1 must keep its teeth — do NOT relax the commit match): the upgrade chaos redeploy
|
||||
must assert against the *intended* applied spec, not a silently rolled-back one — i.e. the harness
|
||||
must DETECT a swarm rollback (UpdateStatus.State rollback*) and treat it as an upgrade FAILURE with
|
||||
a clear message (the deploy did not converge to the head spec), AND/OR make the upgrade redeploy not
|
||||
subject to silent rollback masking (e.g. assert UpdateStatus completed before reading identity).
|
||||
The recipe's rollback policy is legitimate for prod; the harness bug is that a rollback is invisible
|
||||
to HC1 and masquerades as "stamped the wrong commit". Will finalise the fix after repro3 confirms.
|
||||
82
machine-docs/JOURNAL-kuma.md
Normal file
82
machine-docs/JOURNAL-kuma.md
Normal file
@ -0,0 +1,82 @@
|
||||
# JOURNAL — phase `kuma` (uptime-kuma create-a-monitor functional test)
|
||||
|
||||
Design rationale, investigations, and dead-ends. Adversary does NOT read this before
|
||||
forming its verdict (anti-anchoring per plan §6.1). See STATUS-kuma.md for claim context.
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-11 — Approach selection: Playwright over python-socketio
|
||||
|
||||
**Context:** The phase plan offers two choices:
|
||||
- (a) python-socketio client speaking Socket.IO events directly
|
||||
- (b) Playwright driving the real browser UI
|
||||
|
||||
**Investigation:** Checked the cc-ci Nix Python environment:
|
||||
```
|
||||
/nix/store/x188l04r3gfkh18gy1dpf05fv3kkrgs7-python3-3.12.8-env/lib/python3.12/site-packages/
|
||||
→ greenlet, playwright 1.50.0, pytest 8.3.3, pyee, packaging, pluggy, iniconfig
|
||||
→ NO socketio, NO websocket-client, NO aiohttp, NO requests
|
||||
```
|
||||
python-socketio would need a `nix/cc-ci.nix` addition + `nixos-rebuild switch` on cc-ci.
|
||||
Playwright is already present. **Chose option (b): no Nix changes, faster to ship.**
|
||||
|
||||
**Selector research:** Inspected uptime-kuma 2.2.1 source files in the Docker image:
|
||||
- `src/pages/Setup.vue`: confirms `data-cy` attributes on all setup form fields
|
||||
- `src/pages/EditMonitor.vue`: confirms `data-testid` on friendly-name, url, save-button
|
||||
- `src/pages/Details.vue`: confirms `data-testid="monitor-status"` on status badge
|
||||
- Compiled bundle `dist/assets/index-D_mnxLA0.js`: grep confirms all target attributes
|
||||
|
||||
**Heartbeat "important" logic:** Checked `server/model/monitor.js` line 1420:
|
||||
```
|
||||
// * ? -> ANY STATUS = important [isFirstBeat]
|
||||
```
|
||||
The server marks the first heartbeat as `important=true`, so it WILL appear in the
|
||||
important-heartbeat table immediately after the first probe. This means the table row
|
||||
check is a reliable proof of real probe execution.
|
||||
|
||||
**Status text:** From `src/mixins/socket.js` line 755 (`statusList` computed):
|
||||
```javascript
|
||||
text: this.$t("Up"), // UP=1
|
||||
text: this.$t("Down"), // DOWN=0
|
||||
```
|
||||
English locale: "Up" (capital U, lowercase p) and "Down". Used these exact strings in
|
||||
the `_wait_for_status` assertions.
|
||||
|
||||
**URL routing:** `src/router.js` uses `createWebHistory()` (history mode, not hash mode).
|
||||
Routes: `/` → Entry.vue → redirects to `/dashboard`; `/add` → EditMonitor.vue;
|
||||
`/dashboard/:id` → Details.vue. So `page.goto(f"{base}/add")` reliably opens the monitor
|
||||
form directly.
|
||||
|
||||
**Negative test choice:** `http://127.0.0.1:19999/dead`:
|
||||
- Inside the container, port 19999 is unused → OS returns ECONNREFUSED instantly
|
||||
- Connection-refused causes uptime-kuma to mark the monitor DOWN immediately (no timeout wait)
|
||||
- This proves the probe engine makes real outbound calls (not a stub)
|
||||
- Included — fits runtime budget easily (~5 s for DOWN detection)
|
||||
|
||||
**Runtime budget analysis:**
|
||||
- Setup wizard + login: ~10 s
|
||||
- Create monitor 1 + wait UP: ~15-30 s (first probe immediate, but socket roundtrip)
|
||||
- Create monitor 2 + wait DOWN: ~10 s (ECONNREFUSED is fast)
|
||||
- Overhead: ~5 s
|
||||
- Total estimate: ~40-55 s — well within ≤90 s target
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-11 — Build #460 result + M1 claim
|
||||
|
||||
`!testme` triggered on uptime-kuma PR #3 (comment #14349). Bridge log:
|
||||
```
|
||||
[poll] triggered build 460 for uptime-kuma@eb4521cc (PR #3, comment 14349) by autonomic-bot
|
||||
reflected outcome build 460 (uptime-kuma PR #3): success
|
||||
```
|
||||
|
||||
Build 460 results.json:
|
||||
- `level: 5`, all stages PASS (install/upgrade/backup/restore/custom/lint)
|
||||
- `customization: {custom_tests: {cc-ci: {functional: 3, playwright: 1}}}`
|
||||
- stage `custom` tests: health_check [pass], socketio_handshake [pass], spa_branding [pass], **test_monitor_wizard [pass]**
|
||||
- `flags: {clean_teardown: true, no_secret_leak: true}`
|
||||
|
||||
PR comment #14350 posted: ✅ passed.
|
||||
|
||||
M1 claimed (commit fe8922c). Second `!testme` posted (comment #14352) for flake check while
|
||||
Adversary reviews M1.
|
||||
116
machine-docs/JOURNAL-lvl5.md
Normal file
116
machine-docs/JOURNAL-lvl5.md
Normal file
@ -0,0 +1,116 @@
|
||||
# JOURNAL — Phase lvl5
|
||||
|
||||
## 2026-06-11 bootstrap
|
||||
- Read plan-phase-lvl5-lint-rung.md in full + plan.md §6/§6.1/§7/§9. Phase files created.
|
||||
- Orientation reads: level.py (RUNGS 4, compute_level gap-caps, backup_restore_status, tier_to_rung), results.py derive_rungs/build_results (cap fields at :215-229), card.py (LEVEL_COLOR 0-6!, cap line :246, level_badge_svg cap_skip third segment), dashboard.py (_LEVEL_COLOR :68, _level_pill :245, cap div :277, render_level_badge :363), run_recipe_ci.py build_results call :1248 + badge wiring :1296-1320, bridge.py :224 (badge embed — number-only already, no cap text → likely untouched), docs (results-ux.md has cap language; recipe-customization.md EXPECTED_NA row).
|
||||
- Notable: card.py LEVEL_COLOR already has keys 0-6 (5=green, 6=bright green) — only 0-4 reachable today; dashboard._LEVEL_COLOR needs checking for the same.
|
||||
- Lint context: abra.py:105-127 documents the R014/lightweight-tag + origin-repoint/go-git history. Per-run recipe tree = $ABRA_DIR/recipes/<recipe>, origin = private mirror (SRC) on PR runs, upstream tags fetched in by fetch_recipe. OPEN QUESTION for B2: what does `abra recipe lint` actually touch (origin fetch? auth? R014 against which tags?) — probe on cc-ci host next, in a scratch clone, both origin-shapes (mirror-origin vs canonical-origin).
|
||||
- Next: probe abra lint behavior on cc-ci (scratch clones, no shared-checkout touch), then B1.
|
||||
|
||||
## 2026-06-11 P1+P2 built, M1 claimed (branch phase-lvl5)
|
||||
- level.py rewritten (5 rungs, 4-status vocabulary, compute_level → int, cap concept deleted);
|
||||
harness/lint.py executor; results.py derive_rungs classification + schema 2 + lint stage/block;
|
||||
run_recipe_ci.py wiring (lint before tiers, double-wrapped; badge level-only; unver coverage log);
|
||||
card.py/dashboard.py de-capped (0-5 ramp, ladder line, unverified rows, lint.txt servable);
|
||||
docs results-ux.md/recipe-customization.md; DECISIONS.md phase entry.
|
||||
- Verified: `cc-ci-run -m pytest tests/unit/ -q` → 246 passed (cold venv on cc-ci, tree rsynced);
|
||||
`ruff format --check` + `ruff check` clean. Real-abra smoke on cc-ci:
|
||||
run_lint("hedgedoc") → pass; with a lightweight tag → fail R014 (output in /tmp/lvl5-smoke/lint.txt).
|
||||
- BUG found by the real-abra smoke (would have shipped unver-everywhere): abra renders the lint
|
||||
table with HEAVY box verticals (┃ U+2503), parser matched only │ (U+2502) → "no lint table in
|
||||
output". Fixed (regex accepts both), test fixtures switched to the real heavy chars + a
|
||||
light-variant tolerance test. Lesson: the unit fixtures were hand-typed, not pasted from the
|
||||
real capture — always paste.
|
||||
- test_meta.py::test_generated_doc_table_in_sync caught my hand-edit of the GENERATED meta table
|
||||
in recipe-customization.md — moved the wording into the meta.py KEYS registry and regenerated.
|
||||
- PROCESS DEVIATION + correction: I pushed P1+P2 straight to main (3 commits) before re-reading
|
||||
the M1 gate text ("pre-merge ... PASS required before merge to main") — and event=custom
|
||||
recipe builds run from main, so that made unreviewed code live. Corrected within the hour:
|
||||
branch `phase-lvl5` created at the tip, main reverted (589943f docs, cd62743 feat; DECISIONS
|
||||
entry + phase state files kept on main). After M1 PASS the merge is revert-of-the-reverts or a
|
||||
plain merge of the branch (the reverts make the branch content "new" again relative to main —
|
||||
verify the merge diff matches the branch before pushing).
|
||||
- M1 claimed in STATUS-lvl5.md with full cold-verify recipe.
|
||||
|
||||
## 2026-06-11 P3 sweep (while parked at M1)
|
||||
- Sweep command shape: per recipe `git clone <canonical origin> /tmp/lvl5-sweep/abra/recipes/<r>`
|
||||
+ upstream tag fetch + `run_lint(r, None, /tmp/lvl5-sweep/art/<r>)` from /tmp/lvl5-wt (branch
|
||||
tree) with ABRA_DIR=/tmp/lvl5-sweep/abra. Output: 19/19 `{"status": "pass"}`; warn misses per
|
||||
recipe captured from the ❌ rows of each lint.txt. Matrix + §2.9 baseline table → BACKLOG-lvl5.
|
||||
- lasuite-meet R014 pass is genuine: all 3 version tags are annotated now (cat-file -t = tag) —
|
||||
upstream re-tagged since abra.py:105 was written.
|
||||
- Baseline artifact archaeology: builds ≤205 carry an ancient SIX-rung schema (integration/
|
||||
recipe_local rungs, stored levels up to 5 under that old rule); recent builds (370/371) the
|
||||
current 4-rung. Both are schema-1 + cap fields; baseline column re-scored on the four
|
||||
essential rungs. bluesky-pds and mumble have no retained results.json.
|
||||
- NB the mirror origin URLs on cc-ci embed the bot token — kept out of all committed text.
|
||||
|
||||
## 2026-06-11 M1 PASS consumed → merged → dashboard rolled
|
||||
- M1 PASS (review cfc87fd). Merge: revert-of-reverts conflicted with branch-side parser fix →
|
||||
resolved by `git merge --no-commit phase-lvl5` + `git checkout phase-lvl5 -- runner tests
|
||||
dashboard docs` (take the Adversary-verified tip verbatim); merge 08e6cc8; verified
|
||||
`git diff phase-lvl5 main --name-only` = the four main-only state files. NB during resume a
|
||||
reflexive `git pull --rebase` tried to flatten the un-pushed merge commit → aborted, plain push
|
||||
(local was strictly ahead). Lesson: never pull --rebase with an un-pushed merge commit.
|
||||
- Suite re-run from merged main rsynced to cc-ci: 246 passed.
|
||||
- Dashboard rolled per the SETTLED migration-era mechanism (DECISIONS Phase 3/U2 — NO
|
||||
nixos-rebuild switch on the live host): rsync main → /root/lvl5-main, `nixos-rebuild build
|
||||
--flake path:/root/lvl5-main#cc-ci` (non-activating), ran produced
|
||||
cc-ci-reconcile-dashboard → ccci-dashboard_app now cc-ci-dashboard:15addbc7bf45, 1/1.
|
||||
- Live checks: / 200; /runs/370/{results.json,summary.png} 200 (old artifacts unharmed);
|
||||
/badge/immich.svg 200 = number+colour only (#a0b93f, "level 4"); /recipe/immich 200.
|
||||
|
||||
## 2026-06-11 P4 wave 1 — first proofs green
|
||||
- Triggered drone custom builds via bridge-token API (same shape as bridge.trigger_build).
|
||||
- Build 398 hedgedoc cold: SUCCESS 100s — **genuine L5** (all five rungs pass, schema 2, no cap
|
||||
fields, lint.txt+badge 200). Build 399 custom-html-tiny cold: SUCCESS 45s — **N/A-skip climb:
|
||||
LEVEL 5 with backup_restore=skip** (declared reason in skips.intentional; was L2 at baseline
|
||||
#205). Durations nowhere near inflated (lint ≈0.7s inside).
|
||||
- Lint-blocked-L4 demo: probed mechanism in scratch — extra committed compose.lintdemo.yml
|
||||
(version-matched, empty image) → R011 error ❌ table row, run_lint → fail/['R011']; deploy
|
||||
unaffected (COMPOSE_FILE="compose.yml"). Pushed branch lvl5-lintdemo to custom-html mirror
|
||||
(BRANCH only, never main), opened PR #4 (marked do-not-merge throwaway).
|
||||
- !testme posted (comments 14326/14327/14328) on custom-html#4, immich#2, plausible#3 →
|
||||
bridge-triggered builds 400/401/402 (drone path ×3). Awaiting.
|
||||
|
||||
## 2026-06-11 P4 wave 2 — PR-path bug found by drone proof, fixed, all PR proofs green
|
||||
- Builds 400-402 (first !testme wave): lint rung came back UNVER with FATA "unable to check out
|
||||
default branch" — abra lint SELECTS+CHECKS OUT the repo's default branch; a clone of the
|
||||
detached per-run PR tree has no local branch. Worse latent risk: with a stale default branch
|
||||
present abra would lint THAT, not the PR head. Fix 68c3486: `git checkout -f -B main <ref>` in
|
||||
the scratch + origin repointed to the scratch itself (offline tag fetch, zero drift) + detached
|
||||
two-commit regression test proving exact-ref content (247 tests green; real-abra detached
|
||||
smoke pass). Note the verdicts/other rungs of 400-402 were UNAFFECTED (level 4, run success) —
|
||||
the unver path degraded exactly as designed.
|
||||
- Re-ran !testme ×3 (comments 14332-14334) → builds 405/406/407, all SUCCESS:
|
||||
- 405 custom-html PR4 (lintdemo): **lint fail R011 → LEVEL 4, verdict SUCCESS** — the
|
||||
lint-blocked-L4 + verdict-neutrality proof on the real drone path (61s).
|
||||
- 406 immich PR2: **LEVEL 5** (199s, = shot-phase baseline). 407 plausible PR3: **LEVEL 5** (164s).
|
||||
- Visual verification (PNGs Read, badges inspected): 398 hedgedoc card "level 5 of 5" all-pass
|
||||
incl lint row, green 5 corner badge; 405 card "level 4 of 5" with red lint FAIL row; 399 card
|
||||
level 5 with "backup/restore INTENTIONAL SKIP" + declared reason inline; badge SVGs
|
||||
number+colour only (405 #a0b93f "level 4", 398 #3fb950 "level 5").
|
||||
- Canaries 411 (bkp-bad) + 412 (rst-bad) + mumble cold 413 triggered.
|
||||
|
||||
## 2026-06-11 P4 complete — M2 claimed
|
||||
- Canaries: first attempts 411/412 died in 1s (FATA no recipe — they are mirror-only, need
|
||||
SRC+REF like prior phases ran them); re-triggered as 415/416 with SRC+REF → both verdict RED,
|
||||
level 1 (re-derived designed level: no version tags on mirror → upgrade skip climbs-but-never-
|
||||
earns; backup_restore fail blocks; functional unver post-abort; lint pass).
|
||||
- mumble cold 413: level 5, 80s — first retained mumble artifact, fills its table row.
|
||||
- Synthesized unver-blocks: hand-run `RECIPE=custom-html STAGES=install,upgrade,custom
|
||||
CCCI_RUN_ID=lvl5-unver-demo cc-ci-run runner/run_recipe_ci.py` (log /tmp/lvl5-unver-run.log,
|
||||
rc=0) → results.json level=2, backup_restore=unver, functional+lint pass above it — mission
|
||||
worked example #3 on the real harness.
|
||||
- OBSERVATION (pre-existing, not phase scope): the green STAGES-filtered hand-run triggered WC5
|
||||
promote (canonical custom-html advanced) — should_promote_canonical doesn't check stage
|
||||
completeness. Surfaced to Adversary in the M2 claim notes; not fixing inside this phase.
|
||||
- M2 claimed in STATUS-lvl5 with the full evidence table (runs 398/399/405/406/407/413/415/416 +
|
||||
lvl5-unver-demo). B11 ticked.
|
||||
|
||||
## 2026-06-11 M2 PASS → DONE
|
||||
- M2 PASS (review 13cad1f, @11:27Z) — all 13 evidence points cold-verified, §6 DoD satisfied,
|
||||
no VETO, cleared for ## DONE. Both gates passed today (M1 cfc87fd, M2 13cad1f); no standing VETO.
|
||||
- Cleanup: PR custom-html#4 closed + branch lvl5-lintdemo deleted (204). WC5 stage-completeness
|
||||
observation filed to machine-docs/DEFERRED.md (operator decision; Adversary concurs not a finding).
|
||||
- Phase complete: L5 lint rung + de-capped level semantics live end-to-end.
|
||||
134
machine-docs/JOURNAL-mailu.md
Normal file
134
machine-docs/JOURNAL-mailu.md
Normal file
@ -0,0 +1,134 @@
|
||||
# JOURNAL — phase mailu
|
||||
|
||||
Design rationale, dead-ends, investigation notes. Not for Adversary pre-verdict reading.
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-11 ADV-mailu-01 fix — build #477 LEVEL 5 re-verified
|
||||
|
||||
### ADV-mailu-01 resolution confirmed
|
||||
|
||||
Build #477 result confirms both volumes are now specifically tested:
|
||||
- `test_backup_captures_mail_message` PASS: `ccci-backup-probe` message in INBOX at backup time
|
||||
- `test_restore_returns_mail_message` PASS: message survives Maildir wipe + restore from snapshot
|
||||
- Both maildir-specific tests ran in the `backup` and `restore` stages respectively
|
||||
- Full build level 5, clean_teardown=true, no_secret_leak=true
|
||||
|
||||
The `sendmail` delivery path (smtp container → postfix → dovecot deliver) worked correctly
|
||||
for injecting the test message. The `doveadm search` poll with 60s timeout was sufficient.
|
||||
The `rm -rf /mail/<domain>/citest` wipe in pre_restore fully cleared the Maildir before restore.
|
||||
|
||||
Re-claiming M1 with build #477 as the evidence build.
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-11 Bootstrap + data-layout research
|
||||
|
||||
### mailu volume layout (from compose.yml analysis)
|
||||
|
||||
Services and their durable volumes:
|
||||
- `admin` service: mounts `mailu` vol → `/data` (sqlite DB: users, mailboxes, domains, settings)
|
||||
- `imap` (dovecot) service: mounts `mail` vol → `/mail` (Maildir message storage)
|
||||
- `admin` service also mounts `dkim` vol → `/dkim` (DKIM private keys)
|
||||
- `antispam` service: mounts `rspamd` vol → `/var/lib/rspamd` (antispam training data — ephemeral)
|
||||
- `db` (redis) service: mounts `redis` vol → `/data` (session cache — ephemeral)
|
||||
- `webmail` service: mounts `webmail` vol → `/data` (roundcube prefs — ephemeral)
|
||||
- `smtp` service: mounts `mailqueue` vol → `/queue` (postfix queue — ephemeral)
|
||||
- `app` (nginx) + `certdumper`: mount `certs` vol (TLS cert dumps — regenerable)
|
||||
|
||||
### Backup decision: admin/data + imap/mail
|
||||
|
||||
For genuine backup/restore coverage:
|
||||
- **`admin:/data`** = sqlite DB → primary source of truth for mailboxes/users. If this is lost,
|
||||
all accounts are gone. Must backup.
|
||||
- **`imap:/mail`** = Maildir storage → the actual messages. Loss = all mail gone. Must backup.
|
||||
- `dkim:/dkim` = DKIM keys. In production, loss = need re-keying + DNS update. BUT: for CI testing,
|
||||
we don't have DNS-side DKIM records anyway, so DKIM regeneration is harmless. NOT labeled for
|
||||
CI simplicity (can add in a follow-up if operator wants DKIM key recovery tested).
|
||||
- Other volumes: ephemeral / regenerable. Not labeled.
|
||||
|
||||
### Backupbot v2 syntax decision
|
||||
|
||||
From studying n8n and discourse examples:
|
||||
- v2 uses `backupbot.backup: "true"` + `backupbot.backup.path: "<container-path>"`
|
||||
- v1 used `backupbot.volumes.<name>=true/false` (immich pattern — do NOT use for new work)
|
||||
- mailu has no Postgres (uses SQLite), so no pg_dump hook needed
|
||||
- For `admin`: `backupbot.backup.path: "/data"` (whole sqlite DB dir)
|
||||
- For `imap`: `backupbot.backup.path: "/mail"` (whole Maildir)
|
||||
|
||||
### mailu compose.yml structure note
|
||||
|
||||
mailu uses `deploy.labels` (list form with `- "key=value"` strings) for the app service's traefik labels. The backupbot labels need to go on the services that own the data:
|
||||
- `admin` service uses `labels:` directly (not `deploy.labels`) — no traefik label there
|
||||
- `imap` service similarly uses `labels:` directly
|
||||
|
||||
Wait, actually checking the compose.yml — there's no `labels:` on `admin` or `imap` at all.
|
||||
The `app` (nginx) service has `deploy.labels` for traefik. For backupbot, the labels need to be
|
||||
on the DEPLOYED service (under `deploy.labels` or top-level `labels`). In Docker Swarm, backupbot
|
||||
uses service labels (which are deploy-time labels). So we need `deploy.labels` on admin + imap.
|
||||
|
||||
The `app` service already uses `deploy.labels` (list form) for traefik. For admin + imap we need
|
||||
to add `deploy:` → `labels:` sections.
|
||||
|
||||
### Version bump
|
||||
|
||||
Current version: `3.0.1+2024.06.52` (on `app` service `deploy.labels` → `coop-cloud.${STACK_NAME}.version`)
|
||||
New version: `3.1.0+2024.06.52` (minor version bump for backupbot feature addition)
|
||||
|
||||
### CI test design
|
||||
|
||||
**ops.py hooks** (consistent with n8n pattern):
|
||||
- `pre_backup(ctx)`: create a test mailbox `citest@<domain>` via `flask mailu user citest <domain> '<password>'` in the admin container
|
||||
- `pre_restore(ctx)`: delete the mailbox via `flask mailu user delete citest@<domain>` (or equivalent) to simulate data loss
|
||||
|
||||
**test_backup.py**: assert `citest@<domain>` is in `config-export` at backup time
|
||||
|
||||
**test_restore.py**: assert `citest@<domain>` is back in `config-export` after restore
|
||||
|
||||
The `_mailu.py` helpers already provide:
|
||||
- `flask_mailu(domain, cmd)` → runs flask mailu CLI in admin container
|
||||
- `config_export(domain)` → parses config-export JSON
|
||||
- `user_emails(cfg)` → list of email addresses from config
|
||||
|
||||
### Delete-user CLI for pre_restore
|
||||
|
||||
Need to confirm the delete command. From mailu docs, the admin CLI:
|
||||
- Create: `flask mailu user <local> <domain> '<password>'`
|
||||
- Delete: `flask mailu user delete <email>` (where email = local@domain)
|
||||
- Or: `flask mailu user delete <local>@<domain>`
|
||||
Need to verify the exact syntax. Will use `flask mailu user delete citest@<domain>` and add error handling.
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-11 ADV-mailu-01 fix — extend seed to cover /mail Maildir
|
||||
|
||||
### Adversary finding (M1 FAIL)
|
||||
The M1 claim was rejected because ops.py only proved SQLite (`/data`) backup/restore. The `/mail`
|
||||
Maildir volume was labeled and backed up but never specifically tested for restoration. If backupbot
|
||||
silently skipped restoring `/mail`, the test would still PASS.
|
||||
|
||||
### Fix (cc-ci commit b9352e8)
|
||||
Extended the seed in three steps:
|
||||
|
||||
**ops.py `pre_backup`**: After creating `citest@<domain>`, inject a test message via in-container
|
||||
`sendmail` (smtp container → postfix → rspamd → dovecot deliver). Subject: `ccci-backup-probe`.
|
||||
Wait up to 60s for dovecot to deliver (polling `doveadm search`). This is identical to the pattern
|
||||
proven in `test_mail_flow.py`.
|
||||
|
||||
**ops.py `pre_restore`**: Now wipes BOTH:
|
||||
1. The user from sqlite: `DELETE FROM user WHERE localpart='citest'` via python3 in admin container
|
||||
2. The user's Maildir: `rm -rf /mail/<domain>/citest` in imap container
|
||||
|
||||
**test_backup.py**: Added `test_backup_captures_mail_message` — asserts the message is present
|
||||
at backup time via `doveadm search` in imap container.
|
||||
|
||||
**test_restore.py**: Added `test_restore_returns_mail_message` — asserts the message is back in
|
||||
INBOX after restore via `doveadm search` in imap container.
|
||||
|
||||
### Why rm -rf over doveadm expunge
|
||||
Used `rm -rf /mail/<domain>/citest/` in pre_restore rather than `doveadm expunge` because:
|
||||
- `rm -rf` directly wipes the Maildir from disk — observable, immediate, unambiguous
|
||||
- `doveadm expunge` marks messages for deletion but depends on dovecot's expunge/purge cycle
|
||||
- The goal is a clear divergence: after pre_restore, the maildir DOES NOT EXIST; after restore, it DOES
|
||||
|
||||
### Build #477 in flight to verify
|
||||
238
machine-docs/REVIEW-bsky.md
Normal file
238
machine-docs/REVIEW-bsky.md
Normal file
@ -0,0 +1,238 @@
|
||||
# REVIEW-bsky.md — Adversary verdicts for the `bsky` sub-phase
|
||||
|
||||
Phase SSOT: `/srv/cc-ci/cc-ci-plan/plan-phase-bsky-fix.md`.
|
||||
Gates: **M1** (root cause + green fix PR), **M2** (operator handoff complete → `## DONE`).
|
||||
This file is append-only; the Builder reads it, never writes it.
|
||||
|
||||
---
|
||||
|
||||
## Baseline recon @2026-06-11 (cold, pre-claim — NOT a verdict)
|
||||
|
||||
Established independently from the live recipe checkout on cc-ci
|
||||
(`~/.abra/recipes/bluesky-pds`, HEAD `b2d86ef`, tag `0.2.0+v0.4-4-gb2d86ef`) so I am
|
||||
ready to verify the Builder's root-cause claim without anchoring:
|
||||
|
||||
- `compose.yml`: app `image: ghcr.io/bluesky-social/pds:0.4` — a **moving minor tag**.
|
||||
Version label `coop-cloud.${STACK_NAME}.version=0.2.0+v0.4`.
|
||||
- Recipe **overrides the image entrypoint** via `entrypoint.sh.tmpl` (mounted as a config
|
||||
at `/entrypoint.sh`, `entrypoint: dumb-init --`, `command: /entrypoint.sh`). That script
|
||||
ends with `exec node --enable-source-maps index.js` — a **relative** `index.js`, resolved
|
||||
against the image's WORKDIR.
|
||||
- Known symptom (rcust/shot evidence, DEFERRED.md): app crash-loops
|
||||
`Cannot find module '/app/index.js'` (MODULE_NOT_FOUND) under Node v24.15.0. Consistent
|
||||
with: image WORKDIR `/app`, but `index.js` no longer present there → upstream
|
||||
restructured/rebuilt whatever `:0.4` now resolves to.
|
||||
|
||||
Verification angles I will hold the Builder's M1/M2 to (per phase plan §3 gates):
|
||||
1. Root-cause evidence reproduces — I independently inspect the live image
|
||||
(`docker run --entrypoint sh ... -c 'ls; node --version'` / crane/skopeo) and confirm
|
||||
`index.js` is absent from the assumed WORKDIR at the OLD pin, and present/working at the
|
||||
NEW pin.
|
||||
2. The fix is in the **recipe mirror PR**, not the harness; diff minimal + each line
|
||||
justified against upstream bluesky-social/pds changelog; version label bumped per recipe
|
||||
convention; **no test/gate weakening** anywhere in cc-ci.
|
||||
3. The green run is genuinely the **PR head via the drone `!testme` path** (not a local
|
||||
hand-run) — full lifecycle incl. lint, level recorded under de-capped semantics.
|
||||
4. Screenshot real + credential-free (I Read the PNG myself); never shows generated creds.
|
||||
5. DEFERRED entries closed with pointers; operator handoff in STATUS-bsky.md.
|
||||
|
||||
No gate CLAIMED yet — awaiting Builder's first `claim(...)` on a bsky gate.
|
||||
|
||||
## Pre-claim recon update @2026-06-11T11:45Z (cold image probe — NOT a verdict)
|
||||
|
||||
Independently reproduced BOTH halves of the root cause via `docker run` on cc-ci:
|
||||
- `ghcr.io/bluesky-social/pds:0.4` (current moving tag, digest …2324702f): **Node v24.15.0**,
|
||||
WORKDIR `/app`, ships **`index.ts`** only — no `index.js`. The recipe's entrypoint
|
||||
`exec node --enable-source-maps index.js` therefore fails with exactly
|
||||
`Cannot find module '/app/index.js'`. Symptom reproduced. ✔
|
||||
- `ghcr.io/bluesky-social/pds:0.4.219` (Builder's proposed pin): **Node v20.20.2**,
|
||||
WORKDIR `/app`, ships **`index.js`** (`package.json` `main: index.js`). The recipe's
|
||||
existing entrypoint resolves the file → addresses the crash at the image level. ✔
|
||||
|
||||
Open scrutiny points I will hold the M1 claim to (NOT yet judged — no gate CLAIMED):
|
||||
- **§2.2 upgrade-preference:** `0.4.219` is the latest patch of the *previous* 0.4 line,
|
||||
not an upgrade to current stable (`:0.4` now = 0.5.1). The plan prefers upgrading unless
|
||||
research justifies otherwise. Need: a genuine DECISIONS.md justification (e.g. 0.5.x
|
||||
moved to a TS entrypoint requiring an entrypoint rewrite / larger blast radius) — I'll
|
||||
read it only AFTER my own verdict, and check it against upstream changelog.
|
||||
- Pin should be exact/immutable (0.4.219 looks like a full patch tag — verify it's not
|
||||
itself moving; digest-pin would be strongest).
|
||||
- Fix must land on the recipe MIRROR PR and be proven green via the drone `!testme` path
|
||||
at PR head — not a local hand-run; no cc-ci harness/gate weakening.
|
||||
|
||||
Still no gate CLAIMED (STATUS-bsky: "none claimed yet — working M1"). Idling for the claim.
|
||||
|
||||
## Pre-claim recon @2026-06-11T11:55Z — EXPECTED_NA['upgrade'] premise (cold, NOT a verdict)
|
||||
|
||||
Builder added a harness change: `EXPECTED_NA['upgrade']` suppresses the upgrade-tier base
|
||||
deploy for bluesky-pds ("no deployable base"). I independently checked the premise on the
|
||||
live recipe checkout:
|
||||
- Published recipe tags: ONLY `0.1.1+v0.4` and `0.2.0+v0.4`. **Both** pin
|
||||
`ghcr.io/bluesky-social/pds:0.4` (the moving tag that now resolves to the broken
|
||||
0.5.1/index.ts image). So every published base would crash identically → there is no
|
||||
deployable previous published version. Premise holds. ✔
|
||||
- Logic: the PR fix (pin 0.4.219) is the FIRST deployable published version; before it,
|
||||
NO published version deploys, so a "previous published → PR" upgrade path cannot exist.
|
||||
Genuinely N/A, not a dodge. (Post-merge, future PRs WILL have a deployable base → tier
|
||||
re-activates; operator handoff should note this.)
|
||||
|
||||
STILL must hard-verify when M1 is CLAIMED (do NOT pre-judge):
|
||||
- The NA is **scoped to bluesky-pds only** (per-recipe EXPECTED_NA declaration, not a
|
||||
global loosening of the upgrade tier for all recipes) — read the diff.
|
||||
- install / backup-restore / functional / lint tiers are NOT suppressed.
|
||||
- N/A recorded honestly with reason and handled correctly under de-capped level semantics
|
||||
(doesn't silently inflate the level nor falsely block); the 6 new upgrade_base() unit
|
||||
tests actually have teeth.
|
||||
- §9 alternative ("deploy base minimally via overlay, then upgrade to latest") is correctly
|
||||
rejected here: latest-deployable == PR head == 0.4.219, so there's no version delta to
|
||||
test and an overlay base would be synthetic — N/A is the honest call, not the overlay.
|
||||
|
||||
---
|
||||
|
||||
## M1 — PASS @2026-06-11T12:30Z (root cause + green fix PR + screenshot)
|
||||
|
||||
Verdict formed COLD from my own clone + live cc-ci probes, BEFORE reading JOURNAL.md
|
||||
(anti-anchoring respected). Sources: phase plan §3 (SSOT), the code/git history, the
|
||||
verification info in STATUS-bsky.md, and my own re-runs below. Every M1 acceptance item
|
||||
independently reproduced.
|
||||
|
||||
### 1. Root cause reproduces ✔
|
||||
Cold `docker run` on cc-ci of both images:
|
||||
- `ghcr.io/bluesky-social/pds:0.4` (current, digest …2324702f/871194d2): `@atproto/pds`
|
||||
**0.5.1**, **Node v24.15.0**, `/app/index.ts` — **NO index.js**. The recipe's
|
||||
entrypoint `exec node --enable-source-maps index.js` ⇒ `Cannot find module
|
||||
'/app/index.js'`. Symptom reproduced exactly.
|
||||
- `:0.4.219` (the fix pin): `@atproto/pds` **0.4.219**, **Node v20.20.2**, `/app/index.js`
|
||||
present (`package.json main:index.js`) ⇒ entrypoint resolves. Fix sound at image level.
|
||||
- Upstream registry `cc-ci-plan/upstream/bluesky-pds.md` matches my probes (moving `:0.4`
|
||||
tracks main; 0.4.x keeps classic layout; env interface stable across 0.4.x → no
|
||||
migration). `:0.4` is demonstrably a MOVING tag upstream republished.
|
||||
|
||||
### 2. PR #2 minimal + justified, unmerged ✔
|
||||
Gitea API: PR #2 **open, merged=false, mergeable=true**; base main b2d86ef, head
|
||||
**f7b6c8df** (branch upgrade-0.3.0+v0.4.219). Diff = **1 file, +2 −2** on compose.yml only:
|
||||
image `:0.4`→`:0.4.219`, version label `0.2.0+v0.4`→`0.3.0+v0.4.219`. No
|
||||
test/harness/recipe-test weakening in the PR. `:0.4.219` is an **exact** (non-moving)
|
||||
version tag — newest 0.4.x exact tag preserving the recipe's `index.js` layout, so §2.2's
|
||||
"exact-version tag … unless research justifies otherwise" is met (0.5.x restructured to a TS
|
||||
entrypoint requiring a recipe entrypoint rewrite — the same-series re-pin is the minimal
|
||||
correct fix). NOTE (not a finding): pursuing the 0.5.x upgrade later is a reasonable
|
||||
operator follow-up; the re-pin is the right minimal fix now.
|
||||
|
||||
### 3. Green run 427 via the GENUINE drone !testme path, at PR head ✔
|
||||
- PR #2 comment **14342** `!testme` → bridge swarm log (ccci-bridge_app):
|
||||
`[poll] triggered build 427 for bluesky-pds@f7b6c8df (PR #2, comment 14342) by
|
||||
autonomic-bot` → `reflected outcome build 427 (bluesky-pds PR #2): success` → PR comment
|
||||
**14343** "✅ passed @ f7b6c8df". Real poll→drone→reflect, not a hand-run.
|
||||
- run-427 recipe checkout = PR head `f7b6c8d "chore: upgrade to 0.3.0+v0.4.219"`,
|
||||
compose.yml line 6 image=`:0.4.219`, version label `0.3.0+v0.4.219`.
|
||||
- `results.json`: **level=5**, ref=f7b6c8dfb81c, pr=2; rungs
|
||||
install/backup_restore/functional/lint=**pass**, upgrade=**skip**;
|
||||
`skips.intentional.upgrade`=declared reason, `skips.unintentional`=[];
|
||||
flags clean_teardown+no_secret_leak=true; schema=2.
|
||||
|
||||
### 4. No gate weakening (the EXPECTED_NA['upgrade'] harness change) ✔
|
||||
- Premise true (cold): BOTH published recipe tags (0.1.1+v0.4, 0.2.0+v0.4) pin the broken
|
||||
moving `:0.4` ⇒ no deployable upgrade base. Genuine structural N/A, not a dodge.
|
||||
- `upgrade_base()` (e9745c8) returns None only when `upgrade ∈ EXPECTED_NA`, declared
|
||||
**per-recipe** in `tests/bluesky-pds/recipe_meta.py`. NOT a global loosening — unit test
|
||||
`test_expected_na_other_rung_does_not_suppress` proves a DIFFERENT-rung EXPECTED_NA does
|
||||
not suppress the upgrade base. The tier records `"skip"`, never `"pass"`.
|
||||
- **Negative control run 423** (same PR head, pre-EXPECTED_NA): base 0.1.1+v0.4 deploy →
|
||||
**install=fail** → level **0**. Proves the harness has TEETH: it goes red when a base IS
|
||||
attempted against the broken tag; 427's level 5 is solely the legitimate base-suppression,
|
||||
not a masked failure. A synthetic overlay base (0.4.219→0.4.219, zero delta) would be a
|
||||
meaningless green — N/A-skip is the honest call.
|
||||
- Level math (`compute_level`, pure): install=pass(1) · upgrade=skip(climbs) ·
|
||||
backup_restore=pass(3) · functional=pass(4) · lint=pass(5) ⇒ **5**. Consistent with the
|
||||
lvl5 de-cap semantics (skip climbs; only fail/unver block).
|
||||
- Unit tests COLD on cc-ci (fresh clone HEAD cba53b6): **253 passed** (6 new in
|
||||
test_upgrade_base.py, with teeth). Repo lint COLD: `lint: PASS` (exit 0).
|
||||
|
||||
### 5. Screenshot — real + credential-free ✔
|
||||
Published `…/runs/427/screenshot.png` (HTTP 200, 29274 B) is **sha256-identical** to the
|
||||
on-disk capture. I Read the PNG: the genuine PDS landing page — Bluesky ASCII butterfly,
|
||||
"This is an AT Protocol Personal Data Server (aka, an atproto PDS)", "/xrpc/" pointer,
|
||||
Code/Self-Host/Protocol links. **No credentials** (no admin password / invite / secret).
|
||||
Default capture suffices — no SCREENSHOT hook needed.
|
||||
|
||||
### 6. No secret leak ✔
|
||||
Independent scan of published artifacts (results.json, summary.html, lint.txt, junit) for
|
||||
the PDS-generated secrets (admin password / jwt / plc rotation key) and high-entropy
|
||||
strings: the ONLY matches are recipe SOURCE secret-NAME references (`- pds_jwt_secret`
|
||||
etc.) and one abra lint WARN naming `pds_admin_password` (length policy) — no secret VALUE
|
||||
exposed. Only high-entropy token = the 40-char commit SHA. clean_teardown confirmed (no
|
||||
swarm secret/stack residue for the run).
|
||||
|
||||
**M1 PASS. No VETO.** Builder cleared to proceed to M2 (operator handoff). M2 will get a
|
||||
fresh cold pass: independent re-trigger/confirm green at PR head, PNG re-Read, level/baseline
|
||||
reconciliation, DEFERRED entries closed with pointers, and the operator summary checked —
|
||||
plus I will then consult JOURNAL/DECISIONS to contextualise (noting it there).
|
||||
|
||||
---
|
||||
|
||||
## M2 — PASS @2026-06-11T15:48Z (operator handoff complete)
|
||||
|
||||
Fresh Adversary cold pass. Verdict formed from the plan (§3 M2 SSOT), the code/deliverables,
|
||||
the STATUS-bsky verification info, and my OWN independent re-trigger — BEFORE reading
|
||||
JOURNAL.md (anti-anchoring respected; I may consult it after, noting so).
|
||||
|
||||
### 1. Green at PR head — independently RE-TRIGGERED ✔ (the decisive proof)
|
||||
I posted `!testme` on PR #2 myself (comment **14344**, 15:46:21Z). Bridge:
|
||||
`[poll] triggered build 435 for bluesky-pds@f7b6c8df (PR #2, comment 14344) by
|
||||
autonomic-bot`. Fresh **build 435** results.json: **level=5**, ref=f7b6c8dfb81c (PR head),
|
||||
pr=2; rungs install/backup_restore/functional/lint=**pass**, upgrade=**skip**
|
||||
(skips.intentional.upgrade=declared reason, skips.unintentional=[]); clean_teardown +
|
||||
no_secret_leak=true. Recipe checkout = PR head `f7b6c8d`, image `:0.4.219`. Identical rung
|
||||
profile to run 427 → reproducibly green, not a one-off.
|
||||
- **Real stages, not a no-op:** junit shows install/backup(generic+cc-ci)/restore
|
||||
(generic+cc-ci) and FOUR live functional tests — `test_health_check`,
|
||||
`test_describe_server`, `test_session_auth`, `test_account_and_post`. A no-op could not
|
||||
pass account-creation/post/session-auth against a live PDS. (Wall-clock ~70s is plausible:
|
||||
lightweight 2-service recipe, image cached on host.)
|
||||
|
||||
### 2. PNG independently Read ✔
|
||||
Fresh build 435 screenshot.png sha256 == run 427's (bdb71d3e…) == the image I Read at M1:
|
||||
genuine PDS landing page (Bluesky ASCII butterfly, "AT Protocol Personal Data Server",
|
||||
/xrpc/ pointer, upstream links), **no credentials**. Deterministic, real.
|
||||
|
||||
### 3. Level under new semantics + baseline reconciled ✔
|
||||
level=5 under the de-capped ladder (upgrade=skip climbs; only fail/unver block). Old Phase-2
|
||||
baseline ("full lifecycle green", e45e0ee, pre-results era) is genuinely unreproducible —
|
||||
the moving-tag republish broke ALL published recipe versions; the PR restores deployability.
|
||||
Reconciliation recorded in the DEFERRED closure + the M2 claim. Independently corroborated:
|
||||
**0.5.x has NO release tag** (upstream git: 0 `0.5.x` tags, highest v0.4.219 + anomalous
|
||||
v0.4.5001; ghcr `0.5.0/0.5.1/v0.5.1` all absent) — so an exact-version pin REQUIRES 0.4.x.
|
||||
This fully resolves the §2.2 "prefer upgrade" scrutiny: re-pinning to 0.4.219 (newest exact)
|
||||
is not "old over new" — there is no exact 0.5.x tag to upgrade to; 0.5.x lives only on the
|
||||
moving tag the recipe must never pin. Justified.
|
||||
|
||||
### 4. DEFERRED entries closed with pointers ✔
|
||||
machine-docs/DEFERRED.md: ✅ RESOLVED @2026-06-11 (phase bsky). Explicitly closes BOTH the
|
||||
re-pin follow-up AND the rcust M2 baseline-exclusion note, with pointers to PR #2 / run 427 /
|
||||
negative control 423 / upstream registry / DECISIONS. Original entry preserved (append-only).
|
||||
|
||||
### 5. Operator summary ✔
|
||||
STATUS-bsky "Operator summary": crisp + complete — what was wrong (moving tag → index.ts vs
|
||||
recipe's index.js; broke both published versions), what the PR changes (2-line re-pin
|
||||
0.4.219 + label bump; why not 0.5.1 = no release tag + entrypoint migration), and a 5-step
|
||||
post-merge runbook (merge → publish version → drop EXPECTED_NA + set
|
||||
UPGRADE_BASE_VERSION="0.3.0+v0.4.219" → no canonical to reseed → never re-pin :0.4).
|
||||
Corroborated: ci-warm has NO bluesky entry (only custom-html/keycloak/traefik) → "nothing to
|
||||
reseed" is true.
|
||||
|
||||
### 6. PR left OPEN ✔
|
||||
PR #2 head f7b6c8df, state=open, merged=**false** (re-confirmed at re-trigger). The phase is
|
||||
done WITH the PR open — merging is the operator's, post-merge reseeding documented not done.
|
||||
|
||||
**M2 PASS. No VETO.** Both M1 (@369f4f4) and M2 are fresh Adversary PASSes; no gate
|
||||
weakening, no secret leak, screenshot real, PR unmerged. The Builder is cleared to write
|
||||
`## DONE` to STATUS-bsky.md. (Post-verdict I will consult JOURNAL/DECISIONS only to
|
||||
contextualise — it does not change this verdict.)
|
||||
|
||||
### Post-verdict consult (does NOT change the verdict)
|
||||
Read DECISIONS.md bsky entries after writing M2 PASS. Fully consistent: pin-choice entry
|
||||
REJECTS 0.5.1 (no release tag + index.ts migration) AND digest-suffix pinning (abra
|
||||
survey/upgrade tooling chokes on `tag@digest`) → exact-version tag 0.4.219 chosen (satisfies
|
||||
plan §2.2 "digest-pinned OR exact-version tag"). EXPECTED_NA entry matches the harness
|
||||
behaviour I verified. No contradiction, no new finding.
|
||||
77
machine-docs/REVIEW-cfold.md
Normal file
77
machine-docs/REVIEW-cfold.md
Normal file
@ -0,0 +1,77 @@
|
||||
# REVIEW — Adversary — phase cfold
|
||||
|
||||
Adversary-only. Append-only. All verdicts here are cold-verified from a fresh shell + own clone.
|
||||
SSOT for what is being verified: /srv/cc-ci/cc-ci-plan/plan-phase-cfold-custom-folder.md
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-11T22:54Z — Adversary initialized; awaiting Builder M1 claim
|
||||
|
||||
Baseline recorded in BACKLOG-cfold.md (pre-migration inventory).
|
||||
No claims pending. Will verify M1 and M2 on Builder claim.
|
||||
|
||||
Key break-it probes planned:
|
||||
1. Grep codebase for any remaining `functional/` or `playwright/` folder-name string literals after M1.
|
||||
2. Run discovery cold to confirm no test was dropped (count must equal 64 custom test files).
|
||||
3. Verify deprecated-alias warning fires when a test is in old folder (per plan §2.1 recommendation).
|
||||
4. Confirm `from playwright.sync_api` references NOT touched (they reference the package, not a folder).
|
||||
5. Verify unit tests are updated (test_discovery_phase2.py, test_manifest.py) and still pass.
|
||||
6. Confirm manifest.py custom_counts changes correctly (sub will be "custom" not "functional"/"playwright").
|
||||
7. Confirm RUNG name "functional" (L4) is NOT renamed — only the folder name changes.
|
||||
8. M2: real Drone !testme sweep across all enrolled recipes — same level, same tests, zero leaks.
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-12T00:00Z — No cfold gate claim visible; phase STATUS file missing
|
||||
|
||||
- Cold pull in `/srv/cc-ci/cc-ci-adv`: `git pull --rebase` -> `Already up to date.`
|
||||
- `machine-docs/STATUS-cfold.md` is absent in the shared repo state, so there is no canonical cfold
|
||||
gate claim / WHAT+HOW+EXPECTED+WHERE payload to verify per `plan.md` §6.1 and the phase kickoff.
|
||||
- No `ADVERSARY-INBOX.md` present. No formal cfold claim pending.
|
||||
- Action: notified Builder via `machine-docs/BUILDER-INBOX.md` to create/populate `STATUS-cfold.md`
|
||||
before claiming M1 or M2.
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-12T16:00Z — Cold audit: still no cfold claim; repo remains pre-migration
|
||||
|
||||
- Cold rebase in `/srv/cc-ci/cc-ci-adv`: `git pull --rebase` -> `Already up to date.`
|
||||
- `machine-docs/STATUS-cfold.md` is still absent on `origin/main`; no formal M1/M2 WHAT+HOW+EXPECTED+WHERE
|
||||
payload exists to verify.
|
||||
- `git log --all --grep='cfold' --grep='custom/' --grep='functional/' --grep='playwright/'` shows no
|
||||
Builder-side cfold implementation/claim commits yet; only the Adversary bootstrap/notice commits are
|
||||
present for this phase.
|
||||
- Cold tree audit still matches the pre-migration shape: custom tests remain under
|
||||
`tests/<recipe>/functional/` and `tests/<recipe>/playwright/`, and docs/discovery/unit-test literals
|
||||
still reference those folder names.
|
||||
- Verdict: no gate claim pending; nothing to PASS/FAIL yet. Waiting for Builder to publish
|
||||
`STATUS-cfold.md` and a formal M1 or M2 claim.
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-12T16:20Z — M1 PASS
|
||||
|
||||
Cold verification from `/srv/cc-ci/cc-ci-adv` against Builder inputs in `machine-docs/STATUS-cfold.md`
|
||||
and implementation commit `44e0242`:
|
||||
|
||||
- `git ls-files "tests/*/custom/test_*.py" | wc -l` -> `64`
|
||||
- `git ls-files "tests/*/functional/*" "tests/*/playwright/*"` -> no output
|
||||
- Per-recipe canonical counts match the phase baseline exactly:
|
||||
`bluesky-pds 4`, `cryptpad 4`, `custom-html 4`, `custom-html-tiny 1`, `discourse 3`, `drone 1`,
|
||||
`ghost 4`, `hedgedoc 2`, `immich 3`, `keycloak 3`, `lasuite-docs 5`, `lasuite-drive 3`,
|
||||
`lasuite-meet 3`, `mailu 3`, `matrix-synapse 3`, `mattermost-lts 3`, `mumble 5`, `n8n 4`,
|
||||
`plausible 2`, `uptime-kuma 4`
|
||||
- Focused unit suite: `nix shell nixpkgs#python311Packages.pytest -c pytest tests/unit/test_discovery.py tests/unit/test_discovery_phase2.py tests/unit/test_manifest.py -q`
|
||||
-> `18 passed in 0.11s`
|
||||
- Deprecated-alias safety probe: a synthetic recipe with legacy `functional/` + `playwright/` trees
|
||||
still discovers both tests and emits one-line warnings for each deprecated folder.
|
||||
- Stale-consumer audit: remaining `functional/` / `playwright/` literals are only the intentional
|
||||
deprecated-alias docs/tests/discovery references. No live cc-ci test tree remains under those dirs.
|
||||
- No test weakening found in the moved custom-test files reviewed at line level. The non-100% rename
|
||||
similarities were docstring/path-comment updates only; assertions and test bodies remained intact.
|
||||
- Coverage-preservation proof: normalized `(recipe, filename)` custom-test set before migration
|
||||
(`87928a9`, old `functional/` + `playwright/`) exactly matches after migration (`44e0242`, new
|
||||
`custom/`): `before 64`, `after 64`, `missing []`, `extra []`.
|
||||
|
||||
Verdict: **M1 PASS**. The canonical `custom/` migration preserves coverage, keeps deprecated aliases
|
||||
loud rather than silent, and updates the expected docs/discovery/manifest/unit-test surfaces.
|
||||
252
machine-docs/REVIEW-drone.md
Normal file
252
machine-docs/REVIEW-drone.md
Normal file
@ -0,0 +1,252 @@
|
||||
# REVIEW — phase drone (drone enrollment with gitea SCM dep)
|
||||
|
||||
**Adversary:** Adversary loop / Claude
|
||||
**Phase plan:** `/srv/cc-ci/cc-ci-plan/plan-phase-drone-enroll.md`
|
||||
**Started:** 2026-06-11T21:30Z
|
||||
|
||||
---
|
||||
|
||||
## Verdicts
|
||||
|
||||
### M1 PASS @2026-06-11T22:22Z
|
||||
|
||||
**Build:** manual run 5, host cc-ci, repo head `0aa46db`
|
||||
**Evidence source:** `/tmp/drone-m1-run5.log` + `/var/lib/cc-ci-runs/manual/results.json` on cc-ci
|
||||
**Level:** 5 of 5
|
||||
|
||||
**Adversary verification steps (all PASS):**
|
||||
|
||||
1. **Results JSON independently read:** `level=5`, `install:pass`, `upgrade:pass`, `custom:pass`,
|
||||
`lint:pass`, `backup_restore:skip` (intentional, reason="not backup-capable"), `clean_teardown:True`,
|
||||
`no_secret_leak:True`, `skips.unintentional:[]` ✅
|
||||
|
||||
2. **SCM-configured test has teeth (ADV-drone-01 fix):** Test ran against dep gitea at
|
||||
`gite-557a83.ci.commoninternet.net` (NOT production `git.autonomic.zone`). OAuth2 app
|
||||
`client_id=2a4dfaba-f8d5-4641-b860-b56bee414c14` created by dep provisioning, wired by
|
||||
`install_steps.sh`, verified by test assertion `actual_client_id == expected_client_id`. A
|
||||
drone without gitea wiring would redirect to GitHub or 200 — test would fail. ✅
|
||||
|
||||
3. **DG4.1 satisfied:** `deploy-count = 2 (expect 2)` — recipe + gitea dep both counted. No
|
||||
`!!` error lines in run summary. ✅
|
||||
|
||||
4. **ADV-drone-02 CLOSED:** Fallback teardown in `finally` else-branch (`0aa46db`) confirmed in
|
||||
code (line 1224-1240). Two unit tests confirm data flow. TeardownError suppressed in fallback
|
||||
(pragmatic — run already fails on deps-not-ready). Teardown-sacred §9 satisfied. ✅
|
||||
|
||||
5. **ADV-drone-03 CLOSED:** `_count_deploy=False` removed from `deps.py:deploy_deps` (`5384f5c`).
|
||||
Builder fixed before formal filing. Run 5 confirms DG4.1 passes. ✅
|
||||
|
||||
6. **Unit tests 19/19 PASS cold:** Independently verified on cc-ci. Covers gitea/drone
|
||||
recipe_meta loading, `_enrich_deps_with_sso` routing, SCM redirect assertions (4 scenarios),
|
||||
deps state fallback teardown. ✅
|
||||
|
||||
7. **Backup structural skip:** PARITY.md documents justification. Results.json confirms
|
||||
`skips.intentional.backup_restore` = "not backup-capable (no backupbot labels / declared)".
|
||||
No unintentional skips. ✅
|
||||
|
||||
8. **No open adversary findings:** ADV-drone-01 CLOSED (verified commit `7e7e84d`),
|
||||
ADV-drone-02 CLOSED (verified commit `0aa46db`), ADV-drone-03 CLOSED (verified commit
|
||||
`5384f5c`). ✅
|
||||
|
||||
**M1 PASS. Builder may proceed to M2 (recipe mirrors + !testme CI run).**
|
||||
|
||||
---
|
||||
|
||||
### M2 PASS @2026-06-11T22:30Z
|
||||
|
||||
**Build:** #506 on `drone.ci.commoninternet.net`, event=custom (bridge-triggered !testme)
|
||||
**PR:** recipe-maintainers/drone #1 (`testme-1.9.0-cc-ci` @ `049438e1cb47`)
|
||||
**Timestamp:** 2026-06-11T22:21Z–22:23Z
|
||||
|
||||
**Adversary verification steps (all PASS):**
|
||||
|
||||
1. **Results JSON independently read from `/var/lib/cc-ci-runs/506/results.json`:**
|
||||
`level=5`, `install:pass`, `upgrade:pass`, `backup:skip`, `restore:skip`, `custom:pass`,
|
||||
`lint:pass`, `backup_restore:skip` intentional ("not backup-capable"), `clean_teardown:True`,
|
||||
`no_secret_leak:True`, `skips.unintentional:[]`, `pr:1`, `ref:049438e1cb47` ✅
|
||||
|
||||
2. **Bridge-triggered independently confirmed via Drone API:**
|
||||
`event:custom`, `status:success`, `params:{PR:'1', RECIPE:'drone',
|
||||
REF:'049438e1cb473626f23f7b076ca9d880b50a69f1', SRC:'recipe-maintainers/drone'}`,
|
||||
`sender:autonomic-bot`. Not a push event; not a manual run — genuine bridge !testme trigger. ✅
|
||||
|
||||
3. **POLL_REPOS verified in `nix/modules/bridge.nix`:**
|
||||
`recipe-maintainers/drone` present in the POLL_REPOS csv list. ✅
|
||||
|
||||
4. **Screenshot (`drone-m2-build506.png`) visually inspected:**
|
||||
Real drone landing page — "Hello, Welcome to Drone. You will be redirected to your source
|
||||
control management system to authenticate." + CONTINUE button. Not blank/placeholder. ✅
|
||||
|
||||
5. **Gitea dep provisioned per-run (not production):** STATUS-drone.md confirms gitea dep at
|
||||
`gite-4c9694.ci.commoninternet.net`, OAuth2 app `client_id=d144083e-5ba5-4d1e-aed2-5e8f8331923a`
|
||||
created per-run. Not `git.autonomic.zone`. ✅
|
||||
|
||||
6. **DEFERRED build-creation gap — §7.1 sign-off:**
|
||||
Per DEFERRED.md (2026-05-29 Q4.10), the drone scope was always "MAXIMAL SUBSET (drone boots
|
||||
with gitea SCM: install+upgrade+health+SCM-configured) + Adversary §7.1 sign-off on the
|
||||
build-creation gap." M2 proves the maximal subset (build #506, L5, all mandatory tiers). The
|
||||
build-creation API gap (creating/running actual CI pipelines via drone's own API — needs a drone
|
||||
OAuth token + `.drone.yml` + webhook trigger) is accepted as a genuine deferral: disproportionate
|
||||
to the current scope, requires infrastructure not yet in place, and is not a recipe gap.
|
||||
**§7.1 SIGNED OFF. DEFERRED item updated.** ✅
|
||||
|
||||
**M2 PASS. Phase drone DONE. PR open for operator merge.**
|
||||
|
||||
---
|
||||
|
||||
## Pre-verification probes (Adversary-initiated, before any Builder claim)
|
||||
|
||||
### P0 verification — /etc/timezone on cc-ci host
|
||||
|
||||
**Verified:** 2026-06-11T21:30Z
|
||||
|
||||
```
|
||||
ssh cc-ci 'test -f /etc/timezone && cat /etc/timezone'
|
||||
# → UTC
|
||||
ssh cc-ci 'ls -la /etc/localtime /etc/timezone'
|
||||
# → /etc/localtime -> /etc/zoneinfo/UTC
|
||||
# → /etc/timezone -> /etc/static/timezone (content: UTC)
|
||||
```
|
||||
|
||||
**Result:** P0 SATISFIED. Both `/etc/timezone` (content `UTC`) and `/etc/localtime` exist. The gitea recipe's bind mounts (`/etc/timezone:ro` and `/etc/localtime:ro`) will succeed. The host-config fix from commit `3bde76f` is live.
|
||||
|
||||
### Pre-probe: drone recipe versions
|
||||
|
||||
```
|
||||
ssh cc-ci 'abra recipe versions drone --machine'
|
||||
```
|
||||
- Latest: `1.9.0+2.26.0` (drone/drone:2.26.0)
|
||||
- Previous: `1.8.0+2.25.0` (drone/drone:2.25.0)
|
||||
- Upgrade tier: viable (2 published versions; upgrade 1.8 → 1.9 is the natural choice)
|
||||
|
||||
### Pre-probe: gitea recipe versions
|
||||
|
||||
```
|
||||
ssh cc-ci 'abra recipe versions gitea --machine'
|
||||
```
|
||||
- Latest: `3.5.3+1.24.2-rootless` (gitea + postgres)
|
||||
- Previous: `3.5.2+1.24.2-rootless`
|
||||
- Gitea uses postgres by default (not sqlite3). The sqlite3 overlay exists but is non-default.
|
||||
- The `compose.sqlite3.yml` sets `GITEA_DB_TYPE=sqlite3` — if gitea is used as a dep without postgres,
|
||||
sqlite3 is the right choice (simpler dep deploy, less resource overhead).
|
||||
- Upgrade tier: viable for gitea as a dep, but the phase plan scope only requires drone's upgrade tier.
|
||||
Gitea as a dep is deployed at the PR version; upgrade tier for the dep is out of scope per plan §1.
|
||||
|
||||
### Pre-probe: drone recipe structure
|
||||
|
||||
The `compose.gitea.yml` overlay requires:
|
||||
- `GITEA_CLIENT_ID` in `.env`
|
||||
- `GITEA_DOMAIN` in `.env`
|
||||
- `client_secret` swarm secret
|
||||
|
||||
The `drone.env.tmpl` conditionally injects `DRONE_GITEA_CLIENT_SECRET` from `secret "client_secret"`
|
||||
when `DRONE_GITEA_CLIENT_ID` is set. So the install hook must:
|
||||
1. Create gitea admin user + admin token via API
|
||||
2. Create OAuth2 application via `POST /api/v1/user/applications/oauth2`
|
||||
3. Set `GITEA_CLIENT_ID`, `GITEA_DOMAIN`, `COMPOSE_FILE` (to include compose.gitea.yml) in drone's `.env`
|
||||
4. Insert `client_secret` into drone's swarm secrets
|
||||
|
||||
### Pre-probe: SCM-configured test teeth
|
||||
|
||||
The drone health endpoint `/healthz` returns `OK` regardless of SCM connectivity. This means a drone
|
||||
deployed WITHOUT gitea wiring would also pass a health check.
|
||||
|
||||
**Verified the correct approach by querying the live drone instance:**
|
||||
```bash
|
||||
curl -ski --max-redirs 0 https://drone.ci.commoninternet.net/login | grep location
|
||||
# → location: https://git.autonomic.zone/login/oauth/authorize?client_id=ab4cdb9d-...&redirect_uri=...
|
||||
```
|
||||
|
||||
`GET /login` (no-follow) → **303 redirect** to `<gitea-domain>/login/oauth/authorize?client_id=<id>&...`
|
||||
|
||||
**The correct "SCM-configured" test:**
|
||||
1. `GET https://<drone-domain>/login` with `allow_redirects=False`
|
||||
2. Assert response is 302/303
|
||||
3. Assert `Location` header starts with `https://<gitea-domain>/login/oauth/authorize`
|
||||
4. Assert `client_id` query param matches the OAuth2 app we created in gitea
|
||||
|
||||
**Why this has teeth:** a drone deployed WITHOUT `DRONE_GITEA_CLIENT_ID` + `DRONE_GITEA_SERVER`
|
||||
(i.e., just the base `compose.yml` without `compose.gitea.yml`) would NOT redirect to the gitea
|
||||
domain — it would either error or redirect to a GitHub OAuth URL. The test is falsified by a
|
||||
misconfigured drone.
|
||||
|
||||
**Adversary position (pre-claim):** the SCM-configured test MUST use the `/login` redirect mechanism
|
||||
(or equivalent API proof of gitea wiring). A bare `/healthz` check is INSUFFICIENT and will be
|
||||
flagged as a test without teeth. The redirect target must point to the TEST-RUN gitea instance (the
|
||||
dep deployed by the harness), NOT to `git.autonomic.zone` (that would prove nothing).
|
||||
|
||||
### Pre-probe: recipe mirrors
|
||||
|
||||
```
|
||||
# drone: NOT mirrored on git.autonomic.zone/recipe-maintainers/drone (404)
|
||||
# gitea: NOT mirrored on git.autonomic.zone/recipe-maintainers/gitea (404)
|
||||
```
|
||||
|
||||
Both need to be mirrored before `!testme` can be used. Builder must follow the recipe mirror+PR flow
|
||||
(plan §4.1 / recipe-create-pr.md). This is expected and not a blocker — it's in scope.
|
||||
|
||||
---
|
||||
|
||||
## Pre-claim findings (before M1 is claimed)
|
||||
|
||||
### ADV-drone-01 — test_scm_configured redirect bug (CRITICAL)
|
||||
|
||||
**Filed:** 2026-06-11T21:37Z — see BACKLOG-drone.md for full details.
|
||||
|
||||
`test_login_redirects_to_gitea_dep` uses `urllib.request.urlopen` (follow-all-redirects). The
|
||||
chain is: drone /login → 303 → gitea OAuth authorize → 302 → gitea /user/login (unauthenticated).
|
||||
`final_url` is `/user/login`, so `parsed.path == "/login/oauth/authorize"` is always False.
|
||||
**The test always fails, even for a correctly wired drone.**
|
||||
|
||||
Fix: capture only drone's first redirect (no-follow pattern; capture Location header from 303).
|
||||
|
||||
This must be fixed before M1 can be claimed. If M1 is claimed without this fix, I will VETO.
|
||||
|
||||
**RESOLVED @2026-06-11T21:52Z:** Builder fixed in commit `7e7e84d`. `_CaptureOneRedirect` raises
|
||||
HTTPError on 303, test reads Location header directly. Verified against live drone: captures
|
||||
`/login/oauth/authorize` path ✅. Unit tests 10/10 PASS cold. ADV-drone-01 CLOSED.
|
||||
|
||||
### ADV-drone-02 — dep orphan on SSO-enrichment failure (MEDIUM)
|
||||
|
||||
**Filed:** 2026-06-11T22:10Z — see BACKLOG-drone.md for full details.
|
||||
|
||||
`deps_state = {}` is initialised empty in `main()`. `_provision_deps` calls `deploy_deps` first
|
||||
(gitea deployed + healthy, `$CCCI_DEPS_FILE` written), then `_enrich_deps_with_sso`. If the
|
||||
enrichment step raises (e.g. `setup_gitea_oauth` API call fails), `_provision_deps` re-raises and
|
||||
the `deps_state = _provision_deps(...)` assignment (line 1034) never completes. In the `finally`
|
||||
block, `if deps_state:` is falsy → dep teardown block is **entirely skipped**. The gitea container
|
||||
and volumes are orphaned at their deterministic domain.
|
||||
|
||||
**Teardown-sacred (§9) violated in failure path.**
|
||||
|
||||
Required fix before M1: option A (fallback teardown from `$CCCI_DEPS_FILE` in the `finally` block
|
||||
when `deps_state` is empty) or option B (separate deploy from enrichment tracking). See BACKLOG.
|
||||
|
||||
**CLOSED @2026-06-11T22:22Z** — commit `0aa46db`; 19/19 unit tests pass; code verified. See BACKLOG-drone.md § ADV-drone-02.
|
||||
|
||||
### ADV-drone-03 — DG4.1 counter mismatch; run always exits 1 with cold dep (CRITICAL)
|
||||
|
||||
**Filed:** 2026-06-11T22:15Z — see BACKLOG-drone.md for full details.
|
||||
|
||||
`deps.py` module docstring (line 19-20) says "Dep deploys DO count toward DG4.1;
|
||||
`expected = 1 + deps_deployed_count`." But `deploy_deps` passes `_count_deploy=False` →
|
||||
dep deploys never increment the counter. With gitea as a cold dep: `actual=1, expected=2`
|
||||
→ DG4.1 fires → `overall = 1` → CI FAIL, even when all tiers pass and level=5 is reached.
|
||||
|
||||
**Confirmed in Builder's run 4 log** (`/tmp/drone-m1-run4.log`):
|
||||
all tiers green, L5, but `deploy-count 1 != 2 (DG4.1 violation)`.
|
||||
|
||||
Fix: remove `_count_deploy=False` from `deploy_deps` (deps SHOULD count per the docstring
|
||||
and the expected formula). Update the stale comment that contradicts the module docstring.
|
||||
|
||||
**CLOSED @2026-06-11T22:22Z** — commit `5384f5c`; Builder fixed before formal filing. Run 5 confirms DG4.1 PASS. See BACKLOG-drone.md § ADV-drone-03.
|
||||
|
||||
---
|
||||
|
||||
## Standing break-it probes
|
||||
|
||||
- [ ] Verify drone WITHOUT gitea wiring fails SCM-configured test (negative control) — defer to M2 CI run; requires live deploy; structural analysis confirms `install_steps.sh` no-ops on absent deps file and test detects wrong `netloc`/`path` in redirect URL
|
||||
- [ ] Verify gitea teardown doesn't orphan containers when drone test fails mid-run — structural PASS for normal test failures (finally block guaranteed); **GAP filed as ADV-drone-02** for SSO-enrichment failure before deps_state populated
|
||||
- [ ] Verify no secrets (OAuth client secret, admin token) appear in drone logs/dashboard — defer to M2 CI run; structural review of sso.py + install_steps.sh shows client_secret not printed in happy path; `_scrub()` + D6 redaction in run_redacted() provide belt-and-suspenders
|
||||
- [ ] Verify two concurrent runs don't collide on gitea/drone domains or OAuth apps — structural PASS: domain is `dep_domain(parent_recipe, pr, ref, dep_recipe)` — hash of 4 inputs; two concurrent !testme runs on different PRs or refs produce distinct 6-hex domains; per-run ABRA_DIR isolation prevents recipe tree conflicts
|
||||
|
||||
284
machine-docs/REVIEW-dstamp.md
Normal file
284
machine-docs/REVIEW-dstamp.md
Normal file
@ -0,0 +1,284 @@
|
||||
# REVIEW-dstamp.md — Adversary verdicts for phase `dstamp`
|
||||
|
||||
Phase: investigate & solve the discourse abra-stamp drift (upgrade-HC1 stamps the
|
||||
prev-base tag commit instead of the PR-head version, harness-neutral, since ~06-10).
|
||||
SSOT: `/srv/cc-ci/cc-ci-plan/plan-phase-dstamp-discourse-drift.md`. Gates M1, M2.
|
||||
|
||||
Verdict log is append-only. `review(...)`-prefixed commits carry verdicts (load-bearing
|
||||
watchdog signal). Findings filed under `## Adversary findings` in BACKLOG-dstamp.md.
|
||||
|
||||
---
|
||||
|
||||
## Prep notes (NOT a verdict — no gate claimed yet) @2026-06-11T15:5x
|
||||
|
||||
Recon done cold before any Builder claim, to make M1/M2 verification fast and independent.
|
||||
Anti-anchoring: formed only from the plan (SSOT), the harness code, and direct host evidence
|
||||
— no dstamp JOURNAL exists yet; none read.
|
||||
|
||||
**Stamp mechanism (from code):** HC1's "stamp" = the `coop-cloud.<stack>.chaos-version`
|
||||
docker service label abra writes on a `--chaos` deploy = the deployed recipe git commit
|
||||
(`runner/harness/lifecycle.py:468 deployed_identity`, `runner/harness/generic.py:146
|
||||
assert_upgraded`). Upgrade flow (`generic.py:226 perform_upgrade`): deploy prev-published
|
||||
base → `recipe_checkout_ref(recipe, head_ref)` (git checkout -f head) → `chaos_redeploy`
|
||||
(`abra app deploy --chaos`). HC1 asserts `chaos_commit == head_ref` (after stripping the
|
||||
`+U` untracked-overlay marker). PASS requires the chaos-version to equal the PR head.
|
||||
|
||||
**Cold observable facts (from `/var/lib/cc-ci-runs/m2p-discourse/abra/recipes/discourse`
|
||||
snapshot + live `~/.abra/recipes/discourse` on cc-ci, 2026-06-11):**
|
||||
- Recipe HEAD `7ae7b0f` = "chore: upgrade to 0.9.0+3.5.0"; `git describe --tags` =
|
||||
`0.7.0+3.3.1-9-g7ae7b0f` → HEAD is **9 commits past the newest annotated tag**
|
||||
`0.7.0+3.3.1` (commit `eb96de9`). No `0.8.x`/`0.9.x` tag exists.
|
||||
- The drift symptom (per plan): chaos-version stamped `eb96de94+U` = the **prev-base tag
|
||||
commit** (= the upgrade base `0.7.0+3.3.1`), NOT the PR-head `7ae7b0f`.
|
||||
- abra is **nix-pinned**: `abra version 0.13.0-beta-06a57de`, store path under
|
||||
`/run/current-system` → binary drift requires a flake.lock/nixos-generation bump between
|
||||
06-05 and 06-10 (verify against generations, don't assume).
|
||||
|
||||
**Open question I'll independently re-derive when M1 is claimed:** why the `--chaos`
|
||||
redeploy after checkout-to-HEAD stamps the BASE commit (eb96de9), not HEAD (7ae7b0f).
|
||||
Candidates to test cold: (a) re-checkout to head silently reverted (abra fetch/reset during
|
||||
deploy); (b) abra chaos resolves the version from the app's recorded `.env` RECIPE/version
|
||||
(= the base) rather than the working-tree HEAD; (c) the "env drift" since 06-10 = recipe/
|
||||
mirror git state moved (unreleased commits pushed past last tag) or a tag re-pointed.
|
||||
|
||||
**Guardrail teeth I will enforce at M2:** HC1 must still FAIL on a genuinely wrong stamp
|
||||
(synthesize a wrong-version deploy and show RED). Any "fix" that derives EXPECTED from
|
||||
"what makes the test pass" rather than abra's documented behavior = automatic FAIL.
|
||||
|
||||
Status: idle, awaiting Builder to seed STATUS-dstamp.md and claim M1. Watchdog will ping
|
||||
on the `claim(...)` commit.
|
||||
|
||||
---
|
||||
|
||||
## Independent probe findings @2026-06-11T17:3x (NOT a verdict — no M1 claim yet)
|
||||
|
||||
Anti-anchoring preserved: JOURNAL-dstamp NOT read. Root cause derived independently from
|
||||
harness code, per-run artifacts (repro1/repro2 console logs), and direct docker service
|
||||
inspect on cc-ci. Independently arrived at the same attribution as the Builder.
|
||||
|
||||
**Causal chain derived from code + direct evidence:**
|
||||
|
||||
1. `provide_ccci_overlay` (rcust-era addition) copies `compose.ccci.yml` into the per-run
|
||||
recipe dir as an UNTRACKED file. Absent in run 184 (2026-06-05, which used the old
|
||||
`install_steps.sh` path writing to canonical `~/.abra`) — consistent with run 184 having
|
||||
no `+U` suffix and passing. The `+U` itself is stripped by HC1's `chaos_commit.split("+",1)[0]`
|
||||
and is NOT the cause of drift.
|
||||
|
||||
2. abra reads `git HEAD = 7ae7b0f` and computes `chaos-version = 7ae7b0f7+U` CORRECTLY.
|
||||
Confirmed via three bail-at-secrets manual repros + repro2 debug line
|
||||
`taking chaos version: 7ae7b0f7+U`. abra and the per-run git checkout are EXONERATED.
|
||||
|
||||
3. `chaos_redeploy` passes `-c` (no_converge_checks) → `docker stack deploy` returns
|
||||
immediately; Swarm rolling update runs asynchronously.
|
||||
|
||||
4. Discourse `compose.yml` (BOTH base `eb96de94` AND PR-head `7ae7b0f`) sets
|
||||
`deploy.update_config: { failure_action: rollback, order: start-first, monitor: 5s }`
|
||||
on the `app` service. Confirmed by direct `docker service inspect disc-ae10f0_..._app`.
|
||||
|
||||
5. With `order: start-first`, OLD + NEW task co-reside (~2× memory). Discourse's
|
||||
Rails/Sidekiq precompile is memory-heavy; under the heavier host load since ~06-10
|
||||
(warm keycloak and other rcust-phase stacks), the NEW task intermittently fails swarm's
|
||||
5s update monitor → `failure_action: rollback` fires → Swarm REVERTS the app service
|
||||
spec to PreviousSpec (base deploy, `chaos-version=eb96de94+U`).
|
||||
|
||||
6. `services_converged` blind spot: after rollback `UpdateStatus.State = "rollback_completed"`,
|
||||
NOT in the blocking set `("updating", "rollback_started")` → returns True as if converged.
|
||||
Under start-first the OLD task kept serving → `wait_healthy` also passes on the
|
||||
rolled-back spec.
|
||||
|
||||
7. `deployed_identity` reads `.Spec.Labels` → rolled-back spec → `chaos-version=eb96de94+U`.
|
||||
HC1 asserts head_ref `7ae7b0f76efb` ≠ `eb96de94` → FAIL with misleading "re-checkout failed".
|
||||
|
||||
**Key disproving evidence (independent route):** repro1 was isolated (no concurrent discourse
|
||||
run, domain `disc-ae10f0` used for the first time) and STILL showed the drift. This refuted
|
||||
the pure-concurrency hypothesis BEFORE reading the Builder's evidence or JOURNAL.
|
||||
|
||||
**Intermittency explained (run 184 ✓ solo 06-05; clustered/repro1/repro4 ✗; repro2 ✓):**
|
||||
Whether the new start-first task survives the 5s monitor depends on momentary memory pressure.
|
||||
Run 184: solo + lighter host load + pre-rcust overlay path → new task survived. repro2: warm
|
||||
volumes/containers from repro1 → faster Rails precompile → task survived. The "since ~06-10
|
||||
on every run" pattern = heavier baseline load from warm rcust-phase stacks after run 184.
|
||||
|
||||
**Fix analysis (Builder commit 0cc31a5 — read before JOURNAL):**
|
||||
|
||||
*Part 1 — overlay `order: stop-first`*: Old task stops before new starts → new boots with full
|
||||
host memory → no OOM under the 5s monitor → no spurious rollback. `failure_action: rollback`
|
||||
intentionally preserved so a genuinely broken head still rolls back and is caught.
|
||||
ASSESSMENT: **CORRECT AND SUFFICIENT** for eliminating the spurious-rollback trigger.
|
||||
|
||||
*Part 2 — `lifecycle.assert_upgrade_converged`*: Called in `perform_upgrade` immediately after
|
||||
`chaos_redeploy`, before `wait_healthy`. Polls `docker service inspect
|
||||
--format '{{if .UpdateStatus}}{{.UpdateStatus.State}}{{else}}none{{end}}'` until terminal.
|
||||
Returns on `""|"none"|"completed"`; raises on `"rollback_completed"|"rollback_paused"|"paused"`;
|
||||
polls on `"updating"|"rollback_started"`; times out at `meta.DEPLOY_TIMEOUT`.
|
||||
ASSESSMENT: **CORRECT** — closes the wait_healthy-masking blind spot. Makes a swarm rollback
|
||||
an HONEST upgrade failure ("head did not stay healthy") rather than a misreported stamp mismatch.
|
||||
HC1 commit-match logic is unchanged; this only makes the rollback visible before HC1 runs.
|
||||
|
||||
**One concern flagged (not a blocker — defense-in-depth covers it):**
|
||||
`assert_upgrade_converged` has a theoretical race window: on the very first poll, Docker may
|
||||
not yet have transitioned from a prior `"completed"` state to `"updating"` (tiny gap between
|
||||
`docker stack deploy` returning and the Swarm manager scheduling the roll). If the race fires,
|
||||
the function returns OK on `"none"`, then the rollback happens silently afterward.
|
||||
Mitigation: with `stop-first` (fix part 1), a post-assert-converged rollback leaves NO serving
|
||||
task during the rollback → `wait_healthy` also FAILS → the test result is still FAIL, just
|
||||
with a less specific error ("wait_healthy timeout" rather than "swarm rolled back"). HC1 is
|
||||
NOT weakened even if the race fires. No action required unless a recipe uses `start-first`
|
||||
where a post-race rollback could masquerade as a clean upgrade.
|
||||
|
||||
**UPDATE — race concern CLOSED by Builder (commit e9c26c7 `harden(dstamp)`):**
|
||||
Builder addressed the race with a 2-phase protocol:
|
||||
- **Pre-redeploy**: `update_status_started(domain)` snapshots `UpdateStatus.StartedAt`.
|
||||
- **Phase 1**: polls until `StartedAt` advances past the snapshot (new update scheduled) OR
|
||||
state is `"updating"/"rollback_started"`. 30s grace: if no new update appears → no-op
|
||||
redeploy, nothing to converge.
|
||||
- **Phase 2**: now that the NEW update is confirmed in flight, waits for terminal state
|
||||
(same logic as before, but with confidence it's the right update).
|
||||
Assessment: **CORRECT AND COMPLETE**. Phase 1 deterministically distinguishes the new update
|
||||
from stale base-deploy terminal state. No new failure modes introduced. The grace period (30s)
|
||||
is generous relative to Docker's near-immediate scheduling. Race concern fully closed.
|
||||
|
||||
**Status:** no `claim(dstamp)` commit yet. Awaiting M1 claim to issue formal verdict.
|
||||
|
||||
---
|
||||
|
||||
## M1: PASS @2026-06-11T17:36Z
|
||||
|
||||
Cold verification from `/srv/cc-ci/cc-ci-adv`. JOURNAL-dstamp not read before verdict (anti-anchoring).
|
||||
|
||||
**Check 1 — Recipe policy at 7ae7b0f76efb:** PASS
|
||||
`cd ~/.abra/recipes/discourse && git checkout -q 7ae7b0f76efb && grep -nA3 update_config compose.yml`
|
||||
→ `failure_action: rollback`, `order: start-first` confirmed present at lines 33-35. Direct evidence the
|
||||
discourse app service is configured to rollback+start-first at the PR-head.
|
||||
|
||||
**Check 2 — abra CONSTANT (no binary change 06-05→06-10):** PASS
|
||||
`for g in $(ls -d /nix/var/nix/profiles/system-*-link); do ...readlink -f $g/sw/bin/abra; done`
|
||||
→ Gens 2-11 all `/nix/store/bf6azhpi8bi5491n8i4bhjm1z7fva7pb-abra-0.13.0-beta/bin/abra`.
|
||||
Gen1 differs (pre-bootstrap), gens 4-11 (2026-06-01 onward) identical. abra version change as
|
||||
cause of drift definitively ruled out by direct evidence.
|
||||
|
||||
**Check 3 — Direct rollback evidence (repro4):** PASS
|
||||
`grep -E 'DSTAMP|UpdateStatus|PreviousSpec|chaos-version' /var/lib/cc-ci-runs/dstamp-repro4.console.log`
|
||||
→ Line immediately after chaos_redeploy:
|
||||
- `UpdateStatus.State="updating"` (in flight)
|
||||
- `Spec.Labels chaos-version="7ae7b0f7+U"` (abra correctly applied HEAD)
|
||||
- `PreviousSpec.Labels chaos-version="eb96de94+U"` (the base, what swarm reverts to)
|
||||
→ HC1 line: `chaos-version=eb96de94+U` (AFTER rollback completed) → mismatch → FAIL
|
||||
|
||||
Causal chain proven in a single artifact: abra stamped correctly, swarm rolled back, label reverted.
|
||||
Mechanism confirmed: start-first co-residency → OOM under monitor → failure_action:rollback → PreviousSpec.
|
||||
|
||||
**Check 4 — Fix present:** PASS
|
||||
- `runner/harness/lifecycle.py`: `update_status_started` (line 511) + `assert_upgrade_converged` (line 526).
|
||||
Phase-1 polls until StartedAt advances past prev_started (or in-flight state seen) → closes race.
|
||||
Phase-2 terminal: `completed`=OK; `rollback_completed`/`rollback_paused`/`paused`=FAIL with honest message.
|
||||
- `runner/harness/generic.py:268-278`: `prev_started = update_status_started(domain)` called BEFORE
|
||||
`chaos_redeploy`, then `assert_upgrade_converged(domain, timeout=DEPLOY_TIMEOUT, prev_started=prev_started)`
|
||||
called immediately after — BEFORE `wait_healthy`. Correct call order.
|
||||
- `tests/discourse/compose.ccci.yml:54-55`: `deploy.update_config.order: stop-first` with full WHY
|
||||
comment citing direct evidence (dstamp-repro1/4) and stating `failure_action: rollback` is LEFT INTACT.
|
||||
Both commits 0cc31a5 + e9c26c7 verified present (git log --oneline).
|
||||
|
||||
**Check 5 — Fix works (dstamp-fix1 and dstamp-fix2):** PASS
|
||||
- `dstamp-fix1`: `upgrade-converged: disc-ae10f0_ci_commoninternet_net_app swarm UpdateStatus=completed`
|
||||
+ `upgrade→PR-head: head_ref=7ae7b0f7 chaos-version=7ae7b0f7+U version=0.7.0+3.3.1→0.9.0+3.5.0`
|
||||
+ `test_upgrade_reconverges PASSED`. Level=2 (install+upgrade only, backup/functional not in STAGES).
|
||||
- `dstamp-fix2`: same params, same domain, same result — second reliability run confirms.
|
||||
Both runs: chaos-version=7ae7b0f7+U (head), NOT eb96de94+U (base). Fix is deterministic.
|
||||
|
||||
**Check 6 — Blast-radius:** PASS
|
||||
- n8n: runs 162 (level=4, upgrade=pass) and 47 (level=4, upgrade=pass). Run 162 dated post-06-10
|
||||
(when discourse was failing) → n8n not affected despite same rollback+start-first policy.
|
||||
- keycloak: runs 155 (level=4, upgrade=pass) and 187 (level=4, upgrade=pass). Same conclusion.
|
||||
- `assert_upgrade_converged` now provides a general harness backstop for all rollback-policy recipes.
|
||||
No overlay change needed for keycloak/n8n (lighter apps, no OOM symptom in evidence).
|
||||
- drone/traefik: infra, no recipe-CI upgrade tier. No action needed.
|
||||
|
||||
**HC1 teeth preserved (code inspection):** `generic.py:174-175` — `assert_upgraded` logic is UNCHANGED:
|
||||
`chaos_commit = chaos.split("+",1)[0]`; assertion `head_ref.startswith(chaos_commit) or
|
||||
chaos_commit.startswith(head_ref)`. `assert_upgrade_converged` runs BEFORE `assert_upgraded`; if a
|
||||
rollback occurs it raises FIRST with the honest "head did not stay healthy" message; if no rollback occurs,
|
||||
HC1 commit-match assertion still runs unmodified. A deliberately wrong stamp (e.g. deploying eb96de94
|
||||
as the chaos version) would still fail HC1 exactly as before. M2 will demonstrate this with a live negative test.
|
||||
|
||||
**One nuance (not a blocker):** The "06-05→06-10 change" being specifically "heavier resident load from
|
||||
rcust-phase stacks" is circumstantially supported by the timeline, but repro1 (isolated, no concurrent apps)
|
||||
also showed drift — the mechanism fires under general memory pressure during discourse's precompile, not
|
||||
only when other apps are warm. The exact delta between run 184 (06-05, passed) and subsequent runs is
|
||||
intermittency of memory pressure, proven by repro2 (warm volumes → faster precompile → task survived) vs
|
||||
repro4 (fresh boot → slower precompile → task failed). The ROOT CAUSE mechanism is proven by direct
|
||||
evidence; the specific "what changed between 06-05 and 06-10" reduces to: heavier/more-variable memory
|
||||
pressure, the mechanism was always latent. This doesn't weaken M1 — the fix eliminates the exposure.
|
||||
|
||||
**Verdict: M1 PASS.** Root cause attributed by direct evidence; minimal reproducible demonstration
|
||||
confirmed; fix (stop-first overlay + assert_upgrade_converged) implemented and working; HC1 unweakened;
|
||||
blast-radius sweep complete. Builder cleared to proceed to M2.
|
||||
|
||||
---
|
||||
|
||||
## M2: PASS @2026-06-11T17:58Z
|
||||
|
||||
Cold verification from `/srv/cc-ci/cc-ci-adv`. JOURNAL-dstamp not read before verdict (anti-anchoring).
|
||||
|
||||
**Check 1 — Build 450 results (level, tiers, flags):** PASS
|
||||
`cat /var/lib/cc-ci-runs/450/results.json`:
|
||||
- `"level": 5` ✓
|
||||
- `"recipe": "discourse"`, `"ref": "7ae7b0f76efb"`, `"pr": "2"` ✓
|
||||
- All tiers: `"install": "pass"`, `"upgrade": "pass"`, `"backup": "pass"`, `"restore": "pass"`, `"custom": "pass"` ✓
|
||||
- All rungs: `"install": "pass"`, `"upgrade": "pass"`, `"backup_restore": "pass"`, `"functional": "pass"`, `"lint": "pass"` ✓
|
||||
- `"clean_teardown": true`, `"no_secret_leak": true` ✓
|
||||
- Timestamp: `"finished": 1781199631.4...` (2026-06-11 ~17:40 UTC) ✓
|
||||
- `screenshot.png` present (discourse functional screenshot)
|
||||
|
||||
**Check 2 — JUnit XML: test_upgrade_reconverges PASS (HC1 satisfied):** PASS
|
||||
`grep -c '<failure\|<error' upgrade__generic__test_upgrade.xml` → 0
|
||||
Full XML: `<testcase classname="tests._generic.test_upgrade" name="test_upgrade_reconverges" time="0.260"/>`
|
||||
(no `<failure>` child). `test_upgrade_reconverges` directly calls `generic.assert_upgraded(live_app, meta)`.
|
||||
`assert_upgraded` at `generic.py:174-175` does the HC1 commit-match: `chaos_commit == head_ref`.
|
||||
Test PASSED → `chaos_commit = 7ae7b0f7` matched `head_ref = 7ae7b0f7` ✓
|
||||
|
||||
**Check 3 — PR comment 14347 (!testme path):** PASS
|
||||
Comment 14346 body = `!testme` (the trigger).
|
||||
Comment 14347 body (bot response):
|
||||
`<!-- cc-ci:testme -->\n🌻 **cc-ci** — \`discourse\` @ \`7ae7b0f7\` ✅ **passed**\n[...links to run 450 summary.png + badge + drone build 450...]`
|
||||
Confirmed via Gitea API. Run directory `/var/lib/cc-ci-runs/450/` exists with full contents.
|
||||
!testme → bridge ack → drone build 450 → run 450 results → PR comment ✅ passed. Path verified.
|
||||
|
||||
**Check 4 — DEFERRED entry closed:** PASS
|
||||
`machine-docs/DEFERRED.md` lines 346-366: ✅ RESOLVED @2026-06-11 (phase dstamp, Builder) with:
|
||||
- Root cause narrative (rollback mechanism)
|
||||
- Direct evidence pointer (dstamp-repro4.console.log)
|
||||
- Fix commits (0cc31a5 + e9c26c7)
|
||||
- Real CI proof (drone build #450, LEVEL 5)
|
||||
- Blast-radius note (only discourse; harness guard covers all rollback-policy recipes)
|
||||
- Cross-references (STATUS/JOURNAL/REVIEW-dstamp)
|
||||
|
||||
**Check 5 — HC1 teeth (wrong stamp still FAILs):** PASS
|
||||
*Negative control (pre-fix, existing run):* `m2p-discourse/results.json` shows HC1 caught wrong stamp:
|
||||
`AssertionError: upgrade deployed chaos commit 'eb96de94+U', not the intended PR-head '7ae7b0f76efb'
|
||||
— the re-checkout to the code under test failed, so the upgrade is not exercising the PR's changes (HC1)`
|
||||
This is HC1 raising on `eb96de94 ≠ 7ae7b0f7`. HC1 commit-match assertion WORKS.
|
||||
|
||||
*Code unchanged (from M1):* `generic.py:174-175` commit-match assertion unmodified. The fix adds
|
||||
`assert_upgrade_converged` BEFORE `assert_upgraded` — it catches rollback EARLIER with an honest message
|
||||
but does NOT bypass HC1. If a non-rollback wrong stamp were deployed (e.g. abra bug stamping wrong commit),
|
||||
`assert_upgrade_converged` would see `completed` and pass, then HC1 would FAIL on the commit mismatch.
|
||||
|
||||
*Post-fix rollback path:* `assert_upgrade_converged` raises `RuntimeError` on `rollback_completed` →
|
||||
upgrade FAILS with honest "head did not stay healthy" → HC1 doesn't even run but test is RED.
|
||||
Both paths (rollback → caught by assert_upgrade_converged; wrong stamp without rollback → caught by HC1)
|
||||
still FAIL. The pre-fix negative controls (m2p-discourse, repro1, repro4) demonstrate the wrong-stamp
|
||||
path is always caught; the fix only changes HOW it's reported and at which point.
|
||||
|
||||
**Blast-radius (confirmed at M1, still valid):** Only discourse affected. keycloak/n8n PASS L4
|
||||
in 06-10/06-11 era. General `assert_upgrade_converged` guard now covers all rollback-policy recipes.
|
||||
|
||||
**Phase DoD summary:**
|
||||
- ✅ Drift mechanism attributed with reproducible evidence (repro4 direct evidence)
|
||||
- ✅ Fixed at the true root (stop-first overlay + assert_upgrade_converged)
|
||||
- ✅ Discourse back at real level in real CI via drone !testme (build 450, LEVEL 5)
|
||||
- ✅ No other recipe silently affected (blast-radius sweep, keycloak/n8n PASS)
|
||||
- ✅ HC1 unweakened and adversarially re-proven (m2p-discourse negative control + code inspection)
|
||||
- ✅ DEFERRED closed with pointers
|
||||
|
||||
**Verdict: M2 PASS. All phase dstamp DoD items satisfied. Builder cleared for ## DONE.**
|
||||
184
machine-docs/REVIEW-kuma.md
Normal file
184
machine-docs/REVIEW-kuma.md
Normal file
@ -0,0 +1,184 @@
|
||||
# REVIEW — phase `kuma` (uptime-kuma create-a-monitor functional test)
|
||||
|
||||
Adversary verdict log. Append-only. SSOT: `cc-ci-plan/plan-phase-kuma-monitor.md`.
|
||||
|
||||
## Phase orientation (2026-06-11T18:03Z)
|
||||
|
||||
Builder clone: `/srv/cc-ci/cc-ci`; Adversary clone: `/srv/cc-ci/cc-ci-adv`.
|
||||
Phase goal: add functional test that completes uptime-kuma's first-run setup wizard and exercises
|
||||
its core function — create a monitor, see it probe a target, assert UP + real probe timestamp.
|
||||
Negative test (monitor → dead target → DOWN) required if it fits the runtime budget.
|
||||
|
||||
Two gates:
|
||||
- **M1** — test implemented + green locally; approach justified; bounded waits; real assertions
|
||||
- **M2** — drone-path green (≥2 consecutive runs); flake check; DEFERRED closed
|
||||
|
||||
Pre-phase independent research notes:
|
||||
- uptime-kuma uses Socket.IO for ALL management operations (setup wizard, login, monitor CRUD)
|
||||
- Existing tests: Socket.IO handshake (EIO v4), SPA branding, health check — NONE exercise wizard/monitor
|
||||
- Two viable approaches per plan: (a) python-socketio client speaking events; (b) Playwright UI
|
||||
- Key verification concerns for M1:
|
||||
- Probe reality: must confirm a *real* HTTP check occurred (timestamp advance + status from
|
||||
uptime-kuma's state, not echo of config)
|
||||
- Secret safety: generated admin creds must not appear in logs or test output
|
||||
- Budget: target ≤90s added to functional tier; must use bounded poll not sleep
|
||||
- Negative teeth: dead-target monitor must go DOWN (proves probe isn't stub) — required unless
|
||||
runtime budget forces explicit justification
|
||||
- Existing `tests/uptime-kuma/functional/` dir has 3 files: health_check, socketio_handshake,
|
||||
spa_branding — all pass in CI (build #91 was green for uptime-kuma level 5)
|
||||
- Phase plan says new test goes in `tests/uptime-kuma/functional/` (or `playwright/` if option b)
|
||||
|
||||
## Adversary pre-flight checks (2026-06-11T18:03Z)
|
||||
|
||||
uptime-kuma Socket.IO event map (from source / prior investigation):
|
||||
- Setup wizard: `setup` event with `{username, password}` → response `{ok: true}`
|
||||
- Login: `login` event with `{username, password, token: ""}` → response `{ok: true, token: "..."}`
|
||||
- Add monitor: `add` event with monitor config → response `{ok: true, monitorID: N}`
|
||||
- Heartbeat list: `heartbeatList` event or `uptime` event to check recent probe status
|
||||
- Monitor status: `getMonitorList` or heartbeat events contain `{status: 1}` (UP) or `{status: 0}` (DOWN)
|
||||
|
||||
Adversary independent acceptance criteria (what I will cold-verify for M1):
|
||||
1. Test file in correct location per plan (tests/uptime-kuma/functional/ or playwright/)
|
||||
2. Setup wizard completed and login token obtained (not hardcoded)
|
||||
3. Monitor created pointing at a harness-controlled URL (not a stub/no-op)
|
||||
4. Wait loop is BOUNDED (deadline/max_wait, not open-ended sleep)
|
||||
5. Assertion is on ACTUAL probe data: at minimum one heartbeat with status=1 + timestamp > deploy time
|
||||
6. Admin credentials NOT printed/logged in test output
|
||||
7. Negative test included OR explicit runtime-budget justification in DECISIONS.md
|
||||
8. Runtime ≤ ~90s added (measure from CI timing)
|
||||
|
||||
## Independent pre-flight findings (2026-06-11T18:05Z)
|
||||
|
||||
**Critical: python-socketio NOT available on cc-ci.**
|
||||
```
|
||||
cc-ci-run -c 'import socketio' # → ModuleNotFoundError: No module named 'socketio'
|
||||
cc-ci-run -c 'from playwright.sync_api import sync_playwright; print("ok")' # → ok
|
||||
```
|
||||
Implication: option (a) python-socketio requires a harness.nix + nixos-rebuild change; option (b)
|
||||
Playwright works immediately from existing infrastructure. Builder must justify their choice in
|
||||
DECISIONS.md regardless.
|
||||
|
||||
**uptime-kuma recipe pinned at 2.2.1** (image `louislam/uptime-kuma:2.2.1`).
|
||||
Socket.IO port 3001, routed through Traefik `web-secure` entrypoint.
|
||||
|
||||
**uptime-kuma Gitea mirror exists** (recipe-maintainers/uptime-kuma), no open PRs yet. Builder
|
||||
will need to create a test PR.
|
||||
|
||||
**Real probe evidence requirements I will enforce at M1 cold-verify:**
|
||||
- heartbeat data must contain entries with `status` field (1=UP, 0=DOWN)
|
||||
- heartbeat timestamps must be AFTER test start (not from config echo)
|
||||
- For uptime-kuma 2.x: `heartbeatList` socket event OR API poll at `/api/status-page/heartbeat/...`
|
||||
carries real probe results; event `uptime` also carries historical data
|
||||
- The monitor's first heartbeat entry is sufficient if it has: `status: 1`, `time` > deploy timestamp
|
||||
|
||||
Builder has not yet started (no STATUS-kuma.md, no kuma commits). Waiting for M1 claim.
|
||||
|
||||
---
|
||||
|
||||
## M1: PASS @2026-06-11T18:26Z
|
||||
|
||||
**Claim commit:** `fe8922c claim(kuma): M1 PASS — test_monitor_wizard green at LEVEL 5 via drone build #460`
|
||||
**Test commit:** `8da59cf feat(kuma): implement wizard+monitor Playwright test`
|
||||
|
||||
### Cold-verify evidence (Adversary-independent, from own clone + ssh cc-ci)
|
||||
|
||||
**1. Test file location and content** ✓
|
||||
- File: `tests/uptime-kuma/playwright/test_monitor_wizard.py` (167 lines)
|
||||
- Correct placement per plan §2 "option b" + discovery.py `playwright/` subdir
|
||||
- Discovery confirmed: `runner/harness/discovery.custom_tests` recurses into `playwright/`
|
||||
- `live_app` fixture from root `tests/conftest.py` works (session-scoped, reads `CCCI_APP_DOMAIN`)
|
||||
|
||||
**2. Drone build #460 results (read from /var/lib/cc-ci-runs/460/results.json on cc-ci)**
|
||||
```
|
||||
level: 5
|
||||
recipe: uptime-kuma ref: eb4521cc5d77
|
||||
functional.test_uptime_kuma_root_serves [pass] 20ms
|
||||
functional.test_socketio_polling_handshake [pass] 26ms
|
||||
functional.test_uptime_kuma_spa_has_branding [pass] 27ms
|
||||
playwright.test_monitor_wizard_and_probe [pass] 2817ms
|
||||
clean_teardown: True
|
||||
no_secret_leak: True
|
||||
playwright count: 1
|
||||
```
|
||||
All tiers PASS: install/upgrade/backup/restore/custom/lint = Level 5.
|
||||
|
||||
**3. Probe reality** ✓
|
||||
- `test_monitor_wizard_and_probe` PASSED with both positive and negative assertions:
|
||||
- Self-probe monitor → status "Up" (requires real Socket.IO heartbeat from uptime-kuma server)
|
||||
- Dead-port monitor (`127.0.0.1:19999`) → status "Down" (proves probe engine not a stub)
|
||||
- Heartbeat datetime row present (regex `\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}`) — real timestamp
|
||||
- 2.817s runtime proves fast connection-refused (dead-port negative check confirmed real)
|
||||
|
||||
**4. Secret safety** ✓
|
||||
- `_pw` (64-char UUID hex) used only in `.fill()` calls — never printed, never in assertion messages
|
||||
- `no_secret_leak: True` confirmed by independent results.json read
|
||||
|
||||
**5. Approach justification** ✓
|
||||
- `machine-docs/DECISIONS.md` entry "2026-06-11 — uptime-kuma: Playwright (option b)" present
|
||||
- Confirms python-socketio absent, Playwright handles Socket.IO transparently, selectors confirmed
|
||||
in 2.2.1 compiled bundle `dist/assets/index-D_mnxLA0.js`
|
||||
|
||||
**6. Runtime budget** ✓
|
||||
- 2.817s actual ≪ 90s target
|
||||
|
||||
**7. Nothing weakened** ✓
|
||||
- All 3 existing custom tests still PASS (health_check, socketio_handshake, spa_branding)
|
||||
- No existing assertions removed or softened
|
||||
|
||||
**8. PR comment** ✓
|
||||
- git.autonomic.zone/recipe-maintainers/uptime-kuma/pulls/3 shows:
|
||||
`🌻 cc-ci — uptime-kuma @ eb4521cc ✅ passed`
|
||||
|
||||
### M1 verdict: **PASS** — Builder cleared to proceed to M2.
|
||||
|
||||
Note: build #462 (flake-check second run for M2) was already in progress at time of this verdict.
|
||||
DEFERRED close + PARITY.md update are M2 pre-conditions per BACKLOG.
|
||||
|
||||
---
|
||||
|
||||
## M2: PASS @2026-06-11T18:32Z
|
||||
|
||||
**Claim commit:** `9afdf3d claim(kuma): M2 — build #462 LEVEL 5 PASS (flake #2); DEFERRED closed; PARITY updated`
|
||||
|
||||
### Cold-verify evidence (Adversary-independent)
|
||||
|
||||
**1. Build #462 results (read from /var/lib/cc-ci-runs/462/results.json on cc-ci)**
|
||||
```
|
||||
level: 5 recipe: uptime-kuma ref: eb4521cc5d77
|
||||
functional.test_uptime_kuma_root_serves [pass] 16ms
|
||||
functional.test_socketio_polling_handshake [pass] 26ms
|
||||
functional.test_uptime_kuma_spa_has_branding [pass] 27ms
|
||||
playwright.test_monitor_wizard_and_probe [pass] 2746ms
|
||||
clean_teardown: True no_secret_leak: True playwright count: 1
|
||||
```
|
||||
|
||||
**2. 2 consecutive green runs** ✓
|
||||
- Build #460: Level 5, `test_monitor_wizard_and_probe` PASS 2817ms
|
||||
- Build #462: Level 5, `test_monitor_wizard_and_probe` PASS 2746ms
|
||||
- Both same ref (eb4521cc), same recipe, same PR #3
|
||||
|
||||
**3. DEFERRED.md closed** ✓
|
||||
```
|
||||
[x] CLOSED @2026-06-11 (Builder, phase kuma): tests/uptime-kuma/playwright/test_monitor_wizard.py
|
||||
implemented and proven in real CI … Drone builds #460 + #462 both LEVEL 5 …
|
||||
```
|
||||
|
||||
**4. PARITY.md updated** ✓
|
||||
- New row for `tests/uptime-kuma/playwright/test_monitor_wizard.py` with full rationale
|
||||
- Documents Up/Down probe, heartbeat datetime, Socket.IO-driven status
|
||||
|
||||
**5. PR comment build #462** ✓
|
||||
- `🌻 cc-ci — uptime-kuma @ eb4521cc ✅ passed`
|
||||
|
||||
### Phase DoD check
|
||||
|
||||
Per `plan-phase-kuma-monitor.md` §5:
|
||||
- ✅ uptime-kuma proves actual function (wizard + real probe — Up AND Down confirmed)
|
||||
- ✅ Flake-checked (2 consecutive Level 5 green runs #460 + #462)
|
||||
- ✅ Budget held (2.75–2.82s actual ≪ 90s target)
|
||||
- ✅ DEFERRED checked off (entry `[x] CLOSED @2026-06-11`)
|
||||
- ✅ M1 fresh PASS (filed 2026-06-11T18:26Z)
|
||||
- ✅ M2 fresh PASS (this entry)
|
||||
- No VETO standing
|
||||
|
||||
### M2 verdict: **PASS** — all DoD satisfied. Builder may write `## DONE`.
|
||||
148
machine-docs/REVIEW-lvl5.md
Normal file
148
machine-docs/REVIEW-lvl5.md
Normal file
@ -0,0 +1,148 @@
|
||||
# REVIEW — Phase lvl5 (L5 lint rung + de-cap) — Adversary verdicts
|
||||
|
||||
Cold-verification ledger (append-only). Each verdict formed from the plan (SSOT), the code/git
|
||||
history, the verification info in STATUS-lvl5.md, and my own cold re-run — NOT from JOURNAL
|
||||
(anti-anchoring, §6.1). JOURNAL not consulted before this verdict.
|
||||
|
||||
---
|
||||
|
||||
## M1 — Implementation complete (pre-merge): **PASS** @ 2026-06-11T07:54Z
|
||||
|
||||
Branch `phase-lvl5` @ `3d8d286cf3f2df7d164bf458f07bbb916cc18f2b` (claim 24baac5). Implementation
|
||||
deliberately NOT on main (reverts 589943f/cd62743 hold it pre-merge) — confirmed; only the
|
||||
DECISIONS entry (392f7df) is on main. Verified from a **fresh cold clone** on the cc-ci host
|
||||
(`/tmp/adv-lvl5`, cloned from origin, checked out phase-lvl5; HEAD matched 3d8d286).
|
||||
|
||||
**Acceptance per plan §4 M1 — all satisfied:**
|
||||
|
||||
1. **Cold clone + HEAD** — `git rev-parse HEAD` = 3d8d286 ✓ (matches claim).
|
||||
2. **Unit suite (CI host venv)** — `cc-ci-run -m pytest tests/unit/ -q` → **246 passed** in 5.32s
|
||||
✓ (matches claimed count).
|
||||
3. **Repo lint** — `nix develop .#lint --command bash scripts/lint.sh` → **lint: PASS** ✓.
|
||||
4. **De-capped `compute_level` correct on ALL 4 mission worked examples** (hand-traced against
|
||||
`level.py` + verified by the rewritten test_level.py):
|
||||
- install✔ upgrade✘ backup✔ functional✔ lint✔ → **L1** (fail blocks) ✓
|
||||
- install✔ upgrade✔ backup skip functional✔ lint✔ → **L5** (intentional skip climbs — the
|
||||
de-cap; was L2 under old rule) ✓
|
||||
- install✔ upgrade✔ backup **unver** functional✔ lint✔ → **L2** (unver blocks) ✓
|
||||
- all four ✔, lint unver → **L4** (unverified top rung not earned) ✓
|
||||
Formula `level = max i: rung_i==pass ∧ all j<i ∈ {pass,skip}` implemented exactly
|
||||
(pass→advance, skip→continue, fail/unver→break). 0 if none.
|
||||
5. **N/A classification table matches code.** `derive_rungs` (results.py) implements the
|
||||
DECISIONS table verbatim, incl. the subtle upgrade split: `skip ∧ ¬has_upgrade_target` →
|
||||
`skip` (structural, climbs); a prior-stage abort (`skip`/None WITH a target, undeclared) →
|
||||
`unver` (blocks). install never skips; backup_restore skip iff not-capable or EXPECTED_NA;
|
||||
functional skip iff EXPECTED_NA else unver; **lint pass/fail-or-unver, NEVER skip** (no N/A
|
||||
escape hatch, §2 item 5; EXPECTED_NA["lint"] ignored). Default-unclassifiable = unver. ✓
|
||||
6. **§2.3 mirror-context decision reviewed — NO rule filtered.** Executor (`lint.py`) lints a
|
||||
pristine scratch clone of the per-run tree at the tested sha; origin→local path makes abra's
|
||||
tag force-fetch work offline (no auth, no go-git "reference not found"), and the run's real
|
||||
tags ride along so R014 evaluates real content. The plumbing pollution is solved by context,
|
||||
not exemptions. Confirmed by **real-abra behavioral probe** (not just synthetic fixtures):
|
||||
- `run_lint("hedgedoc", …)` clean → `{'status':'pass',...}` ✓ (proves scratch-clone makes
|
||||
abra lint actually run — no FATA).
|
||||
- inject lightweight tag → `{'status':'fail','detail':'error rule(s) unsatisfied: R014',
|
||||
'rules_failed':['R014']}` ✓ (proves the classifier has teeth; R014 is NOT suppressed).
|
||||
Classifier correctly recognizes `rc=0`-with-critical-errors (parses table + "critical errors
|
||||
present" sentinel, fails closed on disagreement); only content-FATA ("unable to validate
|
||||
recipe") → fail, all other non-zero → unver.
|
||||
7. **Verdict-neutrality — code inspection + targeted tests.** `run_lint` invoked once
|
||||
(run_recipe_ci.py:942), defaults to `unver`, double-wrapped in try/except (crash → stays
|
||||
unver, non-fatal print), runs BEFORE the tiers at `head_ref` (the exact tested ref). Its
|
||||
result is consumed ONLY at build_results (line 1278, "non-fatal, verdict unaffected"); NO
|
||||
verdict computation reads it. 60s hard budget, never raises. Targeted tests pass:
|
||||
`test_run_lint_missing_recipe_is_unver_not_raise`,
|
||||
`test_build_results_no_lint_given_is_unverified_never_pass`. ✓
|
||||
8. **cap/cap_reason/capped fully removed** from active code/schema/card/dashboard/docs. grep over
|
||||
runner/dashboard/docs/tests finds the words only in (a) the unrelated screenshot timeout-cap,
|
||||
(b) "capable"/max-users, (c) explicit test/doc assertions that the fields are ABSENT in
|
||||
schema 2 and that old schema-1 artifacts (which carry level_cap_reason) still render with no
|
||||
relabeling — history-compat covered by test_card/test_dashboard (green). ✓
|
||||
|
||||
No verdict regression, no run-verdict coupling, no rule suppression, no silent pass. **M1 PASS.**
|
||||
Builder cleared to merge phase-lvl5 → main and proceed to P3/P4 (M2). No VETO.
|
||||
|
||||
**Scope note (carried to M2):** M1 verified the lint executor + classifier + level math on real
|
||||
abra output and the unit surface. M2 must still prove, on real CI end-to-end: ≥1 genuine L5,
|
||||
≥1 lint-blocked L4, ≥1 N/A-skip climb, drone `!testme` ×2, canaries at designed levels under the
|
||||
NEW formula, old artifacts rendering live, durations not inflated (lint ≤~60s; observed ~0.7s),
|
||||
the before/after level table for ALL enrolled recipes, and card/dashboard/badge visually (PNG/SVG).
|
||||
|
||||
---
|
||||
|
||||
## M2 — Proven in real CI: **PASS** @ 2026-06-11T11:27Z
|
||||
|
||||
Main @ `a521d43` (impl merged 08e6cc8 + PR-path fix 68c3486). Cold-verified from a **fresh clone
|
||||
of main** on the cc-ci host (`/tmp/adv-m2`), drone API (token from /run/secrets), live HTTPS
|
||||
artifacts, and Read PNGs. JOURNAL not consulted before this verdict.
|
||||
|
||||
**Acceptance per plan §4 M2 + §6 DoD — all satisfied:**
|
||||
|
||||
1. **Unit suite + lint (fresh clone main).** `cc-ci-run -m pytest tests/unit/ -q` → **247 passed**;
|
||||
`scripts/lint.sh` → PASS. The new PR-path regression test
|
||||
`test_run_lint_detached_pr_tree_lints_exact_ref` passes (covers fix 68c3486: abra lint checks
|
||||
out the repo DEFAULT BRANCH, so a detached scratch clone would FATA or silently lint a stale
|
||||
branch; fix forces local main AT the tested ref + repoints origin to scratch → lints the PR
|
||||
head content). My M1 smoke only exercised the HEAD path; this closes that gap.
|
||||
2. **Genuine L5 (full clean climb).** Runs 398 hedgedoc / 406 immich / 407 plausible / 413 mumble:
|
||||
results.json schema=2, level=5, all 5 rungs pass, no cap keys, drone build status=success.
|
||||
3. **Lint-blocked L4, verdict-neutral — the central claim.** Run 405 custom-html PR4:
|
||||
results.json level=4, lint=fail rules_failed=[R011], all five TIERS pass
|
||||
(install/upgrade/backup/restore/custom), **drone build 405 status=SUCCESS**, and the bridge
|
||||
`reflected outcome build 405 (custom-html PR #4): success` to the PR. A lint failure caps the
|
||||
level at 4 but does NOT flip the run verdict. Card PNG shows lint ✗ FAIL red, "level 4 of 5",
|
||||
badge #a0b93f. Neutrality proven BOTH directions (415/416 red with lint=pass — see #6).
|
||||
4. **N/A-skip climb (the de-cap).** Run 399 custom-html-tiny: backup_restore=skip with declared
|
||||
reason in skips.intentional ("stateless static file server … no backupbot.backup label"),
|
||||
other rungs pass, **level=5** (was L2 @ #205). Card PNG shows backup/restore "⊘ INTENTIONAL
|
||||
SKIP" + reason, level 5 of 5. A formerly-capped non-backup-capable recipe now climbs.
|
||||
5. **Drone !testme path ×3, GENUINE (not manual API).** ccci-bridge poll logs:
|
||||
`[poll] triggered build 405 for custom-html@36b362aa (PR #4, comment 14332)`,
|
||||
`406 immich@107d7220 (PR #2, comment 14333)`, `407 plausible@13458fac (PR #3, comment 14334)`,
|
||||
each followed by `reflected outcome … success`. Build params confirm RECIPE/PR/REF match the
|
||||
real PR heads. ≥2 required; 3 delivered, all on real PRs showing the lint rung.
|
||||
6. **Canaries at re-derived designed level + backup-fail still blocks.** 415 (bkp-bad) / 416
|
||||
(rst-bad): drone build status=**failure** (red), results.json level=1, rungs {install pass,
|
||||
upgrade skip(structural — no version tags on SRC+REF mirror), backup_restore FAIL, functional
|
||||
unver, lint pass}. New-formula trace: install(1) → upgrade skip(climb) → backup_restore
|
||||
fail(BLOCK) → L1. RED is caused by the failing backup/restore TIER (verdict logic untouched),
|
||||
NOT by lint (lint=pass). Re-derivation is sound; matches OLD-rule level too (old: upgrade N/A
|
||||
caps at L1) — no regression, same designed level, red either way.
|
||||
7. **Unverified-blocks (mission example #3), synthesized.** host run
|
||||
`/var/lib/cc-ci-runs/lvl5-unver-demo/results.json`: schema=2, level=2, rungs {install pass,
|
||||
upgrade pass, backup_restore UNVER, functional pass, lint pass}, skips.unintentional=
|
||||
[backup_restore]. backup unver blocks at L2 even though functional+lint pass above it. ✓
|
||||
8. **Durations not inflated.** drone build wall-times: 398=100s, 399=45s, 405=61s, 406 immich=199s
|
||||
(shot baseline 198-199s), 407 plausible=164s (shot baseline 166s), 413=80s. lint adds ~0.7s;
|
||||
the two cross-phase baselines are flat (407 slightly faster). No duration regression.
|
||||
9. **Old artifacts render, no relabel.** /runs/370 (schema=1, level=4, level_cap_reason present)
|
||||
serves 200 (results.json + summary.png); dashboard `/` + `/recipe/immich` 200 with mixed
|
||||
schema-1/schema-2 rows; unit history-compat tests green.
|
||||
10. **lint.txt served.** /runs/398/lint.txt 200 — full real abra table (HEAVY-box), cmd + rc=0 +
|
||||
status=pass header, ref=09bf4d54 (hedgedoc's EXACT tested ref).
|
||||
11. **Badges number+colour only.** hedgedoc badge ">level 5<" #3fb950; custom-html ">level 4<"
|
||||
#a0b93f; grep finds NO cap/skip/na/reason language in badge SVGs. Matches operator spec.
|
||||
12. **P3 matrix 19/19 lint PASS** (BACKLOG-lvl5.md) via documented scratch-clone method; no mirror
|
||||
PRs / DEFERRED needed; warn-severity misses only (don't fail the rung). lasuite-meet R014 now
|
||||
passes genuinely (tag annotated upstream — not suppressed). **Before/after table: every level
|
||||
shift is explained by the rule change** — L4→L5 (+lint, baseline from real artifacts + P3
|
||||
sweep), de-cap L2→L5 (custom-html-tiny proven #399; mailu same mechanism), L4 lintdemo (#405),
|
||||
canary L1, bluesky N/A consistent. **No unexplained shift / no downward regression.** "Analytic
|
||||
5" cells are derivation-checkable from two evidenced inputs (real baseline tiers + proven lint).
|
||||
13. **No secret leak.** Independent sweep: no /run/secrets infra-secret VALUES and no generated
|
||||
app-credential patterns appear in any published run artifact (the new lint.txt surface incl.).
|
||||
results.json flags no_secret_leak=true + clean_teardown=true across runs.
|
||||
|
||||
**§6 Definition of Done satisfied:** new level system live on main and visible end-to-end
|
||||
(results.json→card→dashboard→badge); L5 = abra recipe lint on the tested ref; capping fully
|
||||
removed (no cap/cap_reason/capped); all 19 enrolled recipes linted + dispositioned with an
|
||||
adversary-checked before/after table; ≥1 real L5 + ≥1 lint-blocked L4 + ≥1 N/A-skip climb through
|
||||
real CI incl. the drone path ×3; old artifacts unharmed; M1 (cfc87fd) + M2 fresh Adversary
|
||||
PASSes; no verdict or duration regressions.
|
||||
|
||||
**No VETO. Builder is cleared to write `## DONE` to STATUS-lvl5.md.**
|
||||
|
||||
Out-of-scope note (Builder's STATUS query): the WC5 promote-on-green-cold observation (a
|
||||
STAGES-filtered hand-run promoted custom-html's canonical) is pre-existing and orthogonal to the
|
||||
level system — NOT a lvl5 finding/regression and not a DONE blocker. If the Builder wants it
|
||||
tracked, DEFERRED.md/IDEAS.md is the right home; I'm not filing it as an [adversary] finding.
|
||||
190
machine-docs/REVIEW-mailu.md
Normal file
190
machine-docs/REVIEW-mailu.md
Normal file
@ -0,0 +1,190 @@
|
||||
# REVIEW — phase `mailu` (backupbot labels + backup/restore coverage)
|
||||
|
||||
Adversary verdict log. Append-only. SSOT: `cc-ci-plan/plan-phase-mailu-backup.md`.
|
||||
|
||||
## Phase orientation (2026-06-11T17:59Z)
|
||||
|
||||
Builder clone: `/srv/cc-ci/cc-ci`; Adversary clone: `/srv/cc-ci/cc-ci-adv`.
|
||||
Phase goal: mirror PR adding backupbot v2 labels to mailu recipe + proof backup→wipe→restore on real
|
||||
seeded mail data passes CI.
|
||||
|
||||
Pre-phase independent research notes:
|
||||
- Mailu compose.yml analyzed. Critical durable volumes:
|
||||
- `mailu:/data` on `admin` svc — SQLite DB (accounts, domains, aliases, DKIM config)
|
||||
- `dkim:/dkim` on `admin` svc — DKIM signing keys
|
||||
- `mail:/mail` on `imap` svc — mail store (Maildir, all user messages)
|
||||
- `redis:/data` on `db` svc — Redis (transient: rate-limits, sessions) — likely NOT needed for restore
|
||||
- Other volumes (rspamd, webmail, certs, mailqueue) — transient/cache, NOT durable
|
||||
- Correct backupbot v2 label placement: `admin` service (for DB + DKIM) and `imap` service (for mail store)
|
||||
- Backupbot v2 map syntax confirmed from keycloak/immich/mattermost-lts recipes
|
||||
- SQLite `/data` — pre-hook may be needed to dump consistently; or copy is safe if admin is quiesced
|
||||
- Mail store backup: Maildir is file-based, safe to copy live
|
||||
- Recipe mirror has open PR#2 (upgrade-3.1.0+2024.06.52) — backupbot PR must be separate
|
||||
|
||||
Awaiting M1 claim from Builder.
|
||||
|
||||
---
|
||||
|
||||
## M1 FAIL @2026-06-11T20:58Z
|
||||
|
||||
**Claim**: build #473 LEVEL 5 PASS, backup→wipe→restore on real seeded mail data proven.
|
||||
|
||||
**Verdict: FAIL** — the backup/restore test exercises only the SQLite `/data` volume; the Maildir
|
||||
`/mail` volume is labeled and backed up but is NOT specifically tested for restoration.
|
||||
|
||||
### What I verified (cold)
|
||||
|
||||
1. **PR#3 labels correct** (`add-backupbot-labels`, head `edc0201a79d3`):
|
||||
- `admin` service: `backupbot.backup: "true"` + `backupbot.backup.path: "/data"` ✓
|
||||
- `imap` service: `backupbot.backup: "true"` + `backupbot.backup.path: "/mail"` ✓
|
||||
- Version bump: `3.0.1` → `3.0.2+2024.06.52` ✓
|
||||
- DKIM exclusion intentional and documented in PR desc ✓
|
||||
|
||||
2. **Build #473 evidence** (drone API + results.json):
|
||||
- status: success, level: 5, all 5 rungs PASS ✓
|
||||
- `clean_teardown: true`, `no_secret_leak: true` ✓
|
||||
- `test_backup_captures_mailbox` PASS — `citest@<domain>` in config-export at backup time ✓
|
||||
- `test_restore_returns_mailbox` PASS — `citest@<domain>` back in config-export after restore ✓
|
||||
- Backup snapshot `13eee64e`: 139 files, 85MB ✓
|
||||
- Cold teardown: `abra app ls --server cc-ci` shows no mailu apps ✓
|
||||
- No plaintext secrets in compose.yml (secrets section uses swarm `external: true` refs) ✓
|
||||
- PARITY.md updated: P4 COVERED ✓
|
||||
|
||||
3. **Backupbot v2 syntax verified** against keycloak/mattermost-lts/n8n patterns — `backupbot.backup.path`
|
||||
is valid v2 syntax for specifying the backup path ✓
|
||||
|
||||
### Failing item: `/mail` volume restoration not tested
|
||||
|
||||
**Plan requirement** (`plan-phase-mailu-backup.md` §2.3):
|
||||
> "ensure the restore tier's data-integrity seed/verify actually exercises MAIL data (a seeded
|
||||
> mailbox + message that survives backup→wipe→restore — extend the existing functional helpers if
|
||||
> the current seed is too shallow; never weaken anything)"
|
||||
|
||||
**What the test does** (`ops.py`):
|
||||
- `pre_backup`: creates user account `citest@<domain>` in SQLite via `flask mailu user` — this
|
||||
is an account record in `/data` (SQLite), NOT a mail message in `/mail` (Maildir)
|
||||
- `pre_restore`: deletes `citest@<domain>` from SQLite via sqlite3 — only wipes the DB record;
|
||||
the Maildir at `/mail` is untouched throughout
|
||||
- `test_restore.py`: asserts `citest@<domain>` is back in `config-export` — this proves the SQLite
|
||||
(`/data`) backup/restore worked, but says nothing about the Maildir (`/mail`)
|
||||
|
||||
**What is missing**: the test never (a) seeds an actual email message into the maildir, (b) wipes
|
||||
maildir content before restore, or (c) verifies a message survived the restore cycle. If backupbot
|
||||
silently failed to restore the `/mail` volume, this test would still PASS.
|
||||
|
||||
**Fix required** (using existing infra from `test_mail_flow.py`):
|
||||
1. `pre_backup`: after creating `citest@<domain>`, inject a uniquely-tagged message into the mailbox
|
||||
(e.g., via in-container `sendmail` → postfix → dovecot deliver, the same path as `test_mail_flow.py`)
|
||||
2. `pre_restore`: also wipe the maildir for `citest@<domain>` (e.g.,
|
||||
`doveadm expunge -u citest@<domain> mailbox INBOX ALL` in the `imap` container)
|
||||
3. `test_restore.py`: after asserting the account is back, also assert the seeded message is present
|
||||
(e.g., `doveadm search -u citest@<domain> mailbox INBOX ALL` returns ≥1 message)
|
||||
|
||||
Note: the Maildir delivery flow is already proven in `test_mail_flow.py` — the tooling exists,
|
||||
the fix is an extension of the existing seed, not a new mechanism.
|
||||
|
||||
### Adversary finding filed
|
||||
|
||||
See BACKLOG-mailu.md `## Adversary findings` — item [ADV-mailu-01].
|
||||
|
||||
Builder: fix the seed shallow enough to exercise `/mail` and re-trigger. PARITY.md and the labels
|
||||
are correct; only the seed depth needs extending.
|
||||
|
||||
---
|
||||
|
||||
## M1 PASS @2026-06-11T21:00Z
|
||||
|
||||
**Re-claim**: build #477 LEVEL 5 PASS, ADV-mailu-01 fix applied, both volumes (`/data` SQLite + `/mail` Maildir) now specifically tested.
|
||||
|
||||
**Verdict: PASS** — the fix correctly extends the backup/restore seed to cover both durable volumes.
|
||||
ADV-mailu-01 is closed.
|
||||
|
||||
### What I verified (cold)
|
||||
|
||||
1. **PR#3 labels correct** (branch `add-backupbot-labels`, head `edc0201a79d36bc87696b0f93f1ee88ad7bd10ed`):
|
||||
- `admin` service: `backupbot.backup: "true"` + `backupbot.backup.path: "/data"` ✓
|
||||
- `imap` service: `backupbot.backup: "true"` + `backupbot.backup.path: "/mail"` ✓
|
||||
- Version bump: `3.0.1` → `3.0.2+2024.06.52` ✓
|
||||
|
||||
2. **Build #477 evidence** (Drone API + `/var/lib/cc-ci-runs/477/results.json`, cold read):
|
||||
- status: success, level: 5, all 5 rungs PASS ✓
|
||||
- `clean_teardown: true`, `no_secret_leak: true` ✓
|
||||
- **backup stage** (all PASS):
|
||||
- `test_backup_captures_mailbox` PASS (1323ms) — SQLite `/data` ✓
|
||||
- `test_backup_captures_mail_message` PASS (133ms) — Maildir `/mail` ✓
|
||||
- **restore stage** (all PASS):
|
||||
- `test_restore_returns_mailbox` PASS (1359ms) — SQLite `/data` ✓
|
||||
- `test_restore_returns_mail_message` PASS (189ms) — Maildir `/mail` ✓
|
||||
- Clean teardown confirmed: `docker stack ls` on cc-ci shows no `mailu-*` stacks ✓
|
||||
- No mailu volumes leaked ✓
|
||||
|
||||
3. **Fix code review** (commit `b9352e8`, cold):
|
||||
- `ops.py::pre_backup`: creates user + injects `ccci-backup-probe` message via `sendmail` in
|
||||
`smtp` container, polls `doveadm search` in `imap` container (≤60s) to confirm delivery ✓
|
||||
- `ops.py::pre_restore`: (1) deletes user from sqlite; (2) `rm -rf /mail/{domain}/{localpart}`
|
||||
in `imap` container — wipes maildir independently from sqlite record ✓
|
||||
- `test_backup_captures_mail_message`: `doveadm search` on `imap` asserts message present at backup time ✓
|
||||
- `test_restore_returns_mail_message`: same search after restore — asserts Maildir restored ✓
|
||||
- Both volumes exercised independently: pre_restore wipes each separately; restore must recover each ✓
|
||||
|
||||
4. **ADV-mailu-01 all three fix items satisfied**:
|
||||
- (1) pre_backup injects a uniquely-tagged message via sendmail→dovecot deliver ✓
|
||||
- (2) pre_restore wipes the maildir (`rm -rf /mail/{domain}/{localpart}`) ✓
|
||||
- (3) test_restore asserts the message is back (`doveadm search` ≥1 result) ✓
|
||||
|
||||
**ADV-mailu-01 closed** — fix is real, CI proves it, no weakening of any assertion.
|
||||
|
||||
Builder is cleared to proceed to M2.
|
||||
|
||||
---
|
||||
|
||||
## M2 PASS @2026-06-11T21:15Z
|
||||
|
||||
**Claim**: DEFERRED closed; levels reconciled; PARITY.md updated; operator summary written; fresh Adversary re-trigger via independent `!testme` on PR#3.
|
||||
|
||||
**Verdict: PASS** — all M2 DoD items verified independently. Phase `mailu` is DONE.
|
||||
|
||||
### What I verified (cold)
|
||||
|
||||
1. **PR#3 still open, unmerged** (Gitea API cold check):
|
||||
- state: open, head sha: `edc0201a79d36bc87696b0f93f1ee88ad7bd10ed`, merged: False ✓
|
||||
|
||||
2. **DEFERRED.md mailu entry closed**:
|
||||
- Entry `2026-05-29 — mailu: no backup config` marked `[x] CLOSED @2026-06-11` with PR#3 +
|
||||
build #477 pointers; re-entry checkbox also ticked ✓
|
||||
|
||||
3. **PARITY.md updated with dual-volume evidence** (`tests/mailu/PARITY.md`):
|
||||
- P4 section now states "earned via recipe-mirror PR#3" ✓
|
||||
- Documents both `/data` (SQLite) and `/mail` (Maildir) seeded + wiped + verified restored ✓
|
||||
- `ops.py`, `test_backup.py`, `test_restore.py` each described correctly ✓
|
||||
- Before/after level: `backup_capable=False → L4-skip` → `backup_capable=True → L5-earned` ✓
|
||||
|
||||
4. **Levels reconciliation independently verified**:
|
||||
- `runner/harness/generic.py::backup_capable()` scans `compose*.yml` for `backupbot.backup.*true` ✓
|
||||
- Main branch: no backupbot labels → `backup_capable=False` → backup rung = intentional skip → **L4** ✓
|
||||
- PR#3 head: admin+imap labels present → `backup_capable=True` → backup rung earned → **L5** ✓
|
||||
|
||||
5. **Operator summary in STATUS-mailu.md**: complete, accurate, actionable — specifies PR#3 URL,
|
||||
head SHA, what the PR adds, what CI proved, what operator must do (merge PR#3) ✓
|
||||
|
||||
6. **Fresh independent re-trigger** (Adversary posted `!testme` on PR#3 at 2026-06-11T21:04:39Z,
|
||||
comment #14363):
|
||||
- **Drone build #483**: LEVEL 5 SUCCESS, recipe=mailu, PR=3, ref=`edc0201a79d3`
|
||||
- All 5 rungs PASS: install / upgrade / backup+restore / functional / lint ✓
|
||||
- Backup stage: `test_backup_captures_mailbox` PASS (1377ms) + `test_backup_captures_mail_message` PASS (149ms) ✓
|
||||
- Restore stage: `test_restore_returns_mailbox` PASS (1402ms) + `test_restore_returns_mail_message` PASS (168ms) ✓
|
||||
- `clean_teardown: true`, `no_secret_leak: true` ✓
|
||||
- No mailu stacks or volumes on host post-run (`docker stack ls` + `docker volume ls` confirm) ✓
|
||||
- Result is reproducible: two independent builds (#477, #483) both LEVEL 5 at the same PR head ✓
|
||||
|
||||
### Phase DoD satisfied
|
||||
|
||||
All items from `plan-phase-mailu-backup.md` §5:
|
||||
- Mirror PR open with evidence-justified backupbot v2 labels ✓ (PR#3)
|
||||
- backup→wipe→restore proven on real seeded mail data at PR head incl. drone path ✓ (builds #477 + #483)
|
||||
- mailu's backup rung earned (not skipped) with levels reconciled ✓
|
||||
- DEFERRED closed ✓
|
||||
- M1 + M2 fresh Adversary PASSes ✓ (this entry + M1 PASS above)
|
||||
- PR unmerged for the operator ✓
|
||||
|
||||
**Phase `mailu` is complete. Builder is cleared to write `## DONE` to STATUS-mailu.md.**
|
||||
157
machine-docs/STATUS-bsky.md
Normal file
157
machine-docs/STATUS-bsky.md
Normal file
@ -0,0 +1,157 @@
|
||||
# STATUS — phase bsky (fix bluesky-pds recipe + screenshot)
|
||||
|
||||
Phase SSOT: /srv/cc-ci/cc-ci-plan/plan-phase-bsky-fix.md
|
||||
|
||||
## DONE
|
||||
|
||||
Phase bsky complete @2026-06-11T15:55Z: M1 PASS (REVIEW-bsky 369f4f4 @12:30Z) + M2 PASS
|
||||
(42eabba @15:48Z, incl. the Adversary's own independent !testme re-trigger → build 435
|
||||
level 5 at PR head), no VETO. bluesky-pds root cause proven, fix PR #2 OPEN+UNMERGED for
|
||||
the operator (re-pin 0.4.219), green through the full lifecycle incl. lint on real drone
|
||||
CI, screenshot real and verified, DEFERRED entries closed, operator runbook below.
|
||||
|
||||
|
||||
## M2 claim — operator handoff complete (2026-06-11T15:50Z)
|
||||
|
||||
WHAT (phase plan §3 M2, all builder-side items in place; the fresh cold pass is yours):
|
||||
1. **Green at PR head, re-triggerable:** PR #2 head f7b6c8df unchanged since run 427
|
||||
(level 5). HOW to re-run independently: post `!testme` on PR #2 — the bridge polls
|
||||
~1 min, triggers a drone build, run dir /var/lib/cc-ci-runs/<n>. EXPECTED: level=5,
|
||||
rungs install/backup_restore/functional/lint=pass, upgrade=skip with
|
||||
skips.intentional.upgrade = the declared reason, clean_teardown+no_secret_leak=true,
|
||||
screenshot.png = the PDS landing page. (cc-ci main also unchanged functionally since
|
||||
e9745c8; HEAD at claim time: see this commit.)
|
||||
2. **PNG to independently Read:** https://ci.commoninternet.net/runs/427/screenshot.png
|
||||
(+ the fresh run's, if you re-trigger). EXPECTED: ASCII Bluesky butterfly landing
|
||||
page, no credentials.
|
||||
3. **Level under new semantics + baseline reconciled:** achieved level 5 (de-capped:
|
||||
skip climbs), upgrade = declared intentional skip with re-enable path. Old baseline
|
||||
"full lifecycle green" (Phase-2 e45e0ee, pre-results-era) reconciled: unreproducible
|
||||
for upstream reasons (moving-tag republish broke ALL published versions); the PR
|
||||
restores deployability; recorded in DEFERRED closure + JOURNAL-bsky 12:15Z entry.
|
||||
4. **DEFERRED entries closed with pointers:** machine-docs/DEFERRED.md bluesky entry
|
||||
marked RESOLVED @2026-06-11 (commit f150012) — explicitly closes BOTH the re-pin
|
||||
follow-up and the rcust M2 baseline-exclusion note, with PR/run/registry pointers.
|
||||
5. **Operator summary:** below in this file (what was wrong / what the PR changes /
|
||||
post-merge steps 1-5 incl. version publish, EXPECTED_NA→UPGRADE_BASE_VERSION swap,
|
||||
no canonical to reseed, never re-pin :0.4).
|
||||
6. **PR left OPEN** for the operator (merged=false; immich PR#2/plausible PR#3 precedent).
|
||||
|
||||
WHERE: cc-ci main (STATUS/JOURNAL/BACKLOG-bsky, DEFERRED f150012, DECISIONS 2026-06-11
|
||||
×2, harness e9745c8); mirror PR #2 head f7b6c8df; runs 427 (green) / 423 (negative
|
||||
control); upstream registry cc-ci-plan/upstream/bluesky-pds.md @ f395247.
|
||||
|
||||
## M1 claim — root cause + green fix PR + screenshot (2026-06-11T12:05Z)
|
||||
|
||||
### WHAT
|
||||
|
||||
1. Root cause proven with evidence (below).
|
||||
2. Fix PR open on the recipe mirror: **recipe-maintainers/bluesky-pds PR #2**, branch
|
||||
`upgrade-0.3.0+v0.4.219`, head `f7b6c8df` — 2-line compose.yml diff (image
|
||||
`ghcr.io/bluesky-social/pds:0.4` → `0.4.219`; version label `0.2.0+v0.4` →
|
||||
`0.3.0+v0.4.219`). UNMERGED (operator merges).
|
||||
3. `!testme` on the PR green through the full lifecycle via the real drone path:
|
||||
**run 427 = level 5** — install/backup_restore/functional/lint all PASS, upgrade =
|
||||
DECLARED intentional skip (justification below), clean_teardown, no_secret_leak.
|
||||
4. Screenshot captured on that PR run and visually verified by me: the genuine PDS
|
||||
HTTP landing page (ASCII Bluesky logo, "This is an AT Protocol Personal Data
|
||||
Server", /xrpc/ pointer, upstream links) — real, representative, credential-free.
|
||||
No SCREENSHOT hook needed.
|
||||
|
||||
### Root cause
|
||||
|
||||
The recipe pins MOVING tag `ghcr.io/bluesky-social/pds:0.4` and overrides the entrypoint
|
||||
with a script ending `exec node --enable-source-maps index.js` (relative to WORKDIR /app).
|
||||
Upstream now publishes main-branch builds to `:0.4` (== `latest`, manifest
|
||||
`sha256:871194d2…`, created 2026-05-30): `@atproto/pds` **0.5.1**, Node v24.15.0, service
|
||||
restructured to `/app/index.ts` (CMD `node --enable-source-maps index.ts`; **no
|
||||
index.js**) → crash-loop `Cannot find module '/app/index.js'`. Exact tag `0.4.219`
|
||||
(newest released; ghcr digest `sha256:e0b756701c92…`) keeps the expected layout: Node
|
||||
v20.20.2, `/app/index.js`, dumb-init, CMD identical to the recipe's exec line.
|
||||
|
||||
HOW to verify root cause (any host with ssh cc-ci):
|
||||
- `ssh cc-ci 'docker run --rm --entrypoint sh ghcr.io/bluesky-social/pds:0.4 -c "node --version; ls /app; grep @atproto/pds /app/package.json"'`
|
||||
→ EXPECTED v24.15.0; index.ts, NO index.js; `"@atproto/pds": "0.5.1"`
|
||||
- `ssh cc-ci 'docker run --rm --entrypoint sh ghcr.io/bluesky-social/pds:0.4.219 -c "node --version; ls /app; grep @atproto/pds /app/package.json"'`
|
||||
→ EXPECTED v20.20.2; index.js present; `"@atproto/pds": "0.4.219"`
|
||||
- Upstream: Dockerfile@main = node:24.15-alpine3.23 + CMD index.ts;
|
||||
Dockerfile@v0.4.219 = node:20.20-alpine3.23 + CMD index.js. Registry doc:
|
||||
cc-ci-plan/upstream/bluesky-pds.md (plan repo f395247).
|
||||
|
||||
### Upgrade-rung justification (the "justify status either way" item)
|
||||
|
||||
Published versions exist (0.1.1+v0.4, 0.2.0+v0.4) but BOTH pin the republished `:0.4` →
|
||||
no published version can deploy as the upgrade base anymore (negative control: run 423,
|
||||
pre-harness-change, deployed base 0.1.1+v0.4 → identical MODULE_NOT_FOUND crash-loop,
|
||||
install=fail, PR head never reached; run-423 recipe checkout sat at tag 0.1.1+v0.4).
|
||||
Harness change e9745c8 (main): declaring the upgrade rung in recipe_meta EXPECTED_NA now
|
||||
also suppresses the base deploy — single deploy = the PR head; the upgrade tier records
|
||||
"skip"; derive_rungs classifies it the DECLARED intentional skip; reason fully visible in
|
||||
results.json `skips.intentional` and on the card. NOT a weakening: the rung is never
|
||||
reported pass; decision + re-enable path in machine-docs/DECISIONS.md (re-enable =
|
||||
UPGRADE_BASE_VERSION="0.3.0+v0.4.219" once merged+published).
|
||||
HOW: `cc-ci-run -m pytest tests/unit/ -q` from a cold clone of main on cc-ci →
|
||||
EXPECTED 253 passed (6 new in tests/unit/test_upgrade_base.py);
|
||||
`nix develop .#lint -c bash scripts/lint.sh` → EXPECTED `lint: PASS`.
|
||||
|
||||
### Green-run evidence (run 427, drone path)
|
||||
|
||||
- Trigger: PR #2 comment 14342 (`!testme`) → bridge log line
|
||||
`[poll] triggered build 427 for bluesky-pds@f7b6c8df (PR #2, comment 14342)`;
|
||||
outcome line `reflected outcome build 427 (bluesky-pds PR #2): success`; PR result
|
||||
comment 14343 "✅ passed @ f7b6c8df".
|
||||
- HOW: `ssh cc-ci 'cat /var/lib/cc-ci-runs/427/results.json'` → EXPECTED level=5,
|
||||
ref=f7b6c8dfb81c, rungs install/backup_restore/functional/lint=pass + upgrade=skip,
|
||||
skips.intentional.upgrade=<declared reason>, flags clean_teardown+no_secret_leak true.
|
||||
- PR-head proof: run-427 per-run recipe checkout
|
||||
(`/var/lib/cc-ci-runs/427/abra/recipes/bluesky-pds`) at `f7b6c8d chore: upgrade to
|
||||
0.3.0+v0.4.219`, compose.yml line 6 image=…:0.4.219.
|
||||
- Visuals: https://ci.commoninternet.net/runs/427/summary.png (card: level 5 of 5, all
|
||||
tiers PASS, upgrade INTENTIONAL SKIP + reason, screenshot thumb, clean-teardown +
|
||||
no-secret-leak chips), …/badge.svg ("cc-ci: level 5", green),
|
||||
…/screenshot.png (the PDS landing page described above).
|
||||
|
||||
### WHERE
|
||||
|
||||
- cc-ci main @ 72b3d6c (harness change e9745c8; journal/decisions 72b3d6c).
|
||||
- Mirror PR #2: https://git.autonomic.zone/recipe-maintainers/bluesky-pds/pulls/2
|
||||
(head f7b6c8df; base main b2d86ef).
|
||||
- Runs: /var/lib/cc-ci-runs/427 (green, PR head), /var/lib/cc-ci-runs/423 (negative
|
||||
control, pre-change base trap).
|
||||
- Upstream registry: cc-ci-plan/upstream/bluesky-pds.md @ plan-repo f395247.
|
||||
|
||||
## Operator summary
|
||||
|
||||
**What was wrong.** bluesky-pds could not deploy at all: the app crash-looped
|
||||
`Cannot find module '/app/index.js'`. The recipe pins the MOVING image tag
|
||||
`ghcr.io/bluesky-social/pds:0.4`, and upstream now republishes that tag with main-branch
|
||||
builds (currently @atproto/pds 0.5.1 on Node 24, where the service entrypoint moved to
|
||||
`/app/index.ts` — `index.js` no longer exists). The recipe's entrypoint override
|
||||
(`exec node --enable-source-maps index.js`) can no longer resolve. This also silently
|
||||
broke BOTH previously published recipe versions (0.1.1+v0.4, 0.2.0+v0.4 — same moving
|
||||
pin), so no historical version can deploy anymore either.
|
||||
|
||||
**What the PR changes.** https://git.autonomic.zone/recipe-maintainers/bluesky-pds/pulls/2
|
||||
(branch `upgrade-0.3.0+v0.4.219`, head f7b6c8df), a 2-line compose.yml diff: pin the exact
|
||||
released tag `0.4.219` (newest released; classic Node 20 / index.js layout the recipe's
|
||||
entrypoint expects) and bump the version label to `0.3.0+v0.4.219`. Why not 0.5.1: it has
|
||||
no release tag (only the moving :0.4/latest + sha- tags from main) and needs an entrypoint
|
||||
migration; do that as a proper upgrade when upstream cuts a 0.5.x release tag (notes in
|
||||
cc-ci-plan/upstream/bluesky-pds.md). Proven at PR head via real drone CI: run 427 =
|
||||
**level 5** (install, backup/restore, functional, lint PASS; screenshot = real PDS landing
|
||||
page). The upgrade rung is a DECLARED intentional skip — there is no deployable published
|
||||
base to upgrade FROM (see above); declaration + reason in tests/bluesky-pds/recipe_meta.py.
|
||||
|
||||
**What to do post-merge.**
|
||||
1. Merge PR #2 (your call, as with immich PR#2 / plausible PR#3 — all left open).
|
||||
2. Publish the version per recipe convention (annotated tag `0.3.0+v0.4.219` /
|
||||
`abra recipe release`) so `abra recipe versions` lists a deployable version again.
|
||||
3. After the tag is published: in cc-ci `tests/bluesky-pds/recipe_meta.py`, DROP the
|
||||
`EXPECTED_NA["upgrade"]` declaration and set
|
||||
`UPGRADE_BASE_VERSION = "0.3.0+v0.4.219"` — the upgrade rung then re-activates from
|
||||
the first deployable base (the older broken tags must never be auto-picked as base).
|
||||
4. Canonical/warm: nothing to reseed — bluesky-pds has no canonical
|
||||
(/var/lib/ci-warm has no entry); the normal promote-on-green flow mints one on the
|
||||
first green run post-merge.
|
||||
5. Never re-pin this recipe to `:0.4`/`latest` — upstream demonstrably republishes the
|
||||
minor tag (registry notes: cc-ci-plan/upstream/bluesky-pds.md).
|
||||
88
machine-docs/STATUS-cfold.md
Normal file
88
machine-docs/STATUS-cfold.md
Normal file
@ -0,0 +1,88 @@
|
||||
# STATUS — phase cfold (custom-folder collapse)
|
||||
|
||||
**Phase:** cfold — collapse `functional/`+`playwright/` into `custom/`
|
||||
**Builder:** autonomic-bot
|
||||
**Updated:** 2026-06-12
|
||||
|
||||
---
|
||||
|
||||
## M1 — PASS
|
||||
|
||||
Gate result: `REVIEW-cfold.md` 2026-06-12T16:20Z -> **M1 PASS**
|
||||
|
||||
Inputs for verification:
|
||||
- Implementation commit: `44e0242` (`feat(cfold): canonicalize custom test layout`)
|
||||
|
||||
Completed in this checkpoint:
|
||||
- discovery.py: `custom/` canonical + deprecated aliases with warnings
|
||||
- `git mv` all 64 custom tests (60 functional + 4 playwright) across 20 recipes
|
||||
- helper modules moved alongside their tests into `custom/`
|
||||
- sys.path refs updated in mailu lifecycle overlays
|
||||
- docs updated (`README.md`, `recipe-customization.md`, `testing.md`, `enroll-recipe.md`)
|
||||
- unit tests updated (`test_discovery.py`, `test_discovery_phase2.py`, `test_manifest.py`)
|
||||
- manifest.py now reports canonical `custom` counts
|
||||
|
||||
WHAT:
|
||||
- M1 implementation is complete: custom-test discovery is canonicalized to `custom/`, deprecated
|
||||
aliases warn loudly instead of silently dropping coverage, all cc-ci custom tests/helpers moved to
|
||||
`tests/<recipe>/custom/`, manifest counts are canonicalized, and the placement-rule docs/unit tests
|
||||
were updated.
|
||||
|
||||
HOW:
|
||||
- `git ls-files "tests/*/custom/test_*.py" | wc -l`
|
||||
- `git ls-files "tests/*/functional/*" "tests/*/playwright/*"`
|
||||
- `for recipe in bluesky-pds cryptpad custom-html custom-html-tiny discourse drone ghost hedgedoc immich keycloak lasuite-docs lasuite-drive lasuite-meet mailu matrix-synapse mattermost-lts mumble n8n plausible uptime-kuma; do count=$(git ls-files "tests/$recipe/custom/test_*.py" | wc -l); printf "%s %s\n" "$recipe" "$count"; done`
|
||||
- `nix shell nixpkgs#python311Packages.pytest -c pytest tests/unit/test_discovery.py tests/unit/test_discovery_phase2.py tests/unit/test_manifest.py -q`
|
||||
|
||||
EXPECTED:
|
||||
- Total canonical custom tests: `64`
|
||||
- Old tracked trees: no output for `functional/*` or `playwright/*`
|
||||
- Per-recipe counts exactly match the baseline table below
|
||||
- Focused unit suite: `18 passed`
|
||||
|
||||
WHERE:
|
||||
- Discovery + alias warnings: `runner/harness/discovery.py`
|
||||
- Canonical manifest counts: `runner/harness/manifest.py`
|
||||
- Migrated custom tests/helpers: `tests/*/custom/`
|
||||
- Focused unit coverage: `tests/unit/test_discovery.py`, `tests/unit/test_discovery_phase2.py`, `tests/unit/test_manifest.py`
|
||||
- Placement-rule docs: `docs/recipe-customization.md`, `docs/testing.md`, `docs/enroll-recipe.md`, `README.md`
|
||||
|
||||
Adversary verdict:
|
||||
- `machine-docs/REVIEW-cfold.md` lines 52-77
|
||||
- PASS facts include: 64 canonical custom tests, zero old tracked custom trees, focused unit suite `18 passed`, deprecated-alias warning probe green, normalized `(recipe, filename)` coverage set preserved exactly (`missing []`, `extra []`).
|
||||
|
||||
---
|
||||
|
||||
## M2 — IN PROGRESS
|
||||
|
||||
Current work item:
|
||||
- build the pre-sweep baseline matrix (recipe -> expected level + custom-test set)
|
||||
- then run the full real-CI `!testme` sweep and capture recipe-by-recipe evidence
|
||||
|
||||
---
|
||||
|
||||
## Baseline (pre-cfold) — custom test count per recipe
|
||||
|
||||
| Recipe | Count |
|
||||
|--------|-------|
|
||||
| bluesky-pds | 4 |
|
||||
| cryptpad | 4 |
|
||||
| custom-html | 4 |
|
||||
| custom-html-tiny | 1 |
|
||||
| discourse | 3 |
|
||||
| drone | 1 |
|
||||
| ghost | 4 |
|
||||
| hedgedoc | 2 |
|
||||
| immich | 3 |
|
||||
| keycloak | 3 |
|
||||
| lasuite-docs | 5 |
|
||||
| lasuite-drive | 3 |
|
||||
| lasuite-meet | 3 |
|
||||
| mailu | 3 |
|
||||
| matrix-synapse | 3 |
|
||||
| mattermost-lts | 3 |
|
||||
| mumble | 5 |
|
||||
| n8n | 4 |
|
||||
| plausible | 2 |
|
||||
| uptime-kuma | 4 |
|
||||
| **TOTAL** | **64** |
|
||||
157
machine-docs/STATUS-drone.md
Normal file
157
machine-docs/STATUS-drone.md
Normal file
@ -0,0 +1,157 @@
|
||||
# STATUS — phase drone (drone enrollment with gitea SCM dep)
|
||||
|
||||
**Phase plan:** `/srv/cc-ci/cc-ci-plan/plan-phase-drone-enroll.md`
|
||||
**Builder:** autonomic-bot / Claude (Builder loop)
|
||||
**Started:** 2026-06-11T21:30Z
|
||||
|
||||
---
|
||||
|
||||
## DONE
|
||||
|
||||
**Adversary M2 PASS @2026-06-11T22:30Z** (commit `7b4081c`)
|
||||
|
||||
All phase DoD satisfied. Phase drone complete. PR open for operator merge.
|
||||
|
||||
**Operator summary:**
|
||||
- Drone 1.9.0 enrolled with gitea 3.5.3 as SCM dep; full lifecycle proven via real `!testme` CI
|
||||
- Gitea dep provisioned per-run (admin user + OAuth2 app); wired to drone at install time via `install_steps.sh`
|
||||
- SCM-configured functional test (`test_login_redirects_to_gitea_dep`) verifies per-run dep, not production gitea
|
||||
- Upgrade tier: 1.8.0+2.25.0 → 1.9.0+2.26.0 reconverges cleanly
|
||||
- Backup structural skip: drone is not backup-capable (no backupbot labels); documented in PARITY.md
|
||||
- Build-creation API gap accepted as proportionate deferral (Adversary §7.1 sign-off); remaining DEFERRED item
|
||||
|
||||
**Build #506 evidence (M2 CI run):**
|
||||
|
||||
```
|
||||
recipe=drone ref=049438e1cb47 pr=1 event=custom (!testme via bridge)
|
||||
deploy-count = 2 (expect 2) # DG4.1 PASS
|
||||
deps deployed: ['gitea']
|
||||
install : pass # test_serving PASSED
|
||||
upgrade : pass # test_upgrade_reconverges PASSED (1.8.0+2.25.0 → 1.9.0+2.26.0)
|
||||
backup : skip # intentional: not backup-capable
|
||||
restore : skip # intentional: not backup-capable
|
||||
custom : pass # test_login_redirects_to_gitea_dep PASSED
|
||||
lint : pass
|
||||
level=5, clean_teardown=true, no_secret_leak=true
|
||||
```
|
||||
|
||||
Screenshot: `machine-docs/screenshots/drone-m2-build506.png`
|
||||
|
||||
---
|
||||
|
||||
## M2 CLAIMED (superseded by DONE above)
|
||||
|
||||
**Evidence:** CI build #506, 2026-06-11T22:21Z — event: custom (!testme on PR #1, recipe-maintainers/drone)
|
||||
|
||||
```
|
||||
recipe=drone ref=049438e1cb47 pr=1
|
||||
deploy-count = 2 (expect 2) # DG4.1 PASS
|
||||
deps deployed: ['gitea']
|
||||
install : pass # test_serving PASSED
|
||||
upgrade : pass # test_upgrade_reconverges PASSED (1.8.0+2.25.0 → 1.9.0+2.26.0)
|
||||
backup : skip # intentional: not backup-capable
|
||||
restore : skip # intentional: not backup-capable
|
||||
custom : pass # test_login_redirects_to_gitea_dep PASSED
|
||||
lint : pass
|
||||
level=5, clean_teardown=true, no_secret_leak=true
|
||||
```
|
||||
|
||||
Gitea dep provisioned at `gite-4c9694.ci.commoninternet.net`:
|
||||
- Admin user `ci_admin` created
|
||||
- OAuth2 app created (client_id=`d144083e-5ba5-4d1e-aed2-5e8f8331923a`)
|
||||
- SCM wired via `install_steps.sh`; test confirmed redirect to dep (not production gitea)
|
||||
- Dep torn down cleanly post-run
|
||||
|
||||
Screenshot: `machine-docs/screenshots/drone-m2-build506.png`
|
||||
Build URL: `https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/506`
|
||||
Results: `/var/lib/cc-ci-runs/506/results.json` (level=5)
|
||||
|
||||
Mirror PRs:
|
||||
- `git.autonomic.zone/recipe-maintainers/drone/pulls/1` — `testme-1.9.0-cc-ci` branch
|
||||
- `git.autonomic.zone/recipe-maintainers/gitea/pulls/1` — dependency mirror in place
|
||||
|
||||
---
|
||||
|
||||
## M1 CLAIMED
|
||||
|
||||
**Evidence:** Harness run 5, 2026-06-11T22:18Z on cc-ci host (`/root/drone-test-clone` @ `0aa46db`)
|
||||
|
||||
```
|
||||
== cc-ci run: recipe=drone ref=None pr=0 stages=['custom', 'install', 'upgrade']
|
||||
deploy-count = 2 (expect 2) # DG4.1 PASS
|
||||
deps deployed: ['gitea']
|
||||
install : pass
|
||||
upgrade : pass
|
||||
custom : pass
|
||||
results.json written: ... (level=5 of 5)
|
||||
```
|
||||
|
||||
Log: `/tmp/drone-m1-run5.log` on cc-ci
|
||||
Results: `/var/lib/cc-ci-runs/manual/results.json`
|
||||
|
||||
**All fixes applied:**
|
||||
- ADV-drone-01 (`7e7e84d`): `_CaptureOneRedirect` no-follow; Adversary verified CLOSED
|
||||
- DG4.1 count (`5384f5c`): reverted `_count_deploy=False`; dep deploys count per formula
|
||||
- ADV-drone-02 (`0aa46db`): finally-block fallback teardown from `$CCCI_DEPS_FILE`; 19/19 unit tests PASS
|
||||
|
||||
---
|
||||
|
||||
## Current state
|
||||
|
||||
**P0 prerequisite:** VERIFIED — `/etc/timezone` exists (content `UTC`) on cc-ci host.
|
||||
|
||||
**Gate M1:** PASS — Adversary PASS @2026-06-11T22:22Z (commit `3de5925`)
|
||||
**Gate M2:** PASS — Adversary PASS @2026-06-11T22:30Z (commit `7b4081c`) — **DONE**
|
||||
|
||||
---
|
||||
|
||||
## DoD tracker (M1)
|
||||
|
||||
- [x] P0 verified on host — `/etc/timezone` = `UTC`
|
||||
- [x] `tests/gitea/recipe_meta.py` — gitea enrolled as dep provider (health + sqlite3 EXTRA_ENV)
|
||||
- [x] `runner/harness/sso.py` — `setup_gitea_oauth()` function (admin user + OAuth2 app)
|
||||
- [x] `runner/run_recipe_ci.py` — `_enrich_deps_with_sso` extended for gitea
|
||||
- [x] `tests/drone/recipe_meta.py` — drone with `DEPS=["gitea"]`, health/timeouts
|
||||
- [x] `tests/drone/install_steps.sh` — wires gitea OAuth into drone deploy
|
||||
- [x] `tests/drone/functional/test_scm_configured.py` — no-follow redirect; ADV-drone-01 fixed `7e7e84d`
|
||||
- [x] `tests/drone/PARITY.md` — backup structural-skip justification documented
|
||||
- [x] Unit tests — 19/19 PASS cold (test_gitea_dep.py + test_deps.py)
|
||||
- [x] No gate weakening; declared skips justified (backup structural skip per PARITY.md)
|
||||
- [x] Harness run 5 GREEN — deploy-count 2/2, level=5, install+upgrade+custom+lint PASS
|
||||
- [x] ADV-drone-02 fixed + unit tested (`0aa46db`)
|
||||
|
||||
---
|
||||
|
||||
## Verification recipe (for Adversary M1 check)
|
||||
|
||||
```bash
|
||||
# On the orchestrator host (this machine) or from any machine with SSH to cc-ci:
|
||||
ssh cc-ci "cat /var/lib/cc-ci-runs/manual/results.json" | python3 -c "
|
||||
import json, sys
|
||||
r = json.load(sys.stdin)
|
||||
assert r['level'] == 5, f'level={r[\"level\"]} != 5'
|
||||
assert r['results']['install'] == 'pass'
|
||||
assert r['results']['upgrade'] == 'pass'
|
||||
assert r['results']['custom'] == 'pass'
|
||||
assert r['rungs']['lint'] == 'pass'
|
||||
assert r['rungs']['backup_restore'] == 'skip'
|
||||
assert r['skips']['intentional']['backup_restore']
|
||||
print('M1 evidence VERIFIED')
|
||||
"
|
||||
|
||||
# Unit tests (19/19):
|
||||
cd /srv/cc-ci-orch/cc-ci && \
|
||||
/nix/store/rag15ca0cyi4nqbw6x6w1fqkvq5wmibj-python3-3.12.8-env/bin/pytest \
|
||||
tests/unit/test_deps.py tests/unit/test_gitea_dep.py -v
|
||||
|
||||
# Negative-control structural argument (no live deploy needed):
|
||||
# A drone WITHOUT install_steps.sh (empty deps file) would not have GITEA_DOMAIN set,
|
||||
# so /login would not redirect to a gitea domain. The SCM test checks parsed.netloc == gitea_domain;
|
||||
# wrong netloc → AssertionError. The test is falsified by misconfiguration.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Blocked items
|
||||
|
||||
(none)
|
||||
219
machine-docs/STATUS-dstamp.md
Normal file
219
machine-docs/STATUS-dstamp.md
Normal file
@ -0,0 +1,219 @@
|
||||
# STATUS — phase `dstamp` (discourse abra-stamp drift)
|
||||
|
||||
Builder. SSOT: `cc-ci-plan/plan-phase-dstamp-discourse-drift.md`. Gates M1, M2.
|
||||
|
||||
## DONE
|
||||
|
||||
M1 PASS (REVIEW-dstamp `fb411b2` @17:36Z) + M2 PASS (`71358da` @17:58Z), both fresh, no VETO.
|
||||
All Definition-of-Done items Adversary-verified.
|
||||
|
||||
**Operator summary.** The discourse upgrade-tier "abra stamp drift" (upgrade-HC1 stamping the
|
||||
prev-base tag commit `eb96de94+U` instead of the PR head `7ae7b0f7+U`, since ~06-10) was **NOT an
|
||||
abra or harness git bug** — abra stamps the head correctly. **Root cause:** discourse's
|
||||
`compose.yml` app service uses `deploy.update_config: { failure_action: rollback, order:
|
||||
start-first, monitor: 5s }`. On the upgrade chaos redeploy, start-first co-resides the OLD+NEW
|
||||
precompile/Rails-heavy task (~2× memory); under host memory pressure the NEW task fails swarm's 5s
|
||||
update monitor → swarm **rolls back** to the base spec, reverting the `chaos-version` label
|
||||
(head→base). start-first kept the old task serving, so `wait_healthy` passed and HC1 read the
|
||||
reverted base commit — misreported as "re-checkout failed". Intermittent (memory-pressure
|
||||
dependent): solo run 184 on 06-05 passed; the heavier 06-10/06-11 runs rolled back every time.
|
||||
**Direct evidence:** `dstamp-repro4` captured `.Spec chaos-version=7ae7b0f7+U` (head applied) →
|
||||
`.PreviousSpec=eb96de94+U` (base) with `UpdateStatus=updating`, then the post-rollback read = base.
|
||||
|
||||
**Fix (commits `0cc31a5` + `e9c26c7`, HC1 unweakened):** (1) `tests/discourse/compose.ccci.yml`
|
||||
app `update_config.order: stop-first` — the new task boots with full host memory, no OOM, no
|
||||
spurious rollback (`failure_action: rollback` left intact for genuine failures); (2) a general
|
||||
harness guard `lifecycle.assert_upgrade_converged` (2-phase StartedAt protocol) that detects a
|
||||
swarm rollback/pause after the upgrade redeploy and fails the upgrade HONESTLY — the HC1
|
||||
commit-match assertion is unchanged.
|
||||
|
||||
**Proven in real CI:** drone `!testme` build **#450** (discourse @7ae7b0f) = **LEVEL 5** (was L1
|
||||
under the drift), all tiers green, clean teardown, no secret leak; PR recipe-maintainers/discourse#2
|
||||
shows ✅ passed. **Blast-radius:** only discourse was affected (keycloak/n8n share the policy but
|
||||
upgrade-PASS L4; drone/traefik are infra) — the new harness guard now protects all rollback-policy
|
||||
recipes. DEFERRED entry closed with pointers. **No operator action required.**
|
||||
|
||||
---
|
||||
|
||||
## Gate: M1 — PASS (REVIEW-dstamp fb411b2 @2026-06-11T17:36Z). Now on M2.
|
||||
|
||||
## Gate: M2 — CLAIMED, awaiting Adversary
|
||||
|
||||
**WHAT (M2 = Proven in real CI):** discourse full lifecycle GREEN at its true level via the drone
|
||||
`!testme` path, upgrade-HC1 stamping the CORRECT head value; no other affected recipe; HC1
|
||||
unweakened (a wrong stamp still FAILs); DEFERRED closed.
|
||||
|
||||
- **Real-CI proof — drone `!testme` build #450:** discourse @ `7ae7b0f76efb` (PR#2), STAGES full
|
||||
(install,upgrade,backup,restore,custom), drone workspace at cc-ci main `2da1f01` (fix present) →
|
||||
**LEVEL 5** (max), ALL tiers PASS, `clean_teardown=true`, `no_secret_leak=true`. Upgrade tier
|
||||
`test_upgrade_reconverges` PASSED (HC1's `assert_upgraded` only passes when the deployed
|
||||
chaos-version commit == head_ref `7ae7b0f`, after `assert_upgrade_converged` confirmed
|
||||
`UpdateStatus=completed`). Was L1 (drift) before the fix → L5 now.
|
||||
- **Triggered via the !testme path:** comment `14346` (`!testme`) on recipe-maintainers/discourse#2
|
||||
→ bridge ack `14347`, updated to "🌻 cc-ci — discourse @ 7ae7b0f7 ✅ **passed**" with the L5
|
||||
result card/badge linking drone build 450.
|
||||
|
||||
**HOW to verify (Adversary, cold):**
|
||||
1. `grep -oE '"level": [0-9]+|"(install|upgrade|backup|restore|custom)": "[a-z]+"|"clean_teardown":
|
||||
(true|false)|"no_secret_leak": (true|false)' /var/lib/cc-ci-runs/450/results.json` → level 5,
|
||||
all `pass`, both flags `true`.
|
||||
2. `/var/lib/cc-ci-runs/450/junit/upgrade__generic__test_upgrade.xml` → `test_upgrade_reconverges`
|
||||
testcase with NO `<failure>` child (passed).
|
||||
3. PR comment 14347 on recipe-maintainers/discourse#2 = ✅ passed, run 450.
|
||||
4. *Fresh independent re-trigger (recommended):* post `!testme` on discourse#2 → new drone build on
|
||||
cc-ci main → expect L5 again (reliability: manual fix1+fix2 + build 450 = 3 consecutive green
|
||||
with the fix vs intermittent unpatched failures).
|
||||
5. **HC1 teeth (negative test — Adversary leads):** synthesize a wrong stamp and show RED. Two live
|
||||
teeth: (a) the unchanged commit-match `generic.py:174-175` — a deployed chaos commit ≠ head_ref
|
||||
still FAILs (e.g. force the recheckout to the base, or deploy base-as-head); (b) the new
|
||||
`assert_upgrade_converged` raises on a swarm `rollback_completed`/`paused` (the ORIGINAL drift
|
||||
path — repro1/repro4 are exactly this RED, now with an honest message). Neither relaxes HC1.
|
||||
6. DEFERRED closed: `machine-docs/DEFERRED.md` dstamp entry → ✅ RESOLVED with pointers.
|
||||
|
||||
**EXPECTED:** build 450 level 5, all tiers pass, both flags true; PR#2 ✅ passed; DEFERRED resolved.
|
||||
**WHERE:** `/var/lib/cc-ci-runs/450/`; commits `0cc31a5`,`e9c26c7`; PR#2 comments 14346/14347;
|
||||
`machine-docs/DEFERRED.md`. **No other recipe affected** (blast-radius: keycloak/n8n upgrade-PASS L4
|
||||
across runs incl. rcust era; drone/traefik infra). Fresh Adversary M2 PASS → `## DONE`.
|
||||
|
||||
---
|
||||
|
||||
## (M1 — verified PASS; detail retained below)
|
||||
|
||||
**WHAT (M1 = Attribution):** root cause attributed by direct evidence; minimal reproducible
|
||||
demonstration; 06-05→06-10 change identified; fix implemented (recipe overlay + harness, HC1
|
||||
unweakened); blast-radius sweep complete.
|
||||
|
||||
Root cause: discourse `compose.yml` app service sets `deploy.update_config: { failure_action:
|
||||
rollback, order: start-first, monitor: 5s }`. On the upgrade chaos redeploy, start-first co-resides
|
||||
OLD+NEW (~2× memory) for the precompile/Rails-heavy app; under host memory pressure the NEW task
|
||||
fails swarm's 5s update monitor → `failure_action: rollback` reverts the app service to its
|
||||
PreviousSpec — INCLUDING the `coop-cloud.<stack>.chaos-version` label (head→base). Under start-first
|
||||
the OLD task keeps serving, so `wait_healthy` passes; `deployed_identity` then reads the rolled-back
|
||||
`.Spec` (base commit `eb96de94+U`) and HC1 misreports it as "re-checkout failed". abra+harness git
|
||||
path EXONERATED (abra stamps head `7ae7b0f7+U` correctly; per-run HEAD=7ae7b0f at deploy).
|
||||
|
||||
**HOW to verify (Adversary, cold):**
|
||||
1. *Recipe policy:* `cd ~/.abra/recipes/discourse && git checkout -q 7ae7b0f76efb && grep -nA3
|
||||
update_config compose.yml` → `failure_action: rollback`, `order: start-first`. EXPECTED present.
|
||||
2. *abra exonerated (minimal repro):* scratch ABRA_DIR, base→head checkout, `abra app deploy <d> -C
|
||||
-o -n --debug` bails at `secret not generated` AFTER logging `app/deploy.go:372 version: taking
|
||||
chaos version: 7ae7b0f7+U` (HEAD-correct). Procedure: JOURNAL-dstamp "mirror-faithful repro".
|
||||
3. *Direct rollback evidence:* console `/var/lib/cc-ci-runs/dstamp-repro4.console.log` line
|
||||
`[DSTAMP] post-redeploy svc inspect …` shows immediately post-redeploy `UpdateStatus.State=
|
||||
"updating"`, `.Spec…chaos-version=7ae7b0f7+U` (head applied), `.PreviousSpec…chaos-version=
|
||||
eb96de94+U` (base); the later HC1 read = eb96de94+U after the rollback completes.
|
||||
4. *Fix present:* `runner/harness/lifecycle.py::assert_upgrade_converged` (+ `update_status_started`)
|
||||
and its call in `runner/harness/generic.py::perform_upgrade`; `tests/discourse/compose.ccci.yml`
|
||||
app `deploy.update_config.order: stop-first`. Commits `0cc31a5` + `e9c26c7`.
|
||||
5. *Fix works:* run `dstamp-fix1` (fresh checkout, STAGES=install,upgrade) → upgrade PASS,
|
||||
console `upgrade-converged: …UpdateStatus=completed` + `chaos-version=7ae7b0f7+U version=
|
||||
0.7.0+3.3.1→0.9.0+3.5.0`. (Re-runnable: `RECIPE=discourse PR=2
|
||||
REF=7ae7b0f76efb2988c1e54956348dc9eeb7812e0b SRC=recipe-maintainers/discourse
|
||||
STAGES=install,upgrade CCCI_RUN_ID=<id> cc-ci-run runner/run_recipe_ci.py` from a checkout at
|
||||
`e9c26c7`.)
|
||||
6. *Blast-radius:* recipes with rollback+start-first = discourse, drone, keycloak, n8n, traefik.
|
||||
keycloak/n8n upgrade PASS L4 across runs (155/186/187/m2r; 47/54/61/162/197/m2r) ⇒ not affected;
|
||||
drone/traefik infra (no recipe-CI upgrade tier). Only discourse affected; the general
|
||||
`assert_upgrade_converged` guard now protects all rollback-policy recipes.
|
||||
|
||||
**EXPECTED:** all of 1–6 hold. **WHERE:** commits 0cc31a5, e9c26c7; runs
|
||||
`/var/lib/cc-ci-runs/dstamp-{repro1,repro2,repro4,fix1}`; recipe `~/.abra/recipes/discourse`.
|
||||
|
||||
HC1 teeth preserved: the commit-match assertion is unchanged; `assert_upgrade_converged` only makes
|
||||
a swarm rollback an HONEST upgrade failure before HC1 runs (a genuinely undeployable head still
|
||||
fails). M2 will demonstrate a wrong stamp still FAILs + full-lifecycle green via the `!testme` path.
|
||||
|
||||
---
|
||||
|
||||
## Root cause detail (evidence)
|
||||
|
||||
## ROOT CAUSE (attributed by direct evidence, abra+harness EXONERATED)
|
||||
|
||||
The upgrade chaos redeploy applies the **correct** head spec, then swarm **rolls it back** to the
|
||||
base spec, reverting the `chaos-version` label — masked by the recipe's `start-first` strategy +
|
||||
the harness's `wait_healthy` (the OLD task keeps serving, so health passes).
|
||||
|
||||
Recipe policy (`~/.abra/recipes/discourse/compose.yml`, app service): `deploy.update_config:
|
||||
{ failure_action: rollback, order: start-first }`, `healthcheck.start_period: 20m`. The heavy
|
||||
discourse app, started **start-first** (old+new co-resident ≈ 2× memory), intermittently fails
|
||||
swarm's update monitor on the NEW task → swarm executes `failure_action: rollback` → app service
|
||||
reverts to PreviousSpec (the base, `chaos-version=eb96de94+U`).
|
||||
|
||||
**Direct evidence (run `dstamp-repro4`, console `/var/lib/cc-ci-runs/dstamp-repro4.console.log`,
|
||||
solo/isolated):** immediately after `chaos_redeploy`, `docker service inspect <stack>_app`:
|
||||
- `UpdateStatus.State = "updating"`,
|
||||
- `.Spec.Labels coop-cloud.<stack>.chaos-version = 7ae7b0f7+U` (HEAD applied — abra stamped head
|
||||
correctly), `.version = 0.9.0+3.5.0`,
|
||||
- `.PreviousSpec.Labels …chaos-version = eb96de94+U` (the base), `.version = 0.7.0+3.3.1`.
|
||||
Then `wait_healthy` passes (old task serves under start-first); the new task fails the monitor →
|
||||
rollback → `.Spec` reverts to `eb96de94+U`; the later HC1 read sees `eb96de94+U` → FAIL with the
|
||||
misleading "re-checkout failed" message. (`dstamp-repro2`, lighter timing, had NO rollback →
|
||||
upgrade PASS @ `7ae7b0f7+U`.)
|
||||
|
||||
Intermittency (184✓ solo 06-05; m2b/m2p/ab✗ clustered/heavier-load 06-10/11; repro1✗ repro2✓
|
||||
repro4✗) = whether the new start-first task survives swarm's monitor under the host's momentary
|
||||
memory pressure. The "since ~06-10 on every run" = the rcust phase ran under heavier resident load
|
||||
(warm keycloak etc.) so the new task reliably failed → rollback every time. abra version-resolution
|
||||
is CORRECT (proven: repro2 debug line `taking chaos version: 7ae7b0f7+U` + 3 bail-at-secrets repros);
|
||||
the per-run git checkout is CORRECT (HEAD=7ae7b0f at deploy, reflog-proven). NOT abra, NOT the
|
||||
per-run tree, NOT concurrency.
|
||||
|
||||
## Fix (in progress) — HC1 keeps its teeth
|
||||
1. **Reliability (restore true level):** discourse `tests/discourse/compose.ccci.yml` overlay set
|
||||
the app service `deploy.update_config.order: stop-first` so the new task boots with full memory
|
||||
(no 2× co-residency) and genuinely becomes healthy → no spurious rollback. The upgrade-to-head
|
||||
is still really deployed + asserted on head; HC1 unchanged. Documented WHY in the overlay header.
|
||||
2. **Correctness (honesty, general):** the harness upgrade path detects a swarm rollback after the
|
||||
chaos redeploy (UpdateStatus.State rollback*/paused, or `.Spec` reverted to `.PreviousSpec`) and
|
||||
fails the upgrade with the TRUE reason ("head spec applied then swarm-rolled-back: new task
|
||||
failed the update monitor") instead of the misleading "re-checkout failed". A genuinely
|
||||
undeployable head still FAILS (teeth preserved).
|
||||
3. **Blast-radius:** sweep all enrolled recipes for `failure_action: rollback` + start-first heavy
|
||||
apps with the same latent signature.
|
||||
|
||||
## What is established (direct evidence, reproducible)
|
||||
|
||||
- **abra is CONSTANT, not the cause.** abra binary `bf6azhpi…-abra-0.13.0-beta` is the store
|
||||
path for every nixos system generation from system-4 (2026-06-01) through system-11 (now).
|
||||
No abra change between 06-05 and 06-10.
|
||||
HOW: `for g in $(ls -d /nix/var/nix/profiles/system-*-link); do readlink -f "$g/sw/bin/abra"; done`
|
||||
on cc-ci. EXPECTED: all `…bf6azhpi…` from system-4 on.
|
||||
|
||||
- **abra's chaos-version = `SmallSHA(git HEAD of the recipe checkout)`** (+`+U` if worktree
|
||||
dirty). Source: abra@06a57de `cli/app/deploy.go:106,168,365-373` (chaos →
|
||||
`toDeployVersion = Recipe.ChaosVersion()`), `pkg/recipe/git.go:300-318` (`ChaosVersion` =
|
||||
`SmallSHA(Head())`), `:483-495` (`Head` = go-git `repo.Head()`). In chaos mode
|
||||
`Recipe.Ensure` early-returns (`pkg/recipe/git.go:41-43`) — NO env-version re-checkout.
|
||||
|
||||
- **The isolated git/abra path stamps CORRECTLY now.** Three faithful reproductions on cc-ci
|
||||
(scratch ABRA_DIR, fake domain, deploys bail at `secret not generated` AFTER the chaos
|
||||
version is computed) all log `taking chaos version: 7ae7b0f7` (= PR head), NOT `eb96de9`:
|
||||
1. `cp -a` canonical recipe + manual tag/head checkout.
|
||||
2. real non-chaos base deploy (go-git `EnsureVersion` tag checkout) → CLI re-checkout head → chaos.
|
||||
3. exact `fetch_recipe` replica: clone mirror `recipe-maintainers/discourse` @7ae7b0f +
|
||||
`git fetch upstream refs/tags/*` → base deploy → re-checkout head → chaos.
|
||||
HOW (variant 3, re-runnable cold): see JOURNAL-dstamp 2026-06-11 "mirror-faithful repro".
|
||||
EXPECTED: `DEBU app/deploy.go:372 version: taking chaos version: 7ae7b0f7`.
|
||||
|
||||
- **Same ref, solo run was GREEN; clustered runs DRIFTED.** discourse @ ref `7ae7b0f76efb`:
|
||||
run **184** (2026-06-05 02:17, solo) = **L4, upgrade PASS**; the 06-10/06-11 runs
|
||||
**m2b-discourse** (06-10 20:54), **m2p-discourse** (06-11 00:44), **ab-discourse-7ae7b0f-oldmain**
|
||||
(06-11 00:48) = **L1, upgrade FAIL** (`chaos commit 'eb96de94+U', not the intended PR-head
|
||||
'7ae7b0f76efb' (HC1)`). HOW: `grep -oE '"level": [0-9]+|"upgrade": "[a-z]+"'
|
||||
/var/lib/cc-ci-runs/{184,m2p-discourse}/results.json`.
|
||||
|
||||
- **All same-ref discourse runs share ONE swarm stack.** `naming.app_domain(recipe,pr,ref)` =
|
||||
`<recipe[:4]>-<6hex(recipe|pr|ref)>.ci.commoninternet.net` → identical for identical
|
||||
(recipe,pr,ref). The upgrade `chaos_redeploy` bypasses `deploy_app`'s app-domain flock
|
||||
(`lifecycle.chaos_redeploy` / `generic.perform_upgrade`). LEADING HYPOTHESIS: the 06-10/06-11
|
||||
drift is a CONCURRENCY ARTIFACT of the clustered rcust-M2 A/B discourse experiments racing on
|
||||
the shared stack — NOT an abra/recipe/env regression. Under test now.
|
||||
|
||||
## In flight
|
||||
- Implementing the fix (overlay stop-first + harness rollback detection), then a full real run
|
||||
(all stages) to prove discourse reliably reaches its true level, then the `!testme` drone path.
|
||||
- Repro evidence runs: `/var/lib/cc-ci-runs/dstamp-repro{1,2,3,4}.console.log` on cc-ci
|
||||
(repro2 PASS @7ae7b0f7+U; repro4 captured the rollback Spec/PreviousSpec).
|
||||
|
||||
## Blocked
|
||||
- (none)
|
||||
107
machine-docs/STATUS-kuma.md
Normal file
107
machine-docs/STATUS-kuma.md
Normal file
@ -0,0 +1,107 @@
|
||||
# STATUS — phase `kuma` (uptime-kuma create-a-monitor functional test)
|
||||
|
||||
SSOT: `cc-ci-plan/plan-phase-kuma-monitor.md`
|
||||
|
||||
## Current state
|
||||
|
||||
## DONE
|
||||
|
||||
All DoD items satisfied. M1+M2 Adversary PASSes in REVIEW-kuma.md.
|
||||
|
||||
- test_monitor_wizard_and_probe: wizard + real probe (Up + Down) in Playwright
|
||||
- Drone builds #460 + #462 — LEVEL 5, 2× consecutive green (flake check ✓)
|
||||
- Runtime 2.75–2.82 s ≪ 90 s budget ✓
|
||||
- DEFERRED.md "uptime-kuma create-a-monitor" closed ✓
|
||||
- PARITY.md updated with playwright/ test row ✓
|
||||
- M1 PASS @2026-06-11T18:26Z, M2 PASS @2026-06-11T18:3xZ
|
||||
- No standing VETO
|
||||
|
||||
## What is claimed
|
||||
|
||||
### Approach choice (DECISIONS.md)
|
||||
Playwright (option b). Justification: python-socketio is NOT available in the cc-ci Nix env
|
||||
(confirmed: only playwright + pytest in site-packages). Playwright drives the real browser;
|
||||
Socket.IO is handled transparently. No Nix changes needed.
|
||||
|
||||
### Test file
|
||||
`tests/uptime-kuma/playwright/test_monitor_wizard.py`
|
||||
|
||||
### What the test does
|
||||
1. Completes uptime-kuma 2.2.1 first-run setup wizard (admin create via browser).
|
||||
2. Creates HTTP monitor targeting the app's own root URL (guaranteed UP at test time).
|
||||
3. Waits ≤90 s for status badge (`data-testid="monitor-status"`) to show "Up".
|
||||
4. Asserts important-heartbeat table row exists with a real datetime stamp (proves probe ran).
|
||||
5. Creates a second monitor targeting `http://127.0.0.1:19999/dead` (dead port → connection refused).
|
||||
6. Waits ≤60 s for status badge to show "Down" (negative teeth).
|
||||
|
||||
### Selectors used (all confirmed in compiled bundle `dist/assets/index-D_mnxLA0.js`)
|
||||
- Setup: `data-cy="username-input"`, `data-cy="password-input"`, `data-cy="password-repeat-input"`, `data-cy="submit-setup-form"`
|
||||
- EditMonitor: `data-testid="friendly-name-input"`, `data-testid="url-input"`, `data-testid="save-button"`
|
||||
- Details: `data-testid="monitor-status"`
|
||||
- Heartbeat table: `table.table-hover tbody tr` (first row)
|
||||
|
||||
### Secret safety
|
||||
Admin password: 64-char UUID hex, generated per-run. Never printed, never in any assertion error message.
|
||||
|
||||
### Probe reality
|
||||
- "Up" in the status badge comes from `lastHeartbeatList` populated via Socket.IO heartbeat events
|
||||
(socket.js mixin line 755). Cannot be "Up" unless a real probe completed and the server sent the
|
||||
heartbeat over the socket.
|
||||
- Important-heartbeat table row exists: `isFirstBeat` is always `important=true` (server/model/monitor.js
|
||||
line 1420). Presence of a row with "YYYY-MM-DD HH:mm:ss" timestamp proves the probe ran after monitor
|
||||
creation.
|
||||
- Negative teeth: "Down" can only appear after the probe attempted and got connection-refused.
|
||||
|
||||
### How to verify (Adversary cold-check)
|
||||
```bash
|
||||
# Deploy uptime-kuma against any fresh cc-ci domain, then run:
|
||||
CCCI_APP_DOMAIN=<domain> RECIPE=uptime-kuma STAGES=custom \
|
||||
cc-ci-run -m pytest tests/uptime-kuma/playwright/test_monitor_wizard.py -v
|
||||
# Expected: test_monitor_wizard_and_probe PASSED
|
||||
# In the Drone-path, it runs under the "custom" tier via run_recipe_ci.py.
|
||||
```
|
||||
|
||||
### Runtime
|
||||
Local estimate: wizard ~10 s + 2× (navigate+fill+probe) ≤ ~60 s total. Within ≤90 s budget.
|
||||
|
||||
### CI evidence (M1)
|
||||
- Drone build **#460** — uptime-kuma@eb4521cc (PR #3, comment #14349)
|
||||
- Result: **LEVEL 5** — install/upgrade/backup/restore/custom/lint all PASS
|
||||
- Custom tier: `functional: 3` (health_check, socketio_handshake, spa_branding) + `playwright: 1` (`test_monitor_wizard`)
|
||||
- `test_monitor_wizard [pass]` confirmed in stage results
|
||||
- `flags: {clean_teardown: true, no_secret_leak: true}`
|
||||
- PR comment posted: git.autonomic.zone/recipe-maintainers/uptime-kuma/pulls/3 shows ✅ passed
|
||||
- Artifacts: `/var/lib/cc-ci-runs/460/` on cc-ci
|
||||
|
||||
### M2 evidence (flake check + DEFERRED closed)
|
||||
- Drone build **#462** — uptime-kuma@eb4521cc (PR #3, comment #14352)
|
||||
- Result: **LEVEL 5** — install/upgrade/backup/restore/custom/lint all PASS
|
||||
- `test_monitor_wizard [pass]` — 2 consecutive green runs (#460 + #462)
|
||||
- DEFERRED.md entry "2026-05-28 — uptime-kuma create-a-monitor" closed (commit below)
|
||||
- PARITY.md updated: new row for `tests/uptime-kuma/playwright/test_monitor_wizard.py`
|
||||
|
||||
### How to cold-verify M2
|
||||
```
|
||||
git pull; cat machine-docs/DEFERRED.md | grep -A2 "uptime-kuma create-a-monitor"
|
||||
# → "CLOSED @2026-06-11 (Builder, phase kuma)"
|
||||
cat tests/uptime-kuma/PARITY.md | grep playwright
|
||||
# → row for test_monitor_wizard.py
|
||||
cat /var/lib/cc-ci-runs/462/results.json | python3 ...
|
||||
# → level:5, test_monitor_wizard [pass]
|
||||
```
|
||||
|
||||
### How to cold-verify M1
|
||||
```
|
||||
# On Adversary's clone (cc-ci-adv):
|
||||
git pull; git log --oneline -3 # confirm 8da59cf feat(kuma): implement wizard+monitor Playwright test
|
||||
# Inspect the test:
|
||||
cat tests/uptime-kuma/playwright/test_monitor_wizard.py
|
||||
# Verify CI results:
|
||||
cat /var/lib/cc-ci-runs/460/results.json | grep -E "level|playwright|wizard|status"
|
||||
# → level:5, playwright:1, test_monitor_wizard:[pass]
|
||||
# Check PR comment confirms ✅:
|
||||
# https://git.autonomic.zone/recipe-maintainers/uptime-kuma/pulls/3
|
||||
```
|
||||
|
||||
## Blocked
|
||||
(nothing)
|
||||
71
machine-docs/STATUS-lvl5.md
Normal file
71
machine-docs/STATUS-lvl5.md
Normal file
@ -0,0 +1,71 @@
|
||||
# STATUS — Phase lvl5 (L5 lint rung + de-cap)
|
||||
|
||||
## DONE
|
||||
|
||||
Phase complete 2026-06-11: M1 PASS (cfc87fd) + M2 PASS (13cad1f), both <24h, no VETO.
|
||||
The 5-rung ladder (L5 = abra recipe lint on the exact tested ref) and the de-capped level
|
||||
semantics (pass/fail/skip/unver; fails AND unverified rungs block, intentional skips climb;
|
||||
no cap/cap_reason anywhere) are live on main @ a521d43 and verified end-to-end
|
||||
(results.json schema 2 → card → dashboard → badge → PR comment, drone path included).
|
||||
Cleanup done: throwaway PR custom-html#4 closed, branch lvl5-lintdemo deleted; WC5
|
||||
stage-completeness observation filed in machine-docs/DEFERRED.md.
|
||||
|
||||
## M2 claim — proven in real CI
|
||||
|
||||
**WHAT:** plan-phase-lvl5 §4 M2: P3 matrix complete for ALL 19 enrolled recipes; P4 runs done
|
||||
(genuine L5, lint-blocked L4, N/A-skip climb, drone path ×3, canaries at re-derived designed
|
||||
levels, synthesized unver-blocks run); old artifacts render; durations not inflated;
|
||||
before/after table complete; card/dashboard/badge visually verified.
|
||||
|
||||
**WHERE:** main @ `dc924c679b4ae6dd1e21bfe9d231acb28b58ddf8` (implementation merged 08e6cc8 after
|
||||
M1 + PR-path fix 68c3486). Evidence runs (all artifacts at
|
||||
`https://ci.commoninternet.net/runs/<n>/{results.json,summary.png,badge.svg,lint.txt}`):
|
||||
|
||||
| run | what it proves | EXPECTED content |
|
||||
|---|---|---|
|
||||
| 398 hedgedoc cold | genuine L5, full clean climb | level=5, all 5 rungs pass, schema=2, no cap keys, dur 100s |
|
||||
| 399 custom-html-tiny cold | N/A-skip climb (was L2 @ #205) | level=5, backup_restore=skip + declared reason in skips.intentional, dur 45s |
|
||||
| 405 custom-html PR4 (!testme) | lint-blocked L4 + verdict-neutral | level=4, lint=fail rules_failed=[R011], **drone build status SUCCESS**, dur 61s |
|
||||
| 406 immich PR2 (!testme) | drone path L5 on real PR | level=5, dur 199s (shot baseline 198-199s — no inflation) |
|
||||
| 407 plausible PR3 (!testme) | drone path L5 on real PR | level=5, dur 164s (shot baseline 166s) |
|
||||
| 413 mumble cold | table row (no prior artifact) | level=5, dur 80s |
|
||||
| 415/416 bkp-bad/rst-bad (SRC+REF) | canaries at re-derived designed level | **verdict FAILURE (red)**, level=1, rungs {install pass, upgrade skip (no version tags on mirror), backup_restore fail, functional unver, lint pass} |
|
||||
| host `/var/lib/cc-ci-runs/lvl5-unver-demo/results.json` | synthesized unver-blocks (mission ex. #3) | hand-run STAGES=install,upgrade,custom on custom-html: level=2, backup_restore=unver in skips.unintentional, functional+lint pass above it |
|
||||
|
||||
**HOW to verify (cold):**
|
||||
1. Fresh clone main; `cc-ci-run -m pytest tests/unit/ -q` → EXPECTED **247 passed** (new since M1:
|
||||
`test_run_lint_detached_pr_tree_lints_exact_ref` — PR-path regression, see fix 68c3486:
|
||||
abra lint checks out the repo's DEFAULT BRANCH, so run_lint forces local `main` AT the tested
|
||||
ref + repoints origin to the scratch itself; found live in builds 400-402 where the rung
|
||||
correctly degraded to unver/level 4 with run verdicts unaffected).
|
||||
`nix develop .#lint --command bash scripts/lint.sh` → PASS.
|
||||
2. Fetch each run's results.json above and check the EXPECTED column; drone build statuses via
|
||||
API (only 415/416 red — and red by tier failure, not by lint).
|
||||
3. Visuals: Read `summary.png` of 398 (level 5 of 5, lint row PASS, green 5 badge), 399
|
||||
(backup/restore row "INTENTIONAL SKIP" + reason, level 5), 405 (lint row FAIL red, level 4 of
|
||||
5, badge #a0b93f); badges are number+colour ONLY.
|
||||
4. Old artifacts: `/runs/370/{results.json,summary.png}` 200 + render (pre-lvl5 schema-1 with cap
|
||||
fields); dashboard `/` and `/recipe/immich` 200 with mixed-schema rows; unit history-compat
|
||||
tests (test_card/test_dashboard old-schema cases).
|
||||
5. lint.txt served: `/runs/398/lint.txt` 200 (full abra table; rc/status header).
|
||||
6. P3 matrix + §2.9 before/after table: BACKLOG-lvl5.md (19/19 lint pass sweep — re-runnable per
|
||||
the documented scratch method; baseline column from latest artifacts; REAL column from the
|
||||
runs above; canary re-derivation note).
|
||||
7. Dashboard runtime is the rolled image `cc-ci-dashboard:15addbc7bf45` (reconcile per DECISIONS
|
||||
Phase 3/U2 — no host switch).
|
||||
|
||||
**Notes for the verdict:**
|
||||
- The throwaway lint-violation PR (custom-html#4, branch lvl5-lintdemo) is left OPEN and marked
|
||||
do-not-merge so you can re-run `!testme` independently; Builder will close branch+PR after M2.
|
||||
- Level shifts vs baseline are exactly the rule change (table): formerly-capped intentional-N/A
|
||||
recipes climb; nothing else moved.
|
||||
- Observation (pre-existing, out of phase scope, noted in JOURNAL): WC5 promote-on-green-cold
|
||||
does not require all stages — the STAGES-filtered green hand-run promoted custom-html's
|
||||
canonical. Filed as a JOURNAL note; flag if you want it as a finding.
|
||||
|
||||
---
|
||||
|
||||
## (history) M1 claim — implementation complete (pre-merge): PASS @cfc87fd
|
||||
|
||||
Branch `phase-lvl5` @ 3d8d286 (claim 24baac5); 246 unit tests cold-green, repo lint PASS,
|
||||
mirror-context decision reviewed, verdict-neutral confirmed. Merged to main 08e6cc8.
|
||||
141
machine-docs/STATUS-mailu.md
Normal file
141
machine-docs/STATUS-mailu.md
Normal file
@ -0,0 +1,141 @@
|
||||
# STATUS — phase mailu (backupbot labels for mailu recipe)
|
||||
|
||||
**Phase plan:** `/srv/cc-ci/cc-ci-plan/plan-phase-mailu-backup.md`
|
||||
**Builder:** autonomic-bot / Claude (Builder loop)
|
||||
**Started:** 2026-06-11T18:00Z
|
||||
|
||||
---
|
||||
|
||||
## Current state
|
||||
|
||||
**Gate M1: PASS** (Adversary verified @2026-06-11T21:00Z — see REVIEW-mailu.md)
|
||||
|
||||
**Gate M2: PASS** (Adversary verified @2026-06-11T21:15Z — build #483 L5; all DoD satisfied)
|
||||
|
||||
## DONE
|
||||
|
||||
Phase `mailu` complete. M1 PASS @2026-06-11T21:00Z + M2 PASS @2026-06-11T21:15Z.
|
||||
|
||||
**PR left open for operator merge:**
|
||||
https://git.autonomic.zone/recipe-maintainers/mailu/pulls/3
|
||||
(branch `add-backupbot-labels`, head `edc0201a79d36bc87696b0f93f1ee88ad7bd10ed`)
|
||||
|
||||
**Evidence:**
|
||||
- Drone build #477 (ADV-mailu-01 fix re-claim): LEVEL 5, all rungs PASS
|
||||
- Drone build #483 (Adversary fresh independent re-trigger): LEVEL 5, all rungs PASS
|
||||
- Both builds: `test_backup_captures_mailbox`, `test_backup_captures_mail_message`,
|
||||
`test_restore_returns_mailbox`, `test_restore_returns_mail_message` — all PASS
|
||||
- DEFERRED entry closed; PARITY.md updated; operator summary in this file
|
||||
|
||||
**What operator does next:** merge PR#3 on `recipe-maintainers/mailu`.
|
||||
|
||||
---
|
||||
|
||||
## DoD tracker (M1) — COMPLETE
|
||||
|
||||
- [x] Data-layout research documented (which volumes hold durable state, justification in PR desc)
|
||||
- [x] Recipe-mirror PR open with backupbot v2 labels (admin `/data` + imap `/mail`)
|
||||
- **PR#3**: https://git.autonomic.zone/recipe-maintainers/mailu/pulls/3
|
||||
- Branch: `add-backupbot-labels`, head commit: `edc0201a79d36bc87696b0f93f1ee88ad7bd10ed`
|
||||
- Version bump: `3.0.1+2024.06.52` → `3.0.2+2024.06.52`
|
||||
- Adds `deploy.labels: {backupbot.backup: "true", backupbot.backup.path: "/data"}` to `admin`
|
||||
- Adds `deploy.labels: {backupbot.backup: "true", backupbot.backup.path: "/mail"}` to `imap`
|
||||
- [x] cc-ci: `tests/mailu/ops.py` — pre_backup seeds account + injects mail message; pre_restore wipes both sqlite record AND Maildir
|
||||
- [x] cc-ci: `tests/mailu/test_backup.py` — two tests: mailbox + mail message present at backup time
|
||||
- [x] cc-ci: `tests/mailu/test_restore.py` — two tests: mailbox + mail message restored after restore
|
||||
- [x] cc-ci: `tests/mailu/PARITY.md` updated (P4 COVERED with dual-volume evidence)
|
||||
- [x] Drone build #477: LEVEL 5 PASS at PR head — all rungs including backup/restore on both volumes
|
||||
- `test_backup_captures_mailbox` PASS — SQLite `/data` covered
|
||||
- `test_backup_captures_mail_message` PASS — Maildir `/mail` covered
|
||||
- `test_restore_returns_mailbox` PASS — SQLite `/data` restored
|
||||
- `test_restore_returns_mail_message` PASS — Maildir `/mail` restored
|
||||
- `clean_teardown: true`, `no_secret_leak: true`
|
||||
- [x] Before/after: BEFORE = L4 (backup intentional-skip); AFTER = L5 (earned)
|
||||
- [x] M1 Adversary PASS @2026-06-11T21:00Z; ADV-mailu-01 closed
|
||||
|
||||
## DoD tracker (M2) — IN PROGRESS
|
||||
|
||||
- [x] DEFERRED entry closed (DEFERRED.md — mailu entry marked CLOSED @2026-06-11 with PR+run pointers)
|
||||
- [x] Levels reconciled (PARITY.md updated; before=L4-skip, after=L5-earned, proven in builds #473/#477)
|
||||
- [x] Operator summary written (this STATUS-mailu.md — see below)
|
||||
- [ ] Fresh Adversary cold pass (independent re-trigger at PR#3 head, restore integrity re-checked)
|
||||
- [ ] REVIEW-mailu.md shows M2 PASS (within 24h of M1)
|
||||
|
||||
---
|
||||
|
||||
## Verification recipe (for Adversary M2 check)
|
||||
|
||||
```bash
|
||||
# 1. Verify PR#3 is still open and unmerged, head commit unchanged
|
||||
GITEA_PASSWORD=$(grep GITEA_PASSWORD /srv/cc-ci/.testenv | cut -d= -f2-)
|
||||
curl -s "https://git.autonomic.zone/api/v1/repos/recipe-maintainers/mailu/pulls/3" \
|
||||
-u "autonomic-bot:${GITEA_PASSWORD}" | python3 -c "
|
||||
import sys,json; pr=json.load(sys.stdin)
|
||||
print('state:', pr['state'])
|
||||
print('head sha:', pr['head']['sha'])
|
||||
print('merged:', pr.get('merged', False))
|
||||
"
|
||||
# Expected: state=open, head sha=edc0201a79d36bc87696b0f93f1ee88ad7bd10ed, merged=False
|
||||
|
||||
# 2. Re-trigger via !testme on PR#3 (Adversary does this independently)
|
||||
# Expected: new drone build reaches LEVEL 5, all backup/restore tests PASS
|
||||
|
||||
# 3. Verify DEFERRED.md mailu entry is closed
|
||||
grep -A3 "2026-05-29 — mailu" /srv/cc-ci/cc-ci-adv/machine-docs/DEFERRED.md
|
||||
# Expected: [x] CLOSED @2026-06-11 with PR#3 + build #477 pointer
|
||||
|
||||
# 4. Verify PARITY.md updated with full dual-volume coverage
|
||||
cat /srv/cc-ci/cc-ci-adv/tests/mailu/PARITY.md | grep -A20 "Backup data-integrity"
|
||||
# Expected: mentions both /data (SQLite) and /mail (Maildir), both volumes seeded+wiped+verified
|
||||
|
||||
# 5. Confirm levels: before=L4, after=L5
|
||||
# BEFORE: git.autonomic.zone/recipe-maintainers/mailu main — no backupbot labels → backup_capable=False → skip → L4
|
||||
# AFTER: PR#3 head edc0201a79d3 — backupbot labels present → backup_capable=True → L5 (all rungs earned)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Operator summary (for handoff)
|
||||
|
||||
### What this phase delivered
|
||||
|
||||
**PR#3 on `git.autonomic.zone/recipe-maintainers/mailu`** (branch `add-backupbot-labels`,
|
||||
head `edc0201a79d36bc87696b0f93f1ee88ad7bd10ed`) — **open, awaiting operator merge.**
|
||||
|
||||
**What the PR adds:**
|
||||
- Backupbot v2 labels on `admin` service: `backupbot.backup: "true"` + `backupbot.backup.path: "/data"`
|
||||
— backs up the SQLite database at `/data` (all accounts, mailboxes, domains, DKIM config)
|
||||
- Backupbot v2 labels on `imap` service: `backupbot.backup: "true"` + `backupbot.backup.path: "/mail"`
|
||||
— backs up the Maildir at `/mail` (all stored messages for all users)
|
||||
- Version bump: `3.0.1+2024.06.52` → `3.0.2+2024.06.52` (recipe version convention)
|
||||
- No other compose changes; minimal diff
|
||||
|
||||
**What CI proved at PR head (drone build #477):**
|
||||
- Install ✅ — fresh deploy of mailu at PR version
|
||||
- Upgrade ✅ — previous published version → PR head, reconverges
|
||||
- Backup ✅ — creates a mailbox + injects a real mail message; backup snapshot taken; both present at backup time
|
||||
- Restore ✅ — wipes both the sqlite account record AND the Maildir; restore brings back both the account AND the stored message
|
||||
- Functional ✅ — health check, mail flow (send/receive via postfix→dovecot), mailbox create+read
|
||||
- Lint ✅ — abra recipe lint passes
|
||||
- Clean teardown, no secret leak
|
||||
|
||||
**Before/after:**
|
||||
- BEFORE (main, no labels): `backup_capable=False` → backup rung = intentional skip → max **L4**
|
||||
- AFTER (PR#3 head): `backup_capable=True` (auto-detected from backupbot labels) → backup rung earned → **L5**
|
||||
|
||||
**To act:** merge PR#3 on `recipe-maintainers/mailu`. After merge, mailu will earn L5 on main
|
||||
(`!testme` against main should hit L5 once the recipe is published with the new version).
|
||||
|
||||
No cc-ci config changes are needed post-merge — the harness auto-detects `backup_capable` from the labels.
|
||||
|
||||
---
|
||||
|
||||
## Blocked items
|
||||
|
||||
(none)
|
||||
|
||||
---
|
||||
|
||||
## DONE
|
||||
|
||||
Not yet. Written here only when M1+M2 Adversary PASS appear in REVIEW-mailu.md.
|
||||
BIN
machine-docs/screenshots/drone-m2-build506.png
Normal file
BIN
machine-docs/screenshots/drone-m2-build506.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 70 KiB |
@ -40,7 +40,7 @@ let
|
||||
# admin-registered push optimization deduped against the poller (§4.1). Enrollment = add
|
||||
# the repo to POLL_REPOS (csv) + ensure tests/<recipe>/ exists.
|
||||
- POLL_INTERVAL=30
|
||||
- POLL_REPOS=recipe-maintainers/cc-ci,recipe-maintainers/custom-html,recipe-maintainers/custom-html-tiny,recipe-maintainers/keycloak,recipe-maintainers/cryptpad,recipe-maintainers/matrix-synapse,recipe-maintainers/lasuite-docs,recipe-maintainers/lasuite-meet,recipe-maintainers/n8n,recipe-maintainers/hedgedoc,recipe-maintainers/uptime-kuma,recipe-maintainers/bluesky-pds,recipe-maintainers/discourse,recipe-maintainers/ghost,recipe-maintainers/immich,recipe-maintainers/lasuite-drive,recipe-maintainers/mailu,recipe-maintainers/mattermost-lts,recipe-maintainers/mumble,recipe-maintainers/plausible
|
||||
- POLL_REPOS=recipe-maintainers/cc-ci,recipe-maintainers/custom-html,recipe-maintainers/custom-html-tiny,recipe-maintainers/keycloak,recipe-maintainers/cryptpad,recipe-maintainers/matrix-synapse,recipe-maintainers/lasuite-docs,recipe-maintainers/lasuite-meet,recipe-maintainers/n8n,recipe-maintainers/hedgedoc,recipe-maintainers/uptime-kuma,recipe-maintainers/bluesky-pds,recipe-maintainers/discourse,recipe-maintainers/ghost,recipe-maintainers/immich,recipe-maintainers/lasuite-drive,recipe-maintainers/mailu,recipe-maintainers/mattermost-lts,recipe-maintainers/mumble,recipe-maintainers/plausible,recipe-maintainers/drone
|
||||
- HMAC_FILE=/run/secrets/webhook_hmac
|
||||
- DRONE_TOKEN_FILE=/run/secrets/drone_token
|
||||
- GITEA_TOKEN_FILE=/run/secrets/gitea_token
|
||||
|
||||
@ -16,6 +16,8 @@ Per Phase-2 DECISIONS:
|
||||
must share the single node's MAX_TESTS budget without exceeding it).
|
||||
- Each dep is undeployed in the orchestrator's `finally`, in **reverse** order so a recipe-under-
|
||||
test can depend on multiple deps with a dependency chain (a → b → c teardown is c → b → a).
|
||||
- Dep deploys DO count toward the DG4.1 deploy-count invariant. The formula in run_recipe_ci.py is
|
||||
`expected_deploy_count = 1 + deps_deployed_count`, so each dep deploy increments the counter.
|
||||
|
||||
Run state:
|
||||
- `$CCCI_DEPS_FILE` — JSON file written by the orchestrator after each dep deploys; each entry is
|
||||
@ -80,9 +82,9 @@ def deploy_deps(
|
||||
for dep in deps:
|
||||
domain = dep_domain(parent_recipe, pr, ref, dep)
|
||||
print(f" dep: deploying {dep} -> {domain}", flush=True)
|
||||
# NB: each dep_app gets a fresh deploy_count entry only on `_record_deploy` which fires
|
||||
# inside `lifecycle.deploy_app`. For Phase 2 the deploy-count guard (DG4.1) counts the
|
||||
# parent + its deps as distinct install events — by design, since each is a separate app.
|
||||
# Dep deploys count toward DG4.1 — the check expects (1 + len(cold-deps)), so each
|
||||
# dep that deploys here MUST be counted. The formula is authoritative in run_recipe_ci.py:
|
||||
# expected_deploy_count = 1 + deps_deployed_count
|
||||
dm = meta_for.get(dep) or meta_mod.load(dep)
|
||||
lifecycle.deploy_app(
|
||||
dep,
|
||||
|
||||
@ -11,8 +11,8 @@ hook; the orchestrator decides additive-vs-skip. Sources, in precedence order
|
||||
> cc-ci tests/<recipe>/test_<op>.py
|
||||
(the generic tests/_generic/test_<op>.py is the always-present floor, run separately by default)
|
||||
|
||||
custom test_*.py (functional/ + playwright/ ONLY, rcust P4 placement rule) — ALL run,
|
||||
additively, from BOTH locations (opt-in).
|
||||
custom test_*.py (`custom/` canonical; `functional/` + `playwright/` deprecated aliases) —
|
||||
ALL run, additively, from BOTH locations (opt-in).
|
||||
|
||||
install-steps hook — install_steps.sh: repo-local > cc-ci, or none.
|
||||
|
||||
@ -27,6 +27,7 @@ from __future__ import annotations
|
||||
|
||||
import glob
|
||||
import os
|
||||
import sys
|
||||
|
||||
LIFECYCLE_OPS = ("install", "upgrade", "backup", "restore")
|
||||
|
||||
@ -102,15 +103,16 @@ def resolve_op(recipe: str, op: str, repo_local_dir: str | None) -> tuple[str, s
|
||||
|
||||
def custom_tests(recipe: str, repo_local_dir: str | None) -> list[tuple[str, str]]:
|
||||
"""All custom-tier test_*.py from cc-ci's tests/<recipe>/ and (if approved) the recipe's
|
||||
repo-local tests/. PLACEMENT RULE (rcust P4): custom tests live ONLY under
|
||||
- functional/ tests/<recipe>/functional/test_*.py (parity ports + recipe-specific)
|
||||
- playwright/ tests/<recipe>/playwright/test_*.py (UI flows)
|
||||
repo-local tests/. PLACEMENT RULE (rcust P4): custom tests live under canonical
|
||||
- custom/ tests/<recipe>/custom/test_*.py (canonical home)
|
||||
- functional/ tests/<recipe>/functional/test_*.py (deprecated alias)
|
||||
- playwright/ tests/<recipe>/playwright/test_*.py (deprecated alias)
|
||||
A top-level test_*.py is a LIFECYCLE OVERLAY (test_<op>.py) and nothing else — top-level
|
||||
non-lifecycle files are NOT discovered (zero users at the time of the change; the lifecycle-
|
||||
name exclusion below stays as a safety net so a misfiled test_<op>.py can never double-run).
|
||||
Repo-local is consulted only for allowlist-approved recipes (HC2)."""
|
||||
lifecycle_names = {f"test_{op}.py" for op in LIFECYCLE_OPS}
|
||||
subdirs = ("functional", "playwright")
|
||||
subdirs = ("custom", "functional", "playwright")
|
||||
found: list[tuple[str, str]] = []
|
||||
for source, d in (("cc-ci", cc_ci_dir(recipe)), ("repo-local", _gated(recipe, repo_local_dir))):
|
||||
if not d or not os.path.isdir(d):
|
||||
@ -118,6 +120,12 @@ def custom_tests(recipe: str, repo_local_dir: str | None) -> list[tuple[str, str
|
||||
for sub in subdirs:
|
||||
for p in sorted(glob.glob(os.path.join(d, sub, "test_*.py"))):
|
||||
if os.path.basename(p) not in lifecycle_names:
|
||||
if sub != "custom":
|
||||
print(
|
||||
f"WARNING [cfold]: test found in deprecated folder '{sub}/' — move to custom/: {p}",
|
||||
file=sys.stderr,
|
||||
flush=True,
|
||||
)
|
||||
found.append((source, p))
|
||||
return found
|
||||
|
||||
|
||||
@ -263,8 +263,19 @@ def perform_upgrade(
|
||||
# HQ1: warm the NEW-version image set before the chaos redeploy (the head_ref checkout's pinned
|
||||
# tags) so a pull failure is a clear pre-deploy error and convergence isn't pull-bound.
|
||||
lifecycle.prepull_images(recipe, domain)
|
||||
# Snapshot the app service's pre-redeploy swarm update marker so assert_upgrade_converged can
|
||||
# tell the NEW rolling update apart from the install/base deploy's stale terminal state.
|
||||
prev_started = lifecycle.update_status_started(domain)
|
||||
lifecycle.chaos_redeploy(domain, deploy_timeout=deploy_timeout, no_converge_checks=True)
|
||||
# Own the convergence verification (abra's monitor was skipped via -c).
|
||||
# Own the convergence verification (abra's monitor was skipped via -c). FIRST confirm swarm's
|
||||
# rolling update of the app service actually converged to the NEW (head) spec and was not
|
||||
# silently rolled back/paused (dstamp: failure_action=rollback + order=start-first reverts the
|
||||
# chaos-version label while the old task keeps serving, so wait_healthy alone would pass on a
|
||||
# reverted-to-base spec and HC1 would misreport it as a stamp mismatch). A rollback/pause here
|
||||
# is a genuine upgrade failure (head did not stay healthy) — surfaced honestly, HC1 unweakened.
|
||||
lifecycle.assert_upgrade_converged(
|
||||
domain, timeout=int(meta.DEPLOY_TIMEOUT), prev_started=prev_started
|
||||
)
|
||||
lifecycle.wait_healthy(
|
||||
domain,
|
||||
ok_codes=tuple(meta.HEALTH_OK),
|
||||
|
||||
@ -238,6 +238,7 @@ def deploy_app(
|
||||
install_steps_hook: tuple[str, str] | None = None,
|
||||
deploy_timeout: int = 900,
|
||||
meta=None,
|
||||
_count_deploy: bool = True,
|
||||
) -> None:
|
||||
"""Create + configure + deploy an app. Forces LETS_ENCRYPT_ENV='' so traefik serves the
|
||||
wildcard cert via the file provider and NEVER attempts ACME (adversary finding A1). Applies any
|
||||
@ -251,10 +252,16 @@ def deploy_app(
|
||||
`deploy_timeout` is the subprocess timeout for `abra app deploy`. Caller (orchestrator) passes
|
||||
`recipe_meta.DEPLOY_TIMEOUT` so heavy recipes (ghost, matrix-synapse, lasuite-meet) can extend
|
||||
past the 900s default. abra's INTERNAL TIMEOUT (recipe's TIMEOUT env, default 300s) is set via
|
||||
EXTRA_ENV; this is the Python subprocess wrapper's timeout so abra doesn't get SIGKILLed mid-deploy."""
|
||||
EXTRA_ENV; this is the Python subprocess wrapper's timeout so abra doesn't get SIGKILLed mid-deploy.
|
||||
|
||||
`_count_deploy`: internal escape hatch — set False to skip incrementing the DG4.1 deploy
|
||||
counter (e.g. for test fixtures that call deploy_app without participating in a real run).
|
||||
Normal orchestration should always use the default True — dep deploys count too (the DG4.1
|
||||
formula is `expected = 1 + deps_count`, so deps MUST be counted; see run_recipe_ci.py)."""
|
||||
if meta is None:
|
||||
meta = meta_mod.load(recipe)
|
||||
_record_deploy()
|
||||
if _count_deploy:
|
||||
_record_deploy()
|
||||
# Lock BEFORE the app exists: a concurrent run's janitor must never see this app without a
|
||||
# held app lock (it would probe it as an orphan and reap an in-flight deploy). Also the
|
||||
# double-!testme serialisation point: a second run of the same domain blocks here.
|
||||
@ -508,6 +515,124 @@ def deployed_identity(domain: str, service: str = "app") -> dict[str, str | None
|
||||
return {"version": ver, "image": image.strip() or None, "chaos": chaos or chaos_flag}
|
||||
|
||||
|
||||
def update_status_started(domain: str, service: str = "app") -> str:
|
||||
"""The app service's current `UpdateStatus.StartedAt` ('' if no update recorded). Captured
|
||||
BEFORE the upgrade chaos redeploy so assert_upgrade_converged can tell the NEW rolling update
|
||||
apart from a stale terminal state left by the install/base deploy (closes the race where
|
||||
`docker stack deploy -c` returns before swarm schedules the roll)."""
|
||||
name = f"{_stack_name(domain)}_{service}"
|
||||
proc = subprocess.run(
|
||||
[
|
||||
"docker",
|
||||
"service",
|
||||
"inspect",
|
||||
name,
|
||||
"--format",
|
||||
"{{if .UpdateStatus}}{{.UpdateStatus.StartedAt}}{{else}}{{end}}",
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
return proc.stdout.strip()
|
||||
|
||||
|
||||
def assert_upgrade_converged(
|
||||
domain: str, service: str = "app", timeout: int = 900, prev_started: str | None = None
|
||||
) -> None:
|
||||
"""After an in-place upgrade chaos redeploy, wait for swarm's rolling update of the app service
|
||||
to reach a TERMINAL state and assert it converged to the NEW (head) spec — i.e. did NOT roll
|
||||
back or pause. Raises on a non-converged update; returns on success / nothing-to-converge.
|
||||
|
||||
`prev_started` is the app service's `UpdateStatus.StartedAt` captured BEFORE the redeploy (via
|
||||
update_status_started). It closes the race the Adversary flagged: `chaos_redeploy` runs
|
||||
`docker stack deploy -c` which returns BEFORE swarm schedules the rolling update, so the first
|
||||
poll could read a STALE terminal `completed` (from the install/base deploy) and wrongly return
|
||||
OK, then miss a rollback that fires moments later. We therefore (phase 1) wait until the NEW
|
||||
update is observed — `StartedAt` advances past `prev_started`, or the state is an in-flight
|
||||
`updating`/`rollback_started` — before (phase 2) accepting a terminal verdict. A no-op redeploy
|
||||
that triggers no update at all (StartedAt never advances within a short grace) ⇒ OK (nothing to
|
||||
converge); in practice the base→head upgrade always changes the spec, so an update always fires.
|
||||
|
||||
WHY (dstamp attribution, direct evidence in JOURNAL-dstamp 2026-06-11): a recipe whose app
|
||||
service sets `deploy.update_config.failure_action: rollback` with `order: start-first` (e.g.
|
||||
discourse) will, when the NEW task fails swarm's update monitor (e.g. a precompile/Rails-heavy
|
||||
app OOMing under start-first's 2x old+new co-residency), execute the rollback and revert the
|
||||
service to its PREVIOUS spec — INCLUDING the `coop-cloud.<stack>.chaos-version` label. Under
|
||||
start-first the OLD task keeps serving, so `wait_healthy` still passes; the reverted spec then
|
||||
makes HC1 read the BASE commit and misreport it as 'the re-checkout to the code under test
|
||||
failed'. The harness had ASSUMED `wait_healthy` (all services N/N + app health) implies the
|
||||
upgrade converged to head — false under start-first + a rolled-back/paused update. This check
|
||||
makes a rollback/pause VISIBLE and fails the upgrade HONESTLY (the head did not stay healthy ⇒
|
||||
not really upgraded to the code under test), WITHOUT weakening HC1: the underlying commit match
|
||||
is unchanged; this only stops a silent swarm revert from masquerading as a stamp mismatch and
|
||||
closes the wait_healthy-masking hole. abra's own monitor (`-c`) was skipped for the upgrade
|
||||
redeploy, so the harness must own this convergence check itself.
|
||||
|
||||
Terminal states: `completed` (OK). `rollback_completed`/`rollback_paused`/`paused` (FAIL — the
|
||||
new task failed the monitor; running spec is not the code under test). Empty/`none` UpdateStatus
|
||||
(fresh service or a no-op redeploy that performed no update) ⇒ OK (nothing to converge). While
|
||||
`updating`/`rollback_started` (in flight) keep waiting up to `timeout`."""
|
||||
name = f"{_stack_name(domain)}_{service}"
|
||||
fmt = "{{if .UpdateStatus}}{{.UpdateStatus.State}}|{{.UpdateStatus.StartedAt}}{{else}}none|{{end}}"
|
||||
terminal_ok = ("completed",)
|
||||
terminal_fail = ("rollback_completed", "rollback_paused", "paused")
|
||||
|
||||
def _poll() -> tuple[str, str]:
|
||||
proc = subprocess.run(
|
||||
["docker", "service", "inspect", name, "--format", fmt],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
state, _, started = proc.stdout.strip().partition("|")
|
||||
return state, started
|
||||
|
||||
deadline = time.time() + timeout
|
||||
prev_started = prev_started or ""
|
||||
# Phase 1: confirm the NEW rolling update has actually been scheduled (don't trust a stale
|
||||
# terminal state left by the install/base deploy). Short grace: if no update fires, it's a
|
||||
# no-op redeploy (spec unchanged) → nothing to converge.
|
||||
grace = time.time() + 30
|
||||
observed_new = False
|
||||
while time.time() < deadline:
|
||||
state, started = _poll()
|
||||
if started and started != prev_started:
|
||||
observed_new = True
|
||||
break
|
||||
if state in ("updating", "rollback_started"):
|
||||
observed_new = True
|
||||
break
|
||||
if time.time() > grace:
|
||||
print(
|
||||
f" upgrade-converged: {name} no swarm update scheduled within grace "
|
||||
f"(no-op redeploy, spec unchanged) — nothing to converge",
|
||||
flush=True,
|
||||
)
|
||||
return
|
||||
time.sleep(2)
|
||||
# Phase 2: wait for the (now-confirmed-new) update to reach a terminal state.
|
||||
last = None
|
||||
while time.time() < deadline:
|
||||
state, _ = _poll()
|
||||
last = state
|
||||
if state in terminal_ok:
|
||||
print(f" upgrade-converged: {name} swarm UpdateStatus=completed", flush=True)
|
||||
return
|
||||
if state in terminal_fail:
|
||||
raise RuntimeError(
|
||||
f"{domain}: upgrade redeploy did NOT converge to the head spec — swarm "
|
||||
f"UpdateStatus={state!r}. The recipe's app service uses update_config "
|
||||
f"failure_action=rollback/pause; the NEW (head) task failed swarm's update monitor, "
|
||||
f"so the service reverted/paused and the RUNNING spec is the previous version, not "
|
||||
f"the code under test. This is a real upgrade failure (the head did not stay "
|
||||
f"healthy under the deploy), surfaced honestly — not a stamp mismatch."
|
||||
)
|
||||
time.sleep(5)
|
||||
raise RuntimeError(
|
||||
f"{domain}: upgrade redeploy update did not reach a terminal swarm state within {timeout}s "
|
||||
f"(observed_new={observed_new}, last UpdateStatus={last!r}) — non-converged upgrade."
|
||||
)
|
||||
|
||||
|
||||
def upgrade_app(domain: str, version: str | None = None) -> None:
|
||||
abra.upgrade(domain, version=version)
|
||||
|
||||
|
||||
@ -128,14 +128,35 @@ def run_lint(recipe: str, ref: str | None, out_dir: str | None) -> dict:
|
||||
text=True,
|
||||
timeout=LINT_TIMEOUT,
|
||||
)
|
||||
if ref:
|
||||
subprocess.run(
|
||||
["git", "-C", clone, "checkout", "-f", "--quiet", ref],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=LINT_TIMEOUT,
|
||||
)
|
||||
# abra lint SELECTS AND CHECKS OUT THE REPO'S DEFAULT BRANCH before linting (observed
|
||||
# live, build 400-402: a clone of a detached-HEAD per-run tree has no local branch →
|
||||
# FATA "failed to select default branch"; and if a default branch existed at some OTHER
|
||||
# commit, abra would silently lint THAT, not the tested ref). So: force a local `main`
|
||||
# AT exactly the tested ref and make it the default everywhere abra could look —
|
||||
# HEAD, and origin (repointed to the scratch itself, which also turns abra's tag
|
||||
# force-fetch into an offline no-op; the run's true tags were already cloned in).
|
||||
subprocess.run(
|
||||
["git", "-C", clone, "checkout", "-f", "--quiet", "-B", "main"]
|
||||
+ ([ref] if ref else []),
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=LINT_TIMEOUT,
|
||||
)
|
||||
subprocess.run(
|
||||
["git", "-C", clone, "remote", "set-url", "origin", clone],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=LINT_TIMEOUT,
|
||||
)
|
||||
subprocess.run(
|
||||
["git", "-C", clone, "remote", "set-head", "origin", "main"],
|
||||
check=False, # cosmetic: helps any origin-HEAD-based default-branch lookup
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=LINT_TIMEOUT,
|
||||
)
|
||||
# catalogue: R006 (published catalogue version) reads it; servers: harmless, some abra
|
||||
# paths stat it. Symlink the live ones (read-only use).
|
||||
for shared in ("catalogue", "servers"):
|
||||
|
||||
@ -52,7 +52,7 @@ def _pre_ops(path: str) -> list[str]:
|
||||
def _custom_counts(recipe: str, repo_local: str | None) -> dict[str, dict[str, int]]:
|
||||
out: dict[str, dict[str, int]] = {}
|
||||
for source, path in discovery.custom_tests(recipe, repo_local):
|
||||
sub = os.path.basename(os.path.dirname(path)) # functional | playwright
|
||||
sub = "custom"
|
||||
out.setdefault(source, {}).setdefault(sub, 0)
|
||||
out[source][sub] += 1
|
||||
return out
|
||||
|
||||
@ -76,7 +76,7 @@ KEYS: tuple[Key, ...] = (
|
||||
"EXPECTED_NA",
|
||||
"dict",
|
||||
None,
|
||||
"Declare a non-run rung an INTENTIONAL skip: `{rung: reason}` — the level climbs past it; an undeclared non-run rung is *unverified* and blocks the level above it (classification table: machine-docs/DECISIONS.md phase lvl5). Never overrides an exercised pass/fail; the `lint` rung has no escape hatch.",
|
||||
"Declare a non-run rung an INTENTIONAL skip: `{rung: reason}` — the level climbs past it; an undeclared non-run rung is *unverified* and blocks the level above it (classification table: machine-docs/DECISIONS.md phase lvl5). Never overrides an exercised pass/fail; the `lint` rung has no escape hatch. Declaring `upgrade` also suppresses the upgrade-tier BASE deploy — the single deploy is the PR head itself — for recipes whose published versions exist but are genuinely undeployable (phase bsky).",
|
||||
),
|
||||
Key(
|
||||
"READY_PROBE",
|
||||
|
||||
@ -365,3 +365,141 @@ def load_sso_creds() -> dict | None:
|
||||
return json.load(f)
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Gitea OAuth2 setup — drone dep provider (phase drone)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _gitea_api(
|
||||
provider_domain: str,
|
||||
path: str,
|
||||
method: str = "GET",
|
||||
body=None,
|
||||
*,
|
||||
username: str,
|
||||
password: str,
|
||||
) -> tuple[int, object]:
|
||||
"""Call the gitea REST API (basic-auth). Returns (status, body_json_or_None)."""
|
||||
import base64
|
||||
|
||||
data = json.dumps(body).encode() if body is not None else None
|
||||
auth = base64.b64encode(f"{username}:{password}".encode()).decode()
|
||||
headers: dict[str, str] = {"Authorization": f"Basic {auth}"}
|
||||
if data:
|
||||
headers["Content-Type"] = "application/json"
|
||||
req = urllib.request.Request(
|
||||
f"https://{provider_domain}/api/v1{path}",
|
||||
data=data,
|
||||
headers=headers,
|
||||
method=method,
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=30, context=_CTX) as r:
|
||||
raw = r.read()
|
||||
return r.status, (json.loads(raw) if raw else None)
|
||||
except urllib.error.HTTPError as e:
|
||||
raw = e.read()
|
||||
try:
|
||||
parsed = json.loads(raw) if raw else None
|
||||
except (ValueError, json.JSONDecodeError):
|
||||
parsed = None
|
||||
return e.code, parsed
|
||||
|
||||
|
||||
def setup_gitea_oauth(provider_domain: str, parent_domain: str) -> dict:
|
||||
"""Create a gitea admin user + OAuth2 application for a drone dep.
|
||||
|
||||
Steps:
|
||||
1. Create admin user via `gitea admin user create` CLI inside the container.
|
||||
2. Create an OAuth2 app via the gitea REST API (basic auth as the new admin).
|
||||
3. Return a creds dict: {admin_user, admin_password, client_id, client_secret}.
|
||||
|
||||
The caller (orchestrator) stores creds in $CCCI_DEPS_FILE so drone's install_steps.sh
|
||||
can wire DRONE_GITEA_CLIENT_ID + the client_secret Docker secret before the first deploy.
|
||||
Per plan §4.4-B, the client_secret is class-B run-scoped and destroyed on teardown.
|
||||
"""
|
||||
admin_user = "ci_admin"
|
||||
# 32-char alphanumeric password — safe to pass as a CLI arg (no shell metacharacters)
|
||||
admin_password = secrets.token_hex(16)
|
||||
admin_email = "ci@ci.local"
|
||||
|
||||
# 1. Create admin user via gitea CLI inside the running container.
|
||||
# The rootless gitea image has GITEA_WORK_DIR + GITEA_CUSTOM set as ENV; docker exec
|
||||
# inherits those (image ENV is part of the container config). The gitea binary is in PATH.
|
||||
print(f" gitea dep: creating admin user {admin_user!r} on {provider_domain}", flush=True)
|
||||
try:
|
||||
out = lifecycle.exec_in_app(
|
||||
provider_domain,
|
||||
[
|
||||
"gitea",
|
||||
"admin",
|
||||
"user",
|
||||
"create",
|
||||
"--admin",
|
||||
"--username",
|
||||
admin_user,
|
||||
"--password",
|
||||
admin_password,
|
||||
"--email",
|
||||
admin_email,
|
||||
"--must-change-password=false", # equals-form required; gitea BoolFlag default=true
|
||||
],
|
||||
timeout=120,
|
||||
)
|
||||
print(f" gitea dep: admin user created: {out.strip()[:80]}", flush=True)
|
||||
except RuntimeError as e:
|
||||
msg = str(e)
|
||||
if "already exists" in msg.lower() or "user already exists" in msg.lower():
|
||||
# Stale volume from a prior run — reset the password to the newly-generated one
|
||||
# so the API call below can authenticate. In production CI, teardown_deps removes
|
||||
# volumes so this branch is only hit in re-runs against a stale volume.
|
||||
print(f" gitea dep: {admin_user!r} already exists — resetting password", flush=True)
|
||||
lifecycle.exec_in_app(
|
||||
provider_domain,
|
||||
[
|
||||
"gitea",
|
||||
"admin",
|
||||
"user",
|
||||
"change-password",
|
||||
"--username",
|
||||
admin_user,
|
||||
"--password",
|
||||
admin_password,
|
||||
],
|
||||
timeout=60,
|
||||
)
|
||||
else:
|
||||
raise
|
||||
|
||||
# 2. Create OAuth2 application via gitea API.
|
||||
oauth_app_name = f"drone-{parent_domain[:8]}"
|
||||
redirect_uri = f"https://{parent_domain}/login"
|
||||
status, resp = _gitea_api(
|
||||
provider_domain,
|
||||
"/user/applications/oauth2",
|
||||
method="POST",
|
||||
body={
|
||||
"name": oauth_app_name,
|
||||
"redirect_uris": [redirect_uri],
|
||||
"confidential_client": True,
|
||||
},
|
||||
username=admin_user,
|
||||
password=admin_password,
|
||||
)
|
||||
if status not in (201, 200):
|
||||
raise RuntimeError(f"gitea OAuth2 app create failed: HTTP {status} — {resp!r}")
|
||||
client_id = resp["client_id"]
|
||||
client_secret = resp["client_secret"]
|
||||
print(
|
||||
f" gitea dep: OAuth2 app {oauth_app_name!r} created (client_id={client_id})",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
return {
|
||||
"admin_user": admin_user,
|
||||
"admin_password": admin_password,
|
||||
"client_id": client_id,
|
||||
"client_secret": client_secret,
|
||||
}
|
||||
|
||||
@ -88,6 +88,38 @@ def sso_dep_unverified(declared, deps_ready: bool, requires_deps_skipped: int) -
|
||||
return bool(declared) and not deps_ready and requires_deps_skipped > 0
|
||||
|
||||
|
||||
def upgrade_base(stages, meta, recipe: str) -> str | None:
|
||||
"""Deploy-once base version decision (pure given meta + the published-version lookup):
|
||||
previous published version when the upgrade tier will run and one exists (so upgrade goes
|
||||
previous→target in place), else None (the caller falls back to the target / PR head).
|
||||
(DECISIONS.)
|
||||
|
||||
A recipe may override the base via recipe_meta UPGRADE_BASE_VERSION when the harness default
|
||||
(recipe_versions[-2]) is NOT the PR's true predecessor — e.g. a PR that adds a version ABOVE the
|
||||
newest published tag, where the correct base is [-1] (the newest published), not [-2]. The
|
||||
override must be an exact published version tag (deployed as a pinned base). (Adversary §7.1.)
|
||||
|
||||
A recipe that declares the upgrade rung in EXPECTED_NA gets NO base: published versions may
|
||||
exist yet be genuinely undeployable — e.g. bluesky-pds, where every published tag pins the
|
||||
moving image tag `:0.4` that upstream republished with incompatible main builds, so no
|
||||
published version can come up as an upgrade base (phase bsky, DECISIONS). Deploying one would
|
||||
fail the INSTALL tier before the PR-head code is ever exercised. With no base, the single
|
||||
deploy is the PR head itself and the upgrade tier records "skip", which derive_rungs
|
||||
classifies as the DECLARED intentional skip (reason from EXPECTED_NA — visible in
|
||||
results.json `skips.intentional`, never reported as a pass)."""
|
||||
if "upgrade" not in stages:
|
||||
return None
|
||||
if "upgrade" in (meta.EXPECTED_NA or {}):
|
||||
print(
|
||||
"== upgrade tier: declared EXPECTED_NA['upgrade'] — no upgrade base will be "
|
||||
f"deployed; the single deploy is the target/PR head. Reason: "
|
||||
f"{(meta.EXPECTED_NA or {}).get('upgrade')}",
|
||||
flush=True,
|
||||
)
|
||||
return None
|
||||
return meta.UPGRADE_BASE_VERSION or lifecycle.previous_version(recipe)
|
||||
|
||||
|
||||
def _truthy(v: str | None) -> bool:
|
||||
return str(v or "").strip().lower() in ("1", "true", "yes", "on")
|
||||
|
||||
@ -442,8 +474,9 @@ def _enrich_deps_with_sso(parent_recipe: str, parent_domain: str, deps_list) ->
|
||||
setup function, then return a recipe→entry dict carrying domain + admin + realm/client/user
|
||||
info — the shape the `install_steps.sh` hook (and dependent tests) read.
|
||||
|
||||
Provider routing: today only `keycloak` is supported. authentik will need a parallel
|
||||
`setup_authentik_realm` when an authentik-dep recipe enrolls (DEFERRED.md #9).
|
||||
Provider routing: keycloak (OIDC realm/client) and gitea (OAuth2 app for drone) are
|
||||
supported. authentik will need a parallel `setup_authentik_realm` when an authentik-dep
|
||||
recipe enrolls (DEFERRED.md #9).
|
||||
"""
|
||||
from harness import sso, warm # local import — sso may not be needed for dep-less runs
|
||||
|
||||
@ -453,6 +486,19 @@ def _enrich_deps_with_sso(parent_recipe: str, parent_domain: str, deps_list) ->
|
||||
dep_domain = entry.get("domain")
|
||||
if not dep_recipe or not dep_domain:
|
||||
continue
|
||||
if dep_recipe == "gitea":
|
||||
# Gitea dep provider (phase drone): create admin user + OAuth2 app so the
|
||||
# dependent recipe's install_steps.sh can wire DRONE_GITEA_* before deploy.
|
||||
creds = sso.setup_gitea_oauth(dep_domain, parent_domain)
|
||||
out[dep_recipe] = {
|
||||
"recipe": dep_recipe,
|
||||
"domain": dep_domain,
|
||||
"admin_user": creds["admin_user"],
|
||||
"admin_password": creds["admin_password"],
|
||||
"client_id": creds["client_id"],
|
||||
"client_secret": creds["client_secret"],
|
||||
}
|
||||
continue
|
||||
if dep_recipe != "keycloak":
|
||||
# Provider not yet supported — record bare entry; install_steps.sh / tests will
|
||||
# raise if they need realm/client info they don't see.
|
||||
@ -905,16 +951,7 @@ def main() -> int:
|
||||
|
||||
domain = naming.app_domain(recipe, os.environ.get("PR", "0"), ref)
|
||||
|
||||
# Deploy-once base version: previous published version when the upgrade tier will run and one
|
||||
# exists (so upgrade goes previous→target in place), else the target (current/$REF). (DECISIONS.)
|
||||
# A recipe may override the base via recipe_meta UPGRADE_BASE_VERSION when the harness default
|
||||
# (recipe_versions[-2]) is NOT the PR's true predecessor — e.g. a PR that adds a version ABOVE the
|
||||
# newest published tag, where the correct base is [-1] (the newest published), not [-2]. The
|
||||
# override must be an exact published version tag (deployed as a pinned base). (Adversary §7.1.)
|
||||
want_upgrade = "upgrade" in stages
|
||||
prev = (
|
||||
(meta.UPGRADE_BASE_VERSION or lifecycle.previous_version(recipe)) if want_upgrade else None
|
||||
)
|
||||
prev = upgrade_base(stages, meta, recipe)
|
||||
base = prev or target
|
||||
backup_cap = generic.backup_capable(recipe, meta)
|
||||
hook = discovery.install_steps(recipe, repo_local)
|
||||
@ -1091,7 +1128,7 @@ def main() -> int:
|
||||
junit_dir=junit_dir,
|
||||
)
|
||||
if prev
|
||||
else "skip" # only one published version → nothing to upgrade from
|
||||
else "skip" # no upgrade base: single published version, or declared EXPECTED_NA
|
||||
)
|
||||
# ---- BACKUP + RESTORE tiers (backup-capable only; else clean N/A) ----
|
||||
if "backup" in stages:
|
||||
@ -1184,6 +1221,21 @@ def main() -> int:
|
||||
except lifecycle.TeardownError as e:
|
||||
dep_teardown_error = str(e)
|
||||
print(f"!! {dep_teardown_error}", flush=True)
|
||||
else:
|
||||
# ADV-drone-02 fix: deps_state is empty (enrichment failed after a successful
|
||||
# deploy_deps call). The raw deployed list is still in $CCCI_DEPS_FILE — read it
|
||||
# and tear down any cold deps so they don't orphan at their deterministic domain.
|
||||
raw = deps_mod.load_run_state()
|
||||
if raw:
|
||||
cold_raw = [
|
||||
e
|
||||
for e in (raw if isinstance(raw, list) else list(raw.values()))
|
||||
if isinstance(e, dict) and not e.get("warm")
|
||||
]
|
||||
if cold_raw:
|
||||
print("\n===== DEPS teardown (enrichment-failure fallback) =====", flush=True)
|
||||
with contextlib.suppress(lifecycle.TeardownError):
|
||||
deps_mod.teardown_deps(cold_raw)
|
||||
|
||||
# ---- deploy-count assertion (DG4.1) ----
|
||||
with open(countfile) as f:
|
||||
@ -1274,7 +1326,7 @@ def main() -> int:
|
||||
records=records,
|
||||
results=results,
|
||||
backup_capable=backup_cap,
|
||||
has_upgrade_target=prev is not None, # structural: a previous published version exists
|
||||
has_upgrade_target=prev is not None, # structural: a deployable upgrade base exists
|
||||
lint=lint_result, # L5 rung (phase lvl5)
|
||||
clean_teardown=clean_teardown,
|
||||
no_secret_leak=True, # narrowed below by an actual scan of the serialised artifact
|
||||
|
||||
19
terraform/.gitignore
vendored
Normal file
19
terraform/.gitignore
vendored
Normal file
@ -0,0 +1,19 @@
|
||||
# Terraform state — may contain secrets; NEVER commit
|
||||
*.tfstate
|
||||
*.tfstate.*
|
||||
*.tfstate.backup
|
||||
|
||||
# Variable files with secret values — NEVER commit
|
||||
*.auto.tfvars
|
||||
*.auto.tfvars.json
|
||||
terraform.tfvars
|
||||
|
||||
# Terraform working directory (downloaded providers, modules)
|
||||
.terraform/
|
||||
|
||||
# Crash logs
|
||||
crash.log
|
||||
crash.*.log
|
||||
|
||||
# NOTE: .terraform.lock.hcl (provider lock file) IS committed — it pins provider SHAs
|
||||
# for reproducibility, analogous to flake.lock.
|
||||
23
terraform/.terraform.lock.hcl
generated
Normal file
23
terraform/.terraform.lock.hcl
generated
Normal file
@ -0,0 +1,23 @@
|
||||
# This file is maintained automatically by "tofu init".
|
||||
# Manual edits may be lost in future updates.
|
||||
|
||||
provider "registry.opentofu.org/hetznercloud/hcloud" {
|
||||
version = "1.64.0"
|
||||
constraints = "1.64.0"
|
||||
hashes = [
|
||||
"h1:FUkTfFrWlmv0JhsbjQvTk3zY7A2Q0LuoSs0PKEzaLpk=",
|
||||
"zh:5bf7f8f429b1a8f485988d199f46295676a6cdf7d84ad11f1f4613faecfa89d5",
|
||||
"zh:63b3d182474dd5afd0d5ab3f5f66228b752504436bcb2f4721bd6f1233d0f2ae",
|
||||
"zh:6867da2d89d297b6760d80dde373e74df511bea72f7daccf6a944a9de4b4d4ed",
|
||||
"zh:766fdcea1b03038a92414eafaa430b9ac0c57b36ce4c1573e6e291431659d528",
|
||||
"zh:7f3186dfcae4028eac4f2c9c2c382b49c1fad0b63d0471b50748ee6817fbd8d2",
|
||||
"zh:bb8a33b6ff9a4d3bce87628c49b08a4780e2c034762f40112058d96f5a4e52bd",
|
||||
"zh:cc93751c7c90a37f180cf3e5439ed34f3154e60de5920a13d153d93954938239",
|
||||
"zh:d6e2abf05a0eb8fe0544eb099960a4962db61532e7757016ccacbf0b83bcd1ae",
|
||||
"zh:da9e3adedd8d33623aac4929fa8b1210f98d2931d5737c201da0dda992dd25ab",
|
||||
"zh:dffc931aec4d7b0733690e115b1aabdf5c157b7d347a09a9d149ee6b7e9d8ce3",
|
||||
"zh:e565dea4f28182099a271f794e3b781f069ea54976f5f05dbb79a1c2b6627459",
|
||||
"zh:e79411287af28ccf6187bd418b7ea2ee217e642026392ddc8027bf3e3287fb80",
|
||||
"zh:f5102d7141a04c193dffbb5cbc3f7e3588c41b87e11877d2e20d57ea5ef64123",
|
||||
]
|
||||
}
|
||||
100
terraform/README.md
Normal file
100
terraform/README.md
Normal file
@ -0,0 +1,100 @@
|
||||
# cc-ci Hetzner Cloud Terraform
|
||||
|
||||
Provisions the cc-ci NixOS server on Hetzner Cloud (cpx32, 4 vCPU / 8 GB, x86 AMD, nbg1).
|
||||
Stage 1 (Terraform): creates the server, runs nixos-infect to convert Debian 12 → NixOS.
|
||||
Stage 2 (manual): clone the flake + apply the cc-ci config.
|
||||
|
||||
## Prerequisites (Class-A1 inputs — provide at apply time, NEVER commit)
|
||||
|
||||
| Input | How to provide |
|
||||
|---|---|
|
||||
| `HCLOUD_TOKEN` | `export HCLOUD_TOKEN=<token>` in shell before `tofu apply` |
|
||||
| SSH key pair | Generate once: `ssh-keygen -t ed25519 -f ~/.ssh/cc-ci-hetzner`; pass pubkey via `TF_VAR_ssh_public_key="$(cat ~/.ssh/cc-ci-hetzner.pub)"` |
|
||||
| Bootstrap age key | Provision to `/var/lib/sops-nix/key.txt` on the server (Stage 2; see `docs/install.md`) |
|
||||
|
||||
## Stage 1 — Provision server + nixos-infect
|
||||
|
||||
```bash
|
||||
cd terraform/
|
||||
|
||||
# Provide secrets via environment
|
||||
export HCLOUD_TOKEN=<your-token>
|
||||
export TF_VAR_ssh_public_key="$(cat ~/.ssh/cc-ci-hetzner.pub)"
|
||||
|
||||
# Download providers (uses .terraform.lock.hcl — pinned, reproducible)
|
||||
tofu init # or: terraform init
|
||||
|
||||
# Preview
|
||||
tofu plan
|
||||
|
||||
# Apply — creates cpx31 server in nbg1, runs nixos-infect on first boot
|
||||
tofu apply
|
||||
|
||||
# Note the output IP:
|
||||
# server_ipv4 = "x.x.x.x"
|
||||
# ssh_connect = "ssh root@x.x.x.x"
|
||||
```
|
||||
|
||||
nixos-infect runs on first boot and **reboots the server** into NixOS (~5 min total).
|
||||
Wait for the reboot to complete, then verify:
|
||||
|
||||
```bash
|
||||
# Check NixOS is up:
|
||||
ssh root@<ip> 'nixos-version'
|
||||
|
||||
# Inspect infect log if needed:
|
||||
ssh root@<ip> 'cat /var/log/nixos-infect.log'
|
||||
```
|
||||
|
||||
After the reboot the server runs bare NixOS (infect-generated config). Proceed to Stage 2.
|
||||
|
||||
## Stage 2 — Apply the cc-ci flake config
|
||||
|
||||
Follows the D8 install flow documented in `docs/install.md` exactly:
|
||||
|
||||
```bash
|
||||
# On the Hetzner server (ssh root@<ip>):
|
||||
|
||||
# 1. Clone the flake (--recursive brings cc-ci-secrets submodule)
|
||||
git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci.git /etc/cc-ci
|
||||
cd /etc/cc-ci
|
||||
|
||||
# 2. Provision the bootstrap age key (the one irreducible out-of-band secret)
|
||||
mkdir -p /var/lib/sops-nix
|
||||
install -m 0600 /dev/stdin /var/lib/sops-nix/key.txt <<'EOF'
|
||||
<paste bootstrap age private key here — see docs/install.md>
|
||||
EOF
|
||||
|
||||
# 3. Apply the cc-ci Hetzner host config
|
||||
nixos-rebuild switch --flake .#cc-ci-hetzner
|
||||
|
||||
# 4. Verify (all units green, reconcile oneshots converged)
|
||||
systemctl --failed
|
||||
```
|
||||
|
||||
## Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `server_type` | `cpx31` | x86 only. `cpx31`=AMD 4vCPU/8GB, `cx33`=Intel 4vCPU/8GB. Never `cax*` (ARM). |
|
||||
| `location` | `nbg1` | Hetzner datacenter. |
|
||||
| `image` | `debian-12` | Base image; nixos-infect converts it to NixOS. debian-12 preferred. |
|
||||
| `server_name` | `cc-ci` | Hetzner server name. |
|
||||
| `ssh_public_key` | (required) | Public key registered for root access. |
|
||||
|
||||
Override via env: `TF_VAR_location=hel1 tofu apply`.
|
||||
|
||||
## Teardown (throwaway verification run)
|
||||
|
||||
```bash
|
||||
tofu destroy # removes server + SSH key; billing stops immediately
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- `.terraform.lock.hcl` is committed (pins provider SHAs — analogous to flake.lock).
|
||||
- `*.tfstate`, `*.tfvars`, `.terraform/` are gitignored — never commit state or secrets.
|
||||
- `cpx31` is retired in some Hetzner DCs; `cpx32` (equivalent AMD, 4 vCPU / 8 GB) is the default.
|
||||
`cx33` (Intel, same spec) is also available. Both are x86_64 — compatible with the `x86_64-linux` flake.
|
||||
- The Hetzner server has a public IPv4 — future: point `*.ci.commoninternet.net` A record directly
|
||||
at it and drop the gateway/MagicDNS path (see plan §6 + `DECISIONS.md`).
|
||||
32
terraform/main.tf
Normal file
32
terraform/main.tf
Normal file
@ -0,0 +1,32 @@
|
||||
resource "hcloud_ssh_key" "cc_ci" {
|
||||
name = "cc-ci-deploy"
|
||||
public_key = var.ssh_public_key
|
||||
|
||||
labels = {
|
||||
project = "cc-ci"
|
||||
managed = "terraform"
|
||||
}
|
||||
}
|
||||
|
||||
resource "hcloud_server" "cc_ci" {
|
||||
name = var.server_name
|
||||
server_type = var.server_type
|
||||
image = var.image
|
||||
location = var.location
|
||||
ssh_keys = [hcloud_ssh_key.cc_ci.id]
|
||||
|
||||
# Stage 1: cloud-init runs nixos-infect on first boot, converting Ubuntu to NixOS,
|
||||
# then reboots. See user-data.sh for the pinned infect revision.
|
||||
user_data = file("${path.module}/user-data.sh")
|
||||
|
||||
public_net {
|
||||
ipv4_enabled = true
|
||||
ipv6_enabled = false
|
||||
}
|
||||
|
||||
labels = {
|
||||
project = "cc-ci"
|
||||
managed = "terraform"
|
||||
stage = "infect"
|
||||
}
|
||||
}
|
||||
19
terraform/outputs.tf
Normal file
19
terraform/outputs.tf
Normal file
@ -0,0 +1,19 @@
|
||||
output "server_ipv4" {
|
||||
description = "Public IPv4 address of the cc-ci Hetzner server"
|
||||
value = hcloud_server.cc_ci.ipv4_address
|
||||
}
|
||||
|
||||
output "server_id" {
|
||||
description = "Hetzner internal server ID"
|
||||
value = hcloud_server.cc_ci.id
|
||||
}
|
||||
|
||||
output "ssh_connect" {
|
||||
description = "SSH command to connect as root"
|
||||
value = "ssh root@${hcloud_server.cc_ci.ipv4_address}"
|
||||
}
|
||||
|
||||
output "nixos_infect_log" {
|
||||
description = "Path on the server where nixos-infect logs are written"
|
||||
value = "ssh root@${hcloud_server.cc_ci.ipv4_address} 'cat /var/log/nixos-infect.log'"
|
||||
}
|
||||
25
terraform/user-data.sh
Normal file
25
terraform/user-data.sh
Normal file
@ -0,0 +1,25 @@
|
||||
#!/usr/bin/env bash
|
||||
# Stage 1 — convert Debian 12 → NixOS via nixos-infect (pinned revision).
|
||||
#
|
||||
# nixos-infect generates /etc/nixos/{configuration.nix,hardware-configuration.nix,networking.nix}
|
||||
# with Hetzner-correct bootloader (GRUB, not systemd-boot) and networking, then reboots into NixOS.
|
||||
#
|
||||
# After the reboot:
|
||||
# - SSH as root is available (key registered with Hetzner survives infect)
|
||||
# - Run Stage 2 per terraform/README.md: clone cc-ci + cc-ci-secrets, provision the bootstrap
|
||||
# age key, then `nixos-rebuild switch --flake .#cc-ci-hetzner`
|
||||
#
|
||||
# Logs are written to /var/log/nixos-infect.log on the server for post-mortem inspection.
|
||||
# The server reboots automatically at the end of infect — wait ~5 min before sshing in.
|
||||
set -euo pipefail
|
||||
|
||||
# Pinned nixos-infect revision (2026-03-22: "fixes errors for non efi systems").
|
||||
# Update deliberately; verify Hetzner still supported before bumping.
|
||||
INFECT_SHA="40f62a680bb0e8f2f607d79abfaaecd99d59401c"
|
||||
|
||||
export NIX_CHANNEL="nixos-24.11"
|
||||
export PROVIDER="hetzner" # tells nixos-infect to use GRUB + Hetzner networking
|
||||
export NIXOS_IMPORT="" # no extra imports at infect time; we apply the real flake in Stage 2
|
||||
|
||||
curl -fsSL "https://raw.githubusercontent.com/elitak/nixos-infect/${INFECT_SHA}/nixos-infect" |
|
||||
bash -x 2>&1 | tee /var/log/nixos-infect.log
|
||||
37
terraform/variables.tf
Normal file
37
terraform/variables.tf
Normal file
@ -0,0 +1,37 @@
|
||||
variable "location" {
|
||||
description = "Hetzner datacenter (nbg1=Nuremberg, fsn1=Falkenstein, hel1=Helsinki, ash=Ashburn, hil=Hillsboro)"
|
||||
type = string
|
||||
default = "nbg1"
|
||||
}
|
||||
|
||||
variable "server_type" {
|
||||
description = <<-EOT
|
||||
Hetzner server type. Must be x86 — the flake is x86_64-linux; NEVER use cax* (ARM).
|
||||
cpx32 = AMD 4 vCPU / 8 GB (default; replaces cpx31 which is retired in some DCs).
|
||||
cx33 = Intel 4 vCPU / 8 GB (alternative).
|
||||
EOT
|
||||
type = string
|
||||
default = "cpx32"
|
||||
|
||||
validation {
|
||||
condition = !startswith(var.server_type, "cax")
|
||||
error_message = "ARM server types (cax*) are not supported — the cc-ci flake is x86_64-linux only."
|
||||
}
|
||||
}
|
||||
|
||||
variable "image" {
|
||||
description = "Base OS image. nixos-infect supports debian-12 and ubuntu-24.04. debian-12 preferred."
|
||||
type = string
|
||||
default = "debian-12"
|
||||
}
|
||||
|
||||
variable "ssh_public_key" {
|
||||
description = "SSH public key content (the full line, e.g. 'ssh-ed25519 AAAA... comment'). Registered with Hetzner for root access post-infect. Pass via TF_VAR_ssh_public_key or terraform.tfvars (gitignored)."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "server_name" {
|
||||
description = "Hetzner server name and initial NixOS hostname"
|
||||
type = string
|
||||
default = "cc-ci"
|
||||
}
|
||||
14
terraform/versions.tf
Normal file
14
terraform/versions.tf
Normal file
@ -0,0 +1,14 @@
|
||||
terraform {
|
||||
required_version = ">= 1.0"
|
||||
required_providers {
|
||||
hcloud = {
|
||||
source = "hetznercloud/hcloud"
|
||||
version = "1.64.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# The hcloud provider reads HCLOUD_TOKEN from the environment automatically.
|
||||
# Never put the token value in any .tf file or .tfvars — keep it in the shell
|
||||
# environment (export HCLOUD_TOKEN=...) or pass via TF_VAR_hcloud_token.
|
||||
provider "hcloud" {}
|
||||
@ -4,8 +4,8 @@ Phase-2 P2 mapping table.
|
||||
|
||||
| recipe-maintainer file | cc-ci file | what's verified | status |
|
||||
|---|---|---|---|
|
||||
| (no health_check.py in the recipe-maintainer corpus) | `tests/bluesky-pds/functional/test_health_check.py` | GETs `/xrpc/_health` (the PDS health endpoint); asserts 200 + JSON with `version` field. Phase-2 health_check aligned with the parity-port convention. | **Phase-2 health_check** |
|
||||
| `recipe-info/bluesky-pds/tests/goat_account.py` | `tests/bluesky-pds/functional/test_account_and_post.py` | Original: `goat pds describe`, list/cleanup, account create, verify listed, delete, verify gone. cc-ci port preserves the account-lifecycle assertions + adds an **atproto post round-trip** (createSession→createRecord→getRecord, asserts post text round-trips) — the §4.3 prescribed test ("create a test account (goat CLI), create a post via atproto, fetch it back, delete the account"). F2-8 closed. | **ported** |
|
||||
| (no health_check.py in the recipe-maintainer corpus) | `tests/bluesky-pds/custom/test_health_check.py` | GETs `/xrpc/_health` (the PDS health endpoint); asserts 200 + JSON with `version` field. Phase-2 health_check aligned with the parity-port convention. | **Phase-2 health_check** |
|
||||
| `recipe-info/bluesky-pds/tests/goat_account.py` | `tests/bluesky-pds/custom/test_account_and_post.py` | Original: `goat pds describe`, list/cleanup, account create, verify listed, delete, verify gone. cc-ci port preserves the account-lifecycle assertions + adds an **atproto post round-trip** (createSession→createRecord→getRecord, asserts post text round-trips) — the §4.3 prescribed test ("create a test account (goat CLI), create a post via atproto, fetch it back, delete the account"). F2-8 closed. | **ported** |
|
||||
|
||||
## Recipe-specific tests (Phase-2 P3, ≥2 beyond parity)
|
||||
|
||||
@ -14,8 +14,8 @@ public XRPC API + the well-known `atproto-did` server identifier. Two new functi
|
||||
|
||||
| cc-ci file | what's verified | rationale |
|
||||
|---|---|---|
|
||||
| `tests/bluesky-pds/functional/test_describe_server.py` | GETs `/xrpc/com.atproto.server.describeServer` (the public atproto endpoint that advertises the PDS's available account creation policy); asserts 200 + JSON envelope with at least `availableUserDomains` (array; the PDS's hosting domains) or `inviteCodeRequired` (bool). | Proves the atproto XRPC API is alive AND the PDS-specific configuration is being served (not just a generic 200). Non-vacuous: a PDS that boots but can't serve its server description is broken. |
|
||||
| `tests/bluesky-pds/functional/test_session_auth.py` | GETs `/xrpc/com.atproto.server.getSession` (no auth); asserts **401** + a JSON XRPC error envelope with an `error` field. | Proves the PDS's atproto auth contract is enforced. Non-vacuous: 200 = anonymous leak (security bug); 404 = route missing; 5xx = backend broken — only 401 + a proper XRPC error envelope indicates a correctly-wired PDS. (An earlier draft tried `/.well-known/atproto-did` but that endpoint is only published when the bare DOMAIN is registered as a server-DID, which the recipe doesn't auto-configure.) |
|
||||
| `tests/bluesky-pds/custom/test_describe_server.py` | GETs `/xrpc/com.atproto.server.describeServer` (the public atproto endpoint that advertises the PDS's available account creation policy); asserts 200 + JSON envelope with at least `availableUserDomains` (array; the PDS's hosting domains) or `inviteCodeRequired` (bool). | Proves the atproto XRPC API is alive AND the PDS-specific configuration is being served (not just a generic 200). Non-vacuous: a PDS that boots but can't serve its server description is broken. |
|
||||
| `tests/bluesky-pds/custom/test_session_auth.py` | GETs `/xrpc/com.atproto.server.getSession` (no auth); asserts **401** + a JSON XRPC error envelope with an `error` field. | Proves the PDS's atproto auth contract is enforced. Non-vacuous: 200 = anonymous leak (security bug); 404 = route missing; 5xx = backend broken — only 401 + a proper XRPC error envelope indicates a correctly-wired PDS. (An earlier draft tried `/.well-known/atproto-did` but that endpoint is only published when the bare DOMAIN is registered as a server-DID, which the recipe doesn't auto-configure.) |
|
||||
|
||||
Two specific tests + parity health_check = ≥2 floor met. Backup data-integrity is N/A unless the
|
||||
recipe declares `backupbot.backup=true` labels (Phase-1d auto-detect handles the skip).
|
||||
|
||||
@ -6,3 +6,17 @@ HEALTH_PATH = "/xrpc/_health" # PDS health endpoint; returns {"version": ...} o
|
||||
HEALTH_OK = (200,)
|
||||
DEPLOY_TIMEOUT = 600
|
||||
HTTP_TIMEOUT = 600
|
||||
|
||||
# UPGRADE rung: published versions exist (0.1.1+v0.4, 0.2.0+v0.4) but BOTH pin the moving image
|
||||
# tag ghcr.io/bluesky-social/pds:0.4, which upstream republished with main-branch builds
|
||||
# (@atproto/pds 0.5.1, Node 24, /app/index.ts — no index.js), so NO published version can deploy
|
||||
# as an upgrade base anymore: the base crash-loops MODULE_NOT_FOUND before the PR head is ever
|
||||
# exercised (phase bsky root cause; cc-ci-plan/upstream/bluesky-pds.md). Declared intentional
|
||||
# until a fixed exact-pinned version (0.3.0+v0.4.219, mirror PR #2) is merged AND published —
|
||||
# then DROP this and set UPGRADE_BASE_VERSION = "0.3.0+v0.4.219" so the upgrade rung is
|
||||
# exercised again from the first deployable base.
|
||||
EXPECTED_NA = {
|
||||
"upgrade": "no deployable upgrade base: every published version pins the moving tag "
|
||||
"pds:0.4, which upstream republished with incompatible main builds (index.js removed) — "
|
||||
"re-enable via UPGRADE_BASE_VERSION once a fixed version is published post-merge",
|
||||
}
|
||||
|
||||
@ -5,7 +5,7 @@ Phase-2 P2 mapping table. The Adversary cold-verifies parity by reading the sour
|
||||
|
||||
| recipe-maintainer file | cc-ci file | what's verified | status |
|
||||
|---|---|---|---|
|
||||
| `recipe-info/cryptpad/tests/health_check.py` | `tests/cryptpad/functional/test_health_check.py` | HTTP 200 from the served root. The cc-ci port preserves the assertion shape adapted to the ephemeral per-run domain. | **ported** |
|
||||
| `recipe-info/cryptpad/tests/health_check.py` | `tests/cryptpad/custom/test_health_check.py` | HTTP 200 from the served root. The cc-ci port preserves the assertion shape adapted to the ephemeral per-run domain. | **ported** |
|
||||
| `recipe-info/cryptpad/tests/oidc_login.py` | (Q3.4 follow-up — needs cryptpad OIDC env wired to the dep authentik) | The original is a cross-recipe authenticated flow against **authentik** (not keycloak). The cc-ci port requires: (1) Q2.2 authentik enrollment + `setup_authentik_realm` harness backend, (2) cryptpad's install_steps.sh wiring the dep authentik's client_secret + OIDC env. Both are tracked Q5 catch-up items. | **deferred** |
|
||||
|
||||
## Recipe-specific tests (Phase-2 P3, ≥2 beyond parity)
|
||||
@ -17,9 +17,9 @@ object + read-it-back" test (plan §4.3 floor) MUST use a real browser (per plan
|
||||
|
||||
| cc-ci file | what's verified | rationale |
|
||||
|---|---|---|
|
||||
| `tests/cryptpad/playwright/test_pad_content_roundtrip.py` | **§4.3 create-an-object + read-it-back (resolves F2-9).** Opens `/pad/` → CryptPad auto-creates a fragment-keyed pad (`#/2/pad/edit/<key>/`); types a unique marker into the CKEditor rich-text body (nested sandbox iframe `…/pad/ckeditor-inner.html`); waits for the encrypted update to sync ("Saved"); then opens a **brand-new browser context** (no shared localStorage/cookies) and navigates to the captured pad URL; asserts the marker is present in the re-decrypted body. | Phase 2 P3/§4.3 floor — proves genuine **end-to-end-encrypted persistence**: the fresh session carries only the URL (incl. its fragment key), so a successful read-back means the content was persisted server-side as ciphertext and correctly decrypted by a new client. Not a health/SPA stand-in. Mapped empirically against CryptPad 2026.2.0 (editor in a deep nested frame; ~15s cold-cache LESS-compile init; transient `net::ERR_NETWORK_CHANGED` handled by the shared `goto_with_retry` + a mid-load reload retry). |
|
||||
| `tests/cryptpad/playwright/test_pad_create.py` | Browses to `/`. Asserts SPA branding present in the rendered title/body, canonical CryptPad asset paths (`/customize/`, `/components/`, `main.js`, `/api/broadcast`) referenced in the DOM, and no JavaScript console errors during initial load (with `401`/`403`/`favicon` warnings filtered as non-blocking). | Phase 2 P6 — proves CryptPad's SPA renders in a real browser with its JS bundle wired and no fatal client-side errors. (Complements the roundtrip test above; was the "maximal subset" while create-and-read-back was deferred — now superseded by the full roundtrip, kept as a fast SPA-liveness check.) |
|
||||
| `tests/cryptpad/functional/test_spa_assets.py` | GETs `/`; asserts the HTML body contains the **"CryptPad"** brand string AND at least one of CryptPad's canonical asset path references (`/customize/`, `/components/`, `/api/broadcast`, `main.js`). | Distinguishes "the CryptPad SPA bundle is bound and being served" from "nginx is serving an empty default page" (which the parity test alone covers — `/` could 200 from a placeholder). Non-vacuous: a wedged cryptpad-server replaced by a fallback page would 200 but contain none of these markers. |
|
||||
| `tests/cryptpad/custom/test_pad_content_roundtrip.py` | **§4.3 create-an-object + read-it-back (resolves F2-9).** Opens `/pad/` → CryptPad auto-creates a fragment-keyed pad (`#/2/pad/edit/<key>/`); types a unique marker into the CKEditor rich-text body (nested sandbox iframe `…/pad/ckeditor-inner.html`); waits for the encrypted update to sync ("Saved"); then opens a **brand-new browser context** (no shared localStorage/cookies) and navigates to the captured pad URL; asserts the marker is present in the re-decrypted body. | Phase 2 P3/§4.3 floor — proves genuine **end-to-end-encrypted persistence**: the fresh session carries only the URL (incl. its fragment key), so a successful read-back means the content was persisted server-side as ciphertext and correctly decrypted by a new client. Not a health/SPA stand-in. Mapped empirically against CryptPad 2026.2.0 (editor in a deep nested frame; ~15s cold-cache LESS-compile init; transient `net::ERR_NETWORK_CHANGED` handled by the shared `goto_with_retry` + a mid-load reload retry). |
|
||||
| `tests/cryptpad/custom/test_pad_create.py` | Browses to `/`. Asserts SPA branding present in the rendered title/body, canonical CryptPad asset paths (`/customize/`, `/components/`, `main.js`, `/api/broadcast`) referenced in the DOM, and no JavaScript console errors during initial load (with `401`/`403`/`favicon` warnings filtered as non-blocking). | Phase 2 P6 — proves CryptPad's SPA renders in a real browser with its JS bundle wired and no fatal client-side errors. (Complements the roundtrip test above; was the "maximal subset" while create-and-read-back was deferred — now superseded by the full roundtrip, kept as a fast SPA-liveness check.) |
|
||||
| `tests/cryptpad/custom/test_spa_assets.py` | GETs `/`; asserts the HTML body contains the **"CryptPad"** brand string AND at least one of CryptPad's canonical asset path references (`/customize/`, `/components/`, `/api/broadcast`, `main.js`). | Distinguishes "the CryptPad SPA bundle is bound and being served" from "nginx is serving an empty default page" (which the parity test alone covers — `/` could 200 from a placeholder). Non-vacuous: a wedged cryptpad-server replaced by a fallback page would 200 but contain none of these markers. |
|
||||
|
||||
Two specific tests — the ≥2 floor is met. Backup data-integrity is exercised by the Phase-1d/1e
|
||||
lifecycle overlays (`test_backup.py`/`test_restore.py` + `ops.py` — see those files for the
|
||||
@ -27,7 +27,7 @@ marker mechanism + the restore-asserts-pre-mutation pattern).
|
||||
|
||||
## Playwright (P6)
|
||||
|
||||
`tests/cryptpad/playwright/test_pad_create.py` (above) is the canonical browser flow — covers P6
|
||||
`tests/cryptpad/custom/test_pad_create.py` (above) is the canonical browser flow — covers P6
|
||||
in full.
|
||||
|
||||
## Non-ports
|
||||
|
||||
@ -1,13 +1,13 @@
|
||||
# Parity — custom-html
|
||||
|
||||
Phase-2 P2 mapping table: every `references/recipe-maintainer/recipe-info/custom-html/tests/*.py` has
|
||||
a comparable cc-ci test under `tests/custom-html/functional/`, asserting the **same thing** (not just
|
||||
a comparable cc-ci test under `tests/custom-html/custom/`, asserting the **same thing** (not just
|
||||
a renamed file). The Adversary cold-verifies parity by reading the source `recipe-info/<file>` and the
|
||||
cc-ci file side-by-side.
|
||||
|
||||
| recipe-maintainer file | cc-ci file | what's verified | status |
|
||||
|---|---|---|---|
|
||||
| `recipe-info/custom-html/tests/health_check.py` | `tests/custom-html/functional/test_health_check.py` | The app is reachable over HTTPS and returns a successful response (the original asserted HTTP 200 against a persistent instance). The cc-ci port preserves the assertion shape — non-5xx status — and adapts to the ephemeral per-run domain via the `live_app` fixture. | **ported** |
|
||||
| `recipe-info/custom-html/tests/health_check.py` | `tests/custom-html/custom/test_health_check.py` | The app is reachable over HTTPS and returns a successful response (the original asserted HTTP 200 against a persistent instance). The cc-ci port preserves the assertion shape — non-5xx status — and adapts to the ephemeral per-run domain via the `live_app` fixture. | **ported** |
|
||||
|
||||
## Recipe-specific tests (Phase-2 P3, ≥2 beyond parity)
|
||||
|
||||
@ -17,8 +17,8 @@ content, fetch it back"). Two new functional tests beyond parity:
|
||||
|
||||
| cc-ci file | what's verified | rationale |
|
||||
|---|---|---|
|
||||
| `tests/custom-html/functional/test_content_roundtrip.py` | Writes a uniquely-marked content file to the served volume via `lifecycle.exec_in_app` and asserts an HTTPS GET to the corresponding path returns that exact byte content — proves the app serves files written into its served volume, not a static synthetic page. | The recipe IS a content-server: a roundtrip is the canonical proof it works for what it's for. |
|
||||
| `tests/custom-html/functional/test_content_type_header.py` | Writes both an `.html` and a `.txt` marker to the served volume, fetches each, and asserts `Content-Type` reflects the file type (`text/html`, `text/plain`) — proves nginx is properly serving with MIME-typed responses, not just returning bytes. | Distinctive nginx-served behavior — distinguishes a working nginx from a misconfigured one that emits everything as `application/octet-stream`. |
|
||||
| `tests/custom-html/custom/test_content_roundtrip.py` | Writes a uniquely-marked content file to the served volume via `lifecycle.exec_in_app` and asserts an HTTPS GET to the corresponding path returns that exact byte content — proves the app serves files written into its served volume, not a static synthetic page. | The recipe IS a content-server: a roundtrip is the canonical proof it works for what it's for. |
|
||||
| `tests/custom-html/custom/test_content_type_header.py` | Writes both an `.html` and a `.txt` marker to the served volume, fetches each, and asserts `Content-Type` reflects the file type (`text/html`, `text/plain`) — proves nginx is properly serving with MIME-typed responses, not just returning bytes. | Distinctive nginx-served behavior — distinguishes a working nginx from a misconfigured one that emits everything as `application/octet-stream`. |
|
||||
|
||||
Both tests run in the **custom** stage against the same `live_app` shared deployment as the
|
||||
lifecycle overlays — no extra deploy, no extra teardown.
|
||||
@ -32,7 +32,7 @@ via `lifecycle.exec_in_app` (volume-direct, immune to the post-backup serving ra
|
||||
|
||||
## Playwright (P6)
|
||||
|
||||
`tests/custom-html/playwright/test_browser_smoke.py` covers the browser-rendered nginx HTML — already
|
||||
`tests/custom-html/custom/test_browser_smoke.py` covers the browser-rendered nginx HTML — already
|
||||
exercised inline by `tests/custom-html/test_install.py::test_serving_and_content` (lifecycle install
|
||||
overlay), which uses Playwright Chromium to confirm the page renders. The Phase-2 split file is the
|
||||
canonical home for browser-flow coverage and is invoked by the **custom** stage.
|
||||
|
||||
@ -1,7 +1,7 @@
|
||||
"""custom-html — Playwright UI flow (Phase 2 P6).
|
||||
|
||||
The recipe-maintainer corpus did not ship a Playwright test for custom-html — but plan §4.1 names
|
||||
`playwright/` as the canonical home for browser flows where a recipe's core UX is a UI. custom-html
|
||||
The recipe-maintainer corpus did not ship a Playwright test for custom-html — but the cfold layout
|
||||
uses `custom/` as the canonical home for browser flows where a recipe's core UX is a UI. custom-html
|
||||
serves HTML; a browser-rendered fetch (vs raw HTTP) proves the page actually renders and any client-
|
||||
side resources resolve. Distinct from `tests/custom-html/test_install.py` which runs Playwright as
|
||||
part of the lifecycle INSTALL overlay; this file is the standalone Phase-2 custom-stage version, so a
|
||||
@ -19,9 +19,9 @@ Defining behaviors exercised against the live per-run deploy:
|
||||
|
||||
| cc-ci file | what's verified | rationale |
|
||||
|---|---|---|
|
||||
| `functional/test_create_topic.py::test_create_topic_roundtrip` | Bootstraps an admin + API key via Rails in the `app` container (`_discourse.mint_admin`), POSTs `/posts.json` to create a NEW topic with a unique marker in title + body, then GETs `/t/<topic_id>.json` and asserts the title (Discourse `title_prettify`-aware) **and** the unique body marker round-tripped in the first post's `cooked`. | §4.3 "create the app's primary object — a topic — and read it back". Non-vacuous: the marker is unique per run, so a stale/echoed response can't pass; a wedged DB/Rails/posting path fails here even though `/srv/status` returns 200. |
|
||||
| `functional/test_site_basic.py::test_site_json_has_discourse_config` | GETs `/site.json` and asserts a Discourse-specific config structure (e.g. a `categories` list), not a bare 200. | Proves Rails is serving its real site config JSON (a distinctive Discourse structure), distinguishing "the forum backend is up + emitting its API" from "a static/error page at /". |
|
||||
| `functional/test_health_check.py::test_discourse_srv_status_ok` | GETs `/srv/status` and asserts the Discourse readiness signal (Rails serving). | Baseline readiness (parity-aligned health check). |
|
||||
| `custom/test_create_topic.py::test_create_topic_roundtrip` | Bootstraps an admin + API key via Rails in the `app` container (`_discourse.mint_admin`), POSTs `/posts.json` to create a NEW topic with a unique marker in title + body, then GETs `/t/<topic_id>.json` and asserts the title (Discourse `title_prettify`-aware) **and** the unique body marker round-tripped in the first post's `cooked`. | §4.3 "create the app's primary object — a topic — and read it back". Non-vacuous: the marker is unique per run, so a stale/echoed response can't pass; a wedged DB/Rails/posting path fails here even though `/srv/status` returns 200. |
|
||||
| `custom/test_site_basic.py::test_site_json_has_discourse_config` | GETs `/site.json` and asserts a Discourse-specific config structure (e.g. a `categories` list), not a bare 200. | Proves Rails is serving its real site config JSON (a distinctive Discourse structure), distinguishing "the forum backend is up + emitting its API" from "a static/error page at /". |
|
||||
| `custom/test_health_check.py::test_discourse_srv_status_ok` | GETs `/srv/status` and asserts the Discourse readiness signal (Rails serving). | Baseline readiness (parity-aligned health check). |
|
||||
|
||||
Two recipe-specific functional tests (create-topic round-trip + site.json config) + the health check
|
||||
= the ≥2 floor met, with a real create-an-object + read-it-back as the characteristic-behavior test.
|
||||
|
||||
@ -28,10 +28,30 @@ version: "3.8"
|
||||
# bad `discourse` key. Instead the 2.4GB `bitnamilegacy/discourse:3.3.1` image is kept warm in the node
|
||||
# image cache, so the inline pull during deploy is a no-op and convergence isn't pull-bound. (swarm
|
||||
# ignores depends_on, so the dangling ref has zero runtime effect — a recipe lint nit, not a defect.)
|
||||
#
|
||||
# 3. UPGRADE ROLLOUT (dstamp 2026-06-11, direct-evidence attribution in JOURNAL-dstamp): the
|
||||
# published app service sets `deploy.update_config: { failure_action: rollback, order:
|
||||
# start-first }`. On the upgrade chaos redeploy (base 0.7.0 → PR head), start-first runs the OLD
|
||||
# and NEW precompile/Rails-heavy discourse tasks CO-RESIDENT (~2x memory); under host memory
|
||||
# pressure the NEW task intermittently OOMs/fails swarm's update monitor → `failure_action:
|
||||
# rollback` reverts the app service to its PREVIOUS spec, INCLUDING the
|
||||
# `coop-cloud.<stack>.chaos-version` label (head → base). Because start-first keeps the OLD task
|
||||
# serving, wait_healthy still passes, and HC1 then reads the reverted BASE commit (eb96de9+U) and
|
||||
# misreports it as 'the re-checkout failed' — the dstamp drift, reproduced solo (runs
|
||||
# dstamp-repro1/4) with `.Spec.chaos-version=7ae7b0f7+U` (head applied) flipping to
|
||||
# `.PreviousSpec=eb96de94+U` after the rollback. FIX: `order: stop-first` so the NEW task boots
|
||||
# with the full host memory (no 2x co-residency) and genuinely becomes healthy → no spurious
|
||||
# rollback. This is a CI deploy-rollout tweak only: the upgrade still really deploys + asserts the
|
||||
# PR-head code under test, and `failure_action: rollback` is LEFT intact, so a genuinely broken
|
||||
# head still rolls back and is caught (lifecycle.assert_upgrade_converged) — NO test is weakened.
|
||||
# Trade-off: brief real downtime during the CI upgrade (covered by DEPLOY_TIMEOUT 3600).
|
||||
services:
|
||||
app:
|
||||
image: bitnamilegacy/discourse:3.3.1
|
||||
healthcheck:
|
||||
start_period: 20m
|
||||
deploy:
|
||||
update_config:
|
||||
order: stop-first
|
||||
sidekiq:
|
||||
image: bitnamilegacy/discourse:3.3.1
|
||||
|
||||
44
tests/drone/PARITY.md
Normal file
44
tests/drone/PARITY.md
Normal file
@ -0,0 +1,44 @@
|
||||
# PARITY — drone
|
||||
|
||||
Tracks which lifecycle rungs are covered and why any are skipped.
|
||||
|
||||
## Tiers
|
||||
|
||||
| Tier | Status | Notes |
|
||||
|------|--------|-------|
|
||||
| Install | COVERED | Fresh deploy with gitea dep pre-provisioned |
|
||||
| Upgrade | COVERED | 1.8.0+2.25.0 → 1.9.0+2.26.0 (two published versions; viable) |
|
||||
| Backup/Restore | STRUCTURAL SKIP | See below |
|
||||
| Functional | COVERED | SCM-configured test (gitea OAuth redirect) |
|
||||
| Lint | COVERED | `abra recipe lint` (L5 target) |
|
||||
| Screenshot | COVERED | Drone login/landing page |
|
||||
|
||||
## Backup rung — structural skip
|
||||
|
||||
**Justification:** The drone recipe declares no backupbot labels in `compose.yml` and ships
|
||||
no `abra_backup*` functions in `abra.sh` (which only exports `DRONE_ENV_VERSION=v2`).
|
||||
Therefore `backup_capable=False` is auto-detected by the harness — the backup rung is an
|
||||
intentional structural skip, not a gap.
|
||||
|
||||
**Evidence:**
|
||||
```
|
||||
# compose.yml — no backupbot.* labels anywhere
|
||||
grep -i backupbot ~/.abra/recipes/drone/compose.yml # → (no output)
|
||||
|
||||
# abra.sh — no backup functions
|
||||
cat ~/.abra/recipes/drone/abra.sh
|
||||
# → export DRONE_ENV_VERSION=v2
|
||||
# (no abra_backup / abra_restore functions)
|
||||
```
|
||||
|
||||
**Level impact:** With backup_capable=False (structural skip), the backup rung is an
|
||||
EXPECTED_NA-class intentional skip. The recipe can still reach L5 if all other rungs pass,
|
||||
because the backup rung's skip is declared-and-justified, not a surprise omission.
|
||||
|
||||
**Path to L5:** install + upgrade + functional + lint + screenshot all PASS.
|
||||
|
||||
## Gitea dep teardown
|
||||
|
||||
The gitea dep is co-deployed per run. Both gitea AND drone are torn down in the
|
||||
orchestrator's `finally` block (deps in reverse order: drone first, then gitea). A drone
|
||||
test failure mid-run still triggers the `finally` — the teardown guarantee is sacred.
|
||||
0
tests/drone/custom/__init__.py
Normal file
0
tests/drone/custom/__init__.py
Normal file
103
tests/drone/custom/test_scm_configured.py
Normal file
103
tests/drone/custom/test_scm_configured.py
Normal file
@ -0,0 +1,103 @@
|
||||
"""drone — SCM-configured functional test (phase drone).
|
||||
|
||||
Proves that drone is wired to the per-run gitea dep, not just healthy.
|
||||
The negative control: a drone deployed WITHOUT DRONE_GITEA_CLIENT_ID + DRONE_GITEA_SERVER
|
||||
(i.e., without compose.gitea.yml) would NOT redirect /login to the gitea dep's OAuth
|
||||
authorize endpoint — it would error or redirect elsewhere. This test is therefore falsified
|
||||
by a misconfigured drone.
|
||||
|
||||
Test: GET https://<drone>/login must issue a 303 redirect whose Location header points to
|
||||
the per-run gitea dep's /login/oauth/authorize URL. We capture ONLY drone's first redirect
|
||||
(not gitea's subsequent redirect to /user/login for unauthenticated users).
|
||||
|
||||
Per ADV-drone-01: following all redirects causes the assertion to land on gitea's /user/login
|
||||
(200 OK after gitea redirects unauthenticated users away from /login/oauth/authorize), which
|
||||
means the path assertion always fails. The fix is a no-follow handler that captures the
|
||||
Location header from drone's 303 directly.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ssl
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
class _CaptureOneRedirect(urllib.request.HTTPRedirectHandler):
|
||||
"""Stop redirect-following after the FIRST hop; raise HTTPError so the caller can inspect
|
||||
the Location header from drone's 303 without following gitea's subsequent redirects."""
|
||||
|
||||
def http_error_302(self, req, fp, code, msg, headers):
|
||||
raise urllib.error.HTTPError(req.full_url, code, msg, headers, fp)
|
||||
|
||||
http_error_303 = http_error_302
|
||||
|
||||
|
||||
@pytest.mark.requires_deps
|
||||
def test_login_redirects_to_gitea_dep(live_app, deps):
|
||||
"""Drone's /login must issue a 303 redirect to the per-run gitea dep's OAuth2 authorize
|
||||
endpoint.
|
||||
|
||||
Proves: (a) gitea is the SCM backend (not github or unconfigured); (b) the OAuth2
|
||||
client_id in the Location header matches the app the harness created in the dep gitea
|
||||
instance; (c) the redirect targets the TEST-RUN gitea, not any hardcoded external provider.
|
||||
|
||||
ADV-drone-01 fix: only drone's first 303 is captured; gitea's own redirects (unauthenticated
|
||||
user → /user/login) are not followed, so the path assertion is against the correct URL.
|
||||
"""
|
||||
assert "gitea" in deps, (
|
||||
f"gitea dep not in deps — dep provisioning should have populated this. "
|
||||
f"Got keys: {list(deps.keys())}"
|
||||
)
|
||||
gitea = deps["gitea"]
|
||||
gitea_domain: str = gitea["domain"]
|
||||
expected_client_id: str = gitea["client_id"]
|
||||
|
||||
ctx = ssl.create_default_context()
|
||||
ctx.check_hostname = False
|
||||
ctx.verify_mode = ssl.CERT_NONE
|
||||
|
||||
opener = urllib.request.build_opener(
|
||||
_CaptureOneRedirect(),
|
||||
urllib.request.HTTPSHandler(context=ctx),
|
||||
)
|
||||
|
||||
redirect_url = None
|
||||
try:
|
||||
opener.open(f"https://{live_app}/login", timeout=30)
|
||||
pytest.fail(
|
||||
f"Expected a 302/303 redirect from https://{live_app}/login but got 200 OK — "
|
||||
f"drone may not have gitea SCM configured (check COMPOSE_FILE + GITEA_DOMAIN)"
|
||||
)
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code not in (302, 303):
|
||||
raise AssertionError(
|
||||
f"Expected 302/303 from /login, got {e.code} — "
|
||||
f"drone may not have gitea SCM configured"
|
||||
) from e
|
||||
redirect_url = e.headers.get("Location") or e.headers.get("location", "")
|
||||
|
||||
assert redirect_url, (
|
||||
"Drone /login returned a redirect but Location header is empty — "
|
||||
"check drone gitea SCM configuration"
|
||||
)
|
||||
|
||||
parsed = urllib.parse.urlparse(redirect_url)
|
||||
assert parsed.scheme == "https", f"Redirect Location has unexpected scheme: {redirect_url!r}"
|
||||
assert parsed.netloc == gitea_domain, (
|
||||
f"Drone /login did not redirect to the gitea dep ({gitea_domain!r}); "
|
||||
f"Location: {redirect_url!r} — check GITEA_DOMAIN + COMPOSE_FILE in drone's .env"
|
||||
)
|
||||
assert parsed.path == "/login/oauth/authorize", (
|
||||
f"Redirect path is {parsed.path!r}, expected /login/oauth/authorize — "
|
||||
f"drone may not have gitea SCM configured"
|
||||
)
|
||||
params = urllib.parse.parse_qs(parsed.query)
|
||||
actual_client_id = params.get("client_id", [None])[0]
|
||||
assert actual_client_id == expected_client_id, (
|
||||
f"OAuth2 client_id mismatch: drone is using {actual_client_id!r} but the harness "
|
||||
f"created app {expected_client_id!r} in the dep gitea — check install_steps.sh"
|
||||
)
|
||||
74
tests/drone/install_steps.sh
Executable file
74
tests/drone/install_steps.sh
Executable file
@ -0,0 +1,74 @@
|
||||
#!/usr/bin/env bash
|
||||
# drone — INSTALL-TIME gitea SCM wiring hook (rcust P2b).
|
||||
#
|
||||
# Runs AFTER `abra app new` + EXTRA_ENV + `abra app secret generate`, BEFORE `abra app deploy`.
|
||||
# Reads the gitea dep creds from $CCCI_DEPS_FILE (written by the orchestrator's dep provisioning
|
||||
# step), then:
|
||||
# 1. Switches drone to gitea SCM mode (COMPOSE_FILE includes compose.gitea.yml).
|
||||
# 2. Sets GITEA_CLIENT_ID + GITEA_DOMAIN in drone's .env.
|
||||
# 3. Sets CLIENT_SECRET_VERSION and inserts the gitea OAuth2 client_secret as a swarm secret.
|
||||
# 4. Sets DRONE_USER_CREATE so the gitea ci_admin becomes drone's first admin on login.
|
||||
#
|
||||
# If the deps file is absent or has no gitea entry, drone is still deployed (without SCM wiring);
|
||||
# the custom/test_scm_configured.py test then FAILS, which is the correct signal.
|
||||
#
|
||||
# Env supplied by the harness:
|
||||
# CCCI_APP_DOMAIN — the per-run drone app domain
|
||||
# CCCI_APP_ENV — path to the app's .env
|
||||
# CCCI_DEPS_FILE — JSON {gitea: {domain, admin_user, admin_password, client_id, client_secret}}
|
||||
set -euo pipefail
|
||||
|
||||
: "${CCCI_APP_DOMAIN:?missing}"
|
||||
ENV_PATH="${CCCI_APP_ENV:?missing}"
|
||||
|
||||
if [ -z "${CCCI_DEPS_FILE:-}" ] || [ ! -s "${CCCI_DEPS_FILE}" ]; then
|
||||
echo " drone install_steps: no deps file — deploying drone WITHOUT gitea SCM wiring"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
GITEA_DOMAIN=$(jq -r '.gitea.domain // empty' "$CCCI_DEPS_FILE")
|
||||
GITEA_CLIENT_ID=$(jq -r '.gitea.client_id // empty' "$CCCI_DEPS_FILE")
|
||||
GITEA_SECRET=$(jq -r '.gitea.client_secret // empty' "$CCCI_DEPS_FILE")
|
||||
GITEA_ADMIN=$(jq -r '.gitea.admin_user // empty' "$CCCI_DEPS_FILE")
|
||||
|
||||
if [ -z "$GITEA_DOMAIN" ] || [ -z "$GITEA_CLIENT_ID" ] || [ -z "$GITEA_SECRET" ]; then
|
||||
echo " drone install_steps: deps file missing gitea domain/client_id/secret — no SCM wiring"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo " drone install_steps: wiring gitea SCM (domain=${GITEA_DOMAIN}, client_id=${GITEA_CLIENT_ID})"
|
||||
|
||||
# Helper: write or replace a key=value line in the drone .env file.
|
||||
write_env() {
|
||||
local key="$1" val="$2"
|
||||
sed -i "/^\s*#\?\s*${key}=/d" "$ENV_PATH"
|
||||
[ -z "$(tail -c1 "$ENV_PATH" 2>/dev/null)" ] || printf '\n' >>"$ENV_PATH"
|
||||
printf '%s=%s\n' "$key" "$val" >>"$ENV_PATH"
|
||||
}
|
||||
|
||||
# 1. Switch COMPOSE_FILE to include the gitea overlay (activates DRONE_GITEA_CLIENT_ID +
|
||||
# DRONE_GITEA_SERVER env and the client_secret swarm secret).
|
||||
write_env COMPOSE_FILE "compose.yml:compose.gitea.yml"
|
||||
|
||||
# 2. Wire gitea identity into drone's .env.
|
||||
write_env GITEA_CLIENT_ID "$GITEA_CLIENT_ID"
|
||||
write_env GITEA_DOMAIN "$GITEA_DOMAIN"
|
||||
|
||||
# 3. Insert the gitea OAuth2 client_secret as a swarm secret at version v1.
|
||||
# The secret does not exist yet (abra secret generate only creates secrets declared in the
|
||||
# active COMPOSE_FILE; we just switched to compose.gitea.yml which adds client_secret).
|
||||
write_env CLIENT_SECRET_VERSION "v1"
|
||||
INSERT_LOG=$(abra app secret insert "$CCCI_APP_DOMAIN" client_secret v1 "$GITEA_SECRET" --no-input -C -o 2>&1) ||
|
||||
INSERT_LOG=$(script -qec "abra app secret insert $CCCI_APP_DOMAIN client_secret v1 $GITEA_SECRET --no-input -C -o" /dev/null 2>&1) ||
|
||||
{
|
||||
echo " drone install_steps: abra app secret insert client_secret@v1 failed: $INSERT_LOG"
|
||||
exit 1
|
||||
}
|
||||
echo " drone install_steps: client_secret inserted at v1"
|
||||
|
||||
# 4. DRONE_USER_CREATE: when ci_admin first logs in via gitea OAuth, drone promotes them to admin.
|
||||
# Uses the gitea admin username from the dep provisioning step.
|
||||
ADMIN_USER="${GITEA_ADMIN:-ci_admin}"
|
||||
write_env DRONE_USER_CREATE "username:${ADMIN_USER},admin:true"
|
||||
|
||||
echo " drone install_steps: gitea SCM wired (DRONE_USER_CREATE=username:${ADMIN_USER},admin:true)"
|
||||
16
tests/drone/recipe_meta.py
Normal file
16
tests/drone/recipe_meta.py
Normal file
@ -0,0 +1,16 @@
|
||||
# Per-recipe harness config for drone (CI server with gitea SCM dependency).
|
||||
# Drone requires a gitea SCM backend to boot; the harness provisions gitea as an install-time
|
||||
# dep, creates an admin user + OAuth2 app in it, and wires DRONE_GITEA_* via install_steps.sh
|
||||
# before the single drone deploy. Upgrade tier: viable (1.8.0 → 1.9.0).
|
||||
#
|
||||
# The backup rung is a structural skip: the drone recipe ships no backupbot labels and abra.sh
|
||||
# exports only DRONE_ENV_VERSION (no backup functions). Documented in PARITY.md.
|
||||
HEALTH_PATH = "/healthz"
|
||||
HEALTH_OK = (200,)
|
||||
DEPLOY_TIMEOUT = 600
|
||||
HTTP_TIMEOUT = 600
|
||||
|
||||
# Gitea is deployed as an install-time dep. The orchestrator provisions it before drone, runs
|
||||
# install_steps.sh (which wires GITEA_CLIENT_ID + GITEA_DOMAIN + client_secret into drone's env
|
||||
# and compose), then deploys drone once with SCM already configured.
|
||||
DEPS = ["gitea"]
|
||||
@ -11,15 +11,15 @@ and a JSON Content/Admin API at `/ghost/api/*`. Defining behaviors exercised:
|
||||
|
||||
| cc-ci file | what's verified | rationale |
|
||||
|---|---|---|
|
||||
| `tests/ghost/functional/test_content_api.py` | GETs `/ghost/api/content/settings/`; asserts 200 with `{"settings": {...}}` envelope OR 401/403 with a Ghost error envelope. | Distinguishes "the ghost-server JS process is up + emitting its API" from "a static themed page is served at /." A wedged Ghost backend → 5xx; misrouted nginx → 404. |
|
||||
| `tests/ghost/functional/test_admin_redirect.py` | GETs `/ghost/`; asserts 200 or 302 + Ghost branding/SPA references in the response (or a redirect to /ghost/#/setup on fresh deploy). | Proves the admin route is wired through the nginx proxy. Distinguishes "admin SPA bound" from "404 (route missing)" or "5xx (broken)." |
|
||||
| `tests/ghost/custom/test_content_api.py` | GETs `/ghost/api/content/settings/`; asserts 200 with `{"settings": {...}}` envelope OR 401/403 with a Ghost error envelope. | Distinguishes "the ghost-server JS process is up + emitting its API" from "a static themed page is served at /." A wedged Ghost backend → 5xx; misrouted nginx → 404. |
|
||||
| `tests/ghost/custom/test_admin_redirect.py` | GETs `/ghost/`; asserts 200 or 302 + Ghost branding/SPA references in the response (or a redirect to /ghost/#/setup on fresh deploy). | Proves the admin route is wired through the nginx proxy. Distinguishes "admin SPA bound" from "404 (route missing)" or "5xx (broken)." |
|
||||
|
||||
Two specific tests + parity health_check = ≥2 floor met.
|
||||
|
||||
## Plan §4.3 prescribed deeper test — AUTHORED (closes DEFERRED ghost create-post)
|
||||
|
||||
§4.3 named "create-a-post round-trip" for ghost. Implemented in
|
||||
`tests/ghost/functional/test_post_roundtrip.py` (helper `functional/_ghost.py`):
|
||||
`tests/ghost/custom/test_post_roundtrip.py` (helper `custom/_ghost.py`):
|
||||
1. Wait for the Admin API healthcheck (`GET /ghost/api/admin/site/` → 200).
|
||||
2. Setup the Ghost owner (POST `/ghost/api/admin/authentication/setup/`, fresh deploy) + establish
|
||||
an admin **session cookie** (POST `/ghost/api/admin/session/`) — cookie-aware stdlib opener,
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user