keycloak is the always-on shared OIDC dep provider at warm-keycloak.ci..., the SAME stable domain a
data-warm canonical would use → the sweep's promote would collide with the live provider that
lasuite-*/drone depend on. keycloak is kept current by roll_warm_infra (WC1.1) instead.
WARM_CANONICAL=False; exception recorded in DECISIONS. Enrolled set now 20.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
WARM_CANONICAL=True added to every recipe in cc-ci-plan/used-recipes.md (20 weekly +
uptime-kuma external). enrolled_recipes() now returns all 21. Test fixtures
(custom-html-*-bad, concurrency, regression) intentionally left unenrolled.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- should_promote_canonical gains a `tagged` requirement (canon §2.A): a green cold
latest run promotes only when the tested head version is a published release tag;
an untagged main commit never becomes a canonical.
- warm_reconcile.is_released_version(recipe, version): release-tag membership (exact or
by version_key). Caller computes `tagged` so the gate stays pure.
- unit tests: untagged -> no promote; is_released_version cases.
- drive-by (pre-existing reds, unrelated to canon, now green): test_warm_reconcile
traefik assertion was stale vs the phase-pxgate spec (probes /api/version, no
health_domain); meta.py UPGRADE_BASE_VERSION KEYS help synced to the prevb doc text.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
resolve_upgrade_base now reads the head's published version (abra.head_compose_version,
the coop-cloud.<stack>.version label) and, when the last-green warm-canonical version
equals it, steps back to the newest published version strictly older than head instead
of deploying a same-version no-op. warm_reconcile gains version_key + newest_older_version
(single coop-cloud ordering source; sort_versions refactored onto version_key, no behavior
change). Skip only when no older published predecessor exists. Step-back returns kind=version
so it inherits F1d-2 pinned-tag checkout. Extends tests/unit/test_upgrade_base.py (13 pass).
The custom tier runs on the PR head — now genuinely the official discourse/discourse image (prevb
stopped the overlay reverting it to bitnamilegacy). mint_admin hardcoded /opt/bitnami/discourse (404 on
official) → create-topic roundtrip failed. Detect /var/www/discourse, re-export DISCOURSE_DB_PASSWORD
from /run/secrets (entrypoint exports it only for boot), run bin/rails; keep bitnami fallback.
docker stack deploy doesn't prune services the head compose dropped (discourse PR#4 drops sidekiq),
leaving them orphaned on the base image. perform_upgrade now reconciles the live stack to the head
compose service set (lifecycle.prune_orphan_services). Makes the deployed stack faithfully reflect
the head — no test weakened. No-op when service sets match / compose unresolvable.
abra does NOT write STACK_NAME to the app's .env file — it derives it at runtime
by replacing dots with underscores (e.g. gite-e1cb78.ci.commoninternet.net →
gite-e1cb78_ci_commoninternet_net). Build #691 failed with 'STACK_NAME not found'
because the env file read was looking for a key that doesn't exist.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Blocker 4 fix: abra `secret generate --all` uses .env.sample for length hints; the
lfs-plain-gitea PR has SECRET_LFS_JWT_SECRET_VERSION=v1 COMMENTED OUT, so abra produces
a wrong-length secret. gitea requires exactly 43 chars (32 bytes base64 URL-safe); wrong
length → gitea fatals trying to save the JWT secret to the read-only Docker Config
app.ini → health check fails → swarm rolls back.
Fix: new UPGRADE_SECRET_PREP hook (meta.py) called before `abra secret generate --all`
in the upgrade path. abra's `--all` is idempotent (skips existing secrets), so the
correctly pre-inserted secret survives. gitea's recipe_meta.py implements the hook using
`docker secret create` directly to guarantee correct format regardless of .env.sample.
Also consumes machine-docs/BUILDER-INBOX.md (Adversary Blocker 4 digest).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Blocker 1 (LFS roundtrip fails on PR #1):
- Add UPGRADE_EXTRA_ENV to gitea recipe_meta.py — after PR-head checkout
(compose.lfs.yml now in ABRA_DIR), add compose.lfs.yml to COMPOSE_FILE
and set SECRET_LFS_JWT_SECRET_VERSION=v1 so the upgrade chaos redeploy
actually runs with LFS enabled. Without this, the base install checks out
the 3.5.x tag (compose.lfs.yml removed), EXTRA_ENV sees no LFS, and the
upgrade chaos redeploy inherits the no-LFS .env — so the LFS test runs
(compose.lfs.yml is restored by recipe_checkout_ref) but LFS is off.
- Add abra.secret_generate(domain) in generic.perform_upgrade when
upgrade_env is non-empty — generates lfs_jwt_secret before chaos redeploy.
Blocker 2 (REF=main upgrade fails HC1):
- Always use recipe_head_commit (git rev-parse HEAD) for head_ref instead
of using ref directly. When ref="main" (a branch name), the HC1 commit
check "head_ref.startswith(chaos_commit)" always fails since "main" ≠ SHA.
recipe_head_commit returns the actual SHA after the fetch/checkout.
Side-fix (stale creds — build #675):
- ops.py pre_install: delete the per-domain creds file before calling
_ensure_admin. A fresh install wipes gitea's DB; any creds file from a
prior run on the same domain is stale and causes 401s in all API calls.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Gitea 1.22+ (including 1.24.2 on cc-ci) requires explicit scopes
when creating API tokens. Add read:user + read:organization to satisfy
the token creation endpoint and the read-back assertions that follow.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Empty-repo HTTPS push with git clone exits 0 but silently fails (remote
branch creation on an empty clone is unreliable). Fix:
- Create repo with auto_init=True + default_branch=main (initial commit present)
- Clone into a non-existing subdir (git clone must target non-existing path)
- Push via explicit cred_url (bypasses remote config; no tracking needed)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- test_git_push.py + test_lfs_roundtrip.py: use cred_url (https://user:pass@host/...)
instead of GIT_CONFIG_COUNT insteadOf rewriting, which silently failed to
propagate credentials to the push step (repo remained empty after push exit 0).
Also add GIT_SSL_NO_VERIFY=true and GIT_TERMINAL_PROMPT=0.
- test_lfs_roundtrip.py: fix restart health-poll path /api/v1/version → /version
(_api() already prepends /api/v1; double prefix produced 404 and a 120s timeout).
- nix/hosts/cc-ci/configuration.nix: add git-lfs to systemPackages (required for
the LFS capstone test on the lfs-plain-gitea PR branch).
Adversary pre-M1 findings: Issue 1 (git-lfs absent) + Issue 2 (double path) both fixed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_csrf is a hidden field; wait_for_selector defaults to state=visible
and times out. Switch to the visible username input which proves the
login form rendered.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When _enrich_deps_with_sso raises after deploy_deps succeeds (e.g., gitea API
call fails), deps_state stays {} and the finally block's `if deps_state:` guard
skips teardown, orphaning the dep at its deterministic domain.
Fix: add an `else` branch after the `if deps_state:` block that reads
$CCCI_DEPS_FILE (the legacy-list written by deploy_deps) and calls
teardown_deps on the cold entries so no dep is left running.
Unit tests: test_load_run_state_provides_fallback_for_enrichment_failure and
test_fallback_skips_warm_entries verify the data-flow that the fallback relies on.
19/19 unit tests pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
test_scm_configured.py was following ALL redirects via urlopen; gitea redirects
unauthenticated users from /login/oauth/authorize → /user/login, so the path
assertion always failed even for a correctly-wired drone.
Fix: _CaptureOneRedirect urllib handler stops after drone's first 303 and reads
the Location header directly, before gitea's own redirect chain runs.
- Consume BUILDER-INBOX.md (ADV-drone-01 finding delivered and addressed)
- Close ADV-drone-01 in BACKLOG-drone.md
- Update test_gitea_dep.py terminology: "location_url" not "final_url"
- All 10 unit tests pass
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
level.py: RUNGS += lint; statuses {pass,fail,skip,unver}; compute_level = max passed
rung with all below pass-or-skip (fail/unver block); cap_reason/capped DELETED.
harness/lint.py: lint executor — pristine scratch clone of the per-run tree at the
exact tested ref (mirror-origin + untracked-overlay pollution solved by context, no
rule filtered), PTY via script -qec, 60s hard budget, lint.txt artifact, table-parse
classifier (rc only signals FATA), unver on any non-run (never silent pass).
results.py: derive_rungs classifies every N/A source (structural/declared → skip,
else unver), lint rung + synthetic lint stage + lint block in results.json, schema 2,
cap fields removed. run_recipe_ci.py: lint call before tiers (double-wrapped,
verdict-neutral), badge = level only. card/dashboard: 0-5 ramp, cap line → 'level N
of {4|5}', unverified rows, badge number+colour only, lint.txt servable, old schema-1
artifacts render untouched. Unit suite rewritten: 245 passed on cc-ci venv.
The P2b port of setup_custom_tests.sh -> ops.py::pre_install made the 90s bucket-poll timeout a
fatal AssertionError; the original shell hook fell through on timeout BY DESIGN (best-effort) and
the custom-tier MinIO storage test is the real gate for a genuinely missing bucket. Live evidence:
in both M2 sweep failures the bucket landed just after the window and every later tier including
the custom MinIO test passed. Warn loudly + continue, exactly the old semantics.
Adversary-approved fix-forward (REVIEW-rcust 57c66ad, scoped to this raise).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adversary heads-up (inbox 2026-06-10T19:06Z): meta values are repo-public by construction, but
the manifest lands on the dashboard — a field literally named SECRET_KEY_BASE showing a value
(plausible's committed CI dummy) is needless secret-scan noise. Mask values whose key NAME is
secret-shaped (SECRET|PASSWORD|TOKEN|CREDENTIAL|word-segment KEY), top-level and nested dict
keys; the key name stays visible. Unit test pins redacted vs passthrough (KEYCLOAK_URL).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One block at run start answering "what does this recipe customize?" across every surface
(non-default recipe_meta keys, ops.py pre-ops, install_steps.sh, compose.ccci.yml, lifecycle
overlays by source, custom-test counts, active CCCI_SKIP_GENERIC* env overrides — !!-flagged when
riding a CI run, P2c), printed to the run log and embedded verbatim in results.json under
"customization". Pure presentation — building/printing it never influences a verdict; the
manifest honors the HC2 repo-local gate so it never advertises code the run will not execute.
Unit tests: synthetic recipe exercising every surface -> complete + deterministic + JSON-clean;
HC2 invisibility; env-override flagging; render golden lines; build_results threads the dict
verbatim (key always present, None when absent).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Placement RULE: discovery.custom_tests covers ONLY functional/ + playwright/ —
the top-level test_*.py glob for recipe dirs is removed (top level is reserved
for lifecycle overlays; zero in-repo users of top-level custom tests, verified
by sweep). Lifecycle-name exclusion inside the subdirs stays as the double-run
safety net. HC2 default-deny unchanged (repo-local custom now pinned via
functional/ in the gate test).
New conftest fixture op_state: parses $CCCI_OP_STATE_FILE (op context: versions,
artifact paths), skipping with a clear reason when unset/absent/unparseable —
overlay tests read op facts from the fixture instead of hand-parsing env (zero
existing hand-parsers found; the fixture is the documented path forward). deps
fixture landed in P2d.
Unit tests: placement-rule discovery tests (top-level custom NOT discovered;
functional/playwright are; misfiled lifecycle names excluded), op_state fixture
contract (reads file / skips without env / skips on missing file), deps fixture
attribute sugar.
Verified on cc-ci: cc-ci-run -m pytest tests/unit -q -> 184 passed; scripts/lint.sh -> PASS.
harness.meta.HookCtx (frozen): .domain, .base_url, .meta (RecipeMeta), .deps
(provisioned dep creds from $CCCI_DEPS_FILE or None), .op (current lifecycle op
or None); built via meta.hook_ctx() at each hook call site.
All recipe callables now take ctx: EXTRA_ENV(ctx), UPGRADE_EXTRA_ENV(ctx),
READY_PROBE(ctx), BACKUP_VERIFY(ctx), SCREENSHOT(page, ctx), ops.py pre_<op>(ctx).
Dict-valued EXTRA_ENV/UPGRADE_EXTRA_ENV unchanged (only the callable signature
moved). Call sites converted: deploy_app env shaping, perform_upgrade,
wait_ready_probes (gains op=), _perform_op BACKUP_VERIFY, screenshot.capture,
_run_pre_hook.
Legacy signatures fail FAST with a clear migration message: the registry carries
hook_params per hook key, enforced at meta.load() (MetaError names the old vs new
signature); ops.py pre-op hooks get the same check at the orchestrator call site
(meta.check_hook_signature) — no silent TypeError mid-run.
Migrated every in-repo user mechanically (17 ops.py files; cryptpad/lasuite-*/
mailu EXTRA_ENV; mumble+lasuite-drive READY_PROBE; ghost/discourse BACKUP_VERIFY)
— seeded values, probes and assertions byte-identical (domain -> ctx.domain;
keycloak pre_restore's meta arg -> ctx.meta).
Unit tests: hook_ctx field contract, ctx.deps from the run deps file, legacy-
signature MetaError (READY_PROBE/EXTRA_ENV/SCREENSHOT + pre-op checker), ctx
signatures accepted. Docs table regenerated (signature docs in key docs).
Verified on cc-ci: cc-ci-run -m pytest tests/unit -q -> 180 passed; scripts/lint.sh -> PASS.
a) compose.ccci.yml is FIRST-CLASS: the harness auto-copies tests/<recipe>/
compose.ccci.yml into the run's recipe checkout (ABRA_DIR-aware, lifecycle.
provide_ccci_overlay) and auto-chaoses the pinned base deploy on its presence
(kills the R7 implicit coupling). ghost/discourse install_steps.sh (copy-only
boilerplate) deleted; CHAOS_BASE_DEPLOY removed from both metas + the registry.
b) install-time deps wiring is the ONLY mode: deps with DEPS provision BEFORE the
single deploy; legacy post-deploy provisioning + the setup_custom_tests.sh
invocation machinery deleted. lasuite-docs migrated to install_steps.sh OIDC
wiring (same env names/values as the old hook — only the timing moved);
lasuite-drive's remaining post-deploy MinIO bucket one-shot moved to ops.py
pre_install; both setup_custom_tests.sh files deleted; OIDC_AT_INSTALL removed
from drive/meet metas + the registry.
c) SKIP_GENERIC meta key deleted (zero users). Env form CCCI_SKIP_GENERIC* stays
as the documented dev-only escape hatch; when active in a drone CI run the
orchestrator prints a loud !! warning (manifest embedding lands in P5).
d) conftest cleanup: dead pre-deploy-once fixtures deployed/deployed_app deleted
(zero users), app_domain + _short + _wait_healthy dropped (only users were the
deleted fixtures); deps_apps+deps_creds consolidated into ONE deps fixture
(entries expose .domain etc. as attributes; dict access intact); the 6 lasuite
test files renamed deps_creds->deps (fixture name only — assertions and flows
byte-identical). requires_deps marker + F2-11 skip-report plumbing unchanged.
Registry is now exactly the 14 final keys; docs §4 table regenerated. Stale
setup_custom_tests/OIDC_AT_INSTALL prose in docstrings/comments/assert MESSAGES
updated (no assert logic or expected value touched).
Verified on cc-ci: cc-ci-run -m pytest tests/unit -q -> 175 passed; scripts/lint.sh -> PASS.
One loader: runner/harness/meta.py::load(recipe) -> RecipeMeta (frozen dataclass,
attribute access), backed by the declarative KEYS registry (14 final keys + 3
P2-deprecated). The ONLY exec() of tests/<recipe>/recipe_meta.py. Validation per
the locked decision: unknown ALL-CAPS top-level name or type mismatch = MetaError
(hard error at load); underscore-prefixed names recipe-private; callables only on
hook-typed keys.
Migrated all six legacy loaders (spec §4 L1–L6):
- run_recipe_ci.py::_load_meta deleted; orchestrator loads once, passes meta down
- tests/conftest.py::_recipe_meta deleted; meta fixture returns full RecipeMeta (R3)
- lifecycle.py::_recipe_extra_env/_recipe_meta_flag deleted; deploy_app takes meta
- deps.py::declared_deps deleted; callers read meta.DEPS
- canonical.py::is_enrolled reads through meta.load()
- screenshot.py now actually receives SCREENSHOT through the orchestrator path (R2
fix; proven by unit test through the real load path)
Mumble private constants underscore-prefixed (_WELCOME_TEXT_MARKER/_MAX_USERS) +
importers fixed. New tests/unit/test_meta.py (all-recipes-load-clean typo gate,
MetaError cases, spec §2 baseline defaults, underscore exemption, doc sync). Docs
§4 key table now GENERATED from the registry (scripts/gen-meta-docs.py); drift
fails CI.
Verified on cc-ci: cc-ci-run -m pytest tests/unit -q -> 175 passed; scripts/lint.sh -> PASS.
The four CCCI state files (deploys countfile, opstate, deps, depskip) were keyed
by app domain in shared /tmp. A second run of the same domain executes its main()
preamble + deploy_app's pre-lock _record_deploy BEFORE blocking at the app lock,
so it reset/polluted the live first run's counter (false DG4.1 deploy-count=2,
build 279) and the first run's end-of-run os.remove crashed the second
(FileNotFoundError, build 281). Masked pre-restructure by the end-to-end recipe
flock. Now keyed by run id + harness pid via _run_state_path(); children receive
exact paths via the CCCI_*_FILE env vars, so domain keying was never load-bearing.
tests/concurrency/test_run_state.py: path-invariant cases + a real-process
regression (helpers.py deploy-count-run) reproducing the live interleaving —
verified to FAIL under simulated shared keying. docs/concurrency.md §3 updated.
