F1e-1 (Adversary): exec_in_app silently returned '' on a failed docker exec, flipping a healthy
recipe RED under opt-out (post-backup container cycle, no readiness buffer). Now polls (re-resolve
container + re-exec) until rc==0 or 90s, then RAISES — never masks an exec failure as empty data.
No assertion weakened. Verified: opt-out install,backup,restore on custom-html now PASS.
HC1: head_ref = ref or recipe_head_commit (prefer explicit PR head sha $REF — robust, no git race;
production !testme always sets REF). assert_upgraded, when head_ref known, REQUIRES the deployed
chaos-version commit to MATCH head_ref (direct + non-vacuous proof the PR-head code was deployed; a
stale prev-checkout chaos redeploy fails). Falls back to version/image/chaos move check otherwise.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
default run: every tier ran generic+overlay (op once, deploy-count=1); CCCI_SKIP_GENERIC=1 run:
generic skipped, overlays only. Clean teardown both. E0/HC2 recorded as Adversary PASS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- orchestrator: per mutating tier, run optional pre-op seed hook (ops.py pre_<op>) → perform the op
ONCE (harness-owned) → run generic assertion (unless opted out) AND overlay assertion, both against
the shared post-op deployment. Op results passed op→assertion via run-scoped CCCI_OP_STATE_FILE.
- opt-out: CCCI_SKIP_GENERIC / CCCI_SKIP_GENERIC_<OP> / recipe_meta.SKIP_GENERIC (declarative).
- generic.py: split do_* into op primitives (perform_upgrade/backup/restore) + assertions
(assert_upgraded/backup_artifact/restore_healthy) reading op_state(); deployed_identity now returns
{version,image,chaos} (chaos label ready for HC1).
- generic test_<op>.py + all 6 recipe overlays migrated to assertion-only; pre-op seeding moved to
per-recipe ops.py (pre_upgrade/pre_backup/pre_restore). install overlays unchanged (no op).
- deploy-count stays 1 (op primitives never call deploy_app). lint PASS; 8 unit tests PASS on cc-ci.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
hedgedoc mirrored to recipe-maintainers/hedgedoc with probe PR #1; add it to the bridge poll list so
!testme triggers the full generic suite (no cc-ci/repo-local overlay -> pure generic). Rebuild pending.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mechanical port to the assertion-only contract (no softened/skipped assertions): install uses
live_app + generic.assert_serving (extend) + the recipe's http/playwright/api checks; upgrade seeds
its data marker then generic.do_upgrade + asserts survival; backup/restore split into test_backup.py
(seed->do_backup->mutate) + new test_restore.py (do_restore->assert original). Recipe-specifics
preserved verbatim (keycloak realm+admin-console+kc_admin, matrix/lasuite db-service psql markers,
cryptpad/n8n volume markers). No recipe now double-deploys under the deploy-once orchestrator.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
tests/custom-html-tiny/install_steps.sh seeds content into the volume pre-deploy. Proof: install
FAILS without the hook (404, graceful-generic), PASSES with it. Same run shows backup/restore=skip
(custom-html-tiny non-backup-capable) — DG3 N/A-skip. deploy-count=1. recipe_meta shortens timeouts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
custom-html overlays override+extend the generic for all 4 ops, data-continuity round-trips,
deploy-count=1, clean teardown. Discovery precedence unit tests 5/5. hedgedoc generic lifecycle
green with genuine 1.10.7->1.10.8 upgrade (move-assertion).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
http_fetch raced the serving layer right after backup-bot cycled the app container (served '' for a
moment). Backup/restore preserve the VOLUME, so read the marker in-container via exec_in_app — correct
and race-free. Serving is proven separately by install/upgrade assert_serving.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
abra app backup create (backup-bot-two) stops/cycles the app container, so a mutate exec_in_app
right after backup hit an empty docker ps and raised. _app_container now polls (no bare sleep) for
the container to reappear within a timeout. Recipe-agnostic harness robustness.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- deploy_app: checkout the pinned tag + deploy NON-chaos when a version is pinned (chaos only for
version=None / PR-head). Was always -C, which ignored the pin and deployed LATEST -> upgrade no-op.
- do_upgrade: assert the deployment actually MOVED (coop-cloud version label and/or image changed)
via lifecycle.deployed_identity -> a vacuous no-op upgrade can no longer pass (DG2).
- G2: migrate custom-html overlays to the assertion-only contract (override + extend-by-composition
+ data-continuity; split backup/restore). tests/unit/test_discovery.py proves precedence (5/5).
Probe (Adversary's F1d-2 test): hedgedoc deploy-prev=1.10.7 -> upgrade=1.10.8, CHANGED=True.
hedgedoc full generic lifecycle green (install/upgrade/backup/restore, deploy-count=1).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- backup artifact: read snapshot_id from 'abra app backup create' output (snapshots needs a TTY);
generic.parse_snapshot_id + do_backup assert it
- restore serving race: lifecycle.http_fetch (one request -> status+body, never raises) +
assert_serving is now a bounded poll (settles a post-op reconverge, no bare sleep); drop wait_serving
- F1d-1 (Adversary, low): reframe served_cert/assert_serving honestly as an INFRA TLS sanity check
(catches a lapsed/mis-rotated wildcard cert), NOT app-vs-fallback (Traefik serves the wildcard
zone-wide); the genuine serving proof is services_converged + non-404 status. Awaiting re-test.
DG1 Adversary PASS @ef44d46. G1 full-lifecycle re-verification in flight.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
git mv STATUS*/BACKLOG*/JOURNAL*/DECISIONS.md -> machine-docs/. README.md kept at root (operator
decision). Updated in-repo refs: README (status line + lint section + Loop-state section) and
docs/install.md -> machine-docs/...
Safe to move now: launch.sh already has resolve_state() (prefers machine-docs/ else root) used by
every STATUS/REVIEW read, and the running watchdog (pid 133191) was restarted AFTER that update, so
it is location-agnostic. scripts/lint.sh -> lint: PASS post-move. Adversary moves its own REVIEW*.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
flake.nix/flake.lock STAY at root so the build ref #cc-ci is unchanged; only flake's internal
configuration.nix path updated. Root-relative refs inside moved modules re-based ../X -> ../../X
(secrets/bridge/dashboard); configuration.nix's ../../modules imports unchanged (both dirs under nix/).
Living docs (README, architecture/install/secrets/enroll) + .drone.yml comment updated to nix/...;
append-only history logs left as-is. DECISIONS.md records RL5 + the deferred-coordinated RL6.
Verified on cc-ci: nixos-rebuild build 'path:#cc-ci' -> toplevel 8i3jcad9 (BYTE-IDENTICAL to the
pre-move build — store derivations are content-addressed on file contents, module .nix not in the
runtime closure); scripts/lint.sh -> lint: PASS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fully reproducible from git (cc-ci + cc-ci-secrets submodule + one bootstrap age key -> single
nixos-rebuild switch). D8 honest (static + live throwaway rebuild). Caught+fixed the abra-init race
and the non-deterministic Drone bot token en route.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Clean-room finding caught by the e2e: DRONE_USER_CREATE had no token: => a fresh-DB rebuild's Drone
auto-generates a random bot token, so the committed (sops) bridge_drone_token gets 401 and the bridge
can't trigger builds. The original cc-ci only matched because its token was captured out-of-band. Now
the bot's machine token == bridge_drone_token deterministically on every rebuild. (Evolves the toplevel
again; re-establish byte-identical on cc-ci after the e2e + Adversary re-verifies C1.)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Found during the e2e: when the bot already granted the shared Drone OAuth app, Gitea 302s straight to
the code callback (no consent form), so the consent-form parse yielded empty _csrf/state and set -e
aborted. Now: if authorize returns a Location, use it directly; else POST the consent form.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
On a FRESH host the reconcile oneshots ran abra concurrently against an uninitialised ~/.abra and
raced on catalogue/recipe init, leaving deploy-proxy/deploy-drone failed after a blank-VM rebuild
(observed on the W4 throwaway). Ordering-only `after` chain serializes them so a single
nixos-rebuild switch converges. Logically correct too (all need the proxy/abra state first).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cc-ci /var/lib/sops-nix/key.txt provisioned = host-derived age key (pub == &host recipient), so
adding keyFile is safe (sops-install-secrets aborts if a configured keyFile is missing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- secrets/ is now the private cc-ci-secrets repo (submodule). defaultSopsFile path unchanged.
- secrets.nix: add wildcard_cert/wildcard_key sops secrets -> path=/var/lib/ci-certs/live/*.
- proxy.nix: cert is sops-from-git, not an operator file drop (reframed; FATAL guard kept as decrypt-path check).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Diagnosed via instrumented diag: lasuite-docs upgrade reported 'FATA deploy failed' while all 9
services converged 1/1 — abra's convergence poll gives up too early on the slow stop-first roll
(pulling new images). Disable abra's check; the harness wait_healthy + data-survival assertion is
the real, more-patient gate (a genuine failure still fails the test: app never gets healthy).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
architecture.md: components, the !testme flow, network/TLS, resource safety, enrollment.
runbook.md: where to look, common failure modes (timeout/rate-limit/auth/skip/health/data), orphan
cleanup, re-trigger, cancel. Completes the D9 doc set (README+install+enroll+secrets+arch+runbook).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
git fetch --tags <url> without a refspec errors 'couldn't find remote ref HEAD'; use
'refs/tags/*:refs/tags/*'. Verified: brings custom-html's 18 upstream version tags into the mirror
PR clone so the upgrade stage finds a previous published version (was skipping).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fetch_recipe (SRC+REF/PR path) now read-only fetches published version tags from the public upstream
into the mirror clone, so the upgrade stage finds a previous published version (mirror PR branches
carry no tags → upgrade would skip). Guardrail-safe: only fetches tags, never pushes to the recipe
repo; plain git so the bot token isn't sent to upstream. Adds the 6 D10 recipes to the bridge
POLL_REPOS so !testme on their PRs triggers runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After triggering a build, the bridge spawns a watcher thread that polls the Drone build to
completion and edits its run-link PR comment to ✅ passed / ❌ <status> (Gitea PATCH
issues/comments/{id}, verified). post_comment now returns the comment id. Also gives the bridge
image a content-hash tag so the swarm service actually rolls on bridge.py changes (was stuck on
:latest). Completes the D7 'PR comment reflects outcome' requirement.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
