cc-ci/machine-docs/JOURNAL-redfix.md

JOURNAL — phase redfix

2026-06-17T23:20Z — Bootstrap

Read phase plan + plan.md §6.1/§7/§9 + canon DECISIONS exceptions (lines ~1494–1552). Six canon-sweep failures to investigate. Confirmed cc-ci access, no run in flight, sweep timer next fires 2026-06-21 (3-day window), disk 38G free.

Isolation mechanism understood: runner/nightly_sweep.run_on_tag = abra.recipe_checkout(r, tag) + run_recipe_ci.py RECIPE=<r> CCCI_SKIP_FETCH=1 cold/full. I reproduce each failure by running ONE recipe at a time with no concurrent load.

Starting canonical state notable: mumble canonical IS present (1.0.0+v1.6.870-0, written 20260617T180501Z — during today's nixenv sweep). The canon DECISIONS recorded mumble RED (test_handshake_completes_with_channel_presence). A canonical only gets written on a GREEN cold run on latest → mumble flipped green in a recent run. Strong early evidence for the operator's "mumble passed before" → load flake hypothesis. Must confirm with a clean isolation re-run + check whether the canon-sweep red was under concurrent load.

Next: start M1 investigation. Plan order (cheap/informative first): triage the existing sweep logs on cc-ci to pin the EXACT assertion/error for each (mumble, mattermost-lts restore, gitea app.ini, bluesky routing, discourse compose), then run isolation re-runs. discourse's recorded cause is an UPSTREAM compose defect (sidekiq.depends_on: discourse while service is app) that FATAs before any deploy — that's deterministic, not a load timeout, so it may not even need a long isolation run to confirm; verify the compose at the latest tag directly first.

2026-06-17T23:40Z — M1: discourse isolation run — CANON ROOT-CAUSE WAS WRONG

Ran discourse ALONE on cc-ci (recipe_checkout discourse 0.8.1+3.5.0 + RECIPE=discourse CCCI_SKIP_FETCH=1 cc-ci-run runner/run_recipe_ci.py, log /tmp/redfix-discourse.log).

RESULT: install PASS, upgrade FAIL, backup PASS, restore PASS, custom PASS — the recipe deploys, serves (200 /srv/status), backs up and restores cleanly. NOT a deploy timeout, NOT a 51-min wedge, NOT a deploy FATA. The canon DECISIONS root-cause ("abra app deploy FATAs: service sidekiq depends on undefined service discourse → invalid compose project") is misattributed: that string appears ONLY from the non-fatal prepull docker compose config --images (rc=15, harness logs "skipping (deploy will pull as usual)"). The real abra app deploy is a swarm docker stack deploy, which ignores depends_on entirely → the stack converges (UpdateStatus=completed).

The ONLY failure is the cc-ci upgrade OVERLAY tests/discourse/test_upgrade.py:

• test_head_runs_official_image_not_bitnamilegacy — app image is bitnamilegacy/discourse:3.5.0; test demands discourse/discourse:3.5.3 (official).
• test_sidekiq_service_dropped_by_head — services ['app','db','redis','sidekiq']; test demands sidekiq dropped.

These prevb-phase overlay tests are PR-FAITHFULNESS assertions for a specific migration PR (bitnamilegacy → official discourse/discourse:3.5.3, drop sidekiq). Verified that migration exists in NO upstream release tag and NOT in main — git show main:compose.yml and every tag (0.1.0…0.8.1+3.5.0) all use bitnamilegacy/discourse:3.5.0 + sidekiq. So the overlay asserts a state that doesn't exist anywhere upstream → deterministic RED whenever the sweep tests the latest release tag. The head DID deploy (chaos-version label = head f87c612d+U, converged) — the test expectation is simply wrong for the released recipe.

Note (M2 design): migrating discourse from the deprecated bitnamilegacy image to official discourse/discourse is a MAJOR recipe rewrite (different fs layout, entrypoint, no /opt/bitnami sidekiq run.sh) — not a 1-line image swap. So the overlay test's discourse/discourse:3.5.3 expectation may not be a realistic near-term recipe change. The bitnamilegacy deprecation is real (bitnami sunset legacy images), so a migration is the right long-term direction, but the test as written hard-codes a migration target absent upstream. Classification + fix approach to settle in M1 table / M2.

Classification: stale/PR-specific cc-ci OVERLAY test mismatched to the canonical-sweep context (NOT a flake, NOT a load timeout, NOT a recipe-deploy defect, NOT warm-machinery). Teardown clean (no discourse stack left). Evidence: /tmp/redfix-discourse.log on cc-ci; junit under /var/lib/cc-ci-runs/manual/junit/upgrade__cc-ci__test_upgrade.xml.

2026-06-18T00:05Z — M1: mattermost-lts isolation run — DETERMINISTIC restore failure (recipe defect)

Ran mattermost-lts ALONE (tag 2.1.9+10.11.15, log /tmp/redfix-mattermost-lts.log). RESULT: install/upgrade/backup/custom PASS, restore FAIL — identical to the canon failure: tests/mattermost-lts/test_restore.py::test_restore_returns_state → relation "ci_marker" does not exist after restore. So it is deterministic in isolation, NOT a loaded-node race (canon framing was wrong). The marker logic is sound (postgres table seeded pre-backup, dropped pre-restore, asserted post-restore — same pattern immich uses and PASSES).

ROOT CAUSE (recipe backup/restore labels). Compared mattermost-lts vs immich (immich passes the IDENTICAL test):

• immich database svc: backupbot.backup.pre-hook: /pg_backup.sh backup, backupbot.backup.volumes.postgres.path: backup.sql (backs up ONLY the dump file), and backupbot.restore.post-hook: /pg_backup.sh restore (replays the dump on restore). → round-trips.
• mattermost-lts postgres svc: pre-hook: pg_dump > /var/lib/postgresql/data/postgres-backup.sql, backup.path: /var/lib/postgresql/data/ (backs up the WHOLE live/hot PGDATA dir + the dump), post-hook: rm .../postgres-backup.sql, and NO backupbot.restore.post-hook. So on restore, abra restores the files but NOTHING replays the dump, and a hot-copied live PGDATA over a running postgres does not reload → ci_marker lost. Restore log confirms Restoring Snapshot b0495d36 at / with no post-hook reimport.

Classification: GENUINE RECIPE DEFECT at latest (postgres backup/restore does not round-trip — missing restore post-hook + backs up hot PGDATA instead of dump-only). NOT a flake, NOT cc-ci test weakening (test is correct & unmodified; immich proves the pattern works). Fix (M2) = recipe PR adopting the immich-style postgres backup/restore (a /pg_backup.sh-style dump + restore post-hook). Teardown clean (no matt stack). Evidence: /tmp/redfix-mattermost-lts.log; junit restore__cc-ci__test_restore.xml.

Tooling note: my background "waiter" loop while pgrep -f run_recipe_ci.py self-matched (its own cmdline contains the string) → never exited, falsely showed a run active. Use pgrep -f "[r]un_recipe_ci.py" or match the python invocation. Killed the stuck waiters; node confirmed free.

2026-06-18T00:18Z — M1: mumble isolation run — GREEN (flake confirmed)

Ran mumble ALONE (tag 1.0.0+v1.6.870-0, log /tmp/redfix-mumble.log). RESULT: ALL tiers PASS (install/upgrade/backup/restore/custom), including custom/test_protocol_handshake.py:: test_handshake_completes_with_channel_presence PASSED. No orphan stacks. The canon sweep recorded this RED (test_handshake… failed under concurrent sweep load); it is GREEN here in isolation, and its canonical was already written green TODAY (1.0.0+v1.6.870-0 @20260617T180501Z) under the lighter nixenv sweep. → load/timing FLAKE on the control-channel handshake, NOT a recipe defect.

The handshake test already retries (retry_handshake(attempts=12, interval=5.0) = 60s). So the flake is the voice server not completing the TLS+ServerSync handshake within ~60s under heavy concurrent node load (deploy contention). M2 fix = harness stabilization (stronger readiness gate before the custom tier / longer-or-smarter retry / serialize), based on the load failure mode. Classification: FLAKE (load/concurrency) → harness stabilization.

Reproducibility: 1 green isolation run here + canonical green today + documented red under canon load. Will do 1–2 more isolation repeats before the M1 claim to firm "reproducibly green in isolation."

2026-06-18T00:45Z — M1: bluesky-pds isolation run — 000 REPRODUCES; root cause = app DNS collision on shared proxy

Ran bluesky-pds ALONE (tag 0.3.0+v0.4.219, log /tmp/redfix-bluesky-pds.log). Cold lifecycle GREEN (install/backup/restore/custom pass; upgrade EXPECTED_NA per recipe_meta — moving pds:0.4 tag). Then WC5 promote-on-green-cold FAILED exactly as canon: warm-bluesky-pds.ci.commoninternet.net: not healthy over HTTPS /xrpc/_health (last status 0). So the 000 reproduces deterministically in isolation — NOT a sweep-load/ACME-rate-limit flake (my first hypothesis, refuted).

LIVE DIAGNOSIS (stack left deployed by the failed promote; probed before teardown):

• app service 1/1, healthy: docker exec app wget localhost:3000/xrpc/_health → {"version":"0.4.219"}; app listens on :::3000; no restarts. So the PDS itself is fine.
• HTTPS to warm domain → 000. caddy logs flood: tls "failed to get permission for on-demand certificate" domain=warm-bluesky-pds… error=… Get "http://app:3000/tls-check?domain=…": dial tcp 10.10.0.X:3000: connect: connection refused (X varies: .2 .4 .5 .6 .8 .9 .10 .12).
• bluesky uses caddy on-demand TLS (Caddyfile: on_demand_tls { ask http://app:3000/tls-check }, tls { on_demand }, reverse_proxy app:3000). caddy must reach app:3000/tls-check to be GRANTED a cert before serving TLS. It can't → no cert → TLS handshake fails → 000.
• WHY can't caddy reach app: service-name app collision on the shared proxy overlay.
  • app is on warm-bluesky-pds…_internal ONLY (IP 10.0.3.3). caddy is on proxy (10.10.50.223) + …_internal (10.0.3.6).
  • docker exec caddy getent hosts app → returns ONLY proxy IPs (8/8 tries: 10.10.0.4/.5/.6/.10/.12), NEVER the internal 10.0.3.3. The proxy-net app alias shadows bluesky's own internal app.
  • docker network inspect proxy shows EVERY stack aliases its main service app: drone…_app=10.10.0.2, traefik…_app=10.10.0.5, warm-keycloak…_app=10.10.0.9, ccci-reports/bridge/dashboard_app, … — exactly the IPs caddy hits. None listens a PDS on 3000 → connection refused. So caddy resolves bare app to OTHER stacks' app endpoints on the shared proxy, never its own PDS.

WHY cold passes / warm fails: cold's health window is long (HTTP_TIMEOUT=600) and on first success caddy CACHES the issued cert; the promote's shorter health window doesn't give caddy a chance to ever resolve correctly (and here it provably never resolves to 10.0.3.3 at all). The collision is the root cause; the promote machinery is CORRECT (it refused to write a canonical for an unhealthy 000 — no canonical.json written, verified).

Classification: genuine ROUTING/recipe defect — caddy↔app cross-stack app-alias collision on the shared proxy net, deterministic, reproducible in isolation. NOT a flake; NOT a promote-machinery bug. Fix approach (M2): recipe PR giving the PDS service a UNIQUE name/alias (e.g. rename app→pds) so caddy's reverse_proxy/tls-check resolve only bluesky's own internal service (no shared-proxy app collision). (Alternatively a caddy-side internal-only resolution; renaming is cleanest.) Will confirm the exact fix in M2 + verify the warm domain then serves 200.

Cleanup: removed orphaned warm-bluesky-pds stack + its volumes/secrets (promote had left it deployed; no canonical written). Node clean.

2026-06-18T01:05Z — M1: keycloak — warm-domain namespace collision (harness), classification complete

keycloak was de-enrolled (WARM_CANONICAL=False) because its data-warm canonical domain would collide with the LIVE-warm OIDC provider. Verified the collision STRUCTURALLY (code, no run needed):

• canonical.canonical_domain(r) → warm.stable_domain(r) → f"warm-{r}.ci.commoninternet.net" (runner/harness/canonical.py:42-44, warm.py:44-48).
• warm.WARM_DOMAINS["keycloak"] = "warm-keycloak.ci.commoninternet.net" (warm.py:27-29) — the always-on shared OIDC provider lasuite-*/drone consume for SSO; kept current by roll_warm_infra.
• So canonical_domain("keycloak") == WARM_DOMAINS["keycloak"] EXACTLY. Enrolling keycloak as a data-warm canonical → the sweep's promote deploy/teardown at warm-keycloak collides with the live provider. Confirmed live keycloak healthy (200 /realms/master) — I did not disturb it.

The collision is unique to keycloak: it is the ONLY recipe that is both a live-warm provider (in WARM_DOMAINS) AND would want a canonical. No collision-free canonical namespace exists today.

Classification: HARNESS defect — warm canonical domain namespace can collide with a live-warm provider. NOT a recipe/flake. Fix approach (M2): make canonical_domain(r) collision-free when r is a live-warm provider — e.g. warm-canon-<r> (or unconditionally) so the canonical deploy gets a distinct domain → distinct stack → cannot touch the live warm-keycloak. Then set keycloak WARM_CANONICAL=True and verify it promotes at the collision-free domain WITHOUT disrupting live keycloak. Minimal blast radius: special-case only providers in WARM_DOMAINS (the 15 other canonicals keep warm-<r>); confirm in M2.

2026-06-18T01:05Z — M1: gitea first advance attempt hit a LEFTOVER confound (not the real crash)

First gitea cold@3.6.0 run: cold lifecycle (install/upgrade/backup/restore/custom) ALL PASS; promote advance FAILED with FATA warm-gitea.ci.commoninternet.net is already deployed — NOT the app.ini crash. Cause: warm-gitea was left DEPLOYED at 3.5.3 by the nixenv-phase sweep (registry said status=idle but the stack was actually running — a state inconsistency). The advance does abra app deploy warm-gitea assuming the canonical is idle/undeployed; finding it deployed, abra FATAs. This is the same GREEN-BUT-PROMOTE-FAILED the nixenv phase saw. To reproduce the REAL app.ini issue I undeployed warm-gitea (docker stack rm; retained data+config volumes → proper idle state) and re-ran gitea cold@3.6.0 (gitea2). Result pending. NOTE: the "already deployed" promote-failure-when-left-deployed may be a secondary promote-machinery robustness gap (advance should undeploy-or-chaos an already-deployed canonical) — will assess after confirming the primary app.ini crash.

2026-06-18T00:14Z — M1: gitea warm advance — app.ini read-only JWT crash CONFIRMED (recipe defect)

After restoring warm-gitea to proper idle state (undeployed, 3.5.3 data+config volumes retained), re-ran gitea cold@3.6.0 (gitea2, log /tmp/redfix-gitea2.log). Cold lifecycle ALL PASS (install/upgrade/backup/restore/custom — incl. the cold FRESH 3.5.3→3.6.0 upgrade tier). WC5 promote advance then crash-loops. Live container logs (warm-gitea_..._app, repeated Failed/exit 1):

modules/setting/setting.go:105:LoadCommonSettings() [F] Unable to load settings from config: error saving JWT Secret for custom config: failed to save "/etc/gitea/app.ini": open /etc/gitea/app.ini: read-only file system

EXACTLY the canon-documented crash. Mechanism: the recipe mounts app.ini as a docker config (read-only by design) at /etc/gitea/app.ini (compose configs: - source: app_ini target: /etc/gitea/app.ini, app.ini.tmpl). gitea 1.24.2 (3.6.0), on the warm REATTACH of the retained 3.5.3 config volume, decides to (re)generate+SAVE a JWT secret to app.ini → read-only fs → FATA at config-load, BEFORE any DB migration (so the 3.5.3 data volume stays intact — confirmed canon).

Why cold passes but warm crashes: the cold fresh deploy + cold chaos-upgrade use freshly-generated secrets consistent with a freshly-initialized config, so gitea never needs to rewrite app.ini. The warm advance reattaches an OLDER retained config-volume state (seeded under 3.5.3) against the new run's secrets/3.6.0 binary → gitea reconciles by trying to persist a JWT secret → read-only crash.

Classification: genuine RECIPE defect (gitea 3.6.0/1.24.2 + read-only app.ini docker-config mount on the warm-reattach advance), deterministic, reproduced first-hand. NOT a flake, NOT promote machinery. Fix approach (M2): recipe PR making app.ini writable on the advance path — e.g. render the config into the WRITABLE config:/etc/gitea volume via an entrypoint (not a read-only docker config), OR ensure the persisted secrets are accepted without rewrite. (Secondary harness option: canonical advance falls back to clean re-deploy when in-place config rewrite is impossible — but that loses the reattach data-warm property; recipe fix preferred.) Ties to LFS PR #1 (app.ini secret handling).

ACTION NEEDED after run exits: warm-gitea is left crash-looping at 3.6.0 → restore it to 3.5.3 (redeploy the known-good canonical version) so the canonical is healthy again. Data volume intact.

2026-06-18T00:25Z — M1 CLAIMED (6/6 investigated, isolated, classified)

mumble repeat #2 (mumble2): ALL tiers green again incl. handshake; canonical re-promoted green (ts 20260618T001730Z). So mumble = 2× reproducibly green in isolation → load/timing FLAKE confirmed.

All six classified with first-hand isolation evidence (or code proof for keycloak). Two canon root-causes were CORRECTED by isolation: discourse (not a timeout/deploy-FATA — it's a stale cc-ci overlay test asserting an unreleased migration) and mattermost-lts (not a loaded-node race — a deterministic recipe restore defect: missing backupbot.restore.post-hook). bluesky's 000 is NOT a load/rate-limit flake (my initial hypothesis) but a deterministic caddy↔app app-alias DNS collision on the shared proxy. gitea app.ini read-only JWT crash reproduced first-hand. keycloak collision proven structurally in code.

Node clean: warm-gitea idle@3.5.3 (volumes retained), orphaned warm-bluesky removed, only live warm-keycloak up (healthy 200). Claiming M1; will start M2 fix design while awaiting the Adversary verdict (keep an unblocked item in hand).

2026-06-18T00:25Z — M2 prep (gated on M1 PASS): bluesky fix refinement

While parked at the M1 gate (no node deploys — Adversary cold-verifying), refined the bluesky fix: cc-ci's bluesky tests probe via HTTP (/xrpc/_health), but the GENERIC harness defaults to service="app" (deployed_identity/_app_container). So RENAMING the recipe's app service → pds could break generic harness assumptions. Cleaner fix: keep the service named app but give it a UNIQUE network ALIAS on the internal net (e.g. aliases: [pds-internal]) and point caddy at pds-internal:3000 (reverse_proxy + on_demand_tls ask). A unique alias has no collision on the shared proxy (only the bare app alias collides), and the service name stays app → zero cc-ci-side breakage. Will validate this exact approach in M2 after M1 PASS.

2026-06-18T01:21Z — M1 PASS; starting M2

Adversary M1 verdict: PASS @01:18Z — all 6 classifications cold-verified CORRECT by its OWN isolation re-runs (discourse/mattermost/mumble/bluesky/gitea) + code-verify (keycloak). No VETO. "Builder cleared to proceed to M2." Two canon root-causes corrected and confirmed (discourse: not a timeout, stale overlay; mattermost: not a load race, recipe defect). bluesky reclassification (recipe, not warm-machinery) confirmed against the plan's prior.

Starting M2. Plan: recipe PRs (mattermost-lts, bluesky-pds, gitea) via the recipe mirror+PR flow (!testme-verified, never merge); harness fixes (keycloak collision-free canonical_domain + enroll; mumble handshake stabilization) on a cc-ci branch; discourse overlay-scope decision. Node now mine (Adversary done). Will examine the recipe-create-pr flow first, then execute one fix at a time.

2026-06-18T01:25Z — M2 recon: prior-phase fix PRs already exist for discourse + mattermost

Surveyed open PRs on all 6 mirrors before doing redundant work:

• discourse #4 discourse-official-image ("switch to official discourse/discourse"): created 2026-06-16 by autonomic-bot; !testme PASSED twice, latest @53ba0910 today 16:36Z (run #849) ✅. This migrates off deprecated bitnamilegacy → official image + drops sidekiq = EXACTLY what the upgrade overlay asserts. So the overlay test was correctly demanding the migration; PR #4 IS the discourse fix and is already !testme-green. (Reframes M1 "stale test": the test is right; the release tag predates the migration; the fix is the migration PR, not weakening the test.)
• mattermost-lts #1 ci/pg-restore ("reimport the postgres dump on restore"): correct immich-pattern fix — pg_backup.sh (backup pg_dump|gzip; restore: terminate conns + DROP DATABASE WITH FORCE + createdb + reimport) + dump-only backup.volumes.postgres_data.path: backup.sql + restore.post-hook: /pg_backup.sh restore. Created 2026-05-30; needs a fresh !testme to confirm green NOW. (Also PR #2 upgrade-2.1.11 overlaps — adds restore hook + version bump; #1 is the focused fix.)
• mumble #1 = "cfold sweep probe" (not the fix — mumble is a harness flake, no recipe PR needed).
• bluesky #3 = version bump (not the routing fix — need a NEW PR for the app-alias collision).
• gitea, keycloak = no open PRs (gitea LFS #1 closed; keycloak is a harness fix).

M2 plan refined: VERIFY discourse #4 (re-!testme fresh) + mattermost #1 (!testme); CREATE recipe PRs for bluesky (unique alias) + gitea (app.ini writable); HARNESS fixes for mumble (handshake stab) + keycloak (collision-free canonical_domain + enroll). Starting with mattermost #1 !testme.

2026-06-18T01:30Z — M2: mattermost-lts FIXED (verified) + discourse already green + bluesky PR created

• mattermost-lts: !testme on PR #1 ci/pg-restore (@4ca7f418) → run #901 ALL tiers green (install/upgrade/backup/restore/custom, every junit failures=0 skipped=0). The M1-failing restore__cc-ci__test_restore.py::test_restore_returns_state now PASSES — the pg_backup.sh restore post-hook (terminate conns + DROP DATABASE WITH FORCE + createdb + reimport dump) round-trips postgres state. FIXED + verified. (Nothing merged — operator merges.)
• discourse: PR #4 discourse-official-image already !testme-green @53ba0910 (run #849, today 16:36Z) — the official-image migration makes the upgrade overlay pass. Will re-verify fresh for current evidence before the M2 claim.
• bluesky-pds: created mirror PR #4 ci/warm-routing-alias (unique pds alias on internal + caddy reverse_proxy/ask → pds:3000; service stays app). compose validated (docker compose config rc=0). VERIFICATION NOTE: bluesky's 000 is warm-promote-only (cold path always green), so !testme (cold) won't reproduce/verify it — I'll verify by running the FIXED recipe through the promote path (cold-on-latest with the fix checked out) → warm-bluesky-pds should serve 200 (vs M1's 000), then tear down the phantom canonical.

Remaining M2: bluesky promote-verify, gitea recipe PR (app.ini writable), keycloak harness (collision-free canonical_domain + enroll), mumble harness (handshake stabilization).

2026-06-18T02:10Z — M2 bluesky: alias fix blocked by abra; pivoting to service RENAME

Verified the bluesky pds network-alias fix end-to-end and found a blocker:

• docker stack deploy HONORS compose network aliases (throwaway test: app got Aliases:["pds","app"]).
• docker compose config PRESERVES the alias in its render.
• BUT the harness/abra promote deploy produced an app service with Aliases:["app"] only — the pds alias was DROPPED. The fixed Caddyfile (pds:3000) DID deploy (same per-run tree), so abra read my recipe tree; by elimination, abra's own compose→swarm translation drops service network aliases (it's not docker, not the tree). Also confirmed: the bluesky promote is a non-chaos pinned deploy. (Two stale-config gotchas also hit + fixed: docker configs are immutable+versioned — a stale warm-bluesky..._caddyfile_v1 was reused until I removed it; lesson for gitea = bump config versions.)

→ Pivot to the ROBUST fix: RENAME the PDS service app→pds. Docker auto-adds the service short-name as a network alias (abra can't drop that — the deployed app proved the service-name alias is always applied), so caddy's reverse_proxy pds:3000 resolves THIS stack's PDS (unique on internal; no pds on the shared proxy). Coupled cc-ci change: 2 exec_in_app(...) calls default service="app" (tests/bluesky-pds/_p4.py:40, custom/test_account_and_post.py:49) → must become service="pds" (NOT a weakening — same assertion, correct service). The warm-routing PROOF (warm-bluesky-pds→200) is the promote path (custom exec tests not involved); cold !testme-green needs the cc-ci ref update.

Need to determine how cc-ci-side code reaches a !testme run (also required for keycloak + mumble harness fixes) — investigating CCCI_REPO/Drone checkout next.

2026-06-18T02:15Z — cc-ci-side change verification mechanism (for bluesky-rename/keycloak/mumble)

The Drone !testme build clones cc-ci at main HEAD; the manual runner runs from CCCI_REPO (default /etc/cc-ci). To verify a cc-ci-side change WITHOUT pushing main or disturbing /etc/cc-ci (shared with Adversary): push the change to a cc-ci BRANCH, clone/checkout that branch to a temp dir on cc-ci, and run cd <tmp> && CCCI_REPO=<tmp> cc-ci-run runner/run_recipe_ci.py RECIPE=... CCCI_SKIP_FETCH=1 (cc-ci-run is the deployed nix env; runner/ + tests/ come from my branch checkout). Restores cleanly.

bluesky-rename coupling: the warm-promote only fires on a FULLY-GREEN cold run, and bluesky's custom tier exec_in_app defaults to service="app". So renaming app→pds REQUIRES the cc-ci exec-ref update (service="pds") deployed via the temp-checkout for the cold run to go green and the promote to fire. So: (1) recipe rename PR, (2) cc-ci branch with exec-ref update, (3) verify via temp-checkout run -> cold green -> promote -> warm-bluesky-pds 200.

M2 progress snapshot (2026-06-18T02:15Z)

• mattermost-lts: DONE (PR #1 ci/pg-restore, !testme run #901 all-green incl restore).
• discourse: DONE (PR #4 discourse-official-image, !testme run #849 green; re-verify fresh for claim).
• bluesky-pds: PR #4 (alias) -> superseding with service RENAME app->pds + cc-ci exec-ref update; verify on promote path.
• gitea: fix READY locally (/tmp/redfix-gitea: app.ini->staging + docker-setup seed-once + DOCKER_SETUP_SH_VERSION v2); needs PR push + warm-advance verify.
• keycloak: harness fix (canonical_domain collision-free for WARM_DOMAINS recipes + enroll) NOT STARTED.
• mumble: harness fix (handshake readiness/retry stabilization) NOT STARTED.

2026-06-18T02:45Z — M2 progress: gitea PR + harness branch pushed; bluesky pivoted to rename

• gitea: opened recipe PR #2 ci/app-ini-writable (app.ini->staging + docker-setup seed-once + DOCKER_SETUP_SH_VERSION v2). Advance-path verification RUNNING (fixed 3.6.0 reattach to idle 3.5.3 canonical; expect no app.ini crash + promote). cold lifecycle green so far (install + cold upgrade converged).
• bluesky: PR #4 updated alias->RENAME service app->pds (abra drops aliases). 3-line recipe diff, validates. Coupled cc-ci exec-ref change on branch.
• cc-ci harness branch redfix-m2-harness pushed (3 commits): keycloak (collision-free canonical_domain + WARM_CANONICAL=True), mumble (handshake budget 60s->180s), bluesky-pds (exec_in_app service=pds). Verified via temp-checkout runs (CCCI_REPO=).
• Verification sequencing (node is single, serial): gitea advance (running) -> bluesky rename promote (needs branch exec-refs) -> keycloak canonical at warm-canon-keycloak (needs branch) -> mumble. NOTE: mumble "green under load" is hard to reproduce deterministically; plan = show branch run still green + reason about the budget (or construct concurrent load).

2026-06-18T03:00Z — M2 gitea fix v1 (seed) BROKE the transition — needs rework

gitea advance verification (fixed 3.6.0): install tier PASSED FULLY (fresh 3.6.0 + my fix: API 200, admin auth OK — so the seed works for a FRESH deploy), but upgrade/backup/restore/custom ALL FAILED: READY_PROBE not ready: /api/v1/version (last status 404) within 600s after the 3.5.3->3.6.0 chaos redeploy → gitea came up in INSTALL-WIZARD mode (serves 200 but no API/admin = no valid app.ini). The LFS custom test's repo-create also 404'd (same wizard-mode cause).

So my seed-once fix is fine for fresh install but FAILS the 3.5.3->3.6.0 transition — exactly the path the canon fix needs. Likely cause: on the chaos redeploy from a 3.5.3 stack (docker_setup_sh_v1, no seed) the docker-setup config didn't update to my v2 (seed) while compose moved app.ini to the staging path → /etc/gitea/app.ini empty → wizard. (To confirm: reproduce + inspect the post-redeploy container — is docker_setup_sh_v2 mounted? does /etc/gitea/app.ini exist? gitea log.) Reverted the fix from cc-ci's gitea clone; warm-gitea intact (idle 3.5.3, promote didn't fire on the red cold run). gitea recipe PR #2 stands but the fix needs a rework (likely: a more robust seed that runs regardless of config version, OR provide a 1.24-valid oauth2 JWT secret so gitea never rewrites app.ini — investigate WHY 1.24 regenerates it). Deferring gitea; proceeding to bluesky-rename / keycloak / mumble verifies.

2026-06-18T03:30Z — M2 bluesky verification BLOCKED by abra non-chaos tag-revert; keycloak/mumble next

Root cause of the bluesky rename verify failure: the deployed service was ..._app (not pds). run_recipe_ci CCCI_SKIP_FETCH copies my renamed clone to the per-run tree, BUT abra's NON-CHAOS pinned deploy (bluesky's tag 0.3.0+v0.4.219 is ANNOTATED) does git checkout <tag> in the per-run tree, REVERTING my rename to the tag's app:. So the renamed recipe never deployed; the branch harness then execs service=pds -> "no running container _pds" -> backup/restore/custom red. (This also re-explains the earlier "abra dropped the alias" — it was the same tag-revert, not a drop.) gitea's tag is lightweight -> deploy_app uses chaos -> my gitea fix DID deploy (install passed); its failure is a real transition issue, not a revert.

IMPLICATION: verifying a RECIPE fix (bluesky, gitea) via CCCI_SKIP_FETCH needs a CHAOS deploy (uses the checkout, not the tag). HARNESS fixes (keycloak canonical_domain, mumble retry) are runner/test code from the branch checkout — NO tag-revert — so they verify cleanly. Doing keycloak + mumble next. For bluesky: force chaos (deploy_app does chaos when has_ccci_overlay) OR reconsider a cc-ci-side overlay fix (alias + caddyfile override) — both verifiable; recipe PR #4 (rename) stays as the ideal upstream fix. gitea: rework + reproduce-with-inspection.

2026-06-18T03:40Z — M2 keycloak FIXED + VERIFIED (collision-free canonical)

Ran keycloak cold-on-latest from branch checkout /tmp/cc-ci-m2run (harness fix: canonical_domain -> warm-canon-keycloak for WARM_DOMAINS recipes; WARM_CANONICAL=True). RESULT: all cold tiers PASS (install/upgrade/backup/restore/custom), and WC5 promote SUCCEEDED: canonical keycloak @ 10.8.0+26.6.3, domain="warm-canon-keycloak.ci.commoninternet.net", idle, volume retained.

• Promoted at the COLLISION-FREE domain warm-canon-keycloak (not warm-keycloak). ✓
• Live warm-keycloak (shared OIDC provider) = 200 THROUGHOUT — undisturbed. ✓
• warm-canon-keycloak = 404 now = CORRECT idle state (data-warm canonical undeployed, volume kept). So keycloak is now a full data-warm canonical with zero risk to the live SSO. FIXED + verified. 3/6 verified: mattermost-lts, discourse, keycloak. Doing mumble next (harness, tractable).

2026-06-18T03:50Z — M2 mumble VERIFIED (stabilization); 4/6 done

Ran mumble from branch checkout (handshake budget attempts=36/180s). ALL tiers PASS incl test_handshake_completes_with_channel_presence; promote succeeded (canonical 1.0.0+v1.6.870-0 idle). The longer budget is active + non-regressing. NOTE: mumble is green in isolation regardless of budget (the 60s sufficed in isolation); the budget matters UNDER LOAD, which is hard to reproduce deterministically — so this verifies the stabilization is applied + sound + non-weakening, not a literal load-flake repro. (M1 already established green-isolation/red-under-canon-load; the fix gives the handshake 3x the readiness window.) Stabilization fix verified. 4/6: mattermost, discourse, keycloak, mumble. Remaining: bluesky (force-chaos verify of the rename), gitea (rework).

2026-06-18T03:52Z — M2 bluesky force-chaos verification approach

bluesky's rename can't deploy via the normal path (annotated tag -> non-chaos -> abra checks out the tag, reverting the rename). In PRODUCTION post-merge the new tag would carry the rename (non-chaos deploys it fine). For PRE-merge verification I force chaos via a temporary tests/bluesky-pds/ compose.ccci.yml scaffold on the branch (has_ccci_overlay -> deploy_app uses chaos -> deploys my renamed checkout). Then cold goes green (service pds + branch exec-refs) and the promote deploys the renamed recipe at warm-bluesky-pds via chaos -> caddy resolves the unique pds -> expect 200 (vs M1 000). The overlay is a verification scaffold (NOT part of recipe PR #4); removed after.

2026-06-18T04:05Z — M2 bluesky verification: STRUCTURAL blocker (pre-merge warm-promote)

bluesky rename verification keeps deploying the TAG's app: (not my rename), even with: tag moved to the rename commit AND a force-chaos overlay. Root: the warm-promote/cold-on-latest path resolves the recipe at the UPSTREAM annotated tag (deploy_app recipe_checkout(tag) reverts unmerged content; the chaos+overlay path STILL recipe_checkout's the pinned version). Unlike gitea (lightweight tag -> the upgrade-tier chaos_redeploy uses the CHECKOUT, so the gitea fix deployed), bluesky has NO upgrade tier (EXPECTED_NA) -> no chaos_redeploy path -> the rename never deploys on the promote path.

CONSEQUENCE: an unmerged RECIPE fix whose failure is WARM-PROMOTE-ONLY (bluesky 000) cannot be end-to-end-verified via the standard harness pre-merge. mattermost/discourse were verifiable because their failures are COLD tiers (restore/upgrade-overlay) reachable by !testme on the PR head.

bluesky fix correctness is nonetheless ESTABLISHED by: (1) M1 root cause (Adversary-confirmed): bare app collides on the shared proxy; (2) docker test (proven): a unique service name/alias resolves to the local service (no collision). Renaming app->pds (PR #4) gives a unique name -> caddy resolves THIS PDS -> cert issued -> 200. End-to-end warm-200 needs either a DIRECT abra chaos deploy at warm-bluesky-pds (manual app+secrets+PLC-key setup; next iteration) or operator post-merge verify. Restored the bluesky tag; node clean; warm-keycloak 200.

M2 STATUS (2026-06-18T04:05Z) — 4/6 verified

• mattermost-lts: VERIFIED (PR #1 ci/pg-restore, !testme run #901 all-green incl restore).
• discourse: VERIFIED (PR #4 discourse-official-image, !testme run #849 green).
• keycloak: VERIFIED (branch redfix-m2-harness; canonical promotes at warm-canon-keycloak, live warm-keycloak undisturbed 200).
• mumble: VERIFIED-stabilization (branch; green + budget 180s active; load-flake not deterministically reproducible).
• bluesky-pds: fix correct (PR #4 rename) + mechanically proven; end-to-end warm verify structurally blocked pre-merge -> direct-deploy or operator post-merge.
• gitea: PR #2 seed fix BROKE 3.5.3->3.6.0 transition (wizard mode); testable via chaos; NEEDS REWORK (reproduce+inspect). NOT claiming M2 — bluesky end-to-end + gitea rework outstanding.

2026-06-18T05:53Z — M2 gitea VERIFIED (v3 seed) + bluesky VERIFIED (${STACK_NAME}_app); 6/6

gitea — rework was already done (v3, a0f2db8) but unverified; verified it. The clone's HEAD a0f2db8 ("fix v2 -s seed, v3") already addressed the v1 wizard-mode bug: docker-setup seeds app.ini into the writable /etc/gitea volume if [ ! -s /etc/gitea/app.ini ] (seed-on-EMPTY, not -f seed-on-missing — a 3.5.3-old-recipe canonical leaves a 0-byte app.ini placeholder in the config volume, which -f wrongly treats as present). Also bumps DOCKER_SETUP_SH_VERSION v1->v3 (config names are immutable; forces swarm to re-mount the new docker-setup) + app.ini config target -> /etc/gitea/app.ini.init (staging). Pushed v3 to PR #2 (force-replaced the broken v1 d4145266).

VERIFICATION (direct chaos-deploy onto the REAL idle 3.5.3 canonical volumes; /tmp/redfix-gitea-m2-directproof.log): reattached the retained config volume (0-byte app.ini = genuine pre-fix M1 state) with the v3 recipe. Result: app.ini seeded 0->1862 bytes, INSTALL_LOCK=true (not wizard), service 1/1, /api/v1/version -> 200 {"version":"1.24.2"}, /api/healthz 200, retained 3.5.3 data adopted (data dirs dated 2026-06-17T08:39 = canonical seed time, not fresh), 0 read-only-app.ini crashes (M1 crashed here).

WHY NOT the harness WC5 promote: it is STRUCTURALLY merge-gated. run_recipe_ci.py:373 force-fetches refs/tags/* from upstream even under CCCI_SKIP_FETCH, and abra itself force-fetches tags on deploy (abra.py:135 documents this) — so a LOCAL tag-move to the fix commit is always reverted to the published 357926f. promote_canonical does recipe_checkout(tag)+non-chaos deploy -> deploys the PUBLISHED release, which pre-merge lacks the fix. Confirmed empirically: a full harness run's WC5 promote deployed 357926f (caddyfile/app.ini OLD) -> crashed exactly like M1. So end-to-end canonical-advance needs the operator to merge PR #2 + re-cut 3.6.0; the direct chaos-deploy is the maximal+faithful pre-merge proof (chaos deploys the working-tree checkout = the PR fix). Node left clean: warm-gitea undeployed (idle 3.5.3, volumes retained), app.ini reset to 0-byte for re-verify, canonical.json UNCHANGED (3.5.3 idle e6a1cc79), recipe tag restored to upstream 357926f.

bluesky — operator directive (2026-06-18): NO rename; use ${STACK_NAME}_app. Replaced the rename (PR #4) with the minimal prefix fix: Caddyfile ask http://{$APP_HOST}:3000/tls-check + reverse_proxy {$APP_HOST}:3000 (caddy native {$ENV}, already used for {$DOMAIN}); compose caddy service - APP_HOST=${STACK_NAME}_app; CADDYFILE_VERSION v1->v2. Service stays app -> NO coupled cc-ci exec-ref change (reverted/dropped b96b8a4 from branch redfix-m2-harness; that branch is now mumble+keycloak only). 3-file recipe-PR-only diff. Pushed to PR #4 ci/warm-routing-alias (4987ba9, force-replaced the rename). Pattern per matrix-synapse/mailu/mumble.

VERIFICATION (direct chaos-deploy at warm-bluesky-pds with secrets + PLC key; /tmp/redfix-bluesky-m2-directproof.log): caddy APP_HOST=warm-bluesky-pds_ci_commoninternet_net_app; getent ${STACK_NAME}_app -> 10.0.3.x (bluesky's OWN internal net) while getent app (M1's bare target) -> 10.10.0.12 (FOREIGN proxy net, the collision); caddy log "certificate obtained successfully" (let's-encrypt, via the own-app tls-check) with 0 connection-refused (M1 cycled refused); external HTTPS https://warm-bluesky-pds.../xrpc/_health -> 200 {"version":"0.4.219"} (M1 was 000). GOTCHA: abra secret insert (no -C -o) force-fetches+checks out the .env TYPE tag, reverting the fix checkout -> must re-checkout the fix AFTER secret ops, right before the chaos deploy. Same merge-gating as gitea (bluesky has no upgrade tier -> warm-promote is the only failing path -> end-to-end canonical-advance is operator-merge-gated; direct chaos-deploy is the maximal pre-merge proof). Node left clean (warm-bluesky-pds torn down, volumes+secrets removed; no canonical, matching M1). Live warm-keycloak 200 throughout.

6/6 VERIFIED. Claiming M2.

2026-06-18T06:55Z — M2 re-claim: discourse F-redfix-1 FIXED + level=5 verified (6/6)

Adversary M2 verdict (06:42Z) was FAIL on discourse ONLY — sharp, correct finding F-redfix-1: my official-image migration (PR #4 @53ba0910) dropped sidekiq from compose.yml (correct — sidekiq is internal to the official image) but left a dangling image-less sidekiq: block in compose.smtpauth.yml (it only added SMTP env + the smtp_password secret, inheriting the image from the old base sidekiq). After the drop, the smtpauth-merged compose has an image-less service → abra recipe lint R011 fail (the L5 rung), run level=4; and any SMTP-auth deploy would start an imageless service. My earlier "run #849 green" was deploy-green (level=4), NOT L5-green — the Adversary correctly called this out.

FIX (PR #4 @9ff5e19, force-pushed onto 53ba0910): removed the orphaned sidekiq: block from compose.smtpauth.yml. No SMTP coverage lost — the app: override already carries DISCOURSE_SMTP_PASSWORD_FILE=/var/run/secrets/smtp_password + the smtp_password secret, and compose.yml app has all DISCOURSE_SMTP_* env; the official image runs sidekiq inside app. grep sidekiq compose*.yml = 0 now.

VERIFIED two ways: (1) the Adversary's exact lint.py repro (clone → checkout -B main 9ff5e19 → ABRA_DIR=scratch abra recipe lint -n discourse) → R011 ✅ (was ❌ at 53ba0910). (2) full cold harness run /tmp/redfix-discourse-m2verify.log: lint rung: pass, RUN SUMMARY level=5 of 5, all tiers pass (install/upgrade/backup/restore/custom), both upgrade-overlay tests pass. Node clean: no discourse stack/canonical (untagged migrated head doesn't promote), recipe reset to published tag 0.8.1+3.5.0.

Other 5 (keycloak/mumble/gitea/bluesky-pds/mattermost-lts) Adversary-PASS already, fixes unchanged — not re-run. 6/6. Re-claiming M2.

2026-07-08 — post-reboot re-confirmation #2 (no work performed)

Terminal condition re-checked, unchanged: STATUS-redfix.md = ## DONE; REVIEW-redfix.md shows M1 PASS @2026-06-18T01:18Z and M2 PASS @07:06Z (6/6, supersedes the 06:42Z FAIL after F-redfix-1 was closed); no standing ## VETO; no BUILDER-INBOX.md. Loop remains stopped. No new work claimed.

Node observations (informational — none are phase regressions, none actioned):

• warm-gitea_ci_commoninternet_net_app is crash-looping, 0/1. Cause confirmed from docker service logs: error saving JWT Secret for custom config: failed to save "/etc/gitea/app.ini": open /etc/gitea/app.ini: read-only file system. This is exactly the M1 root cause, and it is the expected pre-merge state: the fix lives in unmerged recipe PR #2 (ci/app-ini-writable), and the phase guardrail is "nothing merged — the operator merges". The warm service deploys the published (unfixed) release, so it will keep crash-looping until PR #2 is merged. Live corroboration of the M1 classification, not a new defect.

• Two stray wedged processes on cc-ci, both blocked on interactive abra prompts, both started AFTER this phase completed (2026-06-18): pid 577140/577187 (dev-bluesky-pds secret insert, started 2026-06-26) and pid 580348 (abra app config dev-drone, started 2026-06-19). They are not this phase's and left nothing deployed (docker stack ls shows no dev- stacks). Not killed — foreign session, not mine to reap.

• Two untracked files in the repo ROOT, not created by this phase (mtime 2026-06-23, five days after DONE): main.go (a hello-world HTTP server) and config.json (LLM provider config containing what looks like a live Tinfoil API key). Deliberately NOT committed — staging config.json would write a credential into git history — and deliberately NOT deleted, since this loop did not create them. Flagged to the operator. Staged nothing via git add -A for this reason.

2026-07-08T23:08:27Z — consumed BUILDER-INBOX; remediated F-redfix-2 (config.json un-gitignored)

Woken by two watchdog pings (a review(...) verdict, and BUILDER-INBOX.md appearing). Both pointed at the same thing. Phase state re-checked first: redfix remains DONE — STATUS ## DONE @2026-06-18T07:09Z, M1 PASS @01:18Z, M2 PASS @07:06Z (6/6), no standing VETO. No gate action was required of me.

The Adversary's new F-redfix-2 (BACKLOG-redfix.md, non-blocking) assigned the remedy to me while reserving closure for itself. I did not take its claims on trust — re-derived each from my own clone:

git ls-files --error-unmatch config.json  -> error: pathspec ... did not match  (untracked)
git log --all --oneline -- config.json    -> (empty)                            (never committed)
git check-ignore -v config.json           -> (no match)                         (NOT ignored)
.provider.tinfoil.options.apiKey          -> tk_bhg…  len=51                    (live-shaped)

All four confirmed. So: a latent credential-push risk, not an existing leak — git history is clean.

Why .gitignore and nothing else. The exposure is that git add -A in either clone stages a live key and pushes it to git.autonomic.zone. One ignore line removes that path entirely. I proved it rather than asserting it — replayed git add -A against a throwaway GIT_INDEX_FILE seeded from HEAD:

before: config.json would be staged
after : .gitignore, main.go only  — config.json NOT staged

What I deliberately did NOT do, and why:

• Did not delete/move/rewrite config.json. This loop did not create it (mtime Jun 23, five days after phase DONE) and it is plausibly live operator state. Not mine to destroy.
• Did not rotate the key. Out of my authority and it touches a live external provider. The Adversary is right that rotation is warranted if the key was ever pasted into a transcript or log — it appears in at least one journal-adjacent context, so I am escalating rotation to the operator rather than silently deciding it is fine. Flagged, not actioned.
• Did not gitignore main.go. Also foreign, but it is a hello-world HTTP server with no secret in it; staging it would be untidy, not dangerous. Ignoring files I did not create is overreach beyond the finding.
• Did not reopen redfix, touch REVIEW-redfix.md, or mark F-redfix-2 closed. The Adversary owns closure after re-test; a .gitignore line is repo hygiene, not a DoD item, so phase state is unchanged.

.gitignore is tracked, so the fix propagates to the sibling clone /srv/cc-ci/cc-ci on pull (done below).

Correction to F-redfix-2's premise: "both clones" is ONE repo (symlink)

The finding says config.json sits at the root of both Builder clones. Verified otherwise — they are the same directory:

stat -c '%i %n' /srv/cc-ci/cc-ci/.gitignore /srv/cc-ci-orch/cc-ci/.gitignore
3252849 /srv/cc-ci/cc-ci/.gitignore
3252849 /srv/cc-ci-orch/cc-ci/.gitignore     <- same inode
git -C /srv/cc-ci/cc-ci rev-parse --show-toplevel -> /srv/cc-ci-orch/cc-ci

Same for the Adversary pair (cc-ci-adv resolves to one dir), which carries no config.json at all. So the exposure was one file in one repo, and the single .gitignore line closes it on every path that reaches it. This does not weaken the finding — the risk was real — it just means no second remediation is pending, which matters because F-redfix-2 stays open until the Adversary re-tests.

Outstanding, for the operator (not actionable by me): rotate the Tinfoil key if it was ever pasted into a transcript or log. Gitignoring prevents a future commit; it cannot un-expose a key already seen elsewhere.

2026-07-08T23:14:36Z — F-redfix-2 CLOSED by Adversary; no action required; phase stays DONE

Watchdog ping on a review(...) commit (67e86de). Pulled and read the verdict. No VETO (0 occurrences of ^## VETO in REVIEW-redfix.md), phase redfix still DONE, M1/M2 PASS intact. Nothing claimed, nothing asked of me. The Adversary cold-verified my .gitignore remedy from a fresh clone of origin/main (not just my working tree — the distinction matters: a local-only edit would have left a fresh clone unprotected) and confirmed git add -A stages main.go only.

Two corrections landed, both worth keeping:

1. Mine, accepted. It confirmed the symlink finding independently (inode compare + rev-parse + a filesystem sweep turning up exactly one in-repo config.json). Its own note: "a finding that overstates its blast radius is a defect in the finding."
2. Its own, self-caught, and the more interesting one. Its original "never committed" evidence grepped history for the 6-char prefix tk_bhg — but that prefix now appears in the finding, the inbox, and this journal, so the grep was self-contaminating and would have reported hits forever regardless of truth. Re-tested against the full 51-char value: 0 commits, longest-ever-committed prefix 6/51 chars, all of it in our own prose. Never leaked. Worth remembering as a general trap: searching history for a secret using a short prefix of that secret, after you have written the prefix into the repo, tests your own prose, not history. Use the full value, or search a commit range that predates your notes.

Residual, unchanged and correctly left with the operator: the key is still on disk, unrotated. Git is not a reason to rotate — it provably never entered history — but gitignore cannot un-expose a value that reached a transcript or log by another path. That call is not either loop's to make.

Backlog hygiene note (did NOT edit — the findings section is Adversary-owned): BACKLOG-redfix.md carries the authoritative CLOSED header at L111 and a verbatim archive of the original finding at L146 whose preserved header still reads "OPEN, NON-BLOCKING". Unambiguous in context (L146 is labelled "original text"), so I left it and did not wake the Adversary over it.

Terminal condition holds: ## DONE + fresh Adversary PASS on M1 and M2 + no standing VETO + no open blocking finding. Loop stopped.

---

2026-07-08 — post-reboot re-confirmation #4: phase still DONE, loop stopped

Rebooted into the redfix Builder role again. Re-checked the terminal condition from primary sources rather than trusting the commit log or the prior journal entry:

• machine-docs/STATUS-redfix.md L10 → ## DONE — 2026-06-18T07:09Z; its ## Blocked section is (none).
• machine-docs/REVIEW-redfix.md → M1 PASS @2026-06-18T01:18Z, M2 PASS @2026-06-18T07:06Z (6/6, superseding the 06:42Z FAIL); no standing ## VETO; F-redfix-2 explicitly CLOSED (non-blocking).
• No BUILDER-INBOX.md and no ADVERSARY-INBOX.md — nothing pending in either direction.
• HEAD == origin/main == bccba44; nothing unpushed.

Independently re-tested the F-redfix-2 remedy instead of assuming it held: git show origin/main:.gitignore carries config.json at line 7, and a scratch-index git add -A -n stages neither config.json (ignored) nor anything else secret. The rule is on origin, so a fresh clone is protected too.

Untracked stray, deliberately left alone: main.go at the repo root — a 16-line "Hello, World!" net/http server, mtime 2026-06-23T00:59Z, i.e. five days after this phase closed on 06-18. Not mine, not referenced by any tracked code, not part of redfix. A scratch-index add -A would stage it, so it is one careless git add -A from a commit. I did not create it and I did not delete it — surfacing it for the operator is the correct move, not silently removing a file whose provenance I cannot establish. Same posture the previous reboot took toward foreign root files.

No unblocked work exists: every DoD item is Adversary-verified green, nothing is claimed, nothing is blocked. Starting a work loop would be make-work. Terminal condition met → loop stopped.

---

2026-07-08 — F-redfix-3 (Adversary, CLOSED/non-blocking): acted anyway, on the record not the gate

Watchdog pinged on 1271c24. Verdict: DONE stands, no VETO; M1+M2 intact; F-redfix-3 filed and CLOSED by the Adversary with "no action required of the Builder now". No inbox file in either direction.

I re-tested its load-bearing claim from a fresh clone rather than accepting a closed finding on trust — the whole point of the finding is that a stale pointer can make a live fix look withdrawn, and that failure mode applies to the Adversary's own note as readily as to mine:

git clone .../recipe-maintainers/discourse && git fetch origin 'refs/heads/*:...' 'refs/pull/*/head:...'
git cat-file -t 9ff5e19   -> fatal: Not a valid object name   (same for 53ba0910)
git show origin/discourse-official-image:compose.yml | grep image:  -> image: discourse/discourse:3.5.3
git show origin/discourse-official-image:compose.smtpauth.yml | grep -c sidekiq -> 0

Both halves confirmed: the shas are genuinely unreachable from all 17 refs, and the fix content is intact at head ede6399. Cause is a later phase force-pushing the shared discourse-official-image branch. M2 was verified against the shas that existed on 06-18, so a subsequent rebase cannot retroactively unfix it.

Why I edited STATUS despite "no action required." The rotted pins live in my file, and a content assertion (file -> expected line + the command to check it) is precisely STATUS material under WHAT+HOW+EXPECTED+WHERE. Leaving it would hand a future auditor a document whose own repro steps fail, which reads as "the discourse fix was withdrawn." So I appended an evidence addendum giving the durable content check, and deliberately did not rewrite the historical sha lines — those record what was verified on 2026-06-18, and editing them to match today would misrepresent the M2 evidence rather than preserve it.

This changes no claim and reopens no gate: ## DONE stands, M1+M2 PASS unchanged, no VETO. The general lesson (shared recipe branches get rewritten; pin content, or push a tag) is the Adversary's, and it is a good one. Terminal condition still met. Loop stopped.

2026-07-08T23:32Z — post-reboot wake #5: Adversary verdict consumed, no action; loop stopped

Woke on a watchdog ping for 805c44e (review(redfix): post-reboot re-confirmation #5). Pulled and read it in full before deciding anything. It is not a finding against me — it is the Adversary's audit of my own F-redfix-3 evidence addendum (612412c), and it came back clean:

• edit shape verified append-only (+54/-0), my STATUS/JOURNAL only, ## DONE + historical shas untouched;
• my published content assertion re-run verbatim in a fresh clone: ede6399 → discourse/discourse:3.5.3, 0 sidekiq in compose.smtpauth.yml — both EXPECTED met;
• it disbelieved its own green (ruled out a vacuous grep -c zero via cat-file -t = blob) and added a negative control (d7c8c47 added sidekiq, 0c4539b removed it), establishing something stronger than re-confirmation #4: the redfix fix commit is still reachable (0c4539b = refs/pull/4/head) — only the sha labels rotted under the later-phase rebase. F-redfix-3's remedy is verified, not merely asserted.

Verdict: DONE stands, no VETO. So there is no work here. Terminal condition holds unchanged: STATUS ## DONE @2026-06-18T07:09Z; M1 PASS @01:18Z + M2 PASS @07:06Z; F-redfix-1/2/3 all CLOSED; ## Blocked = none; no BUILDER-INBOX.md. Both loops have now independently converged on stop.

Two things I deliberately did not do. (1) I did not re-run the six recipe CIs to "freshen" the PASSes — no gate is pending and the node is shared; re-proving a closed gate is the pointless re-verification the protocol forbids, and it is what the Adversary declined for the same reason. (2) I did not delete the stray untracked main.go (16-line Go hello-world, never committed, unrelated to this Python/Nix harness). I did not create it and it is not mine to remove; it is surfaced here and in wake #4 for the operator instead. It is untracked and unignored, so it cannot contaminate a claim or a cold clone.

Loop stopped — no further self-wake scheduled.

Wake #6 — 2026-07-08T23:36Z (post-reboot)

Rebooted by the watchdog after wake #5 stopped the loop. Re-checked the terminal condition rather than assuming it: ## DONE @2026-06-18T07:09Z stands in STATUS-redfix.md; M1 PASS @2026-06-18T01:18Z and M2 PASS @2026-06-18T07:06Z both intact in REVIEW-redfix.md; no standing ## VETO; no machine-docs/BUILDER-INBOX.md; all three adversary findings (F-redfix-1/2/3) CLOSED.

Consumed the Adversary's post-reboot re-confirmation #6 (REVIEW-redfix.md @2026-07-08T23:36Z). It asks nothing of me: it is a verdict, not an inbox message, and it opens no finding. Its break-it probe (cold --bare clone of each recipe mirror) independently re-confirms that the M2 evidence anchors for mattermost-lts (4ca7f418), gitea (a0f2db8), bluesky-pds (4987ba9) and the cc-ci harness branch tip (07fc6d4) all still resolve and carry the subject lines M2 asserted — so the sha rot is confined to discourse, which is exactly what F-redfix-3 already established and closed (content re-verified at ede6399). Nothing to rebut, nothing to remedy.

The stray untracked main.go in the repo root is still present. Unchanged position: it is not mine, I did not create it, and it is not referenced by any phase file, so I surface it rather than delete it. It is untracked, so it cannot affect a cold clone or any Adversary verification.

No action. DONE stands, no VETO. Loop stopped.

Wake #7 — 2026-07-08T23:49Z (post-reboot)

Rebooted by the watchdog after wake #6 stopped the loop. Re-checked the terminal condition from the files rather than assuming it survived: ## DONE @2026-06-18T07:09Z stands in STATUS-redfix.md (## Blocked = none); M1 PASS @2026-06-18T01:18Z and M2 PASS @2026-06-18T07:06Z both intact in REVIEW-redfix.md; no standing ## VETO; no machine-docs/BUILDER-INBOX.md; F-redfix-1/2/3 all CLOSED. git pull --rebase reports already-up-to-date, so origin/main is exactly what I have.

Consumed the Adversary's post-reboot re-confirmation #7 (REVIEW-redfix.md @2026-07-08T23:47Z). Like #6 it is a verdict, not an inbox message, and it opens no finding, so it asks nothing of me. It does strengthen the M2 evidence in a way worth recording: #6 established only that the M2 evidence anchors were reachable with matching subjects, which is compatible with a reachable sha carrying no fix. #7 closes that gap by grepping the TREE at each asserted sha in a cold --bare clone, with the parent commit as a negative control — mattermost-lts 4ca7f418 adds the backupbot.restore.post-hook (parent: 0 matches), gitea a0f2db8 adds app.ini.init + the docker-setup cp (parent: 0 matches), bluesky-pds 4987ba9 adds the fully-qualified APP_HOST with no leftover bare app alias (parent has no APP_HOST at all). With F-redfix-3's content re-verification of discourse at ede6399, all four recipe fixes are now content-anchored rather than sha-anchored — which is precisely the property that makes them robust to the branch drift F-redfix-3 flagged. Nothing to rebut, nothing to remedy.

The stray untracked main.go in the repo root (hello-world Go HTTP server, mtime 2026-06-23) is still present. Unchanged position: it is not mine, I did not create it, no phase file references it, and it appears nowhere in git history on any ref. I surface it rather than delete it. It is untracked, so it cannot affect a cold clone or any Adversary verification.

No action. DONE stands, no VETO. Loop stopped.

---

Wake #8 — 2026-07-09 — VETO F-redfix-4 received, reproduced, fixed, M2 re-claimed

Woken by a watchdog ping on a review(...) commit. After seven consecutive zero-delta re-confirmations, the Adversary changed probe angle — it had only ever checked the two harness fixes for sha reachability, never for content or second-order effects — and found a real defect. That is exactly what the adversarial loop is for, and it landed on the one code path none of my M2 verification exercised.

Why my M2 verification missed it. I verified that the enrollment deployed: warm-canon-keycloak_* volumes exist, canonical_domain() returns a distinct domain, live SSO untouched. I never exercised seed_canonical(), because registry-advance is deliberately deferred to the operator's merge ("nothing merged"). So the first-ever keycloak seed would have happened post-merge, in production, unexercised. The lesson I'm taking: "the artifact exists and the claim matches the code" is not the same as "the code path the change newly switches on has run." An enrollment flag that gates a data path is only verified when the data path executes — a deploy-only check verifies the domain layer and nothing below it.

Order of work. I withdrew ## DONE before doing anything else. The deliverable is a branch the operator merges, and merging is precisely what arms the defect; leaving a DONE marker up while I investigated would have been the one irreversible mistake available to me. Withdrawal is cheap and reversible, so it goes first even before I'd confirmed the finding myself.

I reproduced it independently rather than taking the finding on trust (plan §9: verify against the real server). Cold clone of the branch on cc-ci, scratch CCCI_WARM_ROOT, real idle warm-canon-keycloak stack: snap_dir("keycloak") is one slot for both domains; the second snapshot() replaced the first; restore() raised SnapshotError. Confirmed. I also confirmed the latency of the bug on the real node — /var/lib/ci-warm/keycloak/ holds only last_good, no canonical.json, no snapshot/ — which is both why M2 passed and why the fix needs no migration.

Design. The tempting minimal fix is a WARM_DOMAINS skip-guard on seed_canonical. It is wrong: it silently de-enrolls keycloak and quietly re-opens the DoD item the phase exists to close — a fix that converts a loud bug into a silent hole. The Adversary pre-emptively ruled it out and I agree.

The purest fix is to key every slot by its stack name. I rejected it on blast radius: it relocates last_good for the live keycloak+traefik reconcilers and the registry+snapshot of 15 canonicals on a live node, needing a migration shim, in a phase mandated to fix red rather than re-architect warm storage.

What I shipped is the same model applied exactly where two stacks contend: one canonical_ns() from which BOTH the canonical's domain and its slot derive. The coupling is the point — the previous code had a conditional for the domain and no conditional for the slot, and that asymmetry IS the bug. Deriving both from one function makes the drift unrepresentable rather than merely absent. Every existing canonical keeps ns <recipe>, so nothing on disk moves.

I added _assert_slot_not_foreign() as defence in depth: it compares against the slot's recorded domain, so it is independent of the canon- naming scheme and will catch a future caller that pairs a slot with the wrong stack. Deliberately behind the structural fix, not instead of it — a guard alone would have left the canonical permanently unable to seed. I put it in snapshot() before the destructive swap and in restore() before touching volumes, so it fails early rather than at the next restore.

Scope discipline. The Adversary's consequence (3) chains through a genuine second defect: the reconciler's rollback restore() sits outside the upgrade's try/except, so a raising restore leaves live keycloak undeployed after abra.undeploy(). Removing the shared slot removes the race that made this reachable, but not the structural gap. It is not in F-redfix-4's clearing condition, and choosing between "redeploy last_good anyway" and "die loudly rather than start on unrestored data after a forward DB migration" is a real safety trade-off for a DB-backed app that I should not settle inside a remediation commit. Filed as B-redfix-5 in BACKLOG-redfix.md and named in STATUS so it is visibly deferred, not dropped.

Verification. Unit suite: baseline 315 passed at parent 07fc6d4, 325 passed with the fix — I ran the baseline first, on an unmodified cold clone, so the +10 is attributable and no pre-existing test broke. Two early full-suite runs failed on test_dashboard.py / test_bridge_trigger.py / test_meta.py; that was my own partial scp (missing dashboard/, scripts/), not the change — proven by the clean 315 baseline in a full clone. Worth recording because for a few minutes it looked like I'd broken three unrelated modules.

For the clearing condition I used a throwaway warm-fakelive_…_data docker volume as the live-warm stand-in rather than the real warm-keycloak stack (which cannot be undeployed to snapshot — it is the shared OIDC provider lasuite-*/drone depend on), and the real idle warm-canon-keycloak stack for the canonical side. restore() genuinely rewrites volumes, so I checksummed the canon mariadb volume before and after: 1201440268 48846 both times — the round-trip is byte-identical. Live warm-keycloak…/realms/master returned 200 throughout. Throwaway volume removed, scratch removed, real warm root still last_good only.

Incidental find. My local redfix-m2-harness carried an unpushed commit b96b8a4 ("exec into renamed 'pds' service, pairs with recipe rename app->pds") dated Jun 18. origin was at 07fc6d4, matching the Adversary's pin, so nothing shipped. That commit belongs to an abandoned approach: the bluesky-pds fix that was actually adopted is the caddy ${STACK_NAME}_app prefix (recipe PR #4 @4987ba9), which does NOT rename the service — so service="pds" would have broken both exec call sites. Had I built the F-redfix-4 fix on my local branch tip without checking, I'd have pushed it silently along with the remedy. I reset the local branch to origin/redfix-m2-harness before starting (recoverable via reflog) and built on 07fc6d4. Check git rev-parse <branch> origin/<branch> before building on a long-lived branch you last touched three weeks ago.

## DONE is NOT restored. M2 is re-claimed and awaits an Adversary verdict on b5f2b10.

---

Wake #9 — 2026-07-09T00:18Z — M2 PASS, VETO cleared, phase closed

Consumed the Adversary's verdict on b5f2b10: M2 PASS, VETO CLEARED, F-redfix-4 CLOSED, with an explicit "the Builder may re-assert ## DONE". Re-asserted it. Merge target is redfix-m2-harness@b5f2b10, not 07fc6d4 — the old tip carries the defect, and that distinction is the one thing an operator could still get wrong from a stale note.

Its re-test went past my own in three ways worth recording. (a) Mutation testing: it reverted canonical_ns() and confirmed 4 of my new tests go red, then removed _assert_slot_not_foreign() and confirmed 2 more do — so the 315→325 delta is load-bearing, not vacuous. I had asserted the tests pass; I had not proven they can fail. That is a strictly better verification of a regression test than the one I ran, and I'll reach for it next time I claim "+N tests that would have caught this". (b) It checked the guard in both directions (foreign snapshot AND foreign restore); my probe only exercised the snapshot side, even though I'd written the restore guard. (c) It verified all 21 enrolled recipes still resolve to their existing on-disk dirs (registry_path("bluesky-pds") character-identical at parent and fix), which is the real proof of "zero blast radius / no migration" — I had only checked keycloak's own dir.

It also independently confirmed b5f2b10's diff touches only the four source files plus two test files and tests/keycloak/recipe_meta.py, so the other five recipe fixes are provably untouched by the remedy. That is the check that lets a scoped VETO clear without re-verifying the whole phase.

B-redfix-5 stands as the single deferred item: recorded in BACKLOG-redfix.md and now also in the shared DEFERRED.md, since the phase closes and a phase-namespaced backlog entry would die with it. The Adversary concurred it is not a VETO and was "correctly filed rather than silently fixed".

What the phase cost, and what it bought. The 2026-06-18 DONE was wrong for three weeks. Seven re-confirmations at the same probe angle (does the artifact exist? does it match the claim?) found nothing, because the defect was one layer below: an enrollment flag gated a data path that never ran, so everything observable was consistent. It fell out the moment the Adversary changed angle to "what does this code newly switch on, and has that ever executed?" — a lesson worth more than the fix. Deploy-only verification of an enrollment verifies the domain layer and nothing beneath it.

Terminal condition met: ## DONE + fresh PASS on every gate + no standing VETO. Loop stopped.

---

Wake #10 — 2026-07-09 — watchdog ping on review(redfix) @d55ba23; no-op, DONE stands

Watchdog fired on a review(...) commit. Pulled: the Adversary's change is editorial, not a verdict. It annotated the original ## VETO — keycloak enrollment … heading in place as ## VETO [CLEARED @2026-07-09T00:18Z … NOT standing], and promoted the clearance from bold **## VETO CLEARED** (which rendered as text, not a heading) to a real ## VETO CLEARED @2026-07-09T00:18Z heading. No content beneath either heading changed; M2 PASS stands, F-redfix-4 stays CLOSED.

Why this mattered enough for the Adversary to push it. My ## DONE predicate is "no standing VETO", and the natural way to test that — grep for ^## VETO — matched the superseded heading, because a cleared veto is cleared by a later paragraph, not by editing the earlier one. Verdict history is append-only, so the record was correct while a mechanical reading of it was not. Two agents relying on a grep over a human-readable file had a latent disagreement about the phase's terminal condition, and the file's own history was the thing lying. Fixed by making the heading self-describing rather than by trusting readers to scroll 96 lines to the clearance.

Re-checked the predicate after the pull:

grep -E "^## VETO" machine-docs/REVIEW-redfix.md | grep -v CLEARED   -> (empty)
grep -n "^## DONE" machine-docs/STATUS-redfix.md                     -> 10:## DONE — 2026-07-09T00:18Z

No standing VETO, ## DONE intact, no BUILDER-INBOX.md, merge target redfix-m2-harness@b5f2b10 unchanged. Nothing to build, nothing to re-verify — a verdict-preserving edit does not reopen a gate. Terminal condition still met. Loop remains stopped.

Unrelated, still unexplained and deliberately untouched: an untracked main.go (16-line Go hello-world HTTP server on :8080) sits in the repo root of this Python harness clone. Nothing in this phase created it; it is referenced in no plan or state file. Not committed, not deleted — I did not write it and cannot account for it, so it is the operator's call.

Wake #11 — 2026-07-09T00:55Z — no-op re-check; phase stays closed, loop stays stopped

Woken with the tree exactly as wake #10 left it. Re-ran the terminal predicate rather than trusting the commit log:

grep -E "^## VETO" machine-docs/REVIEW-redfix.md | grep -v CLEARED   -> (empty)
grep -n "^## DONE" machine-docs/STATUS-redfix.md                     -> 10:## DONE — 2026-07-09T00:18Z
git log 0539c46..origin/main -- machine-docs/REVIEW-redfix.md        -> (empty)
ls machine-docs/{BUILDER,ADVERSARY}-INBOX.md                          -> none

## DONE intact, M1 PASS @2026-06-18T01:18Z + M2 PASS @2026-07-09T00:18Z (37 min old), no standing VETO, no unconsumed Adversary verdict, no inbox message, no coordination file stranded at the repo root. Merge target redfix-m2-harness@b5f2b10 unchanged. Nothing to build, nothing to claim, nothing to re-verify.

main.go still sits untracked in the repo root and I still cannot account for it (see wake #10). Left exactly as found — not committed, not deleted, not added to .gitignore, since hiding it would be worse than leaving it visible. Surfaced to the operator.

Wake #12 — 2026-07-09T00:58Z — consumed review(redfix)@83758f5 (break-it probe, no finding); loop stays stopped

Watchdog pinged on a review(...) commit. Read it before re-checking anything: 83758f5 is Adversary re-confirmation #11 — a break-it probe, not a verdict change. It attacked the one claim in b5f2b10 that is empirical rather than diff-checkable: STATUS's "MIGRATION: none required — on cc-ci /var/lib/ci-warm/keycloak/ contains only last_good". Had that been false, the new warmsnap._assert_slot_not_foreign() guard would raise inside the live-warm reconciler's snapshot() on every stateful keycloak auto-upgrade — the F-redfix-4 fix wedging the shared OIDC provider it was meant to protect. The right thing to probe, and it was cheap for me to assert and expensive for me to have wrong.

Cold read-only on cc-ci, the Adversary found the claim TRUE as observed: keycloak/ (and traefik/) hold only last_good; all 17 seeded canonicals carry domain=warm-<recipe>…, none is in WARM_DOMAINS, so canonical_slot(r) == r and the guard passes on every legacy meta — backward compatible, no migration. It also confirmed the dropped meta["recipe"] key has no consumer, slot↔stack is 1:1 across all 21 enrolled recipes, and prune_stale's keep-set covers all 17 dirs while skipping keycloak/ structurally.

Two facts I had not stated and now hold in the record: traefik/ is likewise unseeded (harmless — same never-seeded shape as keycloak, and it is not in WARM_DOMAINS), and prune_stale derives a stale stack's volumes via warm.stable_domain(<ns>), so pruning a canon-keycloak/ canonical could only ever remove warm-canon-keycloak… volumes, never the live warm-keycloak… ones. Consequence-4 is structural on real disk, not just in the unit tests.

No new finding. No verdict changed. Nothing to fix. A confirmatory probe does not reopen a gate. Predicate re-checked after the pull:

grep -E "^## VETO" machine-docs/REVIEW-redfix.md | grep -v CLEARED   -> (empty)
grep -n "^## DONE" machine-docs/STATUS-redfix.md                     -> 10:## DONE — 2026-07-09T00:18Z
git rev-parse redfix-m2-harness                                      -> b5f2b104e6dd…
ls machine-docs/{BUILDER,ADVERSARY}-INBOX.md                          -> none

## DONE intact, M1 + M2 PASS fresh, no standing VETO, no inbox, merge target redfix-m2-harness@b5f2b10 unchanged. Terminal condition still met. Loop remains stopped. main.go still untracked and unexplained (wakes #10/#11) — untouched.

Wake #13 — 2026-07-09T01:11Z — post-reboot re-verification; phase stays closed, loop stays stopped

Builder loop re-invoked from a cold boot (fresh context). Rather than trust wake #12's summary, I re-derived the DONE predicate from the artifacts:

git pull --rebase                                   -> Already up to date (fc059ef)
grep '^## DONE' machine-docs/STATUS-redfix.md       -> "## DONE — 2026-07-09T00:18Z"
grep -E '^#+ .*(M1|M2).*(PASS|VETO)' REVIEW-redfix  -> M1 PASS @2026-06-18T01:18Z
                                                       M2 PASS @2026-06-18T07:06Z (superseded)
                                                       M2 (F-redfix-4 remedy) PASS @2026-07-09T00:18Z,
                                                         VETO CLEARED, F-redfix-4 CLOSED
tail REVIEW-redfix.md                               -> Adversary break-it probe @83758f5:
                                                       "no new finding. DONE stands, no VETO."
ls machine-docs/{BUILDER,ADVERSARY}-INBOX.md        -> none
git rev-parse --verify b5f2b10                      -> exists, on redfix-m2-harness + origin/
git merge-base --is-ancestor b5f2b10 origin/main    -> false (NOT merged — operator merges)

All six recipes (discourse, mattermost-lts, mumble, bluesky-pds, gitea, keycloak) carry a fix + verification in STATUS-redfix.md; no standing exception; nothing merged. Terminal condition met on every clause. No work to do — no new verdict, no inbox, no open blocking finding. Loop remains stopped.

Still unexplained and still deliberately untouched: the untracked 16-line Go hello-world main.go in the repo root of this Python harness clone (wakes #10/#11/#12). Nothing in this phase created it; no plan or state file calls for it. Not committed, not deleted, not gitignored — hiding it would be worse than leaving it visible. Operator's call. Note it is untracked, so it cannot reach a cold clone or affect the redfix-m2-harness@b5f2b10 merge target.

Wake #14 — 2026-07-09T01:17Z — consumed review(redfix)@0fee249 (re-confirmation #12, no finding); loop stays stopped

Watchdog pinged on a review(...) commit. Pulled and read it: Adversary post-reboot re-confirmation #12, whose one new check is merge-target staleness — the failure mode where the Builder pushes to redfix-m2-harness after the M2 PASS, leaving STATUS naming a sha that is no longer the branch tip. It reports the tip is still b5f2b10. No finding, no verdict change, no VETO, no inbox.

I did not take that on trust; re-derived it against the remote from my own clone:

git ls-remote origin 'refs/heads/redfix*'   -> b5f2b104e6dd... refs/heads/redfix-m2-harness
grep 'Merge target' STATUS-redfix.md        -> redfix-m2-harness @ b5f2b10
git merge-base --is-ancestor b5f2b10 origin/main -> false (not merged; operator merges)

Remote tip == the M2-PASS sha == the STATUS merge target. No post-PASS commits on the branch; verified artifact and merge target have not diverged. This is expected — I have pushed nothing to redfix-m2-harness since the remedy, and wakes #10–#13 touched only JOURNAL-redfix.md on main.

Nothing to act on: no verdict to answer, no finding to fix, no gate to claim. Terminal condition still met on every clause (## DONE + fresh M1/M2 PASS + no standing VETO). Loop remains stopped. main.go still untracked and unexplained (wakes #10–#13) — untouched.

Wake #15 — 2026-07-09T01:26Z — consumed review(redfix)@e9bd483 (re-confirmation #13, no finding); loop stays stopped

Watchdog pinged on a review(...) commit. Pulled and read it: Adversary post-reboot re-confirmation #13. Its one genuinely new check is the phase plan's negative "nothing merged" DoD clause — the failure mode every prior re-confirmation was blind to, because a Builder who self-merged redfix-m2-harness would leave STATUS-redfix.md and REVIEW-redfix.md byte-for-byte intact. A file-only predicate cannot see that violation, so it had to be checked against the remote. Good catch on the gap; the clause holds.

Re-derived both of its claims from my own clone rather than trusting the verdict text:

git rev-parse --short origin/redfix-m2-harness            -> b5f2b10   (== M2-PASS sha; staleness clean)
git merge-base --is-ancestor b5f2b104e6dd… origin/main    -> exit 1    (NOT an ancestor => not merged)
# and, stronger than the Adversary checked: no commit in origin/main's last 20 touches any path
# outside machine-docs/ — main carries only review(redfix)/journal(redfix) coordination commits.

The merge remains the operator's to make, as the phase plan requires. I have pushed nothing to redfix-m2-harness since the F-redfix-4 remedy.

Terminal condition re-checked from artifacts, not from wake #14's summary: ## DONE at STATUS-redfix.md:10; M1 PASS + M2 PASS (F-redfix-4 CLOSED); the sole ## VETO heading is annotated CLEARED and is therefore not standing; no machine-docs/BUILDER-INBOX.md. Nothing claimed, nothing pending, no finding to fix. Loop remains stopped.

main.go (repo root, untracked, 16-line "Hello, World!" Go HTTP server, mtime 2026-07-09T00:01) is still present and still unexplained — carried forward from wakes #10–#14. It is not mine, is not referenced by any recipe, harness, or Nix module, and is not a coordination file, so the FILE-LOCATION RULE does not apply to it. I am deliberately NOT deleting it: I did not create it, and a stray file I cannot account for is information, not litter. It cannot affect the DoD (untracked => absent from the Adversary's cold clone). Surfaced to the operator instead.

Wake #16 @2026-07-09 — consumed review(redfix)@1182aa0 (re-confirmation #14); no verdict change

Watchdog pinged on the review(...) commit. Pulled; no BUILDER-INBOX.md. Verdict: no finding, DONE stands, no VETO. Nothing claimed, nothing pending, so nothing to act on.

Re-derived the one novel claim independently rather than trusting the Adversary's transcript — its five-ref drift sweep newly asserted that the redfix fixes survive at the force-pushed discourse tip. Cold shallow clone of discourse-official-image in scratchpad (removed after):

live tip                                         → ede639916c1f08e098767178e444f5edd8668363
show FETCH_HEAD:compose.yml | grep image:discourse → image: discourse/discourse:3.5.3   (M2 claim holds)
show FETCH_HEAD:compose.smtpauth.yml | grep -c sidekiq → 0                (F-redfix-1 claim holds)

Also re-derived the two standing claims: redfix-m2-harness tip == b5f2b10 (the M2-PASS sha), and git merge-base --is-ancestor b5f2b10 origin/main → exit 1, so nothing from redfix is reachable from origin/main. The ## VETO heading is annotated CLEARED (F-redfix-4 CLOSED); ## DONE present. The discourse sha rot is already recorded in STATUS-redfix.md's 2026-07-08 evidence addendum (line 150), so no STATUS edit is warranted — the pins there are annotated as verify-by-CONTENT, not by sha.

Terminal condition still met. Loop remains stopped. Open item is the operator-gated merge, not mine.

Unrelated observation (not a phase artifact). An untracked main.go (16-line hello-world Go HTTP server, md5 8bb1bb1a55fa3c2d568e29fcf8cc5206) sits in BOTH clones, /srv/cc-ci-orch/cc-ci and /srv/cc-ci/cc-ci, stamped 2026-07-09 00:01. Never committed (git log --all -- main.go empty); repo has no go.mod and no tracked .go files. Git cannot propagate an untracked file between clones, so an actor outside the two loops wrote it to both. Inert: nothing listening on :8080, no go process. Left in place (I did not create it; not mine to delete) and surfaced to the operator. Invisible to the Adversary's cold clone, so no DoD item is affected.

Wake #17 — 2026-07-09T01:52Z — consumed review(redfix)@32fefb2 (re-confirmation #15); corrected a stale count in STATUS; filed B-redfix-6

Consumed the Adversary's re-confirmation #15 (32fefb2): "existing canonicals unchanged" verified exhaustively rather than only at keycloak's slot. Verdict: no finding, DONE stands, no VETO. Terminal condition therefore unchanged and the loop remains stopped.

The verdict carried no finding, but it did expose a factual error in an artifact I own: STATUS-redfix.md said "the 15 existing canonicals". I did not take the Adversary's replacement number on trust either — re-derived every figure first-hand:

ssh cc-ci: /var/lib/ci-warm/ slots=20  seeded(canonical.json)=17
           spared = alerts, keycloak, traefik   (reconciler dirs, no canonical.json)
           keycloak/ contains only `last_good`  → migration-none holds
           canon-keycloak/ absent                → created on first post-merge run
git show b5f2b10:runner/harness/warm.py      → WARM_DOMAINS = {"keycloak": ...}  (singleton)
enrolled (WARM_CANONICAL=True) at b5f2b10    → 21, keycloak among them  ⇒ 20 unchanged

So "15" was wrong twice over: the count is 17, and the guarantee was never a count in the first place. It is structural — r not in warm.WARM_DOMAINS — which is why a singleton WARM_DOMAINS makes the re-key provably keycloak -> canon-keycloak and nothing else. Rewrote the STATUS line in the structural form, with 17 as an observation rather than as the load-bearing fact. STATUS is not sha-pinned, so this is free.

WHY I did not fix the identical "15" in the canonical_ns() docstring (canonical.py:52 @ b5f2b10), which is the actual origin of the error and the copy a future reader will hit: amending it moves the branch tip off b5f2b10 — the exact sha the M2 PASS was granted against, that STATUS pins, and that every drift sweep re-checks (tip == b5f2b10). Trading a verified gate sha for a comment fix is a bad bargain, and it would force a re-verification round for zero behaviour change. Filed as B-redfix-6 (cosmetic, docs-only) to be folded into the next commit that moves the branch for a substantive reason — B-redfix-5 being the obvious candidate. Recording it beats both silently fixing it and silently dropping it.

Node untouched (the only remote command was a read-only ls/find under /var/lib/ci-warm). Nothing claimed, nothing pending, no ADVERSARY-INBOX written — this needs no verdict. DONE stands.

Wake #18 — 2026-07-09T02:00Z — consumed review(redfix)@6997c29 (re-confirmation #16); corrected an Adversary fact; found + filed B-redfix-7 (cleartext bot credential on the node)

Consumed re-confirmation #16 (6997c29): refs swept cold via ls-remote, B-redfix-6 independently confirmed REAL-but-cosmetic, and my choice to defer it rather than move b5f2b10 explicitly endorsed ("I do not want the branch moved to fix a comment"). No finding, no VETO, DONE stands, loop stays stopped.

The verdict carried one claim I had not verified, so I checked it instead of inheriting it — precisely because that verdict also retracted a false NOT-MERGED it had earlier derived from a bad-path probe. A sweep that has just caught itself citing a broken command is the last place to take a new fact on trust.

Adversary fact wrong, conclusion right. It reported /etc/cc-ci as holding redfix-m2-harness @b96b8a4c. Actually:

symbolic-ref HEAD              → refs/heads/main
rev-parse HEAD                 → d11f8f56  (2026-06-17)
rev-parse redfix-m2-harness    → b96b8a4c  ← a stale LOCAL BRANCH, not HEAD
cat-file -e b5f2b10            → absent    ✔ (this half of the claim holds)

HEAD is main; the stale redfix-m2-harness merely exists as a local branch alongside it. The inference drawn from it — stale checkout, not branch drift; remote authoritative; NOT-MERGED holds — is untouched. Sent as ADVERSARY-INBOX (non-gate side-channel) rather than left to rot, because these re-confirmations cite one another and an uncorrected fact in a clean sweep gets inherited by the next.

Incidental find, and the more serious one. While inspecting that checkout: /etc/cc-ci/.git/config is mode 644 with the Gitea bot password embedded in cleartext in the origin URL. Before deciding what to do I calibrated rather than reflexively alarming — awk over /etc/passwd shows no non-root login users, so today only root can read it. So: a missing layer of defence, not a live compromise. Also /etc/cc-ci is a real dir (not a /nix/store symlink) with no systemd unit referencing it — orphaned, undeclared state that nothing runs from, which is itself a violation of "keep server state Nix-declared".

WHY I did not fix it. Two independent rules bite. The Gitea bot credential is a Class-A1 EXTERNAL infra input (§4.4) — I do not rotate or invent those. And I did not create /etc/cc-ci, so deleting or chmod'ing it is not mine to do unasked; a chmod 600 would also be undeclared drift papering over the real defect (credential-in-URL). Filed as B-redfix-7 with the remedy ladder for the operator. It is out of redfix scope and does NOT reopen the phase — the DoD concerns the six canon-sweep failures, all of which remain fixed and verified.

Node untouched: every command this wake was read-only (git -C … rev-parse/cat-file/symbolic-ref, stat, awk, readlink). Nothing claimed. DONE stands.

Wake #19 — 2026-07-09T02:08Z — consumed review(redfix)@14c7dee (re-confirmation #17); my B-redfix-7 severity rationale was WRONG on the axis; corrected after re-deriving the Adversary's replacement

Consumed re-confirmation #17 (14c7dee). It accepted my /etc/cc-ci correction (after re-deriving it rather than taking it on trust), confirmed B-redfix-7 as real + LOW, filed it Adversary-side as A-redfix-1 — and corrected my reasoning, which was wrong. No finding, no VETO, DONE stands, loop stays stopped.

Where I was wrong. I justified "severity LOW" with "no non-root login users, so only root can read it." That reasons about the wrong thing. A process does not need a login shell to run as uid 1000, and mode 644 lets any uid read the file regardless. I did not accept the Adversary's replacement on trust either — the whole point of this exchange is that neither of us inherits a fact. Re-derived all three legs myself:

setpriv --reuid=1000 --clear-groups … /etc/cc-ci/.git/config   → SUCCEEDS   ⇒ my premise was false
non-root procs sharing pid1's mnt-ns (4026531841)              → exactly 1: dbus-daemon, uid 4
uid-1000 quarkus java (the live warm-keycloak), pid 2327794    → mnt-ns 4026533305 ≠ host
  ls /proc/2327794/root/etc/cc-ci                              → No such file or directory

So the credential is contained by mount-namespace isolation, not by the absence of login accounts. Same LOW verdict, entirely different load-bearing reason. Rewrote the B-redfix-7 severity bullet accordingly and cross-referenced A-redfix-1; the remedy ladder is unchanged.

Why the distinction matters rather than being pedantry. "No non-root users" reads as a static fact about the box. Mount-namespace containment is a fragile property: it evaporates the moment a non-root host-ns daemon appears, or /etc is bind-mounted into any container. My wrong rationale would have made the risk look permanently dormant; the true one shows it is one config change away from live. That is now the stated reason to fix it, and it strengthens rather than weakens the case for the operator to act.

Also noted: the Adversary recorded a near-miss against itself — its setpriv probe ran in the host ns, so it nearly concluded "internet-facing Keycloak can exfiltrate the bot credential," which is false. Its own tally is that two of its last three probes gave plausible answers for invalid reasons, both caught by re-running the probe against the thing it claimed to measure rather than by re-checking the conclusion. Mine this wake was the same class of error, caught the same way. Worth carrying forward: verifying a conclusion is nearly worthless; verifying that the probe measures what it purports to is what catches these.

Scope unchanged: A-redfix-1 / B-redfix-7 is pre-existing infra state, touches no DoD item, and the credential is a Class-A1 external input — the operator's to rotate. The phase is not reopened. All commands this wake were read-only (readlink, pgrep, ls /proc/…, and a zero-byte head -c 0 for the mode check). Nothing claimed. DONE stands.

Wake #19 addendum — SECURITY: the Adversary's own commit leaked the live bot password into git; redacted + escalated

My pre-commit credential guard fired on machine-docs/BACKLOG-redfix.md — and I pushed anyway, because I had written it grep … || echo "clean" ; git add …. The ; meant the warning printed and the commit ran regardless. A guard whose failure does not halt the pipeline is decoration. Re-run &&-gated.

Traced it: the credential was NOT mine. git log --all -S'<pattern>' attributes it to exactly one commit, 14c7dee — the Adversary's re-confirmation #17 — whose A-redfix-1 "Repro" block inlined the actual bot password as a grep pattern. git branch -r --contains 14c7dee → origin/main. It was pushed.

The documentation of the finding was worse than the finding. A-redfix-1 is a secret in one 0644 file on one host, reachable only from pid1's mount ns (I re-derived that this wake and agree). The repro line moved the same secret into a git repo: replicated to every clone (both loops' + the node's /etc/cc-ci), readable by anyone with mirror read access, durable in history — and the credential grants push to recipe-maintainers/*, so it now sits inside a repo it can write to. Worse on reachability, replication, and durability simultaneously.

Redacted at HEAD (e99e2b3): the repro now greps autonomic-bot:, which I verified returns the same 1 on the node, so the Adversary's finding lost no verifiability. I edited their ## Adversary findings section, which I otherwise treat as read-only — removing a live credential outranks that convention, and I changed nothing else. Notified via ADVERSARY-INBOX.

What I deliberately did NOT do: rewrite history. The secret lives at 14c7dee permanently; excising it needs --force, which the standing rules forbid outright and which would break both clones mid-phase. Redaction stops propagation; it does not undo disclosure. So the honest status is disclosed, contained, not undone — and the only real remediation is rotating the autonomic-bot password, a Class-A1 external input I must not touch. Filed B-redfix-8; escalated to the operator as the one action that actually closes this. Once rotated, 14c7dee is inert and no rewrite is ever needed — which is precisely why rotation, not history surgery, is the right lever.

Scope: no DoD item is touched. The phase is not reopened; DONE stands. But this is the first thing in several wakes that genuinely needed a human, and it would have gone unnoticed had the guard not fired — and nearly did anyway, because the guard was toothless.

Wake #20 — consumed review(redfix)@cf65019 (re-confirmation #18) + BUILDER-INBOX (Adversary wake #20): the leaked credential is PUBLIC. Escalated B-redfix-8 LOW→HIGH.

The Adversary acked my wake-#19 redaction and handed me one material new fact: the git.autonomic.zone mirror is public, so 14c7dee serves the cleartext password to the unauthenticated internet. A fact this consequential I do not relay on trust — I reproduced it myself, and deliberately with a probe that CANNOT smuggle in credentials:

python3 urllib.request.urlopen(.../raw/commit/14c7dee/machine-docs/BACKLOG-redfix.md)
  → HTTP 200, 33408 bytes, body starts "# BACKLOG", contains the A-redfix-1 "Repro" block,
    and the cleartext password IS in the body.
(no auth handler, no netrc, no git cred helper — a bare urllib GET, so the 200 is truly anonymous)
HEAD /raw/branch/main and /raw/commit/e99e2b3  → 200, password ABSENT (my redaction is live publicly).

First attempt used curl, absent on this host — it printed "no match", a false negative from a broken probe. That is the exact failure mode both loops have now logged three times over (curl-missing here; the Adversary's host-ns setpriv; its bad-path git -C). Caught the same way every time: not by re-reading the conclusion but by re-running with a probe that actually measures the thing (python3 urllib, with sanity assertions that I fetched the real file and not an error page). Naming it again because the lesson keeps paying out: verify the instrument, not just the answer.

Consequence: A-redfix-1/B-redfix-8 goes LOW → HIGH. The LOW rating rested on mount-namespace containment — which is simply irrelevant once the identical secret is on the open web, no host access required. Updated B-redfix-8 accordingly and made rotation URGENT. Redaction at HEAD stops propagation but cannot un-publish: a value served publicly must be presumed captured/crawled, so even the forbidden --force rewrite would not close it. Only rotation does, and only the operator can (Class-A1, §4.4).

Consumed the BUILDER-INBOX by git rm (deletion = ack). No verdict change: the leak is orthogonal to the canon-sweep DoD and clearable only by an external action, so neither of us vetoes — a VETO would wedge the phase without helping. M1/M2 PASS, DONE stands, no VETO. This is the one item in this whole tail of re-confirmations that genuinely needs the human, so I am surfacing it to the operator plainly and not burying it in the loop files.

Wake #21 — 2026-07-09T02:14Z — no-op re-check; one recorded fact corrected (main.go mtime)

Re-derived the terminal predicate first-hand rather than trusting the last entry: ## DONE present in STATUS-redfix.md (2026-07-09T00:18Z); the only ## VETO in REVIEW-redfix.md is the F-redfix-4 one, explicitly CLEARED @2026-07-09T00:18Z; no machine-docs/BUILDER-INBOX.md. Terminal condition met. Nothing to build, nothing to re-verify. Loop stays stopped.

Correction to wake #7. That entry described the stray untracked root main.go as having mtime 2026-06-23. That is false as of now, and I am not going to inherit it:

stat -c 'mtime=%y ctime=%z uid=%u mode=%a size=%s' main.go
-> mtime=2026-07-09 00:01:57.842057029 +0000  ctime=2026-07-09 00:01:57.842057029 +0000
   uid=1000 mode=644 size=281
sha256=bdbc3bf167cd20f30c00880005f4f994f17f3722660973c9c65c7bf33e81ffaf

ctime == mtime means the inode was created (or wholly rewritten) at 2026-07-09T00:01:57Z — during today's wake sequence, ~16 min before the DONE marker — not in June. So the file is not the inert leftover "stable since June" that wake #7 implied; something wrote it mid-phase. I cannot attribute the writer: git log --all -- main.go is empty (0 commits, never tracked on any ref), and no plan or phase file references it. It is the sole .go file in a Python/Nix harness.

Also corrected: earlier entries treat /srv/cc-ci/cc-ci and /srv/cc-ci-orch/cc-ci as two clones that each grew a copy. They are one path — same inode (3254604), same dev, same absolute git dir. There is exactly one main.go, not two.

Position unchanged, and unchanged for the right reason. I did not write it, so I surface it rather than delete it — that call is the operator's. It is harmless to verification regardless: git status --porcelain main.go -> ?? (untracked, not ignored), so it cannot reach a cold clone or perturb any Adversary cold-verify. It contains no secret (16-line net/http "Hello, World!" on :8080). The only thing that changed is that "unexplained and old" is now "unexplained and recent" — which is mildly more interesting, not more dangerous.

No DoD item touched. DONE stands, no VETO, loop remains stopped. The one genuinely open item remains B-redfix-8 (HIGH): the live Gitea bot password is on origin/main and served unauthenticated to the public internet. Redaction at HEAD is done; history cannot be rewritten without --force (forbidden), so operator credential rotation is the only remediation and it is outside my authority.

---

Wake #23 @2026-07-09 — Adversary caught a bad number in my "I verified this myself" claim

Consumed BUILDER-INBOX (Adversary @02:24Z, re-confirmation #22). It reports that both my records of the B-redfix-8 public-exposure probe say the fetch returned 33080 bytes, and that the blob is 33408.

I did not take that on trust, for the same reason the Adversary did not take mine:

git cat-file -s 14c7dee:machine-docs/BACKLOG-redfix.md   -> 33408
bare anonymous urllib GET of raw@14c7dee                 -> status 200, 33408 bytes, body starts b'# BACKLOG'

Both agree at 33408, and they must: a raw blob at a fixed commit sha is immutable, so served size == object size. The Adversary is right. Corrected BACKLOG-redfix.md B-redfix-8 and JOURNAL line 1261 to 33408, and added the git cat-file -s cross-check to the backlog entry so the next reader can tell a real leak from an error page without re-deriving it.

WHY this mattered, and what the defect actually was. I prefaced that probe with "a fact this consequential I do not relay on trust — I reproduced it myself." I did run the probe. What I did not do was read the number off the probe: 33080 is a digit-permutation of 33408, i.e. I transcribed it. So the sentence was true about the act and false about the evidence, which is the worst combination — it spends the credibility of first-hand verification on a figure that was never checked. Everything else in that write-up does reproduce (200, anonymous, # BACKLOG body, A-redfix-1 repro block present, secret present; HEAD and e99e2b3 clean).

The failure mode is specifically load-bearing here and not a typo I can wave off. B-redfix-8 is the entry an operator reads before deciding whether to rotate a live, world-readable, push-capable credential. An operator who re-runs the probe, gets 33408, sees 33080 documented, and concludes "I must have fetched an error page" stands down from a real HIGH-severity leak. A wrong number in security evidence doesn't just fail to help — it argues actively for the wrong action.

Note the symmetry worth keeping: the Adversary's own wake-#20 REVIEW narrative had inherited 33080 by citing my report instead of measuring the blob, and it says so. Two loops, same shape of error — a number that was cited rather than measured. The correction only landed because someone finally ran git cat-file -s. Cite the artifact, not the report about the artifact. That is now the third time this phase a claim survived review purely because it was re-derived from the object rather than from the prose about the object.

No DoD item touched. No verdict change. DONE stands, no VETO. B-redfix-8 remains OPEN and re-confirmed LIVE this wake: still served publicly (HTTP 200, anonymous), still unrotated. History cannot be scrubbed without --force (forbidden by the guardrails), so operator rotation remains the only remediation — it is not mine to close.

Wake #23 — no-op re-check; upgraded B-redfix-8's evidence from "still served" to "still VALID"

Phase state on entry: ## DONE (2026-07-09T00:18Z), M1+M2 Adversary PASS, no standing VETO, ## Blocked (none), no BUILDER-INBOX. Nothing to build. Loop stops.

Re-probed B-redfix-8 first-hand rather than inheriting it. Two facts, and the second is new:

1. Still public. Anonymous urllib.request.urlopen of raw@14c7dee → HTTP 200, 33408 bytes, body starts # BACKLOG. git cat-file -s 14c7dee:machine-docs/BACKLOG-redfix.md → 33408. Served size == object size, so the fetch is the blob, not an error page. This independently re-confirms the corrected figure from 0f174a8 (the old 33080 was the transcription slip).

2. Still the LIVE credential — not merely still reachable. Every prior record, mine and the Adversary's, establishes public reachability (HTTP 200) and then asserts "unrotated". Those are different claims, and only the first was ever measured. An operator reading "the file is still up" can reasonably wonder whether the value went inert when someone rotated. So I tested the actual proposition: read GITEA_PASSWORD from /srv/cc-ci/.testenv (the live value) and checked membership in the 14c7dee blob → True. The string the public internet is serving is the credential the harness authenticates with today. Recorded as sha256[:16] = 3fcea78925015fc9 — a commitment the operator can re-derive to confirm rotation later without either of us printing the secret again.

I got the re-check command wrong on the first write, and the phase's own rule caught it. The version I initially put in BACKLOG was broken twice: the nested quote-escaping was an unterminated string literal, and hashlib.sha256() was handed a str instead of bytes. It would have failed for the operator at the exact moment they were trying to establish whether a live HIGH credential leak had been closed — the worst possible time for a copy-paste to raise SyntaxError. It only got caught because "verify every change; paste command + output; no 'should work'" applies to the commands I document, not just the ones I run. I then round-tripped it: extracted the line back out of the committed file with sed and eval-ed that, rather than re-running the version in my head. Prints 3fcea78925015fc9. Same lesson as above, one turn later and self-inflicted: the artifact, not the report about the artifact.

The methodological point is the same one that has now bitten this phase four times: cite the artifact, not the report about the artifact. "Unrotated" had been true-by-repetition for several wakes without anyone comparing the served bytes against the live secret. It happened to be true. It was not known to be true.

Nothing actionable by me. Rotation is Class-A1 (operator-only); excising history needs --force (forbidden), and a value already served publicly must be presumed captured regardless. DONE stands, no VETO. B-redfix-8 stays OPEN on operator rotation.

Wake #24 — 2026-07-09 — phase closed; B-redfix-8 re-measured, still live

Phase is ## DONE (STATUS-redfix.md), M1+M2 both carry Adversary PASSes, REVIEW-redfix.md's only ## VETO is explicitly [CLEARED @2026-07-09T00:18Z], and both inboxes are absent (consumed). The Adversary loop stopped at de126b3. Nothing in the Definition of Done is outstanding, so this loop stops too.

Before stopping I re-measured B-redfix-8 rather than inheriting the claim, because "still live" is exactly the kind of assertion that rots between wakes — and this phase has now been bitten four times by citing a report instead of the artifact:

live digest:  3fcea78925015fc9          # == the committed commitment ⇒ NOT rotated
HTTP 200, 33408 bytes                   # unauthenticated urlopen of raw/commit/14c7dee/…
live password present verbatim in public blob: True

So as of this wake the leaked autonomic-bot Gitea password is still unrotated and still served to the unauthenticated public internet from the historical commit 14c7dee. Both load-bearing numbers reproduce (33408, matching git cat-file -s). B-redfix-8 stays OPEN, blocking on the operator.

Nothing here is actionable by me: rotation is a Class-A1 external input (operator-only), and excising the blob needs a history rewrite + --force, which the standing rules forbid and which would not undo disclosure anyway — a value served publicly must be presumed captured. Loop stopping.

wake #25 — 2026-07-09T02:51Z — reboot after closure; no work to do, loop re-stopped

Rebooted into a phase that is already terminal. I re-checked the terminal condition rather than assuming it survived the restart, and it did, on every limb:

machine-docs/STATUS-redfix.md   -> "## DONE — 2026-07-09T00:18Z"
machine-docs/REVIEW-redfix.md   -> M1 PASS @2026-06-18T01:18Z, M2 PASS @2026-07-09T00:18Z, VETO [CLEARED]
machine-docs/*INBOX*            -> absent (both consumed)
git status                      -> up to date with origin/main, no gate claimed

No gate is claimed, no finding is open against the DoD, and the Adversary loop stopped at de126b3. So there is nothing here for a Builder to advance.

I did not re-probe B-redfix-8 this wake. Wake #24 (e9a9652) measured it first-hand seven minutes ago — digest 3fcea78925015fc9, HTTP 200/33408, live value verbatim in the public blob — and re-running the same probe against a production mirror seven minutes later would be repetition, not evidence. It stays OPEN, HIGH, blocking on operator rotation, which is the one thing neither loop may do (Class-A1 external input; excising the blob needs a --force history rewrite the rules forbid, and would not undo public disclosure).

One housekeeping observation, filed not fixed: main.go (16-line "Hello, World!" HTTP server) sits untracked and un-gitignored at this clone's root. It is not mine and it is not a secret, but it is the same shape as F-redfix-2 — a stray root file that the next git add -A would push to origin. It is in fact the file the Adversary's F-redfix-2 cold replay saw staged. Left in place (not mine to delete); noted so a future phase either gitignores it or removes it deliberately.

Future reboots of this loop need no action. The DoD is met and Adversary-verified; the only open item is operator-only. Stopping again.

---

Wake #27 — 2026-07-09T03:28Z — reboot, no-op, terminal re-confirmed

Rebooted again post-closure. Re-checked the terminal condition from the artifacts rather than from prior commit messages: ## DONE @2026-07-09T00:18Z in STATUS-redfix.md; M1 PASS + M2 PASS in REVIEW-redfix.md; no standing VETO (both ## VETO grep hits are the same finding — the original heading now annotated [CLEARED … NOT standing], plus the ## VETO CLEARED @2026-07-09T00:18Z record); no ADVERSARY-INBOX.md / BUILDER-INBOX.md; no gate CLAIMED; working tree clean but for the known untracked main.go.

Did not re-probe B-redfix-8. The Adversary measured it 17 minutes ago (e75a24a, 03:10Z: still LIVE, digest 3fcea78925015fc9 unrotated; still PUBLIC, HTTP 200/33408). Probing the same public mirror again on that timescale produces a duplicate reading, not new evidence. It remains OPEN, HIGH, operator-only: rotation is a Class-A1 external input, and excising the blob would need the --force history rewrite the rules forbid — which would not undo the public disclosure anyway.

main.go still untracked and un-gitignored at the clone root. Unchanged from wake #25: not a secret, not mine to delete, same shape as F-redfix-2. Still filed, still not fixed.

No work exists for the Builder. Loop stopping; future reboots need no action.

Wake #28 @2026-07-09T03:40Z — reboot no-op; terminal condition re-verified from artifacts

Re-derived the terminal condition rather than trusting the commit log:

• ## DONE — 2026-07-09T00:18Z in STATUS-redfix.md.
• Both gates carry a fresh Adversary PASS: M1 @2026-06-18T01:18Z, M2 @2026-07-09T00:18Z.
• Both ## VETO headings in REVIEW-redfix.md are the same F-redfix-4 finding — line 553 is stamped [CLEARED @2026-07-09T00:18Z ... NOT standing] and line 649 is the clearing entry itself. No standing VETO.
• No ADVERSARY-INBOX.md, no BUILDER-INBOX.md, no gate CLAIMED. Tree clean but for untracked main.go.

Did not re-probe B-redfix-8. The Adversary measured it 30 minutes ago (e75a24a, 03:10Z: still LIVE, digest 3fcea78925015fc9 unrotated; still PUBLIC, HTTP 200/33408). A fourth reading of the same public mirror on that timescale is a duplicate measurement, not evidence. It stays OPEN, HIGH, operator-only: rotation is a Class-A1 external input, and excising the blob would require the --force history rewrite the rules forbid — which would not undo the public disclosure regardless.

main.go still untracked and un-gitignored at the clone root. Not a secret, not mine to delete (F-redfix-2 shape). Still filed, still not fixed.

No work exists for the Builder. Loop stopping; further reboots need no action.

wake #29 — 2026-07-09T03:47Z — reboot, no-op. Terminal condition holds.

Third consecutive reboot into a closed phase. Re-verified from artifacts only (no re-measurement):

• ## DONE — 2026-07-09T00:18Z at STATUS-redfix.md:10.
• M1 PASS @2026-06-18T01:18Z + M2 PASS @2026-07-09T00:18Z. Both ## VETO headings in REVIEW-redfix.md (lines 553, 649) belong to the single CLEARED F-redfix-4. No standing VETO.
• No ADVERSARY-INBOX.md, no BUILDER-INBOX.md, no gate CLAIMED. Tree clean but for untracked main.go.
• Adversary's newest entry is its own wake-#26 re-probe @03:09Z; nothing after it.

Did not re-probe B-redfix-8: the Adversary read it 37 minutes ago (e75a24a — still LIVE, digest 3fcea78925015fc9 unrotated; still PUBLIC, HTTP 200/33408). Wakes #27 and #28 declined for the same reason. A fifth reading on this timescale measures nothing. It stays OPEN, HIGH, operator-only: rotation is a Class-A1 external input, and excising the blob needs the --force rewrite the rules forbid — which would not undo the public disclosure anyway.

main.go still untracked and un-gitignored at the clone root. Not a secret, not a coordination file, not mine to delete. Filed, unchanged.

No work exists for the Builder. Loop stopping; further reboots need no action.

wake #30 — 2026-07-09T04:38Z — reboot, no-op. Terminal condition holds.

Fourth consecutive reboot into a closed phase. Re-verified from artifacts only (no re-measurement):

• ## DONE — 2026-07-09T00:18Z at STATUS-redfix.md:10.
• M1 PASS @2026-06-18T01:18Z + M2 PASS @2026-07-09T00:18Z. Both ## VETO headings in REVIEW-redfix.md (lines 553, 649) belong to the single CLEARED F-redfix-4. No standing VETO.
• No ADVERSARY-INBOX.md, no BUILDER-INBOX.md, no gate CLAIMED. Tree clean but for untracked main.go.
• Newest commit is the Adversary's own wake-#30 re-probe 8fcae0c @04:13Z (25 min ago): B-redfix-8 still LIVE (digest 3fcea78925015fc9 unrotated) and still PUBLIC (HTTP 200 / 33408 bytes, cleartext served unauthenticated). Nothing after it.

Did not re-probe B-redfix-8. The Adversary measured it 25 minutes ago; wakes #27–#29 declined on the same grounds. A sixth reading at this cadence is duplicate measurement, not evidence. It stays OPEN, HIGH, operator-only: rotation is a Class-A1 external input, and excising the blob needs the --force history rewrite the rules forbid — which would not undo the public disclosure anyway.

main.go still untracked and un-gitignored at the clone root. Not a secret, not a coordination file, not mine to delete. Filed, unchanged.

No work exists for the Builder. Loop stopping; further reboots need no action.

wake #31 — 2026-07-09T04:44Z — reboot no-op; terminal condition holds

Re-verified from artifacts only, no probing:

• STATUS-redfix.md → ## DONE — 2026-07-09T00:18Z.
• REVIEW-redfix.md → M1 PASS @2026-06-18T01:18Z, M2 PASS @2026-07-09T00:18Z. Every VETO string in the file is prose about the one cleared F-redfix-4 (or an explicit "no VETO"). No standing VETO.
• No ADVERSARY-INBOX.md / BUILDER-INBOX.md. No gate claimed. Nothing awaiting me.

Did not re-probe B-redfix-8. The Adversary measured it 31 minutes ago (8fcae0c): still LIVE (digest 3fcea78925015fc9 unrotated), still PUBLIC (HTTP 200 / 33408 bytes, cleartext, unauthenticated). Wakes #27–#30 declined on the same grounds and nothing has changed the underlying fact. A seventh reading at this cadence is duplicate measurement, not evidence. Stays OPEN, HIGH, operator-rotation-only — rotation is a Class-A1 external input, and excising the blob needs the --force history rewrite the rules forbid, which would not undo the public disclosure regardless.

main.go still untracked and un-gitignored at the clone root. Not a secret, not a coordination file, not mine to delete. Filed, unchanged.

No work exists for the Builder. Loop stopping; further reboots need no action.

wake #32 — 2026-07-09T04:50Z — reboot no-op; terminal condition holds

Re-verified from artifacts, not memory: STATUS-redfix.md carries ## DONE @2026-07-09T00:18Z; REVIEW-redfix.md shows M1 PASS @2026-06-18T01:18Z and M2 PASS @2026-07-09T00:18Z; both ## VETO strings in REVIEW are the same CLEARED F-redfix-4 (the heading and its "VETO CLEARED" clearance) — no standing VETO. No BUILDER-INBOX.md, no ADVERSARY-INBOX.md, no gate CLAIMED.

Did not re-probe B-redfix-8. The Adversary measured it 37 minutes ago (8fcae0c): still LIVE (digest 3fcea78925015fc9 unrotated), still PUBLIC (HTTP 200 / 33408 bytes, cleartext, unauthenticated). Wakes #27–#31 declined on identical grounds. An eighth reading at this cadence is duplicate measurement, not evidence. Stays OPEN, HIGH, operator-rotation-only — rotation is a Class-A1 external input, and excising the blob needs the --force history rewrite the rules forbid, which would not undo the public disclosure regardless.

main.go still untracked and un-gitignored at the clone root. Not a secret, not a coordination file, not mine to delete. Filed, unchanged.

No work exists for the Builder. Loop stopping; further reboots need no action.

Wake #33 — reboot no-op

Re-verified the terminal condition from artifacts, not from memory:

• machine-docs/STATUS-redfix.md:10 — ## DONE — 2026-07-09T00:18Z, M1 + M2 both PASS.
• machine-docs/REVIEW-redfix.md — every remaining VETO string is prose referring to the single cleared F-redfix-4. No standing veto.
• No ADVERSARY-INBOX.md / BUILDER-INBOX.md. No gate CLAIMED.

Did not re-probe B-redfix-8. The Adversary measured it in 8fcae0c and it was unchanged then; re-reading it on a reboot cadence is duplicate measurement, not evidence. It stays OPEN, HIGH, operator-rotation-only — the leaked value needs a rotation only the operator can perform, and nothing I do to that item from here changes its state.

main.go remains untracked at the repo root. It is not a file this phase created and not mine to delete.

Loop stopping.

Wake #34 —

Reboot no-op. Terminal condition re-verified from artifacts only: ## DONE @00:18Z, M1+M2 PASS, every VETO string in REVIEW-redfix.md is the one CLEARED F-redfix-4, both inboxes absent, no gate CLAIMED. Did NOT re-probe B-redfix-8 and did NOT re-derive anything already in the record. B-redfix-8 stays OPEN, HIGH, operator-rotation-only. main.go still untracked at repo root — not created by this loop, not mine to delete.

This is the last journal entry for phase redfix. The phase is DONE; the loop is stopping and will not self-reschedule. Further reboots that find this state should stop immediately rather than re-journal.

Wake #35 — watchdog ping on review(redfix)@55899e6; found a real gap in STATUS and closed it

The Adversary's verdict was "no VETO, terminal condition intact" — on its face a no-op, and my last four wakes had correctly refused to manufacture work out of one. But I read the finding it re-probed instead of pattern-matching on the verdict, and that surfaced something none of the previous no-op wakes had noticed:

STATUS-redfix.md never mentioned B-redfix-8. The finding lived in BACKLOG, REVIEW, and JOURNAL. STATUS — the artifact an operator actually opens — carried ## DONE and nothing else. So the single action this phase requires from a human (rotate the leaked Gitea bot password) was invisible in the one place a human looks. "DONE with a HIGH credential leak open" reads as "DONE" if the leak isn't on the page.

That is not noise-work, and the distinction is the point: the previous wakes were right to skip re-probing B-redfix-8 (duplicate measurement), and this wake was right to act (missing artifact). The trigger for work is a gap between what's true and what's recorded — not the arrival of a ping.

Added an ## ⚠ OPERATOR ACTION REQUIRED block to STATUS with WHAT / WHERE / HOW / EXPECTED + remedy, kept to the STATUS layer (no rationale — this file is it). Re-ran both repro commands first rather than copying them forward, because a stale repro in the operator's file is worse than no repro, and this exact item already ate one transcription slip (33080 → 33408, caught by the Adversary at re-confirmation #22):

$ git cat-file -s 14c7dee:machine-docs/BACKLOG-redfix.md
33408
$ python3 -c '...sha256(GITEA_PASSWORD)[:16]...'
3fcea78925015fc9

Both match. The credential is still unrotated and still served. No DoD item touched, no gate claimed, no verdict change: M1+M2 PASS stand, ## DONE stands, no VETO. B-redfix-8 stays OPEN, HIGH, operator-only — I cannot rotate a Class-A1 input (§4.4) and cannot rewrite published history (--force forbidden), so rotation by the operator is the only thing that revokes the exposure. main.go still untracked, still not mine to delete. Loop stops again.

Builder wake — 2026-07-09, watchdog ping on review(redfix) wake #39 (f65dfe7)

Pulled and read the verdict. Nothing to advance: no standing VETO (the sole ## VETO heading is the CLEARED F-redfix-4), no new finding, no BUILDER-INBOX.md, no gate CLAIMED, ## DONE intact, M1+M2 PASS. Wake #39 is a re-confirmation, not a verdict against me.

But it exposed a false-close hazard in my own STATUS text, so the ping was not a no-op. STATUS said, flatly: "A different digest ⇒ rotated ⇒ close B-redfix-8." The Adversary then documented that a probe run over ssh cc-ci cannot open /srv/cc-ci/.testenv (it lives on the orchestrator), hashes empty input, and prints e3b0c44298fc1c14 — a different digest, with no rotation. My own instructions therefore licensed closing an open HIGH public-credential exposure on the strength of a broken probe. Given that a rebooted loop or the operator reads STATUS (not REVIEW) for the remedy, that is a live risk of silently declaring a still-live, push-capable, publicly-served credential "inert."

Verified all three digest claims myself before writing them down (no "should work"):

$ printf '' | sha256sum | cut -c1-16
e3b0c44298fc1c14                      # the empty-input tell
$ ssh cc-ci 'ls /srv/cc-ci/.testenv'
ls: cannot access '/srv/cc-ci/.testenv': No such file or directory   # absent on cc-ci, as claimed
$ python3 -c '...sha256(GITEA_PASSWORD)[:16]...'   # on the orchestrator
3fcea78925015fc9                      # still UNROTATED

So STATUS now (a) says where to run the probe, (b) names e3b0c44298fc1c14 as the empty-input tell that must NOT close the item, (c) disambiguates the three digests that have been conflated twice now (value sentinel 3fcea789…, whole-blob 1994fd8d…, empty e3b0c442…), and (d) records that the exposure is confirmed at value level — the live push-capable password appears verbatim in the public body, which is a stronger claim than the blob-equality prior wakes established.

This is the one edit worth making. I did not manufacture a re-confirmation commit: the phase is terminal and re-asserting an unchanging state on a 10-minute cadence is the churn both loops stood down from. B-redfix-8 stays OPEN/HIGH/operator-only; sole close path is a correctly computed sentinel that differs from 3fcea78925015fc9.

Unrelated, surfaced not touched: an untracked main.go (281 B, Go hello-world HTTP server on :8080) sits at the repo root of this clone. Not mine, foreign to this Nix repo, absent from git and from origin/main. Left in place — not committed (would pollute the repo), not deleted (not mine to destroy).

---

Wake #42 — 2026-07-09 — watchdog ping on the Adversary's wake-#41 verdict

Pulled cf1455f. Verdict is a CONFIRM of the merge guidance (b5f2b10, not 07fc6d4), no new finding, no VETO, M1+M2 PASS stand. So: nothing to unblock, no gate to re-claim. But the verdict carried a merge precondition that existed only in REVIEW-redfix.md, which is Adversary-owned and which the operator has no reason to read before merging. STATUS is the operator-facing artifact I own. Moving it across is the work.

I did not take the Adversary's word for it. Re-derived every leg from source at b5f2b10 in a detached worktree, plus one live check on the node:

1. restore() really does call the new guard (warmsnap.py:209), and snapshot() calls it too (:158). So b5f2b10 adds a raise path to warmsnap — it does not only remove one.
2. warm_reconcile.py rollback: abra.undeploy → wait_undeployed → restore → deploy_version at :534-537, all outside the try/except at :520-525. Confirmed — this is B-redfix-5 as filed.
3. cc-ci live slot: ls -A /var/lib/ci-warm/keycloak/ → last_good only; no snapshot/, no meta.json; no canon-* slot. So read_meta → None → guard passes. Composition unreachable today.

Then the part the Adversary understated. It framed the risk as "B-redfix-5 × the new guard can wedge the live OIDC provider on rollback". But the guard is also reached via snapshot() at warm_reconcile.py:514 — and :512-514 is abra.undeploy → wait_undeployed → snapshot, likewise outside the try/except. That site is on the normal upgrade path: it fires on every stateful auto-upgrade, no unhealthy release required. So the blast radius of the precondition is strictly larger than "on rollback", and B-redfix-5 as originally filed named only one of the two sites. Widened it in BACKLOG + DEFERRED.

Proving it needed care, and my first probe was wrong. Seeding a foreign meta.json and calling restore() gave FileNotFoundError: 'docker', not SnapshotError — I briefly read that as "the guard isn't wired into restore after all", which would have contradicted the Adversary. It didn't: restore() runs _assert_undeployed(domain) at :205 before the guard at :209, and that shells out to docker, which is absent on the orchestrator. The probe died upstream of the thing under test. snapshot() puts the guard first (:158, before _assert_undeployed), which is why that half raised cleanly and the asymmetry looked like a code difference rather than a probe artifact. Re-ran with the two docker-dependent preconditions stubbed and the guard itself untouched: both paths raise on a foreign meta; an absent meta passes; a self-consistent meta (domain == warm-keycloak.ci.commoninternet.net) passes. That is the real behaviour.

Same class of error as the e3b0c44298fc1c14 empty-input tell from wake #39 — a probe that fails before reaching the assertion prints something that reads like a finding. Worth the note: on this orchestrator docker, curl, wget, awk, pytest are all absent, and python3 is not on PATH at the store path STATUS's verify-(1) cites. Reach for the stdlib and check what actually executed.

Corrected the reachability wording in BACKLOG + DEFERRED, which both claimed F-redfix-4 "supplied the only reachable trigger and that is now closed". The trigger is not closed — it is unreachable for a different reason (the live slot is empty), and that reason is a property of node state, not of the fix. Stating it the old way would let an operator conclude the composition is impossible by construction. It isn't: one pre-fix canonical seed writes domain=warm-canon-keycloak… into the live slot and arms both wedge sites.

No VETO to answer, no DoD item touched, no gate to claim. Docs-only. B-redfix-8 not re-probed — still operator-rotation-only, and wake #41 deliberately stopped re-probing it for the same reason I do here.

Wake #43 — 2026-07-09 — Adversary CONFIRMED the widening; folded its three new facts into STATUS

543d566: the Adversary re-derived site (a) cold from b5f2b10 and confirmed it — "my wake-#41 framing was too narrow; the Builder is right. Widening CONFIRMED, do not revert." It also independently reproduced my docker-absent probe artifact on restore() (dies at _assert_undeployed:205, upstream of the guard at :209) and accepted the reachability correction. Nothing to revert; my open question is answered.

It asserted three facts I had not established. Symmetry demands I not take those on faith either, so I re-derived all three from source before letting them into STATUS:

• Nothing upstream catches. reconcile() spans :448-551; its only try is :520-525, wrapping deploy_version + wait_healthy. Sole caller main():556 is a bare result = reconcile(argv[1]). So a raise at :514/:536 escapes to SystemExit with keycloak already undeployed. Confirmed.
• Arming condition is real. git show 07fc6d4:tests/keycloak/recipe_meta.py → WARM_CANONICAL = True. A pre-fix canonical seed genuinely writes the foreign domain into the bare-recipe slot. Confirmed.
• Blast radius one unit. WARM_DOMAINS == {"keycloak"}; SPECS["keycloak"]["stateful"] is True, SPECS["traefik"]["stateful"] is False. Confirmed.

Two of those change what an operator would do, so they went into STATUS: the wedge does not self-heal (no code path redeploys; recovery is manual), and it is bounded to keycloak. The third (arming condition real) tightened the existing "do not run a pre-fix canonical seed" line from a hypothetical into a fact. The "blast radius" note also stops the precondition reading as a whole-fleet hazard, which would be its own kind of overstatement.

Both loops have now independently derived the same structure from source and agree. B-redfix-5 stays deferred, correctly scoped to both call sites; it still needs the operator's safety call (redeploy last_good on unrestored data vs. die loudly), which is exactly why it was filed rather than fixed inline. No VETO, no gate, no DoD item touched. Phase remains ## DONE. B-redfix-8 untouched — operator rotation.

Wake #43 (Builder) — 2026-07-09T08:3xZ — I accept A-redfix-2 and A-redfix-3; both of my claims were wrong

The Adversary probed two operator-facing claims I wrote (68b51d5, 42bc4e4) and refuted both. I re-derived each from source/node before editing — not taking the verdict on faith — and both hold up.

A-redfix-2 (mechanism wrong, conclusion right). I wrote "no code path redeploys it. Recovery is manual." The conclusion (keycloak stays down) is right; the reason I gave is false. I reasoned from the absence of a try/except around the abra.undeploy() sites and stopped there, never asking what the next invocation of reconcile() does. It takes the fresh-deploy branch (:471-479): abra.undeploy() doesn't remove the app .env, so current_version still resolves while is_deployed is False. That branch redeploys and never calls warmsnap, so the foreign meta can't block it. What actually stops the healing is that nothing re-runs the unit — warm-keycloak.service is Type=oneshot/RemainAfterExit=true with no timer and no Restart= (confirmed on the node). So it heals on the next reboot/nixos-rebuild switch and re-wedges on the next due upgrade, because the foreign snapshot/meta.json is still in the slot.

Why this correction matters and is not cosmetic: my wording implied the remedy was to redeploy keycloak. An operator who did that would see it come back, conclude it was fixed, and get re-wedged at the next upgrade. The actual remedy is deleting the foreign snapshot/. My text hid the recurrence and pointed at the wrong fix. It also mis-describes the failure's shape: intermittent flake, not permanent outage.

A-redfix-3 (evidence void). I claimed main.go was in "both clones", inferring it was written by something outside git. But /srv/cc-ci is a symlink to /srv/cc-ci-orch (ls -la /srv/) — the two paths are the same inode (3254604, links=1). One file, seen twice. The inference carries zero information; the file's origin is still unexplained. Lesson: I treated two paths as two clones without stat-ing them. Cheap check, skipped.

Both edits are docs-only, in STATUS-redfix.md. No DoD item, gate, or verdict is affected: ## DONE stands, M1 + M2 PASS stand, merge target b5f2b10 unchanged, no VETO. The B-redfix-5 merge precondition itself is unchanged and still correct — A-redfix-2 corrects only the mechanism + remediation sentences beneath it.

Also re-probed B-redfix-8 this wake with the sound (local, not ssh) probe: 3fcea78925015fc9 → still unrotated, and the empty-input control printed e3b0c44298fc1c14, distinct — so the probe read the real file rather than silently hashing nothing. B-redfix-8 remains OPEN/HIGH/operator-rotation-only. Class-A1 credential; I must not rotate it.

---

2026-07-09T08:45Z — loop close-out

Booted, re-read the phase plan + state. Found the phase already at ## DONE (2026-07-09T00:18Z) with M1 PASS (2026-06-18T01:18Z, re-confirmed through Adversary wake #43) and M2 PASS (2026-07-09T00:18Z, F-redfix-4 remedy cold-verified, VETO CLEARED). No standing ## VETO, no gate CLAIMED, no inbox in either direction, HEAD == origin/main.

The one thing I found worth doing: the ## DONE block enumerated F-redfix-1/2/3/4 as CLOSED but said nothing about A-redfix-1/2/3. My previous commit (f64d102) had accepted A-redfix-2 and A-redfix-3 and corrected the affected passages in the body — the mechanism/remediation sentences of the B-redfix-5 precondition, and the retraction of the main.go "both clones" evidence — but an operator reading only the DONE header would see three findings referenced nowhere. Closed that gap in 1d3a70b. It is bookkeeping, not a new claim: all three are non-VETO and outside the DoD, which is the Adversary's own wake-#43 verdict, not my assertion.

Why I am stopping rather than continuing to poll: the standing rule is "stop the loop only when STATUS says ## DONE," and it does, with both gates Adversary-verified and no open blocking finding. The two items that remain open are, by construction, not mine to close:

• B-redfix-8 (HIGH) — the leaked Gitea bot password in 14c7dee. Class-A1 external infra input (plan §4.4); rotation is the operator's, and history rewrite needs --force, which the standing rules forbid. Nothing I can do here advances it.
• B-redfix-5 (deferred, non-blocking) — the two post-undeploy() warmsnap calls outside the upgrade's try/except. Adversary concurred it is not a VETO. It is recorded with a merge precondition for b5f2b10, which is the correct disposition for a phase whose DoD it does not touch.

Neither is unblocked work, so there is no backlog item left to hold. Untracked main.go left in place per the joint Builder/Adversary position: neither of us created it, it is inert (no go.mod, go not installed, nothing on :8080, in no commit on any ref), and the guardrails say I do not delete files I did not create.

---

2026-07-09T08:55Z — wake #44: the Adversary refuted itself, and I had inherited the error

Re-opened the loop on a watchdog ping. The Adversary withdrew half of A-redfix-2 and filed A-redfix-4: warm_reconcile.py has a second invoker — nightly-sweep.timer (OnCalendar=Sun, Persistent=true) → nightly_sweep.roll_warm_infra() → subprocess warm_reconcile.py keycloak. So the "no re-trigger / heals on reboot" mechanism I wrote into STATUS at f64d102 is false.

The part worth recording is how it got in. At f64d102 I accepted A-redfix-2 and copied its two probe commands into STATUS as the HOW-to-verify. Both probes grep the app name — git grep warm-keycloak -- nix/ and list-timers | grep -i keycloak. The timer is named nightly-sweep. So both return empty while a timer drives the thing weekly: they return the all-clear for a reason unrelated to the property under test. I ran them, saw empty, and treated that as confirmation. It confirmed nothing. The Adversary named the class correctly — it is the same shape as its own e3b0c44298fc1c14 empty-input tell on B-redfix-8.

The lesson is not "the Adversary was wrong" — it self-corrected. It is that I re-ran probes handed to me rather than deriving probes from the property. The property is "what invokes reconcile()", so the probe must grep the callee (git grep warm_reconcile -- runner/ nix/), which finds both invokers immediately. A probe supplied alongside the claim it is meant to test is not independent evidence.

So this wake I verified every leg of A-redfix-4 myself before editing:

• git grep warm_reconcile -- runner/ nix/ → nix/modules/warm-keycloak.nix:26 and runner/nightly_sweep.py:60. Two invokers.
• systemctl cat nightly-sweep.timer → OnCalendar=Sun *-*-* 03:00:00, Persistent=true; next 2026-07-12 03:05:34, last 2026-07-05 03:04:50.
• The timing inference, which is the load-bearing one: warm-keycloak.service ExecMainStartTimestamp = 2026-06-17 17:29:31, but /var/lib/ci-warm/keycloak/last_good mtime = 2026-07-05 03:04:51.209 — 18 days after the unit last ran, 1.2 s after the timer fired. write_last_good() has four call sites (b5f2b10:478,484,491,527), all inside reconcile(). Nothing else can write that file. ⇒ the sweep ran it.
• Silence: roll_warm_infra() binds rc and only prints it (:59-62); every write_alert() site (:493,500,503,539) is downstream of the snapshot() raise at :514; main():556 does not catch.
• Cannot self-arm: git show origin/main:tests/keycloak/recipe_meta.py → WARM_CANONICAL = False. Only 07fc6d4 and b5f2b10 carry True. /etc/cc-ci is origin/main, so Sunday's sweep cannot arm the trap.
• ExecStart → /nix/store/fcqh…-runner/warm_reconcile.py; harness/warmsnap.py is in that same derivation, so a switch re-runs reconcile iff runner/** changed.

I also caught a line-number slip while writing: the write_alert / write_last_good sites I first wrote were main's coordinates, but every other ref in that STATUS block is b5f2b10's (they sit two lines apart). Fixed to branch coordinates and said so in the text, since an operator will git show the branch.

Then I falsification-tested each new probe before committing it, which is the whole point: the sound probe must print both invokers, the fresh-deploy grep must return 0 warmsnap hits, the timer must print its schedule. All did.

Severity re-recorded, scope unchanged. B-redfix-5's consequence is worse than I had it — a silent weekly alternating SSO outage rather than a one-shot manual-recovery wedge — but its reachability is unchanged and still nil on this node. I did not fix it: the two-line try/except widening would move the merge target off b5f2b10, which is the sha the M2 PASS was given against, and the Adversary's asks were explicitly "STATUS text only, no code". Fixing it is the operator's call at merge time; the remedy sketch in BACKLOG-redfix.md now also notes roll_warm_infra() should propagate the rc so the failure is visible at all.

Re-anchored the merge precondition per ask 3. "Slot clean at merge time" named the wrong landmark — a git merge executes nothing, and the slot can only go foreign when a canonical seed runs from a tree with the enrollment but without canonical_ns(). That tree is 07fc6d4, and only 07fc6d4. b5f2b10 ships both halves, so the precondition is self-maintaining. The operator-facing rule is now "never deploy 07fc6d4", and the slot check is kept as a cheap not-armed assertion rather than as the rule itself.

Nothing reopens: M1 + M2 PASS stand, ## DONE stands, no VETO, merge target b5f2b10 unchanged.

Wake (post-#45 ping) — 2026-07-09T09:1xZ — folded A-redfix-1 into the B-redfix-8 operator remedy

Adversary wake #45 (f0372b8): no VETO, no asks, A-redfix-2/3/4 CLOSED. Nothing reopens. But it carried a fact my STATUS remedy did not reflect: A-redfix-1 and B-redfix-8 are the same credential, and rotation does not scrub the second copy.

Why I re-derived instead of adopting. A-redfix-2 got into STATUS at f64d102 because I took the Adversary's mechanism and its probe commands on authority; the probes passed for a reason unrelated to the property under test. So every claim here was checked first-hand before it went into STATUS:

• stat -c "%a %U:%G" on /etc/cc-ci/.git/config → 644 root:root; the origin URL carries userinfo.
• nix/hosts/cc-ci-hetzner/configuration.nix:7 documents git clone … /etc/cc-ci ⇒ manual clone, not nix-generated ⇒ no nixos-rebuild regenerates or scrubs it. This is the load-bearing bit: it is why rotation alone leaves exposure #2 standing.
• Value identity, the claim I most wanted to test myself: the password extracted from the .git/config userinfo, hashed on cc-ci with sha256sum (python3 is absent there — that constraint is already recorded), → 3fcea78925015fc9; GITEA_PASSWORD from the orchestrator's .testenv → 3fcea78925015fc9. Same value. Control: printf '' | sha256sum → e3b0c44298fc1c14, so neither reading is the empty-input artifact. This is what upgrades "two similar-looking secrets" to "one credential, two exposures, unrotated."
• Blast radius: git grep -ln GITEA_PASSWORD → only scripts/bootstrap-drone-oauth.sh in the runner; recipe-mirror-sync.sh:30 pushes with an OAuth token from /run/secrets/bridge_gitea_token. So rotation cannot break mirror sync — there is no operational reason for the operator to delay it.

Why "strip the userinfo" rather than "re-embed the new password." Re-embedding recreates a 0644 world-readable copy of a live push-capable credential the moment rotation completes — the fix would reintroduce the finding. I checked the strip is safe before prescribing it: anonymous ls-remote of the mirror returns f0372b8 HEAD rc=0 with credential.helper= cleared, and the clone only fetches. The cc-ci-secrets submodule URL already has no userinfo, so this is the existing pattern, not a new one.

Scope discipline. A-redfix-1 is out of redfix scope and outside the DoD — I did not "fix" it (it is Class-A1 external infra state; the credential is not mine to rotate, and editing /etc/cc-ci is server state outside this phase's remit). What I changed is the operator instructions, so following them closes both exposures instead of one. ## DONE is untouched; no gate reopens; STATUS gained WHAT/HOW/EXPECTED/WHERE only.

Wake (post-#46 ping) — 2026-07-09T09:1xZ — consumed BUILDER-INBOX; "strip is safe" survived refutation

Adversary wake #46 (3f27cc6) cold-confirmed all three claims from my heads-up. Notably it took the one I explicitly asked it to break — "stripping the userinfo does not wedge the node" — and attacked it three ways (push through origin? submodule fetch dependency? automated pull?). All three failed to refute. That is the outcome I wanted: I prescribed an operator command that runs against a real node, and I did not want it resting on my single ls-remote.

The submodule angle was the one I had not independently checked before prescribing the strip — I had only observed that .git/config:15 carries no userinfo, and inferred submodule update would be fine. The Adversary actually fetched cc-ci-secrets.git anonymously (rc=0) and confirmed it. Had that failed, my step 3 would have broken nixos-rebuild on cc-ci. Worth remembering: "the config looks like it doesn't need auth" is an inference, not a probe. I got the right answer with an incomplete proof, which is luck, not method.

Also caught: /root/.git-credentials exists on the node but no credential.helper is wired at global or system scope, so it is inert and the anonymous ls-remote successes are genuinely credential-free — my original probe could in principle have been silently authenticated. It wasn't, but I hadn't ruled it out.

Folded the three refutation results into STATUS step 3 as facts, so the operator sees why the command is safe rather than taking it on my word. Inbox deleted (consumed signal).

Adversary incidental, recorded not actioned: cc-ci-secrets.git is anonymously readable, but secrets.yaml is SOPS ciphertext (ENC[AES256_GCM…, sops: block), key not in the repo — public storage of ciphertext is the intended model. Not a leak, not a finding, no action. Noting it only so a future reader who greps a public repo named "secrets" doesn't re-litigate it.

Phase state unchanged: ## DONE stands, M1+M2 PASS stand, no VETO. A-redfix-1 + B-redfix-8 remain OPEN/HIGH/operator-rotation-only; sentinel 3fcea78925015fc9 ⇒ still unrotated.

Wake #47 — 2026-07-09T09:2xZ — loop termination check. No work performed.

Re-entered the loop, re-read the phase SSOT and plan.md §6.1/§7/§9, and checked whether anything is Builder-actionable. Nothing is. Verified rather than assumed:

• Tree is pushed. git rev-list --left-right --count origin/main...HEAD → 0 0; HEAD == origin/main == 6eef4ff. No un-pushed fix exists only on this host.

• No standing VETO. Adversary wakes #43–#46 each close with "no VETO; ## DONE stands; M1 + M2 PASS stand." Latest (#46, 3f27cc6) cold-confirmed all three of my B-redfix-8/A-redfix-1 remedy claims and failed to refute the strip-userinfo step on three attack paths.

• No inbox. machine-docs/BUILDER-INBOX.md absent — consumed at 6eef4ff.

• B-redfix-8 sentinel re-run, and the probe is sound this time. On the orchestrator (NOT over ssh cc-ci, which is the known-bad probe):

  python3 -c 'import hashlib,re;v=re.search(r"^GITEA_PASSWORD=(.*)$",open("/srv/cc-ci/.testenv").read(),re.M).group(1).strip().strip(chr(34)).strip(chr(39));print(hashlib.sha256(v.encode()).hexdigest()[:16])'
  → 3fcea78925015fc9
  printf '' | sha256sum   → e3b0c44298fc1c14   (empty-input control — DIFFERENT, so the read succeeded)
  

  The control differing from the sentinel is what makes this a real observation and not the e3b0c44298fc1c14 artifact: the file was opened and a non-empty value was hashed. Still unrotated. Rotation is Class-A1 (plan §4.4) — I must not rotate or invent it, and --force history rewrite is forbidden, so this cannot be closed by the Builder under any approach.

• Untracked main.go still present (16-line Go hello-world, repo root, outside git). Unchanged since I flagged it at 42bc4e4; A-redfix-3 corrected my "both clones" evidence (/srv/cc-ci is a symlink to /srv/cc-ci-orch, so it is one file, seen twice). Origin remains unexplained. Left in place deliberately: I did not create it, nothing references it, and deleting an unexplained artifact destroys the only evidence of how it got there. It is untracked, so it cannot reach origin/main; risk inert.

DoD is met and Adversary-verified on both gates. Per the standing rule ("stop the loop only when STATUS says ## DONE"), I am ending the self-paced loop here. The only open item, B-redfix-8 / A-redfix-1, is a single credential with two exposures and is operator-only; its full remedy (rotate → update .testenv → strip the userinfo from /etc/cc-ci/.git/config + chmod 600) is written out in STATUS-redfix.md and has been independently cold-verified by the Adversary.

Wake #48 — 2026-07-09T09:36Z — terminating the loop (re-confirmed, not re-asserted)

Wake #47 concluded the loop should end; the loop fired again, so I re-checked the one condition that could have changed state rather than restating the conclusion.

B-redfix-8 rotation sentinel — still unrotated. Re-ran the sound probe on the orchestrator (where .testenv lives — NOT over ssh cc-ci, which would hash the empty string):

value-digest:                                 3fcea78925015fc9
printf '' | sha256sum | cut -c1-16   →        e3b0c44298fc1c14   (control — differs ⇒ the read succeeded)
exposure 2 (/etc/cc-ci/.git/config userinfo)  3fcea78925015fc9   (same credential, still embedded)

Both exposures still carry the live, push-capable credential. Rotation is Class-A1 (plan §4.4): I must not rotate or invent it, and history rewrite needs --force, which the standing rules forbid. No approach exists by which the Builder can close this — it is operator-only, and the full remedy is written out in STATUS-redfix.md, Adversary-cold-verified at wake #46 (3f27cc6).

Termination conditions, all met:

• STATUS-redfix.md reads ## DONE (2026-07-09T00:18Z), M1 + M2 both with fresh Adversary PASSes.
• No standing ## VETO in REVIEW-redfix.md; the Adversary re-affirmed "no VETO, ## DONE stands" as recently as wake #46 (09:08Z), ~28 min before this wake.
• No BUILDER-INBOX.md, no ADVERSARY-INBOX.md, no gate CLAIMED and awaiting a verdict.
• Working tree clean apart from the untracked, inert main.go (deliberately left in place; see wake #47).

Per the standing rule — "Stop the loop only when STATUS.md says ## DONE" — I am stopping the self-paced loop now. Remaining work on this phase is the operator's: merge redfix-m2-harness @ b5f2b10 (never 07fc6d4), and close B-redfix-8 / A-redfix-1.

---

Wake #49 — 2026-07-09 — watchdog ping on the Adversary's close-out (4d96413); both loops now closed.

Woken by the watchdog on a review(...) commit. Pulled and read it: it is the Adversary's own loop close-out (wake #49, 09:42:34Z), not a verdict requiring Builder action. It carries no VETO, no new finding, and no CLAIMED gate to answer; no BUILDER-INBOX.md accompanied it. The Adversary stopped for the mirror-image reason I did — counterparty terminated, DoD handshake complete, only operator-only items left.

Acted by verifying, not by changing anything. Re-probed the B-redfix-8 rotation sentinel on cc-ci before stopping (hash only, secret never echoed): still 3fcea78925015fc9 → unrotated. Consistent with wakes #47/#48 and with the Adversary's wake-#49 text. Nothing committed beyond this entry; no source, config, or state file changed, and main.go is left untracked and untouched per wake #47.

Terminal state of phase redfix — both loops closed, in agreement:

• STATUS ## DONE (2026-07-09T00:18Z); M1 PASS @2026-06-18T01:18Z, M2 PASS @2026-07-09T00:18Z; no VETO.
• Adversary findings F-redfix-1/2/3/4 and A-redfix-2/3/4 all CLOSED.
• A-redfix-1 + B-redfix-8 OPEN / HIGH / operator-rotation-only — one live push-capable Gitea credential, exposed twice (public mirror blob 14c7dee; /etc/cc-ci/.git/config 0644). Remedy cold-verified safe at wake #46: rotate GITEA_PASSWORD and strip the userinfo via git remote set-url origin — do not re-embed the new password, which would recreate the 0644 exposure. Neither item gates ## DONE.

Operator's to perform: merge redfix-m2-harness @ b5f2b10 (never 07fc6d4 — that tree enrolls keycloak without canonical_ns() and is the only state that can arm the silent weekly wedge); merge the four recipe PRs; rotate + strip the credential. Loop remains stopped.

---

Wake #50 — Builder reboot on a closed phase; verified terminal state; no action taken

A fresh Builder process booted into phase redfix. Re-checked the stop condition from a cold read rather than trusting the prior wake's word, and it holds:

• machine-docs/STATUS-redfix.md → ## DONE (2026-07-09T00:18Z); ## Blocked section is (none).
• machine-docs/REVIEW-redfix.md → M1 PASS @2026-06-18T01:18Z, M2 PASS @2026-07-09T00:18Z. The single ## VETO header (line 553) is explicitly annotated [CLEARED @2026-07-09T00:18Z … NOT standing] and is followed by ## VETO CLEARED @2026-07-09T00:18Z (F-redfix-4). No standing VETO.
• Adversary wake #49 is a formal LOOP CLOSE-OUT; no gate CLAIMED, no BUILDER-INBOX.md, no ADVERSARY-INBOX.md. No counterparty remains to hand off to.
• No coordination file at the repo root (FILE-LOCATION RULE satisfied); working tree clean apart from the untracked main.go noted below.

Per the standing rule — "Stop the loop only when STATUS.md says ## DONE" — the loop stays stopped. I did not re-open, re-verify, or re-run any DoD item: every one already carries a fresh Adversary PASS, and re-running deploys on the shared swarm for a closed phase would burn the node for no verdict.

Sentinel re-probe (the one live open item), run this wake:

$ P=$(grep -oP '^GITEA_PASSWORD=\K.*' /srv/cc-ci/.testenv | tr -d '"\r')
$ printf '%s' "$P" | sha256sum | cut -c1-16
3fcea78925015fc9

Unchanged from wake #48, and not the empty-string artifact e3b0c44298fc1c14 (so this is a real read of a real value, not a silent failure to read the file). A-redfix-1 / B-redfix-8 remain OPEN / HIGH — the Gitea credential is still unrotated. Operator-only; gates nothing.

main.go (untracked, 16-line hello-world HTTP server) is still present and still unexplained — the subject of the retracted A-redfix-3 evidence. I again left it untouched: I did not create it, so per the guardrails I neither commit nor delete it. Inert; affects no gate.

Nothing for the Builder to do. The three outstanding actions are all the operator's: merge redfix-m2-harness @ b5f2b10 (never 07fc6d4), merge the four recipe PRs, and rotate GITEA_PASSWORD

• strip the userinfo via git remote set-url origin (do not re-embed). Loop remains stopped.

Wake #51 — Builder reboot on a closed phase; terminal state re-verified; no action

Rebooted into phase redfix. Re-derived the terminal state from the files rather than trusting the previous wake's summary:

• machine-docs/STATUS-redfix.md:10 → ## DONE — 2026-07-09T00:18Z; ## Blocked = (none).
• machine-docs/REVIEW-redfix.md → M1 PASS @2026-06-18T01:18Z, M2 PASS @2026-07-09T00:18Z. The lone ## VETO header (:553) is annotated [CLEARED … NOT standing] and followed by ## VETO CLEARED @2026-07-09T00:18Z (F-redfix-4). No standing VETO.
• Adversary wake #49 is a formal loop close-out. No gate CLAIMED, no BUILDER-INBOX.md, no ADVERSARY-INBOX.md. No counterparty to hand off to.
• Merge target intact: b5f2b10 = fix(keycloak): key warm state by stack namespace, not bare recipe (F-redfix-4). (Never 07fc6d4.)

Per the standing rule — "Stop the loop only when STATUS.md says ## DONE" — I did not start a loop. No DoD item was re-opened or re-run: each already carries a fresh Adversary PASS, and re-running deploys on the shared swarm for a closed phase would burn the node for no verdict.

Sentinel re-probe (the one live open item), run this wake:

$ P=$(grep -oP '^GITEA_PASSWORD=\K.*' /srv/cc-ci/.testenv | tr -d '"\r')
$ printf '%s' "$P" | sha256sum | cut -c1-16
3fcea78925015fc9

Unchanged from wakes #48/#50, and distinct from the empty-string artifact e3b0c44298fc1c14 — so this is a real read of a real value, not a silent failure to read the file. git remote get-url origin still carries autonomic-bot:<secret>@ userinfo. A-redfix-1 / B-redfix-8 remain OPEN / HIGH. Operator-only; gates nothing.

main.go (untracked, 281-byte hello-world HTTP listener) still present, still unexplained, still inert. Left untouched: I did not create it, so per the guardrails I neither commit nor delete it.

Nothing for the Builder to do. All three outstanding actions are the operator's: merge redfix-m2-harness @ b5f2b10, merge the four recipe PRs, and rotate GITEA_PASSWORD + strip the userinfo via git remote set-url origin (do not re-embed). Loop remains stopped.

Wake #52 — 2026-07-09 — Builder reboot on closed phase; terminal state re-verified; loop STOPPED

Third consecutive watchdog reboot onto a phase that is already closed (cf. wakes #50, #51). Re-verified the terminal state cold before acting, rather than assuming the prior wakes' conclusions:

• grep -c '^## DONE' machine-docs/STATUS-redfix.md → 1.
• grep -c '^## VETO' machine-docs/REVIEW-redfix.md → 2, which on inspection are both historical: L553 ## VETO [CLEARED @2026-07-09T00:18Z … NOT standing] and L649 ## VETO CLEARED @2026-07-09T00:18Z (F-redfix-4). No standing VETO. Noting this because a naive grep -c '^## VETO' on this file returns a non-zero count forever — any future automated stop-condition check must read the headings, not count them.
• M1 PASS @2026-06-18T01:18Z, M2 PASS @2026-07-09T00:18Z; Adversary closed its own loop at wake #49.
• Working tree clean but for the untracked main.go, already flagged in STATUS as unexplained/inert and left in place per guardrails (do not delete/commit files I did not create).

Open items (operator-only, neither blocks ## DONE, no DoD item depends on them): A-redfix-1 + B-redfix-8 — one live push-capable Gitea credential exposed two ways; sentinel 3fcea78925015fc9 last probed unrotated. Merge target is redfix-m2-harness@b5f2b10, never 07fc6d4.

Per the standing rule "Stop the loop only when STATUS.md says ## DONE" — it does, with a fresh Adversary PASS on every DoD item and no standing VETO. Re-waking cannot advance operator-only work. Stopping the loop via ScheduleWakeup(stop) instead of scheduling wake #53. Nothing further to build; a future merge or new phase should start a fresh loop that cold-verifies against the then-current tree.

---

Wake #53 — 2026-07-09T10:14Z — Builder reboot on closed phase; terminal state re-verified; no action

Fourth reboot onto a closed phase (cf. wakes #50–#52). Re-verified the stop condition cold, no cached assumptions: STATUS-redfix.md:10 = ## DONE; M1 PASS @2026-06-18T01:18Z + M2 PASS @2026-07-09T00:18Z; no standing VETO (the two ## VETO headings at REVIEW-redfix.md:553,649 are historical — :553 is annotated [CLEARED … NOT standing] and :649 is the clearing record itself); CLAIMED at STATUS:481 is the superseded historical M2 claim, as the Adversary itself noted at REVIEW-redfix.md:1442; no inbox files on either side; Adversary loop closed at its wake #49.

Working tree carries only the untracked main.go — already flagged in STATUS as unexplained and inert, left in place per the guardrail against deleting or committing files I did not create.

Open items remain operator-only and block nothing: A-redfix-1 + B-redfix-8 (one live push-capable Gitea credential exposed two ways; sentinel 3fcea78925015fc9 last probed unrotated). Merge target is redfix-m2-harness@b5f2b10, never 07fc6d4.

No Builder action exists that advances operator-only work. Stopping the loop.

---

Wake #54 — 2026-07-09T10:2xZ — Adversary findings acted on; insteadOf root cause falsified; STATUS corrected

Woken by watchdog on two review(...) commits (4c62cd0 wake #53, 3450ab5 wake #54). Both reaffirm ## DONE / no VETO. Neither is a gate. But both correct STATUS text, which is mine, so both needed action.

Why I re-verified instead of accepting. The Adversary's wake-#53 addendum rewrote the operator remedy for a live, unrotated credential. If its root cause were wrong, the operator would do the wrong remediation and believe the leak closed. So I probed the causal claim rather than the exposure count.

Exposure: confirmed. 78 world-readable cred-bearing .git/config, all sentinel 3fcea78925015fc9.

Root cause (/root/.gitconfig insteadOf): falsified, three ways. (1) insteadOf rewrites the transport, not what clone writes — tested on-host, prefix form, git 2.47.2: the clone succeeded through the rewrite and stored the original URL. (2) /etc/cc-ci/.git/config (mtime 2026-05-31) predates /root/.gitconfig (2026-06-15) by two weeks. (3) The live fetch_recipe clones a clean URL + http.extraHeader token (landed 9b33fdf, 2026-05-27) — hence 0/215 numeric clones carry the cred and 68/68 cred-bearing run dirs are manual-*. The Adversary's "regenerates on every CI run" does not reproduce.

Real generator: credentialed origin in canonical /root/.abra/recipes/* (safe at /root 0700), copied by CCCI_SKIP_FETCH=1 shutil.copytree into /var/lib/cc-ci-runs (0755). The copy loses its source's protection. Filed as B-redfix-9 (deferred, post-merge). Sent the falsification to ADVERSARY-INBOX.md — the finding is in its section of BACKLOG, which I may not edit.

Two probes of my own were void before I fixed them, worth recording: (a) I curled 127.0.0.1:8080 and read 000 as evidence — it was a connection failure; the dashboard is behind Traefik on 443. (b) My first "guard holds" 404 was on a path with an empty recipe segment, and then on a manual-* run whose results.json also 404s. A 404 on a nonexistent/unserved resource proves nothing about the guard. Only served run 985, whose .git/config exists on disk at 0644 while results.json → 200 and .git/config → 404, is a sound test. The Adversary's own probe had the same ambiguity; its conclusion survives, its evidence needed the control.

Wake #54's correction accepted and verified first-hand: the sweep runs the deployed /etc/cc-ci (d11f8f5, clean), not origin/main; WARM_CANONICAL = False is a real assignment at line 16; 07fc6d4 is not an ancestor of the deployed HEAD. So the disarm is a property of what is deployed, not of what is merged. STATUS line 40 corrected accordingly.

No gate impact, no DoD item touched, no VETO standing. ## DONE still stands. Nothing here is Builder-actionable beyond the doc corrections + B-redfix-9; rotation remains operator-only. Stopping the loop again.

---

Wake #55 — 2026-07-09T10:4xZ — Adversary adjudication: both my corrections were themselves corrected. Conceded both.

The Adversary (40477fd) conceded my insteadOf falsification but corrected ME on two points. I verified both first-hand and both hold against me:

1. "Production CI does not regenerate them" was WRONG. I read the manual-* prefix as "hand-run leftover." It isn't: run_id() returns manual-<pid> for ANY non-Drone run (run_recipe_ci.py:318-319), and the autonomous nightly-sweep.timer runs run_recipe_ci.py outside Drone with CCCI_SKIP_FETCH="1" (nightly_sweep.py:88, the run_on_tag path). So the sweep itself regenerates the exposed copies weekly. Empirical: freshest copies 2026-07-05 03:37–03:59Z vs sweep LastTriggerUSec 2026-07-05 03:04:50Z. My earlier grep of nightly_sweep.py was truncated by head -5, which hid the CCCI_SKIP_FETCH line — a self-inflicted blind spot. The Adversary's wake-#53 "regenerates" CONCLUSION was right all along; only its mechanism (insteadOf) was wrong. I had the mechanism right and the conclusion wrong. Between us the truth is: sweep copytree, weekly.

2. My census missed a second credential. I grepped autonomic-bot:, which cannot match an oauth2: token. Full per-file census: 78 files carry the password, 117 carry a live oauth2 token (62 carry both), 133 distinct world-readable cred files under /var/lib. The token is LIVE + PUSH-CAPABLE (/api/v1/user → 200 autonomic-bot/64; ls-remote to private mirror OK) and is the one recipe-mirror-sync.sh:39 pushes with. That also falsifies my own B-redfix-8 "small blast radius" note (which leaned on mirror-sync using a token "not the password" — that token is itself leaked).

Lesson worth keeping: a census predicated on one credential's literal string cannot discover a second credential; the right probe enumerates ALL userinfo, then buckets by hash. And head -N on a grep during a root-cause hunt can manufacture a false negative — I concluded "no CCCI_SKIP_FETCH in nightly_sweep" from a truncated view.

Fixed: STATUS steps 3–4 (two credentials, weekly regeneration, combined remedy incl. chmod 0750), the B-redfix-8 blast-radius note, and B-redfix-9. Replied via ADVERSARY-INBOX.md. Still no gate/DoD impact, no VETO, ## DONE stands. Rotation of BOTH secrets remains operator-only. Stopping the loop.

---

Wake #56 — 2026-07-09T10:4xZ — Adversary close-out: full agreement; count discrepancy reconciled; no action

Adversary (8849e42) confirmed both my wake-#55 concessions, independently re-verified token liveness (GET /api/v1/user → 200, autonomic-bot/64), and closed with no VETO. The only open thread was the password count (my 78 vs its 68). Reconciled first-hand — it is a scope difference, not the "regex-form" cause the Adversary guessed:

password-bearing world-readable files:  68 in /var/lib/cc-ci-runs  (== Adversary's 68, exact)
                                      + 8 /nix/store (0444) + /tmp/v + /etc/cc-ci = 10 outside /var/lib
                                      = 78 full-FS  (== my STATUS count)

Both numbers are right under their scope; STATUS already documents 78 = 68 + 8 + /tmp/v + /etc/cc-ci, so no STATUS change. Token count 117 agreed exactly.

Nothing actionable remains: both loops agree on the facts, the corrections are in STATUS/BACKLOG, and rotation of both secrets is operator-only. No BUILDER-INBOX (no question posed to me); I am not sending another inbox — agreement is complete and a re-acknowledgment would only ping-pong. ## DONE stands, no VETO. Stopping.

---

Wake #57 — 2026-07-09 — Builder reboot on closed phase; terminal state re-verified cold; loop STOPPED

Reboot after both loops closed out (Adversary wake #56 8849e42: "full agreement, no action"; my wake #56 3d98d91). No machine-docs/BUILDER-INBOX.md, no gate CLAIMED, no new Adversary verdict since 8849e42.

Re-verified the terminal invariants from a fresh cold clone, not from cached state:

• ## DONE present in machine-docs/STATUS-redfix.md (exactly one heading).
• No standing VETO. The two ^## VETO headings in REVIEW-redfix.md are historical: line 553 is self-marked [CLEARED @2026-07-09T00:18Z … NOT standing] and line 649 is the clearing record itself.
• Merge target unmoved: origin/redfix-m2-harness tip == b5f2b104e6dd41f57521f98f2f8793ab523c6e6d. 07fc6d4 is an ancestor of that tip (expected — b5f2b10 ships keycloak enrollment + the canonical_ns() fix together, so the F-redfix-4 precondition is self-maintaining across a merge).
• "Nothing merged" DoD clause holds: git merge-base --is-ancestor origin/redfix-m2-harness origin/main → non-zero (not merged).
• Working tree clean apart from the known untracked main.go (inert, unexplained, operator-flagged in STATUS; not created by this phase, so not deleted or committed).

Only open items are A-redfix-1 / B-redfix-8 — one live Gitea bot credential exposed two ways, plus the second oauth2/9c44a1 token found at Adversary wake #55. Both are operator-rotation-only and depend on no DoD item; no Builder loop action can advance them. The Adversary reached the same conclusion and stood down at its wake #49.

Stop condition (STATUS-redfix.md says ## DONE, no standing VETO, counterparty stopped) is met. Stopping the loop. No action taken this wake.

---

Wake #58 — 2026-07-09 — Adversary verdict 5431252 acted on: their PASS holds, but their probe skipped a surface — and closing it falsified my own STATUS

Watchdog ping on a review(...) commit. The verdict was a PASS, no VETO (oauth2 token does not leak to dashboard / weekly reports / Drone log API). Nothing to rebut on its face — so the useful move was to ask what the probe didn't cover.

Why I didn't just acknowledge. The Adversary's five steps probe run dirs, the dashboard, the report site, and the Drone log API — then conclude "the invariant now holds for both exposed credentials." But B-redfix-8 is not a dashboard leak. It is a public git mirror leak: a blob at a fixed sha, served unauthenticated, permanent. That surface appears nowhere in their method. The conclusion generalised from "published run/report/log surfaces" to "all published surfaces" — and the omitted surface is precisely the one that already burned this project once. A PASS resting on an untested inference is worth testing.

What the test found. Exhaustive literal-value scan of all 3099 blobs in the object db, positive-controlled:

• Token: 0 / 3099. Their conclusion survives, and now extends to the mirror. The 271 blobs matching the string oauth2: are recipe-mirror-sync.sh:39's ${TOKEN} interpolation and prose citing the sha16 sentinel. Token exposure is filesystem-only.
• Password: 2 blobs / 2 published commits — 14c7dee and 223cc16, both ancestors of origin/main. My own STATUS said "only the historical commit 14c7dee serves it." Wrong, and it understated the exposure in the direction that matters: an operator following my STATUS would scrub 14c7dee, believe the leak closed, and leave 223cc16 serving the live push-capable credential. Corrected in 2a2f2f7.

So the wake's real yield was a defect in my artifact, found by taking the Adversary's claim seriously enough to test its edges rather than its centre. The asymmetry now recorded — password on mirror+FS, token on FS only — changes the operator remedy: a history scrub is required for one credential and pointless for the other.

Near-miss worth recording. My first scan pipeline used awk. awk is not installed on the orchestrator. The blob list came back empty, and the scan dutifully reported 0 token hits and 0 password hits — a clean bill of health from a probe that examined nothing. I caught it only because 0 password hits contradicted the known leak. That is exactly the trap STATUS already documents for the empty-string sentinel e3b0c44298fc1c14, in a new costume. A negative result from an unvalidated probe is not evidence. I have promoted the positive control (password must return 2) into STATUS, BACKLOG, and the inbox as a mandatory step, because the next agent to run this scan will not remember this wake.

Hygiene. Token pulled root-only into a chmod 600 file, never transmitted off-node, shred-removed. Re- scanned after committing: still exactly 2 password blobs — my correction added no new exposure.

Sent ADVERSARY-INBOX.md (29d2b22) for cold adjudication. No gate CLAIMED, no VETO sought. ## DONE stands. Operator-scope; the credential rotation and the (now two-commit) scrub remain the operator's to perform. Loop continues until the Adversary adjudicates.

---

Wake #58b — 2026-07-09 — Adversary adjudicated (63f7001): both findings CONFIRMED. One residual discrepancy chased down; loop STOPPED.

The Adversary cold-verified my wake-#58 correction from its own clone and confirmed both results: the oauth2 token is in 0 blobs (filesystem-only, not mirror-served), the password is in exactly 2 (fd21fcb8@14c7dee + bcc31b55@223cc16, both published ancestors of origin/main). It also accepted the catch that its wake-#57 "the invariant now holds for both credentials" sentence overreached — it had not probed the mirror surface. Inbox consumed. No VETO; ## DONE stands.

I did not accept the confirmation as-is. Their scan reported 3061 blobs where mine reported 3099. A verdict that agrees with me on the conclusion but disagrees on the evidence is still a disagreement, and the cheap read ("close enough, both found 2") would have left an unexplained number in a security artifact.

Root cause, verified rather than guessed — three independent drifts, none of them an error:

1. My own wake-#58 commits added blobs after I scanned (same host now reads 3104).
2. --batch-all-objects counts unreachable objects that git rev-list --all omits — 49 here; reachable-only is 3026.
3. git clone /local/path hardlinks the whole object store, so my "cold" local clone inherited those unreachable objects and also read 3104. A clone from the remote receives only reachable objects — which is exactly why the Adversary's number is lower, and why its scope is the more correct one for a question about what the mirror serves.

The real defect this exposed was mine, again. My STATUS documented the reproduction with # → 3099 lines as the expected output. That figure was an incidental measurement recorded as an assertion. A future agent running the documented command would see a different number every time and could read the mismatch as a broken probe or a tampered repo — both false. Corrected: the repro now asserts a wc -l > 100 sanity floor and names the scope-independent invariants — password ⇒ 2, token ⇒ 0 — which held across all four scans (3099 / 3104 / 3026 / 3061) precisely because they don't depend on scan scope.

Same failure shape as this wake's earlier awk near-miss, one level up: there, an unvalidated probe produced a false negative; here, a valid probe's incidental byproduct got frozen into a spec. Record the invariant, not the measurement. Both lessons are now in STATUS and BACKLOG, where the next agent will actually meet them.

Terminal state re-verified: ## DONE present; the two ^## VETO headings are historical (self-marked CLEARED); latest Adversary verdict = "no VETO, ## DONE stands"; both inboxes absent (consumed); tree synced with origin/main; STATUS + BACKLOG re-scanned post-commit and carry no credential value. Only main.go remains untracked — inert, unexplained, operator-flagged, not mine to delete.

Handshake complete, counterparty stopped, no gate pending, nothing an agent loop can advance. The credential rotation and the two-commit history scrub are the operator's. Stopping the loop.

---

Builder wake #59 @2026-07-09T11:33Z — process reboot onto a closed phase; terminal state re-verified cold; loop STOPPED

A Builder process reboot, not new work. I re-derived the stop condition from the tree rather than trusting the prior entry's record:

• HEAD == origin/main == 88cd1dd — unmoved except for the Adversary's own wake-#59 stop record.
• ## DONE — 2026-07-09T00:18Z at STATUS-redfix.md:10.
• M1 PASS @2026-06-18T01:18Z, M2 PASS @2026-07-09T00:18Z.
• No standing VETO. Both ^## VETO headings in REVIEW are historical: :553 is self-annotated [CLEARED … NOT standing], :649 is the clearing record. Latest Adversary verdict (:1817): "no VETO. ## DONE stands. M1 + M2 PASS stand. Loop STOPPED."
• No BUILDER-INBOX.md (nothing to consume), no ADVERSARY-INBOX.md.
• Nothing merged: b5f2b10 still awaits the operator.
• Tree clean apart from untracked main.go — the A-redfix-3 artifact: inert, origin unexplained, not mine to commit or delete.

I did not re-open M1/M2 or manufacture a probe. The counterparty has stopped, no gate is claimed, and the only open items (A-redfix-1 / B-redfix-8 credential rotation + the two-commit history scrub; B-redfix-5 deferred) are operator-scope — no agent loop can advance them. Re-verifying a passing gate on an unmoved tree is idle-filler, not work.

Stop condition met. Loop STOPPED.

Builder wake #63 — 2026-07-09T11:58Z — reboot onto closed phase; watchdog ping was a no-op re-confirmation. Loop STOPPED.

Woke on a watchdog ping ("Adversary pushed a verdict/finding"). Pulled: the only new commit was 3559dc2, the Adversary's wake-#62 post-reboot re-confirmation — no new finding, no VETO, no gate claimed, no inbox. The ping fired on the review( prefix, which the watchdog cannot distinguish from a substantive verdict. Acting on it correctly = confirming there is nothing to act on.

Terminal state re-verified cold rather than assumed:

• ## DONE present at STATUS-redfix.md:10 (@2026-07-09T00:18Z).
• M1 PASS @2026-06-18T01:18Z, M2 PASS @2026-07-09T00:18Z — both intact.
• grep -c '^## VETO' == 2, and neither is standing: line 553 is F-redfix-4 self-annotated [CLEARED @2026-07-09T00:18Z … NOT standing], line 649 is its ## VETO CLEARED record. Checked the annotations, did not infer from the count — the raw count is misleading and has misled a prior wake.
• Local HEAD == origin/main == 3559dc2 after fast-forward. No BUILDER-INBOX.md.
• DoD clause "Nothing merged (operator merges)" holds: git branch -r --no-merged origin/main still lists origin/redfix-m2-harness.

No unblocked Builder work exists. The one open item, B-redfix-8 (live Gitea bot password committed and pushed), is operator-scope and structurally out of Builder reach: the secret is already in pushed history, so remediation is credential rotation plus a history rewrite, and the guardrails forbid --force. Writing a fix here would be theatre. Untracked main.go left in place, still unexplained, still inert (no go.mod, go not installed, nothing on :8080, in no commit on any ref) — flagged for the operator per guardrails, not deleted, since I did not create it.

Declined to re-run M1/M2: the tree has not moved since the PASSes were recorded, so a re-run would re-measure an unchanged input and could only manufacture false confidence. Recorded the invariant, not the measurement.

Loop STOPPED — STATUS-redfix.md says ## DONE and no gate is pending.

---

Wake #64 (Builder) — 2026-07-09T12:04Z — no-op; the moved ref was the Adversary's own no-op

Woke to origin/main ahead by one (9453d8f..28f3a34). Read the commit rather than assuming a handoff: it is the Adversary's wake-#63 chore( re-confirmation — explicitly "no new verdict", no finding, no VETO. It carried no work for me. Notably the Adversary chose chore( over review( precisely so it would not fire a fourth false handoff ping; that discipline worked — I resolved it by reading, in one commit-diff.

Terminal state re-verified cold, not inferred:

• ## DONE — 2026-07-09T00:18Z (STATUS-redfix.md:10). M1 PASS @2026-06-18T01:18Z, M2 PASS @2026-07-09T00:18Z.
• The two ^## VETO hits are the CLEARED F-redfix-4 veto and its clearance record — confirmed by reading both heading annotations, not by counting greps.
• HEAD == origin/main @28f3a34. No BUILDER-INBOX.md. redfix-m2-harness still not an ancestor of main, so the DoD "nothing merged" clause holds.

main.go is not a stray — I checked before reaching for it. git status shows ?? main.go and my first read was "foreign file, delete or commit it". Neither. It is already adjudicated: STATUS-redfix.md:710 ("NOT produced by this phase (operator: unexplained)"), and A-redfix-3 CLOSED after the Adversary voided my own "present in both clones ⇒ written outside git" evidence — /srv/cc-ci is a symlink to /srv/cc-ci-orch, so the two paths are one inode (3254604, links=1). One file, one clone. Risk stays bounded read-only: 281-byte hello-world net/http listener, in no commit on any ref, no go.mod, go not installed, nothing listening on :8080, referenced by no tracked code. Leaving it untracked and in place is the correct action for a file this loop did not create and cannot explain — committing it would launder unexplained state into the repo, and deleting it would destroy the operator's only evidence.

No unblocked Builder work remains. B-redfix-8 needs a git history rewrite plus a credential rotation (--force is forbidden to me, rotation is operator-scope); A-redfix-1 is explicitly out-of-DoD, no-VETO, and its remediation is the same rotation. Declined to re-run M1/M2 on an unmoved tree — that would re-measure an unchanged input and manufacture false confidence, not test anything.

Loop STOPPED.

Wake #65 @2026-07-09 — reboot onto closed phase, zero delta

Woke on a phase that is DONE. Re-verified terminal state cold rather than trusting the previous entry: ## DONE @00:18Z at STATUS-redfix.md:10; M1 PASS @2026-06-18T01:18Z + M2 PASS @2026-07-09T00:18Z both stand; the two ## VETO headings read individually (not counted) — :553 is annotated [CLEARED … NOT standing], :649 is the clearance record itself. No inbox either direction.

Why this wake produced no commit-of-work: HEAD == origin/main == da79d04, which is my own wake-#64 journal. Nothing — not the Adversary, not the tree — has moved since I last stopped. There was no handoff to respond to. Re-running M1/M2 against an unmoved tree would manufacture a fresh timestamp without applying any new adversarial pressure, so I declined; the existing PASSes are the verification, and re-stamping them would misrepresent idle-filler as evidence.

Why nothing is left for the Builder: the two open backlog items are both operator-scope and outside my authority — B-redfix-8 needs a history rewrite (--force is forbidden by the standing rules) and A-redfix-1 needs a live credential rotation. Neither is a Builder action.

Loop STOPPED. Re-entry on this phase should be a no-op until an operator moves the tree.

Builder wake #66 — 2026-07-09T19:13Z — reboot onto closed phase, zero delta

Cold re-verification of terminal state (no inference from commit messages):

• ## DONE @2026-07-09T00:18Z in STATUS-redfix.md.
• M1 PASS @2026-06-18T01:18Z (REVIEW-redfix.md:22, ### heading — an earlier ^## .*PASS grep missed it; checked the heading directly rather than concluding it was absent).
• M2 PASS via ## VETO CLEARED @2026-07-09T00:18Z (REVIEW-redfix.md:649), superseding the 06-18T07:06Z PASS given against parent 07fc6d4.
• Both ## VETO grep hits READ, not counted: one CLEARED veto + its clearance record. None standing.
• HEAD == origin/main @7a294c0. No BUILDER-INBOX.md. No open blocking finding.
• Untracked main.go re-checked by hash (bdbc3bf1…, 281 B) — byte-identical to the file already adjudicated at STATUS:710 + A-redfix-3. Left in place: committing would launder unexplained state, deleting would destroy operator evidence.

No unblocked Builder work remains. B-redfix-8 (credential rotation) and A-redfix-1 (/etc/cc-ci/.git/config 0644 w/ embedded bot password) are operator-scope: both need a history rewrite (--force, forbidden by the standing rules) plus rotation of credentials I must not unilaterally rotate.

Declined to re-run M1/M2 against an unmoved tree — a re-run on identical inputs manufactures a fresh timestamp without fresh evidence. Loop STOPPED.