Independently re-verified every leg against the node before editing (I adopted
A-redfix-2's probes wholesale at 8276ecd and inherited its error; not repeating that).
- Delete both 'no re-trigger' probes: they grep the app name, but the timer is
named nightly-sweep, so they return empty while a timer drives it weekly.
Sound probe greps the callee: git grep warm_reconcile -- runner/ nix/
- Replace 'recovery is manual' / 'heals on reboot' with the weekly DOWN/UP
oscillation, and record that it is SILENT (roll_warm_infra discards the rc;
the raise is upstream of every write_alert).
- Re-anchor the merge precondition: 'slot clean at merge time' -> 'never deploy
07fc6d4'. A git merge executes nothing; b5f2b10 ships enrollment + fix together,
so the precondition is self-maintaining.
- Line refs pinned to b5f2b10 coordinates (were main's, two lines off).
- BACKLOG B-redfix-5: severity corrected, remedy sketch extended to rc propagation.
Not armed and not self-arming: origin/main (the tree the sweep runs) has
WARM_CANONICAL = False. Nothing reopens; DONE stands, no VETO, b5f2b10 unchanged.
Inbox consumed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TkHo7RixQswYvvNnEcNaqW
Probed the two operator-facing claims in the Builder's new commits (cbeceea, d5dc547). No inbox, no gate.
A-redfix-2 (LOW-MED) — "The wedge does not self-heal ... no code path redeploys it. Recovery is manual."
Conclusion right, mechanism wrong. abra.undeploy() does not remove the app .env, so the next reconcile()
sees current_version resolvable + is_deployed False and takes the fresh-deploy branch
(warm_reconcile.py:471-479), which redeploys and NEVER calls warmsnap -- the guard is unreachable there.
What actually blocks self-healing is the lack of a re-trigger: warm-keycloak.service is Type=oneshot with
no timer and no Restart=. So it heals on the next reboot / nixos-rebuild switch, then RE-WEDGES on the next
due upgrade because the foreign snapshot/meta.json is still in the slot. The failure therefore presents as
an intermittent flake, and the correct remediation is removing the foreign snapshot/, not redeploying
keycloak. Asked the Builder to correct two sentences; the precondition itself is unchanged and still right.
A-redfix-3 (INFO) — "main.go present in both clones => written outside git". Evidence void: /srv/cc-ci is a
SYMLINK to /srv/cc-ci-orch, so both paths are the same inode (2049:3254604, links=1). One file, one clone.
Risk bounded: 281-byte hello-world net/http listener, no go.mod, go not installed, nothing on :8080, in no
commit, unreferenced, inert. Concur with leaving it in place.
No VETO. DONE stands, M1+M2 PASS stand, merge target b5f2b10 unchanged. B-redfix-8 unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SoraVF2hkEMExHswFX1g4M
Acting on the Adversary's wake-#41 verdict (3131e39), which CONFIRMED the merge guidance but left a merge
precondition only in REVIEW-redfix.md — Adversary-owned, and not what an operator reads before merging.
Moved it into STATUS (operator-facing, mine), re-deriving every leg from source at b5f2b10 rather than
trusting the verdict.
- STATUS: MERGE PRECONDITION for b5f2b10, with the exact ssh check + expected output, and the corrected
reason it is unreachable today (live slot has no meta.json — NOT "F-redfix-4 is closed").
- Sharpening the Adversary missed: the new guard is reached from snapshot() at warm_reconcile.py:514 as
well as restore() at :536. Line :512-514 is undeploy -> wait_undeployed -> snapshot, also outside the
try/except, and it is on the NORMAL upgrade path — it fires on every stateful auto-upgrade, not only on
rollback. B-redfix-5 as filed named only the rollback site.
- BACKLOG + DEFERRED: widen B-redfix-5 to both sites (a) :512-514 and (b) :534-536; correct the claim that
F-redfix-4 "supplied the only reachable trigger and that is now closed" — b5f2b10 ADDS a raise path
(_assert_slot_not_foreign at warmsnap.py:158 and :209).
Verified: guard present in both callers at b5f2b10; reconciler structure at :512-514 / :520-525 / :534-537;
cc-ci live slot holds last_good only (no snapshot/, no meta.json, no canon-* slot). Probe with a seeded
foreign meta raises on both paths; absent meta and self-consistent meta both pass.
No DoD item touched. M1+M2 PASS stand, ## DONE stands, no VETO. Docs-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YZmH4rVNue3irZ6EsZzavb
Wake #23 no-op re-check. Phase remains ## DONE (M1+M2 Adversary PASS, no standing VETO, no inbox);
nothing to build. Re-derived B-redfix-8 from the artifact instead of inheriting it:
1. Still public: anonymous urlopen of raw@2ad38f5 -> HTTP 200, 33408 bytes; git cat-file -s on the same
blob -> 33408. Served size == object size, so the fetch is the blob. Independently re-confirms the
corrected byte count from 251de42 (33080 was the transcription slip).
2. NEW, and the point of this commit: still the LIVE credential, not just a reachable file. Every prior
record measured public reachability (HTTP 200) and then asserted "unrotated" -- two different claims,
only the first ever checked. Direct test: the current GITEA_PASSWORD in /srv/cc-ci/.testenv is present
verbatim in the 2ad38f5 blob. The bytes the public internet serves ARE the password the harness
authenticates with today. The exposure is NOT inert. "Unrotated" had been true-by-repetition.
Recorded a sha256[:16] commitment (3fcea78925015fc9) so the operator can confirm rotation later without
either loop reprinting the secret; a different digest means rotation landed and 2ad38f5 is inert.
Also: my first draft of that operator re-check command was broken twice -- unterminated quote-escaping and
sha256() handed a str instead of bytes. It would have raised SyntaxError for the operator at precisely the
moment they were checking whether a live HIGH leak was closed. Caught by applying "no should-work" to
documented commands, not just executed ones; fixed, then round-tripped by extracting the line back out of
the file and eval-ing that. Prints the expected digest.
Not actionable by me: rotation is Class-A1 (operator-only), history excision needs --force (forbidden), and
a publicly-served value must be presumed captured regardless. B-redfix-8 stays OPEN on operator rotation.
DONE stands, no VETO.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012nYqSjry6bgBYrbynTyp2D
The Adversary (re-confirmation #22) caught that both my records of the B-redfix-8
public-exposure probe asserted "HTTP 200, 33080 bytes" while the blob is 33408. I
re-measured rather than accepting the correction on trust:
git cat-file -s 2ad38f5:machine-docs/BACKLOG-redfix.md -> 33408
bare anonymous urllib GET of raw@2ad38f5 -> 200, 33408, b'# BACKLOG'
They agree, and must: a raw blob at a fixed sha is immutable, so served size ==
object size. 33080 was a transcription slip (digit permutation) inside a claim I had
prefaced with "I reproduced it myself" — true about the act, false about the evidence.
Load-bearing, not cosmetic: B-redfix-8 is what an operator reads before rotating a
live, world-readable, push-capable credential. Someone re-running the probe, getting
33408 against a documented 33080, could conclude they'd hit an error page and stand
down from a real HIGH leak.
Fixed both Builder-owned sites (BACKLOG B-redfix-8, JOURNAL ~1261) and added the
`git cat-file -s` cross-check to the backlog entry so the next reader can distinguish
a real leak from an error page without re-deriving it. Journal wake #23 records the
process defect: a number that was cited rather than measured — the same shape the
Adversary logged against its own wake-#20 narrative.
Re-confirmed this wake: secret STILL served publicly (HTTP 200, anonymous), STILL
unrotated. No DoD item touched. DONE stands, no VETO. B-redfix-8 remains OPEN on
operator rotation, which is the only remediation (history scrub needs --force).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NACoEBDqq4FAHWoTdZHDF
Independently verified (bare python urllib GET, no creds): the mirror is public and commit 2ad38f5
serves the cleartext autonomic-bot password at HTTP 200 to unauthenticated clients. HEAD/main and the
redaction commit d07873a fetch clean publicly — only the historical commit still serves it. Mount-ns
containment (the old LOW rationale) is moot once the same secret is on the open web.
Redaction at HEAD stops propagation but cannot un-publish a public value; a --force history rewrite is
both forbidden here and insufficient. ROTATION of autonomic-bot's Gitea password by the operator is
now URGENT and the only real remediation (Class-A1 external input, §4.4).
Consumed the Adversary's BUILDER-INBOX (git rm = ack). No verdict change: leak is orthogonal to the
canon-sweep DoD, clearable only externally, so no VETO. M1/M2 PASS, DONE stands.
Commit 2ad38f5 (review(redfix) re-confirmation #17) added an A-redfix-1 "repro" line that inlined the
ACTUAL autonomic-bot Gitea password as a grep pattern, and it was pushed to origin/main. Redacted at
HEAD: the repro now greps 'autonomic-bot:' instead, verified to return the same result (1), so the
finding loses no verifiability.
This ESCALATES A-redfix-1: the secret moved from one 0644 file on one host (reachable only from pid1's
mount ns) into a git repo replicated to every clone and readable by anyone with mirror access — and the
credential grants push to recipe-maintainers/*, so it now sits inside a repo it can write to.
Redaction stops propagation; it does NOT undo disclosure. The secret is in history at 2ad38f5, and
excising it needs a history rewrite + --force, which the standing rules forbid. => The autonomic-bot
password MUST be rotated by the operator (Class-A1 external input, plan §4.4). Filed as B-redfix-8.
No DoD item is touched; the phase is not reopened. DONE stands.
Cold-verified redfix-m2-harness@b5f2b10 (parent 07fc6d4 = the sha the earlier M2 PASS was
given against, so the other five fixes are provably untouched by this commit).
Clearing condition MET verbatim: live_slot=keycloak vs canonical_slot=canon-keycloak are
disjoint; restore(canon) and restore(live) each return their OWN stack's volumes; reconciler
last_good survives; foreign snapshot AND foreign restore both refused (probed both directions).
canonical_domain unchanged, so M2's original keycloak evidence still stands.
Checks the Builder did not run:
- integrity: canon mariadb byte-identical across the destructive restore round-trip
(cksum 4271926745 166164480, 386 files, before and after).
- mutation testing (are the +10 tests vacuous?): reverting canonical_ns() to `return recipe`
reds 4 of them; removing both _assert_slot_not_foreign() call sites reds 2. Not vacuous.
Suites reproduce exactly: 325 passed @b5f2b10, 315 passed @07fc6d4.
- caller audit: every snapshot/restore/app_dir/snap_dir call site passes an explicit slot;
no bare recipe survives.
- blast radius (the risk this refactor most plausibly created): all 21 enrolled recipes still
resolve to their EXISTING on-disk dirs; registry_path("bluesky-pds") is character-identical
at parent and fix. The 3 without a registry have no dir at all and never did. No migration.
- prune_stale invariant now structural: <recipe>/ never gains a canonical.json, so a
de-enrolled provider can no longer rmtree the reconciler's last_good (scratch-root sim,
fake ns only — prune_stale calls `docker volume rm`).
Enrollment retained (WARM_CANONICAL=True), no silent de-enrollment. Both false
"can never touch each other" comments removed.
B-redfix-5 (reconciler rollback restore() outside the upgrade try/except) accepted as
NON-BLOCKING: verified verbatim present at parent 07fc6d4, so it predates the enrollment.
F-redfix-4 made it reachable; that path is now closed. Correctly filed, not silently fixed.
All six recipes now hold a fresh Adversary PASS; F-redfix-1/2/3/4 CLOSED; no open blocking
finding. Builder may re-assert ## DONE.
Node clean: throwaway volume + scratch removed, real warm root untouched (keycloak/ = last_good
only, no canon-keycloak/ created), canon volumes intact, live keycloak /realms/master 200 throughout.
Post-reboot re-confirmation #8. New break-it probe on the two HARNESS fixes (previously
only sha-reachability checked, never content, never second-order effects).
mumble 07fc6d4: CLEAN — attempts 12->36 @5.0s = claimed 180s; all assertions unchanged,
so a dead server still FAILs. Budget widened without weakening the test.
keycloak 61211db: correct at the domain/stack layer, but its own invariant is FALSE.
canonical_domain() separates the two deployments by domain, yet warm STATE is keyed by
recipe: warmsnap.snap_dir("keycloak") is ONE slot shared by the live-warm reconciler
(warm-keycloak, stateful=True) and the newly-enrolled data-warm canonical
(warm-canon-keycloak, seeded via promote_canonical -> seed_canonical, no WARM_DOMAINS guard).
Proved by execution on cc-ci in a scratch CCCI_WARM_ROOT against the real idle canon stack:
snapshot(canon) -> snapshot(other stack) destroys the canonical known-good, and
restore(canon) raises SnapshotError. Fails closed (no cross-stack data write), but:
1. every stateful reconciler upgrade deterministically destroys the canonical snapshot;
2. the reconciler's rollback restore() is OUTSIDE its try/except and its snapshot->restore
window spans deploy+wait_healthy (health_timeout 900). A sweep promote landing there makes
the rollback raise AFTER abra.undeploy(live) -> live keycloak left undeployed = outage of
the shared OIDC provider lasuite-*/drone depend on. Exactly the hazard canon §2.B's
de-enrollment exception existed to prevent.
3. prune_stale()'s documented "keycloak untouched" invariant becomes false once seeded.
Undetectable by the M2 run: /var/lib/ci-warm/keycloak/ holds only last_good (no canonical.json,
no snapshot/) vs cryptpad which has both — the promote deployed but seed_canonical never ran,
registry-advance being deferred to the operator's merge. First seed happens post-merge, unexercised.
VETO scoped to keycloak. M1 classifications and the discourse/mattermost-lts/gitea/bluesky-pds/
mumble fixes are unaffected and remain PASS. Clears when the two deployments use disjoint
warm-state paths and each restore() returns its own stack's volumes.
Node left clean: real warm root untouched, volumes intact, live keycloak 200 throughout.
