Commit Graph

26 Commits

Author SHA1 Message Date
8276ecdfe0 status(redfix): accept A-redfix-2 + A-redfix-3 — correct the self-heal mechanism and retract the "both clones" evidence
Both claims re-derived independently before editing; both of mine were wrong.

A-redfix-2: "no code path redeploys it" is FALSE. abra.undeploy() leaves the app
.env, so the next reconcile() takes the fresh-deploy branch (:471-479), which
redeploys and never calls warmsnap. Self-healing is blocked by the absence of a
re-trigger (warm-keycloak.service: Type=oneshot, RemainAfterExit=true, no timer,
no Restart=), so it heals on reboot/nixos-rebuild and RE-WEDGES on the next due
upgrade while the foreign snapshot/meta.json remains. Operator remediation is to
DELETE the foreign snapshot/, not to redeploy keycloak. Failure presents as an
intermittent flake, not a permanent outage.

A-redfix-3: /srv/cc-ci is a symlink to /srv/cc-ci-orch, so the two main.go paths
are the same inode (3254604, links=1). The "present in both clones" premise is
void and proves nothing about the file's origin (still unexplained; inert).

Docs-only. No DoD item, gate, or verdict affected: ## DONE stands, M1 + M2 PASS
stand, merge target b5f2b10 unchanged, no VETO. The B-redfix-5 merge precondition
is unchanged and still correct.
2026-07-09 08:35:51 +00:00
d5dc547cd9 status(redfix): flag unexplained untracked main.go (both clones, outside git) — no action taken
Not created by this phase; in no commit on any ref; present in both clones with
identical mtime, so written outside git. Left in place per the "don't delete or
commit what you didn't create" guardrail. Affects no gate, DoD item, or verdict.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TXsyKUod2KPmNyBXNf5GPq
2026-07-09 08:27:12 +00:00
cbeceea22f status(redfix): wedge does not self-heal + blast radius bounded to keycloak (Adversary-confirmed widening)
Adversary (75f6655) independently re-derived the upgrade-path site and CONFIRMED the widening of B-redfix-5:
"my wake-#41 framing was too narrow; do not revert". Nothing reverted.

It supplied three facts I had not established. Re-derived each from source rather than accepting them:
- reconcile()'s only try is :520-525; sole caller main():556 does not catch -> a raise at :514/:536 escapes
  to SystemExit with keycloak already undeployed. Nothing redeploys it; recovery is MANUAL.
- WARM_CANONICAL = True at 07fc6d4 -> the pre-fix canonical seed is a real arming action, not hypothetical.
- WARM_DOMAINS == {keycloak}; keycloak is the only stateful:True SPECS entry (traefik is False) -> blast
  radius is exactly one warm unit.

Folded the two operator-actionable ones into STATUS (does-not-self-heal; bounded to keycloak) and tightened
the pre-fix-seed line from hypothetical to fact. The bound also stops the precondition reading as a
fleet-wide hazard.

No DoD item touched. M1+M2 PASS stand, ## DONE stands, no VETO. Docs-only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YZmH4rVNue3irZ6EsZzavb
2026-07-09 08:20:35 +00:00
93614f114e status(redfix): record the b5f2b10 MERGE PRECONDITION; widen B-redfix-5 to both post-undeploy sites
Acting on the Adversary's wake-#41 verdict (3131e39), which CONFIRMED the merge guidance but left a merge
precondition only in REVIEW-redfix.md — Adversary-owned, and not what an operator reads before merging.
Moved it into STATUS (operator-facing, mine), re-deriving every leg from source at b5f2b10 rather than
trusting the verdict.

- STATUS: MERGE PRECONDITION for b5f2b10, with the exact ssh check + expected output, and the corrected
  reason it is unreachable today (live slot has no meta.json — NOT "F-redfix-4 is closed").
- Sharpening the Adversary missed: the new guard is reached from snapshot() at warm_reconcile.py:514 as
  well as restore() at :536. Line :512-514 is undeploy -> wait_undeployed -> snapshot, also outside the
  try/except, and it is on the NORMAL upgrade path — it fires on every stateful auto-upgrade, not only on
  rollback. B-redfix-5 as filed named only the rollback site.
- BACKLOG + DEFERRED: widen B-redfix-5 to both sites (a) :512-514 and (b) :534-536; correct the claim that
  F-redfix-4 "supplied the only reachable trigger and that is now closed" — b5f2b10 ADDS a raise path
  (_assert_slot_not_foreign at warmsnap.py:158 and :209).

Verified: guard present in both callers at b5f2b10; reconciler structure at :512-514 / :520-525 / :534-537;
cc-ci live slot holds last_good only (no snapshot/, no meta.json, no canon-* slot). Probe with a seeded
foreign meta raises on both paths; absent meta and self-consistent meta both pass.

No DoD item touched. M1+M2 PASS stand, ## DONE stands, no VETO. Docs-only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YZmH4rVNue3irZ6EsZzavb
2026-07-09 08:16:06 +00:00
697959daa3 status(redfix): harden B-redfix-8 close-criteria — e3b0c44298fc1c14 is the empty-input tell, not a rotation
Wake #39 (2a61c09) carried no VETO/finding/inbox and nothing to advance, but it did surface a
false-close hazard in STATUS's own operator instructions: "A different digest => rotated => close
B-redfix-8" licensed closing an open HIGH public-credential exposure on a broken probe.

/srv/cc-ci/.testenv lives on the ORCHESTRATOR, not cc-ci. Run over `ssh cc-ci` the sentinel command
hashes empty input and prints sha256("")[:16] = e3b0c44298fc1c14 — a different digest with no rotation.

STATUS now states where to run the probe, names e3b0c44298fc1c14 as the empty-input tell that must NOT
close the item, and disambiguates the three conflated digests: 3fcea78925015fc9 = sha256(credential
value) = the only rotation sentinel; 1994fd8d… = sha256(whole served blob), immutable; e3b0c44298fc1c14
= sha256(empty), a broken probe. Also records the exposure is now confirmed at VALUE level (the live
push-capable password appears verbatim in the public body), stronger than prior blob-equality.

Verified independently before writing: printf '' | sha256sum => e3b0c442…; ssh cc-ci ls .testenv =>
absent; orchestrator sentinel => 3fcea78925015fc9 (still UNROTATED).

Terminal condition unchanged: ## DONE, M1+M2 PASS, no VETO. B-redfix-8 stays OPEN/HIGH/operator-only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M7ZrNUJscccinFq5CM5VmP
2026-07-09 07:35:20 +00:00
33a4021e81 status(redfix): surface B-redfix-8 (HIGH, open) in STATUS as an explicit OPERATOR ACTION REQUIRED block
The one item in this phase that needs a human was recorded only in BACKLOG/REVIEW/JOURNAL.
STATUS is the operator-facing artifact and carried '## DONE' with no mention of it. Adds
WHAT/WHERE/HOW/EXPECTED + remedy, both repro commands re-verified this wake (blob 33408;
digest 3fcea78925015fc9 = still unrotated). No DoD item touched, no gate claimed, no verdict
change: M1+M2 PASS stand, DONE stands, no VETO.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XChUi2CUccznbmzHZBSuXr
2026-07-09 05:11:12 +00:00
594824d9fe journal(redfix): wake #17 — consumed review(redfix)@32fefb2 (re-confirmation #15); fixed stale "15 existing canonicals" in STATUS (real: 17 seeded, and the invariant is structural not numeric — WARM_DOMAINS singleton ⇒ exactly one re-key, 20 of 21 enrolled unchanged); all figures re-derived first-hand off live disk + b5f2b10, not taken from the verdict; filed B-redfix-6 for the same stale count in canonical.py:52's docstring, deliberately NOT fixed inline to avoid moving the branch off the M2-PASS-pinned b5f2b10; no verdict change, DONE stands, loop remains stopped
Some checks failed
continuous-integration/drone/push Build is failing
2026-07-09 01:42:59 +00:00
3c63f3c6d3 status(redfix): ## DONE re-asserted — M2 PASS @2026-07-09T00:18Z, VETO CLEARED, F-redfix-4 CLOSED; merge target redfix-m2-harness@b5f2b10; B-redfix-5 recorded in DEFERRED.md
Some checks failed
continuous-integration/drone/push Build is failing
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FS8p1esg57UAC69riNvuBX
2026-07-09 00:20:54 +00:00
faef8a9685 claim(m2-redfix): M2 RE-CLAIMED — F-redfix-4 remedied at redfix-m2-harness@b5f2b10 (warm state keyed by stack namespace); clearing condition re-run green on cc-ci, unit suite 315->325; awaiting Adversary
Some checks failed
continuous-integration/drone/push Build is failing
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FS8p1esg57UAC69riNvuBX
2026-07-09 00:13:57 +00:00
cfea875b2b status(redfix): claim WITHDRAWN — standing VETO F-redfix-4 (keycloak warm-state slot collision); M2 reopened, do not merge redfix-m2-harness
Some checks failed
continuous-integration/drone/push Build is failing
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FS8p1esg57UAC69riNvuBX
2026-07-09 00:01:57 +00:00
612412c52a status(redfix): evidence addendum for F-redfix-3 — durable content assertion for the discourse fix (sha pins rotted via later-phase force-push; content re-verified at ede6399); historical shas left unedited on purpose. DONE unchanged, no gate reopened
Some checks failed
continuous-integration/drone/push Build is failing
2026-07-08 23:23:33 +00:00
b3bdc291b4 status(redfix): ## DONE — phase complete, M1+M2 fresh Adversary PASS, no VETO
Some checks reported errors
continuous-integration/drone/push Build is failing
continuous-integration/drone Build was killed
All 6 canon-sweep failures fixed + cold-verified green (mattermost-lts,
discourse, keycloak, mumble, gitea, bluesky-pds). No standing exceptions.
Nothing merged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bacy8SJhBebNAGoYsi1Fxt
2026-06-18 07:07:14 +00:00
29a28176a9 claim(redfix-M2): discourse F-redfix-1 FIXED + level=5 verified — re-claim 6/6
Some checks failed
continuous-integration/drone/push Build is failing
Dropped orphaned image-less sidekiq from discourse compose.smtpauth.yml (PR #4
@9ff5e19); R011 lint  (Adversary repro) + own cold run level=5 of 5 all tiers
pass. Other 5 fixes unchanged (Adversary PASS). 6/6 verified green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bacy8SJhBebNAGoYsi1Fxt
2026-06-18 06:55:28 +00:00
0e255d8570 claim(redfix-M2): all 6 canon-sweep failures FIXED + verified green
Some checks failed
continuous-integration/drone/push Build is failing
mattermost-lts (PR #1, !testme #901), discourse (PR #4, !testme #849), keycloak
(harness branch, promotes at warm-canon-keycloak), mumble (harness branch, budget
180s) — already verified. gitea (PR #2 @a0f2db8, app.ini seed-on-empty into writable
volume) + bluesky-pds (PR #4 @4987ba9, caddy ${STACK_NAME}_app per operator, NO
rename) verified by direct chaos-deploy reproducing the exact M1 scenario: gitea
app.ini 0->1862, API 200, 0 RO crashes; bluesky external HTTPS /xrpc/_health 200
(M1 000), caddy resolves own internal app. Both promotes operator-merge-gated (harness
WC5 force-fetches the published tag); direct deploy is the maximal pre-merge proof.
No standing exceptions. Nothing merged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bacy8SJhBebNAGoYsi1Fxt
2026-06-18 05:55:43 +00:00
ed7d897e5f status(redfix): M2 tracker — mattermost+discourse VERIFIED; bluesky rename routing-works-but-backup-fails; gitea needs rework; keycloak/mumble pending verify
Some checks failed
continuous-integration/drone/push Build is failing
2026-06-18 02:16:47 +00:00
c3854a9bcc status+journal(redfix): M2 — mattermost-lts FIXED (run #901 all green, restore fixed); discourse #4 green; bluesky PR #4 created (promote-path verify next)
Some checks failed
continuous-integration/drone/push Build is failing
2026-06-18 01:30:57 +00:00
abfbe8b0aa journal+status(redfix): M2 recon — discourse #4 (official-image) already !testme-green; mattermost #1 (pg-restore) triggered for verify
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone Build is passing
2026-06-18 01:24:48 +00:00
191ddc9fb8 status(redfix): M1 PASS (Adversary cold-verified all 6 classifications CORRECT); begin M2 fixes
Some checks failed
continuous-integration/drone/push Build is failing
2026-06-18 01:20:15 +00:00
0a06c411a6 claim(redfix-M1): all 6 canon-sweep failures investigated in isolation + classified (results table + cold-verify guide). discourse=stale overlay test, mattermost-lts=recipe restore defect, mumble=load FLAKE (2x green), bluesky=app-alias proxy collision, gitea=app.ini RO crash, keycloak=warm-domain collision. 2 canon root-causes corrected.
Some checks failed
continuous-integration/drone/push Build is failing
2026-06-18 00:18:09 +00:00
00fca8a33e journal+status(redfix): M1 gitea app.ini read-only JWT crash CONFIRMED on warm advance (recipe defect); 6/6 classified
Some checks failed
continuous-integration/drone/push Build is failing
2026-06-18 00:14:32 +00:00
88c9ebcce4 status(redfix): M1 tracker — keycloak classified (harness collision); 5/6 done, gitea app.ini advance reproducing
Some checks failed
continuous-integration/drone/push Build is failing
2026-06-18 00:08:40 +00:00
f8ba0c3a1f journal(redfix): M1 bluesky-pds — 000 reproduces deterministically; root cause = caddy↔app cross-stack 'app' alias collision on shared proxy (recipe defect)
Some checks failed
continuous-integration/drone/push Build is failing
2026-06-18 00:02:26 +00:00
41e161a433 status(redfix): M1 tracker — discourse/mattermost/mumble classified; bluesky promote in flight
Some checks failed
continuous-integration/drone/push Build is failing
2026-06-17 23:53:13 +00:00
8df74d7bc0 journal(redfix): M1 mattermost-lts isolation — DETERMINISTIC restore fail; genuine recipe defect (no restore.post-hook vs immich)
Some checks failed
continuous-integration/drone/push Build is failing
2026-06-17 23:41:29 +00:00
23b439db83 journal(redfix): M1 discourse isolation — canon root-cause wrong; deploys fine, only upgrade overlay (unreleased official-image migration) fails
Some checks failed
continuous-integration/drone/push Build is failing
2026-06-17 23:33:18 +00:00
3e61473365 chore(redfix): bootstrap phase state files (STATUS/BACKLOG/JOURNAL); M1 investigation tracker seeded
Some checks failed
continuous-integration/drone/push Build is failing
2026-06-17 23:20:55 +00:00