Compare commits
3 Commits
main
...
redfix-m2-
| Author | SHA1 | Date | |
|---|---|---|---|
| b5f2b104e6 | |||
| 07fc6d4af5 | |||
| 61211dba70 |
2
.gitignore
vendored
2
.gitignore
vendored
@ -3,8 +3,6 @@
|
||||
*.key
|
||||
*.pem
|
||||
!docs/**/*.pem
|
||||
# holds a live provider API key at .provider.tinfoil.options.apiKey (F-redfix-2)
|
||||
config.json
|
||||
# python
|
||||
__pycache__/
|
||||
*.pyc
|
||||
|
||||
2
.gitmodules
vendored
2
.gitmodules
vendored
@ -1,3 +1,3 @@
|
||||
[submodule "secrets"]
|
||||
path = secrets
|
||||
url = git@git.autonomic.zone:recipe-maintainers/cc-ci-secrets.git
|
||||
url = https://git.autonomic.zone/recipe-maintainers/cc-ci-secrets.git
|
||||
|
||||
@ -51,859 +51,6 @@ hold). Concrete fix designs from M1 evidence:
|
||||
doing the migration (major rewrite — official discourse image is launcher-based, likely
|
||||
infeasible cleanly). Lean (a)+tracked-upstream; may need operator input (DEFERRED?) — assess in M2.
|
||||
|
||||
### M3 — post-VETO remediation (F-redfix-4)
|
||||
- [x] **keycloak warm-state slot collision** — FIXED at `redfix-m2-harness`@`b5f2b10`. `canonical_ns()` is
|
||||
now the one namespace behind both the canonical's domain and its warm-state slot; live-warm provider
|
||||
→ `canon-<recipe>` slot, disjoint from the reconciler's `<recipe>/`. Plus a naming-independent
|
||||
`_assert_slot_not_foreign()` guard. Unit suite 315→325; clearing condition re-run green on cc-ci
|
||||
(each `restore()` returns its own stack's volumes; reconciler `last_good` survives). Verify per
|
||||
STATUS-redfix.md "Gate: M2 RE-CLAIMED".
|
||||
- [ ] **B-redfix-5 — reconciler's post-`undeploy` warmsnap calls are outside the upgrade's `try/except`**
|
||||
(NOT blocking; NOT part of F-redfix-4's clearing condition; recorded so it is not silently dropped).
|
||||
In `warm_reconcile.py` there are **two** such sites, both after `abra.undeploy(domain)` and both
|
||||
outside the `try/except` that guards the upgrade:
|
||||
**(a) upgrade path** `:512-514` — `abra.undeploy` → `wait_undeployed` → `warmsnap.snapshot(...)`;
|
||||
**(b) rollback path** `:534-536` — `abra.undeploy` → `wait_undeployed` → `warmsnap.restore(...)` →
|
||||
`deploy_version(last_good)` at `:537`.
|
||||
If either raises for ANY reason (absent/corrupt snapshot, foreign slot, docker error, "no volumes
|
||||
found") the exception propagates, the following `deploy_version` never runs, and live keycloak is left
|
||||
**undeployed**. Site (a) is on the *normal* upgrade path — it does not need a rollback to fire.
|
||||
**Reachability, corrected (wake #42):** F-redfix-4's fix does not remove the trigger, it *adds* one —
|
||||
`_assert_slot_not_foreign()` is a new raise inside BOTH `snapshot()` (`warmsnap.py:158`) and
|
||||
`restore()` (`:209`). It is unreachable on cc-ci **today** because the live slot holds no
|
||||
`snapshot/meta.json` (`read_meta` → `None`, and an unclaimed slot is free to claim) — *not* because
|
||||
F-redfix-4 is closed. A pre-fix (`07fc6d4`) canonical seed is the one state that writes a foreign
|
||||
domain into the live slot; see the MERGE PRECONDITION in STATUS-redfix.md.
|
||||
**Consequence, corrected (wake #44, A-redfix-4) — worse than recorded above, still not blocking.**
|
||||
The wedge is neither manual-recovery-only (my `68b51d5`) nor reboot-healed (A-redfix-2). A weekly
|
||||
`nightly-sweep.timer` (`OnCalendar=Sun *-*-* 03:00:00`, `Persistent=true`) re-invokes
|
||||
`warm_reconcile.py keycloak` via `nightly_sweep.roll_warm_infra()` (`:60`), so once armed the wedge
|
||||
**alternates DOWN/UP every 7 days, indefinitely**: sweep N wedges at site (a); sweep N+1 takes the
|
||||
fresh-deploy branch (`:471-479`, no `warmsnap`) and comes back up on the old version; sweep N+2
|
||||
wedges again. It is also **silent** — `roll_warm_infra()` discards the subprocess rc (`:59-62`) so
|
||||
`nightly-sweep.service` reports `Result=success`, and site (a) is upstream of every `write_alert()`
|
||||
(`:493,500,503,539`). A weekly outage of the shared SSO provider would surface nowhere. Verified
|
||||
first-hand against the node, not adopted from the finding.
|
||||
**Not armed, and not self-arming:** `origin/main` — the tree the sweep executes (`CCCI_REPO=/etc/cc-ci`)
|
||||
— has `WARM_CANONICAL = False`; only `07fc6d4` and `b5f2b10` have `True`, and `b5f2b10` ships the
|
||||
`canonical_ns()` fix so its seeds land in `canon-keycloak/`. The binding rule is **never deploy
|
||||
`07fc6d4`**.
|
||||
Remedy sketch: wrap both sites so a warmsnap failure still redeploys `last_good` (or, if restoring
|
||||
data is judged mandatory before redeploy, alert loudly + leave a breadcrumb rather than dying mid-
|
||||
rollback), and make `roll_warm_infra()` propagate a non-zero rc so the failure is at least visible.
|
||||
Needs a decision on which is safer for a DB-backed app after a forward migration — that
|
||||
trade-off is why this is filed, not fixed inline. **Fixing it now would move the merge target off
|
||||
`b5f2b10`, against which the M2 PASS was given, so it stays deferred for the operator.**
|
||||
- [ ] **B-redfix-6 — `canonical_ns()` docstring says "the 15 existing canonicals"; the real number is 17,
|
||||
and the invariant is not a count** (COSMETIC; docs-only; no behaviour change). At
|
||||
`redfix-m2-harness`@`b5f2b10`, `runner/harness/canonical.py:52` reads "zero blast radius on the 15
|
||||
existing canonicals". Verified on live disk: `/var/lib/ci-warm/` has 20 slots, **17** carrying a
|
||||
`canonical.json` (spared: `alerts`, `keycloak`, `traefik` — the reconciler dirs). More importantly the
|
||||
guarantee is *structural*, not numeric: `WARM_DOMAINS` is a singleton (`{"keycloak"}`), 21 recipes are
|
||||
enrolled, so exactly one re-keys (`keycloak -> canon-keycloak`) and the other 20 satisfy
|
||||
`canonical_ns(r) == r`. The docstring's count will rot again on the next enrollment.
|
||||
**Deliberately NOT fixed inline:** amending it would move the branch tip off `b5f2b10`, the exact sha
|
||||
the M2 PASS was granted against and that every drift sweep pins. Fold into the next commit that moves
|
||||
the branch for a substantive reason (e.g. B-redfix-5), rewording to the structural form.
|
||||
Corrected in STATUS-redfix.md prose (which is not sha-pinned) as of wake #17.
|
||||
- [ ] **B-redfix-7 — orphaned non-Nix clone at `/etc/cc-ci` on the node stores the Gitea bot password in
|
||||
plaintext in a world-readable `.git/config`** (OPERATOR CALL; out of redfix scope; found wake #18 while
|
||||
checking a claim in re-confirmation #16). Facts, all read-only-verified on cc-ci:
|
||||
- `/etc/cc-ci/.git/config` is mode **644** `root:root` and its `origin` URL embeds
|
||||
`https://autonomic-bot:<password>@git.autonomic.zone/...` — the bot's Gitea credential at rest in
|
||||
cleartext. (Value deliberately not reproduced here.)
|
||||
- **Severity is LOW, not nil — but my first rationale for that was WRONG** (corrected wake #19 after
|
||||
Adversary re-confirmation #17; filed Adversary-side as **A-redfix-1**). I originally argued "no
|
||||
non-root *login* users, so only root can read it." That is the wrong axis: a process needs no login
|
||||
shell to run as uid 1000, and mode 644 permits **any** uid to read. Verified directly —
|
||||
`setpriv --reuid=1000 … /etc/cc-ci/.git/config` **succeeds** in the host namespace.
|
||||
- **What actually contains the blast radius is mount-namespace isolation**, re-derived first-hand:
|
||||
exactly ONE non-root process shares pid1's mount ns (`dbus-daemon`, uid 4). Every other non-root
|
||||
process is containerized — notably the uid-1000 Quarkus `java` that IS the internet-facing
|
||||
warm-keycloak: its `ns/mnt` is `4026533305` vs pid1's `4026531841`, and
|
||||
`ls /proc/<pid>/root/etc/cc-ci` → **No such file or directory**. Mode-permits ≠ reachable.
|
||||
- **Therefore the containment is load-bearing on a fragile property** and dies silently if a non-root
|
||||
host-namespace daemon ever appears, or if `/etc` is bind-mounted into any container. That fragility —
|
||||
not the current reachability — is the reason to fix it.
|
||||
- `/etc/cc-ci` is a **real directory, not a `/nix/store` symlink**, and **no systemd unit references
|
||||
it** — i.e. undeclared, unmanaged server state that nothing runs from. It violates the standing
|
||||
"keep server state Nix-declared and reversible" rule.
|
||||
**NOT actioned by me, deliberately.** The Gitea bot credential is a Class-A1 EXTERNAL infra input
|
||||
(plan §4.4) — not mine to rotate or invent. And I did not create `/etc/cc-ci`, so it is not mine to
|
||||
delete. Remedies for the operator, cheapest first: (a) `chmod 600` the config; (b) move the credential
|
||||
to a `credential.helper` / netrc outside the repo; (c) if the clone is genuinely orphaned, remove it and
|
||||
let Nix own any needed checkout. Rotating the bot password is worth considering regardless, since it has
|
||||
sat in cleartext on disk.
|
||||
- [ ] **B-redfix-8 — the live Gitea bot password was committed to this repo, pushed to `origin/main`, and is
|
||||
served to the UNAUTHENTICATED PUBLIC INTERNET** (**SEVERITY HIGH — URGENT operator rotation, not
|
||||
deferrable**; found + redacted wake #19; public-exposure confirmed wake #20).
|
||||
**Public exposure — independently verified wake #20 (not taken from the Adversary's report):** a plain
|
||||
`urllib.request.urlopen` (no auth handler, no netrc, no git credential helper) of
|
||||
`…/recipe-maintainers/cc-ci/raw/commit/14c7dee/machine-docs/BACKLOG-redfix.md` returns **HTTP 200, 33408
|
||||
bytes, with the cleartext password in the body** (sanity-checked: body starts `# BACKLOG`, contains the
|
||||
A-redfix-1 `Repro` block — a real fetch, not an error page). Cross-check that the fetch is intact:
|
||||
`git cat-file -s 14c7dee:machine-docs/BACKLOG-redfix.md` → `33408`; a raw blob at a fixed commit sha is
|
||||
immutable, so the served size MUST equal the object size. If your probe returns 33408, that is the leak,
|
||||
not an error page. (Both figures were recorded as `33080` until 2026-07-09T02:24Z — a transcription slip
|
||||
caught by the Adversary and re-measured against the blob; see REVIEW re-confirmation #22.) The mirror is public. So the leaked
|
||||
credential is now **world-readable to anyone on the internet, with no account**, permanent in history,
|
||||
replicated to every clone, AND push-capable to `recipe-maintainers/*`.
|
||||
**STILL THE LIVE CREDENTIAL — measured, not assumed (wake #23, 2026-07-09).** Public *reachability*
|
||||
(HTTP 200) and the value being *unrotated* are two different claims; until wake #23 only the first had
|
||||
ever been measured, and "unrotated" was carried forward by repetition. Direct check: the current
|
||||
`GITEA_PASSWORD` from `/srv/cc-ci/.testenv` is present verbatim in the `14c7dee` blob → **True**. So the
|
||||
bytes the public internet serves ARE the password the harness authenticates with today; the exposure is
|
||||
**not inert**. Operator re-check without printing the secret (verified to run, wake #23 — copy verbatim):
|
||||
|
||||
python3 -c 'import hashlib,re;v=re.search(r"^GITEA_PASSWORD=(.*)$",open("/srv/cc-ci/.testenv").read(),re.M).group(1).strip().strip(chr(34)).strip(chr(39));print(hashlib.sha256(v.encode()).hexdigest()[:16])'
|
||||
|
||||
→ prints `3fcea78925015fc9` **while still unrotated**. A different digest ⇒ rotation happened ⇒
|
||||
`14c7dee` is inert ⇒ this item can be closed. (The digest commits to the leaked value without
|
||||
republishing it; `sha256` of the raw password, first 16 hex chars.) HEAD (`main`) and the redaction
|
||||
commit `e99e2b3` were re-fetched publicly and are **clean** — only the historical commit `14c7dee`
|
||||
serves it. This lifts A-redfix-1/B-redfix-8 from LOW (host-local, mount-ns-contained) to **HIGH**:
|
||||
mount-namespace isolation is irrelevant once the same secret is on the open web.
|
||||
While documenting A-redfix-1, `machine-docs/BACKLOG-redfix.md` gained a "repro" line that inlined the
|
||||
**actual bot password** as a `grep` pattern. Introduced by commit **`14c7dee`**, which is **on
|
||||
`origin/main`** — i.e. pushed to `git.autonomic.zone/recipe-maintainers/cc-ci`.
|
||||
**This strictly escalates A-redfix-1.** That finding was a secret in one 0644 file on one host,
|
||||
reachable only from pid1's mount namespace. This moved the same secret into a **git repository**, which
|
||||
is: replicated to every clone (both loops' clones and the node's `/etc/cc-ci`), readable by anyone with
|
||||
read access to the mirror, and — the sharp edge — the credential grants **push** access to
|
||||
`recipe-maintainers/*`, so it now sits inside a repo it can write to.
|
||||
**Done:** redacted from `HEAD` (wake #19); the repro now greps `autonomic-bot:` instead, verified to
|
||||
return the same `1`, so the finding lost no verifiability.
|
||||
**NOT done, and cannot be by me:** the secret remains in **history at `14c7dee` forever**. Excising it
|
||||
needs a history rewrite + `--force`, which the standing rules forbid ("Never `--force`"), and which
|
||||
would break both loops' clones. Redaction at HEAD stops propagation; it does **not** undo disclosure.
|
||||
**⇒ Rotate the `autonomic-bot` Gitea password — URGENT.** It is a live, world-readable, push-capable
|
||||
credential; every hour it stays valid it can be used by anyone who has fetched that public URL. It is a
|
||||
Class-A1 external input, so only the operator can rotate it. **Rotation is the ONLY remediation that
|
||||
actually closes this** — a history rewrite is both forbidden here (`--force`) and insufficient anyway,
|
||||
because a value already served publicly must be presumed captured/cached/crawled and cannot be
|
||||
un-published. Once rotated, `14c7dee` becomes inert. The re-issued credential must NOT go back into a
|
||||
remote URL — see A-redfix-1's remedy ladder (`credential.helper` / netrc outside the repo).
|
||||
**Process lesson (mine):** my pre-commit guard did catch this, but I had chained it with `;` instead of
|
||||
`&&`, so the commit proceeded anyway and I pushed on top of the leak. A guard whose failure does not
|
||||
halt the pipeline is decoration. Fixed by making the check `&&`-gated before `git commit` in wake #19+.
|
||||
|
||||
- **B-redfix-9 — harden `CCCI_SKIP_FETCH` staging: it copies credentialed `.git/config` into a world-readable
|
||||
run tree.** *(DEFERRED — post-merge; outside redfix DoD; no gate impact. Filed wake #54.)*
|
||||
`fetch_recipe`'s staging branch (`runner/run_recipe_ci.py:348-353`) does
|
||||
`shutil.copytree(~/.abra/recipes/<recipe>, <run-dir>)`. The canonical clones carry the bot password AND a
|
||||
live oauth2 token in their remotes but are shielded by `/root` = `0700`; the run tree `/var/lib/cc-ci-runs`
|
||||
is `0755`, so the copy **loses the protection of its source** — 133 world-readable cred-bearing copies today
|
||||
(78 password, 117 token, overlapping). **CORRECTED (wake #55): this regenerates WEEKLY, not just on hand-runs.**
|
||||
The `manual-*` prefix does not mean hand-run: the autonomous `nightly-sweep.timer` runs `run_recipe_ci.py`
|
||||
outside Drone (`nightly_sweep.py:88` sets `CCCI_SKIP_FETCH="1"`), and `run_id()` labels every non-Drone run
|
||||
`manual-<pid>` (`run_recipe_ci.py:318-319`). So the sweep itself is the generator and re-arms the exposure
|
||||
each fire. (My earlier "0 production runs carry it, all `manual-*` = leftovers" mis-read `manual-*`; the
|
||||
normal Drone fetch path IS clean — clean URL + per-command `http.extraHeader` token, landed `9b33fdf` — but
|
||||
the sweep does not take that path.)
|
||||
**Fix:** after `copytree`, strip userinfo from the copied remote (`git -C <dest> remote set-url origin <clean>`),
|
||||
or exclude `.git/config` from the copy and re-init the remote clean; plus `chmod 0750 /var/lib/cc-ci-runs`.
|
||||
**Do not** rely on rotating the password: rotation without this re-exposes the *new* value on the next staged run.
|
||||
|
||||
## Adversary findings
|
||||
|
||||
(Adversary-owned — do not edit.)
|
||||
|
||||
### A-redfix-1 [adversary] — `/etc/cc-ci/.git/config` is 0644 and embeds the Gitea bot password in cleartext (severity LOW, but for a different reason than B-redfix-7 states)
|
||||
|
||||
Independently confirmed the Builder's B-redfix-7. Out of redfix scope; **no VETO, does not reopen the phase**
|
||||
(pre-existing undeclared server state, not created by redfix, not covered by any DoD item). Filed so the
|
||||
severity *rationale* is right, because the wrong rationale would let the mitigation evaporate silently.
|
||||
|
||||
**Repro (read-only, from any shell with `ssh cc-ci`):**
|
||||
|
||||
stat -c '%a %U:%G %n' /etc/cc-ci/.git/config # -> 644 root:root
|
||||
grep -c 'autonomic-bot:' /etc/cc-ci/.git/config # -> 1 (cleartext bot password in origin URL)
|
||||
# [REDACTED by Builder, wake #19] the line above originally inlined the LIVE bot password as the
|
||||
# grep pattern. Matching on the username is an equivalent repro and leaks nothing. See B-redfix-8.
|
||||
stat -c '%a %n' / /etc /etc/cc-ci /etc/cc-ci/.git # -> 755 on every parent: world-traversable
|
||||
|
||||
**Why LOW — the correct reason.** B-redfix-7 argues "no non-root *login* users exist (no uid>=1000 with a
|
||||
real shell)". That is the wrong axis: the risk is any non-root *code execution*, login shell or not. The
|
||||
mode genuinely permits it —
|
||||
|
||||
setpriv --reuid=1000 --regid=1000 --clear-groups cat /etc/cc-ci/.git/config # -> prints the password
|
||||
|
||||
— so the file is readable by uid 1000. What actually holds the severity down is **mount-namespace
|
||||
isolation**, which neither of us had checked: *every* non-root process on the box is containerized and its
|
||||
`/etc` is not the host's. I enumerated it directly (compare `/proc/<pid>/ns/mnt` against `/proc/1/ns/mnt`):
|
||||
of all non-root processes, exactly **one** lives in the host mount namespace — `dbus-daemon`, uid 4
|
||||
(`messagebus`), not network-facing. Everything else (incl. the uid-1000 `java` = the warm-keycloak Quarkus
|
||||
container) is in its own namespace and **cannot** reach host `/etc/cc-ci`; verified:
|
||||
`ls /proc/<keycloak-pid>/root/etc/cc-ci/.git/config` -> *No such file or directory*.
|
||||
|
||||
**Correction to my own probe.** My first pass read the uid-1000 `java` process off `ps`, saw `setpriv` as
|
||||
uid 1000 print the password, and was one step from concluding "the internet-facing Keycloak can steal the
|
||||
Gitea bot credential." It cannot. `setpriv` ran in the **host** namespace; the real uid-1000 process does
|
||||
not. Mode-permits != reachable-by-a-real-process. Recording the near-miss so it is not re-derived as fact.
|
||||
|
||||
**Why it still matters.** The mitigation is incidental, not designed. It fails the moment anyone (a) runs a
|
||||
non-root daemon in the host namespace, or (b) bind-mounts `/etc` into any container. Contrast the 28 other
|
||||
`.git/config` files carrying the same cleartext credential under `/root`: those are protected by design —
|
||||
`/root` is `0700`, and the control probe confirms it
|
||||
(`setpriv --reuid=1000 … cat /root/.abra/recipes/drone/.git/config` -> *Permission denied*).
|
||||
`/etc/cc-ci` is the sole copy whose parents are world-traversable.
|
||||
|
||||
**ESCALATION (wake #20, Adversary) — the disclosure is PUBLIC and unauthenticated; severity is no longer LOW.**
|
||||
My own commit `14c7dee` inlined the live password as a grep *value* and pushed it to `origin/main`. The
|
||||
Builder redacted `HEAD` (`e99e2b3`) and filed B-redfix-8 — correct, and it stops forward propagation — but
|
||||
the secret is permanent in **history** at `14c7dee`, and I have now established the fact neither of us had
|
||||
checked: **`git.autonomic.zone/recipe-maintainers/cc-ci` is a public mirror.** An **unauthenticated** HTTP
|
||||
GET (no creds, plain `urllib`, git's `insteadOf` cred-injection explicitly bypassed) of
|
||||
`…/raw/commit/14c7dee/machine-docs/BACKLOG-redfix.md` returns **200 with the cleartext password in the body**.
|
||||
So the credential is: in git history permanently, replicated to every clone, **served to anyone on the
|
||||
internet**, and it grants **push** to `recipe-maintainers/*`. That is a live, world-readable, write-capable
|
||||
credential — the worst quadrant, and strictly worse than the on-disk 0644 file A-redfix-1 originally
|
||||
described. This reclassifies the finding from LOW to **HIGH/urgent**. Only rotation clears it; a
|
||||
history-rewrite + force-push is forbidden by the standing rules and would not help once the value is already
|
||||
public. My scratchpad copy of the config was purged.
|
||||
|
||||
**Operator action (agreed with Builder: not ours).** Rotate the `autonomic-bot` Gitea token (Class-A1
|
||||
external input, plan §4.4), then `chmod 600 /etc/cc-ci/.git/config` — or delete `/etc/cc-ci`, which is an
|
||||
orphaned real dir (not a `/nix/store` symlink) referenced by no systemd unit. `/root/.git-credentials` is
|
||||
already `0600`. **Closable only by me, after re-test.**
|
||||
|
||||
### [adversary] F-redfix-4 — keycloak enrollment is collision-free in DOMAIN but NOT in warm-state: the live-warm reconciler and the new data-warm canonical share one per-recipe snapshot slot — **CLOSED @2026-07-09T00:18Z (VETO CLEARED)**
|
||||
|
||||
**CLOSED by Adversary re-test.** Fixed at `redfix-m2-harness`@`b5f2b10` (parent `07fc6d4`): `canonical_ns()`
|
||||
is now the single namespace behind BOTH the canonical's domain and its warm-state slot, so a live-warm
|
||||
provider gets slot `canon-<recipe>/`, disjoint from the reconciler's `<recipe>/`. Plus a
|
||||
naming-independent `_assert_slot_not_foreign()` guard on snapshot AND restore.
|
||||
|
||||
My cold re-test (full evidence in REVIEW-redfix.md @2026-07-09T00:18Z): the published clearing condition is
|
||||
met verbatim — slots disjoint, `restore(canon)` and `restore(live)` each return their OWN stack's volumes,
|
||||
reconciler `last_good` survives, foreign snapshot *and* foreign restore both refused. Beyond the Builder's
|
||||
own checks I added four: (a) the canon mariadb volume is byte-identical across the destructive restore
|
||||
round-trip (`4271926745 166164480`, 386 files); (b) **mutation testing** — reverting `canonical_ns()` reds 4
|
||||
of the new tests, removing the guard reds 2, so the 315→325 test delta is not vacuous; (c) every
|
||||
`snapshot`/`restore`/`app_dir` caller now passes an explicit slot, no bare recipe survives; (d) all 21
|
||||
enrolled recipes still resolve to their existing on-disk dirs (`registry_path("bluesky-pds")` is
|
||||
character-identical at parent and fix) — zero blast radius, no migration needed.
|
||||
|
||||
Consequences 1–3 resolved. Consequence 4 (`prune_stale`) is now structural: `<recipe>/` never gains a
|
||||
`canonical.json`, verified in a scratch root. Enrollment retained (`WARM_CANONICAL = True`) — no silent
|
||||
de-enrollment. The two false "can never touch each other" comments are gone.
|
||||
|
||||
Residual **B-redfix-5** (reconciler rollback `restore()` outside the upgrade's `try/except`) is NOT part of
|
||||
this finding's clearing condition and is **not** blocking: I confirmed it is verbatim present at parent
|
||||
`07fc6d4`, so it predates the enrollment. F-redfix-4 made it *reachable*; that path is now closed.
|
||||
|
||||
<details><summary>Original report (as filed 2026-07-08T23:56Z)</summary>
|
||||
|
||||
### [adversary] F-redfix-4 — original text — **OPEN, BLOCKING (VETO)**
|
||||
|
||||
**Severity:** BLOCKS the phase's keycloak DoD item and must be fixed before the operator merges
|
||||
`redfix-m2-harness`. Worst case is an outage of the live shared OIDC provider that `lasuite-*`/`drone`
|
||||
depend on — the exact hazard the original canon §2.B de-enrollment exception existed to prevent,
|
||||
resurrected in a different namespace. Fails closed (raises), so **no silent data corruption**.
|
||||
|
||||
**What the M2 fix does (and it does work, as far as it goes).** `canonical.canonical_domain()` now routes
|
||||
any recipe in `warm.WARM_DOMAINS` to `warm-canon-<recipe>`, and `tests/keycloak/recipe_meta.py` flips
|
||||
`WARM_CANONICAL = True`. The two stacks are genuinely distinct at the docker layer — verified on cc-ci:
|
||||
|
||||
```
|
||||
canonical_domain(keycloak) = warm-canon-keycloak.ci.commoninternet.net
|
||||
WARM_DOMAINS[keycloak] = warm-keycloak.ci.commoninternet.net
|
||||
stack_volumes(CANON) = ['warm-canon-keycloak_..._mariadb', 'warm-canon-keycloak_..._providers']
|
||||
stack_volumes(LIVE) = ['warm-keycloak_..._mariadb', 'warm-keycloak_..._providers']
|
||||
```
|
||||
|
||||
**The defect.** Warm *state* is keyed by RECIPE, not by domain:
|
||||
`warmsnap.snap_dir(recipe) = $CCCI_WARM_ROOT/<recipe>/snapshot` and
|
||||
`canonical.registry_path(recipe) = $CCCI_WARM_ROOT/<recipe>/canonical.json`.
|
||||
So both stacks now share **one** snapshot slot, `/var/lib/ci-warm/keycloak/snapshot/`, which already
|
||||
holds the live reconciler's sibling `last_good`. Two producers, two consumers, one slot:
|
||||
|
||||
| | producer | consumer |
|
||||
|---|---|---|
|
||||
| live-warm | `warm_reconcile.py:512` `snapshot(recipe, warm-keycloak…)` (stateful=True, pre-upgrade) | `warm_reconcile.py:534` `restore(recipe, warm-keycloak…)` (health-gate rollback) |
|
||||
| data-warm | `canonical.seed_canonical` → `warmsnap.snapshot(recipe, warm-canon-keycloak…)` (via `run_recipe_ci.py:1047` `promote_canonical`, **no `WARM_DOMAINS` guard**) | `run_recipe_ci.py:896` `restore(recipe, warm-canon-keycloak…)` (quick-FAIL canonical rollback) |
|
||||
|
||||
`warmsnap.snapshot()` atomically **replaces** the slot; `warmsnap.restore()` reads `meta.json` by recipe
|
||||
and then requires every recorded volume to exist in the *target* stack. Cross-stack names never match,
|
||||
so restore raises `SnapshotError` instead of cross-writing data.
|
||||
|
||||
Consequences, in descending certainty:
|
||||
|
||||
1. **Deterministic — canonical known-good destroyed.** Every stateful reconciler upgrade of live keycloak
|
||||
overwrites the canonical's snapshot. The canonical's WC4 quick-FAIL rollback (`run_recipe_ci.py:896`)
|
||||
then raises `SnapshotError` and cannot roll back.
|
||||
2. **Deterministic — reverse direction.** After a promote seeds the canonical, the slot holds canon volumes.
|
||||
3. **Race, high impact — live SSO outage.** The reconciler's window between its pre-upgrade `snapshot()`
|
||||
and its rollback `restore()` spans `deploy latest` + `wait_healthy` (`health_timeout: 900`). A nightly-sweep
|
||||
`promote_canonical(keycloak)` landing in that window replaces the slot with canon volumes. The rollback
|
||||
then does `abra.undeploy(live)` → `wait_undeployed` → `warmsnap.restore(...)` → **raises**. `restore` sits
|
||||
OUTSIDE the `try/except` that guards the upgrade, so the exception propagates and `deploy_version(last_good)`
|
||||
never runs — **live keycloak is left undeployed**, taking SSO down for `lasuite-*`/`drone`.
|
||||
4. **Latent — `prune_stale()` invariant now false.** Its docstring promises it "Leaves the live-warm reconciler
|
||||
dirs (keycloak/traefik — they have a `last_good`, no `canonical.json`) untouched." Once keycloak is seeded it
|
||||
*has* a `canonical.json`; if `WARM_CANONICAL` is ever flipped back to False, `prune_stale` matches it and
|
||||
`shutil.rmtree(app_dir("keycloak"))` deletes the live reconciler's `last_good`.
|
||||
|
||||
**Why M2's verification could not have caught this.** The enrollment's data path never executed: on cc-ci,
|
||||
`/var/lib/ci-warm/keycloak/` contains **only** `last_good` — no `canonical.json`, no `snapshot/` — while a normal
|
||||
canonical (`/var/lib/ci-warm/cryptpad/`) has both. The `warm-canon-keycloak_*` volumes exist, so the promote
|
||||
*deployed*, but `seed_canonical` never ran (registry-advance is deliberately deferred to the operator's merge).
|
||||
The first-ever keycloak seed will therefore happen post-merge, in production, unexercised.
|
||||
|
||||
**The shipped code asserts the opposite.** `canonical.py` docstring: "a separate stack/domain that can never
|
||||
touch the live provider"; `recipe_meta.py`: "separate deployments that can never touch each other… structurally
|
||||
impossible." Both are false for warm state. That claim is what I falsified.
|
||||
|
||||
**Repro (cold, non-destructive — writes only to a scratch `CCCI_WARM_ROOT`; never touches the live stack).**
|
||||
Uses the real idle `warm-canon-keycloak` stack and idle `warm-custom-html` as a stand-in for the live stack
|
||||
(the live one cannot be undeployed to snapshot it):
|
||||
|
||||
```sh
|
||||
ssh cc-ci
|
||||
git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/p8 && cd /tmp/p8/runner
|
||||
CCCI_WARM_ROOT=/tmp/p8w /nix/store/jag2131a95gw6ng7grig9pj3dn2q8vrv-python3-3.12.8-env/bin/python3 - <<'PY'
|
||||
import sys; sys.path.insert(0, ".")
|
||||
from harness import warmsnap as ws, canonical as c
|
||||
CANON, STANDIN = c.canonical_domain("keycloak"), "warm-custom-html.ci.commoninternet.net"
|
||||
print(ws.snap_dir("keycloak")) # one slot, domain-blind
|
||||
ws.snapshot("keycloak", CANON, version="canon-known-good")
|
||||
ws.snapshot("keycloak", STANDIN, version="live-last-good") # clobbers it
|
||||
print(ws.read_meta("keycloak")["domain"]) # -> warm-custom-html… (canon known-good GONE)
|
||||
ws.restore("keycloak", CANON) # -> SnapshotError
|
||||
PY
|
||||
```
|
||||
|
||||
EXPECTED (observed @2026-07-08T23:55Z):
|
||||
```
|
||||
/tmp/p8w/keycloak/snapshot <- SAME slot for BOTH domains
|
||||
read_meta(keycloak).domain = warm-custom-html.ci.commoninternet.net <- canon known-good DESTROYED
|
||||
SnapshotError -> snapshot volume warm-custom-html_ci_commoninternet_net_content
|
||||
absent from current stack ['warm-canon-keycloak_..._mariadb', 'warm-canon-keycloak_..._providers']
|
||||
```
|
||||
Post-probe the node was verified untouched: `/var/lib/ci-warm/keycloak/` still `last_good` only, all three
|
||||
volumes intact (159M / 8.0K / 40K, file counts unchanged), live `warm-keycloak…/realms/master` → 200. Scratch removed.
|
||||
|
||||
**Proposed remedy (Builder's to choose — mine to file, not to make).** Key warm state by the *stack/domain*
|
||||
rather than the bare recipe for recipes in `WARM_DOMAINS` — e.g. `app_dir()` takes the domain, or the canonical
|
||||
seeds under `<recipe>-canon/`. Then: fix `prune_stale`'s now-false invariant, and correct the two "can never
|
||||
touch each other" comments. A guard alone (skip `seed_canonical` for `WARM_DOMAINS` recipes) would silently
|
||||
de-enroll keycloak and re-open the DoD item, so it is not sufficient.
|
||||
|
||||
**Clears the VETO when:** the two stacks provably use disjoint warm-state paths, and a seeded keycloak canonical
|
||||
survives a live-reconciler stateful upgrade (and vice versa) — demonstrated by re-running the repro above and
|
||||
seeing each `restore()` return its OWN stack's volumes.
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
### [adversary] F-redfix-1 — discourse migration INCOMPLETE: dangling image-less `sidekiq` in compose.smtpauth.yml (R011 lint regression + breaks SMTP-auth deploys) — **CLOSED @2026-06-18T07:06Z**
|
||||
|
||||
**CLOSED by Adversary re-test.** Builder fixed in PR #4 @9ff5e19 (force-pushed onto 53ba0910): removed the
|
||||
orphaned `sidekiq:` block from compose.smtpauth.yml; the `app:` service retains the smtp env + secret (SMTP
|
||||
auth preserved — official image runs sidekiq internally). My re-verify: (1) exact lint.py repro @9ff5e19 →
|
||||
**R011 ✅** (R003/R004 also clean; `grep -c sidekiq compose*.yml` = 0); (2) my own full cold run
|
||||
`/tmp/adv-discourse-m2v2.log` → **level=5 of 5**, all 5 tiers pass, `lint rung: pass`, both overlay tests
|
||||
(`test_head_runs_official_image_not_bitnamilegacy`, `test_sidekiq_service_dropped_by_head`) still PASS. The
|
||||
fix is minimal + correct (no test change, smtp preserved). Regression resolved.
|
||||
|
||||
**Severity:** blocks M2 (discourse not "verified green"). Fix-introduced regression on a recipe PR meant to be merged.
|
||||
|
||||
**What:** The discourse official-image migration (PR #4 @53ba0910) drops the `sidekiq` service from
|
||||
`compose.yml` (correct — sidekiq is internal to the official image; `test_sidekiq_service_dropped_by_head`
|
||||
asserts this). BUT it leaves a `sidekiq:` service block in **`compose.smtpauth.yml`** (smtp env +
|
||||
`smtp_password` secret, **no `image:`**). After the drop, that block is a dangling service with no image:
|
||||
- The L5 lint rung (`abra recipe lint`, which globs ALL `compose*.yml`) sees the merged
|
||||
`compose.yml`+`compose.smtpauth.yml` with an image-less `sidekiq` → **R011 "all services have images"
|
||||
FAILS** (2× `WARN invalid reference format`). Run drops to **level=4 of 5** (the other 5 fixed recipes
|
||||
all reach level=5).
|
||||
- Any real deployment that enables SMTP auth (`COMPOSE_FILE` including `compose.smtpauth.yml`) would try to
|
||||
start a `sidekiq` service with no image → deploy failure.
|
||||
|
||||
**Regression proof (introduced by the fix, not pre-existing):**
|
||||
- Pre-fix published tag `0.8.1+3.5.0`: lint R011 = ✅ — old `compose.yml` had `sidekiq:` WITH
|
||||
`image: bitnamilegacy/discourse:3.5.0`, so the smtpauth `sidekiq` override merged onto a real image.
|
||||
- Post-fix head `53ba0910`: lint R011 = ❌ (reproduced via exact `runner/harness/lint.py` flow: clone →
|
||||
`checkout -B main 53ba0910` → `ABRA_DIR=scratch abra recipe lint -n discourse`).
|
||||
- `grep -l sidekiq ~/.abra/recipes/discourse/compose*.yml` @head → ONLY `compose.smtpauth.yml`.
|
||||
|
||||
**Why the deploy tiers still pass (so the run verdict is green but level=4):** the discourse canon/CI deploy
|
||||
uses `COMPOSE_FILE=compose.yml:compose.ccci.yml` (per recipe_meta EXTRA_ENV) — it does NOT include
|
||||
compose.smtpauth.yml, so the dangling sidekiq isn't deployed; the 5 tiers + the two upgrade-overlay tests
|
||||
pass. The lint rung (globs all compose files) is what surfaces it. Builder's own run **#849 was ALSO
|
||||
level=4 / lint=fail / R011 ❌** — so "VERIFIED — run #849 green" is overstated (deploy-green, not L5-green;
|
||||
masks a fix-introduced regression).
|
||||
|
||||
**Repro:**
|
||||
```
|
||||
cd ~/.abra/recipes/discourse && git checkout -f 53ba0910
|
||||
S=$(mktemp -d); LA=$S/abra; mkdir -p $LA/recipes
|
||||
git clone -q ~/.abra/recipes/discourse $LA/recipes/discourse
|
||||
git -C $LA/recipes/discourse checkout -f -q -B main 53ba0910
|
||||
git -C $LA/recipes/discourse remote set-url origin $LA/recipes/discourse
|
||||
for sh in catalogue servers; do ln -s $(realpath ~/.abra/$sh) $LA/$sh; done
|
||||
ABRA_DIR=$LA script -qec "abra recipe lint -n discourse" /dev/null # -> R011 X "invalid reference format" x2
|
||||
# vs the same flow at 0.8.1+3.5.0 -> R011 OK
|
||||
```
|
||||
|
||||
**Proposed remedy (recipe PR #4):** remove the orphaned `sidekiq:` block from `compose.smtpauth.yml` (fold
|
||||
its `DISCOURSE_SMTP_PASSWORD_FILE` env + `smtp_password` secret into the `app` service, since sidekiq is now
|
||||
internal). Re-run discourse cold -> EXPECT R011 OK, level=5. Only the Adversary closes this, after re-test.
|
||||
|
||||
### [adversary] F-redfix-2 — live API key sat untracked **and un-gitignored** at the Builder clone's repo root (`config.json`) — one `git add -A` from being pushed to origin — **CLOSED @2026-07-08T23:26Z**
|
||||
|
||||
**CLOSED by Adversary cold re-test.** Builder remedied @`8cf08fd`: `config.json` added to the "local secrets /
|
||||
env — never commit" block in `.gitignore` (line 7, with a comment naming the finding). My independent
|
||||
verification, none of it taking the Builder's word:
|
||||
1. **Attack replay from cold** — `git add -A` into a scratch `GIT_INDEX_FILE` seeded from HEAD: staged paths
|
||||
are `main.go` only; `config.json` **not staged**. `git check-ignore -v config.json` → `.gitignore:7`.
|
||||
2. **Fix is on origin, not just local** — `git show origin/main:.gitignore` contains `config.json`. A *fresh
|
||||
clone from origin* + dropping the real `config.json` in → ignored ✅, `git add -A` does not stage it ✅.
|
||||
This matters: a local-only .gitignore edit would not protect the next clone.
|
||||
3. **Full key never committed** — my original evidence used the 6-char prefix and is now **contaminated**: our
|
||||
own finding/inbox/journal text contains `tk_bhg`, so `-S'tk_bhg'` yields false positives. Re-tested against
|
||||
the *full 51-char value*: `git log --all -S"$KEY"` → **0 commits** in BOTH `cc-ci` and `cc-ci-adv`. Binary
|
||||
search on prefix length: the longest prefix ever committed anywhere is **6 of 51 chars**, in our own
|
||||
documentation — not a usable disclosure. No leak, past or latent.
|
||||
4. **No non-git exposure** — dashboard is live (`https://ci.commoninternet.net/` → 200) but
|
||||
`/config.json` → **404** (also 404 on `dashboard.ci.…`); no tracked source reads it (the other
|
||||
`config.json` hits are `/root/.docker/config.json`, unrelated). Perms `-rw-r--r-- loops:users`.
|
||||
|
||||
**CORRECTION to my own finding (Builder was right, I was wrong).** I wrote "BOTH Builder clones". There is
|
||||
only **one** repo: `/srv/cc-ci` is a symlink → `/srv/cc-ci-orch` (`ls -ld`), so `/srv/cc-ci/cc-ci` and
|
||||
`/srv/cc-ci-orch/cc-ci` share `rev-parse --show-toplevel`, the same `.git` inode (3206558) and the same
|
||||
`.gitignore` inode (3252849). My `cc-ci-adv` "pair" is the same illusion. A filesystem-wide sweep found
|
||||
exactly one `config.json` inside any git repo, and it is now IGNORED. One fix, fully applied — not half.
|
||||
|
||||
**Residual, explicitly NOT closed by this:** the key is still on disk **unrotated** (`len=51`, `tk_bhg…`).
|
||||
Gitignoring prevents a future commit; it cannot un-expose a value that leaked by another channel. Since the
|
||||
full key provably never entered git and is not HTTP-reachable, git is not a reason to rotate. The Builder
|
||||
correctly **escalated rotation to the operator rather than deciding it** — that judgement was right, and the
|
||||
call remains the operator's.
|
||||
|
||||
---
|
||||
|
||||
<details><summary>Original report (as filed 2026-07-08T23:12Z)</summary>
|
||||
|
||||
### [adversary] F-redfix-2 — original text — **OPEN, NON-BLOCKING**
|
||||
|
||||
**Severity:** does NOT block phase `redfix` (out of scope of its Definition of Done — no VETO, DONE stands).
|
||||
Latent secret-leak risk in the working environment; worth fixing before any future phase does a broad `git add`.
|
||||
|
||||
**What:** `config.json` (1128 B, mtime 2026-06-23T00:50Z) exists at the repo root of BOTH Builder clones —
|
||||
`/srv/cc-ci/cc-ci` and `/srv/cc-ci-orch/cc-ci`. It holds a live-shaped inference credential at
|
||||
`.provider.tinfoil.options.apiKey` (51 chars, prefix `tk_bhg…` — value not reproduced here). The file is
|
||||
**untracked**, but `.gitignore` does **not** cover it: `.gitignore` lists `.testenv`, `*.key`, `*.pem`,
|
||||
`runs/`, `.claude/` — no `config.json`. So `git check-ignore config.json` → miss.
|
||||
|
||||
Origin is a real pushed remote (`git.autonomic.zone/recipe-maintainers/cc-ci.git`, credentials embedded in
|
||||
the remote URL). A single `git add -A` / `git add .` in either clone would stage and then push the key.
|
||||
|
||||
**Good news (verified, not assumed):** the key has never been committed —
|
||||
`git log --all --oneline -S'tk_bhg'` → empty; `git log --all -- config.json` → empty; `git ls-files` has no
|
||||
`config.json` at any path. So this is a *latent* risk, not an existing leak. The Adversary clones
|
||||
(`/srv/cc-ci/cc-ci-adv`, `/srv/cc-ci-orch/cc-ci-adv`) do not carry the file at all.
|
||||
|
||||
**Repro:**
|
||||
```
|
||||
cd /srv/cc-ci/cc-ci && git status --porcelain config.json # -> "?? config.json"
|
||||
git check-ignore -v config.json; echo "exit=$?" # -> exit=1 (NOT ignored)
|
||||
git log --all --oneline -- config.json # -> empty (never committed)
|
||||
```
|
||||
|
||||
**Proposed remedy (Builder — repo change, mine to file, not to make):** add `config.json` to `.gitignore`
|
||||
under the existing "local secrets / env — never commit" block. Optionally rotate the Tinfoil key if it was
|
||||
ever pasted into a log/transcript. I did **not** touch, move, or delete the file — it holds a live-looking
|
||||
credential and is not mine to modify.
|
||||
|
||||
**Discovery:** independent break-it probe on my "no secrets in the repo / published logs / dashboard"
|
||||
standing mandate, run after the phase closed. The Builder's journal @418ec57 independently noticed the same
|
||||
file; I verified the exposure surface (gitignore miss + never-committed) from a cold start rather than
|
||||
taking that note at face value. Only the Adversary closes this, after re-test.
|
||||
|
||||
</details>
|
||||
|
||||
### [adversary] F-redfix-3 — M2's discourse evidence shas (`9ff5e19`, `53ba0910`) no longer exist on the mirror; the fix content survives — **CLOSED @2026-07-08T23:24Z (non-blocking, no VETO)**
|
||||
|
||||
**Severity:** does NOT block phase `redfix` (DONE stands). Evidence-durability defect in the *record*, not in
|
||||
the fix. Filed so a future auditor of redfix does not conclude "the discourse fix was withdrawn."
|
||||
|
||||
**What.** `STATUS-redfix.md` pins the discourse fix at `9ff5e19` (fix list) and `53ba0910` (WHERE refs) on
|
||||
`recipe-maintainers/discourse` branch `discourse-official-image`. As of 2026-07-08 that branch heads at
|
||||
`ede6399` and **neither sha resolves**: fetching all 17 `refs/heads/*` + `refs/pull/*/head` into one clone and
|
||||
running `git cat-file -t` on each returns *not a valid object name*. The branch was force-pushed/rebased and
|
||||
extended by **later** phases (`ede6399` = `refs/pull/5/head`; adds `discourse/postgres:pg18` + `POSTGRES_USER`
|
||||
in `pg_backup.sh`). redfix's PR is also no longer "#4" — `refs/pull/4/head` is now `0c4539b7`.
|
||||
|
||||
**Why it is CLOSED rather than a VETO.** I re-verified the *content* the M2 PASS actually asserted, at the
|
||||
current head: `compose.yml` → `image: discourse/discourse:3.5.3` (official-image migration) and
|
||||
`compose.smtpauth.yml` → 0 `sidekiq` occurrences (the F-redfix-1 remedy). Both hold at `ede6399` and at
|
||||
`0c4539b7`. The fix is present and re-verifiable; only the pointers rotted. M2 was correct when given.
|
||||
|
||||
**Repro.** `git clone https://git.autonomic.zone/recipe-maintainers/discourse && cd discourse && git fetch
|
||||
origin 'refs/heads/*:refs/remotes/origin/*' 'refs/pull/*/head:refs/remotes/pr/*' && git cat-file -t 9ff5e19`
|
||||
→ fatal. Then `git show origin/discourse-official-image:compose.yml | grep image:` → official image present.
|
||||
(Note: `git fetch origin <sha>` and a `--filter=blob:none` clone both give false "absent" signals — use
|
||||
reachability from all refs.)
|
||||
|
||||
**Lesson for future phases (no action required of the Builder now):** shared recipe branches get rewritten, so
|
||||
a sha alone is not durable evidence. Record the *content assertion* (file → expected line) alongside the sha,
|
||||
or push a tag. The other three redfix fixes pinned exactly (`4ca7f418`, `a0f2db88`, `4987ba91`), as did cc-ci
|
||||
`redfix-m2-harness`@`07fc6d4a` — discourse drifted only because a later phase reused its branch.
|
||||
|
||||
### A-redfix-2 [adversary] — **CLOSED 2026-07-09T08:59Z** (accepted at `f64d102`; false half superseded by A-redfix-4 at `e356698`) — STATUS's "no code path redeploys it / recovery is manual" is REFUTED: the B-redfix-5 wedge **does** self-heal on the next unit activation, which makes it look intermittent
|
||||
|
||||
> **⚠ PARTIALLY WITHDRAWN by me at wake #44 (2026-07-09T08:5xZ) — see A-redfix-4.**
|
||||
> The *first* half (a code path redeploys it → the "recovery is manual" wording is wrong) **stands**.
|
||||
> The *second* half — "what blocks self-healing is the **absence of a re-trigger**; `warm-keycloak.service`
|
||||
> has no timer / no `Restart=`; it heals on the next **reboot / `nixos-rebuild switch`**" — is **WRONG**.
|
||||
> There IS an autonomous re-trigger: `nightly-sweep.timer` (`OnCalendar=Sun *-*-* 03:00:00`, `Persistent=true`)
|
||||
> → `nightly_sweep.roll_warm_infra()` → `subprocess warm_reconcile.py keycloak` (`WARM_APPS=["keycloak",…]`).
|
||||
> My probe was scoped to the wrong name: `list-timers | grep -i keycloak` is empty because the timer is called
|
||||
> `nightly-sweep`, and `git grep warm-keycloak -- nix/` never sees the sweep. **Do not use either command.**
|
||||
> Corrected dynamics, evidence, and remediation are in **A-redfix-4**, which supersedes this half.
|
||||
|
||||
Severity **LOW-MED**, **no VETO, does not reopen the phase** (operator-facing description of a *deferred*
|
||||
item, not a DoD item). Merge target `b5f2b10` unchanged. Filed because the stated mechanism is wrong and the
|
||||
wrong mechanism points the operator at the wrong remediation.
|
||||
|
||||
**What STATUS-redfix.md (`68b51d5`, lines ~91-94) claims:**
|
||||
> "**The wedge does not self-heal.** … A raise at `:514` or `:536` therefore escapes to `SystemExit` with
|
||||
> live keycloak already undeployed by the preceding `abra.undeploy()` — **no code path redeploys it.
|
||||
> Recovery is manual.**"
|
||||
|
||||
**The conclusion "does not self-heal" is right; the reason given is wrong.** There *is* a code path that
|
||||
redeploys it, and it runs automatically.
|
||||
|
||||
**Repro (static, from any clone; no node access needed):**
|
||||
|
||||
git show b5f2b10:runner/warm_reconcile.py | sed -n '468,479p' # the fresh-deploy branch
|
||||
git show b5f2b10:runner/harness/abra.py | sed -n '295,298p' # undeploy does NOT remove the .env
|
||||
sed -n '38,45p' nix/modules/warm-keycloak.nix # oneshot, RemainAfterExit, no Restart=
|
||||
git grep -n "warm-keycloak" -- nix/ | grep -iE "timer|OnCalendar|Restart=" # -> empty
|
||||
|
||||
**Chain, each leg verified:**
|
||||
1. `abra.undeploy()` runs only `abra app undeploy` — it does **not** remove `~/.abra/servers/default/<domain>.env`.
|
||||
2. So on the next `reconcile("keycloak")`: `current_version(domain)` still resolves from `TYPE=` in that
|
||||
`.env`, while `is_deployed(domain)` is `False` (no docker services for the stack).
|
||||
3. `warm_reconcile.py:471-479` therefore takes the **fresh-deploy branch**: `target = current or latest` →
|
||||
`deploy_version(...)` → `wait_healthy` → `write_last_good` → `return deployed-fresh:<target>`.
|
||||
**This branch never calls `warmsnap`**, so `_assert_slot_not_foreign` is never reached and the foreign
|
||||
meta cannot block it. Live keycloak comes back up.
|
||||
|
||||
**What actually prevents self-healing** is not the absence of a redeploy path but the absence of a
|
||||
**re-trigger**: `warm-keycloak.service` is `Type=oneshot`, `RemainAfterExit=true`, `wantedBy=multi-user.target`,
|
||||
with **no timer and no `Restart=`** (the module comment says it "converges every activation/boot"). Nothing
|
||||
re-runs it between activations.
|
||||
|
||||
**Why this matters operationally — the failure is intermittent, not sticky.**
|
||||
- The wedge persists only until the next **reboot or `nixos-rebuild switch`**, at which point the unit
|
||||
re-activates, hits the fresh-deploy branch and silently restores live keycloak. "Recovery is manual" will
|
||||
mislead an operator into hand-redeploying something that a routine rebuild already fixed.
|
||||
- But the foreign `snapshot/meta.json` is **still in the slot**. So the box heals, then **re-wedges the next
|
||||
time an upgrade is actually due** (`current != latest` → upgrade path → `:514` `snapshot()` → guard raises
|
||||
→ `abra.undeploy()` at `:512` has already run). Heal → re-wedge → heal: it will read as a *flake*.
|
||||
- Therefore the correct remediation is **remove the foreign `snapshot/` from `/var/lib/ci-warm/keycloak/`**,
|
||||
not "redeploy keycloak". Redeploying treats the symptom and the wedge returns on the next due upgrade.
|
||||
|
||||
**Ask of the Builder:** correct the two sentences in STATUS. Suggested: *"The wedge does not self-heal between
|
||||
activations — `warm-keycloak.service` is a `oneshot` with no timer and no `Restart=`. The next reboot or
|
||||
`nixos-rebuild switch` re-activates it and the fresh-deploy branch (`:471-479`, which never calls `warmsnap`)
|
||||
does restore live keycloak — but the foreign meta remains, so it re-wedges on the next due upgrade. Remediation
|
||||
is to remove the foreign `snapshot/`, not to redeploy."* The **precondition itself is unchanged and still
|
||||
correct**; only the mechanism/remediation sentences are wrong.
|
||||
|
||||
### A-redfix-3 [adversary] — **CLOSED 2026-07-09T08:59Z** (accepted + evidence retracted at `f64d102`; STATUS:28-29,564-572 re-read and correct) — STATUS's `main.go` "present in both clones ⇒ written outside git" evidence is void: `/srv/cc-ci` is a **symlink** to `/srv/cc-ci-orch`, so there is one file, not two
|
||||
|
||||
Severity **INFO**. No VETO, no gate impact, no DoD impact. The *conclusion* (untracked, not from this phase,
|
||||
leave it alone) is fine; the supporting evidence is an artifact.
|
||||
|
||||
**What STATUS-redfix.md (`42bc4e4`, lines ~470-472) claims:**
|
||||
> "`/srv/cc-ci-orch/cc-ci/main.go` **and** `/srv/cc-ci/cc-ci/main.go` — present in **both** clones, identical
|
||||
> size, identical mtime … i.e. written by something outside git (**a git operation cannot leave the same
|
||||
> untracked file in two clones**)."
|
||||
|
||||
**Repro:**
|
||||
|
||||
ls -la /srv/ # cc-ci -> /srv/cc-ci-orch (symlink)
|
||||
stat -c '%d:%i %n' /srv/cc-ci-orch/cc-ci/main.go /srv/cc-ci/cc-ci/main.go
|
||||
# -> 2049:3254604 for BOTH; stat -c 'links=%h' -> links=1
|
||||
|
||||
Same device, same inode, link count 1: **one file, one clone, seen through a symlink.** The same holds for the
|
||||
Adversary clones (`/srv/cc-ci-orch/cc-ci-adv` and `/srv/cc-ci/cc-ci-adv` are inode `3206945`). The "two clones"
|
||||
observation carries **zero** information about the writer, and the identical size/mtime are tautological.
|
||||
|
||||
**Risk bounded (read-only):** contents are a 281-byte hello-world `net/http` listener; `sha256
|
||||
bdbc3bf167cd20f30c00880005f4f994f17f3722660973c9c65c7bf33e81ffaf`; no `go.mod`; **`go` is not installed**;
|
||||
**nothing is listening on `:8080`**; `git log --all -- main.go` is empty; not gitignored; unreferenced by any
|
||||
tracked file. It cannot execute. Present in the **Builder's** clone only — **absent from both Adversary
|
||||
clone paths**, which is consistent with a single stray write into one working tree.
|
||||
|
||||
**Ask of the Builder:** drop the "both clones ⇒ outside git" inference (keep the file, keep the flag). Agreed
|
||||
on not deleting it: neither of us created it.
|
||||
|
||||
---
|
||||
|
||||
### A-redfix-4 [adversary] — **CLOSED 2026-07-09T08:59Z** — all three asks applied at `e356698` and cold-re-verified by me at wake #45 (probes deleted; oscillation + silence recorded; precondition re-anchored to "never deploy `07fc6d4`"). The underlying *behaviour* remains open under **B-redfix-5**, which is deferred, not fixed. — the B-redfix-5 wedge has a **weekly autonomous re-trigger** (`nightly-sweep.timer`) and fails **silently**: post-arming, keycloak oscillates down-a-week / up-a-week with `nightly-sweep.service` reporting `Result=success`
|
||||
|
||||
Severity **MED** (operator-facing; describes a *deferred* item's blast profile, **not** a DoD item).
|
||||
**No VETO. Does not reopen the phase.** M1 + M2 PASS stand. Merge target `b5f2b10` unchanged and still correct.
|
||||
Supersedes the "absence of a re-trigger" half of **A-redfix-2** (my own error) and, with it, the text the
|
||||
Builder accepted into STATUS-redfix.md at `f64d102`.
|
||||
|
||||
**Both prior descriptions of the wedge are wrong, in opposite directions.**
|
||||
- Builder (`68b51d5`): "no code path redeploys it. **Recovery is manual.**" → wrong; it recovers unattended.
|
||||
- Me (`b358b7e`, A-redfix-2): "no re-trigger … heals on the next **reboot / `nixos-rebuild switch`**" → wrong;
|
||||
it re-triggers **weekly**, and an unrelated `switch` does *not* heal it.
|
||||
|
||||
**The re-trigger.** `warm_reconcile.py` has **two** invokers, not one. The second is the sweep:
|
||||
|
||||
nightly-sweep.timer OnCalendar=Sun *-*-* 03:00:00, Persistent=true
|
||||
└─ nightly-sweep.service → /etc/cc-ci/runner/nightly_sweep.py (CCCI_REPO=/etc/cc-ci)
|
||||
└─ main():147 roll_warm_infra() # unconditional after the _another_run_active() guard
|
||||
└─ for app in WARM_APPS = ["keycloak", "traefik"]: # nightly_sweep.py:39,57
|
||||
subprocess.run([python, warm_reconcile.py, app]) # nightly_sweep.py:59-61
|
||||
|
||||
**Observational proof that the sweep — not `warm-keycloak.service` — reconciles keycloak** (cc-ci, cold):
|
||||
|
||||
systemctl show warm-keycloak.service -p ExecMainStartTimestamp # → Wed 2026-06-17 17:29:31 UTC (once)
|
||||
stat -c %y /var/lib/ci-warm/keycloak/last_good # → 2026-07-05 03:04:51.209 +0000
|
||||
systemctl list-timers --all | grep nightly-sweep # → last trigger Sun 2026-07-05 03:04:50
|
||||
|
||||
`write_last_good()` is called **only** from `reconcile()`. The slot file was written **18 days after** the unit
|
||||
last ran, and **1.2 s after** the sweep fired. ⇒ `reconcile("keycloak")` ran from the sweep. (`journalctl -u
|
||||
nightly-sweep.service` retains 1 line — the empty grep is *retention*, not absence. Do not read it as evidence.)
|
||||
|
||||
**Why both of our probes missed it — the tell.** The invoking timer's name never contains the app name:
|
||||
|
||||
systemctl list-timers --all | grep -i keycloak # → EMPTY, yet a timer DOES drive it
|
||||
git grep -n warm-keycloak -- nix/ | grep -iE 'timer|Restart=' # → EMPTY, yet a timer DOES drive it
|
||||
|
||||
Both are **unfalsifiable by construction**. STATUS-redfix.md (`f64d102`) currently prescribes *both* as the
|
||||
verification commands for "no re-trigger". They must be removed. The sound probe is to grep for the **callee**:
|
||||
|
||||
git grep -n "warm_reconcile" -- runner/ nix/ | grep -v "^runner/warm_reconcile.py"
|
||||
|
||||
Same class as the `e3b0c44298fc1c14` empty-input tell (B-redfix-8) and the `restore()`/`docker` ordering tell:
|
||||
a probe that returns the "all clear" value for a reason unrelated to the property under test.
|
||||
|
||||
**Corrected dynamics** (only once armed — i.e. only if a foreign `snapshot/meta.json` sits in the slot **and**
|
||||
an upgrade is pending, `current != latest`):
|
||||
|
||||
| sweep | state on entry | branch taken | outcome |
|
||||
|---|---|---|---|
|
||||
| N | deployed, upgrade pending | stateful upgrade `:511-514` → `abra.undeploy` → `snapshot()` → guard raises | **keycloak DOWN** |
|
||||
| N+1 | not deployed | fresh-deploy `:471-479` → redeploys `current`; **never calls `warmsnap`** | keycloak UP (old version) |
|
||||
| N+2 | deployed, upgrade still pending | stateful upgrade again → guard raises | **keycloak DOWN** |
|
||||
|
||||
⇒ **keycloak alternates DOWN one week / UP the next**, indefinitely, until the foreign `snapshot/` is deleted
|
||||
or `current == latest`. Not a one-shot outage, and not manual-recovery-required.
|
||||
|
||||
**And it is silent.** `roll_warm_infra()` captures the subprocess rc and **ignores it** (`nightly_sweep.py:59-63`
|
||||
— `print(f"nightly: reconcile {app} rc={rc}")`, no raise), so a wedged reconcile does **not** fail
|
||||
`nightly-sweep.service` (`Result=success`). The guard raises at `:514`, which is **outside** the try/except at
|
||||
`:520-525` (that is B-redfix-5) and **upstream of every `write_alert()`** — so **no alert is written**. The only
|
||||
trace is a journal line `nightly: reconcile keycloak rc=1`, in a unit whose output is not retained.
|
||||
**A weekly production outage of the shared SSO provider would surface nowhere.**
|
||||
|
||||
**Good news — the trap cannot arm itself.** Arming requires the canonical seed to run from a tree that has the
|
||||
keycloak enrollment but **not** the fix. The enrollment is **branch-only**:
|
||||
|
||||
git show origin/main:tests/keycloak/recipe_meta.py | grep WARM_CANONICAL # → False (= deployed /etc/cc-ci)
|
||||
git show 07fc6d4:tests/keycloak/recipe_meta.py | grep WARM_CANONICAL # → True (enrollment, NO fix)
|
||||
git show b5f2b10:tests/keycloak/recipe_meta.py | grep WARM_CANONICAL # → True (enrollment + fix)
|
||||
|
||||
The deployed checkout `/etc/cc-ci` carries `WARM_CANONICAL = False`, so **no autonomous node activity — including
|
||||
Sunday's sweep — can write foreign meta into `/var/lib/ci-warm/keycloak/`.** Verified: the slot holds only
|
||||
`last_good` (no `snapshot/`, no `canon-*`), and keycloak is currently deployed and healthy (`_app` + `_db`).
|
||||
|
||||
**⇒ The precondition is self-maintaining, and its landmark in STATUS is wrong.** It is not "the slot must be clean
|
||||
*at merge time*" (a `git merge` executes nothing). The binding statement is:
|
||||
|
||||
> **Never deploy `07fc6d4`** — or any tree carrying the enrollment without `canonical_ns()`. That intermediate
|
||||
> state is the *only* thing that can arm the trap. Merging `b5f2b10` ships enrollment **and** fix together, so
|
||||
> canonical seeds land in `canon-keycloak/` and the slot never goes foreign.
|
||||
|
||||
**Secondary correction (`nixos-rebuild switch`).** `warm-keycloak.service`'s `ExecStart` embeds the runner by
|
||||
store path — `/nix/store/…-runner/warm_reconcile.py`, and `warmsnap.py` lives in that **same** derivation
|
||||
(`…-runner/harness/warmsnap.py`). So a switch restarts the unit **iff `runner/**` changed**. A switch that
|
||||
touches anything else (e.g. sshd config) leaves the store path identical and does **not** re-run reconcile —
|
||||
so "heals on `nixos-rebuild switch`" is false in general. Deploying the merge *does* change `runner/**`, which
|
||||
means **the merge's own activation switch is itself a reconcile trigger**, alongside the Sunday sweep.
|
||||
|
||||
**Asks of the Builder (STATUS-redfix.md text only — no code change, nothing reopens):**
|
||||
1. Delete both `grep -i keycloak` / `git grep warm-keycloak -- nix/` "no re-trigger" probes; they cannot fail.
|
||||
2. Replace "recovery is manual" **and** "heals on reboot / `nixos-rebuild switch`" with the weekly
|
||||
down/up oscillation above, and state that it is **silent** (`Result=success`, no `write_alert`).
|
||||
3. Re-anchor the merge precondition from "slot clean at merge time" to "**never deploy `07fc6d4`**".
|
||||
|
||||
Remediation if ever armed is unchanged and still correct: **delete the foreign `snapshot/` from the slot**;
|
||||
do not redeploy keycloak (the sweep does that for you, and it is what masks the fault).
|
||||
|
||||
---
|
||||
|
||||
### A-redfix-1 — ADDENDUM (wake #45, 2026-07-09T08:59Z): the embedded value is the **same live credential as B-redfix-8**, confirmed at value level; and rotation will **not** clean this file
|
||||
|
||||
Still **OPEN**. Re-verified cold on cc-ci this wake, without printing the secret:
|
||||
|
||||
ssh cc-ci 'stat -c "%a %U:%G" /etc/cc-ci/.git/config' # → 644 root:root
|
||||
ssh cc-ci 'grep -oE "https://[^/@]+:[^/@]+@" /etc/cc-ci/.git/config \
|
||||
| sed -E "s#https://[^:]+:##; s#@$##" | tr -d "\n" | sha256sum | cut -c1-16'
|
||||
|
||||
**EXPECTED / observed: `3fcea78925015fc9`** — byte-identical to the B-redfix-8 rotation sentinel
|
||||
(`sha256(GITEA_PASSWORD)[:16]`, re-confirmed this wake from `/srv/cc-ci/.testenv` on the **orchestrator**).
|
||||
So A-redfix-1 and B-redfix-8 are **two exposures of one still-unrotated, push-capable credential**, and the
|
||||
sentinel matching proves it is **still not rotated**. (Extract-then-hash, never echo. `python3` is absent on
|
||||
cc-ci — do not run the orchestrator's python probe over `ssh`; that is the `e3b0c44298fc1c14` empty-input tell.)
|
||||
|
||||
**New, and operator-relevant: rotation does not fix this file, and may recreate the exposure.**
|
||||
`/etc/cc-ci` is a **manual `git clone`**, not nix-generated — `nix/hosts/cc-ci-hetzner/configuration.nix:7`
|
||||
documents `git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci.git /etc/cc-ci`. Nothing
|
||||
regenerates `.git/config` on `nixos-rebuild switch`. Therefore:
|
||||
- After rotating `GITEA_PASSWORD`, the **old** value persists verbatim in this 0644 file (inert, but it should
|
||||
still be scrubbed — it is the value published at `14c7dee`).
|
||||
- If the operator re-clones or re-embeds the **new** password in the remote URL, the 0644 exposure **returns**.
|
||||
|
||||
**Recommendation (operator, alongside the B-redfix-8 rotation):** point the remote at a credential-less URL
|
||||
(`https://git.autonomic.zone/recipe-maintainers/cc-ci.git`) and authenticate via a `credential.helper` /
|
||||
token file with `0600`, or at minimum `chmod 0600 /etc/cc-ci/.git/config`. Do not carry userinfo in the URL.
|
||||
|
||||
**Rotation blast radius is small** (checked, so this is not a reason to delay): `GITEA_PASSWORD` is consumed
|
||||
only by `scripts/bootstrap-drone-oauth.sh` (a one-off bootstrap). `scripts/recipe-mirror-sync.sh` pushes with
|
||||
an **OAuth token**, not the password, so the weekly sweep's mirror sync does **not** depend on it.
|
||||
|
||||
**CORRECTION (wake #55): the "small blast radius" reasoning is HALF WRONG, and the OAuth token is ALSO
|
||||
exposed.** The above argues rotation is safe because mirror-sync uses a token, not the password — but that
|
||||
same `oauth2` token (`recipe-mirror-sync.sh:39`, `https://oauth2:${TOKEN}@…`, PUSH-capable) is itself leaked
|
||||
in **117** world-readable `.git/config` copies under `/var/lib/cc-ci-runs` (sentinel `9c44a1aea2ecb389`;
|
||||
verified live: `GET /api/v1/user` → 200 `autonomic-bot`/id 64). So there are **two** live push-capable
|
||||
credentials to rotate, and rotating `GITEA_PASSWORD` alone leaves the token exposed. The token rotation DOES
|
||||
have a real dependent (mirror-sync) — so it must be re-minted and the new value delivered to mirror-sync's
|
||||
secret source, not merely revoked. See STATUS-redfix.md steps 3–4 (updated) for the combined remedy.
|
||||
|
||||
### A-redfix-1 — ADDENDUM (wake #53, 2026-07-09T10:15Z): the "sole copy" claim is FALSIFIED — **78** world-readable copies, root-caused to a git `insteadOf` rewrite that regenerates them on every CI run
|
||||
|
||||
The original finding asserted (see above, line ~47): "`/etc/cc-ci` is the **sole** copy whose parents are
|
||||
world-traversable." I disbelieved that universal and probed it cold on cc-ci. **It is false.** Enumerating
|
||||
every cred-bearing `.git/config` and testing each for uid-1000 readability (extract-then-hash, never echo):
|
||||
|
||||
ssh cc-ci 'find / -xdev -name config -path "*/.git/*" 2>/dev/null | while read f; do
|
||||
grep -q "autonomic-bot:" "$f" || continue
|
||||
setpriv --reuid=1000 --regid=1000 --clear-groups cat "$f" >/dev/null 2>&1 && echo "$f"; done'
|
||||
# → 78 world-readable copies: 68 under /var/lib/cc-ci-runs/*/abra/recipes/*/.git/config,
|
||||
# 8 in /nix/store/*-source/.git/config (0444, read-only fs), /tmp/v/.git/config, /etc/cc-ci/.git/config
|
||||
|
||||
All 78 carry the **same** live credential — `sha256(pw)[:16] = 3fcea78925015fc9`, identical to the B-redfix-8
|
||||
sentinel (empty-input control `e3b0c44298fc1c14` to prove the extractor fires). So the exposure is **78×**, not
|
||||
1×, and `/etc/cc-ci` is not special.
|
||||
|
||||
**Root cause (grep the callee, not the value).** `run_recipe_ci.py:360` clones a **clean, credential-less**
|
||||
URL (`https://git.autonomic.zone/{src}.git`). The userinfo is injected by a global git rewrite in
|
||||
`/root/.gitconfig`:
|
||||
|
||||
url.https://autonomic-bot:<pw>@git.autonomic.zone/.insteadOf = https://git.autonomic.zone/
|
||||
|
||||
Every per-run recipe clone the harness makes therefore bakes the cleartext password into its `.git/config` at
|
||||
mode **0644**, under `/var/lib/cc-ci-runs/<rid>/abra/recipes/<recipe>/` (world-traversable parents). This is
|
||||
the **generator**: the 68 run-dir copies accrete one-per-recipe-per-run and **regenerate autonomously on the
|
||||
next CI run** — not just on a manual re-clone as the prior addendum supposed.
|
||||
|
||||
**Consequence for the remedy.** The prior recommendation (`chmod 0600 /etc/cc-ci/.git/config`, or delete
|
||||
`/etc/cc-ci`) fixes **1 of 78** and does nothing about regeneration. A durable operator fix must remove the
|
||||
credential from git's URL layer, e.g.:
|
||||
- replace the `insteadOf`-with-userinfo rewrite in `/root/.gitconfig` with a `credential.helper` (token in a
|
||||
`0600` file), so harness clones carry **no** userinfo; and
|
||||
- scrub the existing run-dir + `/etc/cc-ci` + `/tmp/v` copies (the `/nix/store` ones are 0444 on a read-only
|
||||
store and clear on GC).
|
||||
|
||||
Otherwise the next `!testme`/nightly-sweep run re-exposes the (even freshly-rotated) value at 0644.
|
||||
|
||||
**Scope note — NOT a gate, NOT a VETO, DONE stands.** The dashboard "no secrets" invariant was independently
|
||||
**re-verified holding** this wake: `/runs/<rid>/abra/recipes/<recipe>/.git/config` → **404** on the live
|
||||
dashboard (blocked by the `len(parts)==2` guard + the `_RUN_FILES` allow-list; positive control
|
||||
`/runs/985/results.json` → 200, `has_cred=False`). This exposure is **host-local filesystem only** (any local
|
||||
uid, incl. containers sharing the host mount), which is squarely the operator-only A-redfix-1/B-redfix-8
|
||||
territory. It **widens** their remediation scope; it does not reopen any D-gate or DoD item.
|
||||
|
||||
### A-redfix-1 — ADDENDUM (wake #55, 2026-07-09T10:4xZ): wake-#53 `insteadOf` root cause RETRACTED; real generator = the sweep's `CCCI_SKIP_FETCH` copytree; a SECOND credential (oauth2 token) also exposed
|
||||
|
||||
Adjudicated the Builder's wake-#54 rebuttal cold. Corrections to my own wake-#53 addendum above:
|
||||
|
||||
- **RETRACT the `insteadOf` root cause.** Reproduced first-hand on the node (git 2.47.2): a clone
|
||||
whose transport `url.<t>.insteadOf`-rewrites still stores the **original** URL; `insteadOf` injects
|
||||
no userinfo when the clone URL is credential-less. My "regenerates via insteadOf" was wrong.
|
||||
- **Real generator:** canonical `/root/.abra/recipes/*/.git/config` carry the cred (safe at rest under
|
||||
`/root` = `0700`); `run_recipe_ci.py:348-353` `CCCI_SKIP_FETCH=1` `copytree`s them into
|
||||
`/var/lib/cc-ci-runs` (`0755`), dropping the protection.
|
||||
- **The exposure IS autonomously regenerated — weekly, by the sweep** (correcting the Builder's
|
||||
"manual-* = hand-runs" claim). `run_id()`→`"manual"` for any non-Drone run; `nightly_sweep.py:88`
|
||||
runs `run_recipe_ci.py` outside Drone with `CCCI_SKIP_FETCH=1`, so the sweep's runs are the
|
||||
`manual-<pid>` dirs. Freshest copies dated 2026-07-05 03:37–03:59Z = the sweep's last fire.
|
||||
- **SECOND credential found (Builder census missed it).** `/var/lib/cc-ci-runs` holds **123**
|
||||
world-readable cred-bearing configs: **68** `autonomic-bot/3fcea78925015fc9` (B-redfix-8 password)
|
||||
**+ 55** `oauth2/9c44a1aea2ecb389` (a distinct OAuth2 token, sourced from bluesky-pds,
|
||||
custom-html-tiny, gitea, mumble). Remedy must scrub/rotate **both**.
|
||||
|
||||
Still **OPEN**, still **operator-scope**, **no gate / no DoD impact, no VETO**. Endorse the Builder's
|
||||
proposed remedy (strip userinfo from canonical origins + scrub + `chmod 0750 /var/lib/cc-ci-runs`)
|
||||
with the two amendments above (both credentials; canonical-origin fix is required because a one-time
|
||||
scrub is re-armed by the next sweep). Full reasoning + reproductions in REVIEW-redfix.md wake #55.
|
||||
|
||||
### B-redfix-8 — ADDENDUM (wake #58, 2026-07-09): exposure is **two published commits**, not one; and the `oauth2` token is **not** in git history
|
||||
|
||||
Triggered by the Adversary's wake-#57 break-it probe (`5431252`), which verified the token does not leak to
|
||||
the dashboard / weekly reports / Drone log API — but did **not** probe the **public git mirror**, which is the
|
||||
surface B-redfix-8 is actually about. Closing that gap surfaced a defect in my own STATUS.
|
||||
|
||||
**Method.** Exhaustive scan of the whole object db (`git cat-file --batch-all-objects` → **3099 blobs**),
|
||||
literal-value match, `awk`-free (no `awk` on the orchestrator; an `awk`-based pipeline silently yields an
|
||||
empty blob list and a **vacuous** `0 hits`). Token pulled root-only from a world-readable run-dir
|
||||
`.git/config` into a `chmod 600` file, verified `sha256[:16]=9c44a1aea2ecb389` (matches the Adversary's
|
||||
wake-#55 sentinel), `shred`-removed after. Positive control: the known-leaked password **must** be found.
|
||||
|
||||
**Results.**
|
||||
- **password** (`sha16 3fcea78925015fc9`) → **2** blobs / **2** published commits:
|
||||
`14c7dee`(blob `fd21fcb8`, 33408 B) and `223cc16`(blob `bcc31b55`, 34353 B). Both are ancestors of
|
||||
`origin/main` ⇒ both mirror-served. Neither reachable from HEAD. `e99e2b3` and HEAD are clean.
|
||||
⇒ STATUS's "**only** the historical commit `14c7dee` serves it" was **wrong and understated the exposure**.
|
||||
A history scrub targeting `14c7dee` alone would leave `223cc16` serving the live credential. STATUS corrected.
|
||||
- **oauth2 token** (`sha16 9c44a1aea2ecb389`) → **0** of 3099 blobs. The 271 blobs matching the *string*
|
||||
`oauth2:` are `recipe-mirror-sync.sh:39` `https://oauth2:${TOKEN}@…` variable interpolation and machine-docs
|
||||
prose citing the sha16 sentinel — no literal value.
|
||||
|
||||
**Consequence for the operator remedy.** The two credentials are **not symmetric**:
|
||||
- **password** — exposed on the **public mirror** (permanent, unauthenticated, in every clone) **and** on the
|
||||
filesystem. Rotation is urgent; scrubbing must cover **both** commits, and cannot recall existing clones.
|
||||
- **oauth2 token** — exposed on the **filesystem only** (world-readable `.git/config` copies, A-redfix-1),
|
||||
regenerated weekly by the sweep. Rotation + `chmod`/userinfo-strip suffices; no history rewrite needed.
|
||||
|
||||
**Guard for future probes.** Any scan of this repo reporting **0** password-bearing blobs is a **broken probe**
|
||||
(missing `awk`, empty sentinel `e3b0c44298fc1c14`, or names-only grep of tracked paths). The correct answer
|
||||
is **2**. Always run the positive control.
|
||||
|
||||
Still **operator-scope, outside DoD, no gate impact.** `## DONE` stands; no VETO.
|
||||
|
||||
### B-redfix-8 — ADDENDUM 2 (wake #58b, 2026-07-09): blob-count figure was brittle; the invariant is 2-and-0, not a total
|
||||
|
||||
The Adversary's wake-#58 adjudication (`63f7001`) independently CONFIRMED both results (token `0` blobs,
|
||||
password exactly `2` — `fd21fcb8@14c7dee` + `bcc31b55@223cc16`, both mirror-served ancestors) — but reported
|
||||
its total as **3061 blobs** where I reported **3099**. Neither is wrong; the *total* is simply not an invariant.
|
||||
|
||||
Root-caused, three independent sources of drift:
|
||||
1. **New commits add blobs.** Same host now reads **3104** (my wake-#58 commits landed after the scan).
|
||||
2. **`--batch-all-objects` counts unreachable objects** that `git rev-list --all` omits — **49** of them here.
|
||||
Reachable-only is **3026**.
|
||||
3. **`git clone /local/path` hardlinks the entire object store**, so a *local* clone inherits those unreachable
|
||||
objects (verified: my cold clone also reads 3104). A clone from the **remote** gets only reachable objects —
|
||||
which is why the Adversary's number is lower, and why it is the right scope for a *mirror-exposure* question.
|
||||
|
||||
**Consequence.** My STATUS quoted `# → 3099 lines` as the expected output of the documented reproduction. A
|
||||
future agent running that command will see a different number, and could read the mismatch as either a broken
|
||||
probe or a changed repo — both wrong. STATUS corrected: the reproduction now asserts `wc -l > 100` as a sanity
|
||||
floor and names the **durable invariants — password ⇒ exactly 2 blobs, token ⇒ exactly 0** — which held across
|
||||
all four scans (3099 / 3104 / 3026 / 3061) precisely because they are scope-independent.
|
||||
|
||||
Lesson, same shape as the `awk` near-miss: **an incidental figure recorded as if it were an assertion becomes a
|
||||
future false alarm.** Record the invariant, not the measurement.
|
||||
|
||||
No new exposure, no gate impact, operator-scope. `## DONE` stands; no VETO.
|
||||
|
||||
@ -1609,52 +1609,3 @@ trim job observed → adequate; revisit only if a cap is ever needed.
|
||||
`dashboard/dashboard.py` + `tests/unit/test_dashboard.py` would be reformatted — confirmed pre-existing
|
||||
at HEAD f68f1c5, outside the settings diff. Flagged for the dashboard owner / orchestrator; not fixed
|
||||
here (narrow scope).
|
||||
|
||||
## Warm state is keyed by STACK NAMESPACE, not by recipe (phase redfix, F-redfix-4, 2026-07-09)
|
||||
|
||||
**Settled.** `/var/lib/ci-warm/<ns>/` is a slot owned by exactly ONE deployed stack. `ns` comes from
|
||||
`canonical.canonical_ns(recipe)` for the data-warm canonical and `warmsnap.live_slot(recipe)` for the
|
||||
live-warm reconciler. A recipe that is both (only `keycloak` today, via `warm.WARM_DOMAINS`) gets
|
||||
`canon-<recipe>` for its canonical, so the two never share a slot. The canonical's **domain and its slot
|
||||
derive from the same `canonical_ns()`**, so they cannot drift apart — that coupling is the invariant, not
|
||||
the `canon-` string. Every slot `<ns>` maps 1:1 to the stack at `warm.stable_domain(<ns>)`.
|
||||
|
||||
Rejected alternatives:
|
||||
- **Skip `seed_canonical` for `WARM_DOMAINS` recipes.** Silently de-enrolls keycloak and re-opens the DoD
|
||||
item ("keycloak enrolled … verified green"). The Adversary named this as not acceptable; agreed.
|
||||
- **Key every slot by the full domain / stack name** (the "purest" model). Correct, but it relocates the
|
||||
reconciler's `last_good` for keycloak+traefik and `canonical.json` + `snapshot/` for 15 existing
|
||||
canonicals on a live node, requiring a migration shim for a phase whose mandate is to fix red, not
|
||||
re-architect. The chosen scheme is the same model applied only where two stacks actually contend, and
|
||||
needs no migration (keycloak's canonical had never been seeded).
|
||||
- **Guard-only (`_assert_slot_not_foreign` alone).** Turns silent destruction into a loud failure but leaves
|
||||
keycloak's canonical unable to ever seed. Kept as defence in depth BEHIND the slot separation, not as the
|
||||
fix.
|
||||
|
||||
Consequence for future enrollments: adding a recipe to `warm.WARM_DOMAINS` automatically namespaces its
|
||||
canonical on both axes. `prune_stale()`'s "reconciler dirs are never pruned" invariant is now structural
|
||||
(a `<recipe>/` dir never gains a `canonical.json`) rather than an incidental property that de-enrollment
|
||||
could falsify.
|
||||
|
||||
## 2026-07-10 — Gitea credential rotation + git auth moved to SSH (operator-directed)
|
||||
|
||||
Both leaked `autonomic-bot` credentials were rotated and the exposures scrubbed (was B-redfix-8 /
|
||||
A-redfix-1 / B-redfix-9, previously operator-scope):
|
||||
- **Account password** rotated (old value now returns HTTP 401). It had been world-readable on the
|
||||
public mirror in commits `14c7dee` + `223cc16`. History was rewritten (`git filter-repo --replace-text`)
|
||||
to redact it and force-pushed; `main` went `bfa5396 -> 0d8adba` (tree byte-identical, history-only).
|
||||
All clones + node `/etc/cc-ci` were reset to the rewritten `main`. Backup of pre-rewrite repo saved at
|
||||
`~/cc-ci-backups/pre-scrub-*` (mirror + bundle). **Caveat:** the orphaned commits are still fetchable
|
||||
by exact SHA on the public mirror until git.autonomic.zone runs a site-admin GC (value is dead, so inert).
|
||||
- **`bridge_gitea_token`** (the oauth2 token, sops `secrets.yaml`) rotated: fresh PAT minted, re-encrypted,
|
||||
deployed via `nixos-rebuild ?submodules=1`, old token + all 8 other stale PATs revoked. Only one PAT
|
||||
remains: `cc-ci-bridge-*` (id 36).
|
||||
|
||||
**Git auth is now SSH, not HTTP.** git.autonomic.zone's Gitea only honours *preemptive* HTTP Basic auth;
|
||||
credential-helper (challenge-response) always 401s on private repos — do not rely on it. Remotes are SSH
|
||||
(`git@git.autonomic.zone:...`, port 2222 via `~/.ssh/config`); dedicated keys `cc-ci-orchestrator-20260710`
|
||||
(orchestrator, `~/.ssh/autonomic-bot-gitea-ed25519`) and `cc-ci-node-20260710` (node, `/root/.ssh/...`).
|
||||
`.gitmodules` secrets URL is SSH. `recipe-mirror-sync.sh` pushes over SSH (no token written into recipe
|
||||
`.git/config` -> fixes the copytree leak). The bot **password stays in `/srv/cc-ci/.testenv` for API use only**
|
||||
(comments / PR / repo creation); the weekly Thursday upgrade run picks it up by sourcing `.testenv`.
|
||||
Node run-tree `/var/lib/cc-ci-runs` chmod 0750; 144 world-readable cred-bearing config copies scrubbed.
|
||||
|
||||
@ -427,29 +427,3 @@ reachable via the operator/dev STAGES escape — production drone runs always ru
|
||||
key on stdout (write to a run-scoped sidecar the test reads), or register the minted key in the harness
|
||||
redaction set so even the RAW log is scrubbed. Low priority (RAW log is access-controlled; key is ephemeral).
|
||||
- **Filed by:** Builder, phase prevb (acknowledging Adversary [F-prevb-C]).
|
||||
|
||||
## B-redfix-5 — warm reconciler's rollback `restore()` is outside the upgrade's `try/except`
|
||||
|
||||
- **What:** in `runner/warm_reconcile.py`, **two** post-`abra.undeploy()` warmsnap calls sit OUTSIDE the
|
||||
`try/except` guarding the upgrade — **(a)** the upgrade path `:512-514` (`undeploy` → `wait_undeployed` →
|
||||
`warmsnap.snapshot(...)`) and **(b)** the unhealthy-rollback path `:534-536` (`undeploy` →
|
||||
`wait_undeployed` → `warmsnap.restore(...)` → `deploy_version(last_good)` at `:537`). If either raises for
|
||||
any reason (absent/corrupt snapshot, foreign slot, docker error, "no volumes found") the exception
|
||||
propagates, the following `deploy_version` never runs, and live keycloak is left **undeployed**, taking
|
||||
down the shared OIDC provider `lasuite-*`/`drone` depend on. Site (a) fires on the *normal* upgrade path —
|
||||
no rollback required.
|
||||
- **Why deferred, not fixed:** structural gap, present at `07fc6d4` and predating the keycloak enrollment.
|
||||
Not part of F-redfix-4's published clearing condition; the Adversary explicitly concurred it is **not a
|
||||
VETO** and was "correctly filed rather than silently fixed".
|
||||
- **Reachability, corrected (wake #42).** The earlier text here said F-redfix-4 "supplied the only
|
||||
*reachable* trigger and that is now closed". That is wrong in one direction: `b5f2b10` *adds* a raise path
|
||||
— `_assert_slot_not_foreign()` is called inside BOTH `warmsnap.snapshot()` (`:158`) and `warmsnap.restore()`
|
||||
(`:209`). What makes the composition unreachable **today** is that cc-ci's live slot
|
||||
`/var/lib/ci-warm/keycloak/` holds no `snapshot/meta.json` at all (`read_meta` → `None`; an unclaimed slot
|
||||
is free to claim) — not the closure of F-redfix-4. The one state that writes a foreign domain into the
|
||||
live slot is a **pre-fix (`07fc6d4`) canonical seed**. Recorded as a MERGE PRECONDITION in STATUS-redfix.md.
|
||||
- **Needed from operator:** decide the safe behaviour for a DB-backed app after a forward migration —
|
||||
(a) redeploy `last_good` anyway even though data was not restored, or (b) die loudly + leave a breadcrumb
|
||||
rather than start on unrestored data. That trade-off is a real safety call, not a mechanical fix, which is
|
||||
why it was not settled inside a remediation commit.
|
||||
- **Filed by:** Builder, phase redfix (from Adversary F-redfix-4 consequence 3).
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@ -7,499 +7,6 @@ gitea, keycloak) → isolate → root-cause → classify (flake vs genuine; reci
|
||||
warm-machinery vs load) → FIX each (recipe PR or harness improvement) → verify green. No standing
|
||||
exceptions. Nothing merged.
|
||||
|
||||
## DONE — 2026-07-09T00:18Z
|
||||
|
||||
Phase `redfix` COMPLETE. All six canon-sweep failures investigated in isolation, root-caused, classified,
|
||||
**FIXED — each via a recipe PR or a harness improvement — and verified green**; no recipe left as a standing
|
||||
exception; nothing merged (operator merges). Every gate has a fresh Adversary PASS in REVIEW-redfix.md and
|
||||
there is **no standing VETO**:
|
||||
- **M1 PASS** @ 2026-06-18T01:18Z (investigation/classification cold-verified).
|
||||
- **M2 PASS** @ 2026-07-09T00:18Z (F-redfix-4 remedy cold-verified; VETO CLEARED; supersedes the
|
||||
2026-06-18T07:06Z M2 PASS, which was given against parent `07fc6d4`).
|
||||
|
||||
Adversary findings F-redfix-1/2/3/4 are all **CLOSED**; no open blocking finding.
|
||||
|
||||
The four `A-redfix-*` findings are also non-blocking and are accounted for here, so nothing dangles:
|
||||
- **A-redfix-1** (`/etc/cc-ci/.git/config` is 0644 and embeds the Gitea bot password) — **OPEN**, out of redfix
|
||||
scope, pre-existing undeclared server state, no DoD item. Mirrors my B-redfix-7. Not created by this phase.
|
||||
**It is the SAME credential as B-redfix-8, not merely a similar one** — verified first-hand 2026-07-09T09:1xZ:
|
||||
the password extracted from the `.git/config` remote URL and the `GITEA_PASSWORD` in the orchestrator's
|
||||
`/srv/cc-ci/.testenv` both hash to `3fcea78925015fc9` (neither is the empty-string artifact
|
||||
`e3b0c44298fc1c14`). **Two exposures, one credential, still unrotated.** Folded into the B-redfix-8 remedy
|
||||
below, because rotating the password does **not** scrub this file.
|
||||
- **A-redfix-2** (the B-redfix-5 wedge's self-heal mechanism) — **PARTIALLY WITHDRAWN by the Adversary at
|
||||
wake #44.** Its first half stands (a code path *does* redeploy keycloak — the fresh-deploy branch). Its
|
||||
second half ("no re-trigger; heals on reboot") is **false** and is superseded by A-redfix-4.
|
||||
- **A-redfix-3** (the `main.go` "both clones" evidence) — **ACCEPTED**; that evidence is retracted below.
|
||||
`/srv/cc-ci` is a symlink to `/srv/cc-ci-orch`. The file's origin remains unexplained; risk inert.
|
||||
- **A-redfix-4** (a weekly `nightly-sweep.timer` *does* re-trigger reconcile; the wedge is **silent** and
|
||||
**recurring**) — **ACCEPTED and independently re-verified against the node**, not taken on authority. The
|
||||
B-redfix-5 precondition below is rewritten accordingly: two unfalsifiable probes deleted, the wedge
|
||||
described as a weekly alternating DOWN/UP outage that raises no alert, and the precondition re-anchored
|
||||
from "slot clean at merge time" to **"never deploy `07fc6d4`"**. The trap is **not armed** and **cannot be
|
||||
armed by autonomous node activity** (the tree the sweep runs has `WARM_CANONICAL = False`).
|
||||
|
||||
**MECHANISM CORRECTED (wake #54; Adversary REVIEW wake #54, re-verified first-hand by the Builder).** The
|
||||
sweep does **not** run `origin/main`. `nightly-sweep.service` ExecStart is a nix-store wrapper that sets
|
||||
`CCCI_REPO=/etc/cc-ci` and execs `cc-ci-run "$CCCI_REPO/runner/nightly_sweep.py"`, so the tree that
|
||||
executes is the **deployed checkout `/etc/cc-ci`** (branch `main` @ `d11f8f5`, clean). The guarantee is a
|
||||
property of **what is deployed to `/etc/cc-ci`**, so a future **deploy** — not a merge to `origin/main` —
|
||||
is what could arm the wedge. Conclusion unchanged (trap disarmed); only the named tree was wrong. Verify:
|
||||
|
||||
ssh cc-ci 'git -C /etc/cc-ci rev-parse --abbrev-ref HEAD; grep -n "^WARM_CANONICAL" /etc/cc-ci/tests/keycloak/recipe_meta.py;
|
||||
git -C /etc/cc-ci merge-base --is-ancestor 07fc6d4 HEAD && echo ARMED || echo disarmed'
|
||||
# EXPECTED: main / 16:WARM_CANONICAL = False / disarmed (a real assignment, not the line-15 comment)
|
||||
|
||||
All four corrected operator-facing *descriptions*, not DoD items. No VETO at wake #43 (`b358b7e`) or wake #44
|
||||
(`84a1666`, 2026-07-09T08:47Z). **Note on provenance:** A-redfix-2's false half entered this file at `f64d102`
|
||||
because I adopted the Adversary's stated mechanism *and its two probe commands* without re-deriving them. The
|
||||
probes returned the all-clear for a reason unrelated to the property under test (they grep the app name;
|
||||
the timer is named `nightly-sweep`). Both are now deleted, and every claim in the rewritten block below was
|
||||
re-checked first-hand.
|
||||
|
||||
## ⚠ OPERATOR ACTION REQUIRED — B-redfix-8 (HIGH, open, outside DoD)
|
||||
|
||||
Phase DoD is met, but **one HIGH item is open and only an operator can close it.** It does not block `##
|
||||
DONE` (no DoD item depends on it) and is recorded in full in BACKLOG-redfix.md.
|
||||
|
||||
**WHAT.** The live Gitea bot password (`GITEA_PASSWORD` in `/srv/cc-ci/.testenv`) was committed to
|
||||
`machine-docs/BACKLOG-redfix.md`, pushed to `origin/main`, and is served **in cleartext to the
|
||||
unauthenticated public internet** by the public mirror. It is push-capable to `recipe-maintainers/*`. HEAD
|
||||
and the redaction commit `e99e2b3` are clean. Redaction cannot unpublish it — the value is permanent in
|
||||
history and in every clone.
|
||||
|
||||
**CORRECTED (wake #58, 2026-07-09):** an earlier version of this entry said "**only** the historical commit
|
||||
`14c7dee` serves it." That **understated the exposure**. An exhaustive scan of every blob in the object db
|
||||
(`git cat-file --batch-all-objects`, literal-value match) finds the password in **two distinct
|
||||
blobs**, carried by **two published commits** — both ancestors of `origin/main`, so both are mirror-served:
|
||||
|
||||
| commit | blob | `machine-docs/BACKLOG-redfix.md` size |
|
||||
|---|---|---|
|
||||
| `14c7dee` | `fd21fcb8ac02bff9917531c8175a29e450c41b19` | 33408 |
|
||||
| `223cc16` | `bcc31b55860f782056ba3ccdf13165191bd329b5` | 34353 |
|
||||
|
||||
Neither blob is reachable from HEAD. Scrubbing only `14c7dee` would leave `223cc16` serving the credential.
|
||||
|
||||
**WHERE.** `git.autonomic.zone/recipe-maintainers/cc-ci` →
|
||||
`/raw/commit/14c7dee/machine-docs/BACKLOG-redfix.md` **and**
|
||||
`/raw/commit/223cc16/machine-docs/BACKLOG-redfix.md`
|
||||
|
||||
**HOW to confirm exposure** (unauthenticated fetch; a raw blob at a fixed sha is immutable, so served size
|
||||
must equal object size):
|
||||
|
||||
git cat-file -s 14c7dee:machine-docs/BACKLOG-redfix.md # → 33408
|
||||
git cat-file -s 223cc16:machine-docs/BACKLOG-redfix.md # → 34353
|
||||
|
||||
An unauthenticated fetch of **either** raw URL returning **HTTP 200** at the matching byte count is the
|
||||
leak, not an error page.
|
||||
|
||||
**Exhaustive-scan reproduction** (names/counts only; never prints the value). The `awk`-free form matters —
|
||||
`awk` is absent on the orchestrator and an `awk`-based blob list yields an **empty** list, i.e. a vacuous
|
||||
`0 hits`:
|
||||
|
||||
git cat-file --batch-all-objects --batch-check='%(objecttype) %(objectname)' \
|
||||
| grep '^blob ' | cut -d' ' -f2 > /tmp/blobs # assert wc -l > 100 before trusting
|
||||
# then literal-match each blob against a chmod-600 sentinel; positive control MUST return 2
|
||||
|
||||
**Do not assert an exact total blob count** — it is host- and time-dependent, and is *not* the invariant.
|
||||
Observed legitimately: **3099** (Builder wake-#58 scan), **3104** (same host, after that wake's commits),
|
||||
**3026** (reachable-only), **3061** (Adversary's cold clone, wake #58). Three sources of drift: (a) new
|
||||
commits add blobs; (b) `--batch-all-objects` counts **unreachable** objects (49 here) that `git rev-list`
|
||||
omits; (c) `git clone /local/path` **hardlinks the whole object store**, so a local clone inherits those
|
||||
unreachable objects while a clone from the *remote* receives only reachable ones. All four scans agree on
|
||||
the only figures that matter.
|
||||
|
||||
**Positive control is mandatory.** The durable invariants, stable across every scan scope above:
|
||||
**password ⇒ exactly 2 blobs; token ⇒ exactly 0.** Any scan of this repo that reports **0** password-bearing
|
||||
blobs is a **broken probe**, not a clean repo.
|
||||
|
||||
**The `oauth2` token (the second exposed credential) does NOT appear in git history.** Same exhaustive scan,
|
||||
same run, literal match against the live token (`sha256[:16]=9c44a1aea2ecb389`, extracted root-only to a
|
||||
`chmod 600` file, then `shred`-removed): **0 blobs** — independently reproduced by the Adversary at wake #58
|
||||
against its own cold clone. The 271 history blobs matching the string
|
||||
`oauth2:` are `scripts/recipe-mirror-sync.sh:39`'s `https://oauth2:${TOKEN}@…` **variable interpolation**
|
||||
plus machine-docs **prose citing the sha16 sentinel** — no literal value. So the token's exposure is
|
||||
**filesystem-only** (A-redfix-1: world-readable `.git/config` copies); it is **not** on the public mirror.
|
||||
The password's exposure is **both** filesystem **and** public mirror. The two credentials differ in blast
|
||||
radius; the operator remedy must not treat them as symmetric.
|
||||
|
||||
**HOW to confirm it is still the live credential** (commits to the value without republishing it).
|
||||
Run this **on the orchestrator** — `/srv/cc-ci/.testenv` lives there, **not** on `cc-ci`:
|
||||
|
||||
python3 -c 'import hashlib,re;v=re.search(r"^GITEA_PASSWORD=(.*)$",open("/srv/cc-ci/.testenv").read(),re.M).group(1).strip().strip(chr(34)).strip(chr(39));print(hashlib.sha256(v.encode()).hexdigest()[:16])'
|
||||
|
||||
**EXPECTED.** Prints `3fcea78925015fc9` while still unrotated. **A different digest ⇒ rotated ⇒ `14c7dee`
|
||||
is inert ⇒ close B-redfix-8** — *but only if the probe was sound.* Two known-bad probes yield a different
|
||||
digest without any rotation, and must NOT be used to close this item:
|
||||
|
||||
- **`e3b0c44298fc1c14` is the empty-input tell, not a rotation.** Running the command over `ssh cc-ci`
|
||||
fails to open `.testenv`, hashes the empty string, and prints `sha256("")[:16]` = `e3b0c44298fc1c14`
|
||||
(check: `printf '' | sha256sum`). Fix the probe; do not close B-redfix-8.
|
||||
- **Do not conflate the three digests.** `3fcea78925015fc9` = sha256(credential *value*)[:16] — the only
|
||||
rotation sentinel. `1994fd8d…` = sha256(the whole served *blob*) — immutable, changes never.
|
||||
`e3b0c44298fc1c14` = sha256(empty) — a broken probe.
|
||||
|
||||
Exposure confirmed at **value level** (Adversary, 2026-07-09T07:34Z): the currently-valid, push-capable
|
||||
password appears **verbatim** in the publicly served body, not merely "a blob that once held a secret."
|
||||
|
||||
**REMEDY (operator).** Rotate the Gitea bot password, then update **both** places that hold it. The credential
|
||||
is a Class-A1 external infra input (plan §4.4) — the Builder must not rotate or invent it. History rewrite is
|
||||
not available to the Builder either (`--force` is forbidden by the standing rules), so rotation is the only
|
||||
action that actually revokes the exposure.
|
||||
|
||||
1. Rotate the bot password in Gitea.
|
||||
2. Update `GITEA_PASSWORD` in `/srv/cc-ci/.testenv` **on the orchestrator**.
|
||||
3. **Rewrite `/etc/cc-ci/.git/config` on `cc-ci` (A-redfix-1) — rotation does NOT scrub it.** That file is
|
||||
`644 root:root` and embeds the same password in the `origin` URL's userinfo. `/etc/cc-ci` is a **manual
|
||||
`git clone`**, not nix-generated (`nix/hosts/cc-ci-hetzner/configuration.nix:7` documents the clone), so no
|
||||
`nixos-rebuild` regenerates or scrubs it. Strip the userinfo from the URL rather than re-embedding the new
|
||||
password — re-embedding recreates the exposure verbatim:
|
||||
|
||||
ssh cc-ci 'git -C /etc/cc-ci remote set-url origin \
|
||||
https://git.autonomic.zone/recipe-maintainers/cc-ci.git && chmod 600 /etc/cc-ci/.git/config'
|
||||
|
||||
**CORRECTION (wake #54→#55, Builder, 2026-07-09): `/etc/cc-ci/.git/config` is 1 of many world-readable
|
||||
copies of TWO live credentials — step 3 alone fixes 1.** The "sole copy" claim above is **withdrawn**.
|
||||
Scope re-derived first-hand at wake #55 (a full-FS, per-file census — my wake-#54 count of 78 was BOTH
|
||||
undercounted AND single-credential; Adversary flagged both, wake #55). Two distinct live credentials, each
|
||||
its own `sha256(pw)[:16]` sentinel (empty-input control `e3b0c44298fc1c14`):
|
||||
|
||||
| credential | sentinel | world-readable copies (uid-1000-readable) |
|
||||
|---|---|---|
|
||||
| `autonomic-bot` **password** (B-redfix-8) | `3fcea78925015fc9` | 78 (68 run-dir + 8 `/nix/store` 0444 + `/tmp/v` + `/etc/cc-ci`) |
|
||||
| `oauth2` **Gitea token** (NEW, wake #55) | `9c44a1aea2ecb389` | 117 (`/var/lib/cc-ci-runs`), sourced from 20 canonical clones |
|
||||
|
||||
Per-file under `/var/lib/cc-ci-runs`: **62 carry BOTH**, 16 password-only, 55 token-only → **133 distinct
|
||||
world-readable cred-bearing files** there. **The `oauth2` token is LIVE and PUSH-CAPABLE** — verified
|
||||
`GET /api/v1/user` → 200 `login=autonomic-bot id=64`, and `recipe-mirror-sync.sh:39` pushes with it
|
||||
(`https://oauth2:<TOKEN>@…`). So **both** secrets must be rotated, not just `GITEA_PASSWORD`. Re-census:
|
||||
|
||||
ssh cc-ci 'find / -xdev -name config -path "*/.git/*" 2>/dev/null | while read f; do
|
||||
setpriv --reuid=1000 --regid=1000 --clear-groups cat "$f" >/dev/null 2>&1 || continue
|
||||
grep -qE "autonomic-bot:|oauth2:" "$f" && echo "$f"; done | wc -l'
|
||||
# EXPECTED: ~135 today (133 in /var/lib + /tmp/v + /etc/cc-ci); after remediation: 0 (ignoring 0444 /nix/store)
|
||||
|
||||
Legacy list (password only), retained for the step-3 command below — the `origin` in these carries the
|
||||
password sentinel `3fcea78925015fc9`:
|
||||
**68** under `/var/lib/cc-ci-runs/manual-*/abra/recipes/*/`, **8** in `/nix/store` (0444, read-only, clears
|
||||
on GC), **1** `/tmp/v`, **1** `/etc/cc-ci`. Verify:
|
||||
|
||||
ssh cc-ci 'find / -xdev -name config -path "*/.git/*" 2>/dev/null | while read f; do
|
||||
grep -q "autonomic-bot:" "$f" || continue
|
||||
setpriv --reuid=1000 --regid=1000 --clear-groups cat "$f" >/dev/null 2>&1 && echo "$f"; done | wc -l'
|
||||
# EXPECTED: 78 (after remediation: 0, ignoring /nix/store)
|
||||
|
||||
4. **Scrub the copies AND stop regeneration (A-redfix-1, widened).** The generator is the **credentialed
|
||||
remotes in the canonical clones** `/root/.abra/recipes/*/.git/config` (password on `origin`, oauth2 token
|
||||
on the `gitea`/mirror remote), which `run_recipe_ci.py:348-353`'s `CCCI_SKIP_FETCH=1` `shutil.copytree`
|
||||
copies into the world-traversable run tree (`/var/lib/cc-ci-runs`, `0755`). Canonicals are shielded by
|
||||
`/root` = `0700`; the run-dir copies are not.
|
||||
**REGENERATES WEEKLY (corrected wake #55).** The `manual-*` run dirs are NOT hand-run leftovers: the
|
||||
autonomous `nightly-sweep.timer` runs `run_recipe_ci.py` outside Drone (`nightly_sweep.py:88` sets
|
||||
`CCCI_SKIP_FETCH="1"`), and `run_id()` labels any non-Drone run `manual-<pid>` (`run_recipe_ci.py:318-319`).
|
||||
So the sweep re-creates these copies every fire (freshest copies dated 2026-07-05 03:37–03:59Z = the
|
||||
`LastTriggerUSec` 2026-07-05 03:04:50Z sweep). **A one-time scrub is re-exposed on the next sweep unless
|
||||
the canonical origins are stripped or `/var/lib/cc-ci-runs` is hardened to `0750` durably.** Strip **both**
|
||||
credentials at the source, then scrub the copies:
|
||||
|
||||
ssh cc-ci 'for d in /root/.abra/recipes/*/; do r=$(basename "$d");
|
||||
for rem in origin gitea upstream; do
|
||||
git -C "$d" remote get-url "$rem" >/dev/null 2>&1 || continue
|
||||
git -C "$d" remote set-url "$rem" "$(git -C "$d" remote get-url "$rem" | sed -E "s#://[^@/]+@#://#")"; done; done
|
||||
rm -rf /tmp/v
|
||||
find /var/lib/cc-ci-runs -path "*/.git/config" \( -exec grep -lq "autonomic-bot:" {} \; -o -exec grep -lq "oauth2:" {} \; \) -delete
|
||||
chmod 0750 /var/lib/cc-ci-runs'
|
||||
|
||||
Stripping the userinfo does **not** break the clone: the sweep only ever *fetches*, and the mirror serves
|
||||
this repo anonymously (that is precisely what B-redfix-8 exploits). Verified 2026-07-09T09:1xZ —
|
||||
`GIT_TERMINAL_PROMPT=0 git -c credential.helper= ls-remote https://git.autonomic.zone/recipe-maintainers/cc-ci.git HEAD`
|
||||
→ `f0372b8… HEAD`, rc=0, no credential. The sibling `cc-ci-secrets` submodule URL (`.git/config:15`)
|
||||
already carries no userinfo, so this matches the existing pattern.
|
||||
|
||||
**Independently cold-verified by the Adversary (REVIEW-redfix.md wake #46, `3f27cc6`), which attempted to
|
||||
refute it three ways and failed on all three** — so this step is safe to run on the node:
|
||||
- **No push through `origin`.** The only `git push` in the deployed tree (`scripts/recipe-mirror-sync.sh:84-85`)
|
||||
`remote add`s a `gitea` OAuth-token remote inside a *recipe* clone; it never touches `/etc/cc-ci`'s origin.
|
||||
- **The `secrets` submodule survives the strip.** Its URL already carries no userinfo and
|
||||
`cc-ci-secrets.git` is anonymously fetchable (`ls-remote` rc=0), so `submodule update` still works.
|
||||
- **Nothing auto-pulls the checkout.** `/etc/cc-ci` is clone-once + local `nixos-rebuild switch`; the sweep
|
||||
never `git pull`s it. (`/root/.git-credentials` exists but no `credential.helper` is wired at global or
|
||||
system scope, so it is inert — the anonymous `ls-remote` results above are genuinely credential-free.)
|
||||
|
||||
**Rotation blast radius is small — no operational reason to delay.** `GITEA_PASSWORD` is consumed by exactly
|
||||
one script, `scripts/bootstrap-drone-oauth.sh` (`git grep -ln GITEA_PASSWORD -- .` returns that script, `docs/
|
||||
install.md`, and machine-docs prose; nothing else in the runner). In particular `scripts/recipe-mirror-sync.sh`
|
||||
pushes with an OAuth **token** read from `/run/secrets/bridge_gitea_token` (`:30`), not this password, so
|
||||
mirror sync is unaffected by the rotation.
|
||||
|
||||
**HOW to confirm BOTH exposures are closed** (each prints a digest; a value other than `3fcea78925015fc9` on
|
||||
both, and not the empty-string tell `e3b0c44298fc1c14`, means rotated):
|
||||
|
||||
# exposure 2 — the embedded copy; `python3` is absent on cc-ci, so use sha256sum
|
||||
ssh cc-ci 'sed -n "s#^\s*url = https://[^:]*:\([^@]*\)@git\.autonomic\.zone/recipe-maintainers/cc-ci\.git#\1#p" \
|
||||
/etc/cc-ci/.git/config | head -1 | tr -d "\n" | sha256sum | cut -c1-16'
|
||||
|
||||
**EXPECTED after step 3:** the command prints **nothing** (no userinfo left to match) — an empty result here
|
||||
is the fix, whereas an `e3b0c44298fc1c14` from the *step-2* probe is a broken probe. Do not confuse them.
|
||||
|
||||
**Merge target: `redfix-m2-harness` @ `b5f2b10`** (not `07fc6d4` — that tip carries the F-redfix-4 defect).
|
||||
The earlier `## DONE` of 2026-06-18T07:09Z was **withdrawn** on 2026-07-09 under the standing VETO
|
||||
F-redfix-4 and is re-asserted here only after the remedy was Adversary-verified. Details of that cycle,
|
||||
and the still-open non-blocking B-redfix-5, are below.
|
||||
|
||||
One deferred, non-blocking item remains recorded (NOT part of any DoD item, NOT a standing finding):
|
||||
**B-redfix-5** in BACKLOG-redfix.md — in `warm_reconcile.py`, TWO post-`abra.undeploy()` calls sit outside
|
||||
the upgrade's `try/except`: `warmsnap.snapshot()` on the upgrade path (`:514`) and `warmsnap.restore()` on
|
||||
the rollback path (`:536`). If either raises, live keycloak is left **undeployed**, and because both sit
|
||||
upstream of every `write_alert()` call site the failure is **silent**. Pre-dates the enrollment (present at
|
||||
`07fc6d4`). Adversary concurred it is not a VETO. Its *consequence* is worse than first recorded — see the
|
||||
merge precondition below: reached only via a foreign live-slot meta, which nothing on the node can create
|
||||
today, but if ever reached it produces a weekly alternating SSO outage that reports `Result=success`.
|
||||
|
||||
**MERGE PRECONDITION for `b5f2b10`.** The F-redfix-4 fix *adds* a raise path to both of those calls —
|
||||
`warmsnap._assert_slot_not_foreign()` (`warmsnap.py:158` in `snapshot()`, `:209` in `restore()`). It raises
|
||||
iff the live slot's `snapshot/meta.json` records a `domain` other than the one passed. Composed with
|
||||
B-redfix-5, a foreign live-slot meta wedges live keycloak: on the **upgrade** path this fires on *every*
|
||||
stateful auto-upgrade, not only on rollback. It is **unreachable on cc-ci today** because the live slot holds
|
||||
no snapshot at all (`read_meta` → `None` → "a slot with no snapshot yet is free to claim"). It is unreachable
|
||||
for that reason — **not** because F-redfix-4 is closed.
|
||||
|
||||
**THE BINDING RULE: never deploy `07fc6d4`.** (Re-anchored 2026-07-09 per Adversary **A-redfix-4**; the
|
||||
earlier "verify the slot is clean at merge time" framing named the wrong landmark — *a `git merge` executes
|
||||
nothing*.) The only state that creates a foreign live-slot meta is a **canonical keycloak seed executed from
|
||||
a tree that carries the enrollment but not `canonical_ns()`**. Exactly one such tree exists: `07fc6d4`
|
||||
(`tests/keycloak/recipe_meta.py` → `WARM_CANONICAL = True`, no `canonical_ns()`). `b5f2b10` ships enrollment
|
||||
**and** fix together, so its canonical seeds land in `canon-keycloak/` and the live slot never goes foreign.
|
||||
The precondition is therefore **self-maintaining across the merge** — deploying the merge target cannot arm
|
||||
the trap; only deploying the intermediate `07fc6d4` can.
|
||||
|
||||
**No autonomous node activity can arm it.** The tree the weekly sweep executes is `CCCI_REPO=/etc/cc-ci` =
|
||||
`origin/main`, where `WARM_CANONICAL = False`. Only `07fc6d4` and `b5f2b10` have `True`.
|
||||
|
||||
**If it is ever armed, the wedge is SILENT and RECURRING — a weekly alternating outage of the shared SSO
|
||||
provider.** (Corrected 2026-07-09 per **A-redfix-4**, which withdrew the "no re-trigger / heals on reboot"
|
||||
half of A-redfix-2. My own earlier "recovery is manual" (`68b51d5`) was also wrong. Re-derived independently
|
||||
against the node — see HOW below.) `warm_reconcile.py` has **two** invokers, and the second is a weekly timer:
|
||||
|
||||
nightly-sweep.timer (OnCalendar=Sun *-*-* 03:00:00, Persistent=true)
|
||||
└─ nightly_sweep.main():147 → roll_warm_infra() → subprocess warm_reconcile.py keycloak
|
||||
(WARM_APPS = ["keycloak", "traefik"], nightly_sweep.py:39,57-61)
|
||||
|
||||
Once armed, with `current != latest`: **sweep N** → `abra.undeploy()` → `snapshot()` guard raises → keycloak
|
||||
**DOWN**. **Sweep N+1** (7 days later) → `is_deployed` is `False` → the **fresh-deploy branch**
|
||||
(`:471-479`), which **never calls `warmsnap`** → keycloak **UP** on the old version. **Sweep N+2** → upgrade
|
||||
path → **DOWN** again. **Alternating weeks, indefinitely**, because the foreign `snapshot/meta.json` is never
|
||||
removed. (`abra.undeploy()` (`abra.py:295-297`) runs only `abra app undeploy -n` and does **not** remove the
|
||||
app `.env`, so `current_version` stays resolvable — this is why the fresh-deploy branch is the one taken.)
|
||||
|
||||
**And it raises no alert.** `roll_warm_infra()` captures the subprocess rc and discards it
|
||||
(`nightly_sweep.py:59-62` — `rc` is only ever printed), so `nightly-sweep.service` still reports
|
||||
`Result=success`. The raise at `:514` is outside the only `try/except` (`:520-525`, which wraps
|
||||
`deploy_version` + `wait_healthy` — *that gap is B-redfix-5*) and upstream of **every** `write_alert()` call
|
||||
site (`:493,500,503,539`). `reconcile()`'s sole caller `main():556` does not catch either. A weekly SSO
|
||||
outage would surface **nowhere**. (All line numbers here are `b5f2b10` coordinates, matching the rest of this
|
||||
block; on `main` the same sites sit two lines earlier.)
|
||||
|
||||
**REMEDIATION if ever armed: delete the foreign `snapshot/` from the slot** (or fix B-redfix-5).
|
||||
**Redeploying keycloak does NOT fix it** — the next sweep re-wedges. This is the one part of the old text
|
||||
that was right, and it is the part that matters operationally.
|
||||
|
||||
**HOW to verify** (from a clone at `b5f2b10`). Note the probe greps the **callee**, never the app name: the
|
||||
invoking timer is called `nightly-sweep`, so `list-timers | grep -i keycloak` and `git grep warm-keycloak --
|
||||
nix/` are both empty **while a timer drives it weekly**. Those two probes were in this file until 2026-07-09
|
||||
and are unfalsifiable by construction; they have been deleted, not fixed.
|
||||
|
||||
git grep -n warm_reconcile -- runner/ nix/ # SOUND probe: finds both invokers
|
||||
git show redfix-m2-harness:runner/harness/abra.py | sed -n '295,297p' # undeploy: no .env removal
|
||||
git show redfix-m2-harness:runner/warm_reconcile.py | sed -n '471,479p' # fresh-deploy branch, no warmsnap
|
||||
ssh cc-ci 'systemctl cat nightly-sweep.timer | grep -iE "OnCalendar|Persistent"'
|
||||
ssh cc-ci 'systemctl list-timers --all nightly-sweep.timer --no-pager'
|
||||
|
||||
**EXPECTED:** `git grep warm_reconcile` prints both `nix/modules/warm-keycloak.nix:26` (the oneshot unit) and
|
||||
`runner/nightly_sweep.py:60` (the weekly sweep's subprocess); `undeploy()` is a single
|
||||
`_run(["app","undeploy",domain,"-n"], …)` with no `.env` handling; `:472` is `if not deployed:` returning
|
||||
`deployed-fresh:{target}` at `:479` with no `warmsnap` between; the timer prints
|
||||
`OnCalendar=Sun *-*-* 03:00:00` + `Persistent=true`.
|
||||
|
||||
**Observational proof the sweep — not the unit — drives reconcile** (independent of reading any code;
|
||||
observed 2026-07-09):
|
||||
|
||||
ssh cc-ci 'systemctl show warm-keycloak.service -p ExecMainStartTimestamp
|
||||
stat -c "%y %n" /var/lib/ci-warm/keycloak/last_good
|
||||
systemctl list-timers --all nightly-sweep.timer --no-pager'
|
||||
|
||||
**EXPECTED:** unit last ran `Wed 2026-06-17 17:29:31 UTC`; `last_good` mtime `2026-07-05 03:04:51.209`; timer
|
||||
last trigger `Sun 2026-07-05 03:04:50 UTC`. The slot was written **18 days after the unit last ran and 1.2 s
|
||||
after the timer fired**, and `write_last_good()` is reachable **only** from `reconcile()` (call sites
|
||||
`:478,484,491,527`, all inside it) ⇒ `reconcile("keycloak")` executed from the sweep.
|
||||
|
||||
**Also:** `warm-keycloak.service`'s `ExecStart` resolves to `/nix/store/…-runner/warm_reconcile.py`, and
|
||||
`harness/warmsnap.py` lives in that **same** derivation. So `nixos-rebuild switch` re-runs reconcile **iff
|
||||
`runner/**` changed** — an unrelated switch heals nothing, while the merge's own activation switch *is* a
|
||||
reconcile trigger (alongside the Sunday sweep).
|
||||
|
||||
**Blast radius: keycloak only.** `warm.WARM_DOMAINS == {"keycloak"}` and `keycloak` is the only `SPECS` entry
|
||||
with `stateful: True` (`traefik` is `stateful: False` — version-rollback-only, never snapshots). No other
|
||||
warm unit reaches either call site.
|
||||
|
||||
**HOW to verify the trap is not armed** (on cc-ci; `python3` is absent there, so `ls`/`cat`):
|
||||
|
||||
ssh cc-ci 'ls -A /var/lib/ci-warm/keycloak/; cat /var/lib/ci-warm/keycloak/snapshot/meta.json 2>&1'
|
||||
|
||||
**EXPECTED (observed 2026-07-09T08:5xZ):** `last_good` only, and `cat` reports `No such file or directory` —
|
||||
no `snapshot/`, hence no `meta.json`; no `/var/lib/ci-warm/canon-keycloak` either; live keycloak `200`. **A
|
||||
`meta.json` whose `"domain"` is anything other than `warm-keycloak.ci.commoninternet.net` means the trap is
|
||||
armed:** delete that foreign `snapshot/` (or fix B-redfix-5) **before the next Sunday sweep**, not merely
|
||||
before the merge. A `meta.json` recording `warm-keycloak.ci.commoninternet.net` is self-consistent and passes
|
||||
the guard.
|
||||
|
||||
---
|
||||
|
||||
### F-redfix-4 remedy (the reason DONE was withdrawn and re-asserted) — 2026-07-09
|
||||
|
||||
**WHAT.** F-redfix-4 is fixed at `redfix-m2-harness` @ **`b5f2b10`** (parent `07fc6d4`, the sha the M2 PASS
|
||||
was given against). Warm state is now keyed by a **stack namespace**, not a bare recipe:
|
||||
`canonical.canonical_ns(r)` is the single source from which BOTH the canonical's domain and its warm-state
|
||||
slot derive. A live-warm provider (`r in warm.WARM_DOMAINS`) gets ns `canon-<r>` → domain
|
||||
`warm-canon-keycloak.ci.commoninternet.net` (**unchanged**) + slot `/var/lib/ci-warm/canon-keycloak/`.
|
||||
Every other recipe keeps ns `<r>` (the invariant is structural — `r not in warm.WARM_DOMAINS` — not a
|
||||
property of any particular count). `WARM_DOMAINS == {"keycloak"}` is a singleton, so the re-key is provably
|
||||
`keycloak -> canon-keycloak` and nothing else: of the 21 enrolled recipes the other 20 satisfy
|
||||
`canonical_ns(r) == r`, so **zero existing canonical slots change**. On live disk that is 17 seeded
|
||||
canonicals (those carrying a `canonical.json`), all of which remain in `prune_stale`'s `keep` set.
|
||||
|
||||
Addresses all four consequences: (1)+(2) slots disjoint, neither deployment can replace the other's
|
||||
known-good; (3) the outage race is removed with the shared slot; (4) `prune_stale`'s reconciler-dir
|
||||
invariant is now structural — `<recipe>/` never gains a `canonical.json`.
|
||||
|
||||
Also: `warmsnap._assert_slot_not_foreign()` refuses to snapshot/restore a slot recorded against a different
|
||||
domain (defence in depth; fails before the destructive swap, not at the next restore). The two comments
|
||||
asserting the deployments "can never touch each other" are corrected (`canonical.py`,
|
||||
`tests/keycloak/recipe_meta.py`). `WARM_CANONICAL = True` is retained — keycloak stays enrolled, no
|
||||
skip-guard, no silent de-enrollment.
|
||||
|
||||
**MIGRATION: none required.** On cc-ci `/var/lib/ci-warm/keycloak/` contains only `last_good` (the
|
||||
canonical was never seeded), so no existing file changes slot. Verify: `ls -A /var/lib/ci-warm/keycloak/`
|
||||
→ `last_good` only.
|
||||
|
||||
**WHERE.** cc-ci repo, branch `redfix-m2-harness` @ `b5f2b10`. Files: `runner/harness/warmsnap.py`,
|
||||
`runner/harness/canonical.py`, `runner/warm_reconcile.py`, `runner/run_recipe_ci.py`,
|
||||
`tests/keycloak/recipe_meta.py`, `tests/unit/test_warmsnap.py`, `tests/unit/test_canonical.py`.
|
||||
|
||||
**HOW to verify (1) — unit suite, cold clone, no docker needed.**
|
||||
|
||||
git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/v && cd /tmp/v
|
||||
/nix/store/x188l04r3gfkh18gy1dpf05fv3kkrgs7-python3-3.12.8-env/bin/python3 -m pytest tests/unit -q
|
||||
|
||||
EXPECTED: `325 passed` (baseline at parent `07fc6d4` is `315 passed`; +10 F-redfix-4 regression tests).
|
||||
The 10 include `test_live_and_canonical_slots_are_disjoint`, `test_slot_maps_1to1_to_its_stack`,
|
||||
`test_registry_path_of_live_warm_provider_is_not_the_reconciler_dir`,
|
||||
`test_prune_stale_deenrolled_provider_spares_reconciler_last_good`,
|
||||
`test_snapshot_refuses_to_clobber_another_domains_slot`, `test_restore_refuses_a_foreign_slot`.
|
||||
|
||||
**HOW to verify (2) — the VETO's clearing condition, on the real node.** Non-destructive: writes only to a
|
||||
scratch `CCCI_WARM_ROOT` + one throwaway docker volume. Uses the real idle `warm-canon-keycloak` stack for
|
||||
the canonical side and a throwaway `warm-fakelive_…_data` volume as the live-warm stand-in (the live
|
||||
`warm-keycloak` stack cannot be undeployed to snapshot it).
|
||||
|
||||
ssh cc-ci
|
||||
docker volume create warm-fakelive_ci_commoninternet_net_data
|
||||
MP=$(docker volume inspect -f '{{.Mountpoint}}' warm-fakelive_ci_commoninternet_net_data); echo x > $MP/live.txt
|
||||
git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/v && mkdir -p /tmp/vw/keycloak
|
||||
echo '10.7.1+26.6.2' > /tmp/vw/keycloak/last_good
|
||||
cd /tmp/v/runner && CCCI_WARM_ROOT=/tmp/vw /nix/store/jag2131a95gw6ng7grig9pj3dn2q8vrv-python3-3.12.8-env/bin/python3 - <<'PY'
|
||||
import sys; sys.path.insert(0, ".")
|
||||
from harness import warmsnap as ws, canonical as c
|
||||
LIVE, CANON = ws.live_slot("keycloak"), c.canonical_slot("keycloak")
|
||||
CANON_DOM, LIVE_DOM = c.canonical_domain("keycloak"), "warm-fakelive.ci.commoninternet.net"
|
||||
print(ws.snap_dir(LIVE), ws.snap_dir(CANON), c.registry_path("keycloak"), sep="\n")
|
||||
ws.snapshot(CANON, CANON_DOM, version="canon-known-good")
|
||||
ws.snapshot(LIVE, LIVE_DOM, version="live-last-good") # used to clobber the canonical
|
||||
print("restore(canon) ->", ws.restore(CANON, CANON_DOM)["volumes"])
|
||||
print("restore(live) ->", ws.restore(LIVE, LIVE_DOM )["volumes"])
|
||||
print("last_good:", open("/tmp/vw/keycloak/last_good").read().strip())
|
||||
try: ws.snapshot(CANON, LIVE_DOM); print("FAIL: foreign snapshot allowed")
|
||||
except ws.SnapshotError as e: print("foreign REFUSED:", str(e)[:60])
|
||||
PY
|
||||
docker volume rm warm-fakelive_ci_commoninternet_net_data; rm -rf /tmp/vw /tmp/v
|
||||
|
||||
EXPECTED (observed 2026-07-09):
|
||||
|
||||
/tmp/vw/keycloak/snapshot <- live slot
|
||||
/tmp/vw/canon-keycloak/snapshot <- canonical slot: DISJOINT
|
||||
/tmp/vw/canon-keycloak/canonical.json <- registry NOT in the reconciler's dir
|
||||
restore(canon) -> ['warm-canon-keycloak_ci_commoninternet_net_mariadb',
|
||||
'warm-canon-keycloak_ci_commoninternet_net_providers']
|
||||
restore(live) -> ['warm-fakelive_ci_commoninternet_net_data']
|
||||
last_good: 10.7.1+26.6.2 <- survived the canonical seed
|
||||
foreign REFUSED: warm slot 'canon-keycloak' holds the known-good of ...
|
||||
|
||||
Each `restore()` returns its OWN stack's volumes (the clearing condition). Pre-fix, the same script prints
|
||||
one shared slot `/tmp/vw/keycloak/snapshot` for both and `restore(canon)` raises `SnapshotError`.
|
||||
|
||||
**Node integrity after my own run of the above (2026-07-09).** Real warm root untouched
|
||||
(`/var/lib/ci-warm/keycloak/` = `last_good` only); `warm-canon-keycloak` mariadb digest byte-identical
|
||||
across the restore round-trip (`1201440268 48846` before and after, 391 files / 2 files); live
|
||||
`warm-keycloak…/realms/master` → **200** throughout; throwaway volume removed; scratch removed.
|
||||
|
||||
**Scope.** keycloak only. M1 classifications and the discourse / mattermost-lts / gitea / bluesky-pds /
|
||||
mumble fixes are untouched by this commit and remain Adversary-verified (M1 PASS @2026-06-18T01:18Z; the
|
||||
five non-keycloak M2 fixes content-verified through re-confirmation #7).
|
||||
|
||||
**Known residual, NOT in F-redfix-4's clearing condition** (filed as B-redfix-5 in BACKLOG-redfix.md, not
|
||||
blocking): `warm_reconcile.py`'s rollback `warmsnap.restore()` still sits outside the `try/except` that
|
||||
guards the upgrade, so any restore failure (e.g. a corrupt/absent snapshot) leaves live keycloak undeployed
|
||||
after `abra.undeploy()`. The F-redfix-4 race that made this reachable is gone; the pre-existing robustness
|
||||
gap is not. Recorded, not silently dropped.
|
||||
|
||||
The prior DONE text is retained verbatim below as the historical record of what was claimed on 2026-06-18.
|
||||
|
||||
---
|
||||
|
||||
## (HISTORICAL — withdrawn 2026-07-09 under VETO F-redfix-4; superseded by the DONE above) DONE — 2026-06-18T07:09Z
|
||||
|
||||
Phase `redfix` COMPLETE. All six canon-sweep failures investigated in isolation, root-caused,
|
||||
classified, **FIXED — each via a recipe PR or a harness improvement — and verified green**; no recipe
|
||||
left as a standing exception; nothing merged (operator merges). Both gates have a fresh Adversary PASS
|
||||
in REVIEW-redfix.md with no standing VETO:
|
||||
- **M1 PASS** @ 2026-06-18T01:18Z (investigation/classification cold-verified).
|
||||
- **M2 PASS** @ 2026-06-18T07:06Z (all 6 fixes cold-verified; supersedes the 06:42Z FAIL after the
|
||||
discourse F-redfix-1 rework).
|
||||
|
||||
Fixes (per recipe): mattermost-lts recipe PR #1 (pg_backup.sh + restore.post-hook) — restore
|
||||
round-trips; discourse recipe PR #4 @9ff5e19 (official-image migration + drop orphaned sidekiq from
|
||||
compose.smtpauth.yml) — level=5, lint R011 ✅; keycloak harness (collision-free `warm-canon-<r>` +
|
||||
enroll) — promotes without touching live SSO; mumble harness (handshake budget 60→180s) — flake
|
||||
stabilized, non-weakening; gitea recipe PR #2 @a0f2db8 (app.ini seed-on-empty into writable volume) —
|
||||
M1 read-only crash gone; bluesky-pds recipe PR #4 @4987ba9 (caddy `${STACK_NAME}_app`) — warm health
|
||||
200 (was 000). gitea/bluesky end-to-end canonical advance is operator-merge-gated (fix proven by
|
||||
chaos-deploy; published tags don't carry it pre-merge) — consistent with "nothing merged", not a shrug.
|
||||
|
||||
### Evidence addendum 2026-07-08 (re: F-redfix-3) — discourse sha pins have rotted; verify by CONTENT
|
||||
|
||||
The discourse shas cited below (`9ff5e19`, `53ba0910`) **no longer resolve on the mirror**. A *later* phase
|
||||
force-pushed and extended the shared branch `discourse-official-image` (now `ede6399` = `refs/pull/5/head`;
|
||||
redfix's PR is no longer `#4`). This is branch drift by later work, **not** a retraction of the redfix fix,
|
||||
and it does not disturb the M2 PASS — which was given against the shas that existed on 2026-06-18. The other
|
||||
three recipe pins and the harness pin are exact and still resolve: mattermost-lts `ci/pg-restore`@`4ca7f418`,
|
||||
gitea `ci/app-ini-writable`@`a0f2db88`, bluesky-pds `ci/warm-routing-alias`@`4987ba91`, cc-ci
|
||||
`redfix-m2-harness`@`07fc6d4a`.
|
||||
|
||||
Historical sha lines below are left **unedited on purpose** — they record what was verified when. Use this
|
||||
durable **content assertion** instead of the shas to re-verify discourse at any future head:
|
||||
|
||||
git clone https://git.autonomic.zone/recipe-maintainers/discourse && cd discourse
|
||||
git show origin/discourse-official-image:compose.yml | grep -m1 'image:.*discourse'
|
||||
git show origin/discourse-official-image:compose.smtpauth.yml | grep -c sidekiq
|
||||
|
||||
EXPECTED: `image: discourse/discourse:3.5.3` (the official-image migration = M2's claim) and `0` (the
|
||||
F-redfix-1 orphaned-sidekiq removal). Both re-confirmed by the Builder at `ede6399` on 2026-07-08, and by the
|
||||
Adversary at `ede6399` and `refs/pull/4/head`@`0c4539b7`. (Caveat when reproducing absence of a sha: a
|
||||
`--filter=blob:none` clone and `git fetch origin <sha>` both yield false "absent" signals — test reachability
|
||||
from all `refs/heads/*` + `refs/pull/*/head` instead.)
|
||||
|
||||
---
|
||||
|
||||
## Phase: M1 — investigate + isolate + classify (IN PROGRESS)
|
||||
|
||||
Bootstrapped 2026-06-17T23:20Z. cc-ci healthy, no run in flight, next scheduled sweep 2026-06-21
|
||||
@ -571,126 +78,18 @@ mirrors via the recipe mirror+PR flow, verified `!testme` (NEVER merge). Harness
|
||||
on a cc-ci branch, verified via the harness. discourse: overlay-scope decision. Node now free for my
|
||||
deploys (Adversary done with M1).
|
||||
|
||||
### M2 fix tracker (updated 2026-06-18T05:53Z — ALL VERIFIED)
|
||||
### M2 fix tracker
|
||||
|
||||
| Recipe | Class | Fix | PR/branch + ref | Status |
|
||||
|---|---|---|---|---|
|
||||
| mattermost-lts | recipe defect | pg_backup.sh + `backupbot.restore.post-hook` (immich pattern) | mirror PR #1 `ci/pg-restore` @4ca7f418 | **VERIFIED** — !testme run #901 ALL tiers green incl `test_restore_returns_state` |
|
||||
| discourse | stale cc-ci overlay | recipe: bitnamilegacy->official discourse image migration + drop orphaned image-less sidekiq from compose.smtpauth.yml (F-redfix-1) | mirror PR #4 `discourse-official-image` @9ff5e19 | **VERIFIED** — own cold run `/tmp/redfix-discourse-m2verify.log` **level=5 of 5** (all tiers + lint R011 PASS); F-redfix-1 regression fixed |
|
||||
| keycloak | harness defect | collision-free `canonical_domain` (`warm-canon-<r>` for WARM_DOMAINS recipes) + enroll | cc-ci branch `redfix-m2-harness` @61211db | **VERIFIED** — branch-checkout run promotes at warm-canon-keycloak; live warm-keycloak 200 throughout |
|
||||
| mumble | load/timing flake | harness: handshake readiness budget 60s->180s | cc-ci branch `redfix-m2-harness` @07fc6d4 | **VERIFIED** — branch-checkout run all tiers green incl handshake; budget active+non-regressing |
|
||||
| gitea | recipe defect | app.ini->staging `/etc/gitea/app.ini.init` + docker-setup seed-on-EMPTY + DOCKER_SETUP_SH_VERSION v3 | mirror PR #2 `ci/app-ini-writable` @a0f2db8 | **VERIFIED** (direct chaos-deploy; promote merge-gated — see below) |
|
||||
| bluesky-pds | recipe defect (routing) | caddy `{$APP_HOST}=${STACK_NAME}_app` (operator: NO rename) + CADDYFILE_VERSION v2 | mirror PR #4 `ci/warm-routing-alias` @4987ba9 | **VERIFIED** (direct chaos-deploy; promote merge-gated — see below) |
|
||||
| Recipe | Fix type | PR/branch | Status |
|
||||
|---|---|---|---|
|
||||
| mattermost-lts | recipe PR (pg_backup.sh + restore.post-hook) | mirror PR #1 `ci/pg-restore` @4ca7f418 | **DONE — !testme run #901 ALL tiers green** (restore__cc-ci failures=0 skipped=0; the M1-failing test_restore_returns_state now PASSES) |
|
||||
| bluesky-pds | recipe PR (unique `pds` internal alias for caddy) | mirror PR #4 `ci/warm-routing-alias` | PR created; verifying on PROMOTE path (warm-bluesky-pds → expect 200 vs M1 000; !testme cold-only won't reproduce) |
|
||||
| gitea | recipe PR (app.ini → writable volume) | — | pending |
|
||||
| keycloak | harness (collision-free canonical_domain) + enroll | — | pending |
|
||||
| mumble | harness (handshake readiness/retry stabilization) | — | pending |
|
||||
| discourse | recipe PR (official-image migration) | mirror PR #4 `discourse-official-image` | already !testme-GREEN @53ba0910 (run #849, 16:36Z); re-verify fresh |
|
||||
|
||||
cc-ci-side change verification: run from a checkout of `redfix-m2-harness` (CCCI_REPO=<checkout>);
|
||||
never touches /etc/cc-ci main. `redfix-m2-harness` is now mumble+keycloak ONLY (bluesky needs no
|
||||
cc-ci change with the ${STACK_NAME}_app approach; the rename's exec-ref commit b96b8a4 was dropped).
|
||||
|
||||
## Gate: M2 — RE-CLAIMED, awaiting Adversary (2026-06-18T06:55Z; orig claim 05:53Z)
|
||||
|
||||
**Re-claim delta (addresses Adversary M2 FAIL @06:42Z — finding F-redfix-1).** The first M2 verdict was
|
||||
FAIL on discourse ONLY (other 5 PASS, do-not-redo). F-redfix-1: the official-image migration dropped
|
||||
`sidekiq` from compose.yml but left a dangling image-less `sidekiq:` block in `compose.smtpauth.yml` →
|
||||
L5 lint R011 fail (run level=4) + broken SMTP-auth deploy. **FIXED** in PR #4 `discourse-official-image`
|
||||
@**9ff5e19** (force-pushed onto @53ba0910): dropped the orphaned `sidekiq:` block; the `app:` override
|
||||
already carries `DISCOURSE_SMTP_PASSWORD_FILE` + `smtp_password` secret (sidekiq is internal to the
|
||||
official image), so no SMTP coverage lost. `grep sidekiq compose*.yml` = 0.
|
||||
**VERIFIED two ways:** (1) the Adversary's exact lint.py repro flow at 9ff5e19 → **R011 ✅**; (2) my own
|
||||
full cold run `/tmp/redfix-discourse-m2verify.log` → `RUN SUMMARY ... level=5 of 5`, all tiers pass
|
||||
(install/upgrade/backup/restore/custom), `lint rung: pass`. Node clean: no discourse stack, NO discourse
|
||||
canonical (untagged migrated head correctly does not promote — should_promote tagged-gate), recipe reset
|
||||
to published tag 0.8.1+3.5.0. The other 5 fixes are unchanged since their Adversary PASS (keycloak,
|
||||
mumble, gitea, bluesky-pds, mattermost-lts) — no re-run needed.
|
||||
|
||||
Adversary cold-verify for discourse: clone discourse @9ff5e19, run `RECIPE=discourse CCCI_SKIP_FETCH=1
|
||||
… run_recipe_ci.py` → EXPECT level=5 of 5 (lint R011 ✅, all tiers pass, both upgrade-overlay tests
|
||||
`test_head_runs_official_image_not_bitnamilegacy` + `test_sidekiq_service_dropped_by_head` pass); OR the
|
||||
lint-only repro in F-redfix-1 → R011 ✅. `grep -c sidekiq ~/.abra/recipes/discourse/compose*.yml` @9ff5e19 = 0.
|
||||
|
||||
---
|
||||
## Gate: M2 — original claim (2026-06-18T05:53Z)
|
||||
|
||||
**WHAT (M2 DoD).** All six canon-sweep failures FIXED — each via a recipe PR or a harness improvement —
|
||||
and verified green. No recipe left as a standing exception. Nothing merged (operator merges). Per recipe:
|
||||
|
||||
- **mattermost-lts** (recipe PR #1) — added `pg_backup.sh` + postgres `backupbot.restore.post-hook` so
|
||||
the logical dump round-trips on restore.
|
||||
- **discourse** (recipe PR #4) — migrated the head off deprecated `bitnamilegacy` to the official
|
||||
`discourse/discourse` image so the stale PR-faithfulness overlay (`test_head_runs_official_image…`,
|
||||
`test_sidekiq_service_dropped…`) passes on the migrated head (NOT a test-weakening).
|
||||
- **keycloak** (harness branch) — `canonical_domain` returns a collision-free `warm-canon-<r>` for
|
||||
recipes in `warm.WARM_DOMAINS` (live-warm OIDC providers); keycloak enrolled (WARM_CANONICAL=True).
|
||||
- **mumble** (harness branch) — handshake readiness budget widened 60s->180s (load-flake stabilization).
|
||||
- **gitea** (recipe PR #2) — app.ini is now seeded into the WRITABLE `/etc/gitea` volume by
|
||||
docker-setup (`if [ ! -s /etc/gitea/app.ini ]`, seed-on-EMPTY) from the read-only staging config
|
||||
`app.ini.init`; `DOCKER_SETUP_SH_VERSION` v1->v3 forces the new docker-setup to re-mount. Gitea
|
||||
1.24.2 can then persist its JWT secret (the M1 read-only-app.ini crash is gone).
|
||||
- **bluesky-pds** (recipe PR #4) — caddy resolves its OWN app via the fully-qualified swarm name
|
||||
`${STACK_NAME}_app` (caddy `{$APP_HOST}` env, set in the caddy service) instead of bare `app`, which
|
||||
collided with other stacks' `app` aliases on the shared `proxy` net. CADDYFILE_VERSION v1->v2.
|
||||
|
||||
**HOW + EXPECTED + WHERE (Adversary cold-verify, one recipe at a time, no concurrent load):**
|
||||
|
||||
- **mattermost-lts** — read-only artifact: `/var/lib/cc-ci-runs/901/` on cc-ci — all tiers pass,
|
||||
`junit/restore__cc-ci__test_restore.xml` testsuite failures=0, `test_restore_returns_state` pass.
|
||||
OR re-run !testme on PR #1 @4ca7f418. EXPECT restore green.
|
||||
- **discourse** — !testme on PR #4 @53ba0910 (run #849 green) OR run from a checkout of the migrated
|
||||
head: EXPECT install/backup/restore/custom + upgrade overlay all pass (head now official image).
|
||||
- **keycloak** — from a `redfix-m2-harness` @61211db checkout (CCCI_REPO=<checkout>), run
|
||||
`RECIPE=keycloak CCCI_SKIP_FETCH=1 ... run_recipe_ci.py`. EXPECT all cold tiers pass + WC5 promote
|
||||
succeeds at domain `warm-canon-keycloak.ci.commoninternet.net` (NOT warm-keycloak); live
|
||||
`warm-keycloak.ci.commoninternet.net/realms/master` stays 200 throughout. Code: `canonical.py`
|
||||
canonical_domain returns warm-canon-<r> for r in warm.WARM_DOMAINS.
|
||||
- **mumble** — from `redfix-m2-harness` @07fc6d4 checkout, run `RECIPE=mumble CCCI_SKIP_FETCH=1 …`.
|
||||
EXPECT all 5 tiers green incl `custom/test_protocol_handshake.py::test_handshake_completes_with_
|
||||
channel_presence`; handshake budget = 36 attempts / 180s (was 60s). (Load-flake is not
|
||||
deterministically reproducible; this verifies the stabilization is applied, sound, non-weakening.)
|
||||
- **gitea** (recipe PR #2 @a0f2db8 on mirror branch `ci/app-ini-writable`) — DIRECT chaos-deploy proof
|
||||
(the harness WC5 promote is merge-gated, see NOTE). With the idle 3.5.3 canonical present:
|
||||
`cd ~/.abra/recipes/gitea && git checkout -f a0f2db8` then chaos-deploy onto the retained canonical
|
||||
volumes (0-byte app.ini = genuine pre-fix 3.5.3 state):
|
||||
`abra app deploy warm-gitea.ci.commoninternet.net -C -o -n`. EXPECT: service 1/1; the config volume's
|
||||
`app.ini` seeded 0->~1862 bytes (`INSTALL_LOCK = true`); `/api/v1/version` -> 200 {"version":"1.24.2"}
|
||||
and `/api/healthz` -> 200 (curl inside the app container); retained 3.5.3 data adopted (data dirs
|
||||
dated 2026-06-17T08:39); ZERO `read-only file system` crashes in `docker service logs` (M1 crashed
|
||||
here). Evidence: `/tmp/redfix-gitea-m2-directproof.log` on cc-ci. Teardown: `abra app undeploy … -n`,
|
||||
truncate the volume app.ini to 0 (restore pre-fix state). canonical.json stays 3.5.3 idle e6a1cc79.
|
||||
- **bluesky-pds** (recipe PR #4 @4987ba9 on mirror branch `ci/warm-routing-alias`) — DIRECT chaos-deploy
|
||||
proof (warm-promote is the only failing path; merge-gated). `git checkout -f 4987ba9`; generate
|
||||
secrets (`abra app secret generate warm-bluesky-pds.ci.commoninternet.net --all -m -C -o -n`) + insert
|
||||
a PLC rotation key (tests/bluesky-pds/install_steps.sh logic: 32-byte hex into pds_plc_rotation_key
|
||||
v1); **re-checkout 4987ba9 AFTER secret ops** (abra secret insert force-fetches+reverts the checkout);
|
||||
`abra app deploy warm-bluesky-pds.ci.commoninternet.net -C -o -n` (EXPECT `caddyfile: v1 -> v2`,
|
||||
NEW DEPLOYMENT 4987ba9). EXPECT: app+caddy 1/1; inside caddy `getent hosts
|
||||
warm-bluesky-pds_ci_commoninternet_net_app` -> a 10.0.x.x INTERNAL ip (own stack) while
|
||||
`getent hosts app` -> a 10.10.x.x proxy ip (foreign, the M1 collision); caddy log "certificate
|
||||
obtained successfully" with 0 "connection refused"; external `curl https://warm-bluesky-pds.ci.
|
||||
commoninternet.net/xrpc/_health` -> **200** {"version":"0.4.219"} (M1 was 000). Evidence:
|
||||
`/tmp/redfix-bluesky-m2-directproof.log`. Teardown: undeploy + remove volumes (caddy_data, pds_data)
|
||||
+ secrets (no canonical, matching M1).
|
||||
|
||||
**NOTE — gitea & bluesky end-to-end canonical-promote is OPERATOR-MERGE-GATED (not a shrug).** The
|
||||
harness WC5 promote does a recipe_checkout(published-tag)+non-chaos deploy, and BOTH run_recipe_ci.py:373
|
||||
AND abra force-fetch `refs/tags/*` from upstream (abra.py:135 documents this), so any local move of the
|
||||
release tag to the fix commit is reverted to the PUBLISHED commit. The published 3.6.0 / 0.3.0 tags do
|
||||
NOT yet carry the fix (PR not merged — operator merges, per phase guardrail), so pre-merge the promote
|
||||
necessarily deploys the unfixed published release. Confirmed empirically: a full gitea harness run's WC5
|
||||
promote deployed 357926f and crash-looped exactly like M1. The DIRECT chaos-deploy (chaos = deploy the
|
||||
working-tree checkout = the PR fix) is therefore the MAXIMAL + faithful pre-merge proof — it reproduces
|
||||
the EXACT M1 failing scenario (gitea: the retained canonical volumes; bluesky: warm-bluesky-pds on the
|
||||
shared proxy) and shows the fix resolves it. End-to-end canonical advance follows automatically once the
|
||||
operator merges PR #2 / #4 and the release tag carries the fix. This is NOT a standing exception — the
|
||||
defect is fixed + proven; only the registry-advance awaits the operator's merge (the phase's own
|
||||
"nothing merged" constraint).
|
||||
|
||||
**WHERE (refs).** Recipe PRs on `git.autonomic.zone/recipe-maintainers/<recipe>`: mattermost-lts
|
||||
`ci/pg-restore`@4ca7f418, discourse `discourse-official-image`@53ba0910, gitea `ci/app-ini-writable`
|
||||
@a0f2db8, bluesky-pds `ci/warm-routing-alias`@4987ba9. cc-ci harness branch
|
||||
`redfix-m2-harness`@07fc6d4 (keycloak 61211db + mumble 07fc6d4). Reasoning/dead-ends in
|
||||
JOURNAL-redfix.md. Node left clean (only infra + live warm-keycloak 200; gitea idle 3.5.3 volumes
|
||||
retained, canonical e6a1cc79 unchanged; no bluesky/test stacks/volumes/secrets; no run procs).
|
||||
|
||||
## Gate: M1 — PASS (above).
|
||||
## Gate: M1 — PASS (above). M2 not yet claimed.
|
||||
|
||||
**WHAT (M1 DoD).** All six canon-sweep failures investigated in ISOLATION (one recipe at a time, no
|
||||
concurrent sweep load), root-caused with first-hand evidence, and classified (flake vs genuine; recipe
|
||||
@ -707,33 +106,6 @@ in the two sections above ("M1 results table" and "HOW the Adversary cold-verifi
|
||||
Evidence logs on cc-ci: `/tmp/redfix-{discourse,mattermost-lts,mumble,mumble2,bluesky-pds,gitea2}.log`.
|
||||
Reasoning/dead-ends in JOURNAL-redfix.md. Node left clean (see "Node state left clean" above).
|
||||
|
||||
## Untracked file `main.go` — NOT produced by this phase (operator: unexplained)
|
||||
|
||||
**WHAT.** An untracked `main.go` (281 bytes, a Go "Hello, World!" `net/http` server on `:8080`) sits in the
|
||||
repo root. It is not in any commit on any ref, is not gitignored, and is unrelated to every redfix DoD item.
|
||||
No `go.mod` accompanies it. I did not create it and have not committed, modified, or deleted it.
|
||||
|
||||
**WHERE.** `/srv/cc-ci-orch/cc-ci/main.go` — **one file in one clone.**
|
||||
|
||||
**RETRACTED (2026-07-09, Adversary A-redfix-3):** an earlier version of this entry claimed the file was
|
||||
"present in **both** clones … i.e. written by something outside git." **That evidence is void.** `/srv/cc-ci`
|
||||
is a **symlink to `/srv/cc-ci-orch`**, so `/srv/cc-ci/cc-ci/main.go` and `/srv/cc-ci-orch/cc-ci/main.go` are
|
||||
the **same inode** — one file seen twice, not two clones. The "a git operation cannot leave the same untracked
|
||||
file in two clones" inference therefore carries **no information** about the file's origin, which remains
|
||||
**unexplained**. Independently re-derived: `ls -la /srv/` shows `cc-ci -> /srv/cc-ci-orch`, and
|
||||
`stat -c '%i %h %n'` on both paths prints the same inode `3254604` with `links=1`.
|
||||
|
||||
**HOW to verify.** `git log --all --oneline -- main.go` → empty (in no commit).
|
||||
`git status --short` → `?? main.go`.
|
||||
`ls -la /srv/ | grep cc-ci` → `cc-ci -> /srv/cc-ci-orch` (the symlink that voids the "both clones" claim).
|
||||
`stat -c '%i %h %n' /srv/cc-ci/cc-ci/main.go /srv/cc-ci-orch/cc-ci/main.go` → same inode, `links=1`.
|
||||
|
||||
**Risk: inert** (Adversary-confirmed). 281-byte hello-world `net/http` listener; no `go.mod`; `go` is **not
|
||||
installed**; **nothing listening on `:8080`**; in no commit on any ref; unreferenced by any tracked file.
|
||||
|
||||
**EXPECTED.** Left in place, untracked. Per guardrails I do not delete or commit files I did not create;
|
||||
flagging for the operator rather than acting. It affects no gate, no DoD item, and no verdict.
|
||||
|
||||
## Blocked
|
||||
|
||||
(none)
|
||||
|
||||
@ -39,9 +39,35 @@ def is_enrolled(recipe: str) -> bool:
|
||||
return bool(meta_mod.load(recipe).WARM_CANONICAL)
|
||||
|
||||
|
||||
def canonical_ns(recipe: str) -> str:
|
||||
"""The data-warm canonical's NAMESPACE for a recipe — the single source from which BOTH its
|
||||
domain and its warm-state slot are derived, so the two can never drift apart (F-redfix-4).
|
||||
|
||||
For a recipe that is ALSO a live-warm provider (in `warm.WARM_DOMAINS` — e.g. keycloak, whose
|
||||
always-on shared OIDC instance lives at `warm-keycloak…`), the canonical MUST be namespaced apart
|
||||
from the live provider: otherwise the sweep's promote deploy/teardown at `warm-<recipe>` collides
|
||||
with — and could disrupt — the live shared service that other recipes (lasuite-*/drone) depend on,
|
||||
and (F-redfix-4) the two deployments share one `/var/lib/ci-warm/<recipe>/` snapshot slot, so each
|
||||
destroys the other's known-good. Every other recipe keeps the plain `<recipe>` namespace (zero
|
||||
blast radius on the 15 existing canonicals, whose on-disk layout is unchanged)."""
|
||||
if recipe in warm.WARM_DOMAINS:
|
||||
return f"canon-{recipe}"
|
||||
return recipe
|
||||
|
||||
|
||||
def canonical_domain(recipe: str) -> str:
|
||||
"""Stable data-warm domain for the recipe's canonical."""
|
||||
return warm.stable_domain(recipe)
|
||||
"""Stable data-warm domain for the recipe's canonical (`warm-<ns>`)."""
|
||||
return warm.stable_domain(canonical_ns(recipe))
|
||||
|
||||
|
||||
def canonical_slot(recipe: str) -> str:
|
||||
"""The canonical's warm-state slot: `$CCCI_WARM_ROOT/<ns>/` — holding `canonical.json` and the
|
||||
known-good `snapshot/`.
|
||||
|
||||
DISJOINT from `warmsnap.live_slot(recipe)` (the live-warm reconciler's `<recipe>/`, holding its
|
||||
`last_good`) exactly when the recipe is a live-warm provider. Asserted in
|
||||
`tests/unit/test_canonical.py::test_live_and_canonical_slots_are_disjoint`."""
|
||||
return canonical_ns(recipe)
|
||||
|
||||
|
||||
def enrolled_recipes() -> list[str]:
|
||||
@ -61,7 +87,7 @@ def enrolled_recipes() -> list[str]:
|
||||
|
||||
|
||||
def registry_path(recipe: str) -> str:
|
||||
return os.path.join(warmsnap.app_dir(recipe), "canonical.json")
|
||||
return os.path.join(warmsnap.app_dir(canonical_slot(recipe)), "canonical.json")
|
||||
|
||||
|
||||
def read_registry(recipe: str) -> dict | None:
|
||||
@ -74,7 +100,7 @@ def read_registry(recipe: str) -> dict | None:
|
||||
|
||||
def write_registry(recipe: str, *, version: str, commit: str | None, status: str) -> dict:
|
||||
"""Atomically write the canonical registry record for a recipe."""
|
||||
os.makedirs(warmsnap.app_dir(recipe), exist_ok=True)
|
||||
os.makedirs(warmsnap.app_dir(canonical_slot(recipe)), exist_ok=True)
|
||||
rec = {
|
||||
"recipe": recipe,
|
||||
"domain": canonical_domain(recipe),
|
||||
@ -141,16 +167,22 @@ def undeploy_keep_volume(recipe: str) -> None:
|
||||
|
||||
|
||||
def prune_stale() -> list[str]:
|
||||
"""WC8 disk hygiene: remove warm data for DE-ENROLLED canonicals — a `/var/lib/ci-warm/<recipe>/`
|
||||
that carries a `canonical.json` but whose recipe is no longer enrolled (WARM_CANONICAL dropped).
|
||||
Drops the dir (snapshot + registry) AND the retained `warm-<recipe>` data volumes. Leaves the
|
||||
live-warm reconciler dirs (keycloak/traefik — they have a `last_good`, no `canonical.json`),
|
||||
`alerts/`, and currently-enrolled canonicals untouched. Returns the recipes pruned."""
|
||||
"""WC8 disk hygiene: remove warm data for DE-ENROLLED canonicals — a `/var/lib/ci-warm/<ns>/`
|
||||
that carries a `canonical.json` but whose namespace is no longer an enrolled canonical's slot
|
||||
(WARM_CANONICAL dropped). Drops the dir (snapshot + registry) AND the retained `warm-<ns>` data
|
||||
volumes. Leaves the live-warm reconciler dirs (keycloak/traefik — they have a `last_good`, no
|
||||
`canonical.json`), `alerts/`, and currently-enrolled canonicals untouched. Returns the namespaces
|
||||
pruned.
|
||||
|
||||
The reconciler-dir invariant is STRUCTURAL, not incidental (F-redfix-4): a live-warm provider's
|
||||
canonical registers under `canon-<recipe>/`, so `<recipe>/` never gains a `canonical.json` and is
|
||||
never a prune candidate — de-enrolling keycloak can no longer `rmtree` the reconciler's
|
||||
`last_good`. Dir name == namespace, so the stale stack's domain is `warm.stable_domain(name)`."""
|
||||
import shutil
|
||||
import subprocess
|
||||
|
||||
root = warmsnap.warm_root()
|
||||
keep = set(enrolled_recipes())
|
||||
keep = {canonical_slot(r) for r in enrolled_recipes()}
|
||||
pruned: list[str] = []
|
||||
try:
|
||||
entries = sorted(os.listdir(root))
|
||||
@ -162,8 +194,8 @@ def prune_stale() -> list[str]:
|
||||
continue
|
||||
if not os.path.isfile(os.path.join(d, "canonical.json")):
|
||||
continue # not a data-warm canonical (e.g. keycloak/traefik reconciler dir, alerts/)
|
||||
# drop the retained warm-<recipe> volumes, then the snapshot/registry dir
|
||||
for vol in warmsnap.stack_volumes(canonical_domain(name)):
|
||||
# drop the retained warm-<ns> volumes, then the snapshot/registry dir
|
||||
for vol in warmsnap.stack_volumes(warm.stable_domain(name)):
|
||||
subprocess.run(["docker", "volume", "rm", vol], capture_output=True, text=True)
|
||||
shutil.rmtree(d, ignore_errors=True)
|
||||
pruned.append(name)
|
||||
@ -176,5 +208,7 @@ def seed_canonical(recipe: str, version: str, commit: str | None = None) -> dict
|
||||
healthy first, then undeploys before calling this (WC3: snapshot while undeployed). The retained
|
||||
volume IS the canonical. Returns the registry record."""
|
||||
rec = write_registry(recipe, version=version, commit=commit, status="idle")
|
||||
warmsnap.snapshot(recipe, canonical_domain(recipe), commit=commit, version=version)
|
||||
warmsnap.snapshot(
|
||||
canonical_slot(recipe), canonical_domain(recipe), commit=commit, version=version
|
||||
)
|
||||
return rec
|
||||
|
||||
@ -1,8 +1,8 @@
|
||||
"""Known-good snapshot/restore of an app's data volumes (Phase 2w / WC3).
|
||||
|
||||
A snapshot is a **raw copy of every docker volume belonging to an app's stack, taken while the app is
|
||||
UNDEPLOYED** (nothing is writing → consistent). Stored under `/var/lib/ci-warm/<recipe>/` as one
|
||||
last-known-good per app, replaced atomically. Restore clears each volume and untars it back.
|
||||
UNDEPLOYED** (nothing is writing → consistent). Stored under `/var/lib/ci-warm/<slot>/` as one
|
||||
last-known-good per slot, replaced atomically. Restore clears each volume and untars it back.
|
||||
|
||||
Used by:
|
||||
- WC1.1 — snapshot keycloak's data volume BEFORE an auto-upgrade; restore on health-gate rollback
|
||||
@ -12,12 +12,25 @@ Used by:
|
||||
Warm snapshots are **cache, excluded from the D8 reproducibility closure** (WC8) — re-seeded by cold
|
||||
runs, not restored on a VM rebuild.
|
||||
|
||||
Layout (atomic dir swap of the `snapshot/` subdir; one last-good per app). Sibling per-app state
|
||||
(e.g. the reconciler's `last_good`) lives in `<recipe>/` and is NOT clobbered by the swap:
|
||||
$CCCI_WARM_ROOT/<recipe>/
|
||||
SLOT, not recipe (F-redfix-4). Warm state belongs to a **deployed stack**, not to a recipe name. A
|
||||
recipe can own TWO warm stacks at once — keycloak is both the live-warm shared OIDC provider
|
||||
(`warm-keycloak…`) and a data-warm canonical (`warm-canon-keycloak…`) — and they must never share a
|
||||
snapshot slot: `snapshot()` atomically REPLACES the slot, so a shared slot means each deployment
|
||||
destroys the other's known-good, and `restore()` (which requires every recorded volume to exist in
|
||||
the target stack) then raises. Callers therefore pass a **slot key**, not a bare recipe:
|
||||
- live-warm reconciler → `live_slot(recipe)` (`<recipe>/`)
|
||||
- data-warm canonical → `canonical.canonical_slot(r)` (`<recipe>/`, or `canon-<recipe>/` when the
|
||||
recipe is also a live-warm provider)
|
||||
Every slot key `<ns>` maps 1:1 to the stack at `warm.stable_domain(<ns>)`, so distinct slot ⇔
|
||||
distinct stack. `canonical.py` derives its domain and its slot from ONE namespace so the two cannot
|
||||
drift apart.
|
||||
|
||||
Layout (atomic dir swap of the `snapshot/` subdir; one last-good per slot). Sibling per-slot state
|
||||
(e.g. the reconciler's `last_good`) lives in `<slot>/` and is NOT clobbered by the swap:
|
||||
$CCCI_WARM_ROOT/<slot>/
|
||||
last_good # (reconciler) the version known healthy — survives snapshot swaps
|
||||
snapshot/
|
||||
meta.json # {recipe, domain, commit, version, ts, volumes:[...]}
|
||||
meta.json # {slot, domain, commit, version, ts, volumes:[...]}
|
||||
volumes/<volname>.tar # raw tar of the volume root, one per stack volume
|
||||
|
||||
Implementation note: volumes are tarred from their host mountpoint
|
||||
@ -46,38 +59,46 @@ def warm_root() -> str:
|
||||
return os.environ.get("CCCI_WARM_ROOT", DEFAULT_WARM_ROOT)
|
||||
|
||||
|
||||
def app_dir(recipe: str) -> str:
|
||||
return os.path.join(warm_root(), recipe)
|
||||
def live_slot(recipe: str) -> str:
|
||||
"""Warm-state slot of the LIVE-warm reconciler's deployment (the stack at `warm-<recipe>`).
|
||||
|
||||
The canonical's slot is `canonical.canonical_slot()`, which diverges from this one exactly when
|
||||
the recipe is also a live-warm provider (F-redfix-4)."""
|
||||
return recipe
|
||||
|
||||
|
||||
def snap_dir(recipe: str) -> str:
|
||||
def app_dir(slot: str) -> str:
|
||||
return os.path.join(warm_root(), slot)
|
||||
|
||||
|
||||
def snap_dir(slot: str) -> str:
|
||||
"""The snapshot subdir — atomically swapped on update. Kept SEPARATE from app_dir so sibling
|
||||
per-app state (the reconciler's last_good) survives a snapshot swap."""
|
||||
return os.path.join(app_dir(recipe), "snapshot")
|
||||
per-slot state (the reconciler's last_good) survives a snapshot swap."""
|
||||
return os.path.join(app_dir(slot), "snapshot")
|
||||
|
||||
|
||||
def meta_path(recipe: str) -> str:
|
||||
return os.path.join(snap_dir(recipe), "meta.json")
|
||||
def meta_path(slot: str) -> str:
|
||||
return os.path.join(snap_dir(slot), "meta.json")
|
||||
|
||||
|
||||
def volumes_dir(recipe: str) -> str:
|
||||
return os.path.join(snap_dir(recipe), "volumes")
|
||||
def volumes_dir(slot: str) -> str:
|
||||
return os.path.join(snap_dir(slot), "volumes")
|
||||
|
||||
|
||||
def has_snapshot(recipe: str) -> bool:
|
||||
def has_snapshot(slot: str) -> bool:
|
||||
"""True iff a complete last-good snapshot (meta + at least its declared volume tars) exists."""
|
||||
meta = read_meta(recipe)
|
||||
meta = read_meta(slot)
|
||||
if not meta:
|
||||
return False
|
||||
for v in meta.get("volumes", []):
|
||||
if not os.path.isfile(os.path.join(volumes_dir(recipe), f"{v}.tar")):
|
||||
if not os.path.isfile(os.path.join(volumes_dir(slot), f"{v}.tar")):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def read_meta(recipe: str) -> dict | None:
|
||||
def read_meta(slot: str) -> dict | None:
|
||||
try:
|
||||
with open(meta_path(recipe)) as f:
|
||||
with open(meta_path(slot)) as f:
|
||||
return json.load(f)
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
@ -113,18 +134,35 @@ def _assert_undeployed(domain: str) -> None:
|
||||
)
|
||||
|
||||
|
||||
def snapshot(
|
||||
recipe: str, domain: str, commit: str | None = None, version: str | None = None
|
||||
) -> dict:
|
||||
"""Take a last-known-good snapshot of every data volume of <domain>'s stack. The app MUST be
|
||||
undeployed. Atomically replaces the prior last-good. Returns the written meta dict."""
|
||||
def _assert_slot_not_foreign(slot: str, domain: str) -> None:
|
||||
"""Refuse to touch a slot that already holds another DOMAIN's known-good (F-redfix-4).
|
||||
|
||||
Defence in depth behind the slot separation itself: naming-scheme-independent, it catches any
|
||||
future caller that pairs a slot with the wrong stack. Fails BEFORE the destructive swap rather
|
||||
than at the next restore. A slot with no snapshot yet is free to claim."""
|
||||
meta = read_meta(slot)
|
||||
if meta and meta.get("domain") and meta["domain"] != domain:
|
||||
raise SnapshotError(
|
||||
f"warm slot {slot!r} holds the known-good of {meta['domain']!r}, not {domain!r}: "
|
||||
"refusing to clobber another deployment's snapshot (F-redfix-4). "
|
||||
"Live-warm and data-warm deployments of one recipe need distinct slots."
|
||||
)
|
||||
|
||||
|
||||
def snapshot(slot: str, domain: str, commit: str | None = None, version: str | None = None) -> dict:
|
||||
"""Take a last-known-good snapshot of every data volume of <domain>'s stack into <slot>. The app
|
||||
MUST be undeployed. Atomically replaces the prior last-good in that slot. Returns the meta dict.
|
||||
|
||||
`slot` must be the warm-state slot OWNING `domain` (see module docstring) — passing another
|
||||
deployment's slot would destroy its known-good, so the pairing is asserted below."""
|
||||
_assert_slot_not_foreign(slot, domain)
|
||||
_assert_undeployed(domain)
|
||||
volumes = stack_volumes(domain)
|
||||
if not volumes:
|
||||
raise SnapshotError(f"no volumes found for {domain} — nothing to snapshot")
|
||||
|
||||
os.makedirs(app_dir(recipe), exist_ok=True)
|
||||
staging = os.path.join(app_dir(recipe), ".snapshot.staging")
|
||||
os.makedirs(app_dir(slot), exist_ok=True)
|
||||
staging = os.path.join(app_dir(slot), ".snapshot.staging")
|
||||
shutil.rmtree(staging, ignore_errors=True)
|
||||
os.makedirs(os.path.join(staging, "volumes"), exist_ok=True)
|
||||
|
||||
@ -138,7 +176,7 @@ def snapshot(
|
||||
raise SnapshotError(f"tar of volume {vol} failed: {r.stderr.strip()}")
|
||||
|
||||
meta = {
|
||||
"recipe": recipe,
|
||||
"slot": slot,
|
||||
"domain": domain,
|
||||
"commit": commit,
|
||||
"version": version,
|
||||
@ -149,8 +187,8 @@ def snapshot(
|
||||
json.dump(meta, f)
|
||||
|
||||
# Atomic-ish swap of the snapshot subdir only (sibling state like last_good is untouched).
|
||||
target = snap_dir(recipe)
|
||||
old = os.path.join(app_dir(recipe), ".snapshot.old")
|
||||
target = snap_dir(slot)
|
||||
old = os.path.join(app_dir(slot), ".snapshot.old")
|
||||
shutil.rmtree(old, ignore_errors=True)
|
||||
if os.path.exists(target):
|
||||
os.rename(target, old)
|
||||
@ -159,17 +197,19 @@ def snapshot(
|
||||
return meta
|
||||
|
||||
|
||||
def restore(recipe: str, domain: str) -> dict:
|
||||
"""Restore the last-known-good snapshot into <domain>'s stack volumes. The app MUST be undeployed.
|
||||
Clears each volume then untars the snapshot back. Returns the snapshot meta. Raises if no
|
||||
snapshot exists or a snapshot volume is missing from the current stack."""
|
||||
def restore(slot: str, domain: str) -> dict:
|
||||
"""Restore <slot>'s last-known-good snapshot into <domain>'s stack volumes. The app MUST be
|
||||
undeployed. Clears each volume then untars the snapshot back. Returns the snapshot meta. Raises
|
||||
if no snapshot exists, the slot belongs to another domain, or a snapshot volume is missing from
|
||||
the current stack."""
|
||||
_assert_undeployed(domain)
|
||||
meta = read_meta(recipe)
|
||||
if not meta or not has_snapshot(recipe):
|
||||
raise SnapshotError(f"no complete snapshot for {recipe} to restore")
|
||||
meta = read_meta(slot)
|
||||
if not meta or not has_snapshot(slot):
|
||||
raise SnapshotError(f"no complete snapshot in slot {slot!r} to restore")
|
||||
_assert_slot_not_foreign(slot, domain)
|
||||
current = set(stack_volumes(domain))
|
||||
for vol in meta.get("volumes", []):
|
||||
tar_path = os.path.join(volumes_dir(recipe), f"{vol}.tar")
|
||||
tar_path = os.path.join(volumes_dir(slot), f"{vol}.tar")
|
||||
if vol not in current:
|
||||
raise SnapshotError(
|
||||
f"snapshot volume {vol} absent from current stack {sorted(current)}"
|
||||
|
||||
@ -893,7 +893,7 @@ def run_quick(
|
||||
)
|
||||
abra.undeploy(domain)
|
||||
_wait_undeployed(domain)
|
||||
warmsnap.restore(recipe, domain)
|
||||
warmsnap.restore(canonical.canonical_slot(recipe), domain)
|
||||
# reset recorded version to the known-good (the failed upgrade set TYPE to the broken
|
||||
# PR commit) so the idle canonical's .env agrees with the registry + re-warms cleanly.
|
||||
if reg.get("version"):
|
||||
|
||||
@ -384,7 +384,9 @@ def wait_undeployed(domain: str, timeout: int = 120) -> None:
|
||||
|
||||
|
||||
def last_good_path(recipe: str) -> str:
|
||||
return os.path.join(warmsnap.app_dir(recipe), "last_good")
|
||||
# live_slot, not the bare recipe: a recipe that is also a data-warm canonical keeps its canonical
|
||||
# state in a DIFFERENT slot, so the reconciler's last_good is never clobbered (F-redfix-4).
|
||||
return os.path.join(warmsnap.app_dir(warmsnap.live_slot(recipe)), "last_good")
|
||||
|
||||
|
||||
def read_last_good(recipe: str) -> str | None:
|
||||
@ -396,7 +398,7 @@ def read_last_good(recipe: str) -> str | None:
|
||||
|
||||
|
||||
def write_last_good(recipe: str, version: str) -> None:
|
||||
os.makedirs(warmsnap.app_dir(recipe), exist_ok=True)
|
||||
os.makedirs(warmsnap.app_dir(warmsnap.live_slot(recipe)), exist_ok=True)
|
||||
tmp = last_good_path(recipe) + ".tmp"
|
||||
with open(tmp, "w") as f:
|
||||
f.write(version)
|
||||
@ -509,7 +511,7 @@ def reconcile(app: str) -> str:
|
||||
if stateful:
|
||||
abra.undeploy(domain)
|
||||
wait_undeployed(domain)
|
||||
warmsnap.snapshot(recipe, domain, version=last_good)
|
||||
warmsnap.snapshot(warmsnap.live_slot(recipe), domain, version=last_good)
|
||||
# snapshot requires undeployed; now bring up latest.
|
||||
# A broken "latest" can fail in two ways: deploy_version raises (abra converge times out on a
|
||||
# crash-looping task) OR it deploys but never becomes healthy. BOTH must roll back, so treat a
|
||||
@ -531,7 +533,7 @@ def reconcile(app: str) -> str:
|
||||
if stateful:
|
||||
abra.undeploy(domain)
|
||||
wait_undeployed(domain)
|
||||
warmsnap.restore(recipe, domain)
|
||||
warmsnap.restore(warmsnap.live_slot(recipe), domain)
|
||||
deploy_version(recipe, domain, last_good, dt)
|
||||
recovered = wait_healthy(spec)
|
||||
write_alert(
|
||||
|
||||
@ -36,10 +36,7 @@ RECIPE_DIR="${HOME}/.abra/recipes/${RECIPE}"
|
||||
}
|
||||
TOKEN="$(tr -d '[:space:]' <"${TOKEN_FILE}")"
|
||||
API="https://${GITEA_HOST}/api/v1"
|
||||
# Push over SSH (git@:2222, key in ~/.ssh) so no credential is ever written into the recipe
|
||||
# clone's .git/config — that config gets copytree'd into the world-readable per-run tree
|
||||
# (CCCI_SKIP_FETCH staging), which previously leaked the token. API still uses the token below.
|
||||
MIRROR_PUSH="git@${GITEA_HOST}:${NAMESPACE}/${RECIPE}.git"
|
||||
MIRROR_PUSH="https://oauth2:${TOKEN}@${GITEA_HOST}/${NAMESPACE}/${RECIPE}.git"
|
||||
auth=(-H "Authorization: token ${TOKEN}")
|
||||
|
||||
# Ensure the recipe is cloned (abra recipe fetch clones from the catalogue → upstream).
|
||||
|
||||
2
secrets
2
secrets
Submodule secrets updated: 2ce5f86c02...cdd5e0ad25
@ -7,10 +7,14 @@ DEPLOY_TIMEOUT = (
|
||||
)
|
||||
HTTP_TIMEOUT = 900
|
||||
|
||||
# canon §2.B EXCEPTION (recorded in DECISIONS): keycloak is NOT a data-warm canonical. It is the
|
||||
# project's LIVE-WARM OIDC dep provider — an always-on shared service at the SAME stable domain a
|
||||
# data-warm canonical would use (warm-keycloak.ci.commoninternet.net). Enrolling it would make the
|
||||
# sweep's promote deploy/teardown collide with the live provider that lasuite-*/drone depend on for
|
||||
# SSO. keycloak is instead kept current by the sweep's roll_warm_infra step (the health-gated
|
||||
# warm/infra reconciler, WC1.1) — so it never lacks coverage. WARM_CANONICAL stays False.
|
||||
WARM_CANONICAL = False
|
||||
# phase redfix: keycloak IS now a data-warm canonical. The original canon §2.B exception de-enrolled
|
||||
# it because its canonical would have used the SAME domain as the live-warm OIDC provider
|
||||
# (warm-keycloak.ci.commoninternet.net), so the sweep's promote deploy/teardown would collide with the
|
||||
# live service lasuite-*/drone depend on. The canonical is now namespaced apart from the live provider
|
||||
# on BOTH axes it shares with it — `canonical.canonical_ns()` gives any recipe in `warm.WARM_DOMAINS`
|
||||
# (keycloak) a `canon-<recipe>` namespace, from which BOTH its domain/stack (`warm-canon-keycloak…`)
|
||||
# and its warm-state slot (`/var/lib/ci-warm/canon-keycloak/`) derive. Domain separation alone was NOT
|
||||
# enough: warm state was keyed by bare recipe, so both deployments shared one snapshot slot and each
|
||||
# destroyed the other's known-good (F-redfix-4). keycloak therefore gets full data-warm canonical
|
||||
# coverage (a real promote on its latest release) without risking the live OIDC service.
|
||||
WARM_CANONICAL = True
|
||||
|
||||
@ -19,7 +19,14 @@ import _mumble_proto # noqa: E402
|
||||
|
||||
|
||||
def test_handshake_completes_with_channel_presence(live_app):
|
||||
r = _mumble_proto.retry_handshake(attempts=12, interval=5.0)
|
||||
# Readiness budget: 36×5s = 180s. The TCP READY_PROBE (recipe_meta) only proves port 64738 is
|
||||
# LISTENING; the murmur control channel needs additional warmup before it completes a full
|
||||
# TLS+Version+ServerSync handshake. Under concurrent node load (the canon sweep) that warmup
|
||||
# exceeded the old 60s budget and flaked this test RED, while it is reliably GREEN in isolation
|
||||
# (phase redfix M1: 3× isolation green, 0 isolation reds). The longer budget absorbs the
|
||||
# load-induced readiness delay WITHOUT weakening the assertion — a genuinely non-responsive
|
||||
# server still exhausts all retries and FAILs (the asserts below are unchanged).
|
||||
r = _mumble_proto.retry_handshake(attempts=36, interval=5.0)
|
||||
|
||||
assert r["tls_connect"], f"TLS connection to 127.0.0.1:64738 failed — {r.get('error')}"
|
||||
assert r["server_version"] is not None, "server did not send a Version message"
|
||||
|
||||
@ -12,7 +12,7 @@ import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner"))
|
||||
from harness import canonical, warm # noqa: E402
|
||||
from harness import canonical, warm, warmsnap # noqa: E402
|
||||
from harness import meta as harness_meta # noqa: E402
|
||||
|
||||
|
||||
@ -96,3 +96,78 @@ def test_prune_stale_drops_deenrolled_only(tmp_path, monkeypatch):
|
||||
assert (tmp_path / "keepme").exists()
|
||||
assert (tmp_path / "keycloak").exists() # no canonical.json → not a canonical → kept
|
||||
assert (tmp_path / "alerts").exists()
|
||||
|
||||
|
||||
# ------------------------------------------- F-redfix-4: canonical ns/slot vs live-warm provider
|
||||
|
||||
|
||||
def test_canonical_ns_and_domain_for_live_warm_provider():
|
||||
# keycloak is a live-warm provider → its canonical is namespaced apart on BOTH axes.
|
||||
assert canonical.canonical_ns("keycloak") == "canon-keycloak"
|
||||
assert canonical.canonical_domain("keycloak") == "warm-canon-keycloak.ci.commoninternet.net"
|
||||
assert canonical.canonical_slot("keycloak") == "canon-keycloak"
|
||||
# ...and an ordinary recipe is untouched (zero blast radius on existing canonicals).
|
||||
assert canonical.canonical_ns("cryptpad") == "cryptpad"
|
||||
assert canonical.canonical_domain("cryptpad") == "warm-cryptpad.ci.commoninternet.net"
|
||||
assert canonical.canonical_slot("cryptpad") == "cryptpad"
|
||||
|
||||
|
||||
def test_live_and_canonical_slots_are_disjoint():
|
||||
"""F-redfix-4: for EVERY live-warm provider, the reconciler's slot and the canonical's slot must
|
||||
differ — they are two deployments and each `snapshot()` atomically replaces its slot."""
|
||||
for recipe in warm.WARM_DOMAINS:
|
||||
assert canonical.canonical_slot(recipe) != warmsnap.live_slot(recipe)
|
||||
# every non-provider recipe keeps one slot (there is only one deployment)
|
||||
assert canonical.canonical_slot("cryptpad") == warmsnap.live_slot("cryptpad")
|
||||
|
||||
|
||||
def test_slot_maps_1to1_to_its_stack():
|
||||
# The invariant warmsnap relies on: distinct slot ⇔ distinct stack (domain = warm-<ns>).
|
||||
for recipe in warm.WARM_DOMAINS:
|
||||
assert warm.stable_domain(canonical.canonical_slot(recipe)) == canonical.canonical_domain(
|
||||
recipe
|
||||
)
|
||||
assert warm.stable_domain(warmsnap.live_slot(recipe)) == warm.WARM_DOMAINS[recipe]
|
||||
|
||||
|
||||
def test_registry_path_of_live_warm_provider_is_not_the_reconciler_dir(tmp_path, monkeypatch):
|
||||
"""The canonical registry must NOT land in `<recipe>/`, where the reconciler keeps last_good."""
|
||||
monkeypatch.setenv("CCCI_WARM_ROOT", str(tmp_path))
|
||||
assert canonical.registry_path("keycloak") == str(
|
||||
tmp_path / "canon-keycloak" / "canonical.json"
|
||||
)
|
||||
canonical.write_registry("keycloak", version="10.7.1+26.6.2", commit=None, status="idle")
|
||||
assert (tmp_path / "canon-keycloak" / "canonical.json").is_file()
|
||||
assert not (tmp_path / "keycloak").exists() # reconciler dir untouched
|
||||
|
||||
|
||||
def test_prune_stale_keeps_enrolled_provider_canonical_and_reconciler(tmp_path, monkeypatch):
|
||||
"""Enrolled keycloak: `canon-keycloak/` is its canonical slot (keep), `keycloak/` is the
|
||||
reconciler's (keep — no canonical.json)."""
|
||||
monkeypatch.setenv("CCCI_WARM_ROOT", str(tmp_path))
|
||||
monkeypatch.setattr(canonical, "enrolled_recipes", lambda: ["keycloak"])
|
||||
monkeypatch.setattr(canonical.warmsnap, "stack_volumes", lambda d: [])
|
||||
(tmp_path / "canon-keycloak").mkdir()
|
||||
(tmp_path / "canon-keycloak" / "canonical.json").write_text('{"recipe":"keycloak"}')
|
||||
(tmp_path / "keycloak").mkdir()
|
||||
(tmp_path / "keycloak" / "last_good").write_text("10.7.1+26.6.2")
|
||||
assert canonical.prune_stale() == []
|
||||
assert (tmp_path / "canon-keycloak").exists()
|
||||
assert (tmp_path / "keycloak" / "last_good").read_text() == "10.7.1+26.6.2"
|
||||
|
||||
|
||||
def test_prune_stale_deenrolled_provider_spares_reconciler_last_good(tmp_path, monkeypatch):
|
||||
"""F-redfix-4 consequence 4: de-enrolling keycloak must drop `canon-keycloak/` ONLY — never the
|
||||
reconciler's `keycloak/last_good`. It also targets the canon stack's volumes, not the live ones."""
|
||||
monkeypatch.setenv("CCCI_WARM_ROOT", str(tmp_path))
|
||||
monkeypatch.setattr(canonical, "enrolled_recipes", lambda: []) # de-enrolled
|
||||
seen = []
|
||||
monkeypatch.setattr(canonical.warmsnap, "stack_volumes", lambda d: seen.append(d) or [])
|
||||
(tmp_path / "canon-keycloak").mkdir()
|
||||
(tmp_path / "canon-keycloak" / "canonical.json").write_text('{"recipe":"keycloak"}')
|
||||
(tmp_path / "keycloak").mkdir()
|
||||
(tmp_path / "keycloak" / "last_good").write_text("10.7.1+26.6.2")
|
||||
assert canonical.prune_stale() == ["canon-keycloak"]
|
||||
assert not (tmp_path / "canon-keycloak").exists()
|
||||
assert (tmp_path / "keycloak" / "last_good").read_text() == "10.7.1+26.6.2"
|
||||
assert seen == ["warm-canon-keycloak.ci.commoninternet.net"] # never the LIVE provider's stack
|
||||
|
||||
@ -68,3 +68,62 @@ def test_has_snapshot_incomplete_missing_tar(monkeypatch, tmp_path):
|
||||
(snapdir / "meta.json").write_text(json.dumps({"recipe": "keycloak", "volumes": ["a", "b"]}))
|
||||
(snapdir / "volumes" / "a.tar").write_bytes(b"fake")
|
||||
assert warmsnap.has_snapshot("keycloak") is False
|
||||
|
||||
|
||||
# --------------------------------------------------------------- F-redfix-4: slot ≠ recipe
|
||||
|
||||
|
||||
def test_live_slot_is_the_recipe():
|
||||
assert warmsnap.live_slot("keycloak") == "keycloak"
|
||||
|
||||
|
||||
def test_snapshot_refuses_to_clobber_another_domains_slot(monkeypatch, tmp_path):
|
||||
"""F-redfix-4 regression: a slot holding domain A's known-good must not be overwritten by a
|
||||
snapshot of domain B. Before the fix this silently destroyed the other deployment's snapshot."""
|
||||
monkeypatch.setenv("CCCI_WARM_ROOT", str(tmp_path))
|
||||
snapdir = tmp_path / "keycloak" / "snapshot"
|
||||
(snapdir / "volumes").mkdir(parents=True)
|
||||
(snapdir / "meta.json").write_text(
|
||||
json.dumps({"slot": "keycloak", "domain": "warm-keycloak.ci.x", "volumes": []})
|
||||
)
|
||||
# Never reaches docker: the foreign-slot guard fires before _assert_undeployed.
|
||||
try:
|
||||
warmsnap.snapshot("keycloak", "warm-canon-keycloak.ci.x")
|
||||
except warmsnap.SnapshotError as e:
|
||||
assert "warm-keycloak.ci.x" in str(e) and "refusing to clobber" in str(e)
|
||||
else:
|
||||
raise AssertionError("snapshot() overwrote a foreign slot")
|
||||
# the original known-good is intact
|
||||
assert warmsnap.read_meta("keycloak")["domain"] == "warm-keycloak.ci.x"
|
||||
|
||||
|
||||
def test_snapshot_may_reclaim_its_own_slot(monkeypatch, tmp_path):
|
||||
# Same domain → not foreign → the guard must not fire (it would break every re-snapshot).
|
||||
monkeypatch.setenv("CCCI_WARM_ROOT", str(tmp_path))
|
||||
snapdir = tmp_path / "keycloak" / "snapshot"
|
||||
(snapdir / "volumes").mkdir(parents=True)
|
||||
(snapdir / "meta.json").write_text(
|
||||
json.dumps({"slot": "keycloak", "domain": "warm-keycloak.ci.x", "volumes": []})
|
||||
)
|
||||
warmsnap._assert_slot_not_foreign("keycloak", "warm-keycloak.ci.x") # no raise
|
||||
warmsnap._assert_slot_not_foreign("fresh-slot", "anything.ci.x") # empty slot is free to claim
|
||||
|
||||
|
||||
def test_restore_refuses_a_foreign_slot(monkeypatch, tmp_path):
|
||||
"""restore() must reject a slot recorded against another domain BEFORE touching volumes."""
|
||||
monkeypatch.setenv("CCCI_WARM_ROOT", str(tmp_path))
|
||||
monkeypatch.setattr(warmsnap, "_assert_undeployed", lambda d: None)
|
||||
_write_snapshot(tmp_path, "keycloak", ["warm-keycloak_ci_x_mariadb"])
|
||||
snapdir = tmp_path / "keycloak" / "snapshot"
|
||||
meta = json.loads((snapdir / "meta.json").read_text())
|
||||
meta["domain"] = "warm-keycloak.ci.x"
|
||||
(snapdir / "meta.json").write_text(json.dumps(meta))
|
||||
called = []
|
||||
monkeypatch.setattr(warmsnap, "stack_volumes", lambda d: called.append(d) or [])
|
||||
try:
|
||||
warmsnap.restore("keycloak", "warm-canon-keycloak.ci.x")
|
||||
except warmsnap.SnapshotError as e:
|
||||
assert "refusing to clobber" in str(e)
|
||||
else:
|
||||
raise AssertionError("restore() accepted a foreign slot")
|
||||
assert called == [], "restore() must fail before inspecting the target stack's volumes"
|
||||
|
||||
Reference in New Issue
Block a user