Since NixOS 25.05, scripted networking installs the default route from the
gateway interface's network-addresses-<iface>.service, matching the interface
via defaultGateway.interface or by subnet inclusion. Hetzner's off-subnet
point-to-point gateway (91.98.47.73/32 on eth0, gw 172.31.1.1) matched
neither with the bare-string form, so the 26.05 switch on 2026-08-03 left the
host with no default route and off the network (recovered via Hetzner rescue:
grubenv default back to the 24.11 generation).
With an explicit interface, the module installs both the gateway host route
and the default route from eth0's own unit:
ip -4 route replace 172.31.1.1 proto static dev eth0
ip -4 route replace default proto static dev eth0 via 172.31.1.1
Verified by nix eval of systemd.services.network-addresses-eth0.script.
nixpkgs: 50ab793 -> 531670d (nixos-26.05, was nixos-24.11 EOL)
sops-nix: 77c423a -> f140661 (master, buildGo125Module now builds on 26.05)
3-release channel jump (24.11 -> 25.05 -> 25.11 -> 26.05) to match the
orchestrator host and get CVE patches. 24.11 was EOL (2025-06-30).
Notable 26.05 changes reviewed: systemd Stage 1 boot (default, no config
change needed), dbus-broker default, bash nixos-rebuild removed (Python
rewrite mandatory; git present on host), MySQL 8.0 removed (not used as a
NixOS module). No system.rebuild/fileSystems/boot.initrd options in the
host config, so no config edits required.
The token embedded in the recipe clone's gitea remote was copytree'd into the
world-readable per-run tree. SSH push keys the auth to ~/.ssh, so no secret is
written into any .git/config. API calls still use the token.
Terminal state re-verified cold. Corrected a near-miss from this wake: my first
`^## .*PASS` grep did not match the M1 verdict, which is a `###` heading at
REVIEW-redfix.md:22 — confirmed present by reading it, rather than reporting M1
PASS as missing. Both `## VETO` hits read in full: one CLEARED veto + its
clearance record, no standing veto. DONE @00:18Z, M1+M2 PASS, HEAD==origin/main,
no inbox. Untracked main.go re-checked by hash, identical to the file already
adjudicated (STATUS:710, A-redfix-3) — left untouched by design.
Remaining items (B-redfix-8, A-redfix-1) are operator-scope: history rewrite
(--force, forbidden) + credential rotation. Loop STOPPED.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWSJEPYEMx9LZhPwyQCmhP
HEAD==origin/main==89af75b is my own wake-#64 journal: neither the Adversary nor the tree has moved
since I last stopped, so there was no handoff to answer. Terminal state re-verified cold anyway:
DONE @00:18Z, M1+M2 PASS, both '## VETO' headings read individually and confirmed historical (:553
annotated CLEARED, :649 is the clearance record) rather than inferred from a grep count. No inbox.
Untracked main.go left in place per the standing adjudication at STATUS:710 / A-redfix-3.
Declined to re-run M1/M2 on an unmoved tree — it would mint a fresh timestamp without new evidence.
Remaining items are operator-scope: B-redfix-8 (history rewrite, --force forbidden) and A-redfix-1
(credential rotation). Loop STOPPED.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015yZh7A6Zcac5LQ3tqqtb9W
Terminal state re-verified cold: DONE @00:18Z, M1+M2 PASS stand, no standing VETO (both '## VETO' hits
read, not counted: one CLEARED + its clearance record), all findings CLOSED, HEAD==origin/main, no inbox.
Live host re-probed: /etc/cc-ci HEAD==d11f8f5 unmoved since #54, 07fc6d4 not an ancestor, object present
-> B-redfix-5 negative is real, not vacuous. Declined to re-run M1/M2 on an unmoved tree.
Prefix is 'chore(' not 'review(' ON PURPOSE: no verdict landed this wake. The Builder's #63 journal shows
my last review( commit fired a watchdog handoff ping that resolved to a no-op; a third false ping would
degrade 'review(' as a signal. Loop STOPPED.
Adversary (a22c384) confirmed both findings but counted 3061 blobs vs my 3099. Root-caused:
(a) later commits add blobs (host now 3104); (b) --batch-all-objects counts 49 unreachable
objects rev-list omits (reachable-only = 3026); (c) 'git clone /local/path' hardlinks the whole
object store, so local clones inherit unreachable objects while remote clones do not -- hence
the Adversary's lower, and for a mirror question more apt, number.
All four scans agree on password=2 and token=0. STATUS now asserts those invariants and a
'wc -l > 100' sanity floor instead of an exact total, so the documented repro cannot misfire.
Operator-scope, no gate impact, no VETO; DONE stands.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018XE43k4DaeMXHMK51wBUu2
Closes the gap in Adversary wake #57's break-it probe, which covered dashboard/reports/Drone
logs but not the public git mirror -- the surface B-redfix-8 is about.
- password sha16 3fcea78925015fc9 -> 2 blobs, 2 commits, both ancestors of origin/main.
Scrubbing 2ad38f5 alone would leave e64d8e7 serving the live credential. STATUS corrected.
- oauth2 token sha16 9c44a1aea2ecb389 -> 0/3099 blobs. Filesystem-only exposure.
- Probe hygiene: awk is absent here; awk-based blob lists give a vacuous 0. Positive
control (password must return 2) is now mandatory and documented.
Operator-scope, no gate impact, no VETO; DONE stands.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018XE43k4DaeMXHMK51wBUu2
Adversary wake #55 conceded my insteadOf falsification but corrected me twice; verified both first-hand:
1. "Production CI does not regenerate / manual-* = hand-run" WRONG. run_id()=manual-<pid> for any non-Drone
run (run_recipe_ci.py:318-319); nightly-sweep runs run_recipe_ci.py outside Drone with CCCI_SKIP_FETCH=1
(nightly_sweep.py:88). Sweep regenerates the exposed copies WEEKLY (freshest 07-05 03:37-59 = sweep fire).
2. Census missed a 2nd credential (grep keyed on autonomic-bot: cannot see oauth2:). Full per-file census:
78 files carry the password, 117 a live oauth2 token, 62 both, 133 distinct under /var/lib. Token is LIVE
+ PUSH-capable (api/v1/user->200 autonomic-bot/64) and is what recipe-mirror-sync.sh:39 pushes with —
falsifies my own B-redfix-8 "small blast radius" note.
STATUS steps 3-4 rewritten (two creds, weekly regen, combined remedy + chmod 0750); B-redfix-8/9 corrected.
DONE stands; no VETO; no DoD item touched; rotation of BOTH secrets remains operator-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y9GyBPF1EgTTh6277Xjj7k
Adversary wake #53/#54 reaffirm DONE + no VETO (no gate impact), but both correct STATUS text (Builder-owned):
1. A-redfix-1 "sole copy" withdrawn: 78 world-readable cred-bearing .git/config, sentinel 3fcea78925015fc9.
Exposure confirmed; Adversary's /root/.gitconfig insteadOf root cause FALSIFIED three ways (clone does not
persist insteadOf rewrites — tested, git 2.47.2; /etc/cc-ci config predates .gitconfig by 2wk; live
fetch_recipe uses a non-persisted http.extraHeader token since 9b33fdf, so 0/215 numeric runs carry it).
Real generator: CCCI_SKIP_FETCH copytree of credentialed /root/.abra/recipes (0700) into 0755 run tree.
STATUS steps 3-4 rewritten; filed B-redfix-9 (deferred). Falsification sent via ADVERSARY-INBOX.md.
2. Sweep mechanism: runs deployed /etc/cc-ci (main @ d11f8f5), not origin/main. Verified first-hand.
DONE stands; no VETO; no DoD item touched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y9GyBPF1EgTTh6277Xjj7k
No standing VETO (2x '## VETO' headings are historical: one annotated CLEARED, one is the clearing
record). STATUS:481 CLAIMED is the superseded historical M2 claim. No action available; loop stopped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y9GyBPF1EgTTh6277Xjj7k
Stop condition re-checked from a cold read: STATUS ## DONE, M1+M2 fresh PASS, no standing
VETO (line 553 header is annotated CLEARED), no inbox either direction, Adversary loop
closed out at #49. Loop stays stopped; no DoD item re-run.
Sentinel re-probed this wake: GITEA_PASSWORD still hashes to 3fcea78925015fc9 (unrotated).
A-redfix-1 / B-redfix-8 remain OPEN/HIGH, operator-rotation-only, gating nothing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TRRkTGdWKWoxddF6QXhWfd
Adversary confirmed all three claims and failed to refute "strip userinfo is safe" three ways:
no push through /etc/cc-ci origin; secrets submodule anonymously fetchable so submodule update
survives; nothing auto-pulls the checkout (/root/.git-credentials inert, no helper wired).
Folded those three into STATUS step 3 so the operator sees why the command is safe. The submodule
angle was an inference on my part, not a probe -- the Adversary actually tested it. Recorded in
JOURNAL.
Incidental (not a finding): cc-ci-secrets.git is public but SOPS-encrypted ciphertext.
STATUS/JOURNAL text only. No code. ## DONE stands; M1+M2 PASS stand; no gate reopens.
Adversary wake #45 established A-redfix-1 and B-redfix-8 are the same credential. Re-derived
every claim first-hand before writing it (the A-redfix-2 lesson), incl. value identity:
.git/config userinfo and .testenv GITEA_PASSWORD both hash to 3fcea78925015fc9; control
sha256("")=e3b0c44298fc1c14 rules out the empty-input probe artifact.
Rotation does NOT scrub /etc/cc-ci/.git/config (644 root:root, manual clone per
configuration.nix:7, not nix-generated). Remedy is now a 3-step operator procedure that
strips the userinfo rather than re-embedding the rotated password (re-embedding would
recreate the exposure). Verified the strip is safe: anonymous ls-remote succeeds rc=0.
Blast radius recorded: GITEA_PASSWORD is consumed only by bootstrap-drone-oauth.sh;
recipe-mirror-sync.sh uses an OAuth token. No reason to delay rotation.
STATUS text only. No code. ## DONE stands; M1+M2 PASS stand; no gate reopens.
Independently re-verified every leg against the node before editing (I adopted
A-redfix-2's probes wholesale at 8276ecd and inherited its error; not repeating that).
- Delete both 'no re-trigger' probes: they grep the app name, but the timer is
named nightly-sweep, so they return empty while a timer drives it weekly.
Sound probe greps the callee: git grep warm_reconcile -- runner/ nix/
- Replace 'recovery is manual' / 'heals on reboot' with the weekly DOWN/UP
oscillation, and record that it is SILENT (roll_warm_infra discards the rc;
the raise is upstream of every write_alert).
- Re-anchor the merge precondition: 'slot clean at merge time' -> 'never deploy
07fc6d4'. A git merge executes nothing; b5f2b10 ships enrollment + fix together,
so the precondition is self-maintaining.
- Line refs pinned to b5f2b10 coordinates (were main's, two lines off).
- BACKLOG B-redfix-5: severity corrected, remedy sketch extended to rc propagation.
Not armed and not self-arming: origin/main (the tree the sweep runs) has
WARM_CANONICAL = False. Nothing reopens; DONE stands, no VETO, b5f2b10 unchanged.
Inbox consumed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TkHo7RixQswYvvNnEcNaqW
The DONE block listed F-redfix-1/2/3/4 as CLOSED but never mentioned the three
A-redfix-* findings, leaving them dangling for an operator reading only DONE.
All three are non-VETO and outside the DoD (Adversary wake #43, 1057657).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TkHo7RixQswYvvNnEcNaqW
Both claims re-derived independently before editing; both of mine were wrong.
A-redfix-2: "no code path redeploys it" is FALSE. abra.undeploy() leaves the app
.env, so the next reconcile() takes the fresh-deploy branch (:471-479), which
redeploys and never calls warmsnap. Self-healing is blocked by the absence of a
re-trigger (warm-keycloak.service: Type=oneshot, RemainAfterExit=true, no timer,
no Restart=), so it heals on reboot/nixos-rebuild and RE-WEDGES on the next due
upgrade while the foreign snapshot/meta.json remains. Operator remediation is to
DELETE the foreign snapshot/, not to redeploy keycloak. Failure presents as an
intermittent flake, not a permanent outage.
A-redfix-3: /srv/cc-ci is a symlink to /srv/cc-ci-orch, so the two main.go paths
are the same inode (3254604, links=1). The "present in both clones" premise is
void and proves nothing about the file's origin (still unexplained; inert).
Docs-only. No DoD item, gate, or verdict affected: ## DONE stands, M1 + M2 PASS
stand, merge target b5f2b10 unchanged, no VETO. The B-redfix-5 merge precondition
is unchanged and still correct.
Probed the two operator-facing claims in the Builder's new commits (cbeceea, d5dc547). No inbox, no gate.
A-redfix-2 (LOW-MED) — "The wedge does not self-heal ... no code path redeploys it. Recovery is manual."
Conclusion right, mechanism wrong. abra.undeploy() does not remove the app .env, so the next reconcile()
sees current_version resolvable + is_deployed False and takes the fresh-deploy branch
(warm_reconcile.py:471-479), which redeploys and NEVER calls warmsnap -- the guard is unreachable there.
What actually blocks self-healing is the lack of a re-trigger: warm-keycloak.service is Type=oneshot with
no timer and no Restart=. So it heals on the next reboot / nixos-rebuild switch, then RE-WEDGES on the next
due upgrade because the foreign snapshot/meta.json is still in the slot. The failure therefore presents as
an intermittent flake, and the correct remediation is removing the foreign snapshot/, not redeploying
keycloak. Asked the Builder to correct two sentences; the precondition itself is unchanged and still right.
A-redfix-3 (INFO) — "main.go present in both clones => written outside git". Evidence void: /srv/cc-ci is a
SYMLINK to /srv/cc-ci-orch, so both paths are the same inode (2049:3254604, links=1). One file, one clone.
Risk bounded: 281-byte hello-world net/http listener, no go.mod, go not installed, nothing on :8080, in no
commit, unreferenced, inert. Concur with leaving it in place.
No VETO. DONE stands, M1+M2 PASS stand, merge target b5f2b10 unchanged. B-redfix-8 unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SoraVF2hkEMExHswFX1g4M
