nix/hosts/cc-ci: real hardware + networking from the box; document the infect fixes

hardware.nix / networking.nix are now the nixos-infect output of
195.201.88.249 (ESP E079-7D41, eth0 MAC-pinned), with the one edit that
matters: defaultGateway as an attrset with interface = "eth0".

README §2a records what it actually took to get NixOS running on the
Debian 13 image: NO_SWAP=true (tmpfs /tmp), a manual lustration from rescue
mode because 26.05's systemd initrd did not honour NIXOS_LUSTRATE (Debian's
unit files shadowed every NixOS service), and the gateway fix applied via a
chroot rebuild with the nix sandbox off.

flake: cc-ci input back on main (PR #32 merged as f6dbfa3); loops uid pinned
to 1000 so workspace rsyncs by uid line up.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz

README.md

@@ -82,9 +82,48 @@ It downloads Nix, builds a NixOS system (5–10 min; follow with
 `ssh root@<ip> tail -f /var/log/nixos-infect.log`), then reboots. The SSH host key changes:
 `ssh-keygen -R <ip>` and confirm `ssh root@<ip> nixos-version` prints a 26.05 version.
 
-> If the box does not come back within ~5 minutes, open the Hetzner console (or rescue mode) and
-> look at the boot messages — the two known failure modes are a broken bootloader install and a
-> wrong `networking.nix` (see §3, the `defaultGateway.interface` note).
+### 2a. What went wrong on 2026-09-07, and the fixes (Debian 13 image, NixOS 26.05)
+
+All three bit on the first attempt; the script above and §3 already include the fixes, this is
+so you recognise them if they come back in another form.
+
+1. **`swapon: /tmp/nixos-infect.XXXX.swp: Invalid argument`** right at the start, script exits.
+   The Debian 13 cloud image mounts `/tmp` as tmpfs and a swapfile cannot live there.
+   Fix: `NO_SWAP=true` (in the script above). An 8 GB box does not need the temporary swap.
+2. **The box never comes back after the reboot: it boots NixOS, but nearly every unit fails**
+   (`dbus`, `systemd-logind`, `sshd`, networking …) with
+   `Could not start dynamically linked executable: /usr/bin/dbus-daemon` in the journal.
+   nixos-infect leaves the old Debian root in place and relies on NixOS's first boot to move it
+   to `/old-root` (`/etc/NIXOS_LUSTRATE`). With NixOS 26.05's systemd-based initrd that
+   lustration did not happen, so Debian's `/etc/systemd/system/*.service` files shadowed the
+   NixOS units and started Debian binaries. Fix, from Hetzner **rescue mode**
+   (`enable_rescue` + `reset` in the API/console, ssh in, `mount /dev/sda1 /mnt/root`):
+   move everything except `nix`, `boot`, `swapfile`, `lost+found`, `var/log`, `var/empty`,
+   `etc/nixos`, `etc/resolv.conf`, `etc/NIXOS`, `etc/machine-id`, `etc/ssh/ssh_host_*`,
+   `root/.nix-*`, `root/.ssh` into `/mnt/root/old-root`, delete `etc/NIXOS_LUSTRATE`, unmount,
+   `disable_rescue`, `reset`. (`/old-root`, ~1 GB, can be deleted once the host is in service.)
+3. **Boots, units fine, but no network.** The generated `networking.nix` has
+   `defaultGateway = "172.31.1.1";` — a bare string. Since NixOS 25.05 that yields no default
+   route. Fix: `defaultGateway = { address = "172.31.1.1"; interface = "eth0"; };` (this is what
+   `nix/hosts/cc-ci/networking.nix` carries). To apply it from rescue mode, chroot into the
+   mounted root and rebuild the boot entry — the nix sandbox cannot `pivot_root` inside a chroot,
+   so turn it off for that one build:
+   ```bash
+   for d in proc sys dev dev/pts; do mount --bind /$d /mnt/root/$d; done
+   mount -t tmpfs tmpfs /mnt/root/run; cp -L /etc/resolv.conf /mnt/root/etc/resolv.conf
+   chroot /mnt/root /nix/var/nix/profiles/system/sw/bin/bash -c '
+     export PATH=/nix/var/nix/profiles/system/sw/bin NIX_REMOTE= HOME=/root
+     export NIX_PATH=nixos-config=/etc/nixos/configuration.nix:nixpkgs=/root/.nix-defexpr/channels/nixos
+     ln -sfn /nix/var/nix/profiles/system /run/current-system
+     nixos-rebuild boot --option sandbox false'
+   ```
+   The `journalctl -D /mnt/root/var/log/journal -b 0` trick (reading the dead system's journal
+   from rescue mode) is what told these apart.
+
+> Rescue mode without a console: `POST /servers/<id>/actions/enable_rescue` with your ssh key
+> id, then `…/actions/reset`; afterwards `disable_rescue` **and check `rescue_enabled` is false
+> before** the next `reset`, or it boots the rescue image again. `scripts/recovery/hetzner.py`
+> wraps these (token in `/srv/cc-ci/.hcloud-token`).
 
 ## 3. Capture the machine-specific config into this repo
 
@@ -101,10 +140,9 @@ Then in `nix/hosts/cc-ci/`:
 - `hardware.nix`: keep as generated (GRUB EFI with `efiInstallAsRemovable`, `/boot/efi` by UUID,
   `/dev/sda1` root). Do not copy another host's file — the UUIDs are per machine.
 - `networking.nix`: keep the static IPv4 + Hetzner gateway `172.31.1.1`. Make sure
-  `networking.defaultGateway` has **both** `address` and `interface = "eth0"` — since NixOS 25.05
-  omitting the interface leaves the host without a default route after a rebuild (this took the
-  old CI server offline once; rescue mode fixed it). If the generated IPv6 block has an empty
-  address, delete the IPv6 parts.
+  `networking.defaultGateway` has **both** `address` and `interface = "eth0"` (§2a item 3). If
+  the generated IPv6 block has an empty address, delete the IPv6 parts; a real global address
+  (as on the 2026-09 box) can stay.
 - `configuration.nix`: set `cc-ci.publicIPv4` to the server's IPv4 and check `system.stateVersion`
   is the release you installed (never change it later).
 - `ssh-keys`: the root keys.