Compare commits
19
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
769fd29dcf | ||
|
|
41e80643c0 | ||
|
|
7147d777ee | ||
|
|
10ecb741e7 | ||
|
|
04e50c7c17 | ||
|
|
611e16f62d | ||
|
|
f42dbc3f82 | ||
|
|
1415cc53c6 | ||
|
|
12dee8bf75 | ||
|
|
8de2b125e9 | ||
|
|
1c70b9e61a | ||
|
|
b7bf41057a | ||
|
|
1c2d5e9f7f | ||
|
|
f6e977c69e | ||
|
|
0db8194dd5 | ||
|
|
148d4c9381 | ||
|
|
cb315f8ab4 | ||
|
|
c24bd0c62c | ||
|
|
5380997543 |
@@ -3,5 +3,6 @@
|
|||||||
# Keep credentials, tokens, and keys in /srv/cc-ci/.testenv. The orchestrator
|
# Keep credentials, tokens, and keys in /srv/cc-ci/.testenv. The orchestrator
|
||||||
# loads this file first via cc-ci-plan/load-env.sh.
|
# loads this file first via cc-ci-plan/load-env.sh.
|
||||||
GITEA_USERNAME=autonomic-bot
|
GITEA_USERNAME=autonomic-bot
|
||||||
|
GITEA_URL=git.autonomic.zone
|
||||||
TINFOIL_MODEL=deepseek-v4-pro
|
TINFOIL_MODEL=deepseek-v4-pro
|
||||||
TINFOIL_BASE_URL=https://inference.tinfoil.sh/v1
|
TINFOIL_BASE_URL=https://inference.tinfoil.sh/v1
|
||||||
|
|||||||
@@ -0,0 +1,283 @@
|
|||||||
|
# Plan: restricted ACME DNS renewal for cc-ci
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
Replace the manually issued, sops-stored wildcard certificate with unattended
|
||||||
|
DNS-01 renewal for these exact names:
|
||||||
|
|
||||||
|
```text
|
||||||
|
ci.commoninternet.net
|
||||||
|
*.ci.commoninternet.net
|
||||||
|
```
|
||||||
|
|
||||||
|
The cc-ci host will run an authoritative `acme-dns` instance only for
|
||||||
|
`acme.commoninternet.net`. It will never receive a Gandi credential or any
|
||||||
|
credential that can edit the parent `commoninternet.net` zone.
|
||||||
|
|
||||||
|
The only enduring delegation from the parent zone is:
|
||||||
|
|
||||||
|
```text
|
||||||
|
_acme-challenge.ci.commoninternet.net. CNAME <account-id>.acme.commoninternet.net.
|
||||||
|
```
|
||||||
|
|
||||||
|
That CNAME authorizes the generated acme-dns account to answer ACME TXT
|
||||||
|
challenges for the ci wildcard, not to edit any parent-zone DNS record.
|
||||||
|
|
||||||
|
## Project facts and constraints
|
||||||
|
|
||||||
|
- The target is the production `cc-ci-hetzner` NixOS 26.05 host, not the
|
||||||
|
orchestrator. Its public IPv4 is `91.98.47.73`; it has no public IPv6.
|
||||||
|
- The wildcard currently points at the public gateway, which TLS-passthroughs
|
||||||
|
to cc-ci's Traefik. DNS authority for `acme.commoninternet.net` must point
|
||||||
|
directly to `91.98.47.73`; the gateway is not involved in DNS.
|
||||||
|
- Nothing listens on TCP or UDP 53 today. The Nix firewall permits 22, 80, and
|
||||||
|
443 only; any Hetzner Cloud firewall must also be checked before deployment.
|
||||||
|
- TLS terminates in the Docker Swarm Traefik service. It currently reads
|
||||||
|
`ssl_cert` and `ssl_key` **Swarm secrets** populated from
|
||||||
|
`/var/lib/ci-certs/live/{fullchain.pem,privkey.pem}` by
|
||||||
|
`runner/warm_reconcile.py`. A normal host-service reload cannot install a
|
||||||
|
renewed certificate.
|
||||||
|
- The existing certificate is expired: its served validity ended
|
||||||
|
`2026-08-24 18:18:52 UTC`. Keep the current files as rollback material until
|
||||||
|
the new production certificate and Traefik rotation have both been verified.
|
||||||
|
- `pkgs.acme-dns` and `pkgs.lego` are available in the pinned nixpkgs. NixOS
|
||||||
|
`security.acme` uses Lego and supports a DNS provider plus an environment
|
||||||
|
file and post-renew hook. Confirm the pinned provider spelling with
|
||||||
|
`lego --help` during implementation; Lego's current documented provider code
|
||||||
|
is `acmedns`.
|
||||||
|
|
||||||
|
## Security invariants
|
||||||
|
|
||||||
|
1. Do not request, add, store, or use `GANDI_API_KEY`, a Gandi PAT, or any
|
||||||
|
parent-zone update credential on cc-ci or the orchestrator.
|
||||||
|
2. Bind the acme-dns HTTP API to `127.0.0.1` only. Its API may use plain HTTP
|
||||||
|
because it is loopback-only; do not create a circular API TLS dependency.
|
||||||
|
3. Allow public DNS only on TCP/UDP 53 and only for the authoritative zone.
|
||||||
|
4. The generated acme-dns account data is a secret. Keep it as a root/acme-only
|
||||||
|
persistent state file under `/var/lib/acme/`; never put it in Nix text, the
|
||||||
|
Nix store, git, `.env.public`, or a log.
|
||||||
|
5. After the account exists, set `disable_registration = true`. The existing
|
||||||
|
account must still be able to call `/update`.
|
||||||
|
6. Limit the account's update source with `ACME_DNS_ALLOWLIST=127.0.0.1/32`.
|
||||||
|
This is defence in depth in addition to the loopback API binding.
|
||||||
|
|
||||||
|
## Intended DNS design
|
||||||
|
|
||||||
|
Use an **out-of-bailiwick** nameserver name to avoid in-bailiwick glue
|
||||||
|
ambiguity:
|
||||||
|
|
||||||
|
```text
|
||||||
|
ns-acme.commoninternet.net. A 91.98.47.73
|
||||||
|
acme.commoninternet.net. NS ns-acme.commoninternet.net.
|
||||||
|
```
|
||||||
|
|
||||||
|
`acme-dns` itself serves the delegated zone and returns its matching NS record:
|
||||||
|
|
||||||
|
```text
|
||||||
|
acme.commoninternet.net. NS ns-acme.commoninternet.net.
|
||||||
|
```
|
||||||
|
|
||||||
|
This host is authoritative for `acme.commoninternet.net` and its generated
|
||||||
|
children only. It is not authoritative for `ci.commoninternet.net` or for
|
||||||
|
`commoninternet.net`.
|
||||||
|
|
||||||
|
## Implementation phases
|
||||||
|
|
||||||
|
### 1. Preflight and safety checks
|
||||||
|
|
||||||
|
Before changing Nix configuration, record:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh cc-ci 'ss -lntup "( sport = :53 )"'
|
||||||
|
ssh cc-ci 'systemctl list-units --type=service --all "*acme*" "*dns*"'
|
||||||
|
ssh cc-ci 'nft list ruleset'
|
||||||
|
ssh cc-ci 'docker service ls'
|
||||||
|
```
|
||||||
|
|
||||||
|
Confirm that no service owns port 53, that the Traefik Swarm services are
|
||||||
|
healthy, and that the Hetzner Cloud firewall will permit both 53/tcp and
|
||||||
|
53/udp. Do not replace an existing DNS service.
|
||||||
|
|
||||||
|
Obtain the operator's ACME contact email before enabling `security.acme`.
|
||||||
|
|
||||||
|
### 2. Add a dedicated acme-dns Nix module
|
||||||
|
|
||||||
|
Create `nix/modules/acme-dns.nix` and import it from
|
||||||
|
`nix/hosts/cc-ci-hetzner/configuration.nix`. The module should:
|
||||||
|
|
||||||
|
- create a dedicated unprivileged `acme-dns` user and group;
|
||||||
|
- run `${pkgs.acme-dns}/bin/acme-dns -c <public generated config>` with a
|
||||||
|
persistent working/state directory `/var/lib/acme-dns`;
|
||||||
|
- grant only `CAP_NET_BIND_SERVICE` to bind DNS port 53;
|
||||||
|
- use SQLite at `/var/lib/acme-dns/acme-dns.db` with mode `0600`;
|
||||||
|
- bind DNS to `91.98.47.73:53` with `protocol = "both4"`;
|
||||||
|
- set `domain = "acme.commoninternet.net"`,
|
||||||
|
`nsname = "ns-acme.commoninternet.net"`, and a public hostmaster-style
|
||||||
|
`nsadmin` value;
|
||||||
|
- include the public NS record above in `general.records`;
|
||||||
|
- bind `[api]` to `127.0.0.1:8080`, set `tls = "none"`, use a restrictive
|
||||||
|
CORS list, and initially leave `disable_registration = false`;
|
||||||
|
- use a hardened systemd unit: `NoNewPrivileges`, `PrivateTmp`,
|
||||||
|
`ProtectSystem = "strict"`, `ProtectHome`, `PrivateDevices`, and only the
|
||||||
|
state directory as writable; and
|
||||||
|
- open `networking.firewall.allowedTCPPorts = [ 53 ]` and
|
||||||
|
`allowedUDPPorts = [ 53 ]` in the **cc-ci Hetzner host** configuration.
|
||||||
|
|
||||||
|
The configuration file is public data and may be generated by Nix. It must not
|
||||||
|
contain account credentials.
|
||||||
|
|
||||||
|
Deploy this phase with the normal cc-ci deployment discipline: first
|
||||||
|
`nixos-rebuild test --flake /etc/cc-ci#cc-ci-hetzner`, verify SSH, Traefik, and
|
||||||
|
the host remain healthy, then run the identical `switch` target. Verify local
|
||||||
|
DNS on both transports:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dig @91.98.47.73 acme.commoninternet.net NS
|
||||||
|
dig +tcp @91.98.47.73 acme.commoninternet.net NS
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Operator gate: delegate the narrow DNS zone
|
||||||
|
|
||||||
|
After the service is healthy, ask the operator to add exactly these records at
|
||||||
|
Gandi (using its DNS UI, never a token on this host):
|
||||||
|
|
||||||
|
```dns
|
||||||
|
ns-acme.commoninternet.net. A 91.98.47.73
|
||||||
|
acme.commoninternet.net. NS ns-acme.commoninternet.net.
|
||||||
|
```
|
||||||
|
|
||||||
|
If Gandi models delegation as a nameserver/glue form rather than ordinary zone
|
||||||
|
records, use its equivalent UI flow. Do not proceed until public recursive DNS
|
||||||
|
shows the delegation and direct queries work from an external network:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dig NS acme.commoninternet.net @1.1.1.1
|
||||||
|
dig TXT test.acme.commoninternet.net @91.98.47.73
|
||||||
|
dig +tcp TXT test.acme.commoninternet.net @91.98.47.73
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Configure NixOS ACME in staging mode and obtain the account target
|
||||||
|
|
||||||
|
Extend the new module with one `security.acme.certs` entry for the base name
|
||||||
|
`ci.commoninternet.net`:
|
||||||
|
|
||||||
|
```nix
|
||||||
|
{
|
||||||
|
domain = "ci.commoninternet.net";
|
||||||
|
extraDomainNames = [ "*.ci.commoninternet.net" ];
|
||||||
|
dnsProvider = "acmedns"; # verify against the pinned Lego binary
|
||||||
|
environmentFile = "/etc/acme-dns/lego.env";
|
||||||
|
dnsResolver = "1.1.1.1:53";
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`/etc/acme-dns/lego.env` contains only non-secret wiring:
|
||||||
|
|
||||||
|
```text
|
||||||
|
ACME_DNS_API_BASE=http://127.0.0.1:8080
|
||||||
|
ACME_DNS_STORAGE_PATH=/var/lib/acme/ci.commoninternet.net/acme-dns-accounts.json
|
||||||
|
ACME_DNS_ALLOWLIST=127.0.0.1/32
|
||||||
|
```
|
||||||
|
|
||||||
|
Lego registers and persists its per-domain acme-dns account in the storage
|
||||||
|
path. The path is writable only by the ACME service user and is not Nix-managed
|
||||||
|
content. Do not hand-create its JSON: let the pinned Lego provider establish
|
||||||
|
the account format.
|
||||||
|
|
||||||
|
Set the ACME CA to Let's Encrypt staging for this phase. Start the certificate
|
||||||
|
unit manually after the NS delegation is confirmed. The first staging run is
|
||||||
|
expected to create the account and may fail validation because the CNAME is not
|
||||||
|
yet present. Read the storage file only with a root-only helper that prints the
|
||||||
|
generated **fulldomain** and never its username or password.
|
||||||
|
|
||||||
|
### 5. Operator gate: permanent challenge CNAME
|
||||||
|
|
||||||
|
Ask the operator to create the exact target reported in phase 4:
|
||||||
|
|
||||||
|
```dns
|
||||||
|
_acme-challenge.ci.commoninternet.net. CNAME <generated-id>.acme.commoninternet.net.
|
||||||
|
```
|
||||||
|
|
||||||
|
This is a permanent record. It must not be created, changed, or removed by an
|
||||||
|
agent. Confirm the complete chain through a public recursive resolver before
|
||||||
|
continuing:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dig CNAME _acme-challenge.ci.commoninternet.net @1.1.1.1
|
||||||
|
dig TXT <generated-id>.acme.commoninternet.net @91.98.47.73
|
||||||
|
dig +tcp TXT <generated-id>.acme.commoninternet.net @91.98.47.73
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6. Staging issuance, then production issuance
|
||||||
|
|
||||||
|
Run the NixOS ACME certificate unit against staging and verify all of the
|
||||||
|
following:
|
||||||
|
|
||||||
|
1. it updates only the generated acme-dns TXT target;
|
||||||
|
2. public recursive DNS sees the CNAME and the TXT value;
|
||||||
|
3. staging issues a certificate containing both requested names; and
|
||||||
|
4. no Gandi variable, credential file, or API request appears in the unit.
|
||||||
|
|
||||||
|
Only then select the production Let's Encrypt directory and issue the real
|
||||||
|
certificate. Keep the old sops certificate live during both attempts.
|
||||||
|
|
||||||
|
### 7. Make Traefik consume renewals safely
|
||||||
|
|
||||||
|
Do **not** use only `reloadServices`: Traefik receives Docker Swarm secrets and
|
||||||
|
cannot see an updated host file. Add a root-only renewal handoff service,
|
||||||
|
serialized with all other Traefik reconciliation, and call it from the ACME
|
||||||
|
certificate's `postRun` hook.
|
||||||
|
|
||||||
|
The handoff must:
|
||||||
|
|
||||||
|
1. atomically copy the new `fullchain.pem` and key from the NixOS ACME output
|
||||||
|
into `/var/lib/ci-certs/live`, with the existing `0444`/`0400` modes;
|
||||||
|
2. generate a new, content-derived **non-secret** Swarm secret version;
|
||||||
|
3. insert new `ssl_cert` and `ssl_key` Swarm secrets, update the Traefik recipe
|
||||||
|
environment to reference those versions, and reconcile/redeploy Traefik;
|
||||||
|
4. health-check `https://traefik.ci.commoninternet.net/api/version` with SNI;
|
||||||
|
5. retain the prior secret version until the new task is healthy, then remove
|
||||||
|
it; and
|
||||||
|
6. record a failure clearly without deleting the last-known-good certificate.
|
||||||
|
|
||||||
|
Implement this as a tested extension of `runner/warm_reconcile.py` (or a
|
||||||
|
small, explicitly locked companion) rather than an ad-hoc shell command. The
|
||||||
|
renewal path and the normal `deploy-proxy` path must share a lock so they cannot
|
||||||
|
race over Swarm secret versions.
|
||||||
|
|
||||||
|
After production issuance and a successful Traefik rotation, remove the
|
||||||
|
`wildcard_cert` and `wildcard_key` sops declarations from
|
||||||
|
`nix/modules/secrets.nix`; otherwise later Nix activations would overwrite the
|
||||||
|
renewed host files. Remove the obsolete encrypted values from the private
|
||||||
|
`cc-ci-secrets` repository only after rollback is no longer needed.
|
||||||
|
|
||||||
|
### 8. Lock registration and prove unattended renewal
|
||||||
|
|
||||||
|
In a follow-up Nix change, set `api.disable_registration = true`, test that the
|
||||||
|
existing account can still update its TXT record, and confirm `/register` is
|
||||||
|
rejected. Then verify:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl list-timers 'acme-*'
|
||||||
|
systemctl start acme-ci.commoninternet.net.service
|
||||||
|
journalctl -u acme-ci.commoninternet.net.service -b
|
||||||
|
```
|
||||||
|
|
||||||
|
Perform a controlled staging renewal after registration is disabled, observe
|
||||||
|
the renewed Traefik secret version, and confirm the certificate served through
|
||||||
|
the gateway has the expected names and a new validity window.
|
||||||
|
|
||||||
|
## Final acceptance checklist
|
||||||
|
|
||||||
|
- [ ] cc-ci and the orchestrator contain no Gandi API credential.
|
||||||
|
- [ ] Gandi delegates only `acme.commoninternet.net` to cc-ci.
|
||||||
|
- [ ] Only `_acme-challenge.ci.commoninternet.net` CNAMEs into that zone.
|
||||||
|
- [ ] The acme-dns API is loopback-only; only 53/tcp and 53/udp are public.
|
||||||
|
- [ ] External UDP and TCP authoritative DNS checks pass.
|
||||||
|
- [ ] Registration is disabled after the one account is created.
|
||||||
|
- [ ] The ACME account can update only its generated TXT record.
|
||||||
|
- [ ] The certificate covers both `ci.commoninternet.net` and its wildcard.
|
||||||
|
- [ ] A renewal rotates Traefik's Swarm secrets and preserves a working prior
|
||||||
|
version until the replacement passes health checks.
|
||||||
|
- [ ] No credential or private key has been committed, logged, or written into
|
||||||
|
the Nix store.
|
||||||
@@ -14,6 +14,7 @@
|
|||||||
./networking.nix
|
./networking.nix
|
||||||
../../modules/packages.nix
|
../../modules/packages.nix
|
||||||
../../modules/secrets.nix
|
../../modules/secrets.nix
|
||||||
|
../../modules/acme-dns.nix
|
||||||
../../modules/swarm.nix
|
../../modules/swarm.nix
|
||||||
../../modules/docker-prune.nix
|
../../modules/docker-prune.nix
|
||||||
../../modules/abra.nix
|
../../modules/abra.nix
|
||||||
|
|||||||
@@ -0,0 +1,155 @@
|
|||||||
|
# Restricted DNS-01 certificate issuance for ci.commoninternet.net.
|
||||||
|
#
|
||||||
|
# This host is authoritative only for acme.commoninternet.net. Gandi continues
|
||||||
|
# to own commoninternet.net; it delegates this narrow zone and one permanent
|
||||||
|
# _acme-challenge CNAME manually. No Gandi credential is present here.
|
||||||
|
{ pkgs, ... }:
|
||||||
|
let
|
||||||
|
acmeDnsConfig = pkgs.writeText "cc-ci-acme-dns.conf" ''
|
||||||
|
[general]
|
||||||
|
listen = "91.98.47.73:53"
|
||||||
|
protocol = "both4"
|
||||||
|
domain = "acme.commoninternet.net"
|
||||||
|
nsname = "ns-acme.commoninternet.net"
|
||||||
|
nsadmin = "hostmaster.commoninternet.net"
|
||||||
|
records = [
|
||||||
|
"acme.commoninternet.net. NS ns-acme.commoninternet.net.",
|
||||||
|
"ns-acme.commoninternet.net. A 91.98.47.73",
|
||||||
|
]
|
||||||
|
debug = false
|
||||||
|
|
||||||
|
[database]
|
||||||
|
# acme-dns 2.x registers the embedded driver under `sqlite` (not the
|
||||||
|
# legacy `sqlite3` identifier).
|
||||||
|
engine = "sqlite"
|
||||||
|
connection = "/var/lib/acme-dns/acme-dns.db"
|
||||||
|
|
||||||
|
[api]
|
||||||
|
ip = "127.0.0.1"
|
||||||
|
port = "8080"
|
||||||
|
tls = "none"
|
||||||
|
# The one Lego account was bootstrapped before this configuration was
|
||||||
|
# hardened. Updates authenticated by that account remain available.
|
||||||
|
disable_registration = true
|
||||||
|
corsorigins = []
|
||||||
|
|
||||||
|
[logconfig]
|
||||||
|
loglevel = "info"
|
||||||
|
logtype = "stdout"
|
||||||
|
logformat = "json"
|
||||||
|
'';
|
||||||
|
|
||||||
|
# These are wiring values only. The acme-dns account JSON is generated by
|
||||||
|
# Lego below /var/lib/acme and never enters Nix, git, or /etc.
|
||||||
|
legoEnvironment = pkgs.writeText "cc-ci-acme-dns-lego.env" ''
|
||||||
|
ACME_DNS_API_BASE=http://127.0.0.1:8080
|
||||||
|
ACME_DNS_STORAGE_PATH=/var/lib/acme/ci.commoninternet.net/acme-dns-accounts.json
|
||||||
|
ACME_DNS_ALLOWLIST=127.0.0.1/32
|
||||||
|
'';
|
||||||
|
in
|
||||||
|
{
|
||||||
|
users.groups.acme-dns = { };
|
||||||
|
users.users.acme-dns = {
|
||||||
|
isSystemUser = true;
|
||||||
|
group = "acme-dns";
|
||||||
|
home = "/var/lib/acme-dns";
|
||||||
|
};
|
||||||
|
|
||||||
|
environment.etc."acme-dns/lego.env".source = legoEnvironment;
|
||||||
|
|
||||||
|
# The staging order has completed successfully. This marker permits the
|
||||||
|
# production ACME post-run hook to hand a renewed certificate to Traefik.
|
||||||
|
systemd.tmpfiles.rules = [
|
||||||
|
"f /var/lib/ci-certs/acme-production-enabled 0600 root root -"
|
||||||
|
];
|
||||||
|
|
||||||
|
networking.firewall = {
|
||||||
|
allowedTCPPorts = [ 53 ];
|
||||||
|
allowedUDPPorts = [ 53 ];
|
||||||
|
};
|
||||||
|
|
||||||
|
systemd.services.acme-dns = {
|
||||||
|
description = "Restricted authoritative DNS for cc-ci ACME DNS-01";
|
||||||
|
wantedBy = [ "multi-user.target" ];
|
||||||
|
after = [ "network-online.target" ];
|
||||||
|
wants = [ "network-online.target" ];
|
||||||
|
serviceConfig = {
|
||||||
|
User = "acme-dns";
|
||||||
|
Group = "acme-dns";
|
||||||
|
StateDirectory = "acme-dns";
|
||||||
|
StateDirectoryMode = "0700";
|
||||||
|
WorkingDirectory = "/var/lib/acme-dns";
|
||||||
|
ExecStart = "${pkgs.acme-dns}/bin/acme-dns -c ${acmeDnsConfig}";
|
||||||
|
Restart = "on-failure";
|
||||||
|
RestartSec = "5s";
|
||||||
|
AmbientCapabilities = [ "CAP_NET_BIND_SERVICE" ];
|
||||||
|
CapabilityBoundingSet = [ "CAP_NET_BIND_SERVICE" ];
|
||||||
|
NoNewPrivileges = true;
|
||||||
|
PrivateTmp = true;
|
||||||
|
PrivateDevices = true;
|
||||||
|
ProtectHome = true;
|
||||||
|
ProtectSystem = "strict";
|
||||||
|
ReadWritePaths = [ "/var/lib/acme-dns" ];
|
||||||
|
RestrictAddressFamilies = [ "AF_INET" "AF_UNIX" ];
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
# Traefik consumes its wildcard as immutable Swarm secrets, so a renewed
|
||||||
|
# host certificate must be copied and reconciled rather than merely reloaded.
|
||||||
|
# This service is started only by the production-mode ACME postRun hook.
|
||||||
|
systemd.services.cc-ci-acme-traefik-handoff = {
|
||||||
|
description = "Install renewed cc-ci wildcard into Traefik Swarm secrets";
|
||||||
|
after = [ "docker.service" "deploy-proxy.service" ];
|
||||||
|
requires = [ "docker.service" ];
|
||||||
|
path = [ pkgs.coreutils pkgs.docker pkgs.systemd pkgs.gnugrep ];
|
||||||
|
serviceConfig = {
|
||||||
|
Type = "oneshot";
|
||||||
|
UMask = "0077";
|
||||||
|
};
|
||||||
|
script = ''
|
||||||
|
src=/var/lib/acme/ci.commoninternet.net
|
||||||
|
dst=/var/lib/ci-certs/live
|
||||||
|
test -s "$src/fullchain.pem"
|
||||||
|
test -s "$src/key.pem"
|
||||||
|
install -d -m 0700 "$dst"
|
||||||
|
install -m 0444 "$src/fullchain.pem" "$dst/fullchain.pem.new"
|
||||||
|
install -m 0400 "$src/key.pem" "$dst/privkey.pem.new"
|
||||||
|
mv -f "$dst/fullchain.pem.new" "$dst/fullchain.pem"
|
||||||
|
mv -f "$dst/privkey.pem.new" "$dst/privkey.pem"
|
||||||
|
|
||||||
|
# deploy-proxy performs the health-gated Swarm rollout. Its reconciler
|
||||||
|
# derives a fresh version from the public certificate chain and inserts
|
||||||
|
# the matching ssl_cert/ssl_key secrets before deploying Traefik.
|
||||||
|
systemctl restart deploy-proxy.service
|
||||||
|
|
||||||
|
# A successful rollout no longer references old wildcard versions. Best
|
||||||
|
# effort removal retains any secret Docker still reports as in use.
|
||||||
|
keep="v$(sha256sum "$dst/fullchain.pem" | cut -c1-16)"
|
||||||
|
docker secret ls --format '{{.Name}}' | \
|
||||||
|
grep -E '^traefik_ci_commoninternet_net_ssl_(cert|key)_v' | \
|
||||||
|
grep -v -E "_(ssl_cert|ssl_key)_$keep\$" | \
|
||||||
|
while IFS= read -r stale; do docker secret rm "$stale" || true; done
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
security.acme = {
|
||||||
|
acceptTerms = true;
|
||||||
|
certs."ci.commoninternet.net" = {
|
||||||
|
domain = "ci.commoninternet.net";
|
||||||
|
extraDomainNames = [ "*.ci.commoninternet.net" ];
|
||||||
|
# Staging issuance succeeded using the permanent, narrowly delegated
|
||||||
|
# CNAME. Production uses the same restricted acme-dns account.
|
||||||
|
dnsProvider = "acmedns";
|
||||||
|
environmentFile = "/etc/acme-dns/lego.env";
|
||||||
|
dnsResolver = "1.1.1.1:53";
|
||||||
|
server = "https://acme-v02.api.letsencrypt.org/directory";
|
||||||
|
postRun = ''
|
||||||
|
# The production marker is deployed only after staging proves the
|
||||||
|
# permanent CNAME and restricted acme-dns account work end to end.
|
||||||
|
if [ -e /var/lib/ci-certs/acme-production-enabled ]; then
|
||||||
|
${pkgs.systemd}/bin/systemctl --no-block start cc-ci-acme-traefik-handoff.service
|
||||||
|
fi
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
};
|
||||||
|
}
|
||||||
+4
-11
@@ -37,17 +37,10 @@
|
|||||||
bridge_drone_token = { };
|
bridge_drone_token = { };
|
||||||
bridge_gitea_token = { };
|
bridge_gitea_token = { };
|
||||||
|
|
||||||
# Phase-1c C2: the wildcard TLS cert+key are now sops secrets (in cc-ci-secrets), decrypted at
|
# The wildcard certificate and private key are issued and renewed locally
|
||||||
# activation to /var/lib/ci-certs/live/{fullchain.pem,privkey.pem} — the exact path the traefik
|
# by security.acme. Do not restore the retired SOPS pair here: activation
|
||||||
# reconcile (modules/proxy.nix) already reads. Replaces the prior operator-drops-a-cert-file step.
|
# would overwrite a freshly renewed ACME certificate before Traefik can
|
||||||
wildcard_cert = {
|
# consume it.
|
||||||
path = "/var/lib/ci-certs/live/fullchain.pem";
|
|
||||||
mode = "0444"; # leaf+intermediate chain — not secret
|
|
||||||
};
|
|
||||||
wildcard_key = {
|
|
||||||
path = "/var/lib/ci-certs/live/privkey.pem";
|
|
||||||
mode = "0400"; # private key — root only
|
|
||||||
};
|
|
||||||
|
|
||||||
# Phase-2 rate-limit fix (Class A1 registry creds, operator-2026-05-28). Authenticated Docker
|
# Phase-2 rate-limit fix (Class A1 registry creds, operator-2026-05-28). Authenticated Docker
|
||||||
# Hub pulls (200/6h per-account) replace the exhausted 100/6h shared-IP anonymous limit that
|
# Hub pulls (200/6h per-account) replace the exhausted 100/6h shared-IP anonymous limit that
|
||||||
|
|||||||
@@ -24,6 +24,7 @@ Run as root on cc-ci (direct docker/volume access). CLI: `warm_reconcile.py <app
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
@@ -37,12 +38,51 @@ from harness import abra, lifecycle, warmsnap # noqa: E402
|
|||||||
# --------------------------------------------------------------------------- specs
|
# --------------------------------------------------------------------------- specs
|
||||||
|
|
||||||
|
|
||||||
def _traefik_setup(recipe: str, domain: str, version: str) -> None:
|
CERT_DIR = "/var/lib/ci-certs/live"
|
||||||
|
|
||||||
|
|
||||||
|
def wildcard_secret_version(cert_dir: str = CERT_DIR) -> str:
|
||||||
|
"""Stable Swarm-secret version for the public certificate chain.
|
||||||
|
|
||||||
|
The certificate chain is public material, so its digest is safe to use as a
|
||||||
|
version label. The key is deliberately never read or hashed for logging.
|
||||||
|
"""
|
||||||
|
chain = os.path.join(cert_dir, "fullchain.pem")
|
||||||
|
if not os.path.isfile(chain):
|
||||||
|
raise RuntimeError(f"FATAL: wildcard certificate missing at {chain}")
|
||||||
|
with open(chain, "rb") as certificate:
|
||||||
|
digest = hashlib.sha256(certificate.read()).hexdigest()
|
||||||
|
return "v" + digest[:16]
|
||||||
|
|
||||||
|
|
||||||
|
def _traefik_requires_certificate_rollout(domain: str, secret_version: str) -> bool:
|
||||||
|
"""Whether Traefik's active service still references an older cert version."""
|
||||||
|
stack = lifecycle._stack_name(domain) # noqa: SLF001
|
||||||
|
service = f"{stack}_app"
|
||||||
|
result = _run(
|
||||||
|
[
|
||||||
|
"docker",
|
||||||
|
"service",
|
||||||
|
"inspect",
|
||||||
|
service,
|
||||||
|
"--format",
|
||||||
|
"{{range .Spec.TaskTemplate.ContainerSpec.Secrets}}{{.SecretName}} {{end}}",
|
||||||
|
],
|
||||||
|
timeout=30,
|
||||||
|
)
|
||||||
|
expected = {
|
||||||
|
f"{stack}_ssl_cert_{secret_version}",
|
||||||
|
f"{stack}_ssl_key_{secret_version}",
|
||||||
|
}
|
||||||
|
return not expected.issubset(set(result.stdout.split()))
|
||||||
|
|
||||||
|
|
||||||
|
def _traefik_setup(recipe: str, domain: str, version: str) -> bool:
|
||||||
"""Per-app config for the traefik reverse-proxy reconcile — preserves EXACTLY what the prior
|
"""Per-app config for the traefik reverse-proxy reconcile — preserves EXACTLY what the prior
|
||||||
proxy.nix bash reconcile did (wildcard/file-provider mode serving the pre-issued cert as
|
proxy.nix bash reconcile did (wildcard/file-provider mode serving the pre-issued cert as
|
||||||
ssl_cert/ssl_key swarm secrets; NO ACME). Uses the proven abra.env_set (newline-safe, unlike the
|
ssl_cert/ssl_key swarm secrets; NO ACME). Uses the proven abra.env_set (newline-safe, unlike the
|
||||||
bash set_env that bit keycloak)."""
|
bash set_env that bit keycloak)."""
|
||||||
cert_dir = "/var/lib/ci-certs/live"
|
cert_dir = CERT_DIR
|
||||||
if not (
|
if not (
|
||||||
os.path.isfile(f"{cert_dir}/fullchain.pem") and os.path.isfile(f"{cert_dir}/privkey.pem")
|
os.path.isfile(f"{cert_dir}/fullchain.pem") and os.path.isfile(f"{cert_dir}/privkey.pem")
|
||||||
):
|
):
|
||||||
@@ -56,14 +96,15 @@ def _traefik_setup(recipe: str, domain: str, version: str) -> None:
|
|||||||
abra.env_set(domain, "DOMAIN", domain)
|
abra.env_set(domain, "DOMAIN", domain)
|
||||||
abra.env_set(domain, "LETS_ENCRYPT_ENV", "")
|
abra.env_set(domain, "LETS_ENCRYPT_ENV", "")
|
||||||
abra.env_set(domain, "WILDCARDS_ENABLED", "1")
|
abra.env_set(domain, "WILDCARDS_ENABLED", "1")
|
||||||
abra.env_set(domain, "SECRET_WILDCARD_CERT_VERSION", "v1")
|
secret_version = wildcard_secret_version(cert_dir)
|
||||||
abra.env_set(domain, "SECRET_WILDCARD_KEY_VERSION", "v1")
|
abra.env_set(domain, "SECRET_WILDCARD_CERT_VERSION", secret_version)
|
||||||
|
abra.env_set(domain, "SECRET_WILDCARD_KEY_VERSION", secret_version)
|
||||||
abra.env_set(domain, "COMPOSE_FILE", '"compose.yml:compose.wildcard.yml"')
|
abra.env_set(domain, "COMPOSE_FILE", '"compose.yml:compose.wildcard.yml"')
|
||||||
stack = lifecycle._stack_name(domain) # noqa: SLF001
|
stack = lifecycle._stack_name(domain) # noqa: SLF001
|
||||||
have = set(lifecycle._docker_names("secret", stack)) # noqa: SLF001
|
have = set(lifecycle._docker_names("secret", stack)) # noqa: SLF001
|
||||||
|
|
||||||
def _has(name):
|
def _has(name):
|
||||||
return any(s.endswith(f"_{name}_v1") for s in have)
|
return any(s.endswith(f"_{name}_{secret_version}") for s in have)
|
||||||
|
|
||||||
if not _has("ssl_cert"):
|
if not _has("ssl_cert"):
|
||||||
_run(
|
_run(
|
||||||
@@ -74,7 +115,7 @@ def _traefik_setup(recipe: str, domain: str, version: str) -> None:
|
|||||||
"insert",
|
"insert",
|
||||||
domain,
|
domain,
|
||||||
"ssl_cert",
|
"ssl_cert",
|
||||||
"v1",
|
secret_version,
|
||||||
f"{cert_dir}/fullchain.pem",
|
f"{cert_dir}/fullchain.pem",
|
||||||
"-f",
|
"-f",
|
||||||
"-n",
|
"-n",
|
||||||
@@ -91,7 +132,7 @@ def _traefik_setup(recipe: str, domain: str, version: str) -> None:
|
|||||||
"insert",
|
"insert",
|
||||||
domain,
|
domain,
|
||||||
"ssl_key",
|
"ssl_key",
|
||||||
"v1",
|
secret_version,
|
||||||
f"{cert_dir}/privkey.pem",
|
f"{cert_dir}/privkey.pem",
|
||||||
"-f",
|
"-f",
|
||||||
"-n",
|
"-n",
|
||||||
@@ -99,6 +140,7 @@ def _traefik_setup(recipe: str, domain: str, version: str) -> None:
|
|||||||
timeout=120,
|
timeout=120,
|
||||||
check=True,
|
check=True,
|
||||||
)
|
)
|
||||||
|
return _traefik_requires_certificate_rollout(domain, secret_version)
|
||||||
|
|
||||||
|
|
||||||
SPECS: dict[str, dict] = {
|
SPECS: dict[str, dict] = {
|
||||||
@@ -457,8 +499,9 @@ def reconcile(app: str) -> str:
|
|||||||
# Per-app config/secrets: a spec may provide its own `setup` (traefik's cert/file-provider wiring);
|
# Per-app config/secrets: a spec may provide its own `setup` (traefik's cert/file-provider wiring);
|
||||||
# otherwise the default keycloak-shaped path (app new + DOMAIN/LETS_ENCRYPT + generate secrets).
|
# otherwise the default keycloak-shaped path (app new + DOMAIN/LETS_ENCRYPT + generate secrets).
|
||||||
setup = spec.get("setup")
|
setup = spec.get("setup")
|
||||||
|
setup_needs_rollout = False
|
||||||
if setup:
|
if setup:
|
||||||
setup(recipe, domain, latest)
|
setup_needs_rollout = bool(setup(recipe, domain, latest))
|
||||||
else:
|
else:
|
||||||
ensure_app_config(recipe, domain, latest)
|
ensure_app_config(recipe, domain, latest)
|
||||||
ensure_secrets(domain)
|
ensure_secrets(domain)
|
||||||
@@ -476,6 +519,20 @@ def reconcile(app: str) -> str:
|
|||||||
write_last_good(recipe, target)
|
write_last_good(recipe, target)
|
||||||
return f"deployed-fresh:{target}"
|
return f"deployed-fresh:{target}"
|
||||||
|
|
||||||
|
# A certificate rotation changes Traefik's immutable Swarm secrets but
|
||||||
|
# must not be held hostage by an unrelated recipe-major upgrade policy.
|
||||||
|
# Redeploy the current recipe version so its compose spec references the
|
||||||
|
# just-created cert/key secret pair, then apply the usual health gate.
|
||||||
|
if setup_needs_rollout:
|
||||||
|
if not current:
|
||||||
|
raise RuntimeError(f"{app} has services but no current version")
|
||||||
|
print(f"[{app}] certificate changed → redeploy {current}", flush=True)
|
||||||
|
deploy_version(recipe, domain, current, dt)
|
||||||
|
if not wait_healthy(spec):
|
||||||
|
raise RuntimeError(f"{app} certificate rollout {current} did not become healthy")
|
||||||
|
write_last_good(recipe, current)
|
||||||
|
return f"certificate-rolled-out:{current}"
|
||||||
|
|
||||||
# Deployed & already on latest → converge to a no-op (commit last-good if healthy).
|
# Deployed & already on latest → converge to a no-op (commit last-good if healthy).
|
||||||
if current == latest:
|
if current == latest:
|
||||||
if wait_healthy(spec, timeout=60):
|
if wait_healthy(spec, timeout=60):
|
||||||
|
|||||||
@@ -0,0 +1,21 @@
|
|||||||
|
"""Unit coverage for the public wildcard-secret version label."""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).parents[2] / "runner"))
|
||||||
|
import warm_reconcile as wr # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
def test_wildcard_secret_version_is_stable_and_does_not_need_key(tmp_path):
|
||||||
|
(tmp_path / "fullchain.pem").write_text("public certificate chain\n")
|
||||||
|
assert wr.wildcard_secret_version(str(tmp_path)) == wr.wildcard_secret_version(str(tmp_path))
|
||||||
|
assert wr.wildcard_secret_version(str(tmp_path)).startswith("v")
|
||||||
|
|
||||||
|
|
||||||
|
def test_wildcard_secret_version_changes_with_certificate_chain(tmp_path):
|
||||||
|
chain = tmp_path / "fullchain.pem"
|
||||||
|
chain.write_text("first public certificate chain\n")
|
||||||
|
first = wr.wildcard_secret_version(str(tmp_path))
|
||||||
|
chain.write_text("replacement public certificate chain\n")
|
||||||
|
assert wr.wildcard_secret_version(str(tmp_path)) != first
|
||||||
@@ -9,6 +9,7 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import os
|
import os
|
||||||
import sys
|
import sys
|
||||||
|
from types import SimpleNamespace
|
||||||
|
|
||||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", ".."))
|
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", ".."))
|
||||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner"))
|
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner"))
|
||||||
@@ -115,6 +116,20 @@ def test_traefik_spec_is_stateless_with_setup():
|
|||||||
assert "setup" not in wr.SPECS["keycloak"]
|
assert "setup" not in wr.SPECS["keycloak"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_traefik_certificate_rollout_detects_active_secret_version(monkeypatch):
|
||||||
|
stack = "traefik_ci_commoninternet_net"
|
||||||
|
monkeypatch.setattr(wr.lifecycle, "_stack_name", lambda _domain: stack)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
wr,
|
||||||
|
"_run",
|
||||||
|
lambda *_args, **_kwargs: SimpleNamespace(
|
||||||
|
stdout=f"{stack}_ssl_cert_vnew {stack}_ssl_key_vnew"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
assert not wr._traefik_requires_certificate_rollout("traefik.ci.commoninternet.net", "vnew")
|
||||||
|
assert wr._traefik_requires_certificate_rollout("traefik.ci.commoninternet.net", "vold")
|
||||||
|
|
||||||
|
|
||||||
def test_manual_migration_markers():
|
def test_manual_migration_markers():
|
||||||
assert wr.notes_flag_manual_migration("This release requires a MANUAL MIGRATION of the DB.")
|
assert wr.notes_flag_manual_migration("This release requires a MANUAL MIGRATION of the DB.")
|
||||||
assert wr.notes_flag_manual_migration("Breaking change: action required before upgrade.")
|
assert wr.notes_flag_manual_migration("Breaking change: action required before upgrade.")
|
||||||
|
|||||||
Reference in New Issue
Block a user