From 22bd897a86b175574dda486ec757ba422cbdb776 Mon Sep 17 00:00:00 2001
From: notplants-bot <mfowler.email@protonmail.com>
Date: Thu, 20 Aug 2026 17:10:28 +0000
Subject: [PATCH] gateway-domain: give a tailnet box a real public domain
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

An agent on a box with no public IP frequently needs a reachable HTTPS URL —
an OAuth callback, a webhook receiver, a demo link. The testing gateway
already holds a wildcard record for *.gtest.commoninternet.net and forwards
by SNI, but nothing here knew that, so every agent had to be told by hand.

    tools/gateway-domain.py add myapp
    #   myapp.gtest.commoninternet.net  ->  100.84.190.30

The backend defaults to the running box's own tailscale IP, which is the case
that comes up almost every time.

The admin password comes from gateway.admin_password in the secret store; the
tool reads it itself, so no caller handles the value and there is no second
copy to drift or get committed.

Two things the tool refuses to do, both learned by doing them:

Backends must be a literal IPv4 address. The gateway's validate_ip accepts a
hostname, but put_domain/remove_domain only match lines whose backend is
numeric ([\d.:]+). A hostname mapping can therefore be written once and never
updated or removed through the admin UI — it becomes an orphan that only a
hand-edit of tunnel_map.conf clears. One got created while testing this.

Verification re-reads the mapping table instead of trusting the POST body.
The admin app mutates its in-memory dict and renders that, so a delete that
silently failed still renders as success. Checking the response alone
reported "removed" for an entry that was still on disk.

skills/gateway-domain/ carries the rest: that the gateway does NOT terminate
TLS (your box serves the cert for that name), how ACME still works through
it, that only 22/80/443 are open at the edge, and how to recover if an
interrupted e2e run leaves the admin password reseeded.
---
 README.md                      |  16 +++
 skills/gateway-domain/SKILL.md | 129 ++++++++++++++++++++
 tools/gateway-domain.py        | 216 +++++++++++++++++++++++++++++++++
 3 files changed, 361 insertions(+)
 create mode 100644 skills/gateway-domain/SKILL.md
 create mode 100755 tools/gateway-domain.py

diff --git a/README.md b/README.md
index 68e21ac..68b22fa 100644
--- a/README.md
+++ b/README.md
@@ -379,6 +379,22 @@ so reviewers read the pre-fixup code and no interdiff exists — with no warning
 push itself succeeded. Always: push, resubmit, then reply with the printed interdiff URL. See
 `machine-docs/PR-WORKFLOW.md`, "A push does NOT advance the round".
 
+## The testing gateway — public domains for tailnet boxes
+
+A box on the tailnet with no public IP can still have a real HTTPS domain: the shared testing
+gateway holds a wildcard record for `*.gtest.commoninternet.net` and forwards by SNI.
+
+```bash
+python3 engine/tools/gateway-domain.py add myapp     # -> myapp.gtest.commoninternet.net -> this box
+python3 engine/tools/gateway-domain.py list
+python3 engine/tools/gateway-domain.py remove myapp
+```
+
+The admin password is `gateway.admin_password` in the secret store below; the tool reads it
+itself. **The gateway does not terminate TLS** — it proxies the encrypted stream, so your box
+serves the certificate for that name. Full detail, including how to get a cert and why hostname
+backends are refused, is in the `gateway-domain` skill (`skills/gateway-domain/SKILL.md`).
+
 ## Secrets — one encrypted store, never in git
 
 **Every credential on an orchestrator host lives in one sops+age encrypted file. Do not put a
diff --git a/skills/gateway-domain/SKILL.md b/skills/gateway-domain/SKILL.md
new file mode 100644
index 0000000..e58eb52
--- /dev/null
+++ b/skills/gateway-domain/SKILL.md
@@ -0,0 +1,129 @@
+---
+name: gateway-domain
+description: Give a tailnet box a real public HTTPS domain (<name>.gtest.commoninternet.net) by mapping it on the shared testing gateway. Use when an agent needs a publicly reachable URL for a box with no public IP — an OAuth callback, a webhook receiver, a demo link, an ACME challenge. Covers the add/remove tool, where the admin password lives, and the traps (your box serves the TLS cert, not the gateway; hostname backends are unremovable).
+---
+
+# Giving your box a public domain
+
+Your machine is on the tailnet with no public IP. You need a real HTTPS URL for it. The
+**testing gateway** already owns a wildcard DNS record, so every name under
+`*.gtest.commoninternet.net` resolves to it. Map your name to your tailnet IP and it forwards
+matching traffic to you.
+
+```bash
+python3 engine/tools/gateway-domain.py add myapp
+#   myapp.gtest.commoninternet.net  ->  100.84.190.30
+```
+
+That is the whole happy path. The backend defaults to **this box's own tailscale IP**, so run
+it on the machine that will serve the domain.
+
+```bash
+python3 engine/tools/gateway-domain.py list
+python3 engine/tools/gateway-domain.py add myapp                  # this box, port 443
+python3 engine/tools/gateway-domain.py add myapp 100.64.1.5       # another box
+python3 engine/tools/gateway-domain.py add myapp 100.64.1.5:8443  # backend not on 443
+python3 engine/tools/gateway-domain.py remove myapp
+```
+
+A bare label is expanded to `<label>.gtest.commoninternet.net`. Anything containing a dot is
+used verbatim, so you can map a domain you control elsewhere — but then **you** must point its
+DNS at the gateway (`49.13.156.72`); only `*.gtest.commoninternet.net` is pre-pointed.
+
+## The credential
+
+The admin password is in the orchestrator secret store, **not** in any repo:
+
+```bash
+python3 engine/secrets.py get gateway.admin_password
+```
+
+| | |
+|---|---|
+| store | `/secrets/store.yaml` (sops+age, `0600`, outside every git tree) |
+| keys | `gateway.admin_password`, `gateway.fqdn`, `gateway.admin_user` |
+| age key | `~/.config/sops/age/keys.txt` |
+| add/edit | `sops /secrets/store.yaml` |
+
+`gateway-domain.py` reads it itself — you should never need to handle the value. Do not copy it
+into a config file, an env file, or a repo. If you need it in a subprocess, use
+`engine/secrets.py exec-env` or `with-file` rather than writing a second copy.
+
+## The one thing that surprises people: your box serves the certificate
+
+The gateway does **not** terminate TLS. It reads the SNI name from the TLS handshake and proxies
+the still-encrypted bytes onward:
+
+```
+browser --TLS--> gateway :443 --reads SNI, proxies encrypted--> your box (tailnet)
+```
+
+So after mapping `myapp.gtest.commoninternet.net`, **your box** must serve a certificate valid
+for that exact name, on the backend port (443 unless you set one). The gateway has a cert for
+its own name only; it never sees your plaintext.
+
+Getting a cert on your box works: the gateway passes HTTP-01 challenges through on `:80` for
+mapped names, so ACME can complete normally. Add the domain here **first**, then request the
+cert — issuance needs the mapping to already exist.
+
+If you only need plain HTTP for a quick test, that also passes through on `:80`.
+
+## Traps
+
+**Backends must be a literal IPv4 address** (optionally `IP:port`). The tool refuses hostnames,
+and it is protecting you: the gateway's `validate_ip` accepts a hostname, but its
+`put_domain`/`remove_domain` only match lines whose backend is numeric. A hostname mapping can
+be written once and then **never updated or removed** through the admin UI — it becomes an
+orphan that only a hand-edit of `/var/lib/tunnel-gateway/tunnel_map.conf` on the box can clear.
+
+**Only ports 22/80/443 are open at the edge.** Mapping `IP:8443` changes which port on *your
+box* the gateway connects to; it does not open 8443 to the internet. Exposing a different
+*gateway* port is a config change in `nix/hosts/hetzner-test.nix`
+(`services.tunnelGateway.openTCPPorts`) **and** the Hetzner firewall in `nix/terraform/` — not
+something this tool can do.
+
+**Names are shared.** Anyone with the password can list, overwrite, or delete any mapping.
+Prefix yours with something recognisable and remove it when you are finished.
+
+**This is the test gateway.** Never point this tool at the production one. The box is long-lived
+(it is also the e2e and CI target), but its map is not sacred.
+
+## When the password stops working
+
+The e2e suite reseeds `.htpasswd` with a throwaway credential for the duration of a run and
+restores the previous file afterwards. If a run was killed part-way, the real one is still on
+the box at `/var/lib/tunnel-gateway/.htpasswd.e2e-backup`:
+
+```bash
+ssh -i /srv/gateway-coop/.secrets/id_admin root@49.13.156.72 \
+  'mv -f /var/lib/tunnel-gateway/.htpasswd.e2e-backup /var/lib/tunnel-gateway/.htpasswd'
+```
+
+To reseed it from the store instead — piping so the value never lands on disk:
+
+```bash
+python3 engine/secrets.py get gateway.admin_password | \
+  ssh -i /srv/gateway-coop/.secrets/id_admin root@49.13.156.72 \
+    'htpasswd -ci /var/lib/tunnel-gateway/.htpasswd admin \
+       && chgrp nginx /var/lib/tunnel-gateway/.htpasswd \
+       && chmod 640 /var/lib/tunnel-gateway/.htpasswd'
+```
+
+## If it still does not work
+
+Check in this order — most failures are the last two.
+
+1. `gateway-domain.py list` — is the mapping actually there?
+2. `getent hosts myapp.gtest.commoninternet.net` — should be `49.13.156.72`.
+3. Is your box reachable from the gateway on the tailnet? The gateway is `gateway-test-1`
+   (`100.91.44.90`), tagged `tag:testing-gateway`. If tailnet ACLs block it from reaching your
+   node, the mapping is correct and traffic still will not flow.
+4. **Is your service actually serving TLS for that name on the backend port?** A backend that
+   speaks plain HTTP on 443, or serves a cert for a different name, fails here and nowhere else.
+
+## The gateway itself
+
+`gtest.commoninternet.net` / `49.13.156.72` — a Hetzner `cx23` running NixOS, configured in the
+`tunnel-gateway-server` repo (`nix/hosts/hetzner-test.nix`), which `notplants-nix` pins as a
+submodule under `external/`. It also accepts reverse-SSH tunnels, for boxes not on the tailnet;
+that path is separate from this one and is not covered here.
diff --git a/tools/gateway-domain.py b/tools/gateway-domain.py
new file mode 100755
index 0000000..e350d35
--- /dev/null
+++ b/tools/gateway-domain.py
@@ -0,0 +1,216 @@
+#!/usr/bin/env python3
+"""gateway-domain.py — point a public domain at your box, via the testing gateway.
+
+WHAT THIS IS FOR. Your machine is on the tailnet but has no public IP, and you need a real
+HTTPS domain for it — an OAuth callback, a webhook receiver, a demo someone else can open.
+The testing gateway at gtest.commoninternet.net already holds a wildcard DNS record, so
+*.gtest.commoninternet.net resolves to it. Map your name to your tailnet IP and the gateway
+streams matching connections to you:
+
+    browser --TLS--> gateway :443 --reads SNI--> your box over the tailnet
+
+The gateway never terminates that TLS. It reads the SNI name and proxies the still-encrypted
+bytes, so **your box serves the certificate**, not the gateway. See SKILL.md.
+
+USAGE
+    tools/gateway-domain.py list
+    tools/gateway-domain.py add myapp                    # -> myapp.gtest.commoninternet.net -> this box's tailscale IP
+    tools/gateway-domain.py add myapp 100.64.1.5         # explicit backend
+    tools/gateway-domain.py add myapp 100.64.1.5:8443    # backend listening somewhere other than 443
+    tools/gateway-domain.py remove myapp
+
+CREDENTIALS. The admin password is in the orchestrator secret store, never here:
+
+    python3 engine/secrets.py get gateway.admin_password
+
+This script reads it from there itself. You should not need to handle the value.
+"""
+import argparse
+import base64
+import os
+import re
+import shutil
+import subprocess
+import sys
+import urllib.error
+import urllib.parse
+import urllib.request
+
+# engine/secrets.py lives one directory up.
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+import secrets as secret_store  # noqa: E402  (engine/secrets.py, not the stdlib module)
+
+TIMEOUT = 30
+
+
+def _cfg():
+    """Gateway coordinates. Only the password is genuinely secret; the rest lives
+    alongside it so there is one thing to change if the gateway ever moves."""
+    return (
+        secret_store.get("gateway.fqdn", "gtest.commoninternet.net"),
+        secret_store.get("gateway.admin_user", "admin"),
+        secret_store.get("gateway.admin_password"),
+    )
+
+
+def _request(method="GET", form=None):
+    fqdn, user, password = _cfg()
+    if not password:
+        sys.exit(
+            "no gateway.admin_password in the secret store.\n"
+            "  check: python3 engine/secrets.py list\n"
+            "  see:   skills/gateway-domain/SKILL.md"
+        )
+    url = f"https://{fqdn}/admin/"
+    data = urllib.parse.urlencode(form).encode() if form else None
+    req = urllib.request.Request(url, data=data, method=method)
+    token = base64.b64encode(f"{user}:{password}".encode()).decode()
+    req.add_header("Authorization", f"Basic {token}")
+    try:
+        with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
+            return r.status, r.read().decode("utf-8", "replace")
+    except urllib.error.HTTPError as e:
+        body = e.read().decode("utf-8", "replace")
+        if e.code == 401:
+            sys.exit(
+                f"401 from {url} — the stored password is not what the gateway expects.\n"
+                "The e2e suite reseeds .htpasswd while it runs and restores it afterwards;\n"
+                "if a run was interrupted, /var/lib/tunnel-gateway/.htpasswd.e2e-backup is\n"
+                "still on the box. See SKILL.md, 'When the password stops working'."
+            )
+        sys.exit(f"HTTP {e.code} from {url}\n{body[:600]}")
+    except urllib.error.URLError as e:
+        sys.exit(f"cannot reach {url}: {e.reason}")
+
+
+# The admin app answers 200 and renders a red <p> for a rejected value, rather than
+# using a status code. Checking only the status would report success on a no-op.
+_ERR = re.compile(r'<p style="color:red">(.*?)</p>', re.S)
+
+# Backends are restricted to a literal IPv4 address, optionally with a port.
+#
+# The gateway would accept a hostname -- its validate_ip is ^[a-zA-Z0-9_.-]+$ -- but its
+# put_domain/remove_domain match existing lines with ^<domain>\s+[\d.:]+; which only ever
+# matches a numeric backend. A hostname mapping can therefore be written once and then never
+# updated or removed through the admin UI: it becomes an orphan only a hand-edit of
+# /var/lib/tunnel-gateway/tunnel_map.conf can clear. Refuse to create one.
+_BACKEND = re.compile(r"^(\d{1,3}(?:\.\d{1,3}){3})(?::(\d{1,5}))?$")
+
+
+def _validate_backend(backend):
+    m = _BACKEND.match(backend)
+    if not m:
+        sys.exit(
+            f"backend must be an IPv4 address, optionally IP:port -- got {backend!r}.\n"
+            "Hostnames are rejected on purpose: the gateway can store one but cannot\n"
+            "remove it again. Use the tailnet IP (tailscale ip -4 on the target box)."
+        )
+    ip, port = m.group(1), m.group(2)
+    if any(int(o) > 255 for o in ip.split(".")):
+        sys.exit(f"not a valid IPv4 address: {ip}")
+    if port is not None and not (1 <= int(port) <= 65535):
+        sys.exit(f"port out of range: {port}")
+    return backend
+
+
+def _check(body):
+    m = _ERR.search(body)
+    if m:
+        sys.exit(f"gateway rejected the request: {m.group(1).strip()}")
+
+
+def _parse_domains(body):
+    """Pull the Domain Mappings table out of the admin page."""
+    section = body.split("Domain Mappings", 1)[-1].split("Port Mappings", 1)[0]
+    return re.findall(r"<td>\s*(.*?)\s*</td>\s*<td>\s*(.*?)\s*</td>", section, re.S)
+
+
+def _tailscale_ip():
+    exe = shutil.which("tailscale") or "/run/current-system/sw/bin/tailscale"
+    try:
+        out = subprocess.run([exe, "ip", "-4"], capture_output=True, text=True, timeout=15)
+    except (OSError, subprocess.SubprocessError) as e:
+        sys.exit(f"could not run tailscale to detect this box's IP ({e}); pass the backend explicitly")
+    ip = out.stdout.strip().splitlines()[0].strip() if out.stdout.strip() else ""
+    if not ip:
+        sys.exit(
+            "tailscale reported no IPv4 address — is this box on the tailnet?\n"
+            f"  {exe} status\n"
+            "Or pass the backend explicitly: gateway-domain.py add <name> <ip>"
+        )
+    return ip
+
+
+def _qualify(name, fqdn):
+    """A bare label becomes a subdomain of the gateway; anything with a dot is used as given."""
+    return name if "." in name else f"{name}.{fqdn}"
+
+
+def cmd_list(args):
+    _, body = _request()
+    rows = _parse_domains(body)
+    if not rows:
+        print("no domain mappings")
+        return
+    width = max(len(d) for d, _ in rows)
+    for domain, backend in rows:
+        print(f"  {domain:<{width}}  ->  {backend}")
+
+
+def cmd_add(args):
+    fqdn, _, _ = _cfg()
+    domain = _qualify(args.name, fqdn)
+    backend = args.backend or _tailscale_ip()
+    _validate_backend(backend)
+    _, body = _request("POST", {"domain": domain, "ip": backend})
+    _check(body)
+    # Re-read rather than trusting the POST body: app.py mutates its in-memory dict and
+    # renders that, so the response shows what it meant to do, not what landed on disk.
+    _, fresh = _request()
+    if not any(d == domain for d, _ in _parse_domains(fresh)):
+        sys.exit(f"POST returned 200 but {domain} is not in the mapping table — check `list`")
+    print(f"  {domain}  ->  {backend}")
+    print(f"\nYour box must now serve TLS for {domain} on the backend port (443 unless you")
+    print("set one). The gateway does not terminate TLS; it proxies the encrypted stream.")
+
+
+def cmd_remove(args):
+    fqdn, _, _ = _cfg()
+    domain = _qualify(args.name, fqdn)
+    # The admin app treats a backend of exactly "0" as delete.
+    _, body = _request("POST", {"domain": domain, "ip": "0"})
+    _check(body)
+    _, fresh = _request()
+    if any(d == domain for d, _ in _parse_domains(fresh)):
+        sys.exit(
+            f"{domain} is still in the mapping table after the delete.\n"
+            "If its backend is a hostname rather than an IP, the gateway cannot remove it:\n"
+            "edit /var/lib/tunnel-gateway/tunnel_map.conf on the box and reload nginx."
+        )
+    print(f"  removed {domain}")
+
+
+def main():
+    ap = argparse.ArgumentParser(
+        description="Map a public domain to your tailnet box via the testing gateway.",
+        epilog="Password comes from the secret store (gateway.admin_password); see skills/gateway-domain/SKILL.md.",
+    )
+    sub = ap.add_subparsers(dest="cmd", required=True)
+
+    sub.add_parser("list", help="show current domain mappings").set_defaults(fn=cmd_list)
+
+    a = sub.add_parser("add", help="point a domain at a backend")
+    a.add_argument("name", help="bare label (myapp) or a full domain")
+    a.add_argument("backend", nargs="?", help="IP or IP:port (default: this box's tailscale IP, port 443)")
+    a.set_defaults(fn=cmd_add)
+
+    r = sub.add_parser("remove", help="delete a domain mapping")
+    r.add_argument("name", help="bare label (myapp) or a full domain")
+    r.set_defaults(fn=cmd_remove)
+
+    args = ap.parse_args()
+    args.fn(args)
+
+
+if __name__ == "__main__":
+    main()