Merge Codex Remote Control support

A Tangled 404 renders 200 and carries no pull record, so tangled_comments.py's
"0 comment(s)" was indistinguishable from "no such pull". On 2026-08-22 that
produced a false report that a pull was filed and unreviewed, a retracted claim
that the queue tool was broken, and a reviewer's time spent disproving it.

tangled_comment_post.py never had the defect, because it MUST resolve a
subject-uri to function. So the read tool now requires the same identifier and
exits non-zero when the page does not yield one. Proven: #99999 and a
non-existent #428 exit 1 naming the cause, while a real pull still serves its
comments.

This is the structural fix over the vigilance fix, and the argument for it is
that both of us knew the countermeasure and neither applied it — the trap was
already written in one set of notes and in three of the reviewer's own reviews.

And the post tool's error named only the cookie, so anyone hitting it on a
phantom pull would re-auth, succeed, and still fail. It now names both causes in
likelihood order, the pull first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3LdmEL7CvCYTNpoBq1kce

README.md

@@ -178,6 +178,25 @@ preamble        = "set -a; . ./.env; set +a"   # shell run before launch (e.g. l
 active_re = "esc interrupt|thinking|running tool|preparing patch"
 limit_re  = "usage limit|limit reached"
 
+[backend.codex]                 # Codex TUI, exposed through Codex Remote Control
+bin             = "codex"
+preamble        = "{bin} remote-control start --json >/dev/null"
+remote_addr     = "unix://"    # connect the TUI to that daemon (one shared thread writer)
+flags           = "--dangerously-bypass-approvals-and-sandbox --dangerously-bypass-hook-trust --no-alt-screen"
+supports_resume = false         # thread-ID persistence is not implemented by this harness
+prompt_delivery = "arg"
+process_name    = "codex"
+footer_ui       = true
+log_grace       = 180
+submit_key      = "Enter"
+startup_prompt_re       = "Trusting the directory|Do you trust"
+startup_prompt_response = "1" # explicitly trust the agent's configured working directory
+startup_prompt_delay    = 2
+stall_idle      = 300
+active_re = "esc to interrupt|working|thinking|running|searching|exploring|implementing"
+limit_re  = "usage limit|limit reached|reached your .*limit|out of (credits|tokens)"
+fatal_re  = "connection is errored|not logged in|authentication required"
+
 [backend.demo]                  # a dependency-free backend for testing the harness mechanics
 bin             = "echo '[demo] {session} up'; exec sleep 1000000"
 prompt_delivery = "exec"        # {kickoff}=prompt file, {session}=session name, {model}=model
@@ -185,8 +204,20 @@ prompt_delivery = "exec"        # {kickoff}=prompt file, {session}=session name,
 
 For an `"arg"` backend the flag *templates* are configurable (so you can point at a non-claude
 CLI): `resume_flag` (default `--resume '{id}'`), `model_flag` (default `--model '{model}'`),
-`remote_control_flag` (default `--remote-control '{session}'`). A backend that sets `process_name`
+`remote_control_flag` (default `--remote-control '{session}'`), `remote_addr_flag` (default
+`--remote '{addr}'`), and `remote_cwd_flag` (default `-C {dir}` whenever `remote_addr` is set).
+A backend that sets `process_name`
 participates in backend-mismatch healing; one that doesn't (e.g. `demo`) never does.
+An optional `preamble` runs first and must succeed before the TUI starts; it supports `{bin}`,
+`{session}`, `{model}`, and `{dir}` templates. The Codex backend uses it to idempotently start the
+shared Remote Control daemon, then uses `remote_addr = "unix://"` to connect each tmux-hosted Codex
+TUI to that daemon. The accompanying `-C` is required because the daemon's process directory is
+independent of the agent's. This keeps the app-server as the sole thread-store writer while local
+and Remote clients share the session in the configured project directory.
+For an unattended TUI with a deterministic first-run gate, `startup_prompt_re` opts into one
+screen check after `startup_prompt_delay` seconds; on a match, the harness types the explicitly
+configured `startup_prompt_response` and `submit_key`. Codex uses this to trust the agent's
+configured project directory so its argv kickoff can proceed.
 
 ### `[[agent]]` — one block per agent
 
@@ -357,6 +388,44 @@ Run it by hand with `engine/agents.py up --config agents.toml`.
 
 ---
 
+## Tangled (the atproto forge) — the PR tools
+
+One tool per action, all reading `tangled.cookie` from the secret store (`--cookie-file` is a
+legacy fallback). They exist because Tangled has no client CLI: each action is an htmx POST to
+the appview that only a logged-in session can make.
+
+| tool | action |
+|---|---|
+| `tangled_pr.py` | open a pull (branch-based; `--target` is the branch it merges into) |
+| `tangled_pr_edit.py` | edit a pull's title/body, or `--show` the current one |
+| `tangled_pr_resubmit.py` | **advance a pull to a new round after you pushed a fixup**, and print the interdiff URL |
+| `tangled_pr_merge.py` | `--check` mergeability, then `--merge` or `--close` |
+| `tangled_comments.py` | read a pull's review comments (`--json` for machine use) |
+| `tangled_comment_post.py` | post a comment on a pull |
+| `tangled_repo.py` | create a repo as the bot (see also the `tangled-repo` skill) |
+
+**The trap `tangled_pr_resubmit.py` exists for:** pushing the branch does NOT update the pull.
+The appview keeps serving the patch it fetched when the pull was opened or last resubmitted,
+so reviewers read the pre-fixup code and no interdiff exists — with no warning, because the
+push itself succeeded. Always: push, resubmit, then reply with the printed interdiff URL. See
+`machine-docs/PR-WORKFLOW.md`, "A push does NOT advance the round".
+
+## The testing gateway — public domains for tailnet boxes
+
+A box on the tailnet with no public IP can still have a real HTTPS domain: the shared testing
+gateway holds a wildcard record for `*.gtest.commoninternet.net` and forwards by SNI.
+
+```bash
+python3 engine/tools/gateway-domain.py add myapp     # -> myapp.gtest.commoninternet.net -> this box
+python3 engine/tools/gateway-domain.py list
+python3 engine/tools/gateway-domain.py remove myapp
+```
+
+The admin password is `gateway.admin_password` in the secret store below; the tool reads it
+itself. **The gateway does not terminate TLS** — it proxies the encrypted stream, so your box
+serves the certificate for that name. Full detail, including how to get a cert and why hostname
+backends are refused, is in the `gateway-domain` skill (`skills/gateway-domain/SKILL.md`).
+
 ## Secrets — one encrypted store, never in git
 
 **Every credential on an orchestrator host lives in one sops+age encrypted file. Do not put a
@@ -385,23 +454,37 @@ python3 engine/secrets.py materialize tangled-session   # write a runtime file f
 sops /secrets/store.yaml                   # add/edit: decrypts to $EDITOR, re-encrypts on save
 ```
 
-**No second copies.** A secret must never be written to a second file "so something can read
-it" — copies drift from the store, get committed, and widen what a stray `grep` or an attacker
-finds. Our own code imports this module. Anything else gets the value at **run time**:
+**One home per secret — two shapes.**
 
-```sh
-# a group as environment variables — nothing touches the disk
-python3 engine/secrets.py exec-env cc_ci_testenv -- some-command
+*Values our code reads* live **in the store**; import this module and ask for them. Nothing is
+written to disk (`engine/.tangled-session` is gone — the tangled tools read `tangled.cookie`).
 
-# a consumer that insists on a path: 0600 file in a private tmpdir, deleted when the command exits
-python3 engine/secrets.py with-file ssh_keys.tangled-ed25519 -- ssh -i {} host
+*Secrets a third party reads from a fixed path* (ssh keys, a systemd `EnvironmentFile`, nix's
+`authKeyFile`, a TLS keypair) live as **real files in `/secrets/files/`, symlinked from the path
+the consumer expects**:
+
+```
+~/.ssh/tangled-ed25519                -> /secrets/files/tangled-ed25519
+/etc/ts-auth-key                      -> /secrets/files/ts-auth-key
+/srv/cc-ci/.testenv                   -> /secrets/files/cc-ci.testenv
 ```
 
-For **systemd**, wrap `ExecStart` in `exec-env` rather than using an `EnvironmentFile`: same
-effect, no plaintext at rest. The genuine exceptions are OS-level paths that are read before any
-of this exists — nix's `authKeyFile`, sshd host keys, and ssh client keys used by bare `git push`.
-Those stay where the OS expects them; do not also copy them into the store, or you have two
-sources of truth again.
+The consumer is unchanged and unaware; the file exists once, in one directory, at 0600. Do **not**
+also copy such a secret into `store.yaml` — that is two sources of truth again.
+
+For a one-off where neither shape fits, inject at run time and leave nothing behind:
+
+```sh
+python3 engine/secrets.py exec-env <group> -- some-command        # group as env vars
+python3 engine/secrets.py with-file <group.key> -- cmd -i {}      # 0600 file in a private
+                                                                  # tmpdir, deleted on exit
+```
+
+**The symlink exception: apps that rewrite their own credential file.** An app that refreshes an
+OAuth token by writing `auth.json` atomically (write-temp + rename) **replaces the symlink with a
+regular file**, silently splitting the home again. `~/.local/share/opencode/auth.json` is such a
+file, so it stays where it is and is deliberately *not* centralised. Before symlinking a secret,
+ask whether its owner ever writes it back.
 
 **Rules of thumb**
 
@@ -424,7 +507,7 @@ nix develop -c python3 agents.py selftest      # or run one command in it
 nix flake check                                # evaluate + build the devShell
 ```
 
-The agent CLIs themselves (`claude`, `opencode`) are **external, non-Nix tools** — install them
+The agent CLIs themselves (`claude`, `codex`, `opencode`) are **external, non-Nix tools** — install them
 per their own docs and make sure they are on `PATH` before launching live agents. The devShell
 documents this in its banner.
 
@@ -434,6 +517,17 @@ documents this in its banner.
 
 The `tests/` directory holds the harness's own test suite. One runner drives everything:
 
+- `tests/test_unit.py` — the harness: config load, kickoff, the phase machine, limit parsing,
+  waiting-until, the build-process detector.
+- `tests/test_tools.py` — the standalone tools (`tangled_pr.py`, `tangled_pr_close.py`,
+  `tools/gateway-domain.py`). **No network**: every HTTP boundary is injected as a fake `_fetch`.
+
+**Mock the seam you own, not the tool you depend on.** The build-detector tests used to fake
+`pgrep` and `ps` subprocess calls, so they passed on a host where neither was reachable and the
+real defect — empty output read as "no build running" — was invisible to every test. They now
+patch `_proc_descendants` and `_comms`, the functions this repo owns. A test that mocks a
+dependency proves the mock works.
+
 ```bash
 nix develop -c ./tests/run.sh      # unit tests always; live backend smokes when available
 # or just:  ./tests/run.sh         # (python3 + tmux must be on PATH)
@@ -447,13 +541,15 @@ What it runs:
   parsing, `WAITING-UNTIL` / stall parsing, and the per-backend activity detectors (claude +
   opencode footers). Always run; a failure fails the suite. Run them alone with
   `python3 -m unittest discover -s tests` (or `python3 tests/test_unit.py`).
-- **Live backend smokes** (`tests/smoke_claude.sh`, `tests/smoke_opencode.sh`) — each brings a
+- **Live backend smokes** (`tests/smoke_claude.sh`, `tests/smoke_codex.sh`,
+  `tests/smoke_opencode.sh`) — each brings a
   throwaway scratch project up **through `agents.py`** on a real backend, in a fully isolated
   sandbox (its own unique `session_prefix`, a temp `log_dir`, and — for opencode — a dedicated
   server on a non-default port `AOTEST_OC_PORT`, default `4097`), confirms the session attaches and
   `status` reports it RUNNING, then `down`s it and cleans up (no leftover sessions, port freed).
   Each **SKIPs gracefully** (exit 0) when its backend's binary or creds are unavailable. Useful env:
-  `CLAUDE_BIN` / `OPENCODE_BIN`, `AOTEST_MODEL`, `AOTEST_OC_PORT`, `AOTEST_OC_CREDS`.
+  `CLAUDE_BIN` / `CODEX_BIN` / `OPENCODE_BIN`, `AOTEST_MODEL`, `AOTEST_OC_PORT`,
+  `AOTEST_OC_CREDS`.
 - **Isolation sanity** — after the live runs, the runner asserts no `aotest-*` tmux sessions leaked
   and reports that any live sessions are untouched.
 

agents.example.toml

@@ -6,7 +6,7 @@
 #
 # This example is self-contained: its agents use a dependency-free `demo` backend (a shell that
 # just idles), so the whole project can be brought up and torn down with no external agent CLI —
-# see ./smoke.sh. The `claude` and `opencode` backends below are the real ones; point an agent at
+# see ./smoke.sh. The `claude`, `codex`, and `opencode` backends below are real; point an agent at
 # them with `backend = "claude"` for a live run.
 
 # ─────────────────────────── global watchdog cadence ───────────────────────────
@@ -42,11 +42,30 @@ supports_resume = true
 prompt_delivery = "arg"
 process_name    = "claude"       # used for backend-mismatch healing
 submit_key      = "Enter"
+startup_prompt_re       = "Trusting the directory|Do you trust"
+startup_prompt_response = "1"
+startup_prompt_delay    = 2
 stall_idle      = 300
 active_re = "esc to interrupt|Running tool|⠇|⠙|· \\d+"
 limit_re  = "spend limit|usage limit|limit reached|reached your .*limit|out of (credits|tokens)"
 fatal_re  = "redacted_thinking|blocks cannot be modified|cannot be modified"
 
+[backend.codex]                  # Codex TUI + the shared Remote Control app-server daemon
+bin             = "codex"
+preamble        = "{bin} remote-control start --json >/dev/null"
+remote_addr     = "unix://"     # make the TUI a client of the daemon; it must not own a second writer
+flags           = "--dangerously-bypass-approvals-and-sandbox --dangerously-bypass-hook-trust --no-alt-screen"
+supports_resume = false          # the harness does not yet persist Codex thread IDs
+prompt_delivery = "arg"
+process_name    = "codex"
+footer_ui       = true
+log_grace       = 180
+submit_key      = "Enter"
+stall_idle      = 300
+active_re = "esc to interrupt|working|thinking|running|searching|exploring|implementing"
+limit_re  = "usage limit|limit reached|reached your .*limit|out of (credits|tokens)"
+fatal_re  = "connection is errored|not logged in|authentication required"
+
 [backend.opencode]               # the real opencode backend (a TUI; prompt typed after connect)
 bin             = "opencode"
 attach          = "{bin} attach {server} --dir {dir}"

agents.py

@@ -349,15 +349,45 @@ def start_agent(cfg, agent, *, force=False):
         if backend.get("remote_control"):
             parts.append(_render_template(backend.get("remote_control_flag",
                                           "--remote-control '{session}'"), {"session": session}))
+        if backend.get("remote_addr"):
+            remote_values = {
+                "addr": backend["remote_addr"], "session": session, "model": model,
+                "dir": shlex.quote(cwd),
+            }
+            parts.append(_render_template(
+                backend.get("remote_addr_flag", "--remote '{addr}'"), remote_values))
+            remote_cwd_flag = backend.get("remote_cwd_flag", "-C {dir}")
+            if remote_cwd_flag:
+                parts.append(_render_template(remote_cwd_flag, remote_values))
         if model:
             parts.append(_render_template(backend.get("model_flag", "--model '{model}'"), {"model": model}))
         if backend.get("flags"):
             parts.append(backend["flags"])
         parts.append(f"\"$(cat '{kf}')\"")
         cmd = " ".join(p for p in parts if p)
+        # Some interactive CLIs need an idempotent service prepared before their TUI starts.
+        # The Codex backend starts the shared app-server here, while remote_addr above makes the
+        # pane's TUI a client of that daemon so both local and Remote UIs share one thread writer.
+        if backend.get("preamble"):
+            preamble = _render_template(backend["preamble"], {
+                "bin": backend["bin"], "session": session, "model": model,
+                "dir": shlex.quote(cwd),
+            })
+            cmd = f"{preamble} && {cmd}"
         log(f"starting {session} ({agent['backend']}, kind={agent['kind']}, phase={pid}, "
             f"model={model or 'default'}{', resume' if rid else ''})")
         new_session(session, cwd, cmd, log_path)
+        # An unattended TUI may present a deterministic first-run gate before it processes the
+        # prompt already supplied on argv (Codex asks whether to trust a new project directory).
+        # Backends opt in with both the screen regex and response; nothing is accepted implicitly.
+        startup_re = backend.get("startup_prompt_re")
+        if startup_re:
+            time.sleep(int(backend.get("startup_prompt_delay", 2)))
+            if re.search(startup_re, capture_pane(session, 40), re.I):
+                _run(["tmux", "send-keys", "-t", TP(session), "-l", "--",
+                      str(backend.get("startup_prompt_response", ""))])
+                _run(["tmux", "send-keys", "-t", TP(session),
+                      backend.get("submit_key", "Enter")])
 
 def start_service(cfg, svc):
     session = svc["session"]
@@ -481,16 +511,37 @@ DEFAULT_BUILD_PROCS_RE = (r"^(cargo|cargo-llvm-cov|cargo-mutants|cargo-nextest|n
 def _proc_descendants(roots):
     """All descendant PIDs of the given root PIDs (BFS via pgrep -P), EXCLUDING the roots themselves —
     the pane root is the claude process whose args embed the prompt (which mentions cargo/rustc/…), so
-    matching it would false-positive; we only inspect its real child processes."""
-    roots = [p for p in roots if str(p).isdigit()]
+    matching it would false-positive; we only inspect its real child processes.
+
+    Reads /proc directly rather than shelling out to `pgrep -P`. WHY (2026-08-21): pgrep is not
+    installed on every host we run on, and a missing binary makes `subprocess.run(shell=True)`
+    return rc=127 with EMPTY stdout — indistinguishable from "this process has no children". That
+    made this function return an empty set on such a host, which made `_build_running` always False,
+    which let the stall detector reboot an agent in the middle of a build. Reading /proc removes the
+    dependency and cannot fail silently in that direction.
+    (Note the repo already bans `pgrep -f` for a DIFFERENT reason — it self-matches. Two distinct
+    traps, same tool.)"""
+    roots = [str(p) for p in roots if str(p).isdigit()]
+
+    children = {}
+    for entry in os.listdir("/proc"):
+        if not entry.isdigit():
+            continue
+        try:
+            with open(f"/proc/{entry}/stat") as fh:
+                fields = fh.read().rsplit(") ", 1)[-1].split()   # rsplit: comm may contain spaces
+            ppid = fields[1]
+        except (OSError, IndexError):
+            continue                                             # process exited mid-scan
+        children.setdefault(ppid, []).append(entry)
+
     seen, stack = set(), list(roots)
     while stack:
         pid = stack.pop()
         if pid in seen:
             continue
         seen.add(pid)
-        c = subprocess.run(f"pgrep -P {pid}", shell=True, capture_output=True, text=True)
-        stack += [p for p in c.stdout.split() if p.isdigit()]
+        stack += children.get(pid, [])
     return seen - set(roots)
 
 def _build_running(cfg, agent):
@@ -507,9 +558,25 @@ def _build_running(cfg, agent):
         rx = re.compile(cfg["watchdog"].get("build_procs_re", DEFAULT_BUILD_PROCS_RE))
     except re.error:
         rx = re.compile(DEFAULT_BUILD_PROCS_RE)
-    ps = subprocess.run("ps -o comm= -p " + ",".join(sorted(kids)),
-                        shell=True, capture_output=True, text=True)
-    return any(rx.match(c.strip()) for c in ps.stdout.splitlines() if c.strip())
+    # Read /proc/<pid>/comm rather than `ps -o comm=`: ps is absent on some hosts we run on, and
+    # an absent binary yields EMPTY stdout, which this function cannot tell from "no build is
+    # running" — so the stall detector would reboot an agent mid-build. Same class as the pgrep
+    # dependency removed from _proc_descendants above; both were invisible because the unit tests
+    # MOCKED these subprocess calls, so the suite exercised the fake and never the dependency.
+    return any(rx.match(c) for c in _comms(kids))
+
+def _comms(pids):
+    """The comm (process name) of each pid, skipping any that exited mid-scan. No external tools."""
+    out = []
+    for pid in sorted(pids):
+        try:
+            with open(f"/proc/{pid}/comm") as fh:
+                c = fh.read().strip()
+        except OSError:
+            continue
+        if c:
+            out.append(c)
+    return out
 
 def _done_nudge_msg(cfg, ph):
     """The DONE-nudge: prompts a stalled loop agent to finalize a built-but-unmarked phase."""
@@ -1148,6 +1215,17 @@ def watchdog_loop(cfg_path):
                 if session_alive(a["session"]):
                     limit_tick(cfg, a, capture_pane(a["session"], 40))
 
+        # ...and ADD one that appeared. wake_elapsed was built once at startup, and only ever
+        # shrank (below) — so a `wake` written into agents.toml while the watchdog is running was
+        # invisible to it, silently and forever. Both flat-file agents were given 30-min wakes on
+        # 2026-08-16 against a watchdog up since 2026-08-01: zero wakes fired in the 2 days since,
+        # and with watch="heal" (no stall-reboot) each finished turn parked them until a human
+        # looked. Seed a new wake with a full interval so it fires on its own schedule, not at once.
+        for _n, _a in cfg["agents"].items():
+            if _a.get("wake") and _n not in wake_elapsed:
+                wake_elapsed[_n] = 0
+                log(f"wake registered mid-run for {_n} (interval={_a['wake'].get('interval', 3600)}s)")
+
         for name, el in list(wake_elapsed.items()):
             agent = cfg["agents"].get(name)
             # Config is re-read every tick, but wake_elapsed was built once at startup. If an agent's

flake.nix

@@ -13,7 +13,7 @@
     {
       # Reproducible devShell with the harness runtime deps. The driver itself is pure Python
       # stdlib (it needs tomllib, so python >= 3.11); the rest is what the agents/watchdog shell
-      # out to. Make the agent CLIs (claude / opencode) available on PATH separately — they are
+      # out to. Make the agent CLIs (claude / codex / opencode) available on PATH separately — they are
       # external, non-Nix tools; install them per their own docs, then `nix develop` here.
       devShells = forAllSystems (pkgs: {
         default = pkgs.mkShell {

secrets.py

@@ -6,9 +6,32 @@ remote URLs (`https://user:pass@host/...`, which `git remote -v` happily prints)
 in .env files, a private key at mode 0644. Anything in a repo is one `git add -A` away from
 a push. So: ONE encrypted file, OUTSIDE every git tree, and a helper every project uses.
 
-    store:    /secrets/store.yaml          sops+age ciphertext, mode 0600
-    age key:  ~/.config/sops/age/keys.txt      the ONLY plaintext secret on disk, 0600
-    outside git by construction — /secrets is not a repo and has no remote.
+    /secrets/store.yaml            sops+age ciphertext (0600) — values our code reads
+    /secrets/<project>/            PROJECT-SCOPED secrets (0600): everything belonging to one
+                                   project lives together, e.g. /secrets/lichen/,
+                                   /secrets/b1/, /secrets/notplants-orchestrator/
+    /secrets/files/                CROSS-PROJECT files (0600) SYMLINKED from the fixed path a
+                                   third party insists on: ~/.ssh keys, a systemd
+                                   EnvironmentFile, nix authKeyFile, a TLS keypair
+    ~/.config/sops/age/keys.txt    the age private key, 0600
+
+PROJECT SECRETS GO IN /secrets/<project>/ (operator, 2026-08-20). If a secret belongs to one
+project, it goes in that project's directory — not in files/, and not with the project name
+baked into the filename. `/secrets/lichen/test-pds.env`, not `/secrets/files/lichen-test-pds.env`.
+Reserve files/ for things genuinely shared across projects.
+
+WHY: a flat directory forces every name to carry its own scope, which nobody does consistently,
+and then nobody can answer "what does this project hold?" or "what do I revoke if this project is
+compromised?" without grepping. A directory answers both by listing. Put a README.md in the
+project directory saying what each file is, what consumes it, and what breaks if it is lost —
+the next person to read it will be doing so under time pressure.
+
+The symlink rule is unchanged and applies the same way: a consumer that insists on a fixed path
+gets a SYMLINK into /secrets/<project>/, so the file still exists exactly once.
+
+One home per secret: a value is in the store OR a file under /secrets, never both.
+/secrets is outside every git tree — not a repo, no remote — and outside /srv, which agents
+grep and walk constantly.
 
 USAGE (library):
     from secrets import get, get_group
@@ -18,20 +41,19 @@ USAGE (library):
 USAGE (CLI):
     python3 engine/secrets.py list                    # group/key names only, never values
     python3 engine/secrets.py get tangled.cookie      # value to stdout (careful in logs)
-    python3 engine/secrets.py materialize <name>      # write a runtime file a consumer needs
 
-NO SECOND COPIES. A secret must not be written to a second file "so something can read it" —
-copies drift from the store, get committed, and multiply what an attacker (or a careless
-`grep`) can find. Consumers read the store: our own code imports this module; anything else
-gets the value injected at RUN TIME and nothing is left at rest.
+NO SECOND COPIES. A secret is never written to a second file "so something can read it" —
+copies drift, get committed, and widen what a stray `grep` or an attacker finds. A consumer
+that insists on a path gets a SYMLINK into /secrets/files (see above), so the file still
+exists exactly once. For a one-off, inject at run time and leave nothing behind:
 
-    secrets.py exec-env cc_ci_testenv -- some-command    # group as env vars, no file
-    secrets.py with-file ssh_keys.tangled-ed25519 -- ssh -i {} host   # 0600 file in a private
-                                                        # tmpdir, deleted when the command exits
+    secrets.py exec-env <group> -- some-command       # group as env vars, no file
+    secrets.py with-file <group.key> -- cmd -i {}     # 0600 file in a private tmpdir,
+                                                      # deleted when the command exits
 
-For systemd, wrap ExecStart in `exec-env` instead of using an EnvironmentFile — same effect,
-no plaintext on disk. The few OS-level paths that genuinely cannot be taught this (nix's
-`authKeyFile`, sshd host keys) are the exception, and are noted in engine/README.md.
+Careful with symlinks: an app that rewrites its own credential file (an OAuth refresh writing
+auth.json via write-temp+rename) REPLACES the symlink with a regular file and silently splits
+the home again. Before symlinking, ask whether the owner ever writes it back.
 
 ADDING A SECRET:  sops /secrets/store.yaml   (opens decrypted in $EDITOR, re-encrypts on save)
 """
@@ -41,12 +63,21 @@ STORE = os.environ.get("AO_SECRETS_STORE", "/secrets/store.yaml")
 AGE_KEY = os.environ.get("SOPS_AGE_KEY_FILE", os.path.expanduser("~/.config/sops/age/keys.txt"))
 
 
+def _sops_bin():
+    """Resolve sops. It is on PATH under the systemd unit, but not always in an
+    interactive shell — fall back to the NixOS system profile before failing."""
+    return (shutil.which("sops")
+            or next((p for p in ("/run/current-system/sw/bin/sops",
+                                 "/run/wrappers/bin/sops") if os.path.exists(p)), None)
+            or "sops")
+
+
 def _load():
     """Decrypt the store. Fails loudly: a silent empty dict would look like 'no secrets'."""
     if not pathlib.Path(STORE).exists():
         sys.exit(f"no secret store at {STORE} — see engine/README.md (Secrets)")
     env = {**os.environ, "SOPS_AGE_KEY_FILE": AGE_KEY}
-    r = subprocess.run(["sops", "-d", "--output-type", "json", STORE],
+    r = subprocess.run([_sops_bin(), "-d", "--output-type", "json", STORE],
                        capture_output=True, text=True, env=env)
     if r.returncode != 0:
         sys.exit(f"cannot decrypt {STORE} (age key at {AGE_KEY}?): {r.stderr.strip()[:300]}")

skills/codeberg-pages/SKILL.md

@@ -0,0 +1,138 @@
+---
+name: codeberg-pages
+description: >-
+  Publish a static site to Codeberg Pages, including custom domains on the new
+  "git-pages" server. Use when deploying a site to Codeberg Pages, setting up or
+  debugging a codeberg.page / custom-domain deployment, wiring the DNS records
+  (A/AAAA, CNAME), the _git-pages-repository TXT authorization record, or the
+  deploy webhook — or when a custom domain serves a TLS error / never gets a
+  certificate. Covers the 2025→2026 migration off the old Pages Server v2
+  (.domains file) to git-pages (webhook + TXT authorization).
+---
+
+# Publishing a site to Codeberg Pages
+
+Codeberg Pages migrated from the old **Pages Server v2** (automatic deploy,
+`.domains` file) to the new **git-pages** server. On git-pages a deployment is
+**webhook-triggered** and a custom domain is authorized by a **TXT record**, not
+by a file in the repo. If you're following older docs or a `.domains`-based
+`deploy.sh`, that's why things silently don't work.
+
+> All values below were verified against the official docs
+> (<https://docs.codeberg.org/codeberg-pages/> and `.../using-custom-domain/`).
+> Codeberg changes these; re-check the docs if something behaves unexpectedly.
+
+## The mental model
+
+1. Static site content lives on a branch named **`pages`** (per-repo site) — push
+   your built site there.
+2. On git-pages, pushing alone does **not** deploy. A **webhook** on the repo,
+   pointed at the domain you want, is what triggers a deployment.
+3. A custom domain is bound to the repo by a **TXT authorization record**
+   (`_git-pages-repository.<domain>`) plus the normal A/AAAA/CNAME records.
+4. TLS (Let's Encrypt) is issued **only after the first successful deployment**.
+   Before that, browsers show a TLS error — that is expected, not a bug.
+
+## Basic deploy (no custom domain, `*.codeberg.page`)
+
+- Put the site on a `pages` branch and push it.
+- Add a webhook: repo **Settings → Webhooks → Forgejo**, Target URL
+  `https://<username>.codeberg.page/<repository>/`, **Branch filter: `pages`**.
+- (User/org site: name the repo `pages` and use Target URL
+  `https://<username>.codeberg.page/`.)
+
+## Custom domain setup (git-pages)
+
+Do all four. Missing #2 or #3 is the usual cause of "DNS looks right but the site
+won't serve / no certificate."
+
+### 1. DNS: point the domain at Codeberg
+
+Exact values (verify against the docs — Codeberg has changed IPs before):
+
+- **Apex domain** (`example.org`):
+  - `A`    → `217.197.84.141`
+  - `AAAA` → `2a0a:4580:103f:c0de::2`
+- **Subdomain** (`www.example.org`, `foo.example.org`):
+  - `CNAME` → `codeberg.page.`  ← **note the trailing dot.**
+
+**Trailing-dot trap:** in a zone file / most DNS UIs, a CNAME target *without* a
+trailing dot is treated as relative and the zone is appended — e.g. entering
+`codeberg.page` (or an old `<user>.codeberg.page`) can resolve to
+`codeberg.page.example.org.`, which is broken. Always use the fully-qualified
+`codeberg.page.` with the dot. (ALIAS/ANAME works where CNAME isn't allowed, but
+conflicts with DNSSEC-signed zones.)
+
+### 2. TXT authorization record (this is how git-pages maps domain → repo)
+
+Create one **per domain** you serve:
+
+```
+_git-pages-repository.example.org.   TXT   "https://codeberg.org/<user>/<repo>.git"
+```
+
+- Name: the `_git-pages-repository.` prefix on the exact domain (including each
+  subdomain you serve — apex and `www` each need their own if both are used).
+- Value: the **HTTPS clone URL** of the repo, ending in `.git`.
+- (If you deploy via **Forgejo Actions** instead of a webhook, the record is
+  `_git-pages-forge-allowlist.<domain>` with the same clone-URL value.)
+
+### 3. Deploy webhook (per domain)
+
+Repo **Settings → Webhooks → Forgejo**:
+
+- **Target URL:** the domain itself, and **`http://` (not `https://`) for the
+  first deployment** — this is documented, not a mistake (the cert doesn't exist
+  yet). One webhook per domain, e.g. `http://example.org`, `http://foo.example.org`.
+- **Branch filter:** `pages`.
+- After the first successful deploy and cert issuance, you may switch the Target
+  URLs to `https://`.
+
+### 4. Trigger the first deploy
+
+**Push to the `pages` branch** (re-run your deploy script / `git push origin pages`).
+The push fires the webhook, git-pages pulls and deploys, then requests a
+Let's Encrypt certificate.
+
+- **Do NOT rely on the webhook "Test delivery" button** — the official docs say it
+  fails by design and is not a valid way to verify or trigger a deploy. Verify by
+  pushing and then checking the webhook's recent-deliveries log, or just load the
+  site. (This corrects a common misconception that "Test delivery" triggers a deploy.)
+
+## The `.domains` file is obsolete
+
+Under the old Pages Server v2, a `.domains` file in the branch listed the domains
+and did apex-vs-alias redirects. On git-pages it is **no longer used** — authorization
+comes from the TXT record. It's harmless to leave, but you can delete it (and drop any
+`.domains` handling from `deploy.sh`). Bonus: on git-pages each domain gets its **own**
+deployment, so a second domain serves the site directly instead of 301-redirecting to
+the primary as the old `.domains` system did.
+
+## TLS / certificate notes
+
+- A cert is issued **only after the first successful webhook deployment**. A TLS
+  error before that is expected.
+- If the domain has **CAA records**, they must allow Let's Encrypt (including the
+  staging issuer) or the cert request is refused.
+- Cert still never issues after a successful deploy → confirm the `_git-pages-repository`
+  TXT value exactly matches the repo's HTTPS `.git` URL, and that the webhook Target
+  URL matches the domain.
+
+## Quick troubleshooting checklist
+
+- Browser TLS error, no cert → no successful deploy yet. Check webhook deliveries;
+  push to `pages`; confirm webhook Target URL used `http://` for the first deploy.
+- "DNS is correct but site won't serve" → missing `_git-pages-repository` TXT, or
+  missing/mis-branch-filtered webhook.
+- CNAME resolves to `codeberg.page.<yourzone>` → missing trailing dot; set target to
+  `codeberg.page.`.
+- CAA present → ensure Let's Encrypt is allowed.
+- Old `.domains` behavior expected (redirects) → gone on git-pages; each domain now
+  deploys independently.
+
+## Sources
+
+- Codeberg Pages: <https://docs.codeberg.org/codeberg-pages/>
+- Using custom domains: <https://docs.codeberg.org/codeberg-pages/using-custom-domain/>
+- pages-server (now in maintenance, superseded by git-pages):
+  <https://codeberg.org/Codeberg/pages-server>

skills/gateway-domain/SKILL.md

@@ -0,0 +1,157 @@
+---
+name: gateway-domain
+description: Give a tailnet box a real public HTTPS domain (<name>.gtest.commoninternet.net) by mapping it on the shared testing gateway. Use when an agent needs a publicly reachable URL for a box with no public IP — an OAuth callback, a webhook receiver, a demo link, an ACME challenge. Covers the add/remove tool, where the admin password lives, and the two things that silently break it: your box must carry a tailnet tag the ACL allows (tag:notplants-test-server or tag:orchestrator) or the gateway cannot reach it, and your box serves the TLS cert rather than the gateway.
+---
+
+# Giving your box a public domain
+
+Your machine is on the tailnet with no public IP. You need a real HTTPS URL for it. The
+**testing gateway** already owns a wildcard DNS record, so every name under
+`*.gtest.commoninternet.net` resolves to it. Map your name to your tailnet IP and it forwards
+matching traffic to you.
+
+```bash
+python3 engine/tools/gateway-domain.py add myapp
+#   myapp.gtest.commoninternet.net  ->  100.84.190.30
+```
+
+That is the whole happy path. The backend defaults to **this box's own tailscale IP**, so run
+it on the machine that will serve the domain.
+
+## Your box needs a tailnet tag the ACL allows
+
+The gateway can only open connections to nodes the tailnet ACL lets it reach. Two tags qualify:
+
+| tag | who |
+|---|---|
+| `tag:notplants-test-server` | test servers — the usual case |
+| `tag:orchestrator` | orchestrator boxes (added 2026-08-20, verified end to end) |
+
+Without one of them the gateway accepts your mapping and then simply never connects — which
+looks like a broken gateway and is not one. Check before you start:
+
+```bash
+tailscale status --json | jq -r '.Self.Tags[]?'
+```
+
+If neither tag is listed, add one to that node in the Tailscale admin (a node's tags are set
+when it is authenticated, so this may mean re-authenticating it), or map a backend that already
+has one. `gateway-domain.py` warns when the node it is about to map carries neither, but it
+cannot see the tags of a backend you name explicitly — that one is on you.
+
+The gateway itself is tagged `tag:testing-gateway`; that is the other half of the same ACL rule.
+
+## The commands
+
+```bash
+python3 engine/tools/gateway-domain.py list
+python3 engine/tools/gateway-domain.py add myapp                  # this box, port 443
+python3 engine/tools/gateway-domain.py add myapp 100.64.1.5       # another box
+python3 engine/tools/gateway-domain.py add myapp 100.64.1.5:8443  # backend not on 443
+python3 engine/tools/gateway-domain.py remove myapp
+```
+
+A bare label is expanded to `<label>.gtest.commoninternet.net`. Anything containing a dot is
+used verbatim, so you can map a domain you control elsewhere — but then **you** must point its
+DNS at the gateway (`49.13.156.72`); only `*.gtest.commoninternet.net` is pre-pointed.
+
+## The credential
+
+The admin password is in the orchestrator secret store, **not** in any repo:
+
+```bash
+python3 engine/secrets.py get gateway.admin_password
+```
+
+| | |
+|---|---|
+| store | `/secrets/store.yaml` (sops+age, `0600`, outside every git tree) |
+| keys | `gateway.admin_password`, `gateway.fqdn`, `gateway.admin_user` |
+| age key | `~/.config/sops/age/keys.txt` |
+| add/edit | `sops /secrets/store.yaml` |
+
+`gateway-domain.py` reads it itself — you should never need to handle the value. Do not copy it
+into a config file, an env file, or a repo. If you need it in a subprocess, use
+`engine/secrets.py exec-env` or `with-file` rather than writing a second copy.
+
+## The one thing that surprises people: your box serves the certificate
+
+The gateway does **not** terminate TLS. It reads the SNI name from the TLS handshake and proxies
+the still-encrypted bytes onward:
+
+```
+browser --TLS--> gateway :443 --reads SNI, proxies encrypted--> your box (tailnet)
+```
+
+So after mapping `myapp.gtest.commoninternet.net`, **your box** must serve a certificate valid
+for that exact name, on the backend port (443 unless you set one). The gateway has a cert for
+its own name only; it never sees your plaintext.
+
+Getting a cert on your box works: the gateway passes HTTP-01 challenges through on `:80` for
+mapped names, so ACME can complete normally. Add the domain here **first**, then request the
+cert — issuance needs the mapping to already exist.
+
+If you only need plain HTTP for a quick test, that also passes through on `:80`.
+
+## Traps
+
+**Backends must be a literal IPv4 address** (optionally `IP:port`). The tool refuses hostnames,
+and it is protecting you: the gateway's `validate_ip` accepts a hostname, but its
+`put_domain`/`remove_domain` only match lines whose backend is numeric. A hostname mapping can
+be written once and then **never updated or removed** through the admin UI — it becomes an
+orphan that only a hand-edit of `/var/lib/tunnel-gateway/tunnel_map.conf` on the box can clear.
+
+**Only ports 22/80/443 are open at the edge.** Mapping `IP:8443` changes which port on *your
+box* the gateway connects to; it does not open 8443 to the internet. Exposing a different
+*gateway* port is a config change in `nix/hosts/hetzner-test.nix`
+(`services.tunnelGateway.openTCPPorts`) **and** the Hetzner firewall in `nix/terraform/` — not
+something this tool can do.
+
+**Names are shared.** Anyone with the password can list, overwrite, or delete any mapping.
+Prefix yours with something recognisable and remove it when you are finished.
+
+**This is the test gateway.** Never point this tool at the production one. The box is long-lived
+(it is also the e2e and CI target), but its map is not sacred.
+
+## When the password stops working
+
+The e2e suite reseeds `.htpasswd` with a throwaway credential for the duration of a run and
+restores the previous file afterwards. If a run was killed part-way, the real one is still on
+the box at `/var/lib/tunnel-gateway/.htpasswd.e2e-backup`:
+
+```bash
+ssh -i /srv/gateway-coop/.secrets/id_admin root@49.13.156.72 \
+  'mv -f /var/lib/tunnel-gateway/.htpasswd.e2e-backup /var/lib/tunnel-gateway/.htpasswd'
+```
+
+To reseed it from the store instead — piping so the value never lands on disk:
+
+```bash
+python3 engine/secrets.py get gateway.admin_password | \
+  ssh -i /srv/gateway-coop/.secrets/id_admin root@49.13.156.72 \
+    'htpasswd -ci /var/lib/tunnel-gateway/.htpasswd admin \
+       && chgrp nginx /var/lib/tunnel-gateway/.htpasswd \
+       && chmod 640 /var/lib/tunnel-gateway/.htpasswd'
+```
+
+## If it still does not work
+
+Check in this order — most failures are the last two.
+
+1. `gateway-domain.py list` — is the mapping actually there?
+2. `getent hosts myapp.gtest.commoninternet.net` — should be `49.13.156.72`.
+3. **Does your node carry `tag:notplants-test-server` or `tag:orchestrator`?**
+   (`tailscale status --json | jq -r '.Self.Tags[]?'`) This is the single most common cause.
+   The gateway is `gateway-test-1` (`100.91.44.90`), tagged `tag:testing-gateway`; the ACL
+   pairs that with the tags above, so a backend with neither is unreachable no matter how
+   correct the mapping looks. Quick check from the gateway itself:
+   `ssh root@49.13.156.72 'timeout 5 bash -c "echo > /dev/tcp/<your-tailnet-ip>/<port>"'`
+4. **Is your service actually serving TLS for that name on the backend port?** A backend that
+   speaks plain HTTP on 443, or serves a cert for a different name, fails here and nowhere else.
+
+## The gateway itself
+
+`gtest.commoninternet.net` / `49.13.156.72` — a Hetzner `cx23` running NixOS, configured in the
+`tunnel-gateway-server` repo (`nix/hosts/hetzner-test.nix`), which `notplants-nix` pins as a
+submodule under `external/`. It also accepts reverse-SSH tunnels, for boxes not on the tailnet;
+that path is separate from this one and is not covered here.

tangled_comment_post.py

@@ -0,0 +1,102 @@
+#!/usr/bin/env python3
+"""Post a top-level comment on a Tangled pull as the bot.
+
+WHY: tangled_comments.py only READS. Replying to the operator's review notes on a
+~130-pull stack by hand is not viable, and there is no client API — the appview
+takes an htmx form POST /comment with three fields:
+    subject-uri      at://{did}/sh.tangled.repo.pull/{rkey}   (the pull record)
+    pull-round-idx   which round the comment hangs off
+    body             markdown
+Both hidden fields are only discoverable from the pull page, so this fetches the
+page, scrapes them, and posts. Same session-cookie auth as tangled_pr.py.
+
+USAGE:
+  tangled_comment_post.py --owner notplants-bot.bsky.social --repo lichen.page.review \
+      --pull 76 --body "..."                  # or --body-file reply.md
+  tangled_comment_post.py ... --pull 76 --round 1 --body "..."   # pin to a round
+  tangled_comment_post.py ... --pull 76 --body "..." --dry-run
+"""
+import argparse, os, re, sys, urllib.error, urllib.parse, urllib.request
+
+BASE = "https://tangled.org"
+
+
+def load_cookie():
+    sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+    import secrets as _store
+    c = _store.get("tangled.cookie")
+    if not c:
+        sys.exit("no tangled.cookie in the secret store — add it with: sops /secrets/store.yaml")
+    return c
+
+
+def get(url, cookie):
+    req = urllib.request.Request(url, headers={"Cookie": cookie, "User-Agent": "tangled-pr-bot"})
+    return urllib.request.urlopen(req, timeout=60).read().decode()
+
+
+def scrape_form(doc):
+    """The comment form's two hidden fields. Several rounds' forms can be on the page;
+    take the last (highest round), which is the one the UI shows open."""
+    uris = re.findall(r'name="subject-uri"[^>]*value="([^"]+)"', doc)
+    idxs = re.findall(r'name="pull-round-idx"[^>]*value="([^"]+)"', doc)
+    if not uris or not idxs:
+        # Two causes, and naming only one sends the reader in the wrong direction: a
+        # reviewer hitting this on a phantom pull will re-auth, succeed, and still fail.
+        sys.exit(
+            "could not find the comment form. Two causes, in likelihood order:\n"
+            "  1. THE PULL DOES NOT EXIST — a 404 page renders 200 and carries no form.\n"
+            "     Check the branch on the remote, not the pull number.\n"
+            "  2. the cookie has expired."
+        )
+    return uris[-1], idxs[-1]
+
+
+def main():
+    ap = argparse.ArgumentParser(description="post a comment on a Tangled pull")
+    ap.add_argument("--owner", required=True)
+    ap.add_argument("--repo", required=True)
+    ap.add_argument("--pull", required=True, type=int)
+    ap.add_argument("--round", type=int, default=None, help="round to attach to (default: latest)")
+    ap.add_argument("--body")
+    ap.add_argument("--body-file")
+    ap.add_argument("--dry-run", action="store_true")
+    a = ap.parse_args()
+
+    if not a.body and not a.body_file:
+        sys.exit("need --body or --body-file")
+    body = a.body if a.body else open(a.body_file).read()
+
+    cookie = load_cookie()
+    page = f"{BASE}/{a.owner}/{a.repo}/pulls/{a.pull}"
+    if a.round is not None:
+        page += f"/round/{a.round}"
+    subject_uri, round_idx = scrape_form(get(page, cookie))
+
+    if a.dry_run:
+        print(f"would post to {page}\n  subject-uri={subject_uri}\n  round={round_idx}\n---\n{body}")
+        return
+
+    data = urllib.parse.urlencode(
+        {"subject-uri": subject_uri, "pull-round-idx": round_idx, "body": body}
+    ).encode()
+    req = urllib.request.Request(
+        f"{BASE}/comment",
+        data=data,
+        headers={
+            "Cookie": cookie,
+            "User-Agent": "tangled-pr-bot",
+            "Content-Type": "application/x-www-form-urlencoded",
+            "HX-Request": "true",
+            "Referer": page,
+        },
+    )
+    try:
+        resp = urllib.request.urlopen(req, timeout=60)
+    except urllib.error.HTTPError as e:
+        sys.exit(f"POST /comment failed: {e.code} {e.read().decode()[:400]}")
+    print(f"posted on pull #{a.pull} (round {round_idx}) — HTTP {resp.status}")
+
+
+if __name__ == "__main__":
+    main()

tangled_comments.py

@@ -0,0 +1,117 @@
+#!/usr/bin/env python3
+"""Read the review comments on a Tangled pull (top-level discussion comments).
+
+WHY: the appview has no read API for pull comments, and eyeballing the pull page
+HTML per comment is slow. This fetches the pull (or a specific round) and prints
+every top-level comment as author / time / body — so acting on operator review
+notes is one command, not a WebFetch guess.
+
+Same session-cookie auth as tangled_pr.py (cookie from the encrypted store,
+tangled.cookie). Read-only: it never posts.
+
+USAGE:
+  tangled_comments.py --owner notplants-bot.bsky.social --repo lichen.page.review --pull 75
+  tangled_comments.py ... --pull 75 --round 2        # a specific round's page
+  tangled_comments.py ... --pull 75 --json           # machine-readable
+"""
+import argparse, html, json, os, re, sys, urllib.request
+
+BASE = "https://tangled.org"
+
+
+def load_cookie():
+    """Cookie from the encrypted store (tangled.cookie) — see engine/README.md (Secrets)."""
+    sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+    import secrets as _store
+    c = _store.get("tangled.cookie")
+    if not c:
+        sys.exit("no tangled.cookie in the secret store — add it with: sops /secrets/store.yaml")
+    return c
+
+
+def fetch(url, cookie):
+    req = urllib.request.Request(url, headers={"Cookie": cookie, "User-Agent": "tangled-pr-bot"})
+    return urllib.request.urlopen(req, timeout=60).read().decode()
+
+
+def _text(fragment):
+    """Strip tags from a body fragment down to readable plaintext."""
+    t = re.sub(r"(?is)<br\s*/?>", "\n", fragment)
+    t = re.sub(r"(?is)</p\s*>", "\n\n", t)
+    t = re.sub(r"(?is)<li[^>]*>", "\n- ", t)
+    t = re.sub(r"(?s)<[^>]+>", "", t)
+    return html.unescape(t).strip()
+
+
+def parse_comments(doc):
+    """Return [{cid, author, when, iso, uri, body}] in document order.
+
+    Anchored on the per-comment header block the appview emits:
+      id="comment-header-{cid}" ... <a href="/{handle}">{handle}</a> ...
+      <time datetime="{iso}">{when}</time> ... class="...comment-body"><div class="prose...">{body}</div>
+    """
+    out = []
+    heads = list(re.finditer(r'id="comment-header-([0-9a-z]+)"', doc))
+    for i, hm in enumerate(heads):
+        cid = hm.group(1)
+        # bound by the next comment header, not a fixed window: a comment's own
+        # reaction/button markup can push its body several KB past the header.
+        end = heads[i + 1].start() if i + 1 < len(heads) else len(doc)
+        seg = doc[hm.end(): end]
+        am = re.search(r'href="/([^"/]+)"[^>]*>\s*([^<]+?)\s*</a>', seg)
+        author = html.unescape(am.group(2)) if am else "?"
+        tm = re.search(r'<time datetime="([^"]+)"[^>]*>\s*([^<]+?)\s*</time>', seg)
+        iso = html.unescape(tm.group(1)) if tm else ""
+        when = html.unescape(tm.group(2)) if tm else ""
+        bm = re.search(r'comment-body">\s*<div class="prose[^"]*">(.*?)</div>\s*<div class="reactions', seg, re.S)
+        if not bm:
+            bm = re.search(r'comment-body">\s*<div class="prose[^"]*">(.*?)</div>', seg, re.S)
+        body = _text(bm.group(1)) if bm else ""
+        out.append({"cid": cid, "author": author, "when": when, "iso": iso,
+                    "uri": f"at://.../sh.tangled.feed.comment/{cid}", "body": body})
+    return out
+
+
+def main():
+    ap = argparse.ArgumentParser(description="read a Tangled pull's review comments")
+    ap.add_argument("--owner", required=True)
+    ap.add_argument("--repo", required=True)
+    ap.add_argument("--pull", required=True, type=int)
+    ap.add_argument("--round", type=int, default=None, help="a specific round page (default: latest)")
+    ap.add_argument("--json", action="store_true")
+    a = ap.parse_args()
+
+    cookie = load_cookie()
+    url = f"{BASE}/{a.owner}/{a.repo}/pulls/{a.pull}"
+    if a.round is not None:
+        url += f"/round/{a.round}"
+    doc = fetch(url, cookie)
+
+    # A 404 page renders 200 and carries no pull record, so "0 comments" used to be
+    # indistinguishable from "no such pull". On 2026-08-22 that cost a false report that
+    # #428 was filed and unreviewed, and a reviewer's time chasing it. The post tool never
+    # had the defect because it MUST resolve a subject-uri to work at all — so require the
+    # same identifier here, and refuse rather than print a header for a phantom.
+    if not re.search(r'name="subject-uri"[^>]*value="(at://[^"]+)"', doc):
+        sys.exit(
+            f"pull #{a.pull} does not exist in {a.owner}/{a.repo} "
+            f"(no pull record on {url}).\n"
+            "  This is NOT an auth failure: the page rendered, it simply carries no pull.\n"
+            "  A pull that exists always yields a subject-uri."
+        )
+
+    comments = parse_comments(doc)
+
+    if a.json:
+        print(json.dumps({"url": url, "count": len(comments), "comments": comments}, indent=2))
+        return
+    print(f"# pull #{a.pull}  ({url})  — {len(comments)} comment(s)\n")
+    for i, c in enumerate(comments, 1):
+        print(f"[{i}] {c['author']}  ·  {c['when']}  ({c['iso']})  #{c['cid']}")
+        for line in (c["body"] or "(empty)").splitlines():
+            print(f"    {line}")
+        print()
+
+
+if __name__ == "__main__":
+    main()

tangled_pr.py

@@ -17,10 +17,24 @@ COOKIE FILE (tools/.tangled-session, gitignored), a single line — paste the wh
 Cookie header value from the logged-in browser (both appview-* cookies):
   TANGLED_COOKIE=appview-session-v2=<...>; appview-accounts-v2=<...>
 
+RELIABILITY (rewritten 2026-08-21). This tool used to judge success ONLY by an
+HX-Redirect header on the POST response. A create that SUCCEEDED but answered without
+that header read as a failure, the caller retried, and Tangled grew duplicate pulls —
+that is exactly how #397, #398 and #399 were created for one branch. A response header
+describes what the server meant to say; it is not the artifact. So now:
+
+  * BEFORE posting, look for an existing OPEN pull for this source branch. If one
+    exists, refuse and name it — a duplicate cannot be created even if a caller retries.
+  * AFTER posting, confirm against the PULLS LIST, not the response: a new pull number
+    that did not exist before and whose page names this source branch IS the success,
+    with or without a redirect header.
+  * Only report failure when no such pull appeared. A false failure is worse than a
+    loud error here, because the caller's remedy is to retry.
+
 USAGE:
   tangled_pr.py --owner notplants-bot.bsky.social --repo lichen.page \
       --target main --source hardening-review \
-      [--fork did:plc:<forkRepoDid>] [--title "..."] [--body "..."]
+      [--fork did:plc:<forkRepoDid>] [--title "..."] [--body "..."] [--dry-run]
 """
 import argparse, os, sys, urllib.request, urllib.parse, urllib.error
 
@@ -41,6 +55,70 @@ def load_cookie(path=None):
         sys.exit("no tangled.cookie in the secret store — add it with: sops /secrets/store.yaml")
     return c
 
+
+# ── verification against the artifact, not the response ─────────────────────────
+# These are pure so they can be unit-tested without a network: see tests/test_tools.py.
+
+import re
+
+def pull_numbers(html):
+    """Every pull number linked from a pulls index page."""
+    return {int(n) for n in re.findall(r'/pulls/(\d+)\b', html or "")}
+
+def page_names_branch(html, branch):
+    """True if a pull's page mentions this source branch.
+
+    Substring rather than token: Tangled renders the branch inside markup we do not
+    control, and a false NEGATIVE here would resurrect the duplicate bug. A false
+    positive is caught by the caller, which only asks about pulls it just created.
+    """
+    return bool(branch) and branch in (html or "")
+
+def classify(before, after, branch_pages):
+    """Decide what a POST did, from the pulls list before and after it.
+
+    before/after: sets of pull numbers. branch_pages: {number: names_our_branch}.
+    Returns (verdict, pull_number) where verdict is:
+      "created"    a new pull naming our branch appeared
+      "unrelated"  new pulls appeared but none is ours (someone else was filing)
+      "none"       nothing new appeared — a real failure
+    """
+    fresh = sorted(after - before)
+    ours = [n for n in fresh if branch_pages.get(n)]
+    if ours:
+        return "created", max(ours)
+    if fresh:
+        return "unrelated", None
+    return "none", None
+
+def fetch(url, cookie, timeout=30):
+    """GET a page as the bot. Returns "" on any error: callers treat an unreadable page as
+    'cannot confirm', never as 'confirmed absent' — the difference is the duplicate bug."""
+    try:
+        r = urllib.request.Request(url, headers={"Cookie": cookie, "User-Agent": "tangled-pr-bot"})
+        return urllib.request.urlopen(r, timeout=timeout).read().decode("utf-8", "replace")
+    except Exception:
+        return ""
+
+def index_pulls(owner, repo, cookie, _fetch=fetch):
+    return pull_numbers(_fetch(f"{BASE}/{owner}/{repo}/pulls", cookie))
+
+def pull_names_branch(owner, repo, n, branch, cookie, _fetch=fetch):
+    return page_names_branch(_fetch(f"{BASE}/{owner}/{repo}/pulls/{n}", cookie), branch)
+
+def existing_pull_for(owner, repo, branch, cookie, numbers, scan=12, _fetch=fetch):
+    """The newest pull whose page names this branch, scanning back `scan` pulls.
+
+    Bounded on purpose: an unbounded scan would fetch hundreds of pages on every call.
+    Scanning the newest N is enough for the duplicate this guards against, which is a
+    retry seconds after the original. Returns None if none is found within the window —
+    which is 'not seen', not 'does not exist', and the post-check is the real backstop.
+    """
+    for n in sorted(numbers, reverse=True)[:scan]:
+        if pull_names_branch(owner, repo, n, branch, cookie, _fetch):
+            return n
+    return None
+
 def main():
     ap = argparse.ArgumentParser(description="file a Tangled PR via a reused session cookie")
     ap.add_argument("--owner", required=True)
@@ -51,12 +129,28 @@ def main():
     ap.add_argument("--title", default="")
     ap.add_argument("--body", default="")
     ap.add_argument("--cookie-file", default=None, help="legacy: read the cookie from this file instead of the store")
+    ap.add_argument("--dry-run", action="store_true", help="check for an existing pull and show the request; do not POST")
+    ap.add_argument("--allow-duplicate", action="store_true", help="post even if a pull for this branch already exists")
     a = ap.parse_args()
 
     cookie = load_cookie(a.cookie_file)
     # Tangled's PR form is htmx: it POSTs to /pulls/new (NOT /pulls/, which is 405) and only processes
     # the create when it sees the HX-Request header — otherwise it just re-renders the page (a 200 that
     # creates nothing). Success is signalled by an HX-Redirect header pointing at the new pull.
+    # PRE-CHECK: refuse to create a second pull for a branch that already has one. This is what
+    # makes a retry harmless rather than duplicating — the failure mode that produced #397-#399.
+    before = index_pulls(a.owner, a.repo, cookie)
+    if not before:
+        print("WARNING: could not read the pulls index — cannot check for an existing pull, and "
+              "cannot confirm the result afterwards. Proceeding, but verify by hand.")
+    dup = existing_pull_for(a.owner, a.repo, a.source, cookie, before) if before else None
+    if dup and not a.allow_duplicate:
+        print(f"REFUSING: #{dup} already exists for source branch {a.source!r}.")
+        print(f"  {BASE}/{a.owner}/{a.repo}/pulls/{dup}")
+        print("  To push new commits to an existing pull use tangled_pr_resubmit.py.")
+        print("  To file anyway (rarely right): --allow-duplicate")
+        sys.exit(3)
+
     new_url = f"{BASE}/{a.owner}/{a.repo}/pulls/new"
     form = {
         "source": "branch",           # branch-compare mode (each PR targets the branch below it)
@@ -76,6 +170,12 @@ def main():
         "Referer": new_url,
     })
 
+    if a.dry_run:
+        print(f"DRY RUN: would POST {new_url}")
+        print(f"  target={a.target!r} source={a.source!r} title={a.title[:60]!r}")
+        print(f"  existing pull for this branch: {dup if dup else 'none found in the scan window'}")
+        return
+
     class NoRedirect(urllib.request.HTTPRedirectHandler):
         def redirect_request(self, *args, **kw):  # capture the redirect instead of following it
             return None
@@ -88,15 +188,27 @@ def main():
     # htmx success is signalled by HX-Redirect (…/pulls/<n>), not a normal 3xx Location.
     target = hdrs.get("HX-Redirect", "") or hdrs.get("HX-Location", "") or hdrs.get("Location", "")
     print(f"HTTP {code}" + (f" -> {target}" if target else ""))
-    if target and "/pulls/" in target and "/new" not in target:
-        print("OK: pull created ->", (BASE + target) if target.startswith("/") else target)
+
+    if "/login" in target or "oauth" in target.lower():
+        sys.exit("AUTH FAILED: session cookie expired/invalid — refresh the cookie in the secret "
+                 "store (scripts/get-tangled-cookie.py)")
+
+    # POST-CHECK: the artifact decides, not the header. A create that answered without a redirect
+    # is still a create; reporting it as a failure is what made callers retry into duplicates.
+    after = index_pulls(a.owner, a.repo, cookie)
+    branch_pages = {n: pull_names_branch(a.owner, a.repo, n, a.source, cookie)
+                    for n in sorted(after - before, reverse=True)[:12]}
+    verdict, num = classify(before, after, branch_pages)
+
+    if verdict == "created":
+        print(f"OK: pull #{num} created -> {BASE}/{a.owner}/{a.repo}/pulls/{num}")
+        if not target:
+            print("  (no redirect header — confirmed against the pulls list instead)")
         return
-    if ("/login" in target or "oauth" in target.lower()):
-        sys.exit("AUTH FAILED: session cookie expired/invalid — refresh engine/.tangled-session "
-                 "(scripts/get-tangled-cookie.py)")
-    # otherwise surface whatever the page said (a Notice, etc.)
+    if verdict == "unrelated":
+        print("  new pulls appeared but none names this source branch — not ours.")
     snippet = " ".join(body.split())[:600]
-    print("  no clear success redirect — response snippet:")
+    print("FAILED: no pull for this branch appeared. Response snippet:")
     print("  " + snippet)
     sys.exit(2)
 

tangled_pr_close.py

@@ -0,0 +1,106 @@
+#!/usr/bin/env python3
+"""Close an existing Tangled pull as the bot, via the session cookie.
+
+WHY: engine/ could create, edit, merge, resubmit and comment on a pull, but not CLOSE one. The
+appview exposes close as an htmx POST (the "Close" button on the pull page):
+
+  POST /{owner}/{repo}/pulls/{n}/close      -> closes (hx-swap; success is a 2xx)
+
+Same session-cookie auth as tangled_pr_edit.py. Closing does not create a round and does not
+touch the patch. This tool trusts nothing: after the POST it re-fetches the pull page and
+confirms the state badge reads Closed. Use --comment-file to post one comment FIRST naming the
+successor (the phase-8 rule: every closed pull says which readable PR carries its work).
+
+USAGE:
+  tangled_pr_close.py --owner notplants-bot.bsky.social --repo lichen.page.review --pull 397 \
+      [--comment-file note.md] [--dry-run]
+"""
+import argparse, os, re, subprocess, sys, urllib.error, urllib.parse, urllib.request
+
+BASE = "https://tangled.org"
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, HERE)
+
+
+def load_cookie():
+    import secrets as _store
+    c = _store.get("tangled.cookie")
+    if not c:
+        sys.exit("no tangled.cookie in the secret store — refresh with scripts/get-tangled-cookie.py")
+    return c
+
+
+def state_of(owner, repo, pull, cookie):
+    import tangled_comments as tc
+    doc = tc.fetch(f"{BASE}/{owner}/{repo}/pulls/{pull}", cookie)
+    badges = set(re.findall(r'>\s*(Merged|Closed|Open)\s*<', doc))
+    return badges
+
+
+def main():
+    ap = argparse.ArgumentParser(description="close a Tangled pull via a reused session cookie")
+    ap.add_argument("--owner", required=True)
+    ap.add_argument("--repo", required=True)
+    ap.add_argument("--pull", required=True, type=int)
+    ap.add_argument("--comment-file", default=None, help="post this comment before closing")
+    ap.add_argument("--dry-run", action="store_true", help="show state and the request; do not POST")
+    a = ap.parse_args()
+
+    cookie = load_cookie()
+    before = state_of(a.owner, a.repo, a.pull, cookie)
+    print(f"#{a.pull} state before: {sorted(before) or 'unknown'}")
+    if "Merged" in before:
+        sys.exit(f"#{a.pull} is MERGED — refusing to close a merged pull")
+    if "Closed" in before and "Open" not in before:
+        print(f"#{a.pull} is already Closed; nothing to do")
+        return
+    if a.dry_run:
+        print(f"DRY RUN: would POST {BASE}/{a.owner}/{a.repo}/pulls/{a.pull}/close")
+        return
+
+    if a.comment_file:
+        r = subprocess.run([sys.executable, os.path.join(HERE, "tangled_comment_post.py"),
+                            "--owner", a.owner, "--repo", a.repo, "--pull", str(a.pull),
+                            "--body-file", a.comment_file], capture_output=True, text=True)
+        print(r.stdout.strip())
+        if r.returncode != 0:
+            sys.exit(f"comment failed, NOT closing: {r.stderr.strip()[:300]}")
+
+    close_url = f"{BASE}/{a.owner}/{a.repo}/pulls/{a.pull}/close"
+    page_url = f"{BASE}/{a.owner}/{a.repo}/pulls/{a.pull}"
+    req = urllib.request.Request(close_url, data=b"", method="POST", headers={
+        "Cookie": cookie,
+        "Content-Type": "application/x-www-form-urlencoded",
+        "User-Agent": "tangled-pr-bot",
+        "HX-Request": "true",
+        "HX-Current-URL": page_url,
+        "Referer": page_url,
+    })
+
+    class NoRedirect(urllib.request.HTTPRedirectHandler):
+        def redirect_request(self, *args, **kw):
+            return None
+    opener = urllib.request.build_opener(NoRedirect)
+    try:
+        r = opener.open(req, timeout=90)
+        hdrs, code, resp = r.headers, r.getcode(), r.read(3000).decode("utf-8", "replace")
+    except urllib.error.HTTPError as e:
+        hdrs, code, resp = e.headers, e.code, e.read(3000).decode("utf-8", "replace")
+    target = hdrs.get("HX-Redirect", "") or hdrs.get("HX-Location", "") or hdrs.get("Location", "")
+    print(f"HTTP {code}" + (f" -> {target}" if target else ""))
+    if "/login" in target or "oauth" in target.lower():
+        sys.exit("AUTH FAILED: session cookie expired — refresh with scripts/get-tangled-cookie.py")
+    if code // 100 != 2:
+        print("  close did not return 2xx — response snippet:")
+        print("  " + " ".join(resp.split())[:600])
+        sys.exit(2)
+
+    after = state_of(a.owner, a.repo, a.pull, cookie)
+    print(f"#{a.pull} state after:  {sorted(after) or 'unknown'}")
+    if "Closed" not in after:
+        sys.exit("POST returned 2xx but the pull page does not read Closed — check by hand")
+    print(f"#{a.pull} CLOSED")
+
+
+if __name__ == "__main__":
+    main()

tangled_pr_merge.py

@@ -0,0 +1,106 @@
+#!/usr/bin/env python3
+"""Merge or close a Tangled pull as the bot, with a mergeability pre-check.
+
+WHY: `main` advances only by operator-approved merges of the stacked pulls, and the
+stack must be merged strictly bottom-up. The appview exposes both actions as bare
+htmx POSTs with no body:
+    POST /{owner}/{repo}/pulls/{n}/merge
+    POST /{owner}/{repo}/pulls/{n}/close
+Neither tells you up front whether the pull is mergeable — that lives in a separate
+fragment, GET /{owner}/{repo}/pulls/{n}/round/{r}/actions, which reports "No conflicts,
+ready to merge" / "patch is empty" / a conflict. This checks that fragment first and
+refuses to merge anything it does not understand, so a bad rung stops the frontier
+instead of landing broken.
+
+Note: a pull whose fragment says "patch is empty" (a branch identical to its parent —
+e.g. an index slot whose content moved out of the repo) CANNOT be merged; close it.
+
+USAGE:
+  tangled_pr_merge.py --owner X --repo Y --pull 76 --check
+  tangled_pr_merge.py --owner X --repo Y --pull 76 --merge
+  tangled_pr_merge.py --owner X --repo Y --pull 75 --close --reason "empty after restack"
+"""
+import argparse, os, re, sys, urllib.error, urllib.request
+
+BASE = "https://tangled.org"
+
+
+def load_cookie():
+    sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+    import secrets as _store
+    c = _store.get("tangled.cookie")
+    if not c:
+        sys.exit("no tangled.cookie in the secret store")
+    return c
+
+
+def get(url, cookie):
+    req = urllib.request.Request(url, headers={"Cookie": cookie, "User-Agent": "tangled-pr-bot"})
+    return urllib.request.urlopen(req, timeout=60).read().decode()
+
+
+def post(url, cookie, referer):
+    req = urllib.request.Request(
+        url,
+        data=b"",
+        headers={
+            "Cookie": cookie,
+            "User-Agent": "tangled-pr-bot",
+            "HX-Request": "true",
+            "Content-Type": "application/x-www-form-urlencoded",
+            "Referer": referer,
+        },
+    )
+    return urllib.request.urlopen(req, timeout=120)
+
+
+def latest_round(doc):
+    idxs = re.findall(r'name="pull-round-idx"[^>]*value="(\d+)"', doc)
+    return max((int(i) for i in idxs), default=0)
+
+
+def state(owner, repo, pull, cookie):
+    """(verdict, round) — verdict is the appview's own mergeability text."""
+    page = get(f"{BASE}/{owner}/{repo}/pulls/{pull}", cookie)
+    rnd = latest_round(page)
+    frag = get(f"{BASE}/{owner}/{repo}/pulls/{pull}/round/{rnd}/actions", cookie)
+    txt = re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", frag)).strip()
+    for phrase in ("No conflicts, ready to merge", "patch is empty", "merged", "conflict"):
+        if phrase.lower() in txt.lower():
+            return phrase, rnd, txt
+    return "UNKNOWN", rnd, txt
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--owner", required=True)
+    ap.add_argument("--repo", required=True)
+    ap.add_argument("--pull", required=True, type=int)
+    g = ap.add_mutually_exclusive_group(required=True)
+    g.add_argument("--check", action="store_true")
+    g.add_argument("--merge", action="store_true")
+    g.add_argument("--close", action="store_true")
+    ap.add_argument("--reason", default=None, help="close only: logged, not posted")
+    a = ap.parse_args()
+
+    cookie = load_cookie()
+    verdict, rnd, txt = state(a.owner, a.repo, a.pull, cookie)
+    print(f"#{a.pull} round {rnd}: {verdict}")
+
+    if a.check:
+        print(f"  raw: {txt[:200]}")
+        return
+
+    referer = f"{BASE}/{a.owner}/{a.repo}/pulls/{a.pull}"
+    if a.merge:
+        if verdict != "No conflicts, ready to merge":
+            sys.exit(f"refusing to merge #{a.pull}: appview says {verdict!r}")
+        r = post(f"{referer}/merge", cookie, referer)
+        print(f"  merged #{a.pull} — HTTP {r.status}")
+    else:
+        r = post(f"{referer}/close", cookie, referer)
+        print(f"  closed #{a.pull} — HTTP {r.status}" + (f" ({a.reason})" if a.reason else ""))
+
+
+if __name__ == "__main__":
+    main()

tangled_pr_resubmit.py

@@ -0,0 +1,121 @@
+#!/usr/bin/env python3
+"""Advance a Tangled pull to a new ROUND after you pushed a fixup, and print its interdiff URL.
+
+WHY THIS EXISTS: pushing to the source branch does NOT update the pull. The appview keeps
+serving the diff it fetched when the pull was opened (or last resubmitted), so a reviewer
+reading the pull sees the code as it was BEFORE your fixup, and no
+`/round/<r>/interdiff` exists to show what changed. Nothing warns you — the push
+succeeded, the branch is right, and the pull silently lags. That cost a real review cycle
+on 2026-08-16: three PRs were reviewed against stale trees because "pushed" was assumed to
+mean "resubmitted".
+
+The appview's own action is a bare htmx POST with no body:
+
+    POST /{owner}/{repo}/pulls/{n}/resubmit
+
+which re-fetches the branch patch from the knot and opens round N+1. This wraps it, then
+reports the new round and the interdiff URL to hand to the reviewer.
+
+USAGE:
+  tangled_pr_resubmit.py --owner X --repo Y --pull 376
+  tangled_pr_resubmit.py --owner X --repo Y --pull 371 372 374 375   # several, in order
+  tangled_pr_resubmit.py --owner X --repo Y --pull 376 --check       # rounds only, no POST
+
+AFTER A FIXUP, THE WHOLE DANCE IS:
+  git push review <branch> && tangled_pr_resubmit.py --owner ... --repo ... --pull <n>
+  then reply on the pull, leading with the printed interdiff URL.
+"""
+import argparse, os, re, sys, urllib.error, urllib.request
+
+BASE = "https://tangled.org"
+
+
+def load_cookie(path=None):
+    """The cookie lives in the encrypted store (tangled.cookie) — see engine/README.md.
+    `path` is the legacy escape hatch: a file holding a TANGLED_COOKIE=... line."""
+    if path:
+        for line in open(path):
+            if line.strip().startswith("TANGLED_COOKIE="):
+                return line.strip()[len("TANGLED_COOKIE=") :]
+        sys.exit(f"{path} has no TANGLED_COOKIE= line")
+    sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+    import secrets as _store
+
+    c = _store.get("tangled.cookie")
+    if not c:
+        sys.exit("no tangled.cookie in the secret store — add it with: sops /secrets/store.yaml")
+    return c
+
+
+def get(url, cookie):
+    req = urllib.request.Request(url, headers={"Cookie": cookie, "User-Agent": "tangled-pr-bot"})
+    return urllib.request.urlopen(req, timeout=60).read().decode()
+
+
+def post(url, cookie, referer):
+    req = urllib.request.Request(
+        url,
+        data=b"",
+        headers={
+            "Cookie": cookie,
+            "User-Agent": "tangled-pr-bot",
+            "HX-Request": "true",
+            "Content-Type": "application/x-www-form-urlencoded",
+            "Referer": referer,
+        },
+    )
+    return urllib.request.urlopen(req, timeout=120)
+
+
+def latest_round(owner, repo, pull, cookie):
+    """The highest round the pull has. Read from the round selector, NOT from the page text:
+    a comment that quotes a `/round/2/interdiff` URL would otherwise be counted as a round."""
+    page = get(f"{BASE}/{owner}/{repo}/pulls/{pull}", cookie)
+    idxs = re.findall(r'name="pull-round-idx"[^>]*value="(\d+)"', page)
+    return max((int(i) for i in idxs), default=0)
+
+
+def interdiff_url(owner, repo, pull, rnd):
+    return f"{BASE}/{owner}/{repo}/pulls/{pull}/round/{rnd}/interdiff?diff=unified"
+
+
+def resubmit(owner, repo, pull, cookie, check_only=False):
+    before = latest_round(owner, repo, pull, cookie)
+    if check_only:
+        print(f"#{pull}: round {before}" + (f" — interdiff {interdiff_url(owner, repo, pull, before)}" if before else " — no interdiff yet (round 0)"))
+        return 0
+    referer = f"{BASE}/{owner}/{repo}/pulls/{pull}"
+    try:
+        r = post(f"{referer}/resubmit", cookie, referer)
+    except urllib.error.HTTPError as e:
+        print(f"#{pull}: resubmit FAILED — HTTP {e.code} {e.read()[:200]!r}", file=sys.stderr)
+        return 1
+    after = latest_round(owner, repo, pull, cookie)
+    if after > before:
+        print(f"#{pull}: round {before} -> {after} (HTTP {r.status})")
+        print(f"   interdiff: {interdiff_url(owner, repo, pull, after)}")
+        return 0
+    # the appview accepted the POST but the round did not move: the branch is identical to
+    # what the pull already carries. Say so — silence here reads as success.
+    print(f"#{pull}: still round {after} (HTTP {r.status}) — the branch matches the pull; nothing to resubmit")
+    return 0
+
+
+def main():
+    ap = argparse.ArgumentParser(description="resubmit a Tangled pull after pushing a fixup")
+    ap.add_argument("--owner", required=True)
+    ap.add_argument("--repo", required=True)
+    ap.add_argument("--pull", required=True, type=int, nargs="+", help="one or more pull numbers")
+    ap.add_argument("--check", action="store_true", help="report the current round; do not resubmit")
+    ap.add_argument("--cookie-file", default=None, help="legacy: read the cookie from this file")
+    a = ap.parse_args()
+
+    cookie = load_cookie(a.cookie_file)
+    rc = 0
+    for pull in a.pull:
+        rc |= resubmit(a.owner, a.repo, pull, cookie, a.check)
+    sys.exit(rc)
+
+
+if __name__ == "__main__":
+    main()

tests/run.sh

@@ -4,7 +4,8 @@
 #
 #   • UNIT tests        — always run (pure logic, no agents spawned). A failure fails the suite.
 #   • CLAUDE smoke      — live, run when the `claude` CLI is available; SKIPs otherwise.
-#   • OPENCODE smoke    — live, run when `opencode` + creds are available; SKIPs otherwise.
+#   • CODEX smoke      — live, run when Codex is installed, logged in, and Remote Control connects.
+#   • OPENCODE smoke   — live, run when `opencode` + creds are available; SKIPs otherwise.
 #   • ISOLATION sanity  — after the live runs: assert no leftover aotest-* tmux sessions, and that
 #                         the live cc-ci-* sessions are untouched.
 #
@@ -19,7 +20,7 @@ set -uo pipefail
 HERE="$(cd "$(dirname "$0")" && pwd)"
 REPO="$(cd "$HERE/.." && pwd)"
 RC=0
-UNIT=FAIL CLAUDE=SKIP OPENCODE=SKIP ISO=PASS
+UNIT=FAIL CLAUDE=SKIP CODEX=SKIP OPENCODE=SKIP ISO=PASS
 
 echo "######################################################################"
 echo "# agent-orchestrator test suite"
@@ -47,6 +48,7 @@ run_smoke() {
 
 # ── live smoke tests (when backends available) ──────────────────────────────────────
 run_smoke "CLAUDE" "$HERE/smoke_claude.sh"; case $? in 0) CLAUDE=PASS;; 2) CLAUDE=SKIP;; *) CLAUDE=FAIL; RC=1;; esac
+run_smoke "CODEX" "$HERE/smoke_codex.sh"; case $? in 0) CODEX=PASS;; 2) CODEX=SKIP;; *) CODEX=FAIL; RC=1;; esac
 run_smoke "OPENCODE" "$HERE/smoke_opencode.sh"; case $? in 0) OPENCODE=PASS;; 2) OPENCODE=SKIP;; *) OPENCODE=FAIL; RC=1;; esac
 
 # ── isolation sanity ────────────────────────────────────────────────────────────────
@@ -69,7 +71,7 @@ fi
 
 # ── summary ─────────────────────────────────────────────────────────────────────────
 echo; echo "######################################################################"
-echo "# SUMMARY:  unit=$UNIT  claude=$CLAUDE  opencode=$OPENCODE  isolation=$ISO"
+echo "# SUMMARY:  unit=$UNIT  claude=$CLAUDE  codex=$CODEX  opencode=$OPENCODE  isolation=$ISO"
 echo "######################################################################"
 [ "$RC" -eq 0 ] && echo "ALL RUN TESTS PASSED (skips are OK)" || echo "SUITE FAILED"
 exit "$RC"

tests/smoke_codex.sh

@@ -0,0 +1,121 @@
+#!/usr/bin/env bash
+# Isolated live smoke of the Codex backend, driven entirely through agents.py.
+# It starts one uniquely-named tmux session, verifies Remote Control and a daemon-connected Codex
+# TUI, then removes only that session. The shared Remote Control daemon intentionally remains
+# available.
+set -uo pipefail
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+REPO="$(cd "$HERE/.." && pwd)"
+CODEX_BIN="${CODEX_BIN:-$(command -v codex 2>/dev/null || echo "$HOME/.local/bin/codex")}"
+MODEL="${AOTEST_CODEX_MODEL:-}"
+PREFIX="aotest-x-$$-"
+SANDBOX="$(mktemp -d)"
+CFG="$SANDBOX/agents.toml"
+FAILED=0
+
+pass(){ echo "  PASS: $*"; }
+fail(){ echo "  FAIL: $*"; FAILED=1; }
+
+cleanup(){
+  local rc=$?
+  python3 "$REPO/agents.py" --config "$CFG" down probe >/dev/null 2>&1 || true
+  if command -v tmux >/dev/null 2>&1; then
+    tmux ls 2>/dev/null | sed 's/:.*//' | grep "^${PREFIX}" | while read -r s; do
+      tmux kill-session -t "=$s" 2>/dev/null || true
+    done || true
+  fi
+  rm -rf "$SANDBOX"
+  exit "$rc"
+}
+trap cleanup EXIT INT TERM
+
+echo "=== codex backend smoke (isolated: prefix=${PREFIX}) ==="
+command -v tmux >/dev/null 2>&1 || { echo "SKIP: tmux not on PATH"; exit 0; }
+[ -x "$CODEX_BIN" ] || command -v "$CODEX_BIN" >/dev/null 2>&1 \
+  || { echo "SKIP: codex binary not found ($CODEX_BIN)"; exit 0; }
+"$CODEX_BIN" login status >/dev/null 2>&1 \
+  || { echo "SKIP: Codex is not logged in"; exit 0; }
+remote_status="$("$CODEX_BIN" remote-control start --json 2>&1)"
+echo "$remote_status" | grep -q '"status":"connected"' \
+  || { echo "SKIP: Codex Remote Control is not connected: $remote_status"; exit 0; }
+pass "Codex Remote Control reports connected"
+
+model_line=""
+[ -n "$MODEL" ] && model_line="model = \"$MODEL\""
+cat > "$CFG" <<EOF
+[defaults]
+project_dir    = "$REPO"
+session_prefix = "$PREFIX"
+log_dir        = "$SANDBOX/state"
+backend        = "codex"
+$model_line
+watch          = "none"
+
+[backend.codex]
+bin             = "$CODEX_BIN"
+preamble        = "{bin} remote-control start --json >/dev/null"
+remote_addr     = "unix://"
+flags           = "--dangerously-bypass-approvals-and-sandbox --dangerously-bypass-hook-trust --no-alt-screen"
+supports_resume = false
+prompt_delivery = "arg"
+process_name    = "codex"
+footer_ui       = true
+log_grace       = 180
+submit_key      = "Enter"
+startup_prompt_re       = "Trusting the directory|Do you trust"
+startup_prompt_response = "1"
+startup_prompt_delay    = 2
+stall_idle      = 300
+active_re       = "esc to interrupt|working|thinking|running|searching"
+limit_re        = "usage limit|limit reached"
+
+[[agent]]
+name   = "probe"
+kind   = "persistent"
+prompt = "You are a harness self-test. Reply exactly CODEX_BACKEND_READY and then wait silently. Do not use tools or modify files."
+EOF
+
+python3 "$REPO/agents.py" --config "$CFG" up probe \
+  || { fail "agents.py up probe errored"; echo "=== CODEX BACKEND SMOKE: FAIL ==="; exit 1; }
+
+sleep 12
+if tmux has-session -t "=${PREFIX}probe" 2>/dev/null; then
+  cmd="$(tmux display-message -p -t "=${PREFIX}probe:" '#{pane_current_command}')"
+  pass "session ${PREFIX}probe created via agents.py (pane command: ${cmd})"
+else
+  fail "${PREFIX}probe session was not created"
+  echo "=== CODEX BACKEND SMOKE: FAIL ==="
+  exit 1
+fi
+
+pane="$(tmux capture-pane -p -t "=${PREFIX}probe:" -S -200 2>/dev/null)"
+if [ "$cmd" = "codex" ] && echo "$pane" | grep -q "CODEX_BACKEND_READY"; then
+  pass "Codex TUI attached and completed the probe"
+else
+  fail "Codex probe did not complete (cmd=${cmd}); tail: $(echo "$pane" | grep -vE '^\s*$' | tail -5)"
+fi
+
+if echo "$pane" | grep -Fq "$REPO"; then
+  pass "daemon-created Codex thread uses the requested project directory"
+else
+  fail "Codex thread did not use requested directory $REPO"
+fi
+
+if python3 "$REPO/agents.py" --config "$CFG" status \
+    | grep -E '^\s*probe\b' | grep -q RUNNING; then
+  pass "agents.py status reports probe RUNNING"
+else
+  fail "agents.py status did not report probe RUNNING"
+fi
+
+python3 "$REPO/agents.py" --config "$CFG" down probe >/dev/null 2>&1
+sleep 2
+if tmux has-session -t "=${PREFIX}probe" 2>/dev/null; then
+  fail "${PREFIX}probe still alive after agents.py down"
+else
+  pass "agents.py down cleanly removed the session"
+fi
+
+if [ "$FAILED" = 0 ]; then echo "=== CODEX BACKEND SMOKE: PASS ==="; exit 0
+else echo "=== CODEX BACKEND SMOKE: FAIL ==="; exit 1; fi

tests/test_tools.py

@@ -0,0 +1,169 @@
+#!/usr/bin/env python3
+"""Unit tests for the standalone tools (the tangled_* scripts, secrets, gateway-domain).
+
+Run:  python3 -m unittest tests.test_tools     (from the repo root)
+      ./tests/run.sh                            (discovers this file automatically)
+
+NO NETWORK. Every HTTP boundary is injected as a fake `_fetch`, so these run offline and
+in CI. The point is the DECISION logic — "did the pull get created", "is this backend
+valid" — because that is where the defects have actually been.
+
+The reason this file exists: tangled_pr.py judged success only by a response header, so a
+create that answered without one read as a failure, the caller retried, and duplicate
+pulls #397-#399 were filed for one branch. The tool was never tested; the bug survived
+months of daily use and was found by counting pulls, not by running the script.
+"""
+import os, sys, unittest
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import tangled_pr  # noqa: E402
+
+
+class TestPullNumbers(unittest.TestCase):
+    def test_extracts_every_linked_pull(self):
+        html = '<a href="/o/r/pulls/417">x</a><a href="/o/r/pulls/416">y</a>'
+        self.assertEqual(tangled_pr.pull_numbers(html), {417, 416})
+
+    def test_deduplicates_repeated_links(self):
+        html = '/pulls/12 /pulls/12 /pulls/12'
+        self.assertEqual(tangled_pr.pull_numbers(html), {12})
+
+    def test_ignores_the_new_pull_form(self):
+        # /pulls/new must not parse as a number — it is the page you POST to, not a pull
+        self.assertEqual(tangled_pr.pull_numbers('/pulls/new /pulls/9'), {9})
+
+    def test_empty_and_none_are_empty_not_an_error(self):
+        # an unreadable page must yield "cannot confirm", never a crash mid-create
+        self.assertEqual(tangled_pr.pull_numbers(""), set())
+        self.assertEqual(tangled_pr.pull_numbers(None), set())
+
+
+class TestPageNamesBranch(unittest.TestCase):
+    def test_finds_the_branch(self):
+        self.assertTrue(tangled_pr.page_names_branch("<div>e2e-pds/R4-real</div>", "e2e-pds/R4-real"))
+
+    def test_absent_branch(self):
+        self.assertFalse(tangled_pr.page_names_branch("<div>other</div>", "e2e-pds/R4-real"))
+
+    def test_unreadable_page_is_not_a_match(self):
+        self.assertFalse(tangled_pr.page_names_branch("", "b"))
+        self.assertFalse(tangled_pr.page_names_branch(None, "b"))
+
+    def test_empty_branch_never_matches(self):
+        # guards against a caller passing "" and matching every page
+        self.assertFalse(tangled_pr.page_names_branch("anything", ""))
+
+
+class TestClassify(unittest.TestCase):
+    """The regression that matters: a successful create with NO redirect header."""
+
+    def test_new_pull_naming_our_branch_is_created(self):
+        v, n = tangled_pr.classify({1, 2}, {1, 2, 3}, {3: True})
+        self.assertEqual((v, n), ("created", 3))
+
+    def test_created_even_though_the_header_said_nothing(self):
+        # THE BUG: classify never sees the response at all, so a missing HX-Redirect
+        # cannot turn a real create into a reported failure.
+        v, n = tangled_pr.classify({396}, {396, 397}, {397: True})
+        self.assertEqual(v, "created")
+
+    def test_no_new_pull_is_a_real_failure(self):
+        v, n = tangled_pr.classify({1, 2}, {1, 2}, {})
+        self.assertEqual((v, n), ("none", None))
+
+    def test_someone_elses_pull_is_not_ours(self):
+        v, n = tangled_pr.classify({1}, {1, 2}, {2: False})
+        self.assertEqual((v, n), ("unrelated", None))
+
+    def test_picks_the_highest_of_several_of_ours(self):
+        v, n = tangled_pr.classify({1}, {1, 2, 3}, {2: True, 3: True})
+        self.assertEqual((v, n), ("created", 3))
+
+    def test_a_pull_that_vanished_does_not_confuse_it(self):
+        # after ⊂ before (someone closed+deleted one mid-run): no new pull, so no create
+        v, n = tangled_pr.classify({1, 2}, {1}, {})
+        self.assertEqual((v, n), ("none", None))
+
+
+class TestExistingPullScan(unittest.TestCase):
+    """The pre-check that makes a retry harmless instead of duplicating."""
+
+    def _fetcher(self, pages):
+        def _f(url, cookie, timeout=30):
+            return pages.get(url.rsplit("/", 1)[-1], "")
+        return _f
+
+    def test_finds_an_existing_pull_for_the_branch(self):
+        f = self._fetcher({"396": "branch: feat/x", "395": "branch: other"})
+        got = tangled_pr.existing_pull_for("o", "r", "feat/x", "c", {395, 396}, _fetch=f)
+        self.assertEqual(got, 396)
+
+    def test_returns_none_when_the_branch_is_new(self):
+        f = self._fetcher({"396": "other", "395": "other"})
+        self.assertIsNone(tangled_pr.existing_pull_for("o", "r", "feat/x", "c", {395, 396}, _fetch=f))
+
+    def test_prefers_the_newest_match(self):
+        f = self._fetcher({"10": "feat/x", "20": "feat/x"})
+        self.assertEqual(tangled_pr.existing_pull_for("o", "r", "feat/x", "c", {10, 20}, _fetch=f), 20)
+
+    def test_scan_window_is_bounded(self):
+        # a match older than the window is missed BY DESIGN; the post-check is the backstop.
+        pages = {str(n): ("feat/x" if n == 1 else "other") for n in range(1, 30)}
+        f = self._fetcher(pages)
+        self.assertIsNone(
+            tangled_pr.existing_pull_for("o", "r", "feat/x", "c", set(range(1, 30)), scan=5, _fetch=f))
+
+    def test_unreadable_pages_do_not_claim_absence(self):
+        # every fetch fails -> None, and main() warns rather than silently creating a duplicate
+        f = lambda url, cookie, timeout=30: ""
+        self.assertIsNone(tangled_pr.existing_pull_for("o", "r", "feat/x", "c", {1, 2}, _fetch=f))
+
+
+class TestFetchIsFailSoft(unittest.TestCase):
+    def test_network_error_returns_empty_not_raise(self):
+        # a create must not die between POST and verification
+        self.assertEqual(tangled_pr.fetch("http://127.0.0.1:1/nope", "c", timeout=1), "")
+
+
+class TestCloseToolStateParsing(unittest.TestCase):
+    """tangled_pr_close.state_of decides whether a pull is already closed."""
+
+    def setUp(self):
+        import tangled_pr_close
+        self.mod = tangled_pr_close
+
+    def test_badge_regex_reads_the_states(self):
+        import re
+        badges = lambda doc: set(re.findall(r'>\s*(Merged|Closed|Open)\s*<', doc))
+        self.assertEqual(badges("<span> Closed </span>"), {"Closed"})
+        self.assertEqual(badges("<b>Open</b><b>Merged</b>"), {"Open", "Merged"})
+        self.assertEqual(badges("<p>closed</p>"), set())  # case-sensitive on purpose
+
+
+class TestGatewayBackendValidation(unittest.TestCase):
+    """gateway-domain refuses hostname backends: the gateway can store one and never remove it."""
+
+    def setUp(self):
+        import importlib.util
+        p = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
+                         "tools", "gateway-domain.py")
+        if not os.path.exists(p):
+            self.skipTest("gateway-domain.py not present")
+        spec = importlib.util.spec_from_file_location("gwd", p)
+        self.gwd = importlib.util.module_from_spec(spec)
+        spec.loader.exec_module(self.gwd)
+
+    def test_accepts_ip_and_ip_port(self):
+        self.assertTrue(self.gwd._BACKEND.match("100.74.89.63"))
+        self.assertTrue(self.gwd._BACKEND.match("100.74.89.63:8443"))
+
+    def test_rejects_hostnames(self):
+        for bad in ("example.com", "host:443", "", "1.2.3", "1.2.3.4.5"):
+            self.assertIsNone(self.gwd._BACKEND.match(bad), f"should reject {bad!r}")
+
+    def test_allowed_tags_include_both_known_tags(self):
+        self.assertIn("tag:notplants-test-server", self.gwd.ALLOWED_TAGS)
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_unit.py

@@ -74,6 +74,23 @@ stall_idle      = 900
 active_re = "esc interrupt|thinking|inferring|running tool|tool call|preparing patch|reading|searching"
 limit_re  = "usage limit|limit reached"
 
+[backend.codex]
+bin             = "codex"
+preamble        = "{bin} remote-control start --json >/dev/null"
+remote_addr     = "unix://"
+flags           = "--dangerously-bypass-approvals-and-sandbox --dangerously-bypass-hook-trust --no-alt-screen"
+supports_resume = false
+prompt_delivery = "arg"
+process_name    = "codex"
+footer_ui       = true
+log_grace       = 180
+submit_key      = "Enter"
+startup_prompt_re       = "Trusting the directory|Do you trust"
+startup_prompt_response = "1"
+startup_prompt_delay    = 0
+active_re       = "working|thinking|running"
+limit_re        = "usage limit|limit reached"
+
 [backend.demo]
 bin             = "echo up; exec sleep 100000"
 prompt_delivery = "exec"
@@ -102,6 +119,13 @@ kind    = "persistent"
 backend = "opencode"
 prompt  = "hi"
 
+[[agent]]
+name    = "cx"
+kind    = "persistent"
+backend = "codex"
+model   = "gpt-test"
+prompt  = "hi from codex"
+
 [[agent]]
 name    = "custom"
 kind    = "persistent"
@@ -233,10 +257,42 @@ class TestExampleConfig(unittest.TestCase):
         cfg = agents.load_config(ex)
         self.assertIn("builder", cfg["agents"])
         self.assertIn("adversary", cfg["agents"])
-        for be in ("demo", "claude", "opencode"):
+        for be in ("demo", "claude", "codex", "opencode"):
             self.assertIn(be, cfg["backends"], f"backend {be} missing from example")
         self.assertEqual(len(agents.phases(cfg)), 2)
 
+    def test_codex_launch_prepares_remote_control_then_connects_tui_to_daemon(self):
+        tmp = tempfile.mkdtemp(prefix="aotest-ut-codex-")
+        old_alive, old_new, old_log = agents.session_alive, agents.new_session, agents.log
+        old_capture, old_run = agents.capture_pane, agents._run
+        try:
+            cfg = agents.load_config(_make_project(tmp))
+            launched = []
+            sent = []
+            agents.session_alive = lambda _session: False
+            agents.new_session = lambda session, cwd, cmd, log_path: launched.append(cmd)
+            agents.log = lambda _msg: None
+            agents.capture_pane = lambda *_args, **_kwargs: "Trusting the directory"
+            agents._run = lambda command: sent.append(command)
+            agents.start_agent(cfg, cfg["agents"]["cx"])
+            self.assertEqual(len(launched), 1)
+            cmd = launched[0]
+            self.assertTrue(cmd.startswith(
+                "codex remote-control start --json >/dev/null && "
+                "codex --remote 'unix://' -C "))
+            self.assertEqual(cmd.count("--remote 'unix://'"), 1)
+            self.assertIn(f"-C {agents.shlex.quote(cfg['agents']['cx']['dir'])}", cmd)
+            self.assertIn("--model 'gpt-test'", cmd)
+            self.assertIn(
+                "--dangerously-bypass-approvals-and-sandbox --dangerously-bypass-hook-trust --no-alt-screen",
+                cmd)
+            self.assertIn("kickoff-aotest-ut-cx.txt", cmd)
+            self.assertTrue(any("1" in command for command in sent))
+        finally:
+            agents.session_alive, agents.new_session, agents.log = old_alive, old_new, old_log
+            agents.capture_pane, agents._run = old_capture, old_run
+            shutil.rmtree(tmp, ignore_errors=True)
+
 
 # ── kickoff-template assembly ──────────────────────────────────────────────────────
 
@@ -568,9 +624,17 @@ class TestProcDescendants(unittest.TestCase):
             kids = agents._proc_descendants([str(p.pid)])
             self.assertNotIn(str(p.pid), kids)                # root excluded
             self.assertGreaterEqual(len(kids), 2)             # the two sleeps
-            comms = subprocess.run("ps -o comm= -p " + ",".join(sorted(kids)),
-                                   shell=True, capture_output=True, text=True).stdout
-            self.assertIn("sleep", comms)
+            # Read /proc/<pid>/comm rather than shelling to `ps -o comm=`: this host's ps
+            # returns nothing for that form, and an empty result from a tool that is absent or
+            # unsupported is indistinguishable from "the children are not sleeps". Same class of
+            # bug as the pgrep dependency this test just caught in _proc_descendants.
+            comms = []
+            for k in sorted(kids):
+                try:
+                    comms.append(open(f"/proc/{k}/comm").read().strip())
+                except OSError:
+                    pass
+            self.assertIn("sleep", comms, f"expected a sleep among {comms}")
         finally:
             os.killpg(os.getpgid(p.pid), signal.SIGKILL); p.wait()
 
@@ -594,26 +658,38 @@ class TestBuildRunning(unittest.TestCase):
         self.ps_targets = ""
 
     def tearDown(self):
+        # Restore EXPLICITLY. A cleverer derivation of these names silently matched nothing, so the
+        # monkeypatch leaked into the next test class and made an unrelated test fail — visible only
+        # in a full run, never when that test was run alone.
         agents.subprocess.run = self._orig_run
         shutil.rmtree(self.tmp, ignore_errors=True)
 
     def _patch(self, descendant_comms):
-        """Fake tmux/pgrep/ps: pane_pid=1000; its children are 1001,1002 with the given comms."""
+        """pane_pid=1000; its children are 1001,1002 with the given comms.
+
+        Patches the two internal seams (_proc_descendants, _comms) rather than faking
+        `pgrep`/`ps` subprocess calls. WHY (2026-08-21): the previous version mocked those two
+        commands, so the suite passed on hosts where NEITHER IS INSTALLED — the fake stood in for
+        the broken dependency and the real bug (empty output read as "no build running", which
+        lets the watchdog reboot an agent mid-build) was invisible to every test. Mock the seam
+        you own, not the tool you depend on, or the test proves the mock works.
+        """
         outer = self
         class R:
             def __init__(self, out): self.stdout = out; self.returncode = 0
         def fake_run(cmd, *a, **k):
-            if "list-panes" in cmd:
-                return R("1000\n")
-            if "pgrep -P 1000" in cmd:
-                return R("1001\n1002\n")
-            if "pgrep -P" in cmd:
-                return R("")                                  # 1001/1002 are leaves
-            if cmd.startswith("ps -o comm="):
-                outer.ps_targets = cmd.split("-p", 1)[1].strip()
-                return R("\n".join(descendant_comms) + "\n")
-            return R("")
+            return R("1000\n") if "list-panes" in cmd else R("")
         agents.subprocess.run = fake_run
+        orig_desc, orig_comms = agents._proc_descendants, agents._comms
+        def _restore():
+            agents._proc_descendants, agents._comms = orig_desc, orig_comms
+        self.addCleanup(_restore)      # runs even if the test errors; no tearDown ordering to get wrong
+        agents._proc_descendants = lambda roots: (
+            {"1001", "1002"} if "1000" in list(roots) else set())
+        def fake_comms(pids):
+            outer.ps_targets = ",".join(sorted(pids))
+            return list(descendant_comms)
+        agents._comms = fake_comms
 
     def test_detects_running_build(self):
         self._patch(["bash", "cargo"])

tools/gateway-domain.py

@@ -0,0 +1,247 @@
+#!/usr/bin/env python3
+"""gateway-domain.py — point a public domain at your box, via the testing gateway.
+
+WHAT THIS IS FOR. Your machine is on the tailnet but has no public IP, and you need a real
+HTTPS domain for it — an OAuth callback, a webhook receiver, a demo someone else can open.
+The testing gateway at gtest.commoninternet.net already holds a wildcard DNS record, so
+*.gtest.commoninternet.net resolves to it. Map your name to your tailnet IP and the gateway
+streams matching connections to you:
+
+    browser --TLS--> gateway :443 --reads SNI--> your box over the tailnet
+
+The gateway never terminates that TLS. It reads the SNI name and proxies the still-encrypted
+bytes, so **your box serves the certificate**, not the gateway. See SKILL.md.
+
+USAGE
+    tools/gateway-domain.py list
+    tools/gateway-domain.py add myapp                    # -> myapp.gtest.commoninternet.net -> this box's tailscale IP
+    tools/gateway-domain.py add myapp 100.64.1.5         # explicit backend
+    tools/gateway-domain.py add myapp 100.64.1.5:8443    # backend listening somewhere other than 443
+    tools/gateway-domain.py remove myapp
+
+CREDENTIALS. The admin password is in the orchestrator secret store, never here:
+
+    python3 engine/secrets.py get gateway.admin_password
+
+This script reads it from there itself. You should not need to handle the value.
+"""
+import argparse
+import base64
+import json
+import os
+import re
+import shutil
+import subprocess
+import sys
+import urllib.error
+import urllib.parse
+import urllib.request
+
+# engine/secrets.py lives one directory up.
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+import secrets as secret_store  # noqa: E402  (engine/secrets.py, not the stdlib module)
+
+TIMEOUT = 30
+
+
+def _cfg():
+    """Gateway coordinates. Only the password is genuinely secret; the rest lives
+    alongside it so there is one thing to change if the gateway ever moves."""
+    return (
+        secret_store.get("gateway.fqdn", "gtest.commoninternet.net"),
+        secret_store.get("gateway.admin_user", "admin"),
+        secret_store.get("gateway.admin_password"),
+    )
+
+
+def _request(method="GET", form=None):
+    fqdn, user, password = _cfg()
+    if not password:
+        sys.exit(
+            "no gateway.admin_password in the secret store.\n"
+            "  check: python3 engine/secrets.py list\n"
+            "  see:   skills/gateway-domain/SKILL.md"
+        )
+    url = f"https://{fqdn}/admin/"
+    data = urllib.parse.urlencode(form).encode() if form else None
+    req = urllib.request.Request(url, data=data, method=method)
+    token = base64.b64encode(f"{user}:{password}".encode()).decode()
+    req.add_header("Authorization", f"Basic {token}")
+    try:
+        with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
+            return r.status, r.read().decode("utf-8", "replace")
+    except urllib.error.HTTPError as e:
+        body = e.read().decode("utf-8", "replace")
+        if e.code == 401:
+            sys.exit(
+                f"401 from {url} — the stored password is not what the gateway expects.\n"
+                "The e2e suite reseeds .htpasswd while it runs and restores it afterwards;\n"
+                "if a run was interrupted, /var/lib/tunnel-gateway/.htpasswd.e2e-backup is\n"
+                "still on the box. See SKILL.md, 'When the password stops working'."
+            )
+        sys.exit(f"HTTP {e.code} from {url}\n{body[:600]}")
+    except urllib.error.URLError as e:
+        sys.exit(f"cannot reach {url}: {e.reason}")
+
+
+# The admin app answers 200 and renders a red <p> for a rejected value, rather than
+# using a status code. Checking only the status would report success on a no-op.
+_ERR = re.compile(r'<p style="color:red">(.*?)</p>', re.S)
+
+# Backends are restricted to a literal IPv4 address, optionally with a port.
+#
+# The gateway would accept a hostname -- its validate_ip is ^[a-zA-Z0-9_.-]+$ -- but its
+# put_domain/remove_domain match existing lines with ^<domain>\s+[\d.:]+; which only ever
+# matches a numeric backend. A hostname mapping can therefore be written once and then never
+# updated or removed through the admin UI: it becomes an orphan only a hand-edit of
+# /var/lib/tunnel-gateway/tunnel_map.conf can clear. Refuse to create one.
+_BACKEND = re.compile(r"^(\d{1,3}(?:\.\d{1,3}){3})(?::(\d{1,5}))?$")
+
+# The tailnet ACL only lets the gateway open connections to nodes carrying one of these.
+# A mapping to an untagged node is accepted by the gateway and then simply never
+# connects, which looks like a gateway fault and is not one.
+ALLOWED_TAGS = ("tag:notplants-test-server", "tag:orchestrator")
+
+
+def _validate_backend(backend):
+    m = _BACKEND.match(backend)
+    if not m:
+        sys.exit(
+            f"backend must be an IPv4 address, optionally IP:port -- got {backend!r}.\n"
+            "Hostnames are rejected on purpose: the gateway can store one but cannot\n"
+            "remove it again. Use the tailnet IP (tailscale ip -4 on the target box)."
+        )
+    ip, port = m.group(1), m.group(2)
+    if any(int(o) > 255 for o in ip.split(".")):
+        sys.exit(f"not a valid IPv4 address: {ip}")
+    if port is not None and not (1 <= int(port) <= 65535):
+        sys.exit(f"port out of range: {port}")
+    return backend
+
+
+def _check(body):
+    m = _ERR.search(body)
+    if m:
+        sys.exit(f"gateway rejected the request: {m.group(1).strip()}")
+
+
+def _parse_domains(body):
+    """Pull the Domain Mappings table out of the admin page."""
+    section = body.split("Domain Mappings", 1)[-1].split("Port Mappings", 1)[0]
+    return re.findall(r"<td>\s*(.*?)\s*</td>\s*<td>\s*(.*?)\s*</td>", section, re.S)
+
+
+def _self_tags(exe):
+    """This node's tailnet tags, or None if they cannot be determined."""
+    try:
+        out = subprocess.run([exe, "status", "--json"], capture_output=True, text=True, timeout=15)
+        return json.loads(out.stdout).get("Self", {}).get("Tags") or []
+    except Exception:
+        return None
+
+
+def _warn_untagged(exe):
+    tags = _self_tags(exe)
+    if tags is None:
+        return
+    if not any(t in tags for t in ALLOWED_TAGS):
+        print(
+            f"warning: this node carries none of {' / '.join(ALLOWED_TAGS)} "
+            f"(tags: {', '.join(tags) or 'none'}).\n"
+            "         The gateway will accept the mapping but the tailnet ACL will not let it\n"
+            "         reach this box, so no traffic will flow. Add the tag in the Tailscale\n"
+            "         admin, or map a backend that already has it.",
+            file=sys.stderr,
+        )
+
+
+def _tailscale_ip():
+    exe = shutil.which("tailscale") or "/run/current-system/sw/bin/tailscale"
+    try:
+        out = subprocess.run([exe, "ip", "-4"], capture_output=True, text=True, timeout=15)
+    except (OSError, subprocess.SubprocessError) as e:
+        sys.exit(f"could not run tailscale to detect this box's IP ({e}); pass the backend explicitly")
+    _warn_untagged(exe)
+    ip = out.stdout.strip().splitlines()[0].strip() if out.stdout.strip() else ""
+    if not ip:
+        sys.exit(
+            "tailscale reported no IPv4 address — is this box on the tailnet?\n"
+            f"  {exe} status\n"
+            "Or pass the backend explicitly: gateway-domain.py add <name> <ip>"
+        )
+    return ip
+
+
+def _qualify(name, fqdn):
+    """A bare label becomes a subdomain of the gateway; anything with a dot is used as given."""
+    return name if "." in name else f"{name}.{fqdn}"
+
+
+def cmd_list(args):
+    _, body = _request()
+    rows = _parse_domains(body)
+    if not rows:
+        print("no domain mappings")
+        return
+    width = max(len(d) for d, _ in rows)
+    for domain, backend in rows:
+        print(f"  {domain:<{width}}  ->  {backend}")
+
+
+def cmd_add(args):
+    fqdn, _, _ = _cfg()
+    domain = _qualify(args.name, fqdn)
+    backend = args.backend or _tailscale_ip()
+    _validate_backend(backend)
+    _, body = _request("POST", {"domain": domain, "ip": backend})
+    _check(body)
+    # Re-read rather than trusting the POST body: app.py mutates its in-memory dict and
+    # renders that, so the response shows what it meant to do, not what landed on disk.
+    _, fresh = _request()
+    if not any(d == domain for d, _ in _parse_domains(fresh)):
+        sys.exit(f"POST returned 200 but {domain} is not in the mapping table — check `list`")
+    print(f"  {domain}  ->  {backend}")
+    print(f"\nYour box must now serve TLS for {domain} on the backend port (443 unless you")
+    print("set one). The gateway does not terminate TLS; it proxies the encrypted stream.")
+
+
+def cmd_remove(args):
+    fqdn, _, _ = _cfg()
+    domain = _qualify(args.name, fqdn)
+    # The admin app treats a backend of exactly "0" as delete.
+    _, body = _request("POST", {"domain": domain, "ip": "0"})
+    _check(body)
+    _, fresh = _request()
+    if any(d == domain for d, _ in _parse_domains(fresh)):
+        sys.exit(
+            f"{domain} is still in the mapping table after the delete.\n"
+            "If its backend is a hostname rather than an IP, the gateway cannot remove it:\n"
+            "edit /var/lib/tunnel-gateway/tunnel_map.conf on the box and reload nginx."
+        )
+    print(f"  removed {domain}")
+
+
+def main():
+    ap = argparse.ArgumentParser(
+        description="Map a public domain to your tailnet box via the testing gateway.",
+        epilog="Password comes from the secret store (gateway.admin_password); see skills/gateway-domain/SKILL.md.",
+    )
+    sub = ap.add_subparsers(dest="cmd", required=True)
+
+    sub.add_parser("list", help="show current domain mappings").set_defaults(fn=cmd_list)
+
+    a = sub.add_parser("add", help="point a domain at a backend")
+    a.add_argument("name", help="bare label (myapp) or a full domain")
+    a.add_argument("backend", nargs="?", help="IP or IP:port (default: this box's tailscale IP, port 443)")
+    a.set_defaults(fn=cmd_add)
+
+    r = sub.add_parser("remove", help="delete a domain mapping")
+    r.add_argument("name", help="bare label (myapp) or a full domain")
+    r.set_defaults(fn=cmd_remove)
+
+    args = ap.parse_args()
+    args.fn(args)
+
+
+if __name__ == "__main__":
+    main()

tools/secrets.sh

@@ -0,0 +1,149 @@
+#!/usr/bin/env bash
+# secrets.sh — safe wrapper for sops-encrypted per-host secrets in /secrets/<host>/.
+#
+# Shared agent tooling: any agent on any orchestrator can use this. Pick the host with
+# SECRETS_HOST (default b1), e.g.  SECRETS_HOST=b1 engine/tools/secrets.sh verify
+#
+# Layout it expects:
+#   /secrets/<host>/<host>.yaml     sops-encrypted payload
+#   /secrets/<host>/.sops.yaml      recipients
+#   /secrets/<host>/admin-age.key   the admin private key (0600)
+#
+# WHY THIS EXISTS. On 2026-08-16 a hand-rolled sops pipeline did this:
+#
+#     sops -d b1.yaml > /tmp/p.yaml      # decrypt
+#     echo "new_key: value" >> /tmp/p.yaml
+#     cp /tmp/p.yaml b1.yaml             # <-- plaintext written to the REAL path
+#     sops -e -i b1.yaml                 # <-- FAILED (creation_rules path mismatch)
+#
+# sops exited non-zero, the `cp` had already happened, and the plaintext file was then copied
+# to the host — world-readable — before anyone noticed. Every secret in it was exposed.
+#
+# The invariant here: **plaintext never exists at the destination path.** All mutation happens
+# on a temp file inside a 0700 directory; the result is encrypted, verified to be ciphertext,
+# round-tripped through a decrypt, and only then moved into place atomically. Any failure at
+# any step leaves the original file untouched.
+set -euo pipefail
+
+# PATH hardening. This script has already been bitten by a tool "not existing" merely because
+# it was not on PATH — on NixOS, sops/nix live in /run/current-system/sw/bin and setuid wrappers
+# in /run/wrappers/bin, neither of which is guaranteed in a non-login shell. A false "command not
+# found" here reads as "decryption failed", which is exactly the wrong conclusion to draw.
+export PATH="/run/wrappers/bin:/run/current-system/sw/bin:/nix/var/nix/profiles/default/bin:$PATH"
+
+HOST="${SECRETS_HOST:-b1}"
+DIR="/secrets/$HOST"
+FILE="$DIR/$HOST.yaml"
+AGE_KEY="$DIR/admin-age.key"
+
+die() { echo "error: $*" >&2; exit 1; }
+have_sops() {
+  if command -v sops >/dev/null 2>&1; then SOPS=(sops)
+  elif command -v nix >/dev/null 2>&1; then
+    SOPS=(nix --extra-experimental-features "nix-command flakes" shell nixpkgs#sops -c sops)
+  else
+    die "neither sops nor nix found on PATH ($PATH)"
+  fi
+}
+is_encrypted() { grep -qE 'ENC\[AES256_GCM' "$1" 2>/dev/null; }
+
+# Every mutation goes through here. It refuses to install anything that is not verified ciphertext.
+install_encrypted() {
+  local tmp=$1
+  is_encrypted "$tmp" || die "refusing to install: result is NOT encrypted (this is the bug this script exists to prevent)"
+  SOPS_AGE_KEY_FILE="$AGE_KEY" "${SOPS[@]}" -d "$tmp" >/dev/null 2>&1 \
+    || die "refusing to install: encrypted file does not decrypt with $AGE_KEY"
+  chmod 600 "$tmp"
+  mv -f "$tmp" "$FILE"          # atomic within the same filesystem
+  echo "ok: $FILE updated ($(grep -c 'recipient:' "$FILE") recipients)"
+}
+
+workdir() { local d; d=$(mktemp -d "$DIR/.work.XXXXXX"); chmod 700 "$d"; echo "$d"; }
+# NB: must return 0. As an EXIT trap its status becomes the script's status, and an
+# `[ -n "$WD" ] && ...` that is simply false would make every read-only command (get/list/verify)
+# exit 1 while printing a perfectly correct answer — a silent false failure in callers.
+scrub() {
+  if [ -n "${WD:-}" ]; then
+    find "$WD" -type f -exec shred -u {} + 2>/dev/null || true
+    rm -rf "$WD"
+  fi
+  return 0
+}
+trap scrub EXIT
+
+have_sops
+[ -r "$DIR/.sops.yaml" ] || die "no $DIR/.sops.yaml — cannot know who may decrypt"
+[ -d "$DIR" ] || die "no such secrets dir: $DIR"
+[ -f "$FILE" ] || die "no such secrets file: $FILE"
+
+cmd="${1:-help}"; shift || true
+case "$cmd" in
+
+  list)     # key names only, never values
+    SOPS_AGE_KEY_FILE="$AGE_KEY" "${SOPS[@]}" -d "$FILE" | grep -oE '^[a-zA-Z0-9_]+:' | tr -d ':' ;;
+
+  get)      # print ONE value to stdout, for piping. Nothing is written to disk.
+    [ $# -ge 1 ] || die "usage: $0 get <key>"
+    SOPS_AGE_KEY_FILE="$AGE_KEY" "${SOPS[@]}" -d --extract "[\"$1\"]" "$FILE" ;;
+
+  set)      # set <key> <file|->   value read from a file or stdin
+    [ $# -ge 2 ] || die "usage: $0 set <key> <file|->"
+    key=$1; src=$2
+    WD=$(workdir); p="$WD/plain.yaml"; e="$WD/enc.yaml"
+    SOPS_AGE_KEY_FILE="$AGE_KEY" "${SOPS[@]}" -d "$FILE" > "$p"
+    # drop any existing definition of this key (scalar or block)
+    awk -v k="$key" 'BEGIN{skip=0}
+      $0 ~ "^"k":" {skip=1; next}
+      skip==1 && /^[[:space:]]/ {next}
+      {skip=0; print}' "$p" > "$p.new" && mv "$p.new" "$p"
+    if [ "$src" = "-" ]; then val=$(cat); else [ -r "$src" ] || die "cannot read $src"; val=$(cat "$src"); fi
+    if [ "$(printf '%s' "$val" | wc -l)" -gt 0 ]; then
+      { echo "$key: |"; printf '%s\n' "$val" | sed 's/^/  /'; } >> "$p"   # multi-line block
+    else
+      printf '%s: %s\n' "$key" "$val" >> "$p"
+    fi
+    cp "$p" "$e"
+    # --filename-override makes creation_rules match regardless of the temp path. This is the
+    # exact failure that caused the incident: the rule keyed on the real filename, the temp file
+    # did not match, and encryption silently refused.
+    "${SOPS[@]}" --config "$DIR/.sops.yaml" -e -i --filename-override "$FILE" "$e"
+    install_encrypted "$e" ;;
+
+  unset)    # remove a key entirely
+    [ $# -ge 1 ] || die "usage: $0 unset <key>"
+    key=$1
+    WD=$(workdir); p="$WD/plain.yaml"; e="$WD/enc.yaml"
+    SOPS_AGE_KEY_FILE="$AGE_KEY" "${SOPS[@]}" -d "$FILE" > "$p"
+    awk -v k="$key" 'BEGIN{skip=0}
+      $0 ~ "^"k":" {skip=1; next}
+      skip==1 && /^[[:space:]]/ {next}
+      {skip=0; print}' "$p" > "$e"
+    "${SOPS[@]}" --config "$DIR/.sops.yaml" -e -i --filename-override "$FILE" "$e"
+    install_encrypted "$e" ;;
+
+  edit)
+    SOPS_AGE_KEY_FILE="$AGE_KEY" "${SOPS[@]}" "$FILE"
+    is_encrypted "$FILE" || die "file is not encrypted after edit — restore from git/backup NOW" ;;
+
+  verify)
+    is_encrypted "$FILE" && echo "  encrypted: yes" || die "NOT ENCRYPTED: $FILE"
+    echo "  recipients: $(grep -c 'recipient:' "$FILE")"
+    echo "  decrypts:   $(SOPS_AGE_KEY_FILE="$AGE_KEY" "${SOPS[@]}" -d "$FILE" >/dev/null 2>&1 && echo yes || echo NO)"
+    echo "  keys:       $(SOPS_AGE_KEY_FILE="$AGE_KEY" "${SOPS[@]}" -d "$FILE" | grep -oE '^[a-zA-Z0-9_]+:' | tr -d ':' | tr '\n' ' ')"
+    n=$(grep -cE 'tskey-auth-[A-Za-z0-9]{5}|BEGIN OPENSSH PRIVATE KEY' "$FILE" || true)
+    [ "$n" -eq 0 ] && echo "  plaintext leaks: none" || die "PLAINTEXT SECRETS PRESENT ($n)" ;;
+
+  deploy)   # deploy <user@host> [remote-path] — refuses to ship anything unencrypted
+    [ $# -ge 1 ] || die "usage: $0 deploy <user@host> [remote-path]"
+    target=$1; rpath=${2:-/etc/nixos/secrets/$HOST.yaml}
+    is_encrypted "$FILE" || die "refusing to deploy: local file is not encrypted"
+    # SSH_OPTS lets the caller pass -i/-o without this script guessing at key locations.
+    # shellcheck disable=SC2086
+    scp -q ${SSH_OPTS:-} "$FILE" "$target:$rpath" || die "scp failed"
+    ssh ${SSH_OPTS:-} "$target" "chmod 600 '$rpath'" || die "chmod failed"
+    ssh ${SSH_OPTS:-} "$target" "grep -qE 'ENC\[AES256_GCM' '$rpath'" \
+      && echo "ok: deployed and verified encrypted at $target:$rpath" \
+      || die "remote file is not encrypted after deploy" ;;
+
+  *) sed -n '2,30p' "$0"; echo; echo "commands: list | get <key> | set <key> <file|-> | edit | verify | deploy <user@host> [path]" ;;
+esac