refactor(secrets): consumers read the store; drop materialized copies

Materializing wrote a second plaintext file per secret, which is the problem the store
was meant to solve: two files drift, and the copy is what ends up committed or grepped.

- tangled_pr / tangled_pr_edit / tangled_repo now read tangled.cookie from the store.
  engine/.tangled-session is deleted; --cookie-file remains as a legacy escape hatch.
- materialize() is replaced by run-time injection that leaves nothing at rest:
    exec-env <group> -- cmd      group as env vars (use this instead of a systemd
                                 EnvironmentFile — same effect, no plaintext on disk)
    with-file <key> -- cmd {}    0600 file in a private tmpdir, removed when cmd exits,
                                 for consumers that insist on a path (ssh -i, a TLS key)

Co-Authored-By: Claude <noreply@anthropic.com>

README.md

@@ -385,10 +385,23 @@ python3 engine/secrets.py materialize tangled-session   # write a runtime file f
 sops /secrets/store.yaml                   # add/edit: decrypts to $EDITOR, re-encrypts on save
 ```
 
-**Materialized files.** Some consumers read a fixed path and can't be taught otherwise (a systemd
+**No second copies.** A secret must never be written to a second file "so something can read
-`EnvironmentFile`, an ssh `IdentityFile`, nix's `authKeyFile`). Those files still exist at 0600,
+it" — copies drift from the store, get committed, and widen what a stray `grep` or an attacker
-but **the store is the source of truth** — `materialize` rewrites them from it. Never hand-edit a
+finds. Our own code imports this module. Anything else gets the value at **run time**:
-materialized file: edit the store and re-materialize, or the two silently drift.
+
+```sh
+# a group as environment variables — nothing touches the disk
+python3 engine/secrets.py exec-env cc_ci_testenv -- some-command
+
+# a consumer that insists on a path: 0600 file in a private tmpdir, deleted when the command exits
+python3 engine/secrets.py with-file ssh_keys.tangled-ed25519 -- ssh -i {} host
+```
+
+For **systemd**, wrap `ExecStart` in `exec-env` rather than using an `EnvironmentFile`: same
+effect, no plaintext at rest. The genuine exceptions are OS-level paths that are read before any
+of this exists — nix's `authKeyFile`, sshd host keys, and ssh client keys used by bare `git push`.
+Those stay where the OS expects them; do not also copy them into the store, or you have two
+sources of truth again.
 
 **Rules of thumb**
 

secrets.py

@@ -20,25 +20,26 @@ USAGE (CLI):
     python3 engine/secrets.py get tangled.cookie      # value to stdout (careful in logs)
     python3 engine/secrets.py materialize <name>      # write a runtime file a consumer needs
 
-MATERIALIZED FILES: some consumers read a fixed path and cannot be taught otherwise (a
+NO SECOND COPIES. A secret must not be written to a second file "so something can read it" —
-systemd EnvironmentFile, an ssh IdentityFile, `nix`'s authKeyFile). Those files still exist
+copies drift from the store, get committed, and multiply what an attacker (or a careless
-on disk at 0600, but the STORE IS THE SOURCE OF TRUTH — `materialize` rewrites them from it.
+`grep`) can find. Consumers read the store: our own code imports this module; anything else
-Never edit a materialized file by hand; edit the store (`sops /secrets/store.yaml`) and
+gets the value injected at RUN TIME and nothing is left at rest.
-re-materialize, or the two silently drift.
+
+    secrets.py exec-env cc_ci_testenv -- some-command    # group as env vars, no file
+    secrets.py with-file ssh_keys.tangled-ed25519 -- ssh -i {} host   # 0600 file in a private
+                                                        # tmpdir, deleted when the command exits
+
+For systemd, wrap ExecStart in `exec-env` instead of using an EnvironmentFile — same effect,
+no plaintext on disk. The few OS-level paths that genuinely cannot be taught this (nix's
+`authKeyFile`, sshd host keys) are the exception, and are noted in engine/README.md.
 
 ADDING A SECRET:  sops /secrets/store.yaml   (opens decrypted in $EDITOR, re-encrypts on save)
 """
-import json, os, subprocess, sys, pathlib
+import json, os, subprocess, sys, pathlib, tempfile, shutil
 
 STORE = os.environ.get("AO_SECRETS_STORE", "/secrets/store.yaml")
 AGE_KEY = os.environ.get("SOPS_AGE_KEY_FILE", os.path.expanduser("~/.config/sops/age/keys.txt"))
 
-# name -> (path, mode). Files a consumer reads from a fixed location; see MATERIALIZED FILES.
-MATERIALIZE = {
-    "tangled-session": ("engine/.tangled-session", 0o600),   # relative to the project dir
-    "cc-ci-testenv":   ("/srv/cc-ci/.testenv", 0o600),
-}
-
 
 def _load():
     """Decrypt the store. Fails loudly: a silent empty dict would look like 'no secrets'."""
@@ -67,23 +68,32 @@ def get_group(name):
     return _load().get(name, {})
 
 
-def materialize(name, project_dir="."):
+def exec_env(group, argv):
-    """Write a runtime file from the store. Returns the path written."""
+    """Run argv with `group`'s keys added to the environment. Nothing touches the disk."""
-    if name == "tangled-session":
+    vals = get_group(group)
-        cookie = get("tangled.cookie")
+    if not vals:
-        if not cookie:
+        sys.exit(f"no group {group!r} in the store (secrets.py list)")
-            sys.exit("store has no tangled.cookie")
+    env = {**os.environ, **{k: str(v) for k, v in vals.items()}}
-        p = pathlib.Path(project_dir) / MATERIALIZE[name][0]
+    return subprocess.call(argv, env=env)
-        body = f"TANGLED_COOKIE={cookie}\n"
+
-    elif name == "cc-ci-testenv":
+
-        p = pathlib.Path(MATERIALIZE[name][0])
+def with_file(dotted, argv):
-        body = "".join(f"{k}={v}\n" for k, v in get_group("cc_ci_testenv").items())
+    """Run argv with `{}` replaced by a 0600 temp file holding the value.
-    else:
+
-        sys.exit(f"unknown materialize target {name!r} — known: {', '.join(MATERIALIZE)}")
+    The file lives in a private 0700 dir and is removed when the command exits — so a consumer
-    p.parent.mkdir(parents=True, exist_ok=True)
+    that insists on a path (ssh -i, a TLS key) never leaves a lasting second copy of the secret.
-    p.write_text(body)
+    """
-    p.chmod(MATERIALIZE[name][1])
+    val = get(dotted)
-    return str(p)
+    if val is None:
+        sys.exit(f"no such key: {dotted}")
+    d = tempfile.mkdtemp(prefix="ao-secret-")   # mkdtemp is 0700
+    try:
+        p = pathlib.Path(d) / dotted.split(".")[-1]
+        p.write_text(val if isinstance(val, str) else json.dumps(val))
+        p.chmod(0o600)
+        return subprocess.call([a.replace("{}", str(p)) for a in argv])
+    finally:
+        shutil.rmtree(d, ignore_errors=True)
 
 
 def main():
@@ -102,12 +112,16 @@ def main():
         if v is None:
             sys.exit(f"no such key: {argv[1]}")
         print(v if isinstance(v, str) else json.dumps(v))
-    elif cmd == "materialize":
+    elif cmd in ("exec-env", "with-file"):
-        if len(argv) < 2:
+        if "--" not in argv:
-            sys.exit(f"usage: secrets.py materialize <{'|'.join(MATERIALIZE)}>")
+            sys.exit(f"usage: secrets.py {cmd} <name> -- <command...>")
-        print("wrote", materialize(argv[1], argv[2] if len(argv) > 2 else "."))
+        i = argv.index("--")
+        if i != 2:
+            sys.exit(f"usage: secrets.py {cmd} <name> -- <command...>")
+        run = exec_env if cmd == "exec-env" else with_file
+        sys.exit(run(argv[1], argv[i + 1:]))
     else:
-        sys.exit(f"unknown command {cmd!r} — list | get | materialize")
+        sys.exit(f"unknown command {cmd!r} — list | get | exec-env | with-file")
 
 
 if __name__ == "__main__":

tangled_pr.py

@@ -26,15 +26,20 @@ import argparse, os, sys, urllib.request, urllib.parse, urllib.error
 
 BASE = "https://tangled.org"
 
-def load_cookie(path):
+def load_cookie(path=None):
-    if not os.path.exists(path):
+    """The cookie lives in the encrypted store (tangled.cookie) — see engine/README.md (Secrets).
-        sys.exit(f"no cookie file at {path} — do the one-time browser login "
+    `path` is a legacy escape hatch: a TANGLED_COOKIE=... file, used only if explicitly passed."""
-                 f"(see machine-docs/tangled-pr-automation.md)")
+    if path:
-    for line in open(path):
+        for line in open(path):
-        line = line.strip()
+            if line.strip().startswith("TANGLED_COOKIE="):
-        if line.startswith("TANGLED_COOKIE="):
+                return line.strip()[len("TANGLED_COOKIE="):]
-            return line[len("TANGLED_COOKIE="):]
+        sys.exit(f"{path} has no TANGLED_COOKIE= line")
-    sys.exit("cookie file present but has no TANGLED_COOKIE= line")
+    sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+    import secrets as _store
+    c = _store.get("tangled.cookie")
+    if not c:
+        sys.exit("no tangled.cookie in the secret store — add it with: sops /secrets/store.yaml")
+    return c
 
 def main():
     ap = argparse.ArgumentParser(description="file a Tangled PR via a reused session cookie")
@@ -45,7 +50,7 @@ def main():
     ap.add_argument("--fork", default="", help="fork repoDid for a cross-fork PR (omit for same-repo)")
     ap.add_argument("--title", default="")
     ap.add_argument("--body", default="")
-    ap.add_argument("--cookie-file", default=os.path.join(os.path.dirname(os.path.abspath(__file__)), ".tangled-session"))
+    ap.add_argument("--cookie-file", default=None, help="legacy: read the cookie from this file instead of the store")
     a = ap.parse_args()
 
     cookie = load_cookie(a.cookie_file)

tangled_pr_edit.py

@@ -22,15 +22,20 @@ import argparse, html, os, re, sys, urllib.request, urllib.parse, urllib.error
 
 BASE = "https://tangled.org"
 
-def load_cookie(path):
+def load_cookie(path=None):
-    if not os.path.exists(path):
+    """The cookie lives in the encrypted store (tangled.cookie) — see engine/README.md (Secrets).
-        sys.exit(f"no cookie file at {path} — do the one-time browser login "
+    `path` is a legacy escape hatch: a TANGLED_COOKIE=... file, used only if explicitly passed."""
-                 f"(see machine-docs/tangled-pr-automation.md)")
+    if path:
-    for line in open(path):
+        for line in open(path):
-        line = line.strip()
+            if line.strip().startswith("TANGLED_COOKIE="):
-        if line.startswith("TANGLED_COOKIE="):
+                return line.strip()[len("TANGLED_COOKIE="):]
-            return line[len("TANGLED_COOKIE="):]
+        sys.exit(f"{path} has no TANGLED_COOKIE= line")
-    sys.exit("cookie file present but has no TANGLED_COOKIE= line")
+    sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+    import secrets as _store
+    c = _store.get("tangled.cookie")
+    if not c:
+        sys.exit("no tangled.cookie in the secret store — add it with: sops /secrets/store.yaml")
+    return c
 
 def fetch_form(edit_url, cookie):
     """GET the htmx edit form; return (title, body) as the appview holds them."""
@@ -54,7 +59,7 @@ def main():
     ap.add_argument("--body", default=None)
     ap.add_argument("--body-file", default=None, help="read the new body from a file (overrides --body)")
     ap.add_argument("--show", action="store_true", help="print current title+body and exit (no edit)")
-    ap.add_argument("--cookie-file", default=os.path.join(os.path.dirname(os.path.abspath(__file__)), ".tangled-session"))
+    ap.add_argument("--cookie-file", default=None, help="legacy: read the cookie from this file instead of the store")
     a = ap.parse_args()
 
     cookie = load_cookie(a.cookie_file)

tangled_repo.py

@@ -13,14 +13,20 @@ import argparse, os, sys, urllib.parse, urllib.request
 
 BASE = "https://tangled.org"
 
-def load_cookie(path):
+def load_cookie(path=None):
-    if not os.path.exists(path):
+    """The cookie lives in the encrypted store (tangled.cookie) — see engine/README.md (Secrets).
-        sys.exit(f"no cookie file at {path} — refresh it with scripts/get-tangled-cookie.py")
+    `path` is a legacy escape hatch: a TANGLED_COOKIE=... file, used only if explicitly passed."""
-    for line in open(path):
+    if path:
-        line = line.strip()
+        for line in open(path):
-        if line.startswith("TANGLED_COOKIE="):
+            if line.strip().startswith("TANGLED_COOKIE="):
-            return line[len("TANGLED_COOKIE="):]
+                return line.strip()[len("TANGLED_COOKIE="):]
-    sys.exit("cookie file present but has no TANGLED_COOKIE= line")
+        sys.exit(f"{path} has no TANGLED_COOKIE= line")
+    sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+    import secrets as _store
+    c = _store.get("tangled.cookie")
+    if not c:
+        sys.exit("no tangled.cookie in the secret store — add it with: sops /secrets/store.yaml")
+    return c
 
 def main():
     ap = argparse.ArgumentParser(description="create a Tangled repo via a reused session cookie")
@@ -28,7 +34,7 @@ def main():
     ap.add_argument("--description", default="")
     ap.add_argument("--branch", default="main", help="default branch (form default: main)")
     ap.add_argument("--domain", default="knot1.tangled.sh", help="knot to host on (radio value)")
-    ap.add_argument("--cookie-file", default=os.path.join(os.path.dirname(os.path.abspath(__file__)), ".tangled-session"))
+    ap.add_argument("--cookie-file", default=None, help="legacy: read the cookie from this file instead of the store")
     a = ap.parse_args()
 
     cookie = load_cookie(a.cookie_file)