Operator: rather than documenting the workaround, notplants-nix should put the system profile on
the agent PATH. It already does (two orchestrator units export it); the gap is that the change is
committed and not yet deployed. The /proc rewrite in agents.py stands on its own — harness code
should not depend on an external tool for something the kernel exposes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3LdmEL7CvCYTNpoBq1kce
THE BUG THAT PROMPTED THIS. tangled_pr.py judged success ONLY by an HX-Redirect header on the POST.
A create that SUCCEEDED but answered without that header read as a failure, so the caller retried
and Tangled grew duplicates — that is exactly how #397, #398 and #399 were filed for one branch. A
response header describes what the server meant to say; it is not the artifact.
Now it checks the artifact, in both directions:
* BEFORE posting, refuse if an open pull already exists for this source branch, naming it. A
retry cannot duplicate, whatever the response said. (--allow-duplicate to override.)
* AFTER posting, confirm against the pulls list: a new pull number that did not exist before,
whose page names this source branch, IS the success — with or without a redirect header.
* Failure is reported only when no such pull appeared. A false failure is worse than a loud
error here, because the caller's remedy is to retry.
Verified live: a dry-run against a branch that already has a pull refuses with rc=3, naming #417.
TWO REAL DEFECTS FOUND BY WRITING THE TESTS.
agents.py shelled out to `pgrep -P` and `ps -o comm=`. Neither is on the agent PATH on this host,
and a missing binary under shell=True returns rc=127 with EMPTY stdout — indistinguishable from
"this process has no children" and "no build is running". So _build_running was ALWAYS False and
the stall detector could reboot an agent mid-build. Both now read /proc directly: no PATH
dependency, and it cannot fail silently in that direction.
That shipped because the unit tests MOCKED pgrep and ps. The fakes stood in for the broken
dependency, so the suite passed on a host where neither tool was reachable and never exercised the
real path. The tests now patch _proc_descendants and _comms — the seams this repo owns. A test that
mocks a dependency proves the mock works.
Also fixed a monkeypatch leak those tests had: restoration used a name derivation that silently
matched nothing, so the patch escaped into another test class and failed an unrelated test — only
in a full run, never when that test ran alone. Now addCleanup, which cannot be ordered wrong.
NEW: tests/test_tools.py, 24 tests over tangled_pr, tangled_pr_close and gateway-domain, with every
HTTP boundary injected so they run offline. Mutation-checked: breaking classify(), the pull-number
regex, the branch match, or the scan bound each turns the suite red. Suite is 93 tests, green, and
order-stable across repeated runs.
README: a "PATH on a NixOS host" section. Every one of ps, pgrep, free, cmp, awk, curl, diff,
strings, nm, getent and ping is INSTALLED here and simply not on the agent PATH, so each reports
"command not found" and reads as a missing package. Documents how to check before concluding a tool
is absent, how to add the system profile, `nix shell` for what is genuinely missing, and the rule
that harness code should not shell out for what the kernel already exposes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3LdmEL7CvCYTNpoBq1kce
An agent on a box with no public IP frequently needs a reachable HTTPS URL —
an OAuth callback, a webhook receiver, a demo link. The testing gateway
already holds a wildcard record for *.gtest.commoninternet.net and forwards
by SNI, but nothing here knew that, so every agent had to be told by hand.
tools/gateway-domain.py add myapp
# myapp.gtest.commoninternet.net -> 100.84.190.30
The backend defaults to the running box's own tailscale IP, which is the case
that comes up almost every time.
The admin password comes from gateway.admin_password in the secret store; the
tool reads it itself, so no caller handles the value and there is no second
copy to drift or get committed.
Two things the tool refuses to do, both learned by doing them:
Backends must be a literal IPv4 address. The gateway's validate_ip accepts a
hostname, but put_domain/remove_domain only match lines whose backend is
numeric ([\d.:]+). A hostname mapping can therefore be written once and never
updated or removed through the admin UI — it becomes an orphan that only a
hand-edit of tunnel_map.conf clears. One got created while testing this.
Verification re-reads the mapping table instead of trusting the POST body.
The admin app mutates its in-memory dict and renders that, so a delete that
silently failed still renders as success. Checking the response alone
reported "removed" for an entry that was still on disk.
skills/gateway-domain/ carries the rest: that the gateway does NOT terminate
TLS (your box serves the cert for that name), how ACME still works through
it, that only 22/80/443 are open at the edge, and how to recover if an
interrupted e2e run leaves the admin password reseeded.
Pushing a fixup to the source branch does NOT update the pull: the appview serves the patch
it fetched when the pull was opened or last resubmitted, so a reviewer keeps reading the
pre-fixup code and no `/round/<r>/interdiff` exists. Nothing warns you — the push succeeded,
the branch is right, the pull silently lags. It cost a review cycle on 2026-08-16 (three
flat-file PRs reviewed against stale trees because "pushed" was read as "resubmitted").
The appview's action is a bare htmx POST with no body,
`POST /{owner}/{repo}/pulls/{n}/resubmit`, which re-fetches the branch patch from the knot
and opens the next round. The tool wraps it and:
- takes several `--pull` numbers at once (a stack-wide rebase touches every rung);
- prints the new round and its interdiff URL — what a reply to a reviewer leads with;
- says plainly when the round did NOT move (the branch already matched the pull) instead of
reporting a silent success;
- `--check` reports the current round without posting.
It reads the round from the round selector rather than the page text, because a comment that
quotes a `/round/2/interdiff` URL would otherwise be counted as a round (that fooled me
first).
README gains a table of the Tangled tools, which did not exist, and states the trap.
A secret a third party reads from a fixed path (ssh key, systemd EnvironmentFile, nix
authKeyFile, TLS keypair) now lives ONCE as a real file in /secrets/files and is symlinked
from where the consumer expects it. The consumer is unchanged and unaware; the file exists
in one directory, at 0600, outside /srv and outside every git tree.
That makes the store and the file directory alternatives, not layers: a secret is a value in
store.yaml OR a file in /secrets/files, never both. The copies of the ssh keys, tailscale
auth key, incus keypair, LE cert and cc-ci testenv have been dropped from store.yaml now that
each has a single home.
Documented exception: an app that rewrites its own credential file (OAuth refresh via
write-temp+rename) replaces the symlink with a regular file and silently re-splits the home.
opencode's auth.json is one, so it stays put and is deliberately not centralised.
Co-Authored-By: Claude <noreply@anthropic.com>
Materializing wrote a second plaintext file per secret, which is the problem the store
was meant to solve: two files drift, and the copy is what ends up committed or grepped.
- tangled_pr / tangled_pr_edit / tangled_repo now read tangled.cookie from the store.
engine/.tangled-session is deleted; --cookie-file remains as a legacy escape hatch.
- materialize() is replaced by run-time injection that leaves nothing at rest:
exec-env <group> -- cmd group as env vars (use this instead of a systemd
EnvironmentFile — same effect, no plaintext on disk)
with-file <key> -- cmd {} 0600 file in a private tmpdir, removed when cmd exits,
for consumers that insist on a path (ssh -i, a TLS key)
Co-Authored-By: Claude <noreply@anthropic.com>
/srv is the projects tree — agents grep, find and `ls -R` it constantly, so a store
under it turns up in ordinary searches and risks being read (or pasted) by accident.
/secrets sits outside that blast radius: nothing routinely walks it, and it is still
0700 loops, still not a repo, still ciphertext at rest.
Override with AO_SECRETS_STORE if a host puts it elsewhere.
Co-Authored-By: Claude <noreply@anthropic.com>
Credentials were scattered in plaintext: a gitea password baked into six git remote
URLs (`git remote -v` prints those), API keys in .env files, an incus client key at
0644. Anything living in a repo is one `git add -A` from being pushed.
So: ONE encrypted file outside every git tree, and a helper each project uses.
/srv/secrets/store.yaml sops+age ciphertext, 0600, not a repo, no remote
~/.config/sops/age/keys.txt the only plaintext secret on disk, 0600
secrets.py is stdlib + the sops binary: get("group.key"), get_group("group"), and
materialize() for consumers that must read a fixed path (systemd EnvironmentFile,
ssh IdentityFile, nix authKeyFile) — those keep their file, but the store is the
source of truth, so a materialized file is never hand-edited.
`list` prints names only, never values, so it is safe in a transcript.
Co-Authored-By: Claude <noreply@anthropic.com>
Multi-reviewer setups (e.g. a correctness + a readability adversary) can now have
the watchdog ping ALL reviewers on a claim, each in its own session with its own
submit key. A bare string still works (single agent). _ping_agents() helper.
When [watchdog].log_tokens (or [loop].log_tokens) is true, the watchdog records
for each phase how many tokens each agent used (and the total) and how long the
phase took, appended to <log_dir>/token-log.jsonl. Tokens are summed from each
agent's session transcript, attributed by working dir. View with `agents.py
tokens`. Baseline snapshot at phase start + delta at phase advance/complete;
robust across watchdog restarts. Validated: the transcript sum matches an
independent external collector exactly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Unit tests (no agents/tmux): config load + defaults merge, kickoff-template
assembly, phase machine (advance/idempotent-complete/append-resumes), limit
reset-banner parsing, WAITING-UNTIL/stall parsing, claude+opencode activity
detectors. Live smokes bring a throwaway project up THROUGH agents.py on each
real backend in an isolated sandbox (unique prefix, opencode on a non-4096
port), verify attach + status + down, and clean up. tests/run.sh runs unit
always + smokes when backends present; README documents it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
notplantsandClaude Fable 5
autonomic-botandClaude Opus 4.8