cctest-recipe-upstream: verify merge-base before rebasing; beware stray upstream main vs master

.opencode/skills/cctest-recipe-upstream/SKILL.md

@@ -24,3 +24,65 @@ run `/cctest-intro` / `/cctest-setup-sandbox` first.
   timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
 - The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
   a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
+
+**Sandboxed vs non-sandboxed mode.** This skill needs only git + (optionally) the Gitea API —
+not the test server — so it can run either way. Probe, then follow that branch:
+
+- **Sandboxed** (ARM env configured: `test-ssh/.testenv` with `GITEA_USERNAME`/`GITEA_PASSWORD`/
+  `GITEA_URL`, sandbox/test instances): the canonical `recipe-upstream` script in
+  `references/recipe-maintainer/.claude/commands/recipe-upstream.md` runs as written. Note its
+  WORKSPACE probing expects `/workspace` or `~/Documents/recipe-maintainer`; on a bare host pass
+  the submodule dir explicitly instead.
+- **Non-sandboxed** (no ARM env on the host — e.g. the orchestrator, where the recipe-maintainer
+  checkout is only a pinned submodule): no sandbox/test instances are needed and NONE of the
+  setup skills are. Recipe-maintainer mirrors on `git.autonomic.zone` are publicly readable, so:
+  1. Check out the recipe if missing: `abra recipe fetch <recipe>` (lands in `~/.abra/recipes/<recipe>`)
+     — or a plain anonymous `git clone https://git.autonomic.zone/recipe-maintainers/<recipe>.git`
+     if abra is unavailable.
+  2. Fetch the PR head branch from the mirror **anonymously** — no credentials in the remote URL:
+     `git remote add gitea https://git.autonomic.zone/recipe-maintainers/<recipe>.git`
+     (remote update rather than re-add if it exists), then
+     `git fetch gitea +refs/pull/<N>/head:refs/heads/<head_ref>`.
+  3. Fetch PR metadata (head/base refs, merged flag, release bump line) from
+     `https://git.autonomic.zone/api/v1/repos/recipe-maintainers/<recipe>/pulls/<N>` —
+     unauthenticated; use bot creds only if the repo turns out to be private (orchestrator hosts
+     can read them from `/srv/cc-ci-orch/.testenv` — never written anywhere else).
+  4. Everything else in the canonical script (origin/dev remote setup, release recommendation,
+     emitted next-steps) is identical.
+
+Every time the branch was prepared **here**, remember it exists only on this host — the
+operator's machine must fetch it first. Always emit this **step 0** before the push step
+(anonymous public fetch, no credentials needed):
+
+```
+  # 0. On a machine WITHOUT the branch pre-fetched, get it from the autonomic mirror
+  #    (fetch by URL — works regardless of what the local remotes are named):
+  cd <local checkout of the recipe>
+  git fetch https://git.autonomic.zone/recipe-maintainers/<recipe>.git +refs/pull/<PR_NUM>/head:refs/heads/<HEAD_REF>
+  git checkout <HEAD_REF>
+```
+
+If the operator's checkout does NOT yet have the mirror remote, emit once before the fetch:
+
+```
+  git remote add gitea https://git.autonomic.zone/recipe-maintainers/<recipe>.git
+```
+
+In both modes the final output is a set of commands for the operator to run on a machine **with
+push access to `git.coopcloud.tech`** — always print them, even when everything local is
+already prepared.
+
+**Verify the merge-base BEFORE rebasing — bases can be traps.** Upstream repos can
+carry a stray, divergent `main` ALONGSIDE the real `master` base branch. Rebase targets must be
+chosen by evidence: `git ls-remote origin | grep -E 'refs/heads/(main|master)$'`, then
+`git merge-base <head_ref> origin/<candidate>` — the correct base is the one where the upgrade
+branch's merge-base is its tip's parent (i.e. `<head_ref>` is already directly on that line —
+then NO cherry-pick is needed, the raw branch compares clean). Cherry-picking onto the wrong
+base produces the very messy 3-dot compare this guard exists to prevent (real case: gitea —
+first "rebase" landed on a stray `main` and had to be redone onto `master`).
+
+**Branch-name mismatch mirror vs upstream.** The mirror and upstream can use different base-branch
+names (gitea: mirror `main`, upstream `master`). When emitting step 3 (the post-merge release),
+NEVER hardcode `main` — derive the upstream default branch from the existing remotes (`git
+remote show origin` or `git ls-remote --symref origin HEAD`) and emit `git checkout <that>;
+git fetch origin; git merge --ff-only origin/<that>;` before `abra recipe release`.

.opencode/skills/recipe-enroll/SKILL.md

@@ -43,7 +43,10 @@ On cc-ci, after `abra recipe fetch <recipe>`, read `~/.abra/recipes/<recipe>/`:
 ### 2. Create + sync the mirror
 
 Create `recipe-maintainers/<recipe>` (Gitea API: `POST /orgs/recipe-maintainers/repos`,
-`{"name":…,"private":true,"default_branch":"main","auto_init":false}`), then force-sync from
+`{"name":…,"private":false,"default_branch":"main","auto_init":false}`) — a recipe mirror MUST be
+created **public**: the cc-ci recipe mirrors are public (fleet invariant; the report's live PR-STATUS
+proxy `report./pr/<recipe>/<n>` is tokenless and only sees public repos — private-from-birth mirrors
+like gitea/wordpress left its status cells dark until flipped). Then force-sync from
 coopcloud with the existing helper **run on cc-ci with creds injected via stdin** (the host has
 no `.testenv`):
 ```

cc-ci-plan/JOURNAL.md

@@ -31,8 +31,44 @@ handoff).
 
 ---
 
-## Session 2026-05-31 ~18:30 UTC — Claude Sonnet 4.6
+## Session 2026-09-14 ~16:45 UTC — opencode glm-5.3-flash (orchestrator) — round 2: Anubis UA
 
+**Left off:** The real root cause turned out to be TWO independent layers; the mirror-privacy
+fix (earlier session entry today) was necessary but not sufficient. Operator's browser console
+showed CORS failures redirecting to `anubis.swarm.autonomic.zone/.within.website/?redir=…`.
+Reproduced exactly: the `/pr/` proxy forwards the END browser's User-Agent to Gitea; Gitea sits
+behind **Anubis**, which 307-challenges browser-like UAs to `anubis.swarm.autonomic.zone`
+(no CORS headers) → every fetch throws in the browser → all cells "?" (curl passed clean, which
+is why server-side checks and my earlier headless test never saw it — intermittent/rate-dependent
+for my playwright run). Fix: cc-ci **PR #38** adds
+`proxy_set_header User-Agent "ccci-reports-proxy/1.0";` to the reports.nix `/pr/` location.
+Hot-verified on the host by mount-swapping a fixed conf into the running task (one mis-step:
+`--mount-rm`+`--mount-add` same-target order wiped the mount; re-added), scoped live, all 16
+cells rendering with a real Chromium. Merged PR #38, `nix flake update cc-ci`,
+`nixos-rebuild test` → healthy (reports 200, no failed units) → `switch` (flake.lock commit
+9e7770f). Final verify: browser-UA curl 200 both gitea/9 + full headless-Chromium sweep 16/16
+OPEN, zero non-200 /pr fetches. Also flipped memory: `memory/gitea-anubis-ua-challenge.md` +
+MEMORY.md index.
+**Open:** nothing blocking; next weekly /recipe-report and STATUS live-checks carry the fix.
+
+## Session 2026-09-14 ~15:00 UTC — opencode glm-5.3-flash (orchestrator)
+
+**Left off:** Report STATUS column fix. Operator reported the week-2026-09-11 report's live
+PR-STATUS column all "?" — root cause: the tokenless same-origin proxy
+`report./pr/<recipe>/<n>` (cc-ci `nix/modules/reports.nix`) 404s on **private** mirrors; two
+late-enrolled mirrors, `recipe-maintainers/gitea` (2026-06-11) and `wordpress` (2026-08-03),
+had been created `"private":true` from birth — by the stale instruction in
+`/recipe-enroll`'s mirror step (the other 21 mirrors were flipped public on 2026-06-09, and
+the old 'org is private' blocker is long resolved). Fixed: secret-scanned both repos, flipped
+`private=false` (PATCH with bot creds), patched `.opencode/skills/recipe-enroll/SKILL.md` to
+create mirrors `private:false`, updated memory/recipe-mirrors-public-org-blocker.md +
+MEMORY.md index. **Verified in a real headless Chromium (nixpkgs chromium + playwright)**:
+all 16 STATUS rows render `open`, every `/pr/` fetch 200 JSON. Commit 6e93922 pushed. The
+STATUS column refreshes live every 30s; cells go ✓ when a PR merges. No reports.nix change
+was needed (proxy itself was healthy).
+**Open:** nothing on this; the report index regenerates next weekly run.
+
+## Session 2026-05-31 ~18:30 UTC — Claude Sonnet 4.6
 **Left off:** Got opencode/deepseek-v4-pro working as the loop backend. Both builder and
 adversary are actively running on `tinfoil/deepseek-v4-pro` (via `inference.tinfoil.sh`).
 Phase 5 [11/11] in progress. The operator is debugging the opencode web UI visibility and

flake.lock

@@ -10,11 +10,11 @@
         ]
       },
       "locked": {
-        "lastModified": 1788820034,
-        "narHash": "sha256-mkktTMeGKutfPp3pn3AhoPoGSCm4iRSnPvpmbEpMmwA=",
+        "lastModified": 1789404337,
+        "narHash": "sha256-FY5oOz/C6i6Ct6Qe6DqN6nOq+TXCyiZq5Am1LKLpDss=",
         "ref": "refs/heads/main",
-        "rev": "b11cc0b7385aee9fb89bd9a7ed23896401d683e5",
-        "revCount": 1543,
+        "rev": "eb5fb826114f2b141dffb270fd36f7a22d1bf343",
+        "revCount": 1550,
         "type": "git",
         "url": "https://git.autonomic.zone/recipe-maintainers/cc-ci.git"
       },

memory/MEMORY.md

@@ -3,7 +3,8 @@
 - [Orchestrator host: Hetzner](orchestrator-host-hetzner.md) — runs on Hetzner cpx22; rebuild cmd, loops-service bounce, git-identity gotcha
 - [Push commits to remote](push-commits-to-remote.md) — push to git.autonomic.zone right after every commit in this repo
 - [Regression canary cadence](regression-canary-cadence.md) — server E2E canaries run on polish/review/release, not every commit
-- [Recipe-mirrors public / org blocker](recipe-mirrors-public-org-blocker.md) — mirrors public but recipe-maintainers ORG is private → live PR-STATUS column dark until operator flips org public
+- [Recipe-mirrors MUST be public](recipe-mirrors-public-org-blocker.md) — org public since ~2026-06, but /recipe-enroll created late mirrors (gitea, wordpress) private-from-birth, darkening report STATUS cells; flipped public + skill fixed 2026-09-14
+- [Gitea Anubis UA challenge](gitea-anubis-ua-challenge.md) — git.autonomic.zone 307-challenges browser UAs to an un-CORS-able origin; server-side proxies representing browser JS must pin a non-browser UA (cc-ci PR #38, 2026-09-14)
 - [abra chaos-deploy checkout gotcha](abra-chaos-deploy-checkout-gotcha.md) — `abra app new` moves recipe checkout to release tag; checkout PR branch after, or chaos deploys wrong tree
 - [Shared recipe-checkout race](shared-recipe-checkout-race.md) — never git-checkout ~/.abra/recipes/<recipe> on cc-ci while its CI build runs; harness deploys from that tree
 - [immich pgvecto.rs DROP DATABASE panic](immich-pgvectors-drop-database-panic.md) — DROP DATABASE crashes immich's postgres image; use pg_dump --clean --if-exists + search_path rewrite

memory/gitea-anubis-ua-challenge.md

@@ -0,0 +1,26 @@
+---
+name: gitea-anubis-ua-challenge
+description: "Gitea (git.autonomic.zone) sits behind Anubis, which 307-challenges browser-like User-Agents to an un-CORS-able counter-domain — any server-side proxy that forwards a browser UA on behalf of client JS breaks; pin a non-browser UA"
+metadata: 
+  node_type: memory
+  type: project
+---
+
+`git.autonomic.zone` sits behind **Anubis** (`anubis.swarm.autonomic.zone`). Anubis
+307-challenges requests whose User-Agent looks like a real browser to
+`/.within.website/?redir=…` — a JS proof-of-work challenge page on a *different* origin with
+**no CORS headers**.
+
+Impact: any server-side proxy that serves client-side JS but forwards the *end browser's* UA
+hits this asymmetry — `curl` (non-browser UA) passes through cleanly, a real Firefox/Chrome UA
+gets the 307. Concretely: the Recipe Report's same-origin PR-STATUS proxy
+(`report./pr/<recipe>/<n>`, cc-ci `nix/modules/reports.nix`) forwarded the browser UA, so
+every live cell in the browser rendered `?` (week-2026-09-11, operator-CORS console report).
+
+Fix (cc-ci PR #38, merged 2026-09-14): `proxy_set_header User-Agent "ccci-reports-proxy/1.0";`
+in the `/pr/` nginx location — a stable non-browser UA passes Anubis unmolested. Deployed via
+`nix flake update cc-ci` → `nixos-rebuild test` → health → `switch` (flake.lock commit).
+
+Rule of thumb: server-side callers of the Gitea API (scripts, proxied fetches, harness code
+running in a browser's name) must pin a non-browser User-Agent or cookie-carry; browser-visit
+flows keep the JS challenge and that's fine. Related: [[recipe-mirrors-public-org-blocker]].

memory/recipe-mirrors-public-org-blocker.md

@@ -1,29 +1,26 @@
 ---
 name: recipe-mirrors-public-org-blocker
-description: "Recipe mirrors are public repos but the recipe-maintainers ORG is private-visibility, so anon reads 404; bot can't flip the org"
+description: "Recipe mirrors MUST be public (fleet invariant: the report's tokenless PR-STATUS proxy only sees public repos); org is public since ~2026-06, but late-enrolled mirrors (gitea, wordpress) were created private-from-birth by /recipe-enroll — fixed 2026-09-14"
 metadata:
   node_type: memory
   type: project
   originSessionId: f7960036-d990-4a21-a81e-f7c486d97fea
 ---
 
-As of 2026-06-09 all 21 recipe mirrors under `recipe-maintainers` were flipped `private=false`
-(secret-scanned first), to power the Recipe Report's live PR-STATUS column via the tokenless
-same-origin proxy `report.ci.commoninternet.net/pr/<recipe>/<n>` (shipped in cc-ci
-`nix/modules/reports.nix`). BUT the **org itself is `visibility: private`**, which makes Gitea 404
-all its repos for anonymous users — so the live STATUS column shows a muted "?" instead of open/✓.
+The Recipe Report's live PR-STATUS column (`report.ci.commoninternet.net/pr/<recipe>/<n>`,
+shipped in cc-ci `nix/modules/reports.nix`) is a tokenless same-origin nginx proxy to the Gitea
+API — it can ONLY see public repos. So **`recipe-maintainers/<recipe>` mirrors MUST be public**
+(fleet invariant; cc-ci/cc-ci-orchestrator/archived repos stay deliberately private).
 
-**Blocker:** `autonomic-bot` cannot flip the org (PATCH `/orgs/recipe-maintainers` → 403 "Must be an
-organization owner"; `is_admin=false`; the basic-auth credential lacks `write:organization` scope,
-even though the bot is in the Owners team). Confirmed model: `autonomic-cooperative` is a public org
-and its repos ARE anonymously visible; `recipe-maintainers` is private and they are not.
+History: originally ALL mirrors were private; on 2026-06-09 the 21 then-existing mirrors were
+flipped `private=false` after a secret-scan, and the `recipe-maintainers` org was later flipped
+public by the operator (the old 'org is private' blocker is RESOLVED).
 
-**Why:** the whole live-status feature is dark until this is resolved. Private repos stay hidden even
-in a public org, so flipping the org public does NOT expose the four locked-private repos (`cc-ci`,
-`cc-ci-secrets`, `cc-ci-orchestrator`, `archived-cc-ci-orchestrator`).
-
-**How to apply:** operator (an org owner) must set `recipe-maintainers` org visibility to **public**
-in the Gitea UI (Settings → make org public), OR provision a token with `write:organization` scope.
-The instant that happens, the proxy returns 200 PR JSON and the column lights up — no redeploy needed.
-Verify: `curl https://report.ci.commoninternet.net/pr/cryptpad/5` should return PR JSON, not a 404.
+Recurrence (2026-09-14): /recipe-enroll still carried the old instruction to create mirrors with
+`"private":true` — so the later-enrolled mirrors `gitea` (2026-06-11) and `wordpress` (2026-08-03)
+were **private from birth**, leaving the report STATUS cells for those rows permanently "?"
+(gitea/wordpress rows in week-2026-09-11). Fixed: secret-scanned both, flipped `private=false`
+via PATCH `/repos/recipe-maintainers/<r>`, patched the skill to create `private:false`. The
+report now lights up for every row — verify any new PR row with
+`curl https://report.ci.commoninternet.net/pr/<recipe>/<n>` returning PR JSON, not 404.
 Related: [[push-commits-to-remote]].

scripts/gen-cctest-skills.py

@@ -50,6 +50,73 @@ run `/cctest-intro` / `/cctest-setup-sandbox` first.
   a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
 """
 
+# Per-skill extra body sections appended after BODY_TEMPLATE (survive regeneration).
+PER_SKILL_NOTES: dict[str, str] = {
+    "recipe-upstream": """
+**Sandboxed vs non-sandboxed mode.** This skill needs only git + (optionally) the Gitea API —
+not the test server — so it can run either way. Probe, then follow that branch:
+
+- **Sandboxed** (ARM env configured: `test-ssh/.testenv` with `GITEA_USERNAME`/`GITEA_PASSWORD`/
+  `GITEA_URL`, sandbox/test instances): the canonical `recipe-upstream` script in
+  `references/recipe-maintainer/.claude/commands/recipe-upstream.md` runs as written. Note its
+  WORKSPACE probing expects `/workspace` or `~/Documents/recipe-maintainer`; on a bare host pass
+  the submodule dir explicitly instead.
+- **Non-sandboxed** (no ARM env on the host — e.g. the orchestrator, where the recipe-maintainer
+  checkout is only a pinned submodule): no sandbox/test instances are needed and NONE of the
+  setup skills are. Recipe-maintainer mirrors on `git.autonomic.zone` are publicly readable, so:
+  1. Check out the recipe if missing: `abra recipe fetch <recipe>` (lands in `~/.abra/recipes/<recipe>`)
+     — or a plain anonymous `git clone https://git.autonomic.zone/recipe-maintainers/<recipe>.git`
+     if abra is unavailable.
+  2. Fetch the PR head branch from the mirror **anonymously** — no credentials in the remote URL:
+     `git remote add gitea https://git.autonomic.zone/recipe-maintainers/<recipe>.git`
+     (remote update rather than re-add if it exists), then
+     `git fetch gitea +refs/pull/<N>/head:refs/heads/<head_ref>`.
+  3. Fetch PR metadata (head/base refs, merged flag, release bump line) from
+     `https://git.autonomic.zone/api/v1/repos/recipe-maintainers/<recipe>/pulls/<N>` —
+     unauthenticated; use bot creds only if the repo turns out to be private (orchestrator hosts
+     can read them from `/srv/cc-ci-orch/.testenv` — never written anywhere else).
+  4. Everything else in the canonical script (origin/dev remote setup, release recommendation,
+     emitted next-steps) is identical.
+
+Every time the branch was prepared **here**, remember it exists only on this host — the
+operator's machine must fetch it first. Always emit this **step 0** before the push step
+(anonymous public fetch, no credentials needed):
+
+```
+  # 0. On a machine WITHOUT the branch pre-fetched, get it from the autonomic mirror
+  #    (fetch by URL — works regardless of what the local remotes are named):
+  cd <local checkout of the recipe>
+  git fetch https://git.autonomic.zone/recipe-maintainers/<recipe>.git +refs/pull/<PR_NUM>/head:refs/heads/<HEAD_REF>
+  git checkout <HEAD_REF>
+```
+
+If the operator's checkout does NOT yet have the mirror remote, emit once before the fetch:
+
+```
+  git remote add gitea https://git.autonomic.zone/recipe-maintainers/<recipe>.git
+```
+
+In both modes the final output is a set of commands for the operator to run on a machine **with
+push access to `git.coopcloud.tech`** — always print them, even when everything local is
+already prepared.
+
+**Verify the merge-base BEFORE rebasing — bases can be traps.** Upstream repos can
+carry a stray, divergent `main` ALONGSIDE the real `master` base branch. Rebase targets must be
+chosen by evidence: `git ls-remote origin | grep -E 'refs/heads/(main|master)$'`, then
+`git merge-base <head_ref> origin/<candidate>` — the correct base is the one where the upgrade
+branch's merge-base is its tip's parent (i.e. `<head_ref>` is already directly on that line —
+then NO cherry-pick is needed, the raw branch compares clean). Cherry-picking onto the wrong
+base produces the very messy 3-dot compare this guard exists to prevent (real case: gitea —
+first "rebase" landed on a stray `main` and had to be redone onto `master`).
+
+**Branch-name mismatch mirror vs upstream.** The mirror and upstream can use different base-branch
+names (gitea: mirror `main`, upstream `master`). When emitting step 3 (the post-merge release),
+NEVER hardcode `main` — derive the upstream default branch from the existing remotes (`git
+remote show origin` or `git ls-remote --symref origin HEAD`) and emit `git checkout <that>;
+git fetch origin; git merge --ff-only origin/<that>;` before `abra recipe release`.
+""",
+}
+
 WRAPPER_TEMPLATE = """# {wrapped} (thin wrapper)
 
 The canonical definition of this skill lives in the **opencode** position:
@@ -99,7 +166,7 @@ def main() -> int:
         )
         canon = ROOT / ".opencode/skills" / wrapped / "SKILL.md"
         canon.parent.mkdir(parents=True, exist_ok=True)
-        canon.write_text(frontmatter + BODY_TEMPLATE.format(name=name, wrapped=wrapped))
+        canon.write_text(frontmatter + BODY_TEMPLATE.format(name=name, wrapped=wrapped) + PER_SKILL_NOTES.get(name, ""))
         thin = ROOT / ".claude/skills" / wrapped / "SKILL.md"
         thin.parent.mkdir(parents=True, exist_ok=True)
         thin.write_text(frontmatter + WRAPPER_TEMPLATE.format(name=name, wrapped=wrapped))