cc-ci/machine-docs/REVIEW-gtea.md

REVIEW — phase gtea (gitea full-test enrollment)

Adversary verdict log. Append-only. Only the Adversary writes here. Commit prefix: review(gtea): ...

---

Init @2026-06-15T19:33Z

Phase gtea started. No gates claimed yet by Builder. Baseline orientation run:

• Builder hasn't started (no STATUS-gtea.md, no gtea commits on origin/main as of 3f6d7dc).
• Existing tests/gitea/recipe_meta.py is the dep-provider stub (header: "NOT a standalone recipe-under-test").
• Plan SSOT loaded: plan-phase-gtea-gitea-fulltests.md — M1 = suite green locally; M2 = green in real CI + LFS PR verified.
• Exemplars to check: tests/cryptpad/, tests/keycloak/.
• Will maintain independent break-it probes while Builder builds.

---

Pre-M1 code review @2026-06-15T19:58Z

Builder commit 33561c8 (all files) + 6ac9989 (Playwright fix) read.

PASS items

• recipe_meta.py: READY_PROBE(ctx) and SCREENSHOT(page, ctx) signatures match registry hook_params ✓
• BACKUP_CAPABLE=True explicit (compose.yml backupbot.backup=true confirmed) ✓
• EXTRA_ENV dep path unchanged: sqlite3 + relaxed auth; LFS guard requires RECIPE=gitea AND overlay file ✓
• PARITY.md honest about absent upstream tests (source note says recipe-info corpus, not upstream) ✓
• ops.py pre_restore deletes marker + asserts absence — divergence is real ✓
• test_restore.py asserts marker returned — a no-op restore would fail ✓
• harness.http.retry_http_get, lifecycle.http_fetch, lifecycle.exec_in_app all exist in the harness ✓
• PARITY.md: beyond-parity test rationale non-vacuous ✓
• Playwright fix: wait_for_selector("input#user_name") is visible — correct ✓

ISSUES filed (in BUILDER-INBOX.md @4a4b756)

[critical — M2 blocker] git-lfs not installed on cc-ci: git lfs is not a git subcommand. The LFS test uses git lfs install/track/ls-files — all fail without git-lfs. Fix: add git-lfs to nix/hosts/cc-ci/configuration.nix systemPackages, rebuild, deploy.

[bug in test_lfs_roundtrip.py] Double /api/v1 path: _api(live_app, "/api/v1/version", ...) constructs https://domain/api/v1/api/v1/version → 404. The restart health-poll will spin 120s then fail. Fix: change path argument to "/version".

Both issues affect only the LFS capstone (which skips on main). Do NOT block M1 verdict. M2 verdict will FAIL unless both are fixed before the lfs-plain-gitea run.

Additional pre-M1 cold checks @2026-06-15T20:10Z

Builder addressed inbox findings in commits 893a7b0, 3cc8338, 74bc5f0, 3ec24b0:

• Double /api/v1 path bug: FIXED ("/version" path used correctly) ✓
• git-lfs: added to nix/hosts/cc-ci-hetzner/configuration.nix (correct host config) ✓
• test_git_push: auto_init=True repo, credential URL approach ✓
• test_admin_api: scopes added for gitea 1.22+ ✓

Cold checks run from cc-ci /root/builder-clone (HEAD 3ec24b0):

• recipe_meta.py: all keys load — BACKUP_CAPABLE=True, READY_PROBE callable, SCREENSHOT callable, EXTRA_ENV callable ✓
• unit tests: 53/53 PASS (test_gitea_dep.py 10/10, test_meta.py 43/43) ✓
• LFS conditional (RECIPE=gitea, compose.lfs.yml absent): COMPOSE_FILE=sqlite3 only, LFS=False ✓
• LFS skip mechanism: _lfs_enabled() returns False when compose.lfs.yml absent (main branch) ✓

M1 cold verification @2026-06-15T20:32Z

Builder claim: commit bac3662, all 5 stages PASS locally (RECIPE=gitea), run_id=manual.

Evidence reviewed (independent, from adv-clone at HEAD b2663dc)

results.json (/var/lib/cc-ci-runs/manual/results.json, mtime 20:08 today):

• level: 5/5 ✓
• install/upgrade/backup/restore/custom: all "pass" ✓
• lint: "pass" ✓
• LFS (test_lfs_roundtrip): status="skip", message="compose.lfs.yml absent in gitea recipe checkout — LFS is not enabled on this branch. This test runs on lfs-plain-gitea (PR #1) and is EXPECTED_NA on main." ✓
• flags: clean_teardown=true, no_secret_leak=true ✓
• customization: 4 custom tests, ops.py hooks for all 4 pre-op stages, meta non-default keys all correct ✓
• unintentional skips: [] (no unexpected skips) ✓

Unit tests (Adversary cold run from adv-clone):

• 53/53 PASS (test_gitea_dep.py 10/10, test_meta.py 43/43) ✓
• test_gitea_recipe_meta_extra_env PASS — dep env correct (no LFS when RECIPE≠gitea) ✓
• test_enrich_deps_routes_gitea PASS — dep routing intact ✓
• test_drone_recipe_meta_deps PASS — DEPS=["gitea"] correct ✓

Code review of test hooks:

• test_restore: pre_restore DELETES marker + asserts absence; test asserts marker RETURNED — no-op restore fails ✓
• test_upgrade: marker_repo_exists() hits API with admin creds — data continuity is real ✓
• test_git_push: auto_init=True repo, credential URL embedded, push via git; verifies non-empty response ✓
• test_admin_api: creates user, org, token via API with 1.22+ scopes; teardown cleans up ✓
• test_health: HTTP 200 on root endpoint ✓
• LFS conditional: 2-guard (_lfs_enabled requires RECIPE=gitea AND compose.lfs.yml exists) prevents dep leak ✓

Dep path verification:

• No RECIPE=drone CI run post-Builder changes (last drone run was #506, June 13)
• EXTRA_ENV dep path verified code-level: RECIPE=drone → no LFS flags, standard sqlite3+auth only ✓
• Unit tests cover this path explicitly ✓

Findings

[non-blocking, pre-existing harness bug] Stale screenshot: /var/lib/cc-ci-runs/manual/screenshot.png has mtime June 13 — not from today's M1 run. Root cause: screenshot.capture() checks if not os.path.exists(out_path) after running the SCREENSHOT hook; since the file exists from a prior manual run (run_id="manual" reuses the same dir), _snap_with_blank_retry is never called and the old file persists. results.json reports "screenshot": "screenshot.png" (file exists and is non-empty), but it's a stale image. Non-blocking per R7 (cosmetics never change verdict). M2 will use DRONE_BUILD_NUMBER as run_id → fresh directory → no issue. NOT a Builder error; pre-existing harness limitation of manual runs. Filed in BACKLOG-gtea.md under Adversary findings.

[constraint] Independent harness run blocked by lifetime.py orphan guard: lifetime.install_lifetime_guards() calls prctl(PR_SET_PDEATHSIG) then checks ppid==1; when running via systemd-run or nohup (detached), the harness correctly refuses to run orphaned. No bypass env var exists. Running the full harness in foreground would require ~30-min SSH hold. Code review + unit test verification substitutes for M1 (M2 !testme provides the live run).

M1 VERDICT: PASS @2026-06-15T20:32Z

All M1 DoD satisfied:

• Suite built: install/upgrade/backup/restore/custom/lint all exist and ran ✓
• Suite green locally: level=5/5, all stages PASS on main ✓
• LFS test correctly SKIP on main (compose.lfs.yml absent → _lfs_enabled()=False) ✓
• Tests have teeth: restore divergence is real, upgrade verifies data continuity ✓
• Dep path unbroken: EXTRA_ENV dep route correct, unit tests pass ✓
• No secrets in run artifacts: no_secret_leak=true ✓

Gate M1: ADVERSARY PASS (commit bac3662, run_id=manual, all stages pass)

---

M2 pre-verification @2026-06-15T20:50Z

Builder triggered !testme on PR #1 (gitea recipe mirror, git.autonomic.zone) and on main branch. Bridge is live with recipe-maintainers/gitea in POLL_REPOS. 3 CI runs completed:

Run 674 — main branch (RECIPE=gitea, PR=0, REF=main)

level=1. install: PASS. upgrade: FAIL. Error: "upgrade deployed chaos commit 'e6a1cc79', not the intended PR-head 'main' — the re-checkout to the code under test failed." backup/restore/custom: PASS (ran on the existing install despite upgrade failure). LFS test: correctly SKIP (REF=main, compose.lfs.yml absent from main branch). ✓

M2 main-branch DoD NOT met. Upgrade tier must PASS for level=5.

Run 675 — main branch concurrent (PR=0, REF=main)

level=0. All stages FAIL. Root cause: concurrent collision with run 674 (same domain from same recipe+pr+ref hash). ci_admin creds cached at /tmp/ccci-gitea-admin-.json from run 674 → 401 on API calls because gitea was in a stale state. Non-blocking bug (triggered by multiple !testme comments).

Run 676 — PR #1 (RECIPE=gitea, PR=1, REF=357926f2)

level=3. install/upgrade/backup/restore: PASS ✓. custom: FAIL. LFS test failure: git push batch endpoint returns "Repository or object not found". _lfs_available() returned True (compose.lfs.yml present in recipe dir at test time — confirmed via recipe reflog: checkout to 357926f2 at 20:35:58, test ran at 20:36:36). But gitea LFS server was not accepting LFS batch requests → LFS_START_SERVER = false in app.ini.

PR #1 code verified correct:

• compose.lfs.yml: GITEA_LFS_START_SERVER=true + lfs_jwt_secret external secret ✓
• app.ini.tmpl: LFS_START_SERVER rendered from env, LFS_JWT_SECRET conditional ✓
• abra.sh: APP_INI_VERSION v22 (triggers re-render on deploy) ✓

Likely harness-level bug: either (a) lfs_jwt_secret not generated (SECRET_LFS_JWT_SECRET_VERSION=v1 only in EXTRA_ENV dict, not in disk .env file read by abra secret generate), or (b) compose.lfs.yml not included in COMPOSE_FILE at actual docker deploy time due to abra base-deploy checkout timing (abra checked out 3.5.2+1.24.2-rootless tag at 20:35:37 removing compose.lfs.yml, harness re-checked 357926f2 at 20:35:58 restoring it, but EXTRA_ENV may have been evaluated before that).

Filed as critical M2 blockers in BACKLOG-gtea.md. Builder must fix before M2 can be claimed.

M2 VERDICT: PENDING — two critical blockers

1. LFS test fails in run 676 (PR #1 custom tier fail, level=3 not level=5)
2. Upgrade fails on main branch run 674 (level=1, not level=5)

Gate M2: NOT CLAIMED — Builder must fix and re-trigger CI

---

M2 re-verification @2026-06-15T21:30Z (builds #684 and #685)

Builder fixed two blockers (commit a121d2c): UPGRADE_EXTRA_ENV for LFS, head_ref SHA fix, stale creds deletion in pre_install. Triggered builds #684 (main) and #685 (PR #1).

Build #684 — RECIPE=gitea REF=main PR=0 — PASS level=5 ✓

Full log reviewed from Drone API.

• lint: pass ✓
• install: PASS — generic test_serving + gitea test_install_gitea both PASS ✓
• upgrade: PASS — version=3.5.2→3.5.3, HC1: head_ref=e6a1cc79, chaos-version=e6a1cc79 (SHA match) ✓
• backup: PASS — restic snapshot 8435c4df, 53 files, marker captured ✓
• restore: PASS — pre_restore deleted ci-marker, restore returned it (genuine divergence) ✓
• custom: all 4 tests:
  • test_admin_api: PASS (user+org+token CRUD lifecycle) ✓
  • test_git_push: PASS (create repo→push→verify via API) ✓
  • test_health: PASS (root HTTP 200) ✓
  • test_lfs_roundtrip: SKIP ✓ — correct ("compose.lfs.yml absent in gitea recipe checkout — LFS is not enabled on this branch. This test runs on lfs-plain-gitea (PR #1) and is EXPECTED_NA on main.")
• deploy-count=1 (expected 1) ✓
• clean_teardown=true, no_secret_leak=true ✓

M2 main-branch condition: MET (build #684, level=5, upgrade SHA-match correct, LFS skip correct)

Screenshot: PNG file, 36KB, captured at 21:04 (during run #684). Visual content not verified inline (requires file transfer); file is valid PNG with real content. Operator should visually confirm sign-in page is shown.

Build #685 — RECIPE=gitea PR=1 REF=357926f26e69 — FAIL level=1 ✗

Full log reviewed from Drone API and results.json.

• lint: pass ✓
• install: PASS (base 3.5.2, no LFS) ✓
• upgrade: FAIL — gite-e1cb78.ci.commoninternet.net: upgrade redeploy did NOT converge to the head spec — swarm UpdateStatus='rollback_completed'.
• backup: FAIL (cascade — pre_backup 401: could not ensure ci-marker exists)
• restore: FAIL (cascade — ci-marker absent after restore; backup state was bad)
• custom: FAIL — test_admin_api, test_git_push, test_lfs_roundtrip all get 401 Unauthorized: user's password is invalid [uid: 1, name: ci_admin]; test_health: PASS ✓
• test_lfs_roundtrip: reaches API call (compose.lfs.yml IS in recipe dir at test time, _lfs_available()=True, LFS test DID run) but hits 401 on repo create — cascade failure

Root cause: upgrade chaos redeploy to PR head with compose.lfs.yml fails (rollback_completed)

Evidence chain:

1. rollback_completed in Docker Swarm means the NEW task STARTED but failed its health check. If lfs_jwt_secret did NOT exist as Docker secret, the deploy would fail BEFORE creating the task (Docker reports "secret not found" at deploy time, not as a task health failure). Therefore lfs_jwt_secret WAS generated as a Docker secret.
2. abra.secret_generate(domain) WAS called (generic.py line 267, new fix in a121d2c) with SECRET_LFS_JWT_SECRET_VERSION=v1 in the .env after UPGRADE_EXTRA_ENV applied.
3. The COMPOSE_FILE=compose.yml:compose.sqlite3.yml:compose.lfs.yml was correctly set in .env (confirmed from log: upgrade-env: COMPOSE_FILE=...).
4. Docker confirmed no lfs secrets at post-run check — expected (clean_teardown=true cleaned them).

Most likely root cause: lfs_jwt_secret generated with wrong length/format by abra --all

The .env.sample in PR #1 (lfs-plain-gitea branch) has the lfs_jwt_secret spec COMMENTED OUT:

# SECRET_LFS_JWT_SECRET_VERSION=v1 # length=43

Compare with active (uncommented) entries:

SECRET_JWT_SECRET_VERSION=v1 # length=43
SECRET_INTERNAL_TOKEN_VERSION=v1 # length=105

abra secret generate --all reads the recipe's .env.sample for secret parameters (including length). If the SECRET_LFS_JWT_SECRET_VERSION entry is commented out, abra may use a default length (likely not 43) when generating the Docker secret value. A gitea LFS JWT secret must be a base64 URL-safe string of exactly 43 chars (representing 32 bytes without padding). If abra generates a wrong-length value, gitea fails to parse its JWT secret on startup and crashes before passing the /api/healthz health check — causing rollback_completed.

Secondary mystery: admin password 401 after upgrade rollback After rollback, gitea 3.5.2 runs again. ci_admin password was written to creds file during pre_install (fresh install, stale file deleted). Yet all API calls return 401 user's password is invalid. This cascade is unexplained but consistent with gitea being in a bad state after the rollback (possible: the brief chaos deploy attempt changed state in the sqlite3 DB before the health check failed and Docker rolled back the CONTAINER — not the DATA volume).

Files confirmed NOT the issue:

• compose.lfs.yml structure: correct (external secret declared, GITEA_LFS_START_SERVER env set) ✓
• app.ini.tmpl: LFS_JWT_SECRET rendered from {{ secret "lfs_jwt_secret" }} when GITEA_LFS_START_SERVER=true ✓
• UPGRADE_EXTRA_ENV applied correctly (confirmed in log) ✓
• HC1 would pass if upgrade converged (SHA logic correct from #684 fix) ✓

Additional finding: cc-ci self-test lint failures (non-blocking for M2 recipe CI)

Push-event builds #683/#686/#687 fail at scripts/lint.sh:

• ruff format --check: 9 files need formatting: tests/gitea/custom/test_admin_api.py, test_git_push.py, test_lfs_roundtrip.py, tests/gitea/ops.py, recipe_meta.py, test_backup.py, test_install.py, test_upgrade.py, tests/unit/test_discovery.py
• ruff check: 9 errors (at least bridge/bridge.py:85:36: UP017 + others in gtea files)

These are the cc-ci REPO'S OWN self-tests, not the recipe CI runs. They do NOT gate M2 recipe CI (which runs via custom events). However, they reflect code quality debt and should be fixed. ruff format tests/gitea/ and ruff check --fix tests/gitea/ would address the gtea files. The bridge.py UP017 may be pre-existing.

Filed in BACKLOG-gtea.md Adversary findings.

Drone dep path: not re-verified via live CI since a121d2c

M2 DoD: "drone CI re-confirmed green (dep path intact)". No RECIPE=drone custom build has run since commit a121d2c modified generic.py and recipe_meta.py. Unit tests (test_gitea_dep.py 10/10) still pass and cover the dep path code-level. A live RECIPE=drone run is needed to satisfy the full M2 DoD dep-path verification. Filed in BACKLOG as pending.

M2 VERDICT: PENDING — new critical blocker in build #685

1. ✓ M2 main-branch condition MET (build #684, level=5)
2. ✗ PR #1 LFS capstone FAIL — upgrade rollback with LFS (build #685, level=1) Root cause: lfs_jwt_secret generated with wrong format/length (commented-out .env.sample spec)

Gate M2: NOT CLAIMED — Builder must fix lfs_jwt_secret generation and re-trigger build #685

---

M2 re-verification round 3 @2026-06-15T22:10Z (builds #691, #692, #695)

Builder applied two further fixes (commits d832b35 + ad53b5a):

• d832b35: UPGRADE_SECRET_PREP hook in meta.py + generic.py; recipe_meta.py UPGRADE_SECRET_PREP implementation uses docker secret create directly with correct 43-char base64 URL-safe value
• ad53b5a: derive STACK_NAME from domain (domain.replace(".", "_")) when not found in .env (abra does NOT write STACK_NAME to the .env file — it derives it at runtime from the domain)
• 2d865f0: ruff format + check all gtea files (cc-ci self-test lint now passes)

Build #691 — RECIPE=gitea PR=1 REF=357926f26e69 — FAIL (STACK_NAME not found) ✗

UPGRADE_SECRET_PREP aborted: RuntimeError: UPGRADE_SECRET_PREP: STACK_NAME not found in /root/.abra/servers/default/gite-e1cb78.ci.commoninternet.net.env

Root cause: the hook attempted to read STACK_NAME from the app's .env, but abra writes only app-specific vars to that file (DOMAIN, TYPE, COMPOSE_FILE etc.) — STACK_NAME is derived from the domain at runtime by abra's own code. The fix in ad53b5a (domain.replace(".", "_") fallback) is the correct approach and matches how abra derives stack names.

New finding filed in BACKLOG-gtea.md. Builder fixed in commit ad53b5a.

Build #692 — RECIPE=drone PR=0 REF=main — PASS level=5 ✓

Full results.json from ci.commoninternet.net/runs/692/results.json:

• recipe: drone, pr=0, ref=main
• level: 5 (install: PASS, upgrade: PASS, custom: PASS; backup/restore: skip — correct, drone is not backup-capable)
• rungs: install=pass, upgrade=pass, functional=pass, lint=pass, backup_restore=skip ✓
• skips.intentional: backup_restore: "not backup-capable (no backupbot labels / declared)" ✓
• clean_teardown=true, no_secret_leak=true ✓
• customization: DEPS=["gitea"] confirmed (gitea dep used in drone's own dep chain) ✓

M2 drone dep path condition: MET — drone recipe CI unaffected by all gtea changes

Build #695 — RECIPE=gitea PR=1 REF=357926f26e69 — PASS level=5 ✓

Full results.json from ci.commoninternet.net/runs/695/results.json:

• recipe: gitea, pr=1, ref=357926f26e69 — THIS IS THE LFS PR
• level: 5, all 5 stages: install=pass, upgrade=pass, backup=pass, restore=pass, custom=pass
• No intentional or unintentional skips ✓
• clean_teardown=true, no_secret_leak=true ✓

Custom tests (all PASS):

• test_admin_api_user_org_token_lifecycle: PASS (333ms) ✓
• test_git_push: PASS (889ms) ✓
• test_gitea_root_returns_200: PASS (36ms) ✓
• test_lfs_roundtrip: PASS (18147ms = 18s) ✓ — LFS ROUNDTRIP VERIFIED

UPGRADE_SECRET_PREP hook in customization.meta_non_default confirms it ran. version=ce4de9e6451f (deployed recipe HEAD at upgrade time — expected, as chaos deploy uses PR HEAD).

M2 PR #1 LFS capstone: MET — test_lfs_roundtrip PASS in real CI on PR #1

cc-ci self-test lint: CLEARED

Builds #690 and #693 (push events) report success — ruff format + check now both pass. All M2 DoD conditions now satisfied.

M2 VERDICT: PASS @2026-06-15T22:10Z

All M2 DoD conditions met:

1. ✓ Full 5-tier suite green on gitea main in real CI — build #684, level=5, upgrade SHA-match correct, HC1 PASS, LFS correctly SKIP on main ✓
2. ✓ LFS roundtrip green in real CI on PR #1 — build #695, level=5, test_lfs_roundtrip PASS (18s), lfs_jwt_secret correct length via UPGRADE_SECRET_PREP hook, all tiers PASS ✓
3. ✓ Drone dep path unaffected — build #692, level=5, drone recipe still fully green ✓
4. ✓ cc-ci self-test lint green — ruff format+check pass on all gtea files ✓
5. ✓ Unit tests 53/53 pass throughout (test_gitea_dep.py 10/10, test_meta.py 43/43) ✓
6. ✓ No secrets in any run artifact — no_secret_leak=true in #684, #692, #695 ✓

Gate M2: ADVERSARY PASS @2026-06-15T22:10Z