diff --git a/cc-ci-plan/plan-lasuite-drive-oidc-robustness.md b/cc-ci-plan/plan-lasuite-drive-oidc-robustness.md new file mode 100644 index 0000000..686ad6e --- /dev/null +++ b/cc-ci-plan/plan-lasuite-drive-oidc-robustness.md @@ -0,0 +1,93 @@ +# Plan — lasuite-drive OIDC-setup flakiness: harness restructure + recipe robustness PR + +**Status:** QUEUED — picks up the deferred lasuite-drive `[~]` item (Q3.2) when the loops return to it. +The disk blocker is now lifted (host grew to 70 GB, 2026-05-29), so the upgrade tier can run too. +**Owner:** Builder + Adversary loops. **Two deliverables:** (A) a cc-ci harness change, (B) a +**lasuite-drive recipe PR** (we maintain the recipe via recipe-maintainer). +**This file:** `/srv/cc-ci/cc-ci-plan/plan-lasuite-drive-oidc-robustness.md` + +--- + +## 0. The problem (ground it in real logs FIRST) + +lasuite-drive's `setup_custom_tests` wires OIDC by doing a **full 12-service `abra app deploy --chaos` +redeploy** of an already-running heavy stack. That reconverge is **flaky**: a collabora reconverge +race + a **transient backend gunicorn-perms / WOPI-404 window**. Only **backend/app** consume the +OIDC env, so re-converging collabora/onlyoffice/minio/db is unnecessary exposure. The Builder +correctly did NOT claim Q3.2 (a flaky setup isn't a reliable green) and filed this. + +**Step 0 (do before fixing):** capture the actual failure from a flaky run — collabora WOPI-discovery +timing, the backend log at the 404, and the exact gunicorn perms error — so we fix the real root +cause, not a guess. Record in JOURNAL. + +--- + +## Part A — cc-ci harness: wire OIDC at INSTALL, eliminate the redeploy + +Leverage the now **live-warm keycloak** (WC1): instead of deploy-recipe-then-redeploy-with-OIDC, the +dep is already running, so configure OIDC **before the single deploy**: + +1. Create the per-run namespaced realm+client in the **warm keycloak** (lightweight API calls — no + deploy). +2. Set lasuite-drive's OIDC env in its `.env` **before** the first `abra app deploy`. +3. Deploy lasuite-drive **once**, OIDC already wired. **No mid-run `--chaos` reconverge** → the flaky + window is gone. + +- **Use REAL abra commands only** (operator decision, 2026-05-29): the install is a normal + `abra app deploy`. Do **NOT** use `docker service update`/`docker service scale` to surgically + patch services — CI must exercise the same abra path a real operator uses. If any redeploy ever + remains necessary, it is `abra app deploy` (+ proper readiness gating, §C), never a docker-level + bypass. +- **Generic-first invariant still holds.** keycloak is live-warm (persistent infra, ~always up), so + there's no per-run dep *deploy* that could fail and break a generic tier. Generic tiers still + assert serving recipe-alone; if realm creation or the provider is unreachable, only the OIDC + *custom* test fails (isolated). This is SAFE **iff** the recipe boots with OIDC-env-set even when + the provider isn't reachable at boot — which Part B guarantees (lazy OIDC). A and B reinforce each + other. +- **Revises `plan-sso-dep-testing.md`** for recipes whose app tolerates OIDC-at-boot: prefer + install-time wiring against the warm dep over the post-deploy `setup_custom_tests` redeploy. Keep + the redeploy path only for recipes that genuinely can't take OIDC env at first boot. +- **Verify:** lasuite-drive full suite green **without** a mid-run reconverge; run it repeatedly + (e.g. 3×) to show the flakiness is gone, not just absent once. + +## Part B — lasuite-drive recipe PR (root-cause robustness, we're maintainers) + +Fix the fragility in the recipe itself so it's robust under ANY reconverge — this helps real +operators, not just CI, and is the real payoff of maintaining the recipe. On a branch of the +lasuite-drive recipe (recipe-maintainer): + +1. **Order backend behind collabora's WOPI readiness.** Add a Docker **healthcheck on collabora** + (WOPI discovery endpoint → 200) and make backend **wait for it / retry WOPI discovery with + backoff** instead of failing during collabora startup. Turns the race into deterministic ordering. +2. **Fix the gunicorn-perms race.** The transient perms error is a volume-permission race at startup + (gunicorn can't read/write a mount until perms are set). Fix in the entrypoint: set perms before + exec'ing gunicorn (or a proper init step). +3. **Make OIDC discovery lazy / retrying.** If backend validates the OIDC provider eagerly at boot + and dies when it's briefly unreachable, that's what forces the deploy-with-dep dance. Resolve OIDC + discovery lazily (at first login, with retry) so the app boots without the provider up. (This is + also the property Part A relies on for the generic-first invariant.) + +### The PR is "working" ONLY when cc-ci verifies it green (operator rule, 2026-05-29) +A lasuite-drive recipe change is **not** considered done until **our CI server has run the full suite +on the PR and it passes** — install + upgrade + backup + restore + custom (incl. the OIDC login +test), repeatedly-green (no flakiness). Concretely: push the recipe branch/PR, trigger cc-ci +(`!testme` on the lasuite-drive PR), and require a clean green (Adversary cold-verified). **Only then +does the operator merge** the recipe PR. This dogfoods cc-ci: the CI that surfaced the bug also gates +its fix. + +## C. If any redeploy must remain (fallback only) +If a recipe genuinely needs OIDC env applied after first boot, do it with **`abra app deploy`** (real +abra — never `docker service update`), and **gate readiness properly**: poll collabora WOPI = 200 + +backend health with retries/timeout before declaring ready. This is strictly a fallback; Part A + +Part B should remove the need. + +## Definition of done +- [ ] Step 0 root-cause logs captured + recorded. +- [ ] Part A: lasuite-drive tested via install-time OIDC wiring against the warm keycloak, **no + mid-run reconverge**, full suite green **3× in a row** (flakiness gone), Adversary cold-verified. +- [ ] Part B: lasuite-drive recipe branch with the WOPI-healthcheck-gating + gunicorn-perms + lazy-OIDC + fixes; **cc-ci runs the full suite (incl. upgrade tier) on the PR and it is repeatedly green**; + Adversary cold-verifies; operator merges. +- [ ] Q3.2 (lasuite-drive) claimed + DEFERRED.md entry closed (flaky-OIDC item) — only after the above. +- [ ] `plan-sso-dep-testing.md` updated to prefer install-time OIDC wiring (warm dep) where the recipe + supports it.