cc-ci-orchestrator/cc-ci-plan/plan-lasuite-drive-oidc-robustness.md

# Plan — lasuite-drive OIDC-setup flakiness: harness restructure + recipe robustness PR

**Status:** QUEUED — picks up the deferred lasuite-drive `[~]` item (Q3.2) when the loops return to it.
The disk blocker is now lifted (host grew to 70 GB, 2026-05-29), so the upgrade tier can run too.
**Owner:** Builder + Adversary loops. **Two deliverables:** (A) a cc-ci harness change, (B) a
**lasuite-drive recipe PR** (we maintain the recipe via recipe-maintainer).
**This file:** `/srv/cc-ci/cc-ci-plan/plan-lasuite-drive-oidc-robustness.md`

---

## 0. The problem (ground it in real logs FIRST)

lasuite-drive's `setup_custom_tests` wires OIDC by doing a **full 12-service `abra app deploy --chaos`
redeploy** of an already-running heavy stack. That reconverge is **flaky**: a collabora reconverge
race + a **transient backend gunicorn-perms / WOPI-404 window**. Only **backend/app** consume the
OIDC env, so re-converging collabora/onlyoffice/minio/db is unnecessary exposure. The Builder
correctly did NOT claim Q3.2 (a flaky setup isn't a reliable green) and filed this.

**Step 0 (do before fixing):** capture the actual failure from a flaky run — collabora WOPI-discovery
timing, the backend log at the 404, and the exact gunicorn perms error — so we fix the real root
cause, not a guess. Record in JOURNAL.

---

## Part A — cc-ci harness: wire OIDC at INSTALL, eliminate the redeploy

Leverage the now **live-warm keycloak** (WC1): instead of deploy-recipe-then-redeploy-with-OIDC, the
dep is already running, so configure OIDC **before the single deploy**:

1. Create the per-run namespaced realm+client in the **warm keycloak** (lightweight API calls — no
   deploy).
2. Set lasuite-drive's OIDC env in its `.env` **before** the first `abra app deploy`.
3. Deploy lasuite-drive **once**, OIDC already wired. **No mid-run `--chaos` reconverge** → the flaky
   window is gone.

- **Use REAL abra commands only** (operator decision, 2026-05-29): the install is a normal
  `abra app deploy`. Do **NOT** use `docker service update`/`docker service scale` to surgically
  patch services — CI must exercise the same abra path a real operator uses. If any redeploy ever
  remains necessary, it is `abra app deploy` (+ proper readiness gating, §C), never a docker-level
  bypass.
- **Generic-first invariant still holds.** keycloak is live-warm (persistent infra, ~always up), so
  there's no per-run dep *deploy* that could fail and break a generic tier. Generic tiers still
  assert serving recipe-alone; if realm creation or the provider is unreachable, only the OIDC
  *custom* test fails (isolated). This is SAFE **iff** the recipe boots with OIDC-env-set even when
  the provider isn't reachable at boot — which Part B guarantees (lazy OIDC). A and B reinforce each
  other.
- **Revises `plan-sso-dep-testing.md`** for recipes whose app tolerates OIDC-at-boot: prefer
  install-time wiring against the warm dep over the post-deploy `setup_custom_tests` redeploy. Keep
  the redeploy path only for recipes that genuinely can't take OIDC env at first boot.
- **Verify:** lasuite-drive full suite green **without** a mid-run reconverge; run it repeatedly
  (e.g. 3×) to show the flakiness is gone, not just absent once.

## Part B — lasuite-drive recipe PR (root-cause robustness, we're maintainers)

> **Broken out into its own sub-plan: `plan-lasuite-drive-recipe-pr.md`** (operator, 2026-05-29) —
> it now also folds in the F2-12 upgrade-convergence fix (collabora healthcheck) so cc-ci can return
> to abra-native convergence. The summary below is retained; the sub-plan is authoritative.

Fix the fragility in the recipe itself so it's robust under ANY reconverge — this helps real
operators, not just CI, and is the real payoff of maintaining the recipe. On a branch of the
lasuite-drive recipe (recipe-maintainer):

1. **Order backend behind collabora's WOPI readiness.** Add a Docker **healthcheck on collabora**
   (WOPI discovery endpoint → 200) and make backend **wait for it / retry WOPI discovery with
   backoff** instead of failing during collabora startup. Turns the race into deterministic ordering.
2. **Fix the gunicorn-perms race.** The transient perms error is a volume-permission race at startup
   (gunicorn can't read/write a mount until perms are set). Fix in the entrypoint: set perms before
   exec'ing gunicorn (or a proper init step).
3. **Make OIDC discovery lazy / retrying.** If backend validates the OIDC provider eagerly at boot
   and dies when it's briefly unreachable, that's what forces the deploy-with-dep dance. Resolve OIDC
   discovery lazily (at first login, with retry) so the app boots without the provider up. (This is
   also the property Part A relies on for the generic-first invariant.)

### The PR is "working" ONLY when cc-ci verifies it green (operator rule, 2026-05-29)
A lasuite-drive recipe change is **not** considered done until **our CI server has run the full suite
on the PR and it passes** — install + upgrade + backup + restore + custom (incl. the OIDC login
test), repeatedly-green (no flakiness). Concretely: push the recipe branch/PR, trigger cc-ci
(`!testme` on the lasuite-drive PR), and require a clean green (Adversary cold-verified). **Only then
does the operator merge** the recipe PR. This dogfoods cc-ci: the CI that surfaced the bug also gates
its fix.

## C. If any redeploy must remain (fallback only)
If a recipe genuinely needs OIDC env applied after first boot, do it with **`abra app deploy`** (real
abra — never `docker service update`), and **gate readiness properly**: poll collabora WOPI = 200 +
backend health with retries/timeout before declaring ready. This is strictly a fallback; Part A +
Part B should remove the need.

## Definition of done
- [ ] Step 0 root-cause logs captured + recorded.
- [ ] Part A: lasuite-drive tested via install-time OIDC wiring against the warm keycloak, **no
      mid-run reconverge**, full suite green **3× in a row** (flakiness gone), Adversary cold-verified.
- [ ] Part B: lasuite-drive recipe branch with the WOPI-healthcheck-gating + gunicorn-perms + lazy-OIDC
      fixes; **cc-ci runs the full suite (incl. upgrade tier) on the PR and it is repeatedly green**;
      Adversary cold-verifies; operator merges.
- [ ] Q3.2 (lasuite-drive) claimed + DEFERRED.md entry closed (flaky-OIDC item) — only after the above.
- [ ] `plan-sso-dep-testing.md` updated to prefer install-time OIDC wiring (warm dep) where the recipe
      supports it.