Files

autonomic-bot 294a8a1a9e rename the opt-in heavy-tests flag: --extra-tests -> --extra (operator 2026-05-29)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-05-29 10:36:04 +01:00

13 KiB

Raw Blame History

SSO-dep testing pattern (OIDC + co-deployed provider) — reference plan

Status: active reference for Phase 2 (does not need its own phase; it's a pattern Phase 2 overlays apply per recipe). Operator clarification (2026-05-28): integrating a recipe with an OIDC/SSO dep is loop work, not operator work. Anything that was deferred citing "operator input needed for OIDC" should be re-opened and done autonomously per this plan. This file's path: /srv/cc-ci/cc-ci-plan/plan-sso-dep-testing.md Companion: the running harness in runner/harness/sso.py (existing primitives: setup_keycloak_realm, oidc_password_grant, assert_discovery_endpoint).

0. Why this plan

Several recipes test their authenticated functionality through an OIDC/SSO provider (keycloak, authentik). The cc-ci pattern is to co-deploy the provider with the recipe under test in the same ephemeral run — one shared deployment per dep, configured at install time, used by the recipe-under-test's authenticated tests, torn down with it. This file is the canonical pattern for how to wire that up so any recipe that declares DEPS = ["keycloak"] (or ["authentik"]) Just Works without per-recipe ad-hoc plumbing. Recipes that need OIDC are not blocked on the operator — they follow this plan.

1. The DEPS model — deps deploy AFTER generic tiers (operator-2026-05-28)

Critical ordering rule: generic tiers (install / upgrade / backup / restore) run against the recipe alone, with no dep available, so a failure in dep-deploy or OIDC setup cannot break generic-tier signal. Deps + OIDC wiring move to a setup_custom_tests step that runs after generic tiers and before the custom tier — its failure is isolated to the SSO-marked custom tests.

A recipe's tests/<recipe>/recipe_meta.py declares its SSO dep:

DEPS = ["keycloak"]   # or ["authentik"] when that backend lands

Lifecycle order (single run, per recipe)

1. Deploy recipe-under-test ALONE (no deps, OIDC env unset or stubbed).
   - app_new #1 for the recipe; generic install_steps.sh runs RECIPE-ONLY setup (no deps).
2. INSTALL tier:   generic [+ overlay] assertions against the recipe alone.
3. UPGRADE tier:   abra app upgrade in place, assertions against the recipe alone.
4. BACKUP tier:    in place (if backup-capable), recipe-alone marker.
5. RESTORE tier:   in place, recipe-alone marker.
6. setup_custom_tests step  ← NEW (operator-2026-05-28)
   a. For each dep in DEPS, deploy + provision realm/client via harness.sso.setup_<provider>_realm.
   b. Write $CCCI_DEPS_FILE with each dep's {domain, realm, client_id, client_secret, admin_*}.
   c. Run the per-recipe post-deps hook `tests/<recipe>/setup_custom_tests.sh` to wire the OIDC
      env into the running recipe (abra app config set + abra app secret insert) and trigger an
      in-place redeploy of the affected services so the env takes effect.
   d. Mark deps-ready = True on success; on ANY failure mark deps-ready = False and CONTINUE
      (log the error; do NOT abort the run).
7. CUSTOM tier:
   - If deps-ready: run all custom tests, including those tagged @pytest.mark.requires_deps.
   - If NOT deps-ready: still run custom tests, but tests tagged @pytest.mark.requires_deps are
     reported as ERROR/SKIP (with the captured setup_custom_tests error attached). Non-deps
     custom tests still run normally.
8. Teardown (in finally): recipe first; then each dep in reverse declaration order.

DG4.1 deploy-count guard, generalised

The "one deploy per run" guard becomes one abra app new per app in the run (recipe + each dep). In-place reconfigure-and-redeploy (the step 6c env update) is NOT a fresh app_new and does NOT increment the per-recipe count. So a run with DEPS = ["keycloak"] has exactly 2 app_new calls (recipe + keycloak), no matter how many tiers ran. The per-run summary reports deploy-count per app for verification.

Why this ordering

Generic-tier signal is preserved when SSO/dep setup is broken — the recipe's own deploy/ upgrade/backup/restore behaviour is still tested honestly.
Failure isolation: a recipe whose generic tier passes but whose SSO setup is broken yields per-op pass/pass/pass/pass/skip(deps-not-ready) — far more useful than the previous all-or-nothing.
A recipe that genuinely can't boot without OIDC fails its generic install honestly (the recipe should accept a stubbed/empty OIDC env at install time and only require the env when an authenticated endpoint is hit). That's a real recipe finding, not a CI artifact.

2. Provider pluggability

Provider-agnostic primitives (today, in harness/sso.py) — these stay pluggable:
- oidc_password_grant(discovery_url, client_id, client_secret, username, password) -> token — pure OIDC; works against any compliant provider.
- assert_discovery_endpoint(discovery_url, expected_issuer) — pure OIDC.
Provider-specific setup (admin API calls) — one function per provider:
- setup_keycloak_realm(domain, admin_user, admin_password, realm, client_id, redirect_uris) -> {client_secret, discovery_url} — exists today.
- setup_authentik_realm(...) — same shape, authentik admin API; deferred to a future Q4 enrollment that actually wants authentik (see machine-docs/DEFERRED.md). Pluggable: a recipe declaring DEPS = ["authentik"] would just call this instead. No change to the per-recipe install_steps.sh shape beyond which provider it asks for from $CCCI_DEPS_FILE.
Don't write per-recipe SSO logic. All recipes use the same DEPS+install_steps shape.

3. Per-recipe hooks — two distinct scripts (recipe-only vs post-deps)

A recipe with DEPS = ["keycloak"] ships two optional hook scripts (either may be absent if not needed):

3.1 `tests/<recipe>/install_steps.sh` — RECIPE-ONLY setup, runs at install time

This is the Phase-1d custom-install-steps hook. It runs before the recipe deploys, with no dep available (the dep hasn't been deployed yet at this point). Use it only for recipe-only setup that the recipe needs to boot at all (e.g. seed a fixture, set a non-OIDC env). Do NOT read $CCCI_DEPS_FILE here — it doesn't exist yet. If the recipe requires OIDC to boot at all, set a safe stub here (e.g. disable auth) so the recipe can come up for generic tiers; the real OIDC wiring happens in §3.2.

3.2 `tests/<recipe>/setup_custom_tests.sh` — POST-DEPS wiring, runs after generic tiers

This is the new (operator-2026-05-28) hook that wires the recipe to its already-deployed dep, after the generic tiers have run. The orchestrator has already deployed each dep and written $CCCI_DEPS_FILE by the time this runs. Roughly:

#!/usr/bin/env bash
set -euo pipefail
# Read the dep's connection info from $CCCI_DEPS_FILE (orchestrator-written).
KC_DOMAIN=$(jq -r '.keycloak.domain'        "$CCCI_DEPS_FILE")
KC_CLIENT=$(jq -r '.keycloak.client_id'     "$CCCI_DEPS_FILE")
KC_SECRET=$(jq -r '.keycloak.client_secret' "$CCCI_DEPS_FILE")
KC_REALM=$( jq -r '.keycloak.realm'         "$CCCI_DEPS_FILE")

# Inject the OIDC client secret as an abra app secret (recipe-conventional name varies — match
# the recipe's .env.sample SECRET_*).
echo "$KC_SECRET" | abra app secret insert -n "$CCCI_APP_DOMAIN" oidc_rpcs v1 -

# Write the OIDC env vars to the parent .env (names per the recipe's .env.sample).
abra app config set "$CCCI_APP_DOMAIN" \
  OIDC_REALM="$KC_REALM" \
  OIDC_OP_DISCOVERY_ENDPOINT="https://${KC_DOMAIN}/realms/${KC_REALM}/.well-known/openid-configuration" \
  OIDC_OP_AUTHORIZATION_URL="https://${KC_DOMAIN}/realms/${KC_REALM}/protocol/openid-connect/auth" \
  OIDC_OP_TOKEN_URL="https://${KC_DOMAIN}/realms/${KC_REALM}/protocol/openid-connect/token" \
  OIDC_OP_USER_URL="https://${KC_DOMAIN}/realms/${KC_REALM}/protocol/openid-connect/userinfo" \
  OIDC_RP_CLIENT_ID="$KC_CLIENT" \
  OIDC_RP_REDIRECT_URI="https://${CCCI_APP_DOMAIN}/auth/oidc/callback"

# Force an in-place redeploy of the affected services to pick up the new env. This is NOT a fresh
# app_new (deploy-count guard still 1 for this recipe).
abra app deploy --force --chaos --no-input "$CCCI_APP_DOMAIN"

The OIDC env-var names are recipe-specific (OIDC_OP_* for lasuite-docs, different prefixes elsewhere). Read the recipe's .env.sample to see which keys the recipe expects; the values follow this template. If a recipe needs more than this (extra group/claim mappings, etc.), extend its setup_custom_tests.sh only — never the shared harness.

4. Test pattern: authenticated endpoints (mark + isolate)

Mark dep-requiring tests: every custom test that needs the dep up + OIDC wired must use @pytest.mark.requires_deps. The orchestrator skips these with reason "deps-not-ready: <err>" if setup_custom_tests failed. Non-deps custom tests are unaffected by SSO setup failures.
Headless API tests — use harness.sso.oidc_password_grant to mint an access token, then call the recipe's authenticated endpoint with Authorization: Bearer <token>. Asserts on the response.
Browser flows (Playwright) — navigate to the recipe, follow the redirect to keycloak, fill the pre-provisioned test user's credentials, return to the recipe, exercise the UI. (Use the pre-provisioned ci-user@example.com / known password the realm setup creates.)
The realm/client is fresh per run — no cross-run state, no shared accounts. The realm setup creates one or more test users with known passwords (pass-through from a per-run secret) so the tests can authenticate without prompts.

5. Concrete recipes that use this pattern (Phase-2 scope)

These are loop work under this plan, not deferred:

lasuite-docs — DEPS = ["keycloak"]; ports the upstream oidc_login.py + upload_conversion.py parity tests + the §4.3-prescribed create-a-doc + read-back via authenticated /api/v1.0/documents/. (Re-enters DEFERRED.md entry #5 — this plan IS the re-entry, not operator input.)
cryptpad — DEPS = ["keycloak"] (cryptpad upstream tests use authentik, but a keycloak-backed cryptpad OIDC test is equally valid and uses the same primitives). The cryptpad create-a-pad Playwright test (DEFERRED #6) is a separate concern — that one really does need a stable CryptPad app-launch contract; it stays deferred.
lasuite-drive, lasuite-meet — same pattern when mirrored (recipe-create-pr skill — loop work).
Any future recipe that requires OIDC follows this plan; no operator handoff.

6. What stays deferred (genuinely operator-input)

authentik enrollment + setup_authentik_realm backend (DEFERRED #9) — RESOLVED (operator, 2026-05-29): keycloak is our default SSO provider; default ALL recipe OIDC tests to keycloak. Do NOT test authentik↔keycloak integration, and do NOT enroll authentik just to "prove pluggability" — Phase-2 DONE is NOT gated on authentik. Enroll authentik + add setup_authentik_realm ONLY if/when a recipe genuinely requires authentik (won't work under keycloak). If a recipe works with keycloak, use keycloak. So DEFERRED #9's re-entry trigger narrows to "a recipe requires authentik" — the cross-provider-coverage trigger is dropped. (E.g. cryptpad: its upstream test uses authentik, but test it under keycloak — equally valid.)
The --extra flag IDEA is not a precondition for this plan; OIDC-dep tests are part of the default suite for the recipes that need them.

7. Definition of done for this pattern

DEPS = [...] honored by runner/run_recipe_ci.py, with the deps-AFTER-generic ordering (§1): deps deploy + setup_custom_tests step runs between RESTORE and CUSTOM tiers; $CCCI_DEPS_FILE written; deps torn down LAST in reverse order.
Failure isolation proven: a forced setup_custom_tests failure (e.g. simulate keycloak realm-setup error) yields a run where generic tiers report pass and CUSTOM requires_deps tests report skip(deps-not-ready) — no false fail of the generic tier, no aborted run.
lasuite-docs ships tests/lasuite-docs/setup_custom_tests.sh per §3.2 + authenticated tests per §4 marked @pytest.mark.requires_deps (closes DEFERRED #5 — keep the entry there with the closing commit, do not re-defer).
At least one other OIDC-dep recipe (cryptpad oidc_login or a lasuite-* once mirrored) lands cold-green using the same pattern, demonstrating reuse.
docs/sso-dep-testing.md (in the cc-ci repo) explains the pattern for future recipe enrollments — link to this plan.
Adversary cold-verifies the full run for one such recipe + the forced-failure isolation case, posts PASS in REVIEW-2.md.

8. Mirror+enroll reminder (also loop work)

If a recipe in scope (e.g. lasuite-drive, lasuite-meet, immich) isn't mirrored to git.autonomic.zone/recipe-maintainers/, mirror it autonomously via the recipe-create-pr skill at /srv/recipe-maintainer/.opencode/skills/recipe-create-pr/SKILL.md (see also plan-phase2-recipe-tests.md §0b). Mirror+enroll is not operator-pending; the bot is admin on the org.

13 KiB Raw Blame History