cc-ci-orchestrator/cc-ci-plan/IDEAS.md

Deferred ideas / future enhancements (orchestrator-tracked)

Post-DONE or "revisit later" ideas that are intentionally out of scope for the current build (§2 Definition of Done). Not active work — parked here so they aren't lost. The loops may pull an item into the project BACKLOG.md as [idea] if/when it becomes relevant.

• ALT infra-app model: deploy traefik / warm-keycloak / drone the normal Co-op Cloud way, updated outside Nix via abra by maintainers. (operator-flagged 2026-05-29, alternative to the current Phase-2w design — return to later) The current design Nix-reconciles the infra/warm apps: NixOS systemd oneshots run abra to deploy traefik/keycloak/drone, and a nightly nixos-rebuild auto-updates them to latest with a pre-deploy major/manual-migration gate (WC1.2) + post-deploy health-gated rollback (WC1.1). The alternative: treat the CI server like a normal Co-op Cloud host — traefik, the warm keycloak, and drone are plain abra deployments managed by maintainers, deployed once and upgraded by a human via abra app upgrade (using abra's own release-notes / major-bump caution), with no Nix reconciler and no nightly auto-update for them. Nix would provide only the host substrate (OS, docker/swarm, the harness, secrets), not the infra-app lifecycle.

  • Pros: simpler — no reconciler/rollback/nightly machinery; idiomatic Co-op Cloud ops (maintainers manage these exactly like any other coop-cloud app); updates are normal human abra actions; lower cognitive load (no "infra is special / Nix-driven" layer).
  • Cons: the infra apps are no longer reproducible-from-git — a VM recreate (the D8 throwaway rebuild) would NOT re-establish traefik/keycloak/drone; they'd need manual redeploy after a rebuild (D8 then covers only the host + harness, not the infra apps). Loses the automated nightly-latest + health-gated auto-rollback; infra updates + rollback become manual/operator discipline. Drifts from the project's "declarative, rebuildable from scratch" ethos.
  • Note: it's essentially one-or-the-other for the update path — a hybrid where Nix bootstraps them but maintainers also abra app upgrade them creates the dual-ownership conflict (the Nix reconciler would fight/redeploy over the maintainer's version on the next activation).
  • When to revisit: if the reconciler + rollback + nightly machinery proves high-maintenance or brittle, or if maintainers strongly prefer normal coop-cloud workflow over the Nix layer — weigh that against how much we value full reproducibility (D8) + hands-off auto-updates. Added: 2026-05-29.

• Docker Hub pull-through registry cache (registry:2 proxy) — deferred; single-host makes it marginal. Considered as a Phase-2pc perf win, then dropped (operator, 2026-05-29): on a single host, Docker's own local image store already caches pulled images (re-deploys reuse local layers), so the prune-policy fix (Phase 2pc PC1) recovers ~all the benefit. A separate pull-through cache's distinctive wins don't apply here — multi-node fan-out (one node), surviving prune/VM-rebuild on separate storage (ours would be co-located, lost on recreate), cache-miss auth (daemon already PAT-authenticated). Revisit ONLY if: (a) cc-ci goes multi-node, OR (b) Phase-2b measurement shows cold-cache / fresh-deploy pull time (D8 throwaway-rebuild, fresh-canonical seeding) is a real bottleneck AND the cache lives on recreate-surviving storage (Incus volume / host-b1 path, not the VM's ephemeral disk). Otherwise it's complexity without payoff. Added: 2026-05-29.

• Optional --extra flag for heavy / operational tests (opt-in heavy suite). Some recipe tests are "more than needed" for the default CI signal — state-management / long-running-instance / load / helper-script operational tests that don't fit the ephemeral per-run-deploy model cheaply but are useful occasionally. Today they're deferred to cc-ci/machine-docs/DEFERRED.md (e.g. matrix-synapse compress_state.sh, test_complexity_limit.sh, test_purge.sh) and don't run. Idea: add an opt-in --extra flag (e.g. !testme --extra on a PR comment, or a STAGES=extra / EXTRA_TESTS=1 Drone build parameter) that the orchestrator passes through; recipes declare an extra/ test dir or mark tests with @pytest.mark.extra; on opt-in the orchestrator runs them alongside the default tiers (still one deploy, still teardown). Default off so default CI stays fast; the operator can ask for the heavy suite when reviewing a PR that touches an extra-covered area (e.g. matrix-synapse's abra helpers). When implemented, each matching DEFERRED entry can be CLOSED by porting its test into the recipe's extra/ and noting the commit in DEFERRED.md. Why deferred for now: default coverage is sufficient; this is a later breadth/depth knob, not a critical-path feature. Added: 2026-05-28.

• Optional webhook self-registration (admin-access environments). We deliberately made polling the primary trigger and require the CI server/bot to run on read-level access only — so the server does not auto-register Gitea webhooks (that needs repo-admin), and webhook setup is a documented manual admin task (§4.1, docs/enroll-recipe.md). Later, for environments where the CI server does hold admin on the recipe repos (or an org-level admin token is available), consider adding an opt-in, off-by-default feature (e.g. WEBHOOK_AUTOREGISTER=1) that auto-registers and idempotently reconciles the issue_comment webhook (URL, events, HMAC secret) on enrolled repos — matching our declarative-reconcile pattern (§9) — giving low-latency push triggering with zero manual setup. Must stay off by default and fall back to manual-doc + polling when admin isn't available, so the least-privilege (read-only) default is preserved. Why deferred: polling already satisfies D1 and the read-only posture is the goal; this is a convenience optimization for a different deployment profile. Added: 2026-05-27.

• Docker Hub registry:2 pull-through cache (deferred from Phase 2pc). A local registry in proxy/pull-through mode, daemon registry-mirrors-wired, so all docker.io pulls are cached locally across recipes/runs/prunes. Deferred (operator, 2026-05-29): on the current single, PAT-authenticated, non-pruning host, Docker's own local image store already IS the cache (redeploys reuse local layers — proven in Phase 2pc), so a separate registry adds a service + mirror config + cache GC for marginal gain; its distinctive wins (multi-node fan-out, surviving prune/VM-rebuild on separate storage, cache-miss auth) don't apply here. Revisit ONLY if (a) cc-ci goes multi-node, OR (b) Phase-2b measurement shows cold-deploy pull time is a real bottleneck (e.g. D8 throwaway-rebuild / fresh-canonical seeding) AND the cache lives on recreate-surviving storage (an Incus volume / a path on host b1, not the VM's ephemeral disk). Otherwise it's complexity without payoff. See DECISIONS.md "Phase 2pc". Added: 2026-05-29.

• Phase-2b empirical performance work (moved out of the 2b phase). The original Phase 2b was a full empirical perf program: per-phase timing instrumentation in results.json, a cold/warm baseline across representative recipes, a Pareto attribution, and a menu of software optimizations. Deferred (operator, 2026-05-30): the real deploy-speed bottleneck turned out to be hardware, not software — the cc-ci VM was 2 vCPU on a 4-core host and disk-I/O-bound (load ~8, io pressure ~65%) while running warm-keycloak (JVM) + all infra; RAM was never the constraint. Fixed directly: bumped to 4 vCPU and made cc-nix-test the only running VM on b1. The software micro-opts below are judged unlikely to move the needle enough to justify the work; revisit ONLY if measurement later shows a specific software bottleneck. (Phase 2b is narrowed to just confirming the test sequence already minimizes deploys — see plan-phase2b.) Parked ideas:

  • Per-phase timing instrumentation + cold/warm baseline + attribution — do this first if perf is ever revisited; numbers should drive any change.
  • Image pulls: local registry pull-through cache (see the item above) and/or pre-pull/warm the enrolled recipes' image set so the first run doesn't pay the cold pull.
  • Readiness/convergence: replace fixed sleeps with tight health-endpoint polling; per-recipe readiness probes; parallelize independent readiness checks within a run.
  • Warm shared SSO provider (already partly live as warm-keycloak): saves per-run SSO deploy time but is a steady JVM CPU tax that slows non-SSO recipes — only worth it with proven per-run isolation; consider start-when-needed / stop-when-idle rather than always-on.
  • Runner/build caching: persistent nix store + warm flake eval; cache pip/uv wheels + Playwright browsers in a persistent volume.
  • Concurrency sizing: tune MAX_TESTS/runner capacity + per-recipe weights so light recipes run concurrently while heavy ones serialize, without overcommitting the node.
  • Resources: further vCPU/RAM/disk-I/O sizing (the 4-vCPU bump is done; storage I/O on b1 is the harder co-bottleneck — a faster storage pool if it ever matters).
  • abra/secret overhead: avoid regenerating/re-inserting secrets redundantly across stages. Why deferred: hardware was the real lever and is fixed; these are speculative software gains best validated by measurement, not assumed. Added: 2026-05-30.

• Package cc-ci itself as a Co-op Cloud recipe (deploy it with abra like any other app). (operator idea 2026-06-02 — not implementing now) Today cc-ci is a NixOS flake that declares the whole server. Idea: split it so the app layer (the !testme bridge, dashboard, and runner) ships as a Co-op Cloud recipe, on top of a host/ Swarm provided by NixOS (or any existing Co-op Cloud host). It would then be deployed/upgraded/ backed-up via abra like any recipe — and could test itself (deploy cc-ci-the-recipe, run its own lifecycle = ultimate self-regression; pairs with the regression-canary work).

  • Pros: dogfooding (cc-ci managed by the same tooling it tests); portability/accessibility (any Co-op Cloud operator could abra app deploy cc-ci instead of needing the flake + a dedicated host); cleaner host-vs-CI-app separation.
  • Cons / crux: the runner must orchestrate Swarm deploys of recipes-under-test, which a normal (unprivileged) recipe can't do — needs either a Docker-socket mount (privileged recipe; test stacks share the host → blast-radius/isolation risk), DinD (heavy/fragile), or a control-plane recipe driving a separate disposable test Swarm over SSH (cleanest, biggest change). Also loses the "entire server reproducible from one flake" property (host + CI-app become two artifacts).
  • When to revisit: after the server regression canaries land (want self-tests before re-architecting the thing that runs them). Treat as a design spike — the runner's Swarm-access model (socket / DinD / separate Swarm) is the decision that drives everything else.