feat(1e): HC3 additive generic + op/assertion split (orchestrator owns the op)

- orchestrator: per mutating tier, run optional pre-op seed hook (ops.py pre_<op>) → perform the op
  ONCE (harness-owned) → run generic assertion (unless opted out) AND overlay assertion, both against
  the shared post-op deployment. Op results passed op→assertion via run-scoped CCCI_OP_STATE_FILE.
- opt-out: CCCI_SKIP_GENERIC / CCCI_SKIP_GENERIC_<OP> / recipe_meta.SKIP_GENERIC (declarative).
- generic.py: split do_* into op primitives (perform_upgrade/backup/restore) + assertions
  (assert_upgraded/backup_artifact/restore_healthy) reading op_state(); deployed_identity now returns
  {version,image,chaos} (chaos label ready for HC1).
- generic test_<op>.py + all 6 recipe overlays migrated to assertion-only; pre-op seeding moved to
  per-recipe ops.py (pre_upgrade/pre_backup/pre_restore). install overlays unchanged (no op).
- deploy-count stays 1 (op primitives never call deploy_app). lint PASS; 8 unit tests PASS on cc-ci.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

tests/lasuite-docs/ops.py

@@ -0,0 +1,41 @@
+"""lasuite-docs — pre-op seed hooks (Phase 1e HC3). The orchestrator runs these BEFORE the op; the
+matching test_<op>.py asserts post-op (assertion-only). The marker is a dedicated `ci_marker` row in
+postgres (the app's Django migrations don't touch it), written via psql in the `db` service. The
+backup path exercises the recipe's pg_backup.sh DB-dump hook (postgres + minio are backupbot-labelled)."""
+
+import os
+import sys
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner"))
+from harness import lifecycle  # noqa: E402
+
+
+def _psql(domain, sql):
+    cmd = f'PGPASSWORD=$(cat /run/secrets/postgres_p) psql -U docs -d docs -tAc "{sql}"'
+    return lifecycle.exec_in_app(domain, ["sh", "-c", cmd], service="db").strip()
+
+
+def _seed(domain, value):
+    _psql(
+        domain,
+        "CREATE TABLE IF NOT EXISTS ci_marker(v text); DELETE FROM ci_marker; "
+        f"INSERT INTO ci_marker VALUES('{value}');",
+    )
+    assert _psql(domain, "SELECT v FROM ci_marker;") == value
+
+
+def pre_upgrade(domain, meta):
+    _seed(domain, "upgrade-survives")
+
+
+def pre_backup(domain, meta):
+    _seed(domain, "original")
+
+
+def pre_restore(domain, meta):
+    # drop the marker table (diverge from the backup) so a successful restore is observable
+    _psql(domain, "DROP TABLE ci_marker;")
+    assert _psql(domain, "SELECT to_regclass('public.ci_marker');") in (
+        "",
+        "NULL",
+    ), "drop did not take"

tests/lasuite-docs/test_backup.py

@@ -1,16 +1,15 @@
-"""lasuite-docs — BACKUP overlay (Phase 1d, DG4): seed a postgres marker, back it up (pg_backup.sh
-pre-hook dumps the DB; assert a snapshot artifact), then mutate (drop it) so the RESTORE overlay
-(test_restore.py) can prove the backed-up state returns. Runs on the shared deployment; the mutated
-state persists for the restore tier.
+"""lasuite-docs — BACKUP overlay (Phase 1e HC3): assertion-only + additive.
 
-Exercises the recipe's real DB-dump backup hook (postgres + minio are both backupbot-labelled); the
-postgres marker is the meaningful Docs-metadata data path."""
+ops.pre_backup wrote "original" into postgres before the backup op (pg_backup.sh dumps the DB); the
+orchestrator performed the backup once (generic tier asserted a snapshot artifact). This overlay
+ADDS: the seeded row is intact at backup time. The backup→restore divergence (dropping the table) is
+in ops.pre_restore."""
 
 import os
 import sys
 
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner"))
-from harness import generic, lifecycle  # noqa: E402
+from harness import lifecycle  # noqa: E402
 
 
 def _psql(domain, sql):
@@ -18,23 +17,7 @@ def _psql(domain, sql):
     return lifecycle.exec_in_app(domain, ["sh", "-c", cmd], service="db").strip()
 
 
-def test_backup_captures_state(live_app, meta):
-    domain = live_app
-
-    # 1) establish original state in postgres, then back up (reuse the generic op: backup +
-    #    assert a snapshot artifact; pg_backup.sh dumps the DB)
-    _psql(
-        domain,
-        "CREATE TABLE IF NOT EXISTS ci_marker(v text); DELETE FROM ci_marker; "
-        "INSERT INTO ci_marker VALUES('original');",
-    )
-    assert _psql(domain, "SELECT v FROM ci_marker;") == "original"
-    snap = generic.do_backup(domain)
-    assert snap, "backup produced no snapshot artifact"
-
-    # 2) mutate: drop the marker table (diverge from the backup)
-    _psql(domain, "DROP TABLE ci_marker;")
-    assert _psql(domain, "SELECT to_regclass('public.ci_marker');") in (
-        "",
-        "NULL",
-    ), "drop did not take"
+def test_backup_captures_state(live_app):
+    assert (
+        _psql(live_app, "SELECT v FROM ci_marker;") == "original"
+    ), "the seeded postgres state was not present at backup time"

tests/lasuite-docs/test_restore.py

@@ -1,17 +1,14 @@
-"""lasuite-docs — RESTORE overlay (Phase 1d, DG4): data-integrity, extends the generic restore.
+"""lasuite-docs — RESTORE overlay (Phase 1e HC3): data-integrity, assertion-only + additive.
 
-Runs after the backup overlay (test_backup.py) on the SAME shared deployment, which left the postgres
-marker table dropped after dumping it. This restores the snapshot via the shared op helper
-(`generic.do_restore`, which also asserts the app is healthy + serving afterwards; the recipe's
-restore.post-hook reloads the dump), then asserts the restored DB matches the pre-mutation "original"
-— the app-specific data integrity the generic restore cannot check. Reads via `psql` in the `db`
-service. Assertion-only (no deploy/teardown)."""
+ops.pre_restore dropped the marker table (diverge); the orchestrator restored once (generic tier
+asserted healthy/serving; the recipe's restore.post-hook reloads the dump). This overlay ADDS: the
+restored DB matches the pre-mutation "original". Read via psql in the `db` service."""
 
 import os
 import sys
 
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner"))
-from harness import generic, lifecycle  # noqa: E402
+from harness import lifecycle  # noqa: E402
 
 
 def _psql(domain, sql):
@@ -19,9 +16,7 @@ def _psql(domain, sql):
     return lifecycle.exec_in_app(domain, ["sh", "-c", cmd], service="db").strip()
 
 
-def test_restore_returns_state(live_app, meta):
-    domain = live_app
-    generic.do_restore(domain, meta)  # restore + assert healthy/serving
+def test_restore_returns_state(live_app):
     assert (
-        _psql(domain, "SELECT v FROM ci_marker;") == "original"
+        _psql(live_app, "SELECT v FROM ci_marker;") == "original"
     ), "restore did not return the pre-mutation postgres state"

tests/lasuite-docs/test_upgrade.py

@@ -1,16 +1,14 @@
-"""lasuite-docs — UPGRADE overlay (Phase 1d, DG4): data-continuity, extends the generic upgrade.
+"""lasuite-docs — UPGRADE overlay (Phase 1e HC3): data-continuity, assertion-only + additive.
 
-The orchestrator deployed the previous published version ONCE; this overlay writes a marker row into
-postgres (a dedicated `ci_marker` table the app's own Django migrations don't touch, read back via
-`psql` in the `db` service), performs the in-place upgrade via the shared op helper
-(`generic.do_upgrade`, which also asserts reconverge + serving + that the deployment moved), then
-asserts the postgres data SURVIVED. Assertion-only on the shared deployment."""
+ops.pre_upgrade wrote a postgres marker row before the upgrade; the orchestrator performed the
+upgrade once (generic tier asserted reconverge/serving/moved). This overlay ADDS: the postgres data
+survived. Read via psql in the `db` service."""
 
 import os
 import sys
 
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner"))
-from harness import generic, lifecycle  # noqa: E402
+from harness import lifecycle  # noqa: E402
 
 
 def _psql(domain, sql):
@@ -18,19 +16,7 @@ def _psql(domain, sql):
     return lifecycle.exec_in_app(domain, ["sh", "-c", cmd], service="db").strip()
 
 
-def test_upgrade_preserves_data(live_app, meta):
-    domain = live_app
-    _psql(
-        domain,
-        "CREATE TABLE IF NOT EXISTS ci_marker(v text); DELETE FROM ci_marker; "
-        "INSERT INTO ci_marker VALUES('upgrade-survives');",
-    )
-    assert _psql(domain, "SELECT v FROM ci_marker;") == "upgrade-survives"
-
-    # in-place upgrade previous -> target (reuses the generic op: upgrade + assert reconverge/serving)
-    generic.do_upgrade(domain, os.environ.get("VERSION") or None, meta)
-
-    assert lifecycle.http_get(domain, "/") in (200, 301, 302)
+def test_upgrade_preserves_data(live_app):
     assert (
-        _psql(domain, "SELECT v FROM ci_marker;") == "upgrade-survives"
+        _psql(live_app, "SELECT v FROM ci_marker;") == "upgrade-survives"
     ), "postgres data did not survive the upgrade"