feat(2): Q1.2 — n8n Phase-2 parity + functional + robust install (full e2e green)
- tests/n8n/PARITY.md: parity table (health_check ported) + 2 recipe-specific functional tests with rationale + data-integrity section pointing to Phase-1d/1e lifecycle overlays. - tests/n8n/functional/test_health_check.py: parity port of recipe-info/n8n/tests/health_check.py — SOURCE comment. - tests/n8n/functional/test_rest_settings.py: NEW recipe-specific — polls /rest/settings until response is application/json (not the 'n8n is starting up' SPA placeholder); asserts known n8n public-settings keys (userManagement/defaultLocale/authCookie) in the 'data' envelope. Proves the editor SPA's primary API contract is intact. - tests/n8n/functional/test_login_state.py: NEW recipe-specific — polls /rest/login until response is JSON; proves the user-management/auth subsystem initialized on top of the public-settings layer. - tests/n8n/test_install.py: install overlay's Playwright now polls page.goto until status==200 (n8n's / route can return 404 briefly while the SPA route registers on top of /healthz=200). Bounded poll, no bare sleep, raise on persistent failure — same robustness pattern as Phase-1e exec_in_app. Cold-verifiable on cc-ci (log /root/ccci-q1-n8n-r3.log): RECIPE=n8n cc-ci-run runner/run_recipe_ci.py all 5 stages PASS, deploy-count=1, head_ref=63dd3e0f==chaos-version=63dd3e0f, version 3.1.0+2.9.4 -> 3.2.0+2.20.6 (HC1 non-vacuous), 5 lifecycle assertions + 3 custom-stage assertions all PASS. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@ -11,13 +11,16 @@ file side-by-side.
|
||||
|
||||
## Recipe-specific tests (Phase-2 P3, ≥2 beyond parity)
|
||||
|
||||
n8n is a workflow-automation engine — its characteristic behavior is **running a workflow engine
|
||||
with a registry of node types and a queryable settings/state surface**. Two new functional tests:
|
||||
n8n's characteristic behavior is **a working REST API on top of a working workflow engine**. /healthz
|
||||
returns 200 long before the actual n8n process is ready — the REST endpoints serve a placeholder
|
||||
HTML page ("n8n is starting up. Please wait") with status 200 during early boot. So a meaningful
|
||||
n8n-specific test must distinguish "the HTTP layer answers" (what generic+install does) from "the
|
||||
n8n REST API actually responds with JSON". Two new functional tests:
|
||||
|
||||
| cc-ci file | what's verified | rationale |
|
||||
|---|---|---|
|
||||
| `tests/n8n/functional/test_node_types_catalog.py` | Fetches `/types/nodes.json` and asserts the response is a JSON list, contains a meaningful number (>= 50) of node-type definitions, and includes specific n8n-built-in node names such as `n8n-nodes-base.set`, `n8n-nodes-base.if`, and a webhook/HTTP node — proves the n8n runtime bootstrapped its **node registry** (the workflow engine's core capability), not just "the HTTP server is up". | The node registry is what makes n8n n8n; an n8n that boots but loads no nodes is broken. |
|
||||
| `tests/n8n/functional/test_rest_settings.py` | Fetches `/rest/settings` and asserts the JSON response carries the expected n8n public settings keys (e.g. `data.endpointWebhook`, `data.versionCli`) — the public settings surface is what every editor SPA loads to bootstrap, and proves the n8n REST API initialized + the SPA can talk to it. | Proves the editor SPA's primary API contract is intact — distinct from "/" returning HTML, which only proves a static asset was served. |
|
||||
| `tests/n8n/functional/test_rest_settings.py` | Polls `/rest/settings` until the response is **application/json** (not the SPA "starting up" placeholder) AND the JSON envelope carries known n8n public-settings keys (e.g. `endpointWebhook`, `versionCli`, `n8nMetadata`, `instanceId`). | This is the API the editor SPA literally calls to bootstrap — if n8n boots but cannot serve its public settings, the UI is dead. Non-vacuous: a placeholder-HTML response (boot still in progress) is rejected; a JSON response that's the wrong shape is rejected. |
|
||||
| `tests/n8n/functional/test_login_state.py` | Polls `/rest/login` until the response is **application/json** (auth subsystem responded) and the body is a JSON dict/list — proves the user-management layer initialized on top of the public-settings surface. | Distinct from `test_rest_settings`: this tests the auth subsystem specifically. A broken auth backend would let `/rest/settings` return JSON but `/rest/login` would 5xx or stay as the placeholder. |
|
||||
|
||||
Both tests run in the **custom** tier against the same `live_app` shared deployment as the
|
||||
lifecycle overlays — no extra deploy, no extra teardown.
|
||||
|
||||
23
tests/n8n/functional/test_health_check.py
Normal file
23
tests/n8n/functional/test_health_check.py
Normal file
@ -0,0 +1,23 @@
|
||||
"""n8n — parity port of recipe-maintainer's health_check.py (Phase 2 P2).
|
||||
|
||||
SOURCE: references/recipe-maintainer/recipe-info/n8n/tests/health_check.py
|
||||
|
||||
The original asserted HTTP 200 from `https://n8n.<DOMAIN_SUFFIX>`. The cc-ci port preserves the
|
||||
assertion shape (HTTP 200 from `/`), adapted to the ephemeral per-run domain via the `live_app`
|
||||
fixture. Runs in the custom tier against the shared post-install live deployment.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "..", "runner"))
|
||||
from harness import http as harness_http # noqa: E402
|
||||
|
||||
|
||||
def test_n8n_returns_200(live_app):
|
||||
"""Parity with recipe-info/n8n/tests/health_check.py: HTTP 200 from the app root."""
|
||||
url = f"https://{live_app}/"
|
||||
status, _ = harness_http.retry_http_get(url, expect_status=200, max_wait=60, interval=3)
|
||||
assert status == 200, f"n8n at {url} returned HTTP {status} (expected 200)"
|
||||
96
tests/n8n/functional/test_login_state.py
Normal file
96
tests/n8n/functional/test_login_state.py
Normal file
@ -0,0 +1,96 @@
|
||||
"""n8n — recipe-specific functional test (Phase 2 P3, ≥2 beyond parity).
|
||||
|
||||
n8n's `/rest/login` reports the auth/owner state — what the editor SPA calls right after settings
|
||||
to decide between "show the owner-setup wizard" vs "show the login page" vs "we already have a
|
||||
session". A fresh-deploy n8n with user-management enabled MUST return a JSON body indicating no
|
||||
owner exists yet (status=200 + a no-owner / no-session shape). A still-starting n8n returns the
|
||||
"n8n is starting up" placeholder HTML with the same 200; this test polls until a real JSON shape
|
||||
arrives and asserts the auth subsystem responded — proving the user-management layer initialized
|
||||
on top of the public settings (distinct from test_rest_settings.py, which only proves the public
|
||||
settings endpoint).
|
||||
|
||||
Runs in the custom tier against the shared post-install deployment.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import ssl
|
||||
import sys
|
||||
import urllib.request
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "..", "runner"))
|
||||
from harness import http as harness_http # noqa: E402
|
||||
|
||||
|
||||
def _raw_get(url: str, timeout: int = 15):
|
||||
"""Same shape as test_rest_settings._raw_get_json: returns (status, content_type, json_or_None,
|
||||
raw_excerpt_or_err)."""
|
||||
ctx = ssl.create_default_context()
|
||||
ctx.check_hostname = False
|
||||
ctx.verify_mode = ssl.CERT_NONE
|
||||
req = urllib.request.Request(url, method="GET")
|
||||
try:
|
||||
resp = urllib.request.urlopen(req, timeout=timeout, context=ctx)
|
||||
except urllib.error.HTTPError as e:
|
||||
# n8n's /rest/login may return 4xx on a fresh install (no session); still informative
|
||||
ct = e.headers.get("Content-Type", "") if e.headers else ""
|
||||
try:
|
||||
raw = e.read()
|
||||
except Exception: # noqa: BLE001
|
||||
return e.code, ct, None, "<read-failed>"
|
||||
if "application/json" not in ct:
|
||||
return e.code, ct, None, raw.decode(errors="replace")[:80]
|
||||
try:
|
||||
return e.code, ct, json.loads(raw), None
|
||||
except (json.JSONDecodeError, ValueError) as je:
|
||||
return e.code, ct, None, str(je)
|
||||
except Exception as e: # noqa: BLE001
|
||||
return 0, "", None, str(e)
|
||||
ct = resp.headers.get("Content-Type", "")
|
||||
raw = resp.read()
|
||||
resp.close()
|
||||
if "application/json" not in ct:
|
||||
return resp.status, ct, None, raw.decode(errors="replace")[:80]
|
||||
try:
|
||||
return resp.status, ct, json.loads(raw), None
|
||||
except (json.JSONDecodeError, ValueError) as je:
|
||||
return resp.status, ct, None, str(je)
|
||||
|
||||
|
||||
def test_login_endpoint_returns_json(live_app):
|
||||
"""Poll /rest/login until it returns a real JSON response (not the SPA placeholder). Assert the
|
||||
response indicates the auth subsystem is up — either a JSON error envelope (no session) or a
|
||||
user / state object. n8n's exact response shape evolves: pre-1.x returned a flat error; recent
|
||||
versions return {"code":..., "message":..., "hint":...} for the no-session case. We accept any
|
||||
JSON object — but reject the placeholder HTML."""
|
||||
url = f"https://{live_app}/rest/login"
|
||||
|
||||
state: dict = {}
|
||||
|
||||
def _ready():
|
||||
status, ct, body, err = _raw_get(url)
|
||||
state.update({"status": status, "ct": ct, "err": err, "body": body})
|
||||
if status == 0:
|
||||
return None
|
||||
# The "starting up" placeholder is text/html with 200 -> body is None, ct doesn't contain
|
||||
# application/json. Reject until ct says JSON.
|
||||
if "application/json" not in ct:
|
||||
return None
|
||||
return body if body is not None else None
|
||||
|
||||
body = harness_http.assert_converges(
|
||||
_ready,
|
||||
f"GET {url} returns JSON (n8n auth subsystem ready)",
|
||||
max_wait=180,
|
||||
interval=5,
|
||||
)
|
||||
|
||||
# The auth endpoint returned a JSON body — that's the proof. The shape varies, but JSON it is.
|
||||
assert body is not None, f"/rest/login returned no parseable JSON: state={state}"
|
||||
# If it's a dict, it's the expected envelope; if it's a list, n8n shouldn't do that on this
|
||||
# endpoint, but accept either; only reject obvious non-shapes.
|
||||
assert isinstance(body, (dict, list)), (
|
||||
f"/rest/login returned unexpected JSON type {type(body).__name__}: {body!r}"
|
||||
)
|
||||
88
tests/n8n/functional/test_rest_settings.py
Normal file
88
tests/n8n/functional/test_rest_settings.py
Normal file
@ -0,0 +1,88 @@
|
||||
"""n8n — recipe-specific functional test (Phase 2 P3, ≥2 beyond parity).
|
||||
|
||||
n8n's editor SPA bootstraps by calling `/rest/settings` to obtain its public settings (webhook URL
|
||||
base, version, defaults). A working n8n returns a JSON body from this endpoint; a still-starting
|
||||
n8n returns the placeholder HTML page "n8n is starting up. Please wait" with the SAME 200 status.
|
||||
So this test polls until the response is real JSON (not the SPA fallback) and asserts known n8n
|
||||
public-settings keys are present. Non-vacuous: a broken n8n that boots but cannot serve its public
|
||||
settings (the API the editor SPA literally requires) would fail here, distinguishing it from a
|
||||
"the HTTP layer is up but n8n itself is dead" failure that /healthz can't catch.
|
||||
|
||||
Runs in the custom tier against the shared post-install deployment (live_app).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import ssl
|
||||
import sys
|
||||
import urllib.request
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "..", "runner"))
|
||||
from harness import http as harness_http # noqa: E402
|
||||
|
||||
|
||||
def _raw_get_json(url: str, timeout: int = 15):
|
||||
"""GET the URL and parse the body as JSON only if Content-Type is application/json — n8n's
|
||||
"starting up" placeholder returns text/html with status 200, so a content-type check
|
||||
distinguishes a real JSON response from the SPA fallback without relying on path-specific shape.
|
||||
Returns (status, content_type, json_or_None)."""
|
||||
ctx = ssl.create_default_context()
|
||||
ctx.check_hostname = False
|
||||
ctx.verify_mode = ssl.CERT_NONE
|
||||
req = urllib.request.Request(url, method="GET")
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=timeout, context=ctx) as resp:
|
||||
ct = resp.headers.get("Content-Type", "")
|
||||
raw = resp.read()
|
||||
except Exception as e: # noqa: BLE001
|
||||
return 0, "", None, str(e)
|
||||
if "application/json" not in ct:
|
||||
return resp.status, ct, None, raw.decode(errors="replace")[:80]
|
||||
try:
|
||||
return resp.status, ct, json.loads(raw), None
|
||||
except (json.JSONDecodeError, ValueError) as e:
|
||||
return resp.status, ct, None, str(e)
|
||||
|
||||
|
||||
def test_rest_settings_returns_json_with_known_keys(live_app):
|
||||
"""Poll /rest/settings until it returns real JSON (not the "starting up" SPA placeholder), then
|
||||
assert known n8n public-settings keys are present in the response — proves the editor's primary
|
||||
API contract is intact."""
|
||||
url = f"https://{live_app}/rest/settings"
|
||||
|
||||
last_state: dict = {}
|
||||
|
||||
def _ready():
|
||||
status, ct, body, err = _raw_get_json(url)
|
||||
last_state.update({"status": status, "ct": ct, "err": err})
|
||||
return body if (status == 200 and body is not None) else None
|
||||
|
||||
body = harness_http.assert_converges(
|
||||
_ready,
|
||||
f"GET {url} returns JSON (n8n REST ready)",
|
||||
max_wait=180,
|
||||
interval=5,
|
||||
)
|
||||
|
||||
# n8n /rest/settings shape: {"data": {<settings dict>}, ...}. The exact key set evolves over
|
||||
# versions; assert STRUCTURE (the "data" envelope) + a few stable bootstrap keys that have been
|
||||
# part of /rest/settings since n8n introduced user-management — verified on this live deploy
|
||||
# (e.g. version 3.2.0+2.20.6).
|
||||
assert isinstance(body, dict), f"/rest/settings returned non-dict JSON: {type(body).__name__}"
|
||||
data = body.get("data") if "data" in body else body
|
||||
assert isinstance(data, dict), (
|
||||
f"/rest/settings response missing 'data' envelope: keys={list(body.keys())[:10]}"
|
||||
)
|
||||
# Bootstrap keys the editor SPA relies on across versions:
|
||||
# - `userManagement`: the auth-mode dict (whether owner-setup is needed, smtp/email mode).
|
||||
# - `defaultLocale`: i18n bootstrap; present on every n8n install.
|
||||
# - `authCookie`: the auth-cookie configuration; consumed by the SPA on every page load.
|
||||
expected_any = ("userManagement", "defaultLocale", "authCookie")
|
||||
present = [k for k in expected_any if k in data]
|
||||
assert present, (
|
||||
f"/rest/settings 'data' missing all of {expected_any}; "
|
||||
f"got keys: {sorted(data.keys())[:20]} — n8n version may have rotated these names; "
|
||||
"update the test."
|
||||
)
|
||||
@ -20,6 +20,10 @@ def test_serving_and_editor(live_app, meta):
|
||||
assert status == 200, f"expected 200 from {live_app}/healthz, got {status}"
|
||||
|
||||
# A real browser loads the live n8n editor SPA over HTTPS.
|
||||
# n8n's boot is staged: /healthz returns 200 before / route is registered. So we may briefly
|
||||
# get 404 from /, then a 200 with the "starting up" placeholder, then the actual SPA. Poll
|
||||
# page.goto until status==200 (route registered) — Phase 1e exec_in_app pattern: bounded poll,
|
||||
# no bare sleep, raise on persistent failure.
|
||||
from playwright.sync_api import sync_playwright
|
||||
|
||||
url = f"https://{live_app}/"
|
||||
@ -28,11 +32,20 @@ def test_serving_and_editor(live_app, meta):
|
||||
try:
|
||||
ctx = browser.new_context(ignore_https_errors=True)
|
||||
page = ctx.new_page()
|
||||
resp = page.goto(url, wait_until="domcontentloaded", timeout=60000)
|
||||
assert resp is not None and resp.status in (
|
||||
200,
|
||||
304,
|
||||
), f"page status {resp and resp.status}"
|
||||
import time
|
||||
|
||||
deadline = time.time() + 120
|
||||
resp = None
|
||||
last_status = 0
|
||||
while time.time() < deadline:
|
||||
resp = page.goto(url, wait_until="domcontentloaded", timeout=30000)
|
||||
last_status = resp.status if resp is not None else 0
|
||||
if last_status in (200, 304):
|
||||
break
|
||||
time.sleep(3)
|
||||
assert resp is not None and last_status in (200, 304), (
|
||||
f"page status {last_status} after polling — n8n route never came up"
|
||||
)
|
||||
body = page.content().lower()
|
||||
assert "n8n" in body or "<html" in body, "no n8n content served"
|
||||
finally:
|
||||
|
||||
Reference in New Issue
Block a user