From 916bdd8b683c2f78817395c4c22b734826cef0f0 Mon Sep 17 00:00:00 2001 From: autonomic-bot Date: Fri, 29 May 2026 21:13:56 +0100 Subject: [PATCH] =?UTF-8?q?feat(2):=20Q4.9=20mailu=20=E2=80=94=20recipe=5F?= =?UTF-8?q?meta=20+=20health=20+=203=20functional=20(create-mailbox/imap-l?= =?UTF-8?q?ogin/mail-flow);=20P4=20N/A=20deferred?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit mailu (full email stack). TLS_FLAVOR=notls avoids certdumper/ACME dep (cc-ci file-provider cert); MAIL_DOMAIN/HOSTNAMES=run domain; TRAEFIK_STACK_NAME for the letsencrypt-volume mount. P2 vacuous (no corpus). P3: test_mailbox (flask mailu user create + config-export read-back), test_imap_login (mailbox authenticates over dovecot IMAP:143), test_mail_flow (SMTP submission send → IMAP retrieve, auth to avoid greylisting). P4 N/A (no backupbot label) — DEFERRED.md + PARITY.md, Adversary §7.1 sign-off pending. Smoke-validated: 8 services converge, mail ports 25/587/143/993 host-open, flask CLI. Co-Authored-By: Claude Opus 4.8 (1M context) --- machine-docs/DEFERRED.md | 14 +++ tests/mailu/PARITY.md | 39 ++++++++ tests/mailu/functional/_mailu.py | 45 +++++++++ tests/mailu/functional/test_health_check.py | 21 ++++ tests/mailu/functional/test_imap_login.py | 49 ++++++++++ tests/mailu/functional/test_mail_flow.py | 100 ++++++++++++++++++++ tests/mailu/functional/test_mailbox.py | 29 ++++++ tests/mailu/recipe_meta.py | 32 +++++++ 8 files changed, 329 insertions(+) create mode 100644 tests/mailu/PARITY.md create mode 100644 tests/mailu/functional/_mailu.py create mode 100644 tests/mailu/functional/test_health_check.py create mode 100644 tests/mailu/functional/test_imap_login.py create mode 100644 tests/mailu/functional/test_mail_flow.py create mode 100644 tests/mailu/functional/test_mailbox.py create mode 100644 tests/mailu/recipe_meta.py diff --git a/machine-docs/DEFERRED.md b/machine-docs/DEFERRED.md index 1451e9d..1b3ad12 100644 --- a/machine-docs/DEFERRED.md +++ b/machine-docs/DEFERRED.md @@ -276,3 +276,17 @@ before the build is called done) — but does **not** force closure. - **Re-entry trigger:** upstream discourse recipe ships a deployable image version; OR operator approves a cc-ci-authored discourse recipe-PR + release. - **Linked IDEA / BACKLOG:** Q4.6. + +### 2026-05-29 — mailu: no backup config (P4 N/A) — recipe-PR to add backupbot +- [ ] **What:** mailu (Q4.9) ships **no `backupbot.backup` label** on any service, so cc-ci's + backup/restore tiers cleanly SKIP (`backup_capable=False`) — P4 (backup data-integrity) is N/A + for mailu as published (no backup mechanism to exercise). Durable fix = a recipe-PR adding + backupbot labels (admin sqlite DB at /data + the `mailu` mail volume), mirroring the immich Q3.5 + / Q3.2b pattern. +- **Filed by:** Builder, phase 2 (Q4.9 mailu enrollment). +- **Reason for deferral:** UPSTREAM recipe has no backup config; adding it is a recipe change + (operator-merge-gated via recipe-create-pr), not a cc-ci/test change. mailu install+upgrade+ + functional (create-mailbox + IMAP-login + send/receive mail-flow) are covered. +- **Re-entry trigger:** Adversary §7.1 sign-off accepting P4-N/A for mailu, OR operator approves a + cc-ci-authored mailu backupbot recipe-PR. +- **Linked IDEA / BACKLOG:** Q4.9. diff --git a/tests/mailu/PARITY.md b/tests/mailu/PARITY.md new file mode 100644 index 0000000..d502318 --- /dev/null +++ b/tests/mailu/PARITY.md @@ -0,0 +1,39 @@ +# mailu — recipe-maintainer → cc-ci parity (Phase 2 P2) + +**P2 is VACUOUS for mailu:** there is no `recipe-info/mailu/tests/` corpus in the recipe-maintainer +workspace (`/srv/recipe-maintainer`), so there are no recipe-maintainer tests to port. Coverage is +therefore health + recipe-specific functional tests (P3), authored from what mailu *is* (a full +email stack: nginx front + admin + postfix/smtp + dovecot/imap + rspamd/antispam + webmail + redis). + +## cc-ci deployment notes +- `COMPOSE_FILE=compose.yml` (base). recipe_meta.EXTRA_ENV(domain) pins `MAIL_DOMAIN`/`HOSTNAMES` to + the per-run domain, `TRAEFIK_STACK_NAME=traefik_ci_commoninternet_net` (so the external + `*_letsencrypt` volume the certdumper mounts resolves), and **`TLS_FLAVOR=notls`** — mailu's + mail-port TLS normally comes from `certdumper` dumping traefik's ACME `acme.json`, but cc-ci uses a + file-provider wildcard cert (no ACME), so there is no acme.json; `notls` removes that dependency. + (certdumper still runs idle; harmless — it converges 1/1.) Web/admin is served over the real + wildcard TLS via Traefik. Mail ports 25/465/587/110/143/993/995 are published mode:host → on-host + (cc-ci-run) tests reach SMTP/IMAP at 127.0.0.1. + +## Recipe-specific functional tests (P3 — ≥2; here 3) +1. `functional/test_mailbox.py` — §4.3 create-an-object + read-back: create a mailbox via the admin + container's `flask mailu user` CLI, then read it back from `flask mailu config-export --json` and + assert the address is present (admin-DB provisioning round-trip). +2. `functional/test_imap_login.py` — distinctive (working mail account): the provisioned mailbox + AUTHENTICATES against dovecot over IMAP (127.0.0.1:143) and SELECTs its INBOX — proving the + account is real end-to-end (admin → dovecot auth), not just a DB row. +3. `functional/test_mail_flow.py` — the characteristic end-to-end flow: SEND a uniquely-marked + message to the mailbox over SMTP (authenticated submission to avoid rspamd greylisting), then + RETRIEVE it over IMAP and assert the marker arrived (postfix → rspamd → dovecot deliver/store/fetch). + +## Backup data-integrity (P4) — N/A (recipe ships no backup config) +The upstream mailu recipe declares **no `backupbot.backup` label** on any service, so the cc-ci +backup/restore tiers cleanly SKIP (`backup_capable=False`). There is no recipe backup mechanism to +exercise — P4 is genuinely N/A for mailu as published, not a cut corner. The durable fix (if P4 +coverage is wanted) is a recipe-PR adding backupbot labels (mailu admin sqlite at /data + mail +volume), filed as a deferral mirroring the immich Q3.5 / Q3.2b pattern — see DEFERRED.md. Pending +Adversary §7.1 sign-off on the N/A. + +## Browser flow (P6) +Not added: mailu's user-facing UX (webmail/admin) is a standard web UI; the characteristic behaviour +(mail send/receive, account auth) is covered functionally above. No Playwright flow owed. diff --git a/tests/mailu/functional/_mailu.py b/tests/mailu/functional/_mailu.py new file mode 100644 index 0000000..45702dd --- /dev/null +++ b/tests/mailu/functional/_mailu.py @@ -0,0 +1,45 @@ +"""Shared mailu helpers — headless mailbox provisioning via the admin container's `flask mailu` CLI. + +mailu's admin container ships the management CLI (`flask mailu domain|user|alias|config-export ...`), +the canonical headless way to provision mail objects. We exec it via the harness (service="admin"). +The primary MAIL_DOMAIN is auto-created at boot; `flask mailu domain ... || true` is idempotent. +""" + +from __future__ import annotations + +import json +import os +import sys + +sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "..", "runner")) +from harness import lifecycle # noqa: E402 + + +def flask_mailu(domain: str, shell_cmd: str) -> str: + """Run a `flask mailu ...` shell command in the admin container; return stdout.""" + return lifecycle.exec_in_app(domain, ["sh", "-c", shell_cmd], service="admin") + + +def ensure_domain(domain: str, mail_domain: str) -> None: + # idempotent: the primary domain is auto-created; tolerate "already exists" + flask_mailu(domain, f"flask mailu domain {mail_domain} || true") + + +def create_user(domain: str, local: str, mail_domain: str, password: str) -> str: + email = f"{local}@{mail_domain}" + flask_mailu(domain, f"flask mailu user {local} {mail_domain} '{password}'") + return email + + +def config_export(domain: str) -> dict: + """`flask mailu config-export --json` → parsed dict (keys: domain/user/alias/relay).""" + out = flask_mailu(domain, "flask mailu config-export --json") + # Be robust to any leading banner: extract the JSON object. + start = out.find("{") + end = out.rfind("}") + assert start != -1 and end != -1, f"config-export produced no JSON: {out[:200]!r}" + return json.loads(out[start : end + 1]) + + +def user_emails(cfg: dict) -> list[str]: + return [u.get("email") for u in cfg.get("user", [])] diff --git a/tests/mailu/functional/test_health_check.py b/tests/mailu/functional/test_health_check.py new file mode 100644 index 0000000..f9c35d7 --- /dev/null +++ b/tests/mailu/functional/test_health_check.py @@ -0,0 +1,21 @@ +"""mailu — health/readiness functional test (Phase 2). + +SOURCE: no recipe-maintainer corpus exists for mailu (P2 vacuous — see PARITY.md). The nginx front +serves `/` (301 → /webmail) once the stack is up; this is the canonical "is mailu serving" signal. +""" + +from __future__ import annotations + +import os +import sys + +sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "..", "runner")) +from harness import http as harness_http # noqa: E402 + + +def test_mailu_front_serves(live_app): + url = f"https://{live_app}/" + status, _ = harness_http.retry_http_get( + url, expect_status=(200, 301, 302), max_wait=120, interval=5 + ) + assert status in (200, 301, 302), f"GET {url} HTTP {status} (mailu front not serving)" diff --git a/tests/mailu/functional/test_imap_login.py b/tests/mailu/functional/test_imap_login.py new file mode 100644 index 0000000..5b2be0f --- /dev/null +++ b/tests/mailu/functional/test_imap_login.py @@ -0,0 +1,49 @@ +"""mailu — recipe-specific functional test #2 (Phase 2 P3, distinctive: working IMAP mailbox). + +Beyond "the row exists in config" (test_mailbox), this proves the provisioned mailbox is a REAL, +usable IMAP account: it authenticates against dovecot over IMAP (port 143, host-published) and can +SELECT its INBOX. That exercises mailu's actual mail-account stack (admin provisioning → dovecot auth +backend), not just the admin DB. notls deploy → plain IMAP on 143. Hard assertion, non-vacuous: wrong +credentials / a non-working account fail the login. +""" + +from __future__ import annotations + +import imaplib +import os +import sys +import time +import uuid + +sys.path.insert(0, os.path.dirname(__file__)) +import _mailu # noqa: E402 + + +def test_created_mailbox_authenticates_over_imap(live_app): + mail_domain = live_app + local = "ccci-imap-" + uuid.uuid4().hex[:8] + password = "CcCi-" + uuid.uuid4().hex[:16] + "-Aa1!" + _mailu.ensure_domain(live_app, mail_domain) + email = _mailu.create_user(live_app, local, mail_domain, password) + + # The account may take a moment to be usable by dovecot after admin provisioning; retry login. + deadline = time.time() + 90 + last_err = None + while time.time() < deadline: + try: + imap = imaplib.IMAP4("127.0.0.1", 143) + try: + imap.login(email, password) + typ, _ = imap.select("INBOX") + assert typ == "OK", f"IMAP SELECT INBOX returned {typ} for {email}" + imap.logout() + return # authenticated + selected INBOX → working mailbox + finally: + try: + imap.shutdown() + except Exception: # noqa: BLE001 + pass + except (imaplib.IMAP4.error, OSError) as e: # noqa: PERF203 + last_err = e + time.sleep(5) + raise AssertionError(f"created mailbox {email} could not authenticate over IMAP:143 — {last_err}") diff --git a/tests/mailu/functional/test_mail_flow.py b/tests/mailu/functional/test_mail_flow.py new file mode 100644 index 0000000..eecf505 --- /dev/null +++ b/tests/mailu/functional/test_mail_flow.py @@ -0,0 +1,100 @@ +"""mailu — recipe-specific functional test #3 (Phase 2 P3, the characteristic end-to-end mail flow). + +Provision a mailbox, SEND a uniquely-marked message to it over SMTP, then RETRIEVE it over IMAP and +assert the marker arrived. This is mailu's defining behaviour (it is a mail server): a full +deliver→store→fetch round-trip across postfix + rspamd + dovecot — not an API/health stand-in. + +To avoid rspamd greylisting (which defers an unknown inbound sender's first attempt for minutes), we +send via AUTHENTICATED submission as the user itself (authenticated mail is not greylisted). notls +deploy → submission accepts AUTH on 587; we try 587 (STARTTLS-if-offered) then fall back to 25. +Retrieval polls INBOX (and Junk, in case rspamd files it) for the marker. +""" + +from __future__ import annotations + +import email.message +import imaplib +import os +import smtplib +import sys +import time +import uuid + +sys.path.insert(0, os.path.dirname(__file__)) +import _mailu # noqa: E402 + + +def _send(addr_from: str, addr_to: str, password: str, marker: str) -> None: + msg = email.message.EmailMessage() + msg["From"] = addr_from + msg["To"] = addr_to + msg["Subject"] = marker + msg.set_content(f"cc-ci mail-flow test body {marker}") + + last = None + # Authenticated submission on 587 (no greylist); fall back to plain 25 local delivery. + for port in (587, 25): + try: + smtp = smtplib.SMTP("127.0.0.1", port, timeout=30) + try: + smtp.ehlo() + if smtp.has_extn("starttls"): + smtp.starttls() + smtp.ehlo() + if smtp.has_extn("auth"): + try: + smtp.login(addr_from, password) + except smtplib.SMTPException as e: # noqa: PERF203 + last = e # 25 may not offer/allow AUTH; local delivery still works + smtp.send_message(msg) + return + finally: + try: + smtp.quit() + except Exception: # noqa: BLE001 + pass + except (smtplib.SMTPException, OSError) as e: + last = e + raise AssertionError(f"could not send mail via SMTP 587/25: {last}") + + +def _retrieve(email_addr: str, password: str, marker: str, timeout: int = 150) -> bool: + deadline = time.time() + timeout + while time.time() < deadline: + try: + imap = imaplib.IMAP4("127.0.0.1", 143) + try: + imap.login(email_addr, password) + for box in ("INBOX", "Junk"): + typ, _ = imap.select(box) + if typ != "OK": + continue + typ, data = imap.search(None, "SUBJECT", f'"{marker}"') + if typ == "OK" and data and data[0].split(): + return True + imap.logout() + finally: + try: + imap.shutdown() + except Exception: # noqa: BLE001 + pass + except (imaplib.IMAP4.error, OSError): # noqa: PERF203 + pass + time.sleep(5) + return False + + +def test_send_and_receive_mail(live_app): + mail_domain = live_app + local = "ccci-flow-" + uuid.uuid4().hex[:8] + password = "CcCi-" + uuid.uuid4().hex[:16] + "-Aa1!" + _mailu.ensure_domain(live_app, mail_domain) + email_addr = _mailu.create_user(live_app, local, mail_domain, password) + + marker = "ccci-mailflow-" + uuid.uuid4().hex + _send(email_addr, email_addr, password, marker) + + assert _retrieve(email_addr, password, marker), ( + f"mail with marker {marker!r} was sent to {email_addr} but never arrived in INBOX/Junk " + f"over IMAP within the delivery window (postfix→rspamd→dovecot)" + ) diff --git a/tests/mailu/functional/test_mailbox.py b/tests/mailu/functional/test_mailbox.py new file mode 100644 index 0000000..70db196 --- /dev/null +++ b/tests/mailu/functional/test_mailbox.py @@ -0,0 +1,29 @@ +"""mailu — recipe-specific functional test #1 (Phase 2 P3 §4.3 create-an-object + read-back). + +Create a mailbox (the characteristic mailu object) via the admin CLI, then read it back from the +authoritative config export and assert it is present. Real create→persist→read-back, not health-only. +""" + +from __future__ import annotations + +import os +import sys +import uuid + +sys.path.insert(0, os.path.dirname(__file__)) +import _mailu # noqa: E402 + + +def test_create_mailbox_and_read_back(live_app): + mail_domain = live_app # recipe_meta sets MAIL_DOMAIN = the per-run domain + local = "ccci-" + uuid.uuid4().hex[:8] + password = "CcCi-" + uuid.uuid4().hex[:16] + "-Aa1!" + + _mailu.ensure_domain(live_app, mail_domain) + email = _mailu.create_user(live_app, local, mail_domain, password) + + cfg = _mailu.config_export(live_app) + emails = _mailu.user_emails(cfg) + assert email in emails, ( + f"created mailbox {email} not present in mailu config-export users {emails}" + ) diff --git a/tests/mailu/recipe_meta.py b/tests/mailu/recipe_meta.py new file mode 100644 index 0000000..21cc553 --- /dev/null +++ b/tests/mailu/recipe_meta.py @@ -0,0 +1,32 @@ +# Per-recipe harness config for mailu (Phase 2 Q4.9 — full email stack: nginx front + admin + +# dovecot(imap) + postfix(smtp) + rspamd(antispam) + webmail + redis + certdumper). +# +# cc-ci integration notes: +# - TLS_FLAVOR=notls: mailu's mail-port TLS normally comes from `certdumper`, which dumps certs from +# traefik's ACME acme.json. cc-ci uses a FILE-PROVIDER wildcard cert (no ACME), so there is no +# acme.json — certdumper would never produce certs. notls disables mail-port TLS so the stack does +# not depend on those certs. (certdumper still runs idle, mounting the existing +# traefik_ci_commoninternet_net_letsencrypt volume; harmless.) The web/admin UI is served over the +# real wildcard TLS via Traefik regardless. +# - TRAEFIK_STACK_NAME must match cc-ci's traefik stack so the external `*_letsencrypt` volume resolves. +# - MAIL_DOMAIN/HOSTNAMES are pinned to the per-run domain (callable EXTRA_ENV). +# - Mail ports (25/465/587/110/143/993/995) are published mode:host on the cc-ci host, so on-host +# tests can reach SMTP/IMAP at 127.0.0.1. +# Smoke (mail-smoke deploy) showed the nginx front serves `/` → 301 (redirect to /webmail) once up; +# /admin briefly 502s during admin first-boot, so `/` is the stable readiness signal. (The +# create-mailbox test uses the admin-container `flask mailu` CLI, which bypasses HTTP entirely.) +HEALTH_PATH = "/" +HEALTH_OK = (200, 301, 302) +DEPLOY_TIMEOUT = 900 +HTTP_TIMEOUT = 600 + + +def EXTRA_ENV(domain): + return { + "MAIL_DOMAIN": domain, + "HOSTNAMES": domain, + "TRAEFIK_STACK_NAME": "traefik_ci_commoninternet_net", + "TLS_FLAVOR": "notls", + "SITENAME": "ccci-mail", + "POSTMASTER": "admin", + }