journal(redfix): M1 bluesky-pds — 000 reproduces deterministically; root cause = caddy↔app cross-stack 'app' alias collision on shared proxy (recipe defect)
Some checks failed
continuous-integration/drone/push Build is failing
Some checks failed
continuous-integration/drone/push Build is failing
This commit is contained in:
autonomic-bot
@ -113,3 +113,48 @@ custom tier / longer-or-smarter retry / serialize), based on the load failure mo
|
||||
|
||||
Reproducibility: 1 green isolation run here + canonical green today + documented red under canon load.
|
||||
Will do 1–2 more isolation repeats before the M1 claim to firm "reproducibly green in isolation."
|
||||
|
||||
## 2026-06-18T00:45Z — M1: bluesky-pds isolation run — 000 REPRODUCES; root cause = `app` DNS collision on shared proxy
|
||||
|
||||
Ran bluesky-pds ALONE (tag 0.3.0+v0.4.219, log /tmp/redfix-bluesky-pds.log). Cold lifecycle GREEN
|
||||
(install/backup/restore/custom pass; upgrade EXPECTED_NA per recipe_meta — moving pds:0.4 tag). Then
|
||||
WC5 promote-on-green-cold FAILED exactly as canon: `warm-bluesky-pds.ci.commoninternet.net: not
|
||||
healthy over HTTPS /xrpc/_health (last status 0)`. So **the 000 reproduces deterministically in
|
||||
isolation — NOT a sweep-load/ACME-rate-limit flake** (my first hypothesis, refuted).
|
||||
|
||||
LIVE DIAGNOSIS (stack left deployed by the failed promote; probed before teardown):
|
||||
- app service 1/1, healthy: `docker exec app wget localhost:3000/xrpc/_health` → `{"version":"0.4.219"}`;
|
||||
app listens on `:::3000`; no restarts. So the PDS itself is fine.
|
||||
- HTTPS to warm domain → 000. caddy logs flood:
|
||||
`tls "failed to get permission for on-demand certificate" domain=warm-bluesky-pds…
|
||||
error=… Get "http://app:3000/tls-check?domain=…": dial tcp 10.10.0.X:3000: connect: connection refused`
|
||||
(X varies: .2 .4 .5 .6 .8 .9 .10 .12).
|
||||
- bluesky uses caddy **on-demand TLS** (Caddyfile: `on_demand_tls { ask http://app:3000/tls-check }`,
|
||||
`tls { on_demand }`, `reverse_proxy app:3000`). caddy must reach app:3000/tls-check to be GRANTED a
|
||||
cert before serving TLS. It can't → no cert → TLS handshake fails → 000.
|
||||
- WHY can't caddy reach app: **service-name `app` collision on the shared `proxy` overlay.**
|
||||
- app is on `warm-bluesky-pds…_internal` ONLY (IP 10.0.3.3). caddy is on `proxy` (10.10.50.223) +
|
||||
`…_internal` (10.0.3.6).
|
||||
- `docker exec caddy getent hosts app` → returns ONLY proxy IPs (8/8 tries: 10.10.0.4/.5/.6/.10/.12),
|
||||
**NEVER the internal 10.0.3.3.** The proxy-net `app` alias shadows bluesky's own internal app.
|
||||
- `docker network inspect proxy` shows EVERY stack aliases its main service `app`:
|
||||
`drone…_app=10.10.0.2`, `traefik…_app=10.10.0.5`, `warm-keycloak…_app=10.10.0.9`,
|
||||
`ccci-reports/bridge/dashboard_app`, … — exactly the IPs caddy hits. None listens a PDS on 3000 →
|
||||
connection refused.
|
||||
So caddy resolves bare `app` to OTHER stacks' app endpoints on the shared proxy, never its own PDS.
|
||||
|
||||
WHY cold passes / warm fails: cold's health window is long (HTTP_TIMEOUT=600) and on first success
|
||||
caddy CACHES the issued cert; the promote's shorter health window doesn't give caddy a chance to ever
|
||||
resolve correctly (and here it provably never resolves to 10.0.3.3 at all). The collision is the root
|
||||
cause; the promote machinery is CORRECT (it refused to write a canonical for an unhealthy 000 — no
|
||||
canonical.json written, verified).
|
||||
|
||||
Classification: **genuine ROUTING/recipe defect — caddy↔app cross-stack `app`-alias collision on the
|
||||
shared proxy net**, deterministic, reproducible in isolation. NOT a flake; NOT a promote-machinery bug.
|
||||
Fix approach (M2): recipe PR giving the PDS service a UNIQUE name/alias (e.g. rename `app`→`pds`) so
|
||||
caddy's `reverse_proxy`/`tls-check` resolve only bluesky's own internal service (no shared-proxy `app`
|
||||
collision). (Alternatively a caddy-side internal-only resolution; renaming is cleanest.) Will confirm
|
||||
the exact fix in M2 + verify the warm domain then serves 200.
|
||||
|
||||
Cleanup: removed orphaned warm-bluesky-pds stack + its volumes/secrets (promote had left it deployed;
|
||||
no canonical written). Node clean.
|
||||
|
||||
@ -38,9 +38,9 @@ flake source per phase plan §2.1). Runs execute on cc-ci from `/etc/cc-ci`.
|
||||
| discourse | DONE @23:40Z (`/tmp/redfix-discourse.log` on cc-ci) | install/backup/restore/custom PASS; **upgrade overlay FAIL**. Deploys+serves fine — NOT a timeout/FATA. | cc-ci overlay `tests/discourse/test_upgrade.py` asserts head runs official `discourse/discourse:3.5.3` + drops sidekiq; latest tag `0.8.1+3.5.0` AND main both still `bitnamilegacy/discourse:3.5.0`+sidekiq (migration exists in no release/main). The `depends_on discourse` string is a non-fatal prepull-only warning, not the deploy. | **stale/PR-specific cc-ci OVERLAY test** mismatched to canonical-sweep context (not flake/timeout/recipe-deploy/warm-machinery) |
|
||||
| mattermost-lts | DONE @00:05Z (`/tmp/redfix-mattermost-lts.log`) | install/upgrade/backup/custom PASS; **restore FAIL** `ci_marker does not exist` — **deterministic in isolation** (not a load race) | recipe `postgres` svc backup labels: backs up hot live PGDATA + dump but has **NO `backupbot.restore.post-hook`** to replay the dump → restore doesn't round-trip postgres. Contrast immich (passes): dump-only `backup.volumes.postgres.path: backup.sql` + `restore.post-hook: /pg_backup.sh restore`. | **genuine RECIPE defect** at latest → recipe PR (adopt immich-style dump+restore-post-hook) |
|
||||
| mumble | DONE @00:18Z (`/tmp/redfix-mumble.log`) | **ALL tiers PASS** incl. handshake; no orphans. Canon red under load; canonical written green TODAY | handshake (TLS+ServerSync) not completing within ~60s retry under heavy concurrent sweep load; fine in isolation | **load/timing FLAKE** → harness stabilization (readiness gate / retry). (1 isolation green; will repeat 1-2× before M1 claim) |
|
||||
| bluesky-pds | running (isolation; promote at end) | — | — | — |
|
||||
| gitea | pending | — | — | — |
|
||||
| keycloak | pending | — | — | — |
|
||||
| bluesky-pds | DONE @00:45Z (`/tmp/redfix-bluesky-pds.log` + live diag) | cold lifecycle GREEN; **WC5 promote 000** reproduces (warm /xrpc/_health last status 0). NOT a flake | caddy on-demand TLS (`ask http://app:3000/tls-check`) can't reach app: caddy resolves bare `app` to OTHER stacks' app endpoints on shared `proxy` net (getent app→only 10.10.0.X, never internal 10.0.3.3; proxy has drone/traefik/keycloak/ccci `app` aliases) → no cert → 000. Promote machinery correct (refused to write canonical). | **genuine routing/RECIPE defect** (cross-stack `app`-alias collision on shared proxy) → recipe PR: unique PDS service name/alias. NOT promote-machinery, NOT flake |
|
||||
| gitea | running (isolation; warm 3.6.0 advance) | — | — | — |
|
||||
| keycloak | pending (mostly design; verify collision) | — | — | — |
|
||||
|
||||
Gate: M1 not yet claimed.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user