blocked(2): Q3.2 drive base-deploy hits Docker Hub rate limit + Gitea outage
- recipe_meta: bump drive abra TIMEOUT 900->1500, DEPLOY_TIMEOUT 1200->1800 (12-svc stack w/ onlyoffice+collabora; cold pulls need a wide window). - STATUS-2 ## Blocked: two Class-A1 external blocks documented w/ verify commands — (1) Docker Hub anon pull rate limit (registry-creds finding per plan §1.5; blocks all new deploys), (2) Gitea git.autonomic.zone 404 outage (coordination down; 2 watchdog pings unconsumable until recovery). JOURNAL-2: full disk->prune->rate-limit chain. - Queued locally; push + Adversary-inbox processing deferred to Gitea recovery. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@ -491,3 +491,43 @@ Operator capacity-unblocked cc-ci (RAM 4→8GB, other VMs stopped). Resumed Phas
|
||||
|
||||
Pausing for Adversary cold-verify of Q4.1+Q4.3 (and re-verify of Q3.1+Q3.4 if updated). Will
|
||||
resume on watchdog ping.
|
||||
|
||||
## 2026-05-28 (later) — Q3.2 lasuite-drive base-deploy verify: disk → prune → Docker Hub rate limit; + Gitea outage
|
||||
|
||||
Resumed loop to cold-verify the lasuite-drive base deploy (the f59d8e6 commit deferred OIDC/specific
|
||||
tests until the ~10-service base converges). Chain of events:
|
||||
|
||||
1. **First install run timed out at abra TIMEOUT=900.** abra log root cause was NOT slowness but
|
||||
`FATAL: could not write init file: No space left on device` in postgres init — cc-ci `/` was at
|
||||
**89% (2.9 GB free)**. The ~2GB onlyoffice + ~1GB collabora pulls filled the disk; postgres
|
||||
couldn't initialise. Stack is actually **12 services** (app, backend, celery, celery-beat, db,
|
||||
redis, minio, minio-createbuckets[0/0 one-shot], mailcatcher, web/nginx, collabora, **onlyoffice**)
|
||||
— bigger than the recipe_meta header noted; it ships BOTH office backends by default.
|
||||
|
||||
2. **Freed disk via `docker image prune -af`** → reclaimed 10.1 GB (30 dangling images from prior
|
||||
recipe runs); host went 2.9 GB → 14 GB free. Bumped abra TIMEOUT 900→1500, DEPLOY_TIMEOUT
|
||||
1200→1800 (recipe_meta.py edit; not yet committed — Gitea down, see below).
|
||||
|
||||
3. **Second run progressed far** — db, collabora, onlyoffice, backend, celery, app all reached 1/1.
|
||||
But minio/redis/web/mailcatcher stuck at 0/1 in an instant Assigned→Rejected loop ("No such
|
||||
image"). Manual `docker pull minio/minio:...` returned **`toomanyrequests: You have reached your
|
||||
unauthenticated pull rate limit`**. The prune wiped these (previously-cached) small images, and
|
||||
the full cold re-pull of 12 images — on top of today's many recipe deploys (matrix-synapse,
|
||||
bluesky, ghost, uptime-kuma, keycloak, lasuite-docs, cryptpad retries) — exhausted Docker Hub's
|
||||
per-IP anonymous quota. Big images pulled first; the 4 small ones got starved.
|
||||
|
||||
**Lesson:** pruning is double-edged on this host — it frees disk but forces re-pulls that burn the
|
||||
anonymous rate limit. The real fix is authenticated registry pulls (plan §1.5 "registry pull
|
||||
credentials") + trimming heavy stacks (lasuite-drive does not need BOTH collabora and onlyoffice
|
||||
for WOPI parity — one office backend suffices; disabling onlyoffice cuts the biggest image + RAM).
|
||||
|
||||
4. **Gitea (git.autonomic.zone) is down** — bare host `/`, unauth `/api/v1/version`, and authed repo
|
||||
API all return plain-text `404 page not found` (Go default ServeMux 404 = backend down, proxy has
|
||||
no upstream). Same from both my sandbox and cc-ci (same IP 116.203.211.204), so it's a real
|
||||
instance outage, not my creds/path. Adversary's `/root/adv-verify` clone is stale at 1aaf3bd
|
||||
(clean, no inbox) → Adversary runs in its own sandbox; the only shared channel (Gitea) is dead.
|
||||
**Two watchdog pings arrived (REVIEW-2 update + BUILDER-INBOX.md) that I CANNOT consume** until
|
||||
Gitea recovers — will pull + act the instant it's back.
|
||||
|
||||
Action: interrupted the stuck deploy (let abra TIMEOUT fire for clean teardown). Recording finding;
|
||||
notifying operator (registry creds per §1.5 + Gitea outage). Idle-retry both until recovery.
|
||||
|
||||
Reference in New Issue
Block a user