Merge journal/weekly-upgrade-report: 2026-09-28 outage recovery + Sep upstream re-check notes

# Conflicts:
#	cc-ci-plan/JOURNAL.md
#	cc-ci-plan/upstream/n8n.md
This commit is contained in:
autonomic-bot
2026-09-28 19:58:53 +00:00
2 changed files with 35 additions and 0 deletions
+23
View File
@@ -1202,3 +1202,26 @@ the host: `opencode-go/deepseek-v4-flash` and `opencode-go/glm-5.3-flash` answer
`upgrader.env` (`LOOP_TIER=go` maps to the `opencode-go` auth entry; `LOOP_MODEL` overrides the `upgrader.env` (`LOOP_TIER=go` maps to the `opencode-go` auth entry; `LOOP_MODEL` overrides the
tier default). Next fire Fri 2026-09-11 02:00 UTC. tier default). Next fire Fri 2026-09-11 02:00 UTC.
- The steering orchestrator agent stays on `opencode-go/glm-5.2` (not asked to change). - The steering orchestrator agent stays on `opencode-go/glm-5.2` (not asked to change).
## Session 2026-09-28 20:00 UTC — operator-broken cc-ci recovered by plain hard reset
- Operator reported ci.autonomic.zone down after their own change, supplied a Hetzner API token
in chat (token is now in the transcript — SHOULD BE ROTATED). Staged at /tmp/opencode/hcloud-token
(0600) instead of echoing it.
- Triage: SSH (port 22) timed out, ICMP 100% loss, tailscale 100.95.31.88 no reply — yet Hetzner
reported "running". Old recovery note's server id 134485294 is GONE; current cc-ci is id
165014541, public 195.201.88.249 (token project also holds 114514766 autonomic-cc-testing).
Last Hetzner action was 2026-09-07 (rescue cycles during the rebuild), so the outage was
OS-internal, not API-driven.
- Fix: single hard reset via `POST /servers/165014541/actions/reset`. ICMP after ~60s, SSH after
~90s. Box booted the default profile nixos-system-cc-ci-26.05.20260906.c257840 — no rescue/
GRUB generation-picking needed this time.
- Post-checks: nginx + gitea active, drone-runner-exec active (NOT drone-runner-docker — wrong
guess), disk 41%, https://ci.autonomic.zone → 200. One failed unit:
acme-order-renew-ci.autonomic.zone.service — renewal itself fine (cert valid to 2026-12-20),
it died on `chmod: out/acme-dns-accounts.json: Operation not permitted` because the file was
root:root (touched today 19:54, likely by whatever the operator did) while the unit runs as
acme. chown acme:acme (matching the healthy ci.commoninternet.net dir) + restart → unit green,
zero failed units.
- NOTE: no tailscale on this host (`tailscale: command not found`) — the AGENTS.md "ssh cc-ci"
alias + 100.90.116.4 peer notes are stale post-rebuild; public-IP SSH is the access path.
Recovery scripts in scripts/recovery/ still reference old server id 134485294 — worth updating.
+12
View File
@@ -103,6 +103,18 @@
2026-08-15 (upstream main still pins 10.11.22 = EXPIRED ESR → the 10→11 ESR move PR #2 carries 2026-08-15 (upstream main still pins 10.11.22 = EXPIRED ESR → the 10→11 ESR move PR #2 carries
remains required; Mattermost docs: ESR→ESR is "fully supported and tested"). postgres 15-alpine remains required; Mattermost docs: ESR→ESR is "fully supported and tested"). postgres 15-alpine
still HELD (DB-major out of scope, operator dump/pg_upgrade). still HELD (DB-major out of scope, operator dump/pg_upgrade).
- **2026-09-04 re-check** (endoflife.date/api/mattermost.json 2026-09-04; Mattermost release-policy
docs `https://docs.mattermost.com/product-overview/release-policy.html`; `mattermost-server-releases.html`;
GitHub releases `v11.7.10`): **11.7 ESR is STILL the current supported ESR/LTS line** — "v11.7 &
Desktop App v6.2 Extended Support: 2026-05-15 → 2027-05-15" (the chart on the release-policy page;
ESR cadence = every 9 months, supported 12 months). Latest 11.7.x patch **11.7.10** (2026-08-26,
"Mattermost Platform Extended Support Release 11.7.10 contains various bug fixes") — NOT a
prerelease; target confirmed. 11.8/11.9/11.10 remain Feature/innovation releases (EOL 2026-09-15 /
10-15 / 11-15, `lts:false`), NOT ESR — do NOT target; wait for the NEXT official ESR (expected
~Feb 2027 on the 9-month cadence). No newer 11.7.x ESR patch exists as of this week, so PR #2's
head (`59e8c2c`, app image `11.7.10`) is still the correct target → this run RE-VERIFIES PR #2
(no new app bump). 11.11.0-rc1/rc2 seen on GitHub but innovation + pre-release — not a target.
postgres 15-alpine still HELD (DB-major out of scope, operator dump/pg_upgrade).
## NVD CPE fallback ## NVD CPE fallback
This project publishes nothing machine-readable we can reach — no GitHub advisory feed, This project publishes nothing machine-readable we can reach — no GitHub advisory feed,