Operator 2026-06-16. Replaces the static UPGRADE_BASE_VERSION + leaky single
compose.ccci.yml overlay model: dynamic base = last-green(warm canonical) ->
main fallback -> skip; optional minimal per-recipe previous/ folder for
base-only version repairs (ignored for head, version-guarded, removable when
stale). Validated on discourse PR #4 (official-image switch the current overlay
masks). regall then sweeps all recipes for regressions on sonnet.
Operator-requested 2026-06-15. gitea is currently only a dep provider for
drone; this phase builds the full recipe-under-test suite (install/upgrade/
backup/restore/custom + lint + screenshot), ports the upstream parity corpus
(health_check, git_push), and verifies recipe-maintainers/gitea PR #1
('feat: support Git LFS on plain gitea', branch lfs-plain-gitea) via an LFS
round-trip + JWT-stability capstone test — red/skipped on main, green on the
PR. Central constraint: do NOT break drone's gitea-dep path (shared
recipe_meta.py). builder+adversary on sonnet.
The agents.toml diff also records the already-live aoeng/aotest/porepo/poe2e
queue head (agent-orchestrator workstream; their plan files remain owned by
that effort).
§2 no longer bumps the coop-cloud version label in the recipe PR (no
--dry-run compute, no tag). It records the recommended 'abra recipe release
<recipe> -<x|y|z>' (no --dry-run) in the PR body (§3) as the operator's
final publish step — run after the upstream PR merges, it bumps the label +
tags + pushes in one go. Bumps recipe-maintainer submodule to 9daddac (same
change across its /recipe-upstream, -upgrade-apply, -upgrade-plan, -new-tag).
cc-ci recipe-upgrade skill now computes the version via 'abra recipe release --dry-run'
(not a hand-edit) and requires the PR body to link upstream release notes per service.
Bumps the recipe-maintainer submodule pointer to the matching change.
Operator: uptime-kuma is maintained elsewhere — drop it from the weekly upgrade
but keep it in the used-recipes inventory. New cc-ci-plan/used-recipes.md is the
canonical list of every recipe cc-ci deploys/tests, with a weekly|external tier;
upgrade-all §1 now excludes 'external' rows from the candidate list (explicit
--args still override). uptime-kuma = external; all others weekly.
Re-target the traefik health gate off ci.commoninternet.net (the dashboard,
which is After=deploy-proxy) onto a traefik-self endpoint, breaking the
fresh-boot deadlock while keeping health-gated rollback. M1 controlled repro by
the loops; M2 from-scratch cold-boot proof owned by the orchestrator.
The unified agents.py watchdog kept firing the hourly orchestrator supervision
ping even after SEQUENCE-COMPLETE (the old launch.py watchdog exited on
completion, which stopped them). Gate the wake loop on the SEQUENCE-COMPLETE
marker so a finished build stays fully at rest — no pings. Resumes
automatically when new work is queued (that clears the marker, line 631).
The durable /16 proxy fix landed in phase pvfix (2026-06-13).
Update the guard description from "safety net until that lands"
to "belt-and-suspenders even after the /16 fix" — guard logic
unchanged, description now accurate.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The unification transcribed phases from .phases-spec before cf48 was added,
so the operator's just-requested opus 4.8 cfold review got dropped. Re-append
it after ghost (system is past cf55/on pvfix, so can't insert before pvfix
without shifting the live phase index). agents.py re-reads config each tick.
Second independent review of the cfold custom-folder collapse, by Opus 4.8
instead of GPT-5.5, inserted after cf55 (queue ...cfold;cf55;cf48;pvfix;...).
Per-phase overrides .loop-model[-adv]-cf48=claude-opus-4-8 on the claude backend.
Root-caused (empirically, dockerd logs) the discourse/ghost deploy wedges:
the shared proxy overlay (/24=254 VIPs) exhausts as concurrent stack rm leaks
endpoints over many days -> tasks stuck in Swarm 'New'. Add a per-run safety
net to Step 0 (network prune + docker restart when VIP-allocation failures are
logged). Plans + memory for the durable fix (enlarge proxy to /16 in swarm.nix,
maintenance window) and for debugging/fixing the ghost PR afterward.
Per operator: just work through recipes alphabetically keeping CAP (=
DRONE_RUNNER_CAPACITY=2) subagents running at once, starting the next the moment
one finishes (rolling pool via run_in_background). Removes the wave-barrier and
the heavy/light classification entirely — simpler and no slot ever idles.
Per operator: always fill all CAP slots. Heavy/light alternation only spreads
heavies across waves while a light is available; once only heavies remain, run
two-per-wave (capacity is the tuned ceiling) instead of one-per-wave.
Host memory is the binding limit, so never schedule two HEAVY recipes in the
same capacity wave — pair each heavy (discourse/immich/matrix-synapse/
lasuite-drive/mattermost-lts/ghost) with a light one to bound peak memory while
keeping both slots busy. Heaviest-first could co-schedule two heavies and OOM/
wedge the box (the disc-50cc8a 'New'-state wedge). For CAP>2 cap heavies at
~CAP/2; if only heavies remain, run one-per-wave.
Now that the 2026-06-10 concurrency restructure makes concurrent recipe runs
safe (per-run trees, app-domain locks, isolation), default /upgrade-all to run
up to DRONE_RUNNER_CAPACITY (the drone runner's slots, currently 2) recipe
subagents at a time instead of strictly sequential — using all available
concurrency without oversubscribing. Query the live capacity from
'systemctl show drone-runner-exec' (fallback 2); process recipes in waves of
CAP (emit CAP Agent calls per message, await, next wave). Flags: --capacity N,
--sequential (CAP=1, old default — use when the build loops share the box),
--parallel (unbounded). Applies to the NEXT run; the in-flight run is unaffected.
The functional/playwright split is purely organizational (discovery globs both
with no branching; same custom tier -> L4 rung, same fixtures, same failure
semantics). Migrate all custom tests to one custom/ folder; M1 proves coverage
identical before/after (no silent drops), M2 is a full real-CI !testme sweep
across all recipes confirming levels unchanged. cfold becomes the last phase so
the queued /upgrade-all fires after it (folder change verified before upgrade).
Per AGENTS.md 'Agent memory lives in memory/ (in this repo)' — memory notes
must be committed + pushed like any repo change, not left only in the local
~/.claude symlink target.
Recent phases wrote STATUS/BACKLOG/REVIEW/JOURNAL to the repo ROOT because
build_kickoff + plan.md's tree used bare filenames, even though the loops'
AGENTS.md + INBOX/DECISIONS/DEFERRED conventions already said machine-docs/.
Make machine-docs/ the single mandated home everywhere: build_kickoff now
emits machine-docs/ paths + an explicit FILE-LOCATION RULE; both loop prompts
and plan.md (tree + seed step) updated; orchestrator AGENTS.md documents +
enforces it. resolve_state/INBOX handoff already read machine-docs/ first.
When LOG_DIR/.run-upgrade-on-complete exists, the watchdog launches
launch-upgrader.py start the moment the last phase reaches ## DONE (then
consumes the flag). Lets the operator replace a scheduled weekly cron run with
'run as soon as the current phase queue finishes' — used tonight: the
cc-ci-upgrade-all.timer was stopped (stamp forwarded past tonight's slot) and
this flag set instead.
A Builder scaffolded 'STATUS-mailu.md' with a '## DONE / Not yet. Written
here only when ...' placeholder section; phase_done's startswith('## DONE')
matched it and auto-advanced past mailu without any of its work being done
(no recipe PR, no claim, no review). Harden phase_done: a '## DONE' heading
counts only when its first non-empty body line is not a placeholder/negation
(Not yet / pending / TBD / when all / <...> etc). Verified against all shipped
STATUS files (real DONEs still detected; mailu placeholder rejected).
Lets a single phase pin a different model, read fresh each role_model call so
a phase transition flips it automatically with no watchdog bounce. Operator
wants builder on opus for the complex dstamp phase, reverting to sonnet from
mailu on: .loop-model-dstamp=opus while base .loop-model stays sonnet.
Root-cause the upstream image breakage (Cannot find module /app/index.js,
Node v24 under the pinned tag — proven harness/ref-neutral in rcust M2),
research upstream releases (persist to cc-ci-plan/upstream/bluesky-pds.md),
fix via recipe-mirror PR (NEVER merge — operator does), prove full lifecycle
green incl. the new L5 lint rung via !testme at PR head, then verify a real
credential-free screenshot on those runs (hook only if needed). Close both
DEFERRED bluesky entries; crisp operator handoff in STATUS-bsky.md.
The 2026-06-11 night watch is over: the limit-wait system was verified
end-to-end on a real monthly-spend-limit window (hit -> hold without reboots
-> flat probes -> prompt resume on lift), and the three bugs it surfaced are
fixed (5ea17fc, 969eb60). Standing supervision continues without the extra
check.
The tick whose probe resumed a session was continuing into stall logic with
its pre-resume pane capture; a 4h-old WAITING-UNTIL in that stale data got
the freshly-resumed adversary kill+rebooted (05:52). Treat probe-resume as
handled-this-tick; the next 30s tick sees the live session.
Night-watch findings (monthly-spend-limit window, ~01:49-04:45):
- probe text said 'usage limit' which matches LIMIT_RE, so a submitted probe
kept limited_now true forever -> reworded to 'quota window' with a CAUTION
note (nudge text must never match LIMIT_RE)
- dedupe scanned all 40 captured lines, so once a probe scrolled into the
conversation no further probe ever fired (builder/adv frozen at nudges=1,
orchestrator probes degraded to hourly riding the wake scroll) -> dedupe
now only checks the bottom 8 lines (input area)
Core invariant HELD: zero kill+reboots during the limit window.
plan(lvl5): operator addition - the top-corner level badge (card, dashboard
pill, badge SVG) shows only the level number+color, zero capping info; the
inline per-rung table keeps intentional-skip/unverified detail.
Operator refinement: only declared/structural skips (not backup-capable, no
previous version) let the climb continue; a rung that should have run but
didn't (infra error, abra missing, tier abort, timeout) blocks the level at
the last verified rung. Every N/A source in derive_rungs gets an explicit
classification (DECISIONS.md, adversary-reviewed); unclassifiable defaults to
unverified. Unit tests + one synthesized tier-abort run prove the rule.
