Gitea 1.24+ writes settings back to app.ini on boot (e.g. WORK_PATH). When app.ini is mounted directly as a read-only swarm config, that write fails:
[E] Unable to update WORK_PATH=/var/lib/gitea to config "/etc/gitea/app.ini":
failed to save "/etc/gitea/app.ini": open /etc/gitea/app.ini: read-only file system
It is non-fatal (the app stays healthy) but Gitea warns it "might cause bugs when accessing the git repositories" and cannot persist any runtime config change.
Fix
Mount the rendered config at /etc/gitea/app.ini.init (read-only swarm config).
Seed a writable copy at /etc/gitea/app.ini in the config volume via docker-setup.sh.
Use ! -s (empty), not ! -f: upgrades from the old direct-mount layout leave a 0-byte app.ini placeholder that still needs seeding.
Bump DOCKER_SETUP_SH_VERSION v1 → v3.
Verified on cctest (1.26.2)
After the fix, /etc/gitea/app.ini is -rw------- and Gitea persists to it (file grew 1886 → 1901 bytes as it wrote WORK_PATH); the read-only error is gone; /api/healthz passes.
Split out from the image-update PR (#3) so the update can ship without this behaviour change.
## Make app.ini writable so Gitea can persist config
Gitea 1.24+ writes settings back to `app.ini` on boot (e.g. `WORK_PATH`). When `app.ini` is mounted directly as a read-only swarm config, that write fails:
```
[E] Unable to update WORK_PATH=/var/lib/gitea to config "/etc/gitea/app.ini":
failed to save "/etc/gitea/app.ini": open /etc/gitea/app.ini: read-only file system
```
It is **non-fatal** (the app stays healthy) but Gitea warns it "might cause bugs when accessing the git repositories" and cannot persist any runtime config change.
### Fix
- Mount the rendered config at `/etc/gitea/app.ini.init` (read-only swarm config).
- Seed a writable copy at `/etc/gitea/app.ini` in the `config` volume via `docker-setup.sh`.
- Use `! -s` (empty), not `! -f`: upgrades from the old direct-mount layout leave a 0-byte `app.ini` placeholder that still needs seeding.
- Bump `DOCKER_SETUP_SH_VERSION` v1 → v3.
### Verified on cctest (1.26.2)
After the fix, `/etc/gitea/app.ini` is `-rw-------` and Gitea persists to it (file grew 1886 → 1901 bytes as it wrote `WORK_PATH`); the read-only error is gone; `/api/healthz` passes.
Split out from the image-update PR (#3) so the update can ship without this behaviour change.
Reproduced on cc-ci — this PR is required, and the trigger is LFS (not an upgrade artefact)
The warm-gitea canonical has been down (0/1), crash-looping on exactly what this PR fixes:
LoadCommonSettings() [F] Unable to load settings from config: error saving JWT Secret for
custom config: failed to save "/etc/gitea/app.ini": open /etc/gitea/app.ini: read-only file system
I isolated it today by deploying the current release (3.6.3+1.27.1-rootless) in several
configurations on cc-ci:
So the trigger is compose.lfs.yml, not sqlite and not stale state — I also wiped both of
warm-gitea's volumes and redeployed clean, and it still crashed. With LFS_START_SERVER=true, gitea
writes a secret back to app.ini at boot, which cannot work while the rendered config is mounted
directly as a read-only swarm config. This PR's app.ini.init + seeded writable copy is the right
fix.
Note the recipe cannot simply be run without LFS either: app.ini.tmpl references {{ secret "lfs_jwt_secret" }} unconditionally, so dropping the overlay fails earlier still, at
config expansion — secret target lfs_jwt_secret not found.
Why CI stayed green throughout: the tests deploy the default (mariadb, no LFS), which is the one
combination that works. The defect is invisible to the suite.
This PR is stale — it is based on 1.24.2-rootless / APP_INI_VERSION=v22, while main is now 1.27.1-rootless / v23. It needs reconciling with main before it can go green.
I have left warm-giteaundeployed rather than crash-looping (it was churning containers every
~6s). Its .env is untouched, so redeploying restores its intended config once this merges. Nothing
is blocked meanwhile: canonical resolution falls back to a release tag, and gitea's own CI passes.
### Reproduced on cc-ci — this PR is required, and the trigger is LFS (not an upgrade artefact)
The `warm-gitea` canonical has been down (0/1), crash-looping on exactly what this PR fixes:
```
LoadCommonSettings() [F] Unable to load settings from config: error saving JWT Secret for
custom config: failed to save "/etc/gitea/app.ini": open /etc/gitea/app.ini: read-only file system
```
I isolated it today by deploying the **current release (3.6.3+1.27.1-rootless)** in several
configurations on cc-ci:
| config | result |
|---|---|
| `compose.yml` + `compose.mariadb.yml` (recipe default) | **1/1, healthy** |
| `compose.yml` + `compose.sqlite3.yml` + `compose.lfs.yml` (what warm-gitea used) | crash-loop |
| `compose.yml` + `compose.mariadb.yml` + `compose.lfs.yml` | **crash-loop** |
So the trigger is **`compose.lfs.yml`**, not sqlite and not stale state — I also wiped both of
warm-gitea's volumes and redeployed clean, and it still crashed. With `LFS_START_SERVER=true`, gitea
writes a secret back to `app.ini` at boot, which cannot work while the rendered config is mounted
directly as a read-only swarm config. This PR's `app.ini.init` + seeded writable copy is the right
fix.
Note the recipe cannot simply be run without LFS either: `app.ini.tmpl` references
`{{ secret "lfs_jwt_secret" }}` unconditionally, so dropping the overlay fails earlier still, at
config expansion — `secret target lfs_jwt_secret not found`.
**Why CI stayed green throughout:** the tests deploy the default (mariadb, no LFS), which is the one
combination that works. The defect is invisible to the suite.
**This PR is stale** — it is based on `1.24.2-rootless` / `APP_INI_VERSION=v22`, while main is now
`1.27.1-rootless` / `v23`. It needs reconciling with main before it can go green.
I have left `warm-gitea` **undeployed** rather than crash-looping (it was churning containers every
~6s). Its `.env` is untouched, so redeploying restores its intended config once this merges. Nothing
is blocked meanwhile: canonical resolution falls back to a release tag, and gitea's own CI passes.
Gitea 1.24+ persists settings back to app.ini on boot (e.g. WORK_PATH).
Mounting app.ini directly as a read-only swarm config makes that write
fail (logged error, possible repo-access bugs). Mount the rendered config
as app.ini.init and seed a writable copy in the config volume instead.
Rebased onto main (was 12 commits behind). The branch is now 0 behind, one commit, original
author preserved. Conflict was in abra.sh only, resolved to keep both sides:
main
this branch
APP_INI_VERSION
v23
v23 (main's — the app.ini content is unchanged by this fix)
v3 rather than v2 deliberately: v2 was released on main historically with different content
(036f070, c4154c8), and swarm configs are immutable, so reusing that name would fail to update. v3 has only ever carried this fix.
compose.yml merged cleanly — it now has main's gitea/gitea:1.27.1-rootlessand this fix's target: /etc/gitea/app.ini.init. Total diff vs main is 3 files, +10/-2.
Verification
1. The fix resolves the actual bug. Deployed the exact configuration that crash-loops on main
(compose.yml:compose.sqlite3.yml:compose.lfs.yml) from this branch on cc-ci:
replicas: 1/1 read-only app.ini errors: 0
-rw------- git git 1874 /etc/gitea/app.ini <- writable copy, gitea wrote to it
-r--r--r-- root root 1857 /etc/gitea/app.ini.init <- the read-only swarm config
On main that same config crash-loops indefinitely. Note the seeded copy is larger than the
source — that is gitea successfully writing back the settings it could not write before.
2. Full suite, cold, against this branch head (d0243cf):
Every !testme since build 1244 dies before the harness starts, on any recipe (plausible fails
identically right now), with:
OSError: [Errno 28] No space left on device: '/var/lib/cc-ci-runs/<build>'
while the host has 110 GB free and 16% inodes used. It is not disk: I sampled df every 2s
across a whole failing build and available space never moved. The same mkdir succeeds as root over
ssh, in the runner's own mount namespace, and 61/61 times in a stress loop; and the same harness,
same recipe, same numeric run id, run by hand outside drone, proceeds normally. Restarting drone-runner-exec and recreating the runs directory with a fresh inode both changed nothing.
So it is specific to the drone step's execution context, and it is a pre-existing CI-server fault
unrelated to this change. The green verdict above comes from the same harness !testme invokes,
just driven directly.
### Reconciled with main and verified GREEN
Rebased onto `main` (was 12 commits behind). The branch is now **0 behind**, one commit, original
author preserved. Conflict was in `abra.sh` only, resolved to keep **both** sides:
| | main | this branch |
|---|---|---|
| `APP_INI_VERSION` | v23 | **v23** (main's — the app.ini content is unchanged by this fix) |
| `DOCKER_SETUP_SH_VERSION` | v1 | **v3** (bumped — `docker-setup.sh.tmpl` content changes) |
`v3` rather than `v2` deliberately: `v2` was released on main historically with *different* content
(`036f070`, `c4154c8`), and swarm configs are immutable, so reusing that name would fail to update.
`v3` has only ever carried this fix.
`compose.yml` merged cleanly — it now has main's `gitea/gitea:1.27.1-rootless` **and** this fix's
`target: /etc/gitea/app.ini.init`. Total diff vs main is 3 files, +10/-2.
### Verification
**1. The fix resolves the actual bug.** Deployed the exact configuration that crash-loops on `main`
(`compose.yml:compose.sqlite3.yml:compose.lfs.yml`) from this branch on cc-ci:
```
replicas: 1/1 read-only app.ini errors: 0
-rw------- git git 1874 /etc/gitea/app.ini <- writable copy, gitea wrote to it
-r--r--r-- root root 1857 /etc/gitea/app.ini.init <- the read-only swarm config
```
On `main` that same config crash-loops indefinitely. Note the seeded copy is *larger* than the
source — that is gitea successfully writing back the settings it could not write before.
**2. Full suite, cold, against this branch head (`d0243cf`):**
```
install : pass upgrade : pass backup : pass restore : pass custom : pass
level = 5 of 5 VERDICT: GREEN
```
### About the red `!testme` badge — it is not this PR
Every `!testme` since build 1244 dies before the harness starts, on **any** recipe (plausible fails
identically right now), with:
```
OSError: [Errno 28] No space left on device: '/var/lib/cc-ci-runs/<build>'
```
while the host has **110 GB free and 16% inodes used**. It is not disk: I sampled `df` every 2s
across a whole failing build and available space never moved. The same `mkdir` succeeds as root over
ssh, in the runner's own mount namespace, and 61/61 times in a stress loop; and the **same harness,
same recipe, same numeric run id, run by hand outside drone, proceeds normally**. Restarting
`drone-runner-exec` and recreating the runs directory with a fresh inode both changed nothing.
So it is specific to the drone step's execution context, and it is a pre-existing CI-server fault
unrelated to this change. The green verdict above comes from the same harness `!testme` invokes,
just driven directly.
I wrote that "the recipe cannot simply be run without LFS either: app.ini.tmpl references {{ secret "lfs_jwt_secret" }} unconditionally." That is wrong — it is already conditional:
A deploy without the LFS overlay works fine — I had in fact verified that earlier in this
investigation (default compose.yml + compose.mariadb.yml, healthy 1/1) and failed to reconcile
the two observations.
What actually happened: warm-gitea's .env sets GITEA_LFS_START_SERVER=trueindependently of
the overlay. When I dropped compose.lfs.yml from its COMPOSE_FILE while testing, the env var
stayed true, so the conditional evaluated true while the secret was no longer attached to the
service — hence secret target lfs_jwt_secret not found. That was my misconfiguration, not a recipe
defect.
Nothing in this PR changes, and the rest of the diagnosis stands: LFS is still the trigger for
the read-only app.ini crash (mariadb + LFS reproduces it; mariadb alone does not), and this PR is
still the fix. Verified GREEN at build 1250, level 5 of 5.
### Correction to my earlier comment
I wrote that "the recipe cannot simply be run without LFS either: `app.ini.tmpl` references
`{{ secret "lfs_jwt_secret" }}` unconditionally." **That is wrong — it is already conditional:**
```
LFS_START_SERVER = {{ env "GITEA_LFS_START_SERVER" }}
{{ if (eq (env "GITEA_LFS_START_SERVER") "true") }}
LFS_JWT_SECRET = {{ secret "lfs_jwt_secret" }}
{{ end }}
```
A deploy without the LFS overlay works fine — I had in fact verified that earlier in this
investigation (default `compose.yml` + `compose.mariadb.yml`, healthy 1/1) and failed to reconcile
the two observations.
What actually happened: `warm-gitea`'s `.env` sets `GITEA_LFS_START_SERVER=true` **independently of
the overlay**. When I dropped `compose.lfs.yml` from its `COMPOSE_FILE` while testing, the env var
stayed `true`, so the conditional evaluated true while the secret was no longer attached to the
service — hence `secret target lfs_jwt_secret not found`. That was my misconfiguration, not a recipe
defect.
**Nothing in this PR changes**, and the rest of the diagnosis stands: LFS is still the trigger for
the read-only `app.ini` crash (mariadb + LFS reproduces it; mariadb alone does not), and this PR is
still the fix. Verified GREEN at build 1250, level 5 of 5.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Make app.ini writable so Gitea can persist config
Gitea 1.24+ writes settings back to
app.inion boot (e.g.WORK_PATH). Whenapp.iniis mounted directly as a read-only swarm config, that write fails:It is non-fatal (the app stays healthy) but Gitea warns it "might cause bugs when accessing the git repositories" and cannot persist any runtime config change.
Fix
/etc/gitea/app.ini.init(read-only swarm config)./etc/gitea/app.iniin theconfigvolume viadocker-setup.sh.! -s(empty), not! -f: upgrades from the old direct-mount layout leave a 0-byteapp.iniplaceholder that still needs seeding.DOCKER_SETUP_SH_VERSIONv1 → v3.Verified on cctest (1.26.2)
After the fix,
/etc/gitea/app.iniis-rw-------and Gitea persists to it (file grew 1886 → 1901 bytes as it wroteWORK_PATH); the read-only error is gone;/api/healthzpasses.Split out from the image-update PR (#3) so the update can ship without this behaviour change.
Reproduced on cc-ci — this PR is required, and the trigger is LFS (not an upgrade artefact)
The
warm-giteacanonical has been down (0/1), crash-looping on exactly what this PR fixes:I isolated it today by deploying the current release (3.6.3+1.27.1-rootless) in several
configurations on cc-ci:
compose.yml+compose.mariadb.yml(recipe default)compose.yml+compose.sqlite3.yml+compose.lfs.yml(what warm-gitea used)compose.yml+compose.mariadb.yml+compose.lfs.ymlSo the trigger is
compose.lfs.yml, not sqlite and not stale state — I also wiped both ofwarm-gitea's volumes and redeployed clean, and it still crashed. With
LFS_START_SERVER=true, giteawrites a secret back to
app.iniat boot, which cannot work while the rendered config is mounteddirectly as a read-only swarm config. This PR's
app.ini.init+ seeded writable copy is the rightfix.
Note the recipe cannot simply be run without LFS either:
app.ini.tmplreferences{{ secret "lfs_jwt_secret" }}unconditionally, so dropping the overlay fails earlier still, atconfig expansion —
secret target lfs_jwt_secret not found.Why CI stayed green throughout: the tests deploy the default (mariadb, no LFS), which is the one
combination that works. The defect is invisible to the suite.
This PR is stale — it is based on
1.24.2-rootless/APP_INI_VERSION=v22, while main is now1.27.1-rootless/v23. It needs reconciling with main before it can go green.I have left
warm-giteaundeployed rather than crash-looping (it was churning containers every~6s). Its
.envis untouched, so redeploying restores its intended config once this merges. Nothingis blocked meanwhile: canonical resolution falls back to a release tag, and gitea's own CI passes.
!testme
83605ff3fatod0243cf724🌻 cc-ci —
gitea@d0243cf7❌ failure → https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/1244(summary card unavailable — see the run for details.) full logs · dashboard
!testme
🌻 cc-ci —
gitea@d0243cf7❌ failure → https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/1245(summary card unavailable — see the run for details.) full logs · dashboard
!testme
🌻 cc-ci —
gitea@d0243cf7❌ failure → https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/1246(summary card unavailable — see the run for details.) full logs · dashboard
!testme
🌻 cc-ci —
gitea@d0243cf7❌ failure → https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/1247(summary card unavailable — see the run for details.) full logs · dashboard
!testme
🌻 cc-ci —
gitea@d0243cf7❌ failure → https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/1249(summary card unavailable — see the run for details.) full logs · dashboard
Reconciled with main and verified GREEN
Rebased onto
main(was 12 commits behind). The branch is now 0 behind, one commit, originalauthor preserved. Conflict was in
abra.shonly, resolved to keep both sides:APP_INI_VERSIONDOCKER_SETUP_SH_VERSIONdocker-setup.sh.tmplcontent changes)v3rather thanv2deliberately:v2was released on main historically with different content(
036f070,c4154c8), and swarm configs are immutable, so reusing that name would fail to update.v3has only ever carried this fix.compose.ymlmerged cleanly — it now has main'sgitea/gitea:1.27.1-rootlessand this fix'starget: /etc/gitea/app.ini.init. Total diff vs main is 3 files, +10/-2.Verification
1. The fix resolves the actual bug. Deployed the exact configuration that crash-loops on
main(
compose.yml:compose.sqlite3.yml:compose.lfs.yml) from this branch on cc-ci:On
mainthat same config crash-loops indefinitely. Note the seeded copy is larger than thesource — that is gitea successfully writing back the settings it could not write before.
2. Full suite, cold, against this branch head (
d0243cf):About the red
!testmebadge — it is not this PREvery
!testmesince build 1244 dies before the harness starts, on any recipe (plausible failsidentically right now), with:
while the host has 110 GB free and 16% inodes used. It is not disk: I sampled
dfevery 2sacross a whole failing build and available space never moved. The same
mkdirsucceeds as root overssh, in the runner's own mount namespace, and 61/61 times in a stress loop; and the same harness,
same recipe, same numeric run id, run by hand outside drone, proceeds normally. Restarting
drone-runner-execand recreating the runs directory with a fresh inode both changed nothing.So it is specific to the drone step's execution context, and it is a pre-existing CI-server fault
unrelated to this change. The green verdict above comes from the same harness
!testmeinvokes,just driven directly.
!testme
🌻 cc-ci —
gitea@d0243cf7✅ passed → https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/1250(summary card unavailable — see the run for details.) full logs · dashboard
Correction to my earlier comment
I wrote that "the recipe cannot simply be run without LFS either:
app.ini.tmplreferences{{ secret "lfs_jwt_secret" }}unconditionally." That is wrong — it is already conditional:A deploy without the LFS overlay works fine — I had in fact verified that earlier in this
investigation (default
compose.yml+compose.mariadb.yml, healthy 1/1) and failed to reconcilethe two observations.
What actually happened:
warm-gitea's.envsetsGITEA_LFS_START_SERVER=trueindependently ofthe overlay. When I dropped
compose.lfs.ymlfrom itsCOMPOSE_FILEwhile testing, the env varstayed
true, so the conditional evaluated true while the secret was no longer attached to theservice — hence
secret target lfs_jwt_secret not found. That was my misconfiguration, not a recipedefect.
Nothing in this PR changes, and the rest of the diagnosis stands: LFS is still the trigger for
the read-only
app.inicrash (mariadb + LFS reproduces it; mariadb alone does not), and this PR isstill the fix. Verified GREEN at build 1250, level 5 of 5.
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.