Ok, I sank some time into this. It seems that the flow is this. We have a volume mounted for the gitea container living at /var/lib/docker/volumes/gitea_git/_data. In there, there is a ssh folder. Now, when Gitea turns on, it generates host keys into that folder. The first time, it sets the gogs.rsa.pub to 1000:1000 permissions. Things work. The next time Gitea turns on, it sems the gogs.rsa.pub file to root:root permissions and then complains that it can't access it and then generates an in-memory ssh host key which is different! Then we get the spooky error message. So, unable to find a real cause to this, I went ahead and just ran chattr +i gogs.rsa and chattr +i gogs.rsa.pub so that Gitea can't mistakenly reset the permissions for the host keys and they persist and are used. I don't even know if I should raise a bug about this or what is the root of it but it seems solved for now.
This is now finally fixed! I had to add a path to the app.ini config and re-wire permissions for the avatar folders in /var/lib/docker/volumes/gitea_git/_data/ but I've been able to set the avatar for Autonomic coop namespace and restarted and it persisted. Phew!
ALTER DATABASE gitea CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;
OK, confirmed on charset/collation!
MariaDB [gitea]> SELECT SCHEMA_NAME 'database', default_character_set_name 'charset', DEFAULT_COLLATION_NAME 'collation' FROM information_schema.SCHEMATA;
+--------------------+---------+-------------------+
That sounds good but I think the problem needs investigating first. There are some keys stored on the volume but I think they are from before the migration from Dokku and not used anymore. I think somehow Gitea is generating them again and not even saving them to the file system because we make use of the internally implemented SSH server (yep, that is right, golan programmers re-wrote SSH and it actually works!). I think it all stems from my botched migration basically. Would be cool to pair and sort this out one day.
Maintaining the docker daemon certs for the "deploy to swarm" thing is a bit of a pain so far. Haven't figured out how to auto-refresh them and update the places where those certs need to be loaded. Maybe there is a better approach.
Ah shit, right, this is because I have the new certs (that I refreshed a few days back) setup under the autonomic-cooperative namespace in the drone config and not under yours. The kawaiipunk namespace doesn't have access to the new ones.
Practically speaking, this means doing something like this https://git.autonomic.zone/autonomic-cooperative/infrastructure/src/branch/master/scripts/dockertls.sh#L29.
Will get to this when have a bitta time. Other fix is to move it under the autonomic-cooperative namespace.