5.x image incompatibility with older systems

[steam3d]

Describe the issue

After upgrade, I can't run the Weblate. I got the error:
weblate_1 | The /app/data volume is not writable, please adjust the permissions. Weblate is running as uid 1000

I did not make any permissions changes. How to fix this?

I already tried

• I've read and searched the documentation.
• I've searched for similar issues in this repository.

Steps to reproduce the behavior

# Fetch latest versions of the images
docker compose pull
# Stop and destroy the containers
docker compose down
# Spawn new containers in the background
docker compose up -d
# Follow the logs during upgrade
docker compose logs -f

Expected behavior

No response

Screenshots

Exception traceback

No response

How do you run Weblate?

Docker container

Weblate versions

No response

Weblate deploy checks

No response

Additional context

No response

[nijel]

There was no change in users or permissions between those versions, so this is most likely unrelated to the upgrade. What actual permissions of the volume?

[github-actions]

This issue has been marked as a question by a Weblate team member. Why? Because it belongs more to the professional Weblate Care or community Discussions than here. We strive to answer these reasonably fast here, too, but purchasing the support subscription is more responsible and faster for your business. And it makes Weblate stronger as well. Thanks!

In case your question is already answered, making a donation is the right way to say thank you!

[steam3d]

@nijel

I restored server back to weblate 4.18.2. Here is permissions:

drwxr-xr-x 3 root root  4096 Jan  8  2021 weblate-docker_postgres-data
drwxr-xr-x 3 root root  4096 Jan  8  2021 weblate-docker_redis-data
drwxr-xr-x 3 root root  4096 Jan  8  2021 weblate-docker_weblate-data

weblate 4.18.2 works without problem right now

[nijel]

What matters are files inside the weblate-docker_weblate-data volume. Were there any changes in the setup besides upgrading?

[steam3d]

@nijel

Absolutely nothing. I didn't update any dependencies, didn't update ubuntu. I just ran the commands that I wrote in the first message and that’s it

drwxrwx--- 15 1000 root 4096 Jan 19  2021 _data

Inside _data

drwx------  2 1000 1000 69632 Jul  5 16:25 avatar-cache
drwxr-xr-x  2 1000 1000  4096 Jul 17  2021 backups
drwxr-xr-x  3 1000 1000  4096 Dec 13 03:00 borgbackups
drwxr-xr-x  3 1000 1000  4096 Jan  8  2021 cache
drwxr-xr-x  2 1000 1000  4096 Apr  5  2023 celery
drwxr-xr-x  2 1000 1000  4096 Dec 13 01:55 fonts
drwxr-xr-x  5 1000 1000  4096 Jul  5 16:38 home
drwxr-xr-x  3 1000 1000  4096 Jan 19  2021 media
drwxr-xr-x  3 1000 1000  4096 Jan  8  2021 python
-rw-r--r--  1 1000 1000    51 Jan  8  2021 secret
drwxr-xr-x  6 1000 1000  4096 Jul  5 16:38 ssh
drwxr-xr-x  2 1000 1000  4096 Mar 27  2021 ssl
drwxr-xr-x 14 1000 1000  4096 May 30  2021 static
drwxr-xr-x  7 1000 1000  4096 Oct 13 10:05 vcs

[nijel]

What might be the difference is that 5.x container is based on Debian 12 while 4.x was based on Debian 11. I recall there was some obscure issue running new container on old kernels (AFAIR the host system was Debian 9 back then). What kernel does the host system have?

[steam3d]

Kernel:

5.4.0-54-generic WeblateOrg/weblate#60-Ubuntu SMP Fri Nov 6 10:37:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[nijel]

That isn't that old, but meanwhile I've looked up the original ticket, and it was indeed the same issue, and it was resolved by upgrading the kernel there.

What is the root cause is unclear to me, the related code is still the same and is a simple shell condition:

docker/start

Lines 42 to 46 in fe995be

	# Check whether data volume is writable
	if [ ! -w /app/data ] ; then
	echo "The /app/data volume is not writable, please adjust the permissions. Weblate is running as uid $(id -u)"
	exit 1
	fi

[steam3d]

So should I try updating the system and then try updating the weblate?

[nijel]

Yes, that should help. I quickly tried to debug the issue back then, but I gave up, and I couldn't find anything related in the release notes. The volume is indeed not writable as a non-root user from the Debian 12 based containers, it's not just the test that would be wrong.

[steam3d]

I upgraded kernel to the latest one. Now I have such issue:

weblate_1   | Starting Weblate 5.3.1...
weblate_1   | System check identified no issues (1 silenced).
weblate_1   | Postgres 130013 is up
weblate_1   | Starting database migration...
weblate_1   | Operations to perform:
weblate_1   |   Apply all migrations: accounts, admin, auth, authtoken, configuration, contenttypes, django_celery_beat, gitexport, glossary, sessions, social_django, trans, utils, vcs, weblate_auth
weblate_1   | [2023-12-20 07:30:48,520: WARNING/74] Handled exception: KeyError: ('trans', 'change')
weblate_1   | Traceback (most recent call last):
weblate_1   |   File "/usr/local/bin/weblate", line 8, in <module>
weblate_1   |     sys.exit(main())
weblate_1   |              ^^^^^^
weblate_1   |   File "/usr/local/lib/python3.11/site-packages/weblate/runner.py", line 19, in main
weblate_1   |     utility.execute()
weblate_1   |   File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
weblate_1   |     self.fetch_command(subcommand).run_from_argv(self.argv)
weblate_1   |   File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
weblate_1   |     self.execute(*args, **cmd_options)
weblate_1   |   File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
weblate_1   |     output = self.handle(*args, **options)
weblate_1   |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
weblate_1   |   File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
weblate_1   |     res = handle_func(*args, **kwargs)
weblate_1   |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
weblate_1   |   File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 301, in handle
weblate_1   |     pre_migrate_state = executor._create_project_state(with_applied_migrations=True)
weblate_1   |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
weblate_1   |   File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 91, in _create_project_state
weblate_1   |     migration.mutate_state(state, preserve=False)
weblate_1   |   File "/usr/local/lib/python3.11/site-packages/django/db/migrations/migration.py", line 91, in mutate_state
weblate_1   |     operation.state_forwards(self.app_label, new_state)
weblate_1   |   File "/usr/local/lib/python3.11/site-packages/django/db/migrations/operations/fields.py", line 93, in state_forwards
weblate_1   |     state.add_field(
weblate_1   |   File "/usr/local/lib/python3.11/site-packages/django/db/migrations/state.py", line 248, in add_field
weblate_1   |     self.models[model_key].fields[name] = field
weblate_1   |     ~~~~~~~~~~~^^^^^^^^^^^
weblate_1   | KeyError: ('trans', 'change')
weblate_1   |
weblate_1   | Database migration has failed. Please check the error message above.
weblate_1   | Note: Upgrading across major versions is not supported. In case you are upgrading
weblate_1   |       from an 4.x version, please upgrade to 5.0.2 first.

How to downgrade the docker-compose image to 5.0.2?

[steam3d]

Full solution:

1. Upgrade kernel to the latest one:

sudo apt update && sudo apt -y upgrade

2. Add tag to docker-compose.yml weblate file

...
services:
	weblate:
	image: weblate/weblate:5.0.2.2
...

3. Upgrade weblate from 4.x to 5.0.2.2

docker-compose down
docker-compose pull
docker-compose up

4. Wait while migration complete. Check that everything works before go next.
5. Remove tag from docker-compose.yml weblate file:

...
services:
	weblate:
	image: weblate/weblate
...

Upgrade weblate from 5.0.2.2 to the latest

docker-compose down
docker-compose pull
docker-compose up

[github-actions]

Thank you for your report; the issue you have reported has just been fixed.

• In case you see a problem with the fix, please comment on this issue.
• In case you see a similar problem, please open a separate issue.
• If you are happy with the outcome, don’t hesitate to support Weblate by making a donation.

[nijel]

redis/docker-library-redis#365 (comment) might be the root cause here and can be addressed by updating libseccomp and docker on the host running the containers.

[htdhnsdjetsr]

What I effectly did was to follow @steam3d's comment and then I updated my Debian 10 installation to Debian 11 which solved the volume not writable issue and I somehow ended up with weblate 5.5.5