chown: changing ownership of ‘/var/lib/postgresql/data’: Operation not permitted, when running in kubernetes with mounted "/var/lib/postgres/data" volume #361
Comments
|
If the error is coming from the following line, it is just a red herring and thus not really the problem. (I think it was added for when the directory was group writable to fix the ownership since postgres can be particular about that). Line 52 in 57213ab Since this seems like a general debugging question and not really a problem with the image, it'd be better to post questions like this in the Docker Community Forums, the Docker Community Slack, or Stack Overflow. |
|
Same problem here. It happens on a docker mounted dir. I suspect a file system weirdy. Still investigating. |
|
The same situation. What the problem? Maybe NFS? |
|
Same in a sense that I do have this "chown: changing ownership of ‘/var/lib/postgresql/data’: Operation not permitted" error. Any ideas? |
|
To solve this problem do it:
Very important to export nfs/main and create volume inside of the folder nfs/main/data. |
|
Thx! |
|
No, you don't need it. Cuz not every container tries to change owner and permissions of files. |
|
You're right. |
|
Yes, it is. NFS exported path is a just entry point. And nobody cannot change owner of the path. But, inside of the folder you can do everything you want. It is the main idea of the solution. |
|
... not sure in the end if you are saying that mountPath should be /xxx/xxx/xxx/xxx if I plan to mount that specific folder ;-) Can you clarify? |
We have created volume inside of shared folder. |
|
I deserved that one. My question did sound stupid ;-) |
|
Oops, another question sorry: |
|
Yes, should. I showed another steps with different names to understand better. |
|
cheers! |
|
Closing since this issue is environmental, not something we can really fix in the image. In the future (as @yosifkit mentioned above), these sorts of questions/requests would be more appropriately posted to the Docker Community Forums, the Docker Community Slack, or Stack Overflow. Thanks! |
|
@mikygit Can you please share your configuration snippets. I tried to following the instructions here but still getting stuck for hours. I put one line But when the container db starts, it complains a lot about |
|
@lingping525 (and others struggling with this problem, which I guess is trying get Docker for Mac's NFS sharing to work): the init script will chown the db folder and its contents, unless the container is started with |
|
I want to update this issue because I think people are still hitting it via google. I don't think you should add "no_root_squash" to your exports. The answer is in https://hub.docker.com/_/postgres.
So if you set the environment variable PGDATA to Your container will mount /var/lib/postgresql/data and create the pgdata directory. All of the chown operations will work because it's not trying to chown the base directory (which is the nfs mount). |
|
thanks @shewless it worked for me |
|
@shewless don't work in OKD 3.11 with psql 11 |
|
Apologies for the necro, but Google led me here as the first result. The above didn't fix things for me, but what did was to make a docker file with the following: FROM postgres:11.2
# Make us the same gid/id as the nfs mount.
RUN sed -i 's/:999:/:5081:/g' /etc/group
RUN sed -i 's/:999:999:/:5081:5081:/g' /etc/passwd
CMD [ "postgres", "-c", "max_connections=10000"]Specifically I modified the group/passwd gid/uid respectively to match what was the NFS uid/gid and then things started working. Hopefully this helps any other future time travellers. |
|
@eddieparker you don't need to extend the version: '3.2'
services:
postgres:
image: postgres:11
environment:
- PGDATA=/var/lib/postgresql/data/pgdata
user: "${UID:?You must do 'export UID' to launch}:${GID:?You must do 'export GID' to launch}"
volumes:
- nfsmountdbdata:/var/lib/postgresql/data
ports:
- 5432:5432
volumes:
nfsmountdbdata:
driver: local
driver_opts:
type: nfs
o: addr=host.docker.internal,rw
device: ":/Users/me/db/data" |
no_root_squash from docker-library/postgres#361 (comment)
|
Hello. Wokring for me: HOST: |
|
You may want to try changing security context. Note securityContext:, change to your wants. Example code. See https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ |
Thanks for the input. But, unfortunately does not work on postgres 14.1 image. |
|
@BinitaBharati can you provide a bit more detail for reproducing? I can't seem to: 😕 $ docker pull postgres:14.1
14.1: Pulling from library/postgres
Digest: sha256:3162a6ead070474b27289f09eac4c865e75f93847a2d7098f718ee5a721637c4
Status: Image is up to date for postgres:14.1
docker.io/library/postgres:14.1
$ docker run -d --name pg --env PGDATA=/var/lib/postgresql/data/pgdata --env POSTGRES_PASSWORD=password postgres:14.1
fbfeb257794ff6716265a5625332fdb8f911708b3991ba47f90bb4d4b5bf4a5b
$ docker logs --tail=1 pg
2022-02-04 23:41:42.717 UTC [1] LOG: database system is ready to accept connections
$ docker exec pg ls -l /var/lib/postgresql/data
total 4
drwx------ 19 postgres root 4096 Feb 4 23:41 pgdata
$ docker exec pg ls -l /var/lib/postgresql/data/pgdata
total 128
drwx------ 5 postgres postgres 4096 Feb 4 23:41 base
drwx------ 2 postgres postgres 4096 Feb 4 23:41 global
drwx------ 2 postgres postgres 4096 Feb 4 23:41 pg_commit_ts
drwx------ 2 postgres postgres 4096 Feb 4 23:41 pg_dynshmem
-rw------- 1 postgres postgres 4821 Feb 4 23:41 pg_hba.conf
-rw------- 1 postgres postgres 1636 Feb 4 23:41 pg_ident.conf
drwx------ 4 postgres postgres 4096 Feb 4 23:41 pg_logical
drwx------ 4 postgres postgres 4096 Feb 4 23:41 pg_multixact
drwx------ 2 postgres postgres 4096 Feb 4 23:41 pg_notify
drwx------ 2 postgres postgres 4096 Feb 4 23:41 pg_replslot
drwx------ 2 postgres postgres 4096 Feb 4 23:41 pg_serial
drwx------ 2 postgres postgres 4096 Feb 4 23:41 pg_snapshots
drwx------ 2 postgres postgres 4096 Feb 4 23:41 pg_stat
drwx------ 2 postgres postgres 4096 Feb 4 23:41 pg_stat_tmp
drwx------ 2 postgres postgres 4096 Feb 4 23:41 pg_subtrans
drwx------ 2 postgres postgres 4096 Feb 4 23:41 pg_tblspc
drwx------ 2 postgres postgres 4096 Feb 4 23:41 pg_twophase
-rw------- 1 postgres postgres 3 Feb 4 23:41 PG_VERSION
drwx------ 3 postgres postgres 4096 Feb 4 23:41 pg_wal
drwx------ 2 postgres postgres 4096 Feb 4 23:41 pg_xact
-rw------- 1 postgres postgres 88 Feb 4 23:41 postgresql.auto.conf
-rw------- 1 postgres postgres 28835 Feb 4 23:41 postgresql.conf
-rw------- 1 postgres postgres 36 Feb 4 23:41 postmaster.opts
-rw------- 1 postgres postgres 101 Feb 4 23:41 postmaster.pid |
@tianon - I think there is some misunderstanding 🙂 . What I meant to do is this I get this : I got it working without using PGDATA like this: |
|
This still doesn't work for me either. This is my setup:
in terminal, I'm running: output and error: On the other hand, using docker only works:
But this is not really a good option for me, as I need multiple services working together. Any help would be appreciated 🙏 |
|
If you get here from Google and aren't particularly concerned about using a mounted volume you use a non-mounted volume like this: |
|
|
I switched from a mounted host volume to a named volume, and that did the trick. I had multiple issues going on, namely that I was also using version 14 database files with the version 16 "latest" image, but the solution I came up with still worked. Mind you, it was working with a mounted host volume 6 months to a year ago, with no issues. I'm not sure what changed. Here's what I did: In volumes: # ⇦ Add this, you're specifying named volumes outside of services (when inside,
pgdata: # ⇦ its volume or bind mounts). The name-only specification is the 'named' volume.
# ⇦ We could spec an object here and set params, including device path, but - forget all that...
services:
postgres: # https://hub.docker.com/_/postgres
container_name: 'postgres-environment'
image: postgres:latest
environment:
POSTGRES_USER: myuser
POSTGRES_PASSWORD: mypassword
POSTGRES_DB: my_db
PGDATA: /var/lib/postgresql/data/pgdata # ⇦ The recommended place to feed persisted database files/data
ports:
- "5432:5432"
volumes:
- pgdata:/var/lib/postgresql/data # ⇦ Here, we volume mount our named volume.
networks:
- development
deploy:
# Not important
# Adminer below, removed for this purpose...I tried everything, let me tell you - it was such a pain in the butt. The changes, explained further:
Next, I ran [+] Running 4/4
✔ Network docker-compose_development Created 0.0s
✔ Volume "docker-compose_pgdata" Created 0.0s
✔ Container postgres-environment Started 0.4s
✔ Container adminer-environment Started Now, to see where your named volume technically lives, you can So I went ahead and copied my data over to it. Currently, at this stage I have created a container, but it' s restarting endlessly due to other issues that I tackle later. All that matters at this point, however, is that a container is created - not whether it is running. There are several ways to test this, but anyways - it's a convenience to this end. So with the volume mounted to a container: x@y:~$ sudo chown -R rik:rik /docker-volumes/postgres-environment/datadir/pgdata
[sudo] password for rik:
x@y:~$ docker cp /docker-volumes/postgres-environment/datadir/pgdata postgres-environment:/var/lib/postgresql/data
Successfully copied 52.9MB to postgres-environment:/var/lib/postgresql/dataI Finally, I PostgreSQL Database directory appears to contain a database; Skipping initialization
2023-12-30 05:54:47.912 UTC [1] FATAL: database files are incompatible with server
2023-12-30 05:54:47.912 UTC [1] DETAIL: The data directory was initialized by PostgreSQL version 14, which is not compatible with this version 16.1 (Debian 16.1-1.pgdg120+1).So if you happen to have the right version going on, you'll find that you can easily get up and running this way and all while preserving your existing data and configuration. It didn't matter what the permissions were set to initially on that data, I tried 5 different ways; It simply worked. If you run into trouble just chown -R it to be the In my case, I recreated an empty volume and found that my Docker Desktop VM was out of disk space. So after deleting ~60GB of GitLab docker containers and associated images, I found myself up and running and tests passing on my module with the above method - with persisted data kept on my local host, just not where I originally had it. I consider it a fix since using a named volume seems to get rid of all those permissions issues for the long haul. In my case I did hold onto my v14 database files, but I went ahead and let docker-compose recreate my datadir for v16. Using this method, I was able to use my old v14 data with the Setting UID:GID in every fashion did nothing, and every other manner by which to handle this scenario required deriving a new image from a custom Dockerfile derivative. I was looking for a more streamlined and universal way to handle things - and this was it for me. I hope this helps those of you out that were running into this issue similarly to me - who find your way here. I know its slightly diff in context than the issue here, but I feel confident I'll land here again if I run into the issue again in the future. |
|
I solved it by mounting the folder above the In other words, instead of $ docker ... -v /tmp/pgdata:/var/lib/postgresql/data/pgdata --env PGDATA=/var/lib/postgresql/data/pgdata ...I did $ docker ... -v /tmp/pgdata:/var/lib/postgresql/data --env PGDATA=/var/lib/postgresql/data/pgdata ... |
So what you did, was take the data files and configuration files that you keep locally at However, then you still told postgresql to find the data in So what happened when you started the postgres container, was that postgres did not find any data or configuration in the directory it was told to use for it, and so automatically generated it for you. It's no different than when simply not persisting data - except you're persisting the newly created data. Is that what you intended? Are subsequent startups into new instances of a container working without issue for you now with that newly persisted data? Had you done: $ docker ... -v /tmp/:/var/lib/postgresql/data --env PGDATA=/var/lib/postgresql/data/pgdata ...You'd have been closer to doing what I think you were hoping you were doing. However, Here's how I was doing it previously in my version: '3.9'
services:
postgres: # https://hub.docker.com/_/postgres
container_name: 'postgres-environment'
image: postgres:latest
environment:
POSTGRES_USER: myuser
POSTGRES_PASSWORD: mypassword
POSTGRES_DB: my_db
PGDATA: /var/lib/postgresql/data/pgdata
#user: "999:999"
ports:
- "5432:5432"
volumes:
- /docker-volumes/postgres-environment/datadir/:/var/lib/postgresql/data
networks:
- development
deploy:
# Not important
# Adminer below, removed for this purpose...It's the That's where I specifically was facing the permissions issue this time around. It's how I had it resolved previously, however - but it was not working at this point. The permissions that existed on the locally preserved The only other recommended fix that I found that supposedly works was making a dockerfile that starts with Using a named volume really was the magic solution that I was looking for. I took the original persisted data and configuration from my original v14 instance and copied it into the named volume for my new v14 configuration - and was able to fix/resolve/get around those permissions issue(s) while still continuing to use my historic data/configuration. For v16 I went with a named volume as well, letting it generate new data by simply specifying a new empty named volume; It will persist and continue to use that newly generated data from now on though - without permissions issues. Are you clear on the fact that your latest postgres container instance wasn't using your originally persisted data - but was generating new data? At least, according to what you provided, on the initial first run following that change? That while that newly created data should persist and it will be what you're using going forward if the problem is now gone and you keep that configuration - that it isn't the original persisted data that you were previously using? |
|
I am struggling too, still.. setup is a k8s with a mounted volume (SMB). I did the mentioned "trick" with the mounting PATH-A but setting PGDATA to I cant and I dont want to create a "postgres" user on the SMB server? Oo |
|
There are two ways to run the
The second one might require setting |
|
thanks @yosifkit for helping out! I would be happy to follow along using docker or docker-compose.. but I am not sure how to do it properly using kubernetes. here are some screenshots from my shared folders, both having now the 9000:9000 owner:
and this is my deployment.. tried to use the apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres
namespace: paperless
labels:
app: postgres
spec:
selector:
matchLabels:
app: postgres
replicas: 1
template:
metadata:
labels:
app: postgres
spec:
securityContext:
runAsUser: 9000
runAsGroup: 9000
containers:
- name: postgres
image: postgres:16.1-alpine3.19
# command: [ "/bin/bash", "-c", "--" ]
# args: [ "while true; do sleep 30; done;" ]
ports:
- containerPort: 5432
name: 5432tcp5432
protocol: TCP
env:
- name: PGDATA
value: /var/lib/postgresql/data/data-wrapper-for-smb-to-work
- name: POSTGRES_USER
value: pgadmin
- name: POSTGRES_PASSWORD
value: what-the-heck!
- name: POSTGRES_DB
value: paperless
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: paperless
subPath: ./postgres-data
volumes:
- name: paperless
persistentVolumeClaim:
claimName: paperless-pvc-smband here are the postgres logs |
|
For postgres it should be securityContext:
runAsUser: 70
runAsGroup: 70 |
thanks @LaurentGoderre , tried that too. even building a custom image with 1000 instead of 70. even 9000, ... didnt work either. none of them. in the 70's case: it is complaining then that the rights are missing to create the folder, even if it is created. guess it wants to create the nested stuff.. but it is not allowed to.. sure, cause the file-server has no user with 70:70, which has the rights to edit stuff :) one of the most used DBs and its unusable cause of awkward permission stuff :D I gave up.. |
|
@slaesh I think it's a mismatch happening. When UID 70, postgres is happy but the mount volume might not have the right permission. You could try the following line in your custom image replacing the UID with the one you need (postgres seems to care more about the name of the user than the id) RUN echo 'postgres:x:70:70:Postgres:/var/lib/postgresql:/bin/false' > /passwd \
&& echo 'postgres:x:70:postgres' > /group |
|
thanks @LaurentGoderre , but sadly no success. the folder: the Dockerfile:
modified script with logs:
|
|
@slaesh and you're sure that the user |
|
yes. he has. I even created a postgres user on my dev system.. with the UID 1001:
I dont get it... :D |
|
I just re-read, are you trying to mount a Windows folder via SMB? Also are you using Compose or K8S when trying these changes or both? |
|
no its not a windows folder. shared NAS (linxux) folder. k8s all the time :) |
|
I have similar issue, and I found it's because of rootless docker. If you are using rootless Docker, ensure that: > cat /etc/subuid
mio:165536:65536
sheep:165536:65536and then execute: > chown <cuid>:<you_current_user_group> <you_postgres_data_dir>Here, |
I did exactly the same thing for months and it worked as expected. I'm using colima with the docker engine by the way. But since the update to Docker version 27.3.1, build ce1223035a I get the chmod issue: So what has changed in the new version? (Postgresql 17.0) |
|
On another Mac with Docker Desktop and docker version 27.2.0, build 3ab4256 it works! So is this an issue with docker 27.3.1, build ce1223035a? |
|
@tantweiler it's related to QEMU which Colima uses. Luckily, we can switch to Side note: I've discovered you had a StackOverflow question, and I've answered it here |
|
@yusufkandemir thank you very much! Changing the he vm-type to |
|
Switching the Mount Type in Rancher Desktop from |
and others
I am running docker.io/postgres:9.6.5 in kubernetes, pod crash loop back off all the time.
I found this message in log, "chown: changing ownership of ‘/var/lib/postgresql/data’: Operation not permitted"
The text was updated successfully, but these errors were encountered: