-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
After installing docker-ce 25.0.0 when building Dockerimage, the container won't run because of ulimit error #4807
Comments
Thanks for reporting. It looks like this is an issue with code in the "engine" code, not the CLI itself, so probably the better location would be in https://github.com/moby/moby. Unfortunately, GitHub doesn't allow transfering tickets between orgs, so I cannot move it there (but perhaps you could open a new ticket?) This issue is related to moby/moby@c893010, which is part of this PR; That PR changed the default ulimits to follow systemd's defaults (only raising the hard-limit, not the soft-limit). When running the docker service as a systemd unit, this would be handled by systemd, but the sysvinit scripts are provided for non-standard setups where systemd is not used (likely in your container). I wondered if it was perhaps that ubuntu 20.04 didn't provide the docker run --rm ubuntu:20.04 bash -c 'ulimit --help'
ulimit: ulimit [-SHabcdefiklmnpqrstuvxPT] [limit]
Modify shell resource limits.
Provides control over the resources available to the shell and processes
it creates, on systems that allow such control.
Options:
-S use the `soft' resource limit
-H use the `hard' resource limit
... |
Oh, wait; but you're running |
It works with the previous docker-ce version, yes. We don't get any errors there. |
I have a similar issue i run bitwarden self-hosting at work i run rocky8 Until now it has worked for years Something is wrong with the latest version. |
Maybe the initial startup worked because no containers were started, so it had just enough privileges to run. I wonder if the previous What version of docker is running on the host (i.e., what version of docker is used to build the image)? Can you provide the output of |
|
Thanks! So, I think this is the issue indeed #4807 (comment);
I tried reproducing the issue; I slightly simplified the Dockerfile, and reduced it to only the essential packages, and split the "install docker" step to a separate # syntax=docker/dockerfile:1
FROM ubuntu:20.04
RUN apt-get update -y; \
apt-get install -y \
ca-certificates \
curl \
gnupg \
lsb-release \
software-properties-common; \
rm -rf /var/cache/apt;
RUN curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg; \
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null; \
apt-get update -y; \
apt-get install -y docker-ce; \
rm -rf /var/cache/apt;
RUN \
echo "ulimits: $(ulimit -Sn):$(ulimit -Hn)"; \
service docker start; \
rm -rf /var/cache/apt;
ENTRYPOINT service docker start && /bin/bash When building the Dockerfile on a docker 25.0 engine with BuildKit, the build works without errors; docker build -t foo --no-cache --progress=plain .
#8 22.27 Setting up docker-ce (5:25.0.0-1~ubuntu.20.04~focal) ...
#8 22.30 invoke-rc.d: could not determine current runlevel
#8 22.31 invoke-rc.d: policy-rc.d denied execution of start.
#8 22.44 Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
#8 22.57 Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
#8 22.57 Setting up xauth (1:1.1-0ubuntu1) ...
#8 22.58 Setting up liberror-perl (0.17029-1) ...
#8 22.58 Setting up git (1:2.25.1-1ubuntu3.11) ...
#8 22.61 Processing triggers for libc-bin (2.31-0ubuntu9.14) ...
#8 22.64 Processing triggers for systemd (245.4-4ubuntu3.22) ...
#8 22.65 Processing triggers for mime-support (3.64ubuntu1) ...
#8 DONE 22.7s
#9 [4/4] RUN echo "ulimits: $(ulimit -Sn):$(ulimit -Hn)"; service docker start; rm -rf /var/cache/apt;
#9 0.320 ulimits: 524288:1024
#9 0.334 * Starting Docker: docker
#9 0.336 ...done.
#9 DONE 0.3s
#10 exporting to image
#10 exporting layers
#10 exporting layers 3.0s done
#10 writing image sha256:92263a9e8ab60769e2127898c6c2fa5489aa66a383df8523862dcae96962043e
#10 writing image sha256:92263a9e8ab60769e2127898c6c2fa5489aa66a383df8523862dcae96962043e done
#10 naming to docker.io/library/foo done
#10 DONE 3.1s When using the the legacy builder (BuildKit disabled through The classic builder starts containers through containerd, whereas BuildKit starts containers through BuildKit (in which case DOCKER_BUILDKIT=0 docker build -t foo --no-cache .
...
Setting up docker-ce (5:25.0.0-1~ubuntu.20.04~focal) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
Setting up xauth (1:1.1-0ubuntu1) ...
Setting up liberror-perl (0.17029-1) ...
Setting up git (1:2.25.1-1ubuntu3.11) ...
Processing triggers for libc-bin (2.31-0ubuntu9.14) ...
Processing triggers for systemd (245.4-4ubuntu3.22) ...
Processing triggers for mime-support (3.64ubuntu1) ...
---> Removed intermediate container 968fb0a992f2
---> c3500524a198
Step 4/5 : RUN echo "ulimits: $(ulimit -Sn):$(ulimit -Hn)"; service docker start; rm -rf /var/cache/apt;
---> Running in b8bb305e36bc
ulimits: 1073741816:1073741816
/etc/init.d/docker: 62: ulimit: error setting limit (Invalid argument)
---> Removed intermediate container b8bb305e36bc
---> 59eb60be41b5
Step 5/5 : ENTRYPOINT service docker start && /bin/bash
---> Running in b0455bfa8db2
---> Removed intermediate container b0455bfa8db2
---> 2bcfe0c9b509
Successfully built 2bcfe0c9b509
Successfully tagged foo:latest When running the build on an older version of docker with the previous LIMIT_NOFILE, it fails with BuildKit as well. Here's the same build on a docker 24.0.2 (Ubuntu 18.04) test-machine that I didn't update yet; docker build -t foo --no-cache --progress=plain .
#8 41.47 Setting up docker-ce (5:25.0.0-1~ubuntu.20.04~focal) ...
#8 41.55 invoke-rc.d: could not determine current runlevel
#8 41.56 invoke-rc.d: policy-rc.d denied execution of start.
#8 41.84 Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
#8 42.08 Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
#8 42.09 Setting up xauth (1:1.1-0ubuntu1) ...
#8 42.10 Setting up liberror-perl (0.17029-1) ...
#8 42.12 Setting up git (1:2.25.1-1ubuntu3.11) ...
#8 42.17 Processing triggers for libc-bin (2.31-0ubuntu9.14) ...
#8 42.26 Processing triggers for systemd (245.4-4ubuntu3.22) ...
#8 42.27 Processing triggers for mime-support (3.64ubuntu1) ...
#8 DONE 42.5s
#9 [4/4] RUN echo "ulimits: $(ulimit -Sn):$(ulimit -Hn)"; service docker start; rm -rf /var/cache/apt;
#9 0.822 ulimits: 1048576:1048576
#9 0.857 /etc/init.d/docker: 62: ulimit: error setting limit (Invalid argument)
#9 DONE 1.0s
#10 exporting to image
#10 exporting layers
#10 exporting layers 14.3s done
#10 writing image sha256:f785f12a2297b42af040deca8ca043e025 |
I think your build effectively happened to be "lucky" and managed to start because it didn't have to adjust the Also trying what works and what doesn't; (also see setrlimit(2), and ulimit
docker run --rm --ulimit nofile=1024:524288 ubuntu:20.04 sh -c ' echo "before: $(ulimit -Hn):$(ulimit -Sn)"; ulimit -Sn 2048; ulimit -Hn 1048576; echo "after: $(ulimit -Hn):$(ulimit -Sn)"'
sh: 1: ulimit: error setting limit (Operation not permitted)
before: 524288:1024
after: 524288:2048
docker run --rm --ulimit nofile=2048:1048576 ubuntu:20.04 sh -c ' echo "before: $(ulimit -Hn):$(ulimit -Sn)"; ulimit -Sn 1024; ulimit -Hn 524288; echo "after: $(ulimit -Hn):$(ulimit -Sn)"'
before: 1048576:2048
after: 524288:1024 Keeping both the same also works; docker run --rm --ulimit nofile=1024:524288 ubuntu:20.04 sh -c ' echo "before: $(ulimit -Hn):$(ulimit -Sn)"; ulimit -Sn 1024; ulimit -Hn 524288; echo "after: $(ulimit -Hn):$(ulimit -Sn)"'
before: 524288:1024
after: 524288:1024 Note that these ulimits are configured for the Given that you're trying to run the docker engine in a container and ulimits can be set for the container at runtime (using the sed -i 's/ulimit -Hn/# ulimit -Hn/g' /etc/init.d/docker; Here's my Dockerfile with that applied; # syntax=docker/dockerfile:1
FROM ubuntu:20.04
RUN apt-get update -y; \
apt-get install -y \
ca-certificates \
curl \
gnupg \
lsb-release \
software-properties-common; \
rm -rf /var/cache/apt;
RUN curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg; \
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null; \
apt-get update -y; \
apt-get install -y docker-ce; \
rm -rf /var/cache/apt;
RUN \
echo "ulimits: $(ulimit -Sn):$(ulimit -Hn)"; \
sed -i 's/ulimit -Hn/# ulimit -Hn/g' /etc/init.d/docker; \
service docker start; \
rm -rf /var/cache/apt;
ENTRYPOINT service docker start && /bin/bash With that change, the build succeeds succesfuly;
|
Context and fix are mentioned here docker/cli#4807 (comment) Looks like this was only impacting new image builds, which pulled very recent docker init.d scripts and failed to call new `ulimit` Signed-off-by: Amine Hilaly <[email protected]>
Context and fix are mentioned here docker/cli#4807 (comment) Looks like this was only impacting new image builds, which pulled very recent docker init.d scripts and failed to call new `ulimit`. Signed-off-by: Amine Hilaly <[email protected]> By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
docker/cli#4807 As of 02/01/2024 a change in the "/etc/init.d/docker" script shipped with docker v25 is preventing the cpi image to work. when start-bosh runs `service docker start` it errors with: "/etc/init.d/docker: 62: ulimit: error setting limit (Invalid argument)" Workaround recommended in github issue suggests to disable resetting ulimit. Pre v25 the script contained `ulimit -n 1048576`, with v25 it changed to: `ulimit -Hn 524288`. the default in our base image is: ulimit -a | grep file `open files (-n) 1048576` so it was a noOp before the change shipped with v25 of docker to confirm I tried running `ulimit -Hn 1048576` and it will succeed.. The issue happens when we want to actually change the ulimit. In our case it would attempt to lower it and it's probably fine not doing that in CI?
During review, it was decided to remove `LimitNOFILE` from `docker.service` to rely on the systemd v240 implicit default of `1024:524288`. On supported platforms with systemd prior to v240, packagers will patch the service with an explicit `LimitNOFILE=1024:524288`. - `1024` soft limit is an implicit default, avoiding unexpected breakage. Software that needs a higher limit should request to raise the soft limit for its process. - `524288` hard limit is an implicit default since systemd v240 and is adequate for most processes (_half of the historical limit from `fs.nr_open` of `1048576`_), while 4096 is the implicit default from the kernel (often too low). Individual containers can be started with `--ulimit` when a larger hard limit is required. - The hard limit may not exceed `fs.nr_open` (_which a value of `infinity` will resolve to_). On most systems with systemd v240 or newer, this will resolve to an excessive size of 2^30 (over 1 billion). - When set to `infinity` (usually as the soft limit) software may experience significantly increased resource usage, resulting in a performance regression or runtime failures that are difficult to troubleshoot. - OpenRC current config approach lacks support for different soft/hard limits being set as it adjusts additional limits and `ulimit` does not support mixed usage of `-H` + `-S`. A soft limit of `524288` is not ideal, but 2^19 is much less overhead than 2^30, whilst a hard limit of 4096 would be problematic for Docker. Signed-off-by: Brennan Kinney <[email protected]>
I got the same error following the installation instructions on |
@eloaf what is the issue you're having? Is that running the docker engine as part of a Dockerfile? For that see my comment above. |
Yes - starting the docker service yields the error checking the system's Its weird I remember having the same issue years ago, then nothing, then I get it again today after installing on ubuntu 25.0.0 |
Yes, it's possible these limits changed over time, and (per my comment above) "changing" the limit only worked if no actual changes were applied (so it being a no-op). Curious; is there a reason to start the |
I mean, I need to start the docker service at some point to perform any builds? (I was working on a node that would get spun up and down didnt persist the installation) |
But this is a docker service inside a container, and the container is part of the Dockerfile / ran during It's still possible to run the resulting image that was built as privileged, but in that case the container must be started with the |
https://docs.docker.com/reference/cli/docker/container/run/#privileged |
Description
Inside of my Dockerfile, which uses ubuntu:20.04, we install docker-ce. It is essential for us since we need to build AWS CDK code in a custom CodeBuild container.
Here's the Dockerfile code (simplified):
The container was built successfully up until two days ago. On further investigation, I have found out that this is due to the Docker Engine upgrade to the 25.0.0 version.
If now we try
docker run -t container
, we get the following output (the same is printed when runningservice docker start
while building the container:service docker start /etc/init.d/docker: 62: ulimit: error setting limit (Invalid argument)
This is due to the line 62 in /etc/init.d/docker file which sets the ulimit hard limit:
ulimit -Hn 524288
Before the most recent 25.0.0 version release, it used to be the following line:
ulimit -n 1048576
When checking the
/etc/security/limits.conf
file inside of the Ubuntu image, I found out that the system hard limit is 100000.If I remove the
service docker start
command from the Dockerfile (both in the RUN and ENTRYPOINT commands), the issue persists.The only way I could make my image run is by hardcoding the previous version of docker-ce:
This has fixed the problem but is still a huge obstacle for us since we are now forced to use the older version of docker-ce and cannot get updates.
I hope this case will be helpful to anyone having the same problem as we did. I also hope a fix will be introduced so we could get the most recent updates on our image.
Reproduce
Expected behavior
We expected the container to work on 25.0.0 version of docker engine in the same way it did on 24.0.7
docker version
docker info
Not accessible since the container couldn't run.
Additional Info
No response
The text was updated successfully, but these errors were encountered: