-
Notifications
You must be signed in to change notification settings - Fork 802
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Podman build is very slow compared to docker #1849
Comments
Are you sure you are using fuse-overlayfs? $ podman info |
|
You are using vfs. GraphDriverName: vfs To get to fuse-ovleray, you need to remove all of your storage and config files and start again. $ sudo dnf install -y fuse-overlayfs Now you should be using fuse-overlayfs. |
Hi, |
Fuse-overlay will not be available until RHEL7.8. |
Okay thanks for getting back. Please can you advise how to run the rootfull podman without prefixing sudo before the command podman build, do we have group similar to docker which we can add user to? |
No podman does not have a daemon, and containers are children of podman. So you have to run it as root or non-root. BTW having access to the Docker socket is less secure then running sudo podman. |
I'm using Ubuntu 18.04. It doesn't look like the fuse-overlayfs package is available until Ubuntu 19.04: |
Sorry, not much we can do about that, unless you can get fuse-overlay built for that version of ubuntu. If you run buildah as root, it should be fine. |
We are also using RHEL7. The COPY --chown used to run at "normal" speed when running rootless buildah build until a recent update to RHEL7.7. We have run thousands of rootless builds over the last few months without any issue. We used podman/buildah from rhel-7-server-extras-rpms repository (podman 0.12 / buildah 1.5.2). I am just trying to understand why it used to work... |
Just to let you know that downgrading to buildah-1.5-2.gite94b4f9.el7 makes the builds are fast again on RHEL7.7 I downgraded the whole stack (note sure if it is needed):
|
@jbertozzi as you're on RHEL 7.7, fuse-overlayfs is not in the environment and that's why you hit the speed issues. Downgrading avoids the need for fuse-overlayfs, restricts the usability of rootless containers. @giuseppe anything else? |
I think that this is related to the fact that the
I'm looking for a solution to this problem, ideally we should tar the file by using an exclusion policy (like |
I can confirm that by adding this patch (it is just a workaround that ignores the
And outputs this:
As a workaround, we can maybe implement a deletion after |
@denysvitali thanks for the debugging and snooping. Sounds like you're on the right track. If you'd like to dive further, please assign this issue to yourself. If you don't have time, @QiWang19 PTAL when you return. |
@TomSweeneyRedHat Unfortunately I don't have more time to look into this in depth 😓 |
@denysvitali no worries, thanks again for the investigation. |
#2072 should fix this problem :D |
@denysvitali Thanks. I'll close this one. |
If comparable-or-greater speed is a priority, show us your benchmarks vs. |
Well we know we have issues with COPY and ADD, which we are working on. Other then that, we have lots of strategies to go way faster then Docker for certain workloads. |
It's really too slowly. When I build my image with docker, it took 3 minutes but if with podman, about 1 hour. Too much time is spent on "copying blob ...". My podman info: `host:
|
do you have a reproducer? Could you share your Dockerfile? |
Always. ADD ./myapp /bin/myapp RUN chmod 777 /bin/myapp |
We know we are slow with Huge COPY commands with .dockerignore files. podman 1.4.4 is an ancient version of podman. Podman 1.6 should be available on RHEL7 at this point. |
This hasn't brought a lot of benefit; and by switching to bundles, we need to include the deploy/directory anyway. Also, from containers/buildah#1849, it seems that .dockerignore slows down podman builds.
This hasn't brought a lot of benefit; and by switching to bundles, we need to include the deploy/directory anyway. Also, from containers/buildah#1849, it seems that .dockerignore slows down podman builds.
Is a full rewrite of the code really needed ? apparently it's much faster with older versions of buildah, is there any reason for that ? |
It didn't handle .dockerignore correctly, mainly, and the fix for that was bolted on in a way that slowed things down considerably. That reworking landed in 1.16, though. |
Description
I've previously experienced slowness using
podman build
with images that have labels (#1764). But now it seems building any image is very slow.In case it's relevant, I'm also experiencing an
image not known
error that could be affecting buildah: containers/podman#3982Normally I'd wipe ~/.local/share/containers/storage and compare the results with a clean setup but I didn't want to do that this time in case there's something in there that's needed for the other issue.
Steps to reproduce the issue:
Here's my setup:
Describe the results you received:
podman build
builds images much slower than docker, with or without a cache. I also had high CPU and IO usage.Describe the results you expected:
I would've expected results comparable to docker.
I also found it interesting that the first time I ran
podman build
without--no-cache
, it only seemed to use the cache for one layer. I had to run it a second time for it to use the cache for all layers. Docker doesn't seem to exhibit this behaviour.Output of
rpm -q buildah
orapt list buildah
:Output of
buildah version
:Output of
podman version
if reporting apodman build
issue:Output of
cat /etc/*release
:Output of
uname -a
:Output of
cat /etc/containers/storage.conf
:Thanks!
The text was updated successfully, but these errors were encountered: