Windows, mounting host dirs: Error: statfs /c/tmp/my-build/something: no such file or directory

[Karm]

/kind bug

Description
We would like to use Podman for Windows developer experience, using Linux containers to build our Java applications via Quarkus framework. We are driving the container runtime so as it exposes local project dirs to the container so as the container could operate on those files, e.g. to compile them using the compiler from the container image.

Steps to reproduce the issue:
The following reproducer requires some Java 11+ installed on the Windows machine. It also requires WSL2 backed Podman installed according to this guide: podman-for-windows.md

I am using cmder terminal for my convenience, but the issue is the same with e.g. PowerShell.

C:\tmp
λ curl -O -J  https://code.quarkus.io/d?e=io.quarkus:quarkus-resteasy-reactive
C:\tmp
λ unzip code-with-quarkus.zip
C:\tmp
λ cd code-with-quarkus\
C:\tmp\code-with-quarkus
λ mvnw clean package -Pnative -Dquarkus.native.container-build=true -Dquarkus.native.container-runtime=podman -Dquarkus.native.native-image-xmx=8g
<SNIP for brevity>
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Building native image from C:\tmp\code-with-quarkus\target\code-with-quarkus-1.0.0-SNAPSHOT-native-image-source-jar\code-with-quarkus-1.0.0-SNAPSHOT-runner.jar
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildContainerRunner] Using podman to run the native image builder
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildContainerRunner] Checking image status quay.io/quarkus/ubi-quarkus-native-image:22.0-java11
Trying to pull quay.io/quarkus/ubi-quarkus-native-image:22.0-java11...
Getting image source signatures
Copying blob sha256:c7a9994ad8bb179a74505af79350f5bd3e656603c5f40173564c6281b487351e
Copying blob sha256:c7a9994ad8bb179a74505af79350f5bd3e656603c5f40173564c6281b487351e
Copying blob sha256:54e56e6f85721741ee7bf0336de8ad3bf138a56769a6d0097b600a0e361be58d
Copying blob sha256:54e56e6f85721741ee7bf0336de8ad3bf138a56769a6d0097b600a0e361be58d
Copying blob sha256:4f8ddd7f5a755f537dd9d5f553c8c78171dcf3018c5fc96676a07380d3e14e20
Copying blob sha256:4f8ddd7f5a755f537dd9d5f553c8c78171dcf3018c5fc96676a07380d3e14e20
Copying config sha256:1459002a200043605e9812b13a9609da074e09ce44706588168b10b2f80cc123
Writing manifest to image destination
Storing signatures
1459002a200043605e9812b13a9609da074e09ce44706588168b10b2f80cc123
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on GraalVM 22.0.0.2 Java 11 CE (Java Version 11.0.14+9-jvmci-22.0-b05)
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] podman run --env LANG=C --rm -v //c/tmp/code-with-quarkus/target/code-with-quarkus-1.0.0-SNAPSHOT-native-image-source-jar:/project:z --name build-native-WbneX quay.io/quarkus/ubi-quarkus-native-image:22.0-java11 -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Duser.language=en -J-Duser.country=US -J-Dfile.encoding=UTF-8 -H:-ParseOnce -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED -J--add-opens=java.base/java.text=ALL-UNNAMED -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy\$BySpaceAndTime -H:+JNI -H:+AllowFoldMethods -J-Djava.awt.headless=true -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -J-Xmx8g -H:-AddAllCharsets -H:EnableURLProtocols=http -H:-UseServiceLoaderFeature -H:+StackTrace code-with-quarkus-1.0.0-SNAPSHOT-runner -jar code-with-quarkus-1.0.0-SNAPSHOT-runner.jar
Error: statfs /c/tmp/code-with-quarkus/target/code-with-quarkus-1.0.0-SNAPSHOT-native-image-source-jar: no such file or directory

Describe the results you received:

Error: statfs /c/tmp/code-with-quarkus/target/code-with-quarkus-1.0.0-SNAPSHOT-native-image-source-jar: no such file or directory

Describe the results you expected:
Local project files on the host accessible by the running container. Build should complete normally as it does on a Linux host with podman.

Additional information:
It used to work just fine with Docker Desktop back in the day.

Version:

podman version:   4.1.0
OS Name:             Microsoft Windows 10 Enterprise
OS Version:          10.0.19044 N/A Build 19044
Hotfix(s):               7 Hotfix(s) Installed.
                                  [01]: KB5013624
                                  [02]: KB4562830
                                  [03]: KB5003791
                                  [04]: KB5007401
                                  [05]: KB5013942
                                  [06]: KB5014032
                                  [07]: KB5005699
WSL2 Kernel:        5.10.102.1
Podman machine: podman-machine-default_fedora-35-x86_64

Is there something we could add as an argument to the podman command to make this work on Windows?

[Karm]

FYI @bmaxwell

[Luap99]

@n1hility PTAL

[bmaxwell]

@Karm thanks!
We know that a lot of users run Windows workstations for developing applications. Docker started requiring subscriptions last year, so this would have made Podman more popular :-) Trying to build on Mac has the same issue, which looks to be Podman has issues mounting the local filesystem into the Container. Using sshfs is mentioned as a possible workaround, though Karm mentioned Docker Desktop did not seem to have the issue on Windows.

-v /home/quicklab/quarkus-native/target/quarkus-native-1.0.0-native-image-source-jar:/project:z
#8016 - Podman disscussion

[n1hility]

@bmaxwell @Karm I’m away today but I will take a look when I get a chance. On Windows the issue is probably the extra leading / “//c/tmp” that the native image builder is using. I can look at patching both sides. On Mac as of 4.1 the user home is automatically mounted into the VM for newly created machines, so -v should be working on any path within that

[Karm]

@n1hility Oh, O.K., it seems that was it. Thx.

It seems all right now:

C:\tmp\code-with-quarkus\target\code-with-quarkus-1.0.0-SNAPSHOT-native-image-source-jar
λ  podman run --env LANG=C --rm -v c:/tmp/code-with-quarkus/target/code-with-quarkus-1.0.0-SNAPSHOT-native-image-source-jar:/project:z --name build-native-WbneX quay.io/quarkus/ubi-quarkus-native-image:22.0-java11 -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Duser.language=en -J-Duser.country=US -J-Dfile.encoding=UTF-8 -H:-ParseOnce -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED -J--add-opens=java.base/java.text=ALL-UNNAMED -H:+JNI -H:+AllowFoldMethods -J-Djava.awt.headless=true -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -J-Xmx8g -H:-AddAllCharsets -H:EnableURLProtocols=http -H:-UseServiceLoaderFeature -H:+StackTrace code-with-quarkus-1.0.0-SNAPSHOT-runner -jar code-with-quarkus-1.0.0-SNAPSHOT-runner.jar
========================================================================================================================
GraalVM Native Image: Generating 'code-with-quarkus-1.0.0-SNAPSHOT-runner'...
========================================================================================================================
[1/7] Initializing...                                                                                   (21.1s @ 0.22GB)
 Version info: 'GraalVM 22.0.0.2 Java 11 CE'
 3 user-provided feature(s)
  - io.quarkus.runner.AutoFeature
  - io.quarkus.runtime.graal.DisableLoggingAutoFeature
  - io.quarkus.runtime.graal.ResourcesFeature
[2/7] Performing analysis...  [**********]                                                              (50.6s @ 1.17GB)
   9,305 (93.87%) of  9,913 classes reachable
  13,823 (63.29%) of 21,840 fields reachable
  46,173 (81.60%) of 56,588 methods reachable
     406 classes,     8 fields, and   687 methods registered for reflection
      68 classes,    88 fields, and    54 methods registered for JNI access
[3/7] Building universe...                                                                               (2.1s @ 1.52GB)
[4/7] Parsing methods...      [***]                                                                     (10.5s @ 0.73GB)
[5/7] Inlining methods...     [*****]                                                                    (4.8s @ 2.91GB)
[6/7] Compiling methods...    [*******]                                                                 (52.2s @ 1.48GB)
[7/7] Creating image...                                                                                  (9.2s @ 2.36GB)
  17.53MB (43.21%) for code area:   30,712 compilation units
  18.65MB (45.98%) for image heap:   6,352 classes and 241,580 objects
   4.38MB (10.81%) for other data
  40.56MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 packages in code area:                               Top 10 object types in image heap:
   1.65MB sun.security.ssl                                     6.13MB byte[] for general heap data
 946.03KB java.util                                            2.19MB java.lang.Class
 686.94KB com.sun.crypto.provider                              1.88MB java.lang.String
 490.01KB sun.security.x509                                    1.44MB byte[] for java.lang.String
 422.82KB io.netty.buffer                                    555.47KB java.util.HashMap$Node
 375.23KB java.util.concurrent                               378.84KB java.lang.String[]
 373.01KB java.lang                                          349.53KB com.oracle.svm.core.util.LazyFinalReference
 357.37KB java.io                                            304.69KB java.util.HashMap$Node[]
 353.22KB io.netty.handler.codec.http2                       295.59KB java.util.LinkedHashMap
 350.60KB com.oracle.svm.core.reflect                        282.24KB byte[] for method metadata
      ... 359 additional packages                                 ... 2290 additional object types
                                           (use GraalVM Dashboard to see all)
------------------------------------------------------------------------------------------------------------------------
                        10.6s (6.8% of total time) in 31 GCs | Peak RSS: 4.33GB | CPU load: 3.33
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
 /project/code-with-quarkus-1.0.0-SNAPSHOT-runner (executable)
 /project/code-with-quarkus-1.0.0-SNAPSHOT-runner.build_artifacts.txt
========================================================================================================================
Finished generating 'code-with-quarkus-1.0.0-SNAPSHOT-runner' in 2m 34s.

I will see to it on the Quarkus side then.

[n1hility]

@Karm cool I will look at improving this on the podman side as well, collapse leading slashes into 1 (a single forward slash should work)