Translate Windows-Style path in volume mounting #5458

Elegie · 2019-11-13T22:16:23Z

Rationale

There are two distributions of Docker which can be installed on Windows: the legacy "Docker Toolbox", which runs on Virtualbox, and the more recent "Docker for Windows", which runs on Hyper-V.

The mounting of volumes on the legacy Docker Toolbox requires a certain formatting of windows paths. This PR provides the required path translation.

I could only test on Windows 10 Home Edition, with the Docker Toolbox. I have not made any regression testing on Windows 10 with Docker For Windows, nor Mac or Linux, as I don't own any machine with these environments.

Note that the build still reports an error on the assembly, which seems of no consequence:
[ERROR] OS=Windows and the assembly descriptor contains a *nix-specific root-relative-reference (starting with slash) /

See #5360

Tested locally (Windows 10 Home Edition, with Docker Toolbox).

Install the new plugin:
mvn install

Create a sample projet:
mvn archetype:generate -DarchetypeGroupId=io.quarkus -DarchetypeArtifactId=quarkus-amazon-lambda-archetype -DarchetypeVersion=1.0.0.CR1

Update the pom.xml of the project:
<quarkus.version>999-SNAPSHOT</quarkus.version>

Start the Docker Machine (legacy Docker Toolbox):
docker-machine start

Build:
mvn clean package -Dnative=true -Dnative-image.docker-build=true

Native Build log provided below. There's an error reported by the assembly, but seemingly not relevant.

Deploy locally (based on https://github.com/quarkusio/quarkus/pull/5377/files):
sam local invoke --template sam.native.yml --event payload.json

Native Build log

[INFO] Scanning for projects...
[INFO]
[INFO] --------------------< io.elegie.quarkus:lambdatest >--------------------
[INFO] Building lambdatest 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ lambdatest ---
[INFO] Deleting C:\Users\Elegie\springyme_quarkus\lambdatest\target
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ lambdatest ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ lambdatest ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 5 source files to C:\Users\Elegie\springyme_quarkus\lambdatest\target\classes
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ lambdatest ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory C:\Users\Elegie\springyme_quarkus\lambdatest\src\test\resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ lambdatest ---
[INFO] Changes detected - recompiling the module!
[INFO]
[INFO] --- maven-surefire-plugin:2.22.0:test (default-test) @ lambdatest ---
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ lambdatest ---
[INFO] Building jar: C:\Users\Elegie\springyme_quarkus\lambdatest\target\lambdatest-1.0-SNAPSHOT.jar
[INFO]
[INFO] --- quarkus-maven-plugin:999-SNAPSHOT:build (default) @ lambdatest ---
[INFO] [org.jboss.threads] JBoss Threads version 3.0.0.Final
[INFO] [io.quarkus.deployment.pkg.steps.JarResultBuildStep] Building fat jar: C:\Users\Elegie\springyme_quarkus\lambdatest\target\lambdatest-1.0-SNAPSHOT-runner.jar
[INFO] [io.quarkus.deployment.QuarkusAugmentor] Quarkus augmentation completed in 2217ms
[INFO]
[INFO] --- quarkus-maven-plugin:999-SNAPSHOT:native-image (default) @ lambdatest ---
[INFO] [io.quarkus.deployment.pkg.steps.JarResultBuildStep] Building native image source jar: C:\Users\Elegie\springyme_quarkus\lambdatest\target\lambdatest-1.0-SNAPSHOT-native-image-source-jar\lambdatest-1.0-SNAPSHOT-runner.jar
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Building native image from C:\Users\Elegie\springyme_quarkus\lambdatest\target\lambdatest-1.0-SNAPSHOT-native-image-source-jar\lambdatest-1.0-SNAPSHOT-runner.jar
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on OpenJDK 64-Bit Server VM
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] docker run -v //c/Users/Elegie/springyme_quarkus/lambdatest/target/lambdatest-1.0-SNAPSHOT-native-image-source-jar:/project:z --rm quay.io/quarkus/ubi-quarkus-native-image:19.2.1 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -jar lambdatest-1.0-SNAPSHOT-runner.jar -J-Djava.util.concurrent.ForkJoinPool.common.parallelism=1 -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:EnableURLProtocols=http -H:-JNI -H:-UseServiceLoaderFeature -H:+StackTrace lambdatest-1.0-SNAPSHOT-runner
Build on Server(pid: 22, port: 35519)*
[lambdatest-1.0-SNAPSHOT-runner:22]    classlist:   7,477.13 ms
[lambdatest-1.0-SNAPSHOT-runner:22]        (cap):   2,506.07 ms
[lambdatest-1.0-SNAPSHOT-runner:22]        setup:   5,804.98 ms
21:28:32,744 INFO  [org.jbo.threads] JBoss Threads version 3.0.0.Final
[lambdatest-1.0-SNAPSHOT-runner:22]   (typeflow):  29,296.39 ms
[lambdatest-1.0-SNAPSHOT-runner:22]    (objects):  13,699.08 ms
[lambdatest-1.0-SNAPSHOT-runner:22]   (features):     505.64 ms
[lambdatest-1.0-SNAPSHOT-runner:22]     analysis:  44,208.52 ms
[lambdatest-1.0-SNAPSHOT-runner:22]     (clinit):     656.04 ms
[lambdatest-1.0-SNAPSHOT-runner:22]     universe:   1,831.01 ms
[lambdatest-1.0-SNAPSHOT-runner:22]      (parse):   8,003.72 ms
[lambdatest-1.0-SNAPSHOT-runner:22]     (inline):  10,102.64 ms
[lambdatest-1.0-SNAPSHOT-runner:22]    (compile):  50,126.19 ms
[lambdatest-1.0-SNAPSHOT-runner:22]      compile:  70,029.60 ms
[lambdatest-1.0-SNAPSHOT-runner:22]        image:   3,276.24 ms
[lambdatest-1.0-SNAPSHOT-runner:22]        write:   1,477.43 ms
[lambdatest-1.0-SNAPSHOT-runner:22]      [total]: 134,407.67 ms
[INFO] [io.quarkus.deployment.QuarkusAugmentor] Quarkus augmentation completed in 161453ms
[INFO]
[INFO] --- maven-assembly-plugin:3.1.0:single (zip-assembly) @ lambdatest ---
[INFO] Reading assembly descriptor: src/assembly/zip.xml
[ERROR] OS=Windows and the assembly descriptor contains a *nix-specific root-relative-reference (starting with slash) /
[INFO] Building zip: C:\Users\Elegie\springyme_quarkus\lambdatest\target\function.zip
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:49 min
[INFO] Finished at: 2019-11-13T22:30:32+01:00
[INFO] ------------------------------------------------------------------------

sam.native.yml

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Globals:
  Function:
    Timeout: 5
Resources:
  QuarkusSam:
    Type: AWS::Serverless::Function
    Properties:
      Handler: not.used.in.provided.runtime
      Runtime: provided
      CodeUri: target/function.zip
      MemorySize: 128
      Policies: AWSLambdaBasicExecutionRole
      Timeout: 15
      Environment:
        Variables:
          DISABLE_SIGNAL_HANDLERS: true

Local Invocation log

C:\Users\Elegie\springyme_quarkus\lambdatest>sam local invoke --template sam.native.yml --event payload.json
Invoking not.used.in.provided.runtime (provided)
2019-11-13 22:43:10 Found credentials in shared credentials file: ~/.aws/credentials
Decompressing C:\Users\Elegie\springyme_quarkus\lambdatest\target\function.zip

Fetching lambci/lambda:provided Docker container image......
Mounting C:\Users\Elegie\AppData\Local\Temp\tmp4pkc1cmb as /var/task:ro,delegated inside runtime container
2019-11-13 21:43:12,095 INFO  [io.quarkus] (main) lambdatest 1.0-SNAPSHOT (running on Quarkus 999-SNAPSHOT) started in 0.114s.
2019-11-13 21:43:12,111 INFO  [io.quarkus] (main) Profile prod activated.
2019-11-13 21:43:12,111 INFO  [io.quarkus] (main) Installed features: [amazon-lambda, cdi]
START RequestId: a0225029-cae1-1b62-0c1c-bcde852b6e77 Version: $LATEST
END RequestId: a0225029-cae1-1b62-0c1c-bcde852b6e77
REPORT RequestId: a0225029-cae1-1b62-0c1c-bcde852b6e77  Init Duration: 185.10 ms        Duration: 16.55 ms      Billed Duration: 100 ms                                                                                                     Memory Size: 128 MB      Max Memory Used: 14 MB
{"result":"hello Bill","requestId":"a0225029-cae1-1b62-0c1c-bcde852b6e77"}

machi1990 · 2019-11-14T06:33:40Z

I could only test on Windows 10 Home Edition, with the Docker Toolbox. I have not made any regression testing on Windows 10 with Docker For Windows, nor Mac or Linux, as I don't own any machine with these environments.

Hi @Elegie thanks for the PR. Concerning Linux, we have the setup in our CI which should cover that and since your PR is green we should be safe for Linux. I also won't worry much for Mac, but I will take the liberty to verify it myself later on today.

@stuartwdouglas @jaikiran @gsmet @rsvoboda @maxandersen @cescoffier I do not have a Windows machine, can we get someone with a
Windows with Docker Desktop For Windows to review and try out this PR and see if it works? Thanks

maxandersen · 2019-11-14T16:37:25Z

@machi1990 isn't the issue here on the old legacy docker for windows ? I've not seen any issues like this on the new/current docker desktop...

machi1990 · 2019-11-14T17:55:44Z

@machi1990 isn't the issue here on the old legacy docker for windows ? I've not seen any issues like this on the new/current docker desktop...

Hi @maxandersen thanks for having a look at this one. Yes the issue being fixed is for the Docker Toolbox . We'll have to check that the PR actually fixes the issue for the toolbox and verify that we have not regressed the working Docker Desktop for Windows version. Anything else you would like checking? @Elegie

I am checking on MacOs now.

Elegie · 2019-11-14T18:00:08Z

Hi @machi1990 @maxandersen. I think just building the native image with the maven plugin is fine. I'm sorry I couldn't do it myself, I just don't own the appropriate environments.

gsmet · 2019-11-14T18:31:37Z

core/deployment/src/main/java/io/quarkus/deployment/pkg/steps/NativeImageBuildStep.java

@@ -82,7 +83,7 @@ public NativeImageBuildItem build(NativeConfig nativeConfig, NativeImageSourceJa
            // E.g. "/usr/bin/docker run -v {{PROJECT_DIR}}:/project --rm quarkus/graalvm-native-image"
            nativeImage = new ArrayList<>();
            Collections.addAll(nativeImage, containerRuntime, "run", "-v",
-                    outputDir.toAbsolutePath() + ":/project:z");
+                    FileUtil.translateToUnixStyle(outputDir.toAbsolutePath().toString()) + ":/project:z");


I would be more comfortable if we only did that for Windows, considering we already have platform specific tests below.

+1 doing so we can remove this condition https://github.com/quarkusio/quarkus/pull/5458/files#diff-fa19c024de216cdbcfde070394b65459R71

machi1990 · 2019-11-14T19:31:54Z

Hi @machi1990 @maxandersen. I think just building the native image with the maven plugin is fine. I'm sorry I couldn't do it myself, I just don't own the appropriate environments.

So I compared the two docker run commands for MacOs with your patch and without and there is no diff. Furthermore, I was able to build the jgit module with the patch. 👍

jaikiran · 2019-11-15T04:32:29Z

Hello @machi1990, the Windows system I had borrowed from a friend for testing doesn't have docker installed. So I may not be able to test this out. In general, I agree with what you and @gsmet suggested - making this change only applicable for Windows, so as to reduce the chances of this having any adverse affect on other operating systems.

machi1990 · 2019-11-15T08:39:07Z

Hello @machi1990, the Windows system I had borrowed from a friend for testing doesn't have docker installed. So I may not be able to test this out. In general, I agree with what you and @gsmet suggested - making this change only applicable for Windows, so as to reduce the chances of this having any adverse affect on other operating systems.

Okay thanks for the update.

Elegie · 2019-11-16T10:49:31Z

Hi, thank you for the review.

I have pushed the two requested changes in a new commit, after rebasing the branch on the upstream:

Checking for IS_WINDOWS in NativeImageBuildSteps.
Removing the check for "/" in FileUtil.

I have updated the naming and documentation to reflect the semantics change implied by the "/" condition removal as well.

Do you know if the current plugin actually work on Windows with "Docker for Windows"? If so, what kind of path would be displayed by the console in the volume-mounting part? Besides protecting Unix/Mac, the "/" condition was also a bit of an incantation trying to address this.

…rks in legacy Docker Toolbox.

gsmet · 2019-12-08T10:59:36Z

I just rebased, squashed and force pushed.

gsmet · 2019-12-08T14:13:07Z

Merged, thanks!

Elegie · 2019-12-08T16:54:07Z

Thanks for the discussions, review and merge!

harpreetsingh29 · 2019-12-20T06:42:38Z

@jaikiran , @machi1990 , @Elegie :

I am getting related issue with "docker for windows" and not able escape through this.
Refer : #6259 for details

Not sure if this is tested but do i need to downgrade docker for windows to some lower version? if yes, then what should it be? Current Version is 2.1 and 19.3 engine.

machi1990 requested review from stuartwdouglas, jaikiran and gsmet November 14, 2019 06:33

machi1990 added the area/build label Nov 14, 2019

gastaldi added the env/windows Impacts Windows machines label Nov 14, 2019

gsmet requested changes Nov 14, 2019

View reviewed changes

[5360] Translate Windows-Style path in volume mounting, so that it wo…

b09346c

…rks in legacy Docker Toolbox.

gsmet added the triage/waiting-for-ci Ready to merge when CI successfully finishes label Dec 8, 2019

gsmet added this to the 1.1.0 milestone Dec 8, 2019

gsmet approved these changes Dec 8, 2019

View reviewed changes

gsmet merged commit 4e88d0e into quarkusio:master Dec 8, 2019

gsmet changed the title ~~[5360] Translate Windows-Style path in volume mounting, so that it wo…~~ [5360] Translate Windows-Style path in volume mounting Dec 8, 2019

gsmet changed the title ~~[5360] Translate Windows-Style path in volume mounting~~ Translate Windows-Style path in volume mounting Dec 8, 2019

machi1990 mentioned this pull request Dec 19, 2019

Native Image Generation Failure | NoSuchFileException | Windows 10 #6259

Closed

jaikiran mentioned this pull request Aug 4, 2020

Compilation fails when using reflection-config in quarkus 1.6.x #11184

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Translate Windows-Style path in volume mounting #5458

Translate Windows-Style path in volume mounting #5458

Elegie commented Nov 13, 2019

machi1990 commented Nov 14, 2019

maxandersen commented Nov 14, 2019

machi1990 commented Nov 14, 2019

Elegie commented Nov 14, 2019

gsmet Nov 14, 2019

machi1990 Nov 14, 2019 •

edited

Loading

machi1990 commented Nov 14, 2019

jaikiran commented Nov 15, 2019

machi1990 commented Nov 15, 2019

Elegie commented Nov 16, 2019

gsmet commented Dec 8, 2019

gsmet commented Dec 8, 2019

Elegie commented Dec 8, 2019

harpreetsingh29 commented Dec 20, 2019

Translate Windows-Style path in volume mounting #5458

Translate Windows-Style path in volume mounting #5458

Conversation

Elegie commented Nov 13, 2019

Rationale

Tested locally (Windows 10 Home Edition, with Docker Toolbox).

Native Build log

sam.native.yml

Local Invocation log

machi1990 commented Nov 14, 2019

maxandersen commented Nov 14, 2019

machi1990 commented Nov 14, 2019

Elegie commented Nov 14, 2019

gsmet Nov 14, 2019

Choose a reason for hiding this comment

machi1990 Nov 14, 2019 • edited Loading

Choose a reason for hiding this comment

machi1990 commented Nov 14, 2019

jaikiran commented Nov 15, 2019

machi1990 commented Nov 15, 2019

Elegie commented Nov 16, 2019

gsmet commented Dec 8, 2019

gsmet commented Dec 8, 2019

Elegie commented Dec 8, 2019

harpreetsingh29 commented Dec 20, 2019

machi1990 Nov 14, 2019 •

edited

Loading