[19.03.8] Binding folders are bind from MSYS2/MinGW #1509

eine · 2018-01-09T20:27:07Z

Expected behavior

Bind folders work in both PowerShell and MinGW.

Actual behavior

After updating to 17.12.0-ce-win46 (15048) Stable 0ac7325, local folders bind to container volumes are not properly shown if docker is executed from MinGW.

Information

E9FA3323-7ADF-49BB-A59A-73C64DC4AFC1/2018-01-09_21-24-26.
I'm pretty sure it is due to the update, because reverting to Version 17.09.1-ce-win42 (14687) Stable 3176a6a "fixes" it.

Steps to reproduce the behavior

This works (17.12.0):

PS C:\Program Files\Docker\Docker\resources\bin> .\docker.exe run --rm -v d:/test:/test alpine ls -la test
total 5
drwxr-xr-x    2 root     root             0 Jan  9 19:48 .
drwxr-xr-x    1 root     root          4096 Jan  9 20:10 ..
-rwxr-xr-x    1 root     root            20 Jan  9 19:48 testfile.md

This also works (17.09.1):

user@host MINGW64 ~
$ winpty docker run --rm -itv //d/test:/test alpine ls -la test
total 5
drwxr-xr-x    2 root     root             0 Jan  9 19:48 .
drwxr-xr-x    1 root     root          4096 Jan  9 20:31 ..
-rwxr-xr-x    1 root     root            20 Jan  9 19:48 testfile.md

This does not (17.12.0):

user@host MINGW64 ~
$ docker run --rm -v //d/test://test alpine ls -la test
total 4
drwxr-xr-x    2 root     root            40 Jan  9 20:05 .
drwxr-xr-x    1 root     root          4096 Jan  9 20:11 ..

The text was updated successfully, but these errors were encountered:

sgammill · 2018-01-09T21:28:13Z

Your mingw64 docker run commands are specifying the host directory in the Moby VM, not Windows paths. In the Moby VM, Windows drives are not mounted at the root directory anymore for some reason; they're now mounted in /host_mnt . As such, the command you're looking for is:

docker run --rm -v //host_mnt/d/test:/test alpine ls -la test

The double // after -v is an escape sequence so your shell interprets it as a single /.

eine · 2018-01-09T21:42:38Z

Thanks a lot @sgammill. May I know about any reference you might have found related to the change in the mount path inside the Moby VM?

Until now, adding a slash to pwd allowed to use the same scripts in both GNU/Linux and Windows (with MinGW): -v /$(pwd):/test. This change breaks all these scripts!

Might be related to #1284 and moby/moby#35425.

sgammill · 2018-01-09T22:56:27Z

I don't think it's related to LCOW. It's a change they made to the filesystem structure of the Moby VM. I stumbled on this like you did and then peeked inside the Moby VM and found out where the Windows drives were being mounted.

sgammill · 2018-01-09T23:22:45Z

With Windows docker.exe in Linux container mode, Windows-style host paths are interpreted as as paths on the Windows host, whereas Unix-style host paths are interpreted as paths in the Moby VM.

I have the same use case as you. As a work-around, you can abstract pwd in your scripts:

wd() {
  if [ "$OS" = Windows_NT ]; then
    cygpath -m -a .
  else
    pwd
  fi
}

docker run -v $(wd):/test ....

I'm not trying to prevent any RFE for host paths on Windows to work differently. I'm just offering the facts.

eine · 2018-01-09T23:55:00Z

Thank, again, for the workaround. Since I am going to use it in scripts that need to tell Windows and GNU/Linux hosts apart, it is clean enough for me. I will leave this issue opened anyway, so that maintainers can decide if it is worth an enhancement.

gtardif · 2018-01-10T08:30:33Z

Yes, we had to change where the drives are mounted in the linuxKit VM, but we do some translation of the mounted paths. Windows paths work a mentioned above : -v d:/test:/test.
Unix-style paths are translated, but not with a "//" at the beginning. You should use -v /d/test:/test instead of -v //d/test:/test.

eine · 2018-01-10T11:35:51Z

@gtardif , thanks for confirming that you had to change it. Unluckily, due to how MinGW/MSYS2 manipulates the paths, the format you mention (d:/test:/test) does not work. These are all the combinations I tried:

user@host $ docker run --rm -v /d/test:/test alpine ls -R test
ls: test: No such file or directory

user@host $ docker run --rm -v /d/test://test alpine ls -R test
bash: /c/Program Files/Docker/Docker/resources/bin/docker: Bad address

user@host $ docker run --rm -v //d/test:/test alpine ls -R test
test:

user@host $ docker run --rm -v //d/test://test alpine ls -R test
test:

user@host $ docker run --rm -v D:/test:/test alpine ls -R test
C:\Program Files\Docker\Docker\resources\bin\docker.exe: Error response from daemon: invalid mode: \msys64\test.
See 'C:\Program Files\Docker\Docker\resources\bin\docker.exe run --help'.

user@host $ docker run --rm -v D:\test:/test alpine ls -R test
test:

user@host $ docker run --rm -v D:\test://test alpine ls -R test
test:

user@host $ docker run --rm -v 'D:\test':/test alpine ls -R test
test:
testfile.md

user@host $ docker run --rm -v 'D:\test'://test alpine ls -R test
test:
testfile.md

user@host $ docker run --rm -v d:/test:/test alpine ls -R test
C:\Program Files\Docker\Docker\resources\bin\docker.exe: Error response from daemon: invalid mode: \msys64\test.
See 'C:\Program Files\Docker\Docker\resources\bin\docker.exe run --help'.

user@host $ docker run --rm -v 'd:/test':/test alpine ls -R test
C:\Program Files\Docker\Docker\resources\bin\docker.exe: Error response from daemon: invalid mode: \msys64\test.
See 'C:\Program Files\Docker\Docker\resources\bin\docker.exe run --help'.

user@host $ docker run --rm -v d:/test://test alpine ls -R test
bash: /c/Program Files/Docker/Docker/resources/bin/docker: Bad address

user@host $ docker run --rm -v 'd:/test'://test alpine ls -R test
bash: /c/Program Files/Docker/Docker/resources/bin/docker: Bad address

Note that only these two work as expected: 'D:\test':/test and 'D:\test'://test.

Of course, //host_mnt/d/test://test and //host_mnt/d/test:/test work too. But /host_mnt/d/test:/test does not.

sgammill · 2018-01-10T17:51:50Z

@gtardif Thanks for the mention that /c/path/ still works... I don't know why that stopped working for me when the Linuxkit-based VM was introduced (possible error occurred between chair and keyboard), but I can confirm it works now.

So I'll amend my observation: With Windows docker.exe in Linux container mode, Unix-style host paths are interpreted as paths on the Windows host if they're of the form /c/path/... but otherwise are interpreted as paths in the Moby VM (e.g., /var/run/docker.sock).

... This appears to be hold for host paths in --volume and --mount on the docker command line and in volume/mount specifications in compose yml files for docker stack, but host filenames for secrets and configs must still be Windows paths (forward-slashes permitted).

@1138-4eb MSYS2 shell interprets a leading / (but not //) as a Unix path that it will autoconvert to a Windows path on the fly before executing the command. You need to set MSYS2_ARG_CONV_EXCL='*' to disable this conversion before running docker.exe in order to get the path syntax to work.

export MSYS2_ARG_CONV_EXCL='*'
docker run --rm -v d:/test:/test alpine ls -R test
docker run --rm -v 'd:\test':/test alpine ls -R test
docker run --rm -v /d/test:/test alpine ls -R test
docker run --rm -v /host_mnt/d/test:/test alpine ls -R test

These 4 docker commands should all accomplish the same thing.

(Also, FWIW, to get cygpath to print backslashes, use cygpath -w -a . instead of -m)

eine · 2018-01-10T19:12:11Z

@sgamill, according to your comments, I wrote this test script:

opts="/d/test '/d/test' //d/test '//d/test' D:/test 'D:/test' D:\test 'D:\test' d:/test 'd:/test' d:\test 'd:\test'"

echo "Without setting MSYS2_ARG_CONV_EXCL"

for j in 0 1; do
  for i in $opts; do
    echo "$i"
    docker run --rm -v "$i":/test alpine ls -R test
    echo ""
    docker run --rm -v "$i"://test alpine ls -R test
    echo ""
    echo ""
  done
  export MSYS2_ARG_CONV_EXCL='*'
  echo "After setting MSYS2_ARG_CONV_EXCL"
done

Summary of working formats:

Without setting MSYS2_ARG_CONV_EXCL
- D:\test
- d:\test
After setting MSYS2_ARG_CONV_EXCL
- /d/test
- D:/test
- D:\test
- d:/test
- d:\test

It just confirms what you explained. I added it to have a quick test script.

Thanks for the tip about cygpath -w -a ..

eine · 2018-01-12T22:44:26Z

Closing because this has been fixed in Version 17.12.0-ce-win47 (15139) stable 9c692cd.
winpty docker run --rm -itv /$(pwd):/test alpine ls test works again.

eine · 2020-04-03T14:05:40Z

Reopening because there seems to be a regression in 2.2.0.5 (43884) stable:

# winpty docker run --rm -itv $(pwd):/test alpine ls -la //
total 64
drwxr-xr-x    1 root     root          4096 Apr  3 13:56 .
drwxr-xr-x    1 root     root          4096 Apr  3 13:56 ..
-rwxr-xr-x    1 root     root             0 Apr  3 13:56 .dockerenv
drwxrwxrwx    1 root     root             0 Apr  3 13:56 \msys64\test
drwxr-xr-x    2 root     root          4096 Jan 16 21:52 bin
drwxr-xr-x    5 root     root           360 Apr  3 13:56 dev
drwxr-xr-x    1 root     root          4096 Apr  3 13:56 etc
drwxr-xr-x    2 root     root          4096 Jan 16 21:52 home
drwxr-xr-x    5 root     root          4096 Jan 16 21:52 lib
drwxr-xr-x    5 root     root          4096 Jan 16 21:52 media
drwxr-xr-x    2 root     root          4096 Jan 16 21:52 mnt
drwxr-xr-x    2 root     root          4096 Jan 16 21:52 opt
dr-xr-xr-x  128 root     root             0 Apr  3 13:56 proc
drwx------    2 root     root          4096 Jan 16 21:52 root
drwxr-xr-x    2 root     root          4096 Jan 16 21:52 run
drwxr-xr-x    2 root     root          4096 Jan 16 21:52 sbin
drwxr-xr-x    2 root     root          4096 Jan 16 21:52 srv
dr-xr-xr-x   13 root     root             0 Apr  3 13:56 sys
drwxrwxrwt    2 root     root          4096 Jan 16 21:52 tmp
drwxr-xr-x    7 root     root          4096 Jan 16 21:52 usr
drwxr-xr-x   12 root     root          4096 Jan 16 21:52 var

No error is produced.
The volume is created in \msys64\test inside the container, which is NOT correct.
The directory is EMPTY.
This did NOT work before.

# winpty docker run --rm -itv /$(pwd):/test alpine ls -la //test
total 4
drwxr-xr-x    2 root     root            40 Apr  3 13:57 .
drwxr-xr-x    1 root     root          4096 Apr  3 13:57 ..

No error is produced.
The volume is created in the expected location inside the container.
The directory is EMPTY.
This DID work before.

# winpty docker run --rm -itv /$(pwd)://test alpine ls -la //test
total 4
drwxr-xr-x    2 root     root            40 Apr  3 13:57 .
drwxr-xr-x    1 root     root          4096 Apr  3 13:57 ..

No error is produced.
The volume is created in the expected location inside the container.
The directory is EMPTY.
This DID work before.

# winpty docker run --rm -itv $(pwd -W):/test alpine ls -la //
C:/Program Files/Docker/Docker/resources/bin/docker.exe: Error response from daemon: invalid mode: \msys64\test.
See 'C:/Program Files/Docker/Docker/resources/bin/docker.exe run --help'.

An ERROR is produced.

# winpty docker run --rm -itv /$(pwd -W):/test alpine ls -la //test
total 4
drwxrwxrwx    1 root     root          4096 Apr  3 04:52 .
drwxr-xr-x    1 root     root          4096 Apr  3 13:59 ..
-rwxr-xr-x    1 root     root         12513 Mar 31 23:13 .bash_history
-rwxr-xr-x    1 root     root            21 Dec 15  2018 .bash_logout
-rwxr-xr-x    1 root     root          1255 Dec 15  2018 .bash_profile
-rwxr-xr-x    1 root     root          5901 Jan 16 04:08 .bashrc
drwxrwxrwx    1 root     root             0 Jan 23 01:26 .cache
drwxrwxrwx    1 root     root             0 Jan 12 10:39 .config
-rwxr-xr-x    1 root     root           158 Mar  9 18:11 .gitconfig
drwxrwxrwx    1 root     root             0 Jan 12 10:36 .gnupg
-rwxr-xr-x    1 root     root          3272 Dec 15  2018 .inputrc
-rwxr-xr-x    1 root     root          1574 Dec 15  2018 .profile
-rwxr-xr-x    1 root     root            88 Feb 19 15:12 .python_history
drwxrwxrwx    1 root     root          4096 Mar 17 03:16 .ssh
drwxrwxrwx    1 root     root             0 Mar  9 18:10 .vim
-rwxr-xr-x    1 root     root         35507 Apr  1 00:36 .viminfo

# winpty docker run --rm -itv /$(pwd -W)://test alpine ls -la //test
total 4
drwxrwxrwx    1 root     root          4096 Apr  3 04:52 .
drwxr-xr-x    1 root     root          4096 Apr  3 13:59 ..
-rwxr-xr-x    1 root     root         12513 Mar 31 23:13 .bash_history
-rwxr-xr-x    1 root     root            21 Dec 15  2018 .bash_logout
-rwxr-xr-x    1 root     root          1255 Dec 15  2018 .bash_profile
-rwxr-xr-x    1 root     root          5901 Jan 16 04:08 .bashrc
drwxrwxrwx    1 root     root             0 Jan 23 01:26 .cache
drwxrwxrwx    1 root     root             0 Jan 12 10:39 .config
-rwxr-xr-x    1 root     root           158 Mar  9 18:11 .gitconfig
drwxrwxrwx    1 root     root             0 Jan 12 10:36 .gnupg
-rwxr-xr-x    1 root     root          3272 Dec 15  2018 .inputrc
-rwxr-xr-x    1 root     root          1574 Dec 15  2018 .profile
-rwxr-xr-x    1 root     root            88 Feb 19 15:12 .python_history
drwxrwxrwx    1 root     root          4096 Mar 17 03:16 .ssh
drwxrwxrwx    1 root     root             0 Mar  9 18:10 .vim
-rwxr-xr-x    1 root     root         35507 Apr  1 00:36 .viminfo

Both of these:

No error is produced.
The volume is created in the expected location inside the container.
The directory is not empty, which is good.
This (-W) was not required before.

stephen-turner · 2020-04-03T14:22:22Z

I'm not quite following all your test cases, but is it the same as #6209?

eine · 2020-04-04T04:40:38Z

@stephen-turner, it might be but, honestly, I am not sure. Both MSYS2/MINGW and WSL do provide a similar user experience, in the sense that drives are accesible as linux paths (/c, /d, etc. in MSYS2 and /mnt/c, /mnt/d, etc. in WSL). However, AFAIK WSL is not open source, so I don't know whether it is possible to properly understand the underlying issue.

Regarding MSYS2, there are multiple approaches to use it with Docker for Win, which is (should be) unaware of the path conversion:

A solution is to ensure that paths provided to the docker CLI are provided without conversion through MSYS2_ARG_CONV_EXCL, and ensure that windows syntax is used.
Until the latest update, it was also possible to use the default environment and to espace linux paths (//c or //d) to avoid MSYS2 trying to convert them automatically.

With the recent update, for the second case, there seems to be some issue that mounts the shared folder in the VM and in the container, but none of them is actually synced with the host.

It is still possible to provide a windows syntax path directly (as done with pwd -W) and that works). However, that introduces a difference between regular linux shells and MSYS2. Before this latest update, it was possible to use /$(pwd)://src on either Windows, Ubuntu, Fedora...

Note that there was a similar regression two years ago. At that time, the erroneous behaviour was reported in #1509 (comment)

docker-robott · 2020-07-03T01:00:14Z

Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

eine · 2020-08-02T12:32:58Z

/remove-lifecycle stale

eine · 2020-08-02T12:33:50Z

Is it possible to reopen this issue?

docker-robott · 2020-09-01T03:00:10Z

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked

eine changed the title ~~Content not shown when folders are bind from MinGW~~ [17.12.0] Content not shown when folders are bind from MinGW Jan 9, 2018

eine closed this as completed Jan 12, 2018

eine pushed a commit to FreeJaus/docker-images that referenced this issue Jan 12, 2018

revert wd workaround, docker/for-win#1509

1ab8f33

eine mentioned this issue Jul 15, 2018

Support docker-for-win: --vcxsrv, --xwin, --xming... mviereck/x11docker#55

Closed

figaw mentioned this issue Oct 19, 2018

How to use host storage in kubernetes #1703

Closed

eine mentioned this issue Aug 21, 2019

Git Bash + containerized dive (Windows) wagoodman/dive#83

Open

eine changed the title ~~[17.12.0] Content not shown when folders are bind from MinGW~~ [19.03.8] Binding folders are bind from MSYS2/MinGW Apr 3, 2020

eine reopened this Apr 3, 2020

docker-robott added the lifecycle/stale label Jul 3, 2020

docker-robott closed this as completed Aug 2, 2020

docker-robott removed the lifecycle/stale label Aug 2, 2020

docker locked and limited conversation to collaborators Sep 1, 2020

docker-robott added the lifecycle/locked label Sep 1, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[19.03.8] Binding folders are bind from MSYS2/MinGW #1509

[19.03.8] Binding folders are bind from MSYS2/MinGW #1509

eine commented Jan 9, 2018 •

edited

Loading

sgammill commented Jan 9, 2018

eine commented Jan 9, 2018 •

edited

Loading

sgammill commented Jan 9, 2018

sgammill commented Jan 9, 2018 •

edited

Loading

eine commented Jan 9, 2018 •

edited

Loading

gtardif commented Jan 10, 2018

eine commented Jan 10, 2018

sgammill commented Jan 10, 2018

eine commented Jan 10, 2018

eine commented Jan 12, 2018

eine commented Apr 3, 2020

stephen-turner commented Apr 3, 2020

eine commented Apr 4, 2020 •

edited

Loading

docker-robott commented Jul 3, 2020

eine commented Aug 2, 2020

eine commented Aug 2, 2020

docker-robott commented Sep 1, 2020

[19.03.8] Binding folders are bind from MSYS2/MinGW #1509

[19.03.8] Binding folders are bind from MSYS2/MinGW #1509

Comments

eine commented Jan 9, 2018 • edited Loading

Expected behavior

Actual behavior

Information

Steps to reproduce the behavior

sgammill commented Jan 9, 2018

eine commented Jan 9, 2018 • edited Loading

sgammill commented Jan 9, 2018

sgammill commented Jan 9, 2018 • edited Loading

eine commented Jan 9, 2018 • edited Loading

gtardif commented Jan 10, 2018

eine commented Jan 10, 2018

sgammill commented Jan 10, 2018

eine commented Jan 10, 2018

eine commented Jan 12, 2018

eine commented Apr 3, 2020

stephen-turner commented Apr 3, 2020

eine commented Apr 4, 2020 • edited Loading

docker-robott commented Jul 3, 2020

eine commented Aug 2, 2020

eine commented Aug 2, 2020

docker-robott commented Sep 1, 2020

eine commented Jan 9, 2018 •

edited

Loading

eine commented Jan 9, 2018 •

edited

Loading

sgammill commented Jan 9, 2018 •

edited

Loading

eine commented Jan 9, 2018 •

edited

Loading

eine commented Apr 4, 2020 •

edited

Loading