Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker on Windows hangs building a layer with icu built with vcpkg #40266

Closed
lvoege opened this issue Nov 28, 2019 · 5 comments
Closed

docker on Windows hangs building a layer with icu built with vcpkg #40266

lvoege opened this issue Nov 28, 2019 · 5 comments

Comments

@lvoege
Copy link

lvoege commented Nov 28, 2019

docker build the following Dockerfile on Windows and it will hang. dockerd will be soaking up around 90% CPU on my machine seemingly indefinitely. nothing interesting appears in the logs.

# escape=`
FROM mcr.microsoft.com/windows/servercore:ltsc2019

SHELL ["cmd", "/S", "/C"]

RUN curl -fSLo vs_buildtools.exe https://aka.ms/vs/16/release/vs_buildtools.exe `
    && (vs_buildtools.exe `
            --quiet --wait --norestart --nocache `
            --installPath C:\BuildTools `
            --add Microsoft.VisualStudio.Workload.VCTools `
            --includeRecommended `
        || IF "%ERRORLEVEL%"=="3010" EXIT 0) `
    && del vs_buildtools.exe `
    && powershell -command "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))"

RUN choco install -y git

RUN git clone https://github.com/Microsoft/vcpkg.git `
    && cd vcpkg && bootstrap-vcpkg.bat

RUN cd vcpkg `
    && vcpkg install icu:x64-windows-static `
    && echo THIS WILL DISPLAY AND THEN HANG

CMD [ "cmd" ]

it has something to do specifically with having vcpkg build libicu, in that having it build some other package does not hang.

I've tried docker stable and edge, on windows 10 1903 and 1909. all of them hang.

Steps to reproduce the issue:

  1. docker build
  2. wait until it displays that it's about to hang. about ten to fifteen minutes.
  3. it'll hang.

Describe the results you received:
a hung dockerd

Describe the results you expected:
a successfully built layer

Additional information you deem important (e.g. issue happens only occasionally):
seems ICU specific, somehow.

Output of docker version:

C:\>docker -v
Docker version 19.03.5, build 633a0ea

Output of docker info:

C:\>docker info
Client:
 Debug Mode: false

Server:
 Containers: 3
  Running: 0
  Paused: 0
  Stopped: 3
 Images: 13
 Server Version: 19.03.5
 Storage Driver: windowsfilter
  Windows:
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: inactive
 Default Isolation: hyperv
 Kernel Version: 10.0 18363 (18362.1.amd64fre.19h1_release.190318-1202)
 Operating System: Windows 10 Pro Version 1909 (OS Build 18363.476)
 OSType: windows
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.94GiB
 Name: buildbot-win
 ID: APE3:S52K:2ZOB:SRXF:7AWS:4NLH:N3WC:KZO3:5ZPX:SDXS:SXVQ:PA5W
 Docker Root Dir: C:\ProgramData\Docker
 Debug Mode: true
  File Descriptors: -1
  Goroutines: 63
  System Time: 2019-11-27T19:40:52.6586974-05:00
  EventsListeners: 1
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Additional environment details (AWS, VirtualBox, physical, etc.):
physical

@Simran-B
Copy link

Simran-B commented Mar 3, 2020

Looks like I have a similar issue, but with vcpkg install qt5-base.

Description

Building my image gets stuck right after the successful compilation and installation of Qt (~40min) without finishing this build step (no sign of progress after 120min). docker ps --all --no-trunc --latest shows that the intermediate container stopped at the time it last progressed:

COMMAND                                CREATED        STATUS
"cmd /S /C vcpkg install qt5-base"     3 hours ago    Exited (0) 2 hours ago

No network or disk activity, but dockerd.exe uses ~28% CPU consistently.

Qt5.Dockerfile

# escape=`
FROM mcr.microsoft.com/windows/servercore:ltsc2019 AS BuildTools
COPY InstallBuildTools.cmd C:\TEMP\
ADD https://aka.ms/vscollect.exe C:\TEMP\vscollect.exe
ADD https://aka.ms/vs/15/release/channel C:\TEMP\VisualStudio.chman
ADD https://aka.ms/vs/15/release/vs_buildtools.exe C:\TEMP\vs_buildtools.exe
RUN C:\TEMP\InstallBuildTools.cmd C:\TEMP\vs_buildtools.exe --quiet --wait --norestart --nocache `
    --installPath C:\BuildTools `
    --channelUri C:\TEMP\VisualStudio.chman `
    --installChannelUri C:\TEMP\VisualStudio.chman `
    --add Microsoft.VisualStudio.Workload.VCTools `
    --includeRecommended
RUN powershell -NoProfile -Command `
    Invoke-Expression ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
RUN choco install -y git --params "/NoShellIntegration"

FROM BuildTools AS OliveDeps
ARG CMAKE_VER=3.20.1
ADD https://github.com/Kitware/CMake/releases/download/v${CMAKE_VER}/cmake-${CMAKE_VER}-windows-x86_64.zip cmake-${CMAKE_VER}-windows-x86_64.zip
RUN powershell -NoProfile -Command `
    Expand-Archive cmake-%CMAKE_VER%-windows-x84_64.zip C:\Tools\; `
    Remove-Item -Force cmake-%CMAKE_VER%-windows-x86_64.zip
WORKDIR C:\Tools\
RUN git clone https://github.com/Microsoft/vcpkg.git
WORKDIR vcpkg
RUN .\bootstrap-vcpkg.bat
RUN setx PATH "C:\Tools\cmake-%CMAKE_VER%-windows-x86_64\bin;C:\Tools\vcpkg;%PATH%"
RUN vcpkg install qt5-base:x64-windows

FROM OliveDeps AS Olive
VOLUME C:\olive\
ENTRYPOINT C:\BuildTools\Common7\Tools\VsDevCmd.bat &&
CMD powershell -NoLogo -ExecutionPolicy Bypass

InstallBuildTools.cmd

@if not defined _echo echo off
setlocal enabledelayedexpansion

call %*
if "%ERRORLEVEL%"=="3010" (
    exit /b 0
) else (
    if not "%ERRORLEVEL%"=="0" (
        set ERR=%ERRORLEVEL%
        call C:\TEMP\vscollect.exe -zip:C:\vslogs.zip

        exit /b !ERR!
    )
)

Steps to reproduce the issue:

  1. Create Qt5.Dockerfile and InstallBuildTools.cmd (see above)
  2. docker build -t qt5 -m 4g -f Qt5.Dockerfile .

Describe the results you received:

The last messages it printed:

Building package qt5-base[core]:x64-windows... done
Installing package qt5-base[core]:x64-windows...
Installing package qt5-base[core]:x64-windows... done
Elapsed time for package qt5-base:x64-windows: 22.88 min

Total elapsed time: 41.41 min

The package qt5-base:x64-windows provides CMake targets:

    find_package(Qt5Concurrent CONFIG REQUIRED)
    target_link_libraries(main PRIVATE Qt5::Concurrent Qt5::ConcurrentPrivate)

    find_package(Qt5Core CONFIG REQUIRED)
    target_link_libraries(main PRIVATE Qt5::Core Qt5::WinMain Qt5::CorePrivate)

    find_package(Qt5DBus CONFIG REQUIRED)
    target_link_libraries(main PRIVATE Qt5::DBus Qt5::DBusPrivate)

    find_package(Qt5Gui CONFIG REQUIRED)
    # Note: 7 target(s) were omitted.
    target_link_libraries(main PRIVATE Qt5::Gui Qt5::GuiPrivate Qt5::QGifPlugin Qt5::QICOPlugin)

    find_package(Qt5Network CONFIG REQUIRED)
    target_link_libraries(main PRIVATE Qt5::Network Qt5::NetworkPrivate Qt5::QGenericEnginePlugin)

    find_package(Qt5OpenGL CONFIG REQUIRED)
    target_link_libraries(main PRIVATE Qt5::OpenGL Qt5::OpenGLPrivate)

    find_package(Qt5OpenGLExtensions CONFIG REQUIRED)
    target_link_libraries(main PRIVATE Qt5::OpenGLExtensions Qt5::OpenGLExtensionsPrivate)

    find_package(Qt5PrintSupport CONFIG REQUIRED)
    target_link_libraries(main PRIVATE Qt5::PrintSupport Qt5::PrintSupportPrivate Qt5::QWindowsPrinterSupportPlugin)

    find_package(Qt5Sql CONFIG REQUIRED)
    # Note: 1 target(s) were omitted.
    target_link_libraries(main PRIVATE Qt5::Sql Qt5::SqlPrivate Qt5::QODBCDriverPlugin Qt5::QPSQLDriverPlugin)

    find_package(Qt5Test CONFIG REQUIRED)
    target_link_libraries(main PRIVATE Qt5::Test Qt5::TestPrivate)

    find_package(Qt5Widgets CONFIG REQUIRED)
    target_link_libraries(main PRIVATE Qt5::Widgets Qt5::WidgetsPrivate Qt5::QWindowsVistaStylePlugin)

    find_package(Qt5Xml CONFIG REQUIRED)
    target_link_libraries(main PRIVATE Qt5::Xml Qt5::XmlPrivate)

Describe the results you expected:

Complete the build step RUN vcpkg install qt5-base:x64-windows (in reasonable time) and finalize the image building.

Additional information you deem important (e.g. issue happens only occasionally):

Hung up in this step 3 out of 3 times so far.

Possibly related:

Diagnostics ID: D5969AF1-0D82-4FFC-BCD8-E933C4D8478A/20200303223254

Output of docker version:

Docker version 19.03.5, build 633a0ea

Output of docker info:

Client:
 Debug Mode: false
 Plugins:
  app: Docker Application (Docker Inc., v0.8.0)
  buildx: Build with BuildKit (Docker Inc., v0.3.1-tp-docker)

Server:
 Containers: 71
  Running: 1
  Paused: 0
  Stopped: 70
 Images: 172
 Server Version: 19.03.5
 Storage Driver: windowsfilter (windows) lcow (linux)
  Windows: 
  LCOW: 
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: inactive
 Default Isolation: hyperv
 Kernel Version: 10.0 18362 (18362.1.amd64fre.19h1_release.190318-1202)
 Operating System: Windows 10 Pro Version 1903 (OS Build 18362.657)
 OSType: windows
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.87GiB
 Name: DESKTOP-8K4LQ8Q
 ID: 2WEP:7E3F:S45I:ADI3:QTVV:SBCZ:AUYU:AQCO:NUKL:DEQV:F5XX:IMWD
 Docker Root Dir: C:\ProgramData\Docker
 Debug Mode: true
  File Descriptors: -1
  Goroutines: 65
  System Time: 2020-03-04T00:39:19.3292655+01:00
  EventsListeners: 1
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Additional environment details (AWS, VirtualBox, physical, etc.):

90 GB free disk space (SSD).

daemon.json

{
  "registry-mirrors": [],
  "insecure-registries": [],
  "debug": true,
  "experimental": true
}

@larshg
Copy link

larshg commented Jul 20, 2020

I have also experienced this now and stuck in making a build image.

I saw in logviewer, that it succeeded in saving a state, ie. after ICU is compiled. The docker instance is stopped and it just hangs there, not starting the next RUN instruction.

Log viewer entry

debug: hcsshim::ExportLayer - succeeded [path=C:\ProgramData\Docker\windowsfilter\65de430792517052c65ce290e083b9d2998c914fcf44585778b75fe33100930d exportFolderPath=C:\ProgramData\Docker\tmp\hcs809210632]

What can I do to debug this?

I have most recently tried with versions below, but also the current stable release. Both get stuck.
I did have it working back in April, where I initially started working with a windows docker image.

Dianostics ID: E639E25F-7E9A-4B35-AB07-5D833C390C1E/20200720213720

Output of Docker version
Client: Docker Engine - Community
Version: 19.03.8
API version: 1.40
Go version: go1.12.17
Git commit: afacb8b
Built: Wed Mar 11 01:23:10 2020
OS/Arch: windows/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 19.03.8
API version: 1.40 (minimum version 1.24)
Go version: go1.12.17
Git commit: afacb8b
Built: Wed Mar 11 01:37:20 2020
OS/Arch: windows/amd64
Experimental: true

Output of Docker Info
Client:
Debug Mode: false

Server:
Containers: 2
Running: 1
Paused: 0
Stopped: 1
Images: 8
Server Version: 19.03.8
Storage Driver: lcow (linux) windowsfilter (windows)
LCOW:
Windows:
Logging Driver: json-file
Plugins:
Volume: local
Network: ics internal l2bridge l2tunnel nat null overlay private transparent
Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
Swarm: inactive
Default Isolation: hyperv
Kernel Version: 10.0 19041 (19041.1.amd64fre.vb_release.191206-1406)
Operating System: Windows 10 Pro Version 2004 (OS Build 19041.388)
OSType: windows
Architecture: x86_64
CPUs: 8
Total Memory: 15.82GiB
Name: LG
ID: HWBY:Y2F6:ZG6T:ITXU:LJHI:MU2T:IHPU:7WSK:DN6C:YSQZ:Z6XJ:Y3J4
Docker Root Dir: C:\ProgramData\Docker
Debug Mode: true
File Descriptors: -1
Goroutines: 56
System Time: 2020-07-20T23:26:08.5032085+02:00
EventsListeners: 2
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

@lvoege
Copy link
Author

lvoege commented Apr 20, 2021

I don't know what did it but latest version of Docker Desktop, with engine version 20.10.5, running on Windows 10.0.21359 appears to no longer have this issue.

@Simran-B
Copy link

I tried to build my Qt5.Dockerfile again with Docker Engine v20.10.5 on Windows 10 Pro 2004 Build 19041. It took about 60 minutes to install and build everything, then there was a period of 20 minutes with high disk activity (maxing out IOPS of my SSD). Building the image completed successfully almost instantly after that. The problem appears to be fixed indeed.

@lvoege
Copy link
Author

lvoege commented Apr 20, 2021

right, closing this then

@lvoege lvoege closed this as completed Apr 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants