WCOW: writing to a file in root directory fails with Access is denied for ContainerUser - except on Win11

[profnandaa]

First, this is not strictly a buildkit regression but something that I suspect is a coming from the platform side, it's by design.

When you follow the guide at b7d04a docs/windows.md, you get "access denied" error on only WS2022 but the build is successful both on Windows 11 and WS2019. It's the same case too with the classic docker build.

UPDATE: I must have mistaken it, it also fails on WS2019. It only builds successfully on Win11, which could be treated as an exception...

Minimal dockerfile:

FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
RUN echo "hello and goodbye!" > hello.txt

Results:

#5 [2/2] RUN echo "hello and goodbye!" > hello.txt
#5 1.388 Access is denied.
#5 ERROR: process "cmd /S /C echo \"hello and goodbye!\" > hello.txt" did not complete successfully: exit code: 1
------
 > [2/2] RUN echo "hello and goodbye!" > hello.txt:
1.388 Access is denied.

Similar failure with docker build on server SKUs:

PS> docker build -t repro-47 .    
               
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
 ---> e82f76d70808
Step 2/2 : RUN echo "hello and goodbye!" > hello.txt
 ---> Running in aa81df33c5a3
Access is denied.
The command 'cmd /S /C echo "hello and goodbye!" > hello.txt' returned a non-zero code: 1

Workarounds:

Option 1: Write to a file in the inner directories, e.g.

FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
RUN mkdir \sample
RUN echo "hello and goodbye!" > \sample\hello.txt

Option 2: Use a privileged user, since the default user in nanoserver is ContainerUser.

FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
USER ContainerAdministrator
RUN echo "hello and goodbye!" > hello.txt

[profnandaa]

Also see discussion here - #4729

[TBBle]

How does FROM mcr.microsoft.com/windows/nanoserver:ltsc2022 build successfully on Windows Server 2019? That shouldn't work with either process isolation or HyperV isolation, AFAIK.

The fact that the USER line fixes it suggests that the default user has somehow changed in the platform code, which would be annoying and probably break existing images that assume you're ContainerAdministrator at runtime.

[profnandaa]

How does FROM mcr.microsoft.com/windows/nanoserver:ltsc2022 build successfully on Windows Server 2019? That shouldn't work with either process isolation or HyperV isolation, AFAIK.

No, I meant that an equivalent dockerfile, i.e. FROM mcr.microsoft.com/windows/nanoserver:ltsc2019

The fact that the USER line fixes it suggests that the default user has somehow changed in the platform code, which would be annoying and probably break existing images that assume you're ContainerAdministrator at runtime.

@TBBle -- sorry I missed your reply.

I've notice that this fails when writing to C:\, any inner directories (other than secured ones like C:\windows) work successfully.
The following will run successfully:

FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
RUN mkdir \test
RUN echo "Goodbye!" > \test\hello.txt

[TBBle]

Rereading this ticket, is it only failing at ’RUN’? If so, that just seems like the nanoserver image's root ACLs or default user have changed in the LTSC 2022 release... Except Windows 11 should share the same base so I expect it to work the same way.

Anyway, if this is by-design, then yeah, a simple doc-update seems the way to go. I'd add a ’USER’ call, personally.

I'm not clear why the base image defaults to A low-priv user, since I expect any real user of that image would start by installing stuff and doing other admIn things, and either way would need to switch bsck down afterwards.

And for trivial image cases (package a single static binary) then sticking it in C:\ seems a mInimal-surprise option.

[profnandaa]

UPDATE: I must have mistaken it, it also fails on WS2019. It only builds successfully on Win11. And same behavior with classic docker build. Seems Win11 is the exception. I'll try get what's causing the difference in Win11 (or most likely all client SKUs)...

[TBBle]

You are specifying process isolation, right? The immediate jump-out for client SKU-specific behaviour is the default being HyperV isolation in Docker, although on reflection I don't think BuildKit or containerd honour that distinction (which I just realised may have been overlooked in the Docker/containerd/Windows migration work).

That said, I did all my testing on Windows 10 or Windows 11, and I'm sure I was using process isolation so there may indeed be something else going on, or I somehow never tested this trivial Dockerfile.

[profnandaa]

You are specifying process isolation, right? The immediate jump-out for client SKU-specific behaviour is the default being HyperV isolation in Docker, although on reflection I don't think BuildKit or containerd honour that distinction (which I just realised may have been overlooked in the Docker/containerd/Windows migration work).

That's true, buildkit/containerd runs process-isolated on both client and server SKUs.

I also tried to have docker build run process isolated on Win 11, and also builds successfully:

PS> docker build --isolation process -t repro-47 .

Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
 ---> 6323fc27b7ed
Step 2/2 : RUN echo "hello and goodbye!" > hello.txt
 ---> Using cache
 ---> a43480d71304
Successfully built a43480d71304
Successfully tagged repro-47:latest

Let me take this up with the platform team to get some explanation.

For completeness, I've also update the description to include docker build failing on WS2022 -- just to highlight that this is not a regression:

PS> docker build -t repro-47 . 
                  
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
 ---> e82f76d70808
Step 2/2 : RUN echo "hello and goodbye!" > hello.txt
 ---> Running in aa81df33c5a3
Access is denied.
The command 'cmd /S /C echo "hello and goodbye!" > hello.txt' returned a non-zero code: 1

[profnandaa]

Rounding up the discussion. This issue is due to the difference in ACLs for C:\ on client SKUs (e.g. Win11) and on server SKUs (e.g. WS2022).

Here is the modifified dockerfile that gives a better picture. Using servercore so as to use both whoami and icacls:

FROM mcr.microsoft.com/windows/servercore:ltsc2022
USER ContainerUser
RUN icacls C:\\
RUN whoami /groups
RUN echo "hello and goodbye!" > hello.txt

Results on WS2022

PS C:\dev\dockerfiles\repro-4731> docker build --no-cache -t repro-4731 .
Sending build context to Docker daemon  155.6kB
Step 1/5 : FROM mcr.microsoft.com/windows/servercore:ltsc2022
 ---> 020089e377ea
Step 2/5 : USER ContainerUser
 ---> Running in b08631d2019b
 ---> Removed intermediate container b08631d2019b
 ---> 22bc0c90e8cd
Step 3/5 : RUN icacls C:\\
 ---> Running in fadc04fe695c
C:\\ BUILTIN\Administrators:(OI)(CI)(F)
     NT AUTHORITY\SYSTEM:(OI)(CI)(F)
     CREATOR OWNER:(OI)(CI)(IO)(F)
     BUILTIN\Users:(OI)(CI)(RX)
     BUILTIN\Users:(CI)(AD)
     BUILTIN\Users:(CI)(IO)(WD)
     Everyone:(RX)

Successfully processed 1 files; Failed processing 0 files
 ---> Removed intermediate container fadc04fe695c
 ---> 0903edc45f67
Step 4/5 : RUN whoami /groups
 ---> Running in ec8230ee9f52

GROUP INFORMATION
-----------------

Group Name                             Type             SID          Attributes
====================================== ================ ============ ==================================================
Mandatory Label\Medium Mandatory Level Label            S-1-16-8192
Everyone                               Well-known group S-1-1-0      Mandatory group, Enabled by default, Enabled group
BUILTIN\Users                          Alias            S-1-5-32-545 Mandatory group, Enabled by default, Enabled group
NT AUTHORITY\INTERACTIVE               Well-known group S-1-5-4      Mandatory group, Enabled by default, Enabled group
CONSOLE LOGON                          Well-known group S-1-2-1      Mandatory group, Enabled by default, Enabled group
NT AUTHORITY\Authenticated Users       Well-known group S-1-5-11     Mandatory group, Enabled by default, Enabled group
NT AUTHORITY\This Organization         Well-known group S-1-5-15     Mandatory group, Enabled by default, Enabled group
LOCAL                                  Well-known group S-1-2-0      Mandatory group, Enabled by default, Enabled group
                                       Unknown SID type S-1-5-93-0   Mandatory group, Enabled by default, Enabled group
 ---> Removed intermediate container ec8230ee9f52
 ---> e7850f02479b
Step 5/5 : RUN echo "hello and goodbye!" > hello.txt
 ---> Running in 896eab99a06e
Access is denied.
The command 'cmd /S /C echo "hello and goodbye!" > hello.txt' returned a non-zero code: 1

Results on Win11

Sending build context to Docker daemon  154.6kB
Step 1/5 : FROM mcr.microsoft.com/windows/servercore:ltsc2022
 ---> e64ba0f4256b
Step 2/5 : USER ContainerUser
 ---> Running in 628b901f7b21
 ---> Removed intermediate container 628b901f7b21
 ---> 6f8a9167c41f
Step 3/5 : RUN icacls C:\\
 ---> Running in bf06475451f1
C:\\ BUILTIN\Administrators:(F)
     BUILTIN\Administrators:(OI)(CI)(IO)(F)
     NT AUTHORITY\SYSTEM:(F)
     NT AUTHORITY\SYSTEM:(OI)(CI)(IO)(F)
     NT AUTHORITY\Authenticated Users:(M)
     NT AUTHORITY\Authenticated Users:(OI)(CI)(IO)(M)
     BUILTIN\Users:(RX)
     BUILTIN\Users:(OI)(CI)(IO)(GR,GE)

Successfully processed 1 files; Failed processing 0 files
 ---> Removed intermediate container bf06475451f1
 ---> 2ddc76a619a9
Step 4/5 : RUN whoami /groups
 ---> Running in 85b9b8fd10d8

GROUP INFORMATION
-----------------

Group Name                             Type             SID          Attributes
====================================== ================ ============ ==================================================
Mandatory Label\Medium Mandatory Level Label            S-1-16-8192
Everyone                               Well-known group S-1-1-0      Mandatory group, Enabled by default, Enabled group
BUILTIN\Users                          Alias            S-1-5-32-545 Mandatory group, Enabled by default, Enabled group
NT AUTHORITY\INTERACTIVE               Well-known group S-1-5-4      Mandatory group, Enabled by default, Enabled group
CONSOLE LOGON                          Well-known group S-1-2-1      Mandatory group, Enabled by default, Enabled group
NT AUTHORITY\Authenticated Users       Well-known group S-1-5-11     Mandatory group, Enabled by default, Enabled group
NT AUTHORITY\This Organization         Well-known group S-1-5-15     Mandatory group, Enabled by default, Enabled group
LOCAL                                  Well-known group S-1-2-0      Mandatory group, Enabled by default, Enabled group
                                       Unknown SID type S-1-5-93-0   Mandatory group, Enabled by default, Enabled group
 ---> Removed intermediate container 85b9b8fd10d8
 ---> 801737b192db
Step 5/5 : RUN echo "hello and goodbye!" > hello.txt
 ---> Running in 9c28c00b887d
 ---> Removed intermediate container 9c28c00b887d
 ---> fbf394e4a1e2
Successfully built fbf394e4a1e2
Successfully tagged repro-4731:latest

Analysis and Conclusion

Worth noting:

// on WS2022, ContainerUser belongs to `BUILTIN\Users` which has:
BUILTIN\Users:(CI)(IO)(WD)

// on Win11, ContainerUser also belongs to `NT AUTHORITY\Authenticated Users` which has:
NT AUTHORITY\Authenticated Users:(OI)(CI)(IO)(M)

• (M) is a more powerful permission that is equivalent to GR + GW + GE (global read, write, execute).
• (WD) is not a powerful-enough permission to be able to write to root directory C:\, hence the access denied failure seen.

[profnandaa]

Will therefore close this issue as by-design after updating the documentation.

[TBBle]

If I'm reading the output correctly, in both cases ContainerUser belongs to the same groups (both show NT AUTHORITY\Authenticated Users), the difference is actually that the CACLs list on Windows Server doesn't have an Authenticated Users entry granting (M) as you note.

Anyway, I guess for me the surprising thing is why does the host OS affect the CACLs list of the root of the image? Presumably this is being inherited from outside the container (i.e. I imagine the differing CACLs match host behaviour), but I don't actually understand how that would happen: we're not actually mounting the container image in the host anymore, right?

Or this is some underlying behaviour of Windows that adds certain CACLs to any drive root on top of whatever's in the data, and it's different in Client and Server SKUs?

[profnandaa]

@TBBle

Anyway, I guess for me the surprising thing is why does the host OS affect the CACLs list of the root of the image? Presumably this is being inherited from outside the container (i.e. I imagine the differing CACLs match host behaviour), but I don't actually understand how that would happen: we're not actually mounting the container image in the host anymore, right?

You are right, the permissions are being inherited. Will notice the permissions with CI (Container Inherit) are passed down to the container. This is what I get when I run the icacls C:\ on the hosts:

// on Win11 host
C:\ S-1-15-3-xxx-full-sid :(S,RD,X,RA)
     NT AUTHORITY\Authenticated Users:(AD)
     NT AUTHORITY\Authenticated Users:(OI)(CI)(IO)(M)
     NT AUTHORITY\SYSTEM:(OI)(CI)(F)
     BUILTIN\Administrators:(OI)(CI)(F)
     BUILTIN\Users:(OI)(CI)(RX)
     Mandatory Label\High Mandatory Level:(OI)(NP)(IO)(NW)

// on WS2022 host
C:\ NT AUTHORITY\SYSTEM:(OI)(CI)(F)
    BUILTIN\Administrators:(OI)(CI)(F)
    BUILTIN\Users:(OI)(CI)(RX)
    BUILTIN\Users:(CI)(AD)
    BUILTIN\Users:(CI)(IO)(WD)
    CREATOR OWNER:(OI)(CI)(IO)(F)

[TBBle]

This seems like an unexpected gap in the isolation model, it suggests that even though we're mounting our filesystem over in another silo (if I remember correctly) it's still inheriting permissions from somewhere on the host. There's a few other differences that don't seem to be related to the C:\ differences, e.g., the Win11 BUILTIN\Users:(OI)(CI)(IO)(GR,GE) at the core of this issue didn't come from the drive root.

So is it inheriting permissions from somewhere else? Like %TEMP% or something? Somewhere that has hidden the root's BUILTIN\Users:(OI)(CI)(RX) in favour of BUILTIN\Users:(OI)(CI)(IO)(GR,GE).

Also, it seems like the non-inherited permissions are different too. Looking at WS2022, Everyone:(RX) appears to not have been inherited from anywhere, but it doesn't appear in Win11? Conversely for Win11, half the permissions do not appear to be inherited (no I, IO, or NP flag), yet none of them appear in the WS2022 output.

So I'm guessing that there's platform-specific behaviour inserting default DACLs into a drive root; it'd be interesting to check what actual DACLs are encoded in the image data.

Anyway, although this strikes me as unexpected, if I'm an outlier here, that's fine. Otherwise it might be better to track this as a platform issue over in https://github.com/microsoft/Windows-Containers/ since it is presumably the same under Docker etc.

I also don't currently see any practical problem here except that Win11 hosts give container users weaker restrictions here, as a known caveat in the "Client SKU containers are a nice-to-have but YMMV" approach. (If I could think of a security exploit or container escape opportunity here, I'd be more worried, but since I expect this doesn't affect Hyper-V isolation, even that is a watered-down possibility)

[profnandaa]

Looks like it's inheritance plus something else.

You raise valid points here. Let me take this up with the platform team, and also see if we can actively track this under the Microsoft WC repo.

[profnandaa]

opened the tracking issue here - microsoft/Windows-Containers#539