PersistKeysToFileSystem in Docker hosted on Linux fails

[mahic]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

DataProtection set up fails when running an ASP.NET Core 7.0 app in a Docker container on Linux.
Running on latest Docker Desktop with WSL2 integration (busybox in WSL2)

Related issue: dotnet/aspnetcore#2941

I am assuming the original bug was never fixed, and that it was reintroduced by using File.Move with this merge

Expected Behavior

No errors when running on Windows

Example:

info: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[58]
      Creating key {20576f4a-a7a3-4ef3-8bc7-1a29f9e67ab3} with creation date 2022-12-09 17:06:02Z, activation date 2022-12-09 17:06:02Z, and expiration date 2023-03-09 17:06:02Z.
info: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[39]
      Writing data to file 'D:\data\key-20576f4a-a7a3-4ef3-8bc7-1a29f9e67ab3.xml'.

Steps To Reproduce

Simplest repro

Exceptions (if any)

fail: Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider[48]
      An error occurred while reading the key ring.
      System.UnauthorizedAccessException: Access to the path '/data/e85c02c6-df41-4f60-8a1b-acc504e6cf86.tmp' is denied.
       ---> System.IO.IOException: Permission denied
         --- End of inner exception stack trace ---
         at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
         at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode, Func`4 createOpenException)
         at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Func`4 createOpenException)
         at System.IO.FileSystem.CopyFile(String sourceFullPath, String destFullPath, Boolean overwrite)
         at System.IO.FileSystem.LinkOrCopyFile(String sourceFullPath, String destFullPath)
         at System.IO.FileSystem.MoveFile(String sourceFullPath, String destFullPath, Boolean overwrite)
         at System.IO.File.Move(String sourceFileName, String destFileName, Boolean overwrite)
         at Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository.StoreElementCore(XElement element, String filename)
         at Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository.StoreElement(XElement element, String friendlyName)
         at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.IInternalXmlKeyManager.CreateNewKey(Guid keyId, DateTimeOffset creationDate, DateTimeOffset activationDate, DateTimeOffset expirationDate)
         at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.CreateNewKey(DateTimeOffset activationDate, DateTimeOffset expirationDate)
         at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.CreateCacheableKeyRingCore(DateTimeOffset now, IKey keyJustAdded)
         at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRingCore(DateTime utcNow, Boolean forceRefresh)

.NET Version

7.0.100

Anything else?

No response

[Tratcher]

@halter73

[halter73]

Thanks for the solid repro. At the point File.Move throws, the temp file has already been created, so it's not an issue with $TMPDIR but with /data. When you mount the /data directory created by docker volume create hello, it is only writable by the root user.

The aspnet:7.0-jammy-chiseled doesn't include niceties like bash, ls, or whoami, so I switched to the aspnet:7.0-jammy image to see what was going on. But when I switched to the non-"chiseled" image, I no longer see the permissions issue. The app started completely normally.

$ docker run -it --rm -p 5000:80 -v hello:/data repro-dp-bug:latest
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://[::]:80
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app

$ docker exec -it 4790cb0deda7 bash
root@4790cb0deda7:/app# whoami
root
root@4790cb0deda7:/app# ls -Al /data/
total 4
-rw------- 1 root root 1000 Dec  9 19:30 key-aba0c186-e044-4282-97fb-c682781aa212.xml
root@4790cb0deda7:/app# ls -Al /
total 56
-rwxr-xr-x   1 root root    0 Dec  9 19:42 .dockerenv
drwxr-xr-x   1 root root 4096 Dec  9 19:41 app
lrwxrwxrwx   1 root root    7 Nov 30 02:04 bin -> usr/bin
drwxr-xr-x   2 root root 4096 Apr 18  2022 boot
drwxr-xr-x   2 root root 4096 Dec  9 19:30 data
...

The only reason I could figure this would happen is if for some reason the chiseled image runs as a non-root user, so I switched back to using aspnet:7.0-jammy-chiseled AS base, but added -u root to the docker run -it --rm -p 5000:80 -v hello:/data -u root repro-dp-bug:latest and that fixed it!

The next thing I did is search our chiseled Dockerfile definition to see where we're changing the user from the root default unlike our non-chiseled images, and found it here:

dotnet-docker/eng/dockerfile-templates/runtime-deps/Dockerfile.chiseled-ubuntu

Line 4 in 84b3ced

set username to "app" ^

@mthalman It looks like you made the change to switch away from running chiseled container applications as root in #4115. Why does this only affect aspnet:7.0-jammy-chiseled and not aspnet:7.0-jammy?

It seems far more ergonomic to continue running containers applications as root given the way that mounting volumes works by default as this issue demonstrates. I'm moving this to the dotnet-docker repo for follow up.

[mthalman]

It looks like you made the change to switch away from running chiseled container applications as root in #4115. Why does this only affect aspnet:7.0-jammy-chiseled and not aspnet:7.0-jammy?

Yes, the intention with the Chiseled Ubuntu images is to provide a more security-hardened environment. This differs from our other images because of the nature of having a more open system that can be updated. For example, our regular images have a package manager and shell available to allow you to install additional packages. We leave these as the root user to better enable these types of scenarios and to have a better overall experience. But the Chiseled images are much more constrained environment that lacks the tools that you would use to further customize the image and the need for a root user. And because the intent of Chiseled is to have a stripped down image for security purposes, it just makes more sense to continue that further and have a non-root user as the default.

So, if there is going to be volume mounting with the Chiseled container, you have two options:

1. Run the container as a root user to provide access to the mounted directory. This does reduce the benefits of using a Chiseled container, however, since running as root has more privileges in the system than the user we define.
2. Configure the mounted directory on the host to provide access to the same UID as the non-root user in the container:

   dotnet-docker/src/runtime-deps/6.0/jammy-chiseled/amd64/Dockerfile

   Line 18 in 7399ba6

   --uid 101 \

Let us know if you have further questions on configuring this or any thoughts on the overall Chiseled Ubuntu offering.