Enable HEALTHCHECK with SDK Containers

[richlander]

Over at dotnet/dotnet-docker, we're looking into health checks. It started with needing to enable the scenario in distroless containers due to that being hard (installing curl) and now we're realizing that the same is true (for different reasons) with SDK-built containers. We decided that we'd do the feature if we could establish a partnership with the SDK team to enable a full E2E.

Context: dotnet/dotnet-docker#4300

This is what we've come up with so far, with wget and curl (since these tools are available in various places).

Using curl, which comes with Nano Server and is likely the most common tool to use:

HEALTHCHECK CMD curl -sf --show-error http://localhost:80/healthz || exit 1

Using wget, which comes with Alpine images:

HEALTHCHECK CMD wget -qO- -t1 http://localhost:80/healthz || exit 1

That all looks great. The challenge is that these tools are often not available and they also add size and vulnerability risk. A simple framework-dependent tool could serve the same purpose w/o any of those problems.

Instead, the following would be generated for you:

HEALTHCHECK CMD httpclient http://localhost:80/healthz || exit 1

The basic idea is that we'd develop a tool and push to blob storage. Similar to: https://github.com/microsoft/dotnet-framework-docker/blob/9715ee1b646522ea91d62ac1b6009e778d016abf/src/aspnet/4.8.1/windowsservercore-ltsc2022/Dockerfile#L10. The SDK would then expose a property like EnableHealthChecks, which would then install the tool and add the HEALTHCHECK directive.

There is an open question of whether HEALTHCHECK is even useful in 2023. It enables observing health via docker ps and docker inspect.

• docker ps: Add basic HTTP health check dotnet-docker#4352 (comment)
• docker inspect: Add basic HTTP health check dotnet-docker#4352 (comment)

It's most useful for dev (for folks that do dev with Docker) and for appliance style apps like Pi-Hole. The people asking for this seem to be invested in Docker desktop and/or compose.

Pi-hole use case:

• https://github.com/pi-hole/docker-pi-hole/blob/79f96060b6d896a02011a1ad7567c77c6a831378/src/Dockerfile#L39
• https://github.com/pi-hole/docker-pi-hole/blob/master/examples/docker-compose.yml.example

Is the SDK team interested in partnering on this scenario? I think the project is relatively small, however, I think it's also really just nice-to-have. For the SDK team, it certainly would help expand the tech to more scenarios. We're unlikely to do this feature w/o partnering with the SDK team. We want to either enable an E2E or just provide customers with guidance.

Note: the port thing can be a challenge. We'd need to sort that out. Context: dotnet/dotnet-docker#3968.

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[richlander]

@baronfel @rainersigwald

[baronfel]

May want to consider moving this to https://github.com/dotnet/sdk-container-builds, that's the main repo for SDK container development.

This is a great idea - I love the idea of cross-cutting tooling being simple to add to the user's environment. In my past I would have loved something like this for dotnet-dump, for example.

Broadly there are two options here that I see:

manual download of a tool

In this scenario, the appropriate binaries for a tool would be identified and downloaded to the users' machine. Those binaries would be packaged into a layer tarball - either as a single unit, or part of the overall layer tarball that contains all of the users' other application code. That layer would then be treated like any other layer in the system: it would be added to the container manifest and pushed to the destination registry.

managing layer tarballs as blobs

In this scenario, the binaries for a tool would be packaged into a docker-compatible layer tarball and uploaded as a blob to a registry (likely mcr.microsoft.com). These binaries would have a known digest that could be used to reference them. The user (or SDK build targets, or some other process) would detect that the user asked for this healthcheck utility, discover the appropriate layer digest, then add that layer to the generated container manifest and push it up in the same process described above. This scenario is a bit more work, because we haven't defined the mechanism by which you'd define a Layer item by digest reference as an MSBuild Item with metdata, but if we did that it would be a truly general mechanism for this kind of change in the future, and allow for simple extensions that wouldn't have to be part of the 'core' SDK container build tooling.

I'm biased towards the second scenario quite a bit (partly because I don't think it would be that much work to do), but I could see doing a one-off thing until the use case was proven out.

[richlander]

Transfered the issue. Good call.

We could expand this to other tools, like dotnet-dump. That could be very nice.

Interesting. You'll see that we were considering the second option (kinda) in dotnet/dotnet-docker#4300. We're definitely happy to collaborate on both options. I see why the second one might be better.

[richlander]

Any idea when you folks will make a decision on whether this is a scenario you plan to take on in the .NET 8 cycle?

[Kralizek]

I'm very interested in having health check support in SDK containers for better integration with AWS ECS hosting.

[baronfel]

We're not going to lean into healthcheck support in the SDK container tooling itself because this extension is Docker-only and has no support in OCI images, which the container world is moving towards.

[mthalman]

We're not going to lean into healthcheck support in the SDK container tooling itself because this extension is Docker-only and has no support in OCI images, which the container world is moving towards.

This issue is about more than just HEALTHCHECK. It's also about potentially injecting a tool to do the execution of the check, which would still be relevant in K8s scenarios with a probe configuration. Just want to make sure that is considered before closing for good.

[baronfel]

@mthalman to me that sounds a bit like #325 - if you have a layer with the content in some content addressable store, then the tooling should let you inject that layer and incorporate it into the final product. This is how google did debugging tools for JVM apps in the original Jib presentations a few years ago, and I believe Oryx does the same thing when keeping image runtimes green.