Health checks for chiseled containers

[MikaelElkiaer]

Hiya, great work on the chiseled containers, looking forward for them to be GA.

I am using dotnet Docker images for ASP.NET apps deployed in both Kubernetes and simpler Docker-based set-ups.
For Kubernetes, these chiseled images are awesome, so far without any issues.

However, for Docker (docker-compose) based deployments, I would like to utilize the built-in health checks.
Does anyone have any thoughts on how to do built-in health checks, for an ASP.NET application running in a chiseled image?

[mthalman]

How do you currently do health checks and what is it about chiseled containers that makes that approach problematic?

[MikaelElkiaer]

For ASP.NET, I at least have the basic health probe configured:

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddHealthChecks();

var app = builder.Build();

app.MapHealthChecks("/healthz");

app.Run();

Then I will include this simple Docker health check:

HEALTHCHECK CMD curl --fail http://localhost:5000/healthz || exit

Would I then have to extend the chiseled image to include curl, wget, or similar?
Or is there another way?

[mthalman]

Thanks for explaining your scenario. This is a general problem of chiseled/distroless containers. There's a good discussion on this topic at GoogleContainerTools/distroless#183 with various solutions. Personally, I like the idea of having a small custom healthcheck .NET app that would get called to run the HTTP request.

[MikaelElkiaer]

Thanks - good discussion in the issue that you linked.
I ended up doing something similar to this: GoogleContainerTools/distroless#183 (comment)

A simple ./healthchecker/Program.cs:

var httpClient = new HttpClient();
httpClient.DefaultRequestHeaders.ConnectionClose = true;

if (args.Length > 1 && Uri.TryCreate(args[1], UriKind.Absolute, out var uri))
  return await httpClient.GetAsync(uri).ContinueWith(r => r.Result.IsSuccessStatusCode) ? 0 : 1;
else
  throw new ArgumentNullException("A valid URI must be given as first argument");

And a publish of that:

dotnet publish --configuration Release --runtime linux-x64 --no-self-contained healthchecker -o healthchecker/out

And finally the Docker health check:

# ...

COPY ./healthchecker/out /healthchecker

HEALTHCHECK CMD [ "dotnet", "/healthchecker/healthchecker.dll", "--", "http://localhost:8080/healthz" ]

[mthalman]

I created this issue with a proposal of defining a shared tool that could provide this functionality: #4300

[lkroliko]

Hey. I had the same problem. I made nuget package for easy use.
MrRabbit.HealthChecks.Container.Client

[dustinbrooks]

I feel like I'm missing something trying to get the small dotnet healthcheck app working. Since the chiseled container doesn't have a shell, how does it execute the HEALTHCHECK CMD?
I'm utilizing the 8.0-noble-chiseled image. Here is my healthcheck, which I can run this command successfully if I use docker desktop's debug mode to spin up a shell.

HEALTHCHECK CMD dotnet /app/healthcheck.dll http://localhost:8080/Health/Live || exit 1

Here is my docker inspect on the health check:

{ "Status": "unhealthy", "FailingStreak": 12, "Log": [ { "Start": "2024-12-05T14:49:29.714229238Z", "End": "2024-12-05T14:49:29.746231576Z", "ExitCode": -1, "Output": "OCI runtime exec failed: exec failed: unable to start container process: exec: \"/bin/sh\": stat /bin/sh: no such file or directory: unknown" },

To me that reads like it can't run the dotnet command because it can't find the shell to do so.

[mthalman]

The difference lies in the syntax you're using. In the solution above, the JSON array syntax was used for the HEALTHCHECK instruction. This does not require a shell to execute it; rather, Docker executes the command directly. In your command, it uses the shell form syntax which does require a shell in the container to execute. So you would want to use the JSON array syntax instead. You're limited what you can do there, so you can't add the || exit 1 portion as that is shell syntax.

[lbussell]

Yes, here is the Dockerfile documentation on the "shell" and "exec" forms of instructions: https://docs.docker.com/reference/dockerfile/#shell-and-exec-form

[MikaelElkiaer]

Some time ago I made this: https://github.com/MikaelElkiaer/dotnet-healthchecker

It might need some love and care now, but perhaps it is still usable.

[dustinbrooks]

@mthalman @lbussell Thank you. Much appreciated. Changing my format to exec form was exactly what I needed to learn about and use.