Add Kubernetes monitoring docs

[dedemorton]

Add Kubernetes monitoring documentation to the observability guide.

Preview: https://observability-docs_151.docs-preview.app.elstc.co/guide/en/observability/master/monitor-kubernetes.html

READY FOR REVIEW: Overview, logging, metrics monitoring, and apm sections.

Closes elastic/apm#235.
Closes https://github.com/elastic/obs-dc-team/issues/149.

[apmmachine]

A docs preview will be available soon:
- HTML diff
- Observability guide

[dedemorton]

@ChrsMark Just a heads-up that I will be pushing in-progress edits to the branch. When I'm done with my changes and additions, I'll put this PR in review state and let everyone know that it's ready.

[dedemorton]

Hey everyone. Just an update that I'm working on the Fleet docs for 7.10 right now, so this PR is a bit stalled. I might not have the K8s guide ready for 7.10 GA, but since the content is not tied to a specific release, I think that's OK.

I've made quite a few changes to the content, but still need to run through the steps and write an overview.

I apologize for bouncing back and forth. For the next guide, I'll try to box off some dedicated time so I don't have to bounce back and forth so much. The context shifting makes this harder!

[ChrsMark]

Also related to elastic/beats#22516

[bmorelli25]

@elasticmachine, run elasticsearch-ci/docs

[bmorelli25]

Per @eyalkoren's recommendation, I've added no-code instrumentation instructions for the .NET agent in 81a9ed6.

The approach I've used is similar to Java's, except this init container uses a busybox image to wget the dotnet agent, unzip it, and save it to a shared volume. DOTNET_STARTUP_HOOKS then references the ElasticApmAgentStartupHook.dll file when the application container starts. I've tested this and have it working locally. My working manifest is here if anyone wants to try it or offer feedback/improvements: https://gist.github.com/bmorelli25/a8e31252f218e3db85d1ba72640b8a46.

cc @elastic/apm-agent-net

edit: and a link to the preview (will take ~30 mins from this comment's timestamp to update) https://observability-docs_151.docs-preview.app.elstc.co/guide/en/observability/master/monitor-kubernetes.html#_step_3_install_and_configure_apm_agents

[eyalkoren]

Who are you @bmorelli25 ?? Nice!! 👏

I thought about doing a similar thing in Java and get the agent jar from maven, definitely simpler.
One advantage I think there is for using a Docker container is that it is native to k8s, meaning - you are guaranteed to have an accessible docker registry to pull images from, as opposed to downloading files from external resources that may be restricted in some environments. I don't know enough to tell whether this is a real issue that worth the effort of publishing containers though.

[bmorelli25]

Thanks, it was a fun project!

One advantage I think there is for using a Docker container is that it is native to k8s, meaning - you are guaranteed to have an accessible docker registry to pull images from, as opposed to downloading files from external resources that may be restricted in some environments. I don't know enough to tell whether this is a real issue that worth the effort of publishing containers though.

That's a good point. And unfortunately, I know less than you when it comes to determining if this is an issue 😄. Hopefully our fearless (tech) leader, @felixbarny can provide input when he returns.

EDIT: I've opened an issue for an official docker image in the apm-agent-dotnet repo: elastic/apm-agent-dotnet#1121

[bmorelli25]

I'll open a follow-up PR to add APM content to the Monitoring architecture section, and to address any feedback to Eyal's .NET agent concern in #151 (comment).

In my third and final review, I present to you a couple of take 'em or leave 'em recommendations. LGTM! 👏

[dedemorton]

Suggested change

//TODO: Add APM goals here

[bmorelli25]

I figured I'd remove these in my follow-up PR, but might as well get rid of 'em now

[felixbarny]

I like the wget approach as it's simple and doesn't require us to publish an official image.

you are guaranteed to have an accessible docker registry to pull images from

For restricted environments, I could see organizations not allowing access to the public Docker registry but hosting their own with a subset of the available images.

In the long run, I think that the primary way to install APM Agent under k8s will be to use the agents embedded in APM Server via Elastic Agent and Fleet (elastic/apm#383 (comment)) with a MutatingAdmissionWebhook (elastic/apm#385).

In the meantime, I really like the init container approach but I think the simple wget way of downloading the agent binaries is enough. I doubt that complicating the release process for more agents is going to be worth it.

[eyalkoren]

For restricted environments, I could see organizations not allowing access to the public Docker registry but hosting their own with a subset of the available images.

Yes, I could see that as well. However, they do use a registry, so they can fetch our formal image and push it there, instead of building one. I assume it is a bit of a better experience.
But I agree, it needs to be weighed in terms of effort vs. value.