Add Kubernetes monitoring docs

docs/en/observability/monitor-k8s/diagnose-k8s-bottlenecks.asciidoc

@@ -0,0 +1,10 @@
+[discrete]
+== Part 6: Diagnose bottlenecks and other issues
+
+[Author: TBD? PM?]
+
+TODO: Describe how to explore a real problem by navigating
+observability UIs and dashboards. This section should showcase the power of
+using our observability solution (being able to correlate logs, metrics, and
+traces to solve a specific, real-world problem). The section title needs to
+match whatever scenario we decide to discuss.

docs/en/observability/monitor-k8s/monitor-k8s-application-performance.asciidoc

@@ -0,0 +1,221 @@
+[discrete]
+[[monitor-kubernetes-application-performance]]
+== Part 3: Monitor application performance
+
+Quickly triage and troubleshoot application performance problems with the help of Elastic
+application performance monitoring (APM).
+
+Think of a latency spike -- APM can help you narrow the scope of your investigation to a single service.
+Because you've also ingested and correlated logs and metrics, you can then link the problem to CPU and memory utilization or error log entries of a particular Kubernetes pod.
+
+[discrete]
+=== Step 1: Set up APM Server
+
+Application monitoring data is streamed from your applications running in Kubernetes to APM Server,
+where it is validated, processed, and transformed into {es} documents.
+
+There are many ways to deploy APM Server when working with Kubernetes,
+but this guide assumes that you're using our hosted {ess} on {ecloud}.
+If you haven't done so already, enable APM Server in the {ess-console}[{ess} console].
+
+If you want to manage APM Server yourself, there are a few alternative options:
+
+[%collapsible]
+.Expand alternatives
+====
+* {eck-ref}/[Elastic Cloud on Kubernetes (ECK)] -- The Elastic recommended approach for managing
+APM Server deployed with Kubernetes.
+Built on the Kubernetes Operator pattern, ECK extends basic Kubernetes orchestration capabilities
+to support the setup and management of APM Server on Kubernetes.
+
+* Deploy APM Server as a DaemonSet -- Ensure a running instance of APM Server on each node in your cluster.
+Useful when all pods in a node should share a single APM Server instance.
+
+* Deploy APM Server as a sidecar -- For environments that should not share an APM Server,
+like when directing traces from multiple applications to separate {es} clusters.
+
+* {apm-server-ref-v}/installing.html[Download and install APM Server] -- The classic, non-Kubernetes option.
+====
+
+[discrete]
+=== Step 2: Save your secret token
+
+A {apm-server-ref-v}/secret-token.html[secret token] is used to secure communication between APM agents
+and APM Server. On the {ecloud} deployment page, select *APM* and copy your APM Server secret token.
+To avoid exposing the secret token, you can store it in a Kubernetes secret. For example:
+
+[source, cmd]
+----
+kubectl create secret generic apm-secret --from-literal=ELASTIC_APM_SECRET_TOKEN=asecretpassword --namespace=kube-system <1>
+----
+<1> Create the secret in the same namespace that you'll be deploying your applications in.
+
+If you're managing APM Server yourself,
+see {apm-server-ref-v}/secret-token.html[secret token] for instructions on how to set up your secret token.
+
+If you are using ECK to set up APM Server, the operator automatically generates an `{APM-server-name}-apm-token` secret for you.
+
+[discrete]
+=== Step 3: Install and configure APM Agents
+
+In most cases, setting up APM agents and thereby instrumenting your applications
+is as easy as installing a library and adding a few lines of code.
+
+Select your application's language for details:
+
+include::{shared}/install-apm-agents-kube/widget.asciidoc[]
+
+[discrete]
+=== Step 4: Configure Kubernetes data
+
+In most instances, APM agents automatically read Kubernetes data from inside the
+container and send it to APM Server.
+If this is not the case, or if you wish to override this data,
+you can set environment variables for the agents to read.
+These environment variable are set via the https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#the-downward-api[Downward API]
+in your kubernetes pod spec:
+
+[source,yml]
+----
+      # ...
+      containers:
+      - name: your-app-container
+        env:
+        # ...
+        - name: KUBERNETES_NODE_NAME
+          valueFrom:
+            fieldRef:
+              fieldPath: spec.nodeName
+        - name: KUBERNETES_POD_NAME
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.name
+        - name: KUBERNETES_NAMESPACE
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.namespace
+        - name: KUBERNETES_POD_UID
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.uid
+----
+
+The table below maps these environment variables to the APM metadata event field:
+
+[options="header"]
+|=====
+|Environment variable |Metadata field name
+|KUBERNETES_NODE_NAME |system.kubernetes.node.name
+|KUBERNETES_POD_NAME |system.kubernetes.pod.name
+|KUBERNETES_NAMESPACE |system.kubernetes.namespace
+|KUBERNETES_POD_UID	|system.kubernetes.pod.uid
+|=====
+
+[discrete]
+=== Step 5: Deploy your application
+
+APM agents are deployed with your application.
+
+[%collapsible]
+.Resource configuration file example
+====
+
+A complete resource configuration file based on the previous steps.
+
+[source,yml]
+----
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: <<your-app>>
+  namespace: kube-system
+  labels:
+    app: <<your-app>>
+    service: <<your-app>>
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: <<your-app>>
+  template:
+    metadata:
+      labels:
+        app: <<your-app>>
+        service: <<your-app>>
+    spec:
+      dnsPolicy: ClusterFirstWithHostNet
+      volumes:
+      - name: elastic-apm-agent
+        emptyDir: {}
+      initContainers:
+      - name: elastic-java-agent
+        image: docker.elastic.co/observability/apm-agent-java:1.12.0
+        volumeMounts:
+        - mountPath: /elastic/apm/agent
+          name: elastic-apm-agent
+        command: ['cp', '-v', '/usr/agent/elastic-apm-agent.jar', '/elastic/apm/agent']
+      containers:
+      - name: <<your-app>>
+        image: <<your-app>>
+        volumeMounts:
+        - mountPath: /elastic/apm/agent
+          name: elastic-apm-agent
+        env:
+        - name: ELASTIC_APM_SERVER_URL
+          value: "apm-server-url-goes-here"
+        - name: ELASTIC_APM_SECRET_TOKEN
+          valueFrom:
+            secretKeyRef:
+              name: apm-secret
+              key: ELASTIC_APM_SECRET_TOKEN
+        - name: ELASTIC_APM_SERVICE_NAME
+          value: "petclinic"
+        - name: ELASTIC_APM_APPLICATION_PACKAGES
+          value: "org.springframework.samples.petclinic"
+        - name: ELASTIC_APM_ENVIRONMENT
+          value: test
+        - name: JAVA_TOOL_OPTIONS
+          value: -javaagent:/elastic/apm/agent/elastic-apm-agent.jar
+        - name: KUBERNETES_NODE_NAME
+          valueFrom:
+            fieldRef:
+              fieldPath: spec.nodeName
+        - name: KUBERNETES_POD_NAME
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.name
+        - name: KUBERNETES_NAMESPACE
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.namespace
+        - name: KUBERNETES_POD_UID
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.uid
+----
+====
+
+[source,cmd]
+----
+kubectl apply -f demo.yml
+----
+
+[discrete]
+=== View your traces in Kibana
+
+To view your application's trace data, open Kibana and go to *Observability* *>* *APM*.
+
+The APM app allows you to monitor your software services and applications in real-time:
+visualize detailed performance information on your services, identify and analyze errors,
+and monitor host-level and agent-specific metrics like JVM and Go runtime metrics.
+
+[screenshot]
+image::images/spring-apm-app-2.png[APM app kubernetes]
+
+Having access to application-level insights with just a few clicks can drastically decrease the time you spend debugging errors, slow response times, and crashes.
+
+Best of all, because Kubernetes environment variables have been mapped to APM metadata events,
+you can filter your trace data by Kubernetes `namespace`, `node.name`, `pod.name`, and `pod.uid`.
+
+[screenshot]
+image::images/apm-app-kubernetes-filter.png[APM app kubernetes]