diff --git a/tutorials/katacoda/thanos-pathway.json b/tutorials/katacoda/thanos-pathway.json index b0dd9b40bf..5f78305e33 100644 --- a/tutorials/katacoda/thanos-pathway.json +++ b/tutorials/katacoda/thanos-pathway.json @@ -1,6 +1,6 @@ { "title": "Learn Thanos", - "description": "Introduction, Tips and Advanced Tutorials for Thanos: the CNCF, Global, Scalable System for Prometheus Metrics with cheap, Long-term Storage Capabilities. Course version: v0.3", + "description": "Introduction, Tips and Advanced Tutorials for Thanos: the CNCF, Global, Scalable System for Prometheus Metrics with cheap, Long-term Storage Capabilities. Course version: v0.4", "icon": "fa-thanos", "courses": [ { @@ -15,8 +15,13 @@ }, { "course_id": "3-receiver", - "title": "Intermediate: Ingesting metrics data from unreachable sources with Thanos Receive", - "description": "Learn how to ingest and query metrics data from unreachable sources with Thanos Receive." + "title": "Intermediate: Streaming metrics from remote source with Thanos Receive", + "description": "Learn how to ingest and then query metrics data from egress-only sources with Thanos Receive." + }, + { + "course_id": "4-receiver-agent", + "title": "Bonus: Using Prometheus Agent for streaming metrics to Thanos Receive", + "description": "Learn how to use Prometheus in agent mode to efficiently stream metrics out of cluster." }, { "course_id": "6-query-caching", diff --git a/tutorials/katacoda/thanos/3-receiver/finish.md b/tutorials/katacoda/thanos/3-receiver/finish.md index 8a47bcb6f3..88ec7238fa 100644 --- a/tutorials/katacoda/thanos/3-receiver/finish.md +++ b/tutorials/katacoda/thanos/3-receiver/finish.md @@ -1,10 +1,11 @@ # Summary Congratulations! 🎉🎉🎉 -You completed this Thanos Receive tutorial. Let's summarize what we learned: +You completed this `Thanos Receive` tutorial. Let's summarize what we learned: -* Thanos Receive is a component that implements the `Prometheus Remote Write` protocol. -* Prometheus can be configured to remote write its metric data in real-time to another server that implements the Remote Write protocol. +* `Thanos Receive` is a component that implements the `Prometheus Remote Write` protocol. +* Prometheus can be configured to remote-write its metric data in real-time to another server that implements the `Remote Write` protocol. +* Thanos Receive allows us to ingest the data from sources which have limited accessibility, or that has no querying / storing capabilities and they have to forward data somewhere else (e.g Prometheus in Agent mode explained in next tutorial). See next courses for other tutorials about different deployment models and more advanced features of Thanos! diff --git a/tutorials/katacoda/thanos/3-receiver/index.json b/tutorials/katacoda/thanos/3-receiver/index.json index 9d3aaa1992..e9a2608b1e 100644 --- a/tutorials/katacoda/thanos/3-receiver/index.json +++ b/tutorials/katacoda/thanos/3-receiver/index.json @@ -1,6 +1,6 @@ { - "title": "Intermediate: Ingesting metrics data from unreachable sources with Thanos Receive", - "description": "Learn how to ingest and query metrics data from unreachable sources with Thanos Receive.", + "title": "Intermediate: Streaming metrics from remote source with Thanos Receive", + "description": "Learn how to ingest and then query metrics data from egress-only sources with Thanos Receive.", "difficulty": "Moderate", "details": { "steps": [ diff --git a/tutorials/katacoda/thanos/3-receiver/intro.md b/tutorials/katacoda/thanos/3-receiver/intro.md index 3ba44eb65a..428dbd7218 100644 --- a/tutorials/katacoda/thanos/3-receiver/intro.md +++ b/tutorials/katacoda/thanos/3-receiver/intro.md @@ -1,4 +1,4 @@ -# Intermediate: Ingesting metrics data from unreachable sources with Thanos Receive +# Intermediate: Streaming metrics from remote source with Thanos Receive The [Thanos](thanos.io) project defines a set of components that can be composed together into a highly available metric system with **unlimited storage capacity** that **seamlessly** integrates into your existing Prometheus deployments. @@ -6,12 +6,10 @@ In this course you get first-hand experience building and deploying this infrast In this tutorial, you will learn: -* How to ingest metrics data from Prometheus instances that are unreachable from your infrastructure. +* How to ingest metrics data from Prometheus instances without ingress traffic and need to store data locally for longer time. * How to setup a Thanos Querier to access this data. * How Thanos Receive is different from Thanos Sidecar, and when is the right time to use each of them. -This will allow you to setup infrastucture - > NOTE: This course uses docker containers with pre-built Thanos, Prometheus, and Minio Docker images available publicly. ### Prerequisites diff --git a/tutorials/katacoda/thanos/3-receiver/step1.md b/tutorials/katacoda/thanos/3-receiver/step1.md index d20013ea9a..8e0b6a0300 100644 --- a/tutorials/katacoda/thanos/3-receiver/step1.md +++ b/tutorials/katacoda/thanos/3-receiver/step1.md @@ -7,6 +7,7 @@ Let's imagine that you run a company called `Wayne Enterprises`. This company ru However, these sites are special. For security reasons, they do not expose public endpoints to the Prometheus instances running there, and so cannot be accessed directly from other parts of your infrastructure. As the person responsible for implementing monitoring these sites, you have two requirements to meet: + 1. Implement a global view of this data. `Wayne Enterprises` needs to know what is happening in all parts of the company - including secret ones! 1. Global view must be queryable in near real-time. We can't afford any delay in monitoring these locations! @@ -57,9 +58,10 @@ Verify that `prometheus-batcave` is running by navigating to the [Batcave Promet This lets us tell Prometheus to dynamically reload its configuration, which will be useful later in this tutorial. + ### Batcomputer -Almost exactly the same configuration as above, execpt we run the Prometheus instance on port `9091`. +Almost exactly the same configuration as above, except we run the Prometheus instance on port `9091`. 
 global:
diff --git a/tutorials/katacoda/thanos/3-receiver/step2.md b/tutorials/katacoda/thanos/3-receiver/step2.md
index 9f4a1b9ca2..c22b0cd42d 100644
--- a/tutorials/katacoda/thanos/3-receiver/step2.md
+++ b/tutorials/katacoda/thanos/3-receiver/step2.md
@@ -27,11 +27,12 @@ Since we cannot access the `Thanos Sidecar` directly - we cannot query metrics d
 This means that the Global View would be at least 2 hours out of date, and does not satisfy requirement #2.
 
 
+
 ## Thanos Receive
 
 Enter [Thanos Receive](https://thanos.io/tip/components/receive.md/).
 
-`Thanos Receive` is a component that implements the [Prometheus Remote Write API](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write). This means that it will accept metrics data that is sent to it by other Prometheus instances.
+`Thanos Receive` is a component that implements the [Prometheus Remote Write API](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write). This means that it will accept metrics data that is sent to it by other Prometheus instances (or any other process that implements Remote Write API).
 
 Prometheus can be configured to `Remote Write`. This means that Prometheus will send all of its metrics data to a remote endpoint as they are being ingested - useful for our requirements!
 
@@ -58,6 +59,8 @@ docker run -d --rm \
     --remote-write.address 127.0.0.1:10908
 ```{{execute}}
 
+This starts Thanos Receive that listens on `http://127.0.0.1:10908/api/v1/receive' endpoint for Remote Write and on `127.0.0.1:10907` for Thanos StoreAPI.
+
 Let's talk about some important parameters:
 * `--label` - `Thanos Receive` requires at least one label to be set. These are called 'external labels' and are used to uniquely identify this instance of `Thanos Receive`.
 * `--remote-write.address` - This is the address that `Thanos Receive` is listening on for Prometheus' remote write requests.
@@ -91,10 +94,11 @@ Now we are done right? Try querying for some data...
 ![alt text](./assets/receive-empty-query-result.png)
 
 

- Uh-oh! Why are we seeing 'Empty Query Result' responses?
+ Uh-oh! Do you know why are we seeing 'Empty Query Result' responses?
 
 We have correctly configured `Thanos Receive` & `Thanos Query`, but we have not yet configured Prometheus to write to remote write its data to the right place.
 
 

 
+
 Hit continue and we will fix this setup!
\ No newline at end of file
diff --git a/tutorials/katacoda/thanos/3-receiver/step3.md b/tutorials/katacoda/thanos/3-receiver/step3.md
index 094de74d1d..ae3611e3a5 100644
--- a/tutorials/katacoda/thanos/3-receiver/step3.md
+++ b/tutorials/katacoda/thanos/3-receiver/step3.md
@@ -1,6 +1,6 @@
 # Configure Prometheus Remote Write
 
-Our problem in the last step was that we have have not yet configured Prometheus to `remote_write` to our `Thanos Receive` instance.
+Our problem in the last step was that we have not yet configured Prometheus to `remote_write` to our `Thanos Receive` instance.
 
 We need to tell `prometheus-batcave` & `prometheus-batcomputer` where to write their data to.
 
diff --git a/tutorials/katacoda/thanos/4-receiver-agent/assets/expected.png b/tutorials/katacoda/thanos/4-receiver-agent/assets/expected.png
new file mode 100644
index 0000000000..7a60f90667
Binary files /dev/null and b/tutorials/katacoda/thanos/4-receiver-agent/assets/expected.png differ
diff --git a/tutorials/katacoda/thanos/4-receiver-agent/courseBase.sh b/tutorials/katacoda/thanos/4-receiver-agent/courseBase.sh
new file mode 100644
index 0000000000..f9d5f65609
--- /dev/null
+++ b/tutorials/katacoda/thanos/4-receiver-agent/courseBase.sh
@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+
+docker pull quay.io/bwplotka/prometheus:agent1
+docker pull quay.io/thanos/thanos:v0.21.0
+
+mkdir /root/editor
diff --git a/tutorials/katacoda/thanos/4-receiver-agent/finish.md b/tutorials/katacoda/thanos/4-receiver-agent/finish.md
new file mode 100644
index 0000000000..2514410157
--- /dev/null
+++ b/tutorials/katacoda/thanos/4-receiver-agent/finish.md
@@ -0,0 +1,20 @@
+# Summary
+
+Congratulations! 🎉🎉🎉
+You completed this Prometheus Agent tutorial. Let's summarize what we learned:
+
+* Thanos Receive is a component that implements the `Prometheus Remote Write` protocol.
+* Prometheus Agent can be deployed to remote write its metric data in real-time to another server that implements the Remote Write protocol.
+* Prometheus Agent uses a _fraction_ of the resources as normal Prometheus, by not enabling features like: local metrics storage and local query APIs.
+
+See next courses for other tutorials about different deployment models and more advanced features of Thanos!
+
+## Further Reading
+
+TBD
+
+### Feedback
+
+Do you see any bug, typo in the tutorial or you have some feedback for us?
+
+let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io
\ No newline at end of file
diff --git a/tutorials/katacoda/thanos/4-receiver-agent/index.json b/tutorials/katacoda/thanos/4-receiver-agent/index.json
new file mode 100644
index 0000000000..3c9e763143
--- /dev/null
+++ b/tutorials/katacoda/thanos/4-receiver-agent/index.json
@@ -0,0 +1,51 @@
+{
+  "title": "Bonus: Using Prometheus Agent for streaming metrics to Thanos Receive",
+  "description": "Learn how to use Prometheus in agent mode to efficiently stream metrics out of cluster.",
+  "difficulty": "Moderate",
+  "details": {
+    "steps": [
+      {
+        "title": "Problem Statement & Setup",
+        "text": "step1.md",
+        "verify": "step1-verify.sh"
+      },
+      {
+        "title": "Setup Prometheus Agents",
+        "text": "step2.md",
+        "verify": "step2-verify.sh"
+      },
+      {
+        "title": "Verify Setup",
+        "text": "step3.md",
+        "verify": "step3-verify.sh"
+      }
+    ],
+    "intro": {
+      "text": "intro.md",
+      "courseData": "courseBase.sh",
+      "credits": "https://thanos.io"
+    },
+    "finish": {
+      "text": "finish.md",
+      "credits": "test"
+    }
+  },
+  "files": [
+    "prom-agent-batmobile.yaml",
+    "prom-agent-batcopter.yaml"
+  ],
+  "environment": {
+    "uilayout": "editor-terminal",
+    "uisettings": "yaml",
+    "uieditorpath": "/root/editor",
+    "showdashboard": true,
+    "dashboards": [
+      {"name": "Prometheus Agent Batmobile", "port": 9090},
+      {"name": "Prometheus Agent Batcopter", "port": 9091},
+      {"name": "Thanos Query", "port": 39090}
+  ]
+  },
+  "backend": {
+    "imageid": "docker-direct"
+  }
+}
diff --git a/tutorials/katacoda/thanos/4-receiver-agent/intro.md b/tutorials/katacoda/thanos/4-receiver-agent/intro.md
new file mode 100644
index 0000000000..51dc893b8c
--- /dev/null
+++ b/tutorials/katacoda/thanos/4-receiver-agent/intro.md
@@ -0,0 +1,30 @@
+# Intermediate: (Bonus) Using Prometheus Agent for pure forward-only metric streaming with Thanos Receive
+
+The [Thanos](thanos.io) project defines a set of components that can be composed together into a highly available metric system with **unlimited storage capacity** that **seamlessly** integrates into your existing Prometheus deployments.
+
+But [Prometheus](https://prometheus.io/) project is far from slowing down the development. Together with the community, it constantly evolves to bring value to different network and cluster topologies that we see. Thanos brings distributed and cloud storage and querying to the table, built on the core Prometheus code and design. It allowed Prometheus to focus on critical collection and single cluster monitoring functionalities.
+
+As we learned in the previous tutorial, certain situations require us to collect (pull) data from applications and stream them out of the cluster as soon as possible. `Thanos Receive` allows doing that by ingesting metrics using the Remote Write protocol that the sender can implement. Typically we recommended using Prometheus with short retention and blocked read and query API as a "lightweight" sender.
+
+In November 2021, however, we, the Prometheus community, introduced a brand new Prometheus mode called "Agent mode". The implementation itself was already battle tested on https://github.com/grafana/agent, where it was available and authored by [Robert Fratto](https://github.com/rfratto) since 2020.
+
+The agent mode is optimized for efficient metric scraping and forwarding (i.e. immediate metric removal once it's securely delivered to a remote location). Since this is incredibly helpful for the Thanos community, we wanted to give you first-hand experience deploying Prometheus Agent together with Thanos Receive in this course.
+
+In this tutorial, you will learn:
+
+* How to reduce Prometheus based client-side metric collection to a minimum, using the new Prometheus "Agent mode" with `Thanos Receiver`, explained in the previous tutorial.
+
+> NOTE: This course uses docker containers with pre-built Thanos, Prometheus, and Minio Docker images available publicly.
+
+### Prerequisites
+
+Please complete tutorial #3 first: [Intermediate: Streaming metrics from remote source with Thanos Receive](https://www.katacoda.com/thanos/courses/thanos/3-receiver) 🤗
+
+### Feedback
+
+Do you see any bug, typo in the tutorial, or do you have some feedback for us?
+Let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io
+
+### Contributed by:
+
+* Bartek Plotka [@bwplotka](http://bwplotka.dev)
diff --git a/tutorials/katacoda/thanos/4-receiver-agent/step1-verify.sh b/tutorials/katacoda/thanos/4-receiver-agent/step1-verify.sh
new file mode 100644
index 0000000000..5143a53c61
--- /dev/null
+++ b/tutorials/katacoda/thanos/4-receiver-agent/step1-verify.sh
@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+
+# receive
+curl -s 127.0.0.1:10909/metrics >/dev/null || exit 1
+# query
+curl -s 127.0.0.1:39090/metrics >/dev/null || exit 1
+
+echo '"done"'
diff --git a/tutorials/katacoda/thanos/4-receiver-agent/step1.md b/tutorials/katacoda/thanos/4-receiver-agent/step1.md
new file mode 100644
index 0000000000..465f2974f8
--- /dev/null
+++ b/tutorials/katacoda/thanos/4-receiver-agent/step1.md
@@ -0,0 +1,56 @@
+## Problem Statement
+
+Let's get back to our example from [Tutorial 3](https://www.katacoda.com/thanos/courses/thanos/3-receiver). Imagine you run a company called `Wayne Enterprises`. In tutorial 3, we established monitoring of two special clusters: `Batcave` & `Batcomputer`.  These are special because they do not expose public endpoints to the Prometheus instances running there for security reasons, so we used the Remote Write protocol to stream all metrics to Thanos Receive in our centralized space.
+
+Let's imagine we want to expand our `Wayne Enterprises` by adding metrics collection to applications running on smaller devices inside more mobile Batman tools: `Batmobile` and `Batcopter`.
+
+Each of these vehicles has a smaller computer that runs applications from which we want to scrape Prometheus-like metrics using OpenMetrics format.
+
+As the person responsible for implementing monitoring in these environments, you have three requirements to meet:
+
+1. Implement a global view of this data. `Wayne Enterprises` needs to know what is happening in all company parts - including secret ones!
+2. `Batmobile` and `Batcopter` can be out of network for some duration of time. You don't want to lose precious data.
+3. `Batmobile` and `Batcopter` environments are very **resource constrained**, so you want an efficient solution that avoids extra computations and storage.
+
+Firstly, let us set up Thanos as we explained in the previous tutorial.
+
+## Setup Central Platform
+
+As you might remember from the previous tutorial, in the simplest form for streaming use cases, we need to deploy Thanos Receive and Thanos Querier.
+
+Let's run `Thanos Receive`:
+
+```
+docker run -d --rm \
+    -v $(pwd)/receive-data:/receive/data \
+    --net=host \
+    --name receive \
+    quay.io/thanos/thanos:v0.21.0 \
+    receive \
+    --tsdb.path "/receive/data" \
+    --grpc-address 127.0.0.1:10907 \
+    --http-address 127.0.0.1:10909 \
+    --label "receive_replica=\"0\"" \
+    --label "receive_cluster=\"wayne-enterprises\"" \
+    --remote-write.address 127.0.0.1:10908
+```{{execute}}
+
+This starts Thanos Receive that listens on `http://127.0.0.1:10908/api/v1/receive` endpoint for Remote Write and on `127.0.0.1:10907` for Thanos StoreAPI.
+
+Next, let us run a `Thanos Query` instance connected to Thanos Receive:
+
+```
+docker run -d --rm \
+--net=host \
+--name query \
+quay.io/thanos/thanos:v0.21.0 \
+query \
+--http-address "0.0.0.0:39090" \
+--store "127.0.0.1:10907"
+```{{execute}}
+
+Verify that `Thanos Query` is working and configured correctly by looking at the 'stores' tab [here](https://[[HOST_SUBDOMAIN]]-39090-[[KATACODA_HOST]].environments.katacoda.com/stores) (try refreshing the Thanos UI if it does not show up straight away).
+
+We should see Receive store on this page and, as expected, no metric data since we did not connect any Remote Write sender yet.
+
+With our "central" platform that is ready to ingest metrics, we can now start to architect our collection pipeline that will stream all metrics to our `Wayne Enterprises`.
diff --git a/tutorials/katacoda/thanos/4-receiver-agent/step2-verify.sh b/tutorials/katacoda/thanos/4-receiver-agent/step2-verify.sh
new file mode 100644
index 0000000000..be32a11ac9
--- /dev/null
+++ b/tutorials/katacoda/thanos/4-receiver-agent/step2-verify.sh
@@ -0,0 +1,12 @@
+#!/usr/bin/env bash
+
+# prometheus-batcave
+curl -s 127.0.0.1:9090/metrics >/dev/null || exit 1
+# prometheus-batcomputer
+curl -s 127.0.0.1:9091/metrics >/dev/null || exit 1
+# receive
+curl -s 127.0.0.1:10909/metrics >/dev/null || exit 1
+# query
+curl -s 127.0.0.1:39090/metrics >/dev/null || exit 1
+
+echo '"done"'
diff --git a/tutorials/katacoda/thanos/4-receiver-agent/step2.md b/tutorials/katacoda/thanos/4-receiver-agent/step2.md
new file mode 100644
index 0000000000..1c860ec367
--- /dev/null
+++ b/tutorials/katacoda/thanos/4-receiver-agent/step2.md
@@ -0,0 +1,104 @@
+
+# Setup `batmobile` and `batcopter` lightweight metric collection
+
+With `Wayne Enterprise` platform ready to ingest Remote Write data, we need to think about how we satisfy our three requirements:
+
+1. Implement a global view of this data. `Wayne Enterprises` needs to know what is happening in all company parts - including secret ones!
+2. `Batmobile` and `Batcopter` can be out of network for some duration of time. You don't want to lose precious data.
+3. `Batmobile` and `Batcopter` do not have large compute power, so you want an efficient solution that avoids extra computations and storage.
+
+How are we going to do this?
+
+Previously, the recommended approach was to deploy Prometheus to our edge environment which scrapes required applications, then remote writes all metrics to Thanos Receive. This will work and satisfy 1st and 2nd requirements, however Prometheus does some things that we don't need in this specific scenario:
+
+* We don't want to alert locally or use recording rules from our edge places.
+* We don't want to query data locally, and we want to store it for an as short duration as possible.
+
+Prometheus was designed as a stateful time-series database, and it adds certain mechanics which are not desired for full forward mode. For example:
+
+* Prometheus builds additional memory structures for easy querying from memory.
+* Prometheus does not remove data when it is safely sent via remote write. It waits for at least two hours and only after the TSDB block is persisted to the disk, it may or may not be removed, depending on retention configuration.
+
+This is where Agent mode comes in handy! It is a native Prometheus mode built into the Prometheus binary. If you add the `--agent` flag when running Prometheus, it will run a dedicated, specially streamlined database, optimized for forwarding purposes, yet able to persist scraped data in the event of a crash, restart or network disconnection.
+
+Let's try to deploy it to fulfil `batmobile` and `batcopter` monitoring requirements.
+
+## Deploy `Prometheus Agent` on `batmobile`
+
+Let's use a very simple configuration file that tells prometheus agent to scrape its own metrics page every 5 seconds and forwards it's to our running `Thanos Receive`.
+
+
+global:
+  scrape_interval: 5s
+  external_labels:
+    cluster: batmobile
+    replica: 0
+
+scrape_configs:
+  - job_name: 'prometheus-agent'
+    static_configs:
+      - targets: ['127.0.0.1:9090']
+
+remote_write:
+- url: 'http://127.0.0.1:10908/api/v1/receive'
+

+
+Run the prometheus in agent mode:
+
+```
+docker run -d --net=host --rm \
+-v /root/editor/prom-agent-batmobile.yaml:/etc/prometheus/prometheus.yaml \
+-v /root/prom-batmobile-data:/prometheus \
+-u root \
+--name prom-agent-batmobile \
+quay.io/bwplotka/prometheus:agent1 \
+--agent \
+--config.file=/etc/prometheus/prometheus.yaml \
+--storage.tsdb.path=/prometheus \
+--web.listen-address=:9090
+```{{execute}}
+
+This runs Prometheus Agent, which will scrape itself and forward all to Thanos Receive. It also exposes UI with pages that relate to scraping, service discovery, configuration and build information.
+
+Verify that `prom-agent-batmobile` is running by navigating to the [Batmobile Prometheus Agent UI](https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com/targets).
+
+You should see one target: Prometheus Agent on `batmobile` itself.
+
+## Deploy `Prometheus Agent` on `batcopter`
+
+Similarly, we can configure and deploy the second agent:
+
+
+global:
+  scrape_interval: 5s
+  external_labels:
+    cluster: batcopter
+    replica: 0
+
+scrape_configs:
+  - job_name: 'prometheus-agent'
+    static_configs:
+      - targets: ['127.0.0.1:9091']
+
+remote_write:
+- url: 'http://127.0.0.1:10908/api/v1/receive'
+

+
+```
+docker run -d --net=host --rm \
+-v /root/editor/prom-agent-batcopter.yaml:/etc/prometheus/prometheus.yaml \
+-v /root/prom-batcopter-data:/prometheus \
+-u root \
+--name prom-agent-batcopter \
+quay.io/bwplotka/prometheus:agent1 \
+--agent \
+--config.file=/etc/prometheus/prometheus.yaml \
+--storage.tsdb.path=/prometheus \
+--web.listen-address=:9091
+```{{execute}}
+
+Verify that `prom-agent-batcopter` is running by navigating to the [Batcopter Prometheus Agent UI](https://[[HOST_SUBDOMAIN]]-9091-[[KATACODA_HOST]].environments.katacoda.com/targets).
+
+You should see one target: Prometheus Agent on `batcopter` itself.
+
+Now, let's navigate to the last step to verify our `Wayne Enterprises` setup!
diff --git a/tutorials/katacoda/thanos/4-receiver-agent/step3-verify.sh b/tutorials/katacoda/thanos/4-receiver-agent/step3-verify.sh
new file mode 100644
index 0000000000..be32a11ac9
--- /dev/null
+++ b/tutorials/katacoda/thanos/4-receiver-agent/step3-verify.sh
@@ -0,0 +1,12 @@
+#!/usr/bin/env bash
+
+# prometheus-batcave
+curl -s 127.0.0.1:9090/metrics >/dev/null || exit 1
+# prometheus-batcomputer
+curl -s 127.0.0.1:9091/metrics >/dev/null || exit 1
+# receive
+curl -s 127.0.0.1:10909/metrics >/dev/null || exit 1
+# query
+curl -s 127.0.0.1:39090/metrics >/dev/null || exit 1
+
+echo '"done"'
diff --git a/tutorials/katacoda/thanos/4-receiver-agent/step3.md b/tutorials/katacoda/thanos/4-receiver-agent/step3.md
new file mode 100644
index 0000000000..4f0be5a42f
--- /dev/null
+++ b/tutorials/katacoda/thanos/4-receiver-agent/step3.md
@@ -0,0 +1,25 @@
+# Verify Setup
+
+At this point, we have:
+
+* Two Prometheus instances configured to `remote_write` and running in `agent` mode.
+* `Thanos Receive` component ingesting data from Prometheus
+* `Thanos Query` component configured to query `Thanos Receive`'s Store API.
+
+The final task on our list is to verify that data is flowing correctly.
+
+Stop and think how you could do this before opening the answer below 👇
+
+

+ How are we going to check that the components are wired up correctly?
+
+
+Let's make sure that we can query data from each of our Prometheus instances from our `Thanos Query` instance.
+
+Navigate to the [Thanos Query UI](https://[[HOST_SUBDOMAIN]]-39090-[[KATACODA_HOST]].environments.katacoda.com), and query for a metric like `up` or `go_goroutines` - inspect the output and you should see `batmobile` and `batcopter` in the `cluster` label.
+
+`go_goroutines` should look something like on image below:
+
+![expected](./assets/expected.png)
+
+

\ No newline at end of file