docs: document recent monitoring changes (#17233)

Adds admin and changelog documentation on some recent monitoring changes: - #17072 - #17069 - #17158 - #17070 Dev docs also show up before admin docs if you search for "grafana" or "prometheus", so adds notices that point back to the admin docs.
sourcegraph · Jan 15, 2021 · 34c4095 · 34c4095
1 parent 4428dfd
commit 34c4095
Show file tree

Hide file tree

Showing 6 changed files with 47 additions and 8 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -15,7 +15,9 @@ All notable changes to Sourcegraph are documented in this file.
 
 ### Added
 
-- Panels in the Sourcegraph monitoring dashboards now have links to relevant alerts documentation and the new [monitoring dashboards reference](https://docs.sourcegraph.com/admin/observability/dashboards). [#16939](https://github.com/sourcegraph/sourcegraph/pull/16939)
+- Panels in the [Sourcegraph monitoring dashboards](https://docs.sourcegraph.com/admin/observability/metrics#grafana) now:
+  - include links to relevant alerts documentation and the new [monitoring dashboards reference](https://docs.sourcegraph.com/admin/observability/dashboards). [#16939](https://github.com/sourcegraph/sourcegraph/pull/16939)
+  - include alert events and version changes annotations that can be enabled from the top of each service dashboard. [#17198](https://github.com/sourcegraph/sourcegraph/pull/17198)
 - Suggested filters in the search results page can now be scrolled. [#17097](https://github.com/sourcegraph/sourcegraph/pull/17097)
 
 ### Changed
@@ -37,6 +39,7 @@ All notable changes to Sourcegraph are documented in this file.
 - Fixed search highlighting the wrong line. [#10468](https://github.com/sourcegraph/sourcegraph/issues/10468)
 - Fixed an issue where searches of the form `foo type:file` returned results of type `path` too. [#17076](https://github.com/sourcegraph/sourcegraph/issues/17076)
 - Fixed queries like `(type:commit or type:diff)` so that if the query matches both the commit message and the diff, both are returned as results. [#16899](https://github.com/sourcegraph/sourcegraph/issues/16899)
+- Fixed container monitoring and provisioning dashboard panels not displaying metrics in certain deployment types and environments. If you continue to have issues with these panels not displaying any metrics after upgrading, please [open an issue](https://github.com/sourcegraph/sourcegraph/issues/new).
 - Fixed a nonexistent field in site configuration being marked as "required" when configuring PagerDuty alert notifications. [#17277](https://github.com/sourcegraph/sourcegraph/pull/17277)
 
 ### Removed

diff --git a/doc/admin/observability/metrics.md b/doc/admin/observability/metrics.md
@@ -11,9 +11,34 @@ Site admins can view the Grafana monitoring dashboards on a Sourcegraph instance
 
 <img src="https://user-images.githubusercontent.com/3173176/82078081-65c62780-9695-11ea-954a-84e8e9686970.png" class="screenshot" alt="Sourcegraph dashboard">
 
-### Available dashboards
+### Dashboards
 
-A complete [dashboard reference](dashboards.md) is available for more context on our available dashboards.
+A complete [dashboard reference](dashboards.md) is available for more context on our available service dashboards and panels.
+
+Additional dashboards can also be set up - see [Grafana configuration](#grafana-configuration) for more details.
+
+#### View documentation
+
+On service dashboards, each metric panel has links attached that lead to relevant [alert solutions](alert_solutions.md) or [panel documentation](dashboards.md).
+These can be accessed from the top left corner of each panel.
+
+#### View alerts
+
+The Overview dashboard includes a high-level summary of alert events across all Sourcegraph services.
+
+On service dashboards, a summary of alert events for that service is available.
+Alerts can be filtered by level using the "Filter alert level" selector at the top of the dashboard.
+This information can be overlaid on all panels in that dashboard by enabling the "Alert events" toggle.
+
+#### View version change events
+
+On service dashboards, annotations can be displayed for occasions when a Sourcegraph version change is detected by enabling the "Version changes" toggle at the top of the dashboard.
+This can be useful for correlating metrics and alerts with Sourcegraph deployment and upgrade events.
+
+#### Querying metrics
+
+Specific metrics can be queried using Grafana's Explore panel, available at `/-/debug/grafana/explore` on your Sourcegraph instance.
+The query for each individual panel can be viewed and explored by clicking on the panel and using the "Explore" option.
 
 ### Grafana configuration
 

diff --git a/doc/dev/background-information/observability/grafana.md b/doc/dev/background-information/observability/grafana.md
@@ -1,5 +1,8 @@
 # Sourcegraph Grafana
 
+> WARNING: Looking for Grafana documentation for Sourcegraph administrators?
+> See the [metrics and dashboards documentation](../../../admin/observability/metrics.md#grafana).
+
 We ship a custom Grafana image as part of a standard Sourcegraph distribution.
 Learn more about it in our [monitoring architecture](https://about.sourcegraph.com/handbook/engineering/observability/monitoring_architecture#sourcegraph-grafana).
 
@@ -9,7 +12,9 @@ The image is defined in [`docker-images/grafana`](https://sourcegraph.com/github
 
 ## Dashboards
 
-See [Metrics: Grafana](../../../admin/observability/metrics.md#grafana).
+See the [metrics and dashboards documentation](../../../admin/observability/metrics.md#grafana).
+
+To learn more about building dashboards, see the [observability developer guides](./index.md#guides).
 
 ## Upgrading Grafana
 

diff --git a/doc/dev/background-information/observability/monitoring-generator.md b/doc/dev/background-information/observability/monitoring-generator.md
@@ -35,8 +35,9 @@ It also takes care of the following:
   - Overview graphs for alerts (both Sourcegraph-wide and per-service)
   - Threshold lines for alerts of all levels are rendered in graphs
 - Formatting of units, labels, and more (using either the defaults, or the [`ObservablePanelOptions` API](https://sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/monitoring/monitoring/README.md#type-observablepaneloptions))
-- Maintaining a uniform look and feel across all dashboards
 - Providing links to [generated documentation](#documentation-generation)
+- Configuring annotation layers for alert events and version changes
+- Maintaining a uniform look and feel across all dashboards
 
 Links to generated documentation can be provided in our other generated integrations - for example, [Slack alerts](https://docs.sourcegraph.com/admin/observability/alerting#setting-up-alerting) will provide a link to the appropriate service's dashboard.
 

diff --git a/doc/dev/background-information/observability/prometheus.md b/doc/dev/background-information/observability/prometheus.md
@@ -1,5 +1,8 @@
 # Sourcegraph Prometheus
 
+> WARNING: Looking for Prometheus documentation for Sourcegraph administrators?
+> See the [metrics and dashboards documentation](../../../admin/observability/metrics.md#prometheus).
+
 We ship a custom Prometheus image as part of a standard Sourcegraph distribution.
 It currently bundles Alertmanager as well as integrations to the Sourcegraph web application.
 Learn more about it in our [monitoring architecture](https://about.sourcegraph.com/handbook/engineering/observability/monitoring_architecture#sourcegraph-prometheus).
@@ -10,7 +13,9 @@ The image is defined in [`docker-images/prometheus`](https://sourcegraph.com/git
 
 ## Metrics
 
-See [Metrics: Prometheus](../../../admin/observability/metrics.md#prometheus).
+See the [metrics and dashboards documentation](../../../admin/observability/metrics.md#grafana).
+
+To learn more about developing metrics, see the [observability developer guides](./index.md#guides).
 
 ## Prom-wrapper
 
@@ -19,6 +24,8 @@ Learn more about it [here](https://about.sourcegraph.com/handbook/engineering/ob
 
 The source code for this program is currently kept in [`docker-images/prometheus/cmd/prom-wrapper`](https://sourcegraph.com/github.com/sourcegraph/sourcegraph/-/tree/docker-images/prometheus/cmd/prom-wrapper).
 
+To learn more about developing our observability stack, see the [local Sourcegraph monitoring development guide](../../how-to/monitoring_local_dev.md).
+
 ## Upgrading Prometheus or Alertmanager
 
 To upgrade Prometheus or Alertmanager:

diff --git a/go.sum b/go.sum
@@ -775,8 +775,6 @@ github.com/hexops/autogold v1.2.0 h1:qZN9ARrud+VavYs/lZc+4cZvWrwptj3pCtms9jukeVA
 github.com/hexops/autogold v1.2.0/go.mod h1:8YhEzAjkMWmNx+uO18cGxVirXgQXjdvaMwt0Tswbo7w=
 github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUqJM=
 github.com/hexops/gotextdiff v1.0.3/go.mod h1:pSWU5MAI3yDq+fZBTazCSJysOMbxWL1BSow5/V2vxeg=
-github.com/hexops/valast v1.2.0 h1:BU1kbhKePO+mct+SknfD7RWnAziq4tOZLouIqSeWmOA=
-github.com/hexops/valast v1.2.0/go.mod h1:hQo2H4eaDuesNxXXgI4kzc8+BwD5o2EFTkkEKd5P3zI=
 github.com/hexops/valast v1.3.0 h1:/dVcyUWEaQxUGbS6ImRH6cmF9DbhRyMqIxIo6kyHoPY=
 github.com/hexops/valast v1.3.0/go.mod h1:hQo2H4eaDuesNxXXgI4kzc8+BwD5o2EFTkkEKd5P3zI=
 github.com/honeycombio/libhoney-go v1.14.0 h1:e764aeqc8d1jCcqvhAFCuuZ7S8nqJsg1DMazT/yRlGM=