Skip to content

Commit

Permalink
[metricbeat] meraki module with device_health metricset (#40836)
Browse files Browse the repository at this point in the history
* inital meraki module and metricsets

* initial refactor for single meraki metricset device_health

* added tunnel support aka VPN support by Device

* adding interfaces aka switch ports and switch port status

* processing review comments

* refactored for comments

* fixing default metricset

* Removing unused variables and adding text to required variables

* add go module deps

* fixing for loop and starting on fields yml

* remove %d from loop condition

* fixing review comments and loss latency

* update dashboard-api lib to version with MIT licensed ratelimiter

* remove unused fields in metricset struct

* remove device details for now

* Major refactor to improve overall readability and device status/metric associations.

This commit removes a couple of data points from the original code, namely:
 - network appliance port settings - these settings are not defined per-device,
   but per-network, and as such don't fit into the current naming taxonomy.
 - site-to-site VPN settings - similarly to the above, these are not per-device attibutes.

In addition there is a limitation included on including only a single metric bucket (as returned
from the API) per metricbeat collection loop. There needs to be some better logging/config
validation on this, which will come later; for now, just run with a maximum 5 minute collection
interval.

This commit includes the field mappings too, but it currently causes events to get dropped due
to the index field limit getting hit (max 10k fields). You can get around this by setting
'setup.template.settings.index.mapping.total_fields.limit' to a higher value. We maybe can't
include all these mappings in the metricbeat module, but for now they are here to allow testing
with standalone metricbeat.

* Add maximum collection interval check to avoid missing uplink loss and latency metrics.

Remove 'percent' units, because the data does not fit the 0-1 elastic interpretation of percent.

* revert accidental changes to reference config

* add example in data.json

* review comments: license headers, missing/incorrect mapping

* result of 'go mod tidy' ... trying to unblock the CI builds

* fix linter issues

* update NOTICE.txt

* add release tag to fields.yml, update docs

* review feedback: keyword -> dat

* commit results of 'make update' in x-pack metricbeat for CI

* remove mappings to avoid hitting field limits in global metricbeat index template

* add one field back to please the docs CI

* more linter errors guarding against shared loop var pointers

* fix error introduced in previous commit

* add exception for golangci-lint G601 which is no longer required in go1.22+

* remove the guards against taking addresses of loop vars

* i can't get the golanglint-ci config to get picked up, so i'm reverting that change and putting back the annoying index loop code

* Update go.mod to match toolchain version to .go-version

Co-authored-by: Craig MacKenzie <[email protected]>

---------

Co-authored-by: Dan Hiebert <[email protected]>
Co-authored-by: Craig MacKenzie <[email protected]>
(cherry picked from commit ae71c67)

# Conflicts:
#	NOTICE.txt
#	go.mod
#	go.sum
#	x-pack/metricbeat/metricbeat.reference.yml
  • Loading branch information
tommyers-elastic authored and mergify[bot] committed Sep 17, 2024
1 parent 85e80de commit 93ea5c6
Show file tree
Hide file tree
Showing 23 changed files with 1,481 additions and 35 deletions.
290 changes: 266 additions & 24 deletions NOTICE.txt

Large diffs are not rendered by default.

44 changes: 43 additions & 1 deletion go.mod
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
module github.com/elastic/beats/v7

go 1.22
go 1.22.0

toolchain go1.22.6

require (
cloud.google.com/go/bigquery v1.59.1
Expand Down Expand Up @@ -144,6 +146,7 @@ require (
go.uber.org/atomic v1.11.0 // indirect
go.uber.org/multierr v1.11.0
go.uber.org/zap v1.27.0
<<<<<<< HEAD
golang.org/x/crypto v0.25.0
golang.org/x/lint v0.0.0-20210508222113-6edffad5e616
golang.org/x/mod v0.18.0
Expand All @@ -157,6 +160,20 @@ require (
google.golang.org/api v0.169.0
google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 // indirect
google.golang.org/grpc v1.64.0
=======
golang.org/x/crypto v0.27.0
golang.org/x/mod v0.21.0
golang.org/x/net v0.29.0
golang.org/x/oauth2 v0.22.0
golang.org/x/sync v0.8.0
golang.org/x/sys v0.25.0
golang.org/x/text v0.18.0
golang.org/x/time v0.6.0
golang.org/x/tools v0.25.0
google.golang.org/api v0.191.0
google.golang.org/genproto v0.0.0-20240730163845-b1a4ccb954bf // indirect
google.golang.org/grpc v1.64.1
>>>>>>> ae71c676d8 ([metricbeat] meraki module with device_health metricset (#40836))
google.golang.org/protobuf v1.34.2
gopkg.in/inf.v0 v0.9.1
gopkg.in/jcmturner/aescts.v1 v1.0.1 // indirect
Expand Down Expand Up @@ -214,6 +231,7 @@ require (
github.com/gorilla/mux v1.8.0
github.com/gorilla/websocket v1.5.0
github.com/icholy/digest v0.1.22
github.com/meraki/dashboard-api-go/v3 v3.0.9
github.com/otiai10/copy v1.12.0
github.com/pierrec/lz4/v4 v4.1.18
github.com/pkg/xattr v0.4.9
Expand Down Expand Up @@ -252,6 +270,7 @@ require (
github.com/apache/thrift v0.19.0 // indirect
github.com/armon/go-radix v1.0.0 // indirect
github.com/aws/aws-sdk-go v1.38.60 // indirect
<<<<<<< HEAD
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.2 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.5 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.5 // indirect
Expand All @@ -263,6 +282,20 @@ require (
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.5 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.20.5 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.23.4 // indirect
=======
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.4 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.16 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.16 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.1 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.16 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.18 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.18 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.16 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.22.5 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.26.5 // indirect
github.com/benbjohnson/clock v1.3.0 // indirect
>>>>>>> ae71c676d8 ([metricbeat] meraki module with device_health metricset (#40836))
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash v1.1.0 // indirect
github.com/cilium/ebpf v0.13.2 // indirect
Expand Down Expand Up @@ -292,6 +325,7 @@ require (
github.com/go-openapi/jsonpointer v0.19.6 // indirect
github.com/go-openapi/jsonreference v0.20.2 // indirect
github.com/go-openapi/swag v0.22.3 // indirect
github.com/go-resty/resty/v2 v2.11.0 // indirect
github.com/go-stack/stack v1.8.0 // indirect
github.com/gobuffalo/here v0.6.7 // indirect
github.com/goccy/go-json v0.10.2 // indirect
Expand All @@ -302,6 +336,7 @@ require (
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/google/gnostic-models v0.6.8 // indirect
github.com/google/go-querystring v1.1.0 // indirect
github.com/google/licenseclassifier v0.0.0-20221004142553-c1ed8fcf4bab // indirect
github.com/google/s2a-go v0.1.7 // indirect
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
Expand Down Expand Up @@ -372,8 +407,14 @@ require (
go.opentelemetry.io/otel/metric v1.28.0 // indirect
go.opentelemetry.io/otel/trace v1.28.0 // indirect
go.opentelemetry.io/proto/otlp v1.3.1 // indirect
<<<<<<< HEAD
golang.org/x/exp v0.0.0-20231127185646-65229373498e // indirect
golang.org/x/term v0.22.0 // indirect
=======
go.uber.org/ratelimit v0.3.1 // indirect
golang.org/x/exp v0.0.0-20240205201215-2c58cdc269a3 // indirect
golang.org/x/term v0.24.0 // indirect
>>>>>>> ae71c676d8 ([metricbeat] meraki module with device_health metricset (#40836))
golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 // indirect
google.golang.org/appengine v1.6.8 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240513163218-0867130af1f8 // indirect
Expand Down Expand Up @@ -414,6 +455,7 @@ replace (
github.com/golang/glog => github.com/elastic/glog v1.0.1-0.20210831205241-7d8b5c89dfc4
github.com/google/gopacket => github.com/elastic/gopacket v1.1.20-0.20211202005954-d412fca7f83a
github.com/insomniacslk/dhcp => github.com/elastic/dhcp v0.0.0-20200227161230-57ec251c7eb3 // indirect
github.com/meraki/dashboard-api-go/v3 => github.com/tommyers-elastic/dashboard-api-go/v3 v3.0.0-20240913150833-a945473a8f25
github.com/snowflakedb/gosnowflake => github.com/snowflakedb/gosnowflake v1.6.19
github.com/tonistiigi/fifo => github.com/containerd/fifo v0.0.0-20190816180239-bda0ff6ed73c
k8s.io/kubernetes v1.13.0 => k8s.io/kubernetes v1.24.15
Expand Down
Loading

0 comments on commit 93ea5c6

Please sign in to comment.