Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Infra UI] Add AWS metrics to node detail page #42153

Merged
merged 22 commits into from
Aug 13, 2019

Conversation

skh
Copy link
Contributor

@skh skh commented Jul 29, 2019

Summary

Implements #39281 and #39282.

This adds the following AWS metrics to the node detail page:

  • Overview
  • CPU Utilization
  • Network Traffic
  • Network Packets
  • Disk I/O Operations
  • Disk I/O Bytes.

Because of #42687 we might have to disable Network Traffic and both Disk I/O sections for now. They are still contained in this PR, they can be commented out quickly if necessary.

To test this, use an ES instance with metricbeat data for at least one AWS instance as described in https://github.com/elastic/kibana/blob/master/x-pack/legacy/plugins/infra/docs/test_setups/infra_metricbeat_aws.md . The Observability 8.x.x shared cluster has this data. (Please note that the instance that reports to this cluster doesn't have any storage attached, so the Disk I/O metrics are zero overall.)

View the node detail page for an AWS instance and verify that AWS metrics are shown:

image

In GraphiQL, you can view the data that is returned for each model with a query like this:

{
  source(id: "default") {
    metrics(nodeIds: {nodeId: "ip-172-31-47-9.us-east-2.compute.internal", cloudId: "i-011454f72559c510b"},
      nodeType:host, timerange: {from: 1564758416480, interval: ">=1m", to: 1564762016480},
      metrics: [awsNetworkPackets]) {
      id
      series {
        id
        data {
          value
          timestamp
        }
      }
    }
  }
}

The available metrics that have been added are

  • awsOverview
  • awsCpuUtilization
  • awsNetworkBytes
  • awsNetworkPackets
  • awsDiskioBytes
  • awsDiskioOps

@elasticmachine
Copy link
Contributor

💔 Build Failed

@elasticmachine
Copy link
Contributor

💔 Build Failed

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@skh skh marked this pull request as ready for review August 6, 2019 15:50
@skh skh requested a review from a team as a code owner August 6, 2019 15:50
@skh skh added Feature:Metrics UI Metrics UI feature release_note:enhancement Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services v7.4.0 labels Aug 6, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/infra-logs-ui

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@jasonrhodes jasonrhodes self-requested a review August 8, 2019 13:21
@simianhacker simianhacker self-requested a review August 8, 2019 15:29
{
id: 'awsOverview',
label: 'AWS',
requires: ['aws'],
Copy link
Member

@simianhacker simianhacker Aug 8, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

requires at the top level can be removed, only the section requires arrays are read (see comment above).

@simianhacker
Copy link
Member

@sorantis @kaiyan-sheng Is the network packets value a rate? I know the values from Metricbeat (system.network.out.packets) are monotonically increasing numbers which we usually end up display as a per second value. The only reason I ask is because there isn't a suffix label on the data and I'm not sure what label would be appropriate? Also if it's a rate then we should probably divide by 300 as well.

@simianhacker
Copy link
Member

@sorantis @kaiyan-sheng Looks like I answered my own question: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/viewing_metrics_with_cloudwatch.html

Looks like it's just the number of packets transferred.

@simianhacker simianhacker self-requested a review August 8, 2019 18:33
Copy link
Member

@simianhacker simianhacker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good Job! Everything looks good. Just remove the requires key from the AWS layout and change the layout generator's function signature as noted. Once that's done, everything should be good to go.

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@skh
Copy link
Contributor Author

skh commented Aug 9, 2019

@simianhacker Thanks for the review! This is ready for another look.

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@skh
Copy link
Contributor Author

skh commented Aug 12, 2019

@simianhacker this is ready for another look.

Copy link
Member

@simianhacker simianhacker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work!

.____     ________________________   
|    |   /  _____/\__    ___/     \  
|    |  /   \  ___  |    | /  \ /  \ 
|    |__\    \_\  \ |    |/    Y    \
|_______ \______  / |____|\____|__  /
        \/      \/                \/ 

@skh skh merged commit c970e70 into elastic:master Aug 13, 2019
jloleysens added a commit to jloleysens/kibana that referenced this pull request Aug 13, 2019
…p-metrics-selectall

* 'master' of github.com:elastic/kibana: (27 commits)
  [ML] Data Frames: Analytics job creation. (elastic#43102)
  [Vis Default editor] Fix issue with Rollup (elastic#42430)
  [Vis: Default editor] EUIficate Markdown tab (elastic#42677)
  [New Platform Migration Phase I]: update dateHisogramInterval & parseEsInterval imports (elastic#42917)
  [Infra UI] Add AWS metrics to node detail page (elastic#42153)
  update apm index pattern (elastic#43106)
  [SIEM] Toggle Column / Code Coverage and Cypress (elastic#42766)
  skip failing test (elastic#43163)
  [code] Add option to turn the go dependency download on/off. (elastic#43096)
  disable visual regression jobs
  Removed dead code (elastic#42774)
  fixes csv export of saved searches that have _source field (elastic#43123)
  Export missing Context types (elastic#43051)
  Update dependency supports-color to v7 (elastic#43064)
  switch to icon type string instead of node (elastic#43111)
  [Maps] Enable borders for icon symbols (elastic#43066)
  [ftr] enable visualRegression jobs (elastic#42989)
  [ML] Converting single to multi metric job (elastic#42532)
  fix(NA): dont clean dll module if it is a package json file (elastic#42904)
  [Logs UI] Add link from the sample web logs to the Logs UI (elastic#42635)
  ...
skh added a commit to skh/kibana that referenced this pull request Aug 13, 2019
* Add cloudId to metrics graphql query.

* Add server-side TSVB model for aws cpu utilization

* Use cloud id for cloud metrics

* Add AWS layout to host page

* Add cloudId to useMetadata

* Refactor: move aws modules into separate directory

* Add AWS network chart to node details

* Add AWS Overview to node details

* Add aws network packets chart to node details

* Add TODO reference to aws network bytes model

* Add AWS diskio ops to node details

* Add AWS diskio bytes to node details

* Check that cloudId is given when needed

* Use network packets on node details AWS overview

* Remove requires on layout level.

* Fix layout creator type signature.

* Add gaugeMax to layout types
skh added a commit that referenced this pull request Aug 13, 2019
* Add cloudId to metrics graphql query.

* Add server-side TSVB model for aws cpu utilization

* Use cloud id for cloud metrics

* Add AWS layout to host page

* Add cloudId to useMetadata

* Refactor: move aws modules into separate directory

* Add AWS network chart to node details

* Add AWS Overview to node details

* Add aws network packets chart to node details

* Add TODO reference to aws network bytes model

* Add AWS diskio ops to node details

* Add AWS diskio bytes to node details

* Check that cloudId is given when needed

* Use network packets on node details AWS overview

* Remove requires on layout level.

* Fix layout creator type signature.

* Add gaugeMax to layout types
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Metrics UI Metrics UI feature release_note:enhancement Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services v7.4.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants