Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] ETW input - add basic metrics #38017

Merged
merged 5 commits into from
Mar 6, 2024

Conversation

andrewkroh
Copy link
Member

@andrewkroh andrewkroh commented Feb 14, 2024

Proposed commit message

Add input metrics to for ETW.

Remove the very verbose debug message that was in the hot path for
processing each event.

I think more metrics taken from the ETW session itself should be added
separately (e.g. events lost, buffers lost). This lays the ground work needed
to expose those metrics.

Closes #38007

Author checklist

  • Is the callback function only ever invoke by a single thread? (This influences where it need to protect against races on the last callback time.)

Example data

[
  {
    "arrival_period": {
      "histogram": {
        "count": 174046,
        "max": 35801500,
        "mean": 51979.00390625,
        "median": 900,
        "min": 0,
        "p75": 2300,
        "p95": 10775,
        "p99": 136950,
        "p999": 35187342.50000056,
        "stddev": 1172792.9385769565
      }
    },
    "discarded_events_total": 0,
    "errors_total": 0,
    "id": "win32k",
    "input": "etw",
    "processing_time": {
      "histogram": {
        "count": 174047,
        "max": 203593000,
        "mean": 461956.8359375,
        "median": 0,
        "min": 0,
        "p75": 0,
        "p95": 512375,
        "p99": 988050,
        "p999": 201688782.50000173,
        "stddev": 7506149.117071189
      }
    },
    "session": "Elastic-Microsoft-Windows-Win32k",
    "received_events_total": 174047,
    "source_lag_time": {
      "histogram": {
        "count": 174047,
        "max": 62549995500,
        "mean": 37557071369.04297,
        "median": 41560344000,
        "min": 960816000,
        "p75": 54243826225,
        "p95": 59757085550,
        "p99": 61976473025,
        "p999": 62548924737.5,
        "stddev": 18237039484.0725
      }
    }
  }
]

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 14, 2024
@andrewkroh andrewkroh added enhancement Filebeat Filebeat Team:Security-Windows Platform Windows Platform Team in Security Solution labels Feb 14, 2024
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 14, 2024

This comment was marked as resolved.

@elasticmachine
Copy link
Collaborator

elasticmachine commented Feb 14, 2024

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Duration: 134 min 2 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

Add input metrics to for ETW.

Remove the very verbose debug message that was in the hot path for
processing each event.

I think more metrics taken from the ETW session itself should be added
separately (e.g. events lost, buffers lost). This lays the ground works needed
to expose those metrics.

Closes elastic#38007
@andrewkroh andrewkroh force-pushed the feature/fb/etw-input-metrics branch from 095ebb0 to 3721e57 Compare February 22, 2024 16:38
@elasticmachine

This comment was marked as duplicate.

@elasticmachine

This comment was marked as duplicate.

@elasticmachine

This comment was marked as duplicate.

@elasticmachine

This comment was marked as duplicate.

@elasticmachine

This comment was marked as duplicate.

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

History

cc @andrewkroh

@andrewkroh andrewkroh marked this pull request as ready for review February 22, 2024 18:58
@andrewkroh andrewkroh requested a review from a team as a code owner February 22, 2024 18:58
@elasticmachine
Copy link
Collaborator

Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform)

@andrewkroh andrewkroh added the backport-skip Skip notification from the automated backport with mergify label Feb 22, 2024
Copy link
Contributor

@chemamartinez chemamartinez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@andrewkroh andrewkroh enabled auto-merge (squash) February 23, 2024 13:48
@andrewkroh andrewkroh merged commit ae312c5 into elastic:main Mar 6, 2024
38 of 39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-skip Skip notification from the automated backport with mergify enhancement Filebeat Filebeat Team:Security-Windows Platform Windows Platform Team in Security Solution
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Filebeat] ETW input - add metrics for obserability and troubleshooting
4 participants