Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Google Cloud Module Additional Field Extraction Request #15651

Closed
ghost opened this issue Jan 17, 2020 · 2 comments
Closed

Google Cloud Module Additional Field Extraction Request #15651

ghost opened this issue Jan 17, 2020 · 2 comments
Labels
enhancement good first issue Indicates a good issue for first-time contributors Team:Integrations Label for the Integrations team

Comments

@ghost
Copy link

ghost commented Jan 17, 2020 •
edited by ghost
Loading

The current Google Cloud module doesn't extract "insertId" field from the VPC flow logs in GCP. According to GCP documentation, this is a unique identifier for the log entry.

Optional. A unique identifier for the log entry. If you provide a value, then Logging considers other log entries in the same project, with the same timestamp, and with the same insertId to be duplicates which can be removed. If omitted in new log entries, then Logging assigns its own unique identifier. The insertId is also used to order log entries that have the same timestamp value.
ref: https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry

Extracting the field with the Google Cloud module would help to find the exact log event in Stackdriver. Right now, it's a bit harder to do it.
@ycombinator ycombinator added enhancement Team:Integrations Label for the Integrations team labels Jan 17, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@andrewkroh
Copy link
Member

For the vpcflow dataset I think it would be just one change at:

beats/x-pack/filebeat/module/googlecloud/vpcflow/config/pipeline.js

Line 60 in 9ece104

{from: "json.jsonPayload", to: "json"},

to add {from: "json.insertId", to: "label.insert_id"},.

And it would be a very similar change to the audit and firewall datasets.

@andrewkroh andrewkroh added the good first issue Indicates a good issue for first-time contributors label Jan 18, 2020
leehinman added a commit to leehinman/beats that referenced this issue Feb 24, 2020
@leehinman
Improve ECS categorization field mappings in googlecloud module
01d7cd2 
+ audit
  - event.id
  - event.action
  - event.kind

+ firewall
  - event.kind
  - event.category
  - event.type
  - event.action
  - event.id
  - rule.name

+ vpcflow
  - event.kind
  - event.category
  - event.type
  - event.id

Closes elastic#16030
Closes elastic#15651
leehinman added a commit to leehinman/beats that referenced this issue Feb 24, 2020
@leehinman
[Filebeat] Improve ECS categorization field mappings in googlecloud m…
58cfbcc 
…odule (elastic#16500)

+ audit
  - event.id
  - event.action
  - event.kind

+ firewall
  - event.kind
  - event.category
  - event.type
  - event.action
  - event.id
  - rule.name

+ vpcflow
  - event.kind
  - event.category
  - event.type
  - event.id

Closes elastic#16030
Closes elastic#15651

(cherry picked from commit e1fa198)
leehinman added a commit that referenced this issue Feb 25, 2020
@leehinman
[Filebeat] Improve ECS categorization field mappings in googlecloud m…
29f87aa 
…odule (#16500) (#16528)

+ audit
  - event.id
  - event.action
  - event.kind

+ firewall
  - event.kind
  - event.category
  - event.type
  - event.action
  - event.id
  - rule.name

+ vpcflow
  - event.kind
  - event.category
  - event.type
  - event.id

Closes #16030
Closes #15651

(cherry picked from commit e1fa198)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement good first issue Indicates a good issue for first-time contributors Team:Integrations Label for the Integrations team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@ycombinator @andresrc @andrewkroh @adriansr @elasticmachine