Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop fields in aws vpcflow data stream when value equals to "-" #435

Merged
merged 4 commits into from
Dec 4, 2020
Merged

Drop fields in aws vpcflow data stream when value equals to "-" #435

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6028e01
Drop fields in aws vpcflow data stream when value equals to "-"
kaiyan-sheng Dec 2, 2020
0a84097
Merge remote-tracking branch 'upstream/master' into vpcflow
kaiyan-sheng Dec 3, 2020
d851675
bump aws package version to 0.3.14
kaiyan-sheng Dec 3, 2020
d47df7e
remove empty lines
kaiyan-sheng Dec 4, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions packages/aws/data_stream/vpcflow/_dev/test/pipeline/test-tcp-flag-sequence.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
version vpc-id subnet-id instance-id interface-id account-id type srcaddr dstaddr srcport dstport pkt-srcaddr pkt-dstaddr protocol bytes packets start end action tcp-flags log-status
3 vpc-abcdefab012345678 subnet-aaaaaaaa012345678 i-01234567890123456 eni-1235b8ca123456789 123456789010 IPv4 52.213.180.42 10.0.0.62 43416 5001 52.213.180.42 10.0.0.62 6 568 8 1566848875 1566848933 ACCEPT 2 OK
version vpc-id subnet-id instance-id interface-id account-id type srcaddr dstaddr srcport dstport pkt-srcaddr pkt-dstaddr protocol bytes packets start end action tcp-flags log-status
3 vpc-abcdefab012345678 subnet-aaaaaaaa012345678 i-01234567890123456 eni-1235b8ca123456789 123456789010 - - - - - - - - - - 1566848875 1566848933 - - SKIPDATA
version vpc-id subnet-id instance-id interface-id account-id type srcaddr dstaddr srcport dstport pkt-srcaddr pkt-dstaddr protocol bytes packets start end action tcp-flags log-status
3 vpc-abcdefab012345678 subnet-aaaaaaaa012345678 i-01234567890123456 eni-1235b8ca123456789 123456789010 - - - - - - - - - - 1566848875 1566848933 - - NODATA
58 changes: 58 additions & 0 deletions packages/aws/data_stream/vpcflow/_dev/test/pipeline/test-tcp-flag-sequence.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
{
"expected": [
{
"cloud": {
"provider": "aws"
},
"event": {
"kind": "event",
"original": "version vpc-id subnet-id instance-id interface-id account-id type srcaddr dstaddr srcport dstport pkt-srcaddr pkt-dstaddr protocol bytes packets start end action tcp-flags log-status"
}
},
{
"cloud": {
"provider": "aws"
},
"event": {
"kind": "event",
"original": "3 vpc-abcdefab012345678 subnet-aaaaaaaa012345678 i-01234567890123456 eni-1235b8ca123456789 123456789010 IPv4 52.213.180.42 10.0.0.62 43416 5001 52.213.180.42 10.0.0.62 6 568 8 1566848875 1566848933 ACCEPT 2 OK"
}
},
{
"cloud": {
"provider": "aws"
},
"event": {
"kind": "event",
"original": "version vpc-id subnet-id instance-id interface-id account-id type srcaddr dstaddr srcport dstport pkt-srcaddr pkt-dstaddr protocol bytes packets start end action tcp-flags log-status"
}
},
{
"cloud": {
"provider": "aws"
},
"event": {
"kind": "event",
"original": "3 vpc-abcdefab012345678 subnet-aaaaaaaa012345678 i-01234567890123456 eni-1235b8ca123456789 123456789010 - - - - - - - - - - 1566848875 1566848933 - - SKIPDATA"
}
},
{
"cloud": {
"provider": "aws"
},
"event": {
"kind": "event",
"original": "version vpc-id subnet-id instance-id interface-id account-id type srcaddr dstaddr srcport dstport pkt-srcaddr pkt-dstaddr protocol bytes packets start end action tcp-flags log-status"
}
},
{
"cloud": {
"provider": "aws"
},
"event": {
"kind": "event",
"original": "3 vpc-abcdefab012345678 subnet-aaaaaaaa012345678 i-01234567890123456 eni-1235b8ca123456789 123456789010 - - - - - - - - - - 1566848875 1566848933 - - NODATA"
}
}
]
}
26 changes: 26 additions & 0 deletions packages/aws/data_stream/vpcflow/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,32 @@ processors:
field: ["aws.vpcflow.start", "aws.vpcflow.end"]
ignore_missing: true

- script:
lang: painless
ignore_failure: true
if: ctx?.aws != null
source: >-
void handleMap(Map map) {
for (def x : map.values()) {
if (x instanceof Map) {
handleMap(x);
} else if (x instanceof List) {
handleList(x);
}
}
map.values().removeIf(v -> v instanceof String && v == "-");
}
void handleList(List list) {
for (def x : list) {
if (x instanceof Map) {
handleMap(x);
} else if (x instanceof List) {
handleList(x);
}
}
}
handleMap(ctx.aws);

# IP Geolocation Lookup
- geoip:
field: source.ip
Expand Down
2 changes: 1 addition & 1 deletion packages/aws/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: 1.0.0
name: aws
title: AWS
version: 0.3.13
version: 0.3.14
license: basic
description: AWS Integration
type: integration
Expand Down