Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow users to convert timezone in elasticsearch module filesets #9761

Merged
merged 10 commits into from
Dec 28, 2018
Merged

Allow users to convert timezone in elasticsearch module filesets #9761

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
6b0b1b4
Allow users to convert timezone in elasticsearch/server fileset
ycombinator Dec 21, 2018
e7ddf48
Allow users to convert timezone in elasticsearch/deprecation fileset
ycombinator Dec 21, 2018
c356eba
Allow users to convert timezone in elasticsearch/audit fileset
ycombinator Dec 26, 2018
1990510
Adding CHANGELOG entry
ycombinator Dec 21, 2018
65e635b
Allow users to convert timezone in elasticsearch/slowlog fileset
ycombinator Dec 26, 2018
5a9a959
Cleaning up formatting
ycombinator Dec 26, 2018
d425d58
Updating x-pack filebeat files
ycombinator Dec 27, 2018
6fea5c6
Updating expected results of tests
ycombinator Dec 27, 2018
d9a2bac
Regenerating golden files
ycombinator Dec 28, 2018
d9f5f1f
Moving CHANGELOG entry to right version
ycombinator Dec 28, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha1...v7.0.0-alpha2[Check the
- Rename many `icinga.*` fields to map to ECS. {pull}9294[9294]
- Rename many `postgresql.log.*` fields to map to ECS. {pull}9303[9303]
- Rename many `kafka.log.*` fields to map to ECS. {pull}9297[9297]
- Add `convert_timezone` option to Elasticsearch module to convert dates to UTC. {issue}9756[9756] {pull}9761[9761]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this changelog is in the wrong place as alpha2 is already shipped. Best directly add it to .next changelog.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved in d9f5f1f.


*Metricbeat*

Expand Down
13 changes: 13 additions & 0 deletions filebeat/filebeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,9 @@ filebeat.modules:
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
#var.convert_timezone: false

gc:
enabled: true
# Set custom paths for the log files. If left empty,
Expand All @@ -104,18 +107,28 @@ filebeat.modules:
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
#var.convert_timezone: false

slowlog:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
#var.convert_timezone: false

deprecation:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
#var.convert_timezone: false


#------------------------------- Haproxy Module -------------------------------
- module: haproxy
# All logs
Expand Down
13 changes: 13 additions & 0 deletions filebeat/module/elasticsearch/_meta/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
#var.convert_timezone: false

gc:
enabled: true
# Set custom paths for the log files. If left empty,
Expand All @@ -19,14 +22,24 @@
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
#var.convert_timezone: false

slowlog:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
#var.convert_timezone: false

deprecation:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
#var.convert_timezone: false

5 changes: 5 additions & 0 deletions filebeat/module/elasticsearch/audit/config/audit.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@ paths:
{{ end }}
exclude_files: [".gz$"]

{{ if .convert_timezone }}
processors:
- add_locale: ~
{{ end }}

fields:
service.name: "elasticsearch"
fields_under_root: true
14 changes: 12 additions & 2 deletions filebeat/module/elasticsearch/audit/ingest/pipeline.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,19 @@
}
},
{
"rename": {
"date": {
"field": "elasticsearch.audit.timestamp",
"target_field": "@timestamp"
"target_field": "@timestamp",
"formats": [
"ISO8601"
],
{< if .convert_timezone >}"timezone": "{{ event.timezone }}",{< end >}
"ignore_failure": true
}
},
{
"remove": {
"field": "elasticsearch.audit.timestamp"
}
}
],
Expand Down
7 changes: 7 additions & 0 deletions filebeat/module/elasticsearch/audit/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,13 @@ var:
- /usr/local/var/lib/elasticsearch/*_access.log
os.windows:
- c:/ProgramData/Elastic/Elasticsearch/logs/*_access.log
- name: convert_timezone
default: false
# if ES < 6.1.0, this flag switches to false automatically when evaluating the
# pipeline
min_elasticsearch_version:
version: 6.1.0
value: false

ingest_pipeline: ingest/pipeline.json
input: config/audit.yml
14 changes: 7 additions & 7 deletions filebeat/module/elasticsearch/audit/test/test.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[
{
"@timestamp": "2018-06-19T05:16:15,549",
"@timestamp": "2018-06-19T05:16:15.549Z",
"ecs.version": "1.0.0-beta2",
"elasticsearch.audit.event_type": "authentication_failed",
"elasticsearch.audit.layer": "rest",
Expand All @@ -15,7 +15,7 @@
"service.name": "elasticsearch"
},
{
"@timestamp": "2018-06-19T05:07:52,304",
"@timestamp": "2018-06-19T05:07:52.304Z",
"ecs.version": "1.0.0-beta2",
"elasticsearch.audit.event_type": "authentication_failed",
"elasticsearch.audit.layer": "rest",
Expand All @@ -31,7 +31,7 @@
"service.name": "elasticsearch"
},
{
"@timestamp": "2018-06-19T05:00:15,778",
"@timestamp": "2018-06-19T05:00:15.778Z",
"ecs.version": "1.0.0-beta2",
"elasticsearch.audit.action": "indices:data/read/scroll/clear",
"elasticsearch.audit.event_type": "access_granted",
Expand All @@ -48,7 +48,7 @@
"service.name": "elasticsearch"
},
{
"@timestamp": "2018-06-19T05:07:45,544",
"@timestamp": "2018-06-19T05:07:45.544Z",
"ecs.version": "1.0.0-beta2",
"elasticsearch.audit.event_type": "anonymous_access_denied",
"elasticsearch.audit.layer": "rest",
Expand All @@ -63,7 +63,7 @@
"service.name": "elasticsearch"
},
{
"@timestamp": "2018-06-19T05:26:27,268",
"@timestamp": "2018-06-19T05:26:27.268Z",
"ecs.version": "1.0.0-beta2",
"elasticsearch.audit.event_type": "authentication_failed",
"elasticsearch.audit.layer": "rest",
Expand All @@ -78,7 +78,7 @@
"service.name": "elasticsearch"
},
{
"@timestamp": "2018-06-19T05:55:26,898",
"@timestamp": "2018-06-19T05:55:26.898Z",
"ecs.version": "1.0.0-beta2",
"elasticsearch.audit.action": "cluster:monitor/main",
"elasticsearch.audit.event_type": "access_denied",
Expand All @@ -95,7 +95,7 @@
"service.name": "elasticsearch"
},
{
"@timestamp": "2018-06-19T05:24:15,190",
"@timestamp": "2018-06-19T05:24:15.190Z",
"ecs.version": "1.0.0-beta2",
"elasticsearch.audit.event_type": "authentication_failed",
"elasticsearch.audit.layer": "rest",
Expand Down
5 changes: 5 additions & 0 deletions filebeat/module/elasticsearch/deprecation/config/log.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,11 @@ multiline:
negate: true
match: after

{{ if .convert_timezone }}
processors:
- add_locale: ~
{{ end }}

fields:
service.name: "elasticsearch"

Expand Down
14 changes: 12 additions & 2 deletions filebeat/module/elasticsearch/deprecation/ingest/pipeline.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,19 @@
}
},
{
"rename": {
"date": {
"field": "timestamp",
"target_field": "@timestamp"
"target_field": "@timestamp",
"formats": [
"ISO8601"
],
{< if .convert_timezone >}"timezone": "{{ event.timezone }}",{< end >}
"ignore_failure": true
}
},
{
"remove": {
"field": "timestamp"
}
}
]
Expand Down
7 changes: 7 additions & 0 deletions filebeat/module/elasticsearch/deprecation/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,13 @@ var:
- /usr/local/var/lib/elasticsearch/*_deprecation.log
os.windows:
- c:/ProgramData/Elastic/Elasticsearch/logs/*_deprecation.log
- name: convert_timezone
default: false
# if ES < 6.1.0, this flag switches to false automatically when evaluating the
# pipeline
min_elasticsearch_version:
version: 6.1.0
value: false

ingest_pipeline: ingest/pipeline.json
input: config/log.yml
8 changes: 4 additions & 4 deletions filebeat/module/elasticsearch/deprecation/test/elasticsearch_deprecation.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[
{
"@timestamp": "2018-04-23T16:40:13,737",
"@timestamp": "2018-04-23T16:40:13.737Z",
"ecs.version": "1.0.0-beta2",
"elasticsearch.server.component": "o.e.d.a.a.i.t.p.PutIndexTemplateRequest",
"event.dataset": "deprecation",
Expand All @@ -12,7 +12,7 @@
"service.name": "elasticsearch"
},
{
"@timestamp": "2018-04-23T16:40:13,862",
"@timestamp": "2018-04-23T16:40:13.862Z",
"ecs.version": "1.0.0-beta2",
"elasticsearch.server.component": "o.e.d.a.a.i.t.p.PutIndexTemplateRequest",
"event.dataset": "deprecation",
Expand All @@ -24,7 +24,7 @@
"service.name": "elasticsearch"
},
{
"@timestamp": "2018-04-23T16:40:14,792",
"@timestamp": "2018-04-23T16:40:14.792Z",
"ecs.version": "1.0.0-beta2",
"elasticsearch.server.component": "o.e.d.a.a.i.t.p.PutIndexTemplateRequest",
"event.dataset": "deprecation",
Expand All @@ -36,7 +36,7 @@
"service.name": "elasticsearch"
},
{
"@timestamp": "2018-04-23T16:40:15,127",
"@timestamp": "2018-04-23T16:40:15.127Z",
"ecs.version": "1.0.0-beta2",
"elasticsearch.server.component": "o.e.d.a.a.i.t.p.PutIndexTemplateRequest",
"event.dataset": "deprecation",
Expand Down
Loading