Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Keep unparsed user agent information in user_agent.original #8537

Merged
merged 5 commits into from
Oct 5, 2018
Merged

Keep unparsed user agent information in user_agent.original #8537

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9b9db7d
rename user_agent.raw to user_agent.original
kvch Oct 2, 2018
3b450ed
add changelog entry
kvch Oct 3, 2018
a0cf81e
edit changelog
kvch Oct 3, 2018
5aa6260
do not index user_agent.original
kvch Oct 3, 2018
f3de59a
Merge branch 'master' into rename-user-agent-raw-to-original
ruflin Oct 5, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,12 +120,12 @@ https://github.com/elastic/beats/compare/v6.4.0...master[Check the HEAD diff]

- Make inputsource generic taking bufio.SplitFunc as input {pull}7746[7746]
- Add custom unpack to log hints config to avoid env resolution {pull}7710[7710]
- Keep raw user agent information after parsing as user_agent_raw in Filebeat modules. {pull}7823[7832]
- Make docker input check if container strings are empty {pull}7960[7960]
- Add tag "truncated" to "log.flags" if incoming line is longer than configured limit. {pull}7991[7991]
- Add tag "multiline" to "log.flags" if event consists of multiple lines. {pull}7997[7997]
- Add haproxy module. {pull}8014[8014]
- Release `docker` input as GA. {pull}8328[8328]
- Keep unparsed user agent information in user_agent.original. {pull}8537[8537]
- Better tracking of number of open file descriptors. {pull}7986[7986]

*Heartbeat*
Expand Down
24 changes: 16 additions & 8 deletions filebeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -252,14 +252,16 @@ The name of the operating system.

--

*`apache2.access.user_agent.raw`*::
*`apache2.access.user_agent.original`*::
+
--
type: text

Raw user agent value before parsing by ingest-user-agent plugin.
Original user agent value before parsing by ingest-user-agent plugin.


Field is not indexed.

--

[float]
Expand Down Expand Up @@ -4037,14 +4039,16 @@ The name of the operating system.

--

*`iis.access.user_agent.raw`*::
*`iis.access.user_agent.original`*::
+
--
type: text

Raw user agent value before parsing by ingest-user-agent plugin.
Original user agent value before parsing by ingest-user-agent plugin.


Field is not indexed.

--

[float]
Expand Down Expand Up @@ -5355,14 +5359,16 @@ The name of the operating system.

--

*`nginx.access.user_agent.raw`*::
*`nginx.access.user_agent.original`*::
+
--
type: text

Raw user agent value before parsing by ingest-user-agent plugin.
Original user agent value before parsing by ingest-user-agent plugin.


Field is not indexed.

--

[float]
Expand Down Expand Up @@ -6365,14 +6371,16 @@ The name of the operating system.

--

*`traefik.access.user_agent.raw`*::
*`traefik.access.user_agent.original`*::
+
--
type: text

Raw user agent value before parsing by ingest-user-agent plugin.
Original user agent value before parsing by ingest-user-agent plugin.


Field is not indexed.

--

[float]
Expand Down
2 changes: 1 addition & 1 deletion filebeat/include/fields.go
View file
Open in desktop

Large diffs are not rendered by default.

5 changes: 3 additions & 2 deletions filebeat/module/apache2/access/_meta/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,10 +85,11 @@
type: keyword
description: >
The name of the operating system.
- name: raw
- name: original
type: text
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we set index: false it should not matter here what type is defined. In ECS it seems we put keyword. Only reason I mention this is we should check later what shows up in the docs for non indexed fields.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. Thanks for the info.

index: false
description: >
Raw user agent value before parsing by ingest-user-agent plugin.
Original user agent value before parsing by ingest-user-agent plugin.
- name: geoip
type: group
description: >
Expand Down
2 changes: 1 addition & 1 deletion filebeat/module/apache2/access/ingest/default.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@
}, {
"rename": {
"field": "apache2.access.agent",
"target_field": "apache2.access.user_agent.raw",
"target_field": "apache2.access.user_agent.original",
"ignore_failure": true
}
}, {
Expand Down
4 changes: 2 additions & 2 deletions filebeat/module/apache2/access/test/test.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,11 @@
"apache2.access.user_agent.major": "50",
"apache2.access.user_agent.minor": "0",
"apache2.access.user_agent.name": "Firefox",
"apache2.access.user_agent.original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:50.0) Gecko/20100101 Firefox/50.0",
"apache2.access.user_agent.os": "Mac OS X 10.12",
"apache2.access.user_agent.os_major": "10",
"apache2.access.user_agent.os_minor": "12",
"apache2.access.user_agent.os_name": "Mac OS X",
"apache2.access.user_agent.raw": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:50.0) Gecko/20100101 Firefox/50.0",
"apache2.access.user_name": "-",
"fileset.module": "apache2",
"fileset.name": "access",
Expand Down Expand Up @@ -63,10 +63,10 @@
"apache2.access.user_agent.major": "15",
"apache2.access.user_agent.minor": "0",
"apache2.access.user_agent.name": "Firefox Alpha",
"apache2.access.user_agent.original": "Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2",
"apache2.access.user_agent.os": "Windows 7",
"apache2.access.user_agent.os_name": "Windows 7",
"apache2.access.user_agent.patch": "a2",
"apache2.access.user_agent.raw": "Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2",
"apache2.access.user_name": "-",
"fileset.module": "apache2",
"fileset.name": "access",
Expand Down
5 changes: 3 additions & 2 deletions filebeat/module/iis/access/_meta/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,10 +130,11 @@
type: keyword
description: >
The name of the operating system.
- name: raw
- name: original
type: text
index: false
description: >
Raw user agent value before parsing by ingest-user-agent plugin.
Original user agent value before parsing by ingest-user-agent plugin.
- name: geoip
type: group
description: >
Expand Down
2 changes: 1 addition & 1 deletion filebeat/module/iis/access/ingest/default.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@
}, {
"rename": {
"field": "iis.access.agent",
"target_field": "iis.access.user_agent.raw"
"target_field": "iis.access.user_agent.original"
}
}, {
"geoip": {
Expand Down
6 changes: 3 additions & 3 deletions filebeat/module/iis/access/test/test.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,9 @@
"iis.access.user_agent.major": "57",
"iis.access.user_agent.minor": "0",
"iis.access.user_agent.name": "Firefox",
"iis.access.user_agent.original": "Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64;+rv:57.0)+Gecko/20100101+Firefox/57.0",
"iis.access.user_agent.os": "Windows",
"iis.access.user_agent.os_name": "Windows",
"iis.access.user_agent.raw": "Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64;+rv:57.0)+Gecko/20100101+Firefox/57.0",
"iis.access.user_name": "-",
"iis.access.win32_status": "0",
"input.type": "log",
Expand Down Expand Up @@ -55,9 +55,9 @@
"iis.access.user_agent.major": "57",
"iis.access.user_agent.minor": "0",
"iis.access.user_agent.name": "Firefox",
"iis.access.user_agent.original": "Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64;+rv:57.0)+Gecko/20100101+Firefox/57.0",
"iis.access.user_agent.os": "Windows",
"iis.access.user_agent.os_name": "Windows",
"iis.access.user_agent.raw": "Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64;+rv:57.0)+Gecko/20100101+Firefox/57.0",
"iis.access.user_name": "-",
"iis.access.win32_status": "0",
"input.type": "log",
Expand Down Expand Up @@ -96,9 +96,9 @@
"iis.access.user_agent.major": "57",
"iis.access.user_agent.minor": "0",
"iis.access.user_agent.name": "Firefox",
"iis.access.user_agent.original": "Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64;+rv:57.0)+Gecko/20100101+Firefox/57.0",
"iis.access.user_agent.os": "Windows",
"iis.access.user_agent.os_name": "Windows",
"iis.access.user_agent.raw": "Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64;+rv:57.0)+Gecko/20100101+Firefox/57.0",
"iis.access.user_name": "-",
"iis.access.win32_status": "0",
"input.type": "log",
Expand Down
5 changes: 3 additions & 2 deletions filebeat/module/nginx/access/_meta/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,10 +92,11 @@
type: keyword
description: >
The name of the operating system.
- name: raw
- name: original
type: text
index: false
description: >
Raw user agent value before parsing by ingest-user-agent plugin.
Original user agent value before parsing by ingest-user-agent plugin.
- name: geoip
type: group
description: >
Expand Down
2 changes: 1 addition & 1 deletion filebeat/module/nginx/access/ingest/default.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@
}, {
"rename": {
"field": "nginx.access.agent",
"target_field": "nginx.access.user_agent.raw"
"target_field": "nginx.access.user_agent.original"
}
}, {
"geoip": {
Expand Down
14 changes: 7 additions & 7 deletions filebeat/module/nginx/access/test/test.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,11 @@
"nginx.access.user_agent.major": "49",
"nginx.access.user_agent.minor": "0",
"nginx.access.user_agent.name": "Firefox",
"nginx.access.user_agent.original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:49.0) Gecko/20100101 Firefox/49.0",
"nginx.access.user_agent.os": "Mac OS X 10.12",
"nginx.access.user_agent.os_major": "10",
"nginx.access.user_agent.os_minor": "12",
"nginx.access.user_agent.os_name": "Mac OS X",
"nginx.access.user_agent.raw": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:49.0) Gecko/20100101 Firefox/49.0",
"nginx.access.user_name": "-",
"offset": 0,
"prospector.type": "log"
Expand All @@ -48,10 +48,10 @@
"nginx.access.user_agent.major": "15",
"nginx.access.user_agent.minor": "0",
"nginx.access.user_agent.name": "Firefox Alpha",
"nginx.access.user_agent.original": "Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2",
"nginx.access.user_agent.os": "Windows 7",
"nginx.access.user_agent.os_name": "Windows 7",
"nginx.access.user_agent.patch": "a2",
"nginx.access.user_agent.raw": "Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2",
"nginx.access.user_name": "-",
"offset": 183,
"prospector.type": "log"
Expand Down Expand Up @@ -84,11 +84,11 @@
"nginx.access.user_agent.major": "49",
"nginx.access.user_agent.minor": "0",
"nginx.access.user_agent.name": "Firefox",
"nginx.access.user_agent.original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:49.0) Gecko/20100101 Firefox/49.0",
"nginx.access.user_agent.os": "Mac OS X 10.12",
"nginx.access.user_agent.os_major": "10",
"nginx.access.user_agent.os_minor": "12",
"nginx.access.user_agent.os_name": "Mac OS X",
"nginx.access.user_agent.raw": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:49.0) Gecko/20100101 Firefox/49.0",
"nginx.access.user_name": "-",
"offset": 341,
"prospector.type": "log"
Expand Down Expand Up @@ -119,11 +119,11 @@
"nginx.access.user_agent.major": "49",
"nginx.access.user_agent.minor": "0",
"nginx.access.user_agent.name": "Firefox",
"nginx.access.user_agent.original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:49.0) Gecko/20100101 Firefox/49.0",
"nginx.access.user_agent.os": "Mac OS X 10.12",
"nginx.access.user_agent.os_major": "10",
"nginx.access.user_agent.os_minor": "12",
"nginx.access.user_agent.os_name": "Mac OS X",
"nginx.access.user_agent.raw": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:49.0) Gecko/20100101 Firefox/49.0",
"nginx.access.user_name": "-",
"offset": 527,
"prospector.type": "log"
Expand Down Expand Up @@ -155,9 +155,9 @@
"nginx.access.url": "/assets/xxxx?q=100",
"nginx.access.user_agent.device": "Other",
"nginx.access.user_agent.name": "Other",
"nginx.access.user_agent.original": "Amazon CloudFront",
"nginx.access.user_agent.os": "Other",
"nginx.access.user_agent.os_name": "Other",
"nginx.access.user_agent.raw": "Amazon CloudFront",
"nginx.access.user_name": "-",
"offset": 693,
"prospector.type": "log"
Expand Down Expand Up @@ -187,9 +187,9 @@
"nginx.access.user_agent.major": "1",
"nginx.access.user_agent.minor": "0",
"nginx.access.user_agent.name": "Facebot",
"nginx.access.user_agent.original": "Mozilla/5.0 (compatible; Facebot 1.0; https://developers.facebook.com/docs/sharing/webmasters/crawler)",
"nginx.access.user_agent.os": "Other",
"nginx.access.user_agent.os_name": "Other",
"nginx.access.user_agent.raw": "Mozilla/5.0 (compatible; Facebot 1.0; https://developers.facebook.com/docs/sharing/webmasters/crawler)",
"nginx.access.user_name": "-",
"offset": 845,
"prospector.type": "log"
Expand All @@ -208,9 +208,9 @@
"nginx.access.response_code": "400",
"nginx.access.user_agent.device": "Other",
"nginx.access.user_agent.name": "Other",
"nginx.access.user_agent.original": "-",
"nginx.access.user_agent.os": "Other",
"nginx.access.user_agent.os_name": "Other",
"nginx.access.user_agent.raw": "-",
"nginx.access.user_name": "-",
"offset": 1085,
"prospector.type": "log"
Expand Down
5 changes: 3 additions & 2 deletions filebeat/module/traefik/access/_meta/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,10 +85,11 @@
type: keyword
description: >
The name of the operating system.
- name: raw
- name: original
type: text
index: false
description: >
Raw user agent value before parsing by ingest-user-agent plugin.
Original user agent value before parsing by ingest-user-agent plugin.
- name: geoip
type: group
description: >
Expand Down
2 changes: 1 addition & 1 deletion filebeat/module/traefik/access/ingest/pipeline.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@
{
"rename": {
"field": "traefik.access.agent",
"target_field": "traefik.access.user_agent.raw"
"target_field": "traefik.access.user_agent.original"
}
},
{
Expand Down
4 changes: 2 additions & 2 deletions filebeat/module/traefik/access/test/test.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,10 @@
"traefik.access.user_agent.major": "61",
"traefik.access.user_agent.minor": "0",
"traefik.access.user_agent.name": "Chrome",
"traefik.access.user_agent.original": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36",
"traefik.access.user_agent.os": "Linux",
"traefik.access.user_agent.os_name": "Linux",
"traefik.access.user_agent.patch": "3163",
"traefik.access.user_agent.raw": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36",
"traefik.access.user_name": "-"
},
{
Expand Down Expand Up @@ -48,10 +48,10 @@
"traefik.access.user_agent.major": "61",
"traefik.access.user_agent.minor": "0",
"traefik.access.user_agent.name": "Chrome",
"traefik.access.user_agent.original": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36",
"traefik.access.user_agent.os": "Linux",
"traefik.access.user_agent.os_name": "Linux",
"traefik.access.user_agent.patch": "3163",
"traefik.access.user_agent.raw": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36",
"traefik.access.user_name": "-"
}
]