-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Haproxy filebeat module tcp and default formats #8637
Merged
sayden
merged 4 commits into
elastic:master
from
sayden:haproxy-filebeat-module-tcp-and-default-formats
Oct 23, 2018
Merged
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,52 +1,71 @@ | ||
| { | ||
| "description": "Pipeline for parsing HAProxy http logs in their default format. Requires the geoip plugin.", | ||
| "processors": [{ | ||
| "grok": { | ||
| "field": "message", | ||
| "patterns": [ | ||
| "(%{NOTSPACE:haproxy.process_name}\\[%{NUMBER:haproxy.pid:int}\\]: )?%{IP:haproxy.client_ip}:%{NUMBER:haproxy.client_port:int} \\[%{NOTSPACE:haproxy.http.request_date}\\] %{NOTSPACE:haproxy.frontend_name} %{NOTSPACE:haproxy.backend_name}/%{NOTSPACE:haproxy.server_name} %{NUMBER:haproxy.time_client_req:int}/%{NUMBER:haproxy.time_queue:int}/%{NUMBER:haproxy.time_backend_connect:int}/%{NUMBER:haproxy.time_server_response:int}/%{NUMBER:haproxy.time_duration:int} %{NUMBER:haproxy.http.response.status_code:int} %{NUMBER:haproxy.http.response.bytes_read:int} %{NOTSPACE:haproxy.http.request.captured_cookie} %{NOTSPACE:haproxy.http.response.captured_cookie} %{NOTSPACE:haproxy.termination_state} %{NUMBER:haproxy.connections.active:int}/%{NUMBER:haproxy.connections.frontend:int}/%{NUMBER:haproxy.connections.backend:int}/%{NUMBER:haproxy.connections.server:int}/%{NUMBER:haproxy.connections.retries:int} %{NUMBER:haproxy.server_queue:int}/%{NUMBER:haproxy.backend_queue:int} \\{%{DATA:haproxy.http.request.captured_headers}\\} \\{%{DATA:haproxy.http.response.captured_headers}\\} \"%{GREEDYDATA:haproxy.http.request.raw_request_line}\"", | ||
| "(%{NOTSPACE:haproxy.process_name}\\[%{NUMBER:haproxy.pid:int}\\]: )?%{IP:haproxy.client_ip}:%{NUMBER:haproxy.client_port:int} \\[%{NOTSPACE:haproxy.http.request_date}\\] %{NOTSPACE:haproxy.frontend_name}/%{NOTSPACE:haproxy.bind_name} %{GREEDYDATA:haproxy.error_message}" | ||
| ], | ||
| "ignore_missing": false | ||
| } | ||
| }, | ||
| { | ||
| "date": { | ||
| "field": "haproxy.http.request_date", | ||
| "target_field": "@timestamp", | ||
| "formats": ["dd/MMM/yyyy:HH:mm:ss.SSS"] | ||
| } | ||
| }, | ||
| { | ||
| "remove": { | ||
| "field": "haproxy.http.request_date" | ||
| } | ||
| }, | ||
| { | ||
| "geoip": { | ||
| "field": "haproxy.client_ip", | ||
| "target_field": "haproxy.geoip" | ||
| } | ||
| }, | ||
| { | ||
| "split": { | ||
| "field": "haproxy.http.request.captured_headers", | ||
| "separator": "\\|", | ||
| "ignore_failure": true | ||
| } | ||
| }, | ||
| { | ||
| "split": { | ||
| "field": "haproxy.http.response.captured_headers", | ||
| "separator": "\\|", | ||
| "ignore_failure": true | ||
| } | ||
| } | ||
| ], | ||
| "on_failure" : [{ | ||
| "set" : { | ||
| "field" : "error.message", | ||
| "value" : "{{ _ingest.on_failure_message }}" | ||
| } | ||
| }] | ||
| } | ||
| "description": "Pipeline for parsing HAProxy http logs in their default format. Requires the geoip plugin.", | ||
| "processors": [ | ||
| { | ||
| "grok": { | ||
| "field": "message", | ||
| "patterns": [ | ||
| "%{HAPROXY_DATE:haproxy.request_date} %{IPORHOST:haproxy.source} %{PROG:haproxy.process_name}(?:\\[%{POSINT:haproxy.pid}\\])?: %{GREEDYDATA} %{IPORHOST:haproxy.client.ip}:%{POSINT:haproxy.client.port} %{WORD} %{IPORHOST:haproxy.destination.ip}:%{POSINT:haproxy.destination.port} \\(%{WORD:haproxy.frontend_name}/%{WORD:haproxy.default.mode}\\)", | ||
|
|
||
| "(%{NOTSPACE:haproxy.process_name}\\[%{NUMBER:haproxy.pid:int}\\]: )?%{IP:haproxy.client.ip}:%{NUMBER:haproxy.client.port:int} \\[%{NOTSPACE:haproxy.request_date}\\] %{NOTSPACE:haproxy.frontend_name} %{NOTSPACE:haproxy.backend_name}/%{NOTSPACE:haproxy.server_name} %{NUMBER:haproxy.http.request.time_wait_ms:int}/%{NUMBER:haproxy.total_waiting_time_ms:int}/%{NUMBER:haproxy.connection_wait_time_ms:int}/%{NUMBER:haproxy.http.request.time_wait_without_data_ms:int}/%{NUMBER:haproxy.http.request.time_active_ms:int} %{NUMBER:haproxy.http.response.status_code:int} %{NUMBER:haproxy.bytes_read:int} %{NOTSPACE:haproxy.http.request.captured_cookie} %{NOTSPACE:haproxy.http.response.captured_cookie} %{NOTSPACE:haproxy.termination_state} %{NUMBER:haproxy.connections.active:int}/%{NUMBER:haproxy.connections.frontend:int}/%{NUMBER:haproxy.connections.backend:int}/%{NUMBER:haproxy.connections.server:int}/%{NUMBER:haproxy.connections.retries:int} %{NUMBER:haproxy.server_queue:int}/%{NUMBER:haproxy.backend_queue:int} \\{%{DATA:haproxy.http.request.captured_headers}\\} \\{%{DATA:haproxy.http.response.captured_headers}\\} \"%{GREEDYDATA:haproxy.http.request.raw_request_line}\"", | ||
|
|
||
| "(%{NOTSPACE:haproxy.process_name}\\[%{NUMBER:haproxy.pid:int}\\]: )?%{IP:haproxy.client.ip}:%{NUMBER:haproxy.client.port:int} \\[%{NOTSPACE:haproxy.request_date}\\] %{NOTSPACE:haproxy.frontend_name}/%{NOTSPACE:haproxy.bind_name} %{GREEDYDATA:haproxy.error_message}", | ||
|
|
||
| "%{HAPROXY_DATE} %{IPORHOST:haproxy.source} (%{NOTSPACE:haproxy.process_name}\\[%{NUMBER:haproxy.pid:int}\\]: )?%{IP:haproxy.client.ip}:%{NUMBER:haproxy.client.port:int} \\[%{NOTSPACE:haproxy.request_date}\\] %{NOTSPACE:haproxy.frontend_name} %{NOTSPACE:haproxy.backend_name}/%{NOTSPACE:haproxy.server_name} %{NUMBER:haproxy.total_waiting_time_ms:int}/%{NUMBER:haproxy.connection_wait_time_ms:int}/%{NUMBER:haproxy.tcp.processing_time_ms:int} %{NUMBER:haproxy.bytes_read:int} %{NOTSPACE:haproxy.termination_state} %{NUMBER:haproxy.connections.active:int}/%{NUMBER:haproxy.connections.frontend:int}/%{NUMBER:haproxy.connections.backend:int}/%{NUMBER:haproxy.connections.server:int}/%{NUMBER:haproxy.connections.retries:int} %{NUMBER:haproxy.server_queue:int}/%{NUMBER:haproxy.backend_queue:int}" | ||
| ], | ||
| "ignore_missing": false, | ||
| "pattern_definitions": { | ||
| "HAPROXY_DATE": "(%{MONTHDAY}[/-]%{MONTH}[/-]%{YEAR}:%{HOUR}:%{MINUTE}:%{SECOND})|%{SYSLOGTIMESTAMP}" | ||
| } | ||
| } | ||
| }, | ||
| { | ||
| "date": { | ||
| "field": "haproxy.request_date", | ||
| "target_field": "@timestamp", | ||
| "formats": [ | ||
| "dd/MMM/yyyy:HH:mm:ss.SSS", | ||
| "MMM dd HH:mm:ss" | ||
| ] | ||
| } | ||
| }, | ||
| { | ||
| "remove": { | ||
| "field": "haproxy.request_date" | ||
| } | ||
| }, | ||
| { | ||
| "remove": { | ||
| "field": "message" | ||
| } | ||
| }, | ||
| { | ||
| "geoip": { | ||
| "field": "haproxy.client.ip", | ||
| "target_field": "haproxy.geoip" | ||
| } | ||
| }, | ||
| { | ||
| "split": { | ||
| "field": "haproxy.http.request.captured_headers", | ||
| "separator": "\\|", | ||
| "ignore_failure": true | ||
| } | ||
| }, | ||
| { | ||
| "split": { | ||
| "field": "haproxy.http.response.captured_headers", | ||
| "separator": "\\|", | ||
| "ignore_failure": true | ||
| } | ||
| } | ||
| ], | ||
| "on_failure": [ | ||
| { | ||
| "set": { | ||
| "field": "error.message", | ||
| "value": "{{ _ingest.on_failure_message }}" | ||
| } | ||
| } | ||
| ] | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| Sep 20 15:42:59 1.2.3.4 haproxy[24551]: Connect from 1.2.3.4:40780 to 1.2.3.4:5000 (main/HTTP) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this can also be common to multiple log formats, remove it too from the default namespace, and it can be set to
tcpon the tcp log format.A possible name for this could be
haproxy.protocol, ornetwork.protocolas of ECS.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's not use
network.protocolyet from ECS as it still might change.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As far as I can read in the HAProxy documentation,
modeis only specific for Default format https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#8.2.1 In TCP format there is no "mode" at all (mode is always TCP).About the naming, I thought that we already agreed that we were going to leave ECS apart in this PR and focus on maintain naming of the service, which is the most familiar naming for an HAProxy user who could use this module.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes,
modeseems to be only specified in the default format, it wouldn't make sense in protocol-specific formats, but it can still be useful in events so users can differentiate logs from tcp and http connections.We can set
haproxy.protocolto the value of mode for the default formats, and for protocol-specific formats set it to the specific protocol depending on the matching pattern. I'm not sure how to set a field depending on the matching pattern, we can leave it for a future change, but I'd use a common name likehaproxy.protocolinstead ofhaproxy.default.modein any case, thinking in the future uses of this field.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we are spending too much time and energy in a field that only appears (and gets parsed) in a deprecated log format.
I feel like writing complex conditional parsing for non-straightforward (invented) fields which doesn't usually appear in HAProxy and to get something that we haven't measure and that it has not come from a user requirements is over-engineering things a bit
WDYT?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, as I said we can leave adding this field in other formats for future changes. I was just asking about using a generic name like
haproxy.protocol(instead ofhaproxy.default.mode), no overengineering, just using a more future-proof name, wdyt about this?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Or
haproxy.modeif you want to keep haproxy naming, but without thedefaultnamespacing.