Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate beat.* to agent.* #8873

Merged
merged 1 commit into from
Nov 6, 2018
Merged

Migrate beat.* to agent.* #8873

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ https://github.com/elastic/beats/compare/v6.4.0...master[Check the HEAD diff]
*Auditbeat*

- Use `initial_scan` action for new paths. {pull}7954[7954]
- Rename beat.name to agent.type, beat.hostname to agent.hostname, beat.version to agent.version.

*Filebeat*
- Rename `fileset.name` to `event.name`.
Expand Down
36 changes: 11 additions & 25 deletions auditbeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2470,36 +2470,12 @@ Contains common beat fields available in all event types.



*`beat.name`*::
+
--
The name of the Beat sending the log messages. If the Beat name is set in the configuration file, then that value is used. If it is not set, the hostname is used. To set the Beat name, use the `name` option in the configuration file.


--

*`beat.hostname`*::
+
--
The hostname as returned by the operating system on which the Beat is running.


--

*`beat.timezone`*::
+
--
The timezone as returned by the operating system on which the Beat is running.


--

*`beat.version`*::
+
--
The version of the beat that generated this event.


--

*`@timestamp`*::
Expand Down Expand Up @@ -2797,7 +2773,7 @@ Version of the agent.

--

*`agent.name`*::
*`agent.type`*::
+
--
type: keyword
Expand All @@ -2807,6 +2783,16 @@ example: filebeat
Name of the agent.


--

*`agent.hostname`*::
+
--
type: keyword

Hostname of the agent.


--

*`agent.id`*::
Expand Down
2 changes: 1 addition & 1 deletion auditbeat/include/fields.go
View file
Open in desktop

Large diffs are not rendered by default.

15 changes: 15 additions & 0 deletions dev-tools/ecs-migration.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,18 @@
to: event.module
alias: true
copy_to: false

- from: beat.name
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bleskes This is the file I have in mind to track the migration of the fields so it can be used later to do automatic migrations.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Copy link
Contributor

@webmat webmat Nov 6, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the meaning of this structure? I'm wondering about the direction of the alias this is suggesting:

  • Does it mean field offset is type alias, and points to the data in log.offset?
    • This will be appropriate for 7.x, for backwards compatibility
  • Does it mean field log.offset is type alias, and points to the data in offset?
    • This would be appropriate if we want to start preparing for ECS in 6.x

Since we may actually do both, I think we should rename the keys of the structures slightly.

I currently read them as "old field is in from" and "new field is in to". However when you read with the strategy in mind, it reads like "alias from offset to log.offset", which is confusing.

What do you think about the following?

- old: offset
  new: log.offset
  strategy: alias # or copy_to

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I messed up the meaning of which strategy made sense in which version. Updated my comment above :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a blocker for this PR, just to be clear. Just maybe a thing to rethink in a separate PR

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The structure can still change, what is important right now is that we have a structure way of documenting it. We can focus on naming later.

to: agent.type
alias: true
copy_to: false

- from: beat.hostname
to: agent.hostname
alias: true
copy_to: false

- from: beat.version
to: agent.version
alias: true
copy_to: false
36 changes: 11 additions & 25 deletions filebeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -599,36 +599,12 @@ Contains common beat fields available in all event types.



*`beat.name`*::
+
--
The name of the Beat sending the log messages. If the Beat name is set in the configuration file, then that value is used. If it is not set, the hostname is used. To set the Beat name, use the `name` option in the configuration file.


--

*`beat.hostname`*::
+
--
The hostname as returned by the operating system on which the Beat is running.


--

*`beat.timezone`*::
+
--
The timezone as returned by the operating system on which the Beat is running.


--

*`beat.version`*::
+
--
The version of the beat that generated this event.


--

*`@timestamp`*::
Expand Down Expand Up @@ -827,7 +803,7 @@ Version of the agent.

--

*`agent.name`*::
*`agent.type`*::
+
--
type: keyword
Expand All @@ -837,6 +813,16 @@ example: filebeat
Name of the agent.


--

*`agent.hostname`*::
+
--
type: keyword

Hostname of the agent.


--

*`agent.id`*::
Expand Down
2 changes: 1 addition & 1 deletion filebeat/include/fields.go
View file
Open in desktop

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions filebeat/module/icinga/startup/test/test.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[
{
"@timestamp": "2018-11-06T08:49:49.239Z",
"@timestamp": "2018-11-06T15:26:20.085Z",
"event.dataset": "startup",
"event.module": "icinga",
"icinga.startup.facility": "cli",
Expand All @@ -10,7 +10,7 @@
"log.offset": 0
},
{
"@timestamp": "2018-11-06T08:49:49.239Z",
"@timestamp": "2018-11-06T15:26:20.085Z",
"event.dataset": "startup",
"event.module": "icinga",
"icinga.startup.facility": "cli",
Expand Down
4 changes: 2 additions & 2 deletions filebeat/tests/system/test_fields.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,6 @@ def test_beat_fields(self):

output = self.read_output()
doc = output[0]
assert doc["beat.name"] == "testShipperName"
assert doc["beat.hostname"] == socket.gethostname()
assert doc["host.name"] == "testShipperName"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be updated to "host.hostname", no?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have host.name in Beats today since 6.4 and is what is used at the moment everywhere.

assert doc["agent.hostname"] == socket.gethostname()
assert "fields" not in doc
4 changes: 2 additions & 2 deletions filebeat/tests/system/test_modules.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,11 +190,11 @@ def _test_expected_events(self, test_file, objects):

def clean_keys(obj):
# These keys are host dependent
host_keys = ["host.name", "beat.hostname", "beat.name"]
host_keys = ["host.name", "agent.hostname", "agent.type"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same: "host.name" => "host.hostname"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see above

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the intention is to stick to "host.name" in 6.x and move to "host.hostname" in 7.x?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We still need to figure that one out. This PR keeps the same behaviour as we had.

# The create timestamps area always new
time_keys = ["read_timestamp", "event.created"]
# source path and beat.version can be different for each run
other_keys = ["source", "beat.version"]
other_keys = ["source", "agent.version"]

for key in host_keys + time_keys + other_keys:
delete_key(obj, key)
Expand Down
10 changes: 5 additions & 5 deletions filebeat/tests/system/test_processors.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ def test_dropfields(self):
path=os.path.abspath(self.working_dir) + "/test.log",
processors=[{
"drop_fields": {
"fields": ["beat"],
"fields": ["agent"],
},
}]
)
Expand All @@ -30,7 +30,7 @@ def test_dropfields(self):
output = self.read_output(
required_fields=["@timestamp"],
)[0]
assert "beat.name" not in output
assert "agent.type" not in output
assert "message" in output

def test_include_fields(self):
Expand All @@ -55,7 +55,7 @@ def test_include_fields(self):
output = self.read_output(
required_fields=["@timestamp"],
)[0]
assert "beat.name" not in output
assert "agent.type" not in output
assert "message" in output

def test_drop_event(self):
Expand Down Expand Up @@ -83,7 +83,7 @@ def test_drop_event(self):
output = self.read_output(
required_fields=["@timestamp"],
)[0]
assert "beat.name" in output
assert "agent.type" in output
assert "message" in output
assert "test" in output["message"]

Expand Down Expand Up @@ -112,7 +112,7 @@ def test_condition(self):
output = self.read_output(
required_fields=["@timestamp"],
)[0]
assert "beat.name" in output
assert "agent.type" in output
assert "message" in output
assert "test" in output["message"]

Expand Down
36 changes: 11 additions & 25 deletions heartbeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,36 +34,12 @@ Contains common beat fields available in all event types.



*`beat.name`*::
+
--
The name of the Beat sending the log messages. If the Beat name is set in the configuration file, then that value is used. If it is not set, the hostname is used. To set the Beat name, use the `name` option in the configuration file.


--

*`beat.hostname`*::
+
--
The hostname as returned by the operating system on which the Beat is running.


--

*`beat.timezone`*::
+
--
The timezone as returned by the operating system on which the Beat is running.


--

*`beat.version`*::
+
--
The version of the beat that generated this event.


--

*`@timestamp`*::
Expand Down Expand Up @@ -362,7 +338,7 @@ Version of the agent.

--

*`agent.name`*::
*`agent.type`*::
+
--
type: keyword
Expand All @@ -372,6 +348,16 @@ example: filebeat
Name of the agent.


--

*`agent.hostname`*::
+
--
type: keyword

Hostname of the agent.


--

*`agent.id`*::
Expand Down
2 changes: 1 addition & 1 deletion heartbeat/include/fields.go
View file
Open in desktop

Large diffs are not rendered by default.

36 changes: 11 additions & 25 deletions journalbeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,36 +28,12 @@ Contains common beat fields available in all event types.



*`beat.name`*::
+
--
The name of the Beat sending the log messages. If the Beat name is set in the configuration file, then that value is used. If it is not set, the hostname is used. To set the Beat name, use the `name` option in the configuration file.


--

*`beat.hostname`*::
+
--
The hostname as returned by the operating system on which the Beat is running.


--

*`beat.timezone`*::
+
--
The timezone as returned by the operating system on which the Beat is running.


--

*`beat.version`*::
+
--
The version of the beat that generated this event.


--

*`@timestamp`*::
Expand Down Expand Up @@ -655,7 +631,7 @@ Version of the agent.

--

*`agent.name`*::
*`agent.type`*::
+
--
type: keyword
Expand All @@ -665,6 +641,16 @@ example: filebeat
Name of the agent.


--

*`agent.hostname`*::
+
--
type: keyword

Hostname of the agent.


--

*`agent.id`*::
Expand Down
2 changes: 1 addition & 1 deletion journalbeat/include/fields.go
View file
Open in desktop

Large diffs are not rendered by default.

14 changes: 0 additions & 14 deletions libbeat/_meta/fields.common.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,24 +4,10 @@
description: >
Contains common beat fields available in all event types.
fields:

- name: beat.name
description: >
The name of the Beat sending the log messages. If the Beat name is
set in the configuration file, then that value is used. If it is not
set, the hostname is used. To set the Beat name, use the `name`
option in the configuration file.
- name: beat.hostname
description: >
The hostname as returned by the operating system on which the Beat is
running.
- name: beat.timezone
description: >
The timezone as returned by the operating system on which the Beat is
running.
- name: beat.version
description: >
The version of the beat that generated this event.

- name: "@timestamp"
type: date
Expand Down
Loading