Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add event.ingested as the ingest timestamp #582

Merged
merged 6 commits into from
Nov 19, 2019
Merged

Add event.ingested as the ingest timestamp #582

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
ed86967
Add event.ingested as the ingest timestamp.
Oct 9, 2019
443da9b
Changelog
Oct 9, 2019
18b1d0c
spacing tweak in changelog
Nov 13, 2019
8e5023e
Flesh out the description wrt related timestamps
Nov 13, 2019
b536afc
Let's set the example by adding examples :-)
Nov 13, 2019
9f41717
Merge branch 'master' into event_ingested
Nov 19, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ Thanks, you're awesome :-) -->
### Bugfixes

### Added
* Add `event.ingested` as the ingest timestamp. #582

### Improvements

Expand Down
4 changes: 4 additions & 0 deletions code/go/ecs/event.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 11 additions & 0 deletions docs/field-details.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1222,6 +1222,17 @@ type: keyword

example: `8a4f500d`

| core

// ===============================================================

| event.ingested
| Time when the event was ingested. This is different from `@timestamp` which is when the event originally occurred.

type: date



| core

// ===============================================================
Expand Down
5 changes: 5 additions & 0 deletions generated/beats/fields.ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -987,6 +987,11 @@
ignore_above: 1024
description: Unique ID to describe the event.
example: 8a4f500d
- name: ingested
level: core
type: date
description: Time when the event was ingested. This is different from `@timestamp`
which is when the event originally occurred.
- name: kind
level: extended
type: keyword
Expand Down
1 change: 1 addition & 0 deletions generated/csv/fields.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,7 @@ event.duration,long,core,,1.2.0-dev
event.end,date,extended,,1.2.0-dev
event.hash,keyword,extended,123456789012345678901234567890ABCD,1.2.0-dev
event.id,keyword,core,8a4f500d,1.2.0-dev
event.ingested,date,core,,1.2.0-dev
event.kind,keyword,extended,state,1.2.0-dev
event.module,keyword,core,apache,1.2.0-dev
event.original,keyword,core,Sep 19 08:26:10 host CEF:0|Security| threatmanager|1.0|100| worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2spt=1232,1.2.0-dev
Expand Down
9 changes: 9 additions & 0 deletions generated/ecs/ecs_flat.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1335,6 +1335,15 @@ event.id:
order: 0
short: Unique ID to describe the event.
type: keyword
event.ingested:
description: Time when the event was ingested. This is different from `@timestamp`
which is when the event originally occurred.
flat_name: event.ingested
level: core
name: ingested
order: 21
short: Ingest timestamp
type: date
event.kind:
description: 'The kind of the event.

Expand Down
9 changes: 9 additions & 0 deletions generated/ecs/ecs_nested.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1547,6 +1547,15 @@ event:
order: 0
short: Unique ID to describe the event.
type: keyword
ingested:
description: Time when the event was ingested. This is different from `@timestamp`
which is when the event originally occurred.
flat_name: event.ingested
level: core
name: ingested
order: 21
short: Ingest timestamp
type: date
kind:
description: 'The kind of the event.

Expand Down
3 changes: 3 additions & 0 deletions generated/elasticsearch/6/template.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -564,6 +564,9 @@
"ignore_above": 1024,
"type": "keyword"
},
"ingested": {
"type": "date"
},
"kind": {
"ignore_above": 1024,
"type": "keyword"
Expand Down
3 changes: 3 additions & 0 deletions generated/elasticsearch/7/template.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -563,6 +563,9 @@
"ignore_above": 1024,
"type": "keyword"
},
"ingested": {
"type": "date"
},
"kind": {
"ignore_above": 1024,
"type": "keyword"
Expand Down
3 changes: 3 additions & 0 deletions generated/legacy/template.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -376,6 +376,9 @@
"ignore_above": 1024,
"type": "keyword"
},
"ingested": {
"type": "date"
},
"kind": {
"ignore_above": 1024,
"type": "keyword"
Expand Down
10 changes: 10 additions & 0 deletions schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -882,6 +882,16 @@
"required": false,
"type": "keyword"
},
"event.ingested": {
"description": "Time when the event was ingested. This is different from `@timestamp` which is when the event originally occurred.",
"example": "",
"footnote": "",
"group": 2,
"level": "core",
"name": "event.ingested",
"required": false,
"type": "date"
},
"event.kind": {
"description": "The kind of the event.\nThis gives information about what type of information the event contains, without being specific to the contents of the event. Examples are `event`, `state`, `alarm`. Warning: In future versions of ECS, we plan to provide a list of acceptable values for this field, please use with caution.",
"example": "state",
Expand Down
8 changes: 8 additions & 0 deletions schemas/event.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -276,3 +276,11 @@

This is mainly useful if you use more than one system that assigns
risk scores, and you want to see a normalized value across all systems.

- name: ingested
level: core
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extended?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to introduce it directly as core. From the feedback we've seen, this timestamp is considered more useful than event.created.

A few reasons why that is so:

  • It's another system's timestamp, which can help detect tampering of the clock on the monitored machine
  • It can also be used to detect slowdowns in the overall pipeline, assuming no tampering

PRs such as elastic/beats#14001 could also help populating it broadly and reliably, without having to revisit all modules or all beats.

If you have strong feelings and would really prefer to start by introducing as extended, I can go with that, in order to get this in quickly. But I think it would send the wrong message wrt to this timestamp's importance vs event.created.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll wait for your response on this, and would like to merge this tomorrow if possible

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM

type: date
short: Ingest timestamp
description: >
Time when the event was ingested. This is different from `@timestamp`
which is when the event originally occurred.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should be a little more explicit in the relationship with event.created as well here.

Here's what I'm thinking:

      short: Timestamp when an event arrived in the central data store
      description: >
        Timestamp when an event arrived in the central data store.

        This is different from `@timestamp`, which is when the event originally occurred.
        It's also different from `event.created`, which is meant to capture the first time an agent saw the event.

        In normal conditions, assuming no tampering, the timestamps should chronologically look like this: `@timestamp` < `event.created` < `event.ingested`.

Just like @timestamp and event.created, the description is pretty verbose in order to clarify the relationship. I think we should fast-track this PR and we may not necessarily need to adjust @timestamp and event.created descriptions for now.