Add event.ingested as the ingest timestamp (elastic#582)

CHANGELOG.next.md

@@ -11,6 +11,7 @@ Thanks, you're awesome :-) -->
 
 ### Added
 
+* Add `event.ingested` as the ingest timestamp. #582
 * Added `package.reference`. #585
 * Added `package.build_version`. #586
 * Added `package.type`. #587

docs/field-details.asciidoc

@@ -1159,7 +1159,7 @@ In case the two timestamps are identical, @timestamp should be used.
 
 type: date
 
-
+example: `2016-05-23 08:05:34.857000`
 
 | core
 
@@ -1226,6 +1226,21 @@ example: `8a4f500d`
 
 // ===============================================================
 
+| event.ingested
+| Timestamp when an event arrived in the central data store.
+
+This is different from `@timestamp`, which is when the event originally occurred.  It's also different from `event.created`, which is meant to capture the first time an agent saw the event.
+
+In normal conditions, assuming no tampering, the timestamps should chronologically look like this: `@timestamp` < `event.created` < `event.ingested`.
+
+type: date
+
+example: `2016-05-23 08:05:35.101000`
+
+| core
+
+// ===============================================================
+
 | event.kind
 | The kind of the event.
 

generated/beats/fields.ecs.yml

@@ -945,6 +945,7 @@
         your agent''s or pipeline''s ability to keep up with your event source.
 
         In case the two timestamps are identical, @timestamp should be used.'
+      example: 2016-05-23 08:05:34.857000
     - name: dataset
       level: core
       type: keyword
@@ -987,6 +988,18 @@
       ignore_above: 1024
       description: Unique ID to describe the event.
       example: 8a4f500d
+    - name: ingested
+      level: core
+      type: date
+      description: 'Timestamp when an event arrived in the central data store.
+
+        This is different from `@timestamp`, which is when the event originally occurred.  It''s
+        also different from `event.created`, which is meant to capture the first time
+        an agent saw the event.
+
+        In normal conditions, assuming no tampering, the timestamps should chronologically
+        look like this: `@timestamp` < `event.created` < `event.ingested`.'
+      example: 2016-05-23 08:05:35.101000
     - name: kind
       level: extended
       type: keyword

generated/csv/fields.csv

@@ -110,12 +110,13 @@ error.type,keyword,extended,java.lang.NullPointerException,1.2.0-dev
 event.action,keyword,core,user-password-change,1.2.0-dev
 event.category,keyword,core,user-management,1.2.0-dev
 event.code,keyword,extended,4648,1.2.0-dev
-event.created,date,core,,1.2.0-dev
+event.created,date,core,2016-05-23 08:05:34.857000,1.2.0-dev
 event.dataset,keyword,core,apache.access,1.2.0-dev
 event.duration,long,core,,1.2.0-dev
 event.end,date,extended,,1.2.0-dev
 event.hash,keyword,extended,123456789012345678901234567890ABCD,1.2.0-dev
 event.id,keyword,core,8a4f500d,1.2.0-dev
+event.ingested,date,core,2016-05-23 08:05:35.101000,1.2.0-dev
 event.kind,keyword,extended,state,1.2.0-dev
 event.module,keyword,core,apache,1.2.0-dev
 event.original,keyword,core,Sep 19 08:26:10 host CEF:0|Security| threatmanager|1.0|100| worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2spt=1232,1.2.0-dev

generated/ecs/ecs_flat.yml

@@ -1266,6 +1266,7 @@ event.created:
     agent''s or pipeline''s ability to keep up with your event source.
 
     In case the two timestamps are identical, @timestamp should be used.'
+  example: 2016-05-23 08:05:34.857000
   flat_name: event.created
   level: core
   name: created
@@ -1335,6 +1336,22 @@ event.id:
   order: 0
   short: Unique ID to describe the event.
   type: keyword
+event.ingested:
+  description: 'Timestamp when an event arrived in the central data store.
+
+    This is different from `@timestamp`, which is when the event originally occurred.  It''s
+    also different from `event.created`, which is meant to capture the first time
+    an agent saw the event.
+
+    In normal conditions, assuming no tampering, the timestamps should chronologically
+    look like this: `@timestamp` < `event.created` < `event.ingested`.'
+  example: 2016-05-23 08:05:35.101000
+  flat_name: event.ingested
+  level: core
+  name: ingested
+  order: 21
+  short: Timestamp when an event arrived in the central data store.
+  type: date
 event.kind:
   description: 'The kind of the event.
 

generated/ecs/ecs_nested.yml

@@ -1477,6 +1477,7 @@ event:
         your agent''s or pipeline''s ability to keep up with your event source.
 
         In case the two timestamps are identical, @timestamp should be used.'
+      example: 2016-05-23 08:05:34.857000
       flat_name: event.created
       level: core
       name: created
@@ -1547,6 +1548,22 @@ event:
       order: 0
       short: Unique ID to describe the event.
       type: keyword
+    ingested:
+      description: 'Timestamp when an event arrived in the central data store.
+
+        This is different from `@timestamp`, which is when the event originally occurred.  It''s
+        also different from `event.created`, which is meant to capture the first time
+        an agent saw the event.
+
+        In normal conditions, assuming no tampering, the timestamps should chronologically
+        look like this: `@timestamp` < `event.created` < `event.ingested`.'
+      example: 2016-05-23 08:05:35.101000
+      flat_name: event.ingested
+      level: core
+      name: ingested
+      order: 21
+      short: Timestamp when an event arrived in the central data store.
+      type: date
     kind:
       description: 'The kind of the event.
 

generated/elasticsearch/6/template.json

@@ -564,6 +564,9 @@
               "ignore_above": 1024, 
               "type": "keyword"
             }, 
+            "ingested": {
+              "type": "date"
+            }, 
             "kind": {
               "ignore_above": 1024, 
               "type": "keyword"

generated/elasticsearch/7/template.json

@@ -563,6 +563,9 @@
             "ignore_above": 1024, 
             "type": "keyword"
           }, 
+          "ingested": {
+            "type": "date"
+          }, 
           "kind": {
             "ignore_above": 1024, 
             "type": "keyword"

generated/legacy/template.json

@@ -376,6 +376,9 @@
               "ignore_above": 1024,
               "type": "keyword"
             },
+            "ingested": {
+              "type": "date"
+            },
             "kind": {
               "ignore_above": 1024,
               "type": "keyword"

schema.json

@@ -824,7 +824,7 @@
       }, 
       "event.created": {
         "description": "event.created contains the date/time when the event was first read by an agent, or by your pipeline.\nThis field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event.\nIn most situations, these two timestamps will be slightly different. The difference can be used to calculate the delay between your source generating an event, and the time when your agent first processed it. This can be used to monitor your agent's or pipeline's ability to keep up with your event source.\nIn case the two timestamps are identical, @timestamp should be used.", 
-        "example": "", 
+        "example": "2016-05-23 08:05:34.857000", 
         "footnote": "", 
         "group": 2, 
         "level": "core", 
@@ -882,6 +882,16 @@
         "required": false, 
         "type": "keyword"
       }, 
+      "event.ingested": {
+        "description": "Timestamp when an event arrived in the central data store.\nThis is different from `@timestamp`, which is when the event originally occurred.  It's also different from `event.created`, which is meant to capture the first time an agent saw the event.\nIn normal conditions, assuming no tampering, the timestamps should chronologically look like this: `@timestamp` < `event.created` < `event.ingested`.", 
+        "example": "2016-05-23 08:05:35.101000", 
+        "footnote": "", 
+        "group": 2, 
+        "level": "core", 
+        "name": "event.ingested", 
+        "required": false, 
+        "type": "date"
+      }, 
       "event.kind": {
         "description": "The kind of the event.\nThis gives information about what type of information the event contains, without being specific to the contents of the event.  Examples are `event`, `state`, `alarm`. Warning: In future versions of ECS, we plan to provide a list of acceptable values for this field, please use with caution.", 
         "example": "state", 

schemas/event.yml

@@ -231,6 +231,7 @@
       level: core
       type: date
       short: Time when the event was first read by an agent or by your pipeline.
+      example: 2016-05-23T08:05:34.857Z
       description: >
         event.created contains the date/time when the event was first read by an
         agent, or by your pipeline.
@@ -276,3 +277,18 @@
 
         This is mainly useful if you use more than one system that assigns
         risk scores, and you want to see a normalized value across all systems.
+
+    - name: ingested
+      level: core
+      type: date
+      short: Timestamp when an event arrived in the central data store.
+      example: 2016-05-23T08:05:35.101Z
+      description: >
+        Timestamp when an event arrived in the central data store.
+
+        This is different from `@timestamp`, which is when the event originally
+        occurred.  It's also different from `event.created`, which is meant
+        to capture the first time an agent saw the event.
+
+        In normal conditions, assuming no tampering, the timestamps should
+        chronologically look like this: `@timestamp` < `event.created` < `event.ingested`.