Added top_level_domain (#572, #542)

Added at: - client.top_level_domain - destination.top_level_domain - dns.question.top_level_domain - server.top_level_domain - source.top_level_domain - url.top_level_domain
elastic · Oct 3, 2019 · 9c4f380 · 9c4f380
1 parent 204d4de
commit 9c4f380
Show file tree

Hide file tree

Showing 22 changed files with 678 additions and 58 deletions.
diff --git a/CHANGELOG.next.md b/CHANGELOG.next.md
@@ -13,6 +13,8 @@ Thanks, you're awesome :-) -->
 * Add support for installed software packages. #532
 
 * Added fields in `log.*` to allow for full Syslog mapping. #525
+* Added `top_level_domain` field to `url`, `dns.question`,
+    `source`, `destination`, `client`, and `server`. #562, #572
 * Add group.domain field #547
 * Added `observer.name` and `observer.product` #557, #571
 * Added `error.stack_trace` field. #562

diff --git a/code/go/ecs/client.go b/code/go/ecs/client.go
diff --git a/code/go/ecs/destination.go b/code/go/ecs/destination.go
diff --git a/code/go/ecs/dns.go b/code/go/ecs/dns.go
diff --git a/code/go/ecs/server.go b/code/go/ecs/server.go
diff --git a/code/go/ecs/source.go b/code/go/ecs/source.go
diff --git a/code/go/ecs/url.go b/code/go/ecs/url.go
diff --git a/docs/field-details.asciidoc b/docs/field-details.asciidoc
@@ -339,6 +339,19 @@ example: `google.com`
 
 // ===============================================================
 
+| client.top_level_domain
+| The effective top level domain (eTLD), also known as the domain suffix, is the last part of the domain name. For example, the top level domain for google.com is "com".
+
+This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last label will not work well for effective TLDs such as "co.uk".
+
+type: keyword
+
+example: `co.uk`
+
+| extended
+
+// ===============================================================
+
 |=====
 
 ==== Field Reuse
@@ -690,6 +703,19 @@ example: `google.com`
 
 // ===============================================================
 
+| destination.top_level_domain
+| The effective top level domain (eTLD), also known as the domain suffix, is the last part of the domain name. For example, the top level domain for google.com is "com".
+
+This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last label will not work well for effective TLDs such as "co.uk".
+
+type: keyword
+
+example: `co.uk`
+
+| extended
+
+// ===============================================================
+
 |=====
 
 ==== Field Reuse
@@ -890,6 +916,19 @@ example: `google.com`
 
 // ===============================================================
 
+| dns.question.top_level_domain
+| The effective top level domain (eTLD), also known as the domain suffix, is the last part of the domain name. For example, the top level domain for google.com is "com".
+
+This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last label will not work well for effective TLDs such as "co.uk".
+
+type: keyword
+
+example: `co.uk`
+
+| extended
+
+// ===============================================================
+
 | dns.question.type
 | The type of record being queried.
 
@@ -3191,6 +3230,19 @@ example: `google.com`
 
 // ===============================================================
 
+| server.top_level_domain
+| The effective top level domain (eTLD), also known as the domain suffix, is the last part of the domain name. For example, the top level domain for google.com is "com".
+
+This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last label will not work well for effective TLDs such as "co.uk".
+
+type: keyword
+
+example: `co.uk`
+
+| extended
+
+// ===============================================================
+
 |=====
 
 ==== Field Reuse
@@ -3479,6 +3531,19 @@ example: `google.com`
 
 // ===============================================================
 
+| source.top_level_domain
+| The effective top level domain (eTLD), also known as the domain suffix, is the last part of the domain name. For example, the top level domain for google.com is "com".
+
+This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last label will not work well for effective TLDs such as "co.uk".
+
+type: keyword
+
+example: `co.uk`
+
+| extended
+
+// ===============================================================
+
 |=====
 
 ==== Field Reuse
@@ -3790,6 +3855,19 @@ example: `https`
 
 // ===============================================================
 
+| url.top_level_domain
+| The effective top level domain (eTLD), also known as the domain suffix, is the last part of the domain name. For example, the top level domain for google.com is "com".
+
+This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last label will not work well for effective TLDs such as "co.uk".
+
+type: keyword
+
+example: `co.uk`
+
+| extended
+
+// ===============================================================
+
 | url.username
 | Username of the request.
 

diff --git a/generated/beats/fields.ecs.yml b/generated/beats/fields.ecs.yml
@@ -285,6 +285,18 @@
         list (http://publicsuffix.org). Trying to approximate this by simply taking
         the last two labels will not work well for TLDs such as "co.uk".'
       example: google.com
+    - name: top_level_domain
+      level: extended
+      type: keyword
+      ignore_above: 1024
+      description: 'The effective top level domain (eTLD), also known as the domain
+        suffix, is the last part of the domain name. For example, the top level domain
+        for google.com is "com".
+
+        This value can be determined precisely with a list like the public suffix
+        list (http://publicsuffix.org). Trying to approximate this by simply taking
+        the last label will not work well for effective TLDs such as "co.uk".'
+      example: co.uk
     - name: user.domain
       level: extended
       type: keyword
@@ -578,6 +590,18 @@
         list (http://publicsuffix.org). Trying to approximate this by simply taking
         the last two labels will not work well for TLDs such as "co.uk".'
       example: google.com
+    - name: top_level_domain
+      level: extended
+      type: keyword
+      ignore_above: 1024
+      description: 'The effective top level domain (eTLD), also known as the domain
+        suffix, is the last part of the domain name. For example, the top level domain
+        for google.com is "com".
+
+        This value can be determined precisely with a list like the public suffix
+        list (http://publicsuffix.org). Trying to approximate this by simply taking
+        the last label will not work well for effective TLDs such as "co.uk".'
+      example: co.uk
     - name: user.domain
       level: extended
       type: keyword
@@ -749,6 +773,18 @@
         list (http://publicsuffix.org). Trying to approximate this by simply taking
         the last two labels will not work well for TLDs such as "co.uk".'
       example: google.com
+    - name: question.top_level_domain
+      level: extended
+      type: keyword
+      ignore_above: 1024
+      description: 'The effective top level domain (eTLD), also known as the domain
+        suffix, is the last part of the domain name. For example, the top level domain
+        for google.com is "com".
+
+        This value can be determined precisely with a list like the public suffix
+        list (http://publicsuffix.org). Trying to approximate this by simply taking
+        the last label will not work well for effective TLDs such as "co.uk".'
+      example: co.uk
     - name: question.type
       level: extended
       type: keyword
@@ -2390,6 +2426,18 @@
         list (http://publicsuffix.org). Trying to approximate this by simply taking
         the last two labels will not work well for TLDs such as "co.uk".'
       example: google.com
+    - name: top_level_domain
+      level: extended
+      type: keyword
+      ignore_above: 1024
+      description: 'The effective top level domain (eTLD), also known as the domain
+        suffix, is the last part of the domain name. For example, the top level domain
+        for google.com is "com".
+
+        This value can be determined precisely with a list like the public suffix
+        list (http://publicsuffix.org). Trying to approximate this by simply taking
+        the last label will not work well for effective TLDs such as "co.uk".'
+      example: co.uk
     - name: user.domain
       level: extended
       type: keyword
@@ -2674,6 +2722,18 @@
         list (http://publicsuffix.org). Trying to approximate this by simply taking
         the last two labels will not work well for TLDs such as "co.uk".'
       example: google.com
+    - name: top_level_domain
+      level: extended
+      type: keyword
+      ignore_above: 1024
+      description: 'The effective top level domain (eTLD), also known as the domain
+        suffix, is the last part of the domain name. For example, the top level domain
+        for google.com is "com".
+
+        This value can be determined precisely with a list like the public suffix
+        list (http://publicsuffix.org). Trying to approximate this by simply taking
+        the last label will not work well for effective TLDs such as "co.uk".'
+      example: co.uk
     - name: user.domain
       level: extended
       type: keyword
@@ -2915,6 +2975,18 @@
 
         Note: The `:` is not part of the scheme.'
       example: https
+    - name: top_level_domain
+      level: extended
+      type: keyword
+      ignore_above: 1024
+      description: 'The effective top level domain (eTLD), also known as the domain
+        suffix, is the last part of the domain name. For example, the top level domain
+        for google.com is "com".
+
+        This value can be determined precisely with a list like the public suffix
+        list (http://publicsuffix.org). Trying to approximate this by simply taking
+        the last label will not work well for effective TLDs such as "co.uk".'
+      example: co.uk
     - name: username
       level: extended
       type: keyword

diff --git a/generated/csv/fields.csv b/generated/csv/fields.csv
@@ -30,6 +30,7 @@ client.nat.port,long,extended,,1.2.0-dev
 client.packets,long,core,12,1.2.0-dev
 client.port,long,core,,1.2.0-dev
 client.registered_domain,keyword,extended,google.com,1.2.0-dev
+client.top_level_domain,keyword,extended,co.uk,1.2.0-dev
 client.user.domain,keyword,extended,,1.2.0-dev
 client.user.email,keyword,extended,,1.2.0-dev
 client.user.full_name,keyword,extended,Albert Einstein,1.2.0-dev
@@ -72,6 +73,7 @@ destination.nat.port,long,extended,,1.2.0-dev
 destination.packets,long,core,12,1.2.0-dev
 destination.port,long,core,,1.2.0-dev
 destination.registered_domain,keyword,extended,google.com,1.2.0-dev
+destination.top_level_domain,keyword,extended,co.uk,1.2.0-dev
 destination.user.domain,keyword,extended,,1.2.0-dev
 destination.user.email,keyword,extended,,1.2.0-dev
 destination.user.full_name,keyword,extended,Albert Einstein,1.2.0-dev
@@ -93,6 +95,7 @@ dns.op_code,keyword,extended,QUERY,1.2.0-dev
 dns.question.class,keyword,extended,IN,1.2.0-dev
 dns.question.name,keyword,extended,www.google.com,1.2.0-dev
 dns.question.registered_domain,keyword,extended,google.com,1.2.0-dev
+dns.question.top_level_domain,keyword,extended,co.uk,1.2.0-dev
 dns.question.type,keyword,extended,AAAA,1.2.0-dev
 dns.resolved_ip,ip,extended,"['10.10.10.10', '10.10.10.11']",1.2.0-dev
 dns.response_code,keyword,extended,NOERROR,1.2.0-dev
@@ -303,6 +306,7 @@ server.nat.port,long,extended,,1.2.0-dev
 server.packets,long,core,12,1.2.0-dev
 server.port,long,core,,1.2.0-dev
 server.registered_domain,keyword,extended,google.com,1.2.0-dev
+server.top_level_domain,keyword,extended,co.uk,1.2.0-dev
 server.user.domain,keyword,extended,,1.2.0-dev
 server.user.email,keyword,extended,,1.2.0-dev
 server.user.full_name,keyword,extended,Albert Einstein,1.2.0-dev
@@ -339,6 +343,7 @@ source.nat.port,long,extended,,1.2.0-dev
 source.packets,long,core,12,1.2.0-dev
 source.port,long,core,,1.2.0-dev
 source.registered_domain,keyword,extended,google.com,1.2.0-dev
+source.top_level_domain,keyword,extended,co.uk,1.2.0-dev
 source.user.domain,keyword,extended,,1.2.0-dev
 source.user.email,keyword,extended,,1.2.0-dev
 source.user.full_name,keyword,extended,Albert Einstein,1.2.0-dev
@@ -367,6 +372,7 @@ url.port,long,extended,443,1.2.0-dev
 url.query,keyword,extended,,1.2.0-dev
 url.registered_domain,keyword,extended,google.com,1.2.0-dev
 url.scheme,keyword,extended,https,1.2.0-dev
+url.top_level_domain,keyword,extended,co.uk,1.2.0-dev
 url.username,keyword,extended,,1.2.0-dev
 user.domain,keyword,extended,,1.2.0-dev
 user.email,keyword,extended,,1.2.0-dev