[Filebeat] Add ThreatQuotient to Threat Intel Module elastic#27423 (#…

…28314) * [Filebeat] Add ThreatQuotient to Threat Intel Module elastic#27423 * generating golden files * updating pipeline, adding some more configuration options and such * updating dashboard import, and adding filter to dashboard * mage update * update docs and add image * Update CHANGELOG.next.asciidoc Co-authored-by: Marc Guasch <[email protected]> Co-authored-by: Marius Iversen <[email protected]> Co-authored-by: Marc Guasch <[email protected]>
elastic · Oct 19, 2021 · 0971fe7 · 0971fe7
1 parent 04ce8a5
commit 0971fe7
Show file tree

Hide file tree

Showing 24 changed files with 2,120 additions and 7 deletions.
diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
@@ -95,6 +95,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Remove deprecated fields in Kafka module. {pull}27938[27938]
 - Remove deprecated fields in coredns module. {pull}28196[28196]
 - Remove old `httpjson` config implementation. {pull}28054[28054]
+- Added dataset `threatq` to the `threatintel` module to ingest indicators from ThreatQ {issue}27423[27423]
 
 *Heartbeat*
 

diff --git a/filebeat/docs/fields.asciidoc b/filebeat/docs/fields.asciidoc
@@ -151786,6 +151786,103 @@ type: long
 
 --
 
+[float]
+=== threatq
+
+Fields for ThreatQ Threat Library
+
+
+
+*`threatintel.threatq.updated_at`*::
++
+--
+Last modification time
+
+
+type: date
+
+--
+
+*`threatintel.threatq.created_at`*::
++
+--
+Object creation time
+
+
+type: date
+
+--
+
+*`threatintel.threatq.expires_at`*::
++
+--
+Expiration time
+
+
+type: date
+
+--
+
+*`threatintel.threatq.expires_calculated_at`*::
++
+--
+Expiration calculation time
+
+
+type: date
+
+--
+
+*`threatintel.threatq.published_at`*::
++
+--
+Object publication time
+
+
+type: date
+
+--
+
+*`threatintel.threatq.status`*::
++
+--
+Object status within the Threat Library
+
+
+type: keyword
+
+--
+
+*`threatintel.threatq.indicator_value`*::
++
+--
+Original indicator value
+
+
+type: keyword
+
+--
+
+*`threatintel.threatq.adversaries`*::
++
+--
+Adversaries that are linked to the object
+
+
+type: keyword
+
+--
+
+*`threatintel.threatq.attributes`*::
++
+--
+These provide additional context about an object
+
+
+type: flattened
+
+--
+
 [[exported-fields-tomcat]]
 == Apache Tomcat fields
 

diff --git a/filebeat/docs/images/filebeat-threatintel-threatq.png b/filebeat/docs/images/filebeat-threatintel-threatq.png
diff --git a/filebeat/docs/modules/threatintel.asciidoc b/filebeat/docs/modules/threatintel.asciidoc
@@ -30,6 +30,7 @@ The available filesets are:
 * <<anomali,anomali>>: Supports gathering threat intel attributes from Anomali Limo.
 * <<anomalithreatstream,anomalithreatstream>>: Supports gathering threat intel attributes from Anomali ThreatStream.
 * <<recordedfuture,recordedfuture>>: Supports gathering threat intel attributes from Recorded Future.
+* <<threatq,threatq>>: Supports gathering threat intel attributes from ThreatQuotient.
 
 include::../include/gs-link.asciidoc[]
 
@@ -627,6 +628,124 @@ image::./images/filebeat-threatintel-recordedfuture.png[]
 [float]
 Overview of the information provided by the Recorded Future feed.
 
+[[threatq]]
+[float]
+==== `threatq` fileset settings
+
+The `threatq` fileset fetches intelligence from the ThreatQuotient API.
+
+The ThreatQ module requires you to set a valid URL, combination of Oauth2 credentials and the ID of the collection to retrieve
+indicators from.
+By default the indicators will be collected every 1 minute, and deduplication is handled by the API itself.
+
+Sample configuration:
+[source,yaml]
+----
+- module: threatintel
+  threatq:
+    enabled: true
+    var.input: httpjson
+    var.host: https://testurl.threatq.com/
+    var.token_url: https://testurl.threatq.com/api/token
+    var.client_id: oauthclient
+    var.client_secret: 123abcd
+    var.interval: 1m
+    var.data_collection_id: "fsd2f54fsg2sf"
+----
+
+*`var.url`*::
+
+The URL of the API endpoint to connect with.
+
+*`var.client_id`*::
+
+The Oauth2 client ID to be used for authentication.
+
+*`var.client_secret`*::
+
+The Oauth2 secret related to the client_id.
+
+*`var.interval`*::
+
+How often the API is polled for updated information.
+
+*`var.proxy_url`*::
+
+Optional URL to use as HTTP proxy.
+
+*`var.http_client_timeout`*::
+
+Optional value to override the default HTTP timeout of 30 seconds.
+
+
+Recorded Future fields are mapped to the following ECS fields:
+
+[options="header"]
+|=============================================================
+| ThreatQ fields         | ECS Fields
+| type.name              | threatintel.indicator.type
+| description            | threatintel.indicator.description
+| score                  | threatintel.indicator.confidence
+| value                  | threatintel.indicator.{url,ip,domain,file.hash}
+| sources                | threatintel.indicator.provider
+|=============================================================
+
+:has-dashboards!:
+
+[float]
+=== Dashboards
+
+This module comes with dashboards for the threat information feeds.
+
+[role="screenshot"]
+image::./images/filebeat-threatintel-overview.png[]
+
+[float]
+Overview of the information provided, and the health of, the Threat Intel module.
+
+[role="screenshot"]
+image::./images/filebeat-threatintel-abuse-malware.png[]
+
+[float]
+Overview of the information provided by the Abuse.ch Malware feed.
+
+[role="screenshot"]
+image::./images/filebeat-threatintel-abuse-url.png[]
+
+[float]
+Overview of the information provided by the Abuse.ch URL feed.
+
+[role="screenshot"]
+image::./images/filebeat-threatintel-alienvault-otx.png[]
+
+[float]
+Overview of the information provided by the AlienVault OTX feed.
+
+[role="screenshot"]
+image::./images/filebeat-threatintel-anomali.png[]
+
+[float]
+Overview of the information provided by the Anomali Limo and
+Anomali ThreatStream feeds.
+
+[role="screenshot"]
+image::./images/filebeat-threatintel-misp.png[]
+
+[float]
+Overview of the information provided by the MISP feed.
+
+[role="screenshot"]
+image::./images/filebeat-threatintel-recordedfuture.png[]
+
+[float]
+Overview of the information provided by the Recorded Future feed.
+
+[role="screenshot"]
+image::./images/filebeat-threatintel-threatq.png[]
+
+[float]
+Overview of the information provided by the ThreatQuotient feed.
+
 :modulename!:
 
 

diff --git a/x-pack/filebeat/filebeat.reference.yml b/x-pack/filebeat/filebeat.reference.yml
@@ -2139,7 +2139,7 @@ filebeat.modules:
     var.listen_port: 8080
 
     # Secret key to authenticate requests from the SDK.
-    var.secret: '<Add your secret here>'
+    var.secret: "<Add your secret here>"
 
     # Uncomment the following and set the absolute paths
     # to the server SSL certificate and private key to
@@ -2180,6 +2180,37 @@ filebeat.modules:
     # Set your API Token.
     var.api_token: "<RF_TOKEN>"
 
+  threatq:
+    enabled: false
+
+    # Input used for ingesting threat intel data
+    var.input: httpjson
+
+    # The URL used for ThreatQ ThreatLibrary API calls.
+    # Remember to put a slash at the end of the host URL
+    var.host: https://www.threatq.com/
+
+    # Oauth 2.0 Access Token URL
+    var.token_url: https://www.threatq.com/api/token
+
+    # Oauth 2.0 Client ID
+    var.client_id: "INSERT_CLIENT_ID"
+
+    # Oauth 2.0 Client Secret
+    var.client_secret: "INSERT_CLIENT_SECRET"
+
+    # The interval to poll the API for updates
+    var.interval: 1m
+
+    # The ID for the ThreatQ smart data collection
+    var.data_collection_id: "INSERT_THREATQ_DATA_COLLECTION_ID"
+
+    # The URL of the proxy if used
+    #var.proxy_url: http://proxy:8000
+
+    # Customize the HTTP timeout configured for the API requests
+    #var.http_client_timeout: 30s
+
 #---------------------------- Apache Tomcat Module ----------------------------
 - module: tomcat
   log:

diff --git a/x-pack/filebeat/module/threatintel/_meta/config.yml b/x-pack/filebeat/module/threatintel/_meta/config.yml
@@ -129,7 +129,7 @@
     var.listen_port: 8080
 
     # Secret key to authenticate requests from the SDK.
-    var.secret: '<Add your secret here>'
+    var.secret: "<Add your secret here>"
 
     # Uncomment the following and set the absolute paths
     # to the server SSL certificate and private key to
@@ -169,3 +169,34 @@
 
     # Set your API Token.
     var.api_token: "<RF_TOKEN>"
+
+  threatq:
+    enabled: false
+
+    # Input used for ingesting threat intel data
+    var.input: httpjson
+
+    # The URL used for ThreatQ ThreatLibrary API calls.
+    # Remember to put a slash at the end of the host URL
+    var.host: https://www.threatq.com/
+
+    # Oauth 2.0 Access Token URL
+    var.token_url: https://www.threatq.com/api/token
+
+    # Oauth 2.0 Client ID
+    var.client_id: "INSERT_CLIENT_ID"
+
+    # Oauth 2.0 Client Secret
+    var.client_secret: "INSERT_CLIENT_SECRET"
+
+    # The interval to poll the API for updates
+    var.interval: 1m
+
+    # The ID for the ThreatQ smart data collection
+    var.data_collection_id: "INSERT_THREATQ_DATA_COLLECTION_ID"
+
+    # The URL of the proxy if used
+    #var.proxy_url: http://proxy:8000
+
+    # Customize the HTTP timeout configured for the API requests
+    #var.http_client_timeout: 30s