Fix missing field in transforms of ti packages (elastic#11008)

Issue detected in elastic#10919, there is a field in transforms that is not documented.

This was only detected when enabling synthetic source (through logsdb), because
only in this case we check fields that are not in the source, but are in the index,
such as constant_keywords.

Not having these fields defined will be a problem when we enable testing with
logsdb, and/or when we fix validation of these fields.

packages/ti_abusech/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "2.3.2"
+  changes:
+    - description: Add missing fields in transform
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/11008
 - version: "2.3.1"
   changes:
     - description: Fix ECS date mapping on threat fields.

packages/ti_abusech/elasticsearch/transform/latest_malware/fields/fields.yml

@@ -30,3 +30,8 @@
       type: keyword
       description: |
         The configured expiration duration.
+
+- name: labels.is_ioc_transform_source
+  type: constant_keyword
+  value: "true"
+  description: In the source index it indicates if the document is a source for the transform.

packages/ti_abusech/elasticsearch/transform/latest_malwarebazaar/fields/fields.yml

@@ -102,3 +102,8 @@
       type: keyword
       description: |
         The configured expiration duration.
+
+- name: labels.is_ioc_transform_source
+  type: constant_keyword
+  value: "true"
+  description: In the source index it indicates if the document is a source for the transform.

packages/ti_abusech/elasticsearch/transform/latest_threatfox/fields/fields.yml

@@ -34,3 +34,8 @@
       type: keyword
       description: |
         The configured expiration duration.
+
+- name: labels.is_ioc_transform_source
+  type: constant_keyword
+  value: "true"
+  description: In the source index it indicates if the document is a source for the transform.

packages/ti_abusech/elasticsearch/transform/latest_url/fields/fields.yml

@@ -60,3 +60,8 @@
 - name: labels.interval
   type: keyword
   description: User-configured value for `Interval` setting. This is used in calculation of indicator expiration time.
+
+- name: labels.is_ioc_transform_source
+  type: constant_keyword
+  value: "true"
+  description: In the source index it indicates if the document is a source for the transform.

packages/ti_abusech/manifest.yml

@@ -1,6 +1,6 @@
 name: ti_abusech
 title: AbuseCH
-version: "2.3.1"
+version: "2.3.2"
 description: Ingest threat intelligence indicators from URL Haus, Malware Bazaar, and Threat Fox feeds with Elastic Agent.
 type: integration
 format_version: "3.0.3"

packages/ti_anomali/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.22.2"
+  changes:
+    - description: Add missing fields in transform
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/11008
 - version: "1.22.1"
   changes:
     - description: Fix ECS date mapping on threat fields.

packages/ti_anomali/elasticsearch/transform/latest_ioc/fields/fields.yml

@@ -505,3 +505,8 @@
       description: >
         OS codename, if any.
 
+
+- name: labels.is_ioc_transform_source
+  type: constant_keyword
+  value: "true"
+  description: In the source index it indicates if the document is a source for the transform.

packages/ti_anomali/manifest.yml

@@ -1,6 +1,6 @@
 name: ti_anomali
 title: Anomali
-version: "1.22.1"
+version: "1.22.2"
 description: Ingest threat intelligence indicators from Anomali with Elastic Agent.
 type: integration
 format_version: 3.0.2

packages/ti_cif3/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.14.2"
+  changes:
+    - description: Add missing fields in transform
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/11008
 - version: "1.14.1"
   changes:
     - description: Fix ECS date mapping on threat fields.

packages/ti_cif3/elasticsearch/transform/latest_threat/fields/fields.yml

@@ -112,3 +112,8 @@
       type: keyword
       description: |
         The configured expiration duration.
+
+- name: labels.is_ioc_transform_source
+  type: constant_keyword
+  value: "true"
+  description: In the source index it indicates if the document is a source for the transform.

packages/ti_cif3/manifest.yml

@@ -1,7 +1,7 @@
 format_version: "3.0.2"
 name: ti_cif3
 title: "Collective Intelligence Framework v3"
-version: "1.14.1"
+version: "1.14.2"
 description: "Ingest threat indicators from a Collective Intelligence Framework v3 instance with Elastic Agent."
 type: integration
 categories:

packages/ti_crowdstrike/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.1.5"
+  changes:
+    - description: Add missing fields in transform
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/11008
 - version: "1.1.4"
   changes:
     - description: Fix max() calculation on empty resources leading to error.

packages/ti_crowdstrike/elasticsearch/transform/latest_intel/fields/fields.yml

@@ -87,3 +87,8 @@
     - name: vulnerabilities
       type: keyword
       description: Information related to vulnerabilities associated with the Intel Indicator.
+
+- name: labels.is_ioc_transform_source
+  type: constant_keyword
+  value: "true"
+  description: In the source index it indicates if the document is a source for the transform.

packages/ti_crowdstrike/elasticsearch/transform/latest_ioc/fields/fields.yml

@@ -57,3 +57,8 @@
     - name: value
       type: ip
       description: The specific value of the indicator.
+
+- name: labels.is_ioc_transform_source
+  type: constant_keyword
+  value: "true"
+  description: In the source index it indicates if the document is a source for the transform.

packages/ti_crowdstrike/manifest.yml

@@ -1,7 +1,7 @@
 format_version: 3.0.3
 name: ti_crowdstrike
 title: CrowdStrike Falcon Intelligence
-version: "1.1.4"
+version: "1.1.5"
 description: Collect logs from CrowdStrike Falcon Intelligence with Elastic Agent.
 type: integration
 categories:

packages/ti_cybersixgill/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.30.2"
+  changes:
+    - description: Add missing fields in transform
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/11008
 - version: "1.30.1"
   changes:
     - description: Fix ECS date mapping on threat fields.

packages/ti_cybersixgill/elasticsearch/transform/latest_ioc/fields/fields.yml

@@ -37,3 +37,8 @@
       type: keyword
       description: |
         The configured expiration duration.
+
+- name: labels.is_ioc_transform_source
+  type: constant_keyword
+  value: "true"
+  description: In the source index it indicates if the document is a source for the transform.

packages/ti_cybersixgill/manifest.yml

@@ -1,6 +1,6 @@
 name: ti_cybersixgill
 title: Cybersixgill
-version: "1.30.1"
+version: "1.30.2"
 description: Ingest threat intelligence indicators from Cybersixgill with Elastic Agent.
 type: integration
 format_version: "3.0.2"

packages/ti_eclecticiq/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.2.2"
+  changes:
+    - description: Add missing fields in transform
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/11008
 - version: "1.2.1"
   changes:
     - description: Fix ECS date mapping on threat fields.

packages/ti_eclecticiq/elasticsearch/transform/latest_ioc/fields/fields.yml

@@ -11,3 +11,8 @@
     - name: deleted_at
       type: date
       description: Date when observable was removed from dataset
+
+- name: labels.is_ioc_transform_source
+  type: constant_keyword
+  value: "true"
+  description: In the source index it indicates if the document is a source for the transform.

packages/ti_eclecticiq/manifest.yml

@@ -1,7 +1,7 @@
 format_version: 3.0.3
 name: ti_eclecticiq
 title: EclecticIQ
-version: "1.2.1"
+version: "1.2.2"
 description: Ingest threat intelligence from EclecticIQ with Elastic Agent
 type: integration
 categories:

packages/ti_eset/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.2.3"
+  changes:
+    - description: Add missing fields in transform
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/11008
 - version: "1.2.2"
   changes:
     - description: Fix ECS date mapping on threat fields.

packages/ti_eset/elasticsearch/transform/apt_latest_ioc/fields/fields.yml

@@ -31,3 +31,8 @@
       type: date
       description: >-
         Event expiration date.
+
+- name: labels.is_ioc_transform_source
+  type: constant_keyword
+  value: "true"
+  description: In the source index it indicates if the document is a source for the transform.

packages/ti_eset/elasticsearch/transform/botnet_latest_ioc/fields/fields.yml

@@ -15,3 +15,8 @@
       type: keyword
       description: >-
         Threat labels.
+
+- name: labels.is_ioc_transform_source
+  type: constant_keyword
+  value: "true"
+  description: In the source index it indicates if the document is a source for the transform.

packages/ti_eset/elasticsearch/transform/cc_latest_ioc/fields/fields.yml

@@ -15,3 +15,8 @@
       type: keyword
       description: >-
         Threat labels.
+
+- name: labels.is_ioc_transform_source
+  type: constant_keyword
+  value: "true"
+  description: In the source index it indicates if the document is a source for the transform.

packages/ti_eset/elasticsearch/transform/domains_latest_ioc/fields/fields.yml

@@ -15,3 +15,8 @@
       type: keyword
       description: >-
         Threat labels.
+
+- name: labels.is_ioc_transform_source
+  type: constant_keyword
+  value: "true"
+  description: In the source index it indicates if the document is a source for the transform.

packages/ti_eset/elasticsearch/transform/files_latest_ioc/fields/fields.yml

@@ -15,3 +15,8 @@
       type: keyword
       description: >-
         Threat labels.
+
+- name: labels.is_ioc_transform_source
+  type: constant_keyword
+  value: "true"
+  description: In the source index it indicates if the document is a source for the transform.

packages/ti_eset/elasticsearch/transform/ip_latest_ioc/fields/fields.yml

@@ -15,3 +15,8 @@
       type: keyword
       description: >-
         Threat labels.
+
+- name: labels.is_ioc_transform_source
+  type: constant_keyword
+  value: "true"
+  description: In the source index it indicates if the document is a source for the transform.

packages/ti_eset/elasticsearch/transform/url_latest_ioc/fields/fields.yml

@@ -15,3 +15,8 @@
       type: keyword
       description: >-
         Threat labels.
+
+- name: labels.is_ioc_transform_source
+  type: constant_keyword
+  value: "true"
+  description: In the source index it indicates if the document is a source for the transform.

packages/ti_eset/manifest.yml

@@ -1,7 +1,7 @@
 format_version: 3.0.3
 name: ti_eset
 title: "ESET Threat Intelligence"
-version: "1.2.2"
+version: "1.2.3"
 description: "Ingest threat intelligence indicators from ESET Threat Intelligence with Elastic Agent."
 type: integration
 categories:

packages/ti_maltiverse/changelog.yml

@@ -1,3 +1,8 @@
+- version: "1.2.2"
+  changes:
+    - description: Add missing fields in transform
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/11008
 - version: "1.2.1"
   changes:
     - description: Fix ECS date mapping on threat fields.

packages/ti_maltiverse/elasticsearch/transform/latest/fields/fields.yml

@@ -381,3 +381,8 @@
       type: keyword
     - name: urlchecksum
       type: keyword
+
+- name: labels.is_ioc_transform_source
+  type: constant_keyword
+  value: "true"
+  description: In the source index it indicates if the document is a source for the transform.

packages/ti_maltiverse/manifest.yml

@@ -1,6 +1,6 @@
 name: ti_maltiverse
 title: Maltiverse
-version: "1.2.1"
+version: "1.2.2"
 description: Ingest threat intelligence indicators from Maltiverse feeds with Elastic Agent
 type: integration
 format_version: 3.0.2

packages/ti_mandiant_advantage/changelog.yml

@@ -1,4 +1,12 @@
-# newer versions go on top - update version numbers in httpjson.yml.hbs
+# newer versions go on top
+- version: "1.4.1"
+  changes:
+    - description: Add missing fields in transform
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/11008
+    - description: Fix mapping of sources subfields.
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/11008
 - version: "1.4.0"
   changes:
     - description: Add optional extra fields.

packages/ti_mandiant_advantage/data_stream/threat_intelligence/_dev/test/pipeline/test-common-config.yml

@@ -0,0 +1,2 @@
+numeric_keyword_fields:
+  - "mandiant.threat_intelligence.ioc.sources.osint"

packages/ti_mandiant_advantage/data_stream/threat_intelligence/_dev/test/system/test-default-config.yml

@@ -15,3 +15,5 @@ data_stream:
     include_campaigns: true
 assert:
   hit_count: 4
+numeric_keyword_fields:
+  - "mandiant.threat_intelligence.ioc.sources.osint"

packages/ti_mandiant_advantage/data_stream/threat_intelligence/fields/fields.yml

@@ -11,9 +11,13 @@
       type: date
       description: IOC last update date.
     - name: sources
-      type: object
-      object_type: keyword
+      type: group
       description: List of the indicator sources.
+      fields:
+        - name: osint
+          type: boolean
+        - name: "*"
+          type: keyword
     - name: campaigns
       type: object
       object_type: keyword

packages/ti_mandiant_advantage/docs/README.md

@@ -319,7 +319,8 @@ An example event for `threat_intelligence` looks as following:
 | mandiant.threat_intelligence.ioc.misp_warning_list_misses | Which MISP warning lists the indicator was not found in. | keyword |
 | mandiant.threat_intelligence.ioc.mscore | M-Score (IC-Score) between 0 - 100. | integer |
 | mandiant.threat_intelligence.ioc.reports | List of related reports. | object |
-| mandiant.threat_intelligence.ioc.sources | List of the indicator sources. | object |
+| mandiant.threat_intelligence.ioc.sources.\* |  | keyword |
+| mandiant.threat_intelligence.ioc.sources.osint |  | boolean |
 | mandiant.threat_intelligence.ioc.type | IOC type. | keyword |
 | mandiant.threat_intelligence.ioc.value | IOC value. | keyword |
 | threat.indicator.first_seen | The date and time when intelligence source first reported sighting this indicator. | date |

packages/ti_mandiant_advantage/manifest.yml

@@ -1,7 +1,7 @@
 format_version: 3.0.2
 name: ti_mandiant_advantage
 title: "Mandiant Advantage"
-version: "1.4.0"
+version: "1.4.1"
 source:
   license: "Elastic-2.0"
 description: "Collect Threat Intelligence from products within the Mandiant Advantage platform."

packages/ti_misp/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.35.2"
+  changes:
+    - description: Add missing fields in transform
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/11008
 - version: "1.35.1"
   changes:
     - description: Fix ECS date mapping on threat fields.

packages/ti_misp/elasticsearch/transform/latest_ioc/fields/fields.yml

@@ -295,3 +295,8 @@
           description: >
             List of attributes of the object in which the attribute is attached.
 
+
+- name: labels.is_ioc_transform_source
+  type: constant_keyword
+  value: "true"
+  description: In the source index it indicates if the document is a source for the transform.

packages/ti_misp/manifest.yml

@@ -1,6 +1,6 @@
 name: ti_misp
 title: MISP
-version: "1.35.1"
+version: "1.35.2"
 description: Ingest threat intelligence indicators from MISP platform with Elastic Agent.
 type: integration
 format_version: "3.0.2"