[meta] Upgrade integrations to ECS 8.17

[taylor-swanson]

Guide

It is recommended to split the changes into smaller PRs to limit the number of changed files. Generally speaking, 10 integrations per PR is a good number to target.

Automated method

Use the ecs-update tool which can be found here.

cd packages/
ecs-update -pr 999999 -ecs-git-ref v8.17.0 -ecs-version 8.17.0 -owner elastic/owner-name-here package_1 [package_2 ...]

Once PR is filed, the changelogs will need to be updated with the correct PR number.

Manual method

Update ECS references in integrations to version 8.17.

• Update reference in _dev/build/build.yml to v8.17.0
• Update package changelog/manifest and regenerate README.

dependencies:
  ecs:
-    reference: "[email protected]"
+    reference: "[email protected]"

ECS 8.17 Changes

TODO: Awaiting 8.17 release

Integrations

@elastic/cloud-security-posture:

• cloud_security_posture

@elastic/ecosystem:

• elastic_package_registry

@elastic/elastic-agent:

• elastic_agent

@elastic/elastic-agent-data-plane:

• filestream
• journald
• linux
• log
• windows

@elastic/fleet:

• fleet_server

@elastic/obs-cloudnative-monitoring:

• containerd
• docker
• istio
• kubernetes
• kubernetes_otel
• nginx_ingress_controller

@elastic/obs-ds-hosted-services:

• aws
• aws_logs
• awsfirehose
• azure
• azure_metrics
• gcp
• gcp_metrics

@elastic/obs-ds-intake-services:

• apm
• profiler_collector
• profiler_symbolizer

@elastic/obs-infraobs-integrations:

• activemq
• airflow
• apache
• apache_spark
• apache_tomcat
• awsfargate
• azure_app_service
• azure_application_insights
• azure_billing
• azure_functions
• azure_logs
• azure_openai
• cassandra
• ceph
• cisco_meraki_metrics
• citrix_adc
• cockroachdb
• coredns
• couchbase
• couchdb
• etcd
• gcp_vertexai
• golang
• hadoop
• haproxy
• ibmmq
• iis
• influxdb
• jolokia
• kafka
• kafka_log
• memcached
• microsoft_sqlserver
• mongodb
• mongodb_atlas
• mysql
• nagios_xi
• nats
• nginx
• nginx_ingress_controller_otel
• oracle
• oracle_weblogic
• panw_metrics
• php_fpm
• postgresql
• prometheus
• prometheus_input
• rabbitmq
• redis
• redisenterprise
• salesforce
• spring_boot
• sql
• stan
• statsd_input
• system
• tomcat
• traefik
• vsphere
• websphere_application_server
• zookeeper

@elastic/sec-deployment-and-devices:

• Update Deployment and Devices integrations to ECS 8.16.0 (part 1)  #12119

  • arista_ngfw
  • cef
  • checkpoint
  • cisco_aironet
  • cisco_asa
  • cisco_ftd
  • cisco_ios
  • cisco_ise
  • cisco_nexus
  • cisco_secure_email_gateway
  • citrix_waf

• Update Deployment and Devices integrations to ECS 8.16.0 (part 2) #12120

  • fortinet_forticlient
  • fortinet_fortiedr
  • fortinet_fortigate
  • fortinet_fortimail
  • fortinet_fortimanager
  • fortinet_fortiproxy
  • goflow2
  • hashicorp_vault
  • imperva
  • iptables

• Update Deployment and Devices integrations to ECS 8.16.0 (part 3) #12121

  • juniper_srx
  • modsecurity
  • netflow
  • osquery
  • panw
  • pfsense
  • proxysg

• Update Deployment and Devices integrations to ECS 8.16.0 (part 4) #12122

  • qnap_nas
  • snort
  • sonicwall_firewall
  • sophos
  • squid
  • stormshield
  • suricata
  • tcp
  • udp
  • watchguard_firebox
  • zeek

@elastic/sec-linux-platform:

• auditd
• auditd_manager
• cloud_defend
• fim
• network_traffic
• sysmon_linux
• system_audit

@elastic/sec-windows-platform:

• hid_bravura_monitor
• microsoft_dhcp
• microsoft_dnsserver
• microsoft_exchange_server
• mysql_enterprise
• windows_etw
• winlog

@elastic/security-asset-management:

• osquery_manager

@elastic/security-service-integrations:

• 1password
• abnormal_security
• akamai
• amazon_security_lake
• atlassian_bitbucket
• atlassian_confluence
• atlassian_jira
• auth0
• authentik
• aws_bedrock
• azure_blob_storage
• azure_frontdoor
• azure_network_watcher_nsg
• azure_network_watcher_vnet
• barracuda
• barracuda_cloudgen_firewall
• bbot
• bitdefender
• bitwarden
• blacklens
• box_events
• canva
• carbon_black_cloud
• carbonblack_edr
• cel
• checkpoint_email
• checkpoint_harmony_endpoint
• cisa_kevs
• cisco_duo
• cisco_meraki
• cisco_secure_endpoint
• cisco_umbrella
• claroty_ctd
• cloudflare
• cloudflare_logpush
• corelight
• cribl
• crowdstrike
• cyberark_pta
• cyberarkpas
• cybereason
• cylance
• darktrace
• digital_guardian
• entityanalytics_ad
• entityanalytics_entra_id
• entityanalytics_okta
• eset_protect
• f5
• f5_bigip
• falco
• fireeye
• first_epss
• forcepoint_web
• forgerock
• gcp_pubsub
• gigamon
• github
• gitlab
• google_cloud_storage
• google_scc
• google_workspace
• http_endpoint
• httpjson
• imperva_cloud_waf
• infoblox_bloxone_ddi
• infoblox_nios
• jamf_compliance_reporter
• jamf_pro
• jamf_protect
• jumpcloud
• keycloak
• lastpass
• lumos
• lyve_cloud
• m365_defender
• mattermost
• menlo
• microsoft_defender_cloud
• microsoft_defender_endpoint
• microsoft_exchange_online_message_trace
• microsoft_sentinel
• mimecast
• netskope
• o365
• okta
• opencanary
• panw_cortex_xdr
• ping_one
• pps
• prisma_access
• prisma_cloud
• proofpoint_on_demand
• proofpoint_tap
• pulse_connect_secure
• qualys_vmdr
• rapid7_insightvm
• santa
• sentinel_one
• sentinel_one_cloud_funnel
• servicenow
• slack
• snyk
• sophos_central
• spycloud
• sublime_security
• symantec_edr_cloud
• symantec_endpoint
• symantec_endpoint_security
• sysdig
• tanium
• teleport
• tenable_io
• tenable_sc
• threat_map
• thycotic_ss
• ti_abusech
• ti_anomali
• ti_cif3
• ti_crowdstrike
• ti_custom
• ti_cybersixgill
• ti_eclecticiq
• ti_eset
• ti_maltiverse
• ti_mandiant_advantage
• ti_misp
• ti_opencti
• ti_otx
• ti_rapid7_threat_command
• ti_recordedfuture
• ti_threatconnect
• ti_threatq
• ti_util
• tines
• trellix_edr_cloud
• trellix_epo_cloud
• trend_micro_vision_one
• trendmicro
• tychon
• vectra_detect
• websocket
• wiz
• zerofox
• zeronetworks
• zoom
• zscaler_zia
• zscaler_zpa

@elastic/stack-monitoring:

• beat
• elasticsearch
• enterprisesearch
• kibana
• logstash
• platform_observability

[consulthys]

@elastic/stack-monitoring:

• beat[ ] elasticsearch[ ] enterprisesearch[ ] kibana[ ] logstash[ ] platform_observability

Hey @taylor-swanson thanks for initiating this. What is the deadline for this change?
Also, can you share whether this should be done before or after https://github.com/elastic/ingest-dev/issues/4484 ?

[taylor-swanson]

Hey @consulthys!

While it is nice to have ECS updated, it isn't critical it needs to be done immediately, unless an integration needs to take advantage of a change in the new version. We haven't issued a new ECS release in a while (8.11.0 was the previous version), but at least back then, we'd work on getting the integrations updated within a short period of time (perhaps a week or so?) after the release.

The ecs-update tool (see the issue description above) definitely helps make the process of updating integrations less painful.

Regarding how it relates to the issue you linked, that's actually a good question. As far as I know, I don't think 9.0 will have any issues using a 8.16 (or any 8.x version) ECS. There isn't a strong version enforcement like there is for kibana.version in a package manifest. I'll raise that question with the ECS team, though. If we do need a 9.0 version, we'll for sure get another issue out like this one.

[taylor-swanson]

After discussing with the ECS working group, we've decided to hold off on upgrading to 8.16.0 due to link rendering issues. We will be fixing these issues for 8.17.0 and then work can resume on upgrading integrations. If the integrations you are working on don't have any issues with links and you would still like to upgrade to 8.16.0, that is okay.

I don't have a date for the ECS release, but I'd imagine it's relatively soon given that the 8.17 stack is already out.