Skip to content

Latest commit

 

History

History
399 lines (331 loc) · 24.2 KB

CHANGELOG.next.asciidoc

File metadata and controls

399 lines (331 loc) · 24.2 KB

Beats version HEAD

Breaking changes

Affecting all Beats

  • Update add_cloud_metadata fields to adjust to ECS. 9265

  • Automaticall cap signed integers to 63bits. 8991

  • Rename beat.timezone to event.timezone. 9458

  • Use _doc as document type. 9056https://github.com/elastic/beats/pull/9573[9573]

  • Update to Golang 1.11.3. 9560

  • Embedded html is not escaped anymore by default. 9914

  • Remove port settings from Logstash and Redis output. 9934

  • Fix registry handle leak on Windows (elastic/go-sysinfo#33). 9920

  • Rename process.exe to process.executable in add_process_metadata to align with ECS. 9949

  • Import ECS change ecs#308: leaf field user.group is now the group field set. 10275

  • Update the code of Central Management to align with the new returned format. 10019

  • Docker and Kubernetes labels/annotations will be "dedoted" by default. 10338

  • Remove --setup command line flag. 10138

  • Remove --version command line flag. 10138

  • Remove --configtest command line flag. 10138

  • Move output.elasticsearch.ilm settings to setup.ilm. 10347

  • ILM will be available by default if Elasticsearch > 7.0 is used. 10347

  • Allow Central Management to send events back to kibana. 9382

  • Initialize the Paths before the keystore and save the keystore into data/{beatname}.keystore. 10706

  • Add cleanup_timeout option to docker autodiscover, to wait some time before removing configurations after a container is stopped. {issue]10374[10374] 10905

  • On Google Cloud Engine (GCE) the add_cloud_metadata will now trim the project info from the cloud.machine.type and cloud.availability_zone. 10968

  • Empty meta.json file will be treated as a missing meta file. 8558

  • Rename migration.enabled config to migration.6_to_7.enabled. 11284

  • Beats Xpack now checks for Basic license on connect. 11296

Auditbeat

  • Rename process.exe to process.executable in auditd module to align with ECS. 9949

  • Rename process.cwd to process.working_directory in auditd module to align with ECS. 10195

  • Change data type of process.pid and process.ppid to number in JSON output of the auditd module. 10195

  • Change data type of file.uid and file.gid to string in JSON output of the FIM module. 10195

  • Field file.origin changed type from text to keyword. 10544

  • Rename user fields to ECS in auditd module. 10456

  • Rename event.type to auditd.message_type in auditd module because event.type is reserved for future use by ECS. 10536

  • Rename auditd.messages to event.original and auditd.warnings to error.message. 10577

  • Process dataset: Only report processes with executable. 11232

Filebeat

  • Set ecs: true in user_agent processors when loading pipelines with Filebeat 7.0.x into Elasticsearch 6.7.x. 10655 10875

Heartbeat

  • Remove monitor generator script that was rarely used. 9648

  • monitor IDs are now configurable. Auto generated monitor IDs now use a different formula based on a hash of their config values. If you wish to have continuity with the old format of monitor IDs you’ll need to set the id property explicitly. 9697

  • A number of fields have been aliased to their relevant counterparts in the url.* field. Existing visualizations should mostly work. The fields that have been moved are monitor.scheme → url.scheme, monitor.host → url.domain, resolve.host → url.domain, http.url → url.full, tcp.port → url.port. In addition to these moves the new fields url.username, url.password, url.path, and url.query are now present. It should be noted that the url.password field does not contain actual password values, but rather the text <hidden> 9570.

  • The included Kibana HTTP dashboard is now removed in favor of the Uptime app in Kibana. 10294

  • Removed the add_host_metadata and add_cloud_metadata processors from the default config. These don’t fit well with ECS for Heartbeat and were rarely used.

Journalbeat

  • Rename read_timestamp to event.created to align with ECS. 10043, 10139

  • Rename host.name to host.hostname to align with ECS. 10043

  • Fix typo in the field name container.id_truncated. 10525

  • Rename container.image.tag to container.log.tag. 10561

  • Change type of text fields to keyword. 10542

Metricbeat

  • Migrate system process metricset fields to ECS. 10332

  • Refactor Prometheus metric mappings 9948

  • Removed Prometheus stats metricset in favor of just using Prometheus collector 9948

  • Migrate system socket metricset fields to ECS. 10339

  • Renamed direction values in sockets to ECS recommendations, from incoming/outcoming to inbound/outbound. 10339

  • Adjust Redis.info metricset fields to ECS. 10319

  • Change type of field docker.container.ip_addresses to ip instead of keyword. 10364

  • Rename http.request.body field to http.request.body.content. 10315

  • Adjust php_fpm.process metricset fields to ECS. 10366

  • Adjust mongodb.status metricset to to ECS. 10368

  • Refactor munin module to collect an event per plugin and to have more strict field mappings. namespace option has been removed, and will be replaced by service.name. 10322

  • Change the following fields from type text to keyword: 10318

  • ceph.osd_df.name

  • ceph.osd_tree.name

  • ceph.osd_tree.children

  • kafka.consumergroup.meta

  • kibana.stats.name

  • mongodb.metrics.replication.executor.network_interface

  • php_fpm.process.request_uri

  • php_fpm.process.script

  • Add service.name option to all modules to explicitly set service.name if it is unset. 10427

  • Update a few elasticsearch.* fields to map to ECS. 10350

  • Update a few logstash.* fields to map to ECS. 10350

  • Update a few kibana.* fields to map to ECS. 10350

  • Update rabbitmq.* fields to map to ECS. 10563

  • Update haproxy.* fields to map to ECS. 10558 10568

  • Collect all EC2 meta data from all instances in all states. 10628

  • Migrate docker module to ECS. 10927

  • Add connection and request timeouts for HTTP helper. 11032

  • Add new option OpMultiplyBuckets to scale histogram buckets to avoid decimal points in final events 10994

Packetbeat

  • Adjust Packetbeat http fields to ECS Beta 2 9645

  • http.request.body moves to http.request.body.content

  • http.response.body moves to http.response.body.content

  • Changed Packetbeat fields to align with ECS. 7968

  • Removed trailing dot from domain names reported by the DNS protocol. 9941

Winlogbeat

  • Adjust Winlogbeat fields to map to ECS. 10333

Functionbeat

  • Mark Functionbeat as GA. 10564

  • Correctly normalize Cloudformation resource name. 10087

  • Functionbeat can now deploy a function for Kinesis. {10116}10116[10116]

  • Allow functionbeat to use the keystore. 9009

  • Correctly extract Kinesis Data field from the Kinesis Record. 11141

Bugfixes

Affecting all Beats

  • Enforce validation for the Central Management access token. 9621

  • Fix config appender registration. 9873

  • Gracefully handle TLS options when enrolling a Beat. 9129

  • The backing off now implements jitter to better distribute the load. 10172

  • Fix TLS certificate DoS vulnerability. 10302

  • Fix panic and file unlock in spool on atomic operation (arm, x86-32). File lock was not released when panic occurs, leading to the beat deadlocking on startup. 10289

  • Fix encoding of timestamps when using disk spool. 10099

  • Fix stopping of modules started by kubernetes autodiscover. 10476

  • Fix a issue when remote and local configuration didn’t match when fetching configuration from Central Management. 10587

  • Fix unauthorized error when loading dashboards by adding username and password into kibana config. 10513 10675

  • Ensure all beat commands respect configured settings. 10721

  • Allow to configure Kafka fetching strategy for the topic metadata. 10682

  • Using an environment variable for the password when enrolling a beat will now raise an error if the variable doesn’t exist. 10936

  • Add missing host.containerized and host.os.build to fields.ecs.yml. 11016

  • Reconnections of Kubernetes watchers are now logged at debug level when they are harmless. 10988

  • Include ip and boolean type when generating index pattern. 10995

  • Cancelling enrollment of a beat will not enroll the beat. 10150

  • Add missing fields and test cases for libbeat add_kubernetes_metadata processor. 11133, 11134

  • Report faulting file when config reload fails. 1130411304

Auditbeat

  • Enable System module config on Windows. 10237

  • Package: Disable librpm signal handlers. 10694

  • Login: Handle different bad login UTMP types. 10865

  • System module: Fix and unify bucket closing logic. 10897

  • User dataset: Numerous fixes to error handling. 10942

Filebeat

  • Add convert_timezone option to Elasticsearch module to convert dates to UTC. 9756 9761

  • Support IPv6 addresses with zone id in IIS ingest pipeline. 9836 error log: 9869, access log: 9955.

  • Support haproxy log lines without captured headers. 9463 9958

  • Make elasticsearch/audit fileset be more lenient in parsing node name. 10035 10135

  • Fix bad bytes count in docker input when filtering by stream. 10211

  • Fixed data types for roles and indices fields in elasticsearch/audit fileset 10307

  • Ensure source.address is always populated by the nginx module (ECS). 10418

  • Add support for Cisco syslog format used by their switch. 10760

  • Cover empty request data, url and version in Apache2 modulehttps://github.com//pull/10730[10730]

  • Fix registry entries not being cleaned due to race conditions. 10747

  • Improve detection of file deletion on Windows. 10747

  • Fix errors in filebeat Zeek dashboard and README files. Add notice.log support. 10916

  • Fix a bug when converting NetFlow fields to snake_case. 10950

  • Add on_failure handler for Zeek ingest pipelines. Fix one field name error for notice and add an additional test case. 11004 11105

  • Fix goroutine leak happening when harvesters are dynamically stopped. 11263

  • Fix issue preventing docker container events to be stored if the container has a network interface without ip address. 11225 11247

  • Add on_failure handler for Zeek ingest pipelines. Fix one field name error for notice and add an additional test case. 11004 11105

  • Change URLPATH grok pattern to support brackets. 11135 11252

  • Add support for iis log with different address format. 11255 11256

  • Add fix to parse syslog message with priority value 0. 11010

  • Don’t apply multiline rules in Logstash json logs. 11346

Heartbeat

  • Made monitors.d configuration part of the default config. 9004

  • Fixed rare issue where TLS connections to endpoints with x509 certificates missing either notBefore or notAfter would cause the check to fail with a stacktrace. 9566

  • Fix checks for TCP send/receive data 11118

Journalbeat

  • Do not stop collecting events when journal entries change. 9994

Metricbeat

  • Fix panics in vsphere module when certain values where not returned by the API. 9784

  • Fix pod UID metadata enrichment in Kubernetes module. 10081

  • Fix issue that would prevent collection of processes without command line on Windows. 10196

  • Fixed data type for tags field in docker/container metricset 10307

  • Fixed data type for tags field in docker/image metricset 10307

  • Fixed data type for isr field in kafka/partition metricset 10307

  • Fixed data types for various hosts fields in mongodb/replstatus metricset 10307

  • Added function to close sql database connection. 10355

  • Fix issue with elasticsearch/node_stats metricset (x-pack) not indexing source_node field. 10639

  • Migrate docker autodiscover to ECS. 10757 10862

  • Fix issue in kubernetes module preventing usage percentages to be properly calculated. 10946

  • Fix for not reusable http client leading to connection leaks in Jolokia module 11014

  • Fix parsing error using GET in Jolokia module. 11075 11071

  • Collect metrics when EC2 instances are not in running state. 11008 11023

  • Change ECS field cloud.provider to aws. 11023

  • Add documentation about jolokia autodiscover fields. 10925 10979

  • Add missing aws.ec2.instance.state.name into fields.yml. 11219 11221

  • Fix ec2 metricset to collect metrics from Cloudwatch with the same timestamp. 11142

  • Fix potential memory leak in stopped docker metricsets 11294

Packetbeat

  • Fix DHCPv4 dashboard that wouldn’t load in Kibana. 9850

  • Fixed a crash when using af_packet capture 10477

  • Prevent duplicate packet loss error messages in HTTP events. 10709

  • Avoid reporting unknown MongoDB opcodes more than once. 10878

Winlogbeat

Functionbeat

  • Ensure that functionbeat is logging at info level not debug. 10262

  • Add the required permissions to the role when deployment SQS functions. 9152

Added

Affecting all Beats

  • Update field definitions for http to ECS Beta 2 9645

  • Add agent.id and agent.ephemeral_id fields to all beats. 9404

  • Add name config option to add_host_metadata processor. 9943

  • Add add_labels and add_tags processors. 9973

  • Add missing file encoding to readers. 10080

  • Introduce migration.enabled configuration. 9805

  • Add alias field support in Kibana index pattern. 10075

  • Add add_fields processor. 10119

  • Add Kibana field formatter to bytes fields. 10184

  • Document a few more auditd.log.* fields. 10192

  • Support Kafka 2.1.0. 10440

  • Add ILM mode auto to setup.ilm.enabled setting. This new default value detects if ILM is available 10347

  • Add support to read ILM policy from external JSON file. 10347

  • Add overwrite and check_exists settings to ILM support. 10347

  • Generate Kibana index pattern on demand instead of using a local file. 10478

  • Calls to Elasticsearch X-Pack APIs made by Beats won’t cause deprecation logs in Elasticsearch logs. {9656}9656[9656]

  • Add network condition to processors for matching IP addresses against CIDRs. 10743

  • Add if/then/else support to processors. 10744

  • Add community_id processor for computing network flow hashes. 10745

  • Add output test to kafka output 10834

  • Add ip fields to default_field in Elasticsearch template. 11035

  • Gracefully shut down on SIGHUP 10704

  • New processor: copy_fields. 11303

  • Add error.message to events when fail_on_error is set in rename and copy_fields processors. 11303

Auditbeat

  • Add system module. 9546

  • Add user.id (UID) and user.name for ECS. 10195

  • Add group.id (GID) and group.name for ECS. 10195

  • System module process dataset: Add user information to processes. 9963

  • Add system package dataset. 10225

  • Add system module login dataset. 9327

  • Add entity_id fields. 10500

  • Add seven dashboards for the system module. 10511

  • Move System module to beta. 10800

  • Login dataset: Add event category and type. 11339

Filebeat

  • Added module for parsing Google Santa logs. 9540

  • Added netflow input type that supports NetFlow v1, v5, v6, v7, v8, v9 and IPFIX. 9399

  • Add option to modules.yml file to indicate that a module has been moved 9432.

  • Fix parsing of GC entries in elasticsearch server log. 9513 9810

  • Support mysql 5.7.22 slowlog starting with time information. 7892 9647

  • Add support for ssl_request_log in apache2 module. 8088 9833

  • Add support for iis 7.5 log format. 9753 9967

  • Add service.type field to all Modules. By default the field is set with the module name. It can be overwritten with service.type config. 10042

  • Add support for MariaDB in the slowlog fileset of mysql module. 9731

  • Apache module’s error fileset now performs GeoIP lookup, like the access fileset. 10273

  • Elasticsearch module’s slowlog now populates event.duration (ECS). 9293

  • HAProxy module now populates event.duration and http.response.bytes (ECS). 10143

  • Teach elasticsearch/audit fileset to parse out some more fields. 10134 10137

  • Add convert_timezone to nginx module. 9839 10148

  • Add support for Percona in the slowlog fileset of mysql module. 6665 10227

  • Added support for ingesting structured Elasticsearch audit logs 10352

  • Added support for ingesting structured Elasticsearch slow logs 10445

  • Added support for ingesting structured Elasticsearch deprecation logs 10445

  • New iptables module that receives iptables/ip6tables logs over syslog or file. Supports Ubiquiti Firewall extensions. 8781 10176

  • Added support for ingesting structured Elasticsearch server logs 10428

  • Populate more ECS fields in the Suricata module. 10006

  • Add ISO8601 timestamp support in syslog metricset. 8716 10736

  • Add more info to message logged when a duplicated symlink file is found 10845

  • Add option to configure docker input with paths 10687

  • Add Netflow module to enrich flow events with geoip data. 10877

  • Set event.category: network_traffic for Suricata. 10882

  • Add configuration knob for auto-discover hints to control whether log harvesting is enabled for the pod/container. 10811 10911

  • Change Suricata module pipeline to handle destination.domain being set if a reverse DNS processor is used. 10510

  • Add the network.community_id flow identifier to field to the IPTables, Suricata, and Zeek modules. 11005

  • Add support for loading custom NetFlow and IPFIX field definitions to netflow input. 10945

  • Added categorization fields for SSH login events in the system/auth fileset. 11334

Heartbeat

  • Autodiscover metadata is now included in events by default. So, if you are using the docker provider for instance, you’ll see the correct fields under the docker key. 10258

Journalbeat

  • Migrate registry from previously incorrect path. 10486

Metricbeat

  • Add key metricset to the Redis module. 9582 9657 9746

  • Add socket_summary metricset to system defaults, removing experimental tag and supporting Windows 9709

  • Add docker event metricset. 9856

  • Add 'performance' metricset to x-pack mssql module 9826

  • Add DeDot for kubernetes labels and annotations. 9860 9939

  • Add more meaningful metrics to 'performance' Metricset on 'MSSQL' module 10011

  • Rename some fields in performance Metricset on MSSQL module to match the updated documentation from Microsoft 10074

  • Add AWS EC2 module. 9257 9300

  • Release windows Metricbeat module as GA. 10163

  • Release traefik Metricbeat module as GA. 10166

  • Release Elastic stack modules (Elasticsearch, Logstash, and Kibana) as GA. 10094

  • List filesystems on Windows that have an access path but not an assigned letter 8916 10196

  • Add nats module. 10071

  • Release uswgi Metricbeat module GA. 10164

  • Release php_fpm module as GA. 10198

  • Release Memcached module as GA. 10199

  • Release etcd module as GA. 10200

  • Release Ceph module as GA. 10202

  • Release aerospike module as GA. 10203

  • Release kubernetes apiserver and event metricsets as GA 10212

  • Release Couchbase module as GA. 10201

  • Release RabbitMQ module GA. 10165

  • Release envoyproxy module GA. 10223

  • Release mongodb.metrics and mongodb.replstatus as GA. 10242

  • Release mysql.galera_status as GA. 10242

  • Release postgresql.statement as GA. 10242

  • Release RabbitMQ Metricbeat module GA. 10165

  • Release Dropwizard module as GA. 10240

  • Release Graphite module as GA. 10240

  • Release kvm module as beta. 10279

  • Release http.server metricset as GA. 10240

  • Release Nats module as GA. 10281

  • Release munin module as GA. 10311

  • Release Golang module as GA. 10312

  • Release use of xpack.enabled: true flag in Elasticsearch and Kibana modules as GA. 10222

  • Add support for MySQL 8.0 and tests also for Percona and MariaDB. 10261

  • Rename 'db' Metricset to 'transaction_log' in MSSQL Metricbeat module 10109

  • Add process arguments and the path to its executable file in the system process metricset 10332

  • Added 'server' Metricset to Zookeeper Metricbeat module 8938 10341

  • Release AWS module as GA. 10345

  • Add overview dashboard to Zookeeper Metricbeat module 10379

  • Add Consul Metricbeat module with Agent Metricset 8631

  • Add filters and pie chart for AWS EC2 dashboard. 10596

  • Add AWS SQS metricset. 10684 10053

  • Add AWS s3_request metricset. 10949 10055

  • Add s3_daily_storage metricset. 10940 10055

  • Add coredns metricbeat module. https://github.com/elastic/beats/pull/10585{10585]

Packetbeat

  • Add network.community_id to Packetbeat flow events. 10061

  • Add aliases for flow fields that were renamed. 7968 10063

Functionbeat

Deprecated

Affecting all Beats

Filebeat

Heartbeat

Journalbeat

Metricbeat

Packetbeat

Winlogbeat

  • Close handle on signalEvent. 9838

Functionbeat

Known Issue

Journalbeat

  • Journalbeat requires at least systemd v233 in order to follow entries after journal changes (rotation, vacuum).