diff --git a/config/logstash.yml b/config/logstash.yml index 1e93cf6dacc..e625ef230fe 100644 --- a/config/logstash.yml +++ b/config/logstash.yml @@ -70,17 +70,15 @@ # available to plugins that implement an ECS Compatibility mode for use with # the Elastic Common Schema. # Possible values are: -# - disabled (default) +# - disabled # - v1 -# - v8 -# CAVEAT: use of this configuration for anything other than `disabled` -# is considered BETA until the General Availability of -# Logstash 8.0.0. As we continue to release updated plugins with ECS-Compatibility -# modes, opting into them at a pipeline or process level will cause you to -# automatically consume breaking changes with each upgrade, which may change the -# shape of data your pipeline produces. -# -# pipeline.ecs_compatibility: disabled +# - v8 (default) +# Pipelines defined before Logstash 8 operated without ECS in mind. To ensure a +# migrated pipeline continues to operate as it did before your upgrade, opt-OUT +# of ECS for the individual pipeline in its `pipelines.yml` definition. Setting +# it here will set the default for _all_ pipelines, including new ones. +# +# pipeline.ecs_compatibility: v8 # # ------------ Pipeline Configuration Settings -------------- # diff --git a/docs/static/breaking-changes.asciidoc b/docs/static/breaking-changes.asciidoc index 4adc0312e7d..aab26377099 100644 --- a/docs/static/breaking-changes.asciidoc +++ b/docs/static/breaking-changes.asciidoc @@ -46,6 +46,15 @@ Users who need to use a version other than the bundled JDK should set the value of `LS_JAVA_HOME` to the path of their preferred JDK. The value of `JAVA_HOME` will be ignored. +[discrete] +[[bc-ecs-compatibility]] +===== ECS compatibility is now on by default +Many plugins can now be run in a mode that avoids implicit conflict with the Elastic Common Schema. +This mode is controlled individually with each plugin's `ecs_compatibility` option, which defaults to the value of the Logstash `pipeline.ecs_compatibility` setting. +In Logstash 8, this compatibility mode will be on-by-default for all pipelines. https://github.com/elastic/logstash/issues/11623[#11623] + +If you wish to _lock in_ a pipeline's behaviour from Logstash 7.x before upgrading to Logstash 8, you can set `pipeline.ecs_compatibility: disabled` to its definition in `pipelines.yml` (or globally in `logstash.yml`). + [discrete] [[bc-ruby-engine]] ===== Ruby Execution Engine removed diff --git a/docs/static/ecs-compatibility.asciidoc b/docs/static/ecs-compatibility.asciidoc index 1da3cc88b0d..10d4a220228 100644 --- a/docs/static/ecs-compatibility.asciidoc +++ b/docs/static/ecs-compatibility.asciidoc @@ -2,7 +2,7 @@ === ECS in Logstash // LS8 will ship with ECS v8, but until ECS v8 is ready we rely on ECS v1 as an approximation. -:ls8-ecs-major-version: v1 +:ls8-ecs-major-version: v8 The {ecs-ref}/index.html[Elastic Common Schema (ECS)] is an open source specification, developed with support from the Elastic user community. ECS defines a common set of fields to be used for storing event data, such as logs and metrics, in {es}. @@ -15,25 +15,16 @@ Many plugins implement an ECS-compatibility mode, which causes them to produce a Any plugin that supports this mode will also have an `ecs_compatibility` option, which allows you to configure which mode the individual plugin instance should operate in. If left unspecified for an individual plugin, the pipeline's `pipeline.ecs_compatibility` setting will be observed. -This allows you to configure plugins to use ECS -- or to lock in your existing non-ECS behavior. +This allows you to configure plugins to use a specific version of ECS or to use their legacy non-ECS behavior. ECS compatibility modes do not prevent you from explicitly configuring a plugin in a manner that conflicts with ECS. Instead, they ensure that _implicit_ configuration avoids conflicts. -NOTE: Until {ls} 8.0 and the final 7.x are released, any value for `pipeline.ecs_compatibility` other than `disabled` is considered BETA and unsupported. - As we continue to release plugins with ECS compatibility modes, having this flag set will cause even patch-level upgrades to _automatically_ consume breaking changes in the upgraded plugins, changing the shape of data the plugin produces. - -ifeval::["{ls8-ecs-major-version}"!="v8"] -NOTE: ECS `v8` will be available alongside the GA release of {ls} 8.0.0, and will be available at or before the final minor release of {ls} 7. - We expect the scope of breaking changes in ECS 8 to be limited. - We are https://github.com/elastic/ecs/issues/839[tracking progress toward ECS v8] in a GitHub issue. -endif::[] - [[ecs-configuration]] ===== Configuring ECS -ECS will be on by default in a future release of {ls}, but you can begin using it now by configuring individual plugins with `ecs_compatibility`. -You can also "lock in" the existing non-ECS behavior for an entire pipeline to ensure its behavior doesn't change when you perform future upgrades. +In {ls} 8, all plugins are run in ECS compatibility {ls8-ecs-major-version} mode by default, but you can opt out at the plugin, pipeline, or system level to maintain legacy behavior. +This can be helpful if you have very complex pipelines that were defined pre-ECS, to allow you to either upgrade them or to avoid doing so independently of your {ls} 8.x upgrade. ====== Specific plugin instance @@ -73,6 +64,8 @@ If you wish to provide a specific default value for `ecs_compatibility` to _all_ This setting will be used unless overridden by a specific plugin instance. If unspecified for an individual pipeline, the global value will be used. +For example, setting `pipeline.ecs_compatibility: disabled` for a pipeline _locks in_ that pipeline's pre-{ls} 8 behavior. + [source,yaml,subs="attributes"] ----- - pipeline.id: my-legacy-pipeline @@ -83,9 +76,6 @@ If unspecified for an individual pipeline, the global value will be used. pipeline.ecs_compatibility: {ls8-ecs-major-version} ----- -NOTE: Until the General Availability of {ls} 8.0.0 that coincides with the final minor release of {ls} 7, any value for `pipeline.ecs_compatibility` other than `disabled` is considered BETA and unsupported because it will produce undesirable consequences when performing upgrades. - As we continue to release updated plugins with ECS-Compatibility modes, opting into them at a pipeline or process level will cause the affected plugins to silently and automatically consume breaking changes with each upgrade, which may change the shape of data your pipeline produces. - [[ecs-configuration-all]] ====== All plugins in all pipelines diff --git a/logstash-core/lib/logstash/agent.rb b/logstash-core/lib/logstash/agent.rb index e5d3469f532..bc42bda0443 100644 --- a/logstash-core/lib/logstash/agent.rb +++ b/logstash-core/lib/logstash/agent.rb @@ -65,14 +65,6 @@ def initialize(settings = LogStash::SETTINGS, source_loader = nil) # Generate / load the persistent uuid id - if @settings.set?('pipeline.ecs_compatibility') - ecs_compatibility_value = settings.get('pipeline.ecs_compatibility') - if ecs_compatibility_value != 'disabled' - logger.warn("Setting `pipeline.ecs_compatibility` given as `#{ecs_compatibility_value}`; " + - "values other than `disabled` are currently considered BETA and may have unintended consequences when upgrading minor versions of Logstash.") - end - end - # Initialize, but do not start the webserver. @webserver = LogStash::WebServer.from_settings(@logger, self, settings) diff --git a/logstash-core/lib/logstash/environment.rb b/logstash-core/lib/logstash/environment.rb index 6a34b8debeb..8df3237f4c9 100644 --- a/logstash-core/lib/logstash/environment.rb +++ b/logstash-core/lib/logstash/environment.rb @@ -60,7 +60,7 @@ module Environment Setting::Boolean.new("pipeline.plugin_classloaders", false), Setting::Boolean.new("pipeline.separate_logs", false), Setting::CoercibleString.new("pipeline.ordered", "auto", true, ["auto", "true", "false"]), - Setting::CoercibleString.new("pipeline.ecs_compatibility", "disabled", true, %w(disabled v1 v8)), + Setting::CoercibleString.new("pipeline.ecs_compatibility", "v8", true, %w(disabled v1 v8)), Setting.new("path.plugins", Array, []), Setting::NullableString.new("interactive", nil, false), Setting::Boolean.new("config.debug", false), diff --git a/logstash-core/lib/logstash/java_pipeline.rb b/logstash-core/lib/logstash/java_pipeline.rb index 6880185184c..220a54d1a1a 100644 --- a/logstash-core/lib/logstash/java_pipeline.rb +++ b/logstash-core/lib/logstash/java_pipeline.rb @@ -72,6 +72,10 @@ def initialize(pipeline_config, namespaced_metric = nil, agent = nil) # is by design and necessary for the wait_until_started semantic @finished_run = Concurrent::AtomicBoolean.new(false) + @logger.info(I18n.t('logstash.pipeline.effective_ecs_compatibility', + :pipeline_id => pipeline_id, + :ecs_compatibility => settings.get('pipeline.ecs_compatibility'))) + @thread = nil end # def initialize diff --git a/logstash-core/lib/logstash/plugins/ecs_compatibility_support.rb b/logstash-core/lib/logstash/plugins/ecs_compatibility_support.rb index 164486b7ef0..21398ba776a 100644 --- a/logstash-core/lib/logstash/plugins/ecs_compatibility_support.rb +++ b/logstash-core/lib/logstash/plugins/ecs_compatibility_support.rb @@ -17,11 +17,6 @@ def ecs_compatibility pipeline_settings = pipeline && pipeline.settings pipeline_settings ||= LogStash::SETTINGS - if !pipeline_settings.set?('pipeline.ecs_compatibility') - deprecation_logger.deprecated("Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. " + - "To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.") - end - pipeline_settings.get_value('pipeline.ecs_compatibility').to_sym end end diff --git a/logstash-core/locales/en.yml b/logstash-core/locales/en.yml index 493ece00538..4bbae030cfc 100644 --- a/logstash-core/locales/en.yml +++ b/logstash-core/locales/en.yml @@ -45,6 +45,9 @@ en: %{plugin} output plugin: setting 'workers => %{worker_count}' is not supported by this plugin. I will continue working as if you had not set this setting. Reason: %{message} + effective_ecs_compatibility: >- + Pipeline `%{pipeline_id}` is configured with `pipeline.ecs_compatibility: %{ecs_compatibility}` setting. + All plugins in this pipeline will default to `ecs_compatibility => %{ecs_compatibility}` unless explicitly configured otherwise. plugin: deprecated_milestone: >- %{plugin} plugin is using the 'milestone' method to declare the version diff --git a/logstash-core/spec/logstash/plugin_spec.rb b/logstash-core/spec/logstash/plugin_spec.rb index 1656a3855b6..5b1176d801a 100644 --- a/logstash-core/spec/logstash/plugin_spec.rb +++ b/logstash-core/spec/logstash/plugin_spec.rb @@ -461,16 +461,9 @@ def register; end end context 'and pipeline-level setting is not specified' do - it 'emits a deprecation warning about using the default which may change' do - instance.ecs_compatibility - - expect(deprecation_logger_stub).to have_received(:deprecated) do |message| - expect(message).to include("Relying on default value of `pipeline.ecs_compatibility`") - end - end - it 'returns `disabled`' do + it 'returns `v8`' do # Default value of `pipeline.ecs_compatibility` - expect(instance.ecs_compatibility).to eq(:disabled) + expect(instance.ecs_compatibility).to eq(:v8) end end end