Moving to the logstash-output-opensearch plugin

[dlvenable]

The OpenSearch project has created a new Logstash output plugin which can send events to OpenSearch 1.x or Elasticsearch 7.x clusters - logstash-output-opensearch. Starting with logstash-output-opensearch 1.2 it also supports SigV4 signing.

Having both the logstash-output-opensearch and logstash-output-amazon_es plugins can be confusing since it becomes unclear which one to use. With that in mind, we are looking into eventually replacing the logstash-output-amazon_es plugin with the logstash-output-opensearch plugin.

For now, we will place the logstash-output-amazon_es plugin into a maintenance mode. We will supply critical bug fixes and security patches. But, new features and functionality should be considered for the logstash-output-opensearch plugin instead.

Both plugins have some overlapping compatibility:

OpenSearch cluster	logstash-output-amazon_es	logstash-output-opensearch
OpenSearch 1.x	✅ (with compatibility mode enabled)	✅
Elasticsearch 7.x	✅	✅
Elasticsearch 6.5	✅	❌

As noted in the above table, the logstash-output-opensearch plugin does not support Elasticsearch 6.x. I've opened a feature request on that repo to support it - opensearch-project/logstash-output-opensearch#123.

We will provide a migration guide to help teams and individuals migrate to the logstash-output-opensearch plugin.

We will also triage issues from this GitHub repository and add them to the logstash-output-opensearch repository as needed. We will mostly focus on feature requests.

[sshivanii]

Please see below for common functionality and Plugin differences between the logstash-output-amazon_es and logtstash-output-opensearch plugins.

Common Functionality

• user
• password
• ssl
• Port: Default is 443 for both Amazon Elasticsearch Service and OpenSearch
• Batch size: 20MB. If single document exceeds 20MB it is sent as a single request.
• Errors retried infinitely: Network Errors, 429, 503
• Valid APIs: index, delete, create, update, sprintf
• Default service name: es

Plugin Differences

	logstash-output-amazon_es	logstash-output-opensearch
Plugin name	amazon_es	opensearch
Plugin support	Stores logs in Elasticsearch, compatible with Kibana	Stores logs in OpenSearch, compatible with OpenSearch Dashboards
Auth Type	AWS IAM	AWS IAM, extensible to custom auth_type
Default index value	logstash-%{+YYYY.MM.dd}	ecs_compatibility disabled: logstash-%{+yyyy.MM.dd} ecs_compatibility enabled: ecs-logstash-%{+yyyy.MM.dd}
ECS compatibility	Not Compatible	Compatible
Example config	output { amazon_es { hosts => ["foo.us-east-1.es.amazonaws.com"] region => "us-east-1" # aws_access_key_id and aws_secret_access_key are optional if instance profile is configured aws_access_key_id => 'ACCESS_KEY' aws_secret_access_key => 'SECRET_KEY' index => "production-logs-%{+YYYY.MM.dd}" }}	output { opensearch { hosts => ["hostname:port"] auth_type => { type => 'aws_iam' aws_access_key_id => 'ACCESS_KEY' aws_secret_access_key => 'SECRET_KEY' region => 'us-west-2' } index => "logstash-logs-%{+YYYY.MM.dd}" } }