upgrade aws-sdk to v3 to support webidentitytoken via service account…

… IAM (#195) * upgrade aws-sdk to v3 to support ec2 instance profiles (IMDS) * fix signing call * update version to 7.2.0 * update README * update to 8.0.0
awslabs · Sep 1, 2022 · d8a693d · d8a693d
1 parent cc08520
commit d8a693d
Show file tree

Hide file tree

Showing 3 changed files with 45 additions and 31 deletions.
diff --git a/README.md b/README.md
@@ -1,19 +1,19 @@
 # Logstash Output Plugin
 
 
-This plugin is now in maintenance mode. We will supply bug fixes and security patches for v7.1.X, older versions are no longer supported. This change is because the OpenSearch Project created a new Logstash output plugin 
-[logstash-output-opensearch](https://github.com/opensearch-project/logstash-output-opensearch) which ships events from 
+This plugin is now in maintenance mode. We will supply bug fixes and security patches for v7.2.X, older versions are no longer supported. This change is because the OpenSearch Project created a new Logstash output plugin
+[logstash-output-opensearch](https://github.com/opensearch-project/logstash-output-opensearch) which ships events from
 Logstash to OpenSearch 1.x and Elasticsearch 7.x clusters, and also supports SigV4 signing. Having similar functionality
 plugins can be redundant, so we plan to eventually replace this logstash-output-amazon_es plugin with the logstash-output-opensearch
 plugin.
 
-To help you migrate to [logstash-output-opensearch](https://github.com/opensearch-project/logstash-output-opensearch) plugin, please 
+To help you migrate to [logstash-output-opensearch](https://github.com/opensearch-project/logstash-output-opensearch) plugin, please
 find below a brief migration guide.
 
 ## Migrating to logstash-output-opensearch plugin
 
 
-This guide provides instructions for existing users of logstash-output-amazon_es plugin to migrate to 
+This guide provides instructions for existing users of logstash-output-amazon_es plugin to migrate to
 logstash-output-opensearch plugin.
 
 ### Configuration Changes
@@ -27,7 +27,7 @@ logstash-output-opensearch plugin.
 * The `type` value for `auth_type` for SigV4 signing will be set to `aws_iam`.
 
 For the Logstash configuration provided in [Configuration for Amazon Elasticsearch Service Output Plugin
-](#configuration-for-amazon-elasticsearch-service-output-plugin), here's a mapped example configuration for 
+](#configuration-for-amazon-elasticsearch-service-output-plugin), here's a mapped example configuration for
 logstash-output-opensearch plugin:
 
 ```
@@ -63,7 +63,7 @@ The remainder of this document is for using or developing the logstash-output-am
 ## Overview
 
 This is a plugin for [Logstash](https://github.com/elastic/logstash) which outputs
-to [Amazon OpenSearch Service](https://aws.amazon.com/opensearch-service/) 
+to [Amazon OpenSearch Service](https://aws.amazon.com/opensearch-service/)
 (successor to Amazon Elasticsearch Service) using
 [SigV4 signing](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html).
 
@@ -81,9 +81,9 @@ The following table shows the versions of logstash and logstash-output-amazon_es
 | 6.4.2                     | >= 6.0.0 |
 | 7.0.1                     | >= 7.0.0 |
 | 7.1.0                     | >= 7.0.0 |
+| 8.0.0                     | >= 7.0.0 |
 
-
-Also, logstash-output-amazon_es plugin versions 6.4.0 and newer are tested to be compatible with Elasticsearch 6.5 and greater. 
+Also, logstash-output-amazon_es plugin versions 6.4.0 and newer are tested to be compatible with Elasticsearch 6.5 and greater.
 
 |  logstash-output-amazon_es | Elasticsearch |
 | ------------- |----------|
@@ -105,8 +105,20 @@ flag to specify the version. For example:
 bin/logstash-plugin install --version 6.4.2 logstash-output-amazon_es
 ```
 
-
-
+Starting in 8.0.0, the aws sdk version is bumped to v3. In order for all other AWS plugins to work together, please remove pre-installed plugins and install logstash-integration-aws plugin as follows. See also https://github.com/logstash-plugins/logstash-mixin-aws/issues/38
+```
+# Remove existing logstash aws plugins and install logstash-integration-aws to keep sdk dependency the same
+# https://github.com/logstash-plugins/logstash-mixin-aws/issues/38
+/usr/share/logstash/bin/logstash-plugin remove logstash-input-s3
+/usr/share/logstash/bin/logstash-plugin remove logstash-input-sqs
+/usr/share/logstash/bin/logstash-plugin remove logstash-output-s3
+/usr/share/logstash/bin/logstash-plugin remove logstash-output-sns
+/usr/share/logstash/bin/logstash-plugin remove logstash-output-sqs
+/usr/share/logstash/bin/logstash-plugin remove logstash-output-cloudwatch
+
+/usr/share/logstash/bin/logstash-plugin install --version 0.1.0.pre logstash-integration-aws
+bin/logstash-plugin install --version 8.0.0 logstash-output-amazon_es
+```
 
 ## Configuration for Amazon Elasticsearch Service Output Plugin
 

diff --git a/lib/logstash/outputs/amazon_es/http_client/manticore_adapter.rb b/lib/logstash/outputs/amazon_es/http_client/manticore_adapter.rb
@@ -51,10 +51,10 @@ def initialize(logger, options={})
       if options[:proxy]
         options[:proxy] = manticore_proxy_hash(options[:proxy])
       end
-      
+
       @manticore = ::Manticore::Client.new(options)
     end
-    
+
     # Transform the proxy option to a hash. Manticore's support for non-hash
     # proxy options is broken. This was fixed in https://github.com/cheald/manticore/commit/34a00cee57a56148629ed0a47c329181e7319af5
     # but this is not yet released
@@ -89,12 +89,12 @@ def perform_request(url, method, path, params={}, body=nil)
       params[:body] = body if body
 
       if url.user
-        params[:auth] = { 
+        params[:auth] = {
           :user => CGI.unescape(url.user),
           # We have to unescape the password here since manticore won't do it
           # for us unless its part of the URL
-          :password => CGI.unescape(url.password), 
-          :eager => true 
+          :password => CGI.unescape(url.password),
+          :eager => true
         }
       end
 
@@ -107,16 +107,18 @@ def perform_request(url, method, path, params={}, body=nil)
       end
 
 
-      key = Seahorse::Client::Http::Request.new(options={:endpoint=>url, :http_method => method.to_s.upcase,
+      request = Seahorse::Client::Http::Request.new(options={:endpoint=>url, :http_method => method.to_s.upcase,
                                                        :headers => params[:headers],:body => params[:body]})
 
-      aws_signer = Aws::Signers::V4.new(@credentials, @service_name, @region )
-
-
-      signed_key =  aws_signer.sign(key)
-      params[:headers] =  params[:headers].merge(signed_key.headers)
-
+      aws_signer = Aws::Sigv4::Signer.new(service: @service_name, region: @region, credentials_provider: @credentials)
 
+      signed_key = aws_signer.sign_request(
+        http_method: request.http_method,
+        url: url,
+        headers: params[:headers],
+        body: params[:body]
+      )
+      params[:headers] = params[:headers].merge(signed_key.headers)
 
       resp = @manticore.send(method.downcase, request_uri.to_s, params)
 
@@ -136,27 +138,27 @@ def perform_request(url, method, path, params={}, body=nil)
 
     def format_url(url, path_and_query=nil)
       request_uri = url.clone
-      
+
       # We excise auth info from the URL in case manticore itself tries to stick
       # sensitive data in a thrown exception or log data
       request_uri.user = nil
       request_uri.password = nil
 
       return request_uri.to_s if path_and_query.nil?
-      
+
       parsed_path_and_query = java.net.URI.new(path_and_query)
-      
+
       query = request_uri.query
       parsed_query = parsed_path_and_query.query
-      
+
       new_query_parts = [request_uri.query, parsed_path_and_query.query].select do |part|
         part && !part.empty? # Skip empty nil and ""
       end
-      
+
       request_uri.query = new_query_parts.join("&") unless new_query_parts.empty?
-      
+
       request_uri.path = "#{request_uri.path}/#{parsed_path_and_query.path}".gsub(/\/{2,}/, "/")
-        
+
       request_uri
     end
 

diff --git a/logstash-output-amazon_es.gemspec b/logstash-output-amazon_es.gemspec
@@ -4,7 +4,7 @@
 
 Gem::Specification.new do |s|
   s.name            = 'logstash-output-amazon_es'
-  s.version         = '7.1.0'
+  s.version         = '8.0.0'
   s.licenses        = ['Apache-2.0']
   s.summary         = "Logstash Output to Amazon Elasticsearch Service"
   s.description     = "Output events to Amazon Elasticsearch Service with V4 signing"
@@ -28,7 +28,7 @@ Gem::Specification.new do |s|
   s.add_runtime_dependency 'stud', ['>= 0.0.17', '~> 0.0']
   s.add_runtime_dependency 'cabin', ['~> 0.6']
   s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
-  s.add_runtime_dependency 'aws-sdk', '>= 2.3.22', '~> 2'
+  s.add_runtime_dependency 'aws-sdk', '~> 3'
 
   s.add_development_dependency 'logstash-codec-plain'
   s.add_development_dependency 'logstash-devutils', "~> 1.3", ">= 1.3.1"