IMDS v2 not supported

[HarshadRanganathan]

Logstash aws es output plugin is not working when IMDS v1 is disabled.

Logstash version: 7.6.1
Plugin version: 6.4.1

Error:

[2021-05-03T08:47:04,071][ERROR][logstash.javapipeline ][main] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<NoMethodError: undefined method `credentials' for nil:NilClass>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/signers/v4.rb:45:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client/manticore_adapter.rb:108:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:291:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:245:in `block in healthcheck!'", "org/jruby/RubyHash.java:1428:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:241:in `healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:341:in `update_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:71:in `start'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client.rb:304:in `build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client.rb:54:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client_builder.rb:110:in `create_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client_builder.rb:98:in `build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es.rb:256:in `build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/common.rb:25:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:106:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:48:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:200:in `block in register_plugins'", "org/jruby/RubyArray.java:1814:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:199:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:501:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:212:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:154:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:109:in `block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x4dbdc20d run>"}

[2021-05-03T08:47:04,146][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}

[AustinTag]

This appears to be an issue with the plugin pulling the credentials from the instance. 2 questions:

1. What type of security are you using for your Amazon Elasticsearch cluster? VPC/Internet, username/password or IAM auth?
2. Can you post your logstash configuration file here? (With all identifying information or credentials removed of course)

[HarshadRanganathan]

This appears to be an issue with the plugin pulling the credentials from the instance. 2 questions:

1. What type of security are you using for your Amazon Elasticsearch cluster? VPC/Internet, username/password or IAM auth?
2. Can you post your logstash configuration file here? (With all identifying information or credentials removed of course)

1. We are using VPC security. Our Domain access policy is pretty basic at the moment -

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": [
        "arn:aws:es:us-east-1:<account-id>:domain/<es-domain>/*",
        "arn:aws:es:us-east-1:<account-id>:domain/<es-domain>"
      ]
    }
  ]
}

2. Logstash configuration (removed identifying information)

input {
  kafka {
    auto_offset_reset => "latest"
    bootstrap_servers => ""
    security_protocol => "SSL"
    consumer_threads => 2
    topics => [""]
    decorate_events => true
    max_poll_interval_ms => "2147483647"
    max_poll_records => "100"
    request_timeout_ms => "50000"
    session_timeout_ms => "40000"
    group_id => ""
    codec => "json"
  }
}


output {
  amazon_es {
    hosts => ["vpc-:443"]
    region => "us-east-1"
    ssl => true
    index => ""
    document_id =>""
  }
}

[HarshadRanganathan]

Fine-grained access control -> Disabled
SAML authentication -> Disabled
Amazon Cognito for authentication -> Disabled

[AustinTag]

Gotcha. So if I am understanding correctly, you can actually use the regular logstash output plugin for Elasticsearch:

https://github.com/logstash-plugins/logstash-output-elasticsearch

This amazon-es plugin is needed if you are using IAM auth for access to your cluster, because it will add the credentials needed for IAM auth with an Amazon Elasticsearch Cluster.

[HarshadRanganathan]

Gotcha. So if I am understanding correctly, you can actually use the regular logstash output plugin for Elasticsearch:

https://github.com/logstash-plugins/logstash-output-elasticsearch

This amazon-es plugin is needed if you are using IAM auth for access to your cluster, because it will add the credentials needed for IAM auth with an Amazon Elasticsearch Cluster.

Yes, we could use it. But, we might want to use fine-grained access control with IAM master user for RBAC/Security/Kibana auth in the future.

Is the suggestion to go with logstash-output-elasticsearch because we have ES in a VPC and that we don't necessarily need RBAC/Security/Kibana auth?

Also, all the requests need to be signed even if we use an open domain access policy (not v4 signature). I hope that's supported in the logstash-output-elasticsearch plugin?

[HarshadRanganathan]

We've switched to logstash-output-elasticsearch plugin. Will keep this ticket open as the plugin needs a fix for anyone planning to use IAM auth with IMDS v2.

[HarshadRanganathan]

Since logstash-output-elasticsearch has added license compatibility checks, we're switching to Opensearch logstash plugin - https://opensearch.org/docs/latest/clients/logstash/index/