User/Password Authentication

[Oscarchase]

Hi,

I'm trying to build an ELK infrastructure, using AWS Fine Grained Authentication, the problem I'm encountering right now is making Logstash connect to the ES cluster.

When launching logstash I get the following error:

[2020-03-30T14:14:48,731][WARN ][logstash.outputs.amazonelasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://user:xxxxxx@ES_ENDPOINT:443/", :error_type=>LogStash::Outputs::AmazonElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '403' contacting Elasticsearch at URL 'https://ES_ENDPOINT:443/'"}

Although when I try to curl the ES_ENDPOINT from the instance with the same credentials it works fine.

Here is the output configuration file I'm using:

output {
        amazon_es {
            hosts => "${ES_ENDPOINT}"
            index => "%{type}-%{+YYYY.MM.dd}"
            region => "${AWS_REGION}"
            user => username
            password => password
        }
}

The user being a master user created when creating the ES cluster.

[wayne00]

encounter the same question

[ghost]

i have the same problem and how did you do?

[lwiedenhoeft]

We are facing the same issue.

[rlewkowicz]

@SharpEdgeMarshall

I'm pinging random people with commits on this repo. There's a lot of questions around authentication and how different methods play together which means your documentation or functionality is problematic. We're all paying customers if we're using this service, is there any channel for meaningful support on this?

[AustinTag]

This appears to be a bug. We will prioritize a fix for this.

[SharpEdgeMarshall]

@rlewkowicz I'm not an Amazon employee, just a contributor 😂

[mahgoyal]

Need this feature to index data to Amazon ES 7.4 with FGAC enabled.

[salim23]

+1 having this issue

[zqyong]

+1

[AustinTag]

Hi all, in the meantime, if you are using fine grained access control with username/password and have made your access policy open, then you can use the regular Logstash Elasticsearch output plugin to unblock yourselves while we work on a fix: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html

[AustinTag]

The above is assuming you are not using IAM to control access in addition to username/password

[iamanush]

Hi all, in the meantime, if you are using fine grained access control with username/password and have made your access policy open, then you can use the regular Logstash Elasticsearch output plugin to unblock yourselves while we work on a fix: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html

Please use this as a temporary workaround. We will work on a fix to the root-cause.

[zeke13210]

How does one define an open access policy for amazon @iamanush? I made mine to be something like below however I'm still encountering the below error

ERROR:
[2021-01-27T11:40:13,408][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://unboxed:xxxxxx@search-unboxed-prod-wumt3rinwbyvje4bwaoyhemcxq.us-east-1.es.amazonaws.com:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://unboxed:xxxxxx@search-unboxed-prod-wumt3rinwbyvje4bwaoyhemcxq.us-east-1.es.amazonaws.com:9200/][Manticore::ConnectTimeout] connect timed out"}

ACCESS POLICY
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": ""
},
"Action": "es:",
"Resource": "*"
}
]
}

LOGSTASH OUTPUT
output {
elasticsearch {
hosts => "https://search-unboxed-prod-wumt3rinwbyvje4bwaoyhemcxq.us-east-1.es.amazonaws.com/"
ssl => true
user => 'unboxed'
password => 'fakepassword'
index => "shoe"
document_id => "%{id}"
}
stdout { codec => rubydebug }
}

[HiranPeri]

@zeke13210 - I am struggling with the same problem
Anyone here can offer help?

[AustinTag]

The following access policy is what I have tested. This was generated using the dropdown menu for "Domain Access Policy" and choosing "Allow open access to domain":

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": ""
},
"Action": "es:",
"Resource": "YOUR_DOMAIN_ARN/*"
}
]
}

Please see below screenshot for dropdown menu. This can be found by going to Actions -> Modify access policy from the domain dashboard.

Please be sure that your domain is accessible from the host from which you are using logstash on. You can do this by trying a curl request to the domain from that host with the username and password:

curl -u username:password DOMAIN_ENDPOINT/_cluster/health - for example

[aruandre]

this issue occurs with instance profile credentials aswell

[T0m-Ph]

Hi! Is there any update on this? I am facing the same today

[ecout]

Your problem is right there in the conf file output block...is obvious as of 06/04/2021 now that the Readme contains the logstash configuration instructions. At this point the solution is a matter of RTM:
Why are you using username and password when you've enabled Fine Grained Access Control and are using this plugin to make AWS SigV4 requests to their API? You ought to be using Programmatic Access keys from an IAM user that's authorized in the cluster as a backend role.

Your config:

output {
        amazon_es {
            hosts => "${ES_ENDPOINT}"
            index => "%{type}-%{+YYYY.MM.dd}"
            region => "${AWS_REGION}"
            **user => username
            password => password**
        }
}

plugin config.

output {
    amazon_es {
        hosts => ["foo.us-east-1.es.amazonaws.com"]
        region => "us-east-1"
        # aws_access_key_id and aws_secret_access_key are optional if instance profile is configured
        **aws_access_key_id => 'ACCESS_KEY'
        aws_secret_access_key => 'SECRET_KEY'**
        index => "production-logs-%{+YYYY.MM.dd}"
    }
}