Add the AWS guide

[melvynator]

This is the pull request to add the AWS field guide. It contains instructions for the users that want to:

• Monitor their AWS infrastructure
• Monitor their AWS logs

https://observability-docs_610.docs-preview.app.elstc.co/guide/en/observability/master/monitor-aws.html

Closes #454

[apmmachine]

A documentation preview will be available soon:

• 📚 HTML diff
• 📘 Fleet User Guide
• 📙 Observability Guide

[bmorelli25]

This is looking great! I was able to create a new AWS account and follow along with all of the steps. I have a lot of small suggestions for improvement, mostly related to grammar and small order changes.

I also have these two questions. Take these with a grain of salt as this is my first time trying out SQS and the AWS module.

1. In step two I ran into trouble because my S3 bucket and my SQS queue were deployed in different regions. I couldn't figure out how to create an SQS queue in a region other than EU-CENTRAL, so I finally just had to delete my Oregon S3 and create a new S3 in EU-CENTRAL. Should we mention that they have to be in the same region?

2. In step 5, I get the following error: awss3/collector.go:101 SQS ReceiveMessageRequest failed: EC2RoleRequestError: no EC2 instance role found. It wasn't clear to me that the fb-aws role needs to be defined. But upon rereading, you do link to documentation about it. I cannot for the life of me understand what that AWS documentation was trying to tell me to do... CLI commands? But where do I use them? I finally found this link: https://console.aws.amazon.com/iam/home?#/users which I think would be useful to document. From there I just started guessing at which permission policies were needed. I was finally able to connect after adding the following permissions:

I also had to add additional fields to the aws.yml config:

  ec2:
    enabled: true
    var.credential_profile_name: test-user
    var.access_key_id: key_id
    var.secret_access_key: my_access_key
    var.queue_url: my_url

EDIT: It looks like you included permissions for metricbeat. See #610 (comment).

[melvynator]

Thanks for the review. Really appreciate it.

In step two I ran into trouble because my S3 bucket and my SQS queue were deployed in different regions. I couldn't figure out how to create an SQS queue in a region other than EU-CENTRAL, so I finally just had to delete my Oregon S3 and create a new S3 in EU-CENTRAL. Should we mention that they have to be in the same region?

Yeah, you are right I will add it to the guide.

It wasn't clear to me that the fb-aws role needs to be defined. But upon rereading, you do link to documentation about it.

I will try to come up with the same set of minimal permission that I defined for Metricbeat. Will that solve your issue?

[bmorelli25]

I will try to come up with the same set of minimal permission that I defined for Metricbeat. Will that solve your issue?

Yes, it would. But only if you think it's useful to add. I'm new to the world of AWS, so I'm not sure what users would expect here.

[melvynator]

@bmorelli25 I finished reviewing all the suggested edits. One small note with regards to the security settings:

 ec2:
    enabled: true
    var.credential_profile_name: test-user
    var.access_key_id: key_id
    var.secret_access_key: my_access_key
    var.queue_url: my_url

The setting credential_profile_name will not be used if you populate the access_key_id and secret_access_key. The credential_profile_name is the recommended way to use AWS credentials. This setting is actually using the AWS credentials stored on your computer in ~/.aws/credentials this file can be formatted in a way were you can define different profiles with their different credential. A little bit like this:

[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

[user1]
aws_access_key_id=AKIAI44QH8DHBEXAMPLE
aws_secret_access_key=je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY

[bmorelli25]

This setting is actually using the AWS credentials stored on your computer in ~/.aws/credentials this file can be formatted in a way were you can define different profiles with their different credential.

Ahh! That makes so much more sense now. Let me give that method a try and take one more quick look at everything.

[bmorelli25]

LGTM 🎉

Let me know if you'd like me to take care of the backports for you.