Cherry-pick #22034 to 7.x: Add awsfargate module for AWS ECS Fargate

[kaiyan-sheng]

Cherry-pick of PR #22034 to 7.x branch. Original message:

What does this PR do?

This PR is to add awsfargate module to collect monitoring metrics from AWS ECS Fargate task management endpoint for each container running under the same task as the Metricbeat is running.

Note: ~internal~ecs~pause container is skipped.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

I created an AWS CloudFormation file to create a cluster, define IAM role, create a task definition, and start the actual service.

Create an Elastic cloud env for testing and store the credentials into AWS secret manager.

aws --region us-east-2 ecretsmanager create-secret --name ELASTIC_CLOUD_AUTH --secret-string elastic:XXX

and

aws --region us-east-2 ecretsmanager create-secret --name ELASTIC_CLOUD_ID --secret-string YYYY

After storing the credentials into AWS secret manager, copy this file locally and copy the ARNs of these two secrets and put them into AWS CloudFormation yml file under TaskDefinition -> ContainerDefinition.

aws --region us-east-2 cloudformation create-stack --stack-name test-metricbeat-deployment --template-body file://./module/awsfargate/cloudformation.yml --parameters 'ParameterKey=SubnetID,ParameterValue=<subnet-id>' --capabilities CAPABILITY_NAMED_IAM

Make sure to replace<subnet-id> with your own subnet in this command. Please go to Services -> VPC -> Subnets to find subnet ID to use. You can also add several more containers under the TaskDefinition section.

Then you should see your task definition and cluster getting created, and containers start running in a minute or two.

Make sure there is no error log for the container and then go to Elastic Cloud to check for events from this module.

Clean up after test:

aws cloudformation delete-stack --stack-name test-metricbeat-deployment

Sample Event Output

data.json

Screenshots

closes #22210

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[botelastic]

This pull request doesn't have a Team:<team> label.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: [Pull request #22720 updated]

• Start Time: 2020-11-23T22:23:42.526+0000

• Duration: 69 min 43 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2117
Skipped	476
Total	2593

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	2117
Skipped	476
Total	2593