[Metricbeat] Add ability to collect tags for aws services

[kaiyan-sheng]

This PR is to add the ability of collecting tags for AWS services using AWS Resource Groups Tagging API. With using get-resources, we don't have to add different code into each metricsets in order to collect from different services (Like the previous implementation: #12372).

Instead, we just need to specify resource-type-filters to specify on the resources that you want this api call to return in order to get the resources that you want with their tags. For example, specifying a resource type of ec2 returns all tagged Amazon EC2 resources, which includes tagged EC2 instances, and security groups, subnets and etc. Specifying a resource type of ec2:instance returns only EC2 instances.

Introduction on tags:
A tag is a label(with a key and a value) that users assign to an AWS resource. If there are two EC2 instances, they can be both assigned a tag key of "Stack" and the value of "Stack" might be "Testing" for one and "Production" for the other. Tagging can help organizing resources and simplifying resource management.

With this change, tags will be a part of the metrics reported by aws module:

  "aws": {
        "cloudwatch": {
            "dimensions": {
                "DBInstanceIdentifier": "test1"
            },
            "metrics": {
                "ActiveTransactions": 0,
                "AuroraBinlogReplicaLag": 0,
                "AuroraReplicaLagMaximum": 18.62599983215332,
                "AuroraReplicaLagMinimum": 18.62599983215332,
                "BinLogDiskUsage": 0,
                "BlockedTransactions": 0,
                "BufferCacheHitRatio": 100,
                "CPUUtilization": 3.2,
                "CommitLatency": 7.324593333333333,
                "CommitThroughput": 0.49995937707131555,
            },
            "namespace": "AWS/RDS"
        },
        "tags": {
            "workload-type": "other",
            "owner": "ks"
            "dept": "engr"
        }
    },

Right now, this PR is only adding tags in cloudwatch metricset. For existing s3 and sqs metricsets, we can call the same function. But for ec2, since the response from DescribeInstances api call already includes tags information, there is no need to make another get-resources api call to obtain the same information.

closes #12263

How to test it?

If you have services which aws supports tags, then add tags onto some of them. Or create services in AWS first, then add tags to them. Start metricbeat with aws module cloudwatch metricset. Please remember to specify tags.resource_type_filter in configuration.

[ruflin]

We should find a way to keep track of our binary size to be aware of changes. Just one line in such libraries could have a big impact.

@exekias @kuisathaverat Would be great to have this as a metric in the CI job so we see in Kibana when the binary size increases (ci stats)

[exekias]

Agreed, I would expect MB binary to keep growing in the short term, is that possible with jenkins already? I know we could log the binary size, but wondering if we can get it stored as a KPI we can graph.

[kaiyan-sheng]

One problem in this approach/PR that I'm not sure about is: https://github.com/elastic/beats/pull/12480/files#diff-149c3579568dabccec12ddeca87f73c7R371 how to pass in the proper --resource-type-filters into get-resources api?

Resource type format: service[:resourceType]
For example, specifying a resource type of ec2 returns all tagged Amazon EC2 resources (which includes tagged EC2 instances). Specifying a resource type of ec2:instance returns only EC2 instances.
Please see https://docs.aws.amazon.com/cli/latest/reference/resourcegroupstaggingapi/get-resources.html for more details.

What I currently have in the PR is a function called getResourceTypeUsingNamespace: most of the time, the resourceType/service name can be obtained from namespace, which is a config input for aws cloudwatch metricset. For example, AWS/EC2 is the namespace and ec2 is the resourceType/service name. But there are exceptions for sure, such as AWS/ELB is the namespace but elasticloadbalancing is the resourceType/service name.

Another solution for this is to add a separate parameter in configuration called resource-type: this will make users' life probably a bit more confusing because we are asking for both namespace and resource-type.

Or we can add the parameter for resource-type in the config but not required. If it's empty, then we will use getResourceTypeUsingNamespace method.

[jsoriano]

But there are exceptions for sure, such as AWS/ELB is the namespace but elasticloadbalancing is the resourceType/service name.

Could we have a prebuilt mapping for the ones we know, and query some API to get the rest of exceptions?

Or we can add the parameter for resource-type in the config but not required. If it's empty, then we will use getResourceTypeUsingNamespace method.

We could go for this, specially if we have some automatic mechanism so getResourceTypeUsingNamespace can always work, even if it is doing some additional query for the namespaces it doesn't know.

Also when we have light modules we can have modules for the most common resource types.

[exekias]

Or we can add the parameter for resource-type in the config but not required. If it's empty, then we will use getResourceTypeUsingNamespace method.

We could go for this, specially if we have some automatic mechanism so getResourceTypeUsingNamespace can always work, even if it is doing some additional query for the namespaces it doesn't know.

++ On having this as a config parameter, I wouldn't put much effort on a magic mechanism if it's not trivial. If I understand this correctly, probably we are using this in combination with lightweight modules, that means we can configure the resource type in the same way we are already defining the namespace.

Another note: this will need to update docs on the new needed permissions

[exekias]

btw, it is great you found a generic way of retrieving tags!! 👏 👏

[jsoriano]

LGTM, I would only consider making resource_type_filter required by now to avoid unexpected behaviours if we cannot guess the correct filter.

[jsoriano]

Not sure if this should be a struct, but if it works, I am fine with this.

[kaiyan-sheng]

jenkins, test this please