Add more keystore backends to support Functionbeat cloud providers

[kvch]

Right now the keystore only supports file backend. However, this is not suited to run on serverless environments where less restrictive permissions are required to read a file.

Possible backends for providers:

• AWS: https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html
• GCP: Secret manager (https://cloud.google.com/secret-manager/docs)
• Both: HashiCorp Vault

In the meantime, users should use environment variables to pass secrets to Functionbeat.

Original issue: #15808

[exekias]

ping @kaiyan-sheng @ChrsMark for awareness

[ChrsMark]

It would interesting for Kubernetes secrets too, related to #8847. Also related to #5832

[kaiyan-sheng]

For AWS credentials, IAM roles can also be used for authentication without requiring a file at the backend: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-api.html. Also, #12464 might be related too.

For other credentials/variables, AWS parameter store seems to be a good option!

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[toby-lego]

This is still an issue. We're trying to use functionbeats but don't want to put secure credentials in environment variables. Any integration with AWS Secrets Manager or System Parameter would be very useful.

[elasticmachine]

Pinging @elastic/agent (Team:Elastic-Agent)

[matthewbarreiro]

This is still an issue for me as well. We use AWS roles for AWS resources, but Functionbeat still needs a password for Elastic Cloud. We can't use environment variables because Lambda does not obfuscate in the AWS console.