out_cloudwatch_logs: Support EMF format for metrics type of events

[cosmo0920]

This PR depends on EMF encoder support on cmetrics. So, the current PR status is canary.

This PR supports metrics type of plugins handling on out_cloudwatch_logs plugin.
On ingested the metrics type of events, cmetrics will encode JSON/EMF format to prepare ingesting to the Cloudwatch Logs' metrics.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change

[SERVICE]
    Flush        10
    Daemon       Off
    Log_Level    debug
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020
    Hot_Reload   On
    Grace 5

[INPUT]
    Name fluentbit_metrics

[FILTER]
    Name stdout
    Match *

[OUTPUT]
    Name cloudwatch_logs
    Match *
    log_stream_name fluent-bit-cloudwatch
    log_group_name fluent-bit-cloudwatch
    region us-west-1
    log_format json/emf
    metric_namespace fluent-bit-metrics
    auto_create_group true

• Debug log output from testing the change

Fluent Bit v3.0.0
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________  
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \ 
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  < 
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/ 

[2024/02/24 22:30:24] [ info] Configuration:
[2024/02/24 22:30:24] [ info]  flush time     | 10.000000 seconds
[2024/02/24 22:30:24] [ info]  grace          | 5 seconds
[2024/02/24 22:30:24] [ info]  daemon         | 0
[2024/02/24 22:30:24] [ info] ___________
[2024/02/24 22:30:24] [ info]  inputs:
[2024/02/24 22:30:24] [ info]      fluentbit_metrics
[2024/02/24 22:30:24] [ info] ___________
[2024/02/24 22:30:24] [ info]  filters:
[2024/02/24 22:30:24] [ info]      stdout.0
[2024/02/24 22:30:24] [ info] ___________
[2024/02/24 22:30:24] [ info]  outputs:
[2024/02/24 22:30:24] [ info]      cloudwatch_logs.0
[2024/02/24 22:30:24] [ info] ___________
[2024/02/24 22:30:24] [ info]  collectors:
[2024/02/24 22:30:24] [ info] [fluent bit] version=3.0.0, commit=a2e0438e0d, pid=535697
[2024/02/24 22:30:24] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/02/24 22:30:24] [ info] [storage] ver=1.1.6, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/02/24 22:30:24] [ info] [cmetrics] version=0.7.0
[2024/02/24 22:30:24] [ info] [ctraces ] version=0.4.0
[2024/02/24 22:30:24] [ info] [input:fluentbit_metrics:fluentbit_metrics.0] initializing
[2024/02/24 22:30:24] [ info] [input:fluentbit_metrics:fluentbit_metrics.0] storage_strategy='memory' (memory only)
[2024/02/24 22:30:24] [debug] [fluentbit_metrics:fluentbit_metrics.0] created event channels: read=21 write=22
[2024/02/24 22:30:24] [debug] [cloudwatch_logs:cloudwatch_logs.0] created event channels: read=23 write=24
[2024/02/24 22:30:24] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Metric Namespace=fluent-bit-metrics
[2024/02/24 22:30:24] [debug] [aws_credentials] Initialized Env Provider in standard chain
[2024/02/24 22:30:24] [debug] [aws_credentials] creating profile (null) provider
[2024/02/24 22:30:24] [debug] [aws_credentials] Initialized AWS Profile Provider in standard chain
[2024/02/24 22:30:24] [debug] [aws_credentials] Not initializing EKS provider because AWS_ROLE_ARN was not set
[2024/02/24 22:30:24] [debug] [aws_credentials] Not initializing ECS Provider because AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is not set
[2024/02/24 22:30:24] [debug] [aws_credentials] Initialized EC2 Provider in standard chain
[2024/02/24 22:30:24] [debug] [aws_credentials] Sync called on the EC2 provider
[2024/02/24 22:30:24] [debug] [aws_credentials] Init called on the env provider
[2024/02/24 22:30:24] [debug] [aws_credentials] Init called on the profile provider
[2024/02/24 22:30:24] [debug] [aws_credentials] Reading shared config file.
[2024/02/24 22:30:24] [debug] [aws_credentials] Reading shared credentials file.
[2024/02/24 22:30:24] [debug] [aws_credentials] upstream_set called on the EC2 provider
[2024/02/24 22:30:24] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] worker #0 started
[2024/02/24 22:30:24] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2024/02/24 22:30:24] [ info] [sp] stream processor started
[2024/02/24 22:30:26] [debug] [input chunk] update output instances with new chunk size diff=6926, records=0, input=fluentbit_metrics.0
[2024/02/24 22:30:28] [debug] [input chunk] update output instances with new chunk size diff=6926, records=0, input=fluentbit_metrics.0
[2024/02/24 22:30:30] [debug] [input chunk] update output instances with new chunk size diff=6926, records=0, input=fluentbit_metrics.0
[2024/02/24 22:30:32] [debug] [input chunk] update output instances with new chunk size diff=6926, records=0, input=fluentbit_metrics.0
[2024/02/24 22:30:34] [debug] [task] created task=0x63cfa80 id=0 OK
[2024/02/24 22:30:34] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] task_id=0 assigned to thread #0
[2024/02/24 22:30:34] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:34] [debug] [input chunk] update output instances with new chunk size diff=6926, records=0, input=fluentbit_metrics.0
[2024/02/24 22:30:34] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream fluent-bit-cloudwatch in log group fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [upstream] KA connection #52 to logs.us-west-1.amazonaws.com:443 is connected
[2024/02/24 22:30:35] [debug] [http_client] not using http_proxy for header
[2024/02/24 22:30:35] [debug] [http_client] server logs.us-west-1.amazonaws.com:443 will close connection #52
[2024/02/24 22:30:35] [debug] [aws_client] logs.us-west-1.amazonaws.com: http_do=0, HTTP Status: 400
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] CreateLogStream http status=400
[2024/02/24 22:30:35] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Log Stream fluent-bit-cloudwatch already exists
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] cloudwatch:PutLogEvents: events=34, payload=12374 bytes
[2024/02/24 22:30:35] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Sending log events to log stream fluent-bit-cloudwatch
[2024/02/24 22:30:36] [debug] [upstream] KA connection #52 to logs.us-west-1.amazonaws.com:443 is connected
[2024/02/24 22:30:36] [debug] [http_client] not using http_proxy for header
[2024/02/24 22:30:36] [debug] [upstream] KA connection #52 to logs.us-west-1.amazonaws.com:443 is now available
[2024/02/24 22:30:36] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] PutLogEvents http status=200
[2024/02/24 22:30:36] [debug] [task] destroy task=0x63cfa80 (task_id=0)
[2024/02/24 22:30:36] [debug] [out flush] cb_destroy coro_id=0
[2024/02/24 22:30:36] [debug] [input chunk] update output instances with new chunk size diff=6926, records=0, input=fluentbit_metrics.0
[2024/02/24 22:30:38] [debug] [input chunk] update output instances with new chunk size diff=6926, records=0, input=fluentbit_metrics.0
^C[2024/02/24 22:30:39] [engine] caught signal (SIGINT)
[2024/02/24 22:30:39] [debug] [task] created task=0x9977d90 id=0 OK
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] task_id=0 assigned to thread #0
[2024/02/24 22:30:39] [ warn] [engine] service will shutdown in max 5 seconds
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [ info] [input] pausing fluentbit_metrics.0
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=fluent-bit-cloudwatch, group=fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] cloudwatch:PutLogEvents: events=34, payload=12379 bytes
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Sending log events to log stream fluent-bit-cloudwatch
[2024/02/24 22:30:39] [debug] [upstream] KA connection #52 to logs.us-west-1.amazonaws.com:443 has been assigned (recycled)
[2024/02/24 22:30:39] [debug] [http_client] not using http_proxy for header
[2024/02/24 22:30:39] [debug] [upstream] KA connection #52 to logs.us-west-1.amazonaws.com:443 is now available
[2024/02/24 22:30:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] PutLogEvents http status=200
[2024/02/24 22:30:39] [debug] [task] destroy task=0x9977d90 (task_id=0)
[2024/02/24 22:30:39] [debug] [out flush] cb_destroy coro_id=1
[2024/02/24 22:30:40] [ info] [engine] service has stopped (0 pending tasks)
[2024/02/24 22:30:40] [ info] [input] pausing fluentbit_metrics.0
[2024/02/24 22:30:40] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] thread worker #0 stopping...
[2024/02/24 22:30:40] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] thread worker #0 stopped

• Attached Valgrind output that shows no leaks or memory corruption was found

==535697== 
==535697== HEAP SUMMARY:
==535697==     in use at exit: 0 bytes in 0 blocks
==535697==   total heap usage: 46,140 allocs, 46,140 frees, 9,702,482 bytes allocated
==535697== 
==535697== All heap blocks were freed -- no leaks are possible
==535697== 
==535697== For lists of detected and suppressed errors, rerun with: -s
==535697== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.