Bug: Kafka envelope return an array of arrays of values

[dreamorosi]

Expected Behavior

When parsing an event coming from Kafka, either Self managed or from MSK, the schemas for the Parser utility should return a list of values. These values should be coming from the items within each record and should be base64-decoded into strings.

Current Behavior

Currently the result of the parsing is this: [['{"key":"value"}']] aka an array of arrays with the values inside.

This is because during our initial implementation we followed the event structure and assumed that there was a relationship many:many between the record keys and the record items. I.e. considering the following sample event:

{
  "eventSource": "aws:kafka",
  "eventSourceArn": "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4",
  "bootstrapServers": "b-2.demo-cluster-1.a1bcde.c1.kafka.us-east-1.amazonaws.com:9092,b-1.demo-cluster-1.a1bcde.c1.kafka.us-east-1.amazonaws.com:9092",
  "records": {
    "mytopic-0": [
      {
        "topic": "mytopic",
        "partition": 0,
        "offset": 15,
        "timestamp": 1545084650987,
        "timestampType": "CREATE_TIME",
        "key": "cmVjb3JkS2V5",
        "value": "eyJrZXkiOiJ2YWx1ZSJ9",
        "headers": [
          {
            "headerKey": [104, 101, 97, 100, 101, 114, 86, 97, 108, 117, 101]
          }
        ]
      }
    ]
  }
}

We wrote the envelope result to accommodate the potential of multiple groups of values, one for each record, with a record being mytopic-0 and it having a list of items.

In reality this can never happen in Lambda because the integration enforces a relationship between 1 topic and one consumer, so in practice the result can be flattened to a simple list of values, i.e. ['{"key":"value"}'].

Code snippet

N/A

Steps to Reproduce

N/A

Possible Solution

No response

Powertools for AWS Lambda (TypeScript) version

latest

AWS Lambda function runtime

22.x

Packaging format used

npm

Execution logs

[github-actions]

⚠️ COMMENT VISIBILITY WARNING ⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.