SerializationError using Lex v1 bot from aws-sdk-go-1.33.21 onwards

[chrbsg]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone though Developer Guide and API reference
• I've checked AWS Forums and StackOverflow for answers
• I've searched for previous similar issues and didn't find any solution

Describe the bug
Create a lex bot using the "OrderFlowers" example

Version of AWS SDK for Go?
1.33.21 up to 1.42.41

Version of Go (go version)?
go version go1.17.4 linux/amd64

To Reproduce (observed behavior)
I don't have a standalone example, but could create one if required. There's nothing fancy here, just:

                pci := &lexruntimeservice.PostContentInput{
                        Accept:      aws.String(accept),
                        BotAlias:    aws.String(lexBotAlias),
                        BotName:     aws.String(lexBotName),
                        ContentType: aws.String(contentType),
                        InputStream: aws.ReadSeekCloser(pr),
                        UserId:      aws.String(sessionID),
                }

                req, res := lexRuntime1.PostContentRequest(pci)
                err := req.Send()

Now send an audio stream and say "I'd like to book a room"
aws-sdk-1.33.20 will return no error.
aws-sdk-1.33.21 and later req.Send will return:

SerializationError: failed to unmarshal response error
        status code: 200, request id: 6591487f-cfb4-44d8-b50d-76f4206b6f42
caused by: UnmarshalError: failed decoding error message
        00000000  00 00 00 00 ff ff 00 00  00 00 00 00 00 00 00 00  |................|
00000010  00 00 00 00 ff ff 00 00  00 00 00 00 ff ff 00 00  |................|
00000020  ff ff 00 00 ff ff ff ff  00 00 ff ff ff ff ff ff  |................|
00000030  ff ff ff ff 00 00 ff ff  00 00 ff ff 00 00 00 00  |................| 
00000040  00 00 ff ff ff ff ff ff  ff ff 00 00 ff ff 00 00  |................|
00000050  00 00 ff ff 00 00 ff ff  00 00 ff ff ff ff 00 00  |................| 
00000060  00 00 ff ff 00 00 00 00  ff ff ff ff 00 00 00 00  |................|
00000070  00 00 ff ff 00 00 00 00  ff ff fe ff ff ff 00 00  |................| 
00000080  00 00 00 00 00 00 00 00  ff ff 00 00 ff ff 00 00  |................|
00000090  ff ff fe ff 01 00 00 00  00 00 ff ff ff ff ff ff  |................| 
000000a0  ff ff fe ff ff ff 00 00  00 00 00 00 ff ff 00 00  |................|
000000b0  01 00 00 00 00 00 ff ff  ff ff 00 00 ff ff 00 00  |................| 
000000c0  00 00 ff ff ff ff ff ff  ff ff 00 00 00 00 01 00  |................|
000000d0  ff ff ff ff 00 00 01 00  00 00 00 00 01 00 ff ff  |................| 
000000e0  ff ff 00 00 fe ff ff ff  fd ff fe ff ff ff fe ff  |................|
000000f0  ff ff fe ff 00 00 ff ff  ff ff ff ff 00 00 01 00  |................| 
00000100  00 00 01 00 01 00 02 00  03 00 04 00 05 00 08 00  |................|
00000110  0b 00 0e 00 11 00 16 00  1b 00 22 00 28 00 30 00  |..........".(.0.| 
00000120  38 00 3f 00 46 00 4e 00  55 00 5f 00 6c 00 7a 00  |8.?.F.N.U._.l.z.|
00000130  8b 00 9b 00 ac 00 b9 00  c2 00 c9 00 cb 00 c9 00  |................| 
00000140  c3 00 b9 00 ab 00 9d 00  8c 00 79 00 63 00 4b 00  |..........y.c.K.|
00000150  32 00 13 00 ec ff bd ff  8e ff 56 ff 1d ff e4 fe  |2.........V.....| 
00000160  ab fe 71 fe 36 fe f4 fd  aa fd 63 fd 23 fd e4 fc  |..q.6.....c.#...|
00000170  ab fc 75 fc 49 fc 2a fc  11 fc e5 fb b3 fb 77 fb  |..u.I.*.......w.| 
00000180  3d fb 0c fb d3 fa 99 fa  62 fa 31 fa 0c fa f0 f9  |=.......b.1.....|
00000190  ed f9 f2 f9 09 fa 38 fa  85 fa ed fa 6d fb 16 fc  |......8.....m...| 
000001a0  e3 fc c0 fd 9e fe 70 ff  1f 00 b7 00 49 01 d5 01  |......p.....I...|

Expected behavior
No error

Additional context
I've verified that merely changing github.com/aws/aws-sdk-go v1.33.20 to github.com/aws/aws-sdk-go v1.33.21 in go.mod is sufficient to trigger the bug, so it seems likely that this is a bug that was introduced in to the SDK in v1.33.21.

[chrbsg]

It looks like this was already reported way back in April 2021 (!) - see issue #3847

To be clear, lexruntimeservice PostContent always returns an error when the input is audio or text and the response contains a X-Amz-Lex-Alternative-Intents header. This header is a base64 encoded JSON array. The base64 text decodes to something like:

[
  {
    "intentName": "BookCar",
    "nluIntentConfidence": {
      "score": 0.61
    },
    "slots": {
      "PickUpDate": null,
      "ReturnDate": null,
      "DriverAge": null,
      "CarType": null,
      "PickUpCity": null
    }
  },
  {
    "intentName": "AMAZON.FallbackIntent",
    "nluIntentConfidence": null,
    "slots": {}
  }
]

This can't be unmarshalled to aws.JSONValue because aws.JSONValue is a map (type JSONValue map[string]interface{}). The error is:

SerializationError: failed to decode REST response
caused by: json: cannot unmarshal array into Go value of type aws.JSONValue

[jasdel]

Thanks for taking the time to create this issue I've created #4269 to address this issue. Though it will require a breaking change to the API model for this parameter. We're choosing to make a breaking change here because the API never could work with the SDK given the JSONValue implementation.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.