Handle base64 encoding

[cmuto09]

Re-raising this as it turns out latest versions did not include a fix for the request encoding issue that was causing #464.

This is the same general concept as PR #743 that I closed, but I pulled the other issue in that code into another PR so that this could stay focused on mimicking API Gateway's binary support, including providing isBase64Encoded with the event.

I bubbled the definition of base64 encoded content types up to the serverless config, but I'm not sure that's the best way. The intent was to make an easy interface with plugins like serverless-apigw-binary, but definitely open to other ways of handling it!

[dnalborczyk]

thank you @cmuto09 !

could you provide a full integration test case as well so I could understand what the plugin does not support at this time? like a serverless.yml and a handler.js would be great! it doesn't need to be hooked up to jest yet, as we haven't done it yet to full extend. we need more end-to-end tests, and I'd be happy to add one for this.

I also want to make sure that yet another option is really needed to support this. (I'm in the middle of removing useless options)

[cmuto09]

Hey @dnalborczyk, no problem!

Just to clarify- you want me to add an example of a handler/serverless.yml combo that implement this feature into the integration folder? I think I understand, just want to make sure.

As for the option question, totally agreed. I was trying to figure out if all the encoding should happen behind the scenes or whether an option was necessary.

My current deploy configuration includes this:

apigwBinary:
    types:
      - 'application/octet-stream'
      - 'image/png'
      - 'image/jpg'
      - 'image/jpeg'
      - 'application/bson'
      - '*/*' # need this for browser conversion of images

so it was easy for me to pass that as a variable into the option, but if that set of types suffice then I could change detectEncoding to look for them specifically (obviously except */*), and convert to base64. What do you think?

[dnalborczyk]

Just to clarify- you want me to add an example of a handler/serverless.yml combo that implement this feature into the integration folder? I think I understand, just want to make sure.

yeah, that would be great. something like this, or similar, if you could:
https://github.com/dherault/serverless-offline/tree/master/__tests__/integration/uncategorized

nothing too complicated. just the minimum what's needed to outline usage and expectations.

if not, no problem, at least something manually I could look at, to better understand what's needed.

As for the option question, totally agreed. I was trying to figure out if all the encoding should happen behind the scenes or whether an option was necessary.

My current deploy configuration includes this:

apigwBinary:
    types:
      - 'application/octet-stream'
      - 'image/png'
      - 'image/jpg'
      - 'image/jpeg'
      - 'application/bson'
      - '*/*' # need this for browser conversion of images

so it was easy for me to pass that as a variable into the option, but if that set of types suffice then I could change detectEncoding to look for them specifically (obviously except */*), and convert to base64. What do you think?

I probably have a better answer to your question once I see your use case.

[cmuto09]

@dnalborczyk great, can do! I'm gonna finish up working on some pesky issues with my other PR (didn't mean to link) first then I'll add my use case

[cmuto09]

@dnalborczyk added a basic use case that decodes a base64 encoded image, fakes some processing, and returns the processed image.

The part that failed in my actual code was the BytesIO(image) call as the utf-8 conversion in detectEncoding was corrupting it, so if that call is working then everything should be okay

[dnalborczyk]

@cmuto09 thanks for the test, I got it running!

couple observations: import json is missing, as well as base64.base64encode is I guess base64.b64encode? keep in mind, I'm not a Python guy.

Also one image test upload caused an endless? loop, or it wasn't processing at all, not sure. I think that's a python issue from the other PR.

[cmuto09]

Hey @dnalborczyk- sorry for the bad python- it was such a simple script that I didn't end up running it, turns out I wrote pseudocode. It is b64decode/b64encode. My bad!

Happy to take a look at the infinitely hanging image to see if it was an issue with not recognizing JSON in the response body (though the symptom of that issue was getting the log Failed! <json response> not hanging indefinitely), or something else.

Let me know how I can be helpful!

[Joebayld]

Hey all. I'm hoping to use this PR. Any idea when it will get merged?

[thetumper]

Looks like all tests passing, and just needing conflicts resolved. Any chance this could get done and released soon? Really needing this one.

[racedale]

Also looking to get this merged

[gabmontes]

@cmuto09 @dnalborczyk I've been dealing with this issue and before finding this PR, I came up to a solution that is very similar.

But instead of adding a new option to serverless-offline, I chose to get the media types from serverless.yml at provider.apiGateway.binaryMediaTypes. That is inline with what serverless uses to configure the API Gateway on a deploy (see the docs).

You can take a look at my initial draft here: master...gabmontes:binary-media-types

This is an awesome project! Hope we can have this implemented/merged soon and let me know if/how I can help for this to happen.

[dherault]

2 yo PR, closing sorry, please feel free to reopen