Documentation for Various Lambda Use-Cases

[naftulikay]

I currently have a Lambda in Rust using the Python 3 binding using the excellent ilianaw/rust-crowbar crate.

I also spent a lot of time creating internal serde mappings for various data types, but the majority of this work is on a private repository.

What isn't clear here in the documentation is whether it's possible to catch all of the various use-cases that Lambdas can perform. My particular use case is using a Lambda for:

• Scheduled jobs via CloudWatch Scheduled Events.
• API Gateway HTTP Endpoints.
• API Gateway Custom Authorizer
• SES for Bounce Actions via Lambda.
• SES to SNS for Receiving Emails via Lambda
• S3 to SNS for Receiving S3 Events via Lambda

My Lambda essentially acts as an event bus allowing me to do just about everything a typical web application would need to do, and it's awesome. I had to do a lot of very strange things to get a build environment that mimicked the official Lambda runtime environment, so it's cool to see that now it's possible to just build an entirely statically linked binary using musl 👍

What is not clear in the existing documentation is how to integrate all of these services inside of this new Rust-native Lambda runtime. Can I use a single Rust Lambda function to address all of the use cases listed above, or is this only for API Gateway endpoints?

I've seen that in other issues, Tower is listed as a potential replacement for Hyper here, and that would be great! Basically I'd like to take advantage of native HTTP server libraries/frameworks for intelligent request routing and yet still be able to handle SNS/SES/CloudWatch/etc. events within the same Rust Lambda function. If all of this is possible, can the documentation be updated to reflect this and provide guidance on how to use it?

[naftulikay]

Also, I'm using match statements and enums to do static dispatch and it runs really fast, like 500 microseconds (half a millisecond) fast. The implementation, parts of which exist in this PR, looks something like this:

lambda!(|event, _context| {
    // dispatch event based on its type
    match serde_json::from_value::<Event>(event) {
        Ok(Event::Auth(event)) => Ok(to_value(&service::auth::route(&context, &event)).unwrap()),
        Ok(Event::CloudWatch(event)) => Ok(service::cloudwatch::route(&context, &event)),
        Ok(Event::Http(event)) => Ok(to_value(&service::http::route(&context, &event)).unwrap()),
        Ok(Event::Records(records)) => Ok(service::multi::route_all(&context, records.entries)),
        Ok(Event::Unknown(event)) => Ok(service::unknown::route(&context, &event)),
        Err(_) => {
            error!("Unable to convert event.");
            Ok(json!({ "statusCode": 400 ,"message": "Unable to convert event." }))
        }
    }
});

Above is the entrypoint into the Lambda function. Next, an implementation of dispatching events from SNS:

pub fn route(context: &Context, record: &sns::Record) {
    info!("Received SNS Event: {}", record.event.message_id);

    match from_str(&record.event.message) {
        Ok(SnsEventKind::Ses(message)) => service::ses::receive(&context, &message),
        Ok(SnsEventKind::Unknown(value)) => warn!("Unknown JSON Event: {}", to_string_pretty(&value).unwrap()),
        Err(_) => error!("Unknown non-JSON event: {}", record.event.message),
    };
}

When receiving a list of Record types (e.g. multiple events within a single payload/call to Lambda), the following is done:

/// Dispatch multiple events.
pub fn route_all(context: &Context, events: Vec<Record>) -> Value {
    for event in events {
        match event {
            Record::S3(event_record) => s3::route(&context, &event_record),
            Record::Ses(event_record) => {
                // if it's an SES request/response, a response may be required, so only process one
                // of them if the type is RequestResponse
                match event_record.event.receipt.action {
                    Action::Lambda(ref action) if action.invocation_type == RequestResponse => {
                        return to_value(ses::filter(&context, &event_record)).unwrap();
                    },
                    _ => ses::route(&context, &event_record.event),
                };
            },
            Record::Sns(event_record) => sns::route(&context, &event_record),
            Record::Unknown(value) => {
                error!("Unknown event record type: {}", to_string_pretty(&value).unwrap())
            }
        };
    }

    Value::Null
}

I've found this to be extremely flexible and powerful. Would love to see the documentation demonstrate routing API Gateway endpoints as well as at least one other type of event for clarity. Also, I see that hyper is mentioned, but is that only for an HTTP client or for a server implementation as well?

[davidbarsky]

Thanks for opening this issue! I'll try to address some of the questions—not all it's a bit late, sorry!

I also spent a lot of time creating internal serde mappings for various data types, but the majority of this work is on a private repository.

We're working with people internally to generate CloudWatch Event types for all services. I believe they've picked it up during sprint planning today. I'll ask them to confirm it, either via email or here.

What is not clear in the existing documentation is how to integrate all of these services inside of this new Rust-native Lambda runtime. Can I use a single Rust Lambda function to address all of the use cases listed above, or is this only for API Gateway endpoints?

Yes, you can a single Rust-based Lambda function to address those use-cases. If you can't please file a bug.

I've seen that in other issues, Tower is listed as a potential replacement for Hyper here, and that would be great! Basically I'd like to take advantage of native HTTP server libraries/frameworks for intelligent request routing and yet still be able to handle SNS/SES/CloudWatch/etc. events within the same Rust Lambda function. If all of this is possible, can the documentation be updated to reflect this and provide guidance on how to use it?

We have to first migrate to Tower! When we re-define the the lambda runtime/handler in terms of Tower's Service trait (as opposed to how we're kinda doing things now, defining the Lambda function in terms of a single, synchronous function), a lot of nice patterns will fall out. Those include the points you raised about using Tower as a de-facto router between various AWS services.

I've found this to be extremely flexible and powerful. Would love to see the documentation demonstrate routing API Gateway endpoints as well as at least one other type of event for clarity.

Agreed, yes. I think we want to have #18 land first.

Also, I see that hyper is mentioned, but is that only for an HTTP client or for a server implementation as well?

Right now, we're mostly using it in a client implementation. I'm not entirely sure how you'd be able to user Hyper as an HTTP server in Lambda because of runtime limitations, but if you're able to figure that out, I think our Infosec teams would be very interested in knowing how you pulled that off.

[sapessi]

This often comes down to preference. Many customers prefer to use a separate function for each event. Others, prefer to group multiple events in a single function. The downside to the former approach is that there a fair bit of code duplication. However, the downside of the latter is that you end up deploying the code to handle all event types each time, potentially increasing the blast radius of even small changes.

The way the runtime is built right now only allows to handle a single event type. We could make your use-case work by having a separate handler type that receives the raw payload from the runtime APIs as a &[u8], you could then use serde to deserialize it yourself.

[lukaspustina]

No sure, this helps, but I currently do process different event types using the 0.1.0 runtime version using Serde's Value type as input.

For example, this shows the important part of a lambda function that reacts to autoscaling events. In addition to the autoscaling event type, all our functions react to a "ping" event for manual invocation. In this way, we check that all permissions and configurations for the function are set correctly.

use serde_json::{self, Value};
use aws_lambda_events::event::autoscaling::AutoScalingEvent;

#[derive(Debug, Deserialize)]
#[serde(untagged)]
enum Event {
    ASG(AutoScalingEvent),
    Ping(Ping),
}

#[derive(Debug, Deserialize)]
struct Ping {
    ping: String,
}

pub fn handler(input: Value, ctx: Context) -> Result<(), HandlerError> {
    let aws_request_id = ctx.aws_request_id.clone();

    let event: Event = serde_json::from_value(input)
        .map_err(|e| ctx.new_error(e.to_string().as_str()))?;
    debug!("Parsed event = {:?}", event);

    let res = match event {
        Event::Ping(ping) => handle_ping(ping, &ctx),
        Event::ASG(asg) => handle_asg(asg),
    };

    ....
}

[davidbarsky]

@lukaspustina That does help, thank you!

[sapessi]

Hey @naftulikay - Yesterday we release version 0.2 of the runtime that includes a lambda_runtime_core crate. The core crate accepts a Handler that receives a Vec<u8> and returns a Vec<u8>. This would enable your use-case with different events in a single Lambda function. Take a look at the basic example.

[naftulikay]

@sapessi Excellent! Does this still start up hyper internally? If not, what's the overhead as far as the stack is concerned?

If it's just the couple method calls in the example then that's AWESOME.

For my architecture, I'm going to try to find a Rust web framework that allows triggering endpoints without any network I/O, similar to how NGINX does internal requests when you're working with OpenResty.

[sapessi]

The core runtime performs the same work of the original runtime. Uses the hyper client to poll the runtime APIs and then serves events (without reading them) to the Handler method. It does not add any overhead because we do not look at the payload at all, we simply send you the buffer we received from the APIs (through hyper). The runtime crate has changed so that now it is only a wrapper on top of the core runtime.

[naftulikay]

So, from what I can find, Handler is a trait and the implementation that uses Handler uses impl Trait to provide static dispatch which should be really fast 👍

In lambda_runtime_core::runtime, I can't seem to find any mention of hyper, which is good.

I originally had thought that at runtime, Hyper was actually binding a port and then making a call over the loopback interface to reach the handler. This no longer seems to be the case. From what I was able to discern in about ten minutes of digging, the only overhead seems to be two to four stack frames, which is amazing 🎉

Is there anything I'm missing or is my understanding correct?

[softprops]

You'll want to look at the client https://github.com/awslabs/aws-lambda-rust-runtime/blob/50b71ae765a198eb62d0888958633bdf4421c5f6/lambda-runtime-client/src/client.rs

[davidbarsky]

Yep, the client makes extensive use of Hyper and Tokio. With 0.3, I intend to make the internals entirely asynchronous, which will allow us to use systems like Tokio Trace for various instrumentation purposes.

I’m curious as to what complaints/concerns you have about Hyper—would you be willing to elaborate?

[davidbarsky]

Ah, I missed this bit:

For my architecture, I'm going to try to find a Rust web framework that allows triggering endpoints without any network I/O, similar to how NGINX does internal requests when you're working with OpenResty.

I'm not too familiar with how Nginx, but you might be able to use HTTParse for the sans-IO implementation.

[softprops]

I'm not that familiar with how you can build networked services with involving io but if you're looking for something more domain socket based for local communication https://github.com/softprops/hyperlocal may be relevant. Original usecase for this docker daemon style host-local communication but it would be interesting to try something like that in lambda like context

[naftulikay]

So I've done some digging but I'd like a clarification if possible on what actually happens at runtime.

I was previously using crowbar to compile a Rust CPython extension which used the Python runtime to execute Rust. I can't remember what the architecture was, but I don't think that any networking was required for the Python client, from what I can tell, Lambda simply imported the Python module and called the entrypoint method from within the Python VM, just passing essentially a Python data structure via a method call.

For the new Rust support provided here, does Rust bind a TCP port and listen via HTTP, or is something else going on here? If it's handling a HTTP request, has anyone done any benchmarks on the overhead, however slight? With crowbar, I was seeing around 500 microsecond executions via Serde dispatch over the received JSON type.

As for my other use-case: I'd like to route API Gateway events using some HTTP framework so that I don't end up with a massive match statement with regular expressions as I currently have.

[softprops]

For the new Rust support provided here, does Rust bind a TCP port and listen via HTTP?

Yes. Runtimes resolve the host and port from an envelope variable provided by the lambda execution
task https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html#runtimes-custom-build

The lifecycle of a runtime can be generalized as a loop that executes the steps under the "Processing Tasks" section of the page linked above.

If it's handling a HTTP request, has anyone done any benchmarks on the overhead, however slight? With crowbar, I was seeing around 500 microsecond executions via Serde dispatch over the received JSON type.

Not that I'm aware of any. I was also a user of crowbar. I wasn't able to benchmark its cpython => serde bridge but I did benchmark a serde optimized schema for apigateway serde value => structures and that was about 8 microseconds.

I think (only guessing here) what you'll see is that the aws custom runtimes are userland executions of use the same internal API/loop processing that existing language specific runtimes aws implemented in the past.

What I have observed to be the biggest difference between crowbar and the rust runtime deployments is deployment artifact sizes.

Today's rust runtime gets included as a library dependency which is a difference kind of performance cost in the overall lambda task life cycle. With crowbar the runtime doesn't get redeployed with your function changes. I think there are ways to slim rusts runtimes down with dependency trimming where dependencies are not absolutely needed but I feel like there may also be a story for having it live in a lambda layer with your function code linking to a library that speaks a protocol for communicating with that layer. It's an optimization with its own trade offs though. You give up local function calls for some IPC ( which requires more serialization overhead and more network io complexity ) I'm actually kind of okay with the runtime as a library for now because I'm sure AWS likely has some magic internal s3 apis for optimizing downloading for larger artifacts a problem you down have to solve.

I think it is worth measuring before making extra optimizations at this point though

[dnagir]

It would be great to be able to have a very simple routing layer specifically for the API Gateway events.

This should allow using a wildcard API Gateway endpoint and do the routing within Rust, similarly how actix-web routing or rocket routing work.

[naftulikay]

@dnagir yes, this is exactly what I'm looking for: there should be a way to translate API Gateway events into some format to be able to internally submit requests to a more mature HTTP backend, preferably without going over the network.

Have you found anything that you can use to route HTTP internally?

[sapessi]

Pinging @softprops on this. Doug originally built the lambda-http crate. I'm assuming you are looking for an additional crate similar to serverless-java-container and serverless-express? We should track it in a separate issue.

[naftulikay]

Just asked around in Actix Web, and it doesn't seem that it's possible right now (APIs don't exist) to:

1. take an ApiGatewayProxyRequest
2. convert it into an actix_web::HttpRequest
3. dispatch it into actix-web
4. receive the response and return it

Step three, dispatching internally into actix-web, is not possible right now.

Chat logs here.

no, you can not use actix for this purpose, you have to build it
but you can take parts of actix
for example router https://github.com/actix/actix-net/tree/master/router
then you can use actix-net for building middlewares
something similar to how actix-web::App does middlewares (1.0 branch)
should be relatively easy to do

"should be relatively easy to do" remains to be seen.

Rocket also doesn't seem to have an API for this.

Again to clarify my use-case, I want to:

• Receive arbitrary Lambda events of many different types (API Gateway, SNS, SES, S3, CloudWatch)
• Take API Gateway Proxy events, convert them into something, and dispatch them locally in-memory to a sophisticated HTTP framework with bells and whistles like middleware, etc.
• In my HTTP handlers, I'll be doing async stuff to call AWS APIs and other things where having parallelism and concurrency will help.
• After receiving a response from the HTTP framework, convert that into an appropriate API Gateway response.

[softprops]

I can take a stab at poc but I'm going to echo that in using this, you are losing a value by not fully leveraging what the platform provides for you.

Apigateway is a request router and is the "server" that replaces the need for a server like actix and rocket.

Building an http router inside of a system that's essentially an http router should really be a special case and not a norm. A core idea behind lambda is that functions should really just serve one operation. If that model doesn't fit your application you may actually want to consider fargate, an equally good system, as a deployment target.

Sometimes it worth picking a deployment target that fits your application over trying make your application fit a deployment target.

[naftulikay]

I understand your line of reasoning, but there are (I believe) valid use-cases for what I'm doing:

• Fargate cannot compete with Lambda in terms of pricing. 1/4 vCPU costs around $7-8 per month. My Lambda usage is pennies on the dollar. I'm building a serverless web API for my own personal use and cost is critical. I may consider using Fargate for things that can't run in Lambda.
• Fargate doesn't have a good story around mapping Route 53 records to container public IP addresses. There are some third-party Lambda functions which bind this functionality together.
• I was hoping to use CoreOS for its coordinated auto-upgrade feature as a backend to ECS, but CoreOS doesn't support the awsvpc networking mode.
• If you had to write a web API in Lambda, would you honestly write different Lambda functions for every resource? I have about 12 different resources, and this would result in me having to maintain 12 CI/CD systems and the Terraform to bind API Gateway together like this.

I understand that perhaps my use-case isn't the most common one. I have only posted the above as potential documentation for anyone trying to do what I'm doing. I fully understand that we can't possibly ask the AWS team to implement anything like this.

[dnagir]

A core idea behind lambda is that functions should really just serve one operation.

@softprops it's great in theory to deploy one lambda per endpoint but it's horrible in practice.
For 100 or so endpoints (which isn't a large API), uploading and deploying 100 different 100MB binaries that share most of the code is extremely inefficient, hard and error prone even with automatic pipeline.

There are examples that run on a single lambda (Ruby on Jets for example).

I can understand a couple different lambdas pet logical set of functionality though.

[naftulikay]

I did some legwork and found a way to do what I'm looking to do. There's a rather obscene document that I made while researching this, but I'll summarize my findings. My admittedly foul language is no reflection on the help that I've received here (can't thank you all enough), rather it's Monday and I'm tired.

• Tide is built on Hyper, which is what the Lambda runtime crate uses internally.
• Internally, the lambda! macro blocks indefinitely polling for new requests via Hyper.
• Tide's App::serve() method also blocks indefinitely.
• Tide's App can be converted into a http_service::HttpService, which has a respond method, which can be served requests without any network overhead 🎉
• Strategy: create a background thread for Tide, but need to somehow bring the Tokio executor over to that thread somehow. Tide is Hyper and Hyper uses tokio::spawn to spawn things using the local TLS Tokio executor.
• Lambda and Tide can both use Tokio by means of Hyper.
• Write a trait implementation for Into<http::Request<T>> for the API Gateway HTTP Proxy request type and a From<http::Response<T>> for converting the result back into something that Lambda understands.
• ❓ ❓ ❓
• Profit

I haven't tried this yet and I hope it works.

If it does, this gets me exactly what I need:

• Very limited overhead.
• A real HTTP framework to make things sane, including middleware.

[davidbarsky]

@naftulikay It's in progress, but I'm re-writing the Lambda runtime to be fully asynchronous—you can see some of in-progress work here. When published to crates.io, most of the internal would be implemented in terms of a Tower service, which is a more generalized form of http_service::HttpService. I think the two things in tandem would allow allow for supporting arbitrary frameworks, so long they build on Tower (or have some sort of From<tower::Service> implementation).

[naftulikay]

@davidbarsky awesome!

So what I see here is that there's a specific path that the Lambda runtime listens on:

/2018-06-01/runtime/invocation/next

With your reimplementation, will it be possible to expose Tower so that users can register routes?

Thank you for putting in the (hard) work of making it possible to use Rust asynchrony in Lambda functions 👍 I'm sure that this will drive costs even further down and dramatically increase performance!

[davidbarsky]

@naftulikay

That path is used for communicating with the Lambda Runtime APIs—Tower allows you to compose middleware that can be used on both clients and servers.

With your reimplementation, will it be possible to expose Tower so that users can register routes?

Maybe. I don't know enough for sure, but I don't see why not.

[OJFord]

@lukaspustina #29 (comment)

Many thanks for sharing that - aws_lambda_events is exactly what I was looking for when I found this issue. IMO it should be re-exported by this crate, to aid discoverability. (It would then show at https://docs.rs/lambda_runtime, even if not used in the example.)

[calavera]

I'm going to close this conversation since it looks stale. We've made several improvements lately, including integrating with Tower, and we have examples in the https://github.com/aws-samples organization. Feel free to leave a comment here if you think we should discuss this topic further.