Create Data Mappings for Lambda Input and Output Events

[naftulikay]

TODO

• CloudWatch Events
  • API Call Events (CloudTrail)
  • Autoscaling Events
  • Batch Events
  • CodeBuild Events
  • CodeCommit Events
  • CodeDeploy Events
  • CodePipeline Events
  • Management Console Sign-in Events
  • EBS Events
  • EC2 Events
    • Maintenance Windows Events
  • ECS Events
  • GameLift Events
  • EMR Events
  • Glue Events
  • GuardDuty Events
  • Health Events
  • KMS Events
  • Macie Events
  • OpsWorks Events
  • Scheduled Events
  • Server Migration Service Events
  • Systems Manager Events
    • Configuration Compliance Events
    • Parameter Store Events
  • Trusted Advisor Events

JFC CloudWatch 💀

• Lambda Events (sample events)
  • Alexa
  • API Gateway
  • CloudFormation
  • CloudFront
  • CloudWatch Logs
  • CloudWatch Events
  • CodeCommit
  • Cognito
  • Config
  • DynamoDB
  • Invocation
  • IoT Button
  • Kinesis Data Firehose
  • Kinesis Data Streams
  • Lex
  • S3
  • SES
  • SNS
  • Scheduled Events

Pull #31 is going to get some of this.

[naftulikay]

Please be advised that Amazon has since added more metadata to their calls, so the difficulty here is that things will be masked by serde only deserializing the fields it knows about.

[softprops]

This is awesome! I was actually motivated enough to start a bridge project https://github.com/softprops/lando to add event types ( api gateway is all I needed ) . What is the likely hood we could iterate and cut releases quickly with this. I would love to just start using this today. That's a big list of checkboxes and I'd love to just get going with a release version.

Aside. I took a slightly different approach that users might appreciate. When deploying a lambdas its atypical for a single function to service multiple types of events. Rather is more common for a single lambda to target a single type of trigger.

A slightly different approach I was taking was to have a named macro per event type, in this case

gateway!(
  |request, context| Ok(lando::Response::default())  
);

So that a lambda targeting a certain trigger would only have to focus on a function types that are relevant vs having to on one extra level of pattern matching. I think something like that could be added in addition to this approach separately.

Either way this looks great and I don't want to duplicate effort. How likely would it be that we could break up this large list into smaller lists and release more often?

[naftulikay]

Yeah, let's break things into pieces. Packages should be nested as represented above.

FWIW my use case is a single Lambda function dispatching all the events.

[softprops]

I think thing the single lambda is a really good default. I'm excited to see where this goes!

[softprops]

I wasn't sure where the best place to post this would be this seemed reasonable. I just announced a crate release that builds on on crowbar that adds bindings for the standard http crate to crowbar https://twitter.com/softprops/status/1003458005852196870. I can likely simplify my implementation when some of these the changes in this issue get addressed.

My workflows for aws lambda mostly revolve around the serverless framework so I made a serverless plugin to help facilitate rustlang applications https://github.com/softprops/serverless-rust. This should also by extension work well with any crowbar application.

[LegNeato]

Ya'll may be interested in https://github.com/srijs/rust-aws-lambda/tree/master/aws_lambda_events. I generate the event structs from Go. I'm going to publish it as its own crate soon.

[softprops]

What's the status on this and how I help push it forward?

[naftulikay]

@softprops I think we need to consolidate around a singular place for these definitions and I'm not sure that this is the best repository for it, we should probably put it in rusoto or something like that. FWIW here is some sampling of what I'm doing with my own Lambda API:

lambda!(|event, _context| {
    INITIALIZE.call_once(|| {
        logging::initialize();
        threading::initialize();
        debug!("Finished one-time initialization.");
    });

    if env::var("DEBUG").map(|s| s == "true").unwrap_or(false) {
        // if we're in debug mode, dump
        debug!("Event: {}", to_string_pretty(&event).unwrap());
    }

    // dispatch event based on its type
    match serde_json::from_value::<Event>(event) {
        Ok(Event::Auth(event)) => Ok(to_value(&service::auth::route(&event)).unwrap()),
        Ok(Event::CloudWatch(event)) => Ok(service::cloudwatch::route(&event)),
        Ok(Event::Http(event)) => Ok(to_value(&service::http::route(&event)).unwrap()),
        Ok(Event::Records(records)) => Ok(service::multi::route_all(records.entries)),
        Ok(Event::Unknown(event)) => Ok(service::unknown::route(&event)),
        Err(_) => {
            error!("Unable to convert event.");
            Ok(json!({ "statusCode": 400 ,"message": "Unable to convert event." }))
        }
    }
});

Here's my Event enum:

#[derive(Deserialize)]
#[serde(untagged)]
pub enum Event {
    CloudWatch(cloudwatch::Event),
    Auth(auth::Event),
    Http(HttpEvent),
    Records(Records),
    Unknown(Value),
}

The ergonomics of Rust are amazing for this purpose. Static routing like this gives me 500 microsecond response times, which is almost unthinkably fast, Amazon bills by 100ms intervals, I'm 200x faster than that 💯

I'd love to see a hierarchy expressed like this, perhaps using the underlying data objects from rusoto in enums for the Lambda entry point. It's really hard to get things working for Rust + Lambda (largely because the underlying Lambda system image is really old, no fault of crowbar), but once it's working, it's a joy to use!

I'd love to see more forward progress on this, but I'm currently pretty busy and haven't been able to devote time to my API as much as I'd like.

[softprops]

Rusoto is a boto client for rust. Boto only describes aws services. If it also describes lambda event data models 👍 if not I feel like a stand alone crate on its own release schedule would do. I'd go so far as to say especially for the release schedule. I think rusoto is a bit reserved in releasing often because it rereleases all of its crates in batches bumping versions when nothing's changed. For something like a crate of lambda event datastuctures I'd imagine the change cycle to release time to be super short.

[naftulikay]

Yup, we're on the same page! I thought that somebody had done it already, but I could be mistaken. If anyone wants to start that, please do and I'll contribute some work at it.

[LegNeato]

I did: https://github.com/srijs/rust-aws-lambda/tree/master/aws_lambda_events. It doesn't have any deps on a particular rust framework and is auto-generated from the official go typedefs. It even autogenerates tests and uses the example json from the official go lib.

The generator is https://github.com/srijs/rust-aws-lambda/tree/master/aws_lambda_events_codegen.

That being said, all those cloudwatch events are out of scope....I'm only focusing on Lambda events.

[softprops]

@naftulikay would you be open to publishing what you have now in a stand alone crate. I'm using crowbar at work in a growing number of places. I would use said crate as soon as you'd publish it :)

[softprops]

@LegNeato ha I started typing before you posted. Would you be able to put that in a repo and on its own release schedule? I fine is convenient for maintainers but less so for users to have projects that release multiple artifacts because of the release coordination involved ( publishing new versions of things when they haven't changed as an artifact :) ). It also makes it less daunting for outside contributors. If you published that today I'd use it.. tomorrow :)

[softprops]

Knowing this exists it'd be great to consolidate efforts. A separate create would keep crowbar super small for cases when you may not gain much from the structures, scheduled crons ECT.

[softprops]

@LegNeato this looks great! I could shed some fur I've grown in another crate with your data structs

https://github.com/softprops/lando/blob/master/src/request.rs

What dependencies does your data structures crate have on its parent project?

[LegNeato]

@softprops Zero deps on the parent project. The parent project consumes it and reexports. I was doing my own runtime with gob encoding and decided to join forces. It also helps to have a project that is testing it while developing, hence why it is part of the workspace. It was always envisioned as a crate that can be used with all rust lambda runtimes even though it lives in a specific one's repo...which is why I posted #32 (comment) 😄

I added a README and do intend to publish it soon. I wanted to get srijs/rust-aws-lambda#8 in first though.

[softprops]

@LegNeato sounds great! Having it embedded in another crate without actually depending on that the other crates makes it harder to discover and potentially more tricky with ci since itd be coupled to its parent projects ci. I'm not sure if how much value you're getting from the embedding now but I assume it would actually make it easier to manage independently from a contributor/ci/release cycle stand point in its own repo. Either way I look forward to its release!

[softprops]

srijs/rust-aws-lambda#8 is an great example. It seems this would be easier to manage changes specific to the data structures in its own crate which you can manage/release independently

[naftulikay]

I will shortly post my data types that I have internally defined. Unfortunately they're in a non-public repository that I don't plan on making public.

[naftulikay]

Here it is everybody, I apologize it's in a Gist, but it's what I can ship for now: https://gist.github.com/naftulikay/99f4ab201c7efddb2210930f411e60e4

I have a series of test cases which should make things clear. Basically mod.rs/Event is the root value that the Lambda function will receive. It disambiguates into HTTP events (from API Gateway), CloudWatch events directly, API Gateway Authorization Events, or to a series of mod.rs/Records which are mod.rs/Record objects.

Records can be S3, SES, SNS, and other types that I haven't implemented yet.

Some of the more complicated flows are found in SES, where it's necessary to define types for full messages (basically SES -> SNS -> Lambda is required to get a full message) versus a "filter" action which allows bouncing emails, but only gives you access to basic metadata about the message.

I have unit tests stashed away and I have the Terraform required to create all of this, but it is likewise not open source.

[LegNeato]

@softprops FWIW I published an initial version of the events package: https://crates.io/crates/aws_lambda_events

[softprops]

@LegNeato that's great news!