Serverless Data Lake Framework 2.0.0-beta.0

[cnfait]

Work is ongoing on a new major version of the Serverless Data Lake Framework. This is a pre-release, not ready for production workloads.

What’s New

• SDLF components are now CloudFormation modules
  • there is one module per component: foundations, team, pipeline, stageA, stageB, dataset.
  • datalakeLibrary and pipLibrary are used to build Lambda layers, they’re not CloudFormation modules.
  • deploy.sh takes care of deploying the CICD infrastructure used to build these modules, and register them in the private CloudFormation registry of each account. Modules are updated whenever there is a change to their source repository.
• SDLF CICD pipelines now live in the Shared DevOps account
  • CloudFormation stacks are created in child accounts through crossaccount IAM roles.
• SDLF can deploy an arbitrary number of child accounts driven from a single devops account.
  • pDomain (which defaults to datalake) can be provided when deploying foundations.
  • each domain can have the usual three environments (dev, test, prod).
• Deploying foundations and teams is now done from a new repository called sdlf-main.
  • this repository is created during the initial setup with deploy.sh.
  • foundations deployment happens in foundations-{domain}-{env}.yaml and teams in teams-{domain}-{env}.yaml.
  • sdlf-main works the same way everything works in SDLF - master, test and dev branches are expected.
  • it is easier to know which teams have been created, and to remove them as they don’t share the same set of parameters in parameters-{env}.json.
• Deploying pipelines and datasets is now done from a new repository called sdlf-{domain}-{team name}-main.
  • this repository is created when a new team is created.
  • pipelines deployment happens in pipelines-{env}.yaml and datasets in datasets-{env}.yaml.
  • sdlf-{team name}-main works the same way everything works in SDLF - master, test and dev branches are expected.
  • it is easier to know which pipelines and teams have been created, and to remove them as they don’t share the same set of parameters in parameters-{env}.json.
• Mappings between datasets and transforms in stageB is done directly when defining a dataset.
  • this mapping used to be done by a CodeBuild project and a script in sdlf-datalakeLibrary. They are no longer needed and have been removed.
  • it is now defined through the pPipelineDetails parameter when defining a dataset in sdlf-dataset. This parameter goes even further and can be used to store more information that stages can use. These details are stored in the Datasets DynamoDB table (as was already the case in SDLFv1).
• Stages in a pipeline are now driven by EventBridge rules exclusively.
  • the rule can be an event pattern or a schedule (cron expression).
  • stageA is no longer sending messages to a queue for stageB to process. StageB is configured with an event pattern to listen for stageA runs (pEventPattern in the example), and then process these events on a schedule (pSchedule)
  • it is easier now to have pipelines with a single stage, pipelines with dependent stages and overall more complex pipelines than in SDLFv1, as long as there is an event pattern to listen for.
• New optional component: sdlf-monitoring, with CloudTrail, ELK and SNS.
  • in SDLFv1 Cloudtrail is optional but enabled by default. Here it is optional and not enabled as long as sdlf-monitoring is not deployed.
• New optional stage: sdlf-stage-dataquality
  • deequ is now entirely optional. While it wasn’t enabled by default in SDLFv1, dedicated infrastructure was still created while deploying sdlf-foundations. This is no longer the case.
  • sdlf-stage-dataquality can now be used as an example on how to add a third stage to the default stageA and stageB pipeline.
• Outside the initial deploy.sh, there is no more shell scripts.

Full Changelog: 1.5.2...2.0.0-beta.0

---

This discussion was created from the release Serverless Data Lake Framework 2.0.0-beta.0.

[cnfait]

To test this version, here is a quick how-to:

Initial setup

Clone the branch where the code for SDLFv2 currently sits:

git clone --single-branch --branch 2.0.0-beta.0 https://github.com/awslabs/aws-serverless-data-lake-framework.git
cd ./aws-serverless-data-lake-framework/

Print the deployment script’s help as it may be useful to understand the commands from the rest of this post:

./deploy.sh -h

Deploy crossaccount IAM roles in child accounts, necessary for DevOps CICD pipelines:

./deploy.sh crossaccount-cicd-roles -r <aws-region> -d <devops-aws-account-id> -p <child-aws-profile>

Seven roles starting with sdlf- should have been created. They can be assumed by pipelines from <devops-aws-account-id>.

Continue deploying SDLF - this time, deploy the DevOps account resources:

./deploy.sh devops-account -r <aws-region> -d <child-account-id1,child-account-id2,...> -p <devops-aws-profile>

The following CodeCommit repositories should have been created:

• sdlf-cicd
  • this repository wasn’t created in SDLFv1
• sdlf-foundations
• sdlf-team
• sdlf-pipeline
• sdlf-dataset
• sdlf-datalakeLibrary
• sdlf-pipLibrary
• sdlf-stageA
• sdlf-stageB
• sdlf-utils

Two CloudFormation stacks should have been created: sdlf-cicd-prerequisites and sdlf-cicd-repositories with one nested stack.

Child Accounts and Teams

In CodeCommit, an empty repository called sdlf-main should have been created. Clone it locally (or edit directly on the AWS web console), move to the dev branch:

git checkout -b dev

and create a file named foundations-datalake-dev.yaml with the following content (replacing the value for pChildAccountId with the account where you created the crossaccount roles in the previous section):

AWSTemplateFormatVersion: 2010-09-09
Description: SDLF Foundations in Dev

Parameters:
  pPipelineReference:
    Type: String
    Default: none

Resources:
  rForecourt:
    Type: awslabs::sdlf::foundations::MODULE
    Properties:
      pPipelineReference: !Ref pPipelineReference
      pChildAccountId: <child-account-id>
      pOrganizationName: forecourt
      pDomain: datalake
      pEnvironment: dev
      pNumBuckets: 3

Push the file to the repository dev branch. 4 CodePipeline pipelines named sdlf-cicd-domain-datalake-dev-* will be created. Make sure that they have all been triggered and executed successfully (rDomainCodePipeline last step will fail - that’s expected for now). rCommonPipLibs and rCommonDatalakeLibs create Lambda layers used in SDLF pipelines. rModulesPipeline builds CloudFormation modules for each SDLF components (foundations, team, pipeline and stages, dataset). Finally, rDomainCodePipeline takes care of deploying the templates created in sdlf-main to the relevant child account.

In SDLFv1, this was done editing parameters-dev.json in the sdlf-foundations repository. In SDLFv2, components such as sdlf-foundations, sdlf-team and sdlf-pipeline are CloudFormation modules, built from their repository with the sdlf-cicd-build-cfn-module-codebuild job. Then these modules can be used to deploy the components needed for the datalake.

In the same repository, deploy a new team. Create a file name teams-datalake-dev.yaml with the following content:

AWSTemplateFormatVersion: 2010-09-09
Description: SDLF Teams in Dev

Parameters:
  pPipelineReference:
    Type: String
    Default: none

Resources:
  rEngineering:
    Type: awslabs::sdlf::team::MODULE
    Properties:
      pPipelineReference: !Ref pPipelineReference
      pTeamName: engineering
      pEnvironment: dev
      pSNSNotificationsEmail: [email protected]

Push to the repository dev branch, and watch rDomainCodePipeline in CodePipeline. In the devops account, a new CodePipeline can be seen: sdlf-cicd-teams-datalake-dev-engineering-rTeamCodePipeline.

Pipelines and Datasets

A new CodeCommit repository called sdlf-datalake-engineering-main should have been created in the devops account. That’s where the team deploys new pipelines and datasets. Clone it locally (or edit directly on the AWS web console), move to the dev branch:

git checkout -b dev

Create a file named pipeline-main-dev.yaml with the following content:

AWSTemplateFormatVersion: "2010-09-09"
Description: Engineering team Main pipeline

Parameters:
  pPipelineReference:
    Type: String
    Default: none

Resources:
  rMainA:
    Type: awslabs::sdlf::stageA::MODULE
    Properties:
      pStageName: A
      pPipeline: main
      pTeamName: engineering
      pEntryPoint: "true"
      pTriggerType: event
      pEventPattern: >-
        {
          "source": ["sdlf.s3"],
          "detail-type": ["SdlfObjectInS3"],
          "detail": {
            "team": ["engineering"],
            "pipeline": ["main"],
            "pipeline_stage": ["A"]
          }
        }
      pEnableTracing: false

  rMainB:
    Type: awslabs::sdlf::stageB::MODULE
    Properties:
      pDatasetBucket: "{{resolve:ssm:/SDLF/S3/StageBucket}}"
      pStageName: B
      pPipeline: main
      pTeamName: engineering
      pTriggerType: schedule
      pEventPattern: !Sub >-
        {
          "source": ["aws.states"],
          "detail-type": ["Step Functions Execution Status Change"],
          "detail": {
            "status": ["SUCCEEDED"],
            "stateMachineArn": ["arn:aws:states:${AWS::Region}:${AWS::AccountId}:stateMachine:sdlf-engineering-main-sm-a"]
          }
        }
      pSchedule: "cron(*/5 * * * ? *)"
      pEnableTracing: false

Then another file named pipelines-dev.yaml with the following content:

AWSTemplateFormatVersion: "2010-09-09"
Description: Engineering team pipelines

Parameters:
  pPipelineReference:
    Type: String
    Default: none

Resources:
  rMain:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: ./pipeline-main-dev.yaml
      Parameters:
        pPipelineReference: !Ref pPipelineReference

Push both files to the repository dev branch, and watch sdlf-cicd-teams-datalake-dev-engineering-rTeamCodePipeline in CodePipeline which should have been triggered automatically. It will take care of deploying all sdlf-stageA and sdlf-stageB resources to the dev account.

There are several changes here compared to SDLFv1. Each stage are declared in a block and attached to a pipeline using pPipeline. pTriggerType, pEventPattern and pSchedule are the main parameters to look at - this is how going from one stage to the next happens. In this example the first stage is event-based, the second schedule-based. pEventPattern is still defined in the second stage as it needs to know what events to listen for before processing them every 5 minutes.

In the same repository, deploy a new dataset. Create a file name datasets-dev.yaml with the following content:

AWSTemplateFormatVersion: 2010-09-09
Description: Engineering team datasets

Parameters:
  pPipelineReference:
    Type: String
    Default: none

Resources:
  rLegislators:
    Type: awslabs::sdlf::dataset::MODULE
    Properties:
      pPipelineReference: !Ref pPipelineReference
      pTeamName: engineering
      pDatasetName: legislators
      pPipelineDetails: >-
        {
          "main": {
            "transforms": {
              "stage_A_transform": "light_transform_blueprint",
              "stage_B_transform": "heavy_transform_blueprint"
            }
          }
        }

Push the file to the repository dev branch, and watch sdlf-cicd-teams-datalake-dev-engineering-rTeamCodePipeline again in CodePipeline which should have been triggered automatically.

The main different here compared to SDLFv1 is that all details a pipeline may require are passed as a JSON object to the pPipelineDetails parameter. main is the pipeline name the details will be available to. These details are added to the dataset record in DynamoDB (octagon-Datasets table) - same as in SDLFv1. There is no longer a need to edit dataset_mappings.json in sdlf-datalakeLibrary.

Feel free to deploy https://github.com/awslabs/aws-serverless-data-lake-framework/tree/main/sdlf-utils/pipeline-examples/legislators and confirm everything works as intended. Thanks for testing!