-
Notifications
You must be signed in to change notification settings - Fork 4k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Skeleton code structure for Glue L2 construct
- Loading branch information
Janardhan Molumuri
committed
Aug 9, 2023
1 parent
0e2c169
commit ead3104
Showing
11 changed files
with
99 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
/** | ||
* Conditional Trigger Class | ||
* | ||
* Conditional triggers have a predicate and actions associated with them. | ||
* When the predicateCondition is true, the trigger actions will be executed. | ||
* | ||
*/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
/** | ||
* Enums used by Job Types/Triggers | ||
* | ||
*/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
/** | ||
* Flex Jobs class | ||
* | ||
* Flex jobs supports Python and Scala language. | ||
* The flexible execution class is appropriate for non-urgent jobs such as | ||
* pre-production jobs, testing, and one-time data loads. | ||
* Flexible job runs are supported for jobs using AWS Glue version 3.0 or later and G.1X or | ||
* G.2X worker types but will default to the latest version of Glue (currently Glue 3.0.) | ||
* | ||
* Similar to ETL, we’ll enable these features: —enable-metrics, —enable-spark-ui, | ||
* —enable-continuous-cloudwatch-log | ||
* | ||
*/ |
10 changes: 10 additions & 0 deletions
10
packages/@aws-cdk/aws-glue-alpha/lib/notifyeventtrigger.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
/** | ||
* Notify Event Trigger Class | ||
* | ||
* Workflows are mandatory for this trigger type. There are two types of notify event triggers, | ||
* batching and non-batching trigger. | ||
* For batching triggers, developers must specify BatchSize but for non-batching BatchSize will | ||
* be set to 1. | ||
* For both triggers, BatchWindow will be default to 900 seconds. | ||
* | ||
*/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
/** | ||
* On Demand Trigger Class | ||
* | ||
* On demand triggers can start glue jobs or crawlers. | ||
* The trigger method will take an optional description but abstract the requirement of an | ||
* actions list using the job or crawler objects using conditional types. | ||
* | ||
*/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
/** | ||
* Python Shell Jobs class | ||
* | ||
* A Python shell job runs Python scripts as a shell and supports a Python version that | ||
* depends on the AWS Glue version you are using. | ||
* This can be used to schedule and run tasks that don't require an Apache Spark environment. | ||
* | ||
*/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
/** | ||
* Ray Jobs class | ||
* | ||
* Glue ray only supports worker type Z.2X and Glue version 4.0. | ||
* Runtime will default to Ray2.3 and min workers will default to 3. | ||
* | ||
*/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
/** | ||
* Scheduled Trigger Base Class | ||
* | ||
* Schedule triggers are a way for developers to create jobs using cron expressions. | ||
* We’ll provide daily, weekly, and monthly convenience functions, as well as a custom function | ||
* that will allow developers to create their own custom timing using the existing | ||
* event Schedule object without having to build their own cron expressions. | ||
* | ||
* The trigger method will take an optional description and list of Actions | ||
* which can refer to Jobs or crawlers via conditional types. | ||
* | ||
*/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
/** | ||
* Spark ETL Jobs class | ||
* | ||
* ETL jobs supports Python and Scala language. | ||
* ETL job type supports G1, G2, G4 and G8 worker type default as G2, which customer can override. | ||
* It wil default to the best practice version of ETL 4.0, but allow developers to override to 3.0. | ||
* We will also default to best practice enablement the following ETL features: | ||
* —enable-metrics, —enable-spark-ui, —enable-continuous-cloudwatch-log. | ||
* | ||
*/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
/** | ||
* Streaming Jobs class | ||
* | ||
* A Streaming job is similar to an ETL job, except that it performs ETL on data streams | ||
* using the Apache Spark Structured Streaming framework. | ||
* These jobs will default to use Python 3.9. | ||
* | ||
* Similar to ETL jobs, streaming job supports Scala and Python languages. Similar to ETL, | ||
* it supports G1 and G2 worker type and 2.0, 3.0 and 4.0 version. We’ll default to G2 worker | ||
* and 4.0 version for streaming jobs which developers can override. | ||
* We will enable —enable-metrics, —enable-spark-ui, —enable-continuous-cloudwatch-log. | ||
* | ||
*/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
/** | ||
* Workflow Trigger Base Class | ||
* | ||
* In AWS Glue, developers can use workflows to create and visualize complex extract, | ||
* transform, and load (ETL) activities involving multiple crawlers, jobs, and triggers. | ||
* | ||
*/ |