Skip to content

Commit

Permalink
Merge branch 'main' into add-description-input
Browse files Browse the repository at this point in the history
  • Loading branch information
sunkickr authored Dec 14, 2023
2 parents 544eb04 + 4e8464c commit 76b3a2e
Show file tree
Hide file tree
Showing 6 changed files with 77 additions and 54 deletions.
Binary file removed .DS_Store
Binary file not shown.
13 changes: 13 additions & 0 deletions .github/workflows/lint_yaml.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
name: Lint YAML
on:
pull_request:
branches: ['main']
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: yaml-lint
uses: ibiqlik/action-yamllint@v3
with:
file_or_dir: action.yaml
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
.DS_Store
.idea/
11 changes: 11 additions & 0 deletions .yamllint
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
extends: default

rules:
line-length: disable
document-start:
present: false
indentation:
spaces: 2
indent-sequences: true
check-multi-line-strings: false # disabled because Bash scripts contain indented code
38 changes: 19 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,12 @@ You can use and configure this GitHub action to easily deploy Apache Airflow DAG
- Avoid manually running `astro deploy` with the Astro CLI every time you make a change to your Astro project.
- Automate deploying code to Astro when you merge changes to a certain branch in your repository.
- Incorporate unit tests for your DAGs as part of the deploy process.
- Create/delete a Deployment Preview. A Deployment Preveiw is an Astro Deployment that mirrors the configuration of your orginal Deployment.
- Create/delete a Deployment Preview. A Deployment Preview is an Astro Deployment that mirrors the configuration of your original Deployment.

This GitHub action runs as a step within a GitHub workflow file. When your CI/CD pipeline is triggered, this action:

- Checks out your GitHub repository.
- Optionaly create or delete a Deployment Preview to test your code changes on before deploying to production.
- Optionally create or delete a Deployment Preview to test your code changes on before deploying to production.
- Checks whether your commit only changed DAG code.
- Optional. Tests DAG code with `pytest`. See [Run tests with pytest](https://docs.astronomer.io/astro/test-and-troubleshoot-locally#run-tests-with-pytest).
- Runs `astro deploy --dags` if the commit only includes DAG code changes.
Expand All @@ -23,8 +23,8 @@ To use this GitHub action, you need:

- An Astro project. See [Create a project](https://docs.astronomer.io/astro/create-project).
- A Deployment on Astro. See [Create a Deployment](https://docs.astronomer.io/astro/create-deployment).
- A Workspace or Organization API Token. See [API Tokens](https://docs.astronomer.io/astro/workspace-api-tokens)
- Or a Deployment API key ID and secret. See [Deployment API keys](https://docs.astronomer.io/astro/api-keys).
- An Organization, Workspace, or Deployment API Token. See [API Tokens](https://docs.astronomer.io/astro/workspace-api-tokens)
- Or (deprecated) a Deployment API key ID and secret. See [Deployment API keys](https://docs.astronomer.io/astro/api-keys).

Astronomer recommends using [GitHub Actions secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets) to store `ASTRO_API_TOKEN` or Deployment API Keys. See the example in [Workflow file examples](https://github.com/astronomer/deploy-action#workflow-file-examples).

Expand Down Expand Up @@ -55,7 +55,7 @@ The following table lists the configuration options for the Deploy to Astro acti
| `pytest-file` | (all tests run) | Specifies a custom pytest file to run with the pytest command. For example, you could specify `/tests/test-tags.py`|
| `force` | `false` | When set to `true`, your code is deployed and skips any pytest or parsing errors |
| `image-name` | | Specifies a custom, locally built image to deploy |
| `workspace` | `false` | If you are using an organization token you will need to provide a workspace-name or id |
| `workspace` | | Workspace id to select. Only required when `ASTRO_API_TOKEN` is given an organization token. |
| `preview-name` | `false` | Specifies custom preview name. By default this is branch name “_” deployment name |
| `checkout` | `true` | Whether to checkout the repo as the first step. Set this to false if you want to modify repo contents before invoking the action |

Expand All @@ -65,7 +65,7 @@ The following table lists the configuration options for the Deploy to Astro acti

In the following example, the GitHub action deploys code to Astro. This example assumes that you have one Astro Deployment and one branch. When a change is merged to the `main` branch, your Astro project is deployed to Astro. DAG files are parsed on every deploy and no pytests are ran.

```
```yaml
name: Astronomer CI - Deploy code

on:
Expand Down Expand Up @@ -94,7 +94,7 @@ Use the following topics to further configure the action based on your needs.
In the following example, the folder `/example-dags/dags` is specified as the DAG folder.

```
```yaml
steps:
- name: Deploy to Astro
uses: astronomer/[email protected]
Expand All @@ -107,7 +107,7 @@ steps:

In the following example, the pytest located at `/tests/test-tags.py` runs before deploying to Astro.

```
```yaml
steps:
- name: Deploy to Astro
uses: astronomer/[email protected]
Expand All @@ -121,7 +121,7 @@ steps:

In the following example, `force` is enabled and both the DAG parse and pytest processes are skipped.

```
```yaml
steps:
- name: Deploy to Astro
uses: astronomer/[email protected]
Expand All @@ -134,7 +134,7 @@ steps:

In the following example, a custom Docker image is built and deployed to an Astro Deployment.

```
```yaml
name: Astronomer CI - Additional build-time args
on:
Expand Down Expand Up @@ -176,16 +176,16 @@ jobs:

## Deployment Preview Templates

This section contains four workflow files that you will need in your repository to have a full Deployment Preview Cycle running for your Deployment. A Deployment Preview is an Astro Deployment that mirrors the configuration of your original Deployment. This Deployment Preview can be used to test your new pipelines changes before pushing them to your orginal Deployment. The scripts below will take your pipeline changes through the following flow:
This section contains four workflow files that you will need in your repository to have a full Deployment Preview Cycle running for your Deployment. A Deployment Preview is an Astro Deployment that mirrors the configuration of your original Deployment. This Deployment Preview can be used to test your new pipelines changes before pushing them to your original Deployment. The scripts below will take your pipeline changes through the following flow:

1. When a new branch is created a Deployment Preview will be created based off your orginal Deployment
1. When a new branch is created a Deployment Preview will be created based off your original Deployment
2. When a PR is created from a branch code changes will be deployed to the Deployment Preview
3. When a PR is merged into your "main" branch code changes will be deployed to the orginal Deployment
4. When a branch is deleted the the corresponding Deployment Preview will also be deleted
3. When a PR is merged into your "main" branch code changes will be deployed to the original Deployment
4. When a branch is deleted the corresponding Deployment Preview will also be deleted

## Create Deployment Preview

```
```yaml
name: Astronomer CI - Create deployment preview

on:
Expand All @@ -210,7 +210,7 @@ jobs:
## Deploy to Deployment Preview
```
```yaml
name: Astronomer CI - Deploy code to Preview

on:
Expand All @@ -235,7 +235,7 @@ jobs:
## Delete Deployment Preview
```
```yaml
name: Astronomer CI - Delete Deployment Preview

on:
Expand All @@ -258,9 +258,9 @@ jobs:
deployment-id: <orginal deployment id>
```
## Deploy to Orginal Deployment
## Deploy to Original Deployment
```
```yaml
name: Astronomer CI - Deploy code to Astro

on:
Expand Down
67 changes: 32 additions & 35 deletions action.yaml
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
name: "Deploy Apache Airflow DAGs to Astro"
description: "Test your DAGs and deploy your Astro project to a Deployment on Astro, Astronomer's managed Airflow service."
author: "Astronomer"
branding:
icon: 'upload-cloud'
color: 'purple'
icon: 'upload-cloud'
color: 'purple'
inputs:
root-folder:
required: false
default: ./
description: "Path to the Astro project folder that contains the 'dags' folder"
description: "Path to the Astro project folder that contains the 'dags' folder."
parse:
required: false
default: false
Expand All @@ -31,22 +32,18 @@ inputs:
action:
required: false
default: deploy
description: "Specify what action you would like to take. Use this option to create or delete deployment previews. Specify either 'create-deployment-preview', 'delete-deployment-preview' or 'deploy-deployment-preview'. Don't sepcify anything if you are deploying to a regular Deployment."
description: "Specify what action you would like to take. Use this option to create or delete deployment previews. Specify either 'create-deployment-preview', 'delete-deployment-preview' or 'deploy-deployment-preview'. Don't specify anything if you are deploying to a regular Deployment."
deployment-name:
required: false
default: false
description: "The name of the Deployment you want to make preview from or are deploying to."
deployment-id:
required: false
default: false
description: "The id of the Deployment you to make a preview from or are deploying to."
workspace:
required: false
default: false
description: "If you are using an organization token you will need to provide a workspace name or id."
description: "Workspace id to select. Only required when `ASTRO_API_TOKEN` is given an organization token."
preview-name:
required: false
default: false
description: "Custom preview name. By default this is <branch-name>_<deployment-name>."
copy-connections:
required: false
Expand All @@ -60,6 +57,10 @@ inputs:
required: false
default: true
description: "Copy pools from the original Deployment to the new deployment preview."
cli-version:
required: false
default: ""
description: "The desired Astro CLI version to use"
checkout:
required: false
default: true
Expand Down Expand Up @@ -91,45 +92,42 @@ runs:
# Determine if only DAGs have changes
- name: Install Astro
run: |
curl -sSL https://install.astronomer.io | sudo bash -s
curl -sSL https://install.astronomer.io | sudo bash -s ${{ inputs.cli-version }}
shell: bash
- name: Deployment Preview action
run: |
# set action
ACTION=nothing
# switch workspace if specified
if [[ ${{ inputs.workspace }} != false ]]; then
# Select workspace if specified
if [[ "${{ inputs.workspace }}" != "" ]]; then
astro workspace switch ${{ inputs.workspace }}
fi
# error if both deployment name and id are used
if [[ "${{ inputs.deployment-name }}" != false && ${{ inputs.deployment-id }} != false ]]; then
if [[ "${{ inputs.deployment-name }}" != "" && "${{ inputs.deployment-id }}" != "" ]]; then
echo ERROR: cannot specify both a Deployment ID and Name
exit 1 # terminate and indicate error
fi
# figure out deployment id
if [[ "${{ inputs.deployment-name }}" != false ]]; then
if [[ "${{ inputs.deployment-name }}" != "" ]]; then
# get deployment-id
DEPLOYMENT_ID="$(astro deployment inspect --clean-output -n "${{ inputs.deployment-name }}" --key metadata.deployment_id)"
fi
if [[ ${{ inputs.deployment-id }} != false ]]; then
DEPLOYMENT_ID=${{ inputs.deployment-id }}
if [[ "${{ inputs.deployment-id }}" != "" ]]; then
DEPLOYMENT_ID="${{ inputs.deployment-id }}"
fi
# create deployment preview if action is create-deployment-preview
if [[ ${{ inputs.action }} == create-deployment-preview ]]; then
if [[ "${{ inputs.deployment-name }}" == false && ${{ inputs.deployment-id }} == false ]]; then
if [[ "${{ inputs.deployment-name }}" == "" && "${{ inputs.deployment-id }}" == "" ]]; then
echo ERROR: cannot create a deployment preview without specifying a deployment name or id
exit 1 # terminate and indicate error
fi
if [[ ${{ inputs.preview-name }} != false ]]; then
BRANCH_DEPLOYMENT_NAME=${{ inputs.preview-name }}
if [[ "${{ inputs.preview-name }}" != "" ]]; then
BRANCH_DEPLOYMENT_NAME="${{ inputs.preview-name }}"
else
# get branch name
DEPLOYMENT_NAME="$(astro deployment inspect $DEPLOYMENT_ID --clean-output --key configuration.name)"
Expand All @@ -149,7 +147,7 @@ runs:
# Add name to deployment template file
sed -i "s| name:.*| name: ${BRANCH_DEPLOYMENT_NAME}|g" deployment-preview-template.yaml
# Create new deployment preview based on the deployment template file
astro deployment create --deployment-file deployment-preview-template.yaml
Expand All @@ -172,15 +170,15 @@ runs:
# delete deployment preview and skip deploy if action is delete-deployment-preview
if [[ ${{ inputs.action }} == delete-deployment-preview ]]; then
if [[ "${{ inputs.deployment-name }}" == false && ${{ inputs.deployment-id }} == false ]]; then
if [[ "${{ inputs.deployment-name }}" == "" && "${{ inputs.deployment-id }}" == "" ]]; then
echo ERROR: cannot delete a deployment preview without specifying a deployment name or id
exit 1 # terminate and indicate error
fi
if [[ ${{ inputs.preview-name }} != false ]]; then
BRANCH_DEPLOYMENT_NAME=${{ inputs.preview-name }}
if [[ "${{ inputs.preview-name }}" != "" ]]; then
BRANCH_DEPLOYMENT_NAME="${{ inputs.preview-name }}"
else
else
DEPLOYMENT_NAME="$(astro deployment inspect $DEPLOYMENT_ID --clean-output --key configuration.name)"
BRANCH_DEPLOYMENT_NAME=${{ github.event.ref }}_$DEPLOYMENT_NAME
BRANCH_DEPLOYMENT_NAME="${BRANCH_DEPLOYMENT_NAME// /_}"
Expand All @@ -197,17 +195,17 @@ runs:
# set action
ACTION=delete
fi
# # deploy to deployment preview if action is deploy-deployment-preview
if [[ ${{ inputs.action }} == deploy-deployment-preview ]]; then
if [[ "${{ inputs.deployment-name }}" == false && ${{ inputs.deployment-id }} == false ]]; then
if [[ "${{ inputs.deployment-name }}" == "" && "${{ inputs.deployment-id }}" == "" ]]; then
echo ERROR: cannot deploy to a deployment preview without specifying a deployment name or id
exit 1 # terminate and indicate error
fi
if [[ ${{ inputs.preview-name }} != false ]]; then
BRANCH_DEPLOYMENT_NAME=${{ inputs.preview-name }}
if [[ "${{ inputs.preview-name }}" != "" ]]; then
BRANCH_DEPLOYMENT_NAME="${{ inputs.preview-name }}"
else
DEPLOYMENT_NAME="$(astro deployment inspect $DEPLOYMENT_ID --clean-output --key configuration.name)"
if [[ ${GITHUB_HEAD_REF##*/} != "" ]]; then
Expand All @@ -226,7 +224,7 @@ runs:
# don't skip deploy
echo "SKIP_DEPLOY=false" >> $GITHUB_OUTPUT
# not image deploy only
echo "IMAGE_DEPLOY_ONLY=false" >> $GITHUB_OUTPUT
Expand Down Expand Up @@ -282,7 +280,7 @@ runs:
fi
if [[ ${{steps.deployment-preview.outputs.SKIP_DEPLOY}} == true ]]; then
# skip all deploy steps
# skip all deploy steps
dags_only=2
fi
Expand All @@ -308,7 +306,7 @@ runs:
if [[ ${{ inputs.image-name }} != no-custom-image ]]; then
options="$options --image-name ${{ inputs.image-name }}"
fi
# add force option
if [[ ${{ inputs.force }} == true ]]; then
options="$options --force"
Expand Down Expand Up @@ -349,4 +347,3 @@ runs:
fi
fi
shell: bash

0 comments on commit 76b3a2e

Please sign in to comment.