diff --git a/README.md b/README.md index 812568d..25d893b 100644 --- a/README.md +++ b/README.md @@ -6,10 +6,12 @@ You can use and configure this GitHub action to easily deploy Apache Airflow DAG - Avoid manually running `astro deploy` with the Astro CLI every time you make a change to your Astro project. - Automate deploying code to Astro when you merge changes to a certain branch in your repository. - Incorporate unit tests for your DAGs as part of the deploy process. +- Create/delete a Deployment Preview. A Deployment Preveiw is an Astro Deployment that mirrors the configuration of your orginal Deployment. This GitHub action runs as a step within a GitHub workflow file. When your CI/CD pipeline is triggered, this action: - Checks out your GitHub repository. +- Optionaly create or delete a Deployment Preview to test your code changes on before deploying to production. - Checks whether your commit only changed DAG code. - Optional. Tests DAG code with `pytest`. See [Run tests with pytest](https://docs.astronomer.io/astro/test-and-troubleshoot-locally#run-tests-with-pytest). - Runs `astro deploy --dags` if the commit only includes DAG code changes. @@ -21,16 +23,17 @@ To use this GitHub action, you need: - An Astro project. See [Create a project](https://docs.astronomer.io/astro/create-project). - A Deployment on Astro. See [Create a Deployment](https://docs.astronomer.io/astro/create-deployment). -- A Deployment API key ID and secret. See [Deployment API keys](https://docs.astronomer.io/astro/api-keys). +- A Deployment API Token. See [API Tokens]() +- Or a Deployment API key ID and secret. See [Deployment API keys](https://docs.astronomer.io/astro/api-keys). -Astronomer recommends using [GitHub Actions secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets) to store `ASTRONOMER_KEY_ID` and `ASTRONOMER_KEY_SECRET`. See the example in [Workflow file examples](https://github.com/astronomer/deploy-action#workflow-file-examples). +Astronomer recommends using [GitHub Actions secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets) to store `ASTRO_API_TOKEN` or Deployment API Keys. See the example in [Workflow file examples](https://github.com/astronomer/deploy-action#workflow-file-examples). ## Use this action To use this action, read [Automate code deploys with CI/CD](https://docs.astronomer.io/astro/ci-cd?tab=multiple%20branch#github-actions-dag-based-deploy). You will: 1. Create a GitHub Actions workflow in your repository that uses the latest version of this action. For example, `astronomer/deploy-action@v0.1`. -2. Configure the workflow to fit your team's use case. This could include disabling DAG-only deploys or adding tests. See [Configuration options](https://github.com/astronomer/deploy-action#configuration-options). +2. Configure the workflow to fit your team's use case. This could include creating a deployment preview or adding tests. See [Configuration options](https://github.com/astronomer/deploy-action#configuration-options). 3. Make changes to your Astro project files in GitHub and let this GitHub Actions workflow take care of deploying your code to Astro. Astronomer recommends setting up multiple environments on Astro. See the [Multiple branch GitHub Actions workflow](https://docs.astronomer.io/astro/ci-cd?tab=multibranch#github-actions-image-only-deploys) in Astronomer documentation. @@ -42,19 +45,23 @@ The following table lists the configuration options for the Deploy to Astro acti | Name | Default | Description | | ---|---|--- | -| `dag-deploy-enabled` | `false` | When set to `true`, this action includes conditional logic that deploys only DAG files to Astro when only the DAGs directory changes. Only set this to `true` when the DAG-only deploy feature is enabled on your Astro Deployment. See [Deploy DAGs only](https://docs.astronomer.io/astro/deploy-code#deploy-dags-only) | -| `root-folder` | `.` | Specifies the path to the Astro project directory that contains the `dags` folder | +| `action` | `deploy` | Specify what action you would like to take. Use this option to create or delete deployment previews. Specify either `create-deployment-preview`, `delete-deployment-preview` or `deploy-deployment-preview`. Don't sepcify anything if you are deploying to a regular deployment | +| `deployment-id` | `false` | Specifies the id of the deployment you to make a preview from or are deploying too | +| `deployment-name` | `false` | Specifies The name of the deployment you want to make preview from or are deploying too. Cannot be used with `deployment-id` | +| `root-folder` | `.` | Specifies the path to the Astro project directory that contains the `dags` folder | | `parse` | `false` | When set to `true`, DAGs are parsed for errors before deploying to Astro | | `pytest` | `false` | When set to `true`, all pytests in the `tests` directory of your Astro project are run before deploying to Astro. See [Run tests with pytest](https://docs.astronomer.io/astro/test-and-troubleshoot-locally#run-tests-with-pytest) | | `pytest-file` | (all tests run) | Specifies a custom pytest file to run with the pytest command. For example, you could specify `/tests/test-tags.py`| | `force` | `false` | When set to `true`, your code is deployed and skips any pytest or parsing errors | -| `image-name` | | Specifies a custom, locally built image to deploy | +| `image-name` | | Specifies a custom, locally built image to deploy | +| `workspace` | `false` | If you are using an organization token you will need to provide a workspace-name or id | +| `preview-name` | `false` | Specifies custom preview name. By default this is branch name “_” deployment name | ## Workflow file examples -In the following example, the GitHub action deploys code to Astro with DAG-only deploys enabled. This example assumes that you have one Astro Deployment and one branch. When a change is merged to the `main` branch, your Astro project is deployed to Astro. DAG files are parsed on every deploy and no pytests are ran. +In the following example, the GitHub action deploys code to Astro. This example assumes that you have one Astro Deployment and one branch. When a change is merged to the `main` branch, your Astro project is deployed to Astro. DAG files are parsed on every deploy and no pytests are ran. ``` name: Astronomer CI - Deploy code @@ -65,9 +72,8 @@ on: - main env: - ## Sets Deployment API key credentials as environment variables - ASTRONOMER_KEY_ID: ${{ secrets.ASTRONOMER_KEY_ID }} - ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRONOMER_KEY_SECRET }} + ## Set API Token as an environment variable + ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }} jobs: deploy: @@ -76,7 +82,7 @@ jobs: - name: Deploy to Astro uses: astronomer/deploy-action@v0.1 with: - dag-deploy-enabled: true + deployment-id: parse: true ``` @@ -91,6 +97,7 @@ steps: - name: Deploy to Astro uses: astronomer/deploy-action@v0.1 with: + deployment-id: root-folder: /example-dags/dags/ ``` @@ -103,6 +110,7 @@ steps: - name: Deploy to Astro uses: astronomer/deploy-action@v0.1 with: + deployment-id: pytest: true pytest-file: /tests/test-tags.py ``` @@ -116,6 +124,7 @@ steps: - name: Deploy to Astro uses: astronomer/deploy-action@v0.1 with: + deployment-id: force: true ``` @@ -135,8 +144,8 @@ jobs: build: runs-on: ubuntu-latest env: - ASTRONOMER_KEY_ID: ${{ secrets.ASTRO_ACCESS_KEY_ID_DEV }} - ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRO_SECRET_ACCESS_KEY_DEV }} + ## Set API Token as an environment variable + ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN steps: - name: Check out the repo uses: actions/checkout@v3 @@ -158,6 +167,139 @@ jobs: - name: Deploy to Astro uses: astronomer/deploy-action@v0.1 with: + deployment-id: image-name: ${{ steps.image_tag.outputs.image_tag }} ``` + +## Deployment Preview Templates + +This section contains four workflow files that you will need in your repository to have a full Deployment Preview Cycle running for your Deployment. A Deployment Preview is an Astro Deployment that mirrors the configuration of your original Deployment. This Deployment Preview can be used to test your new pipelines changes before pushing them to your orginal Deployment. The scripts below will take your pipeline changes through the following flow: + +1. When a new branch is created a Deployment Preview will be created based off your orginal Deployment +2. When a PR is created from a branch code changes will be deployed to the Deployment Preivew +3. When a PR is merged into your "main" branch code changes will be deployed to the orginal Deployment +4. When a branch is deleted the the corresponding Deployment Preview will also be deleted + +## Create Deployment Preivew + +``` +name: Astronomer CI - Create deployment preview + +on: + create: + branches: + - "**" + +env: + ## Sets Deployment API key credentials as environment variables + ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }} + +jobs: + deploy: + runs-on: ubuntu-latest + steps: + - name: Create Deployment Preivew + uses: astronomer/deploy-action@v0.2 + with: + action: create-deployment-preview + deployment-id: +``` + +## Deploy to Deployment Preview + +``` +name: Astronomer CI - Deploy code to Preview + +on: + pull_request: + branches: + - main + +env: + ## Sets Deployment API key credentials as environment variables + ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }} + +jobs: + deploy: + runs-on: ubuntu-latest + steps: + - name: Deploy to Deployment Preivew + uses: astronomer/deploy-action@v0.2 + with: + action: deploy-deployment-preview + deployment-id: +``` + +## Delete Deployment Preview + +``` +name: Astronomer CI - Delete Deployment Preview + +on: + delete: + branches: + - "**" + +env: + ## Sets Deployment API key credentials as environment variables + ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }} + +jobs: + deploy: + runs-on: ubuntu-latest + steps: + - name: Create Deployment Preivew + uses: astronomer/deploy-action@v0.2 + with: + action: delete-deployment-preview + deployment-id: +``` + +## Deploy to Orginal Deployment + +``` +name: Astronomer CI - Deploy code to Astro + +on: + push: + branches: + - main + +env: + ## Sets Deployment API key credentials as environment variables + ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }} + +jobs: + deploy: + runs-on: ubuntu-latest + steps: + - name: Deploy to Astro + uses: astronomer/deploy-action@v0.2 + with: + deployment-id: +``` +## Delete Deployment Preview + +``` +name: Astronomer CI - Delete Deployment Preview + +on: + delete: + branches: + - "**" + +env: + ## Sets Deployment API key credentials as environment variables + ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }} + +jobs: + deploy: + runs-on: ubuntu-latest + steps: + - name: Create Deployment Preivew + uses: astronomer/deploy-action@v0.2 + with: + action: delete-deployment-preview + deployment-id: +``` diff --git a/action.yaml b/action.yaml index dfb760d..c74ddfd 100644 --- a/action.yaml +++ b/action.yaml @@ -4,10 +4,6 @@ branding: icon: 'upload-cloud' color: 'purple' inputs: - dag-deploy-enabled: - required: false - default: false - description: "If true only DAGs will be deployed when dag files are pushed." root-folder: required: false default: ./ @@ -30,7 +26,28 @@ inputs: image-name: required: false default: no-custom-image - description: Specify a custom built image to deploy to an Asto Deployment. + description: "Specify a custom built image to deploy to an Asto Deployment." + action: + required: false + default: deploy + description: "Specify what action you would like to take. Use this option to create or delete deployment previews. Specify either 'create-deployment-preview', 'delete-deployment-preview' or 'deploy-deployment-preview'. Don't sepcify anything if you are deploying to a regular deployment." + deployment-name: + required: false + default: false + description: "The name of the deployment you want to make preview from or are deploying too." + deployment-id: + required: false + default: false + description: "The id of the deployment you to make a preview from or are deploying too." + workspace: + required: false + default: false + description: "If you are using an organization token you will need to provide a workspace-name or id." + preview-name: + required: false + default: false + description: "Custom preview name. By default this is branch name “_” deployment name" + runs: using: "composite" steps: @@ -38,7 +55,167 @@ runs: uses: actions/checkout@v3 with: fetch-depth: 2 - # Determine if only DAGs have changes + # Determine if only DAGs have changes + - name: Install Astro + run: | + curl -sSL https://install.astronomer.io | sudo bash -s + shell: bash + - name: Deployment Preview action + run: | + + # set action + ACTION=nothing + + # switch workspace if specified + if [[ ${{ inputs.workspace }} != false ]]; then + astro workspace switch ${{ inputs.workspace }} + fi + # error if both deployment name and id are you used + if [[ "${{ inputs.deployment-name }}" != false && ${{ inputs.deployment-id }} != false ]]; then + echo ERROR: cannot specify both a Deployment ID and Name + exit 1 # terminate and indicate error + fi + # figure out deployment id + if [[ "${{ inputs.deployment-name }}" != false ]]; then + # get deployment-id + DEPLOYMENT_ID="$(astro deployment inspect --clean-output -n "${{ inputs.deployment-name }}" --key metadata.deployment_id)" + fi + + if [[ ${{ inputs.deployment-id }} != false ]]; then + DEPLOYMENT_ID=${{ inputs.deployment-id }} + fi + + # create deployment preview if action is create-deployment-preview + if [[ ${{ inputs.action }} == create-deployment-preview ]]; then + + if [[ "${{ inputs.deployment-name }}" == false && ${{ inputs.deployment-id }} == false ]]; then + echo ERROR: cannot create a deployment preview without specifying a deployment name or id + exit 1 # terminate and indicate error + fi + + if [[ ${{ inputs.preview-name }} != false ]]; then + BRANCH_DEPLOYMENT_NAME=${{ inputs.preview-name }} + + else + # get branch name + branch_ref="${{ github.ref }}" + branch_name="${branch_ref##*/}" + DEPLOYMENT_NAME="$(./astro deployment inspect --clean-output $DEPLOYMENT_ID --key configuration.name)" + BRANCH_DEPLOYMENT_NAME=${branch_name}_$DEPLOYMENT_NAME + BRANCH_DEPLOYMENT_NAME="${BRANCH_DEPLOYMENT_NAME// /_}" + echo $BRANCH_DEPLOYMENT_NAME + fi + + # Create template of deployment to be copied with + ./astro deployment inspect --clean-output $DEPLOYMENT_ID --template > deployment-preview-template.yaml # autmatically creates deployment-preview-template.yaml file + + # Add name to deployment template file + sed -i "s| name:.*| name: ${BRANCH_DEPLOYMENT_NAME}|g" deployment-preview-template.yaml + + # Create new deploymeent preview based on the deployment template file + ./astro deployment create --clean-output --deployment-file deployment-preview-template.yaml + + # Get deployment ID + echo "FINAL_DEPLOYMENT_ID=$(astro deployment inspect --clean-output -n $BRANCH_DEPLOYMENT_NAME --key metadata.deployment_id)" >> $GITHUB_OUTPUT + + # don't skip deploy + echo "SKIP_DEPLOY=false" >> $GITHUB_OUTPUT + + # image deploy only + echo "IMAGE_DEPLOY_ONLY=true" >> $GITHUB_OUTPUT + + # set action + ACTION=create + fi + + # delete deployment preview and skip deploy if action is delete-deployment-preview + if [[ ${{ inputs.action }} == delete-deployment-preview ]]; then + + if [[ "${{ inputs.deployment-name }}" == false && ${{ inputs.deployment-id }} == false ]]; then + echo ERROR: cannot delete a deployment preview without specifying a deployment name or id + exit 1 # terminate and indicate error + fi + + if [[ ${{ inputs.preview-name }} != false ]]; then + BRANCH_DEPLOYMENT_NAME=${{ inputs.preview-name }} + + else + DEPLOYMENT_NAME="$(./astro deployment inspect --clean-output $DEPLOYMENT_ID --key configuration.name)" + BRANCH_DEPLOYMENT_NAME=${{ github.event.ref }}_$DEPLOYMENT_NAME + BRANCH_DEPLOYMENT_NAME="${BRANCH_DEPLOYMENT_NAME// /_}" + fi + # delete branch deployment + ./astro deployment delete -n $BRANCH_DEPLOYMENT_NAME -f + + # skip deploy + echo "SKIP_DEPLOY=true" >> $GITHUB_OUTPUT + + # not image deploy only + echo "IMAGE_DEPLOY_ONLY=false" >> $GITHUB_OUTPUT + + # set action + ACTION=delete + fi + + # # deploy to deployment preview if action is cdeploy-deployment-preview + if [[ ${{ inputs.action }} == deploy-deployment-preview ]]; then + + if [[ "${{ inputs.deployment-name }}" == false && ${{ inputs.deployment-id }} == false ]]; then + echo ERROR: cannot deploy to a deployment preview without specifying a deployment name or id + exit 1 # terminate and indicate error + fi + + if [[ ${{ inputs.preview-name }} != false ]]; then + BRANCH_DEPLOYMENT_NAME=${{ inputs.preview-name }} + + else + DEPLOYMENT_NAME="$(./astro deployment inspect --clean-output $DEPLOYMENT_ID --key configuration.name)" + BRANCH_DEPLOYMENT_NAME=${GITHUB_HEAD_REF##*/}_$DEPLOYMENT_NAME + BRANCH_DEPLOYMENT_NAME="${BRANCH_DEPLOYMENT_NAME// /_}" + fi + + # Get deployment ID + echo "FINAL_DEPLOYMENT_ID=$(astro deployment inspect --clean-output -n $BRANCH_DEPLOYMENT_NAME --key metadata.deployment_id)" >> $GITHUB_OUTPUT + + # don't skip deploy + echo "SKIP_DEPLOY=false" >> $GITHUB_OUTPUT + + # not image deploy only + echo "IMAGE_DEPLOY_ONLY=false" >> $GITHUB_OUTPUT + + # set action + ACTION=deploy-preview + fi + + # if action is deploy set final deployment id to deployment id + if [[ ${{ inputs.action }} == deploy ]]; then + echo "FINAL_DEPLOYMENT_ID=$DEPLOYMENT_ID" >> $GITHUB_OUTPUT + + # don't skip deploy + echo "SKIP_DEPLOY=false" >> $GITHUB_OUTPUT + + # not image deploy only + echo "IMAGE_DEPLOY_ONLY=false" >> $GITHUB_OUTPUT + + # set action + ACTION=deploy + fi + + if [[ $ACTION == nothing ]]; then + echo ERROR: you specfied an improper action input. Action must be deploy, deploy-deployment-preview, create-deployment-preview, or delete-deployment-preview. + exit 1 # terminate and indicate error + fi + shell: bash + id: deployment-preview + - name: Determine if DAG Deploy is enabled + run: | + if [[ ${{steps.deployment-preview.outputs.SKIP_DEPLOY}} == false && ${{steps.deployment-preview.outputs.IMAGE_DEPLOY_ONLY}} == false ]]; then + echo "DAG_DEPLOY_ENABLED=$(astro deployment inspect --clean-output ${{steps.deployment-preview.outputs.FINAL_DEPLOYMENT_ID}} --key configuration.dag_deploy_enabled)" >> $GITHUB_OUTPUT + else + echo "DAG_DEPLOY_ENABLED=false" >> $GITHUB_OUTPUT + fi + shell: bash + id: dag-deploy-enabled - name: Get Deployment Type run: | cd ${{ inputs.root-folder }} @@ -53,10 +230,15 @@ runs: fi done - if [[ ${{ inputs.dag-deploy-enabled }} == false ]]; then + if [[ ${{steps.dag-deploy-enabled.outputs.DAG_DEPLOY_ENABLED}} == false ]]; then dags_only=0 fi + if [[ ${{steps.deployment-preview.outputs.SKIP_DEPLOY}} == true ]]; then + # skip all deploy steps + dags_only=2 + fi + echo "DAGS_ONLY=$dags_only" >> $GITHUB_OUTPUT shell: bash id: deployment-type @@ -92,15 +274,13 @@ runs: if: steps.deployment-type.outputs.DAGS_ONLY == 1 run: | cd ${{ inputs.root-folder }} - curl -sSL https://install.astronomer.io | sudo bash -s - astro deploy --dags ${{steps.deploy-options.outputs.OPTIONS}} + astro deploy ${{steps.deployment-preview.outputs.FINAL_DEPLOYMENT_ID}} --dags ${{steps.deploy-options.outputs.OPTIONS}} shell: bash # If any other files changed or dag deploys is disabled, deploy the entire Astro project - name: Image and DAG Deploy to Astro if: steps.deployment-type.outputs.DAGS_ONLY == 0 run: | cd ${{ inputs.root-folder }} - curl -sSL https://install.astronomer.io | sudo bash -s - astro deploy ${{steps.deploy-options.outputs.OPTIONS}} + astro deploy ${{steps.deployment-preview.outputs.FINAL_DEPLOYMENT_ID}} ${{steps.deploy-options.outputs.OPTIONS}} shell: bash