Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase instance termination time limit to 3 hours. #14181

Merged
merged 3 commits into from
Jun 27, 2022
Merged

Increase instance termination time limit to 3 hours. #14181

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
23cc7ab
Update to 3 hours.
davinchia Jun 27, 2022
67fd7f7
Typo.
davinchia Jun 27, 2022
c2398c1
Update terminate-zombie-build-instances.yml
davinchia Jun 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions .github/workflows/terminate-zombie-build-instances.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Required since we cannot guarantee instances are always terminated.
# Also a failsafe against a dev writing a workflow that does not terminate build instances.
# The average runtime as of this commit is ~20 mins. Set this to an hour for some buffer.
# Though the average Airbyte build runtime as of this commit is ~20 mins, connector builds
# can take up to 3 hours. Set this to 3 hours to include these longer runs.
name: Terminate Zombie Build Instances

on:
Expand All @@ -12,7 +13,7 @@ jobs:
terminate:
runs-on: ubuntu-latest
steps:
- name: List and Terminate Instances Older Than an Hour
- name: List and Terminate Instances Older Than 3 Hours
env:
AWS_ACCESS_KEY_ID: ${{ secrets.SELF_RUNNER_AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.SELF_RUNNER_AWS_SECRET_ACCESS_KEY }}
Expand All @@ -21,14 +22,16 @@ jobs:
run: |
set -euxo pipefail

TIME_LIMIT=10800 // 3 hours

aws configure set default.region us-east-2

# See https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2/describe-instances.html for describe command.
# Since the AWS cli returns an ISO HH:MM timestamp, and Jq only accepts Z timestamps, we define a function toZ to convert this.
export to_terminate=$(aws ec2 describe-instances --no-paginate --filters Name=instance-type,Values=c5.2xlarge Name=instance-state-name,Values=running \
--query 'Reservations[*].Instances[*].{Instance:InstanceId,LaunchTime:LaunchTime}' --output json \
| jq 'def toZ(str): str | (split("+")[0] + "Z") | fromdate ;
flatten | map( { InstanceId: .Instance, LaunchTime: toZ(.LaunchTime) } ) | map( select ( .LaunchTime < (now - 3600) ) )')
flatten | map( { InstanceId: .Instance, LaunchTime: toZ(.LaunchTime) } ) | map( select ( .LaunchTime < (now - $TIME_LIMIT) ) )')

echo "MARKED FOR TERMINATION: ${to_terminate}"

Expand Down