Update ingest CLI to support scheduled ingest #1103
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Re: #1079
These changes are in support of having a cron job in Rancher for periodic automated ingests. The idea is that cron job will use the existing
serverDocker image to do an ingest via the existingnmdc-serverCLI. Typically when we initiate an ingest manually we do so via the API with aPOST /api/jobs/ingestrequest. I'm proposing not using the API for the automated ingest because 1) the/api/jobs/ingestendpoint requires authentication and that would be challenging without persistent API authentication tokens and 2) we would probably need to add an additional endpoint to check on the job status after is has been finished in order to poll for success or failure.The changes here move the guts of the
ingestcelery task to a separate plain function (do_ingest) that can be called by both the celery task and the CLI to ensure they do the same operations. It also adds a--swap-rancher-secretsto the ingest CLI. If used, it will use the Rancher API to do the A/B database swap and redeploy the server and worker workloads. A number of environment variables need to be set in order to use that flag.