Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cron jobs to do automatic ingest #1079

Closed
pkalita-lbl opened this issue Nov 3, 2023 · 3 comments
Closed

Add cron jobs to do automatic ingest #1079

pkalita-lbl opened this issue Nov 3, 2023 · 3 comments

Comments

@pkalita-lbl
Copy link
Collaborator

As proposed in the Release Management Squad, it would be useful to do periodic, automatic ingests from Mongo. This would have a few benefits:

  • On dev (assuming ingest is done from the dev Mongo deployment -- which is not currently the case!) a failing daily ingest would give us an earlier heads-up if the ingest code needs to be updated to accommodate recent runtime changes.
  • On production (assuming ingest is done from the production Mongo deployment which is more stable see Reconfigure GitHub Actions to have separate dev and production deployments nmdc-runtime#357) a weekly ingest would alleviate the need for team members to make and respond to one-off requests for ingest.

Most of the ingest is already pretty well-automated (either via the API or CLI). The manual steps currently are doing the A/B database swap and restarting the service in Rancher. I assume those can be automated via the Rancher CLI.

@eecavanna
Copy link
Collaborator

I ran into some User ID-related issues when trying to set up a CronJob recently. Here's a link to my notes about it: https://github.com/microbiomedata/infra-admin/issues/27#issuecomment-1863740052

@ssarrafan
Copy link

Sprint over, no updates, removing from sprint and adding backlog label. If still active please move to new sprint.

@ssarrafan ssarrafan added the backlog Issues not assigned to a sprint or not finished during a sprint. Needs to be reprioritized. label Jan 16, 2024
@pkalita-lbl
Copy link
Collaborator Author

Oops sorry I think this can actually be closed now. The necessary code changes are in and Rancher configuration is done for the dev system (see the portal-ingest CronJob in nmdc-dev). The only outstanding thing to do is set up a similar CronJob in the prod namespace. But that needs to happen after the next release. Instead of keeping this issue open I'll add that as a post-release activity in microbiomedata/issues#549

@pkalita-lbl pkalita-lbl removed the backlog Issues not assigned to a sprint or not finished during a sprint. Needs to be reprioritized. label Jan 16, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in Release Management Squad Jan 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

No branches or pull requests

3 participants