Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Issue #3520] Add the ability to only process a subset of opportunity attachments #3564

Merged
merged 31 commits into from
Jan 21, 2025

Conversation

chouinar
Copy link
Collaborator

Summary

Fixes #3520

Time to review: 3 mins

Changes proposed

Add an env var that lets us run the opportunity attachment transform job only process a smaller set of attachments

Context for reviewers

Attachments are going to take a long time to run, and I wanted an easy way when testing to just tell the job to "stop after X attachments". This is that mechanism. It's pretty hacky/straightforward, just tell it to stop after processing a number of values, instead of finishing. It will still save these changes.

Additional information

If you want to test it locally, you can do the following.

First set the following env vars in override.env:

TRANSFORM_ORACLE_DATA_ENABLE_OPPORTUNITY_ATTACHMENT=1
TOTAL_ATTACHMENTS_TO_PROCESS=5

make volume-recreate to drop your DB state
make console and then:

f.StagingTsynopsisAttachmentFactory.create_batch(size=50)
exit()

Finally: make cmd args="data-migration load-transform --no-load --transform --no-set-current"

Copy link
Collaborator

@mikehgrantsgov mikehgrantsgov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting the env var in terraform would be in another PR? Or does it need to be in TF? Either way, this one looks good.

@chouinar
Copy link
Collaborator Author

Setting the env var in terraform would be in another PR? Or does it need to be in TF? Either way, this one looks good.

Won't ever set in terraform, can set manually when I run the job during testing. Can run jobs with the run-command.sh script we have.

A different command, but I ran: bin/run-command.sh --environment-variables "$(jq -c . ~/env_vars/load_job.json)" api dev '["poetry", "run", "flask", "task", "create-analytics-db-csvs"]'

Set the overriden env vars in a file that I load with jq on the fly. Sorta like an improved version of the run_ecs_task script we had on Mass PFML if you ever used that.

@chouinar chouinar merged commit ebb22a3 into main Jan 21, 2025
2 checks passed
@chouinar chouinar deleted the chouinar/3520-attachment-subset branch January 21, 2025 14:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make it possible to run opportunity attachment logic for a subset of attachments
2 participants