B-21322 TPPS ECS Scheduled Task - INT #14790
Conversation
|
Also re-requesting @cameroncaci since I added a commit to deploy latest 21322 int branch to exp |
.circleci/config.yml
Outdated
| @@ -470,6 +470,11 @@ commands: | |||
| command: scripts/do-exclusively --job-name ${CIRCLE_JOB} scripts/ecs-deploy-task-container process-edis "${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/app-tasks@${ECR_DIGEST}" "${APP_ENVIRONMENT}" | |||
| no_output_timeout: 20m | |||
| - announce_failure | |||
| # - run: | |||
There was a problem hiding this comment.
Leaving higher level env script updates as commented out/not added yet. Will have a follow up PR when necessary higher level AWS changes are in place to pull in the script updates to this file.
There was a problem hiding this comment.
changes in this file is unnecessary
circle no more
There was a problem hiding this comment.
Removed those changes here
| @@ -1394,6 +1394,8 @@ deploy_tasks_dp3: | |||
| - ./scripts/ecs-deploy-task-container save-ghc-fuel-price-data "${ECR_REPOSITORY_URI}/app-tasks@${ECR_DIGEST}" "${APP_ENVIRONMENT}" | |||
| - echo "Deploying payment reminder email task service" | |||
| - ./scripts/ecs-deploy-task-container send-payment-reminder "${ECR_REPOSITORY_URI}/app-tasks@${ECR_DIGEST}" "${APP_ENVIRONMENT}" | |||
| - echo "Deploying process TPPS task service" | |||
There was a problem hiding this comment.
Leaving higher level env script updates as commented out/not added yet. Will have a follow up PR when necessary higher level AWS changes are in place to pull in the script updates to this file.
|
Parameters added to loadtest environment: |
There was a problem hiding this comment.
Re-requesting reviews from @cameroncaci and @deandreJones, removed changes from config.yml
https://dp3.atlassian.net/wiki/spaces/MT/pages/1469284485/How+to+create+an+ECS+Scheduled+Task
Agility ticket
Summary
The responsibility of this scheduled task is to grab a TPPS payment file from an S3 bucket, check its ClamAV status to ensure it's CLEAN, and if CLEAN, process the file line by line and store it in our database. Each line in the payment file has an invoice number that will match up to a payment request number.
If no matching payment request number is found, or some error with one line is found, we will continue to process the rest of the entries in the file so that one bad row doesn't keep the rest of the good data in that file from being processed.
This task will run once daily, processing the payment file for the previous day's date. This matches the cadence of the TPPS lambda, which is the service responsible for putting the payment files into the S3 bucket for this new task service to process from that point.
AWS Config
S3 permissions needed in each environment for this task:
Parameters added for this work to the AWS parameter store:
expat time of int review, and at the time of main review, instg, andprdenvironments. The below links are for theexpenv/app-exp/process_tpps_custom_date_file: Parameter than can be modified if we want to process a custom dated TPPS payment file as opposed to the scheduled daily run where we process the previous day's payment file/app-exp/tpps_s3_bucket: S3 bucket path for the TPPS payment files/app-exp/tpps_s3_folder: S3 subfolder inside bucket path for the TPPS payment filesTesting in the exp environment
To test:
Run the task locally (it will fail) with
go run ./cmd/milmove-tasks process-tpps.Verify that you see that we are trying to process yesterday's date - look for "Starting processing of TPPS data for" the date in the logs.
Log in to the experimental environment with
aws-vault login transcom-gov-milmove-expVerify that the
/app-exp/process_tpps_custom_date_file,/app-exp/tpps_s3_bucket, and/app-exp/tpps_s3_foldervariables exist and have defined values in the AWS parameter store.Review the ECS task revisions, created by pushing changes to
exp, here: https://us-gov-west-1.console.amazonaws-us-gov.com/ecs/v2/task-definitions/app-exp-process-tpps?region=us-gov-west-1&status=ACTIVE#Click on the most recent revision, then check the JSON for the task. Scroll down to
secrets. Ensure you see that we linked the parameter store values with the valueFrom attribute, similar to this:Review the (many) process-tpps logs here. This was set to run hourly while we test this, but will be running once a day in prod.
It's a little tough to test this without full AWS permissions, but the basic way to test in experimental is to
https://us-gov-west-1.console.amazonaws-us-gov.com/cloudwatch/home?region=us-gov-west-1#logsV2:log-groups/log-group/ecs-tasks-app-exp/log-events/app-tasks$252Fapp-tasks-process-tpps-exp$252F6641579e32df4ba98f7061581425af26
Note that the logs show that we are processing the MILMOVE-en20250208.csv file, and inserted 3 rows into the database from that file (since there were matching payment requests for that file).
Verify that the tpps_paid_invoice_reports table contains these three rows (Cameron, Dre, or someone with elevated permissions may need to assist here):
Checking out the move that had the payment requests added to it (which is in sort of a weird state because before adding the payment request to it, was only in Service Counseling Completed status) we can see that the payment request is now:
Another test case, testing out the AC:
I created a move in experimental, move GG6BG8 with move id 3c282057-b14a-48a7-aaf8-e4b81237fd10
I looked at the payment file of CSFTP1771/Pickup/MILMOVE-en20250122.csv, and picked 3 payment requests I wanted to process from the file:
I asked Bright Stars to update the /app-exp/process_tpps_custom_date_file parameter store value to be MILMOVE-en20250122.csv.
I asked Bright Stars to manually run the task and verified in the logs that:
I asked Bright Stars to set the /app-exp/process_tpps_custom_date_file parameter store value back to MILMOVE-enYYYYMMDD.csv
I asked Cameron to check the exp DB for me to confirm that those payment requests were inserted:
Next steps after testing in exp
Higher environment to-dos after this works in experimental:
Make sure S3 bucket is created and pulled in files on the daily run of the lambda
Check that the retention policy is set to 30 days for the TPPS files (see proof of 30 day retention policy for exp below)
Add the two parameter store values
Set both of those as environment variables keys in the ecs task
Add this to
.gitlab-ci.ymlunderdeploy_tasks_stganddeploy_tasks_prd(was already added todeploy_tasks_dp3forexpwork):config.ymlunderdeploy_tasks_steps(was already added todeploy_dp3_tasks_stepsforexpwork):Dockerfile.tasks( was already added toDockerfile.tasks_dp3):Bright Stars work that enables the work here:
In the terraform-aws-app-environment repo:
In the transcom-infrasec-gov-nonato repo:
POA Notes:
Proof of 30 day file retention on the S3 bucket in the exp environment:
