Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

B-21322 TPPS ECS Scheduled Task - INT #14790

Merged
merged 141 commits into from
Feb 17, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
141 commits
Select commit Hold shift + click to select a range
ad2c048
cleanup from older tpps feature work
traskowskycaci Nov 15, 2024
0a6a812
cli works to run but needs more
traskowskycaci Nov 18, 2024
a4920f7
add step to deploy process-tpps task
traskowskycaci Nov 18, 2024
900a116
additional steps for deploying process-tpps task based on https://git…
traskowskycaci Nov 18, 2024
9893072
cleanup and some fine-tuning
traskowskycaci Nov 19, 2024
ce0b61c
rename process-tpps to process_tpps
traskowskycaci Nov 19, 2024
c4299cd
Merge branch 'main' into B-21322-MAIN
traskowskycaci Jan 6, 2025
b8a21f1
environment variable updates for pickup dir
traskowskycaci Jan 6, 2025
136b023
return out of processing if no valid filepath found
traskowskycaci Jan 7, 2025
0066beb
add temp logging for s3 path
traskowskycaci Jan 7, 2025
5828df2
circle config for demo
traskowskycaci Jan 7, 2025
437f3e3
Merge branch 'main' into B-21322-MAIN
traskowskycaci Jan 7, 2025
3046c07
add demo env process tips
traskowskycaci Jan 7, 2025
bc7ac26
release demo env
traskowskycaci Jan 8, 2025
27adca0
deploy to exp
traskowskycaci Jan 8, 2025
0c744e8
release exp
traskowskycaci Jan 8, 2025
9d7e15a
deploy to exp
traskowskycaci Jan 9, 2025
d48b52d
release exp
traskowskycaci Jan 9, 2025
eb37ded
Merge branch 'main' into B-21322-MAIN
traskowskycaci Jan 14, 2025
d4070bb
update filepath and deploy_dp3_tasks
traskowskycaci Jan 15, 2025
3fdad8f
deploy to exp
traskowskycaci Jan 15, 2025
76a9742
release exp
traskowskycaci Jan 15, 2025
87213e7
comment out some things in config for now
traskowskycaci Jan 15, 2025
a8af25b
add some temp logging to ecs deploy script for debugging
traskowskycaci Jan 17, 2025
40e70e1
deploy to exp
traskowskycaci Jan 17, 2025
a329f8d
add process-edis back in to deploy_dp3_tasks_steps
traskowskycaci Jan 17, 2025
44d0238
remove temp logging
traskowskycaci Jan 17, 2025
07f3d70
release exp
traskowskycaci Jan 17, 2025
28e11d1
init process tpps flags before adding command
traskowskycaci Jan 20, 2025
622b671
add pseudocode plan for processing specific filenames
traskowskycaci Jan 20, 2025
3f5e66d
get the filepath to process in process_tpps.go to pass filepath to Pr…
traskowskycaci Jan 20, 2025
65399c1
add logging to process_tpps.go and update SilenceUsage to false for p…
traskowskycaci Jan 20, 2025
55f7530
deploy to exp
traskowskycaci Jan 20, 2025
7e98df4
release exp
traskowskycaci Jan 20, 2025
4c4b08a
add logging for env var troubleshooting
traskowskycaci Jan 22, 2025
9f3b472
deploy to exp
traskowskycaci Jan 22, 2025
8eff3b4
swap order for db config call
traskowskycaci Jan 22, 2025
e00a34c
db init updates
traskowskycaci Jan 22, 2025
e69e4e6
updates
traskowskycaci Jan 22, 2025
eafa86b
logging updates
traskowskycaci Jan 22, 2025
cdec799
init those dang flags again I guess
traskowskycaci Jan 22, 2025
e884c23
update logging
traskowskycaci Jan 22, 2025
0a7883e
add db_region to exp.process-tpps.env
traskowskycaci Jan 23, 2025
9e0e391
add dod_ca_package back in
traskowskycaci Jan 23, 2025
649b480
remove second call to initProcessTPPSFlags in case that is causing re…
traskowskycaci Jan 23, 2025
0913e7e
add logging to ecs deploy script
traskowskycaci Jan 23, 2025
85a4a5b
fix filepath to hard-coded value for test
traskowskycaci Jan 23, 2025
194f713
filepath fixes that may work
traskowskycaci Jan 23, 2025
5858f50
release exp
traskowskycaci Jan 23, 2025
0f6528f
s3 filepath doesn't need cleaned up, remove call to that
traskowskycaci Jan 23, 2025
be90c57
deploy to exp
traskowskycaci Jan 23, 2025
c9ba4bd
make process-tpps.env match process-edis.env to see if task revision …
traskowskycaci Jan 23, 2025
f6c544b
remove some logging from ecs deploy
traskowskycaci Jan 24, 2025
0b3621c
config file changes as a test
traskowskycaci Jan 24, 2025
03f00bd
add a bunch of logging to put target and task def
traskowskycaci Jan 24, 2025
b964eff
Revert "add a bunch of logging to put target and task def"
traskowskycaci Jan 24, 2025
e72fa94
release exp
traskowskycaci Jan 24, 2025
d8f9d86
cleanup unneeded logging statements
traskowskycaci Jan 24, 2025
88e6fd9
filepath date improvements
traskowskycaci Jan 24, 2025
2967aea
Merge branch 'main' into B-21322-MAIN
traskowskycaci Jan 24, 2025
8225d79
deploy to exp
traskowskycaci Jan 24, 2025
b47efb5
release exp
traskowskycaci Jan 24, 2025
3ca0c7f
deploy to exp
traskowskycaci Feb 3, 2025
f4ddf1c
Merge branch 'main' into B-21322-MAIN
traskowskycaci Feb 3, 2025
542a18a
add logging to figure out go time.now issue with weird year
traskowskycaci Feb 3, 2025
81e7935
add clamAV status check and s3 download to tmp file
traskowskycaci Feb 3, 2025
9cded67
add logging to test deploy again
traskowskycaci Feb 4, 2025
2acf786
modify gitlab.yml to deploy to exp
traskowskycaci Feb 4, 2025
9489140
changes for make nonato_deploy_restore
traskowskycaci Feb 4, 2025
2bf777b
update deploy of process tpps to deploy_tasks_dp3 in gitlab-ci.yml
traskowskycaci Feb 4, 2025
aaa7865
hard code stuff for testing purposes for now
traskowskycaci Feb 4, 2025
df4c9dc
add logging of s3 getObject response
traskowskycaci Feb 4, 2025
80ccc3f
more general logging
traskowskycaci Feb 4, 2025
09655ac
convert to utf-8 encoding and log metadata so we can see
traskowskycaci Feb 5, 2025
abde80e
get s3 object tags for av-status, not metadata
traskowskycaci Feb 5, 2025
c0546cb
adding a few new env vars for s3 bucket and s3 folder, logging, cleanup
traskowskycaci Feb 5, 2025
cb12e78
download file only if scan is clean
traskowskycaci Feb 5, 2025
95de363
mutable tmp volume for tasks_dp3
traskowskycaci Feb 5, 2025
25410a5
renaming func, some cleanup
traskowskycaci Feb 5, 2025
dc648fd
try with 0116 file with data in it
traskowskycaci Feb 5, 2025
729c74f
add logging of contents of local file
traskowskycaci Feb 5, 2025
adf6091
still trying to log contents of tmp file
traskowskycaci Feb 6, 2025
1fe9bf4
convert the logged output to utf16
traskowskycaci Feb 6, 2025
d664e0a
Merge branch 'main' into B-21322-MAIN
traskowskycaci Feb 6, 2025
2ae6dc8
undo deploy to exp
traskowskycaci Feb 6, 2025
07b9d73
changing how we process entries to fail gracefully if no matching pay…
traskowskycaci Feb 6, 2025
464abf4
deploy to exp
traskowskycaci Feb 6, 2025
309014b
cleanup some logs and remove hard-coded s3 key
traskowskycaci Feb 7, 2025
42f4802
make the logged output preview more readable
traskowskycaci Feb 7, 2025
dc9b769
more log cleanup
traskowskycaci Feb 7, 2025
d7398d0
log count of successful and unsuccessful rows added to DB and log cou…
traskowskycaci Feb 7, 2025
ac730cb
Merge branch 'main' into B-21322-MAIN
traskowskycaci Feb 7, 2025
246842d
additions and updates for testing
traskowskycaci Feb 10, 2025
31313bb
removing unusued env var
traskowskycaci Feb 10, 2025
46ccede
test updates
traskowskycaci Feb 10, 2025
c942d11
Merge branch 'main' into B-21322-MAIN
traskowskycaci Feb 10, 2025
fca8ab1
update timezone from utc to America/Chicago (central time)
traskowskycaci Feb 10, 2025
3d3be68
more tests
traskowskycaci Feb 10, 2025
9e55d69
temp hard coding of 0208 file
traskowskycaci Feb 10, 2025
2f026a5
Revert "more tests"
traskowskycaci Feb 11, 2025
c11e466
Revert "test updates"
traskowskycaci Feb 11, 2025
ae95a26
refactor parser to use csv reader to handle complex messages
traskowskycaci Feb 11, 2025
71e7806
Reapply "more tests"
traskowskycaci Feb 11, 2025
9c11ebf
Reapply "test updates"
traskowskycaci Feb 11, 2025
4afb196
remove unnecessary logs
traskowskycaci Feb 11, 2025
35489c9
Merge branch 'main' into B-21322-MAIN
traskowskycaci Feb 11, 2025
830020c
Merge branch 'B-21322-MAIN' into B-21322-INT
traskowskycaci Feb 11, 2025
ec9d2fd
remove unused arg
traskowskycaci Feb 11, 2025
9308dd2
Merge branch 'B-21322-MAIN' into B-21322-INT
traskowskycaci Feb 11, 2025
dbc4c17
undo deploy to exp
traskowskycaci Feb 11, 2025
3b704ef
Merge branch 'B-21322-MAIN' into B-21322-INT
traskowskycaci Feb 11, 2025
6cdb07c
int specific commit - deploying int branch to exp
traskowskycaci Feb 11, 2025
e1631c4
Merge branch 'integrationTesting' into B-21322-INT
traskowskycaci Feb 11, 2025
c44796f
Merge branch 'main' into B-21322-MAIN
traskowskycaci Feb 11, 2025
856eeef
Merge branch 'B-21322-MAIN' into B-21322-INT
traskowskycaci Feb 11, 2025
26c0cb0
int specific commit - fixing gitlab merge issue for docker_app_image
traskowskycaci Feb 11, 2025
de4c3f7
int specific commit - deploy to exp
traskowskycaci Feb 11, 2025
35e4990
test updates
traskowskycaci Feb 12, 2025
b5f42e5
Merge branch 'B-21322-MAIN' into B-21322-INT
traskowskycaci Feb 12, 2025
7843d3e
Merge branch 'integrationTesting' into B-21322-INT
traskowskycaci Feb 12, 2025
b19737f
int specific commit - deploy integrationTesting to exp
traskowskycaci Feb 12, 2025
afee218
update AVStatusUNKNOWN comment
traskowskycaci Feb 12, 2025
4b87765
lowercase err messages
traskowskycaci Feb 12, 2025
c7f7d5c
Merge branch 'B-21322-MAIN' into B-21322-INT
traskowskycaci Feb 12, 2025
72dfefc
set processTPPS SilenceUsage to true
traskowskycaci Feb 12, 2025
0bac6d8
add comment explaining AVStatusUNKNOWN usage
traskowskycaci Feb 12, 2025
d51739b
Merge branch 'B-21322-MAIN' into B-21322-INT
traskowskycaci Feb 12, 2025
0e5e0da
check tpps flags in config check
traskowskycaci Feb 12, 2025
121aaa4
Merge branch 'B-21322-MAIN' into B-21322-INT
traskowskycaci Feb 12, 2025
3dd4545
int specific commit - deploying latest 21322 int to exp
traskowskycaci Feb 12, 2025
ef8db0b
int specific commit - deploy integrationTesting to exp
traskowskycaci Feb 13, 2025
c4ace72
Merge branch 'integrationTesting' into B-21322-INT
traskowskycaci Feb 14, 2025
37c6afc
Merge branch 'integrationTesting' into B-21322-INT
traskowskycaci Feb 14, 2025
f4f891f
int specific commit - deploy B-21322-int to exp
traskowskycaci Feb 14, 2025
b0d4aed
add init flags check
traskowskycaci Feb 14, 2025
f98a957
Merge branch 'B-21322-MAIN' into B-21322-INT
traskowskycaci Feb 14, 2025
31dba9b
int specific commit - undo deploy 21322 to exp
traskowskycaci Feb 17, 2025
7c62f49
remove changes from config.yml as it's not used anymore
traskowskycaci Feb 17, 2025
32d1753
Merge branch 'B-21322-MAIN' into B-21322-INT
traskowskycaci Feb 17, 2025
42ec54f
Merge branch 'integrationTesting' into B-21322-INT
traskowskycaci Feb 17, 2025
5cd0ee3
Merge branch 'integrationTesting' into B-21322-INT
traskowskycaci Feb 17, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitlab-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1394,6 +1394,8 @@ deploy_tasks_dp3:
- ./scripts/ecs-deploy-task-container save-ghc-fuel-price-data "${ECR_REPOSITORY_URI}/app-tasks@${ECR_DIGEST}" "${APP_ENVIRONMENT}"
- echo "Deploying payment reminder email task service"
- ./scripts/ecs-deploy-task-container send-payment-reminder "${ECR_REPOSITORY_URI}/app-tasks@${ECR_DIGEST}" "${APP_ENVIRONMENT}"
- echo "Deploying process TPPS task service"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving higher level env script updates as commented out/not added yet. Will have a follow up PR when necessary higher level AWS changes are in place to pull in the script updates to this file.

- ./scripts/ecs-deploy-task-container process-tpps "${ECR_REPOSITORY_URI}/app-tasks@${ECR_DIGEST}" "${APP_ENVIRONMENT}"
after_script:
- *announce_failure
rules:
Expand Down
4 changes: 4 additions & 0 deletions Dockerfile.tasks_dp3
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,8 @@ COPY bin/rds-ca-rsa4096-g1.pem /bin/rds-ca-rsa4096-g1.pem

COPY bin/milmove-tasks /bin/milmove-tasks

# Mount mutable tmp for process-tpps
# hadolint ignore=DL3007
VOLUME ["/tmp"]

WORKDIR /bin
16 changes: 16 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -831,6 +831,22 @@ tasks_process_edis: tasks_build_linux_docker ## Run process-edis from inside doc
$(TASKS_DOCKER_CONTAINER):latest \
milmove-tasks process-edis

.PHONY: tasks_process_tpps
tasks_process_tpps: tasks_build_linux_docker ## Run process-tpps from inside docker container
@echo "Processing TPPS files with docker command..."
DB_NAME=$(DB_NAME_DEV) DB_DOCKER_CONTAINER=$(DB_DOCKER_CONTAINER_DEV) scripts/wait-for-db-docker
docker run \
-t \
-e DB_HOST="database" \
-e DB_NAME \
-e DB_PORT \
-e DB_USER \
-e DB_PASSWORD \
--link="$(DB_DOCKER_CONTAINER_DEV):database" \
--rm \
$(TASKS_DOCKER_CONTAINER):latest \
milmove-tasks process-tpps

.PHONY: tasks_save_ghc_fuel_price_data
tasks_save_ghc_fuel_price_data: tasks_build_linux_docker ## Run save-ghc-fuel-price-data from inside docker container
@echo "Saving the fuel price data to the ${DB_NAME_DEV} database with docker command..."
Expand Down
1 change: 1 addition & 0 deletions cmd/ecs-deploy/put_target.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ var names = []string{
"connect-to-gex-via-sftp",
"post-file-to-gex",
"process-edis",
"process-tpps",
"save-ghc-fuel-price-data",
"send-payment-reminder",
}
Expand Down
1 change: 1 addition & 0 deletions cmd/ecs-deploy/task_def.go
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ var servicesToEntryPoints = map[string][]string{
fmt.Sprintf("%s connect-to-gex-via-sftp", binMilMoveTasks),
fmt.Sprintf("%s post-file-to-gex", binMilMoveTasks),
fmt.Sprintf("%s process-edis", binMilMoveTasks),
fmt.Sprintf("%s process-tpps", binMilMoveTasks),
fmt.Sprintf("%s save-ghc-fuel-price-data", binMilMoveTasks),
fmt.Sprintf("%s send-payment-reminder", binMilMoveTasks),
},
Expand Down
10 changes: 10 additions & 0 deletions cmd/milmove-tasks/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,16 @@ func main() {
initConnectToGEXViaSFTPFlags(processEDIsCommand.Flags())
root.AddCommand(processEDIsCommand)

processTPPSCommand := &cobra.Command{
Use: "process-tpps",
Short: "process TPPS files asynchrounously",
Long: "process TPPS files asynchrounously",
RunE: processTPPS,
SilenceUsage: true,
}
initProcessTPPSFlags(processTPPSCommand.Flags())
root.AddCommand(processTPPSCommand)

completionCommand := &cobra.Command{
Use: "completion",
Short: "Generates bash completion scripts",
Expand Down
11 changes: 0 additions & 11 deletions cmd/milmove-tasks/process_edis.go
Original file line number Diff line number Diff line change
Expand Up @@ -244,16 +244,5 @@ func processEDIs(_ *cobra.Command, _ []string) error {
logger.Info("Successfully processed EDI824 application advice responses")
}

// Pending completion of B-20560, uncomment the code below
/*
// Process TPPS paid invoice report
pathTPPSPaidInvoiceReport := v.GetString(cli.SFTPTPPSPaidInvoiceReportPickupDirectory)
_, err = syncadaSFTPSession.FetchAndProcessSyncadaFiles(appCtx, pathTPPSPaidInvoiceReport, lastReadTime, invoice.NewTPPSPaidInvoiceReportProcessor())
if err != nil {
logger.Error("Error reading TPPS Paid Invoice Report application advice responses", zap.Error(err))
} else {
logger.Info("Successfully processed TPPS Paid Invoice Report application advice responses")
}
*/
return nil
}
Loading