Clean up resources even though AWS resources are not used (bug fix) #4541

youngbupark · 2022-11-09T03:17:39Z

Description

This is the fix to clean up resource even though AWS resources are not used.

Issue reference

Fixes: #issue_number

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

Code compiles correctly
Adds necessary unit tests for change
Adds necessary E2E tests for change
Unit tests passing
Extended the documentation / Created issue for it

youngbupark · 2022-11-09T03:20:21Z

test/functional/corerp/corerptest.go

-			t.Logf("validating deletion of AWS resource for %s", ct.Description)
-			validation.ValidateNoAWSResource(ctx, t, &resource, ct.Options.AWSClient)
-			t.Logf("finished validation of deletion of AWS resource %s for %s", resource.Name, ct.Description)
+		if step.AWSResources != nil && len(step.AWSResources.Resources) > 0 {


@vinayada1 I think this is the bug. If there is no AWS resources, then it skips all resource clean up. Please let me know if there is reason we skip the non-aws resource clean up

Yes, this is a bug! Could we change the logic in line 292 as well just in case we end up adding more cleanup after that?

When I fix this, then I do see lots of failures from dapr related functional tests! I cannot merge this until I figure it out the root cause. I will set up the meeting with @vishwahiremat @vinayada1 @kachawla

Do we need to check if step.AWSResources.Resources is nil?

This reverts commit e525d25.

youngbupark · 2022-11-09T17:49:40Z

After this change, I do see lots of failures from dapr related functional tests. I do not see any issues in the other tests. I think we need to look at why it happens before releasing v0.15 - cc/ @vinayada1 @kachawla @vishwahiremat

=== CONT  Test_DaprInvokeHttpRoute/deploy_testdata/corerp-resources-dapr-httproute.bicep
    k8s.go:423: success! pod dapr-invokehttproute-dapr-frontend-698d7b784b-gljck has status: {Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-11-09 17:19:16 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-11-09 17:19:16 +0000 UTC ContainersNotReady containers with unready status: [dapr-frontend daprd]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-11-09 17:19:16 +0000 UTC ContainersNotReady containers with unready status: [dapr-frontend daprd]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-11-09 17:19:16 +0000 UTC  }]    172.18.0.2 10.244.0.58 [{10.244.0.58}] 2022-11-09 17:19:16 +0000 UTC [] [{dapr-frontend {nil &ContainerStateRunning{StartedAt:2022-11-09 17:19:17 +0000 UTC,} nil} {nil nil nil} false 0 radiusdev.azurecr.io/magpiego:pr-ado-20221109.10 radiusdev.azurecr.io/magpiego@sha256:5684f8ad14e830a50ac4e2b45a1592a248c1611793035b82262c93d99c7bb110 containerd://3b886732fad591a08438374a09d804def9c2c346dd4195c04decc924f3f649fd 0xc001298bea} {daprd {nil &ContainerStateRunning{StartedAt:2022-11-09 17:24:59 +0000 UTC,} nil} {nil nil &ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-11-09 17:22:12 +0000 UTC,FinishedAt:2022-11-09 17:22:18 +0000 UTC,ContainerID:containerd://2895b2bdece30ba6f3769bee3575780e60e109e4e66dd30e7d140a1314034158,}} false 6 docker.io/daprio/daprd:1.6.0 docker.io/daprio/daprd@sha256:db7ce4723d61566a2e6be24bd7d136e317023cf11ee78b9c6f55c00ec0c6223f containerd://0235f097e4395c13934ddc635b85fda2e2f5518de2a631a7bc544c3b7a8ea5b2 0xc001298c25}] BestEffort []}
    k8s.go:428: 
        	Error Trace:	/home/vsts/work/1/s/test/functional/corerp/resources/k8s.go:428
        	            				/home/vsts/work/1/s/test/functional/corerp/resources/k8s.go:311
        	            				/home/vsts/work/1/s/test/functional/corerp/resources/corerptest.go:259
        	Error:      	pod %v failed readiness checks
        	Test:       	Test_DaprInvokeHttpRoute/deploy_testdata/corerp-resources-dapr-httproute.bicep
        	Messages:   	dapr-invokehttproute-dapr-frontend-698d7b784b-gljck
    k8s.go:438: watching pod dapr-invokehttproute-dapr-frontend-698d7b784b-gljck for status.. current: {Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-11-09 17:19:16 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-11-09 17:19:16 +0000 UTC ContainersNotReady containers with unready status: [dapr-frontend daprd]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-11-09 17:19:16 +0000 UTC ContainersNotReady containers with unready status: [dapr-frontend daprd]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-11-09 17:19:16 +0000 UTC  }]    172.18.0.2 10.244.0.58 [{10.244.0.58}] 2022-11-09 17:19:16 +0000 UTC [] [{dapr-frontend {nil &ContainerStateRunning{StartedAt:2022-11-09 17:19:17 +0000 UTC,} nil} {nil nil nil} false 0 radiusdev.azurecr.io/magpiego:pr-ado-20221109.10 radiusdev.azurecr.io/magpiego@sha256:5684f8ad14e830a50ac4e2b45a1592a248c1611793035b82262c93d99c7bb110 containerd://3b886732fad591a08438374a09d804def9c2c346dd4195c04decc924f3f649fd 0xc001298bea} {daprd {nil &ContainerStateRunning{StartedAt:2022-11-09 17:24:59 +0000 UTC,} nil} {nil nil &ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-11-09 17:22:12 +0000 UTC,FinishedAt:2022-11-09 17:22:18 +0000 UTC,ContainerID:containerd://2895b2bdece30ba6f3769bee3575780e60e109e4e66dd30e7d140a1314034158,}} false 6 docker.io/daprio/daprd:1.6.0 docker.io/daprio/daprd@sha256:db7ce4723d61566a2e6be24bd7d136e317023cf11ee78b9c6f55c00ec0c6223f containerd://0235f097e4395c13934ddc635b85fda2e2f5518de2a631a7bc544c3b7a8ea5b2 0xc001298c25}] BestEffort []}
=== CONT  Test_DaprPubSubGeneric/deploy_testdata/corerp-resources-dapr-pubsub-generic.bicep
    k8s.go:423: success! pod corerp-resources-dapr-pubsub-generic-gnrc-publisher-7c8497rtx6c has status: {Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-11-09 17:19:07 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-11-09 17:19:07 +0000 UTC ContainersNotReady containers with unready status: [gnrc-publisher daprd]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-11-09 17:19:07 +0000 UTC ContainersNotReady containers with unready status: [gnrc-publisher daprd]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-11-09 17:19:07 +0000 UTC  }]    172.18.0.2 10.244.0.56 [{10.244.0.56}] 2022-11-09 17:19:07 +0000 UTC [] [{daprd {nil &ContainerStateRunning{StartedAt:2022-11-09 17:24:59 +0000 UTC,} nil} {nil nil &ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-11-09 17:22:11 +0000 UTC,FinishedAt:2022-11-09 17:22:17 +0000 UTC,ContainerID:containerd://da69ef0e8e01503f2b01fb0ee9cab229725b09a8f102e62e6fd010bc91cb3f08,}} false 6 docker.io/daprio/daprd:1.6.0 docker.io/daprio/daprd@sha256:db7ce4723d61566a2e6be24bd7d136e317023cf11ee78b9c6f55c00ec0c6223f containerd://8d00bd017398f8a3401a44ed59ebe44b58f94d13c5a6e980ab8b52ca8daa71dd 0xc000fab265} {gnrc-publisher {nil &ContainerStateRunning{StartedAt:2022-11-09 17:19:08 +0000 UTC,} nil} {nil nil nil} false 0 radiusdev.azurecr.io/magpiego:pr-ado-20221109.10 radiusdev.azurecr.io/magpiego@sha256:5684f8ad14e830a50ac4e2b45a1592a248c1611793035b82262c93d99c7bb110 containerd://e9790cad46aa71769fbce3c0843d433d0347ed9d7ff1e89d67d392c7a269e6c7 0xc000fab266}] BestEffort []}
    k8s.go:428: 
        	Error Trace:	/home/vsts/work/1/s/test/functional/corerp/resources/k8s.go:428
        	            				/home/vsts/work/1/s/test/functional/corerp/resources/k8s.go:311
        	            				/home/vsts/work/1/s/test/functional/corerp/resources/corerptest.go:259
        	Error:      	pod %v failed readiness checks
        	Test:       	Test_DaprPubSubGeneric/deploy_testdata/corerp-resources-dapr-pubsub-generic.bicep
        	Messages:   	corerp-resources-dapr-pubsub-generic-gnrc-publisher-7c8497rtx6c
    k8s.go:438: watching pod corerp-resources-dapr-pubsub-generic-gnrc-publisher-7c8497rtx6c for status.. current: {Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-11-09 17:19:07 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-11-09 17:19:07 +0000 UTC ContainersNotReady containers with unready status: [gnrc-publisher daprd]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-11-09 17:19:07 +0000 UTC ContainersNotReady containers with unready status: [gnrc-publisher daprd]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-11-09 17:19:07 +0000 UTC  }]    172.18.0.2 10.244.0.56 [{10.244.0.56}] 2022-11-09 17:19:07 +0000 UTC [] [{daprd {nil &ContainerStateRunning{StartedAt:2022-11-09 17:24:59 +0000 UTC,} nil} {nil nil &ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-11-09 17:22:11 +0000 UTC,FinishedAt:2022-11-09 17:22:17 +0000 UTC,ContainerID:containerd://da69ef0e8e01503f2b01fb0ee9cab229725b09a8f102e62e6fd010bc91cb3f08,}} false 6 docker.io/daprio/daprd:1.6.0 docker.io/daprio/daprd@sha256:db7ce4723d61566a2e6be24bd7d136e317023cf11ee78b9c6f55c00ec0c6223f containerd://8d00bd017398f8a3401a44ed59ebe44b58f94d13c5a6e980ab8b52ca8daa71dd 0xc000fab265} {gnrc-publisher {nil &ContainerStateRunning{StartedAt:2022-11-09 17:19:08 +0000 UTC,} nil} {nil nil nil} false 0 radiusdev.azurecr.io/magpiego:pr-ado-20221109.10 radiusdev.azurecr.io/magpiego@sha256:5684f8ad14e830a50ac4e2b45a1592a248c1611793035b82262c93d99c7bb110 containerd://e9790cad46aa71769fbce3c0843d433d0347ed9d7ff1e89d67d392c7a269e6c7 0xc000fab266}] BestEffort []}
=== CONT  Test_AzureConnections/deploy_testdata/corerp-azure-connection-database-service.bicep

youngbupark · 2022-11-10T22:32:11Z

closing with #4612

fix aws resource clean up

3603bda

youngbupark requested a review from a team as a code owner November 9, 2022 03:17

youngbupark commented Nov 9, 2022

View reviewed changes

youngbupark mentioned this pull request Nov 9, 2022

functional test does not clean up core rp resources #4543

Closed

youngbupark added 2 commits November 8, 2022 20:20

revert for testing

e525d25

Merge branch 'main' into youngp/fix-teardown

8caaa65

vinayada1 approved these changes Nov 9, 2022

View reviewed changes

Revert "revert for testing"

f31a095

This reverts commit e525d25.

jkotalik approved these changes Nov 9, 2022

View reviewed changes

youngbupark added 2 commits November 9, 2022 14:53

Merge branch 'main' into youngp/fix-teardown

562bd1c

Merge branch 'main' into youngp/fix-teardown

868911f

youngbupark closed this Nov 10, 2022

youngbupark deleted the youngp/fix-teardown branch February 7, 2023 20:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clean up resources even though AWS resources are not used (bug fix) #4541

Clean up resources even though AWS resources are not used (bug fix) #4541

youngbupark commented Nov 9, 2022

youngbupark Nov 9, 2022

vinayada1 Nov 9, 2022

youngbupark Nov 9, 2022

ytimocin Nov 10, 2022

youngbupark commented Nov 9, 2022

youngbupark commented Nov 10, 2022

Clean up resources even though AWS resources are not used (bug fix) #4541

Clean up resources even though AWS resources are not used (bug fix) #4541

Conversation

youngbupark commented Nov 9, 2022

Description

Issue reference

Checklist

youngbupark Nov 9, 2022

Choose a reason for hiding this comment

vinayada1 Nov 9, 2022

Choose a reason for hiding this comment

youngbupark Nov 9, 2022

Choose a reason for hiding this comment

ytimocin Nov 10, 2022

Choose a reason for hiding this comment

youngbupark commented Nov 9, 2022

youngbupark commented Nov 10, 2022