[WIP] Allow lazy AWSClient initialization.

[dariko]

This PR tries to solve the problem reported in #20127

AWSClient is currently statically initialized in Config.Client(),
called by providerConfigure.
This makes using the provider in a configuration-driven template
dependant on having a valid set of aws credentials, even if the
configuration provides for no aws resources to be managed.

This PR, while keeping the same approach as a default, allows the
user to configure this initialization to be delayed until (and if)
necessary.

In more detail, this commit/PR:

• adds the provider configuration to AWSClient.providerConfig
• moves the AWSClient initialization process to a new
  AWSClient.Init() error function.
• adds a new provider parameter lazy_init (default false)
• calls AWSClient.Init from Config.Client() only if lazy_init
  if true
• adds a series of
  ClientInit{CrudBase,CrudContext,Exists,MigrateState,CustomizeDiff,StateUpgrade}Func
  wrapper functions
• these wrapper functions are used in each resource definition to integrate
  the current Provider arg function functionalities with a call to
  AWSClient.Init().

I have no experience with go nor with terraform plugin development so I "might" have made some "unlucky" choises about
how to approach this problem; I opened this PR to see if this approach might be acceptable AND to gather some insight about how this problem could be approached.

I would have liked to be able to have a MUCH smaller impact in the codebase, but the need to handle the possible AWSClient.Init() errors (having no exception management in go) left me unable to find a more delicate solution.

Community Note

• Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Acceptance tests

I might be willing to run the whole acceptance test suite but I'd need to get at least a ballpark extimate of how much executing them all could cost: I don't think running them blindly would be wise.

As of now I have just run some random (small) subset of the suite, just to verify there were no error globally breaking functionalities; I just got returned errors which seems to be not linked to my changes but to the environment(my amazon account/target region) or timeouts.

Some of these tests took a "not-short" time to complete, making me weary about the cost of running the whole suite.

$ TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run="^($(grep -r "^func TestAcc" aws|sed -r 's/.*(TestAcc[^\(]*).*/\1/'|shuf -n 3|paste -sd \|))$"
=== RUN   TestAccAWSEc2InstanceTypeOfferingsDataSource_LocationType
=== PAUSE TestAccAWSEc2InstanceTypeOfferingsDataSource_LocationType
=== RUN   TestAccAwsDmsEndpoint_MongoDb
=== PAUSE TestAccAwsDmsEndpoint_MongoDb
=== RUN   TestAccAWSSagemakerNotebookInstanceLifecycleConfiguration_basic
=== PAUSE TestAccAWSSagemakerNotebookInstanceLifecycleConfiguration_basic
=== CONT  TestAccAWSEc2InstanceTypeOfferingsDataSource_LocationType
=== CONT  TestAccAWSSagemakerNotebookInstanceLifecycleConfiguration_basic
=== CONT  TestAccAwsDmsEndpoint_MongoDb
    resource_aws_dms_endpoint_test.go:420: Step 1/2 error: Error running pre-apply refresh: exit status 1
        
        Error: No alias with name "alias/aws/dms" found in this region.
        
          with data.aws_kms_alias.dms,
          on terraform_plugin_test.tf line 2, in data "aws_kms_alias" "dms":
           2: data "aws_kms_alias" "dms" {
        
--- FAIL: TestAccAwsDmsEndpoint_MongoDb (4.70s)
--- PASS: TestAccAWSEc2InstanceTypeOfferingsDataSource_LocationType (20.83s)
--- PASS: TestAccAWSSagemakerNotebookInstanceLifecycleConfiguration_basic (21.58s)
FAIL
FAIL	github.com/terraform-providers/terraform-provider-aws/aws	21.648s
FAIL

some more output here: https://gist.github.com/dariko/5d0a5467582735f5473321b4025f8ebb

I should also write tests for the new provider lazy_init parameter: I have not begun yet, I just run some local basic tests.

Closes #20127

[github-actions]

Welcome @dariko 👋

It looks like this is your first Pull Request submission to the Terraform AWS Provider! If you haven’t already done so please make sure you have checked out our CONTRIBUTING guide and FAQ to make sure your contribution is adhering to best practice and has all the necessary elements in place for a successful approval.

Also take a look at our FAQ which details how we prioritize Pull Requests for inclusion.

Thanks again, and welcome to the community! 😃

[dariko]

I suppose the labeler job is failing because it's trying to generate to many labels?

from https://github.com/hashicorp/terraform-provider-aws/pull/20388/checks?check_run_id=3217790550 :

HttpError: Validation Failed. Issues cannot have more than 100 labels: [CUT]

[zhelding]

Pull request #21306 has significantly refactored the AWS Provider codebase. As a result, most PRs opened prior to the refactor now have merge conflicts that must be resolved before proceeding.

Specifically, PR #21306 relocated the code for all AWS resources and data sources from a single aws directory to a large number of separate directories in internal/service, each corresponding to a particular AWS service. This separation of code has also allowed for us to simplify the names of underlying functions -- while still avoiding namespace collisions.

We recognize that many pull requests have been open for some time without yet being addressed by our maintainers. Therefore, we want to make it clear that resolving these conflicts in no way affects the prioritization of a particular pull request. Once a pull request has been prioritized for review, the necessary changes will be made by a maintainer -- either directly or in collaboration with the pull request author.

For a more complete description of this refactor, including examples of how old filepaths and function names correspond to their new counterparts: please refer to issue #20000.

For a quick guide on how to amend your pull request to resolve the merge conflicts resulting from this refactor and bring it in line with our new code patterns: please refer to our Service Package Refactor Pull Request Guide.

[ewbankkit]

@dariko Thanks for the ideas implemented here 🎉 👏.
The addition of the AWS SDK session to our provider instance state via #23560 has started an internal discussion around lazy-loading the API clients.
I'm going to close this PR in the interest of keeping the repo tidy.

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.