connect: Fix an issue with updating CA config in a secondary datacenter

[kyhavlov]

This PR fixes a case where updating the CA config in a secondary datacenter would cause an error when it should trigger the creation of a new intermediate certificate.

The test passes but I think this still needs some extra logic to prevent races between the RPC endpoint and secondaryCARootWatch/intermediateCertRenewalWatch. Currently secondaryCARootWatch will set things right on its next iteration if there's a race where it gets overwritten by a call to the RPC endpoint, but since it spends most of its time waiting on a blocking query that could be up to 5-10 minutes with the wrong intermediate/root, which is a pretty long time to wait for things to converge.

Fixes #7009.

[kyhavlov]

Discussed fixing the racey parts of the secondary logic with @rboyer and decided a good solution is to pull some of the intermediate update code out into a separate struct to manage the different states (ready/signing/reconfig) in a safer/more understandable way, so I'm going to update the PR with that next.

[kyhavlov]

I've updated the PR with the refactored CA logic - it still needs some test updates/fixes and a writeup of how the state machine logic avoids race conditions in specific situations.

[rboyer]

You could instead use the current time and the time portion of (*testing.T).Deadline() instead to compute remaining test time instead of using a made up "not forever" number here and several other places

[rboyer]

LGTM assuming CI passes.

Further tests can come in a followup.

[hashicorp-ci]

🍒 If backport labels were added before merging, cherry-picking will start automatically.

To retroactively trigger a backport after merging, add backport labels and re-run https://circleci.com/gh/hashicorp/consul/290464.

[hashicorp-ci]

🍒✅ Cherry pick of commit c4eff42 onto release/1.9.x succeeded!

[hashicorp-ci]

🍒❌ Cherry pick of commit c4eff42 onto release/1.8.x failed! Build Log