[FIX] Terraform provider version inconsistency within stages

[smokestacklightnin]

Reference Issues or PRs

Fixes #2614

What does this implement/fix?

Put a x in the boxes that apply

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds a feature)
• Breaking change (fix or feature that would cause existing features not to work as expected)
• Documentation Update
• Code style update (formatting, renaming)
• Refactoring (no functional changes, no API changes)
• Build related changes
• Other (please describe):

Testing

• Did you test the pull request locally?
• Did you add new tests?

How to test this PR?

Any other comments?

The main change in this PR is that Terraform/OpenTofu versions and required providers are removed from the stage templates and instead injected via stages.tf_objects with version data from constants.py. stages.tf_objects.NebariOpentofuRequiredVersion and stages.tf_objects.NebariOpentofuRequiredProvider were added.

Now all subclasses of NebariTerraformStage present with a tf_objects method except terraform_state will call include the result of super().tf_objects in their return value:

class FooStage(NebariTerraformStage):
    def tf_objects(self) -> List[Dict]:
        return [
            *super().tf_objects(),
            ...,
        ]    

This way we can also inject NebariOpentofuRequiredVersion into NebariTerraformStage and have subclasses inherit it.

[smokestacklightnin]

I've executed the changes I intended to make, however, when running Nebari, submodules are not inheriting required providers from the parent module. In particular, this is happening in some of the following modules:

./kubernetes_ingress/template/modules/kubernetes/ingress/
./kubernetes_services/template/modules/kubernetes/cephfs-mount/
./kubernetes_services/template/modules/kubernetes/
./kubernetes_services/template/modules/kubernetes/nfs-mount/
./kubernetes_services/template/modules/kubernetes/forwardauth/
./kubernetes_services/template/modules/kubernetes/nfs-server/
./kubernetes_services/template/modules/kubernetes/services/jupyterhub-ssh/
./kubernetes_services/template/modules/kubernetes/services/jupyterhub/
./kubernetes_services/template/modules/kubernetes/services/keycloak-client/
./kubernetes_services/template/modules/kubernetes/services/
./kubernetes_services/template/modules/kubernetes/services/redis/
./kubernetes_services/template/modules/kubernetes/services/monitoring/
./kubernetes_services/template/modules/kubernetes/services/monitoring/loki/
./kubernetes_services/template/modules/kubernetes/services/postgresql/
./kubernetes_services/template/modules/kubernetes/services/rook-ceph/
./kubernetes_services/template/modules/kubernetes/services/dask-gateway/
./kubernetes_services/template/modules/kubernetes/services/minio/
./kubernetes_services/template/modules/kubernetes/services/argo-workflows/
./kubernetes_services/template/modules/kubernetes/services/conda-store/
./nebari_tf_extensions/template/modules/nebariextension/
./nebari_tf_extensions/template/modules/helm-extensions/
./kubernetes_initialize/template/modules/
./kubernetes_initialize/template/modules/initialization/
./kubernetes_initialize/template/modules/cluster-autoscaler/
./kubernetes_initialize/template/modules/traefik_crds/
./kubernetes_initialize/template/modules/extcr/
./kubernetes_initialize/template/modules/nvidia-installer/
./kubernetes_keycloak/template/modules/kubernetes/keycloak-helm/

Just to see if it would work, I tried adding the correct required providers to those locations. It worked, so this suggest that it might be a good option to be able to dynamically inject required providers from constants.py this would be generalizing what is currently done at the root module to submodules as well.

@viniciusdc Do you have any input?

CC: @marcelovilla

[smokestacklightnin]

My honest opinion is that it is easier to inject submodule required providers manually now and change to using inherited required providers from parent modules in the future if/when OpenTofu supports this, rather than waiting for another project to merge our changes. I also don't know Go, but am activetly trying to learn.

The code I have written would be easy to remove for the injection process in submodules if OpenTofu started supporting version requirement inheritance in the future. This is because it requires minimal additional machinery, just _tf_objects_required_providers.

We need this feature soon because other issues depend on updating versions (#2806, #2870, #2872).

Ideas? @marcelovilla @viniciusdc @dcmcand

[viniciusdc]

I will have a look, we need to make the issues with the double version we noticed before are not happening anymore. Thanks for all the fantastic work so far @smokestacklightnin

[smokestacklightnin]

I will have a look, we need to make the issues with the double version we noticed before are not happening anymore. Thanks for all the fantastic work so far @smokestacklightnin

Pinging @viniciusdc

[viniciusdc]

To be included in 2025.1.2 release, for ref. check the parent issue.

[viniciusdc]

I am reviewing this now