terraform/module: Introduce new job to parse provider versions from lock file

[radeksimko]

Background

Currently we treat provider lock file (and installed provider binaries) as opaque and only use them in the watcher to know whether they have changed, to re-run terraform version and terraform providers schema -json to refresh provider versions and obtain new schemas.

The scheduler already de-duplicates jobs (including the two mentioned above), such that we never run the same job for the same folder more than once, if it's already in the queue. This is a good barrier, but we anticipate the following features to put more pressure on the scheduler and create more situations of duplicate work (same job for the same folder) scheduled more sparsely, which the scheduler itself has no chance of catching.

• Recognize inputs and outputs of uninitialized local modules #598
• Index uninitalized modules #724

both of which increase the chance of us scheduling the same jobs for the same directory repeatedly.

The root problem is that we don't know if we need the schema before we run these commands and so we potentially waste CPU and IO on both jobs unless they end up deduplicated within the same job queue.

There are different ways that Terraform persists the installed provider versions between TF versions:

terraform-ls/internal/terraform/datadir/paths.go

Lines 12 to 19 in 9fcc1c9

	var pluginLockFilePathElements = [][]string{
	// Terraform >= 0.14
	{".terraform.lock.hcl"},
	// Terraform >= v0.13
	{DataDirName, "plugins", "selections.json"},
	// Terraform >= v0.12
	{DataDirName, "plugins", runtime.GOOS + "_" + runtime.GOARCH, "lock.json"},
	}

While relying on more of Terraform's internal state is not great, the cost/risk seems well out-weighted by our ability to prevent wasteful CLI commands if we know that we already have the same provider schema for the same version already available in memdb cache.

Proposal

• datadir: Introduce a function to parse provider versions from the data dir (.terraform)
  • we can attempt to parse different .terraform structures from the latest TF version to oldest and return the first successful parse. This will allow us to keep this logic entirely decoupled from parsing and understanding the core version or constraint
• terraform/module: Introduce a new job ParseProviderVersions to https://github.com/hashicorp/terraform-ls/blob/66c8742/internal/terraform/module/module_ops.go
• Schedule ParseProviderVersions from the walker and workspace/didChangeWatchedFiles and make ObtainSchema & GetTerraformVersion dependent on this new job, such that we don't run either of them if we already have the same provider schema version cached in memdb
• Ensure that module.providers command returns this parsed data https://github.com/hashicorp/terraform-ls/blob/main/internal/langserver/handlers/command/module_providers.go

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.