Go: should _go_external_package dependency inference move to target generation?

[Eric-Arellano]

Target generation opens up a new possibility: dynamically inferring deps at target generation time, rather than dependency resolution time. Would it make sense to do that?

I think this would imply that _go_external_package targets would only depend on other targets defined in the same go.mod, which maybe we want?

Possible pros to explore:

• Can we run fewer processes for better performance?
• ./pants peek is more interesting because it includes values in the dependencies field, whereas you'd otherwise have to use ./pants dependencies

A con is that manually declared _go_external_package targets are much less practical. Although we might not care about this.

[Eric-Arellano]

If we remove the -find arg from here:

pants/src/python/pants/backend/go/util_rules/external_module.py

Lines 271 to 283 in 6d7f305

	json_result = await Get(
	ProcessResult,
	GoSdkProcess(
	input_digest=downloaded_module.digest,
	# "-find" skips determining dependencies and imports for each package.
	command=("list", "-find", "-mod=readonly", "-json", "./..."),
	env={"GOPROXY": "off"},
	description=(
	"Determine packages belonging to Go external module "
	f"{request.module_path}@{request.version}"
	),
	),
	)

Then we can get all the metadata we need in a single go list invocation. That's great because Go tooling is very fast, whereas Pants has some overhead for running more Processes like setting up the sandbox. I'll try implementing this.

[stuhood]

Target generation opens up a new possibility: dynamically inferring deps at target generation time, rather than dependency resolution time. Would it make sense to do that?

In general, I don't think so... Specs expansion is a really fundamental operation that would be good to keep fast. If we can defer things to when dependencies have actually been requested, that would be good I think.

On the other hand, it would probably be fine for the Go tooling to use a shared @rule to extract a shared metadata type containing all of the relevant info in the two codepaths. Effectively equivalent.

[Eric-Arellano]

#13123 attempted this, but #13128 was a much better factoring. So, the answer is: no, it does not make sense to move to target generation.