Parsing of the WORKSPACE file should be consistent with other parsing contexts

[shs96c]

Description of the problem / feature request:

The WORKSPACE file is parsed with a different context to other files. Notably, it's possible to mingle load and other function calls together. The same is not true of files loaded into the WORKSPACE. This makes modularising the WORKSPACE a taxing problem, littered with boilerplate as we try and avoid this inconsistency.

Feature requests: what underlying problem are you trying to solve with this feature?

As an example, Bazel Federation requires someone to load the repositories, then call a function, then load another file, then call another function:

load("@bazel_federation//:repositories.bzl", "rules_java")
rules_java()
load("@bazel_federation//setup:rules_java.bzl", "rules_java_setup")
rules_java_setup()

A similar pattern needs to be followed for people attempting to modularise any WORKSPACE file.

This approach is repetitious and filled with boilerplate, and should really be:

load("@bazel_federation//setup:rules_java.bzl", "rules_java_setup")
rules_java_setup()

Where rules_java_setup first of all loads the respositories.bzl and calls rules_java().

As another example, using maybe in a file that is loaded (but outside of a function) is unsupported, so conditional loading of repositories and subsequent calls to functions defined within those repos is unsupported.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Use Bazel Federation and boggle at the boilerplate.

What's the output of bazel info release?

0.29.0

Have you found anything relevant by searching the web?

Issues that relate to this:

#1550 If load worked within a function, this problem could be mitigated.

#1943 Recursive WORKSPACE loading would allow a clunky and inelegant solution to be implemented.

Any other information, logs, or outputs that you want to share?

I realise that someone is going to suggest the rationale for closing #1550 is still valid. On a technical level, I can buy that for now, but I think that the usability impact of this decision on Bazel users is growing, and as adoption spreads will continue to grow worse. Providing a solution to modularising WORKSPACE files in a way that minimises boilerplate is going to be crucial to enabling a pleasant developer experience.

[fweikert]

Related design doc: https://docs.google.com/document/d/1QQTu-v5yfZiE_heu2wnc65igOO1dNXWCJxUXBNhIxUo/edit?usp=sharing

[fweikert]

@keith : That's the issue I mentioned to you at BazelCon.

[keith]

Thanks. We're interested in this because we depend on Envoy with Bazel through other repos and we end up having to make sure we load the rules that are dependencies of Envoy from their workspace which requires pulling out WORKSPACE dependencies into bzl file and making sure to call those loads too.

[laurentlb]

It's a known pain-point. Unfortunately, this is non-trivial to fix and might require a lot of design work to decide where we want to go with workspace files.

[fweikert]

Related: https://github.com/bazelbuild/proposals/blob/master/designs/2019-01-14-delayed-load.md

[philwo]

I'm marking this as P2, because with the new external deps design that @meteorcloudy and @Wyverald are working on, I assume that this will become irrelevant.

(@keith Please let me know if I'm judging this wrongly.)

[keith]

As long as recursive workspaces are supported for that, I think it should be fine!

[philwo]

@keith I don't think we'll add recursive workspaces or any other new features to the WORKSPACE file parsing. Instead, we're aiming for the WORKSPACE file to be a flat namespace and prevent any transitive loading in the future (like the current deps.bzl pattern). Ideally, we can then also remove all current HTTP, Git, etc. repository features from Bazel and only leave the "local repository".

Instead, the complexity of recursive / transitive dependency resolution, clean modularization and downloading modules from various sources will be solved via the new external tool bzlmod, which will generate a flat WORKSPACE file and handle downloads.

The design for that is here: https://groups.google.com/g/bazel-dev/c/6TEyoGb9ys0 - please comment on the doc if you think this doesn't address your requirements! 😊

[shs96c]

Tagging at @hicksjoseph after discussing with him away from GitHub.

[hicksjoseph]

Hi Simon, I remember that we discussed this issue, but now I forget the next steps or desired outcome. Can you recap your thoughts here? Thanks, Joe

[shs96c]

The key feature was the ability to interleave load statements and other statements from files loaded from the WORKSPACE, allowing the .bzl files to be used to prevent the WORKSPACE file from becoming overwhelming and unmaintainable.

[Wyverald]

Thanks for the context, @shs96c. I realize we're years late here, but the new external dependency system "bzlmod" should solve the pains you mentioned without requiring interleaving load in .bzl files. (In fact, it's such a technical burden that we want to get rid of it altogether.) With the new system, you don't need all that boilerplate to set up a dependency; simply adding the dependency will trigger all its setup logic.

With that in mind, I'm going to close this issue.

[hicksjoseph]

Hi Xudong, Do you have a public link to information on bzlmod that also has a summary and background section? If not, perhaps we can make one together, post it, and link it to this issue. Thanks, Joe

[Wyverald]

The most up-to-date information remains the original design doc Philipp linked to in #9387 (comment). It needs a bit of refreshing but the general idea is still intact.

[hicksjoseph]

Thanks much Xudong. I will follow up with Simon directly if he has any questions.