Support for Bzlmod in a monorepo

[chandraaditya]

What version of gazelle are you using?

v0.33.0

What version of rules_go are you using?

v0.41.0

What version of Bazel are you using?

7.0.0-pre.20231011.2

Does this issue reproduce with the latest releases of all the above?

Yes

What operating system and processor architecture are you using?

macos 14.1 Beta (23B5056e), x86

What did you do?

I'm basically trying to move to Bzlmod from WORKSPACE, and I'm confused about how to use gazelle with it in a monorepo.

The docs at https://github.com/bazelbuild/rules_go/blob/master/docs/go/core/bzlmod.md say that gazelle will use go_deps, and that I don't need to run gazelle update-repos anymore but the problem is that I can only define one go_deps.from_file in my MODULE.bazel file, but I have multiple go projects in my monorepo with different dependencies for each.

With the WORKSPACE file it was at least possible to run update-repos for each go.mod file, but I do know that that's not the recommended way of doing it, and that gazelle has a tool to combine all go.mod files into a single go.mod file in the repo's root.

But is there a way to still use multiple go.mod files?

[fmeum]

@tyler-french What do you think, are there any footguns in allowing multiple go.mod files per module? Proper go.work support would of course be more involved.

[tyler-french]

@tyler-french What do you think, are there any footguns in allowing multiple go.mod files per module? Proper go.work support would of course be more involved.

I want to better understand the reason behind having the multiple go.mod files.

Are the different projects built with different resolved versions of the different modules? In this case, supporting multiple go.mod files would not have the desired effect, as they would deterministically resolve to a single set of dependencies.

Are there any exclude or replace directives in these go.mod files? If so, allowing multiple go.mod files is actually a bit dangerous, because it might make the resolved versions from MVS non-deterministic (or dependent on the ordering of tags)

I'm wondering if it might be more practical to have one MODULE.bazel per go.mod file, each as a separate "module", and then you could have a root module /MODULE.bazel that imports the others (this should work, right @fmeum?), does resolution, and applies overrides. This would also be more "correct", in terms of following the duality of a go.mod module and a Bazel module.

[chandraaditya]

I'm wondering if it might be more practical to have one MODULE.bazel per go.mod file, each as a separate "module", and then you could have a root module /MODULE.bazel that imports the others (this should work, right @fmeum?), does resolution, and applies overrides. This would also be more "correct", in terms of following the duality of a go.mod module and a Bazel module.

Actually I think yes, this would be a better question to ask as there's no good documentation on the best practices with gazelle and Bzlmod anywhere(specifically when it comes to monorepos).

[opicaud]

Hello,

I am currently using a monorepo with some multiple go modules inside, managed via bzlmod.
I can share my approach, not optimal but at least it seems working..
see here: https://github.com/opicaud/monorepo
FYI I don't use any exclude or replace directives, but i must duplicate a lot of configuration, specifically extensions go_deps and directive bazel_deps
I have also those kind of warnings, which are "normal" root module are not using any kind of go modules..

WARNING: /__w/monorepo/monorepo/MODULE.bazel:112:24: The module extension go_deps defined in @bazel_gazelle//:extensions.bzl reported incorrect imports of repositories via use_repo():

Imported, but reported as indirect dependencies by the extension:
    com_github_beorn7_floats, com_github_cucumber_godog, com_github_davecgh_go_spew, com_github_google_uuid, com_github_grpc_ecosystem_grpc_gateway_v2, com_github_hashicorp_logutils, com_github_pact_foundation_pact_go_v2, com_github_smarty_assertions, com_github_spf13_pflag, com_github_spf13_viper, com_github_stretchr_testify, com_github_urfave_cli_v3, org_golang_google_grpc, org_golang_x_exp, org_golang_x_mod

 ** You can use the following buildozer command to fix these issues:

buildozer 'use_repo_remove @bazel_gazelle//:extensions.bzl go_deps com_github_beorn7_floats com_github_cucumber_godog com_github_davecgh_go_spew com_github_google_uuid com_github_grpc_ecosystem_grpc_gateway_v2 com_github_hashicorp_logutils com_github_pact_foundation_pact_go_v2 com_github_smarty_assertions com_github_spf13_pflag com_github_spf13_viper com_github_stretchr_testify com_github_urfave_cli_v3 org_golang_google_grpc org_golang_x_exp org_golang_x_mod' //MODULE.bazel:all

Hope it can help

[aran]

I'm also trying to understand how rules_go is supposed to work with bzlmod + multiple interconnected go modules and I ended up here.

(as well as relatedly #1715)

Concretely -

• I am looking for a clear answer about whether I "should" have one MODULE.bazel at my repository root alongside WORKSPACE, or many MODULE.bazel files, one per go.mod, and perhaps need to introduce tooling to interconnect them. @fmeum says "one" here: Missing documentation and best practices for bzlmod #1715 (comment) and @tyler-french says "many" here: Support for Bzlmod in a monorepo #1658 (comment). Or maybe it is supposed to be only one MODULE.bazel + one go.mod for a whole monorepo?
• I'm trying to understand the blockers / plan for the unsupported feature documented here: https://github.com/bazelbuild/rules_go/blob/master/docs/go/core/bzlmod.md ("go.mod replace directives referencing local files")

Here is a setup that is not pleasant but it works (without bzlmod). I can't see how to port it to bzlmod.

• WORKSPACE with the following stanza:

# gazelle:repository_macro go_repositories.bzl%go_repositories
go_repositories()`

• Many go modules representing different modules of functionality. Each go module has its own subfolder, a go.mod file and a BUILD.bazel file. To add an external dep, run go get (to get transitive dependencies), then bazel run //:gazelle to move those to BUILD.bazel.
• Internally-developed go modules use replace statements to point to each other via their paths in the monorepo, to allow go get to work.
• Use the github.com/airyhq/airy/tools/update-deps tool to sync all go.mod files to a consistent, repo-wide version.
• Some gazelle:prefix and gazelle:proto_strip_import_prefix fixes to manage the names/go module paths of shared .proto files that aren't in the same part of the tree as most of the go code.

[fmeum]

A true monorepo would only have a single MODULE.bazel and a single go.mod file. If you would like to still have multiple go.mod files, you might be interested in go.work support (see #1731).

Replace directives mentioning local paths are waiting for Bazel features that would allow us to overlay a local repo with generated build files. It's also actively being worked on.

[aran]

My current state is multiple go.mod files with no go.work. I'm experimenting with cutting down to one go.mod file for the whole repo based on your suggestion. Are you aware of any published examples of multi-language monorepos that follow this design? It would help me a lot, and I'm sure others. I think it would be very helpful to have some expert documentation or example about the recommended workflows.

To satisfy both go mod and gazelle, it looks like I need to

1. Colocate all go code into one go code directory that has the single go.mod.
2. Create a shadow directory structure in the go code directory, matching the directory structure of the shared repo-wide protobuf directory, and put a dummy .go file in each.

Does that sound right? The protobuf shadow directory makes me suspicious I'm still missing some best practices.

[stefanpenner]

This PR may be relevant: #1731