-
-
Notifications
You must be signed in to change notification settings - Fork 259
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Duplicate maven repositories when importing bazel_deps that use maven.install #916
Comments
All the repositories with the same name are resolved together. This allows rulesets to contribute to the user’s jars easily and efficiently. This also implies that the default “maven” name should only be used for this purpose. TBH, relying on the “maven” namespace being as you declared it in your workspace file for anything than the top-level project was always a risky thing to do. Bzlmod has made that more obvious. Inter-ruleset dependencies have to be handled this way because constants cannot be loaded in a MODULE.bazel file. If a rulesets wants to have dependencies only it uses (for example for custom tooling) then it’s recommended that they declare a custom namespace for themselves and depend upon that. If you’d like to see an example, then “contrib_rules_jvm” does this. On 21 Jul 2023, at 15:08, Yun Peng ***@***.***> wrote:
@shs96c Are maven.install with the same name resolved together or separately? How do you recommend to address the warning message?
/cc @Wyverald
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
I just ran into this too, and I think it's specifically a problem with the protobuf module. Other modules take care to instantiate their maven artifacts with different names to avoid polluting the default "maven" name:
Is there a good reason why protobuf should make its internal java dependencies visible in this way? If not, we should ask the protobuf module maintainers to change it. |
Not really, it was added as a patch file at https://github.com/bazelbuild/bazel-central-registry/blob/918ea9ff0a8c5d3f8719c84c2109c5a25631fc47/modules/protobuf/21.7/patches/add_module_dot_bazel.patch#L37, we can make a BCR patch version for protobuf to fix this issue. |
I also ran into this issue and I agree with @meteorcloudy. Let’s go and make the patch to protobuf in BCR 💪 |
I just ran into this too trying to build https://github.com/entur/schema2proto in bazel. Thanks for working on this! Update: No longer urgent for us since we ended up disabling bzlmod. |
Resolves these warnings: ```txt DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14: The maven repository 'maven' is used in two different bazel modules, originally in 'com_engflow_bazel_invocation_analyzer' and now in 'protobuf' DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14: The maven repository 'maven' is used in two different bazel modules, originally in 'com_engflow_bazel_invocation_analyzer' and now in 'protobuf' DEBUG: .../external/rules_jvm_external~/coursier.bzl:593:18: Found duplicate artifact versions com.google.code.gson:gson has multiple versions 2.11.0, 2.8.9 com.google.guava:guava has multiple versions 33.2.1-jre, 31.1-jre com.google.truth:truth has multiple versions 1.4.3, 1.1.2 org.mockito:mockito-core has multiple versions 5.12.0, 4.3.1 Please remove duplicate artifacts from the artifact list so you do not get unexpected artifact versions ``` See also: - Duplicate maven repositories when importing bazel_deps that use maven.install bazel-contrib/rules_jvm_external#916 - Using maven as the repo name causes duplicate warnings when using bzlmod protocolbuffers/protobuf#16839 - MODULE.bazel doesn't define @maven repository protocolbuffers/protobuf#17176 - Stop including extra artifacts on maven.install() bazel-contrib/rules_jvm_external#1168 - Use a custom name for the maven repository (maven_protobuf) protocolbuffers/protobuf#17190
Resolves these warnings: ```txt DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14: The maven repository 'maven' is used in two different bazel modules, originally in 'com_engflow_bazel_invocation_analyzer' and now in 'protobuf' DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14: The maven repository 'maven' is used in two different bazel modules, originally in 'com_engflow_bazel_invocation_analyzer' and now in 'protobuf' DEBUG: .../external/rules_jvm_external~/coursier.bzl:593:18: Found duplicate artifact versions com.google.code.gson:gson has multiple versions 2.11.0, 2.8.9 com.google.guava:guava has multiple versions 33.2.1-jre, 31.1-jre com.google.truth:truth has multiple versions 1.4.3, 1.1.2 org.mockito:mockito-core has multiple versions 5.12.0, 4.3.1 Please remove duplicate artifacts from the artifact list so you do not get unexpected artifact versions ``` See also: - Duplicate maven repositories when importing bazel_deps that use maven.install bazel-contrib/rules_jvm_external#916 - Using maven as the repo name causes duplicate warnings when using bzlmod protocolbuffers/protobuf#16839 - MODULE.bazel doesn't define @maven repository protocolbuffers/protobuf#17176 - Stop including extra artifacts on maven.install() bazel-contrib/rules_jvm_external#1168 - Use a custom name for the maven repository (maven_protobuf) protocolbuffers/protobuf#17190 Signed-off-by: Mike Bland <[email protected]>
Updates as much as possible except rules_jvm_external. The latest rules_jvm_external v6.1 breaks this project, both under the previous Bazel version (7.0.2) and the new one (7.2.1). I've filed bazel-contrib/rules_jvm_external#1189, which uses this repo and this PR as an example. See the last section below for details. --- Bazel update: - Bazel v7.2.1 https://github.com/bazelbuild/bazel/releases/tag/7.2.1 Bazel module updates: - bazel-skylib v1.7.1 https://github.com/bazelbuild/bazel-skylib/releases/tag/1.7.1 - platforms v0.0.10 https://github.com/bazelbuild/platforms/releases/tag/0.0.10 - rules_proto v6.0.2 https://github.com/bazelbuild/rules_proto/releases/tag/6.0.2 JAR updates: - com.google.code.gson:gson:2.11.0 - com.google.guava:guava:33.2.1-jre - commons-cli:commons-cli:1.8.0 Test JAR updates: - com.google.googlejavaformat:google-java-format:1.22.0 - com.google.truth:truth:1.4.3 - com.google.truth.extensions:truth-java8-extension:1.4.3 - org.mockito:mockito-core:5.12.0 --- Added explicit module spec and repinned the maven deps for: - protobuf v27.2 https://github.com/protocolbuffers/protobuf/releases/tag/v27.2 Adding protobuf explicitly resolves these warnings: ```txt DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14: The maven repository 'maven' is used in two different bazel modules, originally in 'com_engflow_bazel_invocation_analyzer' and now in 'protobuf' DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14: The maven repository 'maven' is used in two different bazel modules, originally in 'com_engflow_bazel_invocation_analyzer' and now in 'protobuf' DEBUG: .../external/rules_jvm_external~/coursier.bzl:593:18: Found duplicate artifact versions com.google.code.gson:gson has multiple versions 2.11.0, 2.8.9 com.google.guava:guava has multiple versions 33.2.1-jre, 31.1-jre com.google.truth:truth has multiple versions 1.4.3, 1.1.2 org.mockito:mockito-core has multiple versions 5.12.0, 4.3.1 Please remove duplicate artifacts from the artifact list so you do not get unexpected artifact versions ``` See also: - Duplicate maven repositories when importing bazel_deps that use maven.install bazel-contrib/rules_jvm_external#916 - Using maven as the repo name causes duplicate warnings when using bzlmod protocolbuffers/protobuf#16839 - MODULE.bazel doesn't define @maven repository protocolbuffers/protobuf#17176 - Stop including extra artifacts on maven.install() bazel-contrib/rules_jvm_external#1168 - Use a custom name for the maven repository (maven_protobuf) protocolbuffers/protobuf#17190 --- rules_jvm_external v6.1 somehow creates duplicate `jvm_import` rules for binary and source jars, instead of using the `srcjar` attribute: ```txt ERROR: Traceback (most recent call last): File ".../external/rules_jvm_external~~maven~maven/BUILD", line 34, column 11, in <toplevel> jvm_import( Error in jvm_import: jvm_import rule 'com_google_auto_value_auto_value_annotations' in package '' conflicts with existing jvm_import rule, defined at .../external/rules_jvm_external~~maven~maven/BUILD:9:11 ``` The content of rules_jvm_external~~maven~maven/BUILD at lines 9 and 34: ```bzl jvm_import( name = "com_google_auto_value_auto_value_annotations", jars = ["com/google/auto/value/auto-value-annotations/1.10.1/auto-value-annotations-1.10.1.jar"], ``` ```bzl jvm_import( name = "com_google_auto_value_auto_value_annotations", jars = ["com/google/auto/value/auto-value-annotations/1.10.1/auto-value-annotations-1.10.1-sources.jar"], ``` This pattern repeats for all the JAR targets. The BUILD contents from v5.3, which builds successfully both before and after applying the PR changes: ```bzl jvm_import( name = "com_google_auto_value_auto_value_annotations", jars = ["com/google/auto/value/auto-value-annotations/1.10.1/auto-value-annotations-1.10.1.jar"], srcjar = "com/google/auto/value/auto-value-annotations/1.10.1/auto-value-annotations-1.10.1-sources.jar", ``` --------- Signed-off-by: Mike Bland <[email protected]>
As someone who maintains a non-root Bazel module, I remain confused about how to correctly use the maven module extension. Based on the discussion above, I believe I should not use the default name in But I'm worried about the comment above by @shs96c :
This makes it sound like a unique name for It would be great to update the documentation to explain this kind of nuance. Currently it just recommends |
@tpudlik, there are a couple of things here that appear to be causing confusion. Our docs are primarily written for owners of root modules, but I can add extra details to help explain how best to use The important thing to bear in mind is that each of the tag classes in The With that out the way, the idea is that the |
The default name for the rules_jvm_external.maven rule is "maven". When not set, it defaults to "maven". For root modules also using rules_jvm_external, the name clash causes a warning: ``` DEBUG: $TMP/external/rules_jvm_external~/private/extensions/maven.bzl:154:14: The maven repository 'maven' is used in two different bazel modules, originally in '<my_workspace>' and now in 'protobuf' ``` Summarizing @shs96c in [1]: > The common maven repo name allows rulesets to contribute to the user's JARs. > However, this implies that maven is for the end user, not for transitive > dependencies. If a ruleset needs private dependencies, it should use a custom > namespace rather than the maven namespace. Since protobuf is not contributing to user's JARs, we'll use a custom namespace. There's precedent for using a custom namespace for library modules: - rules_jvm_external uses `rules_jvm_external_deps` instead of `maven`. - rules_kotlin uses `kotlin_rules_maven` instead of `maven`. [1]: bazel-contrib/rules_jvm_external#916 (comment) Fixes protocolbuffers#16839.
The default name for the rules_jvm_external.maven rule is "maven". When not set, it defaults to "maven". For root modules also using rules_jvm_external, the name clash causes a warning: ``` DEBUG: $TMP/external/rules_jvm_external~/private/extensions/maven.bzl:154:14: The maven repository 'maven' is used in two different bazel modules, originally in '<my_workspace>' and now in 'protobuf' ``` Summarizing @shs96c in [1]: > The common maven repo name allows rulesets to contribute to the user's JARs. > However, this implies that maven is for the end user, not for transitive > dependencies. If a ruleset needs private dependencies, it should use a custom > namespace rather than the maven namespace. Since protobuf is not contributing to user's JARs, we'll use a custom namespace. There's precedent for using a custom namespace for library modules: - rules_jvm_external uses `rules_jvm_external_deps` instead of `maven`. - rules_kotlin uses `kotlin_rules_maven` instead of `maven`. [1]: bazel-contrib/rules_jvm_external#916 (comment) Fixes #16839. Closes #18641 COPYBARA_INTEGRATE_REVIEW=#18641 from jschaf:joe/protobuf-maven bd2c62f PiperOrigin-RevId: 684625084
It seems the problem here just usability? The "Found duplicate artifact versions" warning has been happening since the creation of This comment thread is actually creating problems at runtime, as there must be only a single version of each dependency at runtime. See If the user defines their own java_proto_library toolchain (and we ignore gRPC for the moment), then some of this can function. But I don't see a single example suggesting to do that. |
I spent several hours investigating a mysterious Bazel failure, only to discover that it was due to this behavior. For example, the official Bazel worker API library is also affected—it uses It would be beneficial to reconsider this initial design choice (say, made (thanks a lot for the debug logging—it helped point me in the right direction for investigation.) |
I'll happily merge a PR to update the documentation if you get there before I have a chance to write it. |
…-contrib/rules_jvm_external#916) GitOrigin-RevId: 1378657735b438684e934c8aca721d43980869c8
The default name for the rules_jvm_external.maven rule is "maven". When not set, it defaults to "maven". For root modules also using rules_jvm_external, the name clash causes a warning: ``` DEBUG: $TMP/external/rules_jvm_external~/private/extensions/maven.bzl:154:14: The maven repository 'maven' is used in two different bazel modules, originally in '<my_workspace>' and now in 'protobuf' ``` Summarizing @shs96c in [1]: > The common maven repo name allows rulesets to contribute to the user's JARs. > However, this implies that maven is for the end user, not for transitive > dependencies. If a ruleset needs private dependencies, it should use a custom > namespace rather than the maven namespace. Since protobuf is not contributing to user's JARs, we'll use a custom namespace. There's precedent for using a custom namespace for library modules: - rules_jvm_external uses `rules_jvm_external_deps` instead of `maven`. - rules_kotlin uses `kotlin_rules_maven` instead of `maven`. [1]: bazel-contrib/rules_jvm_external#916 (comment) Fixes #16839. Closes #18641 COPYBARA_INTEGRATE_REVIEW=#18641 from jschaf:joe/protobuf-maven bd2c62f PiperOrigin-RevId: 684625084
The default name for the rules_jvm_external.maven rule is "maven". When not set, it defaults to "maven". For root modules also using rules_jvm_external, the name clash causes a warning: ``` DEBUG: $TMP/external/rules_jvm_external~/private/extensions/maven.bzl:154:14: The maven repository 'maven' is used in two different bazel modules, originally in '<my_workspace>' and now in 'protobuf' ``` Summarizing @shs96c in [1]: > The common maven repo name allows rulesets to contribute to the user's JARs. > However, this implies that maven is for the end user, not for transitive > dependencies. If a ruleset needs private dependencies, it should use a custom > namespace rather than the maven namespace. Since protobuf is not contributing to user's JARs, we'll use a custom namespace. There's precedent for using a custom namespace for library modules: - rules_jvm_external uses `rules_jvm_external_deps` instead of `maven`. - rules_kotlin uses `kotlin_rules_maven` instead of `maven`. [1]: bazel-contrib/rules_jvm_external#916 (comment) Fixes #16839. Closes #18641 COPYBARA_INTEGRATE_REVIEW=#18641 from jschaf:joe/protobuf-maven bd2c62f PiperOrigin-RevId: 684625084 Co-authored-by: Joe Schafer <[email protected]>
I've proposed a docs update in #1303 to at least make this a bit clearer and avoid some future confusion for others. |
When using
maven.install
via bzlmod in the root module, anybazel_dep
which also usesmaven.install
(with the default repository name) can create duplicate artifacts/conflicts.For instance, when including a bazel_dep for protobuf 21.7 , the following message is logged:
The maven repository 'maven' is used in two different bazel modules, originally in '' and now in 'protobuf'
This happens because protobuf does a
maven.install
usingmaven
as the repository name pinning various common dependenciesThis is maybe benign in and of itself, but it can create artifact conflicts. For example, pinning different versions of
mockito
/guava
/etc to the root module, the following messages are logged:It's possible to work around this by using a separate
name
formaven.install
, but it is quite annoying (since you need to setrepository_name
everywhere) and likewise incredibly difficult to understand where these duplicates were introduced. In my case, I wasn't even directly importingprotobuf
in my root module, but transitively through another bazel_dep, so this was incredibly confusing.Is there a better way to understand and resolve these conflicts?
The text was updated successfully, but these errors were encountered: