Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Duplicate maven repositories when importing bazel_deps that use maven.install #916

Open
aarontsharp opened this issue Jun 11, 2023 · 13 comments

Comments

@aarontsharp
Copy link

When using maven.install via bzlmod in the root module, any bazel_dep which also uses maven.install (with the default repository name) can create duplicate artifacts/conflicts.

For instance, when including a bazel_dep for protobuf 21.7 , the following message is logged:

The maven repository 'maven' is used in two different bazel modules, originally in '' and now in 'protobuf'

This happens because protobuf does a maven.install using maven as the repository name pinning various common dependencies

This is maybe benign in and of itself, but it can create artifact conflicts. For example, pinning different versions of mockito/guava/etc to the root module, the following messages are logged:

Found duplicate artifact versions
    com.google.guava:guava has multiple versions 32.0.0-jre, 31.1-jre
    org.mockito:mockito-core has multiple versions 5.3.1, 4.3.1

It's possible to work around this by using a separate name for maven.install, but it is quite annoying (since you need to set repository_name everywhere) and likewise incredibly difficult to understand where these duplicates were introduced. In my case, I wasn't even directly importing protobuf in my root module, but transitively through another bazel_dep, so this was incredibly confusing.

Is there a better way to understand and resolve these conflicts?

@meteorcloudy
Copy link
Contributor

@shs96c Are maven.install with the same name resolved together or separately? How do you recommend to address the warning message?

/cc @Wyverald

@shs96c
Copy link
Collaborator

shs96c commented Jul 21, 2023 via email

@Ubehebe
Copy link

Ubehebe commented Jan 4, 2024

I just ran into this too, and I think it's specifically a problem with the protobuf module. Other modules take care to instantiate their maven artifacts with different names to avoid polluting the default "maven" name:

Is there a good reason why protobuf should make its internal java dependencies visible in this way? If not, we should ask the protobuf module maintainers to change it.

@meteorcloudy
Copy link
Contributor

meteorcloudy commented Jan 5, 2024

Not really, it was added as a patch file at https://github.com/bazelbuild/bazel-central-registry/blob/918ea9ff0a8c5d3f8719c84c2109c5a25631fc47/modules/protobuf/21.7/patches/add_module_dot_bazel.patch#L37, we can make a BCR patch version for protobuf to fix this issue.

@luangong
Copy link

I also ran into this issue and I agree with @meteorcloudy. Let’s go and make the patch to protobuf in BCR 💪

@andrew-otiv
Copy link

andrew-otiv commented May 14, 2024

I just ran into this too trying to build https://github.com/entur/schema2proto in bazel. Thanks for working on this!

Update: No longer urgent for us since we ended up disabling bzlmod.

mbland added a commit to EngFlow/bazel_invocation_analyzer that referenced this issue Jul 2, 2024
Resolves these warnings:

```txt
DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/coursier.bzl:593:18:
    Found duplicate artifact versions
    com.google.code.gson:gson has multiple versions 2.11.0, 2.8.9
    com.google.guava:guava has multiple versions 33.2.1-jre, 31.1-jre
    com.google.truth:truth has multiple versions 1.4.3, 1.1.2
    org.mockito:mockito-core has multiple versions 5.12.0, 4.3.1
Please remove duplicate artifacts from the artifact list so you do not
get unexpected artifact versions
```

See also:

- Duplicate maven repositories when importing bazel_deps that use
  maven.install
  bazel-contrib/rules_jvm_external#916

- Using maven as the repo name causes duplicate warnings when using
  bzlmod
  protocolbuffers/protobuf#16839

- MODULE.bazel doesn't define @maven repository
  protocolbuffers/protobuf#17176

- Stop including extra artifacts on maven.install()
  bazel-contrib/rules_jvm_external#1168

- Use a custom name for the maven repository (maven_protobuf)
  protocolbuffers/protobuf#17190
mbland added a commit to EngFlow/bazel_invocation_analyzer that referenced this issue Jul 2, 2024
Resolves these warnings:

```txt
DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/coursier.bzl:593:18:
    Found duplicate artifact versions
    com.google.code.gson:gson has multiple versions 2.11.0, 2.8.9
    com.google.guava:guava has multiple versions 33.2.1-jre, 31.1-jre
    com.google.truth:truth has multiple versions 1.4.3, 1.1.2
    org.mockito:mockito-core has multiple versions 5.12.0, 4.3.1
Please remove duplicate artifacts from the artifact list so you do not
get unexpected artifact versions
```

See also:

- Duplicate maven repositories when importing bazel_deps that use
  maven.install
  bazel-contrib/rules_jvm_external#916

- Using maven as the repo name causes duplicate warnings when using
  bzlmod
  protocolbuffers/protobuf#16839

- MODULE.bazel doesn't define @maven repository
  protocolbuffers/protobuf#17176

- Stop including extra artifacts on maven.install()
  bazel-contrib/rules_jvm_external#1168

- Use a custom name for the maven repository (maven_protobuf)
  protocolbuffers/protobuf#17190

Signed-off-by: Mike Bland <[email protected]>
mbland added a commit to EngFlow/bazel_invocation_analyzer that referenced this issue Jul 3, 2024
Updates as much as possible except rules_jvm_external.

The latest rules_jvm_external v6.1 breaks this project, both under the
previous Bazel version (7.0.2) and the new one (7.2.1). I've filed
bazel-contrib/rules_jvm_external#1189, which uses this repo and this PR as
an example.

See the last section below for details.

---

Bazel update:
- Bazel v7.2.1
  https://github.com/bazelbuild/bazel/releases/tag/7.2.1

Bazel module updates:
- bazel-skylib v1.7.1
  https://github.com/bazelbuild/bazel-skylib/releases/tag/1.7.1
- platforms v0.0.10
  https://github.com/bazelbuild/platforms/releases/tag/0.0.10
- rules_proto v6.0.2
  https://github.com/bazelbuild/rules_proto/releases/tag/6.0.2

JAR updates:
- com.google.code.gson:gson:2.11.0
- com.google.guava:guava:33.2.1-jre
- commons-cli:commons-cli:1.8.0

Test JAR updates:
- com.google.googlejavaformat:google-java-format:1.22.0
- com.google.truth:truth:1.4.3
- com.google.truth.extensions:truth-java8-extension:1.4.3
- org.mockito:mockito-core:5.12.0

---

Added explicit module spec and repinned the maven deps for:
- protobuf v27.2
  https://github.com/protocolbuffers/protobuf/releases/tag/v27.2

Adding protobuf explicitly resolves these warnings:

```txt
DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/coursier.bzl:593:18:
    Found duplicate artifact versions
    com.google.code.gson:gson has multiple versions 2.11.0, 2.8.9
    com.google.guava:guava has multiple versions 33.2.1-jre, 31.1-jre
    com.google.truth:truth has multiple versions 1.4.3, 1.1.2
    org.mockito:mockito-core has multiple versions 5.12.0, 4.3.1
Please remove duplicate artifacts from the artifact list so you do not
get unexpected artifact versions
```

See also:

- Duplicate maven repositories when importing bazel_deps that use
  maven.install
  bazel-contrib/rules_jvm_external#916

- Using maven as the repo name causes duplicate warnings when using
  bzlmod
  protocolbuffers/protobuf#16839

- MODULE.bazel doesn't define @maven repository
  protocolbuffers/protobuf#17176

- Stop including extra artifacts on maven.install()
  bazel-contrib/rules_jvm_external#1168

- Use a custom name for the maven repository (maven_protobuf)
  protocolbuffers/protobuf#17190

---

rules_jvm_external v6.1 somehow creates duplicate `jvm_import` rules for
binary and source jars, instead of using the `srcjar` attribute:

```txt
ERROR: Traceback (most recent call last):
  File ".../external/rules_jvm_external~~maven~maven/BUILD",
    line 34, column 11, in <toplevel>
      jvm_import(

Error in jvm_import: jvm_import rule
  'com_google_auto_value_auto_value_annotations' in package ''
  conflicts with existing jvm_import rule, defined at
  .../external/rules_jvm_external~~maven~maven/BUILD:9:11
```

The content of rules_jvm_external~~maven~maven/BUILD at lines 9 and 34:

```bzl
jvm_import(
  name = "com_google_auto_value_auto_value_annotations",
  jars = ["com/google/auto/value/auto-value-annotations/1.10.1/auto-value-annotations-1.10.1.jar"],
```

```bzl
jvm_import(
  name = "com_google_auto_value_auto_value_annotations",
  jars = ["com/google/auto/value/auto-value-annotations/1.10.1/auto-value-annotations-1.10.1-sources.jar"],
```

This pattern repeats for all the JAR targets.

The BUILD contents from v5.3, which builds successfully both before and
after applying the PR changes:

```bzl
jvm_import(
  name = "com_google_auto_value_auto_value_annotations",
  jars = ["com/google/auto/value/auto-value-annotations/1.10.1/auto-value-annotations-1.10.1.jar"],
  srcjar = "com/google/auto/value/auto-value-annotations/1.10.1/auto-value-annotations-1.10.1-sources.jar",
```

---------

Signed-off-by: Mike Bland <[email protected]>
@tpudlik
Copy link

tpudlik commented Sep 28, 2024

As someone who maintains a non-root Bazel module, I remain confused about how to correctly use the maven module extension. Based on the discussion above, I believe I should not use the default name in maven.install. Should I just use a unique name instead?

But I'm worried about the comment above by @shs96c :

Inter-ruleset dependencies have to be handled this way because constants cannot be loaded in a MODULE.bazel file. If a rulesets wants to have dependencies only it uses (for example for custom tooling) then it’s recommended that they declare a custom namespace for themselves and depend upon that.

This makes it sound like a unique name for maven.install is only recommended "if a rulesets wants to have dependencies only it uses". To clarify, does this mean that if my module @foo has a java_library //:library with a Maven dep like @foo_maven//:somepackage, and another module depends on my library as @foo//:library, they'll run into some kind of trouble? So the custom namespace is only suitable if //:library is private to my module?

It would be great to update the documentation to explain this kind of nuance. Currently it just recommends use_repo(maven, "maven") and that's that.

@shs96c
Copy link
Collaborator

shs96c commented Sep 30, 2024

@tpudlik, there are a couple of things here that appear to be causing confusion. Our docs are primarily written for owners of root modules, but I can add extra details to help explain how best to use rules_jvm_external in a non-root module.

The important thing to bear in mind is that each of the tag classes in rules_jvm_external have a name attribute, and this defaults to maven. This name is used when creating the workspace and merging attributes, and it's advised that non-root module always set this to be something unique to the ruleset (as we do in our MODULE.bazel, and as is done in contrib_rules_jvm)

The install tag class has a lock_file attribute. Using this is a requirement if you're using the maven resolver, but is optional when using the default coursier resolver (you can select that using the resolver attribute of the install tag) It is strongly advised that all modules (root or non-root) use a lock file since it makes builds faster and more reliable by skipping the artifact resolution in each build. This file can point to a value in your own ruleset (eg. here and here). Within your own ruleset, you can call that file anything you like, including maven_install.json -- the value is a Label, and Bazel will know how to resolve these properly. If you're using a workspace-based build, maven_install is a macro, so I would lean to using the fully-qualified label.

With that out the way, the idea is that the maven namespace is reserved for a top-level ruleset. The only reason to add to it is if you require users to use your dependencies at runtime for their own code, and they can't use an external resource (eg. an artifact from a regular maven repo) in their build for some reason.

jschaf added a commit to jschaf/protobuf that referenced this issue Oct 7, 2024
The default name for the rules_jvm_external.maven rule is "maven". When not set,
it defaults to "maven". For root modules also using rules_jvm_external, the name
clash causes a warning:

```
DEBUG: $TMP/external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
The maven repository 'maven' is used in two different bazel modules,
originally in '<my_workspace>' and now in 'protobuf'
```

Summarizing @shs96c in [1]:

> The common maven repo name allows rulesets to contribute to the user's JARs.
> However, this implies that maven is for the end user, not for transitive
> dependencies. If a ruleset needs private dependencies, it should use a custom
> namespace rather than the maven namespace.

Since protobuf is not contributing to user's JARs, we'll use a custom namespace.
There's precedent for using a custom namespace for library modules:

- rules_jvm_external uses `rules_jvm_external_deps` instead of `maven`.
- rules_kotlin uses `kotlin_rules_maven` instead of `maven`.

[1]: bazel-contrib/rules_jvm_external#916 (comment)

Fixes protocolbuffers#16839.
copybara-service bot pushed a commit to protocolbuffers/protobuf that referenced this issue Oct 11, 2024
The default name for the rules_jvm_external.maven rule is "maven". When not set, it defaults to "maven". For root modules also using rules_jvm_external, the name clash causes a warning:

```
DEBUG: $TMP/external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
The maven repository 'maven' is used in two different bazel modules,
originally in '<my_workspace>' and now in 'protobuf'
```

Summarizing @shs96c in [1]:

> The common maven repo name allows rulesets to contribute to the user's JARs.
> However, this implies that maven is for the end user, not for transitive
> dependencies. If a ruleset needs private dependencies, it should use a custom
> namespace rather than the maven namespace.

Since protobuf is not contributing to user's JARs, we'll use a custom namespace. There's precedent for using a custom namespace for library modules:

- rules_jvm_external uses `rules_jvm_external_deps` instead of `maven`.
- rules_kotlin uses `kotlin_rules_maven` instead of `maven`.

[1]: bazel-contrib/rules_jvm_external#916 (comment)

Fixes #16839.

Closes #18641

COPYBARA_INTEGRATE_REVIEW=#18641 from jschaf:joe/protobuf-maven bd2c62f
PiperOrigin-RevId: 684625084
@ejona86
Copy link

ejona86 commented Oct 16, 2024

It seems the problem here just usability? The "Found duplicate artifact versions" warning has been happening since the creation of maven_install, and I've not heard of any actual problem because of it. It is 100% normal for the dependency tree not to align on versions and you choose the latest and life moves on. So with bzlmod it just is hard to find the source of the extra deps?

This comment thread is actually creating problems at runtime, as there must be only a single version of each dependency at runtime. See
protocolbuffers/protobuf#18641 (comment)
grpc/grpc-java#11614 (comment)

If the user defines their own java_proto_library toolchain (and we ignore gRPC for the moment), then some of this can function. But I don't see a single example suggesting to do that.

@develar
Copy link

develar commented Nov 7, 2024

I spent several hours investigating a mysterious Bazel failure, only to discover that it was due to this behavior.

For example, the official Bazel worker API library is also affected—it uses maven as its default repository name (see here). In my project, I had also used maven as a repository name, which led to the same mysterious Bazel failure. This is the default name, and there are no warnings in the documentation advising users to set a unique name.

It would be beneficial to reconsider this initial design choice (say, made name for maven.install as a required field) — or at least update the documentation to emphasize the importance of using a unique name for Maven repositories.

(thanks a lot for the debug logging—it helped point me in the right direction for investigation.)

@shs96c
Copy link
Collaborator

shs96c commented Nov 7, 2024

I'll happily merge a PR to update the documentation if you get there before I have a chance to write it.

intellij-monorepo-bot pushed a commit to JetBrains/intellij-community that referenced this issue Nov 9, 2024
zhangskz pushed a commit to protocolbuffers/protobuf that referenced this issue Dec 2, 2024
The default name for the rules_jvm_external.maven rule is "maven". When not set, it defaults to "maven". For root modules also using rules_jvm_external, the name clash causes a warning:

```
DEBUG: $TMP/external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
The maven repository 'maven' is used in two different bazel modules,
originally in '<my_workspace>' and now in 'protobuf'
```

Summarizing @shs96c in [1]:

> The common maven repo name allows rulesets to contribute to the user's JARs.
> However, this implies that maven is for the end user, not for transitive
> dependencies. If a ruleset needs private dependencies, it should use a custom
> namespace rather than the maven namespace.

Since protobuf is not contributing to user's JARs, we'll use a custom namespace. There's precedent for using a custom namespace for library modules:

- rules_jvm_external uses `rules_jvm_external_deps` instead of `maven`.
- rules_kotlin uses `kotlin_rules_maven` instead of `maven`.

[1]: bazel-contrib/rules_jvm_external#916 (comment)

Fixes #16839.

Closes #18641

COPYBARA_INTEGRATE_REVIEW=#18641 from jschaf:joe/protobuf-maven bd2c62f
PiperOrigin-RevId: 684625084
zhangskz added a commit to protocolbuffers/protobuf that referenced this issue Dec 2, 2024
The default name for the rules_jvm_external.maven rule is "maven". When not set, it defaults to "maven". For root modules also using rules_jvm_external, the name clash causes a warning:

```
DEBUG: $TMP/external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
The maven repository 'maven' is used in two different bazel modules,
originally in '<my_workspace>' and now in 'protobuf'
```

Summarizing @shs96c in [1]:

> The common maven repo name allows rulesets to contribute to the user's JARs.
> However, this implies that maven is for the end user, not for transitive
> dependencies. If a ruleset needs private dependencies, it should use a custom
> namespace rather than the maven namespace.

Since protobuf is not contributing to user's JARs, we'll use a custom namespace. There's precedent for using a custom namespace for library modules:

- rules_jvm_external uses `rules_jvm_external_deps` instead of `maven`.
- rules_kotlin uses `kotlin_rules_maven` instead of `maven`.

[1]: bazel-contrib/rules_jvm_external#916 (comment)

Fixes #16839.

Closes #18641

COPYBARA_INTEGRATE_REVIEW=#18641 from jschaf:joe/protobuf-maven bd2c62f
PiperOrigin-RevId: 684625084

Co-authored-by: Joe Schafer <[email protected]>
@vorburger
Copy link
Contributor

I've proposed a docs update in #1303 to at least make this a bit clearer and avoid some future confusion for others.

@ejona86
Copy link

ejona86 commented Jan 3, 2025

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

10 participants