Cannot publish jvm_binary

[jhump]

I am converting a repo from using Maven to using Pants. The repo includes a protoc plugin, implemented in Java. The resulting artifact is a runnable program (e.g. can do java -jar plugin.jar).

In Maven, this is configured in the pom.xml file by providing configuration for the maven shade plugin. The configuration tells it to include a shaded artifact (e.g. the self-contained fat jar) and also configures it to include the Main-Class entry in the resulting JAR's META-INF/MANIFEST.MF file.

When I first got the repo running and CI working correctly with Pants, I had accidentally defined the main artifact as a java_library. Everything worked fine, but the published artifact was not a fat JAR and didn't indicate the main class in the manifest.

So I changed it to jvm_binary. This step required a couple of changes:

• Kept the java_library target, which references the sources and dependencies.
• Created a new jvm_binary that simply referred to the java_library (since jvm_binary targets can't actually have sources other than a single main class). I also moved the provides attribute here since this is the target I want to publish.

Several things went wrong at this point:

1. Attempting to publish failed silently: it printed "SUCCESS" but just did nothing. The online doc talks about this symptom, but only indicates the need for provides attributes on targets to publish.
2. After playing around with command-line incantations to get something to happen, I found that indicating the target in an --override argument produced an error: FAILURE: src/main/java:protobuf-plugin is not an exported target
3. After not understanding this at all, I began wading through Pants source code and found that jvm_binary targets do not have an exportable label. @ericzundel helped me out by giving me a custom PEX where jvm_binary targets are exportable.
4. With the custom PEX, I ran into different errors:

FAILURE: The following errors must be resolved to publish.
  Cannot publish BuildFileAddress(FilesystemBuildFile(/Users/jh/Development/protobuf-plugins/src/main/java/BUILD), protobuf-plugin) due to:
    BuildFileAddress(FilesystemBuildFile(/Users/jh/Development/protobuf-plugins/src/main/java/BUILD), protobuf-plugin) - No sources.
    BuildFileAddress(FilesystemBuildFile(/Users/jh/Development/protobuf-plugins/src/main/java/BUILD), protobuf-plugin-lib) - Does not provide a binary artifact.

I'm okay with just omitting source artifacts for this. And the thing I really want published is a fat jar, so I shouldn't have to publish internal dependencies since they'll be bundled into the published artifact. (It would also be nice if I just had a java_binary target that let me include all of my sources. Then I wouldn't need an internal unpublished target and I'd have sources to publish...)

[jsirois]

Historically pants has not allowed publishing jvm_binary objects with intention. The idea being an unshaded fatjar can only cause chaos when it contains other libraries and gets on a classpath that has those other libraries mixed, say [my binary using guava, guava, other things needing guava]. You know the rest of the story...

So either pants would need to be able to verify all 3rdparty deps in the fat jar are shaded or pants needs to loosen up and allow folks to shoot themselves in the foot.

Opinons on these paths are welcome. I'm also interested in the use case. How is publishing a fat jar useful for you and your teams?

[jhump]

To prevent the dependency problem, the fat jar artifact has a classifier. So it wouldn't inadvertently be used if the target were simply added to something else as a dependency. That would still use the standard jar artifact. We currently use a convention of classifier = "shaded" for the fat jar files (not sure if there's a standard.)

The reason it is useful is so that other places that want to use the protoc plugin can just download the fat jar and run it via java -jar. That is far simpler than having to crawl its associated pom.xml file, download all of its transitive dependencies, and build a classpath before being able to run the program. (I suppose a tool that encapsulates all of these steps might suffice, too...)

Why is it necessary to split binaries that have more than one source file into two targets (the binary and accompanying library)? This seems like an arbitrary constraint whose value isn't immediately apparent.

[jsirois]

OK - thanks for the example. Just plopping the binaries on a fileserver or standard dev or prod binary distrbution mechanism would work as well OOB from pants, but it may be the case that you or someone else uses an artifactory or nexus install for that purpose.

I'll try to explain the current state of things here more fully - in short this presents as a bug to you, but pants has not yet been designed to publish anything but library level jars, from the internal pants product scheduling pipeline to the target UIs.

In more detail, the jvm_binary single file restriction was not motivated as a restriction but as an affordance - I don't think publish even existed as a goal at that point. The idea was jvm_binary was fundamentally just a metadata target saying this classpath over here has this main, but then its grew the ability to contribute the actual source file that carried the main to permit both simple binaries of 1 file and encourage most code to be structured (code-wise) as a library even if just to present an API to the main. IIRC the push here was to get folks at Twitter to write mainly testable code and just have the main gather flags and positional args and instantiate and use the api. The former being hard to test w/o shelling out, the latter being standard unit test fodder.

On to solving this. Right now publish does not allow combination of libraries in a single publish artifact at all. It does this to ensure in the simplest way possible that the same classfiles don't get published in 2 different jars. So that structure and aim works against combining jars natively, dooming any attempts by Eric to help you out here at the very end. There is though a customization mechanism, and its designed explicitly to allow attachment of additional artifacts beyond primary + sources + javadoc (+ CHANGELOG). I think this is the way to go. If this works well enough for you all, then it may make sense to land this in contrib so it can be a standard feature repos can turn on.

The high level docs are here: https://pantsbuild.github.io/dev_tasks_publish_extras.html
If this sounds sane to you and @ericzundel I can provide more help getting a publish plugin to attach "shaded" fatjars.

[Eric-Arellano]

Closing due to being stale.