Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[internal] Unify JVM thirdparty resolves #13771

Merged
merged 4 commits into from
Dec 1, 2021

Conversation

stuhood
Copy link
Member

@stuhood stuhood commented Dec 1, 2021

  • Move the testprojects lockfile to 3rdparty, to further the convention of lockfiles living there.
  • Move jvm_artifact definitions to 3rdparty, also for the purposes of convention.
  • Use dependency inference for java_parser.
  • Add targets for scala_parser.

[ci skip-rust]

[ci skip-build-wheels]
@stuhood stuhood requested review from chrisjrn and tdyas December 1, 2021 21:09
@stuhood
Copy link
Member Author

stuhood commented Dec 1, 2021

@tdyas : This shouldn't conflict much with #13754, sorry about that. But one thing that I noticed here is that the java/scala parser code should actually be able to load the lockfile itself as resources (I added them to the glob here)... that would immediately eliminate the need to manually specify the transitive deps, and ensure that they were always in sync.

This also aligns a bit with how tool lockfiles work on the Python side (cc @Eric-Arellano )

Copy link
Contributor

@chrisjrn chrisjrn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems OK, barring one comment

@@ -0,0 +1,6 @@
jvm_artifact(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would probably be a better example to have all of the com.fasterxml.jackson declarations in the same file -- from a maintenance point of view, the versions of these packages are usually kept in lockstep, and having to change a version in multiple files seems obtuse.

Copy link
Member Author

@stuhood stuhood Dec 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, there's an interesting tradeoff there right now which I think that we can maybe remove via #13767: it's not possible to share a variable between different BUILD files without a macro. Allowing macros to be declared hierarchically would help to keep these from going out of sync.

But from experience with a thirdparty layout that had exceptions to the org/group-as-hierarchy (Twitter had ~1k direct thirdparty deps): folks ended up confused about where to put things, and so you ended up with multiple copies of some libraries in parent/child directories. If anything, I'd love to be able to enforce it, although that probably goes too far for casual usage.

I'll ... add a comment for now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's unfortunate that Jackson has such a fine-grained group structure, which still doesn't make a whole lot of sense to me, tbqh. It really doesn't lend itself well to this multiple files situation.

[ci skip-rust]

[ci skip-build-wheels]
[ci skip-rust]

[ci skip-build-wheels]
# Rust tests and lints will be skipped. Delete if not intended.
[ci skip-rust]

# Building wheels and fs_util will be skipped. Delete if not intended.
[ci skip-build-wheels]
@stuhood stuhood force-pushed the stuhood/unify-thirdparty-resolves branch from 2954ded to 6056483 Compare December 1, 2021 21:43
@stuhood stuhood enabled auto-merge (squash) December 1, 2021 21:45
@stuhood stuhood merged commit 8423acc into pantsbuild:main Dec 1, 2021
@stuhood stuhood deleted the stuhood/unify-thirdparty-resolves branch December 1, 2021 21:53
stuhood added a commit that referenced this pull request Dec 1, 2021
#13771 added usage of a target from the `scala` backend, which is not enabled during dryruns. Enable it.

[ci skip-rust]
illicitonion added a commit that referenced this pull request Dec 7, 2021
Internal changes:

* [internal] Remove superfluous f-string specifiers ([#13821](#13821))

* [internal] scala: extract annotations as consumed types ([#13810](#13810))

* [jvm] Split nailgun digest from input file digests ([#13813](#13813))

* [internal] jvm: add jre_major_version and use stderr to properly extract version ([#13812](#13812))

* [internal] Clean up Go `embed` support's handling of dependencies ([#13801](#13801))

* [internal] scala: handle package object syntax in parser ([#13809](#13809))

* [internal] java: fix junit sentinel rule setup ([#13815](#13815))

* [internal] upgrade to rust v1.57.0 ([#13807](#13807))

* [internal] add failing test for FrozenDict equality issue ([#13389](#13389))

* [internal] Use `PyObject` instead of `Value` in more places ([#13802](#13802))

* Remove MultiPlatform Process abstractions ([#12725](#12725))

* [internal] add `JvmToolBase` for lockfile handling for JVM tools ([#13777](#13777))

* [internal] Port `MergeDigests` to Rust ([#13773](#13773))

* [jvm] Spawn nailgun servers outside the pool lock ([#13796](#13796))

* [internal] DRY loading internal Go binaries ([#13800](#13800))

* [internal] Convert unit tests to use pytest ([#13798](#13798))

* [internal] remove dead code - socket util. ([#13797](#13797))

* [internal] Reorganize Go parser scripts ([#13791](#13791))

* Adds Jackson core/datatype to `JVM_ARTIFACT_MAPPINGS` ([#13792](#13792))

* [internal] go: initial support for embedding resources ([#13743](#13743))

* [internal] Refer to `go.mod` path when downloading packages ([#13786](#13786))

* [internal] More robust Go dependency inference tests ([#13785](#13785))

* [internal] `tailor` doesn't add `go_package` for `testdata` folder ([#13783](#13783))

* [internal] Add Scala backend to dryrun for wheel builds. ([#13772](#13772))

* [internal] Unify JVM thirdparty resolves ([#13771](#13771))

* [internal] scala: infer dependencies from consumed symbols and intermediate scopes ([#13758](#13758))

* [internal] java: infer scala encoded symbols ([#13739](#13739))

* [internal] scala: parse and report package scopes ([#13738](#13738))

* [internal] go: configure included env vars for `GoSdkProcess` ([#13734](#13734))

* Fix some `./cargo audit` vulnerabilities. ([#13728](#13728))

* [internal] fix non-empty __init__.py ([#13730](#13730))

* Compute RepositoryPex directly from addresses. ([#13716](#13716))

* Upgrade to cargo-audit 0.16.0. ([#13729](#13729))

* Simplify `NativeHandler`. ([#13727](#13727))

* [internal] Switch to a maintained fork of the `fuse` crate for `brfs`. ([#13679](#13679))

* [internal] Add infrastructure to support deprecating field names ([#13666](#13666))

* [internal] Introduce OptionalPex/OptionalPexRequest. ([#13712](#13712))

* [internal] tailor adds go_package targets ([#13703](#13703))

* [internal] Remove unused testproject for pants-plugin ([#13704](#13704))

* [internal] Rename ambiguous `subpath` variable for Go code ([#13701](#13701))

* [internal] scala: generate the JVM names seen by Java code for Scala code ([#13696](#13696))

* Use RequirementsPexRequest in run_pex_binary.py. ([#13693](#13693))

* [internal] Refactor finding owning `go_mod` for first-party packages ([#13695](#13695))

* [internal] repl: add append_only_caches / run_in_workspace attributes to ReplRequest ([#13599](#13599))

* [internal] switch back to official `cargo-ensure-prefix` crate ([#13692](#13692))

* [internal] scala: extract type names from all Type.* AST nodes ([#13685](#13685))

* [internal] Convert unit tests to use pytest ([#13652](#13652))

* Unblock codegen support for java export analysis (#13645) ([#13675](#13675))

* [internal] upgrade to Rust 2021 Edition ([#13644](#13644))

* [internal] Don't store derived values on `go_first_party_package` targets ([#13676](#13676))

* Upgrade to py03 0.15.1. ([#13725](#13725))

* Add PyPDF2 to module mapping ([#13717](#13717))

* Upgrade console and indacatif. ([#13726](#13726))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants