Use stack ls dependencies json #1364
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* JSON with unicode chars * strings with escapes * tests using package.json from some major npms
aherrmann
force-pushed
the
stack-json
branch
2 times, most recently
from
29ac2ef to
5c440c6
Compare
hanshoglund
approved these changes
Jun 15, 2020
aherrmann
added a commit
that referenced
this pull request
Jun 22, 2020
Addressing review comments #1364 (comment) and #1364 (comment)
buildifier chokes on a nested WORKSPACE file.
ls dependencies json was introduced in 2.3
Previously, the strategy was to first fetch all packages listed by the user in the `packages` attribute to `stack_snapshot` and then use them as "local packages" to determine all transitive dependencies. This interleaves the fetching of remote packages with the dependency resolution which prevents us from persisting the results of dependency resolution in a lock file. The new strategy implemented by this commit is to instead generate a local "dummy" package that lists all packages specified by the user as its dependencies and generate a `stack.yaml` file that captures all Cabal flags, package version overrides, and vendored packages. Stack is then invoked on these two files to determine the transitive dependency graph without first fetching any packages into the external Bazel workspace. The package sources are only fetched in a later step. This separation allows to store the package metadata in a lock file and let Bazel fetch package sources separately based on the information in the lock file in future. I have verified that the resulting external workspaces are equivalent up to expected differences by comparing the results of `bazel fetch @stackage//:all` in the `example` directory, similarly for `@stackage` and `@ghcide` in the project directory. Note, dictionaries are iterated in insertion order. To compare the results before and after this commit one has to sort the contents of `package.bzl` and `BUILD.bazel` by package name first.
It's no longer necessary to separate the loop that determines the dependencies from the loop that discovers the packages. With this change the loops are merged into a single loop that just converts the metadata returned by `stack ls dependencies json` into the format that `stack_snapshot` expects.
Addressing review comments #1364 (comment) and #1364 (comment)
|
/azp run |
|
@Mergifyio refresh |
|
Command |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a step towards #1310
Version 2.3.1 of
stackintroducedstack ls dependencies json. With this PR we make use of that feature to determine the package dependency graph and package versions, instead of the previous approach of parsing thedotformatted dependency graph.Following the example of
rules_jvm_externalthis vendors the Starlark JSON parser from https://github.com/erickj/bazel_json.This PR also separates the step that determines the dependency graph from the step that fetches the package sources. This is in preparation for #1310 where we'll want to store the dependency graph and package locations in a lock file and perform the fetching in a separate repository rule that can depend on a lock file alone.
I have verified that the resulting external workspaces are equivalent
up to expected differences by comparing the results of
bazel fetch @stackage//:allin theexampledirectory and@stackage//:alland@ghcide//:allin the project root. Note, dictionaries are iterated in insertion order. To compare the results before and after this commit one has to sort the contents ofpackage.bzlandBUILD.bazelby package name first.The old mechanism did not respect Cabal revisions. This new implementation fetches the correct Cabal revisions as defined in the Stackage snapshot.(This difference was due to not passing
--resolvertostack unpackas is required.)