Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Time-based resolution mode #2

Merged
merged 3 commits into from
Sep 2, 2022
Merged

Time-based resolution mode #2

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bb8ac1a
Time-based resolution mode
zkochan Aug 16, 2022
7fd013a
minor improvements
zkochan Aug 16, 2022
ddabe45
add delta
zkochan Aug 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions text/0002-time-based-resolution-mode.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Time-based resolution mode

## Summary

A new resolution mode that resolves versions not to their highest versions but to versions that were published in a given time frame.

## Motivation

The resolution mode that is currently used by pnpm is not optimized for caching. During installation we need to always fetch the metadata of each package as we don't know if the metadata in the cache has the highest version of the package. Also, installation of the highest version is dangerous as someone may hijack a subdependency, publish a new version, and pnpm will immediately install it.

## Detailed Explanation

Resolution will be divided into two stages. The first stage will resolve all the direct dependencies of all the workspace projects. This stage may work the same way as it works now (highest version is picked). When all the direct dependencies are resolved, we check when were the picked versions released. This information is present in the package metadata at the "time" field. For example, if we install webpack and eslint, we get webpack resolved to v5.74.0 and eslint resolved to v8.22.0. `[email protected]` was released at "2022-07-25T08:00:33.823Z". `[email protected]` was released at "2022-08-14T01:23:41.730Z". Now we compare the dates of each released packages and pick the nearest date. In this case, the nearest date is the date eslint was released: "2022-08-14T01:23:41.730Z". Let's call this date T.

At the second stage, we resolve all the subdependencies. At this stage, instead of resolving a range to the highest available version, we filter out any versions that were released after T and pick the highest version from those. Let's say we need to resolve `ajv@^6.10.0`. We already have a metadata of ajv in cache and it was saved after T, so we don't need to redownload it. This are the versions of ajv that match `^6.10.0`:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In some monorepos packages are published in random order. So it might happen that a subdep was released later than the parent dep. Maybe we should add 10 minutes to the T.

Copy link

@d3lm d3lm Aug 17, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, adding some delta might be a good idea. In the most common cases it will still not fetch the metadata from sub deps.

Copy link

@dmichon-msft dmichon-msft Aug 31, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we get a config-level setting to prefer the minimum version that matches the set of merged specifiers, rather than the maximum, for the sake of stability? With how most monorepos operate, this would also provide the same guarantee, because it is common to publish with other-workspace-package: ^current in dependencies.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried this. It doesn't work as you would expect. There are many projects that update the dependencies but keep the old ranges. So the "current" doesn't really work anymore with the project.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, it is hard or not possible to implement with pnpm as I have stated in the alternatives section.


| version | release date |
|--|--|
| 6.10.0 | 2021-01-08 |
| 6.10.1 | 2021-12-22 |
| 6.10.2 | 2022-01-25 |
| 6.11.0 | 2022-04-05 |
| 6.11.1 | 2022-08-16 |

As you can see, v6.11.1 was released after T, so we pick v6.11.0, which is the highest version that satisfies the range and the time frame.
juanpicado marked this conversation as resolved.
Show resolved Hide resolved

## Rationale and Alternatives

An alternative solution might be to resolve to the lowest version. This is a resolution mode that [Maël Nison](https://github.com/arcanis) came up with and he already works on the [implementation in Yarn](https://github.com/yarnpkg/berry/pull/4351). It is called "stable" resolution mode. This resolution mode is harder to implement in pnpm. pnpm starts to fetch a package immediately after resolving it. This is fine with "hightest" resolution mode because different ranges are resolved to the same highest version. But when the lowest version is picked, the package manager should do some deduplication before starting the fetching process. For instance, in the root, `foo@^1.0.0` might be in dependencies. pnpm will immediately resolve it to `1.0.0` and fetch it from the registry. But then in the subdependencies there might be `foo@^1.1.0` in the dependencies, so pnpm will need to download a second version of foo even though 1.1.0 would have satisfied the root as well. This is easier to do with Yarn, which has a separate stage for resolution.

## Implementation

{{Give a high-level overview of implementation requirements and concerns. Be specific about areas of code that need to change, and what their potential effects are. Discuss which repositories and sub-components will be affected, and what its overall code effect might be.}}

{{THIS SECTION IS REQUIRED FOR RATIFICATION -- you can skip it if you don't know the technical details when first submitting the proposal, but it must be there before it's accepted}}

## Prior Art

{{This section is optional if there are no actual prior examples in other tools}}

{{Discuss existing examples of this change in other tools, and how they've addressed various concerns discussed above, and what the effect of those decisions has been}}

## Unresolved Questions and Bikeshedding

{{Write about any arbitrary decisions that need to be made (syntax, colors, formatting, minor UX decisions), and any questions for the proposal that have not been answered.}}

{{THIS SECTION SHOULD BE REMOVED BEFORE RATIFICATION}}