Add support for generating a package-lock.json file for each project

[jrista]

• I'd be willing to implement this feature (contributing guide)

Description

Currently, Nx supports the build option generatePackageJson which is dropped into the build output (usually under dist/). This is great, as it allows only the necessary dependencies to be installed for each application, which becomes paramount when trying to configure and deploy docker images for each app in a monorepo.

I would like to see support for package-lock.json added, so that we may perform package installs using the lock file, rather than package.json, via the npm ci command.

Motivation

One of the drawbacks with just having package.json is that we have to run npm i to install, which may bump versions if ^ or ~ are used in the version numbers in the dependencies. There is another wonderful command that npm has, npm ci, which does a "clean install" by referencing ONLY package-lock.json. This will guarantee that only the exact versions of dependencies that are known to build correctly will be used when building in any other environment. An added benefit of using npm ci is it can and usually is faster than npm i as it does not have to search for and determine IF there are newer packages to be installed...it installs exact versions, and can do so as quickly as possible, using any configured caching mechanisms optimally, etc.

Using npm ci has become standard for me anywhere outside of my own local. In fact, when I first clone a repo onto a new machine, I will use npm ci to install with the same deps I had on other machines. I am an avid fan of CircleCI, which has configured their system to support near-instantaneous installs with npm ci as well...even when you have gigs and gigs of dependencies, npm ci can run in a matter of 20-30 seconds with a properly configured npm repo and a fast network.

I would like to leverage the same benefits of npm ci in my Dockerfile configs for each of my apps in an nx monorepo. This will guarantee that I am installing KNOWN and TESTED packages, and never run the risk of unexpectedly updating a package to a newer and potentially broken version during image creation.

Suggested Implementation

I am not certain as to how this could be implemented. Given that the package.json that is generated has a specific list of direct dependencies, I've wondered if it would be possible to walk the root package-lock.json, mark each "reachable" dependency (starting with the roots from the generated package.json) and rebuild a new one with just the dependencies that are "reachable."

Alternate Implementations

Potentially just run an npm install in the apps dist directory after generating the app-specific package.json. The downside of this is that we could also still update packages with ~ or ^ to newer versions.

[gioragutt]

@jrista based on this thread in slack, you can just copy over the lock file (package-lock.json, yarn.lock or pnpm-lock.yaml) from the root of your repo.

Package managers will use the versions specified in the lock file only for the packages specified in package.json, so this gives you the same behavior.

[jrista]

@gioragutt Are you sure that is the behavior when using npm ci? I can see it being the case if you do npm i, but the latter runs the risk of updating package-lock.json to newer versions.

[Ionaru]

@gioragutt Are you sure that is the behavior when using npm ci? I can see it being the case if you do npm i, but the latter runs the risk of updating package-lock.json to newer versions.

I can confirm that npm ci will only install the packages mentioned in package.json while using the exact versions from package-lock.json, even if there are more packages mentioned in package-lock.json than there are in package.json.

[melMass]

@gioragutt Is right, I'm relying on this behavior for my packages

[BeyondEvil]

While copying the lock file works, I would classify that as a workaround.

One possible solution, seeing how parsing the lockfile and generating an app-specific one can be quite expensive, an alternative solution could be to generate the package.json file with strict versions (fetched from the lock-file).

That way, it would fill the same purpose as the lock-file.

[jrista]

Sorry for the late reply, got sucked into the world of modularizing a TF config. I did some testing, and copying the package-lock.json file manually did seem to work.

Regarding the idea @BeyondEvil came up with...while that might lock down root level versions, I suspect it would still allow the child dependencies of any direct dependencies to update, if their own package.json files do not use explicit versions.

I agree that manually copying in package-lock.json is a workaround, as it is manual, and would have to be done pretty much every time. Would be nice if NX could just make sure that the package-lock is included regardless, so that npm ci will work if it is preferred over npm i.

[gioragutt]

Might I say, I just thought of it.
If you want yarn.lock in your dist, add it to your projects "assets", it should do the trick

[jrista]

I'll give that a try...

FWIW, yarn does not actually have any direct corollary to npm ci that I've ever been able to find. It references its lock file, but that does not actually prevent yarn from updating existing packages. There is an option to cause yarn install to exit with an error code IF it "would have" updated packages, but that is not the same as npm ci which will only install exactly the packages referenced in package-lock.json as yarn fails to install.

[gioragutt]

@jrista what about yarn --frozen-lockfile?

[jrista]

@jrista what about yarn --frozen-lockfile?

That is what I am referring to. Unlike npm ci which will install only based on the lockfile, the yarn docs state this about --frozen-lockfile:

"Don’t generate a yarn.lock lockfile and fail if an update is needed."

[jackpordi]

Might I say, I just thought of it. If you want yarn.lock in your dist, add it to your projects "assets", it should do the trick

@gioragutt I get a "asset path must start with the project source root" error if I do that, as I'm using a project root level lockfile.

[jrista]

It seems that there may still need to be work done here to make this function. I tried adding my project-lock.json as an asset, which isn't allowed. I then tried looking around for any kind of build hooks I could hook into. I found something about using the architect section of a project config and the @nrwl/workspace:run-commands builder...but, that only allows you to configure preparation commands, which are a pre- step, not a post- step. Assuming it did have any kind if post-build hook, the @nrwl/workspace:run-commands stuff has to be added to each project individually, which is a bummer.

I have not yet found a way to hook into the nx build cycle to make sure that project-lock.json is copied into the dist/* directories as necessary to make sure I can npm ci my builds.

[ericwooley]

I haven't used it in nx, as I no longer have access to some of my old projects. But I recently reimplemented packge.json generation from a yarn monorepo, and I think I found a solution to this.

After building your project, cd into the dist folder, with your generated package json. Run, npm i --package-lock-only, which appears to resolve the packages in your package.json to the ones which are installed, and generates a package-lock.json.

There might be some extra steps with regards to resolving properly in nx, depending on your setup. But in a yarn workspace, I haven't found any version discrepancies between package-lock.json and yarn.lock.

Your mileage may vary, but for me, using npm ci with a package-lock.json is a clear win over no lock file at all, especially with all the package supply chain attacks these days.

[jrista]

Thanks for the tip, @ericwooley! Never knew about the --package-lock-only option.

[galah92]

@ericwooley this is great!

Is there an easy way to run this as a post-step after the @nrwl/node:webpack for all the projects in my workspace?

[BeyondEvil]

@ericwooley this is great!

Is there an easy way to run this as a post-step after the @nrwl/node:webpack for all the projects in my workspace?

Easy or not, you could create a target that has @nrwl/node:webpack as a pre-step.

[joeflateau]

@ericwooley amazing... i'm now copying ./project/package-lock.json into ./project/dist/app/my-app then running npm i --package-lock-only and i get a package-lock.json trimmed to the appropriate packages 🎉

[joeflateau]

here's my quick and dirty implementation of generating a package-lock.json

removed as it doesn't actually work

[joeflateau]

npm i --package-lock-only doesn't actually seem to work for me. npm ci complains that package.json and package-lock.json are out of sync. presumably because there is no node_modules for --package-lock-only to reference

[10kc-emilyk]

I've been able to successfully generate package-locks in each project using @npmcli/arborist, via the technique used in npm-lockfile. It needs a bit of cleanup, but I'll be able to submit a PR for nx to solve this issue.

[gioragutt]

@10kc-emilyk I'll be happy to review the PR when it's up (or provide any help otherwise)

[joeflateau]

I'll also gladly review/assist in any way

[zh3ngyuan]

wondering if there is any update on this?

[jrista]

I haven't found any solution yet myself. I can't seem to find a hook that I could use that would work, and I'm not familiar enough with the Nx code to know where to begin as far as trying to implement a built-in solution. I think this may need to be handled internally by Nx, or one of its existing builders, with a new flag (say generatePackageLock or copyPackageLock or something.

[10kc-emilyk]

https://github.com/ALexander4295502
wondering if there is any update on this?

Sorry, I havent had a chance to look at integrating it into nx yet.

[conioX]

any news about this feature guys? or can we create something like how next js standalone?

[meeroslav]

@conioX, it's being implemented.

You can look at the current progress here:
#13155

You can already use it, but you would have to call nx internal functions. This is not recommended, though as this code is still under development and will likely move around.

The public API will be available soon (in a few weeks).

[kroeder]

@conioX, it's being implemented.

You can look at the current progress here: #13155

You can already use it, but you would have to call nx internal functions. This is not recommended, though as this code is still under development and will likely move around.

The public API will be available soon (in a few weeks).

@meeroslav Does this feature also support npm workspaces? I only see mentions for yarn workspaces and pnpm workspaces

[meeroslav]

@kroeder, not yet. I will add that feature probably on Monday

[simllll]

is there anything to test already? really looking forward to this feature, and willing to support :-)

we use a lerna / npm workspace

[joeflateau]

@meeroslav @FrozenPandaz Is this really "Closed"? I see it's implemented in core now 🙌 but shouldn't the issue stay open until pruning makes it out to the actual builders/executors? Maybe I'm misunderstanding and they already inherit this from devkit

[jrista]

Same questions here. Excited that work is being done. Is this only going to land in 15, or will it be backported to older versions as well?

[meeroslav]

@joeflateau, in v15 this is currently an opt-in functionality that you can call manually

require(‘@nrwl/devkit’).pruneLockFile(‘project-name’); // returns stringified pruned lock file

We will add it to executors that already support the generatePackageJson flag.

A blog post and guide will follow soon that describes this in more detail. There are still some parts in progress (e.g. npm workspaces and some minor fixes).

@jrista, unfortunately, due to the scope of changes involved in this effort, it will not be backported to v14.

[joeflateau]

Wonderful. Thank you

[github-actions]

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.