Pruned lock file generation with yarn "file:" and "link:" package dependencies

[cwoebker]

Current Behavior

We are getting this error message

 >  NX   Pruned lock file creation failed. The following package was not found in the root lock file: internal-package-name@link:./packages/internal-package-name

Expected Behavior

We would expect the package.json and yarn.lock files to be generated correctly and the build to succeed.

GitHub Repo

No response

Steps to Reproduce

We recently upgraded from NX 15.1.1 to 15.7.0 and think that the changes introduced with 15.3.1 and this issue #9761 are responsible.

We used to copy over the entire yarn.lock file manually to the different dist/app folders but now want to use the lock file pruning feature.

Our package.json file contains a dependency like this:

{
  ...
  "dependencies": {
    ...
    "internal-package-name": "link:./packages/internal-package-name",
  }
}

We are getting this error message

 >  NX   Pruned lock file creation failed. The following package was not found in the root lock file: internal-package-name@link:./packages/internal-package-name

If we use file: instead of link: we are getting the same issue. Seems like this dependency is removed completely while the lock file is being pruned.

In the previous version of nx, we already faced the issue that the package.json file did not correctly reflect these dependencies but we were able to find a manual workaround.

Or are these local file: and link: dependencies simply not supported here?

Nx Report

>  NX   Report complete - copy this into the issue template

   Node : 16.15.1
   OS   : darwin arm64
   yarn : 1.22.19

   nx                      : 15.7.0
   @nrwl/jest              : 15.7.0
   @nrwl/linter            : 15.7.0
   @nrwl/workspace         : 15.7.0
   @nrwl/angular           : 15.7.0
   @nrwl/cli               : 15.7.0
   @nrwl/cypress           : 15.7.0
   @nrwl/devkit            : 15.7.0
   @nrwl/eslint-plugin-nx  : 15.7.0
   @nrwl/express           : 15.7.0
   @nrwl/js                : 15.7.0
   @nrwl/nest              : 15.7.0
   @nrwl/node              : 15.7.0
   @nrwl/tao               : 15.7.0
   @nrwl/webpack           : 15.7.0
   typescript              : 4.7.4
   ---------------------------------------
   Community plugins:
   @ionic/angular         : 6.5.3
   @ngrx/effects          : 14.3.3
   @ngrx/entity           : 14.3.3
   @ngrx/router-store     : 14.3.3
   @ngrx/store            : 14.3.3
   ngx-ui-loader          : 13.0.0
   @ionic/angular-toolkit : 7.0.0
   @ngrx/store-devtools   : 14.3.3

✨  Done in 1.85s.

Failure Logs

No response

Additional Information

Thanks for taking a look here.

[meeroslav]

Hey @cwoebker,

Thank you for reporting this. You are correct, using file: or link: is not supported. The reason is that such package.json cannot be deployed or published, nor lock file can be generated for it (that can work outside of your repo).

In our repo, we have the same requirement and use @nrwl/js:tsc which maps the missing internal packages to versions:

// this
    "@nrwl/devkit": "file:../devkit",
// becomes e.g.
    "@nrwl/devkit": "15.7.0",    

But this is only for publishing which doesn't require lock file generation. For deployment, all the packages need to be deployed first so targeting local packages from your app would not make sense, unless you bundle them. In that case there should be some intermediate step that should modify package.json to strip off bundled local packages.

Can you tell me what type of application you are having a problem with and what build executor you are using?

[meeroslav]

In the meantime, can you try this workaround: module-federation/core#578?

[CyanoFresh]

same for the using urls as versions. For example I want to use fork:

  "dependencies": {
...
    "jsonpatch-to-mongodb": "https://github.com/mrcranky/jsonpatch-to-mongodb#feature/issue-12-use-add-op-without-position",

error:

> nx run api:build

INSIDE WEBPACK CONFIG

 >  NX   Pruned lock file creation failed. The following package was not found in the root lock file: jsonpatch-to-mongodb@https://github.com/mrcranky/jsonpatch-to-mongodb#feature/issue-12-use-add-op-without-position


HookWebpackError: Pruned lock file creation failed. The following package was not found in the root lock file: jsonpatch-to-mongodb@https://github.com/mrcranky/jsonpatch-to-mongodb#feature/issue-12-use-add-op-without-position
    at makeWebpackError (project/node_modules/webpack/lib/HookWebpackError.js:48:9)
    at project/node_modules/webpack/lib/Compilation.js:3060:12
    at eval (eval at create (project/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:23:1)
    at fn (project/node_modules/webpack/lib/Compilation.js:480:17)
    at _next0 (eval at create (project/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:21:1)
    at eval (eval at create (project/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:34:1)
    at eval (eval at create (project/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:11:1)
    at project/node_modules/webpack/lib/Compilation.js:502:10
    at Hook.eval [as callAsync] (eval at create (project/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:7:1)
    at Hook.CALL_ASYNC_DELEGATE [as _callAsync] (project/node_modules/tapable/lib/Hook.js:18:14)
-- inner error --
Error: Pruned lock file creation failed. The following package was not found in the root lock file: jsonpatch-to-mongodb@https://github.com/mrcranky/jsonpatch-to-mongodb#feature/issue-12-use-add-op-without-position
    at project/node_modules/nx/src/lock-file/project-graph-pruning.js:40:19
    at Array.forEach (<anonymous>)
    at normalizeDependencies (project/node_modules/nx/src/lock-file/project-graph-pruning.js:26:42)
    at pruneProjectGraph (project/node_modules/nx/src/lock-file/project-graph-pruning.js:13:34)
    at createLockFile (project/node_modules/nx/src/lock-file/lock-file.js:112:75)
    at project/node_modules/@nrwl/webpack/src/plugins/generate-package-json-plugin.js:42:131
    at fn (project/node_modules/webpack/lib/Compilation.js:478:10)
    at _next0 (eval at create (project/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:21:1)
    at eval (eval at create (project/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:34:1)
    at eval (eval at create (project/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:11:1)

Seems like your findNodeMatchingVersion() function doesn't support anything else than valid semver string.

[cwoebker]

@meeroslav Thanks for the extensive comment from your end. By now we have integrated one linked dependency with a more integrated approach using WebPack and moved the other dependency to a published npm package since it did not contain any proprietary code anyways.
With the new release of 15.8.0 we now have everything working again since we were also blocked by #15019.

Thanks a lot!

[meeroslav]

@CyanoFresh this should actualy work. You can look at our tests targeting exactly these types of versions:
https://github.com/nrwl/nx/tree/master/packages/nx/src/lock-file/__fixtures__/auxiliary-packages

You can see something like this in the root package.json:

"postgres": "charsleysa/postgres#fix-errors-compiled",

becomes this in the pruned package.json:

"postgres": "git+ssh://[email protected]/charsleysa/postgres.git#3b1a01b2da3e2fafb1a79006f838eff11a8de3cb",

Generating a lock file converts the repo URL to the exact commit. This is then used as the source for the pruned lock file. If your project has again path to repo branch, that would potentially result in different commit (version) so the pruning fails.

Unfortunately, Nx cannot tell what is your intention since you might be having multiple versions of the same package pointing to the different branches:

"postgres1": "git+ssh://[email protected]/charsleysa/postgres.git#3b1a01b2da3e2fafb1a79006f838eff11a8de3cb",
"postgres2": "git+ssh://[email protected]/charsleysa/postgres.git#85bca49fab1251b88f6b79e1da9d7abcf4133182",

[CyanoFresh]

I have fixed that by adding git+ before the link:

    "jsonpatch-to-mongodb": "git+https://github.com/mrcranky/jsonpatch-to-mongodb#25f74bf997a65e18683433b7eeb43ba5c44f60a1",

Just looked that semver uses this format, tried and it works. Maybe someone will find this issue too

[github-actions]

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.