Nx & [email protected] errors using "ng build --prod" #1706
Comments
|
@FrozenPandaz can I somehow help with this issue? It seems that Nx builder is trying to bundle all dependencies, even lazy loaded ones. For instance, this const cacheManager = loadPackage('cache-manager', 'CacheModule', () =>
require('cache-manager'),
);shouldn't be bundled if |
|
Edit: Below is not entirely correct after further investigation. Refer to @kamilmysliwiec comment below. After spending a few hours on this, I have found the root of the problem! I believe removing the line @FrozenPandaz Fortunately server side build sizes are not important for most scenarios. Would it be possible to make a change to Nx so that |
|
So basically I have dig into this issue a little bit and I have some takeaways to share. Firstly, this issue is not caused by 6.0.0 release - it wasn't working properly in 5.x either. However, in 5.x you were able to compile your Hence, this for instance: const cacheManager = loadPackage('cache-manager', 'CacheModule', () =>
require('cache-manager'),
);leads to compilation errors, because webpack traverse modules graph and tries to resolve How to fix this? The easiest solution would be to use externals: [nodeExternals()], And it should be fine in 99% cases. However, sometimes you may prefer to bundle entire application (one file, smaller size, somewhat lower memory consumption, slightly faster bootstrap time [serverless]) - in this case, you could use plugins: [
...options.plugins,
new webpack.IgnorePlugin({
checkResource(resource) {
const lazyImports = ['@nestjs/microservices', '@nestjs/platform-express', 'cache-manager', 'class-validator', 'class-transformer'];
if (!lazyImports.includes(resource)) {
return false;
}
try {
require.resolve(resource);
} catch (err) {
return true;
}
return false;
},
}),
],This workaround would ignore whitelisted packages IF they aren't present ( |
|
Edit: The following is a messy solution. Simply setting Thanks @kamilmysliwiec for all the hard work. The community loves you for it ^_^ My build process is a bit more convoluted than I was hoping for, but I ended up using @kamilmysliwiec recommendation to use webpack-node-externals.
var nodeExternals = require('webpack-node-externals');
module.exports = {
externals: [nodeExternals()]
};
|
|
looks like nx already support excluding some or all node modules from bundling for my case |
|
What if I want to build a dockerfile for the Backend? I don't want all installs of my package.json in the docker image, so I think bundling the nest.js' dependencies should be possible. Starting the resulting container will result in this output: "externalDependencies": "none" seems like the option I need, but if fails the build. Anybody got an idea how to handle this? |
|
Thanks for the debugging @kamilmysliwiec. We have the We also accept a /**
* @param config is the config Nx generates
* @param context some context about the task
**/
module.exports = (config, context) {
// extend, mutate, create a new config (whatever you want)
return config;
}For the root of the issue @kamilmysliwiec: I feel like (not sure) dependencies/ tree-shaking isn't what is expected. If the user does not use |
See my response here, especially this part:
Basically, Nest in some cases is using Another, more important example is The solution that I have included in my post (with So reassuming, if we still want to have tree-shaking + bundling, the only thing is to put this |
|
testing with var IgnorePlugin = require('webpack').IgnorePlugin;
module.exports = (config, context) => {
// extend, mutate, create a new config (whatever you want)
config.plugins = [
...config.plugins,
new IgnorePlugin({
checkResource(resource) {
const lazyImports = [
'@nestjs/microservices',
'@nestjs/platform-express',
'cache-manager',
'class-validator',
'class-transformer',
];
if (!lazyImports.includes(resource)) {
return false;
}
try {
require.resolve(resource);
} catch (err) {
return true;
}
return false;
},
}),
];
return config;
};@FrozenPandaz is this correct way to customize webpack ? |
|
I created a PR to revert the change for now. I still suggest people look into bundling in their dependencies but I guess it's hard to provide something that works for NestJS out of the box. @xmlking @kamilmysliwiec Maybe the |
|
Actually removing the external externalDependencies, and adding all and none - still no go, so i decided to add the packages it was complaining about, it was all going well but then I had to add @nestjs/microservices and then mqtt because it complained that was missing and then i run into this. It appears there is a hashbank in the js file and webpack doesn't understand it. I am stuck, anyone have a workaround, i tried everything above but no go. |
|
Also tried @FrozenPandaz suggestion of and this produces :-( |
|
@kamilmysliwiec @FrozenPandaz Tried another solution previously mentioned but I get the following error passing in a webpackconfig to nrwl. Any help anyone, I am not able to build anything. If all else fails - I suppose i need to manually build each library ... rather than use ng . |
|
The change has been reverted in Nx 7.8.0. If you would like to bundle your nest application, you can look into providing an Array of modules to remain external. The update does not run any migrations so if you are still running into issues, remove |
|
got the same issue... and working with Nx workspace to manage multiples nestjs application.. |
|
Stumbled upon this issue when I tried to get webpack.server.config.js work in my nx workspace with a seperated nest app for ssr.. (i was missing ContextReplacementPlugin -> Demo+Explanation) Maybe this helps someone. |
|
i'm trying to bundle my apps with webpack to reduce package size as i'm using aws lambda functions and i don't manage to make it work using the suggested solution .... WARNING in ./node_modules/@nestjs/core/helpers/load-adapter.js 8:39-63
Critical dependency: the request of a dependency is an expression
@ ./node_modules/@nestjs/core/nest-factory.js
@ ./node_modules/@nestjs/core/index.js
@ ./src/sls.ts
WARNING in ./node_modules/@nestjs/common/utils/load-package.util.js 8:39-59
Critical dependency: the request of a dependency is an expression
@ ./node_modules/@nestjs/core/nest-application.js
@ ./node_modules/@nestjs/core/index.js
@ ./src/sls.ts
WARNING in ./node_modules/optional/optional.js 6:11-26
Critical dependency: the request of a dependency is an expression
@ ./node_modules/@nestjs/core/nest-application.js
@ ./node_modules/@nestjs/core/index.js
@ ./src/sls.ts
WARNING in ./node_modules/express/lib/view.js 81:13-25
Critical dependency: the request of a dependency is an expression
@ ./node_modules/express/lib/application.js
@ ./node_modules/express/lib/express.js
@ ./node_modules/express/index.js
@ ./src/sls.ts
WARNING in ./node_modules/type-graphql/dist/helpers/loadResolversFromGlob.js 10:46-63
Critical dependency: the request of a dependency is an expression
@ ./node_modules/type-graphql/dist/utils/buildSchema.js
@ ./node_modules/type-graphql/dist/utils/index.js
@ ./node_modules/type-graphql/dist/index.js
@ ./node_modules/@nestjs/graphql/dist/graphql-schema-builder.js
@ ./node_modules/@nestjs/graphql/dist/graphql.factory.js
@ ./node_modules/@nestjs/graphql/dist/index.js
@ ./node_modules/@nestjs/graphql/index.js
@ ./src/app.module.ts
@ ./src/sls.ts
WARNING in ./node_modules/merge-graphql-schemas/dist/index.esm.js 3782:21-31
Critical dependency: the request of a dependency is an expression
@ ./node_modules/@nestjs/graphql/dist/graphql-types.loader.js
@ ./node_modules/@nestjs/graphql/dist/index.js
@ ./node_modules/@nestjs/graphql/index.js
@ ./src/app.module.ts
@ ./src/sls.ts
WARNING in ./node_modules/ts-morph/node_modules/typescript/lib/typescript.js 94814:19-45
Critical dependency: the request of a dependency is an expression
@ ./node_modules/ts-morph/dist/typescript/public.js
@ ./node_modules/ts-morph/dist/main.js
@ ./node_modules/@nestjs/graphql/dist/graphql-ast.explorer.js
@ ./node_modules/@nestjs/graphql/dist/index.js
@ ./node_modules/@nestjs/graphql/index.js
@ ./src/app.module.ts
@ ./src/sls.ts
WARNING in ./node_modules/subscriptions-transport-ws/node_modules/ws/lib/buffer-util.js
Module not found: Error: Can't resolve 'bufferutil' in '/Users/odifisIH/Sites/cashmetrics/packages/core/node_modules/subscriptions-transport-ws/node_modules/ws/lib'
@ ./node_modules/subscriptions-transport-ws/node_modules/ws/lib/buffer-util.js
@ ./node_modules/subscriptions-transport-ws/node_modules/ws/lib/receiver.js
@ ./node_modules/subscriptions-transport-ws/node_modules/ws/index.js
@ ./node_modules/subscriptions-transport-ws/dist/server.js
@ ./node_modules/subscriptions-transport-ws/dist/index.js
@ ./node_modules/apollo-server-core/dist/ApolloServer.js
@ ./node_modules/apollo-server-core/dist/index.js
@ ./node_modules/apollo-server-express/dist/index.js
@ ./node_modules/@nestjs/graphql/dist/graphql.factory.js
@ ./node_modules/@nestjs/graphql/dist/index.js
@ ./node_modules/@nestjs/graphql/index.js
@ ./src/app.module.ts
@ ./src/sls.ts
WARNING in ./node_modules/subscriptions-transport-ws/node_modules/ws/lib/validation.js
Module not found: Error: Can't resolve 'utf-8-validate' in '/Users/odifisIH/Sites/cashmetrics/packages/core/node_modules/subscriptions-transport-ws/node_modules/ws/lib'
@ ./node_modules/subscriptions-transport-ws/node_modules/ws/lib/validation.js
@ ./node_modules/subscriptions-transport-ws/node_modules/ws/lib/receiver.js
@ ./node_modules/subscriptions-transport-ws/node_modules/ws/index.js
@ ./node_modules/subscriptions-transport-ws/dist/server.js
@ ./node_modules/subscriptions-transport-ws/dist/index.js
@ ./node_modules/apollo-server-core/dist/ApolloServer.js
@ ./node_modules/apollo-server-core/dist/index.js
@ ./node_modules/apollo-server-express/dist/index.js
@ ./node_modules/@nestjs/graphql/dist/graphql.factory.js
@ ./node_modules/@nestjs/graphql/dist/index.js
@ ./node_modules/@nestjs/graphql/index.js
@ ./src/app.module.ts
@ ./src/sls.ts
ERROR in ./node_modules/chokidar/node_modules/fsevents/fsevents.node 1:0
Module parse failed: Unexpected character '�' (1:0)
You may need an appropriate loader to handle this file type.
(Source code omitted for this binary file)
@ ./node_modules/chokidar/node_modules/fsevents/fsevents.js 13:15-41
@ ./node_modules/chokidar/lib/fsevents-handler.js
@ ./node_modules/chokidar/index.js
@ ./node_modules/@nestjs/graphql/dist/graphql-definitions.factory.js
@ ./node_modules/@nestjs/graphql/dist/index.js
@ ./node_modules/@nestjs/graphql/index.js
@ ./src/app.module.ts
@ ./src/sls.tsthis is my webpack config : const webpack = require('webpack');
const path = require('path');
const nodeExternals = require('webpack-node-externals');
module.exports = {
mode: 'production',
entry: './src/sls.ts',
target: 'node',
module: {
rules: [
{
test: /\.tsx?$/,
use: 'ts-loader',
exclude: /node_modules/
}
]
},
resolve: {
extensions: [ '.mjs','.tsx', '.ts', '.js' ]
},
// externals: [nodeExternals()],
plugins: [
new webpack.IgnorePlugin({
checkResource(resource) {
const lazyImports = ['@nestjs/microservices', '@nestjs/platform-express','@nestjs/grahpql', 'cache-manager', 'class-validator', 'class-transformer', 'graphql'];
if (!lazyImports.includes(resource)) {
return false;
}
try {
require.resolve(resource);
} catch (err) {
return true;
}
return false;
},
}),
],
output: {
libraryTarget: 'commonjs',
filename: 'build.js',
path: path.resolve(__dirname, 'dist')
}
};im just using the graphql example from nestJS and is worth mentioning that without adding '.mjs' to the extensions list webpack will also complain: graphql/graphql-js#1272 thanks ! |
Where do you insert this? I'm struggling to find the option. An example snippet of surrounding code would help a lot. |
Under |
|
I would like to deploy a standalone bundle without any node_modules. If I understand correctly, the intended way of doing so would be to add Can someone confirm my following understanding of the work around suggested above:
Did I miss something or is there any plan to correct the |
|
@FrozenPandaz , I tried the solution you proposed in your comment. |
|
Come on, any updates on this issue? This is really urgent... |
|
I have just published a repo which demonstrates how one can bundle Nest and containerize it for Docker. It attempts to address the issues that most people are having here. You can find it here, cheers! 💥 Nest 🔰 Webpack 🔰 Docker 💥 |
|
I really don't like the solution with |
|
I'm just a humble member of the Nest community. Though, I do think we need to sympathize with the complexity of integrating many 3rd party systems together (such as Nest does), and how there can be problems that may not have eloquent solutions. Bundling and tree shaking code has been something the industry has been struggling with since the inception of modern Javascript (ES6+). Webpack and Rollup seem to be the current front running technologies of the industry. Historically, there was no exiting Javascript framework that managed to solve the bundling/tree shaking problem, due to the fact there were various competing Javascript module systems (Commonjs vs AMD vs ES6). Furthermore, there are framework specific module systems, such as Google's Angular module system, which is what Nest's own module system is inspired after. The Angular team took on the challenge to design a module system that would allow for the construction of a deterministic abstract syntax tree (AST). It contains the information needed to appropriately tree shake out unused code and determine what should actually be bundled. The industry has come a long way in a short amount of time, but there are still long standing problems to be resolved. This of course leads us to the issue of this thread. The problem lies deeper than just at the framework level (Nest), or at the bundling level (Webpack). Therefore, we should be sympathetic to the fact that we may need some explicit configuration for the time being. The industry as a whole needs more time to mature. Until then, all of us require some proficiency with configuring bundling systems like Webpack. |
|
In a lambda-context it would be really cool to use Nestjs, even for functions which do not expose a REST-API, but listen on different event-triggers...and for these lambdas we would only need |
|
@ZenSoftware Thanks for providing an example of bundling a Nest app. Is there a chance you could provide a similar example in the context of NX workspace? There's a bit of added complexity that has to do with specifying configuration via angular.json builders there that I don't quite understand yet. |
This approach worked for me, but I noticed my app failed when running with the following error: I had to remove the Just wanted to point out the error I ran into in-case someone was trying to get this to work. |
|
@DmitryEfimenko For a clean install of an Nx project, using the Angular + Nest schematic, most people seem to be using the I myself am having difficulties getting bundling working for our Nest app managed with Nx. The reason is due to the fact that we have 108 npm packages, and we're planning on adding more. Webpack is failing to cope with the complexity of the Javascript ecosystem. Ultimately, the lowest denominator is correctness. So we are excluding all of our dependencies and installing them in the Docker image instead. I spent the last 2 days trying to get bundling working. I couldn't produce reliable builds given the number of 3rd party libraries we are using. The builds seem to be missing modules here and there. At any rate, I've determined that the maintenance of the bundling process is not worth it. Ensuring correctness is more important, as to allow for the integration of new 3rd party libraries with agility. Bundling sort of becomes a bottleneck. Our Kubernetese pods are not as lightweight as they could possibly be, but computer science is full of trade offs... 😅 I published 💥 Nest 🔰 Webpack 🔰 Docker 💥 as a bare minimum example of what is involved in manually setting up a bundling system, without the complexities of many 3rd party libraries. It's actually not that much configuration. I think it is perfectly appropriate to invest in effort in getting bundling working. Stateless micro services are the first thing that come to mind. Small Docker image sizes are necessary for that purpose. You want to have minimal footprint to efficiently spin up replicas of Kubernetes pods to meet compute demands, and dispose of them efficiently. |
|
I tried with externalDependencies but I've got this error: |
|
I still have problem with Silly question: what kind of config you're taking about with those The solution with ignore plugin works, but there are 2 additional submodule imports that need ignoring ( Pasting for posterity 😄 // webpack.config.js
module.exports = config => {
const ignorePlugin = new webpack.IgnorePlugin({
checkResource(resource) {
const lazyImports = [
'@nestjs/microservices',
// ADD THIS
'@nestjs/microservices/microservices-module',
'@nestjs/websockets',
// AND THIS
'@nestjs/websockets/socket-module',
'@nestjs/platform-express',
'cache-manager',
'class-validator',
'class-transformer',
];
if (!lazyImports.includes(resource)) {
return false;
}
try {
require.resolve(resource);
} catch (err) {
return true;
}
return false;
},
});
config.plugins = config.plugins.concat(ignorePlugin);
return config
} |
ghost
mentioned this issue
|
I'm hitting the same error message as @mledour with the below project environment. I've experienced little-to-no problems with my SSR Angular app and it's Any direction towards anything that works would be greatly appreciated! |
|
As @vadistic, I'm still running into this issue in a cli generated monorepo. For some reason, it works well when using |
|
To recap and finally close this issue. In many real-world scenarios (depending on what libraries are being used), you should not bundle Node.js applications (not only NestJS applications) with all dependencies (external packages located in the For instance, if you try to build NestJS (or just express) application with MongoDB, you will see the following error in your console: Why? Because Similarly, if you try to build an application which consists of Angular, Express, and Postgres (with, let's say, TypeORM as ORM), you will see this error: Why? Because Lots of popular Node.js packages that are "required" to build a typical REST/GraphQL API depend on native bindings. Hence, you can't bundle them and produce a single executable JS file (which is easy for Front-End apps like Angular). If, despite everything I wrote, you still want to bundle dependencies, AND YOU USE NX there are 2 ways:
Nest CLI (when webpack mode is enabled
If you're lucky enough (libraries that you depend on don't rely on native bindings) and you follow the above instructions, you should still be able to bundle your application. |
I'm submitting a...
Current behavior
Nx project with a bare-bones [email protected] app does not compile when using the
--prodflag.Does not compile
ng build --prodCompiles
ng serveng buildwithout the--prodflag--prodflagExpected behavior
Should compile
Minimal reproduction of the problem with instructions
Nx & [email protected] bug reproduction
Environment
Build log
The text was updated successfully, but these errors were encountered: