-
Notifications
You must be signed in to change notification settings - Fork 30.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Provide some mechanism to conditionally and synchronously import modules (or just builtins) from ESM #52599
Comments
Given that it doesn't depend on the context, |
Thanks for the summary here, in terms of timelines I just want to summarize my own sense of things:
Perhaps in time we do hit all three, or at least two of the above. |
|
Something on const require = process.getBuiltin("node:module").createRequire(import.meta.url) And be no worse off than before for conditionally loading anything outside of the builtins, which is very handy. |
I think the only concern I have for a |
I think whether the access is module-based depends on #52575 (if policy is enabled, access control to builtins is different per module based on the policy. But then if policy is removed altogether...there is no need to worry about it. It's probably another reason to remove policy?) |
This comment was marked as resolved.
This comment was marked as resolved.
Actually thinking through the implementation details of how to implement per-module access control for |
I'm working on a proof of concept; my quick hack for now (until I can get all of it as ESM) is to use a $ cat patchProcess.cjs
process.createRequire = require("node:module").createRequire;
$ head ./built/local/tsc.mjs
var require, __filename, __dirname;
if (typeof process !== "undefined") {
require = process.createRequire(import.meta.url);
__filename = require("node:url").fileURLToPath(new URL(import.meta.url));
__dirname = require("node:path").dirname(__filename);
}
$ node --require ./patchProcess.cjs ./built/local/tsc.mjs -p ./src/compiler --diagnostics
Files: 213
Lines: 245632
Identifiers: 412592
Symbols: 256043
Types: 103601
Instantiations: 189818
Memory used: 496843K
I/O read: 0.01s
I/O write: 0.00s
Parse time: 0.90s
Bind time: 0.38s
Check time: 7.15s
Emit time: 0.00s
Total time: 8.44s Just throwing on |
So, that was a lot easier than I expected. $ cat testRequireESM.cjs // Inject this into the process so that TS can synchronously require stuff.
process.createRequire = require("node:module").createRequire;
const ts = require("./built/local/typescript.js");
console.log(ts.version);
$ node --experimental-require-module ./testRequireESM.cjs
5.5.0-dev
(node:33109) ExperimentalWarning: Support for loading ES Module in require() is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created) The only change to TS I need to make is to change our import extensions and fix up our system implementation to require Code is at: https://github.com/jakebailey/TypeScript/tree/its-esm |
Opened #52762 |
@nodejs/loaders |
Want to throw in a couple of non-builtins use cases I've come across, in case it's helpful:
|
In the first one, you are already importing For the latter, you can actually export from ESLint configs a Promise, such that you can conditionally resolve things in an async IIFE, which may also work for those cases. No matter what, though, you can always achieve loading of external modules even with just const require = process.getBuiltin("module").createRequire(import.meta.url);
// ... |
In the first case, I was referring to the require calls (exceptionally poor choice of words, sorry!). In both cases it didnt register that |
`process.getBuiltinModule(id)` provides a way to load built-in modules in a globally available function. ES Modules that need to support other environments can use it to conditionally load a Node.js built-in when it is run in Node.js, without having to deal with the resolution error that can be thrown by `import` in a non-Node.js environment or having to use dynamic `import()` which either turns the module into an asynchronous module, or turns a synchronous API into an asynchronous one. ```mjs if (globalThis.process.getBuiltinModule) { // Run in Node.js, use the Node.js fs module. const fs = globalThis.process.getBuiltinModule('fs'); // If `require()` is needed to load user-modules, use // createRequire() const module = globalThis.process.getBuiltinModule('module'); const require = module.createRequire(import.meta.url); const foo = require('foo'); } ``` If `id` specifies a built-in module available in the current Node.js process, `process.getBuiltinModule(id)` method returns the corresponding built-in module. If `id` does not correspond to any built-in module, `undefined` is returned. `process.getBuiltinModule(id)` accept built-in module IDs that are recognized by `module.isBuiltin(id)`. Some built-in modules must be loaded with the `node:` prefix. The built-in modules returned by `process.getBuiltinModule(id)` are always the original modules - that is, it's not affected by `require.cache`. PR-URL: #52762 Fixes: #52599 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Stephen Belanger <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]> Reviewed-By: Marco Ippolito <[email protected]> Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Zijian Liu <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]>
PR-URL: #52762 Fixes: #52599 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Stephen Belanger <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]> Reviewed-By: Marco Ippolito <[email protected]> Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Zijian Liu <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]>
`process.getBuiltinModule(id)` provides a way to load built-in modules in a globally available function. ES Modules that need to support other environments can use it to conditionally load a Node.js built-in when it is run in Node.js, without having to deal with the resolution error that can be thrown by `import` in a non-Node.js environment or having to use dynamic `import()` which either turns the module into an asynchronous module, or turns a synchronous API into an asynchronous one. ```mjs if (globalThis.process.getBuiltinModule) { // Run in Node.js, use the Node.js fs module. const fs = globalThis.process.getBuiltinModule('fs'); // If `require()` is needed to load user-modules, use // createRequire() const module = globalThis.process.getBuiltinModule('module'); const require = module.createRequire(import.meta.url); const foo = require('foo'); } ``` If `id` specifies a built-in module available in the current Node.js process, `process.getBuiltinModule(id)` method returns the corresponding built-in module. If `id` does not correspond to any built-in module, `undefined` is returned. `process.getBuiltinModule(id)` accept built-in module IDs that are recognized by `module.isBuiltin(id)`. Some built-in modules must be loaded with the `node:` prefix. The built-in modules returned by `process.getBuiltinModule(id)` are always the original modules - that is, it's not affected by `require.cache`. PR-URL: #52762 Fixes: #52599 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Stephen Belanger <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]> Reviewed-By: Marco Ippolito <[email protected]> Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Zijian Liu <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]>
How does |
I suppose packages like these can just do what they already do with the CJS loader…by monkey patching? (Not a great way to get it done but it gets the job done for now, just like until proper CJS loader hooks are implemented they will just keep monkey patching anyway…and arguably simply wrapping a method on process is less problematic than the magic they do with the CJS module loader). |
PR-URL: nodejs#52762 Fixes: nodejs#52599 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Stephen Belanger <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]> Reviewed-By: Marco Ippolito <[email protected]> Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Zijian Liu <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]>
`process.getBuiltinModule(id)` provides a way to load built-in modules in a globally available function. ES Modules that need to support other environments can use it to conditionally load a Node.js built-in when it is run in Node.js, without having to deal with the resolution error that can be thrown by `import` in a non-Node.js environment or having to use dynamic `import()` which either turns the module into an asynchronous module, or turns a synchronous API into an asynchronous one. ```mjs if (globalThis.process.getBuiltinModule) { // Run in Node.js, use the Node.js fs module. const fs = globalThis.process.getBuiltinModule('fs'); // If `require()` is needed to load user-modules, use // createRequire() const module = globalThis.process.getBuiltinModule('module'); const require = module.createRequire(import.meta.url); const foo = require('foo'); } ``` If `id` specifies a built-in module available in the current Node.js process, `process.getBuiltinModule(id)` method returns the corresponding built-in module. If `id` does not correspond to any built-in module, `undefined` is returned. `process.getBuiltinModule(id)` accept built-in module IDs that are recognized by `module.isBuiltin(id)`. Some built-in modules must be loaded with the `node:` prefix. The built-in modules returned by `process.getBuiltinModule(id)` are always the original modules - that is, it's not affected by `require.cache`. PR-URL: nodejs#52762 Fixes: nodejs#52599 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Stephen Belanger <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]> Reviewed-By: Marco Ippolito <[email protected]> Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Zijian Liu <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]>
PR-URL: nodejs#52762 Fixes: nodejs#52599 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Stephen Belanger <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]> Reviewed-By: Marco Ippolito <[email protected]> Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Zijian Liu <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]>
`process.getBuiltinModule(id)` provides a way to load built-in modules in a globally available function. ES Modules that need to support other environments can use it to conditionally load a Node.js built-in when it is run in Node.js, without having to deal with the resolution error that can be thrown by `import` in a non-Node.js environment or having to use dynamic `import()` which either turns the module into an asynchronous module, or turns a synchronous API into an asynchronous one. ```mjs if (globalThis.process.getBuiltinModule) { // Run in Node.js, use the Node.js fs module. const fs = globalThis.process.getBuiltinModule('fs'); // If `require()` is needed to load user-modules, use // createRequire() const module = globalThis.process.getBuiltinModule('module'); const require = module.createRequire(import.meta.url); const foo = require('foo'); } ``` If `id` specifies a built-in module available in the current Node.js process, `process.getBuiltinModule(id)` method returns the corresponding built-in module. If `id` does not correspond to any built-in module, `undefined` is returned. `process.getBuiltinModule(id)` accept built-in module IDs that are recognized by `module.isBuiltin(id)`. Some built-in modules must be loaded with the `node:` prefix. The built-in modules returned by `process.getBuiltinModule(id)` are always the original modules - that is, it's not affected by `require.cache`. PR-URL: nodejs#52762 Fixes: nodejs#52599 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Stephen Belanger <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]> Reviewed-By: Marco Ippolito <[email protected]> Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Zijian Liu <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]>
PR-URL: #52762 Fixes: #52599 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Stephen Belanger <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]> Reviewed-By: Marco Ippolito <[email protected]> Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Zijian Liu <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]>
`process.getBuiltinModule(id)` provides a way to load built-in modules in a globally available function. ES Modules that need to support other environments can use it to conditionally load a Node.js built-in when it is run in Node.js, without having to deal with the resolution error that can be thrown by `import` in a non-Node.js environment or having to use dynamic `import()` which either turns the module into an asynchronous module, or turns a synchronous API into an asynchronous one. ```mjs if (globalThis.process.getBuiltinModule) { // Run in Node.js, use the Node.js fs module. const fs = globalThis.process.getBuiltinModule('fs'); // If `require()` is needed to load user-modules, use // createRequire() const module = globalThis.process.getBuiltinModule('module'); const require = module.createRequire(import.meta.url); const foo = require('foo'); } ``` If `id` specifies a built-in module available in the current Node.js process, `process.getBuiltinModule(id)` method returns the corresponding built-in module. If `id` does not correspond to any built-in module, `undefined` is returned. `process.getBuiltinModule(id)` accept built-in module IDs that are recognized by `module.isBuiltin(id)`. Some built-in modules must be loaded with the `node:` prefix. The built-in modules returned by `process.getBuiltinModule(id)` are always the original modules - that is, it's not affected by `require.cache`. PR-URL: #52762 Fixes: #52599 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Stephen Belanger <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]> Reviewed-By: Marco Ippolito <[email protected]> Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Zijian Liu <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]>
PR-URL: #52762 Fixes: #52599 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Stephen Belanger <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]> Reviewed-By: Marco Ippolito <[email protected]> Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Zijian Liu <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]>
`process.getBuiltinModule(id)` provides a way to load built-in modules in a globally available function. ES Modules that need to support other environments can use it to conditionally load a Node.js built-in when it is run in Node.js, without having to deal with the resolution error that can be thrown by `import` in a non-Node.js environment or having to use dynamic `import()` which either turns the module into an asynchronous module, or turns a synchronous API into an asynchronous one. ```mjs if (globalThis.process.getBuiltinModule) { // Run in Node.js, use the Node.js fs module. const fs = globalThis.process.getBuiltinModule('fs'); // If `require()` is needed to load user-modules, use // createRequire() const module = globalThis.process.getBuiltinModule('module'); const require = module.createRequire(import.meta.url); const foo = require('foo'); } ``` If `id` specifies a built-in module available in the current Node.js process, `process.getBuiltinModule(id)` method returns the corresponding built-in module. If `id` does not correspond to any built-in module, `undefined` is returned. `process.getBuiltinModule(id)` accept built-in module IDs that are recognized by `module.isBuiltin(id)`. Some built-in modules must be loaded with the `node:` prefix. The built-in modules returned by `process.getBuiltinModule(id)` are always the original modules - that is, it's not affected by `require.cache`. PR-URL: #52762 Fixes: #52599 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Stephen Belanger <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]> Reviewed-By: Marco Ippolito <[email protected]> Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Zijian Liu <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]>
What is the problem this feature will solve?
Thanks to #51977, requiring ESM is looking to be a real possibility. As such, TypeScript is considering transitioning over to ESM in the near future (depending on when
require(ESM)
is unflagged, hopefully in time for TS 6.0?), as that sort of change would no longer pose compatibility problems for the vast number of downstream CJS users. This has a number of benefits, mainly that we could finally share code betweentsc.js
and our public API without a startup perf regression, and that we wouldn't be duplicating code in our package (thankfully only two copies remain as of TS 5.5, down from six copies in TS 4.9).TypeScript's current public API bundle is intentionally "UMD-ish", detecting whether or not
module.exports
exists and using it (declaring a global otherwise), then later conditionally requiring built-in modules likefs
if we believe to be running within Node. This allows us to ship one single bundle that works in Node, browsers, and bundlers alike.However, the code that relies on conditional
require
is executed at the top-level as it's constructing thets.sys
object, the default "system" implementation for most of our APIs. Within CJS, this is fine, but within ESM, the only way to conditionally import something is by either:createRequire
fromnode:module
.Using top-level await breaks
require(ESM)
, the whole reason we think we can use ESM in the first place, and TS is infamously not async and couldn't import it later. Importingnode:module
is a moot choice, since if we could safely importnode:module
, we could have just importednode:fs
and so on.So, we need some mechanism to synchronously import modules, or at least the builtins.
Given
require(ESM)
is now possible, it sure seems like there could be a way to safely synchronously import modules in ESM that are already require-able from CJS after #51977.What is the feature you are proposing to solve the problem?
After discussing this in the TC39 Module Harmony meeting (with @guybedford @joyeecheung @JakobJingleheimer, others), there seemed to be a number of different paths forward:
import fs from "node:fs" with { optional: true }
) could be added; if the module fails to resolve, the imports are all undefined. This may require some sort of TC39 specing or proposal.import.meta.require
. This was previously proposed at Pull request opened for import.meta.require on core modules#130, but unfortunately drags CJS into the ESM world (potentially no more thancreateRequire
, I suppose).import.meta.importSync
orimport.now
, which is effectively justawait import(...)
that only works on sync-loadable ESM.import.meta.builtins
or similar (e.g. onprocess
), which just provide access to Node's builtin modules. Largely, TS only needsfs
,path
,os
, etc, so this would sidestep the "sync import" problem altogether. TS also conditionally importssource-map-support
, so that would not work, though only in development. Thankfully, since one could getnode:module
this way, you can also shimrequire
viacreateRequire
, which is pretty neat.What alternatives have you considered?
TS could also use
package.json
import maps to achieve "conditional" imports of Node-specific code, e.g. have an import like#system
which in the Node condition imports from Node, but is shims otherwise. This seems to have a number of downsides in my view, specifically:package.json
import maps. In the meeting, @JakobJingleheimer mentioned that one could remap imports likenode:fs
to shims, even todata:...
blobs, but I'm definitely not experienced enough in browser ESM to know how to do that.The text was updated successfully, but these errors were encountered: