Add application/javascript+module mime to remove ambiguity

[bmeck]

Right now whatwg/html#443 is overloading the application/javascript mime type for both script and module goals. This creates ambiguity in node if they share the same file extensions but cause different semantics. I suggest adding application/javascript+module and a different filetype to remove this.

[littledan]

We don't have any control over mimetypes here. I think IANA optimally, or maybe the WHATWG HTML repository in practice, would be a better place to go. WDYT about this proposal @domenic ?

[domenic]

See whatwg/html#558. This doesn't seem to add any value besides some theoretical notion of what MIME types are supposed to be used for.

[bmeck]

@domenic there is disagreement on that:

[1:54pm] ChALkeR: https://github.com/nodejs/node-eps/pull/3#discussion_r50746000
[...]
[1:54pm] ChALkeR: I don't want to see everyone using an unregistered `.someotherjs` filetype.
[1:54pm] ljharb: hm, k
[1:54pm] ChALkeR: Instead of *.js
[1:54pm] ljharb: why not?
[1:55pm] ljharb: people already do, with .jsx
[1:55pm] ChALkeR: Which jsx?
[1:55pm] ChALkeR: I know three.
[1:55pm] ljharb: the react one
[1:55pm] ljharb: but also, which js?
[1:55pm] ljharb: .js could be any number of things
[1:55pm] ChALkeR: js is registered.
[1:55pm] ljharb: it could be IE 6-specific JS
[1:55pm] ljharb: sure, .jsm or whatever would become registered too
[1:55pm] ChALkeR: No, jsx means three totally different things.
[1:55pm] ChALkeR: jsm is already used.
[1:55pm] ChALkeR: for js modules
[1:55pm] ljharb: but file extensions have never been and aren't going to be required for javascript, just the mime type
[1:55pm] ChALkeR: that are not ES6.
[1:56pm] ljharb: um, by who
[1:56pm] ChALkeR: firefox.
[1:56pm] ljharb: then they'll break it
[1:56pm] ljharb: things that only affect one browser don't matter
[1:56pm] ChALkeR: https://developer.mozilla.org/en-US/docs/Mozilla/JavaScript_code_modules/Using
[1:56pm] ljharb: mozilla has deprecated many other similar things before
[1:56pm] ljharb: like let and const, for example.
[1:56pm] ChALkeR: Switching from *.js to *.someotherjs would affect the whole ecosystem.
[1:57pm] ljharb: that's the idea
[1:57pm] ChALkeR: DEs and file editors would not get that by default.
[1:57pm] ljharb: so they'll update
[1:57pm] ljharb: the whole ecosystem should be able to move, gradually and seamlessly, to ES6 modules.
[1:57pm] ChALkeR: Some would not get it until it's IANA approved.
[1:57pm] ljharb: using ".js" runs afoul of that
[1:57pm] ChALkeR: How does it?
[1:57pm] ljharb: i can guarantee TC39 can get it IANA approved on a fast track
[1:57pm] ljharb: so that's literally a non-issue
[1:58pm] ljharb: because if we use .js how can we differentiate between a CJS and ES6 module
[1:58pm] ChALkeR: Are you sure that you get a new file extension and mimetype TC39-approved?
[1:58pm] ljharb: "parsing" won't work
[1:58pm] bradleymeck: ljharb: https://github.com/tc39/ecma262/issues/322
[1:58pm] ljharb: because `console.log('foo')` is a totally valid CJS and ES6 module.
[1:58pm] ChALkeR: And?
[1:58pm] ljharb: bradleymeck: yeah i saw that - TC39 doesn't deal with mime types tho.
[1:58pm] ChALkeR: ljharb: https://github.com/nodejs/node-eps/pull/3#discussion_r50746000
[1:59pm] ljharb: ChALkeR: and that's a nonstarter. it *must* be possible to differentiate.
[1:59pm] bradleymeck: ljharb: IESG does, but I can't go and tell IEGS to change things
[1:59pm] ChALkeR: ljharb: 'use module'.
[1:59pm] bradleymeck: TC39 might
[1:59pm] ljharb: ChALkeR: that requires parsing. "if imports or exports is found" requires a full JS parser. nonstarter.
[1:59pm] ChALkeR: ljharb: if there is no import, no export, and no 'use module' — treat as cjs.
[1:59pm] ljharb: the string pragma could work but *only* if that's the sole pivot
[2:00pm] ChALkeR: ljharb: Could you provide a usecase?
[2:00pm] ChALkeR: for the nonstarter.
[2:00pm] ljharb: sprockets: "do i send this file to the transpiler or not"
[2:00pm] ljharb: s/sprockets/any build pipeline that doesn't already have a JS parser
[2:01pm] rendar left the chat room. (Ping timeout: 245 seconds)
[2:01pm] ljharb: for example, airbnb transpiles with a separate node service
[2:01pm] ljharb: our rails app *simply can't* parse the JS to know if it needs transpilation as a module, and we'd want to call the service with different params if it is or not
[2:01pm] ChALkeR: Ah, so they want to decide early whenether they want to transplit or not?
[2:01pm] ChALkeR: And that's just for es6 modules?
[2:01pm] ljharb: i don't object to "use module" but that's not good enough if it can ever be omitted and still be considered an ES6 module
[2:02pm] ljharb: that's an example use case, yes
[2:02pm] ChALkeR: Why don't we have an own file extension for _every_ es6 feature that someone would want to transplit?
[2:02pm] ljharb: because modules are different
[2:02pm] ljharb: dead code elimination, dep graph analysis
[2:02pm] bradleymeck: ChALkeR: cause new features don't change semantics of old code generally, Module does
[2:02pm] ljharb: none of that applies to which ES6 feature you use.
[2:02pm] ljharb: there must be a declarative, no-JS-parse-requirement way to answer the question "is this an ES6 module or not". parsing isn't it.
[2:03pm] ljharb: "use module" might be, a new file extension absolutely is
[2:03pm] bradleymeck: I don't really care about the features it gives, I care that it has an ambiguity
[2:03pm] ljharb: and browsers would ignore all of that and solely use mime type
[2:03pm] ljharb: yes exactly - there must not be an ambiguity.
[2:03pm] bradleymeck: ljharb: browsers are voting no on new mime
[2:03pm] ljharb: eesh
[2:03pm] bradleymeck: they will go ambiguous
[2:03pm] ljharb: we'll see how that plays out tho
[2:03pm] ljharb: the loader spec can require it of them.
[...]
[2:04pm] bradleymeck: it might work for them since they don't have a module system, and they control what mime is accepted
[...]
[2:04pm] ChALkeR: ljharb: Could you please add those comments to https://github.com/nodejs/node-eps/pull/3#discussion_r50746000?
[...]
[2:05pm] konobi: maroons
[2:05pm] ChALkeR: A new registered mime type + extension could work.
[2:05pm] ChALkeR: But shifting the whole ecosystem to using an unregisterd file extension is a no-go.

[ljharb]

I'm not sure of the current status of the suggestion to use a different MIME type for ES6 modules in browsers, but my last understanding was that this was the way browsers would know to pivot on it. From reading whatwg/html#443 it seems that's not likely. Would users only use <script type="module"> to indicate if something is a module or not?

[meandmycode]

The reasons in chat seem to be a domain problem more than anything, whilst I can sympathise, the fix isn't really a fix.. say the new extension gets the go ahead and what? you can start using that suddenly? you could have used it all along within your problem domain.

To really solve your problem you need the ecosystem to agree, that source out of your control also plays into this scheme, but they won't.. why should they when the old default way exists and your domain problem doesn't affect them.

At the company I work for we've spent the last 24 months writing source as ES6 with modules, and back then the how of all that would work within the browser was very in the dark, even today only just happening, we solved it within our domain with "use modules", today this is mostly about speeding up transpilation but with apparently not so recent movement on how modules wont be self identifying we'll need to solve this anew.

[bmeck]

@meandmycode well the proposal in node is asking for a new filetype, and as such a new MIME nodejs/node-eps#3 . I am unclear on the "you could have used it all along within your problem domain." bit, if you could explain further.

[meandmycode]

@bmeck sorry, by that I mean, within your domain you could have used whatever file extension of your own all along to help the files self identify themselves to your build / runtime.

[bmeck]

@meandmycode people are expressing concerns of using an unregistered file extension, though we do realize that it would be possible to do so.

[meandmycode]

@bmeck yep, I realise that, but I think my point is that a new extension.. registered or otherwise really doesn't solve the problem described, say you used .jsm within your own domain, and you now include a modular js library that doesn't (more than within its rights since its a natural default), you are back to step one.

I feel by the time you would introduce and manage to convince an entire ecosystem of developers to follow a new extension, the original problem would have been long solved and we're left with another artefact.

I should specify this is just based on my own experience, as a consumer developer, so don't take any of this as gospel.

[bmeck]

@meandmycode our goal is to be compatible with the module specs around TC39 and WHATWG and try to avoid being limited to our single domain. The problem comes from gammar ambiguity. Feel free to discuss this in the Node proposal as well if you think we don't need a new filetype.

[domenic]

Closing this since it is not relevant to TC39. See https://github.com/tc39/ecma262/blob/master/CONTRIBUTING.md for what are good ways to contribute to this repository.

Individual communities can use whatever file extension or MIME type they want for their JavaScript. Neither WHATWG's HTML nor TC39's ES have any impact on this, except that HTML only accepts JavaScript MIME types to prevent e.g. reinterpreting an image or CSV as an executable script. (And even then, only for module scripts; classic scripts are looser.) It doesn't care about file extension. ES cares about neither and only provides top-level grammar productions to host environments.

If people want to discuss registering a new MIME type with the IANA, they can, but that doesn't impact any real world software and in my personal opinion should not be a goal of anyone if it doesn't have developer-facing consequences. And that discussion shouldn't happen on this issue tracker.

[trevnorris]

@domenic I'm not sure how this isn't a problem for the TC39. The current solution of depending on a DOM feature shuts out non-browser usage, and an alternative solution like using a different extension would break on the browser because it's an unregistered mime type. Thus shutting out scripts that could be compatible with both. So this move clearly shows there is no concern to support anything except the browser.

[domenic]

@trevnorris TC39 does not deal with file extensions or MIME types. That is why it is not a problem for TC39. For all TC39 knows your code is fed to ParseModule and ParseScript via monkeys dictating it to the CPU through song.

Each host environment is responsible for figuring out whether to call ParseModule or ParseScript. HTML as a host environment has one solution. But any other environment can come up with any other solution. TC39 is agnostic to what your environment uses.

[trevnorris]

@domenic I'm not stating "this" as in the request for an additional mime type. I'm stating that there's been no thought as to a reliable/spec compliant way to detect a module. Thus removing avenues that allow the same file to be detected in browser and non-browser situations.

[domenic]

TC39 has decided that the host environment can choose to detect a script or module depending on host-specific hints. TC39 just provides the two entry points and allows a given string to be parsed by both of them. The host environment can restrict certain strings from certain sources to only go through one entry point or another, but that's up to them. Really, you have complete freedom here, and should be glad you don't get your choices constrained by a standards body.

This has been discussed many times on es-discuss over the years. If you would like to reopen those discussions, es-discuss is the place to do so.

[ExE-Boss]

If anything, this should be application/module+javascript, as the bit after the + sign in a MIME type is how the file is actually encoded (hence application/xhtml+xml or image/svg+xml are both interchangeable with application/xml)

[WebReflection]

@ExE-Boss thoughts on how application/module+wasm would work under the same logic?

[ExE-Boss]

@WebReflection application/module+wasm has a suffix of +wasm and therefore it’s a module compiled as application/wasm.

[WebReflection]

@ExE-Boss so it would work and there would be nothing to do for anyone, right ?