Enhancement: Autoloader

[mAAdhaTTah]

I actually migrated from SyntaxHighlighter to Prism because I much preferred the way Prism works as well as finding a number of the plugins quite useful. However, as Prism's plugin library has exploded lately (thanks so much @Golmote!), the goal of making it as lightweight as possible becomes harder to uphold if I have to to roll up all these languages into a single large file.

One of SH's features is an autoloader that will find the languages in use on the page and load up only the required language definitions, rather than requiring that all that language definitions be downloaded at once. I think a similar idea would be really useful for Prism. Few quick questions

1. Is this something that someone is already working on/underway?
2. If not, is this something you'd be interested in including?

I'm a relative beginner when it comes to JS, but I'd be happy to help out if I can, though I'd probably need some direction as to how we'd accomplish this, maybe using the components.js file somehow? Also might be a chance to think through how we expect users to use Prism w/ Bower and the whole mainBowerFiles question (#406), though I haven't really thought that through either.

So I figure I'd throw this out there, see what ideas you guys have, and see what I can do to chip in. Thoughts?

[LeaVerou]

I would definitely be interested in this functionality. Not so sure it should be in the Core though, maybe it would be better as a plugin.
We should also offer an easy link with all languages as separate files, cause the downloader creates only one JS file which is pointless with this approach.

[Golmote]

Couldn't we just add a link to the github components/ directory in the plugin description page?

[vkbansal]

IMO, requirejs can be used to accomplish this task easily

[Golmote]

I started thinking about this. How will the plugin know about the dependencies of the languages?
I hope we don't have to duplicate the languages list.

Here is a first try (that fails on loading the dependencies). Go is highlighted because C-like is bundled into the prism.js file used by the website. The failure can be seen on Objective C.

@LeaVerou Do you have any awesome idea?

(Here is the code of the plugin)

[vkbansal]

@Golmote Prism languages are basically modules that are dependent on prism-core.js. This problem can be solved using UMD as shown here https://github.com/umdjs/umd. This would allow users to use Prism with their favourite module loader.

[Golmote]

@vkbansal I don't understand how this could solve the problem of knowing the dependencies of a language from inside the plugin.

[vkbansal]

I'll explain AMD first. Lets take your example of GO and ObjectiveC.

The GO definition would be something like

define(['clike'] function(clike){
  /** Rest of the code **/
})

The Objective-C definition would be something like

define(['c'] function(c){
  /** Rest of the code **/
})

The C definition would be something like

define(['clike'] function(clike){
  /** Rest of the code **/
})

So when objectivec is required, it will load c and it in turn will check if clike exists and load it if required.

All the dependencies will be managed by the module loader. I can put a working example if you would like to see it in action.

[mAAdhaTTah]

Do we really wan to add a dependency on UMD for this? I think one of the principles of Prism is that it's self-contained.

[Golmote]

@vkbansal Your proposal is for including this feature in Prism core. Here, @LeaVerou explicitely stated that it should preferably be provided as a plugin.

But this kind of syntax might be used to solve the problem, maybe. I'll try something tonight.

[vkbansal]

@mAAdhaTTah The beauty of UMD is that it's backwards compatible and Prism can still be used like it's used currently.

[vkbansal]

@Golmote by creating a plugin, you'll be reinventing the wheel

[uranusjr]

If I understand currently, using UMD would mean that someone needs to maintain a set of definitions outside of Prism (or at least its core) to make sure that the dependencies are resolved correctly. I’m not sure that is a good idea. Prism already has its own way to record dependencies (in components.js), and unless it is to be deprecated or changed or what, having a new set of rules is duplicate logic and isn’t nice. It would be fine if someone wants to build something standalone, but not in Prism itself.

I can think of two ways to do this:

1. Just load components.js (or a subset of it) in somehow, like how autoloader_path currently works, and use it to resolve dependencies. I use this method in some of my projects to implement something similar (add script tags dynamically based on what languages are used in the page).
2. If a dependency is needed, lang would be undefined when Prism tries to find the language definition for extend. We can raise an exception right there, instead of failing later in the process. This wouldn’t affect existing code (unless the user does something funky with the undefined error we currently get when a language is not defined), and the autoloader can catch that exception and run load_script recursively until all dependencies are loaded (or fail completely if a script for the dependency is not found—which means something is terribly wrong in the user’s code).

Personally the second one has an extra appeal of making the error more explicit when I do something wrong (misspell the language name, for example). The current exception (TypeError: undefined is not an object (evaluating 'lang[key] = redef[key]')) is very puzzling when you first see it.

[vkbansal]

The plugin just solves one use case i.e. browser. What about nodejs? How does one handle that except for concatenating the files.

[Golmote]

@uranusjr

the autoloader can catch that exception

I think it can't, unless it overrides entirely the extend() function with something like:

var _extend = Prism.languages.extend;
Prism.languages.extend = function (id, redef) {
    try {
        _extend(id, redef);
    } catch(e) {...}
};

I think I'm starting to understand @vkbansal's point. This feature would require Prism to be much more asynchronous than it currently is.
Consider the case of the Go component: if C-like is asynchronously loaded somehow (because it is not found when extend is called), what happens to the last line in the Go component?

delete Prism.languages.go['class-name'];

At the time the line would be called, the grammar would not exist yet.

[LeaVerou]

I’m also not too fond of the dependency duplication in @vkbansal’s proposal, nor the UMD syntax.
I’d much rather reinvent the wheel than use some ugly complicated truck wheel that I don’t actually need and this has always been one of Prism’s main design principles. Extra long code that is full of design patterns is almost always terrible code, despite what its authors think.

[vkbansal]

@LeaVerou UMD is to support wide variety of platforms.

As an example, Prism does not work in Node.js directly. Anyone would except this code to work in NodeJS

var prism = require('path/to/prism/components/prism-core');
require('path/to/prism/components/prism-markup');

But, It will not. I found this, when I tried to write a script to compare the differences between regexps in Prism.js and the php port I'm working on. The solution was to concatenate the files and that too in particular order, which is what this issue is about.

load up only the required language definitions, rather than requiring that all that language definitions

[mAAdhaTTah]

The solution was to concatenate the files and that too in particular order, which is what this issue is about.

This is basically what I'm doing now, which is why I opened this issue, to see if there's a better way of doing this.

I was thinking something more along the lines of @Golmote's first take, and the asynchronicity was what I was going for. If I'm not mistaken, if we got a hook here:

https://github.com/LeaVerou/prism/blob/gh-pages/components/prism-core.js#L160

or where @uranusjr suggested:

https://github.com/LeaVerou/prism/blob/gh-pages/components/prism-core.js#L62

we'd be able to write in a plugin that hooks in, checks the language for the highlighted snippet, and if it's not loaded yet, loads it up. I think that would work, though I have no idea how to make that work across environments (browser, webworker, node).

Although (not to leave Node out in the cold), how much does that actually matter? Is rolling up a big file on the server a big deal? I wanted to split this out to lower the overhead for site load.

I'm still pretty new at JS, but I'd still like to take a shot at this if we can work out the general idea of how to make this work.

[Golmote]

This is harder than it looks. Everywhere in Prism, we use direct access to the languages (to create them, to extend them), we iterate through them (some plugins do this). All those operations would need to be rewritten in an asynchronous way.
Any kind of access to a language should be done through a Promise or something. IMO, this will not be easy from inside a plugin.

[mAAdhaTTah]

What if we approached it the other way: Instead of starting with core + plugin (assuming the browser for now), we just enqueue the autoloader plugin, which then checks what languages are on the page and loads them up before loading Prism?

[vkbansal]

@mAAdhaTTah Loading languages before Prism would not be possible (in its current state) as the languages are added inside Prism global object

[Golmote]

@mAAdhaTTah Then it would no be a plugin, properly speaking. But it could work, I guess.

Some issues will remain, though, for languages like Jade, which allows other languages to be embedded directly. (See the end of the file https://github.com/LeaVerou/prism/blob/gh-pages/components/prism-pug.js#L156)
An autoloader could not know what languages are to be loaded in advance.

[Golmote]

This should be fixed by now, since the Autoloader plugin was merged.

[mAAdhaTTah]

Awesome! I haven't had a chance to test it, as the plugin I'm using w/ Prism is going through some major BE refactoring. I'm also going to see what I do about pushing Prism towards more a async-friendly operation.