Introduce npm+webpack to build the app

[mossroy]

It would have some advantages. At least :

• ability to use recent javascript APIs (like ES6), and use a transpiler like https://babeljs.io/ to generate a javascript code compatible with all browsers. It can be handy in some cases, but won't magically add features like ServiceWorkers to browsers that don't support it
• ability to code with TypeScript (or some other language) instead of javascript
• running the code through a local webserver is a standard feature of npm (and it's now needed on Firefox and Chrome)
• webpack can handle the dependencies without putting them in our source code (for the common ones like jQuery or Bootstrap), and could certainly replace requirejs
• it would probably enable some automatic reload of the browser window when modifying the source code
• it could probably replace the dirty shell scripts I use to generate the packages for each platform when doing a release

I thought about implementing this several times, but the following drawbacks kept me from doing it, so far :

• first of all, it's not a trivial change to implement : it would take quite some time
• it would be a significant change in the architecture, and would also imply to update the unit tests, Travis CI, github pages etc
• we would need to implement the source maps to be able to debug, and check that the dev tools of our browsers support them (it's probably not the case for some old browsers)
• some pros of using this stack (like minification, handling of browser cache by creating a new javascript file name on each build etc) are not relevant for us, and could on the contrary make our daily work more complicated. But we probably can disable the features we don't want
• setting up a development environment will be a bit more complicated as it involves installing nodejs, letting npm download the libraries etc.

[kelson42]

I strongly support this

[Jaifroid]

This guide seems useful (much appreciated in the comments):

https://gist.github.com/xjamundx/b1c800e9282e16a6a18e

[Bam92]

This is very interesting. Moderne JS development requires at least these two technologies. What do you think?

[Jaifroid]

@Bam92 Yes. Personally I think the time has come for this, not least because it would enable use of frameworks that now expect this structure, automatic updating of dependencies, and possibility of using supersets of JS such as TypeScript, compiling for all targeted browsers with Babel (which makes use of antiquated dependencies like jQuery unnecessary). We believe it is a huge amount of work, however...

[Bam92]

@Jaifroid Yes, it is a huge amount of work. But I'd suggest migrating little by little.

[mossroy]

This might be some work for an upcoming hackathon

[Bam92]

Which hackathon do you mean here?

[mossroy]

I mean it's something I could work on in the following months

[Jaifroid]

Another good howto with webpack: https://github.com/petehunt/webpack-howto .

[Jaifroid]

This tool could be useful in transforming codebase from ES5 to ES6 (including modules / import / export):

https://github.com/lebab/lebab

It is "the opposite of babel", hence its spelling ("lebab").

These codemods could also be useful: https://github.com/5to6/5to6-codemod

[Krinkle]

ability to use recent javascript APIs (like ES6), and use a transpiler like Babel to generate a javascript code compatible with all browsers. […]

Which browsers are you looking to support? ES6 support is quite widely available nowadays. For example, from https://caniuse.com/es6-class and https://caniuse.com/arrow-functions

• Firefox: 45+ (2016)
• Chrome: 45+ (2015)
• Edge: 12+ (all versions)
• Safari: 10+ (2016)
• Android: 5+ (2014)
• iOS: 10+ (2016)
• IE 6-11: (unsupported)

I'm still very new to how Kiwix works, but as I understand it is used as browser extension or local server, not as website. Which means there's a certain amount of freedom in users deciding to upgrade. Which, if the data format remains compatible, should provide a good experience for users on the previous version for a long time. It could even remain offered as a download of that older version.

Looking at the README, however, it seems like maybe this won't be needed since all versions that have "store" entries or other downloads support ES6. And for IE11, it requires a manual installation, which users could do by checking out the relevant Git tag of the last version before this lands.

It might be worth considering as such, to follow the native web platform and adopt features as they come available. We can ship polyfills and shims for significant new APIs that older browsers don't yet have (e.g. Object.assign), but we could let the programmatic syntax be that of the newest we can safely use in all supported browsers - which could be ES6 today if we drop IE11 support for new releases.

[Jaifroid]

@Krinkle Thanks for your thoughts. Much appreciated.

I think the attraction of npm-webpack-bable for us is twofold:

• modern dependency management and updating
• ability to use the latest native JS features and not have to worry about writing workarounds for IE11+.

Yes, it adds a compile step, but usually this is a one-liner, and can be integrated into VS Code or similar for testing and development.

In terms of what we're supporting, currently we're keeping IE11, Windows Mobile (W10M) and an old Firefox OS on life support, but the main targets are indeed ES6-compatible.

But there are bigger structural issues in the code that represent potential watersheds of deprecation for older browsers that core-js/babel would not help with:

• we currently maintain two codebases within the single app ("modes"), one that uses DOM traversal to find assets and extract them from the ZIM (we refer to this as "jQuery" mode, but it doesn't actually require jQuery in principle), and one that uses a Service Worker to intercept browser requests for assets. Many of the older browsers you list do not support Service Worker;

• we currently use ASM instead of WASM because ASM is supported by all target browsers, but WASM is not -- however, we found in practice that ASM is "fast enough" in all browsers that support WASM - subjectively the difference is not appreciable;

• we currently have a a custom JavaScript implementation of many libzim APIs, which is time-consuming to maintain as libzim evloves (major pieces of work have been adding ZSTD support and now WebP Add webp image polyfill #650). There is a project to compile libzim with Emscripten (Compile libzim and kiwix-lib with emscripten in a more sustainable way #508), which is very tricky because of all the dependencies and an issue with reading very large files.

So we're under no illusions that npm-webpack-babel would solve everything, but it does seem like a good way to modernize the codebase and allow us to use new features of JS more freely, and above all to manage dependencies.

[mossroy]

What @Krinkle suggests here is to drop compatibility with the oldest browsers that don't support ES6. They could still run an old version of kiwix-js (that we could still provide). It would allow to code with ES6, and run this code directly, without transpiling.
It's true, and would of course simplify our work.
But the ZIM file format also sometimes evolves (zstd and webp support are very recent breaking changes, for example, and some other will come). This means they will not be able to open newer ZIM files with their old browser/device.

Kiwix-js is currently very conservative on browser support. If there is a reasonable way to keep support for old browsers, we try to do it. So if transpiling is not a big problem, we'll probably keep it.
If it becomes a real issue, we'll consider dropping some old browsers compatibility. We have to keep in mind that users of kiwix-js can have very old devices/browsers, that sometimes can't be updated and/or can't use another kiwix client

[Jaifroid]

OK, so I think the conclusion is: we want npm-webpack, but are not yet prepared to drop IE11 and other old browsers/webviews that are not compatible with ES6. Therefore, the npm-webpack configuration should ideally include core-js/babel and transpilation, or that can be a separate issue.

[Jaifroid]

Just stumbled across a more modern (and lightning fast) module / build solution appropriately called Vite:

https://vitejs.dev/

It has IE11 support via an official plugin (that harnesses babel, I believe).

It also supports TypeScript or vanilla JS. Comes bundled with a node-based dev server and production of course.

Interesting video about it here: https://www.youtube.com/watch?v=LQQ3CR2JTX8

[Jaifroid]

A document on how to convert a JavaScript project to TypeScript, and use TypeScript modules instead of RequireJS/AMD modules:

https://www.typescriptlang.org/docs/handbook/migrating-from-javascript.html

It might be possible not to have a bundler other than TypeScript itself, which can also transpile code down to ES5 natively. Obviously you then have to compile code in order to run it in a browser, but it's just an npm command.

[Jaifroid]

@mossroy We have various possible strategies outlined above, and I'm happy to keep adding useful info to this thread as I find things, but I wonder if we're in a position to choose a path forwards. I quite like the idea of a combination of TypeScript for type checking, and babel for fine-grained control over transpiling, though on its own TrypeScript engines can do most transpiling to ES5, maybe enough for our needs. I don't have any experience, though. To be clear, you can essentially keep writing JavaScript in a TypeScript project.

FWIW, MWoffliner uses TypeScript, as you can see from https://github.com/openzim/mwoffliner/tree/master/src . That could have some advantages for us in terms of shared knowledge of tooling.

[mossroy]

I don't have much experience on this either, so don't have a strong opinion.
I had played a bit with webpack in the past, which would certainly do the job. But vitejs looks very cool (I watched your video), and might be simpler for our needs.
I'm wondering if one of these tools could also ease the generation of our different packages (which have some slight differences in their content), instead of the dirty create_all_packages.sh script

Regarding the language (javascript vs typescript), I don't have a strong opinion either. To me, the main goal is to be able to use recent ECMAscript features with some native transpiling/polyfilling

[Jaifroid]

OK, maybe we can experiment a bit with Vite in a branch. There's a guide on converting an existing project here:

https://css-tricks.com/adding-vite-to-your-existing-web-app/

The guide is for people using webpack, but the concepts should be similar. It seems as if stripping out ReguireJS and changing our define statements to import statements using ES6 module syntax, and similar for export statements, would be the first step. If it runs with native modules in a modern browser, then adding vite as a fast transpiler ought to be relatively easy (famous last words).

[mossroy]

This month is quite busy for me, but I follow you

[Jaifroid]

I also probably can't do anything on this till end of month, but it's good to have a notional path to experiment with even if only mentally.

[Jaifroid]

It looks like Vite has also developed a very fast testing framework called, wait for it..., https://vitest.dev/

[Jaifroid]

Some more useful Vite examples here: https://dev.to/stormkit/migrating-stormkit-from-webpack-to-vite-5429.

[Jaifroid]

Another guide: https://dev.to/tythos/porting-from-requirejs-to-es6-ip

[Jaifroid]

OK, having made a big effort to remove RequireJS from the Kiwix JS Windows/Linux version in kiwix/kiwix-js-pwa#393 and use modern module imports instead, I've had a very frustrating time trying to find a compatible way of packaging the modules. Everything works fine in a browser that supports ES6 modules natively.

I tried vite.js: Its dev server worked fine after configuration, but it uses rollup to bundle the modules, and that makes a hugely mangled mess of our code when bundling for production. Virtually nothing works and the code is a mess of "legacy" and non-legacy modules. It's very fast, but way too aggressive.

So, I backtracked and tried babel transpiling. As long as I usse the --copy option, it correctly transpiles the code back to... Common JS modules. But it doesn't then add any way to run those modules, as it bizarrely assumes that Node's require function is available. Searching on the Web suggests that you have to include RequireJS.... which I spent a lot of time removing. Now, it would be fine if it actually worked, and left the ES6 modules untouched for browsers that support them, but no such luck. Again, it's a mess.

Then I tried to integrate babel and browserify, which is recommended as a bundler that can enable modules to run in the browser for babel. While I can get browserify to convert and back the main entrypoint, the resulting file won't actually run when called from index.html, and the Service Worker is never loaded.

Finally, I tried supposedly trusted workhorse webpack. But, it turns out that webpack throws a major wobbly when trying to bundle our WASM files. There are several known bugs around WASM support in webpack, and no-one has a solution that I could get to work.

So I'm now a bit stumped. I could try TypeScript, but if the others can't work out how to transpile ES6 modules, there doesn't seem much hope that TS will be able to do it, as it's not intended to be a bundling solution.

[Jaifroid]

I tried esbuild, which looked very positive, and makes a bundle that nearly runs. However, it doesn't have support for dynamic imports, so there is no fallback from wasm to asm, and more importantly, if using a static import (of just one or the other), then it warns that it can't use import.meta, and indeed that fails at runtime and the app can't load the WASM.

[Jaifroid]

I've finally managed -- after a lot of fiddling and working around problems with core-js polyfills in IE11 -- to bundle a usable app for both modern and legacy browsers using https://rollupjs.org/ and its Babel plugin. 😊

Usable code is in kiwix/kiwix-js-pwa#393, with the app being built to the dist directory with npm run rollup. Note that our custom WASMs are working, but not yet the libzim WASM (this is not a blocker, it's simply because I haven't yet implemented the loading rewrite that is necessary to load libzim outside of the bundle).

Since Rollup is used as the bundler behind https://vitejs.dev/, there is hope for adding the Vite goodies as well, though for now I'm focusing on getting a fully working app bundle, and the dev front end can be added later.

When it's implemented, working, and fully tested in KJSWL, I'll backport here. 🍀

[Jaifroid]

The PR for this issue is #1025. More discussion there.