Add zstd decompression support

[Jaifroid]

Implements #611.

I'm creating this PR for testing, as the code is now working well on two test ZIMs: wikipedia_bm_all_2020-07.zstd.zim and foo-zstd.zim. Please note that I have not yet compiled zstdec.js with optimizations and minification, but it is very fast even as is.

I have tested this on Edge Chromium 85.0, Firefox 80.0, Edge Legacy, and IE11 - all working. I have not tested it on Firefox OS simulator yet, but I have tested on Firefox ESR which is the same codebase as the simulator, and it works in Firefox ESR. It might need to be memory optimized before it will work on lower-powered mobile devices, and at the least the binary should be compiled with -O2 optimizations to make it smaller.

However, I would appreciate any testing at this stage, even while the PR is in draft form. I'll add links to the test ZIMs in a separate post on this PR.

@mossroy I've put you as a reviewer, but don't review the code yet, just do some tests if you have time.

[Jaifroid]

EDIT: Linked to wrong issue in comment above (now corrected) - I meant #590 "Switch to GitHub Actions instead of Travis".

[kelson42]

@Jaifroid Which zimfarm recipe would you like to have with zstd to secure testing?

[Jaifroid]

@kelson42 I guess for parity, maybe Ray Charles. Though I still would prefer to do tests as a separate issue because of lack of time right now, and it would delay this PR for quite a long time if it has to be part of this PR.

[kelson42]

@Jaifroid OK, I have put the Ray Charles recipe on Zstd, from October it will make ZIMs in zstd.

[mossroy]

I agree with you that, considering the urgency and the lack of a suitable ZIM file, implementing unit test can be done later in another issue. Could you create it? As we need to keep compatibility with both xz and zstd compressions, I think we should keep the current QUnit and UI test, and simply add some QUnit tests on a zstd file. I don't think #590 has an impact on QUnit tests.

The unused files should be removed from this PR before merging (and some may be moved to another branch) : zstddec.wasm, zstddec_*.js, test ZIM files?

[Jaifroid]

OK, thanks @mossroy. Will do. In view of the fact that Mozilla or other stores may want to see the unminified zstddec.js, I'll just keep one copy of that in the emscripten/zstandard compile folder, together with the final minified/optimized version we're actually using. I'm happy to add a wasm branch (which would be wasm-only) that can be used for performane comparison.

[Jaifroid]

Wasm branch (https://github.com/kiwix/kiwix-js/tree/Test-wasm-version-of-zstandard) and issue (#652) now added and wasm removed from this PR.

[Jaifroid]

All test files removed except: foo-zstd.zim, the full unoptimized build of zstddec, and the final optimized build. Source file and compile script remain.

This should be ready to squash and merge after more testing. I have ported the new code to Kiwix JS Windows to test with UWP and on mobile before merging here.

[Jaifroid]

Tested, working and stable on:

• A Windows Mobile VM of a 4 inch 500MB low-end device
• On a Lumia 950XL physical device

Both were tested by porting the latest code from Kiwix JS to Kiwix JS Windows and building the UWP app. (Back-end is identical in Kiwix JS Windows.)

Everything else I've tested on has been stable and I have not had any corrupted pages.

So tomorrow I will squash and merge this, unless anyone else says they want to do more testing.

[mossroy]

I don't think we should keep the unoptimized build of ztddec.
We did not have such file for xzdec and Mozilla did not complain about it.

[Jaifroid]

Ok, will remove.

[Jaifroid]

I've added the zstandard licences to the licences section of index.html. There was also inconsistency of spelling locally in that area between "Licence" and "License". Strictly, "licence" = noun; "license" = verb (like advice/advise), but US spelling tends to ignore the difference. I haven't attempted to apply this throughout the repo, however, as it would mean multiple changes unrelated to this PR, and it's not very important. I just thought I should explain the changes locally to that file for consistency of what the reader sees in that area.