Provide a workaround to enable ServiceWorker mode in Firefox Extension

[Jaifroid]

This is an explanation form #727 of what this would look like:

@mossroy I think I have a workaround for Firefox Extension users that allows them to access the Service Worker mode. I've done a bit of experimenting today which confirms that:

• Service Worker mode works fine in a Firefox Extension if the extension opens a tab that points to a remote PWA (served over https);
• If the PWA is coded correctly (to cache its own code), the extension works entirely offline (with Service Worker of course).

This means that I believe it would be feasible to adopt the same solution I've adopted with the UWP app, viz.:

• In jQuery mode the extension/app runs from local code (in this case from a moz-extension: URL);
• If the user turns on Service Worker, a banner can be shown which warns the user that one-time access to the secure server will be needed to enable SW mode, so that the app can cache its own code;
• If the user agrees, the app will access the remote server;
• On startup, if SW mode is still on, the app will bootstrap itself from local code to its cached (remote) code, effectively with a redirect or simply by opening the remote tab if it detects SW mode;
• If the user wishes to switch back to jQuery mode, the user will be alerted that the app will switch back to local code, and will be asked to give permission to proceed.

Below are some screenshots of how this works in the UWP app. I think it is a feasible model for Firefox Extensions. It would require #388 to be completed first. It is a workaround rathet than a solution, but one that is perfectly viable.

SWITCHING FROM JQUERY TO SW MODE:

SWITCHING BACK FROM SW MODE TO JQUERY MODE:

To be clear, apart from some controlling code (in backgroundscript.js) that detects the mode of operation and opens the appropriate tab, what this workaround would do is to act as a glorified bookmark to the PWA version. The clever work is done by the Service Worker and the Cache API, not by the extension itself. But for the user, this makes little difference other than that they will notice an https:// address in their browser instead of a moz-extension: address. Above all, this workaround is confirmed to work fully offline once the app has cached its code.

Originally posted by @Jaifroid in #727 (comment)

[Jaifroid]

A prerequisite for this is to decide where to serve the https: version from. The obvious place would be https://kiwix.github.io/kiwix-js , because we already keep that synchronized with our releases, so there would be no scope for the codebases to desynchronize. However, do all Mozilla users (I'm thinking of one in particular!) trust GitHub enough to use this? Would we be better off leveraging a docker container, like I do for https://pwa.kiwix.org ? It would probably need to be a different container, since the codebases are different,, so maybe https://pwa.kiwixjs.org ? But that would be more work than just using our existing implementation.

[mossroy]

We have to think about which lifecycle we want to have for this PWA version
If we have only one URL, upgraded each time we have a new release, like https://kiwix.github.io/kiwix-js, it means that people using any version of the extension will get the latest kiwix-js (which can be different from the version displayed by the extension, and used in jQuery mode)
It would seem better to me to have a distinct URL for each version of kiwix-js, so that each version of the extension can point to the corresponding version of the PWA.
It would ease support (we can rely on the version displayed by the extension) and make things more consistent between jQuery and SW mode. It would also allow our users to choose which version they want to use (for example if we drop support for old browsers at some point, and they can't upgrade their browser, if there is a regression etc)
It would force us to keep all subsequent versions of the PWA, but I don't think it's a big issue as they're very small.

Regarding the URL, we don't seem to own kiwixjs.org domain name, it's probably better to stick with kiwix.org.
I don't remember if the URL https://pwa.kiwix.org/ is hard-coded in some of your software? If it is (it seems to me it is), we will have to keep it as-is. Else we could use for example https://pwa.kiwix.org/kiwix-js/3.2.0/ and https://pwa.kiwix.org/kiwix-js-windows/

Another approach would be something like https://pwa-kiwix-js.kiwix.org/3.2.0/ but it would need to handle another SSL certificate (except if we have a wildcard one?)

@kelson42 : what would you prefer/advise? The most important is that whatever URL we choose, it will be hardcoded in the extensions, so we won't be able to change it afterwards.

It might be possible to deploy all versions on github instead. I tend to prefer kiwix.org (more flexible, and trusted), OTOH some countries might block access to kiwix.org domain name, where it might be more difficult for them to block github.com without impacting their companies business.

[Jaifroid]

@mossroy That's an interesting idea, to have frozen versions. It would certainly be possible to do that with kiwix.github.io/kiwix-js/ (we simply create the appropriate directories and copy the files in).

https://pwa.kiwix.org isn't hard-coded in development code, but it is hard coded in prior versions of the software. I have a parameter in init.js, with which I can switch my dev code to use the kiwix-js-windows GitHub pages, and switch back when a new release is done and the docker container updated, but I can't switch all the old instances except by doing a redirect on the server. I can redirect pwa.kiwix.org to pwa.kiwix.org/kiwix-js-windows/ (it is currently redirected to pwa.kiwix.org/www/index.html).

The main issue I see is that we can't easily have two repos re-building the same docker container. It could create confusion rebuilding it at the wrong time, since a rebuild would have to include data from both repos (I'm not even sure if that can work). IMHO it would be easier and cleaner to have a separate docker container for kiwix-js. How about using https://kiwixjs.kiwix.org instead? I don't think it has to be advertised as a PWA, as it will only be used (at least currently) in the Mozilla extension. If later we converge the two code-bases (that would be quite difficult now!), we could produce a unified PWA at https://pwa.kiwix.org.

Let's see what @kelson42 advises about naming.

[Jaifroid]

Another possible name, a bit cryptic, is https://js.kiwix.org . But it's also an interesting point that https://kiwix.github.io is a bit harder to block.

[Jaifroid]

Other possibilities: https://app.kiwix.org or https://webapp.kiwix.org.

[Jaifroid]

@kelson42 Any views on this issue? To summarize, to enable Service Worker mode in the Firefox Extension, we need to serve the app from a secure server (https://) at least one time, so that the Service Worker can cache the app's own code securely. Thereafter the app will work offline in SW mode, but will auto-update (from the same server) if the code is changed and it is online. In practice we might freeze the app version, so it will not update this way, but we can't stop it checking the server when online. (It will always be possible for the user to switch back to (or stay with) 100% local code if they do not want this, but with loss of features.)

The issue is where to serve the app from. There are various possibilities:

• GitHub pages (e.g. https://kiwix.github.io/kiwix-js/v3.3/ )
  • Advantages: easy, no docker build, not easily blockable by entities who might not like free knowledge
  • Disadvantages: some die-hard users might not trust GitHub due to its association with big tech
• A kiwix.org address (suggestions: https://js.kiwix.org, https://app.kiwix.org, https://webapp.kiwix.org, .......)
  • Advantages: trustworthy
  • Disadvantages: easier to block by hostile regimes; requires Docker configuration (not a big issue, we already have it for KJSW)

Can you help us decide?

[Jaifroid]

@kelson42 Did you have any advice for us, or do you want us to decide? See my summary in the comment above this one. There are pros and cons which you will know more about than we do.

[kelson42]

Thank you for your patience and sorry for the lack on feedback of my side here.

What I miss is a description of the following things available at:

• https://pwa.kiwix.org
• https://kiwix.github.io/kiwix-js
• The new stuff you want to put online to allow this workaround

Does all of this comes from the same repo?

For my (little) level of understanding, all of this should be basically the same!?

[Jaifroid]

@kelson42 Here are the descriptions:

• https://pwa.kiwix.org - this comes from the Kiwix JS Windows repo, so I don't think it can be utilized for this purpose. The codebase has diverged too much, and it is updated regularly based on tags released at the KJSW repo
• https://kiwix.gihub.io/kiwix-js - this is the existing gh-pages implementation of this Repo (Kiwix JS). It is updated (manually) after releasing a new tag. We could use this for the Firefox Extension workaround, but it may be better to have frozen versions of the code at https://kiwix.github.io/kiwix-js/v3.2/, https://kiwix.github.io/kiwix-js/v3.3/, etc. This is one of our proposals (it's easy to implement and doesn't require Docker). We think it has the advantage of not being easy to block by restrictive regimes, but may have the disadvantage of lack of a certain type of users' trust in github.io, or unwillingness to run code directly from from a github-controlled domain. Please tell us if you think this is not something we should worry about.
• https://js.kiwix.org/v3.2/ (for example) - this would be a new URL specifically for the purposes of the Firefox Extension workaround. It has the advantage of being (hopefully!) a trusted domain. It may have the disadvantage of being easier to block.

Really, we would just like you to say from the Kiwix perspective, what would be the preferred URL for the Extension to access (when switched to Service Worker mode): 1. https://kiwix.github.io/v3.2/ or 2. https://js.kiwix.org/v3.2/ (or another similar name in the kiwix.org domain). We do not currently envisage this being published as a separate PWA. It is for internal use within the Firefox extension (but it will be visible to users).

[kelson42]

IMO publishing on Github.io might be the best solution... but not sure how you could elegantly solve the versioning issue.

[Jaifroid]

Thank you @kelson42. We have full control over subdirectories of https://kiwix.github.io/kiwix-js/, so versioning shouldn't be problematic I think. We just replicate the required code from master into a subdirectory during the release and testing phase, and everything should run relative to that subdirectory. We would end up with (eventually) a number of old versions, but perhaps they could be deleted after a set number of cycles.

[Jaifroid]

@mossroy Maybe we can weigh up, when we next meet, the pros and cons of freezing the code version vs allowing the Service Worker to update the cached code when there is a new release (and the client is online). Since the Firefox extension auto-updates anyway (like a PWA) maybe the added complexity of versioning is unnecessary: we will never update the gh-pages version without releasing a new extension version, and the local vs online codebases run independently of each other (apart from the local code bootstrapping to the [cached] SW code and vice versa). Anyway, hopefully we can talk about this. There are advantages and disadvantages.

EDIT: Here is the code we need to implement (in the Service Worker) for the offline-first and self-updating logic:

https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps/Offline_Service_workers

The key to self-updating is to ensure a hard-coded version number is added to the Service Worker's javascript (this will also be used to set the Cache version, and delete obsolete versions of the Cache). As long as there is a byte-difference between the currently running Service Worker and the online Service Worker, the SW will update itself in the background, re-cache all the files it needs for offline use, and will wait until the user restarts the browser or tab before taking over.

[mossroy]

@Jaifroid : sure, we can discuss this when we meet.

[Jaifroid]

After #771 we'll need to implement the following:

• In ServiceWorker mode, the external links are opened inside the iframe #404 (possibly via Use generic event listener for link processing #732)
• Add a step in our CI to check the list of pre-cached files for the PWA-ServiceWorker #772

Once the above are complete, we may be able to implement #196, though I think we should probably have a bedding-in period with the SW mode available as an option in the extension before considering how and whether to make it the default (it is somewhat complicated for the Mozilla extension, because it involves the user giving permission to access the remote server).

[Jaifroid]

This issue is complete. For next steps see #196.