Possible race condition causing random failure of WebP polyfill loaded from the ZIM (SW mode)

[Jaifroid]

Following initial investigation in #670 (comment), there is probably a race condition causing random failures in the WebP Polyfill loaded from the ZIM in browsers that support Service Worker mode but do not support WebP (e.g. Firefox 64, which can be installed as a portable package for testing this issue).

The main symptom is that images will fail to be converted by the Polyfill fairly randomly, usually a whole page at a time. It is usually possible to load and convert the images by visiting another page and then returning to the page that evinced the failure. This kind of random and inconsistent failure suggests a race condition.

Once the cause is discovered, if relevant, an issue should be raised on mwoffliner.

EDIT: "The ZIM" I am referring to is http://download.kiwix.org/zim/.hidden/custom_apps/wikipedia_en_medicine-app_maxi_2020-11.zim (which is the ZIM I have tested).

[kelson42]

I confirm aomething is fishy. A bug has been reported in iOS and I migrated it to MWoffliner because I believe the bug is there.... but I have not really a clue what to do next. See openzim/mwoffliner#1453

[Jaifroid]

@kelson42 From the browsers we support here, there is only one old one (we think) that supports Service Worker mode but does not natively support WebP: Firefox 64. All the others either support WebP natively, so don't load the polyfill, or else don't support SW, in which case they use the polyfill we provide in the Kiwix JS package, which doesn't have this race condition. I'd be interested in the outcome of the investigations from the iOS side.

[kelson42]

@Jaifroid The MWoffliner bug has been fixed and released, I wonder therefore about the pertinence of keeping this ticket open?

[Jaifroid]

@kelson42 I've just tested this in Firefox 64 (portable) using wikipedia_en_endless_maxi_2021-07.zim. On article pages (in SW mode, which is where we see the issue) it seems to load all images correctly at first, but after browsing several articles, the engine seems to give up and no images are loading. On the landing page I'm getting no images at all loaded, even after starting the app fresh with that page.

Was the fix applied to this ZIM? I see there is now an August endless I could test. Or can you tell me a good ZIM to test for this issue with the fix applied?

[kelson42]

This zim file has been created only a week after the PR has been merged. I strongly suspect it was the older version of MWoffliner which made it. Would you be able please to rerun the very same test with the most recent version of this ZIM file http://download.kiwix.org/zim/.hidden/endless/wikipedia_en_endless_maxi_2021-08.zim ?

[Jaifroid]

OK, will do, but it will take some time to download on the wifi I'm currently using!

[Jaifroid]

@kelson42 I've now downloaded the latest English-language endless (August), and it appears to be working well on Firefox 64 in SW mode. In this mode, the supplied webp decoder (from the ZIM) is used, and I have been browsing around several articles with many images, and have switched several times from SW mode to jQuery mode (which uses our custom solution) and back again, and I am not seeing any missing images. Everything seems to load fine. I don't see lazy loading in action, but that doesn't matter (it could be the old version of Firefox that can't do lazy loading). As far as I'm concerned this issue is fixed in latest ZIMs. Many thanks!

[Jaifroid]

PS Does sotoki have a similar solution, or do they use jpeg still?

[kelson42]

@Jaifroid To be confirmed by @rgaudin but AFAIK the answer is "yes". So can we close this ticket?

[rgaudin]

@Jaifroid yes, sotoki has a similar solution. If I recall properly I backported the glue I made to manipulate the polyfill from sotoki to mwoffliner (the one bundled with the polyfill was just rejecting requests on race conditions)

[Jaifroid]

I haven't seen this issue in a long time, so I would say it's confirmed fixed.