[Firefox] Use indexedDB instead of browser.local.storage for cached assets

[gorhill]

Issue first reported here: #2684 (comment).

Related bugzilla issue: "Very bad memory efficiency of browser.storage.local".

As per Firefox devs, one should not use browser.storage.local API to store large amount of data, but rather indexedDB should be used.

The change here will also seamlessly migrate all cache-related entries from browser.storage.local to indexedDB.

[gorhill]

I verified that the (yet-another) migration work in the following cases:

When installing webext-hybrid the first time:

                       +-- webext storage (user settings)
                       |
legacy storage (all) --+
                       |
                       +-- indexedDB (cached assets)

When installing webext-hybrid or webext (when legacy storage had already been migrated):

                       +-- webext storage (user settings)
                       |
webext storage (all) --+
                       |
                       +-- indexedDB (cached assets)

I would like this to be tested by other people before I push 1.13.11rc1 to the dev channel of AMO.

I did not test on Firefox for Android.

[quilloss]

Works fine on Firefox 56. However won't this break for private browsing as https://bugzilla.mozilla.org/show_bug.cgi?id=1313401 is not yet fixed.

[gorhill]

However won't this break for private browsing

I tried using a "Private Window" yesterday and it worked fine.

However now I just tried the full "Private Browser" mode in preferences, and it does not work, uBO is not allowed to use indexedDB at all. Consequences of this:

• The stock filter lists will be always obsolete (the packaged versions) at launch time.
• All filter lists which are not packaged into uBO have to be downloaded at launch time.
• All filter lists must be recompiled at launch time.

Concretely, this will lead to longer load time for uBO, and with obsolete packaged filter lists, when launching Firefox in full privacy mode.

But I have no choice, as per Firefox devs: https://bugzilla.mozilla.org/show_bug.cgi?id=1371255.

[zero77]

Is this also a problem in uMatrix.

[streetwolf]

Ray... Is the new IDB now in "C:\Users\garys\AppData\Roaming\Mozilla\Firefox\Profiles\hrzfisk4.gary\storage\default\moz-extension+++41ac9a13-9e45-474b-8948-32f7217fe7f8"?

If so is "C:\Users\garys\AppData\Roaming\Mozilla\Firefox\Profiles\hrzfisk4.gary\browser-extension-data\[email protected]\storage.js" replaced by the above? The storage.js file is still being updated. I'm using the pure WE version of uBO

[gorhill]

Yes, my observation is that the indexedDB storage is at a location like [profile path]/storage/default/moz-extension+++[random]/. The browser.storage.local is at [profile path]/browser-extension-data/[email protected]/storage.js.

As explained above in the ASCII drawing, cached assets (large chunks of data) are in the indexedDB, everything else (all user settings which can be exported in a backup file) is still in browser.storage.local.

[gorhill]

By the way, for reference purpose, my storage.js is ~50K, while my moz-extension+++[...] is about 20MB. Default settings/lists. Keep in mind the size of indexedDB may vary, it's sqlite-based (just like uBO/legacy), and for efficiency purpose the storage might have "holes" in it sometimes causing it to be larger than the sum of data in it. It's normal and nothing to worry about.

[gorhill]

Another note about storage.js size vs. moz-extension+++[...] size, to avoid needless worries about their sizes: uBO 1.14.0 migrates data to indexedDB in an opportunistic way, meaning it transfers from browser.storage.local to indexedDB only when the data is actually fetched by uBO. This just makes the migration process efficient, and the size of storage.js may still be large if uBO loaded from a valid selfie. Eventually all will be migrated.

[streetwolf]

Thanks for all the good info Ray. I too notice that storage.js is a lot smaller now.