Firestore iOS WKWebView unreasonable increase in application data

[mohshraim]

[REQUIRED] Describe your environment

• Operating System version: iOS WKWebView
• Firebase SDK version: 5.0.4
• Firebase Product: firestore

[REQUIRED] Describe the problem

We notice a huge increase on application data when use firestore on hybrid iOS WKWebView application.
Almost Each time run the iOS application an increase in application data size from 1MB to 5MB; reach 140MB in a few hours while my original Firestore DB size not more than 1 MB with less than 100 doc.

notice that on each application load we didn't read and write more than 10 docs.

To detect the increment we download the device application container through xCode; at application container we found the source of increment on the following file; IndexedDB.sqlite3-wal
and this was the location of file at container:
AppData/Library/WebKit/WebsiteData/IndexedDB/file__0/firestore%2F[DEFAULT]%2FappName%2Fmain/IndexedDB.sqlite3-wal

according to https://www.sqlite.org/tempfiles.html

2.2. Write-Ahead Log (WAL) Files
A write-ahead log or WAL file is used in place of a rollback journal when SQLite is operating in WAL mode. As with the rollback journal, the purpose of the WAL file is to implement atomic commit and rollback. The WAL file is always located in the same directory as the database file and has the same name as the database file except with the 4 characters "-wal" appended. The WAL file is created when the first connection to the database is opened and is normally removed when the last connection to the database closes. However, if the last connection does not shutdown cleanly, the WAL file will remain in the filesystem and will be automatically cleaned up the next time the database is opened.

Any idea?

[mikelehen]

Hrm. Thanks for the report. That seems very strange and is unfortunately likely out of our control. We use IndexedDb for storage in a fairly normal way AFAIK and according to the docs you pasted, that WAL file should get deleted either on close or next open... so I have no idea why it would be growing unbounded unless there's some sort of bug in WKWebView.

FWIW- By default we try to call close() the IndexedDb database on the window 'unload' event, but that's not guaranteed to fire (or have enough time to finish). So you could try adding an explicit call to firebase.app().delete() which should also trigger the database close, and see if that makes any difference.

Sorry I don't have anything more helpful to offer at this point.

[mohshraim]

Thanks for suggestion

firebase.app().delete() doesn't work.

Also try to delete the file on native iOS the result was losing the saved data; while i was thinking that
(-WAL) files is just logs file, seems its not.

More Info: we are using the same environment of WKWebView at other projects using PouchDB, and didn't face any size problem before.

My next step id to test Dexie JS api for indexDB on same environment and see if the size will increase or not.
And i will keep you updated with result.

If anyone using WKWebView and firestore please check application data size, and feed us back if you have mentioned increase on size.

[mohshraim]

@mikelehen

I can confirm 99.99% the source of bug on WKWebView.

Just finish testing Dexie on same environment updating 50 doc each time; after running the app 10 times i had 5MB file with only 50 docs with one field.

Sent bug report to apple.

[mohshraim]

@mikelehen

An extra info for anyone

According to what i knew that iOS Chrome using WKWebView, so i make simple page using IndexDB

https://ssbyte.com/dbtest/

On my iPad just keep open and close chrome loading this page increase the Chrome Document Data by 20MB in a few minutes.

It Seems Chrome on iOS facing the same problem!!!!

[larssn]

We've had horrible IndexedDB for about a year now, in a Cordova app. We see no fix in sight, so our main recommendation is that people stay clear of it for now.

EDIT. I should add that I've seen horrible performance in Chromium (Mac) as well, see: https://groups.google.com/forum/#!topic/firebase-talk/u1y7pYVhc48

[mohshraim]

@larssn
For my case until now i didn't find any performance issues for IndexedDB on Mac or Windows.. testing was done on couple hundred of documents from 10 collection and load will be in 2 second.
Maybe you have big data; how many documents you have at local DB??
Or maybe you should check your code!!

And this issue was opened for iOS specific. and close it as it's related to iOS IndexedDB implementation.
https://bugs.webkit.org/show_bug.cgi?id=178204#c3

[larssn]

We're pretty sure its IndexedDB related. We'll do some more tests, but what we saw was this.
Doing a simple get({source: 'cache'}) (on a collection with few (<10) documents) took between 5-10 sec. After we deleted the database, it dropped to 500ms consistently.
We store orders, so we can quickly reach 100k documents (in total, across collections) and garbage collection for IndexedDB isn't implemented yet. So we're looking far and wide for alternatives.

[mohshraim]

@larssn
Have sent a dozen of bug reports to Apple about WKWebView, and read hundred of articles about
sql-wal file...
It seems this problem will not be solved on our environment with direct solution..

I worked on tow work-around.
1- on application start delay initialize Firesbase; for 2 seconds; this will enable sql-light to clear -wal files.
2- add button on your application to clear the -wal file...
this button should call firebase.app().delete()
then wait a few seconds (i wait for 5 seconds)
then reload your html application page...

-> for me i am using the second method, calling firebase.app().delete() was suggested before from mikelehen but wasn't working because i call it from iOS on application before terminate... which doesnt give enough time for iOS WKWebView sql-light to clear the -wal files

Hope this will help somebody and save his time
Regards,

[sengoontoh]

@mohsraim I see that this is fixed in webkit (and note your question on whether it will be in the next release). https://bugs.webkit.org/show_bug.cgi?id=191294

Did you find any other workarounds to this?

[mohshraim]

@sengoontoh
exactly my previous comment..
in other words
1- even you delay 1-2 seconds after your page load then init firebase..
-> you can do this every 10 days or whatever is your conditions
2- add button to your user that enable him to clear storage.
in button you should call firebase.app().delete() then wait 1-5 second then reload your home page..

the target from both solutions to free any listener on our IndexedDB while the page loaded... if we initialize the firestore directly will open IndexedDB connection and prevent GC from clearing -wal files..
also the second solution when you destroy firebase will free the connections and listener to IDB and your -wal files will be cleared..

iam working on second solution since few months and wait the fix to remove this button from App.

[sengoontoh]

@mohshraim I tried solution #1, waited up to 30 seconds before doing firebase.initializeApp and it still didn't clear the -wal file. Is there something else I need to do?

[mohshraim]

@sengoontoh
Nothing at all..
Try the other solution,, very easy

`     firebase.app().delete().then(function () {
            setTimeout(() => {
                    window.location.reload();
            }, 5000);
        });
`

on the other hand what is your environment?
i don't use (cordova,ionic,reactnative) ... i use WKWebView and load my html in

[sengoontoh]

@mohshraim Appreciate the code snippet. I'm using Ionic. Deleting the app database feels slightly heavy handed to me but as you explained, maybe do it once every 50 app opens or so. Thank you so much for pushing this bug all the way through Webkit! I can't believe it has been overlooked for so long.

[mohshraim]

@sengoontoh
note firebase.app().delete() will not delete your firesotre cached DB only clear a local file -wal
this used for transaction log on SqlLight.

On the other hand i open this bug all over the world, even on chrominume, Apple Support Program..
https://bugs.chromium.org/p/chromium/issues/detail?id=855647 this at chromium , they face the same problem on chrome iOS.
Apple is the worst when you report a bug, send 20 email report and wait for 5 months without a single reply then it delete it..
on webkit this the first report https://bugs.webkit.org/show_bug.cgi?id=178204#c3 a guy report it at 2017.