React Native: UI is unresponsive while inserting objects

[mhv1]

Hi!

I'm currently working on a React Native app using Realm JS which has been in production for a while now. So far everything has been working fine, however I've noticed that sometimes the UI is completely unresponsive while an insert transaction is being executed (both develop and production builds). This usually happens with large amounts of data, although I've also experienced this with as little as 50 records.

The app fetches a considerable amount of data when it first starts, this data is then stored and encrypted using Realm. The unresponsiveness can be between 2s up to 30 seconds! In the worst case users cannot do anything until Realm is done with the transaction, which can be irritating for them.

The data models being stored don't have a particularly large number of properties. The sample repository I provide below contains sample data which resembles what's used in reality.

Any tips on how to avoid this would be very helpful. I've searched around an tried multiple approaches with no luck. I've even tried libraries such as this: https://github.com/joltup/react-native-threads but they sometimes don't even compile or cause additional issues to my app.

Goals

Users should be able to navigate freely and use the app normally while Realm is executing a transaction.

Expected Results

The UI shouldn't freeze. Navigation and general app functionality should function normally.

Actual Results

The UI freezes, sometimes for noticeable long periods of time, until Realm is done inserting all records. As users need to access more data the issue becomes more noticeable and frustrating.

Steps to Reproduce

Simply running an insert transaction as follows:

realm.write(() => { sampleModels.forEach(model => { realm.create(Schema.name, model, UpdateMode.All); }); });

The problem is worse the more data records are inserted.

Code Sample

I have created a very simple React Native project which showcases the issue:

https://github.com/MHV1/RealmPerformanceIssue

The issue can be observed when pressing a button which will trigger the Realm transactions. The button animation will freeze. Attempting to press the additional button in the app while the transaction is in progress will cause no response whatsoever.

The amount of data being inserted is similar to that in the real app.

Version of Realm and Tooling

• Realm JS SDK Version: 10.4.0
• Node or React Native: React Native 0.64.0
• Client OS & Version: both iOS and Android, no specific version
• Which debugger for React Native: None

[cristianoccazinsp]

Most likely realm is 100% sync (no bridge) so while realm is running, JS/Native is fully blocked. Your only option is to batch expensive operations in timeouts I think.

[kneth]

You can try small transactions:

sampleModels.forEach(model => {
  realm.write(() => { 
    realm.create(Schema.name, model, UpdateMode.All); 
  }); 
});

Currently Realm doesn't support multi-processes, and despite the name react-native-threads utilitizes multiple processes and not threads.

[mhv1]

@kneth I did try that approach, however the performance is considerably worse, specially if we talk thousands of records.

For instance in the sample project I provided having a large number of realm.write blocks will freeze the UI for several minutes (70000+ records). So in this case a big transaction still seems to be the way to go. Also this makes sense after having a look at these issues with similar scenarios: #1002 #2499

I'm yet to try smaller batches as @cristianoccazinsp suggests, although I'm curious to see if this would indeed ease the UI thread load without taking longer to persist all the data.

Anyway, thanks your help so far! I'll try to find other approaches. Will leave this open for now and see if there's any other suggestions.

[hugo-chq]

one thing that seems to unblock the ui is running realm writes inside requestAnimationFrame,
one write per frame could take a while to complete, but ui stays interactable

edit: performance now seems better using one write transaction in one requestAnimationFrame and looping over all writes, instead of over multiple frames
note: not using the hermes branch

[mfbx9da4]

I am finding that simply creating a write transaction without inserting any objects takes ~20ms. This seems totally unacceptable, is this to be expected? @kneth (Tested on 10.20.0-beta.1 and 10.20.0-beta.5).

[tomduncalf]

@mfbx9da4 That is not expected, though it could be related to the performance regressions in the Hermes branch (#4443). I'll see if I can reproduce that here. Is the UI also unresponsive during this 20ms?

[mfbx9da4]

Well the JS thread is blocked because write transactions are synchronous so yes, all UI interactions which require JS are blocked during that time. The main UI thread is free so I'm able to scroll around but any meaningful interaction is blocked.

[tomduncalf]

Cool, I was just checking that this ticket was the right place for the report. I'll try to repro and let you know how it goes. It might be that we move this to the Hermes performance regression issue if that seems to be the culprit in your case.

[mfbx9da4]

Likely @hugo-chq workaround is not actually "unblock"ing the UI but rather deferring the blocking of the UI.

[mfbx9da4]

Unfortunately I have to use the hermes branch @tomduncalf because I am affected by this issue #3837

[tomduncalf]

@mfbx9da4 We're actively working to resolve the Hermes performance regressions. The ticket has more details, with Hermes enabled the regression is caused by internal Hermes code which times every call across the C++/JS boundary so we are waiting on a new RN release from Meta, with Hermes disabled the regression is on our side so we are looking at ways to mitigate that.

[tomduncalf]

Just to let you know I was able to confirm your findings @mfbx9da4, I see a transaction take ~20ms on our Hermes branch (with Hermes disabled) and ~1/2ms on our master branch. I suspect the root cause is the same as the other performance regressions (the new method for accessing the internal C++ objects is slower). I'll update you when we have something for you to try to resolve this.

[cristiano-linvix]

@kneth any solution on new realm v12?

the UI is crashing because of data entry
I'm inserting every 1000 records (total 100k+), but it's still blocking the UI.

Did I think of any multi thread alternatives?

[nirinchev]

What does it mean for the UI to be crashing? Do you yield the thread between insertions to allow for the UI to render? And how long does a single batch of 1000 documents take and on which device do you measure it?

[cristiano-linvix]

Hello, I'm sorry, Google translate...
The screen freezes on very large insertions.

The problem happens more on somewhat simpler Android devices.

I'm testing on this device: Samsung Galaxy A03 Core.
However, it also happens on an iPhone 13, but less frequently.

The data is received via the Websocket server, then inserted into the database, in this flow, the application hangs for more than 15 seconds, (in the websocket settings there is a timeout of 15 seconds for disconnection), and the app loses connection with the server websocket.

[cristiano-linvix]

One thing I found in other topics is something like opening 2 realms and using sync: true to do this synchronization.

This way, you could implement the data synchronization layer in a separate thread, for example.

However, I didn't find documentation to implement this, in any version of the realm. It looks like realm v12 is done differently.

Does it solve the problem?

[cristiano-linvix]

Hello guys, any ideas about this?

@kneth any solution on new realm v12?

the UI is crashing because of data entry I'm inserting every 1000 records (total 100k+), but it's still blocking the UI.

Did I think of any multi thread alternatives?

[nirinchev]

Can you try yielding the thread between every insertion? Something like:

for (var j = 0; j < 100; j++) {
  realm.write(() => {
    for (let i = 0; i < 1000; i++) {
      realm.create(...);
    }
  });
  await new Promise(resolve => setTimeout(resolve, 1));
}

I.e. let the UI take over between every insertion to make sure we're not hogging the js thread all the time.

[mhv1]

I completely forgot I had opened this issue. I did manage to fix this back in the day but forgot to update... Back then I seemed to be the only one in the whole world with this problem :P

So, basically I ended up using an approach along the lines of what others have been suggesting in this thread, like in the previous comment.

My trick was to leverage React Native's Timers API, more specifically the "requestAnimationFrame" method. Additionally, I had to come up with a way to split my data into chunks to make "create" transactions a bit lighter, instead of trying to create tens of thousands of object at once.

Her's how the code looks like:

function nextFrame() {
  return new Promise<void>(function(resolve, _reject) {
    requestAnimationFrame(function() {
      resolve();
    });
  });
}

async function storeRecords(records: Array<Record>): Promise<void> {
  try {
    const realm = await openRecordRealm();
    const chunkSize = 1000;

    if (records.length > chunkSize) {
       const chunkArray: Record[][] = [];

      for (let i = 0; i < records.length; i += chunkSize) {
        chunkArray.push(records.slice(i, i + chunkSize));
      }

      realm.beginTransaction();
      try {
        for (let i = 0; i < chunkArray.length; i++) {
          chunkArray[i].forEach(dbRecord => {
            realm.create(RecordSchema.name, dbRecord, Realm.UpdateMode.All);
          });

          await nextFrame();
        }
        realm.commitTransaction();
      } catch (error) {
        realm.cancelTransaction();
        throw e;
      }
    } else {
      realm.write(() => {
        records.forEach(dbItem => {
          realm.create(RecordSchema.name, dbRecord, Realm.UpdateMode.All);
        });
      });
    }
  } catch (error) {
    console.error("Something went wrong while inserting records into DB", error);
  }
}

Here we have two functions: "nextFrame" and "storeRecords". The second is the one that handles the Realm object creation logic:

First I split my data set into chunks, in my case I had around 10 000 records to store and splitting them into chunks of 1 000 seemed to yield the best results. You have to play around with this number as it depends on how complex your data is, i.e., I tried with chunks of 5 000 and I still experience performance issues.

Now, the key is in the "nextFrame" function, whose only job is to resolve a promise function before the next UI repaint. It is called inside the loop that goes trough the data chunks. This way I managed to sneak in 1 000 Realm "create" transactions in between animation frames.

So the "requestAnimationFrame" was meant to be used to draw animations but could be used in this kind of scenarios and will provide a consistent behaviour. More info in the official docs: https://developer.mozilla.org/en-US/docs/Web/API/window/requestAnimationFrame

Hope this makes sense and helps someone! I'm going to close this issue now, after almost 3 years, as it doesn't seem to be an issue on Realm itself.

[cristiano-linvix]

Can you try yielding the thread between every insertion? Something like:

for (var j = 0; j < 100; j++) {
  realm.write(() => {
    for (let i = 0; i < 1000; i++) {
      realm.create(...);
    }
  });
  await new Promise(resolve => setTimeout(resolve, 1));
}

I.e. let the UI take over between every insertion to make sure we're not hogging the js thread all the time.

Hello, thanks a lot!!!
I'll test this, let you know soon.

[cristiano-linvix]

I completely forgot I had opened this issue. I did manage to fix this back in the day but forgot to update... Back then I seemed to be the only one in the whole world with this problem :P

So, basically I ended up using an approach along the lines of what others have been suggesting in this thread, like in the previous comment.

My trick was to leverage React Native's Timers API, more specifically the "requestAnimationFrame" method. Additionally, I had to come up with a way to split my data into chunks to make "create" transactions a bit lighter, instead of trying to create tens of thousands of object at once.

Her's how the code looks like:

function nextFrame() {
  return new Promise<void>(function(resolve, _reject) {
    requestAnimationFrame(function() {
      resolve();
    });
  });
}

async function storeRecords(records: Array<Record>): Promise<void> {
  try {
    const realm = await openRecordRealm();
    const chunkSize = 1000;

    if (records.length > chunkSize) {
       const chunkArray: Record[][] = [];

      for (let i = 0; i < records.length; i += chunkSize) {
        chunkArray.push(records.slice(i, i + chunkSize));
      }

      realm.beginTransaction();
      try {
        for (let i = 0; i < chunkArray.length; i++) {
          chunkArray[i].forEach(dbRecord => {
            realm.create(RecordSchema.name, dbRecord, Realm.UpdateMode.All);
          });

          await nextFrame();
        }
        realm.commitTransaction();
      } catch (error) {
        realm.cancelTransaction();
        throw e;
      }
    } else {
      realm.write(() => {
        records.forEach(dbItem => {
          realm.create(RecordSchema.name, dbRecord, Realm.UpdateMode.All);
        });
      });
    }
  } catch (error) {
    console.error("Something went wrong while inserting records into DB", error);
  }
}

Here we have two functions: "nextFrame" and "storeRecords". The second is the one that handles the Realm object creation logic:

First I split my data set into chunks, in my case I had around 10 000 records to store and splitting them into chunks of 1 000 seemed to yield the best results. You have to play around with this number as it depends on how complex your data is, i.e., I tried with chunks of 5 000 and I still experience performance issues.

Now, the key is in the "nextFrame" function, whose only job is to resolve a promise function before the next UI repaint. It is called inside the loop that goes trough the data chunks. This way I managed to sneak in 1 000 Realm "create" transactions in between animation frames.

So the "requestAnimationFrame" was meant to be used to draw animations but could be used in this kind of scenarios and will provide a consistent behaviour. More info in the official docs: https://developer.mozilla.org/en-US/docs/Web/API/window/requestAnimationFrame

Hope this makes sense and helps someone! I'm going to close this issue now, after almost 3 years, as it doesn't seem to be an issue on Realm itself.

Hello, thanks a lot!!!
I'll test this, let you know soon.