The plutus-pab-examples executable consumes a lot of memory and gets killed due to OOM 😱 😭 😵

[vladimirlogachev]

Summary

We have set up a server just to be able to follow the test-node instructions:

x86_64
12 CPU cores
24 GB RAM
80 GB swap

But the plutus-pab-examples executable seems to be killed due to OOM.

At block 11 752 the amount of swap used was 22 GB
At block 15 427 (slot 6 959 416) the amount of swap used was 72 GB
And that's far from completion:
According to the testnet explorer, at the moment of submitting the issue, the latest slot on testnet is 55 601 949, block 3 474 971

If it will help to find the problem, we can provide SSH access to the server.

Steps to reproduce the behavior

https://github.com/input-output-hk/plutus-apps/tree/main/plutus-pab/test-node

Actual Result

The pab examples sync process does not complete due to OOM

If it will help to find the problem, we can provide SSH access to the server.

Expected Result

The pab examples sync process completes

Describe the approach you would take to fix this

No response

System info

OS: Ubuntu
Version: 18.04
plutus-apps: v2022-04-06

[ghost]

Could you please share your observations if there are any? when the memory usage increases? after the integration test contract endpoint activation?

[vladimirlogachev]

I was just starting all 4 executables 1 by one, waiting for previous to sync, and the plutus-pab-examples is the last one.

At block 11 752 the amount of swap used was 22 GB
At block 15 427 (slot 6 959 416) the amount of swap used was 72 GB

I did not even try to start the integration test.
Please tell me your email/telegram/other contact, so that I could give you an access to the server.
It doesn't contain anything private at the moment.

[vladimirlogachev]

I've just tried running plutus-pab-examples with --rollback-history 1000, but I have no idea whether 1000 is a sane value or not.

It is still synchronizing more successfully, and memory consumption looks more healthy.

Current block: 26 939. Current slot: 8 253 200

UPD:

It's slow.

Current block: 47 207. Current slot: 20 223 898 (this last line of output did not change for at least 7 hours!)

And it's still possible to reach OOM soon...

[ghost]

Yeah, reducing the rollback history might help a bit but is not a general case. Working on the fix.

[agocorona]

I have the same problem but in a more modest docker instance with "only" 11 GB of RAM. I have been increasing the amount of RAM of the instance in successive tries until the maximum allowed in my 16 GB machine and the final result has been the same: the plutus_pab_examples being killed. All four executables: the cardano-node, cardano-wallet, plutus-chain-index and the pab examples running in the same docker container.

[waalge]

If it helps anyone: setting the dev options to the current slot, cleaning the contract db (rm my.db + migrate), and rerunning leads to a PAB service that is stable for a while. Enough for developing with at least.

[agocorona]

If it helps anyone: setting the dev options to the current slot, cleaning the contract db (rm my.db + migrate), and rerunning leads to a PAB service that is stable for a while. Enough for developing with at least.

To make it more explicit, unconment the lines here of pab-config.yml to:

developmentOptions:
  pabRollbackHistory: null
  pabResumeFrom:
    #{ "tag" : "PointAtGenesis" }
    { "tag" : "Point"
    , "pointBlockId" : " enter here the last block id from the log of the cardano node"
    , "pointSlot" : { "getSlot" : the slot }
    }

And now it seems to synchronize well. Following Lars lectures: https://youtu.be/X6AyZIZ0vaE?t=769

[agocorona]

I would like to know why the PAB app needs to sync
It may be important for the knowledge of the cardano blockchain.
Is it because of the possibility of deep rollbacks?

[vladimirlogachev]

up