Improve policies reload time

[Haryk96]

Hello everyone!

I am using Casbin in production with casbin-sqlalchemy-adapter==0.5 and i have recently updated to the newest version of casbin 1.17.6, from 1.16.4.

In my MySQL db there are around 20k rows of policies at the moment.

I have noticed that reloading policies takes significantly longer now on the newest version.

On version 1.16.4 it took around 1.3 seconds, and on the newest version it takes around 10 seconds on the same machine. Only difference is version of the casbin library.

Bellow are logs describing my issue as well as the code which you can use to simulate the issue.
The policy reloading is measured like so:

import casbin
from casbin_sqlalchemy_adapter import Adapter
import sqlalchemy as sa
from time import time

import sys
sys.setrecursionlimit(10000)

user = "<<user>>"
pw = "<<pw>>"
host = "<<host>>"
port = 3306
db = "<<db>>"
path_to_model = "<<model>>"

if __name__ == "__main__":
    _engine = sa.create_engine(f'mysql+pymysql://{user}:{pw}@{host}:{port}/{db}',
                                pool_pre_ping=True,
                                pool_size=10,
                                pool_recycle=1800)
    _adapter = Adapter(engine=_engine)
    casbin = casbin.Enforcer(model=path_to_model, adapter=_adapter)
    while True:
        start = time()
        casbin.load_policy()
        print(f'Reloading took {time()-start} seconds...')

And running the script above with different versions of casbin library yields these logs:

casbin==1.16.4

$ pip install casbin==1.16.4 --force && python3 test_reload.py
...
Successfully installed casbin-1.16.4 simpleeval-0.9.13
Reloading took 1.0550739765167236 seconds...
Reloading took 1.121232032775879 seconds...
Reloading took 1.0821707248687744 seconds...

casbin==1.16.5

$ pip install casbin==1.16.5 --force && python3 test_reload.py
...
Successfully installed casbin-1.16.5 simpleeval-0.9.13
Reloading took 9.400760173797607 seconds...
Reloading took 9.236943244934082 seconds...
Reloading took 9.638797044754028 seconds...

casbin==1.17.5

$ pip install casbin==1.17.6 --force && python3 test_reload.py
...
Successfully installed casbin-1.17.6 simpleeval-0.9.13
Reloading took 9.69035005569458 seconds...
Reloading took 10.186222076416016 seconds...
Reloading took 9.863605976104736 seconds...

I have a suspicion that this change between versions 1.16.4 and 1.16.5 might be causing the issue.

Here is the corresponding commit.

Also, when i tried to locally change this piece of code in the newest casbin back to
model.model[sec][key].policy.append(tokens[1:]), the reloading took 1 second again.

Is this normal and expected behaviour?
Thanks for any responses!

Best regards, Honza

[casbin-bot]

@techoner @Nekotoxin