iterating over projects takes over 30 seconds

[glensc]

as marge checks each project it is assigned one by one, and having over 50 projects assigned, the whole cycle (with no jobs found) takes ~2 minutes. how to solve this and make marge process assigned jobs sooner?

marge 0.6.1

# docker-compose logs |grep -E 'INFO Sleeping|INFO Finding out my current projects'
marge-bot_1  | 2018-03-26 20:06:22,903 INFO Finding out my current projects...
marge-bot_1  | 2018-03-26 20:08:14,924 INFO Sleeping for 0 seconds...
marge-bot_1  | 2018-03-26 20:08:14,924 INFO Finding out my current projects...
marge-bot_1  | 2018-03-26 20:10:07,329 INFO Sleeping for 0 seconds...
marge-bot_1  | 2018-03-26 20:10:07,330 INFO Finding out my current projects...

# docker-compose logs | sed -ne 's/.*INFO Fetching merge requests assigned to me in //p' | sort -u | wc -l
82

[JaimeLennox]

If you have many projects, then yes, it will take some time to iterate through all of them. It would be good to have some optimisations for Marge to work on different projects simultaneously (as mentioned in #69).

For now, a workaround would be to run multiple instances of Marge, each looking at a subset of your projects. As long as these don't overlap, they won't conflict with each other.

[glensc]

Thanks @JaimeLennox, i came to same conclusion for the workaround.

[fooishbar]

Another way would also be to use the root /merge_requests?scope=assigned_to_me&state=open endpoint, rather than iterating the list of every project Marge is a member of and then iterating the list of MRs for that project. That should make turnaround much quicker, and reduce the number of API calls as well.

[maikelvl]

@fooishbar Cross-linking for more info about that in #75 (comment)

[fooishbar]

@maikelvl Thanks for the pointer. Ultimately we just went with having a single instance per project (or group of related projects, at least), since mutual starvation was starting to become a problem, e.g. if you have two equally-active projects with 35min CI pipelines, then you could only merge one per project per hour ...