chore: tune test_rss_used_mem_gap

[kostasrim]

It appears that newer versions of the gh runner require more memory. Some cases of the test test_rss_used_mem_gap allocate more than 6.5-7 gb of memory leaving barely 0.5gb to the gh runner (7.5 in total available) which sometimes cause the instance to run out of memory. This produces the following two cases:

1. OOM-killer gets triggered and manages to iterate over all of the processes and kill df (because it consumes the most memory and the process shall have the highest rank for the algorithm) before it runs out of memory.
2. OOM-killer gets triggered but does not manage to kill df because the system run out of memory.

Case 1. will cause the test to fail and case 2 will cause the instance to freeze. Rather unfortunate for both cases.

This PR addresses this by tuning the test, lowering its memory requirements and also shielding dragonfly by setting maxmemory to 5gb such that the test fails if we reach close this number instead of making the machine (and the CI) unresponsive.

Bonus point is that we trim the total running time of the test by a few minutes.

[kostasrim]

https://github.com/dragonflydb/dragonfly/actions/runs/11440024014