shanoir-issue#2560: stats export fails on big request

[pierrehenri-dauvergne]

With this parameters:

Study to include : Cohorte Principale.*|CISCO.*|Cohorte VHD.*
Subject to exclude : ^TestDummyRun.*|^test.*

The statistics export gives an empty file (doesn't even contain the headers).

With others parameters such as:

Study to include : Cohorte Principale - CHU Fort-de-France

It is working well

This PR is used to deploy and test on ofsep-qualif

This error is thrown:
o.h.e.jdbc.spi.SqlExceptionHelper - Java heap space

I think it happens in DatasetRepositoryImpl.queryStatistics method, on this line:
List<Object[]> results = query.getResultList();
Filling the List with 1+ millions line is filling the java heap space. A possible solution could be to increase the java heap space at application startup with something like this: java -Xmx1024m -Xms512m -jar app.jar but that wouldn't solve the issue for other bigger requests.

[michaelkain]

Hi @pierrehenri-dauvergne, please use only one sql file for both procedures, if possible. I have fear the next developer will adapt the getStatistics and forget to adapt getStatisticsSize at the same time. If all in one file, this should be to avoid. Cheers, Michael

[michaelkain]

very close to finish

[michaelkain]

One more thing: please adapt only the 3_add_statistics_procedure.sql, that concerns the ofsep-stats. The 4_add_studyStatistics_procedure.sql does not have to be impacted by your PR. So 3_add_statistics_procedure should contain getStatistics and getStatisticsSize. Thanks!

[michaelkain]

thank you! one minor item remains then we are ready, always better to use instead of an explicit close the try-with-resources, thx!

[michaelkain]

Perfect for the code now. I approve.

[michaelkain]

Hi @pierrehenri-dauvergne, can you please confirm, that it still runs on qualif? Thanks, if yes, I merge. Cheers, Michael