[MER-2731] Remove max limits on Datashop download and add caching #4464

gastonabella · 2023-11-24T15:34:43Z

Second PR for improving the Datashop export feature (first PR #4459)

It introduces the following changes:

Process the Datashop export process in batches, showing the number of batches processed in real time.
Use a Cache for storing intermediate data.
Some UI/CSS improvements.

datashop_demo_2.mov

…imit

darrensiegel

Really nice work, the caching is going to have a big impact on performance here. Just a couple of things that need to be fixed

config/runtime.exs

lib/oli/analytics/datashop_export_worker.ex

gastonabella added 4 commits November 24, 2023 11:26

[MER-2731] Process datashop export in batches to remove max records l…

7a8470b

…imit

[MER-2731] Use cache on datashop export

56e6f5e

[MER-2731] Highlight selected sections on analytics view

ed75243

[MER-2731] Add some tests

27a3d4d

darrensiegel requested changes Nov 27, 2023

View reviewed changes

config/runtime.exs Outdated Show resolved Hide resolved

lib/oli/analytics/datashop_export_worker.ex Outdated Show resolved Hide resolved

[MER-2731] Fix env var import and optimize db calls on datashop export

b1c5cb5

darrensiegel approved these changes Nov 27, 2023

View reviewed changes

darrensiegel merged commit 5aed474 into master Nov 27, 2023

darrensiegel deleted the MER-2731-remove-max-limits-on-datashop-download-and-add-caching branch November 27, 2023 16:56

Provide feedback