Refactor of the export csv button to export more than 500 documents max

[thomas-mauran]

Refactor of the export csv button to export more than 500 documents max

Refactoring the export csv function to know export more than 500 documents max

How should this be manually tested?

• Step 1 : open a kuzzle backend with data
• Step 2 : go on a collection with a lot of document
• Step 3 : try the export button

Other changes

deleted the function convertToCSV() from src/services/cllectionHelper.ts

[netlify]

✅ Deploy Preview for heuristic-goodall-67c4b1 ready!

Name	Link
🔨 Latest commit	22e3fed
🔍 Latest deploy log	https://app.netlify.com/sites/heuristic-goodall-67c4b1/deploys/63d3c86669f39900087866c9
😎 Deploy Preview	https://deploy-preview-993--heuristic-goodall-67c4b1.netlify.app/
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[Aschen]

Can you improve the PR title and also assign it to you please?

[sebtiz13]

Export doesn't work with huge collection (example: more than 10k documents).
The actual solution isn't really good, because the data are stored in client memory before conversion in blob to create download link, this can cause block the JS browser runtime, also this solution has an UX problem because with a large number of documents, user doesn't have indication of status (loading status or progress).

I think an better client solution it's to use Web Worker for non blocking process an split treatment in chunk.
IMHO the best solution, it's to implement a stream solution on backend and client just open a link, so the browser treat the download progress independently.

[Shiranuit]

Export doesn't work with huge collection (example: more than 10k documents). The actual solution isn't really good, because the data are stored in client memory before conversion in blob to create download link, this can cause block the JS browser runtime, also this solution has an UX problem because with a large number of documents, user doesn't have indication of status (loading status or progress).

I think an better client solution it's to use Web Worker for non blocking process an split treatment in chunk. IMHO the best solution, it's to implement a stream solution on backend and client just open a link, so the browser treat the download progress independently.

Using the export method is the proper way to download large amount of documents directly from the backend, this method is able to handle extremely large amount of data by streaming them directly from the backend, you should be able to download the file directly by letting the browser open the following url https://<kuzzle host>/<index>/<collection>/_export and configure every options of the export by putting things into the query params (lang, searchBody, scroll etc...)
The only thing I'm not sure how to do is how to provide the authentication token when going to the export url

[sebtiz13]

@Shiranuit It's not possible to provide the authentication token in GET params of url ?
If it's not possible due to security reason (understandable), one solution can be add an method to generate one temporary token (this token can contain the params of export) to provide in request. This temporary token are valid just one time.
IMHO it's the better solution to export securely without allow export spamming.

[Aschen]

The only thing I'm not sure how to do is how to provide the authentication token when going to the export url

If the SDK use the cookie auth then the auth cookie should be forwarded automatically no?

[Shiranuit]

@Shiranuit It's not possible to provide the authentication token in GET params of url ? If it's not possible due to security reason (understandable), one solution can be add an method to generate one temporary token (this token can contain the params of export) to provide in request. This temporary token are valid just one time. IMHO it's the better solution to export securely without allow export spamming.

@sebtiz13 Actually it's not possible to provide the authentication token directly in the URL, also it's not recommended even though query params are encrypted, the proper solution would be with cookies but it might not be possible if this is disabled by the backend

The only thing I'm not sure how to do is how to provide the authentication token when going to the export url

If the SDK use the cookie auth then the auth cookie should be forwarded automatically no?

@Aschen Yes it might work with the cookie but this will require the admin console to support login with cookies which is not the case right now, I dont think this will be that hard to implement, but this also means that if the cookies are disable on the Kuzzle instance the export will not be possible, and the admin-console should also handle this case

[Aschen]

@Shiranuit I think cookie auth should be default, isn't it the case ?

Otherwise I agree but if the backend didn't enable cookies I don't see how we can properly authenticate the client..

[sebtiz13]

Ok like I say it's understandable not to be able to provide directly token in GET parameters, but what you think of solution of generate a temporary token for export ?
This token would only be valid for one export and not use the user token. So, off course this create a specific case, but don't cause security problem and it's simpler than cookie.

[Aschen]

@sebtiz13 We could do this but it require some backend development for Kuzzle

[thomas-mauran]

Here is what the css looks like