[Proposal] Uppy-based UI for file uploads in RDM

[mirekys]

Motivation

Why are we doing this? What user stories does it support? What is the expected outcome?

RDM users would like to:

1. Choose a single or multiple files for upload
2. See upload progress of each file
3. Abort an ongoing upload
4. Retry failed uploads
5. Manage already uploaded files (remove, update associated metadata - e.g. image caption, description
6. Verify file integrity (checksums)
7. Efficiently upload large (> 10 GiB) files over not-so-reliable networks

We would like to:

1. Directly upload data to (S3) storage, bypassing Invenio server, reducing app server load when processing uploads
2. Leverage (S3) multipart upload through Pluggable transfers for invenio-records-resources #76 to split and upload large files in multiple parallel chunks

Current UI implementation covers just a subset of these requirements, especially
lacks in department of large files uploads

Summary

What are you proposing to change (high-level overview only)?

Integrate Uppy.io, together with the AWS S3 plugin for a file uploader UI.
Can be probably done either by a headless approach (using its functionality, but leaving most of UI representation as is), or
by replacing current UI altogether with full Uppy Dashboard (examples).

Latest Uppy.io version (v4) requires React 18, but previous version compatible with React 16 is still usable for this case

This RFC will rely on features coming from: #76

Resources

Which resources do you have available to implement this RFC and what is your overall timeline?

CESNET is willing to take over implementation, we'll need it done approx. by the end of November

[mirekys]

@slint @ntarocco @zzacharo just pinging this :)

[ntarocco]

@mirekys sorry if you were waiting from us on this. It looks great to me!
As previously discussed, there are 2 points that the implementation should cover:

1. the usage of this UI component should be toggable: in this way, each instance can choose to enable Uppy, and we can develop/merge/test it without having a long-lasting feature branch
2. the criteria when enabling single or multipart upload should be configurable is possible, and if it makes sense. For example, an instance should be able to configure that multipart upload kicks in for files larger than 1 Gb.

For the React 18 upgrade, I think that it should not be too complex to upgrade the various UI components.

[utnapischtim]

i saw the react 18 reference. if we update InvenioRDM to react 18, keep in mind that packages like react-override have to be updated too

[mirekys]

Thanks @ntarocco, I was quite unfamiliar with how the rfc proposal process goes and was waiting for it to be approved. I definitely agree on those points you mentioned.

Nowadays when mentioning React upgrade, I would aim even at v19, which is now stable (not sure how many blockers will there be regarding other library deps, tho).

I have a question regarding where the implementation should live. Currently FileUploader lives as part of invenio-rdm-records. If this feature should be optional/toggleable and basically will be a React-only project, I think it makes sense for it to be a separate repo/NPM package, rather than to be a part of the main codebase with all the extra dependencies, but please let me know your thoughts :).

General idea is, that this package would then export React components needed for UI, and its own DepositFilesService.js implementation. It could then be toggled on by invenio-app-rdm configuration, by passing it in filesService and config (for things like single/multi threshold criteria config) props to DepositFormApp and setting react-overridable mapping to change the default UI layout.

[ntarocco]

Nowadays when mentioning React upgrade, I would aim even at v19, which is now stable (not sure how many blockers will there be regarding other library deps, tho).

Yes, I agree!

II think it makes sense for it to be a separate repo/NPM package, rather than to be a part of the main codebase with all the extra dependencies, but please let me know your thoughts :).

Totally OK with that, if it is simple enough to integrate the deposit form with this other component. This will probably mean publishing the component in npm.

setting react-overridable mapping to change the default UI layout.

Just an idea: can we, instead, have the feature toggle in Python, and somehow in Jinja, inject one or the other? Or inject via Jinja extra props to drive the UI to use one or the other?
In general, whenever possible, I would limit the usage of react-overridable ids to users that are overriding part of the UI, and avoid using it internally. Again, when possible... we are already using react-overridable for many search interfaces.
Anyway, I think that possibilities and answers will be more clear with some prototyping.