Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable zip file downloading #215

Open
yuliujpl opened this issue Jul 9, 2024 · 10 comments
Open

Enable zip file downloading #215

yuliujpl opened this issue Jul 9, 2024 · 10 comments
Labels
🧙‍♀️enhancement New feature or request

Comments

@yuliujpl
Copy link
Collaborator

yuliujpl commented Jul 9, 2024

Backend and frontend needs to be developed to enable zip file downloading.
@yuliujpl yuliujpl added the 🧙‍♀️enhancement New feature or request label Jul 9, 2024
@nutjob4life
Copy link
Member

@dcrichto1 @hoodriverheather @yuliujpl here are some ideas on how this might work

  1. User visits the UI and selects either a set of files, a set of datasets, or a collection for ZIP file downloading
  2. The UI at this point could put in some limits using knowledge it might have as to the size
  3. Assuming the user's request passes muster at this point, the UI makes an API call to the backend passing
    • The files to put into the archive—this would likely be a Solr query
      • The idea is to find all files that match the query
    • The email address to notify once the ZIP archive is ready
    • This API endpoint is protected and accepts the JWT established in the user's session
  4. The backend assigns a UUID to this request and returns it to the UI and immediately and begins a background task
    1. The task uses the Solr query and creates a directory tree of matching files
    2. It ZIPs the archive
    3. It generates a URL to the archive
    4. It starts a timer task to delete the file after a certail number of days (7 days?)
    5. It emails the user a message with the URL to the archive
  5. The user opens their email in time and begins downloading the file
    • As the archive can be large, the user can retry the download any time (within 7 days?)

At any point in step 4, an error could occur

  • The Solr query matches no files; this could be returned synchronously to the UI
  • There's not enough disk space to create the archive; this would be communicated via email to the user
  • There's enough space, but the generated archive is too large to download (say 1GB?); this would be communicated via email to the user
    • The archive is immediately deleted
  • Generating the archive takes too much time (more than a day?); it's aborted and communicated via email to the user
    • The partial archive is immediately deleted

Using the UUID, the UI can ask about the status of the ZIP file assembly:

  • Is it in progress? Can we tell how far it's gotten?
  • Is it complete? If so, can we get the URL in case the email was lost?
    • Is it complete but it was never downloaded, and so the file was deleted?
  • Did one of the error conditions occur?

@dcrichto1 @hoodriverheather @yuliujpl what do you think? What should some of these thresholds (space and time) be?

@nutjob4life
Copy link
Member

@yuliujpl @dcrichto1 @hoodriverheather, reminder:

  1. Please check the approach above ↑
  2. Leave feedback below ↓

No feedback means we're good and I'll start implementing the server side of this

@hoodriverheather
Copy link

@nutjob4life This looks like a good plan to me. Want me to add to our staff agenda for tomorrow?

@nutjob4life
Copy link
Member

@nutjob4life This looks like a good plan to me. Want me to add to our staff agenda for tomorrow?

@hoodriverheather sure!

@nutjob4life
Copy link
Member

nutjob4life commented Jul 23, 2024 •
edited
Loading

@yuliujpl mentioned in Teams chat:

It should probably run on a separate server so it doesn't compete for resources on the labcas-backend server

Probably; but it'll also need access to the /labcas-data disk so it can gather the files and have enough scratch space to create ZIP archives

@nutjob4life
Copy link
Member

nutjob4life commented Feb 4, 2025 •
edited
Loading

@yuliujpl the API for this is available on labcas-dev as follows:

https://labcas-dev.jpl.nasa.gov/data-access-api/zip?email=EMAIL&query=SOLRQUERY

You can (and should) pass the JWT (as a JasonWebToken cookie and/or an Authentication header) in order for the backend to determine what access the user has.

The return is a text/plain response that contains a UUID to identify the request. In the future we might have other APIs that take that UUID and get status, cancel the request, etc.

Example usage:

$ curl --request POST --insecure \
        --url 'https://labcas-dev.jpl.nasa.gov/data-access-api/auth' \
        --header 'Content-type: application/x-www-form-urlencoded' \
        --data-urlencode 'username=kelly' \
        --data-urlencode 'password=REDACTED' > /tmp/jwt
$ curl  --header "Authentication: Bearer `cat /tmp/jwt`" \
    --cookie "JasonWebToken=`cat /tmp/jwt`" \
    'https://labcas-dev.jpl.nasa.gov/data-access-api/[email protected]&query=id:Pre-diagnostic_PDAC_Images/City_of_Hope/COH_0171/COH_01710003/DICOM/I883*'
69484d3b-7f0e-4252-ba1c-26371f8deba2

Also: I'm wondering if passing a Solr query makes sense? Maybe file IDs would be better?

@nutjob4life
Copy link
Member

@yuliujpl looking for feedback on this (see above comment)

Some questions:

  • Does a Solr query make sense?
  • Would it be better to pass a list of file IDs?
    • If so, would it be better to POST a (possibly large) payload of file IDs?
  • What other questions should we discuss?

@yuliujpl
Copy link
Collaborator Author

yuliujpl commented Feb 11, 2025 via email

@nutjob4life
Copy link
Member

@yuliujpl okay, so how about this:

Instead of a GET request, let's make it a POST with a JSON payload provided by the UI with three possible keys:

  • email — where to send notifications (required)
  • query — Solr files query (optional, but required if files is missing)
  • files — A [list] of file IDs

It is an error to provide both query and files.

The response can still be be a text/plain with a UUID unless you'd prefer a JSON response like {"uuid": "123456-abcd-124456…"}

Does this sound okay?

@yuliujpl
Copy link
Collaborator Author

yuliujpl commented Feb 11, 2025 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
🧙‍♀️enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nutjob4life @yuliujpl @hoodriverheather