Smart source-plugin bulk fetching remote file assets

[calcsam]

(from a conversation with @KyleAMathews )

Problem: People are frustrated by long build times driven by downloading data from Wordpress

Cause: Say gatsby-source-wordpress hits a Wordpress site and starts downloading 10,000 images. This creates problems on a number of levels. Often the receiving Wordpress site can’t deal with so many open connections. It overwhelms the Wordpress site, the server starts running out of memory, and the download slows down or fails altogether

Proposed solution: This is a two-part solution -- (1) fundamental technical improvement (2) better UX for users.

• We need more robust downloading of files (for createRemoteFileNode).
  • We need a file throttler / ConnectionHandler to manage connections to the remote Wordpress server.
  • It would be powered by an algorithm deciding how many connections to open. The algorithm would do this by track bytes / second downloaded, and continuing to add open connections until the bytes/second rate stalls out.
  • There’s potential other, simpler, options like “don’t have more than X open connections”
  • Download manager would add new files, make sure that remote server isn’t getting overloaded.
  • Exponential backoffs.
• Provide visibility of download process to user
  • Showing file download count // download progress for remote files -- “we’re downloading 250 of 500 files, eta 5:00” -- reduce user frustration

Similarity to other systems:

• @KyleAMathews has tweeted out looking at this problem, and looked for packages that handle connections in Wordpress side. So far, this search has come up blank.
• Similar problem to the what rsync does -- we’re trying to sync files through smart management of server
• Also similar to the strategy a lot of browsers use -- TCP -- get initial byte, doubling download speed, exponential backoff, fixed max # of simultaneous open connections….

[TylerBarnes]

This plugin handles transferring media files from wp site to wp site.
https://github.com/wp-sync-db/wp-sync-db-media-files
It might be useful for getting some ideas. I've used it and it works well

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

Thanks for being a part of the Gatsby community! 💪💜

[muescha]

This download manager should also include an cache and some strategies how not redownload media assets when nothing is changed.

[muescha]

Is there also a stragegy only to download the used media assets (when maybe more assets are saved in wordpress but only some used in the used articles)?

[cedricdelpoux]

Any news about this issue?

[NickBarreto]

I'm very much experiencing this sort of issue when sourcing data through an API. At the moment I am using a very simple concurrency manager so I don't make too many axios requests at once, but this doesn't really help when handling images.

Are there any updates regarding a downloader for media? I'd love to contribute but I'm a bit new to Gatsby: happy to try things out if pointed in the right direction.

[ascorbic]

In most cases you should use createRemoteFileNode, which is in gatsby-source-filesystem. That handles a lot of that for you.

[NickBarreto]

I am using createRemoteFileNode, yes. It works well but when there is a lot of media I think it is making too many requests at once: I see some image fetch requests timing out as the AWS bucket that holds the media gets hammered all at once and can't quite keep up. Is there a way to manage the concurrency of requests made by createRemoteFileNode?

[spencer-aerian]

createRemoteFile node is all good for assets but it doesn't fix the issue for api's. Any chance someone could put a promise throttle into the gatsby-drupal loop for example as its opaque atm.