uploading tiny files takes HUGE amount of memory and cpu

[npomfret]

My application is maxed out on CPU and is using a large amount of memory when I introduce a file upload (removing it fixes the problem). I've no idea what I'm doing wrong, but here's some sample code with supporting output. It shows a big jump in memory after the @google-cloud/storage api is invoked and the CPU runs very high. In practice, my app CPU hits 100% constantly.

https://gist.github.com/npomfret/68f9e668042da1b61dcee3554060834c

Sample output:

memory: {"rss":"34.64MB","heapTotal":"15.45MB","heapUsed":"10.25MB","external":"1.80MB"}
cpu: 0%
start

memory: {"rss":"691.91MB","heapTotal":"396.30MB","heapUsed":"369.12MB","external":"1.35MB"}
cpu: 87%
memory: {"rss":"722.25MB","heapTotal":"396.80MB","heapUsed":"369.06MB","external":"1.35MB"}
cpu: 46%
memory: {"rss":"735.36MB","heapTotal":"397.05MB","heapUsed":"369.09MB","external":"1.35MB"}
cpu: 64%
time taken: 22364ms

--
memory: {"rss":"735.37MB","heapTotal":"397.05MB","heapUsed":"369.13MB","external":"1.35MB"}
cpu: 30%
--
memory: {"rss":"735.37MB","heapTotal":"397.05MB","heapUsed":"369.14MB","external":"1.35MB"}
cpu: 58%
--
memory: {"rss":"735.37MB","heapTotal":"397.05MB","heapUsed":"369.15MB","external":"1.35MB"}
cpu: 28%
--
memory: {"rss":"735.38MB","heapTotal":"397.05MB","heapUsed":"369.15MB","external":"1.35MB"}
cpu: 54%
--
memory: {"rss":"735.38MB","heapTotal":"397.05MB","heapUsed":"369.16MB","external":"1.35MB"}
cpu: 26%

[stephenplusplus]

Related: #131

Will you try adding “resumable: false” to the options object passed to file.save()?

[npomfret]

OMG that totally fixed it:

memory: {"rss":"34.91MB","heapTotal":"15.45MB","heapUsed":"10.18MB","external":"1.78MB"}
cpu: 0%
start

memory: {"rss":"42.17MB","heapTotal":"17.20MB","heapUsed":"9.50MB","external":"1.39MB"}
cpu: 12%
memory: {"rss":"42.27MB","heapTotal":"17.20MB","heapUsed":"9.73MB","external":"1.40MB"}
cpu: 0%
memory: {"rss":"42.38MB","heapTotal":"17.20MB","heapUsed":"9.96MB","external":"1.42MB"}
cpu: 8%
time taken: 2303ms

--
memory: {"rss":"42.38MB","heapTotal":"17.20MB","heapUsed":"9.97MB","external":"1.42MB"}
cpu: 0%
--
memory: {"rss":"42.38MB","heapTotal":"17.20MB","heapUsed":"9.97MB","external":"1.42MB"}
cpu: 4%
--
memory: {"rss":"42.38MB","heapTotal":"17.20MB","heapUsed":"9.98MB","external":"1.42MB"}
cpu: 0%
--
memory: {"rss":"42.38MB","heapTotal":"17.20MB","heapUsed":"9.98MB","external":"1.42MB"}
cpu: 3%
--
memory: {"rss":"42.38MB","heapTotal":"17.20MB","heapUsed":"9.99MB","external":"1.42MB"}
cpu: 0%

Ok, so the docs say:

There is some overhead when using a resumable upload that can cause noticeable performance degradation while uploading a series of small files.

I don't think this sufficiently covers it! Also, presumably the same performance problems appear for large files too, no? Using this API without the switch makes my app totally unusable, the CPU, memory and garbage collector are all going so crazy there's no room for my actual code to run.