Jimp allocates 200 MB memory on reading a 3.3 MB jpg #153
Comments
|
Thanks for reporting this. Could you post a test case and I'll try and reproduce? |
|
Also having problems with high memory usage. We are using Jimp as a dependency of https://github.com/haydenbleasel/favicons This is a dealbreaker for us. |
|
Can you try this again on the latest release. Can't reproduce using a 10MB file and the following test case: var i = 0;
function loop(){
console.log(i++);
Jimp.read("large.jpg").then(function(image) {
loop();
}).catch(function (err) {
console.log(err);
});
}
loop();Running on Mac. |
|
I used a modified version of your loop to check the memory usage. I added the output showing memory consumption. The image i am using is a 3.3 MB Jpeg. I used the latest version 0.2.27 running with node 5.11 on Mac OS X. Output for memory usage: |
|
Another data point: I'm hitting this on both OS X and Linux, using both the promises-based and callback-based API. ETA: Also when using |
|
Is this still happening? |
|
@Streemo Yep. |
|
Well, I cannot confirm this either. With Node 6.2.0 running at Windows 8 I see this. Heap grows by 20-30 MB every time I read PNG image (974x974 px, 30 KB), which is acceptable in my opinion. Calculator says that for storing 1500x1500 bitmap with 32 bpp I would need only 8.5 MB however, but whatever, could be garbage from decoding phase. GC is kicking in from time to time and does its job. Here is the image I was using if somebody needs it
@LeZuse, in your case high memory usage is actually favicons' fault. It's reading the same image from fs multiple times in parallel for every icon it's generating. If icons were generated in sequence that would not be a problem. I'm working quietly on a series of changes to that package (favicons) which fixes that among many other things. I was able to reduce heap size down to ~300 MB instead of 1.3 GB that it was using before by generating only one icon at the time (this also includes icons that's already generated and stored in memory), so in my case memory usage seems reasonable. Hopefully I'll finish it by the end of the month (also waiting on #159 to get merged) UPD: actually, yes, can confirm. Tried bigger image (JPEG, 3456x2304px, 611KB) and it used about 320 MB every time. What's interesting, same image saved as 3MB PNG needed only 220 MB for a copy. And calculator says we need only 30MB buffer. (Tests performed with latest version published on npm) |
|
@iwasawafag What would the most performant process be? I can read the file once and use the buffer multiple times but running everything in sequence would take forever. |
|
@haydenbleasel No, it wont. Everything Jimp or underlying modules (like png-js or jpegjs) do takes time to process and isn't asynchronous (except reading/writing from fs). So it doesn't really matter if you start generating icons in parallel or in sequence - either way we are waiting untill we're done iterating over pixels of the one image untill we can start iterating pixels of the other. On the other hand when V8 sees high memory usage it's trying to clear garbage, but there's nothing to clear, so I would speculate that we're actually wasting some processing time here while GC runs. On a dirt cheap laptop sequential generation was only 2-3 seconds slower (~14 seconds instead of 16-17) and I would say it's probably caused by reading from fs. But memory usage was 70% lower. I'd say worth it. Keeping copy of the image in-memory and cloning it would definitely make things faster because decoding PNGs is expensive task and it will be run 40 times less often |
|
It seems that root of the problem is somewhere in 'pngjs' and 'jpeg-js' modules. I tried to repeat the test using these modules directly and got very simillar numbers. I then found another JPEG decoder from Mozilla's PNGJS, which is published on Bower, but not on NPM for some reason. Code looks nearly identical, but needs about 4.5 times less memory.
Still not a 30 MB though |
|
Having the same problem here but much worse, with a 18.7mb and 5751×3347px memory goes from ~60mb to over ~630mb |
|
I was not able to solve the memory issue and decided to switch to http://sharp.dimens.io/en/stable/ |
|
Same as @ahaus I was using jimp to create ~50-200 slices of a ~1000 by 1000px image for a project I'm working on but memory usage was getting into the 2GB range and image processing was taking about ten minutes. It seems like it clearly wasn't freeing memory for whatever reason. Ended up switching to Sharp ( http://sharp.dimens.io/en/stable/ ) FWIW this is the code I was using: https://gist.github.com/Ne0nx3r0/656a18a723ad25775dd362ac5924e85b |
|
I was not able to cope with the memory issue and decided to switch to https://www.npmjs.com/package/gm with http://www.graphicsmagick.org/ and also to choose a better hosting. |
|
Same memory issue while loading and resizing a 16M image file on Windows:
The memory never stop to increase and finally the browser explodes. |
|
I have the same issue. |
|
I'm using jpeg-js, and if I run the following code, the node process still has 166MB of memory. let jpegData = fs.readFileSync('test-images/PB085659_0.JPG');
let fileData = jpeg.decode(jpegData, true);
fileData = null;
global.gc(); //take this out and node's memory footprint will be 403M, with gc, 166Mjpeg-js does seem like the memory leak culprit. |
|
To my knowledge a memory leak will get larger and larger. where as this seems to top out at a certain amount of memory used. Javascript implementations of image encoders and decoders are bound to be slow. This would be a great place to write type plugins that use native deps or different JS libraries. If you do make a plugin please make a PR to add it to the readme! |
|
Wow. Spent hours on wondering why I could not get this to run in my server-less environment :( ... was trying out a JPG image as my test image. I switched to a PNG after reading this thread and everything works fine. Any possibility this could get fixed @oliver-moran? @hipstersmoothie do you have any suggestions on wasm powered alternatives? |
|
@hipstersmoothie , jpeg-js did have a solution which allow to config maxMemoryUsageInMB. Is there any way it can be supported? |
|
|
|
@lawchihon Setting
|
|
@hipstersmoothie "jpeg-js" maintainers can't reproduce memory leak on their side, and thinking that's clearly "jimp" issue: |
|
This issue shouldn't be closed |
|
Issue still remains. |
Great !!! then, I can make thumbnail image with 20Mbyte image. |
|
Using |
|
I've still got the issue as well, using v0.6.4 didn't help |
|
Be aware guys that Sharp doesn't support BMP or HEIC (iPhone uploads) |
|
Issue still remains in 2023. Delete this library please... I lost 2 hours of my life. Thank you. |
|
If anyone wants to submit a PR to fix I'm able to review and merge! |
|
Let's at least put this in the readme so if anyone wants to use this library they are aware of this issue. This should def be highlighted. |
|
I'll merge that PR too! |
added jimp-dev#153 to readme for awareness
* Update README.md added #153 to readme for awareness * Update README.md --------- Co-authored-by: Andrew Lisowski <[email protected]>
|
gosh I wish I had seen this before.... |
|
Jesus, the time it took to pinpoint the memory leak to this library is insane... The above README entry, definitely does not do it justice: The fact that this introduces a (significant) memory leak is unacceptable. In my specific use-case I use Jimp, along with several other dependencies, to manipulate PNGs. Specifically, I use Jimp to add custom text to several related PNGs. Because in development environment the server restarts on each change (nodemon) the issue was not at all apparent. Only in production did the process start racking up memory and never clearing it, causing it to eventually run out and crash. |
|
If anymore wants to make the docs clearer or attempt to fix, PRs welcome! |
What do you suggest to make the docs clearer? It seems like the problem is there's a crippling memory leak ... are you suggesting guidance simply not to use the library? Or is there some work around that should be documented? |
I was referring to this. If the library works for you, use it. If it doesn't, don't. If you feel like you could fix it, I'll merge your code. Given this issue is as old as the repo I have a feeling it won't be too easy to fix. |
|
If you want more accurate image codecs please use the WASM ones. The exact issue here is probably an issue with one of the default JS based codecs. |
and others
When I try to read a 3.3 MB sized jpeg image jimp allocates around 200 MB local memory (which is a lot). The main problem however is that the memory is not deallocated.
The nodejs process got killed on reading about 5 images.
I am only using Jimp.read(filename).then(...).catch(...).
The text was updated successfully, but these errors were encountered: