vips__regions_start: failed for image temp-xxx

[Alex-Werner]

Hi,

Thank you for your library, it worked as a a charm for the last months, and have saved my life !
Our usage is this : Whenever a user send a picture to our server, we store the first as original, and crop/save multiple versions : (40x40, 60x60, 120x120, 200x200, 400x400). And it has always worked.

Lastly, we came into this situation, it happened twice (without any code modification on 0.16.2) where all images has been processed, cropped, saved, except the 200x200 version that without any understanding was saved as a 0bytes file displaying nothing about void.

I didn't find any log for the first time, but the second time I saw this:

Error: VipsJpeg: Not a JPEG file: starts with 0x90 0xa8
vips__region_start: start function failed for image temp-11340
VipsRegion: valid clipped to nothing
VipsRegion: valid clipped to nothing
VipsRegion: valid clipped to nothing
VipsRegion: valid clipped to nothing
VipsRegion: valid clipped to nothing
VipsRegion: valid clipped to nothing
VipsRegion: valid clipped to nothing

So my questions are :

1. Why does it tell me it's not a JPEG file where it seems to be
2. Why it says file starts with 0x90 0xa8 where using HxD on original tells me its start with 0xFF 0xD8 0xFF 0xDB.
   I found a 0X90 0xA8 but at the offset 00000E70, but i'm not sure about what it could or not mean... I tought that JPEG doesn't need any particular headers :'( And to be fair, I don't understand much about theses headers / hexa problematics.
3. Why is it only the 200x200 that doesn't works where other size has been successfully created ?

Here is for reference, the code I use.

inputFile.ext = jpg in this exemple.
uuid is something like : b2a60321-7e38-4a5a-8998-a3c1ebd60fde

var paths = {
                original: {
                    dir: __rootPath + '/uploads/profilePictures/original/',
                    file: __rootPath + '/uploads/profilePictures/original/' + uuid + inputFile.ext
                },
                xsmall: {
                    dir: __rootPath + '/uploads/profilePictures/40x40/',
                    file: __rootPath + '/uploads/profilePictures/40x40/' + uuid + inputFile.ext,
                    size: [40, 40]
                },
                small: {
                    dir: __rootPath + '/uploads/profilePictures/60x60/',
                    file: __rootPath + '/uploads/profilePictures/60x60/' + uuid + inputFile.ext,
                    size: [60, 60]
                },
                normal: {
                    dir: __rootPath + '/uploads/profilePictures/120x120/',
                    file: __rootPath + '/uploads/profilePictures/120x120/' + uuid + inputFile.ext,
                    size: [120, 120]
                },
                large: {
                    dir: __rootPath + '/uploads/profilePictures/200x200/',
                    file: __rootPath + '/uploads/profilePictures/200x200/' + uuid + inputFile.ext,
                    size: [200, 200]
                },
                xlarge: {
                    dir: __rootPath + '/uploads/profilePictures/400x400/',
                    file: __rootPath + '/uploads/profilePictures/400x400/' + uuid + inputFile.ext,
                    size: [400, 400]
                }
            };
var processImageUpload = function () {
                //WARNING : THIS IS ALL SYNCHRONOUS STEPS.
                for (var i = 0; i < Object.keys(paths).length; i++) {
                    var _size = paths[Object.keys(paths)[i]].size;
                    var _dir = paths[Object.keys(paths)[i]].dir;
                    var _file = paths[Object.keys(paths)[i]].file;
                    //Step 1 : Verify that the directories exists and create it if not.
                    fse.ensureDirSync(_dir);

                    //Step 2 : Clean the folders (with any other precedent files that could still lies into)
                    for (var j = 0; j < handledExt.length; j++) {
                        var fileToTest = _dir + uuid + handledExt[j];
                        if (FileExist(fileToTest)) {
                            fse.unlinkSync(fileToTest);
                        }
                    }
                    var sharpQuality = 100;
                    //Step 3 : Create the files
                    var sharping = sharp(fs.readFileSync(temporaryPath + temporaryFilename + inputFile.ext));
                    if (_size) {
                        sharping = sharping.resize(_size[0], _size[1]).quality(sharpQuality);
                    }
                    if (ext == ".jpg") {
                        sharping = sharping.jpeg();
                    }
                    if (ext == '.png') {
                        sharping = sharping.png();
                    }
                    sharping.toFile(_file, function (err) {
                        if (err) {
                            console.error(err);
                            replyHandler.response(804, reply);
                        }
                    });
                }
            };

Thank you for any help on this matters, I really don't know what happens as I didn't change it for a long times.
It has worked a numberous time without trouble (~3000 picture uploads), but failed twice in the last 2 days (with approx 100 pictures uploads, so 2% errors only on ONE specific size).

What I know :

• Has been sent from iPhone
• Picture came from Facebook

[lovell]

Hello, the THIS IS ALL SYNCHRONOUS STEPS comment is slightly misleading as sharp will start to process each input asynchronously after toFile is called.

Node (via libuv) provides sharp with a maximum of UV_THREADPOOL_SIZE threads to process images in parallel

The default size of this shared thread pool is 4.

Local filesystem operations also use this thread pool, so it's possible that ensureDirSync will block the 5th iteration until at least one of the first 4 images has been processed.

The fact you're seeing problems with the 5th iteration hints that code not shown here might be altering/removing temporaryFilename before it is read.

[Alex-Werner]

Hello,
Thank you for you answer, yeah, I know, it's mostly a todo asking me to go back one day on this code to make it async about the fse (fs-extra) method : fse.ensureDirSync(); unlinkSync() or readFileSync().
It would be better if I had removed this, but in the other hand, I wanted to keep the code I used in order to give as much information as possible.

I'm aware of the async property of Sharp (even If I don't totally understand the mechanics behind it).

I will look into it given data you gave me (thank you a lot). But do you have an hint about why the sixth iteration doesn't have this problem ? (the 400x400's one).

I'm indeed altering the file temporaryFilename, after my loop, I perform a database action, where in the callback of this method, I delete temporaryFilename (so after the database has updated the field).

Given your explanation, I'm assuming that I'm lacking a callback at the END of all operations made (so I have to be sure that all iterations and file has correctly been "sharped", and instead i'm doing it too quickly (the delete of the tempFile).
I will try that ! I'm sorry for the dumb question.

[lovell]

Closing as I think this is resolved, but please re-open if there are further related questions.

[Alex-Werner]

Sorry for opening it again. Just to keep you in touch with that

I had try during the last 30 days to change multiple time how it works. But yeah, I always have a problem related to this. It is never the same statement.

I changed how my code behave multiple times, adding some fix here and there, but yeah, sometimes picture will be empty. Even with the same picture sometimes.
Error are so much un-understandable to me that I will try to change how it works. But it works in really 99% of all cases.
Thank you for your help, I will move on from this.
Closing it again and thank you again for all your work on this ! I might not have the technical ability to use it properly :x

Edit : Can't close it myself :x Will let you do that ^^'

[lovell]

See #657 for the error reading resolution warnings.