fs: EPERM inconsistency between synchronous and asynchronous API on Windows 7

[jamesshore]

Update: I've characterized this issue better. Skip down to this comment for a better explanation.

If you're writing tests for an HTTP server, you might find yourself doing something like this (pseudocode):

before each test:
  create index.html using fs.writeFileSync()
  start server, using fs.createReadStream().pipe(res); to respond to requests

after each test:
  stop server
  erase index.html using fs.unlinkSync()

during test:
  GET index.html from server

If you run this seemingly-innocuous test, it will work on Windows XP, Mac OS (10.8.5) and Linux (Ubuntu 10.04), but it will fail with an EPERM error on Windows 7.

The core issue is in how fs.unlinkSync() and fs.writeFileSync() interact with pipe(). It's not really about HTTP; that's just a common situation that demonstrates the issue.

The issue is very timing sensitive. It's sensitive to whether the code is synchronous or asynchronous. I've also had a few test runs where the error didn't appear, although it usually does.

Here's code to reproduce the issue.

"use strict";

console.log("Starting timeout...");
setTimeout(function () {      // REQUIRED to reproduce
    var fs = require("fs");

    var READ_PATH = "read.txt";
    var WRITE_PATH = "write.txt";

    fs.writeFileSync(READ_PATH, "foo");

    var readStream = fs.createReadStream(READ_PATH);
    var writeStream = fs.createWriteStream(WRITE_PATH);

    console.log("Starting pipe...");
    readStream.pipe(writeStream);

    readStream.on("close", function () {
        console.log("Read stream closed.");
    });

    writeStream.on("finish", function () {
        console.log("Write stream finished.");

        console.log("File unlink...");
        fs.unlinkSync(READ_PATH);     // MUST BE SYNC to reproduce
        console.log("Unlink successful.");

        console.log("File write...");
        fs.writeFileSync(READ_PATH, "foo");
        console.log("Write successful");
    });
}, 1000);

Here's the output on Windows 7:

Starting timeout...
Starting pipe...
Write stream finished.
File unlink...
Unlink successful.
File write...

fs.js:427
  return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
                 ^
Error: EPERM, operation not permitted 'C:\projects\weewikipaint\read.txt'
    at Object.fs.openSync (fs.js:427:18)
    at Object.fs.writeFileSync (fs.js:966:15)
    at WriteStream.<anonymous> (C:\projects\weewikipaint\eperm_repro.js:30:6)
    at WriteStream.EventEmitter.emit (events.js:117:20)
    at finishMaybe (_stream_writable.js:354:12)
    at afterWrite (_stream_writable.js:274:5)
    at onwrite (_stream_writable.js:264:7)
    at WritableState.onwrite (_stream_writable.js:97:5)
    at fs.js:1681:5
    at Object.wrapper [as oncomplete] (fs.js:510:5)

As I said, the issue is timing sensitive. If the setTimeout() is removed or changed to process.nextTick(), the issue goes away. If the fs.unlinkSync() is changed to fs.unlink(), the issue goes away.

[rlidwka]

before each test:
create index.html using fs.writeFileSync()

I'm just curious, what language is that?

[jamesshore]

It's just pseudocode.

[jamesshore]

I've dug into this a bit further. The root cause appears to be related to a difference in the way synchronous and asynchronous fs calls work.

If you open a read stream on Windows 7, then delete and overwrite the backing file using the fs synchronous API before the stream is closed, you'll get an EPERM error. However, if you use the asynchronous API instead, the code will execute without error.

This code uses the synchronous API and demonstrates the EPERM error:

"use strict";

// This program demonstrates a cross-platform inconsistency in Node.js between Mac and Windows.
// It deletes and overwrites a file before closing it.
// It will work on Mac but fail with an EPERM error on Windows.

var fs = require("fs");

var READ_PATH = "read.txt";

fs.writeFileSync(READ_PATH, "foo");

console.log("Opening read stream...");
var readStream = fs.createReadStream(READ_PATH);

readStream.on("open", function () {
    console.log("Read stream opened.");

    console.log("Unlinking read file...");
    fs.unlinkSync(READ_PATH);
    console.log("Unlink successful.");

    console.log("Overwriting read file...");
    fs.writeFileSync(READ_PATH, "foo2");
    console.log("Overwrite successful.");

    console.log("Closing read stream...");
    readStream.close();
});

readStream.on("close", function () {
    console.log("Read stream closed.");
});

Output on Windows 7:

C:\projects\weewikipaint>node eperm_no_race.js
Opening read stream...
Read stream opened.
Unlinking read file...
Unlink successful.
Overwriting read file...

fs.js:427
  return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
                 ^
Error: EPERM, operation not permitted 'C:\projects\weewikipaint\read.txt'
    at Object.fs.openSync (fs.js:427:18)
    at Object.fs.writeFileSync (fs.js:966:15)
    at ReadStream.<anonymous> (C:\projects\weewikipaint\eperm_no_race.js:24:5)
    at ReadStream.EventEmitter.emit (events.js:95:17)
    at fs.js:1505:10
    at Object.oncomplete (fs.js:107:15)

Output on Mac OS 10.8.5:

jamesshore:weewikipaint jshore$ node eperm_no_race.js 
Opening read stream...
Read stream opened.
Unlinking read file...
Unlink successful.
Overwriting read file...
Overwrite successful.
Closing read stream...
Read stream closed.

However, this code will work fine if the asynchronous form of unlink() and writeFile() are used:

"use strict";

// This program is the same as eperm_no_race.js, but it uses asynchronous fs calls
// rather than synchronous. It works without error on both Mac and Windows.

var fs = require("fs");

var READ_PATH = "read.txt";

fs.writeFileSync(READ_PATH, "foo");

console.log("Opening read stream...");
var readStream = fs.createReadStream(READ_PATH);

readStream.on("open", function () {
    console.log("Read stream opened.");

    console.log("Unlinking read file...");
    fs.unlink(READ_PATH, function (err) {
        if (err) console.log("ERROR: " + err);
        console.log("Unlink successful.");

        console.log("Overwriting read file...");
        fs.writeFile(READ_PATH, "foo2", function (err) {
            if (err) console.log("ERROR: " + err);
            console.log("Overwrite successful.");

            console.log("Closing read stream...");
            readStream.close();
        });
    });
});

readStream.on("close", function () {
    console.log("Read stream closed.");
});

Output on both Mac and Windows:

C:\projects\weewikipaint>node eperm_no_error.js
Opening read stream...
Read stream opened.
Unlinking read file...
Unlink successful.
Overwriting read file...
Overwrite successful.
Closing read stream...
Read stream closed.

Question: What's the Node.js policy regarding cross-platform compatibility? Are these programs supposed to work the same on both platforms? If so, there's a bug when the synchronous fs API is used. If not, why does the asynchronous version work on Windows? You're not supposed to be able to delete open files on Windows.

[jamesshore]

I've renamed this issue to better reflect the underlying problem. (Previous title: "fs: EPERM race condition with pipe() and unlinkSync() interaction on Windows 7".)

[dcsobral]

I wonder what the async version is doing, since there's no way for this to actually work on Windows. An open file cannot be deleted, so either the file is not open, or the file does not get deleted.

[jamesshore]

@dcsobral Equally strange: the sync version will work if you only delete or only overwrite.

[trevnorris]

@indutny know how this is handled in libuv?

[indutny]

@trevnorris I believe it could be some IOCP trickery, summoning @piscisaureus here :)

[jbrumwell]

I just ran into this error when using the following async flow;

fs.exists
fs.unlink
fs.writeFile || fs.rename

resolved it by overwriting the destination file and then unlinking the temp file

[indutny]

Hey @orangemocha , will you be interested in looking into this sometime later?

[orangemocha]

@indutny , certainly, but probably not until after 0.12 as my main priority right now is getting the unit tests to work.

[indutny]

Sure, I understand. Thank you!

[blessanm86]

Is there any progress with this issue? Its been 4 months since the last activity, so just asking.

[orangemocha]

Your code doesn't seem valid to me, both in the sync and async versions. You are trying to write to a file that is still open for reading (becasue the readStream is still open). In short, move the unlink and overwrite operations to the readStream 'close' event handler, and things will work just fine (and they should on all platforms).

I get the same error on my Win8.1 machine both for your sync and async example. Results may vary because the operation is, well, asynchronous and if you are lucky the underlying file might get closed before you attempt the overwrite.

fs.unlinkSync() marks a file for deletion. Actual deletion will happen when all handles are closed, which is when the readStream is closed. Trying to re-open (as in your overwrite operation) while the delete is pending, will cause an error, as documented by MSDN:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa363858(v=vs.85).aspx
If you call CreateFile on a file that is pending deletion as a result of a previous call to DeleteFile, the function fails. The operating system delays file deletion until all handles to the file are closed. GetLastError returns ERROR_ACCESS_DENIED.

If you tried to overwrite the file without marking it for deletion, you wouldn't get the error, but the results would be unpredictable because you would be simultaneously reading and writing to the same
file.

[jamesshore]

@orangemocha Yes, I know the code is doing something it shouldn't. It's carefully crafted to reproduce the issue reliably. (I originally ran into the problem when trying to test code that used the 'send' library.) The point is that the error is inconsistent across platforms and synchronous vs. asynchronous APIs. As I said:

Question: What's the Node.js policy regarding cross-platform compatibility? Are these programs supposed to work the same on both platforms?

Regarding the asynchronous delete, there shouldn't be a race condition with the read stream closure. It isn't closed until after the overwrite is complete.

[orangemocha]

Regarding the asynchronous delete, there shouldn't be a race condition with the read stream closure. It isn't closed until after the overwrite is complete.

The read stream will automatically close the underlying file when there is no more data to read (unless your pass {autoClose: false} to fs.createReadStream). In your second example, this is not happening because you are not reading any data, and in fact I always get the error even in the async case. But if you pipe the read stream or add a 'data' handler (as in the original example) then the stream will auto-close and there will be a race between the closing/deleting of the file and the overwrite call.

Regarding cross-platform compatibility, it is certainly a goal we aim for. But because of inherent differences in the underlying platforms, there will always be areas where Node cannot be 100% compatible in all scenarios.

In this particular case, it seems that the code is doing something it shouldn't, and the results vary based on timing, for which the error may surface on some platforms but not on others. I don't see a Node bug here.

I hope this helps :)

[jamesshore]

@orangemocha Thanks for looking into it.

[misterdjules]

@jamesshore @orangemocha Would it help to add a mention of that behavior in the documentation for fs.unlink, to the FAQ or any other resource?

[orangemocha]

@misterdjules Currently the documentation for fs.unlink points to unlink(2), which states:

unlink() deletes a name from the file system. If that name was the last link to a file and no processes have the file open the file is deleted and the space it was using is made available for reuse.
If the name was the last link to a file but any processes still have the file open the file will remain in existence until the last file descriptor referring to it is closed.

So the basics seem to be covered. There are also some subtle differences between platforms as per #7176, which is still open.

Of course if you can think of ways to improve the documentation, a PR will be welcome!