Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add advisory file locking API #111

Open
wants to merge 21 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 74 additions & 0 deletions Sources/System/FileLock.swift
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
#if !os(Windows)
extension FileDescriptor {
/// Apply an advisory lock to the file associated with this descriptor.
///
/// Advisory locks allow cooperating processes to perform consistent operations on files,
/// but do not guarantee consistency (i.e., processes may still access files without using advisory locks
/// possibly resulting in inconsistencies).
///
/// The locking mechanism allows two types of locks: shared locks and exclusive locks.
/// At any time multiple shared locks may be applied to a file, but at no time are multiple exclusive, or
/// both shared and exclusive, locks allowed simultaneously on a file.
///
/// A shared lock may be upgraded to an exclusive lock, and vice versa, simply by specifying the appropriate
/// lock type; this results in the previous lock being released and the new lock
/// applied (possibly after other processes have gained and released the lock).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I appreciate this is as specific as man 2 flock gets and its implicit that the upgrade only works on the same file descriptor I suppose, i.e. if a process has two independent file descriptors to the same underlying file, they can still block each other.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe flock(2) would be associated between process/file, while OFD locks would be file-description associated (but not descriptor, i.e. dup would share locks).

Copy link
Contributor

@simonjbeaumont simonjbeaumont Oct 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may already be on the same page here, but I believe that, with flock(2), using dup(2) will allow the new fd to upgrade a lock, but using open(2) will not.

Here's a toy program that accepts NEW_FD_METHOD as an environment variable and attempts to upgrade a shared lock to an exclusive one using a new file descriptor pointing to the same file as was already locked:

https://gist.github.com/simonjbeaumont/7bd12f1f97e162734b6fec847d3a969f

Running it with both options and dup is fine, open fails:

clang -o flock-test flock-test.c && NEW_FD_METHOD=dup ./flock-testclang -o flock-test flock-test.c && NEW_FD_METHOD=open ./flock-test
error obtaining exclusive lock with new fd 4: Resource temporarily unavailable

///
/// Requesting a lock on an object that is already locked normally causes the caller to be blocked
/// until the lock may be acquired. If `nonBlocking` is passed as true, then this will not
/// happen; instead the call will fail and `Errno.wouldBlock` will be thrown.
///
/// Locks are on files, not file descriptors. That is, file descriptors duplicated through `FileDescriptor.duplicate`
/// do not result in multiple instances of a lock, but rather multiple references to a
/// single lock. If a process holding a lock on a file forks and the child explicitly unlocks the file, the parent will lose its lock.
///
/// The corresponding C function is `flock()`
@_alwaysEmitIntoClient
public func lock(
exclusive: Bool = false,
nonBlocking: Bool = false,
retryOnInterrupt: Bool = true
) throws {
try _lock(exclusive: exclusive, nonBlocking: nonBlocking, retryOnInterrupt: retryOnInterrupt).get()
}

/// Unlocks an existing advisory lock on the file associated with this descriptor.
///
/// The corresponding C function is `flock` passed `LOCK_UN`
@_alwaysEmitIntoClient
public func unlock(retryOnInterrupt: Bool = true) throws {
try _unlock(retryOnInterrupt: retryOnInterrupt).get()

}

@usableFromInline
internal func _lock(
exclusive: Bool,
nonBlocking: Bool,
retryOnInterrupt: Bool
) -> Result<(), Errno> {
var operation: CInt
if exclusive {
operation = _LOCK_EX
} else {
operation = _LOCK_SH
}
if nonBlocking {
operation |= _LOCK_NB
}
return nothingOrErrno(retryOnInterrupt: retryOnInterrupt) {
system_flock(self.rawValue, operation)
}
}

@usableFromInline
internal func _unlock(
retryOnInterrupt: Bool
) -> Result<(), Errno> {
return nothingOrErrno(retryOnInterrupt: retryOnInterrupt) {
system_flock(self.rawValue, _LOCK_UN)
}
}
}
#endif

13 changes: 13 additions & 0 deletions Sources/System/Internals/Constants.swift
Original file line number Diff line number Diff line change
Expand Up @@ -528,3 +528,16 @@ internal var _SEEK_HOLE: CInt { SEEK_HOLE }
internal var _SEEK_DATA: CInt { SEEK_DATA }
#endif

#if !os(Windows)
@_alwaysEmitIntoClient
internal var _LOCK_SH: CInt { LOCK_SH }

@_alwaysEmitIntoClient
internal var _LOCK_EX: CInt { LOCK_EX }

@_alwaysEmitIntoClient
internal var _LOCK_NB: CInt { LOCK_NB }

@_alwaysEmitIntoClient
internal var _LOCK_UN: CInt { LOCK_UN }
#endif
34 changes: 34 additions & 0 deletions Sources/System/Internals/Syscalls.swift
Original file line number Diff line number Diff line change
Expand Up @@ -133,3 +133,37 @@ internal func system_ftruncate(_ fd: Int32, _ length: off_t) -> Int32 {
return ftruncate(fd, length)
}
#endif

#if !os(Windows)
internal func system_flock(_ fd: Int32, _ operation: Int32) -> Int32 {
#if ENABLE_MOCKING
if mockingEnabled { return _mock(fd, operation) }
#endif
return flock(fd, operation)
}
#endif

#if !os(Windows)
internal func system_fcntl(_ fd: Int32, _ cmd: Int32) -> Int32 {
#if ENABLE_MOCKING
if mockingEnabled { return _mock(fd, cmd) }
#endif
return fcntl(fd, cmd)
}

internal func system_fcntl(_ fd: Int32, _ cmd: Int32, _ arg: Int32) -> Int32 {
#if ENABLE_MOCKING
if mockingEnabled { return _mock(fd, cmd, arg) }
#endif
return fcntl(fd, cmd, arg)
}

internal func system_fcntl(
_ fd: Int32, _ cmd: Int32, _ arg: UnsafeMutableRawPointer
) -> Int32 {
#if ENABLE_MOCKING
if mockingEnabled { return _mock(fd, cmd, arg) }
#endif
return fcntl(fd, cmd, arg)
}
#endif
27 changes: 26 additions & 1 deletion Tests/SystemTests/FileOperationsTest.swift
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ final class FileOperationsTest: XCTestCase {
let writeBuf = UnsafeRawBufferPointer(rawBuf)
let writeBufAddr = writeBuf.baseAddress

let syscallTestCases: Array<MockTestCase> = [
var syscallTestCases: Array<MockTestCase> = [
MockTestCase(name: "open", .interruptable, "a path", O_RDWR | O_APPEND) {
retryOnInterrupt in
_ = try FileDescriptor.open(
Expand Down Expand Up @@ -83,6 +83,27 @@ final class FileOperationsTest: XCTestCase {
},
]

#if !os(Windows)
syscallTestCases.append(contentsOf: [
// flock
MockTestCase(name: "flock", .interruptable, rawFD, LOCK_SH) { retryOnInterrupt in
_ = try fd.lock(exclusive: false, nonBlocking: false, retryOnInterrupt: retryOnInterrupt)
},
MockTestCase(name: "flock", .interruptable, rawFD, LOCK_SH | LOCK_NB) { retryOnInterrupt in
_ = try fd.lock(exclusive: false, nonBlocking: true, retryOnInterrupt: retryOnInterrupt)
},
MockTestCase(name: "flock", .interruptable, rawFD, LOCK_EX) { retryOnInterrupt in
_ = try fd.lock(exclusive: true, nonBlocking: false, retryOnInterrupt: retryOnInterrupt)
},
MockTestCase(name: "flock", .interruptable, rawFD, LOCK_EX | LOCK_NB) { retryOnInterrupt in
_ = try fd.lock(exclusive: true, nonBlocking: true, retryOnInterrupt: retryOnInterrupt)
},
MockTestCase(name: "flock", .interruptable, rawFD, LOCK_UN) { retryOnInterrupt in
_ = try fd.unlock(retryOnInterrupt: retryOnInterrupt)
},
])
#endif

for test in syscallTestCases { test.runAllTests() }
}

Expand Down Expand Up @@ -203,6 +224,10 @@ final class FileOperationsTest: XCTestCase {
XCTAssertEqual(readBytesAfterTruncation, Array("ab".utf8))
}
}

func testFlock() throws {
// TODO: We need multiple processes in order to test blocking behavior
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's been many years since I wrote a flock(2) wrapper, but in those tests I ended up shelling out to flock(1) as an interop test on Linux. I guess we won't be able to do anything like that in this project?

What do you have in mind for this test?

We could potentially at least check the lock-lock, lock-unlock, and lock-upgrade flows?

Given that two independent file descriptors for the same underlying file can still block each other there are some tests we can write in a single process.

Copy link
Contributor Author

@milseman milseman Oct 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should pursue OFD locks which would be easier to check. Two separate open calls in the same process should have different lock sets. When we get fork support we can also test some behavior there.

}
#endif
}