RFC: Optionally support memory-mapping DirectoryStore values

[jakirkham]

Fixes #265
Depends on #378 & #379 & #380

Adds the memmap option to DirectoryStore, which defaults to False (current behavior). If set to True, this will memory-map each value returned instead of loading it all into memory. This behavior is extended to NestedDirectoryStore and TempStore.

Given the recent work in PR ( zarr-developers/numcodecs#128 ) and PR ( #352 ), this just works out-of-the-box. The tests needed some adjustment to be a bit more flexible as they were strictly requiring bytes before. Also had to implement pop and popitem to ensure data was copied into memory before files were removed.

xref: https://github.com/zarr-developers/zarr/issues/320
xref: #321
xref: https://github.com/zarr-developers/zarr/issues/334

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• Docs build locally (e.g., run tox -e docs)
• AppVeyor and Travis CI passes
• Test coverage is 100% (Coveralls passes)

[alimanfoo]

Thanks, very helpful to see this.

I'm realising that I don't really understand memory mapped files very well at all, so these questions might not make sense. But these come to mind:

• What do we gain by using numpy.memmap rather than mmap directly?
• With memory-mapped files, the underlying file needs to be kept open (is that true?). If lots of chunks are being read, could this cause any problems due to lots of open files?
• What about writing? Would it ever make sense to also write data via a memory map?

One thing I'm really not clear about is the division of responsibility between operating system and Python in terms of who manages the memory mapping and owns the memory. I believe it is the operating system, but I'm not clear.

If this is the case, what happens when two threads (or processes) open a memory map onto the same file? Do they access the same memory map, or are two separate memory maps created?

What about if a single thread (or process) sequentially opens a memory map onto the same file multiple times? Will the same memory map be re-used, or will it be created again from scratch each time?

[jakirkham]

Thanks for giving this a quick look. These are good questions. They forced me to think a bit more about this problem, which has been really helpful. Sorry for writing so much.

What do we gain by using numpy.memmap rather than mmap directly?

Initially this just seemed like the easiest way to go. After digging into the NumPy code a bit, I think we are saving a little on boilerplate as NumPy is just using mmap under the hood. You're probably right that using mmap directly is fine.

With memory-mapped files, the underlying file needs to be kept open (is that true?).

At the system level the file does need to be kept open for the memory-mapped file to work.

That said, Python specifically duplicates the file descriptor of the file when constructing the mmap object; so, can close the original file in Python (NumPy leverages this in memmap).

If lots of chunks are being read, could this cause any problems due to lots of open files?

Certainly. Though the same could be said for how we read files now. There can be an issue with having too many file descriptors in use as noted in issue 10897. However NumPy's memmap avoids this by just closing the old file and we could too if we use mmap.

The main problem we can run into is if we don't explicitly close the memory-mapped file. The memory-mapped file will of course close itself when it scopes out (ref count drops to zero or garbage collection gets to it). However we could still have these things lingering in the interim, which would affect the number of used file descriptors. IDK how much of an issue this would be in practice given these likely get closed quickly (due to ref counting), but it is worth keeping in mind.

What about writing? Would it ever make sense to also write data via a memory map?

It could, but I don't think it makes sense in our current implementation of DirectoryStore. This would be more useful if we wrote directly into chunks. It also could make sense if we kept memory-maps open for a while in read-write mode leveraging the cache described above.

Maybe this is a good argument for a separate store? If we did go that way, we might want to give some thought to how we compose a new store like this with other directory layouts. We'd also want to think about what it means to make an atomic change in this setup and whether that is important.

One thing I'm really not clear about is the division of responsibility between operating system and Python in terms of who manages the memory mapping and owns the memory. I believe it is the operating system, but I'm not clear.

That sounds right to me. FWIU the operating system reads from the file into kernel space and shares that memory segment with the process. With a typical read there would be a system call to read followed by a copy of the data from kernel space into process space. Memory-mapping avoids the system calls and copying. Though this is oversimplifying a lot.

Should add the memory usage still gets counted against your process somehow.

If this is the case, what happens when two threads (or processes) open a memory map onto the same file? Do they access the same memory map, or are two separate memory maps created?

As the memory used is in the kernel (not in the process) this can be reused easily not only across threads, but also across processes (as long as sharing is enabled when creating the memory-map). Memory-mapping is actually a convenient way to create process shared memory generally.

What about if a single thread (or process) sequentially opens a memory map onto the same file multiple times? Will the same memory map be re-used, or will it be created again from scratch each time?

It's worth point out that generally opening and closing a file multiple times comes at a cost (whether or not it is memory mapped), which we already do. Certainly memory mapping a file each time has its own cost as well. To cutdown on these we might consider having an LRU cache of file or memory map objects. ( #381 ) This would be a good way to control the file descriptor issue described earlier as well.

[jakirkham]

As a side note, would be good to get the 3 testing PRs this depends on (listed in the OP) reviewed and hopefully merged in the interim. Those seems like useful additions independent of this PR.

[alimanfoo]

Thanks @jakirkham for the info, much appreciated - learning a lot. Funny that you can interact with the mmap after the original file is closed, I did see that in the numpy memmap source code but wasn't sure I was seeing it right. I.e., this works:

In [4]: with open('test-mmap.dat', 'r+b') as f: 
   ...:     mm = mmap.mmap(f.fileno(), 0) 
   ...:                                                                                                                                                       

In [5]: mm[:]                                                                                                                                                 
Out[5]: b'asdlkjdaslkjdasljkdasjlkdas\n'

In [6]: mm[:5] = b'aaaaa'                                                                                                                                     

I'll try to look at the upstream PRs ASAP.

[jakirkham]

Right that’s because mmap duplicates the file descriptor. It’s both useful and odd.

[alimanfoo]

Just a short note to say that in reading around memory-mapped files, I happened again on information about LMDB (home page, wikipedia page). I don't understand the details but get the sense that if we want to explore the potential benefits of memory-mapping then we may do well to build on existing key-value store implementations where a lot of thought may already have gone into managing memory maps while also providing other useful properties like support for concurrent access. Zarr already has an LMDBStore class, which I have found extremely fast in some casual benchmarks (e.g., substantially faster then berkeley DB). LevelDB also comes up in discussions of memory-mapping, Zarr doesn't have a store class for this yet but would probably be straightforward via plyvel.

This is not to stifle exploration of possible uses of Python's mmap module, but just thought I'd make this connection.

[jhamman]

Btw, it also sounds like MongoDB has a memory mapping storage backend. This will also be available once #372 is completed.

[jbaayen]

Is there a plan to merge this PR at some point?

[jakirkham]

I don't personally have time to work on this. Though if this is of interest to you and you are willing to take it up, would encourage you to resubmit and will close this out

[jbaayen]

This can be implemented in recent Zarr versions using a simple subclass:

class MemoryMappedDirectoryStore(DirectoryStore):
    def _fromfile(self, fn):
        return memoryview(np.memmap(fn, mode="r"))

[joshmoore]

Interesting, so @jakirkham fixed it himself in #503? 😄

[joshmoore]

Shall we close then @jakirkham or is there anything you want to capture? Maybe just @jbaayen's example in the tutorial?

[jakirkham]

Possibly. Though we may want this supported in NestedDirectoryStore as well, which would make this still desirable

[joshmoore]

Hmmm.... I thought that would have worked through inheritance. Ok. Happy to leave this open until someone gets a chance to look at it more closely.

[jakirkham]

Ah meaning one should inherit from NestedDirectoryStore and then override this method? Sure I suppose that would work

[jakirkham]

FWIW a variant using the Python builtin mmap instead of np.memmap could be

class MemoryMappedDirectoryStore(DirectoryStore):
    def _fromfile(self, fn):
        with open(fn, "rb") as fh:
            return memoryview(mmap.mmap(fh.fileno(), 0, prot=mmap.PROT_READ))

[jakirkham]

Filed a doc issue ( #1245 )