In-memory FS data structure

[joliss]

Copying from Twitter. (Cc @rwjblue, @hjdivad.)

@joliss:

Making an in-memory FS data structure / API for Broccoli ecosystem (maybe core). Very simple.

file = Buffer
symlink = string
directory = Map (maps file names to Buffer/string/Map)

Must not mutate; thus === tests deep equality.

Future-proof?

Helper methods (walking etc.) would be functions in separate package(s).

diff(dir1, dir2) is cheap.

If we make it core API, Broccoli can automatically (de)materialize to/from FS on plugin boundaries as needed.

I think I'm missing some serious issue. But can't figure out what.

I considered doing proper Dir/File/Symlink classes, or {type: 'symlink', value: '/target'}-style plain objects, but if we can get away with using Map/Buffer/string directly we gain performance.

(I've come around to thinking that "plain directories on the filesystem" might've been a suboptimal choice of abstraction, due to overhead and cross-platform issues. Perhaps in the future most Broccoli plugins just communicate via in-memory objects, and physical FS becomes exception rather than rule?)

@stefanpenner:

So, we have very limited memory on node. Keeping files on disk, is very important (atleast files over some threshold).

I believe keeping the files (over some size) on disk, but keeping the symlinks in memory strikes the balance we need.

often times, we don't see performance issues with reading the real file from disk, rather we see issues:

• dirty checking
• creating/maintaining symlinks
• creating/maintaining a large number of folders

It's also worth pointing out, that reifing an entire directory tree can be costly and can often be avoided. Reducing initial scan/rebuild costs.

Example:

new Concat(new Funnel('bower_components'), { headerFiles: ['moment/moment.js ] });

[joliss]

As for memory usage, I agree most plugins should write to cachePath and return symlinks into it, instead of in-memory Buffers. We could even consider going without Buffer support to encourage this. I'm really unsure about this.

As for reifying directory trees, if we are able to do it incrementally, we should be fine, right? I'm writing a function for that right now.

[stefanpenner]

This seems to have gone stale, there are additional ways to improve build performance. We can drive that work from concrete problems, and iterate to a solution.