Exclude root directory in file watcher(s)

[rental-ninja]

Hi!

We've got an issue while using the Build runner package. We run on Linux 64bit and when executing an script containing the execution of the build runner "watch" method we get a Operating system error:
OS Error: No space left on device, errno = 28

The output of the watch method is within the same project folder, in the main disk partition, with lots of remaining space. I can add that if we run the script just after the computer is re-started, it works, what makes us think that is a temporary space/files related issue. But if we open some other applications like the IDE, we get the erro. In addition as the error we get is an Operating System error, we think that it's like the system is wrongly believing that there is no space left somewhere.

I give you more details about the issue. This is the script containing the build runner watch method that we run:

And this is the Error we get:

And this is the status of the computer disks (main disk is the one with 240GB of space, at the 33%):

Versions details:

-Dart VM version: 2.0.0-dev.63.0 (Unknown timestamp) on "linux_x64"
-Running on Linux 64 bit
-Distro: Linux quick 4.13.0-45-generic dart-lang/watcher#50~16.04.1-Ubuntu SMP Wed May 30 11:18:27 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
-Using Chrome (but not relevant)
-build_runner: ^0.8.10. Full list of dependencies:

[jakemac53]

Some of those dependencies definitely do use temp directories - but they should be cleaning them up.

Can you look at your temp directory and see if you have anything obvious there? I would check specifically for directories with build or scratch_space in the name.

[rental-ninja]

Hi @jakemac53 !

Thanks a lot for your quick reply. I didn't find anything related to those terms in the tmp folder:

Regarding the dependencies using the temp directories, provably yes. The thing here is if the build runner watch command uses the tmp. Do you think so? I think it shouldn't, as only compiles the code from one place to the other.

thanks!

[jakemac53]

@rental-ninja if you look in those other directories that are just a random sequence of 10 characters, what do you see in them? What about the ~*.tmp directories?

[natebosch]

I think we're hitting ulimit errors adding watchers. We do have an issue to add a better error message for this but haven't implemented anything yet.

#867

The reason you get these while your IDE is running is because it also uses a bunch of file watchers.

I think it shouldn't, as only compiles the code from one place to the other.

We need to use a temp directory to make best effort at hermetic builds whenever we run a process that will be reading files. These temp directories are most likely not causing issues, the problem is the watcher limits.

[rental-ninja]

Thank you both for your replies, always very helpful! Yesterday I was taking a deep view into both options you suggested, particularly to @natebosch one. I was taking a look at this post. I could not solve it but I will continue investigating. In the meantime, if you have any other idea, like another option to solve the ulimit error, share it!

See you!

[refi64]

I just ran into this issue because of my node_modules folder, which may or may not have...uhh...91422 files...

In the mean time, is there a way to tell build_runner to skip watching certain directories?

[jakemac53]

Unfortunately the watcher api doesn't let you exclude a specific directory, so we currently watch the entire root package and then filter out the file events for things that don't match your globs.

We do need to watch some files at the top level directory... although we could potentially use a regular file watcher for those and a directory watcher for subdirectories. This would be a breaking change to our apis though and would complicate things a bit, although there would be some other benefits (not having to filter out events under .dart_tool for instance).

[matanlurey]

+1 for the breaking change @jakemac53 outlines. I think it could help lowering resource CPUs too.

[natebosch]

I think this has come up in other issues but I don't have them on hand - I want to take note of one of the difficulties we've discussed.

If we have don't have a director watcher at the root and someone adds a new directory we would care about, we won't know about it and won't be able to add the watcher on it.

[jakemac53]

What if we moved the .dart_tool/build/generated directory to some other place and just had a symlink to it sort of like bazel-out? That might solve a lot of issues.

[natebosch]

We could move it to somewhere under the user's home directory - not sure if we'd need the symlink. We'd need to include the full path to the project root under that directory so we'll have to think about path lengths on windows.

How would that impact things like CI caching, with or without the symlink?

[jakemac53]

We could move it to somewhere under the user's home directory - not sure if we'd need the symlink.

Ya I think the home dir is probably the most sensible place. The symlink would be so its easier to dive into it if you need to (even though most people won't).

We'd need to include the full path to the project root under that directory so we'll have to think about path lengths on windows.

We could just hash it too for a smaller path - in fact overall the paths would likely be shorter in some cases that way since they start at the home dir.

How would that impact things like CI caching, with or without the symlink?

We should only have to change the directory that is cached, everything else should work fine.

[evanweible-wf]

+1 to the symlink, we open .dart_tool/build/ fairly often on our team for debugging purposes so that would be nice to have.

[natebosch]

We just discussed this offline, there are some other negative side effects of a cache outside of the project:

• If you delete the project the build cache would still exist.
• If you move the project the build cache would be orphaned and a new one would be created
• It becomes trickier to share a prepopulated cache along with the project
• (minor) rm .dart_tool/build wouldn't work anymore, you'd be forced to use pub run build_runner clean

Filed dart-lang/tools#1719 to see if we can implement something better without moving the generated file cache.