jbuild-ignore not respected when there's a nested symlink

[jaredly]

Directory structure

jbuild (empty)
jbuild-ignore (contents: node_modules)
node_modules/
  bs-platform (is a symlink to ~/.nvm/versions/node/v7.6.0/lib/node_modules/bs-platform

in this case, jbuilder clearly does a ton of traversing, because it takes 15s to complete

$ time jbuilder build                                                                                                                                                                                                                                  Error: exception Stack overflow
Backtrace:
Raised at file "src/import.ml", line 414, characters 36-37
Called from file "src/future.ml", line 657, characters 6-29
Called from file "vendor/cmdliner/src/cmdliner_term.ml", line 27, characters 19-24
Called from file "vendor/cmdliner/src/cmdliner.ml", line 106, characters 32-39
Called from file "vendor/cmdliner/src/cmdliner.ml", line 136, characters 18-36
Called from file "vendor/cmdliner/src/cmdliner.ml", line 251, characters 22-48
Called from file "bin/main.ml", line 909, characters 10-51
jbuilder build  1.37s user 13.44s system 98% cpu 15.095 tota

If instead of a symlink, bs-plaftorm is copied into the node_modules directory, then there's no problem.

$ time jbuilder build
jbuilder build  0.02s user 0.03s system 20% cpu 0.267 total

It's weird to me that jbuilder would be looking inside of an ignored folder, and behavior would differ based on the contents

[ghost]

Hmm, so the jbuild-ignore files are interpreted after scanning the file system, so it is a bit mysterious to me why we observe this difference in behaviour.

The reason jbuilder looks inside ignored folders is that it is sometimes the case that one ignores a folder but still depends on files inside it, so jbuilder must still setup copy rules for all files inside ignored folders. This is used for instance to sandbox an external build system, and this is used in jbuilder itself for blackbox testing.

We should probably not follow symlinks when scanning the build system, it just seems simpler.

[copy]

I'm running into a related issue: In my case the node_modules is itself a symlink, and it's so large (300k files) that it causes a stackoverflow:

jbuilder build jsoo_hashtbl.bc.js
Error: exception Stack overflow
Backtrace:
Raised by primitive operation at file "pervasives.ml", line 200, characters 10-32
Called from file "src/path.ml", line 183, characters 15-22
Called from file "list.ml", line 67, characters 20-23
Called from file "list.ml", line 67, characters 32-39
Called from file "list.ml", line 67, characters 32-39
Called from file "list.ml", line 67, characters 32-39
…

The reason jbuilder looks inside ignored folders is that it is sometimes the case that one ignores a folder but still depends on files inside it

I'd say ignoring a folder should imply not looking into it.

In my case I the workaround is to simply build in the source directory.

[rgrinberg]

We should probably not follow symlinks when scanning the build system, it just seems simpler.

Some people rely on symlinking to replace pinning as part of their workflow. Seems like this would break things for them.

[ttamttam]

The reason jbuilder looks inside ignored folders is that it is sometimes the case that one ignores a folder but still depends on files inside it, so jbuilder must still setup copy rules for all files inside ignored folders. This is used for instance to sandbox an external build system, and this is used in jbuilder itself for blackbox testing.

We should probably not follow symlinks when scanning the build system, it just seems simpler.

I was beaten by this, because for some tests, I had a huge remote (over ADSL link) shared folder mounted as a jbuild-ignored sub-directory. I would find ridiculous/overkill to add a jbuild-really-ignore file.

The fact that jbuilder looks inside ignored folders is really not intuitive: I suggest that a warning about this is added into the documentation (I'll have a look at this)?

[ghost]

Now that we setup rules lazily, we can scan the file system lazily as well. This way jbuilder will only look inside ignored directories if it needs to.