ENOSPC on linux

[ghost]

Whilst mostly a Babel user, I got interested in TypeScript a few weeks ago and started following @basarat on Twitter. Seeing his tweet about Alm last night prompted me to check out Alm. In subsequent communication, @thebabelion I mentioned to @basarat, that all did not go well and he requested I log issue here. Here is relevant terminal output (my own path names anonymized).

In both cases no UI appeared.

Terminal session in my TypeScript project directory
`
xxx@xxx-VirtualBox ~/xxx/myproject $ cat /etc/*-release
DISTRIB_ID=LinuxMint
DISTRIB_RELEASE=18
DISTRIB_CODENAME=sarah
DISTRIB_DESCRIPTION="Linux Mint 18 Sarah"
NAME="Linux Mint"
VERSION="18 (Sarah)"
ID=linuxmint
ID_LIKE=ubuntu
PRETTY_NAME="Linux Mint 18"
VERSION_ID="18"
HOME_URL="http://www.linuxmint.com/"
SUPPORT_URL="http://forums.linuxmint.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/linuxmint/"
UBUNTU_CODENAME=xenial
cat: /etc/upstream-release: Is a directory

xxx@xxx-VirtualBox ~/xxx/myproject $ node --version
v6.6.0

xxx@xxx-VirtualBox ~/xxx/myproject $ npm --version
3.10.7

xxx@xxx-VirtualBox ~/xxx/myproject $ alm
Version: 2.0.3
DASHBOARD: http://localhost:4444
TSCONFIG: All Good! /home/xxx/xxx/myproject/tsconfig.json
[TSC] Started Initial Error Analysis: /home/xxx/xxx/myproject/tsconfig.json
[LINT] No tslint configuration found.
[TSC] Error Analysis Duration: 1s
[TSC] FileCount: 3 Errors: 2
[TSC] Incremental Error Analysis /home/xxx/xxx//myproject/tsconfig.json
[TSC] Error Analysis Duration: 1s
[TSC] FileCount: 3 Errors: 2
events.js:160
throw er; // Unhandled 'error' event
^

Error: watch /home/xxx/xxx/tests/baselines/reference/project/sourceRootAbsolutePathSubfolderNoOutdir/amd/ref ENOSPC
at exports._errnoException (util.js:1036:11)
at FSWatcher.start (fs.js:1429:19)
at Object.fs.watch (fs.js:1456:11)
at createFsWatchInstance (/home/xxx/lib/node_modules/alm/node_modules/chokidar/lib/nodefs-handler.js:37:15)
at setFsWatchListener (/home/xxx/lib/node_modules/alm/node_modules/chokidar/lib/nodefs-handler.js:80:15)
at FSWatcher.NodeFsHandler._watchWithNodeFs (/home/xxx/lib/node_modules/alm/node_modules/chokidar/lib/nodefs-handler.js:228:14)
at FSWatcher.NodeFsHandler._handleDir (/home/xxx/lib/node_modules/alm/node_modules/chokidar/lib/nodefs-handler.js:407:19)
at FSWatcher. (/home/xxx/lib/node_modules/alm/node_modules/chokidar/lib/nodefs-handler.js:455:19)
at FSWatcher. (/home/xxx/lib/node_modules/alm/node_modules/chokidar/lib/nodefs-handler.js:460:16)
at FSReqWrap.oncomplete (fs.js:123:15)
fileListingWorker worker restarting. Don't know why it stopped with code: 1
`

Terminal session in an empty directory
`
xxx@xxx-VirtualBox ~ $ mkdir tmp
xxx@xxx-VirtualBox ~ $ cd tmp
xxx@xxx-VirtualBox ~/tmp $ alm
Version: 2.0.3
DASHBOARD: http://localhost:4445
[TSCONFIG]: No active project

`

Trusting this information is helpful.

Justin Johansson
Twitter: @thebabelion
GitHub: @johanssj

[basarat]

/home/xxx/xxx/tests/baselines/reference/project/sourceRootAbsolutePathSubfolderNoOutdir/amd/ref ENOSPC

So question : paulmillr/chokidar#294 (comment) ENOSPC means that there is no space on the drive. Is that the case?

[basarat]

Also give 2.0.5 a go npm install alm -g as I just updated chokidar to see if it fixes it 🌹

[ghost]

Just retried with 2.0.5. Alas same result; no UI appears. This time I noticed that when installing npm WARNs about chokidar. I don't know if this is relevant. Terminal transcripts follow.

<<<<< deleted long npm module listing here as proved not to relevant >>>>

Now in project directory

Observation: While my current directory in this instance is ~/workspace/myproject , it seems strange that alm is looking at a sibling directory ~/workspace/TypeScript/ in the trace below where it says 'Error: watch'. Strikes me as odd that it would do that.

`

xxx@xxx-VirtualBox ~/workspace/myproject $ alm
Version: 2.0.5
DASHBOARD: http://localhost:4444
TSCONFIG: All Good! /home/xxx/workspace/myproject/tsconfig.json
[TSC] Started Initial Error Analysis: /home/xxx/workspace/myproject/tsconfig.json
[LINT] No tslint configuration found.
[TSC] Error Analysis Duration: 1s
[TSC] FileCount: 3 Errors: 2
[TSC] Incremental Error Analysis /home/xxx/workspace/myproject/tsconfig.json
[TSC] Error Analysis Duration: 1s
[TSC] FileCount: 3 Errors: 2
events.js:160
throw er; // Unhandled 'error' event
^

Error: watch /home/xxx/workspace/TypeScript/tests/baselines/reference/project/sourceRootAbsolutePathModuleSubfolderSpecifyOutputDirectory/amd/outdir ENOSPC
at exports._errnoException (util.js:1036:11)
at FSWatcher.start (fs.js:1429:19)
at Object.fs.watch (fs.js:1456:11)
at createFsWatchInstance (/home/xxx/lib/node_modules/alm/node_modules/chokidar/lib/nodefs-handler.js:37:15)
at setFsWatchListener (/home/xxx/lib/node_modules/alm/node_modules/chokidar/lib/nodefs-handler.js:80:15)
at FSWatcher.NodeFsHandler._watchWithNodeFs (/home/xxx/lib/node_modules/alm/node_modules/chokidar/lib/nodefs-handler.js:228:14)
at FSWatcher.NodeFsHandler._handleDir (/home/xxx/lib/node_modules/alm/node_modules/chokidar/lib/nodefs-handler.js:407:19)
at FSWatcher. (/home/xxx/lib/node_modules/alm/node_modules/chokidar/lib/nodefs-handler.js:455:19)
at FSWatcher. (/home/xxx/lib/node_modules/alm/node_modules/chokidar/lib/nodefs-handler.js:460:16)
at FSReqWrap.oncomplete (fs.js:123:15)
fileListingWorker worker restarting. Don't know why it stopped with code: 1
`

Now in empty directory
`

xxx@xxx-VirtualBox ~/tmp $ alm
Version: 2.0.5
DASHBOARD: http://localhost:4444
[TSCONFIG]: No active project
`

Again no UI appears; just stares me back. :-(

[basarat]

Now in empty directory

That one is actually expected. You need to open the dashboard URL DASHBOARD: http://localhost:4444 in chrome yourself. Or if you want you can run alm -o which will open chrome for you. Its a nodejs web app in the true sense i.e nodejs backend and chrome frontend. Sorry about any confusion there but its menioned in the usage : https://github.com/alm-tools/alm#usage and the quickstart : https://basarat.gitbooks.io/alm/content/quickstart.html

However why the ENOSPC happens in the first case is still not understood. That said, I think the UI is functional in that case too (you just need to open it ... or run alm -o) 🌹

[ghost]

Yeah, sorry, I just realised that. My expectation was that some UI would happen just like when you launch Atom editor from the command line. Mia culpa for not reading the docs; but then again one cannot assume people will read the docs so at least a clear message in the terminal output on start up might be a good idea. (Yes, it did say DASHBOARD: http://etc, but you know.) At least I can get sort of going with it now.

[basarat]

then again one cannot assume people will read the doc

Agreed 🌹 Fixed and released with 2.0.6 ❤️

(Screenshot shows 2.0.5 as that is the version I had while adding this fix).

[ghost]

Good on you. While I feel like a goose, I don't really think I'm a goose. Thinking about the psychology of it though, probably had I run it the empty directory scenario first up and didn't see all the stack trace messages, I would have opened up the browser straight away. Still like you allude to, the ENOSPC issue in the first case needs understanding as to why.

[basarat]

Still like you allude to, the ENOSPC issue in the first case needs understanding as to why.

Editing the issue to track that better. This is external stuff but googling around the following shows up:

• Hows the space in /tmp: http://stackoverflow.com/a/22476114/390330
• Seems to be helping a lot of people (upvotes) : http://stackoverflow.com/a/32600959/390330 / http://stackoverflow.com/a/17437601/390330

Just as an estimate how many files are in that project folder? 🌹

[ghost]

There were less than a dozen of mine but maybe it was looking into
node_modules etc which could be many.

I will do some further tests/research. Might take a day or so as have
other priorities. Will try to squeeze in asap.

On 19/09/16 14:05, Basarat Ali Syed wrote:

Still like you allude to, the ENOSPC issue in the first case needs
understanding as to why.

Editing the issue to track that better. This is external stuff but
googling around the following shows up:

• Hows the space in |/tmp|: http://stackoverflow.com/a/22476114/390330
• Seems to be helping a lot of people (upvotes) :
  http://stackoverflow.com/a/32600959/390330

Just as an estimate how many files are in that project folder? 🌹

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#202 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGNbx_ZBlwEDPdcO5rvJCmPnqlbmHu6-ks5qrhEugaJpZM4KAC_G.

[ghost]

Got a better handle on the problem now.
A bit tedious & probably a bit specific to my particular machine capacity but nevertheless ...
We are in directory ~/tmp which has nothing else but 2 alm torture-test scripts in it.

  > ~/tmp $ ls
  alm-torture-test-fail.sh  alm-torture-test-pass.sh

These scripts are:

alm-torture-test-pass.sh:

mkdir workspace-september
cd workspace-september
git clone https://github.com/DefinitelyTyped/DefinitelyTyped.git
cp -R DefinitelyTyped DefinitelyTyped2
cp -R DefinitelyTyped DefinitelyTyped3
cp -R DefinitelyTyped DefinitelyTyped4
cp -R DefinitelyTyped DefinitelyTyped5
cp -R DefinitelyTyped DefinitelyTyped6
cp -R DefinitelyTyped DefinitelyTyped7
cp -R DefinitelyTyped DefinitelyTyped8
cp -R DefinitelyTyped DefinitelyTyped9
mkdir empty
cd empty
alm

and

alm-torture-test-fail.sh: (Notice the workspace direcory name now has parens around the abbreviated month name)

rm -rf workspace-september
mkdir "workspace(sep)"
cd "workspace(sep)"
git clone https://github.com/DefinitelyTyped/DefinitelyTyped.git
cp -R DefinitelyTyped DefinitelyTyped2
cp -R DefinitelyTyped DefinitelyTyped3
cp -R DefinitelyTyped DefinitelyTyped4
cp -R DefinitelyTyped DefinitelyTyped5
mkdir empty
cd empty
alm

Running alm-torture-test-pass.sh as the name suggests is no problem.

In a workspace subdirectory, it clones a large repo off GitHub, makes 8 copies of the repo, then creates and changes into an empty directory, and then in the empty directory runs alm. All without any issues.

> ~/tmp $ ./alm-torture-test-pass.sh
Cloning into 'DefinitelyTyped'...
remote: Counting objects: 130601, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 130601 (delta 12), reused 12 (delta 12), pack-reused 130586
Receiving objects: 100% (130601/130601), 121.07 MiB | 238.00 KiB/s, done.
Resolving deltas: 100% (76234/76234), done.
Checking connectivity... done.
Version: 2.0.7
DASHBOARD: (Please open in chrome) http://localhost:4444
[TSCONFIG]: No active project

Now running alm-torture-test-fail.sh as the name suggests is a problem (at least on my machine).

Again, in a workspace subdirectory, it clones the same large repo off GitHub,
makes 4 copies of the repo (4 less than in the previous test), then creates and changes into an empty directory, and then in the empty directory runs alm. This time ENOSPC shows up.

> ~/tmp $ ./alm-torture-test-fail.sh
Cloning into 'DefinitelyTyped'...
remote: Counting objects: 130601, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 130601 (delta 12), reused 12 (delta 12), pack-reused 130586
Receiving objects: 100% (130601/130601), 121.07 MiB | 238.00 KiB/s, done.
Resolving deltas: 100% (76234/76234), done.
Checking connectivity... done.
Version: 2.0.7
DASHBOARD: (Please open in chrome) http://localhost:4444
[TSCONFIG]: No active project
events.js:160
      throw er; // Unhandled 'error' event
      ^

Error: watch /home/mars/tmp/workspace(sep)/DefinitelyTyped4/sencha_touch ENOSPC
    at exports._errnoException (util.js:1036:11)
    at FSWatcher.start (fs.js:1429:19)
    at Object.fs.watch (fs.js:1456:11)
    at createFsWatchInstance (/home/mars/lib/node_modules/alm/node_modules/chokidar/lib/nodefs-handler.js:37:15)
    at setFsWatchListener (/home/mars/lib/node_modules/alm/node_modules/chokidar/lib/nodefs-handler.js:80:15)
    at FSWatcher.NodeFsHandler._watchWithNodeFs (/home/mars/lib/node_modules/alm/node_modules/chokidar/lib/nodefs-handler.js:228:14)
    at FSWatcher.NodeFsHandler._handleDir (/home/mars/lib/node_modules/alm/node_modules/chokidar/lib/nodefs-handler.js:407:19)
    at FSWatcher.<anonymous> (/home/mars/lib/node_modules/alm/node_modules/chokidar/lib/nodefs-handler.js:455:19)
    at FSWatcher.<anonymous> (/home/mars/lib/node_modules/alm/node_modules/chokidar/lib/nodefs-handler.js:460:16)
    at FSReqWrap.oncomplete (fs.js:123:15)
fileListingWorker worker restarting. Don't know why it stopped with code: 1

Further
When the number of copies of the repo are reduced down to two, so we have DefinitelyTyped2 and DefinitelyTyped3, the problem goes away.

Conclusion
The problem manifests when there are a lot of files in sibling directories of the empty directory in which alm is run AND when a parent directory of that (empty) directory has paren characters in its name.

So now some questions, like

• why didn't the -pass script fail when there were twice of many files about?
• what do parens in directory names have to do with it?
• even with parens in directory names why does problem go away with fewer files about?
• what business of alm or it's file watcher is it to be peeking into sibling directories of the current project directory in any case.

btw. I organise my workspaces by years with 12 subdirs being each month in respective year. A symlink in my home directory called current-workspace always points to the latest month. Then because I like my workspace directory listings to appear nicely ordered, I call name individual monthly workspaces like workspace-2016-03(mar) because then I don't have to think what name of month is month 03.

I think I have spent too much time on this now but will be pleased if this information is useful to you.

Cheers
Justin Johansson

Maybe see you on Twitter, thebabellion

[basarat]

I've also added docs : https://basarat.gitbooks.io/alm/content/contributing/file-listing.html to help you and others.

what business of alm or it's file watcher is it to be peeking into sibling directories of the current project directory in any case

Sounds very weird. FWIW our external API access to the watcher is super simple :

alm/src/server/workers/fileListing/fileListingWorker.ts

Line 252 in 13ceedb

let watcher = chokidar.watch(directoryUnderWatch, { ignoreInitial: true });

fingers crossed that chokidar fixes it or someone reports it 🌹

[ghost]

I was thinking of bowing out of this issue for the moment but feel it is
too important to let slip.

Certainly it would be great if "chokidar fixes it or someone [else]
reports it".

If you keep this issue open for a while I'll try to get back to it in a
few days. Might have to fork chokidar & see what's cooking.

On 20/09/16 08:14, Basarat Ali Syed wrote:

Sounds very weird. FWIW our external API access to the watcher is
super simple :

alm/src/server/workers/fileListing/fileListingWorker.ts

Line 252 in 13ceedb

let watcher = chokidar.watch(directoryUnderWatch, { ignoreInitial: true });

fingers crossed that chokidar fixes it or someone reports it 🌹

[ghost]

btw. Were you able to replicate my test scripts with opening alm in the empty directory with all those copies of DefinitelyTyped as siblings, or did you just take my terminal output as "proof"? Admittedly the tests were a bit tedious so I understand if you couldn't afford the time to do so.

[basarat]

Admittedly the tests
were a bit tedious so I understand if you couldn't afford the time to do so.

Haven't tried it 😁

[ghost]

I've had the opportunity to dig deep into this issue having forked both alm and chokidar. Also read up on the Node FS watch API. I feel that these issues will never go away with the current watch architecture as the low-level watch API is intrinsically flakey.

Even with Eclipse IDE which I have configured to use native file system hooks, I find that sometimes my workspace does not get refreshed when it is supposed to.

Having spent over 5 years of my career writing file system drivers from encryption to security applications I think I can speak with qualified experience.

Notice that your profile says you live in Melbourne. Long time ago I was lead engineer for an anti-virus/PC security company in Melbourne. These days I'm back in Adelaide .. an easy phone call away if ever you wish to discuss. Talking is probably a more effective means of communicating than typing too many words. Anyway you are welcome to the offer if you wish. Contact via DM on Twitter, thebabellion.

[basarat]

Having spent over 5 years of my career writing file system drivers from encryption to security applications I think I can speak with qualified experience

Agree 100%. That's why offloaded to chokidar as it is good enough for most projects 🌹

[ghost]

Unfortunately though I will not be able to use alm editor. I put some console.log messages into alm file watch callbacks. Sometimes they are called when files change sometimes (mostly) no. Reboot (LinuxMint) and I'll see those hooks being called again for a while. Copy some files around, do a git clone in a sibling directory (to where alm was launched) and then flop again, no more callbacks. Just dead in the water.

[ishigami]

About the ENOSPC error on Linux, I've tried the following code and fixed the error

echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

More info at: typicode/json-server#361

[ghost]

Thanks for this.

I already had placed

|fs.inotify.max_user_watches=524288|

at the end of the file AND restarting the box

|/etc/sysctl.conf|

but do not recall ever issuing the command

|sudo sysctl -p
|
So in your opinion that should make a difference?

[ghost]

Just saw some emails from issues in paulmillr/chokidar which say
somethings that sound relevant to what I reported on this ENOSPC problem
a few weeks ago. It is encouraging that others have reported these
things & that perhaps some fixes might be happening in chokidar.

"Any news on this? Also a default-off option is welcome. I see there's
already a PR, but it was not accepted. Atm I can't use |livereload|
because I'm working in a folder containing parenthesis..."

and

"Problems with parentheses will be solved by #535
paulmillr/chokidar#535 once the apparent
windows issues there are resolved.

A different fix will still be needed for other sorts of glob characters
like asterisks."

I do remember mentioning the parentheses thing as well .. so I it
definitely was not my imagination afterall.

[basarat]

Closing as external and can't think of a fix short of reinventing chokidar which is not on the roadmap at the moment.