Can't run two instances of Yarn simultaneously: ENOENT: no such file or directory in .yarn-cache

[Taytay]

First, thanks for Yarn!

Using latest Yarn version:
0.16.1

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
When running multiple instances of yarn on the same machine, some instances are crashing with the error: error An unexpected error occured, please open a bug report with the information provided in ...yarn-error.log

I am starting yarn in a background process in multiple folders simultaneously on our build machine. (We used to do this with npm without issue):

Here is the error from one of the commands:

00:02:46.211 cd /home/vagrant/workspace/Evergreen_Branch_Builder/ynab_web
00:02:46.211 Starting yarn install for ynab_web...
00:02:52.751 ynab_web            : yarn install
00:02:52.751 ynab_web            : yarn install v0.16.1
00:02:52.751 ynab_web            : [1/4] Resolving packages...
00:02:52.751 ynab_web            : [2/4] Fetching packages...
00:02:52.751 ynab_web            : error An unexpected error occured, please open a bug report with the information provided in "/home/vagrant/workspace/Evergreen_Branch_Builder/ynab_web/yarn-error.log".
00:02:52.751 ynab_web            : info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

Here is a gist of the log file referenced above: https://gist.github.com/Taytay/59eced959fefc7effe5ae1a842b65218

Relevant error at the end of the log:

Trace:
  Error: https://registry.yarnpkg.com/node-uuid/-/node-uuid-1.4.7.tgz: ENOENT: no such file or directory, open '/home/vagrant/.yarn-cache/npm-node-uuid-1.4.7/.yarn-tarball.tgz'
      at Error (native)

and from the other failed Yarn command:

00:02:47.732 cd /home/vagrant/workspace/Evergreen_Branch_Builder/ynab_shared_library
00:02:47.732 Starting yarn  install for ynab_shared_library...
00:02:47.732 cd 00:02:52.269 ynab_shared_library : yarn install
00:02:52.270 ynab_shared_library : yarn install v0.16.1
00:02:52.270 ynab_shared_library : [1/4] Resolving packages...
00:02:52.270 ynab_shared_library : [2/4] Fetching packages...
00:02:52.270 ynab_shared_library : error An unexpected error occured, please open a bug report with the information provided in "/home/vagrant/workspace/Evergreen_Branch_Builder/ynab_shared_library/yarn-error.log".
00:02:52.270 ynab_shared_library : info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

yarn-log from that folder: https://gist.github.com/Taytay/652691af503445235742767181391b8c

Relevant error:

Trace:
  Error: https://registry.yarnpkg.com/node-uuid/-/node-uuid-1.4.7.tgz: ENOENT: no such file or directory, open '/home/vagrant/.yarn-cache/npm-node-uuid-1.4.7/.yarn-tarball.tgz'
      at Error (native)

If the current behavior is a bug, please provide the steps to reproduce.
It appears to happen when running yarn in separate folders simultaneously, like so:

cd folder1
yarn 2>&1
cd folder2
yarn 2>&1
cd folder3
yarn 2>&1

However, if I go into each folder in turn and run yarn, it succeeds. My guess is that there are race conditions related to downloading the packages for the cache folder. Once I have run yarn in each folder, running them in parallel works without issue.

What is the expected behavior?
No Crash

Please mention your node.js, yarn and operating system version.

Yarn version:
  0.16.1

Node version:
  4.6.1

Platform:
  linux x64

[amandogra]

I am also getting the similar error. Relevant error in yarn_error.logis:

Trace: 
  Error: https://registry.yarnpkg.com/appium-adb/-/appium-adb-1.7.5.tgz: unexpected end of file
      at Zlib._handle.onerror (zlib.js:363:17)

The command I executed:

yarn --ignore-engines --cache-folder="lib/external/yarn_cache"

yarn version: v0.16.0

[mandragorn]

similar error for the same reasons (running multiple yarn commands simultaneously)

Trace: 
  Error: http://nexus.crl.vecna.com:8081/nexus/content/groups/npm/ansi-styles/-/ansi-styles-2.2.1.tgz: ENOENT: no such file or directory, open '/home/jenkins/.yarn-cache/npm-ansi-styles-2.2.1/.yarn-tarball.tgz'

Command:

yarn

Yarn version:
0.16.1

Node version:
4.6.1

[mandragorn]

My current workaround for this is to specify a separate yarn cache for each invocation of yarn and so far its working. Not ideal, but working.

yarn --cache-folder=".yarn-cache"

[adamchainz]

Same error for two parallel invocations:

  Error: ENOTEMPTY: directory not empty, rmdir '/home/backend/.yarn-cache/npm-lodash-4.16.4'
      at Error (native)

This is important to us as our build uses multiple node_modules for different sub-applications .

[whitelynx]

Similar to @adamchainz's use case, this is also important in order for yarn to be usable inside lerna. (as evidenced by #966)

[ctavan]

The same happens when you try to run yarn installs in a docker-compose setup where you have multiple docker containers starting (and running yarn) at the same time which share a common mounted ~/.yarn-cache directory…

Being able to run multiple yarn processes which share a common cache in parallel would be really desirable.

[tmair]

Looks like this is a duplicate of #683?

[wclr]

Looks like this is a duplicate of #683?

Yes this is a duplicate, it would be cool if yarn could handle parallel installations, I believe the problem lays mostly in the yarn cache domain, if there is not special locking mechanics problems obviously will appear.

[ctavan]

Yes it indeed is a duplicate. How about closing this one in favor of the older issue? Also #683 at least provides a workaround.