Error: EXDEV: cross-device link not permitted, rename #19

zachariahtimothy · 2016-11-03T23:34:00Z

Thank you for creating this image, I have seen many people recommending it. When I use this image however my build fails, but only when using a local meteor package. It works fine with no local packages, so I am absolutely stumped at what the problem is. I created a dummy meteor app and build the image just fine, then added a dummy package, verified meteor build works, and then it still fails. Following is the full stack trace if it is useful. I do not believe the error is with your image, but I am at my wits end and I appreciate ANY help or suggestions you have.
Thanks in advance!

`
/root/.meteor/packages/ecmascript/.0.5.9.ui3230++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:190
throw error;
^

Error: EXDEV: cross-device link not permitted, rename '/opt/meteor/src/.meteor/local/isopacks/pigknows_conversion-collections' -> '/opt/meteor/src/.meteor/local/isopacks/pigknows_conversion-collections-garbage-p6wxcy'
at Error (native)
at Object.fs.renameSync (fs.js:681:18)
at Object.wrapper [as rename] (/tools/fs/files.js:1531:35)
at Object.files.renameDirAlmostAtomically (/tools/fs/files.js:963:11)
at Builder.complete (/tools/isobuild/builder.js:608:13)
at /tools/isobuild/isopack.js:1679:15
at Isopack.saveToPath (/tools/isobuild/isopack.js:1191:9)
at /tools/isobuild/isopack-cache.js:378:23
at /tools/utils/buildmessage.js:359:18
at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
at /tools/utils/buildmessage.js:352:34
at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
at /tools/utils/buildmessage.js:350:23
at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
at Object.enterJob (/tools/utils/buildmessage.js:324:26)
at IsopackCache.loadLocalPackage (/tools/isobuild/isopack-cache.js:315:18)
at /tools/isobuild/isopack-cache.js:249:16
at Function.time (/tools/tool-env/profile.js:305:10)
at /tools/isobuild/isopack-cache.js:248:17
at /tools/utils/buildmessage.js:359:18
at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
at /tools/utils/buildmessage.js:352:34
at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
at /tools/utils/buildmessage.js:350:23
at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
at Object.enterJob (/tools/utils/buildmessage.js:324:26)
at IsopackCache.ensurePackageLoaded (/tools/isobuild/isopack-cache.js:239:20)
at /tools/isobuild/isopack-cache.js:76:14
at /tools/packaging/package-map.js:57:7
at Function..each..forEach (/root/.meteor/packages/meteor-tool/.1.4.2.1r0536n++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/lib/node_modules/underscore/underscore.js:87:22)
at [object Object].eachPackage (/tools/packaging/package-map.js:49:7)
at IsopackCache.buildLocalPackages (/tools/isobuild/isopack-cache.js:75:24)
at /tools/project-context.js:809:25
at /tools/utils/buildmessage.js:359:18
at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
at /tools/utils/buildmessage.js:352:34
at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
at /tools/utils/buildmessage.js:350:23
at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
at Object.enterJob (/tools/utils/buildmessage.js:324:26)
at ProjectContext._buildLocalPackages (/tools/project-context.js:808:18)
at /tools/project-context.js:268:9
at /tools/utils/buildmessage.js:359:18
at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
at /tools/utils/buildmessage.js:352:34
at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
at /tools/utils/buildmessage.js:350:23
at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
at Object.enterJob (/tools/utils/buildmessage.js:324:26)
at ProjectContext._completeStagesThrough (/tools/project-context.js:258:18)
at /tools/project-context.js:250:12
at Function.run (/tools/tool-env/profile.js:490:12)
at ProjectContext.prepareProjectForBuild (/tools/project-context.js:249:13)
at /tools/cli/commands.js:854:20
at /tools/utils/buildmessage.js:271:13
at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
at /tools/utils/buildmessage.js:264:29
at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
at /tools/utils/buildmessage.js:262:18
at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
at /tools/utils/buildmessage.js:253:23
at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
at Object.capture (/tools/utils/buildmessage.js:252:19)
at Object.main.captureAndExit (/tools/cli/main.js:273:29)
at buildCommand (/tools/cli/commands.js:853:8)
at Command.func (/tools/cli/commands.js:802:12)
at /tools/cli/main.js:1454:23
`

0o-de-lally · 2016-11-04T15:39:50Z

@zachariahtimothy I've just tested a build on docker cloud, and I don't have this issue.
I'm using the 'classic ui' interface on docker cloud. And I'm building on my own machines instead of dockers system. My machine is a Packet.net standard ubuntu. Ubuntu 14.04 LTS

@jshimko we can close this since I can't reproduce.

zachariahtimothy · 2016-11-04T15:51:14Z

@keyscores The old "works on my machine" excuse eh? I appreciate you looking into it, but to close this because your totally different environment works is unreasonable. I am not using Docker Cloud or Ubuntu.
All I am trying to accomplish is firing up a simple Meteor app (with local packages) in Docker.

jshimko · 2016-11-04T15:55:27Z

I've been using these build scripts for about 6 months with a bunch of different apps and building them on a variety of platforms (Linux, Mac, Windows, Docker Cloud, Docker Hub, etc). I've never personally had the issue above. Where are you trying to build this image?

zachariahtimothy · 2016-11-05T12:59:29Z

I am just trying to build this image locally on my Macbook Pro. I have docker installed and updated via brew cask.

Create a new Meteor project an use this image to build, it works.
Add third party libraries to the project from Atmosphere, it still works.
Create local package via meteor create --package test-pack (without doing meteor add test-pack) it still works.
Once meteor add test-pack is added, the error kills the docker build.

jshimko · 2016-11-05T13:42:12Z

What version of Docker? If it's not at least 1.12.x, I'd start there. Have you tried installing Docker For Mac? That's the officially supported way to install Docker now and I'm pretty sure the VM is significantly different than the old boot2docker VM you'll get with the Homebrew install.

jshimko · 2016-11-05T13:57:45Z

Also, your exact reproduction steps above don't work without modifications. The package.js in the sample package that gets generated links files that don't exist yet within that package. And that crashes Meteor when you add the package (and obviously the Docker build too).

I know you've used actual functioning packages previously, but the details of exact reproductions are important when you're the only one seeing the issue. Nobody else is running your private internal packages, so it's hard to guess at the problem without as much info as possible.

So try installing Docker for Mac first. If that still doesn't work, it'd be helpful if you could push up a Github repo that reproduces the problem so other people can clone/test it. Thanks!

jshimko · 2016-11-05T14:23:30Z

I was able to reproduce this.

Sounds like this is likely the explaination...
npm/npm#9863 (comment)

This appears to be because aufs (Docker's default filesystem) will often cause rename(2) to return EXDEV, regardless of apparent aufs device boundaries. From the aufs manual:

To rename(2) directory may return EXDEV even if both of src and tgt are on the same aufs. When the rename-src dir exists on multiple branches and the lower dir has child(ren), aufs has to copyup all his children. It can be recursive copyup. Current aufs does not support such huge copyup operation at one time in kernel space, instead produces a warning and returns EXDEV. Generally, mv(1) detects this error and tries mkdir(2) and rename(2) or copy/unlink recursively. So the result is harmless. If your application which issues rename(2) for a directory does not support EXDEV, it will not work on aufs. Also this specification is applied to the case when the src directory exists on the lower readonly branch and it has child(ren).

And that's unfortunately a little complicated to address because this is Meteor internals using the version of npm/node that is bundled with Meteor. Patching that would require a PR to get accepted or we'd have to use a fork of Meteor (no thanks).

So it looks like I need to provide a way to build the container with a pre-built app bundle if we want this fixed right now. That would get around installing Meteor and building the app in the container. One more manual step for the end user, but probably much more stable inside the container since you're just adding prebuilt js files and running vanilla Node. No install/build steps.

Let me chew on that for the weekend and get back to you.

zachariahtimothy · 2016-11-06T16:39:16Z

Thank you so much for looking further into this. I am using Docker for Mac
installed via homebrew cask but it is 1.12.3. I read the same with regards
to aufs and I see you can change the storage engine but I could not seem to
find how to do that.
For now your solution is what I am trying. Image lifecycle wise I was
planning to replicate what you have but either commit changes to the image
manually, or do what you mentioned and set it up to install a pre-built
Meteor bundle. Seems like the latter is the most reasonable.
Thanks again and have a great weekend!

Zach Curtis
P: 970.590.6383
E: [email protected]
T: @zachariahCurtis https://twitter.com/#!/@zachariahCurtis

On Sat, Nov 5, 2016 at 8:23 AM, Jeremy Shimko [email protected]
wrote:

I was able to reproduce this.

Sounds like this is likely the explaination...
npm/npm#9863 (comment)
npm/npm#9863 (comment)

This appears to be because aufs (Docker's default filesystem) will often
cause rename(2) to return EXDEV, regardless of apparent aufs device
boundaries. From the aufs manual:

To rename(2) directory may return EXDEV even if both of src and tgt are on
the same aufs. When the rename-src dir exists on multiple branches and the
lower dir has child(ren), aufs has to copyup all his children. It can be
recursive copyup. Current aufs does not support such huge copyup operation
at one time in kernel space, instead produces a warning and returns EXDEV.
Generally, mv(1) detects this error and tries mkdir(2) and rename(2) or
copy/unlink recursively. So the result is harmless. If your application
which issues rename(2) for a directory does not support EXDEV, it will not
work on aufs. Also this specification is applied to the case when the src
directory exists on the lower readonly branch and it has child(ren).

And that's unfortunately a little complicated to address because this is
Meteor internals using the version of npm/node that is bundled with Meteor.
Patching that would require a PR to get accepted or we'd have to use a fork
of Meteor (no thanks).

So it looks like I need to provide a way to build the container with a
pre-built app bundle if we want this fixed right now. That would get around
installing Meteor and building the app in the container. One more manual
step for the end user, but probably much more stable inside the container
since you're just adding prebuilt js files and running vanilla Node. No
install/build steps.

Let me chew on that for the weekend and get back to you.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#19 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABSNi7wgouabo6Zila4C11OXXLCGQYKtks5q7JFigaJpZM4KpCVo
.

Jank1310 · 2016-11-07T18:24:42Z

I think in meteord this issue is handled by copying the files during the build step, but I'm not 100 percent sure...

From https://github.com/abernix/meteord/blob/master/base/scripts/lib/build_app.sh

# sometimes, directly copied folder cause some wierd issues
# this fixes that
echo "=> Copying the app"
cp -R /app $COPIED_APP_PATH
cd $COPIED_APP_PATH

igor90007 · 2016-12-15T16:33:57Z

I had similar error. (Windows 10 Home). Try to delete "local" folder.

rsercano · 2017-01-12T11:21:48Z

Hello @jshimko

thanks for this great image, I've decided to use this image on mongoclient since I'm not an expert on docker. And it unfortunately fails on docker hub (however it's working on local builds) with this error. Check here, at the very bottom of page you can see the details.

p.s. this is an automated build.

Apart from that, as far as I remember on windows if you try to build meteor application placed under D: while you have installed meteor to C: it throws exact same error. Can be related.

Thanks in advance

rsercano · 2017-01-13T06:02:39Z

I copied script files and changed Dockerfile a bit for mongoclient, then somehow it started working

Netherdrake · 2017-02-18T14:36:57Z

I am now getting a similar error:

[-] Building Meteor application...

/root/.meteor/packages/less/.2.5.7.1mii3e6++os+web.browser+web.cordova/plugin.compileLessBatch.os/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:116
      throw error;
      ^

Error: EXDEV: cross-device link not permitted, rename '/opt/meteor/dist/bundle' -> '/opt/meteor/dist/bundle-garbage-zo73m2'
    at Error (native)
    at Object.fs.renameSync (fs.js:681:18)
    at Object.wrapper [as rename] (/tools/fs/files.js:1535:35)
    at Object.files.renameDirAlmostAtomically (/tools/fs/files.js:965:11)
    at Builder.complete (/tools/isobuild/builder.js:687:13)
    at /tools/isobuild/bundler.js:2492:13
    at /tools/isobuild/bundler.js:2399:7
    at /tools/isobuild/bundler.js:2800:22
    at /tools/utils/buildmessage.js:271:13
    at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
    at /tools/utils/buildmessage.js:264:29
    at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
    at /tools/utils/buildmessage.js:262:18
    at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
    at /tools/utils/buildmessage.js:253:23
    at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
    at Object.capture (/tools/utils/buildmessage.js:252:19)
    at bundle (/tools/isobuild/bundler.js:2633:31)
    at /tools/isobuild/bundler.js:2580:32
    at Object.withCache (/tools/fs/files.js:1601:12)
    at Object.exports.bundle (/tools/isobuild/bundler.js:2580:16)
    at buildCommand (/tools/cli/commands.js:952:30)
    at Command.func (/tools/cli/commands.js:827:12)
    at /tools/cli/main.js:1483:23
The command '/bin/sh -c cd $APP_SOURCE_DIR &&             bash $BUILD_SCRIPTS_DIR/install-meteor.sh &&             bash $BUILD_SCRIPTS_DIR/build-meteor.sh &&             bash $BUILD_SCRIPTS_DIR/post-build-cleanup.sh' returned a non-zero code: 1

Docker info:

Containers: 11
 Running: 0
 Paused: 0
 Stopped: 11
Images: 93
Server Version: 1.13.1
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.4.49-1-lts
Operating System: Antergos Linux
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.37 GiB
Name: arch2
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: furion
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

jshimko · 2017-02-18T15:51:34Z

Yeah, this problem has been haunting me for months and Docker 1.13 just started causing it to happen every time I build locally on macOS (seems to work fine when on a linux box).

I'm fairly certain this is the issue here:
meteor/meteor#7852

And the root cause from Docker:
https://docs.docker.com/engine/userguide/storagedriver/aufs-driver/#/renaming-directories-with-the-aufs-storage-driver

This is the code causing it in Meteor: https://github.com/meteor/meteor/blob/b40df12129fa6b3545f89394723fcf044b47f78d/tools/fs/files.js#L950-L978

I think there's a possibility that using the mv method from fs-extra may fix it (since there's error handling specifically for that error)

The only reason I haven't confirmed this yet is because I'm going to have to fork and patch Meteor to test it. I just haven't had the time in the last couple weeks. However, it's officially blocking several projects for me at this point, so I'm bumping it up the priority list. I'll post here and on the Meteor issue with my results.

Thanks for taking the time to report your details!

jshimko · 2017-03-15T17:30:58Z

Well, it took a while, but I finally fixed this. A PR has been submitted to Meteor.

meteor/meteor#8491

And now we wait.

wsimmonds · 2017-03-19T18:24:20Z

@jshimko: Awesome, thanks for working on this and submitting the PR to Meteor - hopefully they will merge/release it shortly.

I temporarily worked around the problem by forking the repo and appending the following to install-meteor.sh:

if [ "$METEOR_VERSION" = "1.4.2.3" ]; then
    printf "\n[-] Setting up temporary workaround for EXDEV issue\n\n.."
    cd ~/.meteor/packages/meteor-tool/1.4.2_3/mt-os.linux.x86_64/tools && npm install fs-extra
    sed -i 's/require("fs");/require("fs");\nfs.move=require("fs-extra").move;\nfs.moveSync=require("fs-extra").moveSync;/g' ~/.meteor/packages/meteor-tool/1.4.2_3/mt-os.linux.x86_64/tools/fs/files.js
    sed -i 's/files.rename(toDir, garbageDir);/files.move(toDir, garbageDir);/g' ~/.meteor/packages/meteor-tool/1.4.2_3/mt-os.linux.x86_64/tools/fs/files.js
    sed -i 's/files.rename(fromDir, toDir);/files.move(fromDir, toDir);/g' ~/.meteor/packages/meteor-tool/1.4.2_3/mt-os.linux.x86_64/tools/fs/files.js
    sed -ie 's/wrapFsFunc("rename", \[0, 1\]);/wrapFsFunc("rename", \[0, 1\]);\nwrapFsFunc("move", \[0\]);/g' ~/.meteor/packages/meteor-tool/1.4.2_3/mt-os.linux.x86_64/tools/fs/files.js
  fi

Which simply:

Steals the code from your Meteor PR and introduces it in a very hacky way
Works for just Meteor 1.4.2.3 (version I'm setting up a build for)

This has got the build functioning on my CI server, so your fix looks good to me!

jshimko · 2017-03-19T20:37:45Z

Ha! Nice. That's probably as hacky as it gets, but whatever works!

I have no idea if they're going to even consider merging my fix (still no response to the PR after 4 days). I'm extremely invested in Meteor for a bunch of projects at this point, so I'm not giving up on finding a solution for this. But I also need my builds to work. If I have to, I'll be maintaining a fork of Meteor that has my fix and stays up to date with upstream moving forward. I really only need the changes inside my Docker builds, so nobody has to deal with local installs of the fork. The process can be baked into this base image. Definitely not my ideal workflow, but I haven't come up with a better solution for dealing with the Meteor folks not being willing to merge my fix. I'm certainly open to suggestions if anyone has any better ideas.

In the meantime, if my PR would solve problems for anyone reading this, it'd be great if you could take a few mins to mention that over in the PR comments. This bug doesn't appear to be an issue in their build environments, so it doesn't seem to be high priority for them at this point. The original issue that was filed for this was tagged as can't-reproduce, has-workaround, and impact:few (all of which I haven't found to be true), so it's likely to get put on the back burner until those assumptions are changed. A little reinforcement from other people that it's actually an issue would probably help things along.

Thanks!

jshimko · 2017-03-22T17:14:17Z

My fix was merged over at meteor/meteor#8491 and tagged for the Release 1.4.4 milestone.

Once Meteor 1.4.4 lands (hopefully soon), this issue should finally be fixed!

kokokenada · 2017-04-16T17:18:20Z

Confirming that after a meteor update to 1.4.4.1 this problem was resolved. Thanks.

jshimko · 2017-04-17T13:48:56Z

Excellent. Thanks for the update @kokokenada.

This appears to finally be fixed by Meteor >= 1.4.4. I'm going to close this issue, but feel free to reopen it if you ever run into the problem again.

Thanks for your patience everyone!

jshimko · 2017-05-04T12:13:57Z

This is not the same error and this issue is closed, so you may want to open a new one to track this problem. It's really easy to miss new comments on a closed issue.

jshimko mentioned this issue Mar 7, 2017

Can't build docker image on MacOS Sierra reactioncommerce/reaction#1946

Closed

rsercano mentioned this issue Mar 17, 2017

How to bind a volume to internally installed mongodb's data path #36

Closed

jshimko changed the title ~~Error: EXDEV: cross-device link not permitted, rename When using local Meteor package~~ Error: EXDEV: cross-device link not permitted, rename Mar 17, 2017

jshimko closed this as completed Apr 17, 2017

maurice-schleussinger mentioned this issue Jul 11, 2018

Error: EXDEV: cross-device link not permitted #107

Open

Maddin-619 mentioned this issue Mar 11, 2020

grafana-toolkit: use fs-extra instead of fs grafana/grafana#22723

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error: EXDEV: cross-device link not permitted, rename #19

Error: EXDEV: cross-device link not permitted, rename #19

zachariahtimothy commented Nov 3, 2016 •

edited

Loading

0o-de-lally commented Nov 4, 2016

zachariahtimothy commented Nov 4, 2016

jshimko commented Nov 4, 2016

zachariahtimothy commented Nov 5, 2016

jshimko commented Nov 5, 2016

jshimko commented Nov 5, 2016

jshimko commented Nov 5, 2016

zachariahtimothy commented Nov 6, 2016

Jank1310 commented Nov 7, 2016

igor90007 commented Dec 15, 2016 •

edited

Loading

rsercano commented Jan 12, 2017 •

edited

Loading

rsercano commented Jan 13, 2017 •

edited

Loading

Netherdrake commented Feb 18, 2017

jshimko commented Feb 18, 2017

jshimko commented Mar 15, 2017

wsimmonds commented Mar 19, 2017

jshimko commented Mar 19, 2017

jshimko commented Mar 22, 2017

kokokenada commented Apr 16, 2017

jshimko commented Apr 17, 2017

jshimko commented May 4, 2017

Error: EXDEV: cross-device link not permitted, rename #19

Error: EXDEV: cross-device link not permitted, rename #19

Comments

zachariahtimothy commented Nov 3, 2016 • edited Loading

0o-de-lally commented Nov 4, 2016

zachariahtimothy commented Nov 4, 2016

jshimko commented Nov 4, 2016

zachariahtimothy commented Nov 5, 2016

jshimko commented Nov 5, 2016

jshimko commented Nov 5, 2016

jshimko commented Nov 5, 2016

zachariahtimothy commented Nov 6, 2016

Jank1310 commented Nov 7, 2016

igor90007 commented Dec 15, 2016 • edited Loading

rsercano commented Jan 12, 2017 • edited Loading

rsercano commented Jan 13, 2017 • edited Loading

Netherdrake commented Feb 18, 2017

jshimko commented Feb 18, 2017

jshimko commented Mar 15, 2017

wsimmonds commented Mar 19, 2017

jshimko commented Mar 19, 2017

jshimko commented Mar 22, 2017

kokokenada commented Apr 16, 2017

jshimko commented Apr 17, 2017

jshimko commented May 4, 2017

zachariahtimothy commented Nov 3, 2016 •

edited

Loading

igor90007 commented Dec 15, 2016 •

edited

Loading

rsercano commented Jan 12, 2017 •

edited

Loading

rsercano commented Jan 13, 2017 •

edited

Loading