CLI - che mount Fails

[sherl0cks]

A number of things seem to go wrong when trying to run che mount on my machine
Reproduction Steps:

1. Install CLI per the docs (fedora 24 in my case)
2. Try to mount a workspace `che mount <local_mount_dir> <codenvy_ssh_port>

Expected behavior:
Mount takes place and file system is synced up to what is in the CodEnvy workspace.

Observed behavior:
A few odd things in the console output below, as well as other investigations

1. docker returns ip: can't find device eth0 which is odd because with centos, fedora and alpine containers on the same host and running ip a show eth0 returns something like

121: eth0@if122: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/16 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe11:2/64 scope link 
       valid_lft forever preferred_lft forever

1. debugging and print statements (including ones I added manually) from get_mount_path do not seem to print to console.
2. the result of get_mount_path in execute_che_mount appears to be empty even though that directory does exist and is 775
3. the output was only available by enabling debugging export CHE_CLI_DEBUG=true, otherwise it just failed silently...
4. docker ps -a does not show a che-mount container

Console Output

[sherl0cks@localhost CommunityProjects]$ che mount /home/sherl0cks/LabsCode/jenkins-plugin 38175
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
**ip: can't find device 'eth0'**

DEBUG: is_boot2docker
DEBUG: parse_command_line
DEBUG: load_profile
DEBUG: has_default_profile
DEBUG: execute_che_mount
**ERROR: che mount: Path provided does not exist.**

Che version: CLI says latest. Please do share if there is a way to get a specific version
OS and version: fedora 24, with selinux disabled per #2441
Docker version: 1.10.3
Che install: cli per the docs

Additional information:

FYI @TylerJewell @bmicklea

• Problem started happening recently, didn't happen in an older version of Che: dunno
• Problem can be reliably reproduced, doesn't happen randomly: 100%

[TylerJewell]

I do not think we have done any exhaustive testing of the Che CLI or our containers on Fedora. It seems that it is likely that the root of the issues are unrelated to Che mount and instead how the CLI was authored in bash.

We need to get a qa cycle readied for fedora 24 to see how it performs.

[sherl0cks]

ok that makes sense. worth noting that the working directly with the che-mount container fails as well (both with and without sudo)

[sherl0cks@localhost workspace]$ sudo docker run --cap-add SYS_ADMIN --device /dev/fuse --name che-mount -v /home/sherl0cks/LabsCode/jenkins-plugin:/mnthost:z codenvy/che-mount 172.17.0.2 39417
fuse: mount failed: Permission denied

Totally possible I'm doing something wrong here, so let me know.

[sherl0cks]

@TylerJewell has any linux distro been tested? I'm happy to try this in a VM as well

[TylerJewell]

The cli and Che mount have been tested on Ubuntu, centos, rhel. Your issue is now going to have me setup some digital ocean vms for testing. Unless you beat me to it!

[TylerJewell]

Btw - your container syntax looks perfect.

[sherl0cks]

Ok - glad the container syntax is right. I'm locked in other stuff right now but I'll free up early next week to try this on a centos and/or Ubuntu machine.

[ghost]

@sherl0cks why do you run docker with sudo?

[JamesDrummond]

@sherl0cks I don't know if it makes a difference but try the following.

sudo mkdir /root/LabsCode
sudo docker run --cap-add SYS_ADMIN --device /dev/fuse --name che-mount -v /root/LabsCode:/mnthost codenvy/che-mount 172.17.0.2 39417

If this doesn't fix it try:

sudo docker run ---cap-add SYS_ADMIN --cap-add MKNOD --device=/dev/fuse --security-opt apparmor:unconfined --name che-mount -v /root/LabsCode:/mnthost codenvy/che-mount 172.17.0.2 39417

Also, using linux/unix you should not need to mount with the above che-mount. I think it is more for systems with sshfs NOT built-in like windows. You should be able to sshfs -o allow_other,defer_permissions -p 39417 [email protected]:/projects /home/sherl0cks/LabsCode/jenkins-plugin with password secret. If you can, this is actually a better solution than che-mount in most cases when a workspace is on the same local desktop machine or same local network.

[sherl0cks]

@eivantsov that command was a bit of copy paste error. The sudo was coming from the ubuntu VM I spun up that didn't have the vagrant user added to the docker group. Thus sudo was needed to talk to the docker.sock.

[sherl0cks]

@JamesDrummond first command has the same error as before fuse: mount failed: Permission denied

running the second command returns:

apparmor: config provided but apparmor not supported
docker: Error response from daemon: Cannot start container 7a229f8f3208e12b807a5f658b88d92b81f20c9b2792682e2d8569428da28872: [9] System error: could not synchronise with container process.

removing the apparmour flag (as this is a fedora machine with selinux, not apparmour) brings us back to the beginning: fuse: mount failed: Permission denied

That said, I agree that using sshfs directly makes more sense if the che-mount image is really just a proxy for it on windows/mac. I'll give that a try in the next day or so.

[JamesDrummond]

@sherl0cks I edited my response above about sshfs directly. When you have low latency between workspace and desktop machine sshfs directly is usually better. Usually this is the case when the workspace and desktop machine are on the same machine or local network. It could also be used for remote che server setups if your latency and download/upload speeds are not a factor.

Sorry you are having issues with selinux. We would love it if you could figure out how to get che-mount to work as other users will get into the same issue. If you cannot, you can use sshfs directly and setup unison on your fedora machine. You can take a look at how we setup by looking at che-mount dockerfile and script file on our github page below:

https://github.com/eclipse/che-dockerfiles/tree/master/che-mount

[sherl0cks]

@JamesDrummond I hear you on working through these issues, and I'd love to contribute. Unfortunately, we'll need to convince my team that such work is a priority first. I'll let you know if I can dedicate more time to this in the near future

FYI @TylerJewell

[ghost]

@TylerJewell @JamesDrummond che mount does not work for me on Ubuntu:

eugene@eugene:~/projects/che (master)$ ./che.sh mount $(pwd) 32781
INFO: Pulling image alpine:latest
WARNING: No swap limit support
Unable to find image 'codenvy/che-mount:nightly' locally
nightly: Pulling from codenvy/che-mount

e110a4a17941: Already exists 
cfc038de5b68: Pull complete 
25c6afaac606: Pull complete 
Digest: sha256:25c4179b27e4549e87f916893cd30904ff69ab67a636ff0e41a29d15032d7092
Status: Downloaded newer image for codenvy/che-mount:nightly
fuse: mount failed: Permission denied

[benoitf]

@eivantsov maybe you need to use latest PR from @l0rd
eclipse-che/che-dockerfiles#18

with improvements including some permissions

Launch it without che.sh
https://github.com/l0rd/che-dockerfiles/blob/1b04d6b804050ed2c94c6b0d6bc31909ac09e818/che-mount/Dockerfile#L15-L22

[l0rd]

@eivantsov I've a couple of PR (#2523 and eclipse-che/che-dockerfiles#18) that should fix che-mount permissions problems on Ubuntu.

Could you try to run this in an empty directory:

# Set the SSH port and workspace
SSH_PORT=32816
SYNC_FOLDER="$(pwd)"

# get files from the PRs
curl -sSL https://raw.githubusercontent.com/l0rd/che-dockerfiles/chemountfixes/che-mount/Dockerfile > Dockerfile
curl -sSL https://raw.githubusercontent.com/l0rd/che-dockerfiles/chemountfixes/che-mount/sync.sh > sync.sh
curl -sSL https://raw.githubusercontent.com/l0rd/che/chemountfixes/che.sh > che.sh
chmod +x ./che.sh
chmod +x ./sync.sh

# Build che-mount and run it
docker build -t codenvy/che-mount:local .
CHE_UTILITY_VERSION=local ./che.sh mount $SYNC_FOLDER $SSH_PORT

[sherl0cks]

misspoke - ignore this comment

[TylerJewell]

@sherl0cks - the new che-mount has been merged. It will probably show up as codenvy/che-mount:nightly docker image in another 24 hours. I think the merge happened after the nightly CI cycle, so we may have to wait 24 hours. Or, you can build it directly into codenvy/che-mount:nightly.

Also - there are some Che CLI improvements that are waiting for a merge - they will have to go in within the next 24 hours to be synchronized with this new image. We will now do discovery to provide a list of workspaces that have SSH running in them that can be mounted. And if there is only a single workspace with SSH, it will just attempt to connect to it. So it simplifies the syntax quite a bit.

#2535

[TylerJewell]

Closed to this being resolved. Performance is screaming now.