zfs mount failed if the cwd contains a symlink with the same name

[zfsnewbie]

Hey folks,

Greetings from germany!
I've (possibly) found a bug while mounting a zfs filesystem, in my example named "data3", from a current working directory (cwd) which contains a symbolic link also called "data3". The destination where the symlink points to is not significant. In my example the cwd is: "/data/home/chriss_ubuntu" and the symblic link called "data3" points to "/data3".
It seems to me that the command "zfs mount data3" tries to follow the symbolic link and so the following error message appears:

---

filesystem '/data3' cannot be mounted, unable to open the dataset
cannot mount 'data3': Invalid argument

---

If I change the cwd to any other directory (without a symlink called "data3") the mount command works just fine.

strace (the short way):

---

11304 execve("/sbin/zfs", ["zfs", "mount", "data3"], [/* 29 vars /]) = 0
11304 lstat("/data3", {st_mode=S_IFDIR|0755, st_size=48, ...}) = 0
11304 openat(AT_FDCWD, "/data3", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 6
11305 execve("/bin/mount", ["/bin/mount", "-t", "zfs", "-o", "defaults,atime,dev,exec,rw,suid,"..., "data3", "/data3"], [/ 29 vars /]) = 0
11305 readlink("/data/home/chriss_ubuntu/data3", "/data3", 4096) = 6
11305 readlink("/data3", 0x7fff5fd0afe0, 4096) = -1 EINVAL (Invalid argument)
11306 execve("/sbin/mount.zfs", ["/sbin/mount.zfs", "/data3", "/data3", "-o", "rw,xattr,zfsutil"], [/ 25 vars */]) = 0
11306 stat("/data3", {st_mode=S_IFDIR|0755, st_size=48, ...}) = 0
11306 open("/data3", O_RDONLY) = 6
11306 lstat("/data3", {st_mode=S_IFDIR|0755, st_size=48, ...}) = 0
11306 write(2, "filesystem '/data3' cannot be mo"..., 66) = 66
11304 write(2, "cannot mount 'data3': Invalid ar"..., 39) = 39

---

Is this a bug or maybe an intentional behaviour?

Used versions:
Ubuntu 13.04 (amd64)
dkms 2.2.0.3-1.1ubuntu2+zfs6raring1
libzfs1 0.6.2-1raring
mountall 2.48build1-zfs2
zfs-dkms 0.6.2-1raring
zfsutils 0.6.2-1raring

[GregorKopka]

I can reproduce what you found with 0.6.2-r1 on gentoo:

~$ cd <empty directory>
~$ zfs create tank/test -o mountpoint=/test
~$ zfs mount | grep test
tank/test                      /test
~$ zfs umount tank/test
~$ zfs mount | grep test
~$ md tank
~$ ln -s /test tank/test
~$ zfs mount tank/test
filesystem '/test' cannot be mounted, unable to open the dataset
cannot mount 'tank/test': Invalid argument

BUT

~$ rm tank/test
~$ ln -s tank/test tank/test
~$ zfs mount tank/test
~$ zfs mount | grep test
tank/test                      /test

Since it's not documented and can lead to unexpected behaviour (depending on the contents of cwd) i would argue that it is a bug.

Just wrote this in another issue, but it fits here too:
Note
The root filesystem of the pool might behave a bit different than the siblings (f.ex. you most likely won't be able to use zfs recv on it in some scenarios since it can't be destroyed). Best practice for zfs is to use the root filesystem of the pool only as a container (can even have canmount=off or mountpoint=none) to just inherit properties to the 'real' filesystems|volumes.

[behlendorf]

We've seen issues like this before, they're caused by Linux's generic mount command which assumes the passed mount argument is a path. We may be able to do something about this in the mount helper.

[loli10K]

This seems to be caused by mount path canonicalization (which is by default enabled).

First zfs mount data3 calls /bin/mount -t zfs -o defaults,atime,dev,exec,rw,suid,nomand,zfsutil data3 /data3

root@debian-8-zfs:~# gdb -q -ex 'set follow-fork-mode child' -ex 'b execve' -ex 'b readlink' -ex 'run' --args zfs mount data3
Reading symbols from zfs...done.
Function "execve" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (execve) pending.
Function "readlink" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 2 (readlink) pending.
Starting program: /usr/local/sbin/zfs mount data3
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New process 5836]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Switching to Thread 0x7ffff7fe4780 (LWP 5836)]

Thread 2.1 "zfs" hit Breakpoint 1, __execve (file=file@entry=0x7ffff7241aea "/bin/mount", argv=argv@entry=0x7fffffff8800, envp=envp@entry=0x7fffffffecf8)
    at ../sysdeps/unix/sysv/linux/execve.c:33
33  ../sysdeps/unix/sysv/linux/execve.c: No such file or directory.
(gdb) bt
#0  __execve (file=file@entry=0x7ffff7241aea "/bin/mount", argv=argv@entry=0x7fffffff8800, envp=envp@entry=0x7fffffffecf8) at ../sysdeps/unix/sysv/linux/execve.c:33
#1  0x00007ffff66d2920 in __execvpe (file=0x7ffff7241aea "/bin/mount", argv=0x7fffffff8800, envp=0x7fffffffecf8) at execvpe.c:63
#2  0x00007ffff723985a in libzfs_run_process (path=0x7ffff7241aea "/bin/mount", argv=0x7fffffff8800, flags=3) at libzfs_util.c:673
#3  0x00007ffff72218dc in do_mount (src=0x61ff60 "data3", mntpt=0x7fffffffa990 "/data3", opts=0x7fffffff9990 "defaults,atime,dev,exec,rw,suid,nomand,zfsutil") at libzfs_mount.c:288
#4  0x00007ffff7221fb2 in zfs_mount (zhp=0x61ff50, options=0x0, flags=0) at libzfs_mount.c:470
#5  0x0000000000410cf7 in share_mount_one (zhp=0x61ff50, op=2, flags=0, protocol=0x0, explicit=B_TRUE, options=0x0) at zfs_main.c:5821
#6  0x000000000041135f in share_mount (op=2, argc=1, argv=0x7fffffffece8) at zfs_main.c:6012
#7  0x000000000041139d in zfs_do_mount (argc=2, argv=0x7fffffffece0) at zfs_main.c:6030
#8  0x0000000000412bd9 in main (argc=3, argv=0x7fffffffecd8) at zfs_main.c:6723
(gdb) print *argv@7
$1 = {0x7ffff7241aea "/bin/mount", 0x7ffff7241af5 "-t", 0x7ffff7241af8 "zfs", 0x7ffff7241afc "-o", 0x7fffffff9990 "defaults,atime,dev,exec,rw,suid,nomand,zfsutil", 0x61ff60 "data3", 
  0x7fffffffa990 "/data3"}
(gdb) c
Continuing.

then the /bin/mount binary canonicalize "data3" into "/data3" (via readlink)

Thread 2.1 "mount" hit Breakpoint 2, readlink () at ../sysdeps/unix/syscall-template.S:81
81  ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0  readlink () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffff73e18e2 in __realpath (name=<optimized out>, resolved=0x0) at canonicalize.c:175
#2  0x00007ffff7bbc8d2 in canonicalize_path (path=0x60b2f0 "data3") at lib/canonicalize.c:76
#3  0x00007ffff7b9beef in canonicalize_path_and_cache (path=0x60b2f0 "data3", cache=0x60b2b0) at libmount/src/cache.c:508
#4  0x00007ffff7b9bff0 in mnt_resolve_path (path=0x60b2f0 "data3", cache=0x60b2b0) at libmount/src/cache.c:553
#5  0x00007ffff7bb1c2d in mnt_context_prepare_srcpath (cxt=0x60b030) at libmount/src/context.c:1485
#6  0x00007ffff7bb8589 in mnt_context_prepare_mount (cxt=0x60b030) at libmount/src/context_mount.c:876
#7  0x00007ffff7bb8b4d in mnt_context_mount (cxt=0x60b030) at libmount/src/context_mount.c:1035
#8  0x0000000000405a22 in main (argc=2, argv=0x7fffffffecc0) at sys-utils/mount.c:1104
(gdb) c
Continuing.

and finally /bin/mount calls /sbin/mount.zfs /data3 /data3 -o rw,zfsutil with the "wrong" parameter.

Thread 3.1 "mount" hit Breakpoint 1, __execve (file=0x60b2f0 "/sbin/mount.zfs", argv=0x7fffffffe9c0, envp=0x7fffffffecd8) at ../sysdeps/unix/sysv/linux/execve.c:33
33  ../sysdeps/unix/sysv/linux/execve.c: No such file or directory.
(gdb) bt
#0  __execve (file=0x60b2f0 "/sbin/mount.zfs", argv=0x7fffffffe9c0, envp=0x7fffffffecd8) at ../sysdeps/unix/sysv/linux/execve.c:33
#1  0x00007ffff7bb765d in exec_helper (cxt=0x60b030) at libmount/src/context_mount.c:594
#2  0x00007ffff7bb7b0b in do_mount (cxt=0x60b030, try_type=0x0) at libmount/src/context_mount.c:685
#3  0x00007ffff7bb888c in mnt_context_do_mount (cxt=0x60b030) at libmount/src/context_mount.c:937
#4  0x00007ffff7bb8b77 in mnt_context_mount (cxt=0x60b030) at libmount/src/context_mount.c:1039
#5  0x0000000000405a22 in main (argc=2, argv=0x7fffffffecc0) at sys-utils/mount.c:1104
(gdb) print *argv@5
$2 = {0x60b2f0 "/sbin/mount.zfs", 0x60b210 "/data3", 0x60b310 "/data3", 0x7ffff7bcf717 "-o", 0x60e010 "rw,zfsutil"}

TL;DR: can this be fixed just by adding -c|--no-canonicalize to the mount helper call to /bin/mount?
Reading the comments in libzfs_mount.c i see that the intention was to use libmount directly instead of relying on the /bin/mount binary, so maybe this is not the right way to solve this issue?