Missing symlinks in /dev/zvol #12507

Tualua · 2021-08-24T03:06:28Z

System information

Type	Version/Name
Distribution Name	Oracle Linux
Distribution Version	8.4
Kernel Version	5.4.17-2102.204.4.3.el8uek
Architecture	x86_64
OpenZFS Version	zfs-2.1.0-1

Describe the problem you're observing

We are serving zvols as iSCSI targets to virtual machines on other servers. We use a lot of clones and snapshots.
For example we have a rather large zvol data/reference (25.4T) and make a bunch of clones which then served as iSCSI targets using scst.

However, sometimes, after destroying a clone and making a new one symlink to this clone is missing from /dev/zvol

# ll /dev/zvol/data/kvm/desktop/games-vm2*
lrwxrwxrwx. 1 root root 18 Aug 24 08:27 /dev/zvol/data/kvm/desktop/games-vm2 -> ../../../../zd1648
lrwxrwxrwx. 1 root root 18 Aug 24 08:29 /dev/zvol/data/kvm/desktop/games-vm20 -> ../../../../zd1936
lrwxrwxrwx. 1 root root 20 Aug 24 08:29 /dev/zvol/data/kvm/desktop/games-vm20-part1 -> ../../../../zd1936p1
lrwxrwxrwx. 1 root root 20 Aug 24 08:29 /dev/zvol/data/kvm/desktop/games-vm20-part2 -> ../../../../zd1936p2
lrwxrwxrwx. 1 root root 20 Aug 24 08:29 /dev/zvol/data/kvm/desktop/games-vm20-part3 -> ../../../../zd1936p3
lrwxrwxrwx. 1 root root 18 Aug 24 08:28 /dev/zvol/data/kvm/desktop/games-vm21 -> ../../../../zd1952
lrwxrwxrwx. 1 root root 20 Aug 24 08:28 /dev/zvol/data/kvm/desktop/games-vm21-part1 -> ../../../../zd1952p1
lrwxrwxrwx. 1 root root 20 Aug 24 08:28 /dev/zvol/data/kvm/desktop/games-vm21-part2 -> ../../../../zd1952p2
lrwxrwxrwx. 1 root root 20 Aug 24 08:28 /dev/zvol/data/kvm/desktop/games-vm21-part3 -> ../../../../zd1952p3
lrwxrwxrwx. 1 root root 18 Aug 24 08:25 /dev/zvol/data/kvm/desktop/games-vm22 -> ../../../../zd1968
lrwxrwxrwx. 1 root root 20 Aug 24 08:25 /dev/zvol/data/kvm/desktop/games-vm22-part1 -> ../../../../zd1968p1
lrwxrwxrwx. 1 root root 20 Aug 24 08:25 /dev/zvol/data/kvm/desktop/games-vm22-part2 -> ../../../../zd1968p2
lrwxrwxrwx. 1 root root 20 Aug 24 08:25 /dev/zvol/data/kvm/desktop/games-vm22-part3 -> ../../../../zd1968p3
lrwxrwxrwx. 1 root root 18 Aug 24 08:26 /dev/zvol/data/kvm/desktop/games-vm24 -> ../../../../zd2000
lrwxrwxrwx. 1 root root 20 Aug 24 08:26 /dev/zvol/data/kvm/desktop/games-vm24-part1 -> ../../../../zd2000p1
lrwxrwxrwx. 1 root root 20 Aug 24 08:26 /dev/zvol/data/kvm/desktop/games-vm24-part2 -> ../../../../zd2000p2
lrwxrwxrwx. 1 root root 20 Aug 24 08:26 /dev/zvol/data/kvm/desktop/games-vm24-part3 -> ../../../../zd2000p3
lrwxrwxrwx. 1 root root 18 Aug 24 08:25 /dev/zvol/data/kvm/desktop/games-vm25 -> ../../../../zd2016
lrwxrwxrwx. 1 root root 20 Aug 24 08:25 /dev/zvol/data/kvm/desktop/games-vm25-part1 -> ../../../../zd2016p1
lrwxrwxrwx. 1 root root 20 Aug 24 08:25 /dev/zvol/data/kvm/desktop/games-vm25-part2 -> ../../../../zd2016p2
lrwxrwxrwx. 1 root root 20 Aug 24 08:25 /dev/zvol/data/kvm/desktop/games-vm25-part3 -> ../../../../zd2016p3
lrwxrwxrwx. 1 root root 18 Aug 24 08:29 /dev/zvol/data/kvm/desktop/games-vm26 -> ../../../../zd2032
lrwxrwxrwx. 1 root root 20 Aug 24 08:29 /dev/zvol/data/kvm/desktop/games-vm26-part1 -> ../../../../zd2032p1
lrwxrwxrwx. 1 root root 20 Aug 24 08:29 /dev/zvol/data/kvm/desktop/games-vm26-part2 -> ../../../../zd2032p2
lrwxrwxrwx. 1 root root 20 Aug 24 08:29 /dev/zvol/data/kvm/desktop/games-vm26-part3 -> ../../../../zd2032p3
lrwxrwxrwx. 1 root root 18 Aug 24 08:24 /dev/zvol/data/kvm/desktop/games-vm27 -> ../../../../zd2048
lrwxrwxrwx. 1 root root 20 Aug 24 08:24 /dev/zvol/data/kvm/desktop/games-vm27-part1 -> ../../../../zd2048p1
lrwxrwxrwx. 1 root root 20 Aug 24 08:24 /dev/zvol/data/kvm/desktop/games-vm27-part2 -> ../../../../zd2048p2
lrwxrwxrwx. 1 root root 20 Aug 24 08:24 /dev/zvol/data/kvm/desktop/games-vm27-part3 -> ../../../../zd2048p3
lrwxrwxrwx. 1 root root 18 Aug 24 08:27 /dev/zvol/data/kvm/desktop/games-vm28 -> ../../../../zd2064
lrwxrwxrwx. 1 root root 20 Aug 24 08:27 /dev/zvol/data/kvm/desktop/games-vm28-part1 -> ../../../../zd2064p1
lrwxrwxrwx. 1 root root 20 Aug 24 08:27 /dev/zvol/data/kvm/desktop/games-vm28-part2 -> ../../../../zd2064p2
lrwxrwxrwx. 1 root root 20 Aug 24 08:27 /dev/zvol/data/kvm/desktop/games-vm28-part3 -> ../../../../zd2064p3
lrwxrwxrwx. 1 root root 18 Aug 24 08:25 /dev/zvol/data/kvm/desktop/games-vm29 -> ../../../../zd2128
lrwxrwxrwx. 1 root root 20 Aug 24 08:25 /dev/zvol/data/kvm/desktop/games-vm29-part1 -> ../../../../zd2128p1
lrwxrwxrwx. 1 root root 20 Aug 24 08:25 /dev/zvol/data/kvm/desktop/games-vm29-part2 -> ../../../../zd2128p2
lrwxrwxrwx. 1 root root 20 Aug 24 08:25 /dev/zvol/data/kvm/desktop/games-vm29-part3 -> ../../../../zd2128p3
lrwxrwxrwx. 1 root root 20 Aug 24 08:27 /dev/zvol/data/kvm/desktop/games-vm2-part1 -> ../../../../zd1648p1
lrwxrwxrwx. 1 root root 20 Aug 24 08:27 /dev/zvol/data/kvm/desktop/games-vm2-part2 -> ../../../../zd1648p2
lrwxrwxrwx. 1 root root 20 Aug 24 08:27 /dev/zvol/data/kvm/desktop/games-vm2-part3 -> ../../../../zd1648p3`

As you can see from the output data/kvm/desktop/games-vm23 is missing but if I check it with zfs list or get all - everyting is looking fine

# zfs get all data/kvm/desktop/games-vm23
NAME                         PROPERTY              VALUE                      SOURCE
data/kvm/desktop/games-vm23  type                  volume                     -
data/kvm/desktop/games-vm23  creation              Mon Aug 23 20:08 2021      -
data/kvm/desktop/games-vm23  used                  0B                         -
data/kvm/desktop/games-vm23  available             15.0T                      -
data/kvm/desktop/games-vm23  referenced            25.2T                      -
data/kvm/desktop/games-vm23  compressratio         1.00x                      -
data/kvm/desktop/games-vm23  origin                data/reference@ver2_14474  -
data/kvm/desktop/games-vm23  reservation           none                       default
data/kvm/desktop/games-vm23  volsize               26T                        local
data/kvm/desktop/games-vm23  volblocksize          64K                        -
data/kvm/desktop/games-vm23  checksum              on                         default
data/kvm/desktop/games-vm23  compression           off                        default
data/kvm/desktop/games-vm23  readonly              off                        default
data/kvm/desktop/games-vm23  createtxg             28878088                   -
data/kvm/desktop/games-vm23  copies                1                          default
data/kvm/desktop/games-vm23  refreservation        none                       default
data/kvm/desktop/games-vm23  guid                  16247426968911119015       -
data/kvm/desktop/games-vm23  primarycache          all                        inherited from data
data/kvm/desktop/games-vm23  secondarycache        all                        default
data/kvm/desktop/games-vm23  usedbysnapshots       0B                         -
data/kvm/desktop/games-vm23  usedbydataset         0B                         -
data/kvm/desktop/games-vm23  usedbychildren        0B                         -
data/kvm/desktop/games-vm23  usedbyrefreservation  0B                         -
data/kvm/desktop/games-vm23  logbias               latency                    default
data/kvm/desktop/games-vm23  objsetid              86827                      -
data/kvm/desktop/games-vm23  dedup                 off                        default
data/kvm/desktop/games-vm23  mlslabel              none                       default
data/kvm/desktop/games-vm23  sync                  standard                   default
data/kvm/desktop/games-vm23  refcompressratio      1.00x                      -
data/kvm/desktop/games-vm23  written               0                          -
data/kvm/desktop/games-vm23  logicalused           0                          -
data/kvm/desktop/games-vm23  logicalreferenced     25.2T                      -
data/kvm/desktop/games-vm23  volmode               default                    default
data/kvm/desktop/games-vm23  snapshot_limit        none                       default
data/kvm/desktop/games-vm23  snapshot_count        none                       default
data/kvm/desktop/games-vm23  snapdev               hidden                     default
data/kvm/desktop/games-vm23  context               none                       default
data/kvm/desktop/games-vm23  fscontext             none                       default
data/kvm/desktop/games-vm23  defcontext            none                       default
data/kvm/desktop/games-vm23  rootcontext           none                       default
data/kvm/desktop/games-vm23  redundant_metadata    all                        default
data/kvm/desktop/games-vm23  encryption            off                        default
data/kvm/desktop/games-vm23  keylocation           none                       default
data/kvm/desktop/games-vm23  keyformat             none                       default
data/kvm/desktop/games-vm23  pbkdf2iters           0                          default

Describe how to reproduce the problem

I don't now how to reproduce this problem. I happens randomly on zfs 2.0.4 and 2.1.0, on different kernels. We have similar setup in other location with higher load and it runs without this problem.

Include any warning/errors/backtraces from the system logs

I could not find any errors except scst error when it tries to create iSCSI lun when this symlink is missing. If I manually destroy and re-create clone - symlink appears again. And if it helps we have more than 600 /dev/zd* devices

[Tue Aug 24 09:32:16 2021] [1719449]: dev_vdisk: ***ERROR***: filp_open(/dev/zvol/data/kvm/desktop/games-vm23) failed: -2

The text was updated successfully, but these errors were encountered:

Tualua · 2021-08-24T04:25:20Z

Some additional info.
This is part of log when clone goes without any problems:

Aug 23 20:08:25 pk-sds2 sudo[511941]: zfsreplica : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/zfs clone data/reference@ver2_14474 data/kvm/desktop/games-vm118
Aug 23 20:08:25 pk-sds2 systemd[1]: Started Session c36528 of user root.
Aug 23 20:08:25 pk-sds2 sudo[511941]: pam_unix(sudo:session): session opened for user root by (uid=0)
Aug 23 20:08:26 pk-sds2 kernel:  zd1984: p1 p2 p3
Aug 23 20:08:26 pk-sds2 sudo[511941]: pam_unix(sudo:session): session closed for user root
Aug 23 20:08:26 pk-sds2 systemd[1]: session-c36528.scope: Succeeded.
Aug 23 20:08:26 pk-sds2 sudo[513803]: zfsreplica : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/zfs snapshot data/kvm/desktop/games-vm118@0
Aug 23 20:08:26 pk-sds2 systemd[1]: Started Session c36529 of user root.
Aug 23 20:08:26 pk-sds2 sudo[513803]: pam_unix(sudo:session): session opened for user root by (uid=0)
Aug 23 20:08:26 pk-sds2 sudo[513803]: pam_unix(sudo:session): session closed for user root
Aug 23 20:08:26 pk-sds2 systemd[1]: session-c36529.scope: Succeeded.

And when it does not create symlink

Aug 23 20:08:10 pk-sds2 sudo[484783]: zfsreplica : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/zfs clone data/reference@ver2_14474 data/kvm/desktop/games-vm23
Aug 23 20:08:10 pk-sds2 systemd[1]: Started Session c36518 of user root.
Aug 23 20:08:10 pk-sds2 sudo[484783]: pam_unix(sudo:session): session opened for user root by (uid=0)
Aug 23 20:08:11 pk-sds2 sudo[484783]: pam_unix(sudo:session): session closed for user root
Aug 23 20:08:11 pk-sds2 systemd[1]: session-c36518.scope: Succeeded.
Aug 23 20:08:11 pk-sds2 sudo[486398]: zfsreplica : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/zfs snapshot data/kvm/desktop/games-vm23@0
Aug 23 20:08:11 pk-sds2 systemd[1]: Started Session c36519 of user root.
Aug 23 20:08:11 pk-sds2 sudo[486398]: pam_unix(sudo:session): session opened for user root by (uid=0)
Aug 23 20:08:12 pk-sds2 sudo[486398]: pam_unix(sudo:session): session closed for user root
Aug 23 20:08:12 pk-sds2 systemd[1]: session-c36519.scope: Succeeded.

Line with kernel is missing

Heyshubham · 2021-08-31T01:52:45Z

Can you please check if there are any udev timeouts in os logs?

mk01 · 2021-09-21T23:39:59Z

@Tualua

can you check if /dev/zvol/Unable symlink exists on a machine with missing links?

Tualua · 2021-09-22T03:07:29Z

OK, will check next time it happens

…

On Wed, Sep 22, 2021 at 7:40 AM Matus Kral ***@***.***> wrote: @Tualua <https://github.com/Tualua> can you check if /dev/zvol/Unable symlink exists on a machine with missing links? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12507 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHX2DHIBCKLIOSAMMDLPCCTUDEJVVANCNFSM5CV4PY7Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

Tualua · 2021-09-22T14:11:11Z

@mk01 no, there is no such symlink

Heyshubham · 2021-09-22T16:46:57Z

@Tualua Can you please also check /dev/zvol/blkid symlink exist?

Tualua · 2021-09-25T16:46:00Z

Also no

…

On Thu, Sep 23, 2021 at 12:47 AM Shubham Agrawal ***@***.***> wrote: @Tualua <https://github.com/Tualua> Can you please also check /dev/zvol/blkid symlink exist? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12507 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHX2DHJVQYLAYV2ZHQ47PLDUDICAXANCNFSM5CV4PY7Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

Tualua · 2021-09-28T08:12:11Z

Can you please check if there are any udev timeouts in os logs?

Some fresh info
zfs history

2021-09-28.14:31:19 zfs destroy -r data/kvm/desktop/games-vm13
2021-09-28.14:31:20 zfs clone data/reference@ver2_14762 data/kvm/desktop/games-vm13
2021-09-28.14:31:21 zfs snapshot data/kvm/desktop/games-vm13@0

The problem is `games-vm14`

2021-09-28.14:31:26 zfs destroy -r data/kvm/desktop/games-vm14
2021-09-28.14:31:27 zfs clone data/reference@ver2_14762 data/kvm/desktop/games-vm14
2021-09-28.14:31:27 zfs snapshot data/kvm/desktop/games-vm14@0

2021-09-28.14:31:33 zfs destroy -r data/kvm/desktop/games-vm111
2021-09-28.14:31:34 zfs clone data/reference@ver2_14762 data/kvm/desktop/games-vm111
2021-09-28.14:31:35 zfs snapshot data/kvm/desktop/games-vm111@0
2021-09-28.14:31:36 zfs destroy -r data/kvm/desktop/games-vm110
2021-09-28.14:31:37 zfs clone data/reference@ver2_14762 data/kvm/desktop/games-vm110
2021-09-28.14:31:37 zfs snapshot data/kvm/desktop/games-vm110@0

Clone created at 14:31:27

journalctl --since="2021-09-28 14:31:27" -u systemd-udevd

-- Logs begin at Fri 2021-09-24 23:20:42 +07, end at Tue 2021-09-28 15:05:24 +07. --
Sep 28 14:31:27  systemd-udevd[929]: seq 14236 queued, 'remove' 'block'
Sep 28 14:31:27  systemd-udevd[929]: Validate module index
Sep 28 14:31:27  systemd-udevd[929]: Check if link configuration needs reloading.
Sep 28 14:31:27  systemd-udevd[929]: seq 14236 forked new worker [1493627]
Sep 28 14:31:27  systemd-udevd[929]: seq 14237 queued, 'remove' 'block'
Sep 28 14:31:27  systemd-udevd[1493627]: seq 14236 running
Sep 28 14:31:27  systemd-udevd[1493627]: IMPORT '/lib/udev/vdev_id -d zd1168p3' /usr/lib/udev/rules.d/69-vdev.rules:6
Sep 28 14:31:27  systemd-udevd[929]: seq 14237 forked new worker [1493628]
Sep 28 14:31:27  systemd-udevd[929]: seq 14238 queued, 'remove' 'block'
Sep 28 14:31:27  [1493629]: starting '/lib/udev/vdev_id -d zd1168p3'
Sep 28 14:31:27  systemd-udevd[1493628]: seq 14237 running
Sep 28 14:31:27  systemd-udevd[1493628]: IMPORT '/lib/udev/vdev_id -d zd1168p2' /usr/lib/udev/rules.d/69-vdev.rules:6
Sep 28 14:31:27  systemd-udevd[929]: seq 14238 forked new worker [1493630]
Sep 28 14:31:27  systemd-udevd[929]: seq 14239 queued, 'remove' 'bdi'
Sep 28 14:31:27  systemd-udevd[1493630]: seq 14238 running
Sep 28 14:31:27  [1493631]: starting '/lib/udev/vdev_id -d zd1168p2'
Sep 28 14:31:27  systemd-udevd[1493630]: IMPORT '/lib/udev/vdev_id -d zd1168p1' /usr/lib/udev/rules.d/69-vdev.rules:6
Sep 28 14:31:27  systemd-udevd[929]: seq 14239 forked new worker [1493632]
Sep 28 14:31:27  systemd-udevd[929]: seq 14240 queued, 'remove' 'block'
Sep 28 14:31:27  [1493633]: starting '/lib/udev/vdev_id -d zd1168p1'
Sep 28 14:31:27  systemd-udevd[1493632]: seq 14239 running
Sep 28 14:31:27  systemd-udevd[1493632]: seq 14239 processed
Sep 28 14:31:27  systemd-udevd[1493627]: '/lib/udev/vdev_id -d zd1168p3'(out) 'Error: Config file "/etc/zfs/vdev_id.conf" not found'
Sep 28 14:31:27  systemd-udevd[1493627]: Process '/lib/udev/vdev_id -d zd1168p3' succeeded.
Sep 28 14:31:27  systemd-udevd[1493628]: '/lib/udev/vdev_id -d zd1168p2'(out) 'Error: Config file "/etc/zfs/vdev_id.conf" not found'
Sep 28 14:31:27  systemd-udevd[1493627]: no reference left, remove '/dev/data/kvm/desktop/games-vm14-part3'
Sep 28 14:31:27  systemd-udevd[1493627]: no reference left, remove '/dev/zvol/data/kvm/desktop/games-vm14-part3'
Sep 28 14:31:27  systemd-udevd[1493628]: Process '/lib/udev/vdev_id -d zd1168p2' succeeded.
Sep 28 14:31:27  systemd-udevd[1493627]: seq 14236 processed
Sep 28 14:31:27  systemd-udevd[1493628]: no reference left, remove '/dev/data/kvm/desktop/games-vm14-part2'
Sep 28 14:31:27  systemd-udevd[1493628]: no reference left, remove '/dev/zvol/data/kvm/desktop/games-vm14-part2'
Sep 28 14:31:27  systemd-udevd[1493628]: seq 14237 processed
Sep 28 14:31:27  systemd-udevd[1493630]: '/lib/udev/vdev_id -d zd1168p1'(out) 'Error: Config file "/etc/zfs/vdev_id.conf" not found'
Sep 28 14:31:27  systemd-udevd[1493630]: Process '/lib/udev/vdev_id -d zd1168p1' succeeded.
Sep 28 14:31:27  systemd-udevd[1493630]: no reference left, remove '/dev/data/kvm/desktop/games-vm14-part1'
Sep 28 14:31:27  systemd-udevd[1493630]: no reference left, remove '/dev/zvol/data/kvm/desktop/games-vm14-part1'
Sep 28 14:31:27  systemd-udevd[1493630]: seq 14238 processed
Sep 28 14:31:27  systemd-udevd[1493627]: seq 14240 running
Sep 28 14:31:27  systemd-udevd[1493627]: IMPORT '/lib/udev/vdev_id -d zd1168' /usr/lib/udev/rules.d/69-vdev.rules:5
Sep 28 14:31:27  [1493636]: starting '/lib/udev/vdev_id -d zd1168'
Sep 28 14:31:27  systemd-udevd[1493627]: '/lib/udev/vdev_id -d zd1168'(out) 'Error: Config file "/etc/zfs/vdev_id.conf" not found'
Sep 28 14:31:27  systemd-udevd[1493627]: Process '/lib/udev/vdev_id -d zd1168' succeeded.
Sep 28 14:31:27  systemd-udevd[1493627]: no reference left, remove '/dev/data/kvm/desktop/games-vm14'
Sep 28 14:31:27  systemd-udevd[1493627]: no reference left, remove '/dev/zvol/data/kvm/desktop/games-vm14'
Sep 28 14:31:27  systemd-udevd[1493627]: seq 14240 processed
Sep 28 14:31:27  systemd-udevd[929]: cleanup idle workers
Sep 28 14:31:27  systemd-udevd[1493630]: Unload module index
Sep 28 14:31:27  systemd-udevd[1493628]: Unload module index
Sep 28 14:31:27  systemd-udevd[1493627]: Unload module index
Sep 28 14:31:27  systemd-udevd[1493632]: Unload module index
Sep 28 14:31:27  systemd-udevd[1493630]: Unloaded link configuration context.
Sep 28 14:31:27  systemd-udevd[1493628]: Unloaded link configuration context.
Sep 28 14:31:27  systemd-udevd[1493627]: Unloaded link configuration context.
Sep 28 14:31:27  systemd-udevd[1493632]: Unloaded link configuration context.
Sep 28 14:31:27  systemd-udevd[929]: worker [1493630] exited
Sep 28 14:31:27  systemd-udevd[929]: worker [1493628] exited
Sep 28 14:31:27  systemd-udevd[929]: cleanup idle workers
Sep 28 14:31:27  systemd-udevd[929]: cleanup idle workers
Sep 28 14:31:27  systemd-udevd[929]: worker [1493627] exited
Sep 28 14:31:27  systemd-udevd[929]: worker [1493632] exited
Sep 28 14:31:33  systemd-udevd[929]: seq 14241 queued, 'remove' 'block'
Sep 28 14:31:33  systemd-udevd[929]: Validate module index
Sep 28 14:31:33  systemd-udevd[929]: Check if link configuration needs reloading.
Sep 28 14:31:33  systemd-udevd[929]: seq 14241 forked new worker [1496510]
Sep 28 14:31:33  systemd-udevd[929]: seq 14242 queued, 'remove' 'block'
Sep 28 14:31:33  systemd-udevd[1496510]: seq 14241 running
Sep 28 14:31:33  systemd-udevd[1496510]: IMPORT '/lib/udev/vdev_id -d zd480p3' /usr/lib/udev/rules.d/69-vdev.rules:6
Sep 28 14:31:33  systemd-udevd[929]: seq 14242 forked new worker [1496520]
Sep 28 14:31:33  systemd-udevd[929]: seq 14243 queued, 'remove' 'block'
Sep 28 14:31:33  systemd-udevd[1496520]: seq 14242 running
Sep 28 14:31:33  [1496522]: starting '/lib/udev/vdev_id -d zd480p3'
Sep 28 14:31:33  systemd-udevd[1496520]: IMPORT '/lib/udev/vdev_id -d zd480p2' /usr/lib/udev/rules.d/69-vdev.rules:6
Sep 28 14:31:33  systemd-udevd[929]: seq 14243 forked new worker [1496531]
Sep 28 14:31:33  systemd-udevd[929]: seq 14244 queued, 'remove' 'bdi'
Sep 28 14:31:33  [1496532]: starting '/lib/udev/vdev_id -d zd480p2'
Sep 28 14:31:33  systemd-udevd[1496531]: seq 14243 running
Sep 28 14:31:33  systemd-udevd[929]: seq 14244 forked new worker [1496533]
Sep 28 14:31:33  systemd-udevd[929]: seq 14245 queued, 'remove' 'block'
Sep 28 14:31:33  systemd-udevd[1496531]: IMPORT '/lib/udev/vdev_id -d zd480p1' /usr/lib/udev/rules.d/69-vdev.rules:6
Sep 28 14:31:33  systemd-udevd[1496533]: seq 14244 running
Sep 28 14:31:33  systemd-udevd[1496533]: seq 14244 processed
Sep 28 14:31:33  [1496535]: starting '/lib/udev/vdev_id -d zd480p1'
Sep 28 14:31:33  systemd-udevd[1496510]: '/lib/udev/vdev_id -d zd480p3'(out) 'Error: Config file "/etc/zfs/vdev_id.conf" not found'
Sep 28 14:31:33  systemd-udevd[1496510]: Process '/lib/udev/vdev_id -d zd480p3' succeeded.
Sep 28 14:31:33  systemd-udevd[1496510]: no reference left, remove '/dev/data/kvm/desktop/games-vm111-part3'
Sep 28 14:31:33  systemd-udevd[1496510]: no reference left, remove '/dev/zvol/data/kvm/desktop/games-vm111-part3'
Sep 28 14:31:33  systemd-udevd[1496510]: seq 14241 processed
Sep 28 14:31:33  systemd-udevd[1496520]: '/lib/udev/vdev_id -d zd480p2'(out) 'Error: Config file "/etc/zfs/vdev_id.conf" not found'

Heyshubham · 2021-09-29T02:21:14Z

This is different from the issue I am facing, in my case, I was seeing timeouts in udev and eventually it was terminating the rule which should be responsible for symlink creation. I only know this is random and have something to do with clone operation.

But with this discussion we are one step further and it is destroy+clone.

behlendorf · 2021-10-04T19:24:19Z

This issue may be a duplicate of #12301 if using a 5.13 or newer kernel.

Tualua · 2021-10-05T17:09:51Z

I'm using 5.4 UEK kernel from Oracle

…

On Tue, Oct 5, 2021 at 3:24 AM Brian Behlendorf ***@***.***> wrote: This issue may be a duplicate of #12301 <#12301> if using a 5.13 or newer kernel. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12507 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHX2DHINB2XF3SVSU5DBF43UFH5O3ANCNFSM5CV4PY7Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

stale · 2022-10-14T04:43:29Z

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

behlendorf · 2022-10-14T18:42:11Z

Closed as a duplicate of #12301.

Tualua added the Type: Defect Incorrect behavior (e.g. crash, hang) label Aug 24, 2021

ChristophSchmidpeter mentioned this issue Nov 1, 2021

Zvols get lost upon host startup and VM shut down #12712

Closed

fabianishere mentioned this issue Nov 6, 2021

Broken zvol links fabianishere/pve-edge-kernel#183

Closed

stale bot added the Status: Stale No recent activity for issue label Oct 14, 2022

behlendorf closed this as completed Oct 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Missing symlinks in /dev/zvol #12507

Missing symlinks in /dev/zvol #12507

Tualua commented Aug 24, 2021 •

edited

Loading

Tualua commented Aug 24, 2021

Heyshubham commented Aug 31, 2021

mk01 commented Sep 21, 2021

Tualua commented Sep 22, 2021 via email

Tualua commented Sep 22, 2021

Heyshubham commented Sep 22, 2021

Tualua commented Sep 25, 2021 via email

Tualua commented Sep 28, 2021

Heyshubham commented Sep 29, 2021

behlendorf commented Oct 4, 2021

Tualua commented Oct 5, 2021 via email

stale bot commented Oct 14, 2022

behlendorf commented Oct 14, 2022

Missing symlinks in /dev/zvol #12507

Missing symlinks in /dev/zvol #12507

Comments

Tualua commented Aug 24, 2021 • edited Loading

System information

Describe the problem you're observing

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

Tualua commented Aug 24, 2021

Heyshubham commented Aug 31, 2021

mk01 commented Sep 21, 2021

Tualua commented Sep 22, 2021 via email

Tualua commented Sep 22, 2021

Heyshubham commented Sep 22, 2021

Tualua commented Sep 25, 2021 via email

Tualua commented Sep 28, 2021

The problem is games-vm14

Heyshubham commented Sep 29, 2021

behlendorf commented Oct 4, 2021

Tualua commented Oct 5, 2021 via email

stale bot commented Oct 14, 2022

behlendorf commented Oct 14, 2022

Tualua commented Aug 24, 2021 •

edited

Loading

The problem is `games-vm14`