clock_nanosleep is called every second even when multipathd isn't configured

[simondeziel]

While investigating CPU consumption on idle VMs, I noticed that multipathd was calling clock_nanosleep even when not configured. Here's the health-check status over 15s:

# health-check -d15 -fcp $(systemctl show --value -p MainPID multipathd)
CPU usage (in terms of 1 CPU):
     PID Process                USR%   SYS% TOTAL%   Duration
     474 multipathd             0.00   0.00   0.00      15.00  (idle)

Page Faults:
     PID Process                 Minor/sec    Major/sec    Total/sec
     474 multipathd                   0.00         0.00         0.00

Context Switches:
     PID Process                Voluntary   Involuntary     Total
                               Ctxt Sw/Sec  Ctxt Sw/Sec  Ctxt Sw/Sec
     483 multipathd                   9.40         0.00         9.40 (low)
     474 multipathd                   5.00         0.00         5.00 (low)
     484 multipathd                   0.00         0.00         0.00 (idle)
     482 multipathd                   0.00         0.00         0.00 (idle)
     481 multipathd                   0.00         0.00         0.00 (idle)
     480 multipathd                   0.00         0.00         0.00 (idle)
     478 multipathd                   0.00         0.00         0.00 (idle)
 Total                             14.40         0.00        14.40

File I/O operations:
 No file I/O operations detected.

System calls traced:
     PID Process              Syscall               Count    Rate/Sec    Total μSecs  % Call Time
     474 multipathd           futex                    31       2.0665      14702669     50.0008
     478 multipathd           futex                     1       0.0667             0      0.0000
     480 multipathd           ppoll                     1       0.0667             0      0.0000
     481 multipathd           restart_syscall           1       0.0667             0      0.0000
     482 multipathd           restart_syscall           1       0.0667             0      0.0000
     483 multipathd           write                    33       2.1998           482      0.0016
     483 multipathd           futex                    15       0.9999           452      0.0015
     483 multipathd           clock_nanosleep          15       0.9999      14001751     47.6171
     483 multipathd           restart_syscall           1       0.0667        699503      2.3789
     484 multipathd           futex                     1       0.0667             0      0.0000
 Total                                              100       6.6661      29404857

Top polling system calls:
     PID Process              Syscall             Rate/Sec   Infinite   Zero     Minimum    Maximum    Average
                                                             Timeouts Timeouts   Timeout    Timeout    Timeout
     480 multipathd           ppoll                   0.0667        1        0   0.0 sec    0.0 sec    0.0 sec 
     483 multipathd           clock_nanosleep         0.9999        0        0   1.0 sec    1.0 sec    1.0 sec 
 Total                                              1.0666        1        0

Distribution of poll timeout times:
                                                            10.0  100.0    1.0   10.0  100.0    1.0   10.0  100.0
                                                    up to    to     to     to     to     to     to     to  or more
                                              Zero    9.9   99.9  999.9    9.9   99.9  999.9    9.9   99.9        Infinite
     PID Process              Syscall            sec   usec   usec   usec   msec   msec   msec    sec    sec    sec   Wait
     480 multipathd           ppoll                0     -      -      -      -      -      -      -      -      -       1
     483 multipathd           clock_nanosleep      0     -      -      -      -      -      -      15     -      -       0

Polling system call analysis:
 No bad polling discovered.

Filesystem Syncs:
 None.

Inotify watches added:
 None.

Memory:
Per Process Memory (K):
     PID Process              Type        Size       RSS       PSS
     474 multipathd           Stack        132       132       132
     474 multipathd           Heap      279884     18300     18300
     474 multipathd           Mapped      9468      9424      5203

Change in memory (K/second):
 No changes found.

Heap Change via brk():
 None.

Memory Change via mmap() and munmap():
 None.

Open Network Connections:
     PID Process              Proto         Send   Receive  Address
     474 multipathd           UNIX        0.00 B    0.00 B  @/org/kernel/linux/storage/multipathd
     474 multipathd           NETLINK     0.00 B    0.00 B  netlink:[1821]
     474 multipathd           UNIX        0.00 B    0.00 B  /run/systemd/journal/stdout
 Total                                  0.00 B    0.00 B

While this isn't much activity, it accumulates over time:

# uptime
 18:18:04 up 5 days, 23:43,  1 users,  load average: 0.02, 0.01, 0.00
# ps -fp $(systemctl show --value -p MainPID multipathd)
UID          PID    PPID  C STIME TTY          TIME CMD
root         474       1  0 Aug08 ?        00:01:37 /sbin/multipathd -d -s

# lscpu | grep Xeon
Model name:                      Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz

multipathd being installed by default in Ubuntu Server images, tuning the idle behavior could result in interesting power savings considering the scale of unconfigured/unused/idle deployments.

[simondeziel]

I unfortunately cannot contribute any PR to improve the situation but I'd be happy to test any proposed solution if someone is kind enough to work on this. Thanks in advance!

[mwilck]

What do you mean with "not configured"? No multipath devices in the system? In that case, why is the multipathd service enabled?

That said, It's true that multipathd's polling mechanism is suboptimal, I've long thought about improving it.

[simondeziel]

The Ubuntu Server image installs and runs multipathd by default, even when there are no multipath devices. Maybe there is a way to change that to only activate when needed, I'll need to do some research.