Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

-x does not seem to honor filesystem boundaries - 5.10.0 #106

Closed
danie-dejager opened this issue Nov 16, 2021 · 20 comments
Closed

-x does not seem to honor filesystem boundaries - 5.10.0 #106

danie-dejager opened this issue Nov 16, 2021 · 20 comments
Assignees
@dundee
Labels
bug Something isn't working

Comments

@danie-dejager
Copy link
Collaborator

danie-dejager commented Nov 16, 2021 •
edited
Loading

I have the following config

┌──┬─────┬──────────────┬────┬────┬─────┬─────────┬─────┬──────┬──────────────────┐
│id│ dev │  filesystem  │disk│type│used │  use%   │ free│ size │mount point       │
├──┼─────┼──────────────┼────┼────┼─────┼─────────┼─────┼──────┼──────────────────┤
│44│259:5│/dev/nvme3n1  │SSD │xfs │118Mi│ 1%      │ 11Gi│11Gi  │/var/app-local    │
│55│259:2│/dev/nvme5n1  │SSD │xfs │100Mi│ 1%      │9.9Gi│10.0Gi│/var/app          │
│38│259:7│/dev/nvme0n1p1│SSD │xfs │7.5Gi│75% ███▊ │2.5Gi│10.0Gi│/                 │
│42│259:4│/dev/nvme4n1  │SSD │xfs │3.1Gi│35% █▊   │5.9Gi│9.0Gi │/usr/local/app    │
│54│259:1│/dev/nvme2n1  │SSD │xfs │568Mi│19% ▉    │2.4Gi│3.0Gi │/var/log          │
│43│259:3│/dev/nvme6n1  │SSD │xfs │614Mi│30% █▌   │1.4Gi│2.0Gi │/tmp              │
│57│259:3│/dev/nvme6n1  │SSD │xfs │614Mi│30% █▌   │1.4Gi│2.0Gi │/var/tmp          │
└──┴─────┴──────────────┴────┴────┴─────┴─────────┴─────┴──────┴──────────────────

when I run gdu from / I do not expect to see gdu counting totals for /var/app-local but that is what happens. Here is output from ncdu and gdu. The sizes differ as -x does not seem to work for gdu.
gdu -x
image

ncdu -x
image
@dundee dundee self-assigned this Nov 16, 2021
@dundee dundee added the bug Something isn't working label Nov 16, 2021
@dundee
Copy link
Owner

dundee commented Nov 16, 2021

Could you paste the contents of /proc/mounts?

@danie-dejager
Copy link
Collaborator Author

danie-dejager commented Nov 16, 2021 •
edited
Loading

Here it is from one of my servers:

sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
devtmpfs /dev devtmpfs rw,nosuid,size=8035508k,nr_inodes=2008877,mode=755 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_prio,net_cls 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
/dev/nvme0n1p1 / xfs rw,relatime,attr2,inode64,usrquota,prjquota 0 0
rpc_pipefs /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=12840 0 0
/dev/loop6 /var/lib/snapd/snap/snapd/13640 squashfs ro,nodev,relatime 0 0
/dev/loop2 /var/lib/snapd/snap/htop/3233 squashfs ro,nodev,relatime 0 0
/dev/loop3 /var/lib/snapd/snap/btop/189 squashfs ro,nodev,relatime 0 0
/dev/loop0 /var/lib/snapd/snap/snapd/13831 squashfs ro,nodev,relatime 0 0
/dev/loop5 /var/lib/snapd/snap/core20/1169 squashfs ro,nodev,relatime 0 0
/dev/loop4 /var/lib/snapd/snap/htop/3161 squashfs ro,nodev,relatime 0 0
/dev/loop7 /var/lib/snapd/snap/btop/193 squashfs ro,nodev,relatime 0 0
/dev/loop1 /var/lib/snapd/snap/certbot/1582 squashfs ro,nodev,relatime 0 0
/dev/loop8 /var/lib/snapd/snap/certbot/1514 squashfs ro,nodev,relatime 0 0
/dev/loop9 /var/lib/snapd/snap/core20/1081 squashfs ro,nodev,relatime 0 0
/dev/loop10 /var/lib/snapd/snap/certbot-dns-cloudflare/1189 squashfs ro,nodev,relatime 0 0
/dev/loop11 /var/lib/snapd/snap/certbot-dns-cloudflare/1121 squashfs ro,nodev,relatime 0 0
/dev/nvme2n1p1 /tmp xfs rw,relatime,attr2,inode64,usrquota,prjquota,grpquota 0 0
/dev/nvme10n1p1 /var/app xfs rw,relatime,attr2,inode64,usrquota,prjquota,grpquota 0 0
/dev/nvme2n1p1 /var/tmp xfs rw,relatime,attr2,inode64,usrquota,prjquota,grpquota 0 0
/dev/nvme9n1p1 /var/log xfs rw,relatime,attr2,inode64,usrquota,prjquota,grpquota 0 0
/dev/nvme4n1p1 /var/app/data xfs rw,relatime,attr2,inode64,usrquota,prjquota,grpquota 0 0
/dev/nvme8n1p1 /usr/local/app xfs rw,relatime,attr2,inode64,usrquota,prjquota,grpquota 0 0
/dev/nvme5n1p1 /var/app-local xfs rw,relatime,attr2,inode64,usrquota,prjquota,grpquota 0 0
/dev/nvme3n1p1 /home xfs rw,relatime,attr2,inode64,usrquota,prjquota,grpquota 0 0
tmpfs /run/user/0 tmpfs rw,nosuid,nodev,relatime,size=1612088k,mode=700 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
tmpfs /run/user/1003 tmpfs rw,nosuid,nodev,relatime,size=1612088k,mode=700,uid=1003,gid=1005 0 0

@dundee
Copy link
Owner

dundee commented Nov 16, 2021

Thanks. I don't see a bug there at first glance, will have to dig in deeper.

@oderwat oderwat mentioned this issue Nov 17, 2021
Closed
@dundee
Copy link
Owner

dundee commented Nov 18, 2021

@daniejstriata Could you run gdu with logging (-l gdu.log) a paste the resulted file here? Thanks!

@danie-dejager
Copy link
Collaborator Author

danie-dejager commented Nov 18, 2021 •
edited
Loading 

time="2021-11-19T02:37:01+05:30" level=info msg="Ignoring dirs /proc, /dev, /sys, /run"
time="2021-11-19T02:37:01+05:30" level=info msg="Max cores set to 4"
time="2021-11-19T02:37:01+05:30" level=info msg="Directory /dev ignored"
time="2021-11-19T02:37:01+05:30" level=info msg="Directory /proc ignored"
time="2021-11-19T02:37:01+05:30" level=info msg="Directory /run ignored"
time="2021-11-19T02:37:01+05:30" level=info msg="Directory /sys ignored"

@dundee
Copy link
Owner

dundee commented Nov 19, 2021

Interesting, it does not ignore any mounted devices. Have you run it with -x option?

@danie-dejager
Copy link
Collaborator Author

Yes. I ran it with -x

@dundee
Copy link
Owner

dundee commented Nov 19, 2021

I have added some more logging. Could you paste the log output again?
gdu_linux_amd64.zip

@danie-dejager
Copy link
Collaborator Author

danie-dejager commented Nov 19, 2021 •
edited
Loading

Log:

time="2021-11-19T20:31:49Z" level=info msg="Runtime flags: {LogFile:gdu.log InputFile: OutputFile: IgnoreDirs:[/proc /dev /sys /run] IgnoreDirPatterns:[] IgnoreFromFile: MaxCores:4 ShowDisks:false ShowApparentSize:false ShowVersion:false NoColor:false NonInteractive:false NoProgress:false NoCross:true NoHidden:false Summarize:false}"
time="2021-11-19T20:31:49Z" level=info msg="Ignoring mount points: "
time="2021-11-19T20:31:49Z" level=info msg="Ignoring dirs /proc, /dev, /sys, /run"
time="2021-11-19T20:31:49Z" level=info msg="Max cores set to 4"
time="2021-11-19T20:31:49Z" level=info msg="Analyzing path: /"
time="2021-11-19T20:31:49Z" level=info msg="Directory /dev ignored"
time="2021-11-19T20:31:49Z" level=info msg="Directory /proc ignored"
time="2021-11-19T20:31:49Z" level=info msg="Directory /run ignored"
time="2021-11-19T20:31:49Z" level=info msg="Directory /sys ignored"

gdu -d

Device name    Size       Used      Used part    Free      Mount point                                                                          
/dev/nvme0n1p1 15.0 GiB   7.4 GiB   [####      ] 7.6 GiB   /                                                                                    
/dev/nvme4n1   9.0 GiB    5.2 GiB   [#####     ] 3.8 GiB   /usr/local/app                                                                   
/dev/nvme6n1   10.0 GiB   1.1 GiB   [#         ] 8.9 GiB   /var/app                                                                  
/dev/nvme7n1   2.0 GiB    742.5 MiB [###       ] 1.3 GiB   /tmp                                                                                 
/dev/nvme1n1   3.0 GiB    571.9 MiB [#         ] 2.4 GiB   /var/log                                                                             
/dev/nvme3n1   11.0 GiB   133.0 MiB [          ] 10.9 GiB  /var/app-local                                                                   
/dev/nvme8n1   1014.0 MiB 54.1 MiB  [          ] 959.9 MiB /home

gdu -x
image
ncdu -x
image
Mounts:
mounts.txt

@dundee
Copy link
Owner

dundee commented Nov 19, 2021

Wow, that is really strange. I tried to simulate it locally but without success.

@dundee
Copy link
Owner

dundee commented Nov 19, 2021

I tried multiple OSes (Debian, Ubuntu, Arch) but without success to replicate the bug. @daniejstriata which operating system do you use?

@danie-dejager
Copy link
Collaborator Author

I'm on Centos 7 and 8 with selinux on and off.

@danie-dejager
Copy link
Collaborator Author

danie-dejager commented Nov 22, 2021 •
edited
Loading

I also get the same on Debian 10
gdu -x
image

ncdu -x
image

@dundee
Copy link
Owner

dundee commented Nov 22, 2021

Strange. It works well for me on Debian.

/dev/sda1 /mnt ext4 rw,relatime,errors=remount-ro 0 0

time="2021-11-22T11:52:02+01:00" level=info msg="Directory /mnt ignored"

@dundee
Copy link
Owner

dundee commented Nov 22, 2021

Even, with the name local-data1.

/dev/sda1 /local-data1 ext4 rw,relatime,errors=remount-ro 0 0

time="2021-11-22T11:55:58+01:00" level=info msg="Directory /local-data1 ignored"

VirtualBox_Debian Sid_22_11_2021_12_00_25
VirtualBox_Debian Sid_22_11_2021_12_00_48

@danie-dejager
Copy link
Collaborator Author

This is the output from gdu -d
image

@dundee
Copy link
Owner

dundee commented Nov 23, 2021

I have added some more logging. Could you try once again and paste the log output?
gdu.zip

@danie-dejager
Copy link
Collaborator Author

danie-dejager commented Nov 23, 2021 •
edited
Loading

This is running your binary on Fedora 34. It should've excluded /Data from the results. Could it be ignoring the drives because they are encrypted? That's the only thing I think would be different in my environment to someone else.

 / $ sudo /home/danie.dejager/gdu -x -l gdu.log
[sudo] password for danie.dejager: 
/ $ sudo cat gdu.log 
time="2021-11-23T14:46:00+02:00" level=info msg="Runtime flags: {LogFile:gdu.log InputFile: OutputFile: IgnoreDirs:[/proc /dev /sys /run] IgnoreDirPatterns:[] IgnoreFromFile: MaxCores:12 ShowDisks:false ShowApparentSize:false ShowVersion:false NoColor:false NonInteractive:false NoProgress:false NoCross:true NoHidden:false Summarize:false}"
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /proc is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /sys is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /dev is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /sys/kernel/security is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /dev/shm is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /dev/pts is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /run is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /sys/fs/cgroup is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /sys/fs/pstore is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /sys/firmware/efi/efivars is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /sys/fs/bpf is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: / is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /sys/fs/selinux is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /proc/sys/fs/binfmt_misc is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /dev/mqueue is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /sys/kernel/debug is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /dev/hugepages is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /sys/kernel/tracing is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /sys/fs/fuse/connections is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /sys/kernel/config is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /tmp is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /home is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /var/lib/snapd/snap/core20/1242 is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /var/lib/snapd/snap/core18/2253 is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /var/lib/snapd/snap/bare/5 is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /var/lib/snapd/snap/gnome-3-34-1804/72 is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /var/lib/snapd/snap/pinta/12 is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /var/lib/snapd/snap/gnome-3-38-2004/76 is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /var/lib/snapd/snap/gtk-common-themes/1519 is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /var/lib/snapd/snap/btop/213 is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /var/lib/snapd/snap/htop/3161 is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /var/lib/snapd/snap/btop/193 is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /var/lib/snapd/snap/htop/3233 is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /var/lib/snapd/snap/core20/1169 is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /var/lib/snapd/snap/core18/2246 is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /var/lib/snapd/snap/snapd/13270 is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /var/lib/snapd/snap/pinta/13 is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /var/lib/snapd/snap/remmina/5073 is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /var/lib/snapd/snap/remmina/5071 is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /var/lib/snapd/snap/ssh-audit/4 is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /var/lib/snapd/snap/ssh-audit/6 is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /var/lib/snapd/snap/snapd/13640 is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /boot is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /boot/efi is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /Data is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /var/lib/nfs/rpc_pipefs is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /opt/sophos-spl/var/sophos-spl-comms/lib is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /opt/sophos-spl/var/sophos-spl-comms/usr/lib is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /opt/sophos-spl/var/sophos-spl-comms/etc/hosts is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /opt/sophos-spl/var/sophos-spl-comms/etc/resolv.conf is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /opt/sophos-spl/var/sophos-spl-comms/usr/lib64 is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /opt/sophos-spl/var/sophos-spl-comms/etc/ssl/certs is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /opt/sophos-spl/var/sophos-spl-comms/etc/pki/tls/certs is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /opt/sophos-spl/var/sophos-spl-comms/etc/pki/ca-trust/extracted is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /opt/sophos-spl/var/sophos-spl-comms/base/mcs/certs is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /opt/sophos-spl/var/sophos-spl-comms/base/remote-diagnose/output is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /run/snapd/ns is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /run/snapd/ns/remmina.mnt is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /run/user/1001 is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /run/user/1001/doc is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /proc/sys/fs/binfmt_misc is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Mount point: /run/user/0 is outside path ."
time="2021-11-23T14:46:00+02:00" level=info msg="Ignoring mount points: "
time="2021-11-23T14:46:00+02:00" level=info msg="Ignoring dirs /proc, /dev, /sys, /run"
time="2021-11-23T14:46:00+02:00" level=info msg="Max cores set to 12"
time="2021-11-23T14:46:00+02:00" level=info msg="Analyzing path: /"
time="2021-11-23T14:46:00+02:00" level=info msg="Directory /dev ignored"
time="2021-11-23T14:46:00+02:00" level=info msg="Directory /proc ignored"
time="2021-11-23T14:46:00+02:00" level=info msg="Directory /run ignored"
time="2021-11-23T14:46:00+02:00" level=info msg="Directory /sys ignored"

image

image

sda 8:0 0 931.5G 0 disk
└─sda 253:1 0 931.5G 0 crypt /Data

@dundee
Copy link
Owner

dundee commented Nov 23, 2021

Ahhh, I see the problem. You are running it without specifing exact path (/). I will fix it.

@dundee dundee closed this as completed in 3b5e3ee Nov 23, 2021
@dundee
Copy link
Owner

dundee commented Nov 23, 2021

Thanks @daniejstriata for debugging!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dundee dundee

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dundee @danie-dejager