Increased Memory Usage in 2.1.11-1

[gbonfiglio]

Hello ZFS Community!

I'm pretty sure there is something wrong with this ZFS version on Debian, but haven't been able to collect definitive data so would appreciate some help.

Situation: I have a small VM (2 cores / 2 GB RAM) which I use as a backup destination. It runs two zpools, compression enabled, dedup disabled. There is zero tuning and literally all config values are Debian's defaults.

NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
library   179G   122G  56.8G        -         -     1%    68%  1.00x    ONLINE  -
tank      496G   424G  72.3G        -         -    55%    85%  1.00x    ONLINE  -

They are single device zpools (the virtual device on the host is in raid), and this setup has served me well for a while on Debian 11 / ZFS 2.0.3-9. Note this VM doesn't literally run anything else, other than a small hourly rsync cron - RAM usage has always been stable at 50% memory for ARC and another 3/400 mb for everything else.

Since the upgrade to Debian 12 / ZFS 2.1.11-1 my system started frequently going OOM. I have doubled this VM's memory allocation expecting to see ARC using half of it and the same 3/400 mb buffer for other processes, but the result is ZFS seems to be eating up about 3.6 GB in total, getting me incredibly close to OOM'ing again.

My ARC summary is here https://pastebin.com/raw/hfDcWD5k, and seems to be fine.

This is a free -m:

               total        used        free      shared  buff/cache   available
Mem:            3915        3586         342           2         208         328
Swap:              0           0           0

A sample OOM is https://pastebin.com/raw/9BkG7CWC

I might be completely wrong (obviously), but have a strong suspicion something changed between 2.0 and 2.1 which is causing this, so would appreciate any idea to collect further data.

Thanks!
G

[rincebrain]

I'm not an expert on reading those, but it seems like the total usage in those caches that is reported that's to blame from ZFS is about 1.25 GiB.

That's not to say it can't be ZFS using the RAM, just that the output there is insufficient to determine that.

It seems like almost all of your ARC caching is from metadata, and it's well under 3 GiB, so what's making you assume it's ZFS doing it? (I'm not, to be clear, saying it's not, just that the numbers you've presented seem to be ARC behaving as expected...)

It could also be an interaction with the new kernel, so you could try 2.1.11 on the bullseye kernel and see if it still misbehaves the way you're seeing here...

[gbonfiglio]

This is a good question - accounting for kernelspace memory usage isn't easy, but I'm basing my focus on ZFS on two things:

• there are no non-default daemons other than zfs
• the vm boots with 350 mb of ram, usage only spikes as file operations start

I've just gone back to 5.10.0-23-amd64, will report back!

[gbonfiglio]

Yeah, 5.10 makes the problem go away. I could even revert to my previous 2GB RAM config. Now off to find where the unaccounted for memory is being used. Any hint? Thanks!

[rincebrain]

Neat.

My guess would be something changed with the folio reworking, but I can't imagine offhand what.

I'd look at the configure check outputs for OpenZFS between 5.10 and 6.1 and see if there's any checks now reporting "no" for any available interface that said "yes" before, as a quick pass, but otherwise, I'd just compare the list of kmem allocation usages periodically and see which one(s) are bloating wildly differently between them, then try to figure out why.

[gbonfiglio]

Moving to #15637