txg_sync at 100% reopen #938
Comments
|
Can you get a back trace from the txg_sync thread so we can determine if this really is the same issue. |
|
I think I'm seeing the same issue. Running kernel 3.4.4 (on OEL 6) and spl/zfs 0.6.0rc10. Cheers, |
|
@phaedrus77 I'd actually be much more interested to hear if using the latest master source resolves this issue. There were substantial memory management improvements merged in to master which should help. The master source is stable and there's no risk to your data so it would be an easy thing to try. |
|
i had it and posted it on spl. It was closed before. it was mentioned to be a copy of that was the oops. El 05/09/2012, a las 18:27, Brian Behlendorf [email protected] escribió:
|
|
@behlendorf I pulled the source from git (zfs+spl) as zipped archives: spl compiles fine, but zfs chokes on spl has kpreempt_* defined in include/sys/disp.h as while zfs has which one is the right one to use in this place? should this part only be compiled when _KERNEL is defined? where should I put the missing ifdefs? did I miss a step or is my build environment broken? |
|
for testing further I commented the kpreempt lines out of fm.c now I get seems to hit the same problem. the error on line 391 is interesting though: |
|
You're getting these errors because ZFS is being built with an old SPL version. Compile the latest SPL from master, then compile latest ZFS from master using |
|
@dechamps thanks this was it. i forgot to install the freshly built spl bits :-\ @behlendorf master looks good! I was able to boot a VM while having firefox running. txg_sync runs up pretty high - consuming about two cores on my maschine while the VM boots up an does loads of IO but returns to normal when the VMs IO is down to normal. |
|
That's great news. I'm going to close out this issue since it sounds like the updated master code is working as expected and addresses the issue. |
|
I'm afraid the problem wasn't 100% solved. After running a few days (with several suspends/resumes) txg_sync came back blocking io to the pool. I can trigger the lockup by heavily stressing memory. I tried to observe as much as possible and noticed that kswapd was doing its thing and arc size went down by a huge amount (~1.5GB, from about 1.8GB down to well below 512MB, where arc_size stabilized) that seems to somehow get txg_sync stuck. |
|
OK, I've reopened the issue. It sounds like things are better but not 100% fixed. If your able to reproduce this issue please grab the contents of |
|
I did a while starting a VM in vbox. txg_sync started consuming lots of cpu time when vbox allocated memory for the vm. all processes accessing data on the same dataset hung when txg_sync went up. |
|
another crash today with an interesting trace in the logs: I hope this helps to narrow it down. |
|
Some more information to narrow this down. It tested zfsonlinux on several of my systems. I encouter this issue not on all of those machines. Debian Squeeze, standard Debian kernel 2.6.32-5-amd64. ZFS 0.6.0-rc9 and 0.6.0-rc11 compiled as described on your web page. ZFS on a single partition. ZFS works here without any problems. I don't see any txg_sync process with high CPU load. Debian Squeeze. Kernel from Debian Wheezy 3.2.0-3-amd64. ZFS 0.6.0-rc9 and 0.6.0-rc11. ZFS on 4 x 2 mirror SSDs, or using raidz3 or just on a single partition. I always see txg_sync process with 100% CPU load after writing 5-10GB to that filesystem. Gentoo Linux. Kernel from vanilla-sources 3.5.0 built with genkernel, default configuration. sys-kernel/spl, sys-fs/zfs-kmod, sys-fs/zfs 0.6.0-rc10. USE flags: kernel_linux, rootfs. Two ZFS pools on single partitions, several subvolumes. No txg_sync process with high CPU load. I will provide output of /proc/spl/kstat/zfs/dmu_tx the week after next because I am on vacation. |
|
It sounds like Debian is applying a patch to Linux 3.2 that causes problems with ZFS. It would be helpful if someone could find out where Debian keeps those patches so that they could be examined. |
|
Sources are available here: http://packages.debian.org/testing/kernel/linux-image-3.2.0-3-amd64 |
|
in my case i pulled the vanilla sources and built the kernel myself... |
|
The patches that debian applies are included as individual files in the following tarball: http://ftp.us.debian.org/debian/pool/main/l/linux/linux_3.2.23-1.debian.tar.xz |
|
@kleini I am running I was running iozone3 to test speed of a set of new hard-disks that would generate files around 8GB each. This had been going for about 36-hours when I did a Traces from syslog: http://fpaste.org/xtL8/ |
|
I can confirm this under Debian Wheezy 3.2.0-4 (3.2.32-1), txg_sync keeps at 100% when starting a VirtualBox guest vm, for example. I'm using latest ZFS/SPL 0.6.0-rc11 on a fresh install. There are no (strange/dump/traces) messages whatsoever on dmesg/syslog, it just hangs at 100% but the rest of the system keeps responsible (but the VM doesn't start after all). My arc max size is set to 512M, the machine has only 4GB of RAM and the VM is set to use 1GB, and it is a mirrored setup (sata/sata). Doesn't matter if I increase or decrease arc size, I get the same result. |
|
decreasing arc (8GB ram, arc set to 512MB) has helped quite a bit for me. |
|
@dajhorn, is there any chance that you could take a look at this? |
|
@ryao, maybe on Friday. Whenever I see Virtual Box in a bug report, I ask the user to install the latest official build from Oracle. It tends to make problems like this go away. |
|
zpool create -m /test mirror /dev/sda /dev/sdb The written file gets stuck at 922M. Kernel: Debian 3.2.23-1 from Debian Wheezy. Happens with 0.6.0-rc9 and 0.6.0-rc11. Output of /proc/spl/kstat/zfs/dmu_tx: cat /proc/spl/kstat/zfs/dmu_tx cat /proc/spl/kstat/zfs/dmu_tx cat /proc/spl/kstat/zfs/dmu_tx |
|
@dajhorn it's been persistant for at least the last 3 VBox builds. I'd say I could trigger this by just eating huge chunk of memory. |
|
@phaedrus77 The changes in |
|
It is a 64-bit system. You are talking about some memory pressure on the system. The system has 16G memory, an 8-core hyperthreading CPU and only the base system was installed and running. So less than 1G of the memory was in use. 15G of free memory should not be a situation of memory pressure. |
|
@behlendorf it's a 64bit system already 8-) I'll grab the latest master tomorrow or so and try that... |
|
On my installation are no virtual machines. The whole memory was available for the base linux system and ZFS. I do not have a swap zvol. I just noticed this is allowed now - my knowledge to date remembers swap on zvol is discouraged. |
|
I have max 4GB allocated a VM + 128MB "Video RAM" Thats the only VM running when it hits the bug. I have some smaller ones running in parallel as well every now and then (3*1GB+2GB) which also triggers the bug. It can happen pretty much anytime when large chunks of RAM are allocated and some process tries to allocate some more or a new process starts. |
|
My configuration does NOT have swap on a zvol. The only thing special about my setup is that one of the zvols (telinbackup2) The second zpool (zfs) is a mirror created out of two lvm partitions on different disks. The backups (to backup2 and to backup) are run sequentially in a round robin fashion. On devel, snapshots are created automatically every hour and older snapshots are cleaned out. All zfs filesystems are compressed using gzip. I hope this helps. |
|
@phaedrus77 When I say memory pressure in this case it's entirely possible that there is no memory pressure and ZFS is just getting the calculation wrong. It tries (for better or worse) to be proactive about throttling things when it observed what it believes to be a low memory situation. Because of how Virtual Box works it's been known to throw off those calculation which may can ZFS to over aggressively throttle the system. In this case it's attempting to ensure there's enough free memory (or reclaimable memory) on the system to construct a full txg. See You may be able to improve this by decreasing the maximum txg size which is set by default to 1/8 of system memory. Try setting the |
|
@phaedrus77 where do we currently stand on this? Presumably your still having issues, did changing |
|
Ehm. I still had issues and Big trouble getting my System back to productive. I used the parameter and for me it works. I only have kmem_cache now a bit busy (5-10%). But the overall overall performance is awefull. |
|
@behlendorf I'm still trying different values for zfs_write_limit_shift. Currently I have set this to 6. I still see txg_sync go up eating cpu like mad and block IO but until now it came back after 10-20seconds. I'm trying to really hammer my machine tonight. as @mcr_ksh said it comes with a performance hit. |
|
Thanks. Could you both include the following patch in your testing. It should help with the large cpu usage by |
|
I had a bad lockup this morning again (VBox and Firefox figthing over the memory). I had zfs_write_limit_shift=6 and arc_max set to 512MB. |
|
Just had the next lockup. This was rc13 + your patch and options set as before. Of my 8GB RAM about 60MB were free which got txg_sync stuck. I'll see if further limiting zfs_write_limit_shift helps. |
|
@phaedrus77 When things get stuck is it always because of the 'dmu_tx_memory_reclaim' count in /proc/spl/kstat/zfs/dmu_tx. If so we should consider disabling that throttle, it's not critical. It's another bit of code we inherited from Solaris and it's completely unclear if it's really needed for Linux. |
|
I think when I watched /proc/spl/kstat/zfs/dmu_tx I saw dmu_tx_memory_reclaim growing. I'll have an eye on it when I next trash the box ;-) |
|
I've found "peace" on VirtualBox by using these module options:
(Debian 3.2.0-4-amd64) |
|
For me the problem seems to be fixed in one of the RC 9-13. For me the additional module options are not necessary. |
|
I still see the problem with rc13. I'll give the options a try. |
|
with @turrini optins rc13 on kernel 3.4.4 hans again. |
|
@phaedrus77 It's clearly stalling because it believes there to be memory pressure (there in fact may not be). Can you try making this simple change which disables this throttle. |
|
@behlendorf Thanks! That's the magic patch. I have firefox and vbox running without txg_sync blocking anything. top reports free memory changing somewhere between 60MB and 150MB while both are running. Before it remaind constant. dmu_tx_memory_reclaim stays at 0. also now I see pages being moved to swap, which never happend when txg_sync blocked. |
|
@phaedrus77 That's good news. This sort of thing has come up a few times now always related to virtual box or in the past 32-bit systems. I'll push a patch in to master today to allow you to disable this code just with a module option so you won't need to carry a custom patch set More broadly disabling this code is probably the right thing for all Linux systems since we have a direct memory reclaim patch where Solaris does not. But before I make that the default I'd like to get some additional real world testing with this code disabled. |
The zfs_arc_memory_throttle_disable module option was introduced by commit 0c5493d to resolve a memory miscalculation which could result in the txg_sync thread spinning. When this was first introduced the default behavior was left unchanged until enough real world usage confirmed there were no unexpected issues. We've now reached that point. Linux's direct reclaim is working as expected so we're enabling this behavior by default. This helps pave the way to retire the spl_kmem_availrmem() functionality in the SPL layer. This was the only caller. Signed-off-by: Brian Behlendorf <[email protected]> Issue #938
The zfs_arc_memory_throttle_disable module option was introduced by commit 0c5493d to resolve a memory miscalculation which could result in the txg_sync thread spinning. When this was first introduced the default behavior was left unchanged until enough real world usage confirmed there were no unexpected issues. We've now reached that point. Linux's direct reclaim is working as expected so we're enabling this behavior by default. This helps pave the way to retire the spl_kmem_availrmem() functionality in the SPL layer. This was the only caller. Signed-off-by: Brian Behlendorf <[email protected]> Issue openzfs#938
The way in which virtual box ab(uses) memory can throw off the free memory calculation in arc_memory_throttle(). The result is the txg_sync thread will effectively spin waiting for memory to be released even though there's lots of memory on the system. To handle this case I'm adding a zfs_arc_memory_throttle_disable module option largely for virtual box users. Setting this option disables free memory checks which allows the txg_sync thread to make progress. By default this option is disabled to preserve the current behavior. However, because Linux supports direct memory reclaim it's doubtful throttling due to perceived memory pressure is ever a good idea. We should enable this option by default once we've done enough real world testing to convince ourselve there aren't any unexpected side effects. Signed-off-by: Brian Behlendorf <[email protected]> Closes openzfs#938
The zfs_arc_memory_throttle_disable module option was introduced by commit 0c5493d to resolve a memory miscalculation which could result in the txg_sync thread spinning. When this was first introduced the default behavior was left unchanged until enough real world usage confirmed there were no unexpected issues. We've now reached that point. Linux's direct reclaim is working as expected so we're enabling this behavior by default. This helps pave the way to retire the spl_kmem_availrmem() functionality in the SPL layer. This was the only caller. Signed-off-by: Brian Behlendorf <[email protected]> Issue openzfs#938
|
@turrini - Thank you for your "peaceful" VirtualBox options! I am running ZFS on Linux on Debian within ESXi, with RDM mapped drives (yeah, it's pretty rough as a configuration goes!) However, your settings really helped restore performance - writes up from 800KB/sec to around 40-50MB/sec - understandable given that the array in question is on an external port multiplier! To save anyone else from searching too far - here's how to apply those options to ZFS on boot:
You can check what each parameter is set to by running cat /sys/module/zfs/parameters/ Hope this helps someone else! |
|
@blueacid In case if you're using ZFS on root, remember that actually on master there aren't anymore some of these module options, so it will result in an unbootable system if you don't remove them.
More info: |
|
Hi There, Thanks for the warning - in my case I'm not using ZFS as my root file
|
Bumps [gimli](https://github.com/gimli-rs/gimli) from 0.27.2 to 0.27.3. - [Changelog](https://github.com/gimli-rs/gimli/blob/master/CHANGELOG.md) - [Commits](gimli-rs/gimli@0.27.2...0.27.3) --- updated-dependencies: - dependency-name: gimli dependency-type: indirect update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Hi,
got the same issue as it was closed before ! i just posted a kernel panic yesterday on SPL.
txg_sync is at 100%
5268 root 0 -20 0 0 0 D 40.8 0.0 7:21.33 txg_sync
devices blocked and are irresponsive. All VMs gone.
I was on latest commit.
Regards,
Chris.
The text was updated successfully, but these errors were encountered: