Heavy lock contention in zfs_zinactive causing ARC metadata to grow #1932
Labels
Component: Memory Management
kernel memory management
Milestone
Comments
|
Are xattrs being used on the filesystems in question? If so is xattr=sa set? EDIT: Also, if using xattrs, are there a lot of unlink operations going on? |
|
No xattrs are in use (as far as I know, some may exist but There's 7 big filesystems with numbers of inodes in the 3 to 15 million ranges. Let's say the average is 8 or 10 million each. Several threads per filesystem are running through various directories. |
|
Pull request 1696, while making things better, doesn't eliminate the issue completely. After an overnight run arcstats show |
|
So I think I just had it deadlock. ZFS is good and jammed but the rest of the OS is okay enough for me to collect all the below data straight from the machine while ZFS is hung. HardwareCPU: Dual-core Xeon 2.33 GHz (core2 Duo based) Disk config:
ZFS/SPL version~/spl# git describe 0.6.2-24_g3d74857 ~/spl# git log --pretty=oneline | head 3d748574863 Use Linux slab allocator for small allocations 4c995417bc4 Remove incorrect use of EXTRA_DIST for man pages d58a99af2f3 Define the needed ISA types for Sparc aeeb4e0c0ae Remove default taskq thread to CPU bindings ~/zfs# git describe 0.6.2-186_g1dbdb3d ~zfs# git log --pretty=oneline 1dbdb3d358 Remove kmutex_t & kcondvar out of arc_buf_hdr_t 9fe73f2368 Set "arc_meta_limit" to 3/4 arc_c_max by default c8170bfd95 Split "data_size" into "meta" and "data" 6c61ec8df2 Prioritize "metadata" in arc_get_data_buf 56f4f84570 Fix l2arc_write_done/l2arc_evict race 51bd84fc6b Remove "arc_meta_used" from arc_adjust calculation 41dff7fcb0 Prune metadata from ghost lists in arc_adjust_meta 35af4e16a0 Revert "Return -1 from arc_shrinker_func()" 9932e4acb2 3 Disable arc_p adapt dampener by default 4c77d71fd3 Allow "arc_p" to drop to zero or grow to "arc_c" c53c04986f Disable aggressive arc_p growth by default 02fab31f39 Adjust arc_p based on "bytes" in arc_shrink da7d1c5235 Set zfs_arc_min to 4MB 881f45c6a8 Add systemd unit files for ZFS startup c5cb66addc Fix corrupted l2_asize in arcstats SPL patch reference openzfs/spl#328 ZFS patch reference #2110 and #1971 Stack tracesSysRq : Show Blocked State task PC stack pid father kswapd0 D 0000000000000001 0 388 2 0x00000000 ffff88041c3d7a20 0000000000000046 ffff8802d3407f08 ffff88041c3d6010 ffff88041da29980 0000000000010900 ffff88041c3d7fd8 0000000000004000 ffff88041c3d7fd8 0000000000010900 ffff88041d863fc0 ffff88041da29980 Call Trace: [<ffffffff810ad987>] ? __slab_free+0x189/0x2a5 [<ffffffffa00d7c58>] ? arc_buf_eviction_needed+0x5f/0x69 [zfs] [<ffffffffa00e0579>] ? dmu_buf_rele+0x25/0x2c [zfs] [<ffffffff8130e2a2>] schedule+0x5f/0x61 [<ffffffff8130e4dc>] schedule_preempt_disabled+0x9/0xb [<ffffffff8130cd58>] __mutex_lock_slowpath+0xde/0x123 [<ffffffff81098347>] ? unmap_mapping_range+0x15d/0x16e [<ffffffff8130cbf2>] mutex_lock+0x16/0x27 [<ffffffffa0156e0d>] zfs_zinactive+0x67/0xd7 [zfs] [<ffffffffa014f533>] zfs_inactive+0x112/0x1ac [zfs] [<ffffffff810881f9>] ? truncate_pagecache+0x51/0x59 [<ffffffffa0165ed5>] zpl_evict_inode+0x23/0x27 [zfs] [<ffffffff810cbfb0>] evict+0xa2/0x155 [<ffffffff810cc50d>] dispose_list+0x3d/0x4a [<ffffffff810cc7e3>] prune_icache_sb+0x2c9/0x2d8 [<ffffffff810ba66b>] prune_super+0xe3/0x158 [<ffffffff8108a3a8>] shrink_slab+0x122/0x19c [<ffffffff8108b23b>] balance_pgdat+0x3a4/0x7f5 [<ffffffff8108f2a0>] ? calculate_pressure_threshold+0x1c/0x40 [<ffffffff8108b93b>] kswapd+0x2af/0x2e4 [<ffffffff8103e9c0>] ? wake_up_bit+0x25/0x25 [<ffffffff8108b68c>] ? balance_pgdat+0x7f5/0x7f5 [<ffffffff8103e5ca>] kthread+0x84/0x8c [<ffffffff81310654>] kernel_thread_helper+0x4/0x10 [<ffffffff8103e546>] ? kthread_freezable_should_stop+0x58/0x58 [<ffffffff81310650>] ? gs_change+0xb/0xb spl_kmem_cache/ D 0000000000000001 0 1664 2 0x00000000 ffff88041c381690 0000000000000046 ffff880049d78000 ffff88041c380010 ffff88041da99980 0000000000010900 ffff88041c381fd8 0000000000004000 ffff88041c381fd8 0000000000010900 ffff88041d863fc0 ffff88041da99980 Call Trace: [<ffffffffa00d7c58>] ? arc_buf_eviction_needed+0x5f/0x69 [zfs] [<ffffffffa00e0579>] ? dmu_buf_rele+0x25/0x2c [zfs] [<ffffffff8130e2a2>] schedule+0x5f/0x61 [<ffffffff8130e4dc>] schedule_preempt_disabled+0x9/0xb [<ffffffff8130cd58>] __mutex_lock_slowpath+0xde/0x123 [<ffffffff81098347>] ? unmap_mapping_range+0x15d/0x16e [<ffffffff8130cbf2>] mutex_lock+0x16/0x27 [<ffffffffa0156e0d>] zfs_zinactive+0x67/0xd7 [zfs] [<ffffffffa014f533>] zfs_inactive+0x112/0x1ac [zfs] [<ffffffff810881f9>] ? truncate_pagecache+0x51/0x59 [<ffffffffa0165ed5>] zpl_evict_inode+0x23/0x27 [zfs] [<ffffffff810cbfb0>] evict+0xa2/0x155 [<ffffffff810cc50d>] dispose_list+0x3d/0x4a [<ffffffff810cc7e3>] prune_icache_sb+0x2c9/0x2d8 [<ffffffff810ba66b>] prune_super+0xe3/0x158 [<ffffffff8108a3a8>] shrink_slab+0x122/0x19c [<ffffffff8108aca1>] do_try_to_free_pages+0x318/0x4a4 [<ffffffff81083951>] ? get_page_from_freelist+0x477/0x4a8 [<ffffffff8108ae95>] try_to_free_pages+0x68/0x6a [<ffffffff81084058>] __alloc_pages_nodemask+0x4cf/0x75c [<ffffffff81084361>] __get_free_pages+0x12/0x41 [<ffffffff810229fe>] pte_alloc_one_kernel+0x10/0x12 [<ffffffff81093f66>] __pte_alloc_kernel+0x16/0x7b [<ffffffff810a16ca>] vmap_page_range_noflush+0x17f/0x270 [<ffffffff810a17e8>] map_vm_area+0x2d/0x40 [<ffffffff810a194d>] __vmalloc_node_range+0x152/0x201 [<ffffffffa00348d1>] ? kv_alloc+0x3e/0x40 [spl] [<ffffffff810a1a2c>] __vmalloc_node+0x30/0x32 [<ffffffffa00348d1>] ? kv_alloc+0x3e/0x40 [spl] [<ffffffff810a1bdf>] __vmalloc+0x1b/0x1d [<ffffffffa00348d1>] kv_alloc+0x3e/0x40 [spl] [<ffffffffa0034904>] spl_cache_grow_work+0x31/0x3bd [spl] [<ffffffffa0037a4f>] taskq_thread+0x2be/0x447 [spl] [<ffffffff81048ceb>] ? try_to_wake_up+0x219/0x219 [<ffffffffa0037791>] ? task_expire+0xf6/0xf6 [spl] [<ffffffff8103e5ca>] kthread+0x84/0x8c [<ffffffff81310654>] kernel_thread_helper+0x4/0x10 [<ffffffff8103e546>] ? kthread_freezable_should_stop+0x58/0x58 [<ffffffff81310650>] ? gs_change+0xb/0xb arc_adapt D 0000000000000000 0 1667 2 0x00000000 ffff8804157c3b00 0000000000000046 0000000000000000 ffff8804157c2010 ffff88041da9cc80 0000000000010900 ffff8804157c3fd8 0000000000004000 ffff8804157c3fd8 0000000000010900 ffffffff81613020 ffff88041da9cc80 Call Trace: [<ffffffffa00d7c58>] ? arc_buf_eviction_needed+0x5f/0x69 [zfs] [<ffffffffa00e0579>] ? dmu_buf_rele+0x25/0x2c [zfs] [<ffffffff8130e2a2>] schedule+0x5f/0x61 [<ffffffff8130e4dc>] schedule_preempt_disabled+0x9/0xb [<ffffffff8130cd58>] __mutex_lock_slowpath+0xde/0x123 [<ffffffff81098347>] ? unmap_mapping_range+0x15d/0x16e [<ffffffff8130cbf2>] mutex_lock+0x16/0x27 [<ffffffffa0156e0d>] zfs_zinactive+0x67/0xd7 [zfs] [<ffffffffa014f533>] zfs_inactive+0x112/0x1ac [zfs] [<ffffffff810881f9>] ? truncate_pagecache+0x51/0x59 [<ffffffffa0165ed5>] zpl_evict_inode+0x23/0x27 [zfs] [<ffffffff810cbfb0>] evict+0xa2/0x155 [<ffffffff810cc50d>] dispose_list+0x3d/0x4a [<ffffffff810cc7e3>] prune_icache_sb+0x2c9/0x2d8 [<ffffffff810ba66b>] prune_super+0xe3/0x158 [<ffffffffa014c4e2>] zfs_sb_prune+0x76/0x9b [zfs] [<ffffffffa0165f96>] ? zpl_prune_sbs+0x37/0x37 [zfs] [<ffffffffa0165fb1>] zpl_prune_sb+0x1b/0x1e [zfs] [<ffffffff810b9c8a>] iterate_supers_type+0x43/0xbf [<ffffffffa0165f8f>] zpl_prune_sbs+0x30/0x37 [zfs] [<ffffffffa00dbcc8>] arc_adapt_thread+0x1e0/0x31d [zfs] [<ffffffffa0165f5f>] ? zpl_inode_alloc+0x70/0x70 [zfs] [<ffffffffa00dbae8>] ? arc_adjust+0x105/0x105 [zfs] [<ffffffffa003691f>] ? __thread_create+0x2e3/0x2e3 [spl] [<ffffffffa003698b>] thread_generic_wrapper+0x6c/0x79 [spl] [<ffffffffa003691f>] ? __thread_create+0x2e3/0x2e3 [spl] [<ffffffff8103e5ca>] kthread+0x84/0x8c [<ffffffff81310654>] kernel_thread_helper+0x4/0x10 [<ffffffff8103e546>] ? kthread_freezable_should_stop+0x58/0x58 [<ffffffff81310650>] ? gs_change+0xb/0xb z_rd_int/0 D 0000000000000001 0 5857 2 0x00000000 ffff88040faf9af0 0000000000000046 ffff88040faf99f0 ffff88040faf8010 ffff8804102bcc80 0000000000010900 ffff88040faf9fd8 0000000000004000 ffff88040faf9fd8 0000000000010900 ffff88041d863fc0 ffff8804102bcc80 Call Trace: [<ffffffff8104bad7>] ? enqueue_entity+0x121/0x1fb [<ffffffff810497c7>] ? wakeup_preempt_entity+0x34/0x40 [<ffffffff8104b871>] ? check_preempt_wakeup+0x169/0x1ed [<ffffffff810315f2>] ? lock_timer_base+0x27/0x4d [<ffffffff8103189d>] ? try_to_del_timer_sync+0x7b/0x87 [<ffffffff8130e2a2>] schedule+0x5f/0x61 [<ffffffff8130c97d>] schedule_timeout+0x169/0x18f [<ffffffff810319f6>] ? add_timer_on+0x84/0x84 [<ffffffffa0034404>] spl_kmem_cache_alloc+0x835/0xcc4 [spl] [<ffffffff810441ea>] ? __wake_up_common+0x49/0x7f [<ffffffff8103e9c0>] ? wake_up_bit+0x25/0x25 [<ffffffffa01206d1>] ? spa_config_exit+0x73/0x8f [zfs] [<ffffffffa015efd3>] zio_data_buf_alloc+0x1d/0x1f [zfs] [<ffffffffa00dda3e>] l2arc_read_done+0xfd/0x2d3 [zfs] [<ffffffff8130e135>] ? __schedule+0x772/0x804 [<ffffffffa0127145>] ? vdev_stat_update+0xe2/0x2a1 [zfs] [<ffffffffa0162111>] zio_done+0x906/0xbe1 [zfs] [<ffffffffa015d8ce>] ? zio_wait_for_children+0x4e/0x5a [zfs] [<ffffffffa015ea6e>] zio_execute+0xdf/0x106 [zfs] [<ffffffffa0037a4f>] taskq_thread+0x2be/0x447 [spl] [<ffffffff81048ceb>] ? try_to_wake_up+0x219/0x219 [<ffffffffa0037791>] ? task_expire+0xf6/0xf6 [spl] [<ffffffff8103e5ca>] kthread+0x84/0x8c [<ffffffff81310654>] kernel_thread_helper+0x4/0x10 [<ffffffff8103e546>] ? kthread_freezable_should_stop+0x58/0x58 [<ffffffff81310650>] ? gs_change+0xb/0xb usage.pl D 0000000000000001 0 7005 3629 0x00000004 ffff8800741036a8 0000000000000082 ffff88042fc50900 ffff880074102010 ffff880414083300 0000000000010900 ffff880074103fd8 0000000000004000 ffff880074103fd8 0000000000010900 ffff88041d863fc0 ffff880414083300 Call Trace: [<ffffffff810315f2>] ? lock_timer_base+0x27/0x4d [<ffffffff8103189d>] ? try_to_del_timer_sync+0x7b/0x87 [<ffffffff8130e2a2>] schedule+0x5f/0x61 [<ffffffff8130c97d>] schedule_timeout+0x169/0x18f [<ffffffff810319f6>] ? add_timer_on+0x84/0x84 [<ffffffffa0034404>] spl_kmem_cache_alloc+0x835/0xcc4 [spl] [<ffffffff8103e9c0>] ? wake_up_bit+0x25/0x25 [<ffffffff8103e9c0>] ? wake_up_bit+0x25/0x25 [<ffffffffa003573b>] ? kmem_alloc_debug+0x2ae/0x42f [spl] [<ffffffffa00f68c7>] dnode_create+0x32/0x14e [zfs] [<ffffffffa00f6ebb>] dnode_hold_impl+0x326/0x46e [zfs] [<ffffffffa00f7017>] dnode_hold+0x14/0x16 [zfs] [<ffffffffa00e8a09>] dmu_bonus_hold+0x24/0x28e [zfs] [<ffffffffa01108a0>] sa_buf_hold+0x9/0xb [zfs] [<ffffffffa0158a4e>] zfs_zget+0x5f/0x1a0 [zfs] [<ffffffffa003573b>] ? kmem_alloc_debug+0x2ae/0x42f [spl] [<ffffffffa013dd43>] zfs_dirent_lock+0x433/0x479 [zfs] [<ffffffffa013e106>] zfs_dirlook+0x21b/0x285 [zfs] [<ffffffffa0138eb7>] ? zfs_zaccess+0x2b0/0x346 [zfs] [<ffffffffa0154f5c>] zfs_lookup+0x267/0x2ad [zfs] [<ffffffffa0165358>] zpl_lookup+0x5b/0xad [zfs] [<ffffffff810c0cf3>] __lookup_hash+0xb9/0xdd [<ffffffff810c0f80>] do_lookup+0x269/0x2a4 [<ffffffff810c23dc>] path_lookupat+0xe9/0x5c5 [<ffffffff810c28d6>] do_path_lookup+0x1e/0x54 [<ffffffff810c2b44>] user_path_at_empty+0x50/0x96 [<ffffffffa015295a>] ? zfs_getattr_fast+0x10c/0x11d [zfs] [<ffffffff810bb898>] ? cp_new_stat+0xde/0xea [<ffffffff810c2b96>] user_path_at+0xc/0xe [<ffffffff810bba19>] vfs_fstatat+0x3d/0x68 [<ffffffff810bbb1d>] vfs_stat+0x16/0x18 [<ffffffff810bbb39>] sys_newstat+0x1a/0x38 [<ffffffff8130f5a2>] system_call_fastpath+0x16/0x1b usage.pl D 0000000000000001 0 6524 4127 0x00000004 ffff8803de0196a8 0000000000000086 ffff88042fc50900 ffff8803de018010 ffff880380ca8cc0 0000000000010900 ffff8803de019fd8 0000000000004000 ffff8803de019fd8 0000000000010900 ffff88041d863fc0 ffff880380ca8cc0 Call Trace: [<ffffffff81083efa>] ? __alloc_pages_nodemask+0x371/0x75c [<ffffffff810315f2>] ? lock_timer_base+0x27/0x4d [<ffffffff8103189d>] ? try_to_del_timer_sync+0x7b/0x87 [<ffffffff8130e2a2>] schedule+0x5f/0x61 [<ffffffff8130c97d>] schedule_timeout+0x169/0x18f [<ffffffff810319f6>] ? add_timer_on+0x84/0x84 [<ffffffffa0034404>] spl_kmem_cache_alloc+0x835/0xcc4 [spl] [<ffffffff8103e9c0>] ? wake_up_bit+0x25/0x25 [<ffffffffa003573b>] ? kmem_alloc_debug+0x2ae/0x42f [spl] [<ffffffffa00f68c7>] dnode_create+0x32/0x14e [zfs] [<ffffffffa00f6ebb>] dnode_hold_impl+0x326/0x46e [zfs] [<ffffffffa00f7017>] dnode_hold+0x14/0x16 [zfs] [<ffffffffa00e8a09>] dmu_bonus_hold+0x24/0x28e [zfs] [<ffffffffa01108a0>] sa_buf_hold+0x9/0xb [zfs] [<ffffffffa0158a4e>] zfs_zget+0x5f/0x1a0 [zfs] [<ffffffffa003573b>] ? kmem_alloc_debug+0x2ae/0x42f [spl] [<ffffffffa013dd43>] zfs_dirent_lock+0x433/0x479 [zfs] [<ffffffffa013e106>] zfs_dirlook+0x21b/0x285 [zfs] [<ffffffffa0138eb7>] ? zfs_zaccess+0x2b0/0x346 [zfs] [<ffffffffa0154f5c>] zfs_lookup+0x267/0x2ad [zfs] [<ffffffffa0165358>] zpl_lookup+0x5b/0xad [zfs] [<ffffffff810c0cf3>] __lookup_hash+0xb9/0xdd [<ffffffff810c0f80>] do_lookup+0x269/0x2a4 [<ffffffff810c23dc>] path_lookupat+0xe9/0x5c5 [<ffffffff810c08bb>] ? path_put+0x1d/0x21 [<ffffffff810c28d6>] do_path_lookup+0x1e/0x54 [<ffffffff810c2b44>] user_path_at_empty+0x50/0x96 [<ffffffff810c2b96>] user_path_at+0xc/0xe [<ffffffff810bba19>] vfs_fstatat+0x3d/0x68 [<ffffffff810bbb1d>] vfs_stat+0x16/0x18 [<ffffffff810bbb39>] sys_newstat+0x1a/0x38 [<ffffffff810b631a>] ? filp_close+0x6d/0x78 [<ffffffff810b63a3>] ? sys_close+0x7e/0xb3 [<ffffffff8130f5a2>] system_call_fastpath+0x16/0x1b usage.pl D 0000000000000000 0 7060 4127 0x00000004 ffff88009bfe76a8 0000000000000086 ffff88042fc10900 ffff88009bfe6010 ffff880414085940 0000000000010900 ffff88009bfe7fd8 0000000000004000 ffff88009bfe7fd8 0000000000010900 ffffffff81613020 ffff880414085940 Call Trace: [<ffffffffa0110252>] ? sa_find_idx_tab+0x61/0x1c9 [zfs] [<ffffffff810315f2>] ? lock_timer_base+0x27/0x4d [<ffffffff8103189d>] ? try_to_del_timer_sync+0x7b/0x87 [<ffffffff8130e2a2>] schedule+0x5f/0x61 [<ffffffff8130c97d>] schedule_timeout+0x169/0x18f [<ffffffff810319f6>] ? add_timer_on+0x84/0x84 [<ffffffffa0034404>] spl_kmem_cache_alloc+0x835/0xcc4 [spl] [<ffffffffa016159c>] ? zio_ready+0xb1/0x320 [zfs] [<ffffffff8103e9c0>] ? wake_up_bit+0x25/0x25 [<ffffffffa003573b>] ? kmem_alloc_debug+0x2ae/0x42f [spl] [<ffffffffa00f68c7>] dnode_create+0x32/0x14e [zfs] [<ffffffffa00f6ebb>] dnode_hold_impl+0x326/0x46e [zfs] [<ffffffffa00f7017>] dnode_hold+0x14/0x16 [zfs] [<ffffffffa00e8a09>] dmu_bonus_hold+0x24/0x28e [zfs] [<ffffffffa01108a0>] sa_buf_hold+0x9/0xb [zfs] [<ffffffffa0158a4e>] zfs_zget+0x5f/0x1a0 [zfs] [<ffffffffa003573b>] ? kmem_alloc_debug+0x2ae/0x42f [spl] [<ffffffffa013dd43>] zfs_dirent_lock+0x433/0x479 [zfs] [<ffffffffa013e106>] zfs_dirlook+0x21b/0x285 [zfs] [<ffffffffa0138eb7>] ? zfs_zaccess+0x2b0/0x346 [zfs] [<ffffffffa0154f5c>] zfs_lookup+0x267/0x2ad [zfs] [<ffffffffa0165358>] zpl_lookup+0x5b/0xad [zfs] [<ffffffff810c0cf3>] __lookup_hash+0xb9/0xdd [<ffffffff810c0f80>] do_lookup+0x269/0x2a4 [<ffffffff810c23dc>] path_lookupat+0xe9/0x5c5 [<ffffffff810c08bb>] ? path_put+0x1d/0x21 [<ffffffff810c28d6>] do_path_lookup+0x1e/0x54 [<ffffffff810c2b44>] user_path_at_empty+0x50/0x96 [<ffffffffa015295a>] ? zfs_getattr_fast+0x10c/0x11d [zfs] [<ffffffff810bb898>] ? cp_new_stat+0xde/0xea [<ffffffff810c2b96>] user_path_at+0xc/0xe [<ffffffff810bba19>] vfs_fstatat+0x3d/0x68 [<ffffffff810bbb1d>] vfs_stat+0x16/0x18 [<ffffffff810bbb39>] sys_newstat+0x1a/0x38 [<ffffffff8130f5a2>] system_call_fastpath+0x16/0x1b (more usage.pl stacks) (functionally the same as 'zpool status' but output suitable for a Nagios plugin) zpool_nagios D 0000000000000001 0 7542 7541 0x00000000 ffff8803502b5758 0000000000000082 ffff8803502b5698 ffff8803502b4010 ffff880049d7dfa0 0000000000010900 ffff8803502b5fd8 0000000000004000 ffff8803502b5fd8 0000000000010900 ffff88041d863fc0 ffff880049d7dfa0 Call Trace: [<ffffffff810da395>] ? __getblk+0x20/0x260 [<ffffffff81125b78>] ? ext4_ext_drop_refs+0x26/0x44 [<ffffffff810315f2>] ? lock_timer_base+0x27/0x4d [<ffffffff8103189d>] ? try_to_del_timer_sync+0x7b/0x87 [<ffffffff8130e2a2>] schedule+0x5f/0x61 [<ffffffff8130c97d>] schedule_timeout+0x169/0x18f [<ffffffff810319f6>] ? add_timer_on+0x84/0x84 [<ffffffffa0034404>] spl_kmem_cache_alloc+0x835/0xcc4 [spl] [<ffffffffa003573b>] ? kmem_alloc_debug+0x2ae/0x42f [spl] [<ffffffff8103e9c0>] ? wake_up_bit+0x25/0x25 [<ffffffffa00dfbd2>] dbuf_create+0x3e/0x346 [zfs] [<ffffffffa00df407>] ? dbuf_find+0xae/0xc0 [zfs] [<ffffffffa00e13f7>] __dbuf_hold_impl+0x26f/0x3c4 [zfs] [<ffffffffa00e15c2>] dbuf_hold_impl+0x76/0x9c [zfs] [<ffffffffa00e2e00>] dbuf_hold+0x1b/0x29 [zfs] [<ffffffffa00e7294>] dmu_buf_hold+0x6c/0x182 [zfs] [<ffffffffa0134de6>] zap_lockdir+0x4e/0x5b7 [zfs] [<ffffffffa003573b>] ? kmem_alloc_debug+0x2ae/0x42f [spl] [<ffffffffa01364e0>] zap_cursor_retrieve+0x59/0x25c [zfs] [<ffffffffa00b2365>] ? nv_alloc_sleep_spl+0x23/0x25 [znvpair] [<ffffffffa00ae460>] ? nv_mem_zalloc+0x16/0x2f [znvpair] [<ffffffffa00ae516>] ? nvlist_xalloc+0x5d/0xaa [znvpair] [<ffffffffa011b092>] spa_add_feature_stats+0xf3/0x1ff [zfs] [<ffffffffa00b022b>] ? nvlist_copy_pairs+0x41/0x54 [znvpair] [<ffffffffa00afe0d>] ? nvlist_remove_all+0x52/0xc5 [znvpair] [<ffffffff8105190d>] ? ktime_get_ts+0xa9/0xb6 [<ffffffffa003b479>] ? __gethrtime+0x11/0x1f [spl] [<ffffffffa012625b>] ? vdev_get_stats+0x95/0x107 [zfs] [<ffffffffa011b323>] spa_get_stats+0x185/0x222 [zfs] [<ffffffffa01434e6>] zfs_ioc_pool_stats+0x23/0x59 [zfs] [<ffffffffa01461a6>] zfsdev_ioctl+0x32e/0x3e1 [zfs] [<ffffffff810c5cb6>] do_vfs_ioctl+0x3e0/0x426 [<ffffffff810b6232>] ? fd_install+0x31/0x5e [<ffffffff810ae477>] ? kmem_cache_free+0x15/0x8d [<ffffffff810c5d3e>] sys_ioctl+0x42/0x65 [<ffffffff8130f5a2>] system_call_fastpath+0x16/0x1b ( echo 3 > /proc/sys/vm/drop_caches ) bash D 0000000000000000 0 7924 22507 0x00000000 ffff8802e598fb88 0000000000000082 ffffea000fc67cc0 ffff8802e598e010 ffff880414080000 0000000000010900 ffff8802e598ffd8 0000000000004000 ffff8802e598ffd8 0000000000010900 ffffffff81613020 ffff880414080000 Call Trace: [<ffffffffa00d7c58>] ? arc_buf_eviction_needed+0x5f/0x69 [zfs] [<ffffffffa00e0579>] ? dmu_buf_rele+0x25/0x2c [zfs] [<ffffffff8130e2a2>] schedule+0x5f/0x61 [<ffffffff8130e4dc>] schedule_preempt_disabled+0x9/0xb [<ffffffff8130cd58>] __mutex_lock_slowpath+0xde/0x123 [<ffffffff81098347>] ? unmap_mapping_range+0x15d/0x16e [<ffffffff8130cbf2>] mutex_lock+0x16/0x27 [<ffffffffa0156e0d>] zfs_zinactive+0x67/0xd7 [zfs] [<ffffffffa014f533>] zfs_inactive+0x112/0x1ac [zfs] [<ffffffff810881f9>] ? truncate_pagecache+0x51/0x59 [<ffffffffa0165ed5>] zpl_evict_inode+0x23/0x27 [zfs] [<ffffffff810cbfb0>] evict+0xa2/0x155 [<ffffffff810cc50d>] dispose_list+0x3d/0x4a [<ffffffff810cc7e3>] prune_icache_sb+0x2c9/0x2d8 [<ffffffff810ba66b>] prune_super+0xe3/0x158 [<ffffffff8108a3a8>] shrink_slab+0x122/0x19c [<ffffffff810d6347>] drop_caches_sysctl_handler+0x67/0x76 [<ffffffff8110683e>] proc_sys_call_handler+0x85/0xaf [<ffffffff81106877>] proc_sys_write+0xf/0x11 [<ffffffff810b8375>] vfs_write+0xae/0x153 [<ffffffff810b84d3>] sys_write+0x45/0x6c [<ffffffff8130f5a2>] system_call_fastpath+0x16/0x1b Memory information# echo 3 > /proc/sys/vm/drop & # cat /proc/spl/kstat/zfs/arcstats 5 1 0x01 85 4080 317522309093 518831101128609 name type data hits 4 860963707 misses 4 787286712 demand_data_hits 4 0 demand_data_misses 4 152882 demand_metadata_hits 4 843324252 demand_metadata_misses 4 710135298 prefetch_data_hits 4 0 prefetch_data_misses 4 0 prefetch_metadata_hits 4 17639455 prefetch_metadata_misses 4 76998532 mru_hits 4 19506866 mru_ghost_hits 4 40954322 mfu_hits 4 836351477 mfu_ghost_hits 4 208201271 deleted 4 546428460 recycle_miss 4 655846797 mutex_miss 4 863920 evict_skip 4 488101822477 evict_l2_cached 4 11207863794176 evict_l2_eligible 4 2411402869248 evict_l2_ineligible 4 145852925952 hash_elements 4 7626298 hash_elements_max 4 7626298 hash_collisions 4 77018887 hash_chains 4 262144 hash_chain_max 4 60 p 4 16384 c 4 3000000000 c_min 4 4194304 c_max 4 3000000000 size 4 4041174944 hdr_size 4 58225248 data_size 4 0 meta_size 4 1941048320 other_size 4 688340704 anon_size 4 16384 anon_evict_data 4 0 anon_evict_metadata 4 0 mru_size 4 9734144 mru_evict_data 4 0 mru_evict_metadata 4 229376 mru_ghost_size 4 2821863424 mru_ghost_evict_data 4 822678016 mru_ghost_evict_metadata 4 1999185408 mfu_size 4 1931297792 mfu_evict_data 4 0 mfu_evict_metadata 4 36264960 mfu_ghost_size 4 1261568 mfu_ghost_evict_data 4 0 mfu_ghost_evict_metadata 4 1261568 l2_hits 4 579849476 l2_misses 4 207437209 l2_feeds 4 496537 l2_rw_clash 4 107975 l2_read_bytes 4 2548327023104 l2_write_bytes 4 21307215360 l2_writes_sent 4 170518 l2_writes_done 4 170518 l2_writes_error 4 0 l2_writes_hdr_miss 4 964 l2_evict_lock_retry 4 0 l2_evict_reading 4 0 l2_free_on_write 4 345969 l2_abort_lowmem 4 0 l2_cksum_bad 4 0 l2_io_error 4 0 l2_size 4 122494297600 l2_asize 4 15137260544 l2_hdr_size 4 1653434912 l2_compress_successes 4 10065114 l2_compress_zeros 4 0 l2_compress_failures 4 0 memory_throttle_count 4 0 duplicate_buffers 4 0 duplicate_buffers_size 4 0 duplicate_reads 4 0 memory_direct_count 4 0 memory_indirect_count 4 73 arc_no_grow 4 0 arc_tempreserve 4 0 arc_loaned_bytes 4 0 arc_prune 4 480173069 arc_meta_used 4 4041174944 arc_meta_limit 4 2000000000 arc_meta_max 4 4041175128 # cat /proc/spl/kmem/slab --------------------- cache ------------------------------------------------------- ----- slab ------ ---- object ----- --- emergency --- name flags size alloc slabsize objsize total alloc max total alloc max dlock alloc max spl_vn_cache 0x100020 0 0 0 88 0 0 0 0 0 0 0 0 0 spl_vn_file_cache 0x100020 0 0 0 96 0 0 0 0 0 0 0 0 0 spl_zlib_workspace_cache 0x00140 0 0 8388608 268104 0 0 1 0 0 16 0 0 0 space_seg_cache 0x100000 0 0 0 64 0 0 0 0 0 0 0 0 0 ddt_cache 0x00040 524288 248400 524288 24840 1 1 1 21 10 21 0 0 0 ddt_entry_cache 0x00020 0 0 8192 448 0 0 0 0 0 0 0 0 0 zio_cache 0x00020 1867776 66528 32768 1056 57 3 670 1653 63 19430 0 0 0 zio_link_cache 0x100020 0 0 0 48 0 0 0 0 0 0 0 0 0 zio_vdev_cache 0x00040 33554432 27528480 4194304 131088 8 8 13 240 210 368 0 0 0 zio_buf_512 0x08020 450789376 218350080 32768 512 13757 13757 13757 426467 426465 426467 0 0 0 zio_data_buf_512 0x08020 0 0 32768 512 0 0 238 0 0 7378 0 0 0 zio_buf_1024 0x00040 0 0 32768 1024 0 0 95 0 0 1995 0 0 0 zio_data_buf_1024 0x00040 0 0 32768 1024 0 0 17 0 0 357 0 0 0 zio_buf_1536 0x0c040 131072 95232 65536 1536 2 2 22 62 62 682 1 0 1 zio_data_buf_1536 0x00040 0 0 65536 1536 0 0 14 0 0 434 0 0 0 zio_buf_2048 0x00040 720896 79872 65536 2048 11 11 39 275 39 975 0 0 0 zio_data_buf_2048 0x00040 196608 8192 65536 2048 3 3 23 75 4 575 0 0 0 zio_buf_2560 0x00040 262144 163840 65536 2560 4 4 44 84 64 924 0 0 0 zio_data_buf_2560 0x00040 0 0 65536 2560 0 0 218 0 0 4578 0 0 0 zio_buf_3072 0x00040 0 0 65536 3072 0 0 37 0 0 666 0 0 0 zio_data_buf_3072 0x00040 0 0 65536 3072 0 0 25 0 0 450 0 0 0 zio_buf_3584 0x00040 131072 3584 131072 3584 1 1 43 31 1 1333 0 0 0 zio_data_buf_3584 0x00040 0 0 131072 3584 0 0 8 0 0 248 0 0 0 zio_buf_4096 0x00040 0 0 262144 4096 0 0 382 0 0 11842 0 0 0 zio_data_buf_4096 0x00040 786432 102400 262144 4096 3 3 502 93 25 15562 0 0 0 zio_buf_5120 0x00040 0 0 131072 5120 0 0 19 0 0 399 0 0 0 zio_data_buf_5120 0x00040 0 0 131072 5120 0 0 13 0 0 273 0 0 0 zio_buf_6144 0x00040 0 0 131072 6144 0 0 14 0 0 252 0 0 0 zio_data_buf_6144 0x00040 0 0 131072 6144 0 0 16 0 0 288 0 0 0 zio_buf_7168 0x00040 0 0 262144 7168 0 0 9 0 0 279 0 0 0 zio_data_buf_7168 0x00040 0 0 262144 7168 0 0 9 0 0 279 0 0 0 zio_buf_8192 0x00040 262144 8192 262144 8192 1 1 18 21 1 378 0 0 0 zio_data_buf_8192 0x00040 0 0 262144 8192 0 0 214 0 0 4494 0 0 0 zio_buf_10240 0x00040 262144 10240 262144 10240 1 1 18 21 1 378 0 0 0 zio_data_buf_10240 0x00040 0 0 262144 10240 0 0 19 0 0 399 0 0 0 zio_buf_12288 0x00040 0 0 524288 12288 0 0 10 0 0 310 0 0 0 zio_data_buf_12288 0x00040 0 0 524288 12288 0 0 18 0 0 558 0 0 0 zio_buf_14336 0x00040 0 0 524288 14336 0 0 9 0 0 279 0 0 0 zio_data_buf_14336 0x00040 0 0 524288 14336 0 0 15 0 0 465 0 0 0 zio_buf_16384 0x0c040 2484600832 1940668416 524288 16384 4739 4739 4739 118475 118449 118475 1 0 0 zio_data_buf_16384 0x00040 6291456 999424 524288 16384 12 12 179 300 61 4475 0 0 0 zio_buf_20480 0x00040 0 0 524288 20480 0 0 17 0 0 357 0 0 0 zio_data_buf_20480 0x00040 0 0 524288 20480 0 0 18 0 0 378 0 0 0 zio_buf_24576 0x00040 0 0 524288 24576 0 0 16 0 0 288 0 0 0 zio_data_buf_24576 0x00040 0 0 524288 24576 0 0 16 0 0 288 0 0 0 zio_buf_28672 0x00040 0 0 1048576 28672 0 0 11 0 0 341 0 0 0 zio_data_buf_28672 0x00040 0 0 1048576 28672 0 0 16 0 0 496 0 0 0 zio_buf_32768 0x00040 0 0 1048576 32768 0 0 10 0 0 280 0 0 0 zio_data_buf_32768 0x00040 0 0 1048576 32768 0 0 8 0 0 213 0 0 0 zio_buf_36864 0x00040 0 0 1048576 36864 0 0 7 0 0 175 0 0 0 zio_data_buf_36864 0x00040 0 0 1048576 36864 0 0 11 0 0 275 0 0 0 zio_buf_40960 0x00040 0 0 1048576 40960 0 0 8 0 0 184 0 0 0 zio_data_buf_40960 0x00040 0 0 1048576 40960 0 0 12 0 0 276 0 0 0 zio_buf_45056 0x00040 1048576 45056 1048576 45056 1 1 8 21 1 168 0 0 0 zio_data_buf_45056 0x00040 0 0 1048576 45056 0 0 11 0 0 231 0 0 0 zio_buf_49152 0x00040 1048576 49152 1048576 49152 1 1 26 19 1 494 0 0 0 zio_data_buf_49152 0x00040 0 0 1048576 49152 0 0 8 0 0 152 0 0 0 zio_buf_53248 0x00040 0 0 1048576 53248 0 0 15 0 0 270 0 0 0 zio_data_buf_53248 0x00040 0 0 1048576 53248 0 0 9 0 0 162 0 0 0 zio_buf_57344 0x00040 0 0 1048576 57344 0 0 8 0 0 136 0 0 0 zio_data_buf_57344 0x00040 0 0 1048576 57344 0 0 6 0 0 102 0 0 0 zio_buf_61440 0x00040 0 0 2097152 61440 0 0 4 0 0 124 0 0 0 zio_data_buf_61440 0x00040 0 0 2097152 61440 0 0 5 0 0 155 0 0 0 zio_buf_65536 0x00040 2097152 65536 2097152 65536 1 1 4 30 1 120 0 0 0 zio_data_buf_65536 0x00040 0 0 2097152 65536 0 0 4 0 0 120 0 0 0 zio_buf_69632 0x00040 0 0 2097152 69632 0 0 4 0 0 112 0 0 0 zio_data_buf_69632 0x00040 0 0 2097152 69632 0 0 5 0 0 140 0 0 0 zio_buf_73728 0x00040 0 0 2097152 73728 0 0 27 0 0 702 0 0 0 zio_data_buf_73728 0x00040 0 0 2097152 73728 0 0 4 0 0 104 0 0 0 zio_buf_77824 0x00040 0 0 2097152 77824 0 0 26 0 0 650 0 0 0 zio_data_buf_77824 0x00040 0 0 2097152 77824 0 0 4 0 0 100 0 0 0 zio_buf_81920 0x00040 0 0 2097152 81920 0 0 19 0 0 456 0 0 0 zio_data_buf_81920 0x00040 0 0 2097152 81920 0 0 7 0 0 163 0 0 0 zio_buf_86016 0x00040 0 0 2097152 86016 0 0 4 0 0 92 0 0 0 zio_data_buf_86016 0x00040 0 0 2097152 86016 0 0 5 0 0 115 0 0 0 zio_buf_90112 0x00040 0 0 2097152 90112 0 0 4 0 0 88 0 0 0 zio_data_buf_90112 0x00040 0 0 2097152 90112 0 0 5 0 0 110 0 0 0 zio_buf_94208 0x00040 0 0 2097152 94208 0 0 6 0 0 126 0 0 0 zio_data_buf_94208 0x00040 0 0 2097152 94208 0 0 11 0 0 231 0 0 0 zio_buf_98304 0x00040 0 0 2097152 98304 0 0 27 0 0 540 0 0 0 zio_data_buf_98304 0x00040 0 0 2097152 98304 0 0 4 0 0 80 0 0 0 zio_buf_102400 0x00040 0 0 2097152 102400 0 0 20 0 0 380 0 0 0 zio_data_buf_102400 0x00040 0 0 2097152 102400 0 0 3 0 0 57 0 0 0 zio_buf_106496 0x00040 0 0 2097152 106496 0 0 50 0 0 900 0 0 0 zio_data_buf_106496 0x00040 0 0 2097152 106496 0 0 8 0 0 144 0 0 0 zio_buf_110592 0x00040 0 0 2097152 110592 0 0 25 0 0 450 0 0 0 zio_data_buf_110592 0x00040 0 0 2097152 110592 0 0 4 0 0 72 0 0 0 zio_buf_114688 0x00040 0 0 2097152 114688 0 0 5 0 0 85 0 0 0 zio_data_buf_114688 0x00040 0 0 2097152 114688 0 0 8 0 0 136 0 0 0 zio_buf_118784 0x00040 0 0 2097152 118784 0 0 27 0 0 459 0 0 0 zio_data_buf_118784 0x00040 0 0 2097152 118784 0 0 7 0 0 119 0 0 0 zio_buf_122880 0x00040 2097152 122880 2097152 122880 1 1 22 16 1 352 0 0 0 zio_data_buf_122880 0x00040 0 0 2097152 122880 0 0 4 0 0 54 0 0 0 zio_buf_126976 0x00040 0 0 4194304 126976 0 0 23 0 0 713 0 0 0 zio_data_buf_126976 0x00040 0 0 4194304 126976 0 0 2 0 0 62 0 0 0 zio_buf_131072 0x00040 0 0 4194304 131072 0 0 17 0 0 504 0 0 0 zio_data_buf_131072 0x00040 54525952 6553600 4194304 131072 13 13 105 403 50 3248 0 0 0 lz4_cache 0x0c040 0 0 524288 16384 0 0 2 0 0 62 1 0 1 sa_cache 0x100000 0 0 0 80 0 0 0 0 0 0 0 0 0 spill_cache 0x00040 0 0 4194304 131072 0 0 0 0 0 0 0 0 0 dnode_t 0x08020 437272576 399694464 16384 936 26689 26689 26689 427024 427024 427024 0 0 0 dmu_buf_impl_t 0x08020 178094080 152180000 8192 280 21740 21740 21740 543500 543500 543500 0 0 0 arc_buf_hdr_t 0x100000 0 0 0 184 0 0 0 0 0 0 0 0 0 arc_buf_t 0x100000 0 0 0 72 0 0 0 0 0 0 0 0 0 l2arc_buf_hdr_t 0x100000 0 0 0 40 0 0 0 0 0 0 0 0 0 zil_lwb_cache 0x100000 0 0 0 200 0 0 0 0 0 0 0 0 0 zfs_znode_cache 0x00020 411500544 382769112 32768 984 12558 12558 12562 389298 388993 389422 0 0 0 # egrep '(missing entries from above)' /proc/slabinfo slabinfo - version: 2.1 # name : tunables : slabdata zil_lwb_cache 0 0 200 20 1 : tunables 0 0 0 : slabdata 0 0 0 l2arc_buf_hdr_t 7496898 7496898 40 102 1 : tunables 0 0 0 : slabdata 73499 73499 0 arc_buf_t 118776 118776 72 56 1 : tunables 0 0 0 : slabdata 2121 2121 0 arc_buf_hdr_t 7626322 7626322 184 22 1 : tunables 0 0 0 : slabdata 346651 346651 0 sa_cache 389325 389385 80 51 1 : tunables 0 0 0 : slabdata 7635 7635 0 space_seg_cache 954019 1653696 64 64 1 : tunables 0 0 0 : slabdata 25839 25839 0 |
|
This is all the stack traces (all processes that would fit in my dmesg buffer). |
|
I updated. Not helping. Here's a big info dump. Job description: 7 filesystems, 6 threads each doing a 'du' style crawl of the filesystem. # uname -a
Linux whoopass1 3.4.73-zfs #1 SMP Tue Dec 10 09:57:47 EST 2013 x86_64 x86_64 x86_64 GNU/Linux
(while job in progress)
# arcstat.py 2
time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c
14:20:10 858 23 2 23 2 0 100 23 2 5.4G 4.7G
14:20:12 26K 4.6K 17 3.9K 15 676 84 4.6K 17 5.5G 4.7G
14:20:14 11K 3.0K 25 1.9K 18 1.1K 96 3.0K 25 5.6G 4.7G
14:20:16 5.2K 568 11 502 9 66 99 568 11 5.6G 4.7G
14:20:18 14K 2.4K 16 1.8K 12 611 94 2.4K 16 5.6G 4.7G
14:20:20 15K 4.0K 25 2.9K 19 1.2K 95 4.0K 25 5.8G 4.7G
14:20:22 3.1K 477 15 450 14 26 81 477 15 5.8G 4.7G
14:20:24 10K 2.6K 23 1.9K 18 731 97 2.6K 23 5.8G 4.7G
14:20:26 13K 2.0K 15 1.7K 13 343 80 2.0K 15 5.9G 4.7G
14:20:29 14K 2.8K 19 2.0K 14 794 91 2.8K 19 6.0G 4.7G
14:20:31 12K 2.4K 19 1.8K 15 565 90 2.4K 19 6.0G 4.7G
14:20:33 6.8K 2.0K 28 1.2K 20 762 90 2.0K 28 6.1G 4.7G
14:20:35 9.4K 2.0K 21 1.3K 14 715 99 2.0K 21 6.1G 4.7G
14:20:37 8.7K 1.2K 14 1.1K 12 199 99 1.2K 14 6.2G 4.7G
14:20:39 16K 3.4K 21 2.4K 15 1.0K 98 3.4K 21 6.2G 4.7G
14:20:41 7.4K 1.4K 18 858 12 502 100 1.4K 18 6.3G 4.7G
14:20:43 15K 3.2K 20 2.3K 16 827 98 3.2K 20 6.4G 4.7G
14:20:45 9.2K 1.3K 14 1.2K 13 112 86 1.3K 14 6.4G 4.7G
14:20:47 14K 2.8K 19 2.0K 14 786 97 2.8K 19 6.5G 4.7G
14:20:49 8.2K 1.9K 23 1.4K 18 533 98 1.9K 23 6.5G 4.7G
14:20:51 18K 3.8K 20 2.6K 15 1.1K 97 3.8K 20 6.6G 4.7G
14:20:53 3.5K 731 20 356 11 375 98 731 20 6.7G 4.7G
14:20:55 14K 3.2K 22 2.1K 16 1.1K 87 3.2K 22 6.7G 4.7G
14:20:57 9.0K 1.8K 19 1.1K 13 685 99 1.8K 19 6.8G 4.7G
14:20:59 10K 1.5K 15 848 9 668 99 1.5K 15 6.8G 4.7G
14:21:01 10K 2.1K 19 1.3K 13 765 99 2.1K 19 6.9G 4.7G
14:21:03 5.3K 1.3K 25 762 16 581 98 1.3K 25 6.9G 4.7G
14:21:05 5.4K 1.4K 26 644 14 779 97 1.4K 26 7.0G 4.7G
14:21:07 10.0K 1.4K 14 1.1K 11 324 91 1.4K 14 7.0G 4.7G
14:21:09 7.3K 1.8K 24 1.1K 16 648 95 1.8K 24 7.0G 4.7G
14:21:11 7.3K 1.3K 17 861 12 438 92 1.3K 17 7.1G 4.7G
14:21:13 10K 1.8K 17 1.4K 14 451 85 1.8K 17 7.1G 4.7G
14:21:15 17K 3.2K 18 2.4K 14 851 96 3.2K 18 7.2G 4.7G
14:21:17 9.7K 1.5K 15 1.1K 11 359 93 1.5K 15 7.3G 4.7G
# cat /proc/spl/kstat/zfs/arcstats
5 1 0x01 85 4080 522427437894326 622769379315579
name type data
hits 4 390348058
misses 4 140336231
demand_data_hits 4 60764317
demand_data_misses 4 27428
demand_metadata_hits 4 313916888
demand_metadata_misses 4 41474559
prefetch_data_hits 4 6
prefetch_data_misses 4 60773326
prefetch_metadata_hits 4 15666847
prefetch_metadata_misses 4 38060918
mru_hits 4 78364976
mru_ghost_hits 4 5375816
mfu_hits 4 302452995
mfu_ghost_hits 4 3547460
deleted 4 132486432
recycle_miss 4 26063509
mutex_miss 4 152598
evict_skip 4 3457639252
evict_l2_cached 4 1034640056832
evict_l2_eligible 4 88532775936
evict_l2_ineligible 4 2039419473920
hash_elements 4 7623821
hash_elements_max 4 7848366
hash_collisions 4 77227047
hash_chains 4 262144
hash_chain_max 4 59
p 4 0
c 4 5000000000
c_min 4 4194304
c_max 4 5000000000
size 4 6487302920
hdr_size 4 100819624
data_size 4 0
meta_size 4 3183800320
other_size 4 1588724568
anon_size 4 16384
anon_evict_data 4 0
anon_evict_metadata 4 0
mru_size 4 10532864
mru_evict_data 4 0
mru_evict_metadata 4 0
mru_ghost_size 4 2494231552
mru_ghost_evict_data 4 95232
mru_ghost_evict_metadata 4 2494136320
mfu_size 4 3173251072
mfu_evict_data 4 0
mfu_evict_metadata 4 1081344
mfu_ghost_size 4 1512239104
mfu_ghost_evict_data 4 128000
mfu_ghost_evict_metadata 4 1512111104
l2_hits 4 32669935
l2_misses 4 107666226
l2_feeds 4 97636
l2_rw_clash 4 4611
l2_read_bytes 4 99249397760
l2_write_bytes 4 16059228160
l2_writes_sent 4 79199
l2_writes_done 4 79199
l2_writes_error 4 0
l2_writes_hdr_miss 4 980
l2_evict_lock_retry 4 0
l2_evict_reading 4 0
l2_free_on_write 4 176633
l2_abort_lowmem 4 36
l2_cksum_bad 4 0
l2_io_error 4 0
l2_size 4 119552991744
l2_asize 4 14347626496
l2_hdr_size 4 1613958408
l2_compress_successes 4 7985904
l2_compress_zeros 4 0
l2_compress_failures 4 0
memory_throttle_count 4 0
duplicate_buffers 4 0
duplicate_buffers_size 4 0
duplicate_reads 4 1
memory_direct_count 4 4856
memory_indirect_count 4 6
arc_no_grow 4 0
arc_tempreserve 4 0
arc_loaned_bytes 4 0
arc_prune 4 1980175
arc_meta_used 4 6487302920
arc_meta_limit 4 4000000000
arc_meta_max 4 9841053168
Perf flame graph of arc_adapt thread which was pegged in the 'D' state during this time: ~/spl# git log --pretty=oneline ~/zfs# git log --pretty=oneline |
|
Quick update: @ryao suggested enabling CONFIG_PREEMPT_VULUNTARY and it doesn't fix it, though it might mitigate it. New flame graph with the updated kernel: http://199.166.6.44/pub/zfs-issue1932-graph3.svg |
|
Bumping to 0.6.4. I think we're reasonably sure this is being caused by inodes pinning dnodes. However, we're not going to have a fix ready for 0.6.3. |
behlendorf
added
Bug - Major
Component: Memory Management
|
I believe this was fixed by 2cbb06b which means this can be closed. Yes? |
|
Yes, you're right. Closing. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This is just mostly throwing fuel on the fire that is the ARC meta limit growing without bounds problem.
Kernel thread
arc_adaptappears to be single-handedly responsible for pruning the metadata buffer, but it's very possible for it to get stuck doing mutex waiting. The following stack trace came from an otherwise fully working system while it overran itszfs_arc_meta_limit.During this time the arcstat
arc_prunewould stop counting. I'm not sure if it's one lock being held for a long time or many locksarc_pruneis slowly trudging through. On my system I findarc_adaptmaking very slow progress and not completely frozen up - maybe one count ofarc_pruneper 5 or 10 seconds.By executing
killall -STOPon the jobs doing I/Oarc_adaptwould unwedge and be able to prune the cache fairly quickly. Collecting stack traces multiple times between pausing/restarting the disk jobs gives the same backtraces.ryao suggested I apply pull request #1696 as a possible solution - early impressions are good.
Other issues (eg: #1224) have similar stack traces. Once the metadata buffer fills too much the system may reach a state of real deadlock. ZFS is the first to go but general memory allcoations across all of Linux may seize as well.
The text was updated successfully, but these errors were encountered: