BUG: soft lockup - CPU#13 stuck for 22s! [kswapd1:149] #415
Comments
|
A screenshot showing that there is a lot of free ram being used by the kernel caching that should be available to zfsonlinux (so it should not be short of ram as far as I can tell). |
|
If I can provide any more information let me know. This is routinely locking my server and making it unresponsive for about 30 minutes each day. I know there are a number of fixes going in related to memory in 0.6.4 so I would be curious if any address this issue. I haven't built ZFS head from source as I'm not very experienced in that manner but if you think there is reason to believe this issue would be fixed on head let me know what changed and how to set it up and I can try it. |
|
@peter64 It looks like your particular workload is suffering from contention on the virtual address space lock. This was significantly improved in earlier releases but additional improvements are being worked on. It you're game it would be helpful if you could try the currently master source. There are directions for installing it here but before you do so make sure you've removed the Ubuntu PPA. There are additional patches which might help but they're still being finalized. |
|
Hey @behlendorf thanks so much for your reply. I'm a little apprehensive to attempt to run master as I don't have an offline backup system in place currently. I'm investigating tape drives but it may be two or three months before I have one up and running and will be comfortable running master. I'll update here once I am running master to let you know if there is any improvement. I am still experiencing daily lockups and weekly total server crashes. But I'm reducing my arc size to try and mitigate them whenever possible. |
|
Closing, no news is good news. This issue is believed to be resolved with the latest code. |
|
Sorry I forgot to update this. As you stated this appears to be resolved On Wed, Apr 27, 2016 at 7:18 AM, Brian Behlendorf [email protected]
|
The tickets I found which are already closed that most closely resemble my issue is.
openzfs/zfs#1579 and
My system config is ubuntu 12.04, 256 GB ram and 16 cores.
apt-show-versions ubuntu-zfs
ubuntu-zfs/precise uptodate 8~precise
cat /var/log/dmesg | grep -E 'SPL:|ZFS:'
[ 11.627621] SPL: Loaded module v0.6.3-1
preciseprecise, ZFS pool version 5000, ZFS filesystem version 5[ 11.762840] ZFS: Loaded module v0.6.3-2
[ 12.242842] SPL: using hostid 0x007f0101
My error is basically identical to the one in the ticket which said it should be resolved in version 0.6.2.
It even occured during a send/receive operation exactly as the original ticket. I also have my arc cache size set to 60 GB. I haven't tried reducing it to be smaller yet to see if it remedies the issue.
I am also using a swappiness sysctl value of 0 just like the original ticket.
I've tried to provide as many logs as possible I apologize if its too verbose.
I'm not entirely sure what the underlying cause is but it does appear that the server may be tight on available ram when the error occurs. If this is an expected result of being tight on ram let me know and I will mark the ticket closed. I'm just trying to figure out what if any special steps are needed to prevent the server from crashing from this error.
Nov 21 10:15:12 tuft kernel: [1191999.874851] BUG: soft lockup - CPU#13 stuck for 22s! [kswapd1:149]
Nov 21 10:15:12 tuft kernel: [1191999.882104] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 10:15:12 tuft kernel: [1191999.882149] CPU 13
Nov 21 10:15:12 tuft kernel: [1191999.882151] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 10:15:12 tuft kernel: [1191999.882182]
Nov 21 10:15:12 tuft kernel: [1191999.882185] Pid: 149, comm: kswapd1 Tainted: P O 3.5.0-54-generic #81
precise1-Ubuntu Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+precise1-Ubuntu Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+Nov 21 10:15:12 tuft kernel: [1191999.882189] RIP: 0010:[] [] remove_vm_area+0x86/0xa0
Nov 21 10:15:12 tuft kernel: [1191999.882201] RSP: 0018:ffff881fd2de3ab0 EFLAGS: 00000283
Nov 21 10:15:12 tuft kernel: [1191999.882203] RAX: ffff881f49c43dc0 RBX: 0000000000000297 RCX: ffff881fd2de3ea0
Nov 21 10:15:12 tuft kernel: [1191999.882205] RDX: ffff881f49c43a80 RSI: 0000000000000001 RDI: ffffffff81c41c20
Nov 21 10:15:12 tuft kernel: [1191999.882207] RBP: ffff881fd2de3ac0 R08: ffff8825390dcf40 R09: 000000018040003f
Nov 21 10:15:12 tuft kernel: [1191999.882208] R10: 0000000000000001 R11: 0000000000000002 R12: ffffffffffffff10
Nov 21 10:15:12 tuft kernel: [1191999.882210] R13: 0000000100000000 R14: 0000000000000000 R15: 000000007fffcd00
Nov 21 10:15:12 tuft kernel: [1191999.882212] FS: 0000000000000000(0000) GS:ffff88407fca0000(0000) knlGS:0000000000000000
Nov 21 10:15:12 tuft kernel: [1191999.882214] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov 21 10:15:12 tuft kernel: [1191999.882215] CR2: 00007f80cc72d0b0 CR3: 0000000001c0b000 CR4: 00000000000407e0
Nov 21 10:15:12 tuft kernel: [1191999.882217] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 21 10:15:12 tuft kernel: [1191999.882219] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Nov 21 10:15:12 tuft kernel: [1191999.882221] Process kswapd1 (pid: 149, threadinfo ffff881fd2de2000, task ffff881fd2dd2e20)
Nov 21 10:15:12 tuft kernel: [1191999.882222] Stack:
Nov 21 10:15:12 tuft kernel: [1191999.882224] ffffc9151d7c4000 ffff883fc8a56848 ffff881fd2de3af0 ffffffff8115f22d
Nov 21 10:15:12 tuft kernel: [1191999.882230] ffff8802b38c0240 ffffc9152c335000 ffff883fc8a56848 ffff883fc8a56000
Nov 21 10:15:12 tuft kernel: [1191999.882235] ffff881fd2de3b00 ffffffff8115f2eb ffff881fd2de3b10 ffffffff8115f41a
Nov 21 10:15:12 tuft kernel: [1191999.882239] Call Trace:
Nov 21 10:15:12 tuft kernel: [1191999.882245] [] __vunmap.part.16+0x1d/0xc0
Nov 21 10:15:12 tuft kernel: [1191999.882248] [] __vunmap+0x1b/0x40
Nov 21 10:15:12 tuft kernel: [1191999.882252] [] vfree+0x2a/0x30
Nov 21 10:15:12 tuft kernel: [1191999.882263] [] kv_free.isra.6+0x68/0x70 [spl]
Nov 21 10:15:12 tuft kernel: [1191999.882270] [] spl_slab_reclaim+0x335/0x440 [spl]
Nov 21 10:15:12 tuft kernel: [1191999.882278] [] spl_kmem_cache_reap_now+0x149/0x340 [spl]
Nov 21 10:15:12 tuft kernel: [1191999.882302] [] arc_kmem_reap_now+0x60/0xb0 [zfs]
Nov 21 10:15:12 tuft kernel: [1191999.882317] [] arc_shrinker_func+0xc7/0x120 [zfs]
Nov 21 10:15:12 tuft kernel: [1191999.882323] [] shrink_slab+0x154/0x300
Nov 21 10:15:12 tuft kernel: [1191999.882328] [] ? mem_cgroup_iter+0xe8/0x200
Nov 21 10:15:12 tuft kernel: [1191999.882333] [] balance_pgdat+0x5a4/0x720
Nov 21 10:15:12 tuft kernel: [1191999.882337] [] kswapd+0x123/0x240
Nov 21 10:15:12 tuft kernel: [1191999.882341] [] ? balance_pgdat+0x720/0x720
Nov 21 10:15:12 tuft kernel: [1191999.882347] [] kthread+0x93/0xa0
Nov 21 10:15:12 tuft kernel: [1191999.882353] [] kernel_thread_helper+0x4/0x10
Nov 21 10:15:12 tuft kernel: [1191999.882358] [] ? flush_kthread_worker+0xb0/0xb0
Nov 21 10:15:12 tuft kernel: [1191999.882361] [] ? gs_change+0x13/0x13
Nov 21 10:15:12 tuft kernel: [1191999.882362] Code: 20 1c c4 81 e8 8c 0f 54 00 48 8b 05 cd 83 d9 00 48 c7 c2 f8 74 ef 81 48 39 c3 74 14 66 0f 1f 84 00 00 00 00 00 48 89 c2 48 8b 00 <48> 39 c3 75 f5 48 8b 00 48 89 02 f0 81 05 c4 2a ae 00 00 00 10
Nov 21 11:00:02 tuft kernel: [1194684.583433] EXT2-fs (dm-1): warning: mounting unchecked fs, running e2fsck is recommended
Nov 21 11:02:52 tuft kernel: [1194854.275577] BUG: soft lockup - CPU#12 stuck for 22s! [kswapd1:149]
Nov 21 11:02:52 tuft kernel: [1194854.282929] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:02:52 tuft kernel: [1194854.282977] CPU 12
Nov 21 11:02:52 tuft kernel: [1194854.282978] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:02:52 tuft kernel: [1194854.283011]
Nov 21 11:02:52 tuft kernel: [1194854.283014] Pid: 149, comm: kswapd1 Tainted: P O 3.5.0-54-generic #81
Nov 21 11:02:52 tuft kernel: [1194854.283018] RIP: 0010:[] [] remove_vm_area+0x86/0xa0
Nov 21 11:02:52 tuft kernel: [1194854.283029] RSP: 0018:ffff881fd2de3ab0 EFLAGS: 00000206
Nov 21 11:02:52 tuft kernel: [1194854.283032] RAX: ffff8824de81a440 RBX: ffff8828674e7540 RCX: ffff881fd2de3ea0
Nov 21 11:02:52 tuft kernel: [1194854.283033] RDX: ffff8818e30efe00 RSI: 0000000000000001 RDI: ffffffff81c41c20
Nov 21 11:02:52 tuft kernel: [1194854.283035] RBP: ffff881fd2de3ac0 R08: ffff8828674e7540 R09: 000000010040003b
Nov 21 11:02:52 tuft kernel: [1194854.283037] R10: 00000000674e7e01 R11: 0000000000000006 R12: 000000007cdb1401
Nov 21 11:02:52 tuft kernel: [1194854.283038] R13: 0000000100000000 R14: 0000000000000000 R15: 000000007fffbd00
Nov 21 11:02:52 tuft kernel: [1194854.283041] FS: 0000000000000000(0000) GS:ffff88407fc80000(0000) knlGS:0000000000000000
Nov 21 11:02:52 tuft kernel: [1194854.283043] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov 21 11:02:52 tuft kernel: [1194854.283044] CR2: 00007fd82674a024 CR3: 0000000001c0b000 CR4: 00000000000407e0
Nov 21 11:02:52 tuft kernel: [1194854.283046] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 21 11:02:52 tuft kernel: [1194854.283048] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Nov 21 11:02:52 tuft kernel: [1194854.283050] Process kswapd1 (pid: 149, threadinfo ffff881fd2de2000, task ffff881fd2dd2e20)
Nov 21 11:02:52 tuft kernel: [1194854.283051] Stack:
Nov 21 11:02:52 tuft kernel: [1194854.283053] ffffc90d9aaa3000 ffff883fc8a56848 ffff881fd2de3af0 ffffffff8115f22d
Nov 21 11:02:52 tuft kernel: [1194854.283059] 0000000000000032 ffffc904a2e9b000 ffff883fc8a56848 ffff883fc8a56000
Nov 21 11:02:52 tuft kernel: [1194854.283064] ffff881fd2de3b00 ffffffff8115f2eb ffff881fd2de3b10 ffffffff8115f41a
Nov 21 11:02:52 tuft kernel: [1194854.283068] Call Trace:
Nov 21 11:02:52 tuft kernel: [1194854.283075] [] __vunmap.part.16+0x1d/0xc0
Nov 21 11:02:52 tuft kernel: [1194854.283078] [] __vunmap+0x1b/0x40
Nov 21 11:02:52 tuft kernel: [1194854.283082] [] vfree+0x2a/0x30
Nov 21 11:02:52 tuft kernel: [1194854.283093] [] kv_free.isra.6+0x68/0x70 [spl]
Nov 21 11:02:52 tuft kernel: [1194854.283101] [] spl_slab_reclaim+0x335/0x440 [spl]
Nov 21 11:02:52 tuft kernel: [1194854.283108] [] spl_kmem_cache_reap_now+0x149/0x340 [spl]
Nov 21 11:02:52 tuft kernel: [1194854.283132] [] arc_kmem_reap_now+0x60/0xb0 [zfs]
Nov 21 11:02:52 tuft kernel: [1194854.283147] [] arc_shrinker_func+0xc7/0x120 [zfs]
Nov 21 11:02:52 tuft kernel: [1194854.283152] [] shrink_slab+0x154/0x300
Nov 21 11:02:52 tuft kernel: [1194854.283157] [] ? mem_cgroup_iter+0xe8/0x200
Nov 21 11:02:52 tuft kernel: [1194854.283162] [] balance_pgdat+0x5a4/0x720
Nov 21 11:02:52 tuft kernel: [1194854.283166] [] kswapd+0x123/0x240
Nov 21 11:02:52 tuft kernel: [1194854.283170] [] ? balance_pgdat+0x720/0x720
Nov 21 11:02:52 tuft kernel: [1194854.283176] [] kthread+0x93/0xa0
Nov 21 11:02:52 tuft kernel: [1194854.283181] [] kernel_thread_helper+0x4/0x10
Nov 21 11:02:52 tuft kernel: [1194854.283186] [] ? flush_kthread_worker+0xb0/0xb0
Nov 21 11:02:52 tuft kernel: [1194854.283189] [] ? gs_change+0x13/0x13
Nov 21 11:02:52 tuft kernel: [1194854.283190] Code: 20 1c c4 81 e8 8c 0f 54 00 48 8b 05 cd 83 d9 00 48 c7 c2 f8 74 ef 81 48 39 c3 74 14 66 0f 1f 84 00 00 00 00 00 48 89 c2 48 8b 00 <48> 39 c3 75 f5 48 8b 00 48 89 02 f0 81 05 c4 2a ae 00 00 00 10
Nov 21 11:03:20 tuft kernel: [1194882.220875] BUG: soft lockup - CPU#12 stuck for 22s! [kswapd1:149]
Nov 21 11:03:20 tuft kernel: [1194882.227983] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:03:20 tuft kernel: [1194882.228027] CPU 12
Nov 21 11:03:20 tuft kernel: [1194882.228028] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:03:20 tuft kernel: [1194882.228059]
Nov 21 11:03:20 tuft kernel: [1194882.228062] Pid: 149, comm: kswapd1 Tainted: P O 3.5.0-54-generic #81
precise1-Ubuntu Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+precise1-Ubuntu Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+Nov 21 11:03:20 tuft kernel: [1194882.228066] RIP: 0010:[] [] remove_vm_area+0x86/0xa0
Nov 21 11:03:20 tuft kernel: [1194882.228077] RSP: 0018:ffff881fd2de3ab0 EFLAGS: 00000206
Nov 21 11:03:20 tuft kernel: [1194882.228079] RAX: ffff88087dc08600 RBX: ffff88286d34fbc0 RCX: ffff881fd2de3ea0
Nov 21 11:03:20 tuft kernel: [1194882.228081] RDX: ffff88087dc08800 RSI: 0000000000000001 RDI: ffffffff81c41c20
Nov 21 11:03:20 tuft kernel: [1194882.228083] RBP: ffff881fd2de3ac0 R08: ffff88286d34fbc0 R09: 000000010040003a
Nov 21 11:03:20 tuft kernel: [1194882.228085] R10: 000000006d34fb01 R11: 0000000000000006 R12: 000000007adc1301
Nov 21 11:03:20 tuft kernel: [1194882.228086] R13: 0000000100000000 R14: 0000000000000000 R15: 000000007fffbd00
Nov 21 11:03:20 tuft kernel: [1194882.228089] FS: 0000000000000000(0000) GS:ffff88407fc80000(0000) knlGS:0000000000000000
Nov 21 11:03:20 tuft kernel: [1194882.228090] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov 21 11:03:20 tuft kernel: [1194882.228092] CR2: 00007fd82674a024 CR3: 0000000001c0b000 CR4: 00000000000407e0
Nov 21 11:03:20 tuft kernel: [1194882.228094] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 21 11:03:20 tuft kernel: [1194882.228095] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Nov 21 11:03:20 tuft kernel: [1194882.228097] Process kswapd1 (pid: 149, threadinfo ffff881fd2de2000, task ffff881fd2dd2e20)
Nov 21 11:03:20 tuft kernel: [1194882.228099] Stack:
Nov 21 11:03:20 tuft kernel: [1194882.228101] ffffc90da5f24000 ffff883fc8a56848 ffff881fd2de3af0 ffffffff8115f22d
Nov 21 11:03:20 tuft kernel: [1194882.228107] 0000000000000032 ffffc90d92f20000 ffff883fc8a56848 ffff883fc8a56000
Nov 21 11:03:20 tuft kernel: [1194882.228111] ffff881fd2de3b00 ffffffff8115f2eb ffff881fd2de3b10 ffffffff8115f41a
Nov 21 11:03:20 tuft kernel: [1194882.228116] Call Trace:
Nov 21 11:03:20 tuft kernel: [1194882.228122] [] __vunmap.part.16+0x1d/0xc0
Nov 21 11:03:20 tuft kernel: [1194882.228125] [] __vunmap+0x1b/0x40
Nov 21 11:03:20 tuft kernel: [1194882.228129] [] vfree+0x2a/0x30
Nov 21 11:03:20 tuft kernel: [1194882.228139] [] kv_free.isra.6+0x68/0x70 [spl]
Nov 21 11:03:20 tuft kernel: [1194882.228147] [] spl_slab_reclaim+0x335/0x440 [spl]
Nov 21 11:03:20 tuft kernel: [1194882.228154] [] spl_kmem_cache_reap_now+0x149/0x340 [spl]
Nov 21 11:03:20 tuft kernel: [1194882.228177] [] arc_kmem_reap_now+0x60/0xb0 [zfs]
Nov 21 11:03:20 tuft kernel: [1194882.228193] [] arc_shrinker_func+0xc7/0x120 [zfs]
Nov 21 11:03:20 tuft kernel: [1194882.228198] [] shrink_slab+0x154/0x300
Nov 21 11:03:20 tuft kernel: [1194882.228203] [] ? mem_cgroup_iter+0xe8/0x200
Nov 21 11:03:20 tuft kernel: [1194882.228208] [] balance_pgdat+0x5a4/0x720
Nov 21 11:03:20 tuft kernel: [1194882.228212] [] kswapd+0x123/0x240
Nov 21 11:03:20 tuft kernel: [1194882.228216] [] ? balance_pgdat+0x720/0x720
Nov 21 11:03:20 tuft kernel: [1194882.228221] [] kthread+0x93/0xa0
Nov 21 11:03:20 tuft kernel: [1194882.228228] [] kernel_thread_helper+0x4/0x10
Nov 21 11:03:20 tuft kernel: [1194882.228233] [] ? flush_kthread_worker+0xb0/0xb0
Nov 21 11:03:20 tuft kernel: [1194882.228236] [] ? gs_change+0x13/0x13
Nov 21 11:03:20 tuft kernel: [1194882.228237] Code: 20 1c c4 81 e8 8c 0f 54 00 48 8b 05 cd 83 d9 00 48 c7 c2 f8 74 ef 81 48 39 c3 74 14 66 0f 1f 84 00 00 00 00 00 48 89 c2 48 8b 00 <48> 39 c3 75 f5 48 8b 00 48 89 02 f0 81 05 c4 2a ae 00 00 00 10
Nov 21 11:03:24 tuft kernel: [1194886.528417] INFO: rcu_sched self-detected stall on CPU { 12} (t=15000 jiffies)
Nov 21 11:03:24 tuft kernel: [1194886.538523] INFO: rcu_sched detected stalls on CPUs/tasks: { 12} (detected by 7, t=15002 jiffies)
Nov 21 11:03:24 tuft kernel: [1194886.538530] INFO: Stall ended before state dump start
Nov 21 11:03:24 tuft kernel: [1194886.548390] sending NMI to all CPUs:
....
Nov 21 11:04:20 tuft kernel: [1194942.103659] BUG: soft lockup - CPU#12 stuck for 22s! [kswapd1:149]
Nov 21 11:04:20 tuft kernel: [1194942.112664] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:04:20 tuft kernel: [1194942.112712] CPU 12
Nov 21 11:04:20 tuft kernel: [1194942.112713] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:04:20 tuft kernel: [1194942.112746]
Nov 21 11:04:20 tuft kernel: [1194942.112750] Pid: 149, comm: kswapd1 Tainted: P O 3.5.0-54-generic #81
Nov 21 11:04:20 tuft kernel: [1194942.112753] RIP: 0010:[] [] remove_vm_area+0x86/0xa0
Nov 21 11:04:20 tuft kernel: [1194942.112765] RSP: 0018:ffff881fd2de3ab0 EFLAGS: 00000202
Nov 21 11:04:20 tuft kernel: [1194942.112767] RAX: ffff88232443abc0 RBX: ffff8804a0203f40 RCX: ffff881fd2de3ea0
Nov 21 11:04:20 tuft kernel: [1194942.112769] RDX: ffff8805fb7bf200 RSI: 0000000000000001 RDI: ffffffff81c41c20
Nov 21 11:04:20 tuft kernel: [1194942.112770] RBP: ffff881fd2de3ac0 R08: ffff8804a0203f40 R09: 0000000100400012
Nov 21 11:04:20 tuft kernel: [1194942.112772] R10: 00000000a0203a01 R11: 0000000000000002 R12: 00000000a0203a01
Nov 21 11:04:20 tuft kernel: [1194942.112774] R13: 0000000000000002 R14: 0000000000000000 R15: ffff881fd2de3a90
Nov 21 11:04:20 tuft kernel: [1194942.112776] FS: 0000000000000000(0000) GS:ffff88407fc80000(0000) knlGS:0000000000000000
Nov 21 11:04:20 tuft kernel: [1194942.112778] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov 21 11:04:20 tuft kernel: [1194942.112779] CR2: 00007fd82674a024 CR3: 0000000001c0b000 CR4: 00000000000407e0
Nov 21 11:04:20 tuft kernel: [1194942.112781] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 21 11:04:20 tuft kernel: [1194942.112783] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Nov 21 11:04:20 tuft kernel: [1194942.112785] Process kswapd1 (pid: 149, threadinfo ffff881fd2de2000, task ffff881fd2dd2e20)
Nov 21 11:04:20 tuft kernel: [1194942.112786] Stack:
Nov 21 11:04:20 tuft kernel: [1194942.112788] ffffc90da4892000 ffff883fc8a56848 ffff881fd2de3af0 ffffffff8115f22d
Nov 21 11:04:20 tuft kernel: [1194942.112794] ffff8826f04f8e80 ffffc90d7c5c8000 ffff883fc8a56848 ffff883fc8a56000
Nov 21 11:04:20 tuft kernel: [1194942.112799] ffff881fd2de3b00 ffffffff8115f2eb ffff881fd2de3b10 ffffffff8115f41a
Nov 21 11:04:20 tuft kernel: [1194942.112803] Call Trace:
Nov 21 11:04:20 tuft kernel: [1194942.112809] [] __vunmap.part.16+0x1d/0xc0
Nov 21 11:04:20 tuft kernel: [1194942.112813] [] __vunmap+0x1b/0x40
Nov 21 11:04:20 tuft kernel: [1194942.112816] [] vfree+0x2a/0x30
Nov 21 11:04:20 tuft kernel: [1194942.112827] [] kv_free.isra.6+0x68/0x70 [spl]
Nov 21 11:04:20 tuft kernel: [1194942.112835] [] spl_slab_reclaim+0x335/0x440 [spl]
Nov 21 11:04:20 tuft kernel: [1194942.112842] [] spl_kmem_cache_reap_now+0x149/0x340 [spl]
Nov 21 11:04:20 tuft kernel: [1194942.112866] [] arc_kmem_reap_now+0x60/0xb0 [zfs]
Nov 21 11:04:20 tuft kernel: [1194942.112882] [] arc_shrinker_func+0xc7/0x120 [zfs]
Nov 21 11:04:20 tuft kernel: [1194942.112888] [] shrink_slab+0x154/0x300
Nov 21 11:04:20 tuft kernel: [1194942.112894] [] ? mem_cgroup_iter+0xe8/0x200
Nov 21 11:04:20 tuft kernel: [1194942.112898] [] balance_pgdat+0x5a4/0x720
Nov 21 11:04:20 tuft kernel: [1194942.112902] [] kswapd+0x123/0x240
Nov 21 11:04:20 tuft kernel: [1194942.112906] [] ? balance_pgdat+0x720/0x720
Nov 21 11:04:20 tuft kernel: [1194942.112912] [] kthread+0x93/0xa0
Nov 21 11:04:20 tuft kernel: [1194942.112918] [] kernel_thread_helper+0x4/0x10
Nov 21 11:04:20 tuft kernel: [1194942.112922] [] ? flush_kthread_worker+0xb0/0xb0
Nov 21 11:04:20 tuft kernel: [1194942.112925] [] ? gs_change+0x13/0x13
Nov 21 11:04:20 tuft kernel: [1194942.112926] Code: 20 1c c4 81 e8 8c 0f 54 00 48 8b 05 cd 83 d9 00 48 c7 c2 f8 74 ef 81 48 39 c3 74 14 66 0f 1f 84 00 00 00 00 00 48 89 c2 48 8b 00 <48> 39 c3 75 f5 48 8b 00 48 89 02 f0 81 05 c4 2a ae 00 00 00 10
Nov 21 11:04:48 tuft kernel: [1194970.048960] BUG: soft lockup - CPU#12 stuck for 22s! [kswapd1:149]
Nov 21 11:04:48 tuft kernel: [1194970.057040] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:04:48 tuft kernel: [1194970.057093] CPU 12
Nov 21 11:04:48 tuft kernel: [1194970.057095] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:04:48 tuft kernel: [1194970.057136]
Nov 21 11:04:48 tuft kernel: [1194970.057139] Pid: 149, comm: kswapd1 Tainted: P O 3.5.0-54-generic #81
precise1-Ubuntu Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+precise1-Ubuntu Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+Nov 21 11:04:48 tuft kernel: [1194970.057144] RIP: 0010:[] [] remove_vm_area+0x86/0xa0
Nov 21 11:04:48 tuft kernel: [1194970.057157] RSP: 0018:ffff881fd2de3ab0 EFLAGS: 00000206
Nov 21 11:04:48 tuft kernel: [1194970.057159] RAX: ffff8824844ca580 RBX: ffff8805fb5a2040 RCX: ffff881fd2de3ea0
Nov 21 11:04:48 tuft kernel: [1194970.057161] RDX: ffff8824844ca300 RSI: 0000000000000001 RDI: ffffffff81c41c20
Nov 21 11:04:48 tuft kernel: [1194970.057163] RBP: ffff881fd2de3ac0 R08: ffff8805fb5a2040 R09: 000000010040001f
Nov 21 11:04:48 tuft kernel: [1194970.057165] R10: 00000000fb5a2901 R11: 0000000000000002 R12: 000000003a200601
Nov 21 11:04:48 tuft kernel: [1194970.057167] R13: 0000000100000000 R14: 0000000000000000 R15: 000000007fffcd00
Nov 21 11:04:48 tuft kernel: [1194970.057170] FS: 0000000000000000(0000) GS:ffff88407fc80000(0000) knlGS:0000000000000000
Nov 21 11:04:48 tuft kernel: [1194970.057172] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov 21 11:04:48 tuft kernel: [1194970.057174] CR2: 00007fd82674a024 CR3: 0000000001c0b000 CR4: 00000000000407e0
Nov 21 11:04:48 tuft kernel: [1194970.057176] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 21 11:04:48 tuft kernel: [1194970.057178] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Nov 21 11:04:48 tuft kernel: [1194970.057181] Process kswapd1 (pid: 149, threadinfo ffff881fd2de2000, task ffff881fd2dd2e20)
Nov 21 11:04:48 tuft kernel: [1194970.057183] Stack:
Nov 21 11:04:48 tuft kernel: [1194970.057184] ffffc90117c15000 ffff883fc8a56848 ffff881fd2de3af0 ffffffff8115f22d
Nov 21 11:04:48 tuft kernel: [1194970.057193] ffff8826f04f8e80 ffffc9014f585000 ffff883fc8a56848 ffff883fc8a56000
Nov 21 11:04:48 tuft kernel: [1194970.057199] ffff881fd2de3b00 ffffffff8115f2eb ffff881fd2de3b10 ffffffff8115f41a
Nov 21 11:04:48 tuft kernel: [1194970.057206] Call Trace:
Nov 21 11:04:48 tuft kernel: [1194970.057213] [] __vunmap.part.16+0x1d/0xc0
Nov 21 11:04:48 tuft kernel: [1194970.057217] [] __vunmap+0x1b/0x40
Nov 21 11:04:48 tuft kernel: [1194970.057221] [] vfree+0x2a/0x30
Nov 21 11:04:48 tuft kernel: [1194970.057234] [] kv_free.isra.6+0x68/0x70 [spl]
Nov 21 11:04:48 tuft kernel: [1194970.057243] [] spl_slab_reclaim+0x335/0x440 [spl]
Nov 21 11:04:48 tuft kernel: [1194970.057252] [] spl_kmem_cache_reap_now+0x149/0x340 [spl]
Nov 21 11:04:48 tuft kernel: [1194970.057281] [] arc_kmem_reap_now+0x60/0xb0 [zfs]
Nov 21 11:04:48 tuft kernel: [1194970.057298] [] arc_shrinker_func+0xc7/0x120 [zfs]
Nov 21 11:04:48 tuft kernel: [1194970.057305] [] shrink_slab+0x154/0x300
Nov 21 11:04:48 tuft kernel: [1194970.057311] [] ? mem_cgroup_iter+0xe8/0x200
Nov 21 11:04:48 tuft kernel: [1194970.057316] [] balance_pgdat+0x5a4/0x720
Nov 21 11:04:48 tuft kernel: [1194970.057322] [] kswapd+0x123/0x240
Nov 21 11:04:48 tuft kernel: [1194970.057326] [] ? balance_pgdat+0x720/0x720
Nov 21 11:04:48 tuft kernel: [1194970.057332] [] kthread+0x93/0xa0
Nov 21 11:04:48 tuft kernel: [1194970.057339] [] kernel_thread_helper+0x4/0x10
Nov 21 11:04:48 tuft kernel: [1194970.057344] [] ? flush_kthread_worker+0xb0/0xb0
Nov 21 11:04:48 tuft kernel: [1194970.057348] [] ? gs_change+0x13/0x13
Nov 21 11:04:48 tuft kernel: [1194970.057349] Code: 20 1c c4 81 e8 8c 0f 54 00 48 8b 05 cd 83 d9 00 48 c7 c2 f8 74 ef 81 48 39 c3 74 14 66 0f 1f 84 00 00 00 00 00 48 89 c2 48 8b 00 <48> 39 c3 75 f5 48 8b 00 48 89 02 f0 81 05 c4 2a ae 00 00 00 10
Nov 21 11:05:16 tuft kernel: [1194997.994260] BUG: soft lockup - CPU#12 stuck for 22s! [kswapd1:149]
Nov 21 11:05:16 tuft kernel: [1194998.002465] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:05:16 tuft kernel: [1194998.002513] CPU 12
Nov 21 11:05:16 tuft kernel: [1194998.002514] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:05:16 tuft kernel: [1194998.002546]
Nov 21 11:05:16 tuft kernel: [1194998.002550] Pid: 149, comm: kswapd1 Tainted: P O 3.5.0-54-generic #81
Nov 21 11:05:16 tuft kernel: [1194998.002554] RIP: 0010:[] [] remove_vm_area+0x86/0xa0
Nov 21 11:05:16 tuft kernel: [1194998.002565] RSP: 0018:ffff881fd2de3ab0 EFLAGS: 00000283
Nov 21 11:05:16 tuft kernel: [1194998.002567] RAX: ffff883f6f9b0bc0 RBX: ffff882491fb8ec0 RCX: ffff881fd2de3ea0
Nov 21 11:05:16 tuft kernel: [1194998.002569] RDX: ffff882a3963eb00 RSI: 0000000000000001 RDI: ffffffff81c41c20
Nov 21 11:05:16 tuft kernel: [1194998.002570] RBP: ffff881fd2de3ac0 R08: ffff882491fb8ec0 R09: 000000018040001b
Nov 21 11:05:16 tuft kernel: [1194998.002572] R10: 0000000091fb8201 R11: 0000000000000006 R12: 0000000000000001
Nov 21 11:05:16 tuft kernel: [1194998.002574] R13: 0000000100000000 R14: 0000000000000000 R15: 000000017fffbd00
Nov 21 11:05:16 tuft kernel: [1194998.002576] FS: 0000000000000000(0000) GS:ffff88407fc80000(0000) knlGS:0000000000000000
Nov 21 11:05:16 tuft kernel: [1194998.002578] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov 21 11:05:16 tuft kernel: [1194998.002580] CR2: 00007fd82674a024 CR3: 0000000001c0b000 CR4: 00000000000407e0
Nov 21 11:05:16 tuft kernel: [1194998.002582] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 21 11:05:16 tuft kernel: [1194998.002583] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Nov 21 11:05:16 tuft kernel: [1194998.002585] Process kswapd1 (pid: 149, threadinfo ffff881fd2de2000, task ffff881fd2dd2e20)
Nov 21 11:05:16 tuft kernel: [1194998.002587] Stack:
Nov 21 11:05:16 tuft kernel: [1194998.002588] ffffc90a7baca000 ffff883fc8a56848 ffff881fd2de3af0 ffffffff8115f22d
Nov 21 11:05:16 tuft kernel: [1194998.002595] ffff8826f04f8e80 ffffc900d94d4000 ffff883fc8a56848 ffff883fc8a56000
Nov 21 11:05:16 tuft kernel: [1194998.002599] ffff881fd2de3b00 ffffffff8115f2eb ffff881fd2de3b10 ffffffff8115f41a
Nov 21 11:05:16 tuft kernel: [1194998.002604] Call Trace:
Nov 21 11:05:16 tuft kernel: [1194998.002610] [] __vunmap.part.16+0x1d/0xc0
Nov 21 11:05:16 tuft kernel: [1194998.002613] [] __vunmap+0x1b/0x40
Nov 21 11:05:16 tuft kernel: [1194998.002617] [] vfree+0x2a/0x30
Nov 21 11:05:16 tuft kernel: [1194998.002628] [] kv_free.isra.6+0x68/0x70 [spl]
Nov 21 11:05:16 tuft kernel: [1194998.002635] [] spl_slab_reclaim+0x335/0x440 [spl]
Nov 21 11:05:16 tuft kernel: [1194998.002642] [] spl_kmem_cache_reap_now+0x149/0x340 [spl]
Nov 21 11:05:16 tuft kernel: [1194998.002666] [] arc_kmem_reap_now+0x60/0xb0 [zfs]
Nov 21 11:05:16 tuft kernel: [1194998.002682] [] arc_shrinker_func+0xc7/0x120 [zfs]
Nov 21 11:05:16 tuft kernel: [1194998.002687] [] shrink_slab+0x154/0x300
Nov 21 11:05:16 tuft kernel: [1194998.002693] [] ? mem_cgroup_iter+0xe8/0x200
Nov 21 11:05:16 tuft kernel: [1194998.002697] [] balance_pgdat+0x5a4/0x720
Nov 21 11:05:16 tuft kernel: [1194998.002701] [] kswapd+0x123/0x240
Nov 21 11:05:16 tuft kernel: [1194998.002705] [] ? balance_pgdat+0x720/0x720
Nov 21 11:05:16 tuft kernel: [1194998.002710] [] kthread+0x93/0xa0
Nov 21 11:05:16 tuft kernel: [1194998.002716] [] kernel_thread_helper+0x4/0x10
Nov 21 11:05:16 tuft kernel: [1194998.002720] [] ? flush_kthread_worker+0xb0/0xb0
Nov 21 11:05:16 tuft kernel: [1194998.002723] [] ? gs_change+0x13/0x13
Nov 21 11:05:16 tuft kernel: [1194998.002724] Code: 20 1c c4 81 e8 8c 0f 54 00 48 8b 05 cd 83 d9 00 48 c7 c2 f8 74 ef 81 48 39 c3 74 14 66 0f 1f 84 00 00 00 00 00 48 89 c2 48 8b 00 <48> 39 c3 75 f5 48 8b 00 48 89 02 f0 81 05 c4 2a ae 00 00 00 10
Nov 21 11:05:44 tuft kernel: [1195025.939558] BUG: soft lockup - CPU#12 stuck for 22s! [kswapd1:149]
Nov 21 11:05:44 tuft kernel: [1195025.948265] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:05:44 tuft kernel: [1195025.948328] CPU 12
Nov 21 11:05:44 tuft kernel: [1195025.948330] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:05:44 tuft kernel: [1195025.948376]
Nov 21 11:05:44 tuft kernel: [1195025.948380] Pid: 149, comm: kswapd1 Tainted: P O 3.5.0-54-generic #81
precise1-Ubuntu Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+precise1-Ubuntu Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+Nov 21 11:05:44 tuft kernel: [1195025.948385] RIP: 0010:[] [] remove_vm_area+0x86/0xa0
Nov 21 11:05:44 tuft kernel: [1195025.948398] RSP: 0018:ffff881fd2de3ab0 EFLAGS: 00000202
Nov 21 11:05:44 tuft kernel: [1195025.948401] RAX: ffff88318b220ac0 RBX: ffff882162224580 RCX: ffff881fd2de3ea0
Nov 21 11:05:44 tuft kernel: [1195025.948403] RDX: ffff8807a3ea9680 RSI: 0000000000000001 RDI: ffffffff81c41c20
Nov 21 11:05:44 tuft kernel: [1195025.948406] RBP: ffff881fd2de3ac0 R08: ffff882162224580 R09: 000000010040003b
Nov 21 11:05:44 tuft kernel: [1195025.948408] R10: 0000000062224001 R11: 0000000000000006 R12: 0000000062224001
Nov 21 11:05:44 tuft kernel: [1195025.948411] R13: 0000000000000006 R14: 0000000000000000 R15: ffff881fd2de3a90
Nov 21 11:05:44 tuft kernel: [1195025.948413] FS: 0000000000000000(0000) GS:ffff88407fc80000(0000) knlGS:0000000000000000
Nov 21 11:05:44 tuft kernel: [1195025.948416] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov 21 11:05:44 tuft kernel: [1195025.948418] CR2: 00007fd82674a024 CR3: 0000000001c0b000 CR4: 00000000000407e0
Nov 21 11:05:44 tuft kernel: [1195025.948421] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 21 11:05:44 tuft kernel: [1195025.948424] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Nov 21 11:05:44 tuft kernel: [1195025.948427] Process kswapd1 (pid: 149, threadinfo ffff881fd2de2000, task ffff881fd2dd2e20)
Nov 21 11:05:44 tuft kernel: [1195025.948428] Stack:
Nov 21 11:05:44 tuft kernel: [1195025.948431] ffffc90c7363f000 ffff883fc8a56848 ffff881fd2de3af0 ffffffff8115f22d
Nov 21 11:05:44 tuft kernel: [1195025.948441] ffff8826f04f8e80 ffffc90c38929000 ffff883fc8a56848 ffff883fc8a56000
Nov 21 11:05:44 tuft kernel: [1195025.948447] ffff881fd2de3b00 ffffffff8115f2eb ffff881fd2de3b10 ffffffff8115f41a
Nov 21 11:05:44 tuft kernel: [1195025.948454] Call Trace:
Nov 21 11:05:44 tuft kernel: [1195025.948462] [] __vunmap.part.16+0x1d/0xc0
Nov 21 11:05:44 tuft kernel: [1195025.948467] [] __vunmap+0x1b/0x40
Nov 21 11:05:44 tuft kernel: [1195025.948472] [] vfree+0x2a/0x30
Nov 21 11:05:44 tuft kernel: [1195025.948487] [] kv_free.isra.6+0x68/0x70 [spl]
Nov 21 11:05:44 tuft kernel: [1195025.948496] [] spl_slab_reclaim+0x335/0x440 [spl]
Nov 21 11:05:44 tuft kernel: [1195025.948506] [] spl_kmem_cache_reap_now+0x149/0x340 [spl]
Nov 21 11:05:44 tuft kernel: [1195025.948537] [] arc_kmem_reap_now+0x60/0xb0 [zfs]
Nov 21 11:05:44 tuft kernel: [1195025.948556] [] arc_shrinker_func+0xc7/0x120 [zfs]
Nov 21 11:05:44 tuft kernel: [1195025.948564] [] shrink_slab+0x154/0x300
Nov 21 11:05:44 tuft kernel: [1195025.948570] [] ? mem_cgroup_iter+0xe8/0x200
Nov 21 11:05:44 tuft kernel: [1195025.948576] [] balance_pgdat+0x5a4/0x720
Nov 21 11:05:44 tuft kernel: [1195025.948581] [] kswapd+0x123/0x240
Nov 21 11:05:44 tuft kernel: [1195025.948586] [] ? balance_pgdat+0x720/0x720
Nov 21 11:05:44 tuft kernel: [1195025.948593] [] kthread+0x93/0xa0
Nov 21 11:05:44 tuft kernel: [1195025.948601] [] kernel_thread_helper+0x4/0x10
Nov 21 11:05:44 tuft kernel: [1195025.948606] [] ? flush_kthread_worker+0xb0/0xb0
Nov 21 11:05:44 tuft kernel: [1195025.948611] [] ? gs_change+0x13/0x13
Nov 21 11:05:44 tuft kernel: [1195025.948613] Code: 20 1c c4 81 e8 8c 0f 54 00 48 8b 05 cd 83 d9 00 48 c7 c2 f8 74 ef 81 48 39 c3 74 14 66 0f 1f 84 00 00 00 00 00 48 89 c2 48 8b 00 <48> 39 c3 75 f5 48 8b 00 48 89 02 f0 81 05 c4 2a ae 00 00 00 10
Nov 21 11:06:12 tuft kernel: [1195053.884853] BUG: soft lockup - CPU#12 stuck for 22s! [kswapd1:149]
Nov 21 11:06:12 tuft kernel: [1195053.891787] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:06:12 tuft kernel: [1195053.891833] CPU 12
Nov 21 11:06:12 tuft kernel: [1195053.891835] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:06:12 tuft kernel: [1195053.891867]
Nov 21 11:06:12 tuft kernel: [1195053.891869] Pid: 149, comm: kswapd1 Tainted: P O 3.5.0-54-generic #81
Nov 21 11:06:12 tuft kernel: [1195053.891873] RIP: 0010:[] [] remove_vm_area+0x86/0xa0
Nov 21 11:06:12 tuft kernel: [1195053.891884] RSP: 0018:ffff881fd2de3ab0 EFLAGS: 00000287
Nov 21 11:06:12 tuft kernel: [1195053.891886] RAX: ffff881fbfc92440 RBX: ffff881fd2de3a70 RCX: ffff881fd2de3ea0
Nov 21 11:06:12 tuft kernel: [1195053.891887] RDX: ffff881fbfc92640 RSI: 0000000000000001 RDI: ffffffff81c41c20
Nov 21 11:06:12 tuft kernel: [1195053.891889] RBP: ffff881fd2de3ac0 R08: ffff8807a0971300 R09: 000000018040003e
Nov 21 11:06:12 tuft kernel: [1195053.891891] R10: 00000000a0971c01 R11: 0000000000000006 R12: ffffffff8115f294
Nov 21 11:06:12 tuft kernel: [1195053.891892] R13: 0000000000000002 R14: ffff8807a0971c00 R15: 00000001a0971c00
Nov 21 11:06:12 tuft kernel: [1195053.891894] FS: 0000000000000000(0000) GS:ffff88407fc80000(0000) knlGS:0000000000000000
Nov 21 11:06:12 tuft kernel: [1195053.891896] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov 21 11:06:12 tuft kernel: [1195053.891898] CR2: 00007fd82674a024 CR3: 0000000001c0b000 CR4: 00000000000407e0
Nov 21 11:06:12 tuft kernel: [1195053.891900] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 21 11:06:12 tuft kernel: [1195053.891901] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Nov 21 11:06:12 tuft kernel: [1195053.891904] Process kswapd1 (pid: 149, threadinfo ffff881fd2de2000, task ffff881fd2dd2e20)
Nov 21 11:06:12 tuft kernel: [1195053.891905] Stack:
Nov 21 11:06:12 tuft kernel: [1195053.891907] ffffc91d9620a000 ffff883fbfd96848 ffff881fd2de3af0 ffffffff8115f22d
Nov 21 11:06:12 tuft kernel: [1195053.891913] ffff8823e0930e80 ffffc9062f96b000 ffff883fbfd96848 ffff883fbfd96000
Nov 21 11:06:12 tuft kernel: [1195053.891917] ffff881fd2de3b00 ffffffff8115f2eb ffff881fd2de3b10 ffffffff8115f41a
Nov 21 11:06:12 tuft kernel: [1195053.891922] Call Trace:
Nov 21 11:06:12 tuft kernel: [1195053.891928] [] __vunmap.part.16+0x1d/0xc0
Nov 21 11:06:12 tuft kernel: [1195053.891932] [] __vunmap+0x1b/0x40
Nov 21 11:06:12 tuft kernel: [1195053.891935] [] vfree+0x2a/0x30
Nov 21 11:06:12 tuft kernel: [1195053.891945] [] kv_free.isra.6+0x68/0x70 [spl]
Nov 21 11:06:12 tuft kernel: [1195053.891953] [] spl_slab_reclaim+0x335/0x440 [spl]
Nov 21 11:06:12 tuft kernel: [1195053.891960] [] spl_kmem_cache_reap_now+0x149/0x340 [spl]
Nov 21 11:06:12 tuft kernel: [1195053.891983] [] arc_kmem_reap_now+0x60/0xb0 [zfs]
Nov 21 11:06:12 tuft kernel: [1195053.891998] [] arc_shrinker_func+0xc7/0x120 [zfs]
Nov 21 11:06:12 tuft kernel: [1195053.892004] [] shrink_slab+0x154/0x300
Nov 21 11:06:12 tuft kernel: [1195053.892009] [] ? mem_cgroup_iter+0xe8/0x200
Nov 21 11:06:12 tuft kernel: [1195053.892014] [] balance_pgdat+0x5a4/0x720
Nov 21 11:06:12 tuft kernel: [1195053.892018] [] kswapd+0x123/0x240
Nov 21 11:06:12 tuft kernel: [1195053.892021] [] ? balance_pgdat+0x720/0x720
Nov 21 11:06:12 tuft kernel: [1195053.892027] [] kthread+0x93/0xa0
Nov 21 11:06:12 tuft kernel: [1195053.892033] [] kernel_thread_helper+0x4/0x10
Nov 21 11:06:12 tuft kernel: [1195053.892037] [] ? flush_kthread_worker+0xb0/0xb0
Nov 21 11:06:12 tuft kernel: [1195053.892040] [] ? gs_change+0x13/0x13
Nov 21 11:06:12 tuft kernel: [1195053.892041] Code: 20 1c c4 81 e8 8c 0f 54 00 48 8b 05 cd 83 d9 00 48 c7 c2 f8 74 ef 81 48 39 c3 74 14 66 0f 1f 84 00 00 00 00 00 48 89 c2 48 8b 00 <48> 39 c3 75 f5 48 8b 00 48 89 02 f0 81 05 c4 2a ae 00 00 00 10
Nov 21 11:06:24 tuft kernel: [1195066.196729] INFO: rcu_sched self-detected stall on CPU { 12} (t=60005 jiffies)
Nov 21 11:06:24 tuft kernel: [1195066.208705] INFO: rcu_sched detected stalls on CPUs/tasks: { 12} (detected by 26, t=60008 jiffies)
Nov 21 11:06:24 tuft kernel: [1195066.208715] INFO: Stall ended before state dump start
Nov 21 11:06:24 tuft kernel: [1195066.221979] sending NMI to all CPUs:
....
keeps happing a bunch more times
....
Nov 21 11:18:44 tuft kernel: [1195804.228102] BUG: soft lockup - CPU#4 stuck for 22s! [kswapd0:148]
Nov 21 11:18:44 tuft kernel: [1195804.234970] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:18:44 tuft kernel: [1195804.235017] CPU 4
Nov 21 11:18:44 tuft kernel: [1195804.235018] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:18:44 tuft kernel: [1195804.235049]
Nov 21 11:18:44 tuft kernel: [1195804.235052] Pid: 148, comm: kswapd0 Tainted: P O 3.5.0-54-generic #81
precise1-Ubuntu Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+precise1-Ubuntu Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+Nov 21 11:18:44 tuft kernel: [1195804.235056] RIP: 0010:[] [] remove_vm_area+0x86/0xa0
Nov 21 11:18:44 tuft kernel: [1195804.235065] RSP: 0018:ffff881fd2dafab0 EFLAGS: 00000202
Nov 21 11:18:44 tuft kernel: [1195804.235067] RAX: ffff8832283aa240 RBX: ffff88242d9ca480 RCX: ffff881fd2dafea0
Nov 21 11:18:44 tuft kernel: [1195804.235068] RDX: ffff8809a26518c0 RSI: 0000000000000001 RDI: ffffffff81c41c20
Nov 21 11:18:44 tuft kernel: [1195804.235070] RBP: ffff881fd2dafac0 R08: ffff88242d9ca480 R09: 000000018040003f
Nov 21 11:18:44 tuft kernel: [1195804.235071] R10: 0000000000000001 R11: 0000000000000006 R12: 0000000000000000
Nov 21 11:18:44 tuft kernel: [1195804.235073] R13: 0000000100000000 R14: 0000000000000000 R15: 000000007fffbd00
Nov 21 11:18:44 tuft kernel: [1195804.235075] FS: 0000000000000000(0000) GS:ffff881fffc80000(0000) knlGS:0000000000000000
Nov 21 11:18:44 tuft kernel: [1195804.235077] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov 21 11:18:44 tuft kernel: [1195804.235079] CR2: 000000000061cd64 CR3: 0000000001c0b000 CR4: 00000000000407e0
Nov 21 11:18:44 tuft kernel: [1195804.235081] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 21 11:18:44 tuft kernel: [1195804.235082] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Nov 21 11:18:44 tuft kernel: [1195804.235084] Process kswapd0 (pid: 148, threadinfo ffff881fd2dae000, task ffff881fd2dd1710)
Nov 21 11:18:44 tuft kernel: [1195804.235085] Stack:
Nov 21 11:18:44 tuft kernel: [1195804.235087] ffffc907b9566000 ffff883fc8a56848 ffff881fd2dafaf0 ffffffff8115f22d
Nov 21 11:18:44 tuft kernel: [1195804.235093] ffff883abef75980 ffffc90e41b5f000 ffff883fc8a56848 ffff883fc8a56000
Nov 21 11:18:44 tuft kernel: [1195804.235097] ffff881fd2dafb00 ffffffff8115f2eb ffff881fd2dafb10 ffffffff8115f41a
Nov 21 11:18:44 tuft kernel: [1195804.235102] Call Trace:
Nov 21 11:18:44 tuft kernel: [1195804.235107] [] __vunmap.part.16+0x1d/0xc0
Nov 21 11:18:44 tuft kernel: [1195804.235111] [] __vunmap+0x1b/0x40
Nov 21 11:18:44 tuft kernel: [1195804.235114] [] vfree+0x2a/0x30
Nov 21 11:18:44 tuft kernel: [1195804.235126] [] kv_free.isra.6+0x68/0x70 [spl]
Nov 21 11:18:44 tuft kernel: [1195804.235134] [] spl_slab_reclaim+0x335/0x440 [spl]
Nov 21 11:18:44 tuft kernel: [1195804.235141] [] spl_kmem_cache_reap_now+0x149/0x340 [spl]
Nov 21 11:18:44 tuft kernel: [1195804.235170] [] arc_kmem_reap_now+0x60/0xb0 [zfs]
Nov 21 11:18:44 tuft kernel: [1195804.235186] [] arc_shrinker_func+0xc7/0x120 [zfs]
Nov 21 11:18:44 tuft kernel: [1195804.235192] [] shrink_slab+0x154/0x300
Nov 21 11:18:44 tuft kernel: [1195804.235196] [] ? mem_cgroup_iter+0xe8/0x200
Nov 21 11:18:44 tuft kernel: [1195804.235200] [] balance_pgdat+0x5a4/0x720
Nov 21 11:18:44 tuft kernel: [1195804.235204] [] kswapd+0x123/0x240
Nov 21 11:18:44 tuft kernel: [1195804.235208] [] ? balance_pgdat+0x720/0x720
Nov 21 11:18:44 tuft kernel: [1195804.235213] [] kthread+0x93/0xa0
Nov 21 11:18:44 tuft kernel: [1195804.235217] [] kernel_thread_helper+0x4/0x10
Nov 21 11:18:44 tuft kernel: [1195804.235221] [] ? flush_kthread_worker+0xb0/0xb0
Nov 21 11:18:44 tuft kernel: [1195804.235224] [] ? gs_change+0x13/0x13
Nov 21 11:18:44 tuft kernel: [1195804.235225] Code: 20 1c c4 81 e8 8c 0f 54 00 48 8b 05 cd 83 d9 00 48 c7 c2 f8 74 ef 81 48 39 c3 74 14 66 0f 1f 84 00 00 00 00 00 48 89 c2 48 8b 00 <48> 39 c3 75 f5 48 8b 00 48 89 02 f0 81 05 c4 2a ae 00 00 00 10
Nov 21 11:19:12 tuft kernel: [1195832.173403] BUG: soft lockup - CPU#4 stuck for 22s! [kswapd0:148]
Nov 21 11:19:12 tuft kernel: [1195832.181760] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:19:12 tuft kernel: [1195832.181819] CPU 4
Nov 21 11:19:12 tuft kernel: [1195832.181821] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:19:12 tuft kernel: [1195832.181861]
Nov 21 11:19:12 tuft kernel: [1195832.181865] Pid: 148, comm: kswapd0 Tainted: P O 3.5.0-54-generic #81
Nov 21 11:19:12 tuft kernel: [1195832.181869] RIP: 0010:[] [] remove_vm_area+0x86/0xa0
Nov 21 11:19:12 tuft kernel: [1195832.181879] RSP: 0018:ffff881fd2dafab0 EFLAGS: 00000206
Nov 21 11:19:12 tuft kernel: [1195832.181882] RAX: ffff88080a281800 RBX: ffff8829645701c0 RCX: ffff881fd2dafea0
Nov 21 11:19:12 tuft kernel: [1195832.181884] RDX: ffff8807d6224240 RSI: 0000000000000001 RDI: ffffffff81c41c20
Nov 21 11:19:12 tuft kernel: [1195832.181886] RBP: ffff881fd2dafac0 R08: ffff8829645701c0 R09: 000000010040003d
Nov 21 11:19:12 tuft kernel: [1195832.181888] R10: 0000000064570001 R11: 0000000000000002 R12: 000000006e4f9501
Nov 21 11:19:12 tuft kernel: [1195832.181890] R13: ffffc912ca2ae000 R14: ffffc912ca2adfff R15: 000000007fffcd00
Nov 21 11:19:12 tuft kernel: [1195832.181892] FS: 0000000000000000(0000) GS:ffff881fffc80000(0000) knlGS:0000000000000000
Nov 21 11:19:12 tuft kernel: [1195832.181895] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov 21 11:19:12 tuft kernel: [1195832.181897] CR2: 000000000061cd64 CR3: 0000000001c0b000 CR4: 00000000000407e0
Nov 21 11:19:12 tuft kernel: [1195832.181899] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 21 11:19:12 tuft kernel: [1195832.181901] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Nov 21 11:19:12 tuft kernel: [1195832.181904] Process kswapd0 (pid: 148, threadinfo ffff881fd2dae000, task ffff881fd2dd1710)
Nov 21 11:19:12 tuft kernel: [1195832.181906] Stack:
Nov 21 11:19:12 tuft kernel: [1195832.181908] ffffc912760a8000 ffff883fc8a56848 ffff881fd2dafaf0 ffffffff8115f22d
Nov 21 11:19:12 tuft kernel: [1195832.181917] ffff883abef75980 ffffc9151cb92000 ffff883fc8a56848 ffff883fc8a56000
Nov 21 11:19:12 tuft kernel: [1195832.181923] ffff881fd2dafb00 ffffffff8115f2eb ffff881fd2dafb10 ffffffff8115f41a
Nov 21 11:19:12 tuft kernel: [1195832.181929] Call Trace:
Nov 21 11:19:12 tuft kernel: [1195832.181936] [] __vunmap.part.16+0x1d/0xc0
Nov 21 11:19:12 tuft kernel: [1195832.181941] [] __vunmap+0x1b/0x40
Nov 21 11:19:12 tuft kernel: [1195832.181945] [] vfree+0x2a/0x30
Nov 21 11:19:12 tuft kernel: [1195832.181961] [] kv_free.isra.6+0x68/0x70 [spl]
Nov 21 11:19:12 tuft kernel: [1195832.181970] [] spl_slab_reclaim+0x335/0x440 [spl]
Nov 21 11:19:12 tuft kernel: [1195832.181979] [] spl_kmem_cache_reap_now+0x149/0x340 [spl]
Nov 21 11:19:12 tuft kernel: [1195832.182013] [] arc_kmem_reap_now+0x60/0xb0 [zfs]
Nov 21 11:19:12 tuft kernel: [1195832.182032] [] arc_shrinker_func+0xc7/0x120 [zfs]
Nov 21 11:19:12 tuft kernel: [1195832.182038] [] shrink_slab+0x154/0x300
Nov 21 11:19:12 tuft kernel: [1195832.182044] [] ? mem_cgroup_iter+0xe8/0x200
Nov 21 11:19:12 tuft kernel: [1195832.182049] [] balance_pgdat+0x5a4/0x720
Nov 21 11:19:12 tuft kernel: [1195832.182054] [] kswapd+0x123/0x240
Nov 21 11:19:12 tuft kernel: [1195832.182059] [] ? balance_pgdat+0x720/0x720
Nov 21 11:19:12 tuft kernel: [1195832.182064] [] kthread+0x93/0xa0
Nov 21 11:19:12 tuft kernel: [1195832.182070] [] kernel_thread_helper+0x4/0x10
Nov 21 11:19:12 tuft kernel: [1195832.182075] [] ? flush_kthread_worker+0xb0/0xb0
Nov 21 11:19:12 tuft kernel: [1195832.182078] [] ? gs_change+0x13/0x13
Nov 21 11:19:12 tuft kernel: [1195832.182080] Code: 20 1c c4 81 e8 8c 0f 54 00 48 8b 05 cd 83 d9 00 48 c7 c2 f8 74 ef 81 48 39 c3 74 14 66 0f 1f 84 00 00 00 00 00 48 89 c2 48 8b 00 <48> 39 c3 75 f5 48 8b 00 48 89 02 f0 81 05 c4 2a ae 00 00 00 10
Nov 21 11:19:40 tuft kernel: [1195860.118701] BUG: soft lockup - CPU#4 stuck for 22s! [kswapd0:148]
Nov 21 11:19:40 tuft kernel: [1195860.125246] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:19:40 tuft kernel: [1195860.125293] CPU 4
Nov 21 11:19:40 tuft kernel: [1195860.125294] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:19:40 tuft kernel: [1195860.125326]
Nov 21 11:19:40 tuft kernel: [1195860.125329] Pid: 148, comm: kswapd0 Tainted: P O 3.5.0-54-generic #81
precise1-Ubuntu Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+precise1-Ubuntu Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+Nov 21 11:19:40 tuft kernel: [1195860.125332] RIP: 0010:[] [] remove_vm_area+0x86/0xa0
Nov 21 11:19:40 tuft kernel: [1195860.125341] RSP: 0018:ffff881fd2dafab0 EFLAGS: 00000206
Nov 21 11:19:40 tuft kernel: [1195860.125343] RAX: ffff8836f774b600 RBX: ffff88146faad140 RCX: ffff881fd2dafea0
Nov 21 11:19:40 tuft kernel: [1195860.125345] RDX: ffff8821314e0780 RSI: 0000000000000001 RDI: ffffffff81c41c20
Nov 21 11:19:40 tuft kernel: [1195860.125346] RBP: ffff881fd2dafac0 R08: ffff88146faad140 R09: 000000018040003f
Nov 21 11:19:40 tuft kernel: [1195860.125348] R10: 0000000000000001 R11: 0000000000000002 R12: 000000007f669101
Nov 21 11:19:40 tuft kernel: [1195860.125349] R13: 0000000100000000 R14: 0000000000000000 R15: 000000007fffcd00
Nov 21 11:19:40 tuft kernel: [1195860.125352] FS: 0000000000000000(0000) GS:ffff881fffc80000(0000) knlGS:0000000000000000
Nov 21 11:19:40 tuft kernel: [1195860.125353] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov 21 11:19:40 tuft kernel: [1195860.125355] CR2: 000000000061cd64 CR3: 0000000001c0b000 CR4: 00000000000407e0
Nov 21 11:19:40 tuft kernel: [1195860.125357] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 21 11:19:40 tuft kernel: [1195860.125358] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Nov 21 11:19:40 tuft kernel: [1195860.125360] Process kswapd0 (pid: 148, threadinfo ffff881fd2dae000, task ffff881fd2dd1710)
Nov 21 11:19:40 tuft kernel: [1195860.125362] Stack:
Nov 21 11:19:40 tuft kernel: [1195860.125363] ffffc91054548000 ffff883fc8a56848 ffff881fd2dafaf0 ffffffff8115f22d
Nov 21 11:19:40 tuft kernel: [1195860.125369] ffff883238d34180 ffffc900d4f3f000 ffff883fc8a56848 ffff883fc8a56000
Nov 21 11:19:40 tuft kernel: [1195860.125374] ffff881fd2dafb00 ffffffff8115f2eb ffff881fd2dafb10 ffffffff8115f41a
Nov 21 11:19:40 tuft kernel: [1195860.125379] Call Trace:
Nov 21 11:19:40 tuft kernel: [1195860.125384] [] __vunmap.part.16+0x1d/0xc0
Nov 21 11:19:40 tuft kernel: [1195860.125388] [] __vunmap+0x1b/0x40
Nov 21 11:19:40 tuft kernel: [1195860.125391] [] vfree+0x2a/0x30
Nov 21 11:19:40 tuft kernel: [1195860.125404] [] kv_free.isra.6+0x68/0x70 [spl]
Nov 21 11:19:40 tuft kernel: [1195860.125413] [] spl_slab_reclaim+0x335/0x440 [spl]
Nov 21 11:19:40 tuft kernel: [1195860.125420] [] spl_kmem_cache_reap_now+0x149/0x340 [spl]
Nov 21 11:19:40 tuft kernel: [1195860.125451] [] arc_kmem_reap_now+0x60/0xb0 [zfs]
Nov 21 11:19:40 tuft kernel: [1195860.125467] [] arc_shrinker_func+0xc7/0x120 [zfs]
Nov 21 11:19:40 tuft kernel: [1195860.125472] [] shrink_slab+0x154/0x300
Nov 21 11:19:40 tuft kernel: [1195860.125476] [] ? mem_cgroup_iter+0xe8/0x200
Nov 21 11:19:40 tuft kernel: [1195860.125481] [] balance_pgdat+0x5a4/0x720
Nov 21 11:19:40 tuft kernel: [1195860.125485] [] kswapd+0x123/0x240
Nov 21 11:19:40 tuft kernel: [1195860.125488] [] ? balance_pgdat+0x720/0x720
Nov 21 11:19:40 tuft kernel: [1195860.125493] [] kthread+0x93/0xa0
Nov 21 11:19:40 tuft kernel: [1195860.125498] [] kernel_thread_helper+0x4/0x10
Nov 21 11:19:40 tuft kernel: [1195860.125502] [] ? flush_kthread_worker+0xb0/0xb0
Nov 21 11:19:40 tuft kernel: [1195860.125505] [] ? gs_change+0x13/0x13
Nov 21 11:19:40 tuft kernel: [1195860.125506] Code: 20 1c c4 81 e8 8c 0f 54 00 48 8b 05 cd 83 d9 00 48 c7 c2 f8 74 ef 81 48 39 c3 74 14 66 0f 1f 84 00 00 00 00 00 48 89 c2 48 8b 00 <48> 39 c3 75 f5 48 8b 00 48 89 02 f0 81 05 c4 2a ae 00 00 00 10
Nov 21 11:20:08 tuft kernel: [1195888.064000] BUG: soft lockup - CPU#4 stuck for 22s! [kswapd0:148]
Nov 21 11:20:08 tuft kernel: [1195888.070809] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:20:08 tuft kernel: [1195888.070857] CPU 4
Nov 21 11:20:08 tuft kernel: [1195888.070859] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:20:08 tuft kernel: [1195888.070890]
Nov 21 11:20:08 tuft kernel: [1195888.070893] Pid: 148, comm: kswapd0 Tainted: P O 3.5.0-54-generic #81
Nov 21 11:20:08 tuft kernel: [1195888.070897] RIP: 0010:[] [] remove_vm_area+0x86/0xa0
Nov 21 11:20:08 tuft kernel: [1195888.070906] RSP: 0018:ffff881fd2dafab0 EFLAGS: 00000206
Nov 21 11:20:08 tuft kernel: [1195888.070908] RAX: ffff8821f1b32280 RBX: ffff880812ad4180 RCX: ffff881fd2dafea0
Nov 21 11:20:08 tuft kernel: [1195888.070909] RDX: ffff8825f17a9dc0 RSI: 0000000000000001 RDI: ffffffff81c41c20
Nov 21 11:20:08 tuft kernel: [1195888.070911] RBP: ffff881fd2dafac0 R08: ffff880812ad4180 R09: 000000010040003d
Nov 21 11:20:08 tuft kernel: [1195888.070913] R10: 0000000012ad4301 R11: 0000000000000006 R12: 00000000bbd77801
Nov 21 11:20:08 tuft kernel: [1195888.070914] R13: 0000000100000000 R14: 0000000000000000 R15: 000000007fffbd00
Nov 21 11:20:08 tuft kernel: [1195888.070916] FS: 0000000000000000(0000) GS:ffff881fffc80000(0000) knlGS:0000000000000000
Nov 21 11:20:08 tuft kernel: [1195888.070918] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov 21 11:20:08 tuft kernel: [1195888.070920] CR2: 000000000061cd64 CR3: 0000000001c0b000 CR4: 00000000000407e0
Nov 21 11:20:08 tuft kernel: [1195888.070921] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 21 11:20:08 tuft kernel: [1195888.070923] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Nov 21 11:20:08 tuft kernel: [1195888.070925] Process kswapd0 (pid: 148, threadinfo ffff881fd2dae000, task ffff881fd2dd1710)
Nov 21 11:20:08 tuft kernel: [1195888.070926] Stack:
Nov 21 11:20:08 tuft kernel: [1195888.070928] ffffc9075237e000 ffff883fc8a56848 ffff881fd2dafaf0 ffffffff8115f22d
Nov 21 11:20:08 tuft kernel: [1195888.070934] ffff883238d34180 ffffc911c5975000 ffff883fc8a56848 ffff883fc8a56000
Nov 21 11:20:08 tuft kernel: [1195888.070939] ffff881fd2dafb00 ffffffff8115f2eb ffff881fd2dafb10 ffffffff8115f41a
Nov 21 11:20:08 tuft kernel: [1195888.070943] Call Trace:
Nov 21 11:20:08 tuft kernel: [1195888.070949] [] __vunmap.part.16+0x1d/0xc0
Nov 21 11:20:08 tuft kernel: [1195888.070952] [] __vunmap+0x1b/0x40
Nov 21 11:20:08 tuft kernel: [1195888.070955] [] vfree+0x2a/0x30
Nov 21 11:20:08 tuft kernel: [1195888.070967] [] kv_free.isra.6+0x68/0x70 [spl]
Nov 21 11:20:08 tuft kernel: [1195888.070975] [] spl_slab_reclaim+0x335/0x440 [spl]
Nov 21 11:20:08 tuft kernel: [1195888.070982] [] spl_kmem_cache_reap_now+0x149/0x340 [spl]
Nov 21 11:20:08 tuft kernel: [1195888.071012] [] arc_kmem_reap_now+0x60/0xb0 [zfs]
Nov 21 11:20:08 tuft kernel: [1195888.071027] [] arc_shrinker_func+0xc7/0x120 [zfs]
Nov 21 11:20:08 tuft kernel: [1195888.071033] [] shrink_slab+0x154/0x300
Nov 21 11:20:08 tuft kernel: [1195888.071038] [] ? mem_cgroup_iter+0xe8/0x200
Nov 21 11:20:08 tuft kernel: [1195888.071042] [] balance_pgdat+0x5a4/0x720
Nov 21 11:20:08 tuft kernel: [1195888.071046] [] kswapd+0x123/0x240
Nov 21 11:20:08 tuft kernel: [1195888.071050] [] ? balance_pgdat+0x720/0x720
Nov 21 11:20:08 tuft kernel: [1195888.071054] [] kthread+0x93/0xa0
Nov 21 11:20:08 tuft kernel: [1195888.071059] [] kernel_thread_helper+0x4/0x10
Nov 21 11:20:08 tuft kernel: [1195888.071063] [] ? flush_kthread_worker+0xb0/0xb0
Nov 21 11:20:08 tuft kernel: [1195888.071066] [] ? gs_change+0x13/0x13
Nov 21 11:20:08 tuft kernel: [1195888.071067] Code: 20 1c c4 81 e8 8c 0f 54 00 48 8b 05 cd 83 d9 00 48 c7 c2 f8 74 ef 81 48 39 c3 74 14 66 0f 1f 84 00 00 00 00 00 48 89 c2 48 8b 00 <48> 39 c3 75 f5 48 8b 00 48 89 02 f0 81 05 c4 2a ae 00 00 00 10
Nov 21 11:20:36 tuft kernel: [1195916.009307] BUG: soft lockup - CPU#4 stuck for 22s! [kswapd0:148]
Nov 21 11:20:36 tuft kernel: [1195916.015504] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:20:36 tuft kernel: [1195916.015553] CPU 4
Nov 21 11:20:36 tuft kernel: [1195916.015554] Modules linked in: dm_snapshot ext2 ipmi_devintf ipmi_si ipmi_msghandler binfmt_misc vesafb kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 microcode sb_edac edac_core joydev lpc_ich mei ioatdma acpi_pad mac_hid coretemp lp parport zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate hid_generic usbhid hid raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov igb ahci libahci dca isci libsas mpt2sas scsi_transport_sas raid_class raid6_pq async_tx raid1 raid0 multipath linear
Nov 21 11:20:36 tuft kernel: [1195916.015586]
Nov 21 11:20:36 tuft kernel: [1195916.015589] Pid: 148, comm: kswapd0 Tainted: P O 3.5.0-54-generic #81~precise1-Ubuntu Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+
Nov 21 11:20:36 tuft kernel: [1195916.015593] RIP: 0010:[] [] free_hot_cold_page+0x11a/0x1a0
Nov 21 11:20:36 tuft kernel: [1195916.015601] RSP: 0018:ffff881fd2dafa60 EFLAGS: 00000297
Nov 21 11:20:36 tuft kernel: [1195916.015603] RAX: 000000000000001f RBX: ffff881fd2dafa80 RCX: 0000000000016e70
Nov 21 11:20:36 tuft kernel: [1195916.015605] RDX: ffffffffffffffa8 RSI: 0000000000000000 RDI: 0000000000000297
Nov 21 11:20:36 tuft kernel: [1195916.015607] RBP: ffff881fd2dafab0 R08: ffff88407fffbd78 R09: ffffea008e84fb00
Nov 21 11:20:36 tuft kernel: [1195916.015608] R10: 00000000000003ef R11: 0000000000000006 R12: ffff881fd2dafa50
Nov 21 11:20:36 tuft kernel: [1195916.015610] R13: 0000000000000001 R14: ffff881fd2daf9d0 R15: ffff881fd2daf9d0
Nov 21 11:20:36 tuft kernel: [1195916.015612] FS: 0000000000000000(0000) GS:ffff881fffc80000(0000) knlGS:0000000000000000
Nov 21 11:20:36 tuft kernel: [1195916.015614] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov 21 11:20:36 tuft kernel: [1195916.015615] CR2: 000000000061cd64 CR3: 0000000001c0b000 CR4: 00000000000407e0
Nov 21 11:20:36 tuft kernel: [1195916.015617] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 21 11:20:36 tuft kernel: [1195916.015619] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Nov 21 11:20:36 tuft kernel: [1195916.015621] Process kswapd0 (pid: 148, threadinfo ffff881fd2dae000, task ffff881fd2dd1710)
Nov 21 11:20:36 tuft kernel: [1195916.015622] Stack:
Nov 21 11:20:36 tuft kernel: [1195916.015623] 0000000000000000 ffff880eb44fcc00 ffffea00a0df6420 0000000000000002
Nov 21 11:20:36 tuft kernel: [1195916.015630] 0000000000000000 000000000000000d ffff88000ef9d100 0000000000000040
Nov 21 11:20:36 tuft kernel: [1195916.015634] ffff883fc8a56848 ffffc906bd31c000 ffff881fd2dafac0 ffffffff81130d0d
Nov 21 11:20:36 tuft kernel: [1195916.015639] Call Trace:
Nov 21 11:20:36 tuft kernel: [1195916.015644] [] __free_pages+0x2d/0x40
Nov 21 11:20:36 tuft kernel: [1195916.015650] [] __vunmap.part.16+0x63/0xc0
Nov 21 11:20:36 tuft kernel: [1195916.015653] [] __vunmap+0x1b/0x40
Nov 21 11:20:36 tuft kernel: [1195916.015657] [] vfree+0x2a/0x30
Nov 21 11:20:36 tuft kernel: [1195916.015668] [] kv_free.isra.6+0x68/0x70 [spl]
Nov 21 11:20:36 tuft kernel: [1195916.015676] [] spl_slab_reclaim+0x335/0x440 [spl]
Nov 21 11:20:36 tuft kernel: [1195916.015683] [] spl_kmem_cache_reap_now+0x149/0x340 [spl]
Nov 21 11:20:36 tuft kernel: [1195916.015713] [] arc_kmem_reap_now+0x60/0xb0 [zfs]
Nov 21 11:20:36 tuft kernel: [1195916.015728] [] arc_shrinker_func+0xc7/0x120 [zfs]
Nov 21 11:20:36 tuft kernel: [1195916.015734] [] shrink_slab+0x154/0x300
Nov 21 11:20:36 tuft kernel: [1195916.015738] [] ? mem_cgroup_iter+0xe8/0x200
Nov 21 11:20:36 tuft kernel: [1195916.015742] [] balance_pgdat+0x5a4/0x720
Nov 21 11:20:36 tuft kernel: [1195916.015746] [] kswapd+0x123/0x240
Nov 21 11:20:36 tuft kernel: [1195916.015750] [] ? balance_pgdat+0x720/0x720
Nov 21 11:20:36 tuft kernel: [1195916.015755] [] kthread+0x93/0xa0
Nov 21 11:20:36 tuft kernel: [1195916.015759] [] kernel_thread_helper+0x4/0x10
Nov 21 11:20:36 tuft kernel: [1195916.015763] [] ? flush_kthread_worker+0xb0/0xb0
Nov 21 11:20:36 tuft kernel: [1195916.015766] [] ? gs_change+0x13/0x13
Nov 21 11:20:36 tuft kernel: [1195916.015767] Code: 8b 0a 48 8d 43 20 48 89 41 08 48 89 4b 20 48 89 53 28 48 89 02 41 8b 45 00 83 c0 01 41 3b 45 04 41 89 45 00 7d 33 4c 89 e7 57 9d <66> 66 90 66 90 e9 29 ff ff ff 0f 1f 40 00 48 8b 42 08 48 8d 4b
Nov 21 11:21:04 tuft kernel: [1195943.954599] BUG: soft lockup - CPU#4 stuck for 22s! [kswapd0:148]
.....
server frozen and totally dead
The text was updated successfully, but these errors were encountered: