Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

soft lockup #382

Closed
ghost opened this issue Aug 31, 2011 · 4 comments
Closed

soft lockup #382

ghost opened this issue Aug 31, 2011 · 4 comments
Milestone
0.6.0-rc7

Comments

@ghost
Copy link

ghost commented Aug 31, 2011

This issue happens on my 3-drive pool, running Ubuntu server 10.04 64-bit. Got a stack trace through the serial port:

[ 3627.100001] BUG: soft lockup - CPU#0 stuck for 61s! [sftp-server:2448]

[ 3653.369956] SysRq : Emergency Sync

[ 3659.111602] SysRq : Trigger a crash

[ 3659.115117] BUG: unable to handle kernel NULL pointer dereference at (null)

[ 3659.120003] IP: [] sysrq_handle_crash+0x16/0x20

[ 3659.120003] PGD 0

[ 3659.120003] Oops: 0002 [#1] SMP

[ 3659.120003] last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:04:00.0/host5/uevent

[ 3659.120003] CPU 0

[ 3659.120003] Modules linked in: zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) fbcon tileblit font bitblit softcursor vga16fb vgastate i915 spl drm_kms_helper zlib_deflate smsc47m192 hwmon_vid ppdev i2c_i801 drm parport_pc i2c_algo_bit lp intel_agp video output parport floppy sata_via r8169 mii

[ 3659.120003] Pid: 2448, comm: sftp-server Tainted: P W 2.6.32-33-server #72-Ubuntu

[ 3659.120003] RIP: 0010:[] [] sysrq_handle_crash+0x16/0x20

[ 3659.120003] RSP: 0018:ffff880001c03c18 EFLAGS: 00010096

[ 3659.120003] RAX: 0000000000000010 RBX: 0000000000000063 RCX: 0000000000000000

[ 3659.120003] RDX: 0000000000000000 RSI: ffff880017d93000 RDI: 0000000000000063

[ 3659.120003] RBP: ffff880001c03c18 R08: 00000000ffffffff R09: 0000000000000000

[ 3659.120003] R10: 0000000000000005 R11: 0000000000000000 R12: ffff880017d93000

[ 3659.120003] R13: ffffffff81814940 R14: 0000000000000082 R15: 0000000000000004

[ 3659.120003] FS: 00007f4543b95700(0000) GS:ffff880001c00000(0000) knlGS:0000000000000000

[ 3659.120003] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

[ 3659.120003] CR2: 0000000000000000 CR3: 00000000281b2000 CR4: 00000000000006f0

[ 3659.120003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000

[ 3659.120003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

[ 3659.120003] Process sftp-server (pid: 2448, threadinfo ffff880029408000, task ffff88002f809700)

[ 3659.120003] Stack:

[ 3659.120003] ffff880001c03c68 ffffffff81349622 000000000000580f 0000000000000001

[ 3659.120003] <0> 0000000001c0fe70 000000000000002e 0000000000000001 ffff88003f004000

[ 3659.120003] <0> ffff880017d93000 0000000000000001 ffff880001c03c78 ffffffff8134970b

[ 3659.120003] Call Trace:

[ 3659.120003]

[ 3659.120003] [] __handle_sysrq+0x132/0x1a0

[ 3659.120003] [] handle_sysrq+0x2b/0x30

[ 3659.120003] [] kbd_keycode+0x491/0x5e0

[ 3659.120003] [] ? mix_pool_bytes_extract+0x16b/0x180

[ 3659.120003] [] kbd_event+0x11b/0x120

[ 3659.120003] [] input_pass_event+0xbe/0xc0

[ 3659.120003] [] input_handle_event+0x93/0x470

[ 3659.120003] [] input_event+0x84/0xa0

[ 3659.120003] [] atkbd_interrupt+0x633/0x6f0

[ 3659.120003] [] serio_interrupt+0x52/0xa0

[ 3659.120003] [] i8042_interrupt+0xe0/0x2b0

[ 3659.120003] [] handle_IRQ_event+0x60/0x170

[ 3659.120003] [] handle_edge_irq+0xd2/0x170

[ 3659.120003] [] handle_irq+0x22/0x30

[ 3659.120003] [] do_IRQ+0x6c/0xf0

[ 3659.120003] [] ret_from_intr+0x0/0x11

[ 3659.120003]

[ 3659.120003] [] ? generic_exec_single+0x82/0xb0

[ 3659.120003] [] ? generic_exec_single+0xa7/0xb0

[ 3659.120003] [] ? drain_local_pages+0x0/0x20

[ 3659.120003] [] ? drain_local_pages+0x0/0x20

[ 3659.120003] [] ? smp_call_function_single+0x162/0x170

[ 3659.120003] [] ? drain_local_pages+0x0/0x20

[ 3659.120003] [] ? smp_call_function_many+0x22a/0x260

[ 3659.120003] [] ? drain_local_pages+0x0/0x20

[ 3659.120003] [] ? smp_call_function+0x22/0x30

[ 3659.120003] [] ? on_each_cpu+0x24/0x50

[ 3659.120003] [] ? drain_all_pages+0x1c/0x20

[ 3659.120003] [] ? __alloc_pages_slowpath+0x31a/0x590

[ 3659.120003] [] ? __alloc_pages_nodemask+0x171/0x180

[ 3659.120003] [] ? alloc_pages_current+0x87/0xd0

[ 3659.120003] [] ? new_slab+0x2f7/0x310

[ 3659.120003] [] ? __slab_alloc+0x169/0x2d0

[ 3659.120003] [] ? kmem_alloc_debug+0xeb/0x130 [spl]

[ 3659.120003] [] ? __kmalloc+0x16d/0x1a0

[ 3659.120003] [] ? kmem_alloc_debug+0xeb/0x130 [spl]

[ 3659.120003] [] ? _spin_lock+0xe/0x20

[ 3659.120003] [] ? zil_itx_create+0x3c/0x60 [zfs]

[ 3659.120003] [] ? zfs_log_write+0xde/0x340 [zfs]

[ 3659.120003] [] ? zfs_write+0x919/0xc90 [zfs]

[ 3659.120003] [] ? __wake_up_common+0x59/0x90

[ 3659.120003] [] ? file_update_time+0x62/0x180

[ 3659.120003] [] ? zpl_write_common+0x52/0x70 [zfs]

[ 3659.120003] [] ? zpl_write+0x68/0xa0 [zfs]

[ 3659.120003] [] ? security_file_permission+0x16/0x20

[ 3659.120003] [] ? vfs_write+0xb8/0x1a0

[ 3659.120003] [] ? sys_write+0x51/0x80

[ 3659.120003] [] ? system_call_fastpath+0x16/0x1b

[ 3659.120003] Code: d0 88 81 63 13 a7 81 c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 1f 44 00 00 c7 05 39 a6 65 00 01 00 00 00 0f ae f8 04 25 00 00 00 00 01 c9 c3 55 48 89 e5 0f 1f 44 00 00 8d 47

[ 3659.120003] RIP [] sysrq_handle_crash+0x16/0x20

[ 3659.120003] RSP

[ 3659.120003] CR2: 0000000000000000

[ 3659.120003] ---[ end trace fc48178a5381684a ]---

[ 3659.120003] Kernel panic - not syncing: Fatal exception in interrupt

[ 3659.120003] Pid: 2448, comm: sftp-server Tainted: P D W 2.6.32-33-server #72-Ubuntu

[ 3659.120003] Call Trace:

[ 3659.120003] [] panic+0x78/0x139

[ 3659.120003] [] oops_end+0xea/0xf0

[ 3659.120003] [] no_context+0xf3/0x190

[ 3659.120003] [] __bad_area_nosemaphore+0x125/0x1e0

[ 3659.120003] [] bad_area_nosemaphore+0x13/0x20

[ 3659.120003] [] do_page_fault+0x2e4/0x3b0

[ 3659.120003] [] page_fault+0x25/0x30

[ 3659.120003] [] ? sysrq_handle_crash+0x16/0x20

[ 3659.120003] [] __handle_sysrq+0x132/0x1a0

[ 3659.120003] [] handle_sysrq+0x2b/0x30

[ 3659.120003] [] kbd_keycode+0x491/0x5e0

[ 3659.120003] [] ? mix_pool_bytes_extract+0x16b/0x180

[ 3659.120003] [] kbd_event+0x11b/0x120

[ 3659.120003] [] input_pass_event+0xbe/0xc0

[ 3659.120003] [] input_handle_event+0x93/0x470

[ 3659.120003] [] input_event+0x84/0xa0

[ 3659.120003] [] atkbd_interrupt+0x633/0x6f0

[ 3659.120003] [] serio_interrupt+0x52/0xa0

[ 3659.120003] [] i8042_interrupt+0xe0/0x2b0

[ 3659.120003] [] handle_IRQ_event+0x60/0x170

[ 3659.120003] [] handle_edge_irq+0xd2/0x170

[ 3659.120003] [] handle_irq+0x22/0x30

[ 3659.120003] [] do_IRQ+0x6c/0xf0

[ 3659.120003] [] ret_from_intr+0x0/0x11

[ 3659.120003] [] ? generic_exec_single+0x82/0xb0

[ 3659.120003] [] ? generic_exec_single+0xa7/0xb0

[ 3659.120003] [] ? drain_local_pages+0x0/0x20

[ 3659.120003] [] ? drain_local_pages+0x0/0x20

[ 3659.120003] [] ? smp_call_function_single+0x162/0x170

[ 3659.120003] [] ? drain_local_pages+0x0/0x20

[ 3659.120003] [] ? smp_call_function_many+0x22a/0x260

[ 3659.120003] [] ? drain_local_pages+0x0/0x20

[ 3659.120003] [] ? smp_call_function+0x22/0x30

[ 3659.120003] [] ? on_each_cpu+0x24/0x50

[ 3659.120003] [] ? drain_all_pages+0x1c/0x20

[ 3659.120003] [] ? __alloc_pages_slowpath+0x31a/0x590

[ 3659.120003] [] ? __alloc_pages_nodemask+0x171/0x180

[ 3659.120003] [] ? alloc_pages_current+0x87/0xd0

[ 3659.120003] [] ? new_slab+0x2f7/0x310

[ 3659.120003] [] ? __slab_alloc+0x169/0x2d0

[ 3659.120003] [] ? kmem_alloc_debug+0xeb/0x130 [spl]

[ 3659.120003] [] ? __kmalloc+0x16d/0x1a0

[ 3659.120003] [] ? kmem_alloc_debug+0xeb/0x130 [spl]

[ 3659.120003] [] ? _spin_lock+0xe/0x20

[ 3659.120003] [] ? zil_itx_create+0x3c/0x60 [zfs]

[ 3659.120003] [] ? zfs_log_write+0xde/0x340 [zfs]

[ 3659.120003] [] ? zfs_write+0x919/0xc90 [zfs]

[ 3659.120003] [] ? __wake_up_common+0x59/0x90

[ 3659.120003] [] ? file_update_time+0x62/0x180

[ 3659.120003] [] ? zpl_write_common+0x52/0x70 [zfs]

[ 3659.120003] [] ? zpl_write+0x68/0xa0 [zfs]

[ 3659.120003] [] ? security_file_permission+0x16/0x20

[ 3659.120003] [] ? vfs_write+0xb8/0x1a0

[ 3659.120003] [] ? sys_write+0x51/0x80

[ 3659.120003] [] ? system_call_fastpath+0x16/0x1b

$ sudo dpkg -s spl
Package: spl
Status: install ok installed
Priority: extra
Section: kernel
Installed-Size: 100
Maintainer: Darik Horn [email protected]
Architecture: amd64
Source: spl-linux
Version: 0.6.0.31-0ubuntu1~lucid1
@behlendorf
Copy link
Contributor

Thanks for the debugging we'll look in to it as best we can. Was this a one time event or are you able to recreate this issue regularly?

@ghost
Copy link
Author

ghost commented Sep 5, 2011

The machine hangs regularly, I can recreate it easily, just wait for an hour

@ghost
Copy link
Author

ghost commented Nov 6, 2011

I have recently updated my server to the latest Ubuntu 11.10, running smooth so far.. I could not wait for a fix because the system was too unstable. I won't be able to test the issue again, sorry about that

@behlendorf
Copy link
Contributor

OK, thanks for the update. We've had a couple stability fixes merged recently as well which could easily explain the original issue you reported. Since your unable to recreate the original problem I'm going to close this issue, please feel free to open a new issue if you observe problems.

kernelOfTruth pushed a commit to kernelOfTruth/zfs that referenced this issue Mar 1, 2015
@behlendorf
Avoid PAGESIZE redefinition
f9bde4f 
Add #ifndef PAGESIZE to avoid redefinition warning on platforms
where this value is already provided.

Signed-off-by: stf <[email protected]>
Signed-off-by: Brian Behlendorf <[email protected]>
Closes openzfs#382
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
0.6.0-rc7
Development

No branches or pull requests
1 participant
@behlendorf