Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752703AbZDMAvQ (ORCPT ); Sun, 12 Apr 2009 20:51:16 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1751242AbZDMAu7 (ORCPT ); Sun, 12 Apr 2009 20:50:59 -0400 Received: from www262.sakura.ne.jp ([202.181.97.72]:52166 "EHLO www262.sakura.ne.jp" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752120AbZDMAu6 (ORCPT ); Sun, 12 Apr 2009 20:50:58 -0400 Message-Id: <200904130048.n3D0mw1f077050@www262.sakura.ne.jp> Subject: Re: [2.6.30-rc1] RCU detected CPU 1 stall From: Tetsuo Handa To: viro@ZenIV.linux.org.uk Cc: linux-kernel@vger.kernel.org, hugh@veritas.com, jmorris@namei.org, akpm@linux-foundation.org, paulmck@linux.vnet.ibm.com MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Date: Mon, 13 Apr 2009 09:48:58 +0900 References: <200904080057.n380vZAH051872@www262.sakura.ne.jp> <20090410142203.GA6719@linux.vnet.ibm.com> <20090410150353.GL26366@ZenIV.linux.org.uk> <20090410153229.GB6719@linux.vnet.ibm.com> <200904110608.IED21123.FQOVMtSOOHFFLJ@I-love.SAKURA.ne.jp> <20090410231245.GF6719@linux.vnet.ibm.com> <20090410233919.GS26366@ZenIV.linux.org.uk> In-Reply-To: <20090410233919.GS26366@ZenIV.linux.org.uk> Content-Type: text/plain; charset="ISO-2022-JP" X-Anti-Virus: K-Prox Anti-Virus Powered by Kaspersky, bases: 12042009 #1843665, status: clean Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 9799 Lines: 184 Hello. Paul E. McKenney wrote: > Is this reproducible? Not always, but it is reproducible. Al Viro wrote: > I'd really love to see results of repeated alt-sysrq-p/alt-sysrq-l, just > to see where was it actually spinning. Below is sysrq message. Maybe something related to khelper's current->mm == NULL warning problem. Full log is at http://I-love.SAKURA.ne.jp/tmp/dmesg-2.6.30-rc1-200904130930.txt . [ 47.412519] SysRq : Show Regs [ 47.413986] [ 47.414584] Pid: 3655, comm: khelper Tainted: G W (2.6.30-rc1 #1) VMware Virtual Platform [ 47.415804] EIP: 0060:[] EFLAGS: 00000293 CPU: 0 [ 47.415804] EIP is at __get_user_4+0x11/0x17 [ 47.415804] EAX: f7150003 EBX: f7150000 ECX: 00000000 EDX: f6744000 [ 47.415804] ESI: 000007b8 EDI: 7fffffff EBP: f6744f20 ESP: f6744f10 [ 47.415804] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 [ 47.415804] CR0: 8005003b CR2: f7150000 CR3: 3599a000 CR4: 000006d0 [ 47.415804] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 47.415804] DR6: ffff0ff0 DR7: 00000400 [ 47.415804] Call Trace: [ 47.415804] [] ? count+0x3e/0xb0 [ 47.415804] [] do_execve+0x621/0x890 [ 47.415804] [] ? getname+0x6b/0xa0 [ 47.415804] [] sys_execve+0x5e/0xb0 [ 47.415804] [] syscall_call+0x7/0xb [ 47.415804] [] ? kernel_execve+0x24/0x30 [ 47.415804] [] ? ____call_usermodehelper+0xff/0x170 [ 47.415804] [] ? ____call_usermodehelper+0x0/0x170 [ 47.415804] [] ? kernel_thread_helper+0x7/0x10 [ 52.551469] SysRq : Show backtrace of all active CPUs [ 52.551803] CPU0: [ 52.551803] [ 52.551803] Pid: 3655, comm: khelper Tainted: G W (2.6.30-rc1 #1) VMware Virtual Platform [ 52.551803] EIP: 0060:[] EFLAGS: 00000293 CPU: 0 [ 52.551803] EIP is at __get_user_4+0x11/0x17 [ 52.551803] EAX: f7150003 EBX: f7150000 ECX: 00000000 EDX: f6744000 [ 52.551803] ESI: 000007b8 EDI: 7fffffff EBP: f6744f20 ESP: f6744f10 [ 52.551803] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 [ 52.551803] CR0: 8005003b CR2: f7150000 CR3: 3599a000 CR4: 000006d0 [ 52.551803] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 52.551803] DR6: ffff0ff0 DR7: 00000400 [ 52.551803] Call Trace: [ 52.551803] [] ? count+0x3e/0xb0 [ 52.551803] [] do_execve+0x621/0x890 [ 52.551803] [] ? getname+0x6b/0xa0 [ 52.551803] [] sys_execve+0x5e/0xb0 [ 52.551803] [] syscall_call+0x7/0xb [ 52.551803] [] ? kernel_execve+0x24/0x30 [ 52.551803] [] ? ____call_usermodehelper+0xff/0x170 [ 52.551803] [] ? ____call_usermodehelper+0x0/0x170 [ 52.551803] [] ? kernel_thread_helper+0x7/0x10 [ 60.956967] SysRq : Show Regs [ 60.958362] [ 60.958896] Pid: 3655, comm: khelper Tainted: G W (2.6.30-rc1 #1) VMware Virtual Platform [ 60.959802] EIP: 0060:[] EFLAGS: 00000293 CPU: 0 [ 60.959802] EIP is at __get_user_4+0x11/0x17 [ 60.959802] EAX: f7150003 EBX: f7150000 ECX: 00000000 EDX: f6744000 [ 60.959802] ESI: 000007b8 EDI: 7fffffff EBP: f6744f20 ESP: f6744f10 [ 60.959802] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 [ 60.959802] CR0: 8005003b CR2: f7150000 CR3: 3599a000 CR4: 000006d0 [ 60.959802] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 60.959802] DR6: ffff0ff0 DR7: 00000400 [ 60.959802] Call Trace: [ 60.959802] [] ? count+0x3e/0xb0 [ 60.959802] [] do_execve+0x621/0x890 [ 60.959802] [] ? getname+0x6b/0xa0 [ 60.959802] [] sys_execve+0x5e/0xb0 [ 60.959802] [] syscall_call+0x7/0xb [ 60.959802] [] ? kernel_execve+0x24/0x30 [ 60.959802] [] ? ____call_usermodehelper+0xff/0x170 [ 60.959802] [] ? ____call_usermodehelper+0x0/0x170 [ 60.959802] [] ? kernel_thread_helper+0x7/0x10 [ 69.014489] SysRq : Show backtrace of all active CPUs [ 69.015802] CPU0: [ 69.015802] [ 69.015802] Pid: 3655, comm: khelper Tainted: G W (2.6.30-rc1 #1) VMware Virtual Platform [ 69.015802] EIP: 0060:[] EFLAGS: 00000293 CPU: 0 [ 69.015802] EIP is at __get_user_4+0x11/0x17 [ 69.015802] EAX: f7150003 EBX: f7150000 ECX: 00000000 EDX: f6744000 [ 69.015802] ESI: 000007b8 EDI: 7fffffff EBP: f6744f20 ESP: f6744f10 [ 69.015802] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 [ 69.015802] CR0: 8005003b CR2: f7150000 CR3: 3599a000 CR4: 000006d0 [ 69.015802] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 69.015802] DR6: ffff0ff0 DR7: 00000400 [ 69.015802] Call Trace: [ 69.015802] [] ? count+0x3e/0xb0 [ 69.015802] [] do_execve+0x621/0x890 [ 69.015802] [] ? getname+0x6b/0xa0 [ 69.015802] [] sys_execve+0x5e/0xb0 [ 69.015802] [] syscall_call+0x7/0xb [ 69.015802] [] ? kernel_execve+0x24/0x30 [ 69.015802] [] ? ____call_usermodehelper+0xff/0x170 [ 69.015802] [] ? ____call_usermodehelper+0x0/0x170 [ 69.015802] [] ? kernel_thread_helper+0x7/0x10 [ 73.695801] INFO: RCU detected CPU 0 stall (t=4294910558/10000 jiffies) [ 73.695801] Pid: 3655, comm: khelper Tainted: G W 2.6.30-rc1 #1 [ 73.695801] Call Trace: [ 73.695801] [] ? printk+0x1d/0x30 [ 73.695801] [] print_cpu_stall+0x45/0xa0 [ 73.695801] [] check_cpu_stall+0xe8/0x190 [ 73.695801] [] ? get_timestamp+0xd/0x20 [ 73.695801] [] __rcu_pending+0x1b/0x160 [ 73.695801] [] rcu_pending+0x2e/0x70 [ 73.695801] [] update_process_times+0x3b/0x80 [ 73.695801] [] tick_periodic+0x40/0x90 [ 73.695801] [] tick_handle_periodic+0x1e/0xa0 [ 73.695801] [] ? apic_timer_interrupt+0x28/0x34 [ 73.695801] [] local_apic_timer_interrupt+0x6f/0x80 [ 73.695801] [] ? irq_enter+0x14/0x60 [ 73.695801] [] smp_apic_timer_interrupt+0x33/0x42 [ 73.695801] [] apic_timer_interrupt+0x2f/0x34 [ 73.695801] [] ? __get_user_4+0x11/0x17 [ 73.695801] [] ? count+0x3e/0xb0 [ 73.695801] [] do_execve+0x621/0x890 [ 73.695801] [] ? getname+0x6b/0xa0 [ 73.695801] [] sys_execve+0x5e/0xb0 [ 73.695801] [] syscall_call+0x7/0xb [ 73.695801] [] ? kernel_execve+0x24/0x30 [ 73.695801] [] ? ____call_usermodehelper+0xff/0x170 [ 73.695801] [] ? ____call_usermodehelper+0x0/0x170 [ 73.695801] [] ? kernel_thread_helper+0x7/0x10 [ 79.029887] SysRq : Show Regs [ 79.030975] [ 79.031578] Pid: 3655, comm: khelper Tainted: G W (2.6.30-rc1 #1) VMware Virtual Platform [ 79.031800] EIP: 0060:[] EFLAGS: 00000293 CPU: 0 [ 79.031800] EIP is at __get_user_4+0x11/0x17 [ 79.031800] EAX: f7150003 EBX: f7150000 ECX: 00000000 EDX: f6744000 [ 79.031800] ESI: 000007b8 EDI: 7fffffff EBP: f6744f20 ESP: f6744f10 [ 79.031800] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 [ 79.031800] CR0: 8005003b CR2: f7150000 CR3: 3599a000 CR4: 000006d0 [ 79.031800] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 79.031800] DR6: ffff0ff0 DR7: 00000400 [ 79.031800] Call Trace: [ 79.031800] [] ? count+0x3e/0xb0 [ 79.031800] [] do_execve+0x621/0x890 [ 79.031800] [] ? getname+0x6b/0xa0 [ 79.031800] [] sys_execve+0x5e/0xb0 [ 79.031800] [] syscall_call+0x7/0xb [ 79.031800] [] ? kernel_execve+0x24/0x30 [ 79.031800] [] ? ____call_usermodehelper+0xff/0x170 [ 79.031800] [] ? ____call_usermodehelper+0x0/0x170 [ 79.031800] [] ? kernel_thread_helper+0x7/0x10 [ 87.925907] SysRq : Show backtrace of all active CPUs [ 87.927799] CPU0: [ 87.927799] [ 87.927799] Pid: 3655, comm: khelper Tainted: G W (2.6.30-rc1 #1) VMware Virtual Platform [ 87.927799] EIP: 0060:[] EFLAGS: 00000293 CPU: 0 [ 87.927799] EIP is at __get_user_4+0x11/0x17 [ 87.927799] EAX: f7150003 EBX: f7150000 ECX: 00000000 EDX: f6744000 [ 87.927799] ESI: 000007b8 EDI: 7fffffff EBP: f6744f20 ESP: f6744f10 [ 87.927799] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 [ 87.927799] CR0: 8005003b CR2: f7150000 CR3: 3599a000 CR4: 000006d0 [ 87.927799] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 87.927799] DR6: ffff0ff0 DR7: 00000400 [ 87.927799] Call Trace: [ 87.927799] [] ? count+0x3e/0xb0 [ 87.927799] [] do_execve+0x621/0x890 [ 87.927799] [] ? getname+0x6b/0xa0 [ 87.927799] [] sys_execve+0x5e/0xb0 [ 87.927799] [] syscall_call+0x7/0xb [ 87.927799] [] ? kernel_execve+0x24/0x30 [ 87.927799] [] ? ____call_usermodehelper+0xff/0x170 [ 87.927799] [] ? ____call_usermodehelper+0x0/0x170 [ 87.927799] [] ? kernel_thread_helper+0x7/0x10 [ 98.785592] BUG: soft lockup - CPU#1 stuck for 61s! [khelper:3657] [ 98.785592] Modules linked in: ipv6 pcnet32 crc32 [ 98.785592] irq event stamp: 0 [ 98.785592] hardirqs last enabled at (0): [<(null)>] (null) [ 98.785592] hardirqs last disabled at (0): [] copy_process+0x2ab/0x11a0 [ 98.785592] softirqs last enabled at (0): [] copy_process+0x2ab/0x11a0 [ 98.785592] softirqs last disabled at (0): [<(null)>] (null) [ 98.785592] Regards. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/