Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753723AbZIONJ5 (ORCPT ); Tue, 15 Sep 2009 09:09:57 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1753588AbZIONJv (ORCPT ); Tue, 15 Sep 2009 09:09:51 -0400 Received: from e23smtp06.au.ibm.com ([202.81.31.148]:46465 "EHLO e23smtp06.au.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750862AbZIONJu (ORCPT ); Tue, 15 Sep 2009 09:09:50 -0400 Date: Tue, 15 Sep 2009 18:28:36 +0530 From: Dhaval Giani To: Andrew Morton Cc: Srivatsa Vaddagiri , Balbir Singh , Ingo Molnar , Peter Zijlstra , bugzilla-daemon@bugzilla.kernel.org, bugme-daemon@bugzilla.kernel.org, "Serge E. Hallyn" , containers@lists.linux-foundation.org, linux-kernel@vger.kernel.org, risrajak@linux.vnet.ibm.com, iranna.ankad@in.ibm.com, Bharata B Rao Subject: Re: [Bugme-new] [Bug 14150] New: BUG: soft lockup - CPU#3 stuck for 61s!, while running cpu controller latency testcase on two containers parallaly Message-ID: <20090915125836.GA4087@linux.vnet.ibm.com> Reply-To: Dhaval Giani References: <20090911142813.dc823bf1.akpm@linux-foundation.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20090911142813.dc823bf1.akpm@linux-foundation.org> User-Agent: Mutt/1.5.18 (2008-05-17) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 6516 Lines: 156 On Fri, Sep 11, 2009 at 02:28:13PM -0700, Andrew Morton wrote: > > (switched to email. Please respond via emailed reply-to-all, not via the > bugzilla web interface). > > On Thu, 10 Sep 2009 09:32:30 GMT > bugzilla-daemon@bugzilla.kernel.org wrote: > > > http://bugzilla.kernel.org/show_bug.cgi?id=14150 > > > > Summary: BUG: soft lockup - CPU#3 stuck for 61s!, while running > > cpu controller latency testcase on two containers > > parallaly > > Product: Process Management > > Version: 2.5 > > Kernel Version: 2.6.31-rc7 > > Platform: All > > OS/Version: Linux > > Tree: Mainline > > Status: NEW > > Severity: high > > Priority: P1 > > Component: Scheduler > > AssignedTo: mingo@elte.hu > > ReportedBy: risrajak@linux.vnet.ibm.com > > CC: serue@us.ibm.com, iranna.ankad@in.ibm.com, > > risrajak@in.ibm.com > > Regression: No > > > > > > Created an attachment (id=23055) > > --> (http://bugzilla.kernel.org/attachment.cgi?id=23055) > > Config-file-used > > > > Hitting this soft lock issue while running this scenario on 2.6.31-rc7 kernel > > on SystemX 32 bit on multiple machines. > > > > Scenario: > > - While running cpu controller latency testcase from LTP same time on two > > containers. > > > > Steps: > > 1. Compile ltp-full-20090731.tgz on host. > > 2. Create two container (Used lxc tool > > (http://sourceforge.net/projects/lxc/lxc-0.6.3.tar.gz) for creating container ) > > e.g: > > lxc-create -n foo1 > > lxc-create -n foo2 > > On first shell: > > lxc-execute -n foo1 -f /usr/etc/lxc/lxc-macvlan.conf /bin/bash > > on Second shell: > > lxc-execute -n foo2 -f /usr/etc/lxc/lxc-macvlan.conf /bin/bash > > > > 3. Either you run cpu_latency testcase alone or run "./runltp -f controllers" > > at same time on both the containers. > > 4. After testcase execution completes, you can see this message in dmesg. > > > > Expected Result: > > - Should not reproduce soft lock up issue. > > - This reproduces 3 times out of 5 tries. > > > > hrtimer: interrupt too slow, forcing clock min delta to 5843235 ns > > hrtimer: interrupt too slow, forcing clock min delta to 5842476 ns > > Clocksource tsc unstable (delta = 18749057581 ns) > > BUG: soft lockup - CPU#3 stuck for 61s! [cpuctl_latency_:17174] > > Modules linked in: bridge stp llc bnep sco l2cap bluetooth sunrpc ipv6 > > p4_clockmod dm_multipath uinput qla2xxx ata_generic pata_acpi usb_storage e1000 > > scsi_transport_fc joydev scsi_tgt i2c_piix4 pata_serverworks pcspkr serio_raw > > mptspi mptscsih mptbase scsi_transport_spi radeon ttm drm i2c_algo_bit i2c_core > > [last unloaded: scsi_wait_scan] > > > > Pid: 17174, comm: cpuctl_latency_ Tainted: G W (2.6.31-rc7 #1) IBM > > eServer BladeCenter HS40 -[883961X]- > > EIP: 0060:[] EFLAGS: 00000283 CPU: 3 > > EIP is at find_next_bit+0x9/0x79 > > EAX: c2c437a0 EBX: f3d433c0 ECX: 00000000 EDX: 00000020 > > ESI: c2c436bc EDI: 00000000 EBP: f063be6c ESP: f063be64 > > DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 > > CR0: 80050033 CR2: 008765a4 CR3: 314d7000 CR4: 000006d0 > > DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 > > DR6: ffff0ff0 DR7: 00000400 > > Call Trace: > > [] cpumask_next+0x17/0x19 > > [] tg_shares_up+0x53/0x149 > > [] ? tg_nop+0x0/0xc > > [] ? tg_nop+0x0/0xc > > [] walk_tg_tree+0x63/0x77 > > [] ? tg_shares_up+0x0/0x149 > > [] update_shares+0x5d/0x65 > > [] rebalance_domains+0x114/0x460 > > [] ? restore_all_notrace+0x0/0x18 > > [] run_rebalance_domains+0x36/0xa3 > > [] __do_softirq+0xbc/0x173 > > [] do_softirq+0x3b/0x5f > > [] irq_exit+0x3a/0x68 > > [] smp_apic_timer_interrupt+0x6d/0x7b > > [] apic_timer_interrupt+0x2f/0x34 > > BUG: soft lockup - CPU#2 stuck for 61s! [watchdog/2:11] > > Modules linked in: bridge stp llc bnep sco l2cap bluetooth sunrpc ipv6 > > p4_clockmod dm_multipath uinput qla2xxx ata_generic pata_acpi usb_storage e1000 > > scsi_transport_fc joydev scsi_tgt i2c_piix4 pata_serverworks pcspkr serio_raw > > mptspi mptscsih mptbase scsi_transport_spi radeon ttm drm i2c_algo_bit i2c_core > > [last unloaded: scsi_wait_scan] > > > > Pid: 11, comm: watchdog/2 Tainted: G W (2.6.31-rc7 #1) IBM eServer > > BladeCenter HS40 -[883961X]- > > EIP: 0060:[] EFLAGS: 00000246 CPU: 2 > > EIP is at tg_shares_up+0xd9/0x149 > > EAX: 00000000 EBX: f09b3c00 ECX: f0baac00 EDX: 00000100 > > ESI: 00000002 EDI: 00000400 EBP: f6cb7de0 ESP: f6cb7db8 > > DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 > > CR0: 8005003b CR2: 08070680 CR3: 009c8000 CR4: 000006d0 > > DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 > > DR6: ffff0ff0 DR7: 00000400 > > Call Trace: > > [] ? tg_nop+0x0/0xc > > [] ? tg_nop+0x0/0xc > > [] walk_tg_tree+0x63/0x77 > > [] ? tg_shares_up+0x0/0x149 > > [] update_shares+0x5d/0x65 > > [] rebalance_domains+0x114/0x460 > > [] run_rebalance_domains+0x36/0xa3 > > [] __do_softirq+0xbc/0x173 > > [] do_softirq+0x3b/0x5f > > [] irq_exit+0x3a/0x68 > > [] smp_apic_timer_interrupt+0x6d/0x7b > > [] apic_timer_interrupt+0x2f/0x34 > > [] ? finish_task_switch+0x5d/0xc4 > > [] schedule+0x74c/0x7b2 > > [] ? trace_hardirqs_on_thunk+0xc/0x10 > > [] ? restore_all_notrace+0x0/0x18 > > [] ? watchdog+0x0/0x79 > > [] ? watchdog+0x0/0x79 > > [] watchdog+0x4a/0x79 > > [] kthread+0x70/0x75 > > [] ? kthread+0x0/0x75 > > [] kernel_thread_helper+0x7/0x10 > > [root@hs40 ltp-full-20090731]# uname -a > > Linux hs40.in.ibm.com 2.6.31-rc7 #1 SMP Thu Sep 3 10:14:41 IST 2009 i686 i686 > > i386 GNU/Linux > > [root@hs40 ltp-full-20090731]# > > > We have been unable to reproduce it on current -tip. Rishi, are you able to reproduce it on -tip? thanks, -- regards, Dhaval -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/