Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753166Ab2JPJD7 (ORCPT ); Tue, 16 Oct 2012 05:03:59 -0400 Received: from mail.skyhub.de ([78.46.96.112]:35482 "EHLO mail.skyhub.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751867Ab2JPJD5 (ORCPT ); Tue, 16 Oct 2012 05:03:57 -0400 Date: Tue, 16 Oct 2012 11:03:53 +0200 From: Borislav Petkov To: Laurent CARON Cc: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org Subject: Re: Strange crash on Dell R720xd Message-ID: <20121016090353.GB7428@liondog.tnic> Mail-Followup-To: Borislav Petkov , Laurent CARON , linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org References: <20121015213522.kaiyahro@trusted.unix-scripts.info> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20121015213522.kaiyahro@trusted.unix-scripts.info> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4915 Lines: 108 On Mon, Oct 15, 2012 at 09:42:58PM +0200, Laurent CARON wrote: > Hi, > > I'm currently replacing an old system (HP DL 380 G5) by new dell R720xd. > On those new boxes I did configure the H310 controler as plain JBOD. > > Those boxes appear to crash more often than not (from 5 mins to a couple > of hours). > I have the impression those crashes appear under heavy IO. > > The setup consists of a few md RAID arrays serving as underlying devices > for either filesystem, or drbd (plus lvm on top). > > I managed to catch a trace over netconsole: > ------------[ cut here ]------------ > kernel BUG at crypto/async_tx/async_tx.c:174! That's: BUG_ON(async_tx_test_ack(depend_tx) || txd_next(depend_tx) || txd_parent(tx)); but probably the b0rkage happens up the stack. And this __raid_run_ops is probably starting the whole TX so maybe we should add linux-raid@vger.kernel.org to CC. Added. > invalid opcode: 0000 [#1] SMP > Modules linked in: drbd lru_cache netconsole iptable_filter ip_tables ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue bonding ipv6 btrfs ioatdma lpc_ich sb_edac dca mfd_core > CPU 0 > Pid: 12580, comm: kworker/u:2 Not tainted 3.6.2-r510-r720xd #1 Dell Inc. PowerEdge R720xd What is that "r510" thing in the kernel version? You have your patches ontop? If yes, please try reproducing this with a kernel.org kernel without anything else ontop. Also, it might be worth trying plain 3.6 to rule out a regression introduced in the stable 3.6 series. (leaving in the rest for reference) > RIP: 0010:[] [] async_tx_submit+0x29/0xab > RSP: 0018:ffff88100940fb30 EFLAGS: 00010202 > RAX: ffff88100b30aeb0 RBX: ffff88080b5cf390 RCX: 0000000000000029 > RDX: ffff88100940fd00 RSI: ffff88080b5cf390 RDI: ffff880809ad0818 > RBP: ffff8808054a7d90 R08: ffff88080b5cf900 R09: 0000000000000001 > R10: 0000000000001000 R11: 0000000000000001 R12: ffff88100940fd00 > R13: 0000000000000002 R14: ffff880809ad0638 R15: ffff880809ad0818 > FS: 0000000000000000(0000) GS:ffff88080fc00000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > CR2: ffffffffff600400 CR3: 0000000e4055f000 CR4: 00000000000407f0 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Process kworker/u:2 (pid: 12580, threadinfo ffff88100940e000, task ffff880804850630) > Stack: > ffff88100940fd00 ffff88100940fc40 0000000000000101 ffffffff8131044b > 0000000000000001 0000000000000246 0000000000000201 ffffffffa0073a00 > ffff8808054a7d90 ffff8808054a7690 ffff88100940fc40 ffff88080bf9e668 > Call Trace: > [] ? do_async_gen_syndrome+0x2f3/0x320 > [] ? ioat2_tx_submit_unlock+0xac/0xb3 [ioatdma] > [] ? ops_complete_compute+0x7b/0x7b > [] ? async_gen_syndrome+0xc8/0x1d6 > [] ? __raid_run_ops+0x9e7/0xb5a > [] ? select_task_rq_fair+0x487/0x74b > [] ? ops_complete_compute+0x7b/0x7b > [] ? __wake_up+0x35/0x46 > [] ? async_schedule+0x12/0x12 > [] ? async_run_ops+0x32/0x3e > [] ? async_run_entry_fn+0xa4/0x17e > [] ? async_schedule+0x12/0x12 > [] ? process_one_work+0x259/0x381 > [] ? worker_thread+0x2ad/0x3e3 > [] ? try_to_wake_up+0x1fc/0x20c > [] ? manage_workers+0x245/0x245 > [] ? manage_workers+0x245/0x245 > [] ? kthread+0x81/0x89 > [] ? kernel_thread_helper+0x4/0x10 > [] ? kthread_freezable_should_stop+0x4e/0x4e > [] ? gs_change+0xb/0xb > Code: 5b c3 41 54 49 89 d4 55 53 48 89 f3 48 8b 6a 08 48 8b 42 10 48 85 ed 48 89 46 20 48 8b 42 18 48 89 46 28 74 5c f6 45 04 02 74 72 <0f> 0b eb fe 48 8b 02 48 8b 48 28 80 e1 40 74 24 31 f6 48 89 d7 > RIP [] async_tx_submit+0x29/0xab > RSP > ---[ end trace 64fb561d16a3b535 ]--- > Kernel panic - not syncing: Fatal exception in interrupt > Rebooting in 5 seconds.. > > Do any of you guys have a clue about it ? > > Thanks > > Laurent > > PS: The very same kernel doesn't cause any trouble on R510 hardware. > > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ -- Regards/Gruss, Boris. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/