Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758928AbYGRTLV (ORCPT ); Fri, 18 Jul 2008 15:11:21 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1754452AbYGRTLK (ORCPT ); Fri, 18 Jul 2008 15:11:10 -0400 Received: from mx2.mail.elte.hu ([157.181.151.9]:36412 "EHLO mx2.mail.elte.hu" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752421AbYGRTLJ (ORCPT ); Fri, 18 Jul 2008 15:11:09 -0400 Date: Fri, 18 Jul 2008 21:10:37 +0200 From: Ingo Molnar To: Vegard Nossum Cc: David Miller , linux-kernel@vger.kernel.org, netdev@vger.kernel.org, penberg@cs.helsinki.fi, rjw@sisk.pl Subject: Re: [bug] Attempt to release alive inet socket f6fac040 Message-ID: <20080718191037.GA25937@elte.hu> References: <20080717214222.GA29449@elte.hu> <19f34abd0807171615s5b477d4cr22d3e9444bcf65df@mail.gmail.com> <20080717.190315.186892939.davem@davemloft.net> <19f34abd0807180003m7bdf832g141777751d7ea563@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <19f34abd0807180003m7bdf832g141777751d7ea563@mail.gmail.com> User-Agent: Mutt/1.5.18 (2008-05-17) X-ELTE-VirusStatus: clean X-ELTE-SpamScore: -1.5 X-ELTE-SpamLevel: X-ELTE-SpamCheck: no X-ELTE-SpamVersion: ELTE 2.0 X-ELTE-SpamCheck-Details: score=-1.5 required=5.9 tests=BAYES_00 autolearn=no SpamAssassin version=3.2.3 -1.5 BAYES_00 BODY: Bayesian spam probability is 0 to 1% [score: 0.0000] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4804 Lines: 92 Hm, just got another skb related weirdness and crash on that testbox: [ 13.495000] usb 1-0:1.0: uevent [ 19.392000] Attempt to release alive inet socket f6fac040 [ 20.387000] EXT3 FS on sda1, internal journal [ 20.762500] eth0: no IPv6 routers present [ 22.431000] warning: process `kudzu' used the deprecated sysctl system call with 1.23. [ 22.906000] ------------[ cut here ]------------ [ 22.906000] kernel BUG at mm/slab.c:2927! [ 22.906000] invalid opcode: 0000 [#1] PREEMPT SMP [ 22.906000] Modules linked in: [ 22.906000] [ 22.906000] Pid: 1831, comm: ip6tables-resto Tainted: G W (2.6.26-tip-03923-g2c96bcb-dirty #3134) [ 22.906000] EIP: 0060:[] EFLAGS: 00010046 CPU: 0 [ 22.906000] EIP is at cache_alloc_refill+0xfa/0x431 [ 22.906000] EAX: 00000004 EBX: f6ec1a00 ECX: f67826c0 EDX: f6fa3c40 [ 22.906000] ESI: f6fac000 EDI: 0000000b EBP: f5c01ec4 ESP: f5c01e70 [ 22.906000] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 [ 22.906000] Process ip6tables-resto (pid: 1831, ti=f5c01000 task=f5d2f060 task.ti=f5c01000) [ 22.906000] Stack: 00000004 00000004 000000d0 000000d0 f67826c8 f67826d0 f6fac01c 000080d0 [ 22.906000] f6fa3c40 f6fa9800 f67826c0 00000000 7f431025 b014a3e9 b2bd6ab8 00000000 [ 22.906000] 00000000 00000000 00000246 f6fa3c40 f6fa3c40 f5c01ee4 b0187793 b07c11d6 [ 22.906000] Call Trace: [ 22.906000] [] ? __wake_up_bit+0xc/0x37 [ 22.906000] [] ? kmem_cache_alloc+0x53/0x98 [ 22.906000] [] ? sk_prot_alloc+0x1e/0xa7 [ 22.906000] [] ? cap_capable+0xa/0x2b [ 22.906000] [] ? sk_alloc+0x20/0xa3 [ 22.906000] [] ? inet6_create+0x187/0x316 [ 22.906000] [] ? __sock_create+0x15b/0x229 [ 22.906000] [] ? sock_create+0x29/0x2e [ 22.906000] [] ? sys_socket+0x1e/0x42 [ 22.906000] [] ? sys_socketcall+0x56/0x163 [ 22.906000] [] ? resume_userspace+0x8/0x30 [ 22.906000] [] ? sysenter_past_esp+0x78/0xb9 [ 22.906000] ======================= [ 22.906000] Code: bb 00 00 00 8b 45 d4 8b 30 39 c6 75 13 c7 40 50 01 00 00 00 8b 70 10 3b 75 c0 0f 84 94 00 00 00 8b 55 cc 8b 42 38 39 46 10 72 37 <0f> 0b eb fe 8b 0b 89 4d ac 8b 56 14 8b 45 cc 0f af 50 2c 03 56 [ 22.906000] EIP: [] cache_alloc_refill+0xfa/0x431 SS:ESP 0068:f5c01e70 [ 22.906000] Kernel panic - not syncing: Fatal exception [ 22.906000] Pid: 1831, comm: ip6tables-resto Tainted: G D W 2.6.26-tip-03923-g2c96bcb-dirty #3134 Config and full crashlog at: http://redhat.com/~mingo/misc/config-Fri_Jul_18_19_59_56_CEST_2008.bad http://redhat.com/~mingo/misc/crash-2.6.26-tip-03923-g2c96bcb.log Note, the first warning the bootlog is here: [ 10.039500] initcall ip6_queue_init+0x0/0x111 returned 0 after 0 msecs [ 10.039500] calling ip6_tunnel_init+0x0/0x91 [ 10.039500] ------------[ cut here ]------------ [ 10.039500] WARNING: at net/core/dst.c:265 dst_release+0x24/0x2b() [ 10.039500] Modules linked in: [ 10.039500] Pid: 7, comm: ksoftirqd/1 Not tainted 2.6.26-tip-03923-g2c96bcb-dirty #3134 [ 10.040500] [] warn_on_slowpath+0x46/0x6a [ 10.040500] [] ? put_lock_stats+0xd/0x21 [ 10.040500] [] ? __lock_acquire+0x5ba/0x618 [ 10.040500] [] ? put_lock_stats+0xd/0x21 [ 10.040500] [] dst_release+0x24/0x2b [ 10.040500] [] skb_release_all+0x13/0x83 [ 10.041500] [] __kfree_skb+0x10/0x6b [ 10.041500] [] net_tx_action+0x6e/0xee [ 10.041500] [] __do_softirq+0x7a/0xe8 [ 10.041500] [] ? __do_softirq+0x0/0xe8 [ 10.041500] [] do_softirq+0x79/0xba [ 10.041500] [] ksoftirqd+0x88/0x13b [ 10.042500] [] ? ksoftirqd+0x0/0x13b [ 10.042500] [] kthread+0x40/0x67 [ 10.042500] [] ? kthread+0x0/0x67 [ 10.042500] [] kernel_thread_helper+0x7/0x10 [ 10.042500] ======================= [ 10.042500] ---[ end trace 5c376f440d7caa15 ]--- [ 10.042500] Attempt to release alive inet socket f6fac040 [ 10.043500] initcall ip6_tunnel_init+0x0/0x91 returned 0 after 3 msecs [ 10.043500] calling packet_init+0x0/0x3e That does seem to be quite specific. Note, this box never had skb problems and was running hundreds of similar boot tests every day - these appeared in this merge window. The upstream base of 2.6.26-tip-03923-g2c96bcb is v2.6.26-3465-g5b664cb. I'll now check how reproducible this crash is. Ingo -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/