Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933463AbcKVTXS (ORCPT ); Tue, 22 Nov 2016 14:23:18 -0500 Received: from mail-io0-f169.google.com ([209.85.223.169]:33707 "EHLO mail-io0-f169.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S933288AbcKVTXB (ORCPT ); Tue, 22 Nov 2016 14:23:01 -0500 MIME-Version: 1.0 In-Reply-To: References: <352443968.11059.1479836663353.JavaMail.zimbra@efficios.com> From: Eric Dumazet Date: Tue, 22 Nov 2016 11:22:47 -0800 Message-ID: Subject: Re: Kernel BUG_ON in stable 4.8 To: Mathieu Desnoyers Cc: Alexander Duyck , Willem de Bruijn , Alexei Starovoitov , "David S. Miller" , Greg Kroah-Hartman , Francis Deslauriers , linux-kernel Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4577 Lines: 85 On Tue, Nov 22, 2016 at 9:57 AM, Eric Dumazet wrote: > On Tue, Nov 22, 2016 at 9:44 AM, Mathieu Desnoyers > wrote: >> ----- On Nov 22, 2016, at 12:01 PM, Francis Deslauriers francis.deslauriers@efficios.com wrote: >> >>> Hi Mathieu, >>> >>> Here is a description of the kernel BUG_ON I have encountered. This bug was >>> triggered by our continuous integration system tracking the stable branches. I >>> was only able to reproduce it on our Lava baremetal x86_64 worker. Running the >>> same kernel image on our KVM worker did not trigger the bug. >>> >>> The bug occurs at boot time and I believe it's during the configuration of one >>> of the network cards. >>> >>> See attached the .config used. >>> See attached the dmesg output of the crash containing the kernel panic >>> output(around line 885 of dmesg.txt) >> >> Hi guys, >> >> Upstream commit 34fad54c2 "net: __skb_flow_dissect() must cap its return value" >> triggers this BUG_ON at boot up on our testing machines. We have observed the >> BUG_ON on v4.9-rc6, as well as stable kernels v4.8.10 and v4.4.34. >> >> Relevant bits: >> >> [ 16.841793] kernel BUG at ./include/linux/skbuff.h:1927! >> [ 16.847101] invalid opcode: 0000 [#1] SMP >> [ 16.851101] Modules linked in: >> [ 16.854172] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.8.10 #1 >> [ 16.860081] Hardware name: Supermicro SYS-6018R-TDW/X10DDW-i, BIOS 1.0a 01/14/2015 >> [ 16.867637] task: ffffffff8220d500 task.stack: ffffffff82200000 >> [ 16.873550] RIP: 0010:[] [] eth_type_trans+0xc9/0x110 >> [ 16.881826] RSP: 0000:ffff88103fc03df0 EFLAGS: 00010297 >> [ 16.887130] RAX: 0000000000000148 RBX: ffff881037b2b5c0 RCX: 0000000000001073 >> [ 16.894254] RDX: ffff88103221fdc0 RSI: ffff881038358000 RDI: ffff881037c5bb00 >> [ 16.901377] RBP: ffff88103fc03df8 R08: 0000000000000001 R09: 0000000000000800 >> [ 16.908503] R10: ffff88103221fec0 R11: ffffea0040de7780 R12: ffff881037c5bb00 >> [ 16.915625] R13: ffff881037458000 R14: 0000000000000156 R15: ffff881038358000 >> [ 16.922751] FS: 0000000000000000(0000) GS:ffff88103fc00000(0000) knlGS:0000000000000000 >> [ 16.930829] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [ 16.936565] CR2: ffff88207ffff000 CR3: 0000000002206000 CR4: 00000000001406f0 >> [ 16.943689] Stack: >> [ 16.945702] ffffffff815c098f ffff88103fc03e90 ffffffff8181c651 ffff88103fc0dc20 >> [ 16.953153] 0000000000000000 0000000000000020 0000000000000002 ffff88103fc03e38 >> [ 16.960607] ffffea0040de7780 ffffea0000000008 ffffffff00000001 ffffea0040de7780 >> [ 16.968061] Call Trace: >> [ 16.970507] >> [ 16.972432] [] ? swiotlb_sync_single_for_device+0xf/0x20 >> [ 16.979575] [] igb_poll+0x691/0xe60 >> [ 16.984714] [] net_rx_action+0x1bb/0x2f0 >> [ 16.990291] [] __do_softirq+0xf6/0x280 >> [ 16.995689] [] irq_exit+0xdc/0xf0 >> [ 17.000648] [] do_IRQ+0x54/0xd0 >> [ 17.005433] [] common_interrupt+0x8c/0x8c >> [ 17.011083] >> [ 17.013014] [] ? mwait_idle+0x76/0x170 >> [ 17.018596] [] arch_cpu_idle+0xf/0x20 >> [ 17.023907] [] default_idle_call+0x2a/0x40 >> [ 17.029648] [] cpu_startup_entry+0x29a/0x300 >> [ 17.035561] [] rest_init+0x77/0x80 >> [ 17.040608] [] start_kernel+0x40b/0x418 >> [ 17.046091] [] ? early_idt_handler_array+0x120/0x120 >> [ 17.052702] [] x86_64_start_reservations+0x2a/0x2c >> [ 17.059132] [] x86_64_start_kernel+0x13b/0x14a >> [ 17.065215] Code: 00 04 00 00 c9 c3 48 33 86 58 03 00 00 48 c1 e0 10 48 85 c0 0f b6 87 80 00 00 00 75 25 83 e0 f8 83 c8 01 88 87 80 00 00 00 eb 9c <0f> 0b 0f b6 87 80 00 00 00 83 e0 f8 83 c8 03 88 87 80 00 00 00 >> [ 17.085165] RIP [] eth_type_trans+0xc9/0x110 >> [ 17.091094] RSP >> [ 17.094586] ---[ end trace e233c88f3b369632 ]--- >> [ 17.101432] Kernel panic - not syncing: Fatal exception in interrupt >> [ 17.107790] Kernel Offset: disabled >> [ 17.113484] ---[ end Kernel panic - not syncing: Fatal exception in interrupt >> >> Do you have clues on what is going on ? >> >> Thanks, >> >> Mathieu >> >> > > Under investigation. Look around other threads. Thanks. Probable fix is : https://patchwork.ozlabs.org/patch/697891/