Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752160AbbKJWnp (ORCPT ); Tue, 10 Nov 2015 17:43:45 -0500 Received: from one.firstfloor.org ([193.170.194.197]:38828 "EHLO one.firstfloor.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751284AbbKJWnm (ORCPT ); Tue, 10 Nov 2015 17:43:42 -0500 Date: Tue, 10 Nov 2015 23:43:40 +0100 From: Andi Kleen To: Andi Kleen Cc: Peter Hurley , gregkh@linuxfoundation.org, linux-serial@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: 4.3 serial driver crashes with console shortly after boot Message-ID: <20151110224340.GA30452@two.firstfloor.org> References: <20151110002939.GG3533@two.firstfloor.org> <56414DE9.20707@hurleysoftware.com> <20151110223957.GI3533@two.firstfloor.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151110223957.GI3533@two.firstfloor.org> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 11647 Lines: 232 On Tue, Nov 10, 2015 at 11:39:57PM +0100, Andi Kleen wrote: > > I've just tried to reproduce this without success on my current > > tree which has some additional patches I just posted this am. They weren't > > intended to fix crashes but they directly impact the area of concern. Could > > you try these three? > > > > [PATCH v2 2/4] n_tty: Ignore all read data when closing > > [PATCH v2 3/4] tty: Abstract and encapsulate tty->closing behavior > > [PATCH v2 4/4] tty: Remove drivers' extra tty_ldisc_flush() > > > Applying the three patches fixes the crash. > I haven't tried to figure out which one did the trick. Actually I was wrong sorry. It still crashes, but now it doesn't hang the system anymore. Here are full oopses: [ 109.350595] BUG: unable to handle kernel NULL pointer dereference at 00000000000001f4 [ 109.358410] IP: [] __uart_start.isra.1+0x1a/0x40 [ 109.364151] PGD 0 [ 109.365216] Oops: 0000 [#1] SMP [ 109.367705] Modules linked in: x86_pkg_temp_thermal crc32c_intel [ 109.373363] CPU: 2 PID: 2957 Comm: kworker/u129:8 Not tainted 4.3.0-dirty #679 [ 109.380206] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS GRNDSDP1.86B.0046.R00.1502111331 02/11/2015 [ 109.390542] Workqueue: events_unbound flush_to_ldisc [ 109.394915] task: ffff88085a2b5c00 ti: ffff880858ad8000 task.ti: ffff880858ad8000 [ 109.402049] RIP: 0010:[] [] __uart_start.isra.1+0x1a/0x40 [ 109.410681] RSP: 0018:ffff880858adbce8 EFLAGS: 00010046 [ 109.415390] RAX: 0000000000000000 RBX: ffffffff81edfd60 RCX: ffffffff817ce300 [ 109.422137] RDX: 0000000000000001 RSI: 0000000000000020 RDI: ffffffff81edfd60 [ 109.428886] RBP: ffff880858adbd08 R08: 0000000000000074 R09: 00000000ffffffff [ 109.435628] R10: ffff880856caa120 R11: 0000000000000074 R12: ffff881059583c00 [ 109.442365] R13: 0000000000000286 R14: ffffc90009c782b0 R15: 0000000000000000 [ 109.449107] FS: 0000000000000000(0000) GS:ffff88085f840000(0000) knlGS:0000000000000000 [ 109.456922] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 109.462116] CR2: 00000000000001f4 CR3: 0000000001af3000 CR4: 00000000001406e0 [ 109.468862] Stack: [ 109.469873] ffffffff813bbe77 ffff881059583c00 ffffc90009c76000 0000000000000074 [ 109.477133] ffff880858adbd18 ffffffff813bbe9e ffff880858adbdc0 ffffffff813a56b9 [ 109.484393] 0000000000015200 ffff881059583cd8 ffff880800000001 ffff880800000074 [ 109.491651] Call Trace: [ 109.493155] [] ? uart_start+0x37/0x50 [ 109.497866] [] uart_flush_chars+0xe/0x10 [ 109.502868] [] n_tty_receive_buf_common+0x6e9/0xc90 [ 109.508938] [] n_tty_receive_buf2+0x14/0x20 [ 109.514232] [] flush_to_ldisc+0xda/0x170 [ 109.519236] [] process_one_work+0x144/0x430 [ 109.524525] [] worker_thread+0x4b/0x4c0 [ 109.529417] [] ? process_one_work+0x430/0x430 [ 109.534892] [] kthread+0xc9/0xe0 [ 109.539111] [] ? flush_kthread_worker+0x70/0x70 [ 109.544798] [] ret_from_fork+0x3f/0x70 [ 109.549602] [] ? flush_kthread_worker+0x70/0x70 [ 109.555271] Code: ff ff 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b bf 90 01 00 00 48 8b 87 a0 00 00 00 48 8b 80 90 00 00 00 80 f4 01 00 00 01 74 01 c3 8b 87 f0 00 00 00 85 c0 75 f5 55 [ 109.579051] RIP [] __uart_start.isra.1+0x1a/0x40 [ 109.584875] RSP [ 109.587537] CR2: 00000000000001f4 [ 109.590008] ---[ end trace 0e4d53c4437868b0 ]--- [ 163.478518] ------------[ cut here ]------------ [ 163.478524] WARNING: CPU: 2 PID: 2957 at /home/ak/lsrc/git/linux-2.6/kernel/watchdog.c:331 watchdog_overflow_callback+0x79/0xa0() [ 163.478526] Watchdog detected hard LOCKUP on cpu 2 [ 163.478528] Modules linked in: x86_pkg_temp_thermal crc32c_intel [ 163.478531] CPU: 2 PID: 2957 Comm: kworker/u129:8 Tainted: G D 4.3.0-dirty #679 [ 163.478532] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS GRNDSDP1.86B.0046.R00.1502111331 02/11/2015 [ 163.478536] Workqueue: events_unbound flush_to_ldisc [ 163.478539] ffffffff81a00b28 ffff88085f845b00 ffffffff81310ce4 ffff88085f845b48 [ 163.478541] ffff88085f845b38 ffffffff810a42b2 ffff88085b9f8000 0000000000000000 [ 163.478543] ffff88085f845c40 ffff88085f845ef8 0000000000000000 ffff88085f845b98 [ 163.478544] Call Trace: [ 163.478552] [] dump_stack+0x44/0x60 [ 163.478557] [] warn_slowpath_common+0x82/0xc0 [ 163.478560] [] warn_slowpath_fmt+0x4c/0x50 [ 163.478562] [] watchdog_overflow_callback+0x79/0xa0 [ 163.478567] [] __perf_event_overflow+0x8c/0x1d0 [ 163.478570] [] perf_event_overflow+0x14/0x20 [ 163.478576] [] intel_pmu_handle_irq+0x1ce/0x430 [ 163.478582] [] perf_event_nmi_handler+0x26/0x40 [ 163.478587] [] nmi_handle+0x7b/0x110 [ 163.478590] [] default_do_nmi+0x40/0x100 [ 163.478592] [] do_nmi+0xe2/0x130 [ 163.478596] [] end_repeat_nmi+0x1a/0x1e [ 163.478602] [] ? native_queued_spin_lock_slowpath+0x15c/0x170 [ 163.478604] [] ? native_queued_spin_lock_slowpath+0x15c/0x170 [ 163.478607] [] ? native_queued_spin_lock_slowpath+0x15c/0x170 [ 163.478612] <> [] _raw_spin_lock_irqsave+0x37/0x40 [ 163.478617] [] serial8250_console_write+0x1ea/0x220 [ 163.478620] [] ? print_prefix+0x50/0x90 [ 163.478623] [] univ8250_console_write+0x26/0x30 [ 163.478627] [] call_console_drivers.constprop.4+0xf2/0x100 [ 163.478630] [] console_unlock+0x301/0x4d0 [ 163.478633] [] vprintk_emit+0x2a4/0x490 [ 163.478636] [] vprintk_default+0x1f/0x30 [ 163.478640] [] printk+0x48/0x50 [ 163.478643] [] print_oops_end_marker+0x2c/0x60 [ 163.478645] [] oops_exit+0x13/0x20 [ 163.478647] [] oops_end+0x7d/0xd0 [ 163.478651] [] no_context+0x10b/0x350 [ 163.478656] [] ? vsnprintf+0x340/0x510 [ 163.478659] [] __bad_area_nosemaphore+0x80/0x1f0 [ 163.478661] [] bad_area_nosemaphore+0x13/0x20 [ 163.478663] [] __do_page_fault+0xa7/0x3e0 [ 163.478665] [] do_page_fault+0x22/0x30 [ 163.478667] [] page_fault+0x28/0x30 [ 163.478671] [] ? __uart_start.isra.1+0x1a/0x40 [ 163.478673] [] ? uart_start+0x37/0x50 [ 163.478676] [] uart_flush_chars+0xe/0x10 [ 163.478679] [] n_tty_receive_buf_common+0x6e9/0xc90 [ 163.478682] [] n_tty_receive_buf2+0x14/0x20 [ 163.478685] [] flush_to_ldisc+0xda/0x170 [ 163.478688] [] process_one_work+0x144/0x430 [ 163.478691] [] worker_thread+0x4b/0x4c0 [ 163.478693] [] ? process_one_work+0x430/0x430 [ 163.478696] [] kthread+0xc9/0xe0 [ 163.478700] [] ? flush_kthread_worker+0x70/0x70 [ 163.478703] [] ret_from_fork+0x3f/0x70 [ 163.478707] [] ? flush_kthread_worker+0x70/0x70 [ 163.478709] ---[ end trace 0e4d53c4437868b1 ]--- [ 178.623346] INFO: rcu_sched detected stalls on CPUs/tasks: [ 178.623351] 2: (71 GPs behind) idle=8d1/140000000000000/0 softirq=826/826 fqs=14905 [ 178.623357] (detected by 33, t=15002 jiffies, g=1537, c=1536, q=11162) [ 178.623358] Task dump for CPU 2: [ 178.623362] kworker/u129:8 R running task 0 2957 2 0x00000008 [ 178.623374] Workqueue: events_unbound flush_to_ldisc [ 178.623378] ffff88085f413400 ffff88085f433600 0000000000000000 ffff88105bac0808 [ 178.623380] ffff880858adbe60 ffffffff810b9684 0000000000000000 ffff88085b7481b0 [ 178.623383] ffff88085f413400 0000000000000088 ffff88085f413418 ffff88085b748180 [ 178.623383] Call Trace: [ 178.623395] [] ? process_one_work+0x144/0x430 [ 178.623398] [] ? worker_thread+0x4b/0x4c0 [ 178.623401] [] ? process_one_work+0x430/0x430 [ 178.623405] [] ? kthread+0xc9/0xe0 [ 178.623409] [] ? flush_kthread_worker+0x70/0x70 [ 178.623420] [] ? ret_from_fork+0x3f/0x70 [ 178.623424] [] ? flush_kthread_worker+0x70/0x70 [ 225.093423] NMI watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [grub2-probe:9298] [ 225.093425] Modules linked in: x86_pkg_temp_thermal crc32c_intel [ 225.093426] CPU: 19 PID: 9298 Comm: grub2-probe Tainted: G D W 4.3.0-dirty #679 [ 225.093427] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS GRNDSDP1.86B.0046.R00.1502111331 02/11/2015 [ 225.093428] task: ffff88105388d080 ti: ffff881056514000 task.ti: ffff881056514000 [ 225.093432] RIP: 0010:[] [] smp_call_function_many+0x1ef/0x240 [ 225.093432] RSP: 0018:ffff881056517d68 EFLAGS: 00000202 [ 225.093433] RAX: 0000000000000003 RBX: 0000000000000040 RCX: 0000000000000002 [ 225.093433] RDX: ffff88085f859960 RSI: 0000000000000040 RDI: ffff88107fa36108 [ 225.093433] RBP: ffff881056517da8 R08: 0000000000000000 R09: ffffffeffff7ffff [ 225.093434] R10: 0000000000000100 R11: 0000000000000206 R12: ffff88107fa36100 [ 225.093434] R13: ffff88107fa36108 R14: ffffffff811e41d0 R15: 0000000000000000 [ 225.093435] FS: 00007fe156fdf800(0000) GS:ffff88107fa20000(0000) knlGS:0000000000000000 [ 225.093435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 225.093436] CR2: 0000003002e42a10 CR3: 0000001057850000 CR4: 00000000001406e0 [ 225.093436] Stack: [ 225.093437] 0000000000000000 00000000000160c0 01ffffff00000001 0000000000000013 [ 225.093438] ffff881056517df8 ffffffff811e41d0 0000000000000000 0000000000000040 [ 225.093455] ffff881056517dd8 ffffffff811040a8 0000000000000000 ffffffff81bfa4d8 [ 225.093455] Call Trace: [ 225.093460] [] ? __brelse+0x30/0x30 [ 225.093461] [] on_each_cpu_mask+0x28/0x60 [ 225.093463] [] ? mark_buffer_async_write+0x20/0x20 [ 225.093464] [] on_each_cpu_cond+0x8c/0xb0 [ 225.093465] [] ? __brelse+0x30/0x30 [ 225.093466] [] invalidate_bh_lrus+0x29/0x30 [ 225.093468] [] invalidate_bdev+0x1e/0x40 [ 225.093473] [] blkdev_ioctl+0x37d/0x690 [ 225.093475] [] block_ioctl+0x3d/0x50 [ 225.093478] [] do_vfs_ioctl+0x285/0x470 [ 225.093481] [] ? SyS_newfstat+0x2a/0x40 [ 225.093483] [] SyS_ioctl+0x41/0x70 [ 225.093485] [] entry_SYSCALL_64_fastpath+0x12/0x71 [ 225.093494] Code: fc 21 00 3b 05 87 76 af 00 89 c1 0f 8d a2 fe ff ff 48 98 49 8b 14 24 48 03 14 c5 c0 9c bf 81 8b 42 18 a8 01 74 ca f3 90 8b 42 18 01 75 f7 eb bf 4c 89 ea 48 89 de 44 89 e7 e8 6d cb 20 00 41 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/