Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S1752733AbbF2X4T (ORCPT <rfc822;w@1wt.eu>);
	Mon, 29 Jun 2015 19:56:19 -0400
Received: from mail.linuxfoundation.org ([140.211.169.12]:54047 "EHLO
	mail.linuxfoundation.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1751080AbbF2X4N (ORCPT
	<rfc822;linux-kernel@vger.kernel.org>);
	Mon, 29 Jun 2015 19:56:13 -0400
Date: Mon, 29 Jun 2015 16:56:12 -0700
From: Greg KH <gregkh@linuxfoundation.org>
To: "long.wanglong" <long.wanglong@huawei.com>
Cc: Petr Mladek <pmladek@suse.cz>, rostedt@goodmis.org, jkosina@suse.cz,
        stable@vger.kernel.org, peifeiyue@huawei.com,
        linux-kernel@vger.kernel.org, paulmck@linux.vnet.ibm.com,
        dzickus@redhat.com, x86@kernel.org, morgan.wang@huawei.com,
        sasha.levin@oracle.com
Subject: Re: [PATCH v2 00/17] [request for stable 3.10 inclusion] x86/nmi:
 Print all cpu stacks from NMI safely
Message-ID: <20150629235612.GB29763@kroah.com>
References: <1432026542-123571-1-git-send-email-long.wanglong@huawei.com>
 <20150519124754.GA12395@pathway.suse.cz>
 <20150520132259.GE2728@pathway.suse.cz>
 <555D370F.2070501@huawei.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <555D370F.2070501@huawei.com>
User-Agent: Mutt/1.5.23+89 (0255b37be491) (2014-03-12)
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 3852
Lines: 95

On Thu, May 21, 2015 at 09:38:23AM +0800, long.wanglong wrote:
> On 2015/5/20 21:22, Petr Mladek wrote:
> > On Tue 2015-05-19 14:57:46, Petr Mladek wrote:
> >> On Tue 2015-05-19 09:08:45, Wang Long wrote:
> >>> This is my backport patch series to Fix the problem(backport to 3.10):
> >>> "
> >>> When trigger_all_cpu_backtrace() is called on x86, it will trigger an
> >>> NMI on each CPU and call show_regs(). But this can lead to a hard lock
> >>> up if the NMI comes in on another printk().
> >>> "
> >>> The solution is described in commit "a9edc88093287183ac934be44f295f183b2c62dd":
> >>> when the NMI triggers, it switches the printk routine for that CPU to call 
> >>> a NMI safe printk function that records the printk in a per_cpu seq_buf 
> >>> descriptor. After all NMIs have finished recording its data, the trace_
> >>> seqs are printed in a safe context.
> >>>
> >>> The solution use "switch printk routine" and "seq_buf" infrastructures, but the
> >>> 3.10 stable have no both of them.
> >>>
> >>> The patch 1-13 backport the "seq_buf" infrastructures. in detail, patch 1, 2
> >>> and 6 only backport "seq_buf" related code.
> >>>
> >>> The patch 14-15 backport the "switch printk routine".
> >>>
> >>> The patch 16-17 is the patch to print all cpu stacks from NMI safely
> >>>
> >>> as discussed in https://lkml.org/lkml/2015/5/13/497, in 3.10 stable, this is 
> >>> the only way to solve the problem and the backport code is a bit more.
> >>>
> >>> v1 -> v2:
> >>>  * fix the indent error.
> >>>  * rebase on 3.10.79
> >>>
> >>> Any thoughts?
> >>
> >> Please, wait with the integration. I am testing it with a storm of
> >> sysrq requests:
> >>
> >>     $> while true ; do echo l >/proc/sysrq-trigger ; done
> >>
> >> with iptables enabled:
> >>
> >>     $> iptables -A INPUT -j LOG --log-prefix "incomming packet:"
> >>
> >> and storm of pings from other machine:
> >>
> >>     $> ping -f <patched-host>
> >>
> >>
> >> The machine somehow freezes. It does not make sense. I am trying to investigate.
> > 
> > OK, it seems that the machine freezes because there are still few
> > messages printed in the NMI context, e.g.:
> > 
> > [ 3080.286277] Uhhuh. NMI received for unknown reason 3d on CPU 12.
> > [ 3637.939276] Uhhuh. NMI received for unknown reason 2d on CPU 13.
> > 
> > I am not exactly sure why I get them on the test machine. But I get
> > such messages from time to time when hammering it by the pings and
> > sysrq-l requests.
> > 
> > I modified vprintk_emit() to do raw_spin_trylock(&logbuf_lock)
> > and do not try to lock console in NMI context. The trylock fails
> > from time to time but it does not longer freeze.
> > 
> > I am going to clean up the vprintk_emit() modification and send it for
> > review.
> > 
> > Anyway, this patch set seems to work as expected. It heavily reduces
> > the risk of NMI/printk-related deadlocks => it is worth having.
> > 
> > Feel free to use the following for the whole patchset (backport):
> > 
> > Reviewed-by: Petr Mladek <pmladek@suse.cz>
> > Tested-by: Petr Mladek <pmladek@suse.cz>
> 
> Hi Greg,
> 
> This patch set is the only way to solve the NMI/printk-related deadlock problems.
> Could you please include them to 3.10 stable?
> 
> Although the code a bit more, most of the code is "seq_buf" infrastructures and
> it does not affect other parts of the kernel.

Yeah, but this is way too much for a -stable kernel.  I suggest that if
a user has this problem, please move to 3.14 or newer kernels, which has
this fixed.  There's too many changes here for me to be confortable
accepting to a -stable kernel, sorry.

greg k-h
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/