Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753279AbbGVX3w (ORCPT ); Wed, 22 Jul 2015 19:29:52 -0400 Received: from mail-pa0-f47.google.com ([209.85.220.47]:33244 "EHLO mail-pa0-f47.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753189AbbGVX3u (ORCPT ); Wed, 22 Jul 2015 19:29:50 -0400 Date: Wed, 22 Jul 2015 16:29:46 -0700 From: =?iso-8859-1?Q?J=F6rn?= Engel To: Andrew Morton Cc: Spencer Baugh , Don Zickus , Ulrich Obergfell , Ingo Molnar , Andrew Jones , chai wen , Chris Metcalf , Stephane Eranian , open list , Spencer Baugh , Joern Engel Subject: Re: [PATCH] soft lockup: kill realtime threads before panic Message-ID: <20150722232946.GA18432@Sligo.logfs.org> References: <1437516477-30554-5-git-send-email-sbaugh@catern.com> <20150722155436.04d66934cd423107b810f2b1@linux-foundation.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20150722155436.04d66934cd423107b810f2b1@linux-foundation.org> User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3065 Lines: 85 On Wed, Jul 22, 2015 at 03:54:36PM -0700, Andrew Morton wrote: > On Tue, 21 Jul 2015 15:07:57 -0700 Spencer Baugh wrote: > > > From: Joern Engel > > > > We have observed cases where the soft lockup detector triggered, but no > > kernel bug existed. Instead we had a buggy realtime thread that > > monopolized a cpu. So let's kill the responsible party and not panic > > the entire system. > > > > ... > > > > --- a/kernel/watchdog.c > > +++ b/kernel/watchdog.c > > @@ -428,7 +428,10 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) > > } > > > > add_taint(TAINT_SOFTLOCKUP, LOCKDEP_STILL_OK); > > - if (softlockup_panic) > > + if (rt_prio(current->prio)) { > > + pr_emerg("killing realtime thread\n"); > > + send_sig(SIGILL, current, 0); > > Why choose SIGILL? It is a random signal that happens to generate a stacktrace in userspace. > > + } else if (softlockup_panic) > > panic("softlockup: hung tasks"); > > __this_cpu_write(soft_watchdog_warn, true); > > But what about a non-buggy realtime thread which happens to > occasionally spend 15 seconds doing stuff? > > Old behaviour: kernel blurts a softlockup message, everything keeps running. > > New behaviour: thread gets killed, plane crashes. > > > Possibly a better approach would be to only kill the thread if > softlockup_panic was set, because the system is going down anyway. > > Also, perhaps some users would prefer that the kernel simply suppress > the softlockup warning in this situation, rather than killing stuff! > > > > > Really, what you're trying to implement here is a watchdog for runaway > realtime threads. And that sounds a worthy project but it's a rather > separate thing from the softlockup detector. A realtime thread > watchdog feature might have things as > > - timeout duration separately configurable from softlockup > > - enabled independently from sotflockup: people might want one and > not the other. > > - configurable signal, perhaps? > > Now, the *implementation* of the realtime thread watchdog may well > share code with the softlockup detector. But from a > conceptual/configuration/documentation point of view, it's a separate > thing, no? Agreed. We needed this patch exactly once and it is a rather quick hack that yielded the necessary results. Realtime threads were well-behaved since and the patch has seen zero polish as a result. I think it is better to drop the patch for now. If someone else keeps running into the same issue, it might be a starting point for a better implementation. They will find it in list archives. J?rn -- I can say that I spend most of my time fixing bugs even if I have lots of new features to implement in mind, but I give bugs more priority. -- Andrea Arcangeli, 2000 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/