Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933080AbaLAXZG (ORCPT ); Mon, 1 Dec 2014 18:25:06 -0500 Received: from mail-qa0-f53.google.com ([209.85.216.53]:33818 "EHLO mail-qa0-f53.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932623AbaLAXZC (ORCPT ); Mon, 1 Dec 2014 18:25:02 -0500 MIME-Version: 1.0 In-Reply-To: <20141201230339.GA20487@ret.masoncoding.com> References: <20141127225637.GA24019@redhat.com> <547b8a45.6e608c0a.20f9.1002@mx.google.com> <547bbe36.48548c0a.105c.779c@mx.google.com> <20141201191431.GA17385@linux.vnet.ibm.com> <547ccf74.a5198c0a.25de.26d9@mx.google.com> <20141201230339.GA20487@ret.masoncoding.com> Date: Mon, 1 Dec 2014 15:25:01 -0800 X-Google-Sender-Auth: I7qUohFu7ZIRhsI9HfZx55TkMdg Message-ID: Subject: Re: frequent lockups in 3.18rc4 From: Linus Torvalds To: Chris Mason , Linus Torvalds , =?UTF-8?Q?D=C3=A2niel_Fraga?= , Dave Jones , Sasha Levin , "Paul E. McKenney" , Linux Kernel Mailing List Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Dec 1, 2014 at 3:08 PM, Chris Mason wrote: > I'm not sure if this is related, but running trinity here, I noticed it > was stuck at 100% system time on every CPU. perf report tells me we are > spending all of our time in spin_lock under the sync system call. > > I think it's coming from contention in the bdi_queue_work() call from > inside sync_inodes_sb, which is spin_lock_bh(). Please do a perf run with -g to get the call chain to make sure.. > I wonder if we're just spinning so hard on this one bh lock that we're > starving the watchdog? If it was that simple, we should see it in the actual soft-lockup stack trace. That said, looking at the bdi_queue_work() function, I don't think you should see any real contention there, although: - spin-lock debugging can make any bad situation about 10x worse by making the spinlocks just that much more horrible from a performance standpoint - the whole "complete(work->done)" thing seems to be pointlessly done inside the spinlock, and that just seems horrible. Do you have a ton of BDI's that might fail that BDI_registered thing? - even the "mod_delayed_work()" is dubious wrt the wb_lock. From what I can tell, the spinlock is supposed to just protect the list. So I think that bdi_queue_work() quite possibly is horribly broken crap and *if* it really is contention on wb_lock, we could rewrite it to not be so bad locking-wise. That said, contention that happens with spinlock debugging enabled really tends to fall under the heading of "that's your own fault". Linus -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/