Received: by 2002:a05:6a10:16a7:0:0:0:0 with SMTP id gp39csp1336024pxb; Fri, 13 Nov 2020 10:00:04 -0800 (PST) X-Google-Smtp-Source: ABdhPJyfk9rgQ02eNmLxrQkTjTxQnVzDfRAHmuDUDhZmg4XpFNCgT67O+Hz4pAoVj/ROpQEYMUyF X-Received: by 2002:a17:906:a002:: with SMTP id p2mr2999704ejy.528.1605290404452; Fri, 13 Nov 2020 10:00:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1605290404; cv=none; d=google.com; s=arc-20160816; b=fFt0MsEPVMfwxf1ngt6a6uOIJDPJc3YD/h3KzXORrJ2YnQQN8tADQ4uWQCZsSG70dm AH3jOc4CrpBqUtuuo99Dtx7LsaqnU+dVMfoYvHZ5YgvBfx+eF6fpr2jZfk3ALQzAze5C IEoQ6FPlSn98b3hbPGoUqrVTpKpGZ3epmdiMILkEgH5XhcZp/rjYFKRXFsgWROAsSll8 lTDFfWAwE3IrBlO1v03031YMGf4p6nLvrBLr8qNxxG5sE/NkweSrxP2Pxj6OSwGFcpU9 1R/LahxrdX36J70Jj2lLNK16atOoPxWjtZXa0AYODTJFDWyUwWUUMIOEWZ06wA5aoygU GlIw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:user-agent:in-reply-to:content-disposition :mime-version:references:reply-to:message-id:subject:cc:to:from:date :dkim-signature; bh=S5YvRFRZMxsWgD31cXBQamHkmgAgVS7zwUG1Nz7ZrK0=; b=wo7q6qvgF4g/Q6FgzUQsrh7NYi57Ht+cKCTu6NRw1C1kVejK8FACSgGhiWTlsWDUPU nArAmJdUSqvePJAK6hon8daLU+USKdOkAuJ5Ot6O9aXlUeVYoUj7kD8Z7E8NcoG9fvKN r1c+BDP+kEJhkd/YZhIFojRROAplwZ4jrZPPrB9GXXx8cpmZMdx0UkzUmEbX9gJg/jYq TAPunh1QOqxhwNjUA8BwOdiu+f0bOsSDP71uYjP7HbZasQFIxN28GD/exwMwoPjOkygG EJQ6bjgHMKXwgmpOw/9vacNVyYUQeS+nuLQu9envbd5ofBg8eyi7r5v/DP8k5GDHR8z7 5FsQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=l+E3WoS3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id q24si3608111eju.589.2020.11.13.09.59.40; Fri, 13 Nov 2020 10:00:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=l+E3WoS3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726146AbgKMR54 (ORCPT + 99 others); Fri, 13 Nov 2020 12:57:56 -0500 Received: from mail.kernel.org ([198.145.29.99]:54392 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725959AbgKMR5z (ORCPT ); Fri, 13 Nov 2020 12:57:55 -0500 Received: from paulmck-ThinkPad-P72.home (50-39-104-11.bvtn.or.frontiernet.net [50.39.104.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 9F5A6208D5; Fri, 13 Nov 2020 17:57:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1605290274; bh=SPs6ifdr8H4sMKD7ax8Sx6hzwFG4f+SmdudJwla/AAk=; h=Date:From:To:Cc:Subject:Reply-To:References:In-Reply-To:From; b=l+E3WoS3zzxVRAuAt1Dro8+KxAduqsliRUf/i4u9RcbveIlUEvlzRAYv0H4v0t5TM S7TXa5flyp+i/Gg7uImRgq36p+Cxphb4t0TdcayxoIm3j1qD6DL8Kja1+NAfJSMLEL VS9Y3NA3dfrIXdyc1vkm5e3GlMNmW+r3b8mWsX9M= Received: by paulmck-ThinkPad-P72.home (Postfix, from userid 1000) id 4C48D35212DC; Fri, 13 Nov 2020 09:57:54 -0800 (PST) Date: Fri, 13 Nov 2020 09:57:54 -0800 From: "Paul E. McKenney" To: Marco Elver Cc: Steven Rostedt , Anders Roxell , Andrew Morton , Alexander Potapenko , Dmitry Vyukov , Jann Horn , Mark Rutland , Linux Kernel Mailing List , Linux-MM , kasan-dev , rcu@vger.kernel.org, Peter Zijlstra , Tejun Heo , Lai Jiangshan Subject: Re: [PATCH] kfence: Avoid stalling work queue task without allocations Message-ID: <20201113175754.GA6273@paulmck-ThinkPad-P72> Reply-To: paulmck@kernel.org References: <20201111133813.GA81547@elver.google.com> <20201111130543.27d29462@gandalf.local.home> <20201111182333.GA3249@paulmck-ThinkPad-P72> <20201111183430.GN517454@elver.google.com> <20201111192123.GB3249@paulmck-ThinkPad-P72> <20201111202153.GT517454@elver.google.com> <20201112001129.GD3249@paulmck-ThinkPad-P72> <20201112161439.GA2989297@elver.google.com> <20201112175406.GF3249@paulmck-ThinkPad-P72> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20201112175406.GF3249@paulmck-ThinkPad-P72> User-Agent: Mutt/1.9.4 (2018-02-28) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Nov 12, 2020 at 09:54:06AM -0800, Paul E. McKenney wrote: > On Thu, Nov 12, 2020 at 05:14:39PM +0100, Marco Elver wrote: [ . . . ] > > | [ 334.160218] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 15s! > > It might be instructive to cause this code to provoke a backtrace. > I suggest adding something like "trigger_single_cpu_backtrace(cpu)" > in kernel/workqueue.c's function named wq_watchdog_timer_fn() > somewhere within its "if" statement that is preceded with the "did we > stall?" comment. Or just search for "BUG: workqueue lockup - pool" > within kernel/workqueue.c. And I did get a small but unexpected gift of time, so here is an (untested) patch. Thanx, Paul ------------------------------------------------------------------------ diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 437935e..f3d4ff7 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -5792,6 +5792,7 @@ static void wq_watchdog_timer_fn(struct timer_list *unused) pr_cont_pool_info(pool); pr_cont(" stuck for %us!\n", jiffies_to_msecs(jiffies - pool_ts) / 1000); + trigger_single_cpu_backtrace(cpu); } }