Received: by 2002:a05:6a10:16a7:0:0:0:0 with SMTP id gp39csp1498933pxb; Fri, 20 Nov 2020 10:59:47 -0800 (PST) X-Google-Smtp-Source: ABdhPJyssfsIXoDj/emL8OaoSVm38n/ErgeVBpzVvwBj4q6B0Qd5IAjF5LGdt0xfHbuMm+ak0bev X-Received: by 2002:a17:906:8559:: with SMTP id h25mr11360161ejy.173.1605898786976; Fri, 20 Nov 2020 10:59:46 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1605898786; cv=none; d=google.com; s=arc-20160816; b=CQcWjokdvJngcPyRA3ya7vKbWzry+bAcdPK85SWfBWsaWaZ+POK0ZT9EL1gwmWqE0V 4hY2nKl1/x7KLSPeQHiKjyQHxeLICk8X+Orjn+jsQMe3UAyaxuD152mIeowGNvEmVmyW ysDDZyjGO14cCZd8GtTmwcFzMCrAT3yb9aPrb1vJXbeRZRXTr/vxneiocEWjDnpZYyXD 8Xqy7jBqPWILBNnqU1SK2qUrBVBf8C0CqSIooMWcAALy1tZBDhCQ8ljUVP795F5/JT9P /vUuYxuhXiOa7txeC2BUbXcAkwjmMuoWD+u3X2vfMCGzbJDaELvReCxLNYxNGuI+2VwL rJHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:user-agent:in-reply-to:content-disposition :mime-version:references:reply-to:message-id:subject:cc:to:from:date :dkim-signature; bh=wU6gkQj7TSeLw0/pJ9KekwpgvUr4XxwhC4H75OmSIpc=; b=LvUvsPguFCnaldZ68O1KB9+wrcKyYdrDBHIp1bn80Rh8/FcYDGoOKuyfR7w96CcVKL cgGM04iLg8x3+ygf5RIFdg/HDCaI2hl4o8NghmpO/8P+4wvlRWna88KhWLo9rcnpUMKd cjHNarIi9XOgtRzuvjMa+2iRrcjIoKesDl+lZYYlWK8d31/DLU5pjPSgBefnWYw7WLDQ NSbBHcQEOs2P3VgvD59/kHKS+rvsIEvAF+vU6TPeVa6D9oxsPr3lmlM1q8UcTyMZE7bJ DEZNkr7RrobSJHE9AXEUH93hUycINUi8bPDivHj3K83kIdrgaAvUdfebLahmDW4xGK0q BTzw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=wJPsspjB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id b14si2111826ejc.581.2020.11.20.10.59.24; Fri, 20 Nov 2020 10:59:46 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=wJPsspjB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730160AbgKTS57 (ORCPT + 99 others); Fri, 20 Nov 2020 13:57:59 -0500 Received: from mail.kernel.org ([198.145.29.99]:36238 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730123AbgKTS57 (ORCPT ); Fri, 20 Nov 2020 13:57:59 -0500 Received: from paulmck-ThinkPad-P72.home (50-39-104-11.bvtn.or.frontiernet.net [50.39.104.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 4C01222464; Fri, 20 Nov 2020 18:57:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1605898678; bh=XFTcuGAvqGRf8jCZ+ckM7jc3MC3vlz4De93JWXGu/Ro=; h=Date:From:To:Cc:Subject:Reply-To:References:In-Reply-To:From; b=wJPsspjBAtc2Maj5KcKF0b+v14djW14eYbsjEMFBRVZNoUZEi2q5wc0RTCHU1qYt0 jQ6XGn+h3U1TMAK7Hh1puIJu2qQCcqYlWwMKw10ke/QJPL7eFZMkuHm4oninm8u0Rt WxdDT/4XnMTOFNIZZF4Uy4osU6s31bBTiWBAv5K4= Received: by paulmck-ThinkPad-P72.home (Postfix, from userid 1000) id E2B203522637; Fri, 20 Nov 2020 10:57:57 -0800 (PST) Date: Fri, 20 Nov 2020 10:57:57 -0800 From: "Paul E. McKenney" To: Mark Rutland Cc: Marco Elver , Steven Rostedt , Anders Roxell , Andrew Morton , Alexander Potapenko , Dmitry Vyukov , Jann Horn , Linux Kernel Mailing List , Linux-MM , kasan-dev , rcu@vger.kernel.org, Peter Zijlstra , Tejun Heo , Lai Jiangshan , linux-arm-kernel@lists.infradead.org Subject: Re: linux-next: stall warnings and deadlock on Arm64 (was: [PATCH] kfence: Avoid stalling...) Message-ID: <20201120185757.GL1437@paulmck-ThinkPad-P72> Reply-To: paulmck@kernel.org References: <20201119151409.GU1437@paulmck-ThinkPad-P72> <20201119170259.GA2134472@elver.google.com> <20201119184854.GY1437@paulmck-ThinkPad-P72> <20201119193819.GA2601289@elver.google.com> <20201119213512.GB1437@paulmck-ThinkPad-P72> <20201120141928.GB3120165@elver.google.com> <20201120143928.GH1437@paulmck-ThinkPad-P72> <20201120152200.GD2328@C02TD0UTHF1T.local> <20201120173824.GJ1437@paulmck-ThinkPad-P72> <20201120180206.GF2328@C02TD0UTHF1T.local> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20201120180206.GF2328@C02TD0UTHF1T.local> User-Agent: Mutt/1.9.4 (2018-02-28) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Nov 20, 2020 at 06:02:06PM +0000, Mark Rutland wrote: > On Fri, Nov 20, 2020 at 09:38:24AM -0800, Paul E. McKenney wrote: > > On Fri, Nov 20, 2020 at 03:22:00PM +0000, Mark Rutland wrote: > > > On Fri, Nov 20, 2020 at 06:39:28AM -0800, Paul E. McKenney wrote: > > > > On Fri, Nov 20, 2020 at 03:19:28PM +0100, Marco Elver wrote: > > > > > I found that disabling ftrace for some of kernel/rcu (see below) solved > > > > > the stalls (and any mention of deadlocks as a side-effect I assume), > > > > > resulting in successful boot. > > > > > > > > > > Does that provide any additional clues? I tried to narrow it down to 1-2 > > > > > files, but that doesn't seem to work. > > > > > > > > There were similar issues during the x86/entry work. Are the ARM guys > > > > doing arm64/entry work now? > > > > > > I'm currently looking at it. I had been trying to shift things to C for > > > a while, and right now I'm trying to fix the lockdep state tracking, > > > which is requiring untangling lockdep/rcu/tracing. > > > > > > The main issue I see remaining atm is that we don't save/restore the > > > lockdep state over exceptions taken from kernel to kernel. That could > > > result in lockdep thinking IRQs are disabled when they're actually > > > enabled (because code in the nested context might do a save/restore > > > while IRQs are disabled, then return to a context where IRQs are > > > enabled), but AFAICT shouldn't result in the inverse in most cases since > > > the non-NMI handlers all call lockdep_hardirqs_disabled(). > > > > > > I'm at a loss to explaim the rcu vs ftrace bits, so if you have any > > > pointers to the issuies ween with the x86 rework that'd be quite handy. > > > > There were several over a number of months. I especially recall issues > > with the direct-from-idle execution of smp_call_function*() handlers, > > and also with some of the special cases in the entry code, for example, > > reentering the kernel from the kernel. This latter could cause RCU to > > not be watching when it should have been or vice versa. > > Ah; those are precisely the cases I'm currently fixing, so if we're > lucky this is an indirect result of one of those rather than a novel > source of pain... Here is hoping! > > I would of course be most aware of the issues that impinged on RCU > > and that were located by rcutorture. This is actually not hard to run, > > especially if the ARM bits in the scripting have managed to avoid bitrot. > > The "modprobe rcutorture" approach has fewer dependencies. Either way: > > https://paulmck.livejournal.com/57769.html and later posts. > > That is a very good idea. I'd been relying on Syzkaller to tickle the > issue, but the torture infrastructure is a much better fit for this > problem. I hadn't realise how comprehensive the scripting was, thanks > for this! But why not both rcutorture and Syzkaller? ;-) > I'll see about giving that a go once I have the irq-from-idle cases > sorted, as those are very obviously broken if you hack > trace_hardirqs_{on,off}() to check that RCU is watching. Sounds good! Thanx, Paul