Received: by 2002:a05:6a10:f347:0:0:0:0 with SMTP id d7csp1898622pxu; Tue, 24 Nov 2020 11:32:32 -0800 (PST) X-Google-Smtp-Source: ABdhPJwbL5rqcy3nwJ9JSyHGkP798n5QcCxD4tuxAB5DK/FZ7ZxB5CqOuXZaJOTKJP29c6gVWX9S X-Received: by 2002:a50:8e0e:: with SMTP id 14mr17581edw.171.1606246351933; Tue, 24 Nov 2020 11:32:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1606246351; cv=none; d=google.com; s=arc-20160816; b=RYOdjQVwe1XCqi0Hf32/UYZ+DmNyNDLrZu80/k2w+JzTPytMp6g6SRuvBD+4SiuV3s WbNEI39kx13IdeuFH+EwD79cne06l0jFTzzSiXXvmWy11hLQ9UPFn1B9+Der7D5PXNzR yMYEDbFrake2P7gDaxZpi+hnXMGUHbQE8s+wLUPn4fG+GlT69GiG6FtLsURHh1TsZc8O bIEbXrc5CbD9UQ7B2mJxg3r2ktaOLIgG/d9BGYhnUrxjQ27l/A2E1a+V3ApSajVnPjmV UUrNEYQjtHQGx4u9CPcs7K3RTgE/8tTA44Tw6dt3hpwcvs6lyf4fypADjJLlVQNsgZ7Q grVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date; bh=MTu7JPVheZSrnStETROp4LHaogLrg9LGx+W88Unjdhk=; b=WfwxURcI6zjQPSRFdwLpOTyxtvQfs7kTNu9d8W60zzaFPkPHcMq6FVjRxwVSB3p7zD guLsaV0m1ZspVMbEZ4sId085xxor3nYE8uO00hObbEdkd+mHZ2KGPCqD+I/7E3F4q5Uk d27d78wIuk3UYpuFGEl+eO87/WwQc5fqDEeTbK+KAGWDu15V35SHck97Wl8OybtcCmZm 8BpW72ow/NeFRTei/8m+FiufFrKdCMiA+cHxXuBXz+NjtuUZ4P6v28b45DkRxZcKBgzi MN0/hoculIuhi7Yvq54+drCLSMzbi4OqU2VtCaPzzJH4YXFQgF/WhaGFqgSKxRpWDum5 MOtQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id e24si5263413ejh.235.2020.11.24.11.32.08; Tue, 24 Nov 2020 11:32:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728153AbgKXTan (ORCPT + 99 others); Tue, 24 Nov 2020 14:30:43 -0500 Received: from foss.arm.com ([217.140.110.172]:50640 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726306AbgKXTam (ORCPT ); Tue, 24 Nov 2020 14:30:42 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 06E501396; Tue, 24 Nov 2020 11:30:41 -0800 (PST) Received: from C02TD0UTHF1T.local (unknown [10.57.26.92]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E2B6E3F70D; Tue, 24 Nov 2020 11:30:36 -0800 (PST) Date: Tue, 24 Nov 2020 19:30:34 +0000 From: Mark Rutland To: Marco Elver Cc: Will Deacon , "Paul E. McKenney" , Steven Rostedt , Anders Roxell , Andrew Morton , Alexander Potapenko , Dmitry Vyukov , Jann Horn , Linux Kernel Mailing List , Linux-MM , kasan-dev , rcu@vger.kernel.org, Peter Zijlstra , Tejun Heo , Lai Jiangshan , linux-arm-kernel@lists.infradead.org, boqun.feng@gmail.com, tglx@linutronix.de Subject: Re: linux-next: stall warnings and deadlock on Arm64 (was: [PATCH] kfence: Avoid stalling...) Message-ID: <20201124193034.GB8957@C02TD0UTHF1T.local> References: <20201119151409.GU1437@paulmck-ThinkPad-P72> <20201119170259.GA2134472@elver.google.com> <20201119184854.GY1437@paulmck-ThinkPad-P72> <20201119193819.GA2601289@elver.google.com> <20201119213512.GB1437@paulmck-ThinkPad-P72> <20201119225352.GA5251@willie-the-truck> <20201120103031.GB2328@C02TD0UTHF1T.local> <20201120140332.GA3120165@elver.google.com> <20201123193241.GA45639@C02TD0UTHF1T.local> <20201124140310.GA811510@elver.google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20201124140310.GA811510@elver.google.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Nov 24, 2020 at 03:03:10PM +0100, Marco Elver wrote: > On Mon, Nov 23, 2020 at 07:32PM +0000, Mark Rutland wrote: > > On Fri, Nov 20, 2020 at 03:03:32PM +0100, Marco Elver wrote: > > > On Fri, Nov 20, 2020 at 10:30AM +0000, Mark Rutland wrote: > > > > On Thu, Nov 19, 2020 at 10:53:53PM +0000, Will Deacon wrote: > > > > > FWIW, arm64 is known broken wrt lockdep and irq tracing atm. Mark has been > > > > > looking at that and I think he is close to having something workable. > > > > > > > > > > Mark -- is there anything Marco and Paul can try out? > > > > > > > > I initially traced some issues back to commit: > > > > > > > > 044d0d6de9f50192 ("lockdep: Only trace IRQ edges") > > > > > > > > ... and that change of semantic could cause us to miss edges in some > > > > cases, but IIUC mostly where we haven't done the right thing in > > > > exception entry/return. > > > > > > > > I don't think my patches address this case yet, but my WIP (currently > > > > just fixing user<->kernel transitions) is at: > > > > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/irq-fixes > > > > > > > > I'm looking into the kernel<->kernel transitions now, and I know that we > > > > mess up RCU management for a small window around arch_cpu_idle, but it's > > > > not immediately clear to me if either of those cases could cause this > > > > report. > > > > > > Thank you -- I tried your irq-fixes, however that didn't seem to fix the > > > problem (still get warnings and then a panic). :-/ > > > > I've just updated that branch with a new version which I hope covers > > kernel<->kernel transitions too. If you get a chance, would you mind > > giving that a spin? > > > > The HEAD commit should be: > > > > a51334f033f8ee88 ("HACK: check IRQ tracing has RCU watching") > > Thank you! Your series appears to work and fixes the stalls and > deadlocks (3 trials)! Thanks for testing! I'm glad that appears to work, as it suggests there's not another massive problem lurking in this area. While cleaning/splitting that up today, I spotted a couple of new problems I introduced, and I'm part-way through sorting that out, but it's not quite ready today after all. :/ Fingers crossed for tomorrow... > I noticed there are a bunch of warnings in the log > that might be relevant (see attached). > [ 91.184432] ============================= > [ 91.188301] WARNING: suspicious RCU usage > [ 91.192316] 5.10.0-rc4-next-20201119-00002-g51c2bf0ac853 #25 Tainted: G W > [ 91.197536] ----------------------------- > [ 91.201431] kernel/trace/trace_preemptirq.c:78 RCU not watching trace_hardirqs_off()! > [ 91.206546] > [ 91.206546] other info that might help us debug this: > [ 91.206546] > [ 91.211790] > [ 91.211790] rcu_scheduler_active = 2, debug_locks = 0 > [ 91.216454] RCU used illegally from extended quiescent state! > [ 91.220890] no locks held by swapper/0/0. > [ 91.224712] > [ 91.224712] stack backtrace: > [ 91.228794] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 5.10.0-rc4-next-20201119-00002-g51c2bf0ac853 #25 > [ 91.234877] Hardware name: linux,dummy-virt (DT) > [ 91.239032] Call trace: > [ 91.242587] dump_backtrace+0x0/0x240 > [ 91.246500] show_stack+0x34/0x88 > [ 91.250295] dump_stack+0x140/0x1bc > [ 91.254159] lockdep_rcu_suspicious+0xe4/0xf8 > [ 91.258332] trace_hardirqs_off+0x214/0x330 > [ 91.262462] trace_graph_return+0x1ac/0x1d8 > [ 91.266564] ftrace_return_to_handler+0xa4/0x170 > [ 91.270809] return_to_handler+0x1c/0x38 > [ 91.274826] default_idle_call+0x94/0x38c > [ 91.278869] do_idle+0x240/0x290 > [ 91.282633] rest_init+0x1e8/0x2dc > [ 91.286529] arch_call_rest_init+0x1c/0x28 > [ 91.290585] start_kernel+0x638/0x670 Hmm... I suspect that arch_cpu_idle() is being traced here, and I reckon we have to mark that and its callees as noinstr, since it doesn't seem sane to have ftrace check whether RCU is watching for every function call. Maybe Paul or Steve can correct me. ;) If you still have the binary lying around, can you check whether default_idle_call+0x94/0x38c is just after the call to arch_cpu_idle()? If you could dump the asm around that, along with whatever faddr2line tells you, that'd be a great help. This looks like it should be reproducible, so I'll enable the boot-time self tests in my kernel and check whether I see the above too. Thanks, Mark.