Received: by 2002:a05:6358:5282:b0:b5:90e7:25cb with SMTP id g2csp3323472rwa; Tue, 23 Aug 2022 02:50:40 -0700 (PDT) X-Google-Smtp-Source: AA6agR402xlpw2fI2+x70AfvUqp3ymdn5mJpUDVx5KOkN6Z9TzZmuna2tlHOvylx4WR+AGa+GiYY X-Received: by 2002:a05:6402:43cf:b0:446:b290:ea84 with SMTP id p15-20020a05640243cf00b00446b290ea84mr2964151edc.119.1661248240293; Tue, 23 Aug 2022 02:50:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1661248240; cv=none; d=google.com; s=arc-20160816; b=cVweIB2AZh4CmYVac/pCsrvvNB0a9W3pfLEmhWhCOhFH08+RlThl2JBFy0gh93rodK YuCfuR0JeaMVpcl/iKx1La+b0KdSKYIWjE4hzZUfSu1I4XdC5XwdlaROxKMmm8/QbFCH 8oz0+frNmxyvTQwsuuZF6Z4D6wqHo/JYnqIYL5a07Tw2qTmawvLsUEpV9hWSFC7wToU9 b+zaPAlWwzKlwek+P1zRVKrFjxC98QKKaioHxtmntlu+8Wzo0Qgb3AfTBpk7urUslRkG yJkQ6meLNVmoOJvaXmgigZ+CsCIQDiNkBvNUej5qXiU3XtvX/8r7qaNBgRIGlSy2uodR BPXw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:dkim-signature; bh=jzL+tYVj4e9+zENm5e4p359zO1COy5i/CDx3a5RuXi0=; b=QNMDZirwAM3d4sFfcp7eeK53sc5j761yJDI+Oahas0qd1PbEu4miFFPLpwRRPY0MU0 X5GVPskr6KIayiaUiiOOlh0Ogq+o/tXASidHZBPWuQI5+V/4wdRA8zLQ7Btf/i7UtkYh qxub5dCja/BeElKKbDEfHGjkvibsLn2c8QRHFVM0DfUJO4oJLgfvddPv1O+ewA/htLid VPLW/d2fmrinLUSffjhn0FaN6TB/7GdDwKgGQ86wNye7JqMQSROsHByfUzw6AhCuzb0o /R64oQbqEbKPJbE49LiIp4Sy15hOS7b+jO9+kgYyuig1rpKnM8XRerF0vuqVpqHUzY0i PE0Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ellerman.id.au header.s=201909 header.b="DWH/riAm"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 16-20020a508750000000b004461cce7560si1481220edv.280.2022.08.23.02.50.13; Tue, 23 Aug 2022 02:50:40 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@ellerman.id.au header.s=201909 header.b="DWH/riAm"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240422AbiHWJXK (ORCPT + 99 others); Tue, 23 Aug 2022 05:23:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37516 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350134AbiHWJVa (ORCPT ); Tue, 23 Aug 2022 05:21:30 -0400 Received: from gandalf.ozlabs.org (gandalf.ozlabs.org [150.107.74.76]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 777A48C456; Tue, 23 Aug 2022 01:34:36 -0700 (PDT) Received: from authenticated.ozlabs.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.ozlabs.org (Postfix) with ESMTPSA id 4MBjDH5BJcz4xYS; Tue, 23 Aug 2022 18:33:19 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ellerman.id.au; s=201909; t=1661243600; bh=jzL+tYVj4e9+zENm5e4p359zO1COy5i/CDx3a5RuXi0=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=DWH/riAmZijXBMyFpMOiCjZL/u14KNbC33rDY2XPlmwRTlIHFZEyB3l4hZ6CutX2M /hxh+Z+F6vNEwwgMah0o1MIoBK/i6qQfhDYcu9Ac4BZf6IxKGhWAcJIfsROANYVH0M 2cJCD7cGrgSm+6Hle+5F8DQiDlE3nNy4QRjFvW+n5WL+2Nkq4AKPDtSCDwkc9uk8QA KEzBaddNdy74sR7hw0/y6OjlExq3oBBYjU97e95jCsGLYAR20kqphSEuAqjx6FngzL qNIH2G9b0IyZaSi49GgtXATGN0SmKeTKjGlFe7UyPR4YxQbjQdI+Z1PfSv2CDedkjL cs//ozxWcigLA== From: Michael Ellerman To: Zhouyi Zhou , npiggin@gmail.com, christophe.leroy@csgroup.eu, atrajeev@linux.vnet.ibm.com, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, lance@osuosl.org, paulmck@kernel.org, rcu@vger.kernel.org Cc: Zhouyi Zhou Subject: Re: [PATCH linux-next] powerpc: disable sanitizer in irq_soft_mask_set In-Reply-To: <20220821010030.97539-1-zhouzhouyi@gmail.com> References: <20220821010030.97539-1-zhouzhouyi@gmail.com> Date: Tue, 23 Aug 2022 18:33:14 +1000 Message-ID: <87edx7l5px.fsf@mpe.ellerman.id.au> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_PASS,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Zhouyi Zhou writes: > In ppc, compiler based sanitizer will generate instrument instructions > around statement WRITE_ONCE(local_paca->irq_soft_mask, mask): > > 0xc000000000295cb0 <+0>: addis r2,r12,774 > 0xc000000000295cb4 <+4>: addi r2,r2,16464 > 0xc000000000295cb8 <+8>: mflr r0 > 0xc000000000295cbc <+12>: bl 0xc00000000008bb4c > 0xc000000000295cc0 <+16>: mflr r0 > 0xc000000000295cc4 <+20>: std r31,-8(r1) > 0xc000000000295cc8 <+24>: addi r3,r13,2354 > 0xc000000000295ccc <+28>: mr r31,r13 > 0xc000000000295cd0 <+32>: std r0,16(r1) > 0xc000000000295cd4 <+36>: stdu r1,-48(r1) > 0xc000000000295cd8 <+40>: bl 0xc000000000609b98 <__asan_store1+8> > 0xc000000000295cdc <+44>: nop > 0xc000000000295ce0 <+48>: li r9,1 > 0xc000000000295ce4 <+52>: stb r9,2354(r31) > 0xc000000000295ce8 <+56>: addi r1,r1,48 > 0xc000000000295cec <+60>: ld r0,16(r1) > 0xc000000000295cf0 <+64>: ld r31,-8(r1) > 0xc000000000295cf4 <+68>: mtlr r0 > > If there is a context switch before "stb r9,2354(r31)", r31 may > not equal to r13, in such case, irq soft mask will not work. > > This patch disable sanitizer in irq_soft_mask_set. > > Signed-off-by: Zhouyi Zhou > --- > Dear PPC developers > > I found this bug when trying to do rcutorture tests in ppc VM of > Open Source Lab of Oregon State University following Paul E. McKenny's guidance. > > console.log report following bug: > > [ 346.527467][ T100] BUG: using smp_processor_id() in preemptible [00000000] code: rcu_torture_rea/100^M > [ 346.529416][ T100] caller is rcu_preempt_deferred_qs_irqrestore+0x74/0xed0^M > [ 346.531157][ T100] CPU: 4 PID: 100 Comm: rcu_torture_rea Tainted: G W 5.19.0-rc5-next-20220708-dirty #253^M > [ 346.533620][ T100] Call Trace:^M > [ 346.534449][ T100] [c0000000094876c0] [c000000000ce2b68] dump_stack_lvl+0xbc/0x108 (unreliable)^M > [ 346.536632][ T100] [c000000009487710] [c000000001712954] check_preemption_disabled+0x154/0x160^M > [ 346.538665][ T100] [c0000000094877a0] [c0000000002ce2d4] rcu_preempt_deferred_qs_irqrestore+0x74/0xed0^M > [ 346.540830][ T100] [c0000000094878b0] [c0000000002cf3c0] __rcu_read_unlock+0x290/0x3b0^M > [ 346.542746][ T100] [c000000009487910] [c0000000002bb330] rcu_torture_read_unlock+0x30/0xb0^M > [ 346.544779][ T100] [c000000009487930] [c0000000002b7ff8] rcutorture_one_extend+0x198/0x810^M > [ 346.546851][ T100] [c000000009487a10] [c0000000002b8bfc] rcu_torture_one_read+0x58c/0xc90^M > [ 346.548844][ T100] [c000000009487ca0] [c0000000002b942c] rcu_torture_reader+0x12c/0x360^M > [ 346.550784][ T100] [c000000009487db0] [c0000000001de978] kthread+0x1e8/0x220^M > [ 346.552555][ T100] [c000000009487e10] [c00000000000cd54] ret_from_kernel_thread+0x5c/0x64^M > > After 12 days debugging, I finally narrow the problem to irq_soft_mask_set. Thanks for spending 12 days debugging it! O_o > diff --git a/arch/powerpc/include/asm/hw_irq.h b/arch/powerpc/include/asm/hw_irq.h > index 26ede09c521d..a5ae8d82cc9d 100644 > --- a/arch/powerpc/include/asm/hw_irq.h > +++ b/arch/powerpc/include/asm/hw_irq.h > @@ -121,7 +121,7 @@ static inline notrace unsigned long irq_soft_mask_return(void) > * for the critical section and as a clobber because > * we changed paca->irq_soft_mask > */ > -static inline notrace void irq_soft_mask_set(unsigned long mask) > +static inline notrace __no_kcsan __no_sanitize_address void irq_soft_mask_set(unsigned long mask) > { > /* > * The irq mask must always include the STD bit if any are set. My worry is that this will force irq_soft_mask_set() out of line, which we would rather avoid. It's meant to be a fast path. In fact with this applied I see nearly 300 out-of-line copies of the function when building a defconfig, and ~1700 calls to it. Normally it is inlined at every call site. So I think I'm inclined to revert ef5b570d3700 ("powerpc/irq: Don't open code irq_soft_mask helpers"). It was a nice looking cleanup, but those loads must not be instrumented by KASAN, but we also want them inlined, and AFAICS the only way to achieve that is to go back to inline asm. cheers