Received: by 2002:a05:6358:3188:b0:123:57c1:9b43 with SMTP id q8csp37980738rwd; Wed, 12 Jul 2023 01:02:11 -0700 (PDT) X-Google-Smtp-Source: APBJJlEJkcD2hWu80qw60zU+217mAgMAYUX664MjgSRcG264SaXcGNXCxPh4khUhTPyKQX/rtP37 X-Received: by 2002:a17:902:e54a:b0:1b8:b564:b531 with SMTP id n10-20020a170902e54a00b001b8b564b531mr17626716plf.60.1689148931530; Wed, 12 Jul 2023 01:02:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689148931; cv=none; d=google.com; s=arc-20160816; b=Pe1qaZpmAetVp19Y7i+jRFHrCBZ/TDwYd/9m2a2ifTqC7qckxOzCGC6lbAM0GwrBa6 il8H5NkS72S6OiTrFIRw8bRe5zwCC0mq401ICqetsHZ6Dzx190aPHUhpjLFEUPkSiF8x MbNkcAq27yDxQlByeEvCuNeeG1jiW3toYN4icifVzVagq6NtEeamYTz2BbIPqqr67ZOA erKR7CKqcoIUn0p9H+rv5t4+aVyI7A3vDOBM31TYZQyh6B1LvFxcWxRoyPrj2STJJYEW do+slH3yzw36LBeRKIgCUz3Ryy5/GL37EvG4Xn30HRSJQoFjNV6nNF0atx8vWcfo8nzJ r9Jg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date; bh=naRj4VhwTzfDqHSDo+pa/071y9AeGrwecERC+bvbWNM=; fh=x/afr8rcjrDlW8vD40JVIV2VGsOBPUbIChD/i8AaAKI=; b=Q9gP8SWJMD8KJlz4NrpaBsYLoHvhnVlP8bg2LjH9cUbI1rsdFxJcVggg0kRdr3c4Y1 zp656JylfZYXLQCKTwsQm/z1SIkeHhAZ0GQM4DrfbJIvOfNGAMfk7xNFfHM8F66fXxvc uaLIiZWTbnRG60KjZ/AK297oazHwEjxMlnb9yULiuD8cgb0k6X2XCqpqSsi5wYqiVqJ+ mD14b+99ujHmi7IMVbCngHebAfSYv4gOFXn29a1zuSraKVqzDV0SUoX+Nm3BbztUC+3n oau0jHmF2FceaBs50pbYOMbk539Xkzxfxt+ev00LjuSlXjPDGr2oTY1g2oZnuOqZoKky QseA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x9-20020a170902a38900b001b85ab48092si2953170pla.499.2023.07.12.01.01.59; Wed, 12 Jul 2023 01:02:11 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232039AbjGLHg5 (ORCPT + 99 others); Wed, 12 Jul 2023 03:36:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58522 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231693AbjGLHg4 (ORCPT ); Wed, 12 Jul 2023 03:36:56 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 5A3DCB8; Wed, 12 Jul 2023 00:36:54 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6916E2F4; Wed, 12 Jul 2023 00:37:36 -0700 (PDT) Received: from FVFF77S0Q05N.emea.arm.com (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 3637D3F67D; Wed, 12 Jul 2023 00:36:52 -0700 (PDT) Date: Wed, 12 Jul 2023 08:36:46 +0100 From: Mark Rutland To: "Aiqun(Maria) Yu" Cc: Will Deacon , corbet@lwn.net, catalin.marinas@arm.com, maz@kernel.org, quic_pkondeti@quicinc.com, quic_kaushalk@quicinc.com, quic_satyap@quicinc.com, quic_shashim@quicinc.com, quic_songxue@quicinc.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Subject: Re: [PATCH] arm64: Add the arm64.nolse_atomics command line option Message-ID: References: <20230710055955.36551-1-quic_aiquny@quicinc.com> <20230710093751.GC32673@willie-the-truck> <5cf15f85-0397-96f7-4110-13494551b53b@quicinc.com> <20230711082226.GA1554@willie-the-truck> <84f0994a-26de-c20a-a32f-ec8fe41df3a3@quicinc.com> <20230711102510.GA1809@willie-the-truck> <67c2621f-4cad-2495-9785-7737246d3e90@quicinc.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <67c2621f-4cad-2495-9785-7737246d3e90@quicinc.com> X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Jul 12, 2023 at 11:09:10AM +0800, Aiqun(Maria) Yu wrote: > On 7/11/2023 6:25 PM, Will Deacon wrote: > > On Tue, Jul 11, 2023 at 06:15:49PM +0800, Aiqun(Maria) Yu wrote: > > > On 7/11/2023 4:22 PM, Will Deacon wrote: > > > > On Tue, Jul 11, 2023 at 12:02:22PM +0800, Aiqun(Maria) Yu wrote: > > > > > On 7/10/2023 5:37 PM, Will Deacon wrote: > > > > > > On Mon, Jul 10, 2023 at 01:59:55PM +0800, Maria Yu wrote: > > > > > > > In order to be able to disable lse_atomic even if cpu > > > > > > > support it, most likely because of memory controller > > > > > > > cannot deal with the lse atomic instructions, use a > > > > > > > new idreg override to deal with it. > > > > > > > > > > > > This should not be a problem for cacheable memory though, right? > > > > > > > > > > > > Given that Linux does not issue atomic operations to non-cacheable mappings, > > > > > > I'm struggling to see why there's a problem here. > > > > > > > > > > The lse atomic operation can be issued on non-cacheable mappings as well. > > > > > Even if it is cached data, with different CPUECTLR_EL1 setting, it can also > > > > > do far lse atomic operations. > > > > > > > > Please can you point me to the place in the kernel sources where this > > > > happens? The architecture doesn't guarantee that atomics to non-cacheable > > > > mappings will work, see "B2.2.6 Possible implementation restrictions on > > > > using atomic instructions". Linux, therefore, doesn't issue atomics > > > > to non-cacheable memory. > > > > > > We encounter the issue on third party kernel modules and third party apps > > > instead of linux kernel itself. > > > > Great, so there's nothing to do in the kernel then! > > > > The third party code needs to be modified not to use atomic instructions > > with non-cacheable mappings. No need to involve us with that. > > > > This is a tradeoff of performance and stability. Per my understanding, > > > options can be used to enable the lse_atomic to have the most performance > > > cared system, and disable the lse_atomic by stability cared most system. > > > > Where do livelock and starvation fit in with "stability"? Disabling LSE > > atomics for things like qspinlock and the scheduler just because of some > > badly written third-party code isn't much of a tradeoff. > We also have requirement to have cpus/system fully support lse atomic and > cpus/system not fully support lse atomic with a generic kernel image. Who *specifically* has this requirement (i.e. what does 'we' mean here)? The upstream kernel does not require that atomics work on non-cacheable memory, and saying "The company I work for want this" doesn't change that. AFAICT the system here is architecturally compliant, and what you're relying upon something that the architecture doesn't guarantee, and Linux doesn't guarantee. > Same kernel module wanted to be used by lse atomic fully support cpu and not > fully support cpu/system as well. Which kernel modules *specifically* need to do atomics to non-cacheable memory? > That's why we want to have a runtime option here. As per other replies, a runtime option doesn't solve the issue you have described, and it will adversely affect the system in other ways (e.g. the livelock and starvation issues will mentioned, which we have seen with LDXR+STXR atomics). Thanks, Mark.