Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <6d1a6691-f858-71bf-97fe-97e13fcb93b6@quicinc.com>
Date:   Fri, 14 Jul 2023 09:56:27 +0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.11.2
Subject: Re: [PATCH] arm64: Add the arm64.nolse_atomics command line option
To:     Mark Rutland <mark.rutland@arm.com>
CC:     Will Deacon <will@kernel.org>, <corbet@lwn.net>,
        <catalin.marinas@arm.com>, <maz@kernel.org>,
        <quic_pkondeti@quicinc.com>, <quic_kaushalk@quicinc.com>,
        <quic_satyap@quicinc.com>, <quic_shashim@quicinc.com>,
        <quic_songxue@quicinc.com>, <linux-doc@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>,
        <linux-arm-kernel@lists.infradead.org>
References: <20230710093751.GC32673@willie-the-truck>
 <5cf15f85-0397-96f7-4110-13494551b53b@quicinc.com>
 <20230711082226.GA1554@willie-the-truck>
 <84f0994a-26de-c20a-a32f-ec8fe41df3a3@quicinc.com>
 <20230711102510.GA1809@willie-the-truck>
 <67c2621f-4cad-2495-9785-7737246d3e90@quicinc.com>
 <ZK5X9bXQT7GBxNHj@FVFF77S0Q05N.emea.arm.com>
 <604ac52d-4336-744f-2ab8-44d1c93fbaa8@quicinc.com>
 <ZK_d86ApI1FCHhTL@FVFF77S0Q05N.cambridge.arm.com>
 <e02b9969-a3ca-a80d-1d32-25d2bf4c72b6@quicinc.com>
 <ZLBLwG2LJ4gZLfbh@FVFF77S0Q05N.cambridge.arm.com>
From:   "Aiqun(Maria) Yu" <quic_aiquny@quicinc.com>
In-Reply-To: <ZLBLwG2LJ4gZLfbh@FVFF77S0Q05N.cambridge.arm.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 7/14/2023 3:08 AM, Mark Rutland wrote:
> On Thu, Jul 13, 2023 at 10:08:34PM +0800, Aiqun(Maria) Yu wrote:
>> On 7/13/2023 7:20 PM, Mark Rutland wrote:
>>> Are you saying that LSE atomics to *cacheable* mappings do not work on your
>>> system?
>>>
>>> Specifically, when using a Normal Inner-Shareable Inner-Writeback
>>> Outer-Writeback mapping, do the LSE atomics work or not work?
>> *cacheable* mapping have the LSE atomic is not working if far atomic is
>> performed.
> 
> Thanks for confirming; the fact that this doesn't work on *cacheable* memory is
> definitely a major issue. I think everyone is confused here because of the
> earlier mention of non-cachable accesses (which don't matter).
> 
Maybe I can have the information collected in a summary to see if that 
helps.
> I know that some CPU implementations have EL3 control bits to force LSE atomics
> to be performed near (e.g. in Cortex-A55, the CPUECTLR.ATOM control bits),
> which would avoid the issue while still allowing the LSE atomics to be used.
> 
> If those can be configured in EL3 firmware, that would be a preferable
> workaround.
> 
> Can you say which CPUs are integrated in this system? and/or can you check if
> such control bits exist?

We have CPUECTLR_EL1.ATOM bit can force LSE atomics to be perform near.
CPUECTLR_EL1 is also an option to EL1 kernel drivers to be configuarable.

Try to a detailed summarise of the whole discussions, anyone can ignore 
some part if you are already know.

* Part 1: Solution for this issue.
While we still want to have options to let third party and end users 
can have options:
   1.Disable lse atomic cap.
   2.*Disallow* far atomic by "CPUECTLR_EL1.atom force near atomic" and 
non-cachable mappling for lse atomic only.


* Part 2: Why we need the solution
1. There is also some case far atomic is better performance than near 
atomic. end user may still can still try to do allow far atomic.
while this driver is also use kerenl LSE ATOMIC macro, so it can be 
running on cpu don't support lse atomic and cpu support lse atomic already.
while current system, cpu have feature register said lse atomic is 
supported, but memory controller is not supported is currently not yet 
supported.
2. cpu feature of lse atomic capbility can be controled via options for 
the same image.
Can have GKI(generic kernel Image) + same third party drivers Images 
support multi systems.
--  *New system* fully support lse atomic
-- *Intermidiate support system* which only have cpu support lse atomic, 
but have memory control/bus don't support lse atomic.* (mainly issue are 
discussed in this thread.)
-- *old system* have cpu don't have this cpu feature at all.
3. better for debugging purpose, it would be easier for verify if it is 
this feature related or not.
4. *Disallow* from the developer side is not easy to control, expecially 
when they have the same code working on *old system* or *new system*, 
but failed on current *Intermidiate support system*.

> 
> Thanks,
> Mark.
>   

Thx for discussion in details.
-- 
Thx and BRs,
Aiqun(Maria) Yu