Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
IronPort-SDR: WJMjrz4/1Dp1ReRR6t1FzyYM5BGXSOqQrrtCxXZDgC6J30xAy0C9/BQ1Cnnxgdhx/LJLNKV+kk
 uWLKSk0bzdhiDzvLf+/tSPk6dOX68YS7cnh+bxdETk5iDzik3JAMix5jp5AKum8ZLe6hWvIFVA
 1myq11xrZ0747Efks3au5klWfEifmt+RlY8GxgkK2OPJI8pHy388zdkqR/YwcfPz5HrASqMXe2
 jglWLRV0oM07yxVjVVvSPhOAfKjNQDajharlo/VJC9T8oDemTXUAK9ZhFGddrG08Np9zSVghTu
 yuk=
From:   "tan.shaopeng@fujitsu.com" <tan.shaopeng@fujitsu.com>
To:     "'fenghua.yu@intel.com'" <fenghua.yu@intel.com>,
        "'reinette.chatre@intel.com'" <reinette.chatre@intel.com>
CC:     "'linux-kernel@vger.kernel.org'" <linux-kernel@vger.kernel.org>,
        "'linux-arm-kernel@lists.infradead.org'" 
        <linux-arm-kernel@lists.infradead.org>,
        'James Morse' <james.morse@arm.com>,
        "misono.tomohiro@fujitsu.com" <misono.tomohiro@fujitsu.com>
Subject: About add an A64FX cache control function into resctrl
Thread-Topic: About add an A64FX cache control function into resctrl
Thread-Index: AdctARUohIvM7pb1S52qTlM+pjp6RA==
Date:   Fri, 9 Apr 2021 05:46:28 +0000
Message-ID: <OSAPR01MB214600C7923AEF7C35B02E648B739@OSAPR01MB2146.jpnprd01.prod.outlook.com>
Accept-Language: ja-JP, en-US
Content-Language: ja-JP
Content-Type: text/plain; charset="iso-2022-jp"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 1fcf883b-389b-4019-80eb-08d8fb1ad239
X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Apr 2021 05:46:28.6691
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: a19f121d-81e1-4858-a9d8-736e267fd4c7
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: GdVT5/sSOGjm+63lmvxrz9wNLoudjmUgcRzbfF9uUSOMH+u84c/ki15gJ/egeSgW/JtDddCkBYtMdQF4mwwJjmLYFdATo9bp3lE1JpJDuDg=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: OS3PR01MB6150
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hello


I'm Tan Shaopeng from Fujitsu Limited.=20

I=1B$B!G=1B(Bm trying to implement Fujitsu A64FX=1B$B!G=1B(Bs cache related=
 features.=20
It is a cache partitioning function we called sector cache function=20
that using the value of the tag that is upper 8 bits of the 64bit=20
address and the value of the sector cache register to control virtual=20
cache capacity of the L1D&L2 cache.=20

A few days ago, when I sent a driver that realizes this function to=20
ARM64 kernel community, Will Deacon and Arnd Bergmann suggested=20
an idea to add the sector cache function of A64FX into resctrl.=20
https://lore.kernel.org/linux-arm-kernel/CAK8P3a2pFcNTw9NpRtQfYr7A5OcZ=3DAs=
2kM0D_sbfFcGQ_J2Q+Q@mail.gmail.com/=20

Based on my study, I think the sector cache function of A64FX can be=20
added into the allocation features of resctrl after James' resctrl=20
rework has finished. But, in order to implement this function,=20
more interfaces for resctrl are need. The details are as follow,=20
and could you give me some advice?=20

[Sector cache function]=20
The sector cache function split cache into multiple sectors and=20
control them separately. It is implemented on the L1D cache and=20
L2 cache in the A64FX processor and can be controlled individually=20
for L1D cache and L2 cache. A64FX has no L3 cache. Each L1D cache=20
and L2 cache has 4 sectors. Which L1D sector is used is specified=20
by the value of [57:56] bits of address, how many ways of sector=20
are specified by the value of register (IMP_SCCR_L1_EL0).=20
Which L2 sector is used is specified by the value of [56] bits of=20
address, and how many ways of sector are specified by value of register=20
(IMP_SCCR_ASSIGN_EL1, IMP_SCCR_SET0_L2_EL1, IMP_SCCR_SET1_L2_EL1).=20

For more details of sector cache function,=20
see A64FX HPC extension specification (1.2. Sector cache) in=20
https://github.com/fujitsu/A64FX=20

[Difference between resctrl(CAT) and this sector cache function]=20
L2/L3 CAT (Cache Allocation Technology) enables the user to specify=20
some physical partition of cache space that an application can fill.=20
A64FX's L1D/L2 cache has 4 sectors and 16ways. This sector function=20
enables a user to specify number of ways each sector uses.=20
Therefore, for CAT it is enough to specify a cache portion for=20
each cache_id (socket). On the other hand, sector cache needs to=20
specify cache portion of each sector for each cache_id, and following=20
extension to resctrl interface is needed to support sector cache.=20

[Idear for A64FX sector cache function control interface (schemata file det=
ails)]=20
L1:<cache_id0>=3D<cwbm>,<cwbm>,<cwbm>,<cwbm>;<cache_id1>=3D<cwbm>,<cwbm>,<c=
wbm>,<cwbm>;=1B$B!D=1B(B =20
L2:<cache_id0>=3D>=3D<cwbm>,<cwbm>,<cwbm>,<cwbm>;<cache_id1>=3D<cwbm>,<cwbm=
>,<cwbm>,<cwbm>;=1B$B!D=1B(B=20

=1B$B!&=1B(BL1: Add a new interface to control the L1D cache.=20
=1B$B!&=1B(B<cwbm>,<cwbm>,<cwbm>,<cwbm>=1B$B!'=1B(BSpecify the number of wa=
ys for each sector.=20
=1B$B!&=1B(Bcwbm=1B$B!'=1B(BSpecify the number of ways in each sector as a =
bitmap (percentage),=20
  but the bitmap does not indicate the location of the cache.=20
* In the sector cache function, L2 sector cache way setting register is=20
  shared among PEs (Processor Element) in shared domain. If two PEs=20
  which share L2 cache belongs to different resource groups, one resource=20
  group's L2 setting will affect to other resource group's L2 setting.=20
* Since A64FX does not support MPAM, it is not necessary to consider=20
  how to switch between MPAM and sector cache function now.=20

Some questions:=20
1.I'm still studying about RDT, could you tell me whether RDT has=20
  the similar mechanism with sector cache function?=20
2.In RDT, L3 cache is shared among cores in socket. If two cores which=20
  share L3 cache belongs to different resource groups, one resource=20
  group's L3 setting will affect to other resource group's L3 setting?=20
3.Is this approach acceptable? could you give me some advice?=20


Best regards=20
Tan Shaopeng=20