Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
IronPort-SDR: xtF3qfTLSQgcXKS7r++r2OcZorwC4ANEMTH3lrbcNZlmTvlpyaYk0oebP1HCWvEavV4xdyXc+E
 D85rumgKCf5Wnbwc74BzrlmxTUaGijOI3eZHda/ctqtWNpIlYBlZ4mWGUFtLr021MTzSz7Q4mk
 qsdL+Agkvdmcr5rDzB19xUMEhr0oJ46J9ZDg3t6YVLS4l/CzuwuSFY0+FBS60GLuzu665fOWt9
 w8JADO8jBsaGy/PTtoMbk8gy+JWd1ko1a4Nm/KL9FLwQunn09B+nRrwS5DNmz9CQsbx4u9kNMh
 ewQ=
From:   "tan.shaopeng@fujitsu.com" <tan.shaopeng@fujitsu.com>
To:     'Reinette Chatre' <reinette.chatre@intel.com>,
        "'fenghua.yu@intel.com'" <fenghua.yu@intel.com>
CC:     "'linux-kernel@vger.kernel.org'" <linux-kernel@vger.kernel.org>,
        "'linux-arm-kernel@lists.infradead.org'" 
        <linux-arm-kernel@lists.infradead.org>,
        'James Morse' <james.morse@arm.com>,
        "misono.tomohiro@fujitsu.com" <misono.tomohiro@fujitsu.com>
Subject: RE: About add an A64FX cache control function into resctrl
Thread-Topic: About add an A64FX cache control function into resctrl
Thread-Index: AdctARUohIvM7pb1S52qTlM+pjp6RAJht/AAABE+WYAAUfQI4A==
Date:   Fri, 23 Apr 2021 08:10:10 +0000
Message-ID: <OSAPR01MB21467099EAA9B25F0A992BD88B459@OSAPR01MB2146.jpnprd01.prod.outlook.com>
References: <OSAPR01MB214600C7923AEF7C35B02E648B739@OSAPR01MB2146.jpnprd01.prod.outlook.com>
 <OSAPR01MB214657641D532FB8D112DD528B479@OSAPR01MB2146.jpnprd01.prod.outlook.com>
 <bb0967c0-5b88-c6c2-0242-1e3928189a04@intel.com>
In-Reply-To: <bb0967c0-5b88-c6c2-0242-1e3928189a04@intel.com>
Accept-Language: ja-JP, en-US
Content-Language: ja-JP
Content-Type: text/plain; charset="iso-2022-jp"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: ddaf17cb-3ff3-4f6e-6e3e-08d9062f373d
X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Apr 2021 08:10:10.9308
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: a19f121d-81e1-4858-a9d8-736e267fd4c7
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: gvL5cHJ0TQvT+Cep3esISqnR6VQ9gDfDYsxxjqcbXSEq65bsregH00nPKM7KnEuWb+wCDzjqKysgfq71C3PpdnbEPN2fw/p5NCWO6wwprPY=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: OSBPR01MB2373
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Reinette,

> On 4/21/2021 1:37 AM, tan.shaopeng@fujitsu.com wrote:
> > Hi,
> >
> > Ping... any comments&advice about add an A64FX cache control function
> into resctrl?
>=20
> My apologies for the delay.
>=20
> >
> > Best regards
> > Tan Shaopeng
> >
> >> Hello
> >>
> >>
> >> I'm Tan Shaopeng from Fujitsu Limited.
> >>
> >> I=1B$B!G=1B(Bm trying to implement Fujitsu A64FX=1B$B!G=1B(Bs cache re=
lated features.
> >> It is a cache partitioning function we called sector cache function
> >> that using the value of the tag that is upper 8 bits of the 64bit
> >> address and the value of the sector cache register to control virtual =
cache
> capacity of the L1D&L2 cache.
> >>
> >> A few days ago, when I sent a driver that realizes this function to
> >> ARM64 kernel community, Will Deacon and Arnd Bergmann suggested an
> >> idea to add the sector cache function of A64FX into resctrl.
> >>
> https://lore.kernel.org/linux-arm-kernel/CAK8P3a2pFcNTw9NpRtQfYr7A5Oc
> >> Z=3DAs2kM0D_sbfFcGQ_J2Q+Q@mail.gmail.com/
> >>
> >> Based on my study, I think the sector cache function of A64FX can be
> >> added into the allocation features of resctrl after James' resctrl rew=
ork has
> finished.
> >> But, in order to implement this function, more interfaces for resctrl =
are
> need.
> >> The details are as follow, and could you give me some advice?
> >>
> >> [Sector cache function]
> >> The sector cache function split cache into multiple sectors and
> >> control them separately. It is implemented on the L1D cache and
> >> L2 cache in the A64FX processor and can be controlled individually
> >> for L1D cache and L2 cache. A64FX has no L3 cache. Each L1D cache and
> >> L2 cache has 4 sectors. Which L1D sector is used is specified by the
> >> value of [57:56] bits of address, how many ways of sector are
> >> specified by the value of register (IMP_SCCR_L1_EL0).
> >> Which L2 sector is used is specified by the value of [56] bits of
> >> address, and how many ways of sector are specified by value of
> >> register (IMP_SCCR_ASSIGN_EL1, IMP_SCCR_SET0_L2_EL1,
> >> IMP_SCCR_SET1_L2_EL1).
> >>
> >> For more details of sector cache function, see A64FX HPC extension
> >> specification (1.2. Sector cache) in https://github.com/fujitsu/A64FX
>=20
> The overview in section 12 was informative but very high level.
> I was not able to find any instance of "IMP_SCCR" in this document to exp=
lore
> how this cache allocation works.
>=20
> Are these cache sectors exposed to the OS in any way? For example, when t=
he
> OS discovers the cache, does it learn about these sectors and expose the
> details to user space (/sys/devices/system/cpuX/cache)?
>=20
> The overview of Sector Cache in that document provides details of how the=
 size
> of the sector itself is dynamically adjusted to usage. That description i=
s quite
> cryptic but it seems like a sector, since the number of ways associated w=
ith it
> can dynamically change, is more equivalent to a class of service or resou=
rce
> group in the resctrl environment.
>=20
> I really may be interpreting things wrong here, could you perhaps point m=
e to
> where I can obtain more details?
>=20
>=20
> >> [Difference between resctrl(CAT) and this sector cache function]
> >> L2/L3 CAT (Cache Allocation Technology) enables the user to specify
> >> some physical partition of cache space that an application can fill.
> >> A64FX's L1D/L2 cache has 4 sectors and 16ways. This sector function
> >> enables a user to specify number of ways each sector uses.
> >> Therefore, for CAT it is enough to specify a cache portion for each
> >> cache_id (socket). On the other hand, sector cache needs to specify
> >> cache portion of each sector for each cache_id, and following
> >> extension to resctrl interface is needed to support sector cache.
> >>
> >> [Idear for A64FX sector cache function control interface (schemata
> >> file details)]
> >>
> L1:<cache_id0>=3D<cwbm>,<cwbm>,<cwbm>,<cwbm>;<cache_id1>=3D<cw
> >> bm>,<cwbm>,<cwbm>,<cwbm>;=1B$B!D=1B(B
> >>
> L2:<cache_id0>=3D>=3D<cwbm>,<cwbm>,<cwbm>,<cwbm>;<cache_id1>=3D
> >> <cwbm>,<cwbm>,<cwbm>,<cwbm>;=1B$B!D=1B(B
> >>
> >> =1B$B!&=1B(BL1: Add a new interface to control the L1D cache.
> >> =1B$B!&=1B(B<cwbm>,<cwbm>,<cwbm>,<cwbm>=1B$B!'=1B(BSpecify the number =
of ways for
> each
> >> sector.
> >> =1B$B!&=1B(Bcwbm=1B$B!'=1B(BSpecify the number of ways in each sector =
as a bitmap
> (percentage),
> >>    but the bitmap does not indicate the location of the cache.
> >> * In the sector cache function, L2 sector cache way setting register i=
s
> >>    shared among PEs (Processor Element) in shared domain. If two PEs
> >>    which share L2 cache belongs to different resource groups, one
> resource
> >>    group's L2 setting will affect to other resource group's L2 setting=
.
>=20
> In resctrl a "resource group" can be viewed as a class of service.
>=20
> >> * Since A64FX does not support MPAM, it is not necessary to consider
> >>    how to switch between MPAM and sector cache function now.
> >>
> >> Some questions:
> >> 1.I'm still studying about RDT, could you tell me whether RDT has
> >>    the similar mechanism with sector cache function?
>=20
> This is not clear to me yet. One thing to keep in mind is that a bit in t=
he capacity
> bitmask could correspond to some number of ways in a cache, but it does n=
ot
> have to. It is essentially a hint to hardware on how much cache space nee=
ds to
> be allocated while also indicating overlap and isolation from other alloc=
ations.
>=20
> resctrl already supports the bitmask being interpreted differently betwee=
n
> architectures and with the MPAM support there will be even more support f=
or
> different interpretations.
>=20
> >> 2.In RDT, L3 cache is shared among cores in socket. If two cores which
> >>    share L3 cache belongs to different resource groups, one resource
> >>    group's L3 setting will affect to other resource group's L3 setting=
?
>=20
> This question is not entirely clear to me. Are you referring to the hardw=
are layout
> or configuration changes via the resctrl "cpus" file?
>=20
> Each resource group is a class of service (CLOS) that is supported by all=
 cache
> instances. By default each resource group would thus contain all cache
> instances on the system (even if some cache instances do not support the
> same number of CLOS resctrl would only support the CLOS supported by all
> resources).

Thanks for your comment.=20

I am sorry that the description about the sector cache function was
difficult to understand. Since all public specifications were shown
in the URL, please give me some time, I will organize the contents of
64FX cache control function.=20

Best regards,=20
Tan Shaopeng