Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
IronPort-SDR: jCI6qzrQXU5R62EF0XTllF0IIX/HwQswt/VJKf4T3jnASfPJate8ZBRxjAiAMDqnp+pFc5TDla
 HuC4oW+ubgielLGQOGZk7GoPLXRlEv7hQ/2y9jFNWuThRBncXtB8sJKvPBUDmY6n45Vw+Cm54P
 +QCvS6zvoNCkjBOGT4ycGoyDS8H1g2ROnYoSZgbRfm/0AYhNEm4fL+AwK0uZz2e13fJ4szEFh2
 /KICWi/hNMWUwqOisiCC6qWufPKNqELLKF4jIYL9Ccvv4ZSgZoTvze3dDXJu6TdOZizu2XC+Ld
 LmM=
From:   "tan.shaopeng@fujitsu.com" <tan.shaopeng@fujitsu.com>
To:     'Reinette Chatre' <reinette.chatre@intel.com>,
        "'fenghua.yu@intel.com'" <fenghua.yu@intel.com>
CC:     "'linux-kernel@vger.kernel.org'" <linux-kernel@vger.kernel.org>,
        "'linux-arm-kernel@lists.infradead.org'" 
        <linux-arm-kernel@lists.infradead.org>,
        'James Morse' <james.morse@arm.com>,
        "misono.tomohiro@fujitsu.com" <misono.tomohiro@fujitsu.com>
Subject: RE: About add an A64FX cache control function into resctrl
Thread-Topic: About add an A64FX cache control function into resctrl
Thread-Index: AdctARUohIvM7pb1S52qTlM+pjp6RAJht/AAABE+WYAFB/HO8A==
Date:   Mon, 17 May 2021 08:37:01 +0000
Message-ID: <TYAPR01MB633050EF44B5071B97917D208B2D9@TYAPR01MB6330.jpnprd01.prod.outlook.com>
References: <OSAPR01MB214600C7923AEF7C35B02E648B739@OSAPR01MB2146.jpnprd01.prod.outlook.com>
 <OSAPR01MB214657641D532FB8D112DD528B479@OSAPR01MB2146.jpnprd01.prod.outlook.com>
 <bb0967c0-5b88-c6c2-0242-1e3928189a04@intel.com>
In-Reply-To: <bb0967c0-5b88-c6c2-0242-1e3928189a04@intel.com>
Accept-Language: ja-JP, en-US
Content-Language: ja-JP
Content-Type: text/plain; charset="iso-2022-jp"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: ea13ea8b-2419-45b7-8e8a-08d9190ef164
X-MS-Exchange-CrossTenant-originalarrivaltime: 17 May 2021 08:37:01.9253
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: a19f121d-81e1-4858-a9d8-736e267fd4c7
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 7adi9bLOf2xULViz56GUsX6barFLNq+eQx5eH9CuPfj4Q1m/9l7IAzCywvSFHzldgXMli9OckYidfnnMTbwnlNPQXd24usCkUPG+rmBjmTs=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: TYAPR01MB3021
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Reinette,

> On 4/21/2021 1:37 AM, tan.shaopeng@fujitsu.com wrote:
> > Hi,
> >
> > Ping... any comments&advice about add an A64FX cache control function
> into resctrl?
>=20
> My apologies for the delay.
>=20
> >
> > Best regards
> > Tan Shaopeng
> >
> >> Hello
> >>
> >>
> >> I'm Tan Shaopeng from Fujitsu Limited.
> >>
> >> I=1B$B!G=1B(Bm trying to implement Fujitsu A64FX=1B$B!G=1B(Bs cache re=
lated features.
> >> It is a cache partitioning function we called sector cache function
> >> that using the value of the tag that is upper 8 bits of the 64bit
> >> address and the value of the sector cache register to control virtual =
cache
> capacity of the L1D&L2 cache.
> >>
> >> A few days ago, when I sent a driver that realizes this function to
> >> ARM64 kernel community, Will Deacon and Arnd Bergmann suggested an
> >> idea to add the sector cache function of A64FX into resctrl.
> >>
> https://lore.kernel.org/linux-arm-kernel/CAK8P3a2pFcNTw9NpRtQfYr7A5Oc
> >> Z=3DAs2kM0D_sbfFcGQ_J2Q+Q@mail.gmail.com/
> >>
> >> Based on my study, I think the sector cache function of A64FX can be
> >> added into the allocation features of resctrl after James' resctrl rew=
ork has
> finished.
> >> But, in order to implement this function, more interfaces for resctrl =
are
> need.
> >> The details are as follow, and could you give me some advice?
> >>
> >> [Sector cache function]
> >> The sector cache function split cache into multiple sectors and
> >> control them separately. It is implemented on the L1D cache and
> >> L2 cache in the A64FX processor and can be controlled individually
> >> for L1D cache and L2 cache. A64FX has no L3 cache. Each L1D cache and
> >> L2 cache has 4 sectors. Which L1D sector is used is specified by the
> >> value of [57:56] bits of address, how many ways of sector are
> >> specified by the value of register (IMP_SCCR_L1_EL0).
> >> Which L2 sector is used is specified by the value of [56] bits of
> >> address, and how many ways of sector are specified by value of
> >> register (IMP_SCCR_ASSIGN_EL1, IMP_SCCR_SET0_L2_EL1,
> >> IMP_SCCR_SET1_L2_EL1).
> >>
> >> For more details of sector cache function, see A64FX HPC extension
> >> specification (1.2. Sector cache) in https://github.com/fujitsu/A64FX
>=20
> The overview in section 12 was informative but very high level.
> I was not able to find any instance of "IMP_SCCR" in this document to exp=
lore
> how this cache allocation works.

Maybe you have already known, the sector cache works as follows.=20
 - Set the maximum number of ways for sector (id =3D 0/1/2/3) of L1&L2=20
   by setting the "IMP_SCCR" register.=20
 - When running a task, the sector id is specified by [56:57] bits of=20
   the virtual address. If the sector id is not specified, the sector=20
   id specified in IMP_SCCR_ASSIGN_EL1.default_sector will be used.

> Are these cache sectors exposed to the OS in any way? For example, when t=
he
> OS discovers the cache, does it learn about these sectors and expose the
> details to user space (/sys/devices/system/cpuX/cache)?

These cache sectors are not exposed to the OS in any way.

> The overview of Sector Cache in that document provides details of how the=
 size
> of the sector itself is dynamically adjusted to usage. That description i=
s quite
> cryptic but it seems like a sector, since the number of ways associated w=
ith it
> can dynamically change, is more equivalent to a class of service or resou=
rce
> group in the resctrl environment.

I explained the difference between "sector" and "class of service"=20
in another email.

> I really may be interpreting things wrong here, could you perhaps point m=
e to
> where I can obtain more details?

I'm sorry, there is no documentation other than the manual and=20
specifications. More details about how sector cache function works=20
as follows.=20
(1) By setting the access control register IMP_SCCR_CTRL_EL1, cache=20
    capacity setting registers (IMP_SCCR_CTRL_EL1, IMP_SCCR_ASSIGN_EL1,=20
    IMP_SCCR_L1_EL0, IMP_SCCR_SET0_L2_EL1, IMP_SCCR_SET1_L2_EL1,=20
    IMP_SCCR_VSCCR_L2_EL0) can be set from user space or kernel space.=20
(2) Set L1 sector cache capacity register from kernel space.=20
    By setting the register IMP_SCCR_L1_EL0, set the maximum number=20
    of ways of sector (id =3D 0/1/2/3) of L1.=20
(3) Set L2 sector cache capacity register.=20
    (one of cases) By setting IMP_SCCR_ASSIGN_EL1.assign =3D 0 from=20
    kernel space, IMP_SCCR_VSCCR_L2_EL0 becomes alias of=20
    IMP_SCCR_SET0_L2_EL1. By setting IMP_SCCR_VSCCR_L2_EL0 from user=20
    space, set the maximum number of ways of sector (id =3D 0/1) of L2.=20
(4) When running a task, the sector ID of L1 is decided by [56:57]=20
    bits of virtual address, and the sector ID of L2 is decided by=20
    [56] bit of the address. These bits are programed by the users.

> >> [Difference between resctrl(CAT) and this sector cache function]
> >> L2/L3 CAT (Cache Allocation Technology) enables the user to specify
> >> some physical partition of cache space that an application can fill.
> >> A64FX's L1D/L2 cache has 4 sectors and 16ways. This sector function
> >> enables a user to specify number of ways each sector uses.
> >> Therefore, for CAT it is enough to specify a cache portion for each
> >> cache_id (socket). On the other hand, sector cache needs to specify
> >> cache portion of each sector for each cache_id, and following
> >> extension to resctrl interface is needed to support sector cache.
> >>
> >> [Idear for A64FX sector cache function control interface (schemata
> >> file details)]
> >>
> L1:<cache_id0>=3D<cwbm>,<cwbm>,<cwbm>,<cwbm>;<cache_id1>=3D<cw
> >> bm>,<cwbm>,<cwbm>,<cwbm>;=1B$B!D=1B(B
> >>
> L2:<cache_id0>=3D>=3D<cwbm>,<cwbm>,<cwbm>,<cwbm>;<cache_id1>=3D
> >> <cwbm>,<cwbm>,<cwbm>,<cwbm>;=1B$B!D=1B(B
> >>
> >> =1B$B!&=1B(BL1: Add a new interface to control the L1D cache.
> >> =1B$B!&=1B(B<cwbm>,<cwbm>,<cwbm>,<cwbm>=1B$B!'=1B(BSpecify the number =
of ways for
> each
> >> sector.
> >> =1B$B!&=1B(Bcwbm=1B$B!'=1B(BSpecify the number of ways in each sector =
as a bitmap
> (percentage),
> >>    but the bitmap does not indicate the location of the cache.
> >> * In the sector cache function, L2 sector cache way setting register i=
s
> >>    shared among PEs (Processor Element) in shared domain. If two PEs
> >>    which share L2 cache belongs to different resource groups, one
> resource
> >>    group's L2 setting will affect to other resource group's L2 setting=
.
>=20
> In resctrl a "resource group" can be viewed as a class of service.

Thanks for your explanation. Adding sector cache function into resctrl,=20
I will use this mechanism of resctrl as it is.

> >> * Since A64FX does not support MPAM, it is not necessary to consider
> >>    how to switch between MPAM and sector cache function now.
> >>
> >> Some questions:
> >> 1.I'm still studying about RDT, could you tell me whether RDT has
> >>    the similar mechanism with sector cache function?
>=20
> This is not clear to me yet. One thing to keep in mind is that a bit in t=
he capacity
> bitmask could correspond to some number of ways in a cache, but it does n=
ot
> have to. It is essentially a hint to hardware on how much cache space nee=
ds to
> be allocated while also indicating overlap and isolation from other alloc=
ations.
>=20
> resctrl already supports the bitmask being interpreted differently betwee=
n
> architectures and with the MPAM support there will be even more support f=
or
> different interpretations.

when adding sector cache function into resctrl,=20
the bitmap will only show the maximum number of ways of sector=20
and does not indicate cache position like in RDT.=20
Sector is a group of cache ways, and one cache line cannot be assigned=20
to different sectors at the same time. Different sectors have different=20
cache space. When different tasks use different sectors,=20
the cache space used can be isolated.

> >> 2.In RDT, L3 cache is shared among cores in socket. If two cores which
> >>    share L3 cache belongs to different resource groups, one resource
> >>    group's L3 setting will affect to other resource group's L3 setting=
?
>=20
> This question is not entirely clear to me. Are you referring to the hardw=
are layout
> or configuration changes via the resctrl "cpus" file?
>=20
> Each resource group is a class of service (CLOS) that is supported by all=
 cache
> instances. By default each resource group would thus contain all cache
> instances on the system (even if some cache instances do not support the
> same number of CLOS resctrl would only support the CLOS supported by all
> resources).

[Idea for A64FX sector cache function control] =20
An example of using the sector function when working on resctrl=20
as follows.=20
  # mount -t resctrl resctrl /sys/fs/resctrl
  # cd /sys/fs/resctrl
  # mkdir p0
  # echo XXXX > /sys/fs/resctrl/p0/cpus *1=20
  # echo =1B$B!H=1B(BL1:0=3D000F,000F,000F,000F;1=3D000F,000F,000F,000F=1B$=
B!I=1B(B > /sys/fs/resctrl/p0/schemata*2
  # echo =1B$B!H=1B(BL2:0=3D000F,000F,0,0;1=3D0,0,000F,000F=1B$B!I=1B(B > /=
sys/fs/resctrl/p0/schemata=1B$B"(=1B(B2=20
  # echo PID > sys/fs/resctrl/p0/tasks

*1=20
   Since the A64FX L2 settings are shared by NUMA, all PEs (cores)=20
   on the same NUMA should be specified at the same time.=20
   In other words, we want to specify NUMAs instead of PEs to the=20
   resource group. Maybe it is better to change the interface to=20
   numas(/sys/fs/resctrl/p0/numas). Could you give me some advice?=20
*2=20
   L1:<cache_id0>=3D<cwbm>,<cwbm>,<cwbm>,<cwbm>;<cache_id1>=3D<cwbm>,<cwbm>=
,<cwbm>,<cwbm>;=1B$B!D=1B(B=20
   L2:<cache_id0>=3D<cwbm>,<cwbm>,<cwbm>,<cwbm>;<cache_id1>=3D<cwbm>,<cwbm>=
,<cwbm>,<cwbm>;=1B$B!D=1B(B=20
  =1B$B!&=1B(B L1:
     Add a new interface to control the L1D cache.=20
  =1B$B!&=1B(B <cwbm>,<cwbm>,<cwbm>,<cwbm>:
     Specify the number of ways for each sector. =20
     Each L1/L2 cache has 4 sectors.
     And it is needed to specify the number of ways for each sector.=20
  =1B$B!&=1B(B cwbm=1B$B!'=1B(B
     Specify the number of ways in each sector as a bitmap (percentage),=20
     but the bitmap does not indicate the position of the cache.=20
     The range is from 0 way to 16 ways.=20

When creating a resource group, the number of ways for 4 sectors of=20
L1&L2 is set. When running a task, the [56:57] bits of virtual=20
address are used for sector selection. Even different tasks running=20
in the same resource group can use different sector caches. Therefore,=20
when running a task that handles a large amount of infrequently used=20
data and a task that handles a large amount of frequently used data=20
at the same time, cache size limitation and cache space isolation can=20
be performed, and cache thrashing also can be reduced.=20

Is this approach acceptable? Could you give me some advice? =20

Best regards,
Tan Shaopeng