Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
IronPort-SDR: nRWTZDV0Dnvm33wDm2gkmL79Zs+9Qasv0FaWlovH/dGUQObNJnLVMsQGTfGXR7l4T3a11SIjg1
 ZxjtH4VSu+pGYWiWmhzRyzBG27/dtJo+q5PJjwpyFNwu4uKDorzRKsA1v95KlSWsviDRQ0HUJ0
 H6UioPsq9iR3PdAsWNWjt83KDiM0eSjC4+dq25osuNeRgIipG1SCl58/rDzewKAhHckWkpi8lX
 2W/J7GF4mkDHbQn4zH5xH85V3ePyoV2sbkrUXV/R7Jqnh2Ni3KjI42ITlDY/dI4z7AWFHjQM1q
 g2w=
From:   "tan.shaopeng@fujitsu.com" <tan.shaopeng@fujitsu.com>
To:     'Reinette Chatre' <reinette.chatre@intel.com>,
        "'fenghua.yu@intel.com'" <fenghua.yu@intel.com>
CC:     "'linux-kernel@vger.kernel.org'" <linux-kernel@vger.kernel.org>,
        "'linux-arm-kernel@lists.infradead.org'" 
        <linux-arm-kernel@lists.infradead.org>,
        'James Morse' <james.morse@arm.com>,
        "misono.tomohiro@fujitsu.com" <misono.tomohiro@fujitsu.com>,
        "Luck, Tony" <tony.luck@intel.com>
Subject: RE: About add an A64FX cache control function into resctrl
Thread-Topic: About add an A64FX cache control function into resctrl
Thread-Index: AdctARUohIvM7pb1S52qTlM+pjp6RAJht/AAABE+WYABTmUCMABGH4GAA3M648AA3z+sgAC1LKkQAEYGy4AAH1dN4A==
Date:   Thu, 27 May 2021 08:45:49 +0000
Message-ID: <TYAPR01MB6330E119C33C0349EB6622308B239@TYAPR01MB6330.jpnprd01.prod.outlook.com>
References: <OSAPR01MB214600C7923AEF7C35B02E648B739@OSAPR01MB2146.jpnprd01.prod.outlook.com>
 <OSAPR01MB214657641D532FB8D112DD528B479@OSAPR01MB2146.jpnprd01.prod.outlook.com>
 <bb0967c0-5b88-c6c2-0242-1e3928189a04@intel.com>
 <OSAPR01MB2146D42FC04779268BA231878B409@OSAPR01MB2146.jpnprd01.prod.outlook.com>
 <14ddf86b-89e1-ba26-b684-f3d5d5f8ade7@intel.com>
 <TYAPR01MB633071CD547B0AAF818520E48B2D9@TYAPR01MB6330.jpnprd01.prod.outlook.com>
 <f53b9bfd-0d55-4cf2-fabb-82b3ec86e52a@intel.com>
 <TYAPR01MB6330758A49FECF90B5290C768B259@TYAPR01MB6330.jpnprd01.prod.outlook.com>
 <89416df1-4e91-8ad2-981c-827808a65229@intel.com>
In-Reply-To: <89416df1-4e91-8ad2-981c-827808a65229@intel.com>
Accept-Language: ja-JP, en-US
Content-Language: ja-JP
Content-Type: text/plain; charset="iso-2022-jp"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: b27dcd05-08ba-4e9c-08e6-08d920ebd3de
X-MS-Exchange-CrossTenant-originalarrivaltime: 27 May 2021 08:45:49.2816
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: a19f121d-81e1-4858-a9d8-736e267fd4c7
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: aihk8XRxa6j5ivnn1vjJy65iyRX/rSrFmU4jOowx0p8EyazfTzTiPBPkkQldAYDnqiU+XO2HTDIP706ca0ZnJJVJBrmTywfKPxEpVKP/cyE=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: TYYPR01MB7118
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Reinette,

> On 5/25/2021 1:45 AM, tan.shaopeng@fujitsu.com wrote:
> > Hi Reinette,
> >
> > Sorry, I have not explained A64FX's sector cache function well yet.
> > I think I need explain this function from different perspective.
>=20
> You have explained the A64FX's sector cache function well. I have also re=
ad
> both specs to understand it better. It appears to me that you are not con=
sidering
> the resctrl architecture as part of your solution but instead just forcin=
g your
> architecture onto the resctrl filesystem. For example, in resctrl the res=
ource
> groups are not just a directory structure but has significance in what is=
 being
> represented within the directory (a class of service). The files within a=
 resource
> group's directory build on that. From your side I have not seen any effor=
t in
> aligning the sector cache function with the resctrl architecture but inst=
ead you
> are just changing resctrl interface to match the A64FX architecture.
>=20
> Could you please take a moment to understand what resctrl is and how it c=
ould
> be mapped to A64FX in a coherent way?

Thanks for your mail.
Sorry, I=1B$B!G=1B(Bm wrong in understanding how to use=20
/sys/fs/resctrl/p0/cpus and /sys/fs/resctrl/tasks.=20
I think I have not understood resctrl yet, and I will learn more about it.
If I have questions, please allow me to mail you.

> >> On 5/17/2021 1:31 AM, tan.shaopeng@fujitsu.com wrote:
>=20
> > --------
> > A64FX NUMA-PE-Cache Architecture:
> > NUMA0:
> >    PE0:
> >      L1sector0,L1sector1,L1sector2,L1sector3
> >    PE1:
> >      L1sector0,L1sector1,L1sector2,L1sector3
> >    ...
> >    PE11:
> >      L1sector0,L1sector1,L1sector2,L1sector3
> >
> >    L2sector0,1/L2sector2,3
> > NUMA1:
> >    PE0:
> >      L1sector0,L1sector1,L1sector2,L1sector3
> >    ...
> >    PE11:
> >      L1sector0,L1sector1,L1sector2,L1sector3
> >
> >    L2sector0,1/L2sector2,3
> > NUMA2:
> >    ...
> > NUMA3:
> >    ...
> > --------
> > In A64FX processor, one L1 sector cache capacity setting register is
> > only for one PE and not shared among PEs. L2 sector cache maximum
> > capacity setting registers are shared among PEs in same NUMA, and it
> > is to be noted that changing these registers in one PE influences other=
 PE.
>=20
> Understood. cache affinity is familiar to resctrl. When a CPU becomes onl=
ine it
> is discovered which caches/resources it has affinity to.
> Resources then have CPU mask associated with them to indicate on which
> CPU a register could be changed to configure the resource/cache. See
> domain_add_cpu() and struct rdt_domain.
>=20
> > The number of ways for L2 Sector ID (0,1 or 2,3) can be set through
> > any PEs in same NUMA. The sector ID 0,1 and 2,3 are not available at
> > the same time in same NUMA.
> >
> >
> > I think, in your idea, a resource group will be created for each sector=
 ID.
> > (> "sectors" could be considered the same as the resctrl "classes of
> > service") Then, an example of resource group is created as follows.
> > =1B$B!&=1B(B L1: NUMAX-PEY-L1sector0 (X =3D 0,1,2,3.Y =3D 0,1,2 ... 11)=
,
> > =1B$B!&=1B(B L2: NUMAX-L2sector0 (X =3D 0,1,2,3)
> >
> > In this example, sector with same ID(0) of all PEs is allocated to
> > resource group. The L1D caches are numbered from
> > NUMA0_PE0-L1sector0(0) to NUMA4_PE11-L1sector0(47) and the L2
> caches
> > numbered from
> > NUMA0-L2sector0(0) to NUM4-L2sector0(3).
> > (NUMA number X is from 0-4, PE number Y is from 0-11)
> > (1) The number of ways of NUMAX-PEY-L1sector0 can be set independently
> >      for each PEs (0-47). When run a task on this resource group,
> >      we cannot control on which PE the task is running on and how many
> >      cache ways the task is using.
>=20
> resctrl does not control the affinity on which PE/CPU a task is run.
> resctrl is an interface with which to configure how resources are allocat=
ed on
> the system. resctrl could thus provide interface with which each sector o=
f each
> cache instance is assigned a number of cache ways.
> resctrl also provides an interface to assign a task with a class of servi=
ce (sector
> id?). Through this the task obtains access to all resources that is alloc=
ated to
> the particular class of service (sector id?). Depending on which CPU the =
task is
> running it may indeed experience different performance if the sector id i=
t is
> running with does not have the same allocations on all cache instances. T=
he
> affinity of the task needs to be managed separately using for example tas=
kset.
> Please see Documentation/x86/resctrl.rst "Examples for RDT allocation usa=
ge"
>=20
> > (2) Since L2 can only use 2 sectors at a time, when creating more than
> >      2 resource groups, L2setctor0 will have to be allocated to a
> >      different resource group. If the L2sector0 is shared by different
> >      resource groups, the L2 sector settings on resource group will be
> >      influenced by each other.
> > etc... there are various problems, and no merit to using resctrl.
> >
> >
> > In my idea, in order to allocate the L1 and L2 cache to a resource
> > group, allocate NUMA to the resource group.
> > An example of resource group is as follows.
> > =1B$B!&=1B(B NUMA0-PEY-L1sectorZ (Y =3D 0,1,2...11. Z =3D 0,1,2,3)
> > =1B$B!&=1B(B NUMA0-L2sectorZZ (ZZ =3D 0,1,2,3)
> >
> >    #cat /sys/fs/resctrl/p0/cpus
> >    0-11 *1
> >    #cat /sys/fs/resctrl/p0/schemata
> >    L1:0=3D0xF,0x3,0x1,x0x0 *2
> >    L2:0=3D0xFFF,0xF,0,0 *3
> >
> > *1: PEs belong one NUMA. (Of course, multiple NUMAs can also be
> >      specified in one resource group)
> > *2: The number of ways for L1sector0,1,2,3. On this resource group
> >      the number of ways of all sector0 is the same(0xF). If 0 way is
> >      specified for one sector, this sector cannot be used. If 4(0xF)
> >      ways are specified for one sector, this sector can use cache fully=
.
> >      If 4 ways are specified for each sector, there will be no
> >      restriction for using cache.
> > *3: The number of ways for L2 sector 0,1. If L2sector0,1 is used,
> >      the number of ways of L2sector2,3 must be set to 0.
> >
> > All sectors with the same ID on the same resource group were set to
> > the same number of ways, and when running a task on A64FX, the sector
> > ID used by task is determined by [56:57] bits of virtual address.
> > By specifying the PID to /sys/fs/resctrl/tasks, the task will be bound
> > to the resource group, and then, the cache size used by task will not
> > be changed never.
>=20
> This completely ignores how this directory and files are currently used.
> What is missing how this implementation maps to the current resctrl
> architecture.

Best regards,
Tan Shaopeng