Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
IronPort-SDR: rak3RM+j+Ym8eVIFSqjG0UXKoN4uhY1nRSetlQ7YjWuVeF685k4GzZVrzM5qKeQ4Kp4HKpcmfK
 vspiRI4Onap3MbUhPxCCm/DqVEtUpRxiz92r2GY7UPkInym5FfvCR+rv76PJrlODJmMNja3Ao7
 SvCmc0RdqbRHs+PYRR+zO2apP+Y9dxm+ObTrPavp8wd3rNPaf6ohCaM0xEO3t3VDRgjWfdRuNA
 KzlhjKZrGKM4kaES7RY7FjLYAtkwVRA9iVL++DoskUt32DjCkbqG722J/lAVAbVfgL7+9Uc3MY
 X1k=
From:   "tan.shaopeng@fujitsu.com" <tan.shaopeng@fujitsu.com>
To:     'Reinette Chatre' <reinette.chatre@intel.com>,
        "'fenghua.yu@intel.com'" <fenghua.yu@intel.com>
CC:     "'linux-kernel@vger.kernel.org'" <linux-kernel@vger.kernel.org>,
        "'linux-arm-kernel@lists.infradead.org'" 
        <linux-arm-kernel@lists.infradead.org>,
        'James Morse' <james.morse@arm.com>,
        "misono.tomohiro@fujitsu.com" <misono.tomohiro@fujitsu.com>,
        "Luck, Tony" <tony.luck@intel.com>
Subject: RE: About add an A64FX cache control function into resctrl
Thread-Topic: About add an A64FX cache control function into resctrl
Thread-Index: AdctARUohIvM7pb1S52qTlM+pjp6RAJht/AAABE+WYABTmUCMABGH4GAA3M648A=
Date:   Mon, 17 May 2021 08:31:30 +0000
Message-ID: <TYAPR01MB633071CD547B0AAF818520E48B2D9@TYAPR01MB6330.jpnprd01.prod.outlook.com>
References: <OSAPR01MB214600C7923AEF7C35B02E648B739@OSAPR01MB2146.jpnprd01.prod.outlook.com>
 <OSAPR01MB214657641D532FB8D112DD528B479@OSAPR01MB2146.jpnprd01.prod.outlook.com>
 <bb0967c0-5b88-c6c2-0242-1e3928189a04@intel.com>
 <OSAPR01MB2146D42FC04779268BA231878B409@OSAPR01MB2146.jpnprd01.prod.outlook.com>
 <14ddf86b-89e1-ba26-b684-f3d5d5f8ade7@intel.com>
In-Reply-To: <14ddf86b-89e1-ba26-b684-f3d5d5f8ade7@intel.com>
Accept-Language: ja-JP, en-US
Content-Language: ja-JP
Content-Type: text/plain; charset="iso-2022-jp"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 218750a8-5057-4450-6460-08d9190e2c1a
X-MS-Exchange-CrossTenant-originalarrivaltime: 17 May 2021 08:31:30.9520
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: a19f121d-81e1-4858-a9d8-736e267fd4c7
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: hS1krldgLwk/iqhKnv5GGYlYczxYClO75CBnSi97nkW2I0+g7GijDh4w9XTsVQ56Qt7iL1gPFGImEjsZ1bGac40jwmal6J0GE5exSCo2SXM=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: TY1PR01MB1883
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Reinette,

I=1B$B!G=1B(Bm sorry for the late reply.=20
I think I could not explain A64FX=1B$B!G=1B(Bs sector cache function well i=
n=20
my first mail. While answering the question, I will also explain=20
this function in more detail. Though maybe you have already learned=20
more about this function by reading specification and manual,=20
in order to better understand this function, some contents may have=20
duplicate explanations.

> >> The overview in section 12 was informative but very high level.
> >
> > I'm considering how to answer your questions from your email which I
> > received before, when I check the email again, I am sorry that the
> > information I provided before are insufficient.
> >
> > To understand the sector cache function of A64FX, could you please see
> > A64FX_Microarchitecture_Manual - section 12. Sector Cache
> >
> https://github.com/fujitsu/A64FX/blob/master/doc/A64FX_Microarchitectu
> > re_Manual_en_1.4.pdf
> > and,
> > A64FX_Specification_HPC_Extension ? section 1.2. Sector Cache
> >
> https://github.com/fujitsu/A64FX/blob/master/doc/A64FX_Specification_H
> > PC_Extension_v1_EN.pdf
>=20
> Thank you for the direct links - I missed that there are two documents av=
ailable.
>=20
> After reading the spec portion it does seem to me even more as though
> "sectors" could be considered the same as the resctrl "classes of service=
". The
> Fujitsu hardware supports four sectors that can be configured with differ=
ent
> number of ways using the registers you mention above. In resctrl this cou=
ld be
> considered as hardware that supports four classes of service and each cla=
ss of
> service can be allocated a different number of ways.

Fujitsu hardware supports four sectors that can be configured with=20
different number of ways by using "IMP_SCCR" registers, and when this=20
function is added into resctrl, the maximum ways of each sector are=20
indicated by bitmap.=20

However, A64FX's L2 cache setting registers are shared among PEs=20
(Processor Element) in NUMA. If two PEs in the same NUMA are assigned=20
to different resource groups, changing one PE's L2 setting on one=20
resource group, the other PE's L2 setting on other resource groups=20
will be influenced. So, adding this function into resctrl, we will=20
assign NUMA to the resource group. (On F64FX, each NUMA has 12 PEs,=20
and each PE has L1 cache setting registers, but these registers are=20
not shared.) There are 4 NUMAs on A64FX, 4 NUMAs could be considered=20
as hardware that supports four classes of service at most, and each=20
class of service has 4 sectors (4 L1 sectors& 4 L2 sectors),=20
and each sector can be allocated a different number of ways.=20
And, when a running task on resource group, the [56:57] bits of=20
virtual address are used for sector selection (cache affinity).

> The other part is how hardware knows which sector is being used at any
> moment in time. In resctrl that is programmed by writing the active class=
 of
> service into needed register at the time the application is context switc=
hed
> (resctrl_sched_in()). This seems different here since as you describe the
> sector is chosen by bits in the address. Even so, which bits to set in th=
e
> address needs to be programmed also and I also understand that there is a
> "default" sector that can be programmed via register. Could these be equi=
valent
> to what is done currently in resctrl?

Adding this function into resctrl, there is no need to write active=20
class of service into needed register. When running a task, the sector=20
id is decided by [56:57] bits of virtual address, and these bits are=20
programed by users. When creating a resource group, the maximum number=20
of ways of each sector are set by "IMP_SCCR" setting registers.=20
As long as the task is running in a certain resource group, the sector=20
and the maximum number of ways of sectors are used will not be changed.=20
Therefore, we need not consider context switches on A64FX.

> (Could you please also consider my original questions?)
I will reply to the original questions mail.=20


Best regards,
Tan Shaopeng