On Tue, Mar 02, 2021 at 11:59:40AM +1300, Barry Song wrote:
> From: Tim Chen <[email protected]>
>
> There are x86 CPU architectures (e.g. Jacobsville) where L2 cahce
> is shared among a cluster of cores instead of being exclusive
> to one single core.
Isn't that most atoms one way or another? Tremont seems to have it per 4
cores, but earlier it was per 2 cores.
On 3/2/21 2:30 AM, Peter Zijlstra wrote:
> On Tue, Mar 02, 2021 at 11:59:40AM +1300, Barry Song wrote:
>> From: Tim Chen <[email protected]>
>>
>> There are x86 CPU architectures (e.g. Jacobsville) where L2 cahce
>> is shared among a cluster of cores instead of being exclusive
>> to one single core.
>
> Isn't that most atoms one way or another? Tremont seems to have it per 4
> cores, but earlier it was per 2 cores.
>
Yes, older Atoms have 2 cores sharing L2. I probably should
rephrase my comments to not leave the impression that sharing
L2 among cores is new for Atoms.
Tremont based Atom CPUs increases the possible load imbalance more
with 4 cores per L2 instead of 2. And also with more overall cores on a die, the
chance increases for packing running tasks on a few clusters while leaving
others empty on light/medium loaded systems. We did see
this effect on Jacobsville.
So load balancing between the L2 clusters is more
useful on Tremont based Atom CPUs compared to the older Atoms.
Tim
> -----Original Message-----
> From: Tim Chen [mailto:[email protected]]
> Sent: Thursday, March 4, 2021 7:34 AM
> To: Peter Zijlstra <[email protected]>; Song Bao Hua (Barry Song)
> <[email protected]>
> Cc: [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; Jonathan Cameron <[email protected]>;
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> xuwei (O) <[email protected]>; Zengtao (B) <[email protected]>;
> [email protected]; yangyicong <[email protected]>; Liguozhu (Kenneth)
> <[email protected]>; [email protected]; [email protected]
> Subject: [Linuxarm] Re: [RFC PATCH v4 3/3] scheduler: Add cluster scheduler
> level for x86
>
>
>
> On 3/2/21 2:30 AM, Peter Zijlstra wrote:
> > On Tue, Mar 02, 2021 at 11:59:40AM +1300, Barry Song wrote:
> >> From: Tim Chen <[email protected]>
> >>
> >> There are x86 CPU architectures (e.g. Jacobsville) where L2 cahce
> >> is shared among a cluster of cores instead of being exclusive
> >> to one single core.
> >
> > Isn't that most atoms one way or another? Tremont seems to have it per 4
> > cores, but earlier it was per 2 cores.
> >
>
> Yes, older Atoms have 2 cores sharing L2. I probably should
> rephrase my comments to not leave the impression that sharing
> L2 among cores is new for Atoms.
>
> Tremont based Atom CPUs increases the possible load imbalance more
> with 4 cores per L2 instead of 2. And also with more overall cores on a die,
> the
> chance increases for packing running tasks on a few clusters while leaving
> others empty on light/medium loaded systems. We did see
> this effect on Jacobsville.
>
> So load balancing between the L2 clusters is more
> useful on Tremont based Atom CPUs compared to the older Atoms.
It seems sensible the more CPU we get in the cluster, the more
we need the kernel to be aware of its existence.
Tim, it is possible for you to bring up the cpu_cluster_mask and
cluster_sibling for x86 so that the topology can be represented
in sysfs and be used by scheduler? It seems your patch lacks this
part.
BTW, I wonder if x86 can do some improvement on your KMP_AFFINITY
by leveraging the cluster topology level.
https://software.intel.com/content/www/us/en/develop/documentation/cpp-compiler-developer-guide-and-reference/top/optimization-and-programming-guide/openmp-support/openmp-library-support/thread-affinity-interface-linux-and-windows.html
KMP_AFFINITY has thread affinity modes like compact and scatter,
it seems this "compact" and "scatter" can also use the cluster
information as you see we are also struggling with the "compact"
and "scatter" issues here in this patchset :-)
Thanks
Barry
> It seems sensible the more CPU we get in the cluster, the more
> we need the kernel to be aware of its existence.
>
> Tim, it is possible for you to bring up the cpu_cluster_mask and
> cluster_sibling for x86 so that the topology can be represented
> in sysfs and be used by scheduler? It seems your patch lacks this
> part.
You mean having something in /sys/devices/system/cpu/cpu0/topology
on cluster information so that an external program can
affinitize to a cluster if it prefers to do so?
Tim
>
> BTW, I wonder if x86 can do some improvement on your KMP_AFFINITY
> by leveraging the cluster topology level.
> https://software.intel.com/content/www/us/en/develop/documentation/cpp-compiler-developer-guide-and-reference/top/optimization-and-programming-guide/openmp-support/openmp-library-support/thread-affinity-interface-linux-and-windows.html
>
> KMP_AFFINITY has thread affinity modes like compact and scatter,
> it seems this "compact" and "scatter" can also use the cluster
> information as you see we are also struggling with the "compact"
> and "scatter" issues here in this patchset :-)
>
> Thanks
> Barry
>