I'm taking over the series Michael sent previously [1] which is smartly
reviewing the initial series I sent [2]. This series is addressing the
comments sent by Thomas and me on the Michael's one.
Here is a short introduction to the issue this series is addressing:
When a new CPU is added, the kernel is activating all its threads. This
leads to weird, but functional, result when adding CPU on a SMT 4 system
for instance.
Here the newly added CPU 1 has 8 threads while the other one has 4 threads
active (system has been booted with the 'smt-enabled=4' kernel option):
ltcden3-lp12:~ # ppc64_cpu --info
Core 0: 0* 1* 2* 3* 4 5 6 7
Core 1: 8* 9* 10* 11* 12* 13* 14* 15*
This mixed SMT level may confused end users and/or some applications.
There is no SMT level recorded in the kernel (common code), neither in user
space, as far as I know. Such a level is helpful when adding new CPU or
when optimizing the energy efficiency (when reactivating CPUs).
When SMP and HOTPLUG_SMT are defined, this series is adding a new SMT level
(cpu_smt_num_threads) and few callbacks allowing the architecture code to
fine control this value, setting a max and a "at boot" level, and
controling whether a thread should be onlined or not.
v2:
As Thomas suggested,
Reword some commit's description
Remove topology_smt_supported()
Remove topology_smt_threads_supported()
Introduce CONFIG_SMT_NUM_THREADS_DYNAMIC
Remove switch() in __store_smt_control()
Update kernel-parameters.txt
[1] https://lore.kernel.org/linuxppc-dev/[email protected]/
[2] https://lore.kernel.org/linuxppc-dev/[email protected]/
Laurent Dufour (1):
cpu/SMT: Remove topology_smt_supported()
Michael Ellerman (8):
cpu/SMT: Move SMT prototypes into cpu_smt.h
cpu/SMT: Move smt/control simple exit cases earlier
cpu/SMT: Store the current/max number of threads
cpu/SMT: Create topology_smt_thread_allowed()
cpu/SMT: Allow enabling partial SMT states via sysfs
powerpc/pseries: Initialise CPU hotplug callbacks earlier
powerpc: Add HOTPLUG_SMT support
powerpc/pseries: Honour current SMT state when DLPAR onlining CPUs
.../ABI/testing/sysfs-devices-system-cpu | 1 +
.../admin-guide/kernel-parameters.txt | 4 +-
arch/Kconfig | 3 +
arch/powerpc/Kconfig | 2 +
arch/powerpc/include/asm/topology.h | 15 +++
arch/powerpc/kernel/smp.c | 8 +-
arch/powerpc/platforms/pseries/hotplug-cpu.c | 30 +++--
arch/powerpc/platforms/pseries/pseries.h | 2 +
arch/powerpc/platforms/pseries/setup.c | 2 +
arch/x86/include/asm/topology.h | 4 +-
arch/x86/kernel/cpu/bugs.c | 3 +-
arch/x86/kernel/smpboot.c | 8 --
include/linux/cpu.h | 25 +---
include/linux/cpu_smt.h | 33 +++++
kernel/cpu.c | 116 ++++++++++++++----
15 files changed, 185 insertions(+), 71 deletions(-)
create mode 100644 include/linux/cpu_smt.h
--
2.41.0
> On 28-Jun-2023, at 3:35 PM, Laurent Dufour <[email protected]> wrote:
>
> I'm taking over the series Michael sent previously [1] which is smartly
> reviewing the initial series I sent [2]. This series is addressing the
> comments sent by Thomas and me on the Michael's one.
>
> Here is a short introduction to the issue this series is addressing:
>
> When a new CPU is added, the kernel is activating all its threads. This
> leads to weird, but functional, result when adding CPU on a SMT 4 system
> for instance.
>
> Here the newly added CPU 1 has 8 threads while the other one has 4 threads
> active (system has been booted with the 'smt-enabled=4' kernel option):
>
> ltcden3-lp12:~ # ppc64_cpu --info
> Core 0: 0* 1* 2* 3* 4 5 6 7
> Core 1: 8* 9* 10* 11* 12* 13* 14* 15*
>
> This mixed SMT level may confused end users and/or some applications.
>
Thanks for the patches Laurent.
Is the SMT level retained even when dynamically changing SMT values?
I am observing difference in behaviour with and without smt-enabled
kernel command line option.
When smt-enabled= option is specified SMT level is retained across
cpu core remove and add.
Without this option but changing SMT level during runtime using
ppc64_cpu —smt=<level>, the SMT level is not retained after
cpu core add.
[root@ltcden8-lp8 ~]# ppc64_cpu —smt=4
[root@ltcden8-lp8 ~]# ppc64_cpu --info
Core 0: 0* 1* 2* 3* 4 5 6 7
Core 1: 8* 9* 10* 11* 12 13 14 15
Core 2: 16* 17* 18* 19* 20 21 22 23
Core 3: 24* 25* 26* 27* 28 29 30 31
Remove a core, SMT level is retained.
[root@ltcden8-lp8 ~]# ppc64_cpu --info
Core 0: 0* 1* 2* 3* 4 5 6 7
Core 1: 8* 9* 10* 11* 12 13 14 15
Core 2: 16* 17* 18* 19* 20 21 22 23
[root@ltcden8-lp8 ~]#
Add 3 cores, SMT level is not retained.
[ 496.600648] Fallback order for Node 1: 1 0
[ 496.600655] Built 1 zonelists, mobility grouping on. Total pages: 1228159
[ 496.600676] Policy zone: Normal
[ 496.661173] WARNING: workqueue cpumask: online intersect > possible intersect
[ 499.530646] Fallback order for Node 3: 3 0
[ 499.530655] Built 2 zonelists, mobility grouping on. Total pages: 1228159
[ 499.530675] Policy zone: Normal
ppc64_cpu --info
Core 0: 0* 1* 2* 3* 4 5 6 7
Core 1: 8* 9* 10* 11* 12 13 14 15
Core 2: 16* 17* 18* 19* 20 21 22 23
Core 3: 24* 25* 26* 27* 28* 29* 30* 31*
Core 4: 32* 33* 34* 35* 36* 37* 38* 39*
Core 5: 40* 41* 42* 43* 44* 45* 46* 47*
[root@ltcden8-lp8 ~]#
- Sachin
Sachin Sant <[email protected]> writes:
>> On 28-Jun-2023, at 3:35 PM, Laurent Dufour <[email protected]> wrote:
>>
>> I'm taking over the series Michael sent previously [1] which is smartly
>> reviewing the initial series I sent [2]. This series is addressing the
>> comments sent by Thomas and me on the Michael's one.
>>
>> Here is a short introduction to the issue this series is addressing:
>>
>> When a new CPU is added, the kernel is activating all its threads. This
>> leads to weird, but functional, result when adding CPU on a SMT 4 system
>> for instance.
>>
>> Here the newly added CPU 1 has 8 threads while the other one has 4 threads
>> active (system has been booted with the 'smt-enabled=4' kernel option):
>>
>> ltcden3-lp12:~ # ppc64_cpu --info
>> Core 0: 0* 1* 2* 3* 4 5 6 7
>> Core 1: 8* 9* 10* 11* 12* 13* 14* 15*
>>
>> This mixed SMT level may confused end users and/or some applications.
>>
>
> Thanks for the patches Laurent.
>
> Is the SMT level retained even when dynamically changing SMT values?
> I am observing difference in behaviour with and without smt-enabled
> kernel command line option.
>
> When smt-enabled= option is specified SMT level is retained across
> cpu core remove and add.
>
> Without this option but changing SMT level during runtime using
> ppc64_cpu —smt=<level>, the SMT level is not retained after
> cpu core add.
That's because ppc64_cpu is not using the sysfs SMT control file, it's
just onlining/offlining threads manually.
If you run:
$ ppc64_cpu --smt=4
And then also do:
$ echo 4 > /sys/devices/system/cpu/smt/control
It should work as expected?
ppc64_cpu will need to be updated to do that automatically.
cheers
Le 29/06/2023 à 13:10, Michael Ellerman a écrit :
> Sachin Sant <[email protected]> writes:
>>> On 28-Jun-2023, at 3:35 PM, Laurent Dufour <[email protected]> wrote:
>>>
>>> I'm taking over the series Michael sent previously [1] which is smartly
>>> reviewing the initial series I sent [2]. This series is addressing the
>>> comments sent by Thomas and me on the Michael's one.
>>>
>>> Here is a short introduction to the issue this series is addressing:
>>>
>>> When a new CPU is added, the kernel is activating all its threads. This
>>> leads to weird, but functional, result when adding CPU on a SMT 4 system
>>> for instance.
>>>
>>> Here the newly added CPU 1 has 8 threads while the other one has 4 threads
>>> active (system has been booted with the 'smt-enabled=4' kernel option):
>>>
>>> ltcden3-lp12:~ # ppc64_cpu --info
>>> Core 0: 0* 1* 2* 3* 4 5 6 7
>>> Core 1: 8* 9* 10* 11* 12* 13* 14* 15*
>>>
>>> This mixed SMT level may confused end users and/or some applications.
>>>
>>
>> Thanks for the patches Laurent.
>>
>> Is the SMT level retained even when dynamically changing SMT values?
>> I am observing difference in behaviour with and without smt-enabled
>> kernel command line option.
>>
>> When smt-enabled= option is specified SMT level is retained across
>> cpu core remove and add.
>>
>> Without this option but changing SMT level during runtime using
>> ppc64_cpu —smt=<level>, the SMT level is not retained after
>> cpu core add.
>
> That's because ppc64_cpu is not using the sysfs SMT control file, it's
> just onlining/offlining threads manually.
>
> If you run:
> $ ppc64_cpu --smt=4
>
> And then also do:
>
> $ echo 4 > /sys/devices/system/cpu/smt/control
>
> It should work as expected?
>
> ppc64_cpu will need to be updated to do that automatically.
Hi Sachin and Michael,
Yes, ppc64_cpu will need an update, and I have a patch ready to be sent
once this series will be accepted.
By the way, I've a fix for the build issue reported against the patch
6/9. I'll send a v3 soon.
Cheers,
Laurent.
>>
>> Without this option but changing SMT level during runtime using
>> ppc64_cpu —smt=<level>, the SMT level is not retained after
>> cpu core add.
>
> That's because ppc64_cpu is not using the sysfs SMT control file, it's
> just onlining/offlining threads manually.
>
> If you run:
> $ ppc64_cpu --smt=4
>
> And then also do:
>
> $ echo 4 > /sys/devices/system/cpu/smt/control
>
> It should work as expected?
Thanks Michael. Yes this works. The SMT level is preserved
after a core add.
- Sachin