When a partition is transferred, once it arrives at the destination node,
the partition is active but much of its memory must be transferred from the
start node.
It depends on the activity in the partition, but the more CPU the partition
has, the more memory to be transferred is likely to be. This causes latency
when accessing pages that need to be transferred, and often, for large
partitions, it triggers the NMI watchdog.
The NMI watchdog causes the CPU stack to dump where it appears to be
stuck. In this case, it does not bring much information since it can happen
during any memory access of the kernel.
In addition, the NMI interrupt mechanism is not secure and can generate a
dump system in the event that the interruption is taken while MSR[RI]=0.
Depending on the LPAR size and load, it may be interesting to extend the
NMI watchdog timer during the LPM.
That's configurable through sysctl with the new introduced variable
(specific to powerpc) nmi_watchdog_factor. This value represents the
percentage added to watchdog_tresh to set the NMI watchdog timeout during a
LPM.
Changes in v3:
- don't export watchdog_mutex
- fix a comment in mobilty.c, wait_for_vasi_session_completed()
- fix a build issue when !CONFIG_PPC_WATCHDOG
- rework some printk and rename the sysctl variable.
v2:
https://lore.kernel.org/all/[email protected]/
Laurent Dufour (4):
powerpc/mobility: wait for memory transfer to complete
watchdog: export lockup_detector_reconfigure
powerpc/watchdog: introduce a NMI watchdog's factor
pseries/mobility: set NMI watchdog factor during LPM
Documentation/admin-guide/sysctl/kernel.rst | 12 +++
arch/powerpc/include/asm/nmi.h | 2 +
arch/powerpc/kernel/watchdog.c | 21 ++++-
arch/powerpc/platforms/pseries/mobility.c | 85 ++++++++++++++++++++-
include/linux/nmi.h | 2 +
kernel/watchdog.c | 21 +++--
6 files changed, 135 insertions(+), 8 deletions(-)
--
2.36.1
Excerpts from Laurent Dufour's message of June 27, 2022 11:53 pm:
> When a partition is transferred, once it arrives at the destination node,
> the partition is active but much of its memory must be transferred from the
> start node.
>
> It depends on the activity in the partition, but the more CPU the partition
> has, the more memory to be transferred is likely to be. This causes latency
> when accessing pages that need to be transferred, and often, for large
> partitions, it triggers the NMI watchdog.
Importantly, it can require page in of code that runs with irqs
disabled, which is unlike a guest normally runs under PowerVM
(but it can under KVM) which is why we enabled the watchdog under
PowerVM but not KVM. So, okay it makes sense to mak an exception
for this case.
Thanks,
Nick
> The NMI watchdog causes the CPU stack to dump where it appears to be
> stuck. In this case, it does not bring much information since it can happen
> during any memory access of the kernel.
>
> In addition, the NMI interrupt mechanism is not secure and can generate a
> dump system in the event that the interruption is taken while MSR[RI]=0.
>
> Depending on the LPAR size and load, it may be interesting to extend the
> NMI watchdog timer during the LPM.
>
> That's configurable through sysctl with the new introduced variable
> (specific to powerpc) nmi_watchdog_factor. This value represents the
> percentage added to watchdog_tresh to set the NMI watchdog timeout during a
> LPM.
>
> Changes in v3:
> - don't export watchdog_mutex
> - fix a comment in mobilty.c, wait_for_vasi_session_completed()
> - fix a build issue when !CONFIG_PPC_WATCHDOG
> - rework some printk and rename the sysctl variable.
>
> v2:
> https://lore.kernel.org/all/[email protected]/
>
> Laurent Dufour (4):
> powerpc/mobility: wait for memory transfer to complete
> watchdog: export lockup_detector_reconfigure
> powerpc/watchdog: introduce a NMI watchdog's factor
> pseries/mobility: set NMI watchdog factor during LPM
>
> Documentation/admin-guide/sysctl/kernel.rst | 12 +++
> arch/powerpc/include/asm/nmi.h | 2 +
> arch/powerpc/kernel/watchdog.c | 21 ++++-
> arch/powerpc/platforms/pseries/mobility.c | 85 ++++++++++++++++++++-
> include/linux/nmi.h | 2 +
> kernel/watchdog.c | 21 +++--
> 6 files changed, 135 insertions(+), 8 deletions(-)
>
> --
> 2.36.1
>
>
Le 12/07/2022 à 03:21, Nicholas Piggin a écrit :
> Excerpts from Laurent Dufour's message of June 27, 2022 11:53 pm:
>> When a partition is transferred, once it arrives at the destination node,
>> the partition is active but much of its memory must be transferred from the
>> start node.
>>
>> It depends on the activity in the partition, but the more CPU the partition
>> has, the more memory to be transferred is likely to be. This causes latency
>> when accessing pages that need to be transferred, and often, for large
>> partitions, it triggers the NMI watchdog.
>
> Importantly, it can require page in of code that runs with irqs
> disabled, which is unlike a guest normally runs under PowerVM
> (but it can under KVM) which is why we enabled the watchdog under
> PowerVM but not KVM. So, okay it makes sense to mak an exception
> for this case.
On PowerVM, irqs disabled code may trigger multiple page in of data, which
is generating latencies too. Furthermore, the hypervisor is currently not
prioritizing page in operation based on the faulting CPU state. So the
kernel may have to wait for user page in operations to last.
> Thanks,
> Nick
>
>> The NMI watchdog causes the CPU stack to dump where it appears to be
>> stuck. In this case, it does not bring much information since it can happen
>> during any memory access of the kernel.
>>
>> In addition, the NMI interrupt mechanism is not secure and can generate a
>> dump system in the event that the interruption is taken while MSR[RI]=0.
>>
>> Depending on the LPAR size and load, it may be interesting to extend the
>> NMI watchdog timer during the LPM.
>>
>> That's configurable through sysctl with the new introduced variable
>> (specific to powerpc) nmi_watchdog_factor. This value represents the
>> percentage added to watchdog_tresh to set the NMI watchdog timeout during a
>> LPM.
>>
>> Changes in v3:
>> - don't export watchdog_mutex
>> - fix a comment in mobilty.c, wait_for_vasi_session_completed()
>> - fix a build issue when !CONFIG_PPC_WATCHDOG
>> - rework some printk and rename the sysctl variable.
>>
>> v2:
>> https://lore.kernel.org/all/[email protected]/
>>
>> Laurent Dufour (4):
>> powerpc/mobility: wait for memory transfer to complete
>> watchdog: export lockup_detector_reconfigure
>> powerpc/watchdog: introduce a NMI watchdog's factor
>> pseries/mobility: set NMI watchdog factor during LPM
>>
>> Documentation/admin-guide/sysctl/kernel.rst | 12 +++
>> arch/powerpc/include/asm/nmi.h | 2 +
>> arch/powerpc/kernel/watchdog.c | 21 ++++-
>> arch/powerpc/platforms/pseries/mobility.c | 85 ++++++++++++++++++++-
>> include/linux/nmi.h | 2 +
>> kernel/watchdog.c | 21 +++--
>> 6 files changed, 135 insertions(+), 8 deletions(-)
>>
>> --
>> 2.36.1
>>
>>