hardlockup_detector_event_create() indirectly calls
kmem_cache_alloc_node(), which is blockable.
So here, the really planned context is is_percpu_thread().
Signed-off-by: Pingfan Liu <[email protected]>
Cc: Petr Mladek <[email protected]>
Cc: Andrew Morton <[email protected]>
Cc: Wang Qing <[email protected]>
Cc: "Peter Zijlstra (Intel)" <[email protected]>
Cc: Santosh Sivaraj <[email protected]>
Cc: Sumit Garg <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Mark Rutland <[email protected]>
To: [email protected]
---
kernel/watchdog_hld.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/kernel/watchdog_hld.c b/kernel/watchdog_hld.c
index 247bf0b1582c..6876e796dbf5 100644
--- a/kernel/watchdog_hld.c
+++ b/kernel/watchdog_hld.c
@@ -165,10 +165,13 @@ static void watchdog_overflow_callback(struct perf_event *event,
static int hardlockup_detector_event_create(void)
{
- unsigned int cpu = smp_processor_id();
+ unsigned int cpu;
struct perf_event_attr *wd_attr;
struct perf_event *evt;
+ /* This function plans to execute in cpu bound kthread */
+ BUG_ON(!is_percpu_thread());
+ cpu = raw_smp_processor_id();
wd_attr = &wd_hw_attr;
wd_attr->sample_period = hw_nmi_get_sample_period(watchdog_thresh);
--
2.31.1
On Wed, 15 Sep 2021 11:51:00 +0800 Pingfan Liu <[email protected]> wrote:
> hardlockup_detector_event_create() indirectly calls
> kmem_cache_alloc_node(), which is blockable.
>
> So here, the really planned context is is_percpu_thread().
>
> ...
>
> --- a/kernel/watchdog_hld.c
> +++ b/kernel/watchdog_hld.c
> @@ -165,10 +165,13 @@ static void watchdog_overflow_callback(struct perf_event *event,
>
> static int hardlockup_detector_event_create(void)
> {
> - unsigned int cpu = smp_processor_id();
> + unsigned int cpu;
> struct perf_event_attr *wd_attr;
> struct perf_event *evt;
>
> + /* This function plans to execute in cpu bound kthread */
> + BUG_ON(!is_percpu_thread());
Can we avoid adding the BUG()? Find a way to emit a WARNing and then
permit the kernel to continue?
> + cpu = raw_smp_processor_id();
> wd_attr = &wd_hw_attr;
> wd_attr->sample_period = hw_nmi_get_sample_period(watchdog_thresh);
On Wed, Sep 15, 2021 at 11:51:00AM +0800, Pingfan Liu wrote:
> hardlockup_detector_event_create() indirectly calls
> kmem_cache_alloc_node(), which is blockable.
>
> So here, the really planned context is is_percpu_thread().
>
> Signed-off-by: Pingfan Liu <[email protected]>
> Cc: Petr Mladek <[email protected]>
> Cc: Andrew Morton <[email protected]>
> Cc: Wang Qing <[email protected]>
> Cc: "Peter Zijlstra (Intel)" <[email protected]>
> Cc: Santosh Sivaraj <[email protected]>
> Cc: Sumit Garg <[email protected]>
> Cc: Will Deacon <[email protected]>
> Cc: Mark Rutland <[email protected]>
> To: [email protected]
> ---
> kernel/watchdog_hld.c | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/watchdog_hld.c b/kernel/watchdog_hld.c
> index 247bf0b1582c..6876e796dbf5 100644
> --- a/kernel/watchdog_hld.c
> +++ b/kernel/watchdog_hld.c
> @@ -165,10 +165,13 @@ static void watchdog_overflow_callback(struct perf_event *event,
>
> static int hardlockup_detector_event_create(void)
> {
> - unsigned int cpu = smp_processor_id();
> + unsigned int cpu;
> struct perf_event_attr *wd_attr;
> struct perf_event *evt;
>
> + /* This function plans to execute in cpu bound kthread */
> + BUG_ON(!is_percpu_thread());
> + cpu = raw_smp_processor_id();
> wd_attr = &wd_hw_attr;
> wd_attr->sample_period = hw_nmi_get_sample_period(watchdog_thresh);
This patch makes no sense.
On Tue, Sep 14, 2021 at 09:06:27PM -0700, Andrew Morton wrote:
> On Wed, 15 Sep 2021 11:51:00 +0800 Pingfan Liu <[email protected]> wrote:
>
> > hardlockup_detector_event_create() indirectly calls
> > kmem_cache_alloc_node(), which is blockable.
> >
> > So here, the really planned context is is_percpu_thread().
> >
> > ...
> >
> > --- a/kernel/watchdog_hld.c
> > +++ b/kernel/watchdog_hld.c
> > @@ -165,10 +165,13 @@ static void watchdog_overflow_callback(struct perf_event *event,
> >
> > static int hardlockup_detector_event_create(void)
> > {
> > - unsigned int cpu = smp_processor_id();
> > + unsigned int cpu;
> > struct perf_event_attr *wd_attr;
> > struct perf_event *evt;
> >
> > + /* This function plans to execute in cpu bound kthread */
> > + BUG_ON(!is_percpu_thread());
>
> Can we avoid adding the BUG()? Find a way to emit a WARNing and then
> permit the kernel to continue?
>
Yes, WARN_ON() can work in this case.
Thanks,
Pingfan
On Wed, Sep 15, 2021 at 03:45:06PM +0200, Peter Zijlstra wrote:
> On Wed, Sep 15, 2021 at 11:51:00AM +0800, Pingfan Liu wrote:
> > hardlockup_detector_event_create() indirectly calls
> > kmem_cache_alloc_node(), which is blockable.
> >
> > So here, the really planned context is is_percpu_thread().
> >
> > Signed-off-by: Pingfan Liu <[email protected]>
> > Cc: Petr Mladek <[email protected]>
> > Cc: Andrew Morton <[email protected]>
> > Cc: Wang Qing <[email protected]>
> > Cc: "Peter Zijlstra (Intel)" <[email protected]>
> > Cc: Santosh Sivaraj <[email protected]>
> > Cc: Sumit Garg <[email protected]>
> > Cc: Will Deacon <[email protected]>
> > Cc: Mark Rutland <[email protected]>
> > To: [email protected]
> > ---
> > kernel/watchdog_hld.c | 5 ++++-
> > 1 file changed, 4 insertions(+), 1 deletion(-)
> >
> > diff --git a/kernel/watchdog_hld.c b/kernel/watchdog_hld.c
> > index 247bf0b1582c..6876e796dbf5 100644
> > --- a/kernel/watchdog_hld.c
> > +++ b/kernel/watchdog_hld.c
> > @@ -165,10 +165,13 @@ static void watchdog_overflow_callback(struct perf_event *event,
> >
> > static int hardlockup_detector_event_create(void)
> > {
> > - unsigned int cpu = smp_processor_id();
> > + unsigned int cpu;
> > struct perf_event_attr *wd_attr;
> > struct perf_event *evt;
> >
> > + /* This function plans to execute in cpu bound kthread */
> > + BUG_ON(!is_percpu_thread());
> > + cpu = raw_smp_processor_id();
> > wd_attr = &wd_hw_attr;
> > wd_attr->sample_period = hw_nmi_get_sample_period(watchdog_thresh);
>
> This patch makes no sense.
This patch aims to disable any attempt such as using get_cpu()/put_cpu() to
shut up the check_preemption_disabled().
But if anybody is familiar with the integration of watchdog_hld and
cpuhp, he should know the right way without this BUG_ON() or warn.
Do you still think it is pointless?
Thanks,
Pingfan
On Thu 2021-09-16 11:57:44, Pingfan Liu wrote:
> On Wed, Sep 15, 2021 at 03:45:06PM +0200, Peter Zijlstra wrote:
> > On Wed, Sep 15, 2021 at 11:51:00AM +0800, Pingfan Liu wrote:
> > > hardlockup_detector_event_create() indirectly calls
> > > kmem_cache_alloc_node(), which is blockable.
> > >
> > > So here, the really planned context is is_percpu_thread().
> > >
> > > Signed-off-by: Pingfan Liu <[email protected]>
> > > Cc: Petr Mladek <[email protected]>
> > > Cc: Andrew Morton <[email protected]>
> > > Cc: Wang Qing <[email protected]>
> > > Cc: "Peter Zijlstra (Intel)" <[email protected]>
> > > Cc: Santosh Sivaraj <[email protected]>
> > > Cc: Sumit Garg <[email protected]>
> > > Cc: Will Deacon <[email protected]>
> > > Cc: Mark Rutland <[email protected]>
> > > To: [email protected]
> > > ---
> > > kernel/watchdog_hld.c | 5 ++++-
> > > 1 file changed, 4 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/kernel/watchdog_hld.c b/kernel/watchdog_hld.c
> > > index 247bf0b1582c..6876e796dbf5 100644
> > > --- a/kernel/watchdog_hld.c
> > > +++ b/kernel/watchdog_hld.c
> > > @@ -165,10 +165,13 @@ static void watchdog_overflow_callback(struct perf_event *event,
> > >
> > > static int hardlockup_detector_event_create(void)
> > > {
> > > - unsigned int cpu = smp_processor_id();
> > > + unsigned int cpu;
> > > struct perf_event_attr *wd_attr;
> > > struct perf_event *evt;
> > >
> > > + /* This function plans to execute in cpu bound kthread */
> > > + BUG_ON(!is_percpu_thread());
> > > + cpu = raw_smp_processor_id();
> > > wd_attr = &wd_hw_attr;
> > > wd_attr->sample_period = hw_nmi_get_sample_period(watchdog_thresh);
> >
> > This patch makes no sense.
>
> This patch aims to disable any attempt such as using get_cpu()/put_cpu() to
> shut up the check_preemption_disabled().
I have to say that the description of the problem is really cryptic.
Please, provide more context, code paths, sample code, next time.
Well, I probably got it. The code might sleep. But it should run on the
same CPU even after waking up. You try to achieve this by running
the code in a process that is bound to a single CPU.
IMHO, this is not reliable. Anyone could change the affinity
of the process in the meantime.
I see two solutions. Either avoid the sleep or making sure
that the code access per-CPU variables on the same CPU
all the time.
For example, you might use
*per_cpu_ptr(watchdog_ev, cpu) = evt;
instead of
this_cpu_write(watchdog_ev, evt);
Best Regards,
Petr
On Thu, Sep 16, 2021 at 10:02:27AM +0200, Petr Mladek wrote:
> On Thu 2021-09-16 11:57:44, Pingfan Liu wrote:
> > On Wed, Sep 15, 2021 at 03:45:06PM +0200, Peter Zijlstra wrote:
> > > On Wed, Sep 15, 2021 at 11:51:00AM +0800, Pingfan Liu wrote:
> > > > hardlockup_detector_event_create() indirectly calls
> > > > kmem_cache_alloc_node(), which is blockable.
> > > >
> > > > So here, the really planned context is is_percpu_thread().
> > > >
> > > > Signed-off-by: Pingfan Liu <[email protected]>
> > > > Cc: Petr Mladek <[email protected]>
> > > > Cc: Andrew Morton <[email protected]>
> > > > Cc: Wang Qing <[email protected]>
> > > > Cc: "Peter Zijlstra (Intel)" <[email protected]>
> > > > Cc: Santosh Sivaraj <[email protected]>
> > > > Cc: Sumit Garg <[email protected]>
> > > > Cc: Will Deacon <[email protected]>
> > > > Cc: Mark Rutland <[email protected]>
> > > > To: [email protected]
> > > > ---
> > > > kernel/watchdog_hld.c | 5 ++++-
> > > > 1 file changed, 4 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/kernel/watchdog_hld.c b/kernel/watchdog_hld.c
> > > > index 247bf0b1582c..6876e796dbf5 100644
> > > > --- a/kernel/watchdog_hld.c
> > > > +++ b/kernel/watchdog_hld.c
> > > > @@ -165,10 +165,13 @@ static void watchdog_overflow_callback(struct perf_event *event,
> > > >
> > > > static int hardlockup_detector_event_create(void)
> > > > {
> > > > - unsigned int cpu = smp_processor_id();
> > > > + unsigned int cpu;
> > > > struct perf_event_attr *wd_attr;
> > > > struct perf_event *evt;
> > > >
> > > > + /* This function plans to execute in cpu bound kthread */
> > > > + BUG_ON(!is_percpu_thread());
> > > > + cpu = raw_smp_processor_id();
> > > > wd_attr = &wd_hw_attr;
> > > > wd_attr->sample_period = hw_nmi_get_sample_period(watchdog_thresh);
> > >
> > > This patch makes no sense.
> >
> > This patch aims to disable any attempt such as using get_cpu()/put_cpu() to
> > shut up the check_preemption_disabled().
>
> I have to say that the description of the problem is really cryptic.
> Please, provide more context, code paths, sample code, next time.
>
Sorry, I will be more straight forward. And I had thought commit log had
mentioned it.
> Well, I probably got it. The code might sleep. But it should run on the
And you do get it.
> same CPU even after waking up. You try to achieve this by running
> the code in a process that is bound to a single CPU.
>
> IMHO, this is not reliable. Anyone could change the affinity
> of the process in the meantime.
>
No, it is not. As the code says: PF_NO_SETAFFINITY.
static inline bool is_percpu_thread(void)
{
#ifdef CONFIG_SMP
return (current->flags & PF_NO_SETAFFINITY) &&
(current->nr_cpus_allowed == 1);
#else
return true;
#endif
}
This is critical for cpuhp_* (kernel/cpu.c). And
hardlockup_detector_event_create() should be planned to run on such a
kthread.
Thanks,
Pingfan
> I see two solutions. Either avoid the sleep or making sure
> that the code access per-CPU variables on the same CPU
> all the time.
>
> For example, you might use
>
> *per_cpu_ptr(watchdog_ev, cpu) = evt;
>
> instead of
>
> this_cpu_write(watchdog_ev, evt);
>
> Best Regards,
> Petr