2020-08-26 14:57:28

by kajoljain

[permalink] [raw]
Subject: [RFC v2] perf/core: Fixes hung issue on perf stat command during cpu hotplug

Commit 2ed6edd33a21 ("perf: Add cond_resched() to task_function_call()")
added assignment of ret value as -EAGAIN in case function
call to 'smp_call_function_single' fails.
For non-zero ret value, it did
'ret = !ret ? data.ret : -EAGAIN;', which always
assign -EAGAIN to ret and make second if condition useless.

In scenarios like when executing a perf stat with --per-thread option, and
if any of the monitoring cpu goes offline, the 'smp_call_function_single'
function could return -ENXIO, and with the above check,
task_function_call hung and increases CPU
usage (because of repeated 'smp_call_function_single()')

Recration scenario:
# perf stat -a --per-thread && (offline a CPU )

Patch here removes the tertiary condition added as part of that
commit and added a check for NULL and -EAGAIN.

Fixes: 2ed6edd33a21("perf: Add cond_resched() to task_function_call()")
Signed-off-by: Kajol Jain <[email protected]>
Reported-by: Srikar Dronamraju <[email protected]>
---
kernel/events/core.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

Changelog:
- Remove addition of else in the first patch for
if(ret != -EAGAIN) condition.

diff --git a/kernel/events/core.c b/kernel/events/core.c
index 856d98c36f56..fe104fee097a 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -115,8 +115,8 @@ task_function_call(struct task_struct *p, remote_function_f func, void *info)
for (;;) {
ret = smp_call_function_single(task_cpu(p), remote_function,
&data, 1);
- ret = !ret ? data.ret : -EAGAIN;
-
+ if(!ret)
+ ret = data.ret;
if (ret != -EAGAIN)
break;

--
2.26.2


2020-08-26 15:16:08

by Barret Rhoden

[permalink] [raw]
Subject: Re: [RFC v2] perf/core: Fixes hung issue on perf stat command during cpu hotplug

Hi -

On 8/26/20 10:54 AM, Kajol Jain wrote:
> Commit 2ed6edd33a21 ("perf: Add cond_resched() to task_function_call()")
> added assignment of ret value as -EAGAIN in case function
> call to 'smp_call_function_single' fails.
> For non-zero ret value, it did
> 'ret = !ret ? data.ret : -EAGAIN;', which always
> assign -EAGAIN to ret and make second if condition useless.
>
> In scenarios like when executing a perf stat with --per-thread option, and
> if any of the monitoring cpu goes offline, the 'smp_call_function_single'
> function could return -ENXIO, and with the above check,
> task_function_call hung and increases CPU
> usage (because of repeated 'smp_call_function_single()')
>
> Recration scenario:
> # perf stat -a --per-thread && (offline a CPU )
>
> Patch here removes the tertiary condition added as part of that
> commit and added a check for NULL and -EAGAIN.
>
> Fixes: 2ed6edd33a21("perf: Add cond_resched() to task_function_call()")
> Signed-off-by: Kajol Jain <[email protected]>
> Reported-by: Srikar Dronamraju <[email protected]>
> ---
> kernel/events/core.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> Changelog:
> - Remove addition of else in the first patch for
> if(ret != -EAGAIN) condition.
>
> diff --git a/kernel/events/core.c b/kernel/events/core.c
> index 856d98c36f56..fe104fee097a 100644
> --- a/kernel/events/core.c
> +++ b/kernel/events/core.c
> @@ -115,8 +115,8 @@ task_function_call(struct task_struct *p, remote_function_f func, void *info)
> for (;;) {
> ret = smp_call_function_single(task_cpu(p), remote_function,
> &data, 1);
> - ret = !ret ? data.ret : -EAGAIN;
> -
> + if(!ret)
^
Minor nit, you need a space after the if.

Also, the comment for the function says it returns @func's return val or
-ESRCH. You could also add -ENXIO to that.

Thanks for the fix.

Reviewed-by: Barret Rhoden <[email protected]>

> + ret = data.ret;
> if (ret != -EAGAIN)
> break;
>
>

2020-08-27 05:50:02

by Srikar Dronamraju

[permalink] [raw]
Subject: Re: [RFC v2] perf/core: Fixes hung issue on perf stat command during cpu hotplug

* Kajol Jain <[email protected]> [2020-08-26 20:24:11]:

> Commit 2ed6edd33a21 ("perf: Add cond_resched() to task_function_call()")
> added assignment of ret value as -EAGAIN in case function
> call to 'smp_call_function_single' fails.
> For non-zero ret value, it did
> 'ret = !ret ? data.ret : -EAGAIN;', which always
> assign -EAGAIN to ret and make second if condition useless.
>
> In scenarios like when executing a perf stat with --per-thread option, and
> if any of the monitoring cpu goes offline, the 'smp_call_function_single'
> function could return -ENXIO, and with the above check,
> task_function_call hung and increases CPU
> usage (because of repeated 'smp_call_function_single()')
>
> Recration scenario:
> # perf stat -a --per-thread && (offline a CPU )
>
> Patch here removes the tertiary condition added as part of that
> commit and added a check for NULL and -EAGAIN.
>
> Fixes: 2ed6edd33a21("perf: Add cond_resched() to task_function_call()")
> Signed-off-by: Kajol Jain <[email protected]>
> Reported-by: Srikar Dronamraju <[email protected]>

Tested-by: Srikar Dronamraju <[email protected]>


--
Thanks and Regards
Srikar Dronamraju