2023-03-17 14:25:59

by Jens Axboe

[permalink] [raw]
Subject: NMI reason 2d when running perf

Hi,

When running perf on my Dell R7525 on a running process, I get a ton of:

[ 504.234782] Dazed and confused, but trying to continue
[ 504.267843] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[ 504.267846] Dazed and confused, but trying to continue
[ 504.335975] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[ 504.335977] Dazed and confused, but trying to continue
[ 504.368031] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[ 504.368033] Dazed and confused, but trying to continue
[ 504.371037] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[ 504.371038] Dazed and confused, but trying to continue
[ 504.439165] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[ 504.439167] Dazed and confused, but trying to continue

spew in dmesg. The box has 2x7763 CPUS. This seems to be a recent
regression, been using this box for a while and haven't seen this
before. The test being traced is pinned to CPU 48. The box is currently
running:

commit 6015b1aca1a233379625385feb01dd014aca60b5 (origin/master, origin/HEAD)
Author: Linus Torvalds <[email protected]>
Date: Tue Mar 14 19:32:38 2023 -0700

sched_getaffinity: don't assume 'cpumask_size()' is fully initialized

with the pending block/io_uring branches merged in for testing.

--
Jens Axboe



2023-03-18 15:59:02

by Dr. David Alan Gilbert

[permalink] [raw]
Subject: Re: NMI reason 2d when running perf

* Jens Axboe ([email protected]) wrote:
> Hi,
>
> When running perf on my Dell R7525 on a running process, I get a ton of:
>
> [ 504.234782] Dazed and confused, but trying to continue
> [ 504.267843] Uhhuh. NMI received for unknown reason 2d on CPU 48.
> [ 504.267846] Dazed and confused, but trying to continue
> [ 504.335975] Uhhuh. NMI received for unknown reason 2d on CPU 48.
> [ 504.335977] Dazed and confused, but trying to continue
> [ 504.368031] Uhhuh. NMI received for unknown reason 2d on CPU 48.
> [ 504.368033] Dazed and confused, but trying to continue
> [ 504.371037] Uhhuh. NMI received for unknown reason 2d on CPU 48.
> [ 504.371038] Dazed and confused, but trying to continue
> [ 504.439165] Uhhuh. NMI received for unknown reason 2d on CPU 48.
> [ 504.439167] Dazed and confused, but trying to continue
>
> spew in dmesg. The box has 2x7763 CPUS. This seems to be a recent
> regression, been using this box for a while and haven't seen this
> before. The test being traced is pinned to CPU 48. The box is currently
> running:

I'm not sure this is new; I've seen this intermittently on AMD boxes
for a few years. Always when running perf or related tools.

Dave

> commit 6015b1aca1a233379625385feb01dd014aca60b5 (origin/master, origin/HEAD)
> Author: Linus Torvalds <[email protected]>
> Date: Tue Mar 14 19:32:38 2023 -0700
>
> sched_getaffinity: don't assume 'cpumask_size()' is fully initialized
>
> with the pending block/io_uring branches merged in for testing.
>
> --
> Jens Axboe
>
--
-----Open up your eyes, open up your mind, open up your code -------
/ Dr. David Alan Gilbert | Running GNU/Linux | Happy \
\ dave @ treblig.org | | In Hex /
\ _________________________|_____ http://www.treblig.org |_______/

2023-03-20 04:45:11

by Ravi Bangoria

[permalink] [raw]
Subject: Re: NMI reason 2d when running perf

On 17-Mar-23 7:55 PM, Jens Axboe wrote:
> Hi,
>
> When running perf on my Dell R7525 on a running process, I get a ton of:
>
> [ 504.234782] Dazed and confused, but trying to continue
> [ 504.267843] Uhhuh. NMI received for unknown reason 2d on CPU 48.
> [ 504.267846] Dazed and confused, but trying to continue
> [ 504.335975] Uhhuh. NMI received for unknown reason 2d on CPU 48.
> [ 504.335977] Dazed and confused, but trying to continue
> [ 504.368031] Uhhuh. NMI received for unknown reason 2d on CPU 48.
> [ 504.368033] Dazed and confused, but trying to continue
> [ 504.371037] Uhhuh. NMI received for unknown reason 2d on CPU 48.
> [ 504.371038] Dazed and confused, but trying to continue
> [ 504.439165] Uhhuh. NMI received for unknown reason 2d on CPU 48.
> [ 504.439167] Dazed and confused, but trying to continue
>
> spew in dmesg. The box has 2x7763 CPUS.

Are you using IBS directly or indirectly (perf record, if run as root
without explicit -e event, internally invokes IBS)? fwiw, Zen2 had
similar hw issue where IBS can raise NMI without sample valid bit set
while exiting from CC6 state[1]. But as per my knowledge, it's already
fixed in Zen3.

[1]: https://lore.kernel.org/all/YFDSSxftYw9tCGC6@krava/

> This seems to be a recent
> regression, been using this box for a while and haven't seen this
> before. The test being traced is pinned to CPU 48. The box is currently
> running:
>
> commit 6015b1aca1a233379625385feb01dd014aca60b5 (origin/master, origin/HEAD)
> Author: Linus Torvalds <[email protected]>
> Date: Tue Mar 14 19:32:38 2023 -0700
>
> sched_getaffinity: don't assume 'cpumask_size()' is fully initialized
>
> with the pending block/io_uring branches merged in for testing.
>