Events can be deleted and the entry can be NULL.
Check event for NULL in amd_pmu_enable_all() before enable to avoid a GPF.
This appears to be an AMD only issue.
Syzkaller reported a GPF in amd_pmu_enable_all.
INFO: NMI handler (perf_event_nmi_handler) took too long to run: 13.170
msecs
perf: interrupt took too long (191950 > 156435), lowering
kernel.perf_event_max_sample_rate to 1000
Oops: general protection fault, probably for non-canonical address
0xdffffc0000000034: 0000 [#1] PREEMPT SMP KASAN NOPTI
KASAN: null-ptr-deref in range [0x00000000000001a0-0x00000000000001a7]
Call Trace:
<IRQ>
amd_pmu_enable_all (arch/x86/events/amd/core.c:1341)
x86_pmu_enable (arch/x86/events/core.c:1276 arch/x86/events/core.c:1335)
__pmu_ctx_sched_out (kernel/events/core.c:8314 (discriminator 1))
ctx_sched_out (kernel/events/core.c:8328)
__perf_install_in_context (kernel/events/core.c:6235)
remote_function (./arch/x86/include/asm/atomic64_64.h:20)
__flush_smp_call_function_queue (kernel/smp.c:189 (discriminator 20)
kernel/smp.c:197 (discriminator 20) kernel/smp.c:540 (discriminator 20))
__sysvec_call_function_single (arch/x86/kernel/smp.c:193 (discriminator 1))
sysvec_call_function_single (lib/maple_tree.c:3155 (discriminator 2))
asm_sysvec_call_function_single (./arch/x86/include/asm/idtentry.h:709)
Reported-by: syzkaller <[email protected]>
Signed-off-by: George Kennedy <[email protected]>
---
arch/x86/events/amd/core.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/arch/x86/events/amd/core.c b/arch/x86/events/amd/core.c
index 1fc4ce44e743..0ead72d3e206 100644
--- a/arch/x86/events/amd/core.c
+++ b/arch/x86/events/amd/core.c
@@ -760,7 +760,8 @@ static void amd_pmu_enable_all(int added)
if (!test_bit(idx, cpuc->active_mask))
continue;
- amd_pmu_enable_event(cpuc->events[idx]);
+ if (cpuc->events[idx])
+ amd_pmu_enable_event(cpuc->events[idx]);
}
}
--
2.39.3
Hi George,
> Events can be deleted and the entry can be NULL.
Can you please also explain "how".
> Check event for NULL in amd_pmu_enable_all() before enable to avoid a GPF.
> This appears to be an AMD only issue.
>
> Syzkaller reported a GPF in amd_pmu_enable_all.
Can you please provide a bug report link? Also, any reproducer?
> @@ -760,7 +760,8 @@ static void amd_pmu_enable_all(int added)
> if (!test_bit(idx, cpuc->active_mask))
> continue;
>
> - amd_pmu_enable_event(cpuc->events[idx]);
> + if (cpuc->events[idx])
> + amd_pmu_enable_event(cpuc->events[idx]);
What if cpuc->events[idx] becomes NULL after if (cpuc->events[idx]) but
before amd_pmu_enable_event(cpuc->events[idx])?
Thanks,
Ravi
Hi Ravi,
On 5/29/2024 11:37 PM, Ravi Bangoria wrote:
> Hi George,
>
>> Events can be deleted and the entry can be NULL.
> Can you please also explain "how".
It looks like x86_pmu_stop() is clearing the bit in active_mask and
setting the events entry to NULL (and doing it in the correct order) for
the same events index that amd_pmu_enable_all() is trying to enable.
>
>> Check event for NULL in amd_pmu_enable_all() before enable to avoid a GPF.
>> This appears to be an AMD only issue.
>>
>> Syzkaller reported a GPF in amd_pmu_enable_all.
> Can you please provide a bug report link? Also, any reproducer?
The Syzkaller reproducer can be found in this link:
https://lore.kernel.org/netdev/CAMt6jhyec7-TSFpr3F+_ikjpu39WV3jnCBBGwpzpBrPx55w20g@mail.gmail.com/T/#u
>
>> @@ -760,7 +760,8 @@ static void amd_pmu_enable_all(int added)
>> if (!test_bit(idx, cpuc->active_mask))
>> continue;
>>
>> - amd_pmu_enable_event(cpuc->events[idx]);
>> + if (cpuc->events[idx])
>> + amd_pmu_enable_event(cpuc->events[idx]);
> What if cpuc->events[idx] becomes NULL after if (cpuc->events[idx]) but
> before amd_pmu_enable_event(cpuc->events[idx])?
Good question, but the crash has not reproduced with the proposed fix in
hours of testing. It usually reproduces within minutes without the fix.
Thank you,
George
>
> Thanks,
> Ravi
Hi Ravi,
On 5/30/2024 2:30 PM, George Kennedy wrote:
> Hi Ravi,
>
> On 5/29/2024 11:37 PM, Ravi Bangoria wrote:
>> Hi George,
>>
>>> Events can be deleted and the entry can be NULL.
>> Can you please also explain "how".
> It looks like x86_pmu_stop() is clearing the bit in active_mask and
> setting the events entry to NULL (and doing it in the correct order)
> for the same events index that amd_pmu_enable_all() is trying to enable.
>>
>>> Check event for NULL in amd_pmu_enable_all() before enable to avoid
>>> a GPF.
>>> This appears to be an AMD only issue.
>>>
>>> Syzkaller reported a GPF in amd_pmu_enable_all.
>> Can you please provide a bug report link? Also, any reproducer?
> The Syzkaller reproducer can be found in this link:
> https://lore.kernel.org/netdev/CAMt6jhyec7-TSFpr3F+_ikjpu39WV3jnCBBGwpzpBrPx55w20g@mail.gmail.com/T/#u
>
>>
>>> @@ -760,7 +760,8 @@ static void amd_pmu_enable_all(int added)
>>> if (!test_bit(idx, cpuc->active_mask))
>>> continue;
>>> - amd_pmu_enable_event(cpuc->events[idx]);
>>> + if (cpuc->events[idx])
>>> + amd_pmu_enable_event(cpuc->events[idx]);
>> What if cpuc->events[idx] becomes NULL after if (cpuc->events[idx]) but
>> before amd_pmu_enable_event(cpuc->events[idx])?
> Good question, but the crash has not reproduced with the proposed fix
> in hours of testing. It usually reproduces within minutes without the
> fix.
Also, a similar fix is done in __intel_pmu_enable_all() in
arch/x86/events/intel/core.c except that a WARN_ON_ONCE is done as well.
See:
https://elixir.bootlin.com/linux/v6.10-rc1/source/arch/x86/events/intel/core.c#L2256
>
> Thank you,
> George
>>
>> Thanks,
>> Ravi
>
Hi George,
>>> Syzkaller reported a GPF in amd_pmu_enable_all.
>> Can you please provide a bug report link? Also, any reproducer?
> The Syzkaller reproducer can be found in this link:
> https://lore.kernel.org/netdev/CAMt6jhyec7-TSFpr3F+_ikjpu39WV3jnCBBGwpzpBrPx55w20g@mail.gmail.com/T/#u
I'm _not_ able to reproduce the issue with this syzkaller reproducer. I ran
it for more than 12 hours. Here is my setup.
- EYPC Zen3 Milan _host_
- Fedora 40
- v6.9 vanilla kernel with attached config
- Syzkaller command running as a root
# ./syz-execprog -executor=./syz-executor -threaded=1 -repeat=0 -procs=8 2-repro.prog
- Also tried with C program generated using:
# ./syz-prog2c -procs=8 -repeat=0 -threaded=1 -tmpdir=1 -segv=true -prog=2-repro.prog > 2-repro.c
Can you please provide your config and system detail in case you are able
to reproduce it.
Thanks,
Ravi
On 6/3/2024 12:39 AM, Ravi Bangoria wrote:
> Hi George,
>
>>>> Syzkaller reported a GPF in amd_pmu_enable_all.
>>> Can you please provide a bug report link? Also, any reproducer?
>> The Syzkaller reproducer can be found in this link:
>> https://lore.kernel.org/netdev/CAMt6jhyec7-TSFpr3F+_ikjpu39WV3jnCBBGwpzpBrPx55w20g@mail.gmail.com/T/#u
> I'm _not_ able to reproduce the issue with this syzkaller reproducer. I ran
> it for more than 12 hours. Here is my setup.
>
> - EYPC Zen3 Milan _host_
> - Fedora 40
> - v6.9 vanilla kernel with attached config
> - Syzkaller command running as a root
> # ./syz-execprog -executor=./syz-executor -threaded=1 -repeat=0 -procs=8 2-repro.prog
> - Also tried with C program generated using:
> # ./syz-prog2c -procs=8 -repeat=0 -threaded=1 -tmpdir=1 -segv=true -prog=2-repro.prog > 2-repro.c
processor : 0
vendor_id : AuthenticAMD
cpu family : 23
model : 1
model name : AMD EPYC 7551 32-Core Processor
stepping : 2
microcode : 0x800126e
cpu MHz : 2000.000
cache size : 512 KB
physical id : 0
siblings : 64
core id : 0
cpu cores : 32
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 ht sysca
ll nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl
nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf rapl pni
pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave
avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_le
gacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext
perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw
_pstate ssbd ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap
clflushopt sha_ni xsaveopt xsavec xgetbv1 nt_good clz
ero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale
vmcb_clean flushbyasid decodeassists pausefilter pfthreshol
d avic v_vmsave_vmload vgif overflow_recov succor smca
bugs : div0 sysret_ss_attrs null_seg spectre_v1 spectre_v2
spec_store_bypass retbleed smt_rsb srso
bogomips : 3992.29
TLB size : 2560 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
Try these repro steps instead:
syz-prog2c -prog repro.prog -threaded -repeat 0 -procs 8 -slowdown 1
-sandbox none -sandbox_arg 0 -enable net_reset -enable net_dev -enable
cgroups -tmpdir -enable binfmt_misc -enable close_fds -enable sysctl
-segv >repro.c
>
> Can you please provide your config and system detail in case you are able
> to reproduce it.
See attached for repro C and config.
Thanks,
George
>
> Thanks,
> Ravi
>>> Events can be deleted and the entry can be NULL.
>> Can you please also explain "how".
> It looks like x86_pmu_stop() is clearing the bit in active_mask and setting the events entry to NULL (and doing it in the correct order) for the same events index that amd_pmu_enable_all() is trying to enable.
Right, so how can it disable the exact same event which it's about to
enable?
If it's happening because of some race condition, we need to find and fix
it. For ex: https://git.kernel.org/torvalds/c/baa014b9543c8
>>> Check event for NULL in amd_pmu_enable_all() before enable to avoid a GPF.
>>> This appears to be an AMD only issue.
>>>
>>> Syzkaller reported a GPF in amd_pmu_enable_all.
>> Can you please provide a bug report link? Also, any reproducer?
> The Syzkaller reproducer can be found in this link:
> https://lore.kernel.org/netdev/CAMt6jhyec7-TSFpr3F+_ikjpu39WV3jnCBBGwpzpBrPx55w20g@mail.gmail.com/T/#u
>>
>>> @@ -760,7 +760,8 @@ static void amd_pmu_enable_all(int added)
>>> if (!test_bit(idx, cpuc->active_mask))
>>> continue;
>>> - amd_pmu_enable_event(cpuc->events[idx]);
>>> + if (cpuc->events[idx])
>>> + amd_pmu_enable_event(cpuc->events[idx]);
>> What if cpuc->events[idx] becomes NULL after if (cpuc->events[idx]) but
>> before amd_pmu_enable_event(cpuc->events[idx])?
> Good question, but the crash has not reproduced with the proposed fix in hours of testing. It usually reproduces within minutes without the fix.
It does not reproduce with the patch because that if() condition makes the
race window smaller?
Thanks,
Ravi
>>>>> Syzkaller reported a GPF in amd_pmu_enable_all.
>>>> Can you please provide a bug report link? Also, any reproducer?
>>> The Syzkaller reproducer can be found in this link:
>>> https://lore.kernel.org/netdev/CAMt6jhyec7-TSFpr3F+_ikjpu39WV3jnCBBGwpzpBrPx55w20g@mail.gmail.com/T/#u
>> I'm _not_ able to reproduce the issue with this syzkaller reproducer. I ran
>> it for more than 12 hours. Here is my setup.
>>
>> - EYPC Zen3 Milan _host_
>> - Fedora 40
>> - v6.9 vanilla kernel with attached config
>> - Syzkaller command running as a root
>> # ./syz-execprog -executor=./syz-executor -threaded=1 -repeat=0 -procs=8 2-repro.prog
>> - Also tried with C program generated using:
>> # ./syz-prog2c -procs=8 -repeat=0 -threaded=1 -tmpdir=1 -segv=true -prog=2-repro.prog > 2-repro.c
> processor : 0
> vendor_id : AuthenticAMD
> cpu family : 23
> model : 1
> model name : AMD EPYC 7551 32-Core Processor
> stepping : 2
> microcode : 0x800126e
> cpu MHz : 2000.000
> cache size : 512 KB
> physical id : 0
> siblings : 64
> core id : 0
> cpu cores : 32
> apicid : 0
> initial apicid : 0
> fpu : yes
> fpu_exception : yes
> cpuid level : 13
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht sysca
> ll nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf rapl pni
> pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_le
> gacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw
> _pstate ssbd ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 nt_good clz
> ero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshol
> d avic v_vmsave_vmload vgif overflow_recov succor smca
> bugs : div0 sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass retbleed smt_rsb srso
> bogomips : 3992.29
> TLB size : 2560 4K pages
> clflush size : 64
> cache_alignment : 64
> address sizes : 48 bits physical, 48 bits virtual
> power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
>
> Try these repro steps instead:
> syz-prog2c -prog repro.prog -threaded -repeat 0 -procs 8 -slowdown 1 -sandbox none -sandbox_arg 0 -enable net_reset -enable net_dev -enable cgroups -tmpdir -enable binfmt_misc -enable close_fds -enable sysctl -segv >repro.c
>
>>
>> Can you please provide your config and system detail in case you are able
>> to reproduce it.
>
> See attached for repro C and config.
No luck with your reproducers as well :(. My kernel config is almost same as
yours:
$ ./scripts/diffconfig 2-config.repro .config
-CC_HAS_AUTO_VAR_INIT_PATTERN y
-CC_HAS_AUTO_VAR_INIT_ZERO y
-CC_HAS_AUTO_VAR_INIT_ZERO_BARE y
-CC_HAS_KASAN_MEMINTRINSIC_PREFIX y
-INIT_STACK_ALL_PATTERN n
-INIT_STACK_ALL_ZERO y
-TOOLS_SUPPORT_RELR y
AS_VERSION 24000 -> 23502
CC_VERSION_TEXT "gcc (GCC) 13.1.1 20230614 (Red Hat 13.1.1-43..0.1)" -> "gcc (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3)"
GCC_VERSION 130101 -> 110401
INIT_STACK_NONE n -> y
LD_VERSION 24000 -> 23502
LOCALVERSION "" -> "-ravi"
PAHOLE_VERSION 0 -> 125
+PAHOLE_HAS_LANG_EXCLUDE y
+PAHOLE_HAS_SPLIT_BTF y
And I'm trying on Zen1 test machine:
processor : 0
vendor_id : AuthenticAMD
cpu family : 23
model : 1
model name : AMD EPYC 7601 32-Core Processor
stepping : 2
microcode : 0x800126e
cpu MHz : 2692.575
cache size : 512 KB
physical id : 0
siblings : 64
core id : 0
cpu cores : 32
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate ssbd ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca sev sev_es
bugs : sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass retbleed smt_rsb srso div0
bogomips : 4391.65
TLB size : 2560 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 43 bits physical, 48 bits virtual
power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
>>>> Events can be deleted and the entry can be NULL.
>>> Can you please also explain "how".
>> It looks like x86_pmu_stop() is clearing the bit in active_mask and setting the events entry to NULL (and doing it in the correct order) for the same events index that amd_pmu_enable_all() is trying to enable.
>>>
>>>> Check event for NULL in amd_pmu_enable_all() before enable to avoid a GPF.
>>>> This appears to be an AMD only issue.
>>>>
>>>> Syzkaller reported a GPF in amd_pmu_enable_all.
>>> Can you please provide a bug report link? Also, any reproducer?
>> The Syzkaller reproducer can be found in this link:
>> https://lore.kernel.org/netdev/CAMt6jhyec7-TSFpr3F+_ikjpu39WV3jnCBBGwpzpBrPx55w20g@mail.gmail.com/T/#u
>>>
>>>> @@ -760,7 +760,8 @@ static void amd_pmu_enable_all(int added)
>>>> if (!test_bit(idx, cpuc->active_mask))
>>>> continue;
>>>> - amd_pmu_enable_event(cpuc->events[idx]);
>>>> + if (cpuc->events[idx])
>>>> + amd_pmu_enable_event(cpuc->events[idx]);
>>> What if cpuc->events[idx] becomes NULL after if (cpuc->events[idx]) but
>>> before amd_pmu_enable_event(cpuc->events[idx])?
>> Good question, but the crash has not reproduced with the proposed fix in hours of testing. It usually reproduces within minutes without the fix.
>
> Also, a similar fix is done in __intel_pmu_enable_all() in arch/x86/events/intel/core.c except that a WARN_ON_ONCE is done as well.
> See: https://elixir.bootlin.com/linux/v6.10-rc1/source/arch/x86/events/intel/core.c#L2256
There are subtle differences between Intel and AMD pmu implementation.
__intel_pmu_enable_all() enables all event with single WRMSR whereas
amd_pmu_enable_all() loops over each PMC and enables it individually.
The WARN_ON_ONCE() is important because it will warn about potential
sw bug somewhere else.
Thanks,
Ravi
> On 6/4/2024 9:16 AM, Ravi Bangoria wrote:
>>>>>> Events can be deleted and the entry can be NULL.
>>>>> Can you please also explain "how".
>>>> It looks like x86_pmu_stop() is clearing the bit in active_mask and setting the events entry to NULL (and doing it in the correct order) for the same events index that amd_pmu_enable_all() is trying to enable.
>>>>>> Check event for NULL in amd_pmu_enable_all() before enable to avoid a GPF.
>>>>>> This appears to be an AMD only issue.
>>>>>>
>>>>>> Syzkaller reported a GPF in amd_pmu_enable_all.
>>>>> Can you please provide a bug report link? Also, any reproducer?
>>>> The Syzkaller reproducer can be found in this link:
>>>> https://lore.kernel.org/netdev/CAMt6jhyec7-TSFpr3F+_ikjpu39WV3jnCBBGwpzpBrPx55w20g@mail.gmail.com/T/#u
>>>>>> @@ -760,7 +760,8 @@ static void amd_pmu_enable_all(int added)
>>>>>> if (!test_bit(idx, cpuc->active_mask))
>>>>>> continue;
>>>>>> - amd_pmu_enable_event(cpuc->events[idx]);
>>>>>> + if (cpuc->events[idx])
>>>>>> + amd_pmu_enable_event(cpuc->events[idx]);
>>>>> What if cpuc->events[idx] becomes NULL after if (cpuc->events[idx]) but
>>>>> before amd_pmu_enable_event(cpuc->events[idx])?
>>>> Good question, but the crash has not reproduced with the proposed fix in hours of testing. It usually reproduces within minutes without the fix.
>>> Also, a similar fix is done in __intel_pmu_enable_all() in arch/x86/events/intel/core.c except that a WARN_ON_ONCE is done as well.
>>> See: https://elixir.bootlin.com/linux/v6.10-rc1/source/arch/x86/events/intel/core.c#L2256
>> There are subtle differences between Intel and AMD pmu implementation.
>> __intel_pmu_enable_all() enables all event with single WRMSR whereas
>> amd_pmu_enable_all() loops over each PMC and enables it individually.
>>
>> The WARN_ON_ONCE() is important because it will warn about potential
>> sw bug somewhere else.
> We could add a similar WARN_ON_ONCE() to the proposed patch.
Sure, that would help in future. But for current splat, can you please
try to rootcause the underlying race condition?
Thanks,
Ravi
On 6/4/2024 9:16 AM, Ravi Bangoria wrote:
>>>>> Events can be deleted and the entry can be NULL.
>>>> Can you please also explain "how".
>>> It looks like x86_pmu_stop() is clearing the bit in active_mask and setting the events entry to NULL (and doing it in the correct order) for the same events index that amd_pmu_enable_all() is trying to enable.
>>>>> Check event for NULL in amd_pmu_enable_all() before enable to avoid a GPF.
>>>>> This appears to be an AMD only issue.
>>>>>
>>>>> Syzkaller reported a GPF in amd_pmu_enable_all.
>>>> Can you please provide a bug report link? Also, any reproducer?
>>> The Syzkaller reproducer can be found in this link:
>>> https://lore.kernel.org/netdev/CAMt6jhyec7-TSFpr3F+_ikjpu39WV3jnCBBGwpzpBrPx55w20g@mail.gmail.com/T/#u
>>>>> @@ -760,7 +760,8 @@ static void amd_pmu_enable_all(int added)
>>>>> if (!test_bit(idx, cpuc->active_mask))
>>>>> continue;
>>>>> - amd_pmu_enable_event(cpuc->events[idx]);
>>>>> + if (cpuc->events[idx])
>>>>> + amd_pmu_enable_event(cpuc->events[idx]);
>>>> What if cpuc->events[idx] becomes NULL after if (cpuc->events[idx]) but
>>>> before amd_pmu_enable_event(cpuc->events[idx])?
>>> Good question, but the crash has not reproduced with the proposed fix in hours of testing. It usually reproduces within minutes without the fix.
>> Also, a similar fix is done in __intel_pmu_enable_all() in arch/x86/events/intel/core.c except that a WARN_ON_ONCE is done as well.
>> See: https://elixir.bootlin.com/linux/v6.10-rc1/source/arch/x86/events/intel/core.c#L2256
> There are subtle differences between Intel and AMD pmu implementation.
> __intel_pmu_enable_all() enables all event with single WRMSR whereas
> amd_pmu_enable_all() loops over each PMC and enables it individually.
>
> The WARN_ON_ONCE() is important because it will warn about potential
> sw bug somewhere else.
We could add a similar WARN_ON_ONCE() to the proposed patch. Thanks, George
>
> Thanks,
> Ravi
On 6/4/2024 9:40 AM, Ravi Bangoria wrote:
>> On 6/4/2024 9:16 AM, Ravi Bangoria wrote:
>>>>>>> Events can be deleted and the entry can be NULL.
>>>>>> Can you please also explain "how".
>>>>> It looks like x86_pmu_stop() is clearing the bit in active_mask and setting the events entry to NULL (and doing it in the correct order) for the same events index that amd_pmu_enable_all() is trying to enable.
>>>>>>> Check event for NULL in amd_pmu_enable_all() before enable to avoid a GPF.
>>>>>>> This appears to be an AMD only issue.
>>>>>>>
>>>>>>> Syzkaller reported a GPF in amd_pmu_enable_all.
>>>>>> Can you please provide a bug report link? Also, any reproducer?
>>>>> The Syzkaller reproducer can be found in this link:
>>>>> https://lore.kernel.org/netdev/CAMt6jhyec7-TSFpr3F+_ikjpu39WV3jnCBBGwpzpBrPx55w20g@mail.gmail.com/T/#u
>>>>>>> @@ -760,7 +760,8 @@ static void amd_pmu_enable_all(int added)
>>>>>>> if (!test_bit(idx, cpuc->active_mask))
>>>>>>> continue;
>>>>>>> - amd_pmu_enable_event(cpuc->events[idx]);
>>>>>>> + if (cpuc->events[idx])
>>>>>>> + amd_pmu_enable_event(cpuc->events[idx]);
>>>>>> What if cpuc->events[idx] becomes NULL after if (cpuc->events[idx]) but
>>>>>> before amd_pmu_enable_event(cpuc->events[idx])?
>>>>> Good question, but the crash has not reproduced with the proposed fix in hours of testing. It usually reproduces within minutes without the fix.
>>>> Also, a similar fix is done in __intel_pmu_enable_all() in arch/x86/events/intel/core.c except that a WARN_ON_ONCE is done as well.
>>>> See: https://elixir.bootlin.com/linux/v6.10-rc1/source/arch/x86/events/intel/core.c#L2256
>>> There are subtle differences between Intel and AMD pmu implementation.
>>> __intel_pmu_enable_all() enables all event with single WRMSR whereas
>>> amd_pmu_enable_all() loops over each PMC and enables it individually.
>>>
>>> The WARN_ON_ONCE() is important because it will warn about potential
>>> sw bug somewhere else.
>> We could add a similar WARN_ON_ONCE() to the proposed patch.
> Sure, that would help in future. But for current splat, can you please
> try to rootcause the underlying race condition?
Sure, I can keep trying to root cause, but will need help from the AMD
perf experts.
In the meantime, the proposed patch with the WARN_ON_ONCE() added like
the Intel version would avoid a GPF and would potentially hint at root
cause. BTW, reproduced on Ubuntu 22.04.4 LTS on AMD Baremetal. processor
: 0 vendor_id : AuthenticAMD cpu family : 23 model : 1 model name : AMD
EPYC 7551 32-Core Processor stepping : 2 microcode : 0x800126e cpu MHz :
1200.000 cache size : 512 KB physical id : 0 siblings : 64 core id : 0
cpu cores : 32 apicid : 0 initial apicid : 0 fpu : yes fpu_exception :
yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8
apic sep mtrr pge mca cmov pat pse36 clf lush mmx fxsr sse sse2 ht
syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl
nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf rapl pni pclmulqdq
monitor ssse3 fma cx 16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c
rdrand lahf_lm cmp_legacy svm extapic cr8_le gacy abm sse4a misalignsse
3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext
perfctr_llc mwaitx cpb hw_pstate ssbd ibpb vmmcall fsgsbase bmi1 avx2
smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1
clzero irperf xsaveerptr arat npt lbrv svm _lock nrip_save tsc_scale
vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v
_vmsave_vmload vgif overflow_recov succor smca bugs : sysret_ss_attrs
null_seg spectre_v1 spectre_v2 spec_store_bypass retbleed s mt_rsb srso
div0 bogomips : 3992.42 TLB size : 2560 4K pages clflush size : 64
cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14] # nproc
128 [ 165.692858] perf: interrupt took too long (36007 > 35837),
lowering kernel.perf_event_max_sample_rate to 5000 [ 188.226736] Oops:
general protection fault, probably for non-canonical address
0xdffffc0000000034: 0000 [#1] PREEMPT SMP KASAN NOPTI [ 188.228803]
KASAN: null-ptr-deref in range [0x00000000000001a0-0x00000000000001a7] [
188.230029] CPU: 0 PID: 20434 Comm: repro_x86_pmu_e Not tainted
6.10.0-rc1-21-ge0cce98fe279-syzk #1 [ 188.231472] Hardware name: QEMU
Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014 [
188.232791] RIP: 0010:x86_pmu_enable_event+0x63/0x280 [ 188.233642]
Code: 41 5c 41 5d 41 5e 41 5f e9 3a 84 99 00 e8 35 84 99 00 48 8d bb a0
01 00 00 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 <0f> b6 04
02 65 4c 8b 25 a9 1e 01 7f 84 c0 74 08 3c 03 0f 8e ac 01 [ 188.236554]
RSP: 0000:ffff888118209a38 EFLAGS: 00010012 [ 188.237406] RAX:
dffffc0000000000 RBX: 0000000000000000 RCX: 0000000000000000 [
188.238546] RDX: 0000000000000034 RSI: 0000000000000000 RDI:
00000000000001a0 [ 188.239691] RBP: 0000000000000001 R08:
0000000000000000 R09: 0000000000000000 [ 188.240830] R10:
0000000000000000 R11: 0000000000000000 R12: ffff88811822a230 [
188.241959] R13: ffff88811822a420 R14: 0000000000000001 R15:
fffffbfff32748b7 [ 188.243097] FS: 00007fa0554fb700(0000)
GS:ffff888118200000(0000) knlGS:0000000000000000 [ 188.244374] CS: 0010
DS: 0000 ES: 0000 CR0: 0000000080050033 [ 188.245291] CR2:
00000000200001c0 CR3: 000000001c646000 CR4: 00000000000006f0 [
188.246418] Call Trace: [ 188.246848] <IRQ> [ 188.247199] ?
show_regs+0x91/0xa0 [ 188.247765] ? die_addr+0x54/0xd0 [ 188.248334] ?
exc_general_protection+0x15c/0x240 [ 188.249123] ?
asm_exc_general_protection+0x26/0x30 [ 188.249922] ?
x86_pmu_enable_event+0x63/0x280 [ 188.250669] ?
x86_pmu_enable_event+0x4b/0x280 [ 188.251421]
amd_pmu_enable_all+0x109/0x180 [ 188.252108] x86_pmu_enable+0x773/0xca0
[ 188.252739] ? amd_pmu_del_event+0x42/0x70 [ 188.253415] ?
perf_event_update_time+0x294/0x3a0 [ 188.254190]
event_sched_out+0x7a1/0xd50 [ 188.254862]
__perf_remove_from_context+0xfa/0xe70 [ 188.255650]
event_function+0x275/0x450 [ 188.256293] ?
__pfx___perf_remove_from_context+0x10/0x10 [ 188.257174] ?
__pfx_event_function+0x10/0x10 [ 188.257891] remote_function+0x12e/0x1c0
[ 188.258552] __flush_smp_call_function_queue+0x1c6/0xcb0 [ 188.259433]
? __pfx_remote_function+0x10/0x10 [ 188.260175]
__sysvec_call_function_single+0x2a/0x210 [ 188.261000]
sysvec_call_function_single+0x36/0x90 [ 188.261789]
asm_sysvec_call_function_single+0x1a/0x20 qemu-system-x86_64 -m 4096
-smp 4 -net nic,model=virtio -net
user,host=10.0.2.10,hostfwd=tcp::1569-:22 -display none -serial
mon:stdio -no-reboot -enable-kvm -initrd
/var/opt/do_syzkaller_setup-1.0/fuzzer/images/initramfs.img -hda
images/syzk_8.img -initrd images/initramfs.img -kernel
images/bzImage.UPSTREAM.v6.10-rc1-21-ge0cce98fe279 -snapshot -append
'console=ttyS0 earlyprintk=serial oops=panic nmi_watchdog=panic
panic_on_warn=0 loglevel=8 panic=86400 ftrace_dump_on_oops=orig_cpu
rodata=n vsyscall=native biosdevname=0 root=/dev/sda console=ttyS0
root=/dev/mapper/ol-root' In your attempts at crash reproduction, you
have all modules built-in with KASAN config'd, correct? Thanks, George
>
> Thanks,
> Ravi
Hi Ravi,
On 6/4/2024 9:40 AM, Ravi Bangoria wrote:
>> On 6/4/2024 9:16 AM, Ravi Bangoria wrote:
>>>>>>> Events can be deleted and the entry can be NULL.
>>>>>> Can you please also explain "how".
>>>>> It looks like x86_pmu_stop() is clearing the bit in active_mask and setting the events entry to NULL (and doing it in the correct order) for the same events index that amd_pmu_enable_all() is trying to enable.
>>>>>>> Check event for NULL in amd_pmu_enable_all() before enable to avoid a GPF.
>>>>>>> This appears to be an AMD only issue.
>>>>>>>
>>>>>>> Syzkaller reported a GPF in amd_pmu_enable_all.
>>>>>> Can you please provide a bug report link? Also, any reproducer?
>>>>> The Syzkaller reproducer can be found in this link:
>>>>> https://lore.kernel.org/netdev/CAMt6jhyec7-TSFpr3F+_ikjpu39WV3jnCBBGwpzpBrPx55w20g@mail.gmail.com/T/#u
>>>>>>> @@ -760,7 +760,8 @@ static void amd_pmu_enable_all(int added)
>>>>>>> if (!test_bit(idx, cpuc->active_mask))
>>>>>>> continue;
>>>>>>> - amd_pmu_enable_event(cpuc->events[idx]);
>>>>>>> + if (cpuc->events[idx])
>>>>>>> + amd_pmu_enable_event(cpuc->events[idx]);
>>>>>> What if cpuc->events[idx] becomes NULL after if (cpuc->events[idx]) but
>>>>>> before amd_pmu_enable_event(cpuc->events[idx])?
>>>>> Good question, but the crash has not reproduced with the proposed fix in hours of testing. It usually reproduces within minutes without the fix.
>>>> Also, a similar fix is done in __intel_pmu_enable_all() in arch/x86/events/intel/core.c except that a WARN_ON_ONCE is done as well.
>>>> See: https://elixir.bootlin.com/linux/v6.10-rc1/source/arch/x86/events/intel/core.c#L2256
>>> There are subtle differences between Intel and AMD pmu implementation.
>>> __intel_pmu_enable_all() enables all event with single WRMSR whereas
>>> amd_pmu_enable_all() loops over each PMC and enables it individually.
>>>
>>> The WARN_ON_ONCE() is important because it will warn about potential
>>> sw bug somewhere else.
>> We could add a similar WARN_ON_ONCE() to the proposed patch.
> Sure, that would help in future. But for current splat, can you please
> try to rootcause the underlying race condition?
Were you able to reproduce the crash on the AMD machine?
Thanks,
George
> Thanks,
> Ravi
On 6/8/2024 12:43 AM, George Kennedy wrote:
> Hi Ravi,
>
> On 6/4/2024 9:40 AM, Ravi Bangoria wrote:
>>> On 6/4/2024 9:16 AM, Ravi Bangoria wrote:
>>>>>>>> Events can be deleted and the entry can be NULL.
>>>>>>> Can you please also explain "how".
>>>>>> It looks like x86_pmu_stop() is clearing the bit in active_mask and setting the events entry to NULL (and doing it in the correct order) for the same events index that amd_pmu_enable_all() is trying to enable.
>>>>>>>> Check event for NULL in amd_pmu_enable_all() before enable to avoid a GPF.
>>>>>>>> This appears to be an AMD only issue.
>>>>>>>>
>>>>>>>> Syzkaller reported a GPF in amd_pmu_enable_all.
>>>>>>> Can you please provide a bug report link? Also, any reproducer?
>>>>>> The Syzkaller reproducer can be found in this link:
>>>>>> https://lore.kernel.org/netdev/CAMt6jhyec7-TSFpr3F+_ikjpu39WV3jnCBBGwpzpBrPx55w20g@mail.gmail.com/T/#u
>>>>>>>> @@ -760,7 +760,8 @@ static void amd_pmu_enable_all(int added)
>>>>>>>> if (!test_bit(idx, cpuc->active_mask))
>>>>>>>> continue;
>>>>>>>> - amd_pmu_enable_event(cpuc->events[idx]);
>>>>>>>> + if (cpuc->events[idx])
>>>>>>>> + amd_pmu_enable_event(cpuc->events[idx]);
>>>>>>> What if cpuc->events[idx] becomes NULL after if (cpuc->events[idx]) but
>>>>>>> before amd_pmu_enable_event(cpuc->events[idx])?
>>>>>> Good question, but the crash has not reproduced with the proposed fix in hours of testing. It usually reproduces within minutes without the fix.
>>>>> Also, a similar fix is done in __intel_pmu_enable_all() in arch/x86/events/intel/core.c except that a WARN_ON_ONCE is done as well.
>>>>> See: https://elixir.bootlin.com/linux/v6.10-rc1/source/arch/x86/events/intel/core.c#L2256
>>>> There are subtle differences between Intel and AMD pmu implementation.
>>>> __intel_pmu_enable_all() enables all event with single WRMSR whereas
>>>> amd_pmu_enable_all() loops over each PMC and enables it individually.
>>>>
>>>> The WARN_ON_ONCE() is important because it will warn about potential
>>>> sw bug somewhere else.
>>> We could add a similar WARN_ON_ONCE() to the proposed patch.
>> Sure, that would help in future. But for current splat, can you please
>> try to rootcause the underlying race condition?
>
> Were you able to reproduce the crash on the AMD machine?
I'm able to reproduce within the KVM guest. Will try to investigate further.
Thanks,
Ravi