Dear developers and maintainers,
We encountered a divide error bug while using our modified syzkaller.
It was tested against the latest upstream kernel (6.9-rc5). The kernel
was compiled by clang 14.0.0, and kernel config and C repro are
attached to this email. Kernel crash log is listed below.
```
divide error: 0000 [#1] PREEMPT SMP KASAN PTI
CPU: 1 PID: 21 Comm: cpuhp/1 Not tainted 6.9.0-rc5 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
RIP: 0010:wq_update_node_max_active+0x369/0x6b0 kernel/workqueue.c:1605
Code: 24 bf 00 00 00 80 44 89 fe e8 83 27 33 00 41 83 fc ff 75 0d 41
81 ff 00 00 00 80 0f 84 68 01 00 00 e8 fb 22 33 00 44 89 f8 99 <41> f7
fc 89 c5 89 c7 44 89 ee e8 a8 24 33 00 89 ef 8b 5c 24 04 89
RSP: 0018:ffffc9000018fbb0 EFLAGS: 00010293
RAX: 00000000000000ff RBX: 0000000000000001 RCX: ffff888100ada500
RDX: 0000000000000000 RSI: 00000000000000ff RDI: 0000000080000000
RBP: 0000000000000001 R08: ffffffff815b1fcd R09: 1ffff1100364ad72
R10: dffffc0000000000 R11: ffffed100364ad73 R12: 0000000000000000
R13: 0000000000000100 R14: 0000000000000000 R15: 00000000000000ff
FS: 0000000000000000(0000) GS:ffff888135c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb8c06ca6f8 CR3: 000000010d6c6000 CR4: 0000000000750ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
<TASK>
workqueue_offline_cpu+0x56f/0x600 kernel/workqueue.c:6525
cpuhp_invoke_callback+0x4e1/0x870 kernel/cpu.c:194
cpuhp_thread_fun+0x411/0x7d0 kernel/cpu.c:1092
smpboot_thread_fn+0x544/0xa10 kernel/smpboot.c:164
kthread+0x2ed/0x390 kernel/kthread.c:388
ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
ret_from_fork_asm+0x11/0x20 arch/x86/entry/entry_64.S:244
</TASK>
Modules linked in:
---[ end trace 0000000000000000 ]---
RIP: 0010:wq_update_node_max_active+0x369/0x6b0 kernel/workqueue.c:1605
Code: 24 bf 00 00 00 80 44 89 fe e8 83 27 33 00 41 83 fc ff 75 0d 41
81 ff 00 00 00 80 0f 84 68 01 00 00 e8 fb 22 33 00 44 89 f8 99 <41> f7
fc 89 c5 89 c7 44 89 ee e8 a8 24 33 00 89 ef 8b 5c 24 04 89
RSP: 0018:ffffc9000018fbb0 EFLAGS: 00010293
RAX: 00000000000000ff RBX: 0000000000000001 RCX: ffff888100ada500
RDX: 0000000000000000 RSI: 00000000000000ff RDI: 0000000080000000
RBP: 0000000000000001 R08: ffffffff815b1fcd R09: 1ffff1100364ad72
R10: dffffc0000000000 R11: ffffed100364ad73 R12: 0000000000000000
R13: 0000000000000100 R14: 0000000000000000 R15: 00000000000000ff
FS: 0000000000000000(0000) GS:ffff888135c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb8c06ca6f8 CR3: 000000010d6c6000 CR4: 0000000000750ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
----------------
Code disassembly (best guess):
0: 24 bf and $0xbf,%al
2: 00 00 add %al,(%rax)
4: 00 80 44 89 fe e8 add %al,-0x170176bc(%rax)
a: 83 27 33 andl $0x33,(%rdi)
d: 00 41 83 add %al,-0x7d(%rcx)
10: fc cld
11: ff 75 0d push 0xd(%rbp)
14: 41 81 ff 00 00 00 80 cmp $0x80000000,%r15d
1b: 0f 84 68 01 00 00 je 0x189
21: e8 fb 22 33 00 call 0x332321
26: 44 89 f8 mov %r15d,%eax
29: 99 cltd
* 2a: 41 f7 fc idiv %r12d <-- trapping instruction
2d: 89 c5 mov %eax,%ebp
2f: 89 c7 mov %eax,%edi
31: 44 89 ee mov %r13d,%esi
34: e8 a8 24 33 00 call 0x3324e1
39: 89 ef mov %ebp,%edi
3b: 8b 5c 24 04 mov 0x4(%rsp),%ebx
3f: 89 .byte 0x89
```
If you have any questions, please contact us.
Reported by Yue Sun <[email protected]>
Reported by xingwei lee <[email protected]>
Best Regards,
Yue
From: Lai Jiangshan <[email protected]>
Yue Sun and xingwei lee reported a divide error bug in
wq_update_node_max_active():
divide error: 0000 [#1] PREEMPT SMP KASAN PTI
CPU: 1 PID: 21 Comm: cpuhp/1 Not tainted 6.9.0-rc5 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
RIP: 0010:wq_update_node_max_active+0x369/0x6b0 kernel/workqueue.c:1605
Code: 24 bf 00 00 00 80 44 89 fe e8 83 27 33 00 41 83 fc ff 75 0d 41
81 ff 00 00 00 80 0f 84 68 01 00 00 e8 fb 22 33 00 44 89 f8 99 <41> f7
fc 89 c5 89 c7 44 89 ee e8 a8 24 33 00 89 ef 8b 5c 24 04 89
RSP: 0018:ffffc9000018fbb0 EFLAGS: 00010293
RAX: 00000000000000ff RBX: 0000000000000001 RCX: ffff888100ada500
RDX: 0000000000000000 RSI: 00000000000000ff RDI: 0000000080000000
RBP: 0000000000000001 R08: ffffffff815b1fcd R09: 1ffff1100364ad72
R10: dffffc0000000000 R11: ffffed100364ad73 R12: 0000000000000000
R13: 0000000000000100 R14: 0000000000000000 R15: 00000000000000ff
FS: 0000000000000000(0000) GS:ffff888135c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb8c06ca6f8 CR3: 000000010d6c6000 CR4: 0000000000750ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
<TASK>
workqueue_offline_cpu+0x56f/0x600 kernel/workqueue.c:6525
cpuhp_invoke_callback+0x4e1/0x870 kernel/cpu.c:194
cpuhp_thread_fun+0x411/0x7d0 kernel/cpu.c:1092
smpboot_thread_fn+0x544/0xa10 kernel/smpboot.c:164
kthread+0x2ed/0x390 kernel/kthread.c:388
ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
ret_from_fork_asm+0x11/0x20 arch/x86/entry/entry_64.S:244
</TASK>
Modules linked in:
---[ end trace 0000000000000000 ]---
After analysis, it happens when all of the CPUs in a workqueue's affinity
get offine.
The problem can be easily reproduced by:
# echo 8 > /sys/devices/virtual/workqueue/<any-wq-name>/cpumask
# echo 0 > /sys/devices/system/cpu/cpu3/online
Force the calculation of total_cpus which is used to be the divisor to
be at least 1 to fix the problem.
Link: https://lore.kernel.org/lkml/CAEkJfYPGS1_4JqvpSo0=FM0S1ytB8CEbyreLTtWpR900dUZymw@mail.gmail.com/
Cc: [email protected]
Fixes: 5797b1c18919 ("workqueue: Implement system-wide nr_active enforcement for unbound workqueues")
Reported-by: Yue Sun <[email protected]>
Reported-by: xingwei lee <[email protected]>
Signed-off-by: Lai Jiangshan <[email protected]>
---
kernel/workqueue.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 0066c8f6c154..b31cd7faeb9f 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -1591,7 +1591,7 @@ static void wq_update_node_max_active(struct workqueue_struct *wq, int off_cpu)
off_cpu = -1;
total_cpus = cpumask_weight_and(effective, cpu_online_mask);
- if (off_cpu >= 0)
+ if (off_cpu >= 0 && total_cpus > 1)
total_cpus--;
for_each_node(node) {
--
2.19.1.6.gb485710b
Hello, Lai.
On Tue, Apr 23, 2024 at 08:45:48PM +0800, Lai Jiangshan wrote:
> diff --git a/kernel/workqueue.c b/kernel/workqueue.c
> index 0066c8f6c154..b31cd7faeb9f 100644
> --- a/kernel/workqueue.c
> +++ b/kernel/workqueue.c
> @@ -1591,7 +1591,7 @@ static void wq_update_node_max_active(struct workqueue_struct *wq, int off_cpu)
> off_cpu = -1;
>
> total_cpus = cpumask_weight_and(effective, cpu_online_mask);
> - if (off_cpu >= 0)
> + if (off_cpu >= 0 && total_cpus > 1)
> total_cpus--;
Can we do this explicitly instead? ie. test total_cpus before using it as
divisor and use max_active explicitly if it's zero.
Thanks.
--
tejun
From: Lai Jiangshan <[email protected]>
Yue Sun and xingwei lee reported a divide error bug in
wq_update_node_max_active():
divide error: 0000 [#1] PREEMPT SMP KASAN PTI
CPU: 1 PID: 21 Comm: cpuhp/1 Not tainted 6.9.0-rc5 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
RIP: 0010:wq_update_node_max_active+0x369/0x6b0 kernel/workqueue.c:1605
Code: 24 bf 00 00 00 80 44 89 fe e8 83 27 33 00 41 83 fc ff 75 0d 41
81 ff 00 00 00 80 0f 84 68 01 00 00 e8 fb 22 33 00 44 89 f8 99 <41> f7
fc 89 c5 89 c7 44 89 ee e8 a8 24 33 00 89 ef 8b 5c 24 04 89
RSP: 0018:ffffc9000018fbb0 EFLAGS: 00010293
RAX: 00000000000000ff RBX: 0000000000000001 RCX: ffff888100ada500
RDX: 0000000000000000 RSI: 00000000000000ff RDI: 0000000080000000
RBP: 0000000000000001 R08: ffffffff815b1fcd R09: 1ffff1100364ad72
R10: dffffc0000000000 R11: ffffed100364ad73 R12: 0000000000000000
R13: 0000000000000100 R14: 0000000000000000 R15: 00000000000000ff
FS: 0000000000000000(0000) GS:ffff888135c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb8c06ca6f8 CR3: 000000010d6c6000 CR4: 0000000000750ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
<TASK>
workqueue_offline_cpu+0x56f/0x600 kernel/workqueue.c:6525
cpuhp_invoke_callback+0x4e1/0x870 kernel/cpu.c:194
cpuhp_thread_fun+0x411/0x7d0 kernel/cpu.c:1092
smpboot_thread_fn+0x544/0xa10 kernel/smpboot.c:164
kthread+0x2ed/0x390 kernel/kthread.c:388
ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
ret_from_fork_asm+0x11/0x20 arch/x86/entry/entry_64.S:244
</TASK>
Modules linked in:
---[ end trace 0000000000000000 ]---
After analysis, it happens when all of the CPUs in a workqueue's affinity
get offine.
The problem can be easily reproduced by:
# echo 8 > /sys/devices/virtual/workqueue/<any-wq-name>/cpumask
# echo 0 > /sys/devices/system/cpu/cpu3/online
Use the default max_actives for nodes when all of the CPUs in the
workqueue's affinity get offline to fix the problem.
Reported-by: Yue Sun <[email protected]>
Reported-by: xingwei lee <[email protected]>
Link: https://lore.kernel.org/lkml/CAEkJfYPGS1_4JqvpSo0=FM0S1ytB8CEbyreLTtWpR900dUZymw@mail.gmail.com/
Fixes: 5797b1c18919 ("workqueue: Implement system-wide nr_active enforcement for unbound workqueues")
Cc: [email protected]
Signed-off-by: Lai Jiangshan <[email protected]>
---
Changed from v1:
when total_cpus==0, use the default values for max_active
instead of forcing total_cpus>=1
[v1]: https://lore.kernel.org/lkml/[email protected]/
kernel/workqueue.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 0066c8f6c154..0105e3c82df4 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -1594,6 +1594,15 @@ static void wq_update_node_max_active(struct workqueue_struct *wq, int off_cpu)
if (off_cpu >= 0)
total_cpus--;
+ /* If all CPUs of the wq get offline, use the default values */
+ if (unlikely(!total_cpus)) {
+ for_each_node(node)
+ wq_node_nr_active(wq, node)->max = min_active;
+
+ wq_node_nr_active(wq, NUMA_NO_NODE)->max = max_active;
+ return;
+ }
+
for_each_node(node) {
int node_cpus;
--
2.19.1.6.gb485710b
Applied to wq/for-6.9-fixes.
Thanks.
--
tejun