From: Zqiang <[email protected]>
If the rescuer worker belong to freezable wq, when this wq is
frozen, the rescuer worker also need to be frozen.
Signed-off-by: Zqiang <[email protected]>
---
v1->v2:
use kthread_freezable_should_stop() replaces try_to_freeze().
kernel/workqueue.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 104e3ef04e33..dd7b78bdb7f1 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -2476,6 +2476,9 @@ static int rescuer_thread(void *__rescuer)
* doesn't participate in concurrency management.
*/
set_pf_worker(true);
+
+ if (wq->flags & WQ_FREEZABLE)
+ set_freezable();
repeat:
set_current_state(TASK_IDLE);
@@ -2487,7 +2490,7 @@ static int rescuer_thread(void *__rescuer)
* @wq->maydays processing before acting on should_stop so that the
* list is always empty on exit.
*/
- should_stop = kthread_should_stop();
+ should_stop = kthread_freezable_should_stop(NULL);
/* see whether any pwq is asking for help */
raw_spin_lock_irq(&wq_mayday_lock);
--
2.25.1
On Tue, Jun 22, 2021 at 8:42 PM <[email protected]> wrote:
>
> From: Zqiang <[email protected]>
>
> If the rescuer worker belong to freezable wq, when this wq is
> frozen, the rescuer worker also need to be frozen.
Hello,
From my understanding, the wq subsystem doesn't use set_freezable(),
try_to_freeze() families. For example, the worker_thread() doesn't
use set_freezable(), neither do in your patch. If kthreads
in the wq subsystem need these functions, please let us know.
The way the wq subsystem to freeze tasks is by freeze_workqueues_begin()
and thaw_workqueues(). When WQs is requested for freezing,
they will stop activating work items. All the workers including
*rescuers* will be scheduled as TASK_IDLE when all the activated work
items are processed and the whole wq subsystem is frozen unless
there is any semantic change about TASK_IDLE V.S. "frozen".
(rescuers can also only process activated work items.)
Thanks
Lai
>
> Signed-off-by: Zqiang <[email protected]>
> ---
> v1->v2:
> use kthread_freezable_should_stop() replaces try_to_freeze().
>
> kernel/workqueue.c | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/workqueue.c b/kernel/workqueue.c
> index 104e3ef04e33..dd7b78bdb7f1 100644
> --- a/kernel/workqueue.c
> +++ b/kernel/workqueue.c
> @@ -2476,6 +2476,9 @@ static int rescuer_thread(void *__rescuer)
> * doesn't participate in concurrency management.
> */
> set_pf_worker(true);
> +
> + if (wq->flags & WQ_FREEZABLE)
> + set_freezable();
> repeat:
> set_current_state(TASK_IDLE);
>
> @@ -2487,7 +2490,7 @@ static int rescuer_thread(void *__rescuer)
> * @wq->maydays processing before acting on should_stop so that the
> * list is always empty on exit.
> */
> - should_stop = kthread_should_stop();
> + should_stop = kthread_freezable_should_stop(NULL);
>
> /* see whether any pwq is asking for help */
> raw_spin_lock_irq(&wq_mayday_lock);
> --
> 2.25.1
>
________________________________________
From: Lai Jiangshan <[email protected]>
Sent: Wednesday, 23 June 2021 11:16
To: Zhang, Qiang
Cc: Tejun Heo; Andrew Morton; LKML
Subject: Re: [PATCH v2] workqueue: set the rescuer worker that belong to freezable wq is freezable
[Please note: This e-mail is from an EXTERNAL e-mail address]
On Tue, Jun 22, 2021 at 8:42 PM <[email protected]> wrote:
>
> From: Zqiang <[email protected]>
>
> If the rescuer worker belong to freezable wq, when this wq is
> frozen, the rescuer worker also need to be frozen.
>Hello,
>
>From my understanding, the wq subsystem doesn't use >set_freezable(),
>try_to_freeze() families. For example, the worker_thread() >doesn't
>use set_freezable(), neither do in your patch. If kthreads
>in the wq subsystem need these functions, please let us know.
>
>The way the wq subsystem to freeze tasks is by >freeze_workqueues_begin()
>and thaw_workqueues(). When WQs is requested for freezing,
>they will stop activating work items. All the workers including
>*rescuers* will be scheduled as TASK_IDLE when all the >activated work
>items are processed and the whole wq subsystem is frozen >unless
>there is any semantic change about TASK_IDLE V.S. "frozen".
>(rescuers can also only process activated work items.)
Hello Lai
My understanding is this:
when the freeze_workqueues_begin() be called, this only set pwq->max_active is zero, it's just to prevent, after that when we queue the work again, we will put it in the pwq->delayed_works,
but if there are still some work in the pool->worklist, the workers in pool are still handle work items, and the rescuer work may be also handle work items, until the work items is processed, the workers enters TASK_IDLE status. the rescuer worker is independent of workerpools , when the system freezing, freeze the rescuer worker directly without waiting for all workitems to be processed.
Thanks
Qiang
>
>Thanks
>Lai
>
> Signed-off-by: Zqiang <[email protected]>
> ---
> v1->v2:
> use kthread_freezable_should_stop() replaces try_to_freeze().
>
> kernel/workqueue.c | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/workqueue.c b/kernel/workqueue.c
> index 104e3ef04e33..dd7b78bdb7f1 100644
> --- a/kernel/workqueue.c
> +++ b/kernel/workqueue.c
> @@ -2476,6 +2476,9 @@ static int rescuer_thread(void *__rescuer)
> * doesn't participate in concurrency management.
> */
> set_pf_worker(true);
> +
> + if (wq->flags & WQ_FREEZABLE)
> + set_freezable();
> repeat:
> set_current_state(TASK_IDLE);
>
> @@ -2487,7 +2490,7 @@ static int rescuer_thread(void *__rescuer)
> * @wq->maydays processing before acting on should_stop so that the
> * list is always empty on exit.
> */
> - should_stop = kthread_should_stop();
> + should_stop = kthread_freezable_should_stop(NULL);
>
> /* see whether any pwq is asking for help */
> raw_spin_lock_irq(&wq_mayday_lock);
> --
> 2.25.1
>
Greeting,
FYI, we noticed the following commit (built with gcc-9):
commit: 72faca7779de7476b780e1406db51f8cfb86f5be ("[PATCH v2] workqueue: set the rescuer worker that belong to freezable wq is freezable")
url: https://github.com/0day-ci/linux/commits/qiang-zhang-windriver-com/workqueue-set-the-rescuer-worker-that-belong-to-freezable-wq-is-freezable/20210622-204319
base: https://git.kernel.org/cgit/linux/kernel/git/tj/wq.git for-next
in testcase: xfstests
version: xfstests-x86_64-820b5e3-1_20210609
with following parameters:
disk: 4HDD
fs: xfs
test: xfs-group-25
ucode: 0x21
test-description: xfstests is a regression test suite for xfs and other files ystems.
test-url: git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git
on test machine: 4 threads 1 sockets Intel(R) Core(TM) i3-3220 CPU @ 3.30GHz with 8G memory
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
+-----------------------------------------------+------------+------------+
| | f2ccd0a922 | 72faca7779 |
+-----------------------------------------------+------------+------------+
| boot_successes | 34 | 0 |
| boot_failures | 0 | 21 |
| WARNING:at_kernel/sched/core.c:#__might_sleep | 0 | 21 |
| RIP:__might_sleep | 0 | 21 |
+-----------------------------------------------+------------+------------+
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <[email protected]>
kern :warn : [ 1.527410] WARNING: CPU: 0 PID: 3 at kernel/sched/core.c:8285 __might_sleep (kbuild/src/consumer/kernel/sched/core.c:8285 (discriminator 9))
kern :warn : [ 1.528407] Modules linked in:
kern :warn : [ 1.529408] CPU: 0 PID: 3 Comm: rcu_gp Not tainted 5.13.0-rc4-00092-g72faca7779de #1
kern :warn : [ 1.530407] Hardware name: Hewlett-Packard HP Pro 3340 MT/17A1, BIOS 8.07 01/24/2013
kern :warn : [ 1.531408] RIP: 0010:__might_sleep (kbuild/src/consumer/kernel/sched/core.c:8285 (discriminator 9))
kern :warn : [ 1.532407] Code: 5d 41 5e 5d e9 2e ff ff ff 48 8b 90 e0 22 00 00 48 8b 70 18 48 c7 c7 10 9f 55 82 c6 05 16 13 c8 01 01 48 89 d1 e8 96 6d b6 00 <0f> 0b eb c6 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 9c 58 f6 c4
All code
========
0: 5d pop %rbp
1: 41 5e pop %r14
3: 5d pop %rbp
4: e9 2e ff ff ff jmpq 0xffffffffffffff37
9: 48 8b 90 e0 22 00 00 mov 0x22e0(%rax),%rdx
10: 48 8b 70 18 mov 0x18(%rax),%rsi
14: 48 c7 c7 10 9f 55 82 mov $0xffffffff82559f10,%rdi
1b: c6 05 16 13 c8 01 01 movb $0x1,0x1c81316(%rip) # 0x1c81338
22: 48 89 d1 mov %rdx,%rcx
25: e8 96 6d b6 00 callq 0xb66dc0
2a:* 0f 0b ud2 <-- trapping instruction
2c: eb c6 jmp 0xfffffffffffffff4
2e: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
35: 00 00
37: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
3c: 9c pushfq
3d: 58 pop %rax
3e: f6 .byte 0xf6
3f: c4 .byte 0xc4
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: eb c6 jmp 0xffffffffffffffca
4: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
b: 00 00
d: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
12: 9c pushfq
13: 58 pop %rax
14: f6 .byte 0xf6
15: c4 .byte 0xc4
kern :warn : [ 1.533407] RSP: 0000:ffffc90000037e60 EFLAGS: 00010286
kern :warn : [ 1.534407] RAX: 0000000000000000 RBX: 0000000000000000 RCX: c0000000ffff7fff
kern :warn : [ 1.535407] RDX: ffffc90000037c88 RSI: 00000000ffff7fff RDI: 0000000000000000
kern :warn : [ 1.536407] RBP: ffffc90000037e78 R08: 0000000000000000 R09: ffffc90000037c80
kern :warn : [ 1.537407] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff82558de7
kern :warn : [ 1.538407] R13: 00000000000000aa R14: 0000000000000000 R15: ffff88810c9dc780
kern :warn : [ 1.539407] FS: 0000000000000000(0000) GS:ffff88821fa00000(0000) knlGS:0000000000000000
kern :warn : [ 1.540407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kern :warn : [ 1.541407] CR2: ffff88821f201000 CR3: 000000021e00a001 CR4: 00000000001706f0
kern :warn : [ 1.542407] Call Trace:
kern :warn : [ 1.543410] ? worker_thread (kbuild/src/consumer/kernel/workqueue.c:2466)
kern :warn : [ 1.544407] kthread_freezable_should_stop (kbuild/src/consumer/kernel/kthread.c:170)
kern :warn : [ 1.545407] rescuer_thread (kbuild/src/consumer/kernel/workqueue.c:2496)
kern :warn : [ 1.546408] ? worker_thread (kbuild/src/consumer/kernel/workqueue.c:2466)
kern :warn : [ 1.547407] ? worker_thread (kbuild/src/consumer/kernel/workqueue.c:2466)
kern :warn : [ 1.548407] kthread (kbuild/src/consumer/kernel/kthread.c:313)
kern :warn : [ 1.549407] ? kthread_park (kbuild/src/consumer/kernel/kthread.c:266)
kern :warn : [ 1.550407] ret_from_fork (kbuild/src/consumer/arch/x86/entry/entry_64.S:300)
kern :warn : [ 1.551410] ---[ end trace 0c2a94da1228593c ]---
kern :info : [ 1.552572] Performance Events: PEBS fmt1+, IvyBridge events, 16-deep LBR, full-width counters, Intel PMU driver.
kern :info : [ 1.553408] ... version: 3
kern :info : [ 1.554407] ... bit width: 48
kern :info : [ 1.555407] ... generic registers: 4
kern :info : [ 1.556407] ... value mask: 0000ffffffffffff
kern :info : [ 1.557407] ... max period: 00007fffffffffff
kern :info : [ 1.558407] ... fixed-purpose events: 3
kern :info : [ 1.559407] ... event mask: 000000070000000f
kern :info : [ 1.560623] rcu: Hierarchical SRCU implementation.
kern :info : [ 1.562267] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
kern :info : [ 1.562497] smp: Bringing up secondary CPUs ...
kern :info : [ 1.563540] x86: Booting SMP configuration:
kern :info : [ 1.564410] .... node #0, CPUs: #1
kern :warn : [ 0.240160] masked ExtINT on CPU#1
kern :warn : [ 1.572568] #2
kern :warn : [ 0.240160] masked ExtINT on CPU#2
kern :warn : [ 1.578531] MDS CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for more details.
kern :warn : [ 1.579578] #3
kern :warn : [ 0.240160] masked ExtINT on CPU#3
kern :info : [ 1.585472] smp: Brought up 1 node, 4 CPUs
kern :info : [ 1.586408] smpboot: Max logical packages: 1
kern :info : [ 1.587408] smpboot: Total of 4 processors activated (26339.58 BogoMIPS)
kern :info : [ 1.599621] node 0 deferred pages initialised in 10ms
kern :info : [ 1.604732] devtmpfs: initialized
kern :info : [ 1.608486] x86/mm: Memory block size: 128MB
kern :info : [ 1.613692] PM: Registering ACPI NVS region [mem 0xd8eda000-0xd8feafff] (1118208 bytes)
kern :info : [ 1.622442] PM: Registering ACPI NVS region [mem 0xd94ab000-0xd94edfff] (274432 bytes)
kern :info : [ 1.630474] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
kern :info : [ 1.640412] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
kern :info : [ 1.648766] pinctrl core: initialized pinctrl subsystem
kern :info : [ 1.653643] NET: Registered protocol family 16
kern :info : [ 1.658828] DMA: preallocated 1024 KiB GFP_KERNEL pool for atomic allocations
kern :info : [ 1.666421] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
kern :info : [ 1.674415] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
kern :info : [ 1.682418] audit: initializing netlink subsys (disabled)
kern :notice: [ 1.687427] audit: type=2000 audit(1295646369.199:1): state=initialized audit_enabled=0 res=1
kern :info : [ 1.687571] thermal_sys: Registered thermal governor 'fair_share'
kern :info : [ 1.696411] thermal_sys: Registered thermal governor 'bang_bang'
kern :info : [ 1.702408] thermal_sys: Registered thermal governor 'step_wise'
kern :info : [ 1.708407] thermal_sys: Registered thermal governor 'user_space'
kern :info : [ 1.714421] cpuidle: using governor menu
kern :info : [ 1.724715] ACPI: bus type PCI registered
kern :info : [ 1.729409] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
kern :info : [ 1.735498] PCI: MMCONFIG for domain 0000 [bus 00-3f] at [mem 0xf8000000-0xfbffffff] (base 0xf8000000)
kern :info : [ 1.744410] PCI: MMCONFIG at [mem 0xf8000000-0xfbffffff] reserved in E820
kern :info : [ 1.751426] PCI: Using configuration type 1 for base access
kern :info : [ 1.757730] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on
kern :info : [ 1.766709] Kprobes globally optimized
kern :info : [ 1.770440] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
kern :info : [ 1.777447] cryptd: max_cpu_qlen set to 1000
kern :info : [ 1.783478] ACPI: Added _OSI(Module Device)
kern :info : [ 1.787410] ACPI: Added _OSI(Processor Device)
kern :info : [ 1.792408] ACPI: Added _OSI(3.0 _SCP Extensions)
kern :info : [ 1.796408] ACPI: Added _OSI(Processor Aggregator Device)
kern :info : [ 1.802409] ACPI: Added _OSI(Linux-Dell-Video)
kern :info : [ 1.806408] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
kern :info : [ 1.812408] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
kern :info : [ 1.825569] ACPI: 5 ACPI AML tables successfully acquired and loaded
kern :info : [ 1.834127] ACPI: Dynamic OEM Table Load:
kern :info : [ 1.837415] ACPI: SSDT 0xFFFF88810D028000 00083B (v01 PmRef Cpu0Cst 00003001 INTL 20051117)
kern :info : [ 1.847280] ACPI: Dynamic OEM Table Load:
kern :info : [ 1.851412] ACPI: SSDT 0xFFFF88821FBF8400 000303 (v01 PmRef ApIst 00003000 INTL 20051117)
kern :info : [ 1.860081] ACPI: Dynamic OEM Table Load:
kern :info : [ 1.864411] ACPI: SSDT 0xFFFF88821F30FC00 000119 (v01 PmRef ApCst 00003000 INTL 20051117)
kern :info : [ 1.875492] ACPI: Interpreter enabled
kern :info : [ 1.878440] ACPI: (supports S0 S3 S4 S5)
kern :info : [ 1.882408] ACPI: Using IOAPIC for interrupt routing
kern :info : [ 1.888447] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
kern :info : [ 1.897983] ACPI: Enabled 14 GPEs in block 00 to 3F
kern :info : [ 1.912951] ACPI: PM: Power Resource [FN00] (off)
kern :info : [ 1.918527] ACPI: PM: Power Resource [FN01] (off)
kern :info : [ 1.922518] ACPI: PM: Power Resource [FN02] (off)
kern :info : [ 1.927518] ACPI: PM: Power Resource [FN03] (off)
kern :info : [ 1.932524] ACPI: PM: Power Resource [FN04] (off)
kern :info : [ 1.938109] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-3e])
kern :info : [ 1.944415] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
kern :info : [ 1.953728] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug SHPCHotplug PME]
kern :info : [ 1.962600] acpi PNP0A08:00: _OSC: OS now controls [AER PCIeCapability LTR]
kern :info : [ 1.970046] PCI host bridge to bus 0000:00
kern :info : [ 1.974412] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]
kern :info : [ 1.980408] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]
kern :info : [ 1.987408] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
kern :info : [ 1.995408] pci_bus 0000:00: root bus resource [mem 0x000d0000-0x000d3fff window]
kern :info : [ 2.002409] pci_bus 0000:00: root bus resource [mem 0x000d4000-0x000d7fff window]
kern :info : [ 2.010412] pci_bus 0000:00: root bus resource [mem 0x000d8000-0x000dbfff window]
kern :info : [ 2.017426] pci_bus 0000:00: root bus resource [mem 0x000dc000-0x000dffff window]
kern :info : [ 2.025411] pci_bus 0000:00: root bus resource [mem 0x000e0000-0x000e3fff window]
kern :info : [ 2.032408] pci_bus 0000:00: root bus resource [mem 0x000e4000-0x000e7fff window]
kern :info : [ 2.040410] pci_bus 0000:00: root bus resource [mem 0xdf200000-0xfeafffff window]
kern :info : [ 2.048408] pci_bus 0000:00: root bus resource [bus 00-3e]
kern :info : [ 2.053425] pci 0000:00:00.0: [8086:0150] type 00 class 0x060000
kern :info : [ 2.059552] pci 0000:00:01.0: [8086:0151] type 01 class 0x060400
kern :info : [ 2.065461] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
kern :info : [ 2.072568] pci 0000:00:02.0: [8086:0152] type 00 class 0x030000
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
bin/lkp run generated-yaml-file
---
0DAY/LKP+ Test Infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/[email protected] Intel Corporation
Thanks,
Oliver Sang