Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
MIME-Version: 1.0
References: <388098b2c03fbf0a732834fc01b2d875c335bc49.1642170196.git.lucien.xin@gmail.com>
In-Reply-To: <388098b2c03fbf0a732834fc01b2d875c335bc49.1642170196.git.lucien.xin@gmail.com>
From:   Maurizio Lombardi <mlombard@redhat.com>
Date:   Wed, 1 Jun 2022 16:36:06 +0200
Message-ID: <CAFL455m6BZW-KVNOkJPo3BSeEK39Vq597F-b5XvQJmU3mrPmZA@mail.gmail.com>
Subject: Re: [PATCH] mm: slub: fix a deadlock warning in kmem_cache_destroy
To:     Xin Long <lucien.xin@gmail.com>
Cc:     linux-kernel@vger.kernel.org, linux-mm@kvack.org,
        Christoph Lameter <cl@linux.com>,
        Pekka Enberg <penberg@kernel.org>,
        David Rientjes <rientjes@google.com>,
        Joonsoo Kim <iamjoonsoo.kim@lge.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Vlastimil Babka <vbabka@suse.cz>,
        Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
        Antoine Tenart <atenart@kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

p=C3=A1 14. 1. 2022 v 15:23 odes=C3=ADlatel Xin Long <lucien.xin@gmail.com>=
 napsal:
>
> cpus_read_lock() is introduced into kmem_cache_destroy() by
> commit 5a836bf6b09f ("mm: slub: move flush_cpu_slab() invocations
> __free_slab() invocations out of IRQ context"), and it could cause
> a deadlock.

FYI,

I received a bug report from one of our customers, he complains that
his system (with nvmefc boot from SAN) hangs when rebooting.
He runs a RHEL-9 kernel based on version 5.14.0.

What is interesting is that, according to him, after reverting commit
5a836bf6b09f
("mm: slub: move flush_cpu_slab() invocations __free_slab()
invocations out of IRQ context")
the reboot operation doesn't hang anymore.

The call trace seems to point to a possible problem due to the fact that
nvme_delete_ctrl_work is allocated with the WQ_MEM_RECLAIM bit set.

[  453.012078] ------------[ cut here ]------------
[  453.016744] workqueue: WQ_MEM_RECLAIM
nvme-delete-wq:nvme_delete_ctrl_work [nvme_core] is flushing
!WQ_MEM_RECLAIM events:flush_cpu_slab
[  453.016789] WARNING: CPU: 37 PID: 410 at kernel/workqueue.c:2637
check_flush_dependency+0x10a/0x120
[...]
[  453.262125] Call Trace:
[  453.264582]  __flush_work.isra.0+0xbf/0x220
[  453.268775]  ? __queue_work+0x1dc/0x420
[  453.272623]  flush_all_cpus_locked+0xfb/0x120
[  453.276992]  __kmem_cache_shutdown+0x2b/0x320
[  453.281361]  kmem_cache_destroy+0x49/0x100
[  453.285465]  bioset_exit+0x143/0x190
[  453.289052]  blk_release_queue+0xb9/0x100
[  453.293075]  kobject_cleanup+0x37/0x130
[  453.296922]  nvme_fc_ctrl_free+0xc6/0x150 [nvme_fc]
[  453.302397]  nvme_free_ctrl+0x1ac/0x2b0 [nvme_core]
[  453.307818]  device_release+0x31/0x90
[  453.312005]  kobject_cleanup+0x37/0x130
[  453.316369]  process_one_work+0x1e5/0x3c0
[  453.320895]  worker_thread+0x50/0x3b0
[  453.325074]  ? rescuer_thread+0x370/0x370
[  453.329592]  kthread+0x146/0x170
[  453.333322]  ? set_kthread_struct+0x40/0x40
[  453.338027]  ret_from_fork+0x1f/0x30
[  453.342082] ---[ end trace 8c9cdd85adbbfc4f ]---

Maurizio Lombardi