From: Sagi Grimberg <[email protected]>
The mpath disk node takes a reference on the request mpath
request queue when adding live path to the mpath gendisk.
However if we connected to an inaccessible path device_add_disk
is not called, so if we disconnect and remove the mpath gendisk
we endup putting an reference on the request queue that was
never taken [1].
Fix that to check if we ever added a live path (using
NVME_NS_HEAD_HAS_DISK flag) and if not, clear the disk->queue
reference.
[1]:
------------[ cut here ]------------
refcount_t: underflow; use-after-free.
WARNING: CPU: 1 PID: 1372 at lib/refcount.c:28 refcount_warn_saturate+0xa6/0xf0
CPU: 1 PID: 1372 Comm: nvme Tainted: G O 5.7.0-rc2+ #3
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.13.0-1ubuntu1 04/01/2014
RIP: 0010:refcount_warn_saturate+0xa6/0xf0
RSP: 0018:ffffb29e8053bdc0 EFLAGS: 00010282
RAX: 0000000000000000 RBX: ffff8b7a2f4fc060 RCX: 0000000000000007
RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff8b7a3ec99980
RBP: ffff8b7a2f4fc000 R08: 00000000000002e1 R09: 0000000000000004
R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000
R13: fffffffffffffff2 R14: ffffb29e8053bf08 R15: ffff8b7a320e2da0
FS: 00007f135d4ca800(0000) GS:ffff8b7a3ec80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005651178c0c30 CR3: 000000003b650005 CR4: 0000000000360ee0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
disk_release+0xa2/0xc0
device_release+0x28/0x80
kobject_put+0xa5/0x1b0
nvme_put_ns_head+0x26/0x70 [nvme_core]
nvme_put_ns+0x30/0x60 [nvme_core]
nvme_remove_namespaces+0x9b/0xe0 [nvme_core]
nvme_do_delete_ctrl+0x43/0x5c [nvme_core]
nvme_sysfs_delete.cold+0x8/0xd [nvme_core]
kernfs_fop_write+0xc1/0x1a0
vfs_write+0xb6/0x1a0
ksys_write+0x5f/0xe0
do_syscall_64+0x52/0x1a0
entry_SYSCALL_64_after_hwframe+0x44/0xa9
Signed-off-by: Sagi Grimberg <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
Signed-off-by: mengfanhui <[email protected]>
---
drivers/nvme/host/multipath.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
index 2dd4137a08b2..3e579e49579c 100644
--- a/drivers/nvme/host/multipath.c
+++ b/drivers/nvme/host/multipath.c
@@ -895,6 +895,14 @@ void nvme_mpath_remove_disk(struct nvme_ns_head *head)
/* make sure all pending bios are cleaned up */
kblockd_schedule_work(&head->requeue_work);
flush_work(&head->requeue_work);
+ if (!test_bit(NVME_NSHEAD_DISK_LIVE, &head->flags)) {
+ /*
+ * if device_add_disk wasn't called, prevent
+ * disk release to put a bogus reference on the
+ * request queue
+ */
+ head->disk->queue = NULL;
+ }
put_disk(head->disk);
}
--
2.25.1
On 1/17/24 08:50, mengfanhui wrote:
> From: Sagi Grimberg <[email protected]>
>
> The mpath disk node takes a reference on the request mpath
> request queue when adding live path to the mpath gendisk.
> However if we connected to an inaccessible path device_add_disk
> is not called, so if we disconnect and remove the mpath gendisk
> we endup putting an reference on the request queue that was
> never taken [1].
>
> Fix that to check if we ever added a live path (using
> NVME_NS_HEAD_HAS_DISK flag) and if not, clear the disk->queue
> reference.
>
> [1]:
> ------------[ cut here ]------------
> refcount_t: underflow; use-after-free.
> WARNING: CPU: 1 PID: 1372 at lib/refcount.c:28 refcount_warn_saturate+0xa6/0xf0
> CPU: 1 PID: 1372 Comm: nvme Tainted: G O 5.7.0-rc2+ #3
> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.13.0-1ubuntu1 04/01/2014
> RIP: 0010:refcount_warn_saturate+0xa6/0xf0
> RSP: 0018:ffffb29e8053bdc0 EFLAGS: 00010282
> RAX: 0000000000000000 RBX: ffff8b7a2f4fc060 RCX: 0000000000000007
> RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff8b7a3ec99980
> RBP: ffff8b7a2f4fc000 R08: 00000000000002e1 R09: 0000000000000004
> R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000
> R13: fffffffffffffff2 R14: ffffb29e8053bf08 R15: ffff8b7a320e2da0
> FS: 00007f135d4ca800(0000) GS:ffff8b7a3ec80000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00005651178c0c30 CR3: 000000003b650005 CR4: 0000000000360ee0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
> disk_release+0xa2/0xc0
> device_release+0x28/0x80
> kobject_put+0xa5/0x1b0
> nvme_put_ns_head+0x26/0x70 [nvme_core]
> nvme_put_ns+0x30/0x60 [nvme_core]
> nvme_remove_namespaces+0x9b/0xe0 [nvme_core]
> nvme_do_delete_ctrl+0x43/0x5c [nvme_core]
> nvme_sysfs_delete.cold+0x8/0xd [nvme_core]
> kernfs_fop_write+0xc1/0x1a0
> vfs_write+0xb6/0x1a0
> ksys_write+0x5f/0xe0
> do_syscall_64+0x52/0x1a0
> entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
> Signed-off-by: Sagi Grimberg <[email protected]>
> Signed-off-by: Christoph Hellwig <[email protected]>
> Signed-off-by: mengfanhui <[email protected]>
> ---
> drivers/nvme/host/multipath.c | 8 ++++++++
> 1 file changed, 8 insertions(+)
>
> diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
> index 2dd4137a08b2..3e579e49579c 100644
> --- a/drivers/nvme/host/multipath.c
> +++ b/drivers/nvme/host/multipath.c
> @@ -895,6 +895,14 @@ void nvme_mpath_remove_disk(struct nvme_ns_head *head)
> /* make sure all pending bios are cleaned up */
> kblockd_schedule_work(&head->requeue_work);
> flush_work(&head->requeue_work);
> + if (!test_bit(NVME_NSHEAD_DISK_LIVE, &head->flags)) {
> + /*
> + * if device_add_disk wasn't called, prevent
> + * disk release to put a bogus reference on the
> + * request queue
> + */
> + head->disk->queue = NULL;
> + }
> put_disk(head->disk);
> }
>
How did you see this? disk->queue is allocated in blk_alloc_disk called
in nvme_mpath_alloc_disk... I don't understand how you saw the same
dereference that was addressed by this commit.
On Wed, Jan 17, 2024 at 04:04:12PM +0200, Sagi Grimberg wrote:
> How did you see this? disk->queue is allocated in blk_alloc_disk called
> in nvme_mpath_alloc_disk... I don't understand how you saw the same
> dereference that was addressed by this commit.
This looks like a backport of an old patch of yours to a geriatric
kernel to me..
Purpose that to check if we ever added a live path (using
NVME_NS_HEAD_HAS_DISK flag) and if not, clear the disk->queue
reference.The purpose is to perform security checks and remove the disk.
在 2024/1/17 22:39, Christoph Hellwig 写道:
> On Wed, Jan 17, 2024 at 04:04:12PM +0200, Sagi Grimberg wrote:
>> How did you see this? disk->queue is allocated in blk_alloc_disk called
>> in nvme_mpath_alloc_disk... I don't understand how you saw the same
>> dereference that was addressed by this commit.
>
> This looks like a backport of an old patch of yours to a geriatric
> kernel to me..
On 14/03/2024 8:13, mengfanhui wrote:
> Purpose that to check if we ever added a live path (using
> NVME_NS_HEAD_HAS_DISK flag) and if not, clear the disk->queue
> reference.The purpose is to perform security checks and remove the disk.
Does this issue happen in upstream? If it isn't I don't see a reason to fix
a non-existing bug here.
In this scenario. upstream should also appear.
在 2024/3/17 15:33, Sagi Grimberg 写道:
>
>
> On 14/03/2024 8:13, mengfanhui wrote:
>> Purpose that to check if we ever added a live path (using
>> NVME_NS_HEAD_HAS_DISK flag) and if not, clear the disk->queue
>> reference.The purpose is to perform security checks and remove the disk.
>
> Does this issue happen in upstream? If it isn't I don't see a reason to fix
> a non-existing bug here.
On 20/03/2024 4:03, mengfanhui wrote:
> In this scenario. upstream should also appear.
I'd be surprised if it is,
The proposed patch was superseded by:
f165fb89b71f ("nvme-multipath: convert to blk_alloc_disk/blk_cleanup_disk")
Please verify that this exists upstream.