The following crash happens when a never-used device is unbound from
uio_hv_generic driver:
kernel BUG at mm/slub.c:321!
invalid opcode: 0000 [#1] SMP PTI
CPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G X --------- --- 5.14.0-0.rc2.23.el9.x86_64 #1
Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018
RIP: 0010:__slab_free+0x1d5/0x3d0
...
Call Trace:
? pick_next_task_fair+0x18e/0x3b0
? __cond_resched+0x16/0x40
? vunmap_pmd_range.isra.0+0x154/0x1c0
? __vunmap+0x22d/0x290
? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
kfree+0x331/0x380
? hv_uio_remove+0x43/0x60 [uio_hv_generic]
hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
vmbus_free_ring+0x21/0x60 [hv_vmbus]
hv_uio_remove+0x4f/0x60 [uio_hv_generic]
vmbus_remove+0x23/0x30 [hv_vmbus]
__device_release_driver+0x17a/0x230
device_driver_detach+0x3c/0xa0
unbind_store+0x113/0x130
...
The problem appears to be that we free 'ring_info->pkt_buffer' twice:
first, when the device is unbound from in-kernel driver (netvsc in this
case) and second from hv_uio_remove(). Normally, ring buffer is supposed
to be re-initialized from hv_uio_open() but this happens when UIO device
is being opened and this is not guaranteed to happen.
Generally, it is OK to call hv_ringbuffer_cleanup() twice for the same
channel (which is being handed over between in-kernel drivers and UIO) even
if we didn't call hv_ringbuffer_init() in between. We, however, need to
avoid kfree() call for an already freed pointer.
Fixes: adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")
Signed-off-by: Vitaly Kuznetsov <[email protected]>
---
drivers/hv/ring_buffer.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c
index 2aee356840a2..314015d9e912 100644
--- a/drivers/hv/ring_buffer.c
+++ b/drivers/hv/ring_buffer.c
@@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)
mutex_unlock(&ring_info->ring_buffer_mutex);
kfree(ring_info->pkt_buffer);
+ ring_info->pkt_buffer = NULL;
ring_info->pkt_buffer_size = 0;
}
--
2.31.1
On Tue, Aug 31, 2021 at 04:39:16PM +0200, Vitaly Kuznetsov wrote:
> The following crash happens when a never-used device is unbound from
> uio_hv_generic driver:
>
> kernel BUG at mm/slub.c:321!
> invalid opcode: 0000 [#1] SMP PTI
> CPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G X --------- --- 5.14.0-0.rc2.23.el9.x86_64 #1
> Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018
> RIP: 0010:__slab_free+0x1d5/0x3d0
> ...
> Call Trace:
> ? pick_next_task_fair+0x18e/0x3b0
> ? __cond_resched+0x16/0x40
> ? vunmap_pmd_range.isra.0+0x154/0x1c0
> ? __vunmap+0x22d/0x290
> ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
> kfree+0x331/0x380
> ? hv_uio_remove+0x43/0x60 [uio_hv_generic]
> hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
> vmbus_free_ring+0x21/0x60 [hv_vmbus]
> hv_uio_remove+0x4f/0x60 [uio_hv_generic]
> vmbus_remove+0x23/0x30 [hv_vmbus]
> __device_release_driver+0x17a/0x230
> device_driver_detach+0x3c/0xa0
> unbind_store+0x113/0x130
> ...
>
> The problem appears to be that we free 'ring_info->pkt_buffer' twice:
> first, when the device is unbound from in-kernel driver (netvsc in this
> case) and second from hv_uio_remove(). Normally, ring buffer is supposed
> to be re-initialized from hv_uio_open() but this happens when UIO device
> is being opened and this is not guaranteed to happen.
>
> Generally, it is OK to call hv_ringbuffer_cleanup() twice for the same
> channel (which is being handed over between in-kernel drivers and UIO) even
> if we didn't call hv_ringbuffer_init() in between. We, however, need to
> avoid kfree() call for an already freed pointer.
>
> Fixes: adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")
> Signed-off-by: Vitaly Kuznetsov <[email protected]>
> ---
> drivers/hv/ring_buffer.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c
> index 2aee356840a2..314015d9e912 100644
> --- a/drivers/hv/ring_buffer.c
> +++ b/drivers/hv/ring_buffer.c
> @@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)
> mutex_unlock(&ring_info->ring_buffer_mutex);
>
> kfree(ring_info->pkt_buffer);
> + ring_info->pkt_buffer = NULL;
This certainly won't hurt.
I would however like to wait till Andrea and Michael go over the
reasoning of this patch before doing anything.
Wei.
> ring_info->pkt_buffer_size = 0;
> }
>
> --
> 2.31.1
>
Wei Liu <[email protected]> writes:
> On Tue, Aug 31, 2021 at 04:39:16PM +0200, Vitaly Kuznetsov wrote:
>> The following crash happens when a never-used device is unbound from
>> uio_hv_generic driver:
>>
>> kernel BUG at mm/slub.c:321!
>> invalid opcode: 0000 [#1] SMP PTI
>> CPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G X --------- --- 5.14.0-0.rc2.23.el9.x86_64 #1
>> Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018
>> RIP: 0010:__slab_free+0x1d5/0x3d0
>> ...
>> Call Trace:
>> ? pick_next_task_fair+0x18e/0x3b0
>> ? __cond_resched+0x16/0x40
>> ? vunmap_pmd_range.isra.0+0x154/0x1c0
>> ? __vunmap+0x22d/0x290
>> ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
>> kfree+0x331/0x380
>> ? hv_uio_remove+0x43/0x60 [uio_hv_generic]
>> hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
>> vmbus_free_ring+0x21/0x60 [hv_vmbus]
>> hv_uio_remove+0x4f/0x60 [uio_hv_generic]
>> vmbus_remove+0x23/0x30 [hv_vmbus]
>> __device_release_driver+0x17a/0x230
>> device_driver_detach+0x3c/0xa0
>> unbind_store+0x113/0x130
>> ...
>>
>> The problem appears to be that we free 'ring_info->pkt_buffer' twice:
>> first, when the device is unbound from in-kernel driver (netvsc in this
>> case) and second from hv_uio_remove(). Normally, ring buffer is supposed
>> to be re-initialized from hv_uio_open() but this happens when UIO device
>> is being opened and this is not guaranteed to happen.
>>
>> Generally, it is OK to call hv_ringbuffer_cleanup() twice for the same
>> channel (which is being handed over between in-kernel drivers and UIO) even
>> if we didn't call hv_ringbuffer_init() in between. We, however, need to
>> avoid kfree() call for an already freed pointer.
>>
>> Fixes: adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")
>> Signed-off-by: Vitaly Kuznetsov <[email protected]>
>> ---
>> drivers/hv/ring_buffer.c | 1 +
>> 1 file changed, 1 insertion(+)
>>
>> diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c
>> index 2aee356840a2..314015d9e912 100644
>> --- a/drivers/hv/ring_buffer.c
>> +++ b/drivers/hv/ring_buffer.c
>> @@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)
>> mutex_unlock(&ring_info->ring_buffer_mutex);
>>
>> kfree(ring_info->pkt_buffer);
>> + ring_info->pkt_buffer = NULL;
>
> This certainly won't hurt.
>
> I would however like to wait till Andrea and Michael go over the
> reasoning of this patch before doing anything.
>
Thanks,
the counter-intuitive thing here is that the channel sometimes continues
to live after hv_ringbuffer_cleanup(): when we unbind a device from
in-kernel driver and give it to uio_hv_generic the channel is handed
over from one driver to another.
> Wei.
>
>> ring_info->pkt_buffer_size = 0;
>> }
>>
>> --
>> 2.31.1
>>
>
--
Vitaly
On Tue, Aug 31, 2021 at 04:39:16PM +0200, Vitaly Kuznetsov wrote:
> The following crash happens when a never-used device is unbound from
> uio_hv_generic driver:
>
> kernel BUG at mm/slub.c:321!
> invalid opcode: 0000 [#1] SMP PTI
> CPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G X --------- --- 5.14.0-0.rc2.23.el9.x86_64 #1
> Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018
> RIP: 0010:__slab_free+0x1d5/0x3d0
> ...
> Call Trace:
> ? pick_next_task_fair+0x18e/0x3b0
> ? __cond_resched+0x16/0x40
> ? vunmap_pmd_range.isra.0+0x154/0x1c0
> ? __vunmap+0x22d/0x290
> ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
> kfree+0x331/0x380
> ? hv_uio_remove+0x43/0x60 [uio_hv_generic]
> hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
> vmbus_free_ring+0x21/0x60 [hv_vmbus]
> hv_uio_remove+0x4f/0x60 [uio_hv_generic]
> vmbus_remove+0x23/0x30 [hv_vmbus]
> __device_release_driver+0x17a/0x230
> device_driver_detach+0x3c/0xa0
> unbind_store+0x113/0x130
> ...
>
> The problem appears to be that we free 'ring_info->pkt_buffer' twice:
> first, when the device is unbound from in-kernel driver (netvsc in this
> case) and second from hv_uio_remove(). Normally, ring buffer is supposed
> to be re-initialized from hv_uio_open() but this happens when UIO device
> is being opened and this is not guaranteed to happen.
>
> Generally, it is OK to call hv_ringbuffer_cleanup() twice for the same
> channel (which is being handed over between in-kernel drivers and UIO) even
> if we didn't call hv_ringbuffer_init() in between. We, however, need to
> avoid kfree() call for an already freed pointer.
>
> Fixes: adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")
> Signed-off-by: Vitaly Kuznetsov <[email protected]>
LGTM.
Reviewed-by: Andrea Parri <[email protected]>
ae6935ed7d424f appears to have anticipated this problem on ->ring_buffer
and adopted the solution proposed here by Vitaly.
Andrea
> ---
> drivers/hv/ring_buffer.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c
> index 2aee356840a2..314015d9e912 100644
> --- a/drivers/hv/ring_buffer.c
> +++ b/drivers/hv/ring_buffer.c
> @@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)
> mutex_unlock(&ring_info->ring_buffer_mutex);
>
> kfree(ring_info->pkt_buffer);
> + ring_info->pkt_buffer = NULL;
> ring_info->pkt_buffer_size = 0;
> }
>
> --
> 2.31.1
>
From: Vitaly Kuznetsov <[email protected]> Sent: Tuesday, August 31, 2021 7:39 AM
>
> The following crash happens when a never-used device is unbound from
> uio_hv_generic driver:
>
> kernel BUG at mm/slub.c:321!
> invalid opcode: 0000 [#1] SMP PTI
> CPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G X --------- --- 5.14.0-0.rc2.23.el9.x86_64 #1
> Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018
> RIP: 0010:__slab_free+0x1d5/0x3d0
> ...
> Call Trace:
> ? pick_next_task_fair+0x18e/0x3b0
> ? __cond_resched+0x16/0x40
> ? vunmap_pmd_range.isra.0+0x154/0x1c0
> ? __vunmap+0x22d/0x290
> ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
> kfree+0x331/0x380
> ? hv_uio_remove+0x43/0x60 [uio_hv_generic]
> hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
> vmbus_free_ring+0x21/0x60 [hv_vmbus]
> hv_uio_remove+0x4f/0x60 [uio_hv_generic]
> vmbus_remove+0x23/0x30 [hv_vmbus]
> __device_release_driver+0x17a/0x230
> device_driver_detach+0x3c/0xa0
> unbind_store+0x113/0x130
> ...
>
> The problem appears to be that we free 'ring_info->pkt_buffer' twice:
> first, when the device is unbound from in-kernel driver (netvsc in this
> case) and second from hv_uio_remove(). Normally, ring buffer is supposed
> to be re-initialized from hv_uio_open() but this happens when UIO device
> is being opened and this is not guaranteed to happen.
>
> Generally, it is OK to call hv_ringbuffer_cleanup() twice for the same
> channel (which is being handed over between in-kernel drivers and UIO) even
> if we didn't call hv_ringbuffer_init() in between. We, however, need to
> avoid kfree() call for an already freed pointer.
>
> Fixes: adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")
> Signed-off-by: Vitaly Kuznetsov <[email protected]>
> ---
> drivers/hv/ring_buffer.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c
> index 2aee356840a2..314015d9e912 100644
> --- a/drivers/hv/ring_buffer.c
> +++ b/drivers/hv/ring_buffer.c
> @@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)
> mutex_unlock(&ring_info->ring_buffer_mutex);
>
> kfree(ring_info->pkt_buffer);
> + ring_info->pkt_buffer = NULL;
> ring_info->pkt_buffer_size = 0;
> }
>
> --
> 2.31.1
Reviewed-by: Michael Kelley <[email protected]>
On Tue, Aug 31, 2021 at 04:39:16PM +0200, Vitaly Kuznetsov wrote:
> The following crash happens when a never-used device is unbound from
> uio_hv_generic driver:
>
[...]
> Fixes: adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")
> Signed-off-by: Vitaly Kuznetsov <[email protected]>
Applied to hyperv-fixes. Thanks.