2021-11-25 17:46:56

by Michael S. Tsirkin

[permalink] [raw]
Subject: [PATCH] Bluetooth: virtio_bt: fix device removal

Device removal is clearly out of virtio spec: it attempts to remove
unused buffers from a VQ before invoking device reset. To fix, make
open/close NOPs and do all cleanup/setup in probe/remove.

The cost here is a single skb wasted on an unused bt device - which
seems modest.

NB: with this fix in place driver still suffers from a race condition if
an interrupt triggers while device is being reset. Work on a fix for
that issue is in progress.

Signed-off-by: Michael S. Tsirkin <[email protected]>
---

Note: completely untested, in particular the device isn't supported in QEMU.
Please do not queue directly - please help review and test and ack,
and I will queue this together with reset fixes.
Thanks!


drivers/bluetooth/virtio_bt.c | 18 ++++++++++++++++--
1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/drivers/bluetooth/virtio_bt.c b/drivers/bluetooth/virtio_bt.c
index 24a9258962fa..aea33ba9522c 100644
--- a/drivers/bluetooth/virtio_bt.c
+++ b/drivers/bluetooth/virtio_bt.c
@@ -50,8 +50,11 @@ static int virtbt_add_inbuf(struct virtio_bluetooth *vbt)

static int virtbt_open(struct hci_dev *hdev)
{
- struct virtio_bluetooth *vbt = hci_get_drvdata(hdev);
+ return 0;
+}

+static int virtbt_open_vdev(struct virtio_bluetooth *vbt)
+{
if (virtbt_add_inbuf(vbt) < 0)
return -EIO;

@@ -61,7 +64,11 @@ static int virtbt_open(struct hci_dev *hdev)

static int virtbt_close(struct hci_dev *hdev)
{
- struct virtio_bluetooth *vbt = hci_get_drvdata(hdev);
+ return 0;
+}
+
+static int virtbt_close_vdev(struct virtio_bluetooth *vbt)
+{
int i;

cancel_work_sync(&vbt->rx);
@@ -351,8 +358,14 @@ static int virtbt_probe(struct virtio_device *vdev)
goto failed;
}

+ virtio_device_ready(vdev);
+ if (virtbt_open_vdev(vbt))
+ goto open_failed;
+
return 0;

+open_failed:
+ hci_free_dev(hdev);
failed:
vdev->config->del_vqs(vdev);
return err;
@@ -365,6 +378,7 @@ static void virtbt_remove(struct virtio_device *vdev)

hci_unregister_dev(hdev);
vdev->config->reset(vdev);
+ virtbt_close_vdev(vbt);

hci_free_dev(hdev);
vbt->hdev = NULL;
--
MST



2021-11-25 20:04:05

by Marcel Holtmann

[permalink] [raw]
Subject: Re: [PATCH] Bluetooth: virtio_bt: fix device removal

Hi Michael,

> Device removal is clearly out of virtio spec: it attempts to remove
> unused buffers from a VQ before invoking device reset. To fix, make
> open/close NOPs and do all cleanup/setup in probe/remove.

so the virtbt_{open,close} as NOP is not really what a driver is suppose
to be doing. These are transport enable/disable callbacks from the BT
Core towards the driver. It maps to a device being enabled/disabled by
something like bluetoothd for example. So if disabled, I expect that no
resources/queues are in use.

Maybe I misunderstand the virtio spec in that regard, but I would like
to keep this fundamental concept of a Bluetooth driver. It does work
with all other transports like USB, SDIO, UART etc.

> The cost here is a single skb wasted on an unused bt device - which
> seems modest.

There should be no buffer used if the device is powered off. We also don’t
have any USB URBs in-flight if the transport is not active.

> NB: with this fix in place driver still suffers from a race condition if
> an interrupt triggers while device is being reset. Work on a fix for
> that issue is in progress.

In the virtbt_close() callback we should deactivate all interrupts.

Regards

Marcel


2021-11-25 20:47:03

by Michael S. Tsirkin

[permalink] [raw]
Subject: Re: [PATCH] Bluetooth: virtio_bt: fix device removal

On Thu, Nov 25, 2021 at 09:02:01PM +0100, Marcel Holtmann wrote:
> Hi Michael,
>
> > Device removal is clearly out of virtio spec: it attempts to remove
> > unused buffers from a VQ before invoking device reset. To fix, make
> > open/close NOPs and do all cleanup/setup in probe/remove.
>
> so the virtbt_{open,close} as NOP is not really what a driver is suppose
> to be doing. These are transport enable/disable callbacks from the BT
> Core towards the driver. It maps to a device being enabled/disabled by
> something like bluetoothd for example. So if disabled, I expect that no
> resources/queues are in use.
>
> Maybe I misunderstand the virtio spec in that regard, but I would like
> to keep this fundamental concept of a Bluetooth driver. It does work
> with all other transports like USB, SDIO, UART etc.
>
> > The cost here is a single skb wasted on an unused bt device - which
> > seems modest.
>
> There should be no buffer used if the device is powered off. We also don’t
> have any USB URBs in-flight if the transport is not active.
>
> > NB: with this fix in place driver still suffers from a race condition if
> > an interrupt triggers while device is being reset. Work on a fix for
> > that issue is in progress.
>
> In the virtbt_close() callback we should deactivate all interrupts.
>
> Regards
>
> Marcel

If you want to do that then device has to be reset on close,
and fully reinitialized on open.
Can you work on a patch like that?
Given I don't have the device such a rework is probably more
than I can undertake.

--
MST


2021-11-25 21:03:33

by Marcel Holtmann

[permalink] [raw]
Subject: Re: [PATCH] Bluetooth: virtio_bt: fix device removal

Hi Michael,

>>> Device removal is clearly out of virtio spec: it attempts to remove
>>> unused buffers from a VQ before invoking device reset. To fix, make
>>> open/close NOPs and do all cleanup/setup in probe/remove.
>>
>> so the virtbt_{open,close} as NOP is not really what a driver is suppose
>> to be doing. These are transport enable/disable callbacks from the BT
>> Core towards the driver. It maps to a device being enabled/disabled by
>> something like bluetoothd for example. So if disabled, I expect that no
>> resources/queues are in use.
>>
>> Maybe I misunderstand the virtio spec in that regard, but I would like
>> to keep this fundamental concept of a Bluetooth driver. It does work
>> with all other transports like USB, SDIO, UART etc.
>>
>>> The cost here is a single skb wasted on an unused bt device - which
>>> seems modest.
>>
>> There should be no buffer used if the device is powered off. We also don’t
>> have any USB URBs in-flight if the transport is not active.
>>
>>> NB: with this fix in place driver still suffers from a race condition if
>>> an interrupt triggers while device is being reset. Work on a fix for
>>> that issue is in progress.
>>
>> In the virtbt_close() callback we should deactivate all interrupts.
>>
>
> If you want to do that then device has to be reset on close,
> and fully reinitialized on open.
> Can you work on a patch like that?
> Given I don't have the device such a rework is probably more
> than I can undertake.

so you mean move virtio_find_vqs() into virtbt_open() and del_vqs() into
virtbt_close()?

Or is there are way to set up the queues without starting them?

However I am failing to understand your initial concern, we do reset()
before del_vqs() in virtbt_remove(). Should we be doing something different
in virtbt_close() other than virtqueue_detach_unused_buf(). Why would I
keep buffers attached if they are not used.

Regards

Marcel


2021-11-25 21:25:21

by Michael S. Tsirkin

[permalink] [raw]
Subject: Re: [PATCH] Bluetooth: virtio_bt: fix device removal

On Thu, Nov 25, 2021 at 10:01:25PM +0100, Marcel Holtmann wrote:
> Hi Michael,
>
> >>> Device removal is clearly out of virtio spec: it attempts to remove
> >>> unused buffers from a VQ before invoking device reset. To fix, make
> >>> open/close NOPs and do all cleanup/setup in probe/remove.
> >>
> >> so the virtbt_{open,close} as NOP is not really what a driver is suppose
> >> to be doing. These are transport enable/disable callbacks from the BT
> >> Core towards the driver. It maps to a device being enabled/disabled by
> >> something like bluetoothd for example. So if disabled, I expect that no
> >> resources/queues are in use.
> >>
> >> Maybe I misunderstand the virtio spec in that regard, but I would like
> >> to keep this fundamental concept of a Bluetooth driver. It does work
> >> with all other transports like USB, SDIO, UART etc.
> >>
> >>> The cost here is a single skb wasted on an unused bt device - which
> >>> seems modest.
> >>
> >> There should be no buffer used if the device is powered off. We also don’t
> >> have any USB URBs in-flight if the transport is not active.
> >>
> >>> NB: with this fix in place driver still suffers from a race condition if
> >>> an interrupt triggers while device is being reset. Work on a fix for
> >>> that issue is in progress.
> >>
> >> In the virtbt_close() callback we should deactivate all interrupts.
> >>
> >
> > If you want to do that then device has to be reset on close,
> > and fully reinitialized on open.
> > Can you work on a patch like that?
> > Given I don't have the device such a rework is probably more
> > than I can undertake.
>
> so you mean move virtio_find_vqs() into virtbt_open() and del_vqs() into
> virtbt_close()?

And reset before del_vqs.

> Or is there are way to set up the queues without starting them?
>
> However I am failing to understand your initial concern, we do reset()
> before del_vqs() in virtbt_remove(). Should we be doing something different
> in virtbt_close() other than virtqueue_detach_unused_buf(). Why would I
> keep buffers attached if they are not used.
>
> Regards
>
> Marcel

They are not used at that point but until device is reset can use them.
Also, if you then proceed to open without a reset, and kick,
device will start by processing the original buffers, crashing
or corrupting memory.

--
MST


2021-11-25 22:00:59

by Marcel Holtmann

[permalink] [raw]
Subject: Re: [PATCH] Bluetooth: virtio_bt: fix device removal

Hi Michael,

>>>>> Device removal is clearly out of virtio spec: it attempts to remove
>>>>> unused buffers from a VQ before invoking device reset. To fix, make
>>>>> open/close NOPs and do all cleanup/setup in probe/remove.
>>>>
>>>> so the virtbt_{open,close} as NOP is not really what a driver is suppose
>>>> to be doing. These are transport enable/disable callbacks from the BT
>>>> Core towards the driver. It maps to a device being enabled/disabled by
>>>> something like bluetoothd for example. So if disabled, I expect that no
>>>> resources/queues are in use.
>>>>
>>>> Maybe I misunderstand the virtio spec in that regard, but I would like
>>>> to keep this fundamental concept of a Bluetooth driver. It does work
>>>> with all other transports like USB, SDIO, UART etc.
>>>>
>>>>> The cost here is a single skb wasted on an unused bt device - which
>>>>> seems modest.
>>>>
>>>> There should be no buffer used if the device is powered off. We also don’t
>>>> have any USB URBs in-flight if the transport is not active.
>>>>
>>>>> NB: with this fix in place driver still suffers from a race condition if
>>>>> an interrupt triggers while device is being reset. Work on a fix for
>>>>> that issue is in progress.
>>>>
>>>> In the virtbt_close() callback we should deactivate all interrupts.
>>>>
>>>
>>> If you want to do that then device has to be reset on close,
>>> and fully reinitialized on open.
>>> Can you work on a patch like that?
>>> Given I don't have the device such a rework is probably more
>>> than I can undertake.
>>
>> so you mean move virtio_find_vqs() into virtbt_open() and del_vqs() into
>> virtbt_close()?
>
> And reset before del_vqs.
>
>> Or is there are way to set up the queues without starting them?
>>
>> However I am failing to understand your initial concern, we do reset()
>> before del_vqs() in virtbt_remove(). Should we be doing something different
>> in virtbt_close() other than virtqueue_detach_unused_buf(). Why would I
>> keep buffers attached if they are not used.
>>
>
> They are not used at that point but until device is reset can use them.
> Also, if you then proceed to open without a reset, and kick,
> device will start by processing the original buffers, crashing
> or corrupting memory.

so the only valid usage is like this:

vdev->config->reset(vdev);

while ((.. = virtqueue_detach_unused_buf(vq))) {
}

vdev->config->del_vqs(vdev);

If I make virtbt_{open,close} a NOP, then I keep adding an extra SKB to inbuf on
every power cycle (ifup/ifdown). How does netdev handle this?

Regards

Marcel


2021-11-26 01:04:53

by Michael S. Tsirkin

[permalink] [raw]
Subject: Re: [PATCH] Bluetooth: virtio_bt: fix device removal

On Thu, Nov 25, 2021 at 10:58:56PM +0100, Marcel Holtmann wrote:
> Hi Michael,
>
> >>>>> Device removal is clearly out of virtio spec: it attempts to remove
> >>>>> unused buffers from a VQ before invoking device reset. To fix, make
> >>>>> open/close NOPs and do all cleanup/setup in probe/remove.
> >>>>
> >>>> so the virtbt_{open,close} as NOP is not really what a driver is suppose
> >>>> to be doing. These are transport enable/disable callbacks from the BT
> >>>> Core towards the driver. It maps to a device being enabled/disabled by
> >>>> something like bluetoothd for example. So if disabled, I expect that no
> >>>> resources/queues are in use.
> >>>>
> >>>> Maybe I misunderstand the virtio spec in that regard, but I would like
> >>>> to keep this fundamental concept of a Bluetooth driver. It does work
> >>>> with all other transports like USB, SDIO, UART etc.
> >>>>
> >>>>> The cost here is a single skb wasted on an unused bt device - which
> >>>>> seems modest.
> >>>>
> >>>> There should be no buffer used if the device is powered off. We also don’t
> >>>> have any USB URBs in-flight if the transport is not active.
> >>>>
> >>>>> NB: with this fix in place driver still suffers from a race condition if
> >>>>> an interrupt triggers while device is being reset. Work on a fix for
> >>>>> that issue is in progress.
> >>>>
> >>>> In the virtbt_close() callback we should deactivate all interrupts.
> >>>>
> >>>
> >>> If you want to do that then device has to be reset on close,
> >>> and fully reinitialized on open.
> >>> Can you work on a patch like that?
> >>> Given I don't have the device such a rework is probably more
> >>> than I can undertake.
> >>
> >> so you mean move virtio_find_vqs() into virtbt_open() and del_vqs() into
> >> virtbt_close()?
> >
> > And reset before del_vqs.
> >
> >> Or is there are way to set up the queues without starting them?
> >>
> >> However I am failing to understand your initial concern, we do reset()
> >> before del_vqs() in virtbt_remove(). Should we be doing something different
> >> in virtbt_close() other than virtqueue_detach_unused_buf(). Why would I
> >> keep buffers attached if they are not used.
> >>
> >
> > They are not used at that point but until device is reset can use them.
> > Also, if you then proceed to open without a reset, and kick,
> > device will start by processing the original buffers, crashing
> > or corrupting memory.
>
> so the only valid usage is like this:
>
> vdev->config->reset(vdev);
>
> while ((.. = virtqueue_detach_unused_buf(vq))) {
> }
>
> vdev->config->del_vqs(vdev);
>
> If I make virtbt_{open,close} a NOP, then I keep adding an extra SKB to inbuf on
> every power cycle (ifup/ifdown).

So make sure you don't :)

> How does netdev handle this?
>
> Regards
>
> Marcel

For net, open adds buffers to vq. close does not free them up -
they stay in the vq until device is removed.

--
MST


2021-12-09 21:23:03

by Michael S. Tsirkin

[permalink] [raw]
Subject: Re: [PATCH] Bluetooth: virtio_bt: fix device removal

On Thu, Nov 25, 2021 at 09:02:01PM +0100, Marcel Holtmann wrote:
> Hi Michael,
>
> > Device removal is clearly out of virtio spec: it attempts to remove
> > unused buffers from a VQ before invoking device reset. To fix, make
> > open/close NOPs and do all cleanup/setup in probe/remove.
>
> so the virtbt_{open,close} as NOP is not really what a driver is suppose
> to be doing. These are transport enable/disable callbacks from the BT
> Core towards the driver. It maps to a device being enabled/disabled by
> something like bluetoothd for example. So if disabled, I expect that no
> resources/queues are in use.
>
> Maybe I misunderstand the virtio spec in that regard, but I would like
> to keep this fundamental concept of a Bluetooth driver. It does work
> with all other transports like USB, SDIO, UART etc.
>
> > The cost here is a single skb wasted on an unused bt device - which
> > seems modest.
>
> There should be no buffer used if the device is powered off. We also don’t
> have any USB URBs in-flight if the transport is not active.
>
> > NB: with this fix in place driver still suffers from a race condition if
> > an interrupt triggers while device is being reset. Work on a fix for
> > that issue is in progress.
>
> In the virtbt_close() callback we should deactivate all interrupts.
>
> Regards
>
> Marcel

So Marcel, do I read it right that you are working on a fix
and I can drop this patch for now?

--
MST


2021-12-13 10:44:17

by Michael S. Tsirkin

[permalink] [raw]
Subject: Re: [PATCH] Bluetooth: virtio_bt: fix device removal

On Thu, Dec 09, 2021 at 04:22:58PM -0500, Michael S. Tsirkin wrote:
> On Thu, Nov 25, 2021 at 09:02:01PM +0100, Marcel Holtmann wrote:
> > Hi Michael,
> >
> > > Device removal is clearly out of virtio spec: it attempts to remove
> > > unused buffers from a VQ before invoking device reset. To fix, make
> > > open/close NOPs and do all cleanup/setup in probe/remove.
> >
> > so the virtbt_{open,close} as NOP is not really what a driver is suppose
> > to be doing. These are transport enable/disable callbacks from the BT
> > Core towards the driver. It maps to a device being enabled/disabled by
> > something like bluetoothd for example. So if disabled, I expect that no
> > resources/queues are in use.
> >
> > Maybe I misunderstand the virtio spec in that regard, but I would like
> > to keep this fundamental concept of a Bluetooth driver. It does work
> > with all other transports like USB, SDIO, UART etc.
> >
> > > The cost here is a single skb wasted on an unused bt device - which
> > > seems modest.
> >
> > There should be no buffer used if the device is powered off. We also don’t
> > have any USB URBs in-flight if the transport is not active.
> >
> > > NB: with this fix in place driver still suffers from a race condition if
> > > an interrupt triggers while device is being reset. Work on a fix for
> > > that issue is in progress.
> >
> > In the virtbt_close() callback we should deactivate all interrupts.
> >
> > Regards
> >
> > Marcel
>
> So Marcel, do I read it right that you are working on a fix
> and I can drop this patch for now?

ping

> --
> MST


2021-12-13 23:57:44

by Michael S. Tsirkin

[permalink] [raw]
Subject: Re: [PATCH] Bluetooth: virtio_bt: fix device removal

On Mon, Dec 13, 2021 at 05:44:13AM -0500, Michael S. Tsirkin wrote:
> On Thu, Dec 09, 2021 at 04:22:58PM -0500, Michael S. Tsirkin wrote:
> > On Thu, Nov 25, 2021 at 09:02:01PM +0100, Marcel Holtmann wrote:
> > > Hi Michael,
> > >
> > > > Device removal is clearly out of virtio spec: it attempts to remove
> > > > unused buffers from a VQ before invoking device reset. To fix, make
> > > > open/close NOPs and do all cleanup/setup in probe/remove.
> > >
> > > so the virtbt_{open,close} as NOP is not really what a driver is suppose
> > > to be doing. These are transport enable/disable callbacks from the BT
> > > Core towards the driver. It maps to a device being enabled/disabled by
> > > something like bluetoothd for example. So if disabled, I expect that no
> > > resources/queues are in use.
> > >
> > > Maybe I misunderstand the virtio spec in that regard, but I would like
> > > to keep this fundamental concept of a Bluetooth driver. It does work
> > > with all other transports like USB, SDIO, UART etc.
> > >
> > > > The cost here is a single skb wasted on an unused bt device - which
> > > > seems modest.
> > >
> > > There should be no buffer used if the device is powered off. We also don’t
> > > have any USB URBs in-flight if the transport is not active.
> > >
> > > > NB: with this fix in place driver still suffers from a race condition if
> > > > an interrupt triggers while device is being reset. Work on a fix for
> > > > that issue is in progress.
> > >
> > > In the virtbt_close() callback we should deactivate all interrupts.
> > >
> > > Regards
> > >
> > > Marcel
> >
> > So Marcel, do I read it right that you are working on a fix
> > and I can drop this patch for now?
>
> ping


If I don't hear otherwise I'll queue my version - it might not
be ideal but it at least does not violate the spec.
We can work on not allocating/freeing buffers later
as appropriate.

> > --
> > MST


2021-12-16 19:58:34

by Marcel Holtmann

[permalink] [raw]
Subject: Re: [PATCH] Bluetooth: virtio_bt: fix device removal

Hi Michael,

>>>>> Device removal is clearly out of virtio spec: it attempts to remove
>>>>> unused buffers from a VQ before invoking device reset. To fix, make
>>>>> open/close NOPs and do all cleanup/setup in probe/remove.
>>>>
>>>> so the virtbt_{open,close} as NOP is not really what a driver is suppose
>>>> to be doing. These are transport enable/disable callbacks from the BT
>>>> Core towards the driver. It maps to a device being enabled/disabled by
>>>> something like bluetoothd for example. So if disabled, I expect that no
>>>> resources/queues are in use.
>>>>
>>>> Maybe I misunderstand the virtio spec in that regard, but I would like
>>>> to keep this fundamental concept of a Bluetooth driver. It does work
>>>> with all other transports like USB, SDIO, UART etc.
>>>>
>>>>> The cost here is a single skb wasted on an unused bt device - which
>>>>> seems modest.
>>>>
>>>> There should be no buffer used if the device is powered off. We also don’t
>>>> have any USB URBs in-flight if the transport is not active.
>>>>
>>>>> NB: with this fix in place driver still suffers from a race condition if
>>>>> an interrupt triggers while device is being reset. Work on a fix for
>>>>> that issue is in progress.
>>>>
>>>> In the virtbt_close() callback we should deactivate all interrupts.
>>>>
>>>> Regards
>>>>
>>>> Marcel
>>>
>>> So Marcel, do I read it right that you are working on a fix
>>> and I can drop this patch for now?
>>
>> ping
>
>
> If I don't hear otherwise I'll queue my version - it might not
> be ideal but it at least does not violate the spec.
> We can work on not allocating/freeing buffers later
> as appropriate.

I have a patch, but it is not fully tested yet.

Regards

Marcel


2022-01-14 23:04:38

by Michael S. Tsirkin

[permalink] [raw]
Subject: Re: [PATCH] Bluetooth: virtio_bt: fix device removal

On Thu, Dec 16, 2021 at 08:58:31PM +0100, Marcel Holtmann wrote:
> Hi Michael,
>
> >>>>> Device removal is clearly out of virtio spec: it attempts to remove
> >>>>> unused buffers from a VQ before invoking device reset. To fix, make
> >>>>> open/close NOPs and do all cleanup/setup in probe/remove.
> >>>>
> >>>> so the virtbt_{open,close} as NOP is not really what a driver is suppose
> >>>> to be doing. These are transport enable/disable callbacks from the BT
> >>>> Core towards the driver. It maps to a device being enabled/disabled by
> >>>> something like bluetoothd for example. So if disabled, I expect that no
> >>>> resources/queues are in use.
> >>>>
> >>>> Maybe I misunderstand the virtio spec in that regard, but I would like
> >>>> to keep this fundamental concept of a Bluetooth driver. It does work
> >>>> with all other transports like USB, SDIO, UART etc.
> >>>>
> >>>>> The cost here is a single skb wasted on an unused bt device - which
> >>>>> seems modest.
> >>>>
> >>>> There should be no buffer used if the device is powered off. We also don’t
> >>>> have any USB URBs in-flight if the transport is not active.
> >>>>
> >>>>> NB: with this fix in place driver still suffers from a race condition if
> >>>>> an interrupt triggers while device is being reset. Work on a fix for
> >>>>> that issue is in progress.
> >>>>
> >>>> In the virtbt_close() callback we should deactivate all interrupts.
> >>>>
> >>>> Regards
> >>>>
> >>>> Marcel
> >>>
> >>> So Marcel, do I read it right that you are working on a fix
> >>> and I can drop this patch for now?
> >>
> >> ping
> >
> >
> > If I don't hear otherwise I'll queue my version - it might not
> > be ideal but it at least does not violate the spec.
> > We can work on not allocating/freeing buffers later
> > as appropriate.
>
> I have a patch, but it is not fully tested yet.
>
> Regards
>
> Marcel

ping

it's been a month ...

I'm working on cleaning up module/device removal in virtio and bt
is kind of sticking out.

--
MST