LinuxLists.cc - [PATCH] usb: dwc3: gadget: Handle dequeuing of non queued URB gracefully

[permalink] [raw]

Subject: Re: [PATCH] usb: dwc3: gadget: Handle dequeuing of non queued URB gracefully

Hi,

"Ardelean, Alexandru" <[email protected]> writes:
> On Tue, 2019-11-12 at 15:41 +0100, Michael Olbrich wrote:
>> [External]
>>
>> On Wed, Nov 06, 2019 at 04:45:53PM +0200, Alexandru Ardelean wrote:
>> > From: Lars-Peter Clausen <[email protected]>
>> >
>> > Trying to dequeue and URB that is currently not queued should be a no-
>> > op
>> > and be handled gracefully.
>> >
>> > Use the list field of the URB to indicate whether it is queued or not
>> > by
>> > setting it to the empty list when it is not queued.
>> >
>> > Handling this gracefully allows for race condition free synchronization
>> > between the complete callback being called to to a completed transfer
>> > and
>> > trying to call usb_ep_dequeue() at the same time.
>> >
>> > Signed-off-by: Lars-Peter Clausen <[email protected]>
>> > Signed-off-by: Alexandru Ardelean <[email protected]>
>>
>> Thanks, no more "dwc3 fe200000.usb: request 00000000cdd42e4a was not
>> queued
>> to ep2in" messages with this patch applied.
>>
>> Tested-by: Michael Olbrich <[email protected]>
>>
>
> I thought I replied here, but I guess I never hit the Send button.
> Many thanks @Michael for testing this.
>
> I'd also use the opportunity to make this a patch-ping message.

https://lore.kernel.org/linux-usb/[email protected]/T/#u

--
balbi

Attachments:

signature.asc (847.00 B)

2020-01-16 13:26:02

by Alexandru Ardelean

[permalink] [raw]

Subject: [PATCH][RESEND] usb: dwc3: gadget: Handle dequeuing of non queued URB gracefully

From: Lars-Peter Clausen <[email protected]>

Trying to dequeue and URB that is currently not queued should be a no-op
and be handled gracefully.

Use the list field of the URB to indicate whether it is queued or not by
setting it to the empty list when it is not queued.

Handling this gracefully allows for race condition free synchronization
between the complete callback being called to to a completed transfer and
trying to call usb_ep_dequeue() at the same time.

Tested-by: Michael Olbrich <[email protected]>
Signed-off-by: Lars-Peter Clausen <[email protected]>
Signed-off-by: Alexandru Ardelean <[email protected]>
---

* Added Michael Olbrich's Tested-by tag
https://lore.kernel.org/linux-usb/[email protected]/

drivers/usb/dwc3/gadget.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
index 1b8014ab0b25..22a78eb41a5b 100644
--- a/drivers/usb/dwc3/gadget.c
+++ b/drivers/usb/dwc3/gadget.c
@@ -177,7 +177,7 @@ static void dwc3_gadget_del_and_unmap_request(struct dwc3_ep *dep,
{
struct dwc3 *dwc = dep->dwc;

- list_del(&req->list);
+ list_del_init(&req->list);
req->remaining = 0;
req->needs_extra_trb = false;

@@ -847,6 +847,7 @@ static struct usb_request *dwc3_gadget_ep_alloc_request(struct usb_ep *ep,
req->epnum = dep->number;
req->dep = dep;
req->status = DWC3_REQUEST_STATUS_UNKNOWN;
+ INIT_LIST_HEAD(&req->list);

trace_dwc3_alloc_request(req);

@@ -1549,6 +1550,10 @@ static int dwc3_gadget_ep_dequeue(struct usb_ep *ep,

spin_lock_irqsave(&dwc->lock, flags);

+ /* Not queued, nothing to do */
+ if (list_empty(&req->list))
+ goto out0;
+
list_for_each_entry(r, &dep->pending_list, list) {
if (r == req)
break;
--
2.20.1

2020-01-30 12:03:36

[permalink] [raw]

Subject: Re: [PATCH][RESEND] usb: dwc3: gadget: Handle dequeuing of non queued URB gracefully

Hi,

Alexandru Ardelean <[email protected]> writes:

> From: Lars-Peter Clausen <[email protected]>
>
> Trying to dequeue and URB that is currently not queued should be a no-op
> and be handled gracefully.
>
> Use the list field of the URB to indicate whether it is queued or not by
> setting it to the empty list when it is not queued.
>
> Handling this gracefully allows for race condition free synchronization
> between the complete callback being called to to a completed transfer and
> trying to call usb_ep_dequeue() at the same time.

We need a little more information here. Can you further explain what
happens and how you caught this?

> Tested-by: Michael Olbrich <[email protected]>
> Signed-off-by: Lars-Peter Clausen <[email protected]>
> Signed-off-by: Alexandru Ardelean <[email protected]>
> ---
>
> * Added Michael Olbrich's Tested-by tag
> https://lore.kernel.org/linux-usb/[email protected]/
>
> drivers/usb/dwc3/gadget.c | 7 ++++++-
> 1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
> index 1b8014ab0b25..22a78eb41a5b 100644
> --- a/drivers/usb/dwc3/gadget.c
> +++ b/drivers/usb/dwc3/gadget.c
> @@ -177,7 +177,7 @@ static void dwc3_gadget_del_and_unmap_request(struct dwc3_ep *dep,
> {
> struct dwc3 *dwc = dep->dwc;
>
> - list_del(&req->list);
> + list_del_init(&req->list);

this should *not* be necessary. Neither the INIT_LIST_HEAD() below.

> req->remaining = 0;
> req->needs_extra_trb = false;
>
> @@ -847,6 +847,7 @@ static struct usb_request *dwc3_gadget_ep_alloc_request(struct usb_ep *ep,
> req->epnum = dep->number;
> req->dep = dep;
> req->status = DWC3_REQUEST_STATUS_UNKNOWN;
> + INIT_LIST_HEAD(&req->list);
>
> trace_dwc3_alloc_request(req);
>
> @@ -1549,6 +1550,10 @@ static int dwc3_gadget_ep_dequeue(struct usb_ep *ep,
>
> spin_lock_irqsave(&dwc->lock, flags);
>
> + /* Not queued, nothing to do */
> + if (list_empty(&req->list))
> + goto out0;

The loop below is actually looking for the request in our lists. You
just made the entire loop below unnecessary, but you didn't change it
accordingly. Moreover, I think that a user dequeueing a request that
wasn't queued for the current endpoint indicates a possible bug in the
gadget driver which needs to be fixed.

If you really disagree, suffice to change "ret = -EINVAL;" to "ret =
0;" and you would get what you want, without any of the extra cruft.

cheers

--
balbi

Attachments:

signature.asc (847.00 B)

2020-03-10 13:25:30

by Alexandru Ardelean

[permalink] [raw]

Subject: Re: [PATCH][RESEND] usb: dwc3: gadget: Handle dequeuing of non queued URB gracefully

On Thu, 2020-01-30 at 14:02 +0200, Felipe Balbi wrote:
> [External]
>
>
> Hi,
>
> Alexandru Ardelean <[email protected]> writes:
>
> > From: Lars-Peter Clausen <[email protected]>
> >
> > Trying to dequeue and URB that is currently not queued should be a no-op
> > and be handled gracefully.
> >
> > Use the list field of the URB to indicate whether it is queued or not by
> > setting it to the empty list when it is not queued.
> >
> > Handling this gracefully allows for race condition free synchronization
> > between the complete callback being called to to a completed transfer and
> > trying to call usb_ep_dequeue() at the same time.
>
> We need a little more information here. Can you further explain what
> happens and how you caught this?

Apologies for the delay [of this reply].
It's been a while since this patch was created, and it was on a 4.14 kernel.
Lars was trying to fix various crashes with USB DWC3 OTG + some Xilinx patches.
I did not track the status of the OTG stuff upstream. I think it's a lot of
patches in the Xilinx tree.

The context has changed from 4.14 [obviously], and there were many things that
could have influenced things.
I've been trying to RFC some of these patches now.
[ yeah I know: maybe I should have [probably] also added an RFC tag :) ]
Some of the patches [including this one] seemed to make sense, even outside of
the context of the crashes that were happening on 4.14.
Atm, we're at 4.19 and we don't see issues, but we still have this patch.
We may drop it and see what happens.
¯\_(ツ)_/¯

But in any case, it does require a bit more re-investigation.
Apologies for the noise that this patch created :)

>
> > Tested-by: Michael Olbrich <[email protected]>
> > Signed-off-by: Lars-Peter Clausen <[email protected]>
> > Signed-off-by: Alexandru Ardelean <[email protected]>
> > ---
> >
> > * Added Michael Olbrich's Tested-by tag
> > https://lore.kernel.org/linux-usb/[email protected]/
> >
> > drivers/usb/dwc3/gadget.c | 7 ++++++-
> > 1 file changed, 6 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
> > index 1b8014ab0b25..22a78eb41a5b 100644
> > --- a/drivers/usb/dwc3/gadget.c
> > +++ b/drivers/usb/dwc3/gadget.c
> > @@ -177,7 +177,7 @@ static void dwc3_gadget_del_and_unmap_request(struct
> > dwc3_ep *dep,
> > {
> > struct dwc3 *dwc = dep->dwc;
> >
> > - list_del(&req->list);
> > + list_del_init(&req->list);
>
> this should *not* be necessary. Neither the INIT_LIST_HEAD() below.
>
> > req->remaining = 0;
> > req->needs_extra_trb = false;
> >
> > @@ -847,6 +847,7 @@ static struct usb_request
> > *dwc3_gadget_ep_alloc_request(struct usb_ep *ep,
> > req->epnum = dep->number;
> > req->dep = dep;
> > req->status = DWC3_REQUEST_STATUS_UNKNOWN;
> > + INIT_LIST_HEAD(&req->list);
> >
> > trace_dwc3_alloc_request(req);
> >
> > @@ -1549,6 +1550,10 @@ static int dwc3_gadget_ep_dequeue(struct usb_ep *ep,
> >
> > spin_lock_irqsave(&dwc->lock, flags);
> >
> > + /* Not queued, nothing to do */
> > + if (list_empty(&req->list))
> > + goto out0;
>
> The loop below is actually looking for the request in our lists. You
> just made the entire loop below unnecessary, but you didn't change it
> accordingly. Moreover, I think that a user dequeueing a request that
> wasn't queued for the current endpoint indicates a possible bug in the
> gadget driver which needs to be fixed.
>

Yeah, that could be.
Will see about reverting the patch on our end, and trying to track this again.

Thanks
Alex

> If you really disagree, suffice to change "ret = -EINVAL;" to "ret =
> 0;" and you would get what you want, without any of the extra cruft.
>
> cheers
>

2020-03-10 13:46:35

[permalink] [raw]

Subject: Re: [PATCH][RESEND] usb: dwc3: gadget: Handle dequeuing of non queued URB gracefully

On 3/10/20 2:22 PM, Ardelean, Alexandru wrote:
> On Thu, 2020-01-30 at 14:02 +0200, Felipe Balbi wrote:
>> [External]
>>
>>
>> Hi,
>>
>> Alexandru Ardelean <[email protected]> writes:
>>
>>> From: Lars-Peter Clausen <[email protected]>
>>>
>>> Trying to dequeue and URB that is currently not queued should be a no-op
>>> and be handled gracefully.
>>>
>>> Use the list field of the URB to indicate whether it is queued or not by
>>> setting it to the empty list when it is not queued.
>>>
>>> Handling this gracefully allows for race condition free synchronization
>>> between the complete callback being called to to a completed transfer and
>>> trying to call usb_ep_dequeue() at the same time.
>> We need a little more information here. Can you further explain what
>> happens and how you caught this?
> Apologies for the delay [of this reply].
> It's been a while since this patch was created, and it was on a 4.14 kernel.
> Lars was trying to fix various crashes with USB DWC3 OTG + some Xilinx patches.
> I did not track the status of the OTG stuff upstream. I think it's a lot of
> patches in the Xilinx tree.
>
> The context has changed from 4.14 [obviously], and there were many things that
> could have influenced things.
> I've been trying to RFC some of these patches now.
> [ yeah I know: maybe I should have [probably] also added an RFC tag :) ]
> Some of the patches [including this one] seemed to make sense, even outside of
> the context of the crashes that were happening on 4.14.
> Atm, we're at 4.19 and we don't see issues, but we still have this patch.
> We may drop it and see what happens.
> ¯\_(ツ)_/¯
>
> But in any case, it does require a bit more re-investigation.
> Apologies for the noise that this patch created :)

The race condition is between a gadget calling usb_ep_dequeue() and the
driver completing the URB.

Lets say in a thread you have a reference to a in-flight URB and you
want to abort the request, e.g. because the application that sent the
request has been closed. But concurrently to that the URB is completed
by the hardware and the interrupt fires and marks the URB as complete.
Your thread is suspended while the interrupt is running, once the
interrupt has finished the thread wakes up, still has the reference to
the URB, but now it has been completed. The thread still calls
usb_ep_dequeue() though and then undefined behavior occurs.

The context in which we observed the issue is when using function fs to
create a userspace gadget and using aio_cancel() to abort a pending URB.
But really any gadget that aborts a transfer before it is completed or
before the timeout occurred can run into this issue.

- Lars

2020-03-10 14:10:31

[permalink] [raw]

Subject: Re: [PATCH][RESEND] usb: dwc3: gadget: Handle dequeuing of non queued URB gracefully

On 3/10/20 2:45 PM, Lars-Peter Clausen wrote:
> On 3/10/20 2:22 PM, Ardelean, Alexandru wrote:
>> On Thu, 2020-01-30 at 14:02 +0200, Felipe Balbi wrote:
>>> [External]
>>>
>>>
>>> Hi,
>>>
>>> Alexandru Ardelean <[email protected]> writes:
>>>
>>>> From: Lars-Peter Clausen <[email protected]>
>>>>
>>>> Trying to dequeue and URB that is currently not queued should be a
>>>> no-op
>>>> and be handled gracefully.
>>>>
>>>> Use the list field of the URB to indicate whether it is queued or
>>>> not by
>>>> setting it to the empty list when it is not queued.
>>>>
>>>> Handling this gracefully allows for race condition free synchronization
>>>> between the complete callback being called to to a completed
>>>> transfer and
>>>> trying to call usb_ep_dequeue() at the same time.
>>> We need a little more information here. Can you further explain what
>>> happens and how you caught this?
>> Apologies for the delay [of this reply].
>> It's been a while since this patch was created, and it was on a 4.14
>> kernel.
>> Lars was trying to fix various crashes with USB DWC3 OTG + some Xilinx
>> patches.
>> I did not track the status of the OTG stuff upstream. I think it's a
>> lot of
>> patches in the Xilinx tree.
>>
>> The context has changed from 4.14 [obviously], and there were many
>> things that
>> could have influenced things.
>> I've been trying to RFC some of these patches now.
>> [ yeah I know: maybe I should have [probably] also added an RFC tag :) ]
>> Some of the patches [including this one] seemed to make sense, even
>> outside of
>> the context of the crashes that were happening on 4.14.
>> Atm, we're at 4.19 and we don't see issues, but we still have this patch.
>> We may drop it and see what happens.
>> ¯\_(ツ)_/¯
>>
>> But in any case, it does require a bit more re-investigation.
>> Apologies for the noise that this patch created :)
>
> The race condition is between a gadget calling usb_ep_dequeue() and the
> driver completing the URB.
>
> Lets say in a thread you have a reference to a in-flight URB and you
> want to abort the request, e.g. because the application that sent the
> request has been closed. But concurrently to that the URB is completed
> by the hardware and the interrupt fires and marks the URB as complete.
> Your thread is suspended while the interrupt is running, once the
> interrupt has finished the thread wakes up, still has the reference to
> the URB, but now it has been completed. The thread still calls
> usb_ep_dequeue() though and then undefined behavior occurs.
>

Sorry, one quick correction. I believe the issue actually occurs when
you have more than one CPU and the thread is not suspended, while the
interrupt is running. In this case it is possible that the IRQ fires the
driver marks the URB as complete, then unlocks, the driver lock and
calls the complete callback, but before the complete callback runs the
other thread calls usb_ep_dequeue(). There is no way to protect against
this condition at the gadget level and it needs to be handled in the driver.

Basically

CPU 1 | CPU 2
--------------------------------------------------------------
URB IRQ fires | |
spin_lock(&dwc->lock); | |
| |
Driver handles completed URB | |
frees resources, etc | spin_lock(&gadget->lock) |
| usb_ep_dequeue() |
spin_unlock(&dwc->lock); | |
usb_gadget_giveback_request() | spin_lock(&dwc->lock) |
| |
Call compelte callback | Try to free URB resources |
| again => Undefined behavior |
| |
| spin_lock(&dwc->lock) |
| spin_unlock(&gadget->lock) |
spin_lock(&gadget->lock) | |
| |
Mark URB as compelte in gadget | |
spin_unlock(&gadget->lock) | |
spin_unlock(&dwc->lock) | |
spin_lock(&dwc->lock) | |
finish IRQ | |

> The context in which we observed the issue is when using function fs to
> create a userspace gadget and using aio_cancel() to abort a pending URB.
> But really any gadget that aborts a transfer before it is completed or
> before the timeout occurred can run into this issue.
>
> - Lars
>

2020-03-27 08:48:18

[permalink] [raw]

Subject: [PATCH] usb: dwc3: gadget: don't dequeue requests on already disabled endpoints

dwc3_gadget_ep_disable gets called before the last request gets
dequeued.

In __dwc3_gadget_ep_disable all started, pending and cancelled
lists for this endpoint will call dwc3_gadget_giveback in
dwc3_remove_requests.

After that no list containing the afterwards dequed request,
therefor it is not necessary to run the dequeue routine.

Signed-off-by: Michael Grzeschik <[email protected]>
---
@Lars-Peter Clausen:

This patch addresses the case that not queued requests get dequeued.
The only case that this happens seems on disabling the gadget.

drivers/usb/dwc3/gadget.c | 3 +++
1 file changed, 3 insertions(+)

diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
index 9a6f741d1db0dc..5d4fa8d6c93e49 100644
--- a/drivers/usb/dwc3/gadget.c
+++ b/drivers/usb/dwc3/gadget.c
@@ -1609,6 +1609,9 @@ static int dwc3_gadget_ep_dequeue(struct usb_ep *ep,

trace_dwc3_ep_dequeue(req);

+ if (!(dep->flags & DWC3_EP_ENABLED))
+ return 0;
+
spin_lock_irqsave(&dwc->lock, flags);

list_for_each_entry(r, &dep->pending_list, list) {
--
2.26.0.rc2

2020-03-27 08:54:19

by Sergei Shtylyov

[permalink] [raw]

Subject: Re: [PATCH] usb: dwc3: gadget: don't dequeue requests on already disabled endpoints

Hello!

On 27.03.2020 11:43, Michael Grzeschik wrote:

> dwc3_gadget_ep_disable gets called before the last request gets
> dequeued.
>
> In __dwc3_gadget_ep_disable all started, pending and cancelled
> lists for this endpoint will call dwc3_gadget_giveback in
> dwc3_remove_requests.
>
> After that no list containing the afterwards dequed request,

Dequeued.

> therefor it is not necessary to run the dequeue routine.

Therefore?

> Signed-off-by: Michael Grzeschik <[email protected]>
[...]

MBR, Sergei

2020-03-27 09:16:07

[permalink] [raw]

Subject: Re: [PATCH] usb: dwc3: gadget: don't dequeue requests on already disabled endpoints

On 3/27/20 9:43 AM, Michael Grzeschik wrote:
> dwc3_gadget_ep_disable gets called before the last request gets
> dequeued.
>
> In __dwc3_gadget_ep_disable all started, pending and cancelled
> lists for this endpoint will call dwc3_gadget_giveback in
> dwc3_remove_requests.
>
> After that no list containing the afterwards dequed request,
> therefor it is not necessary to run the dequeue routine.
>
> Signed-off-by: Michael Grzeschik <[email protected]>
> ---
> @Lars-Peter Clausen:
>
> This patch addresses the case that not queued requests get dequeued.
> The only case that this happens seems on disabling the gadget.

I don't believe it does. Calling usb_ep_dequeue() is not limited to be
called after the endpoint has been disabled. It is part of the API and
can be called at any time. E.g. with function_fs you can abort a queued
transfer from userspace at any time.

- Lars

>
> drivers/usb/dwc3/gadget.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
> index 9a6f741d1db0dc..5d4fa8d6c93e49 100644
> --- a/drivers/usb/dwc3/gadget.c
> +++ b/drivers/usb/dwc3/gadget.c
> @@ -1609,6 +1609,9 @@ static int dwc3_gadget_ep_dequeue(struct usb_ep *ep,
>
> trace_dwc3_ep_dequeue(req);
>
> + if (!(dep->flags & DWC3_EP_ENABLED))
> + return 0;
> +
> spin_lock_irqsave(&dwc->lock, flags);
>
> list_for_each_entry(r, &dep->pending_list, list) {

2020-03-27 10:44:33

[permalink] [raw]

Subject: Re: [PATCH] usb: dwc3: gadget: don't dequeue requests on already disabled endpoints

On Fri, Mar 27, 2020 at 10:14:10AM +0100, Lars-Peter Clausen wrote:
> On 3/27/20 9:43 AM, Michael Grzeschik wrote:
> > dwc3_gadget_ep_disable gets called before the last request gets
> > dequeued.
> >
> > In __dwc3_gadget_ep_disable all started, pending and cancelled
> > lists for this endpoint will call dwc3_gadget_giveback in
> > dwc3_remove_requests.
> >
> > After that no list containing the afterwards dequed request,
> > therefor it is not necessary to run the dequeue routine.
> >
> > Signed-off-by: Michael Grzeschik <[email protected]>
> > ---
> > @Lars-Peter Clausen:
> >
> > This patch addresses the case that not queued requests get dequeued.
> > The only case that this happens seems on disabling the gadget.
>
>
> I don't believe it does. Calling usb_ep_dequeue() is not limited to be
> called after the endpoint has been disabled. It is part of the API and can
> be called at any time. E.g. with function_fs you can abort a queued transfer
> from userspace at any time.

OK, can you give me an Userspace example how to simply trigger the
issue. I tried to figure your mentioned function stack but it would
be much easier if it could be reproduced.

The patch on the other hand can stand on itself, as it then
fixes another code path that is much more common.

Regads,
Michael

> >
> > drivers/usb/dwc3/gadget.c | 3 +++
> > 1 file changed, 3 insertions(+)
> >
> > diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
> > index 9a6f741d1db0dc..5d4fa8d6c93e49 100644
> > --- a/drivers/usb/dwc3/gadget.c
> > +++ b/drivers/usb/dwc3/gadget.c
> > @@ -1609,6 +1609,9 @@ static int dwc3_gadget_ep_dequeue(struct usb_ep *ep,
> > trace_dwc3_ep_dequeue(req);
> > + if (!(dep->flags & DWC3_EP_ENABLED))
> > + return 0;
> > +
> > spin_lock_irqsave(&dwc->lock, flags);
> > list_for_each_entry(r, &dep->pending_list, list) {
>
>
>

--
Pengutronix e.K. | |
Steuerwalder Str. 21 | http://www.pengutronix.de/ |
31137 Hildesheim, Germany | Phone: +49-5121-206917-0 |
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |

Attachments:

(No filename) (2.19 kB)
signature.asc (849.00 B)
Download all attachments

2020-03-27 10:56:22

[permalink] [raw]

Subject: Re: [PATCH] usb: dwc3: gadget: don't dequeue requests on already disabled endpoints

On 3/27/20 11:43 AM, Michael Grzeschik wrote:
> On Fri, Mar 27, 2020 at 10:14:10AM +0100, Lars-Peter Clausen wrote:
>> On 3/27/20 9:43 AM, Michael Grzeschik wrote:
>>> dwc3_gadget_ep_disable gets called before the last request gets
>>> dequeued.
>>>
>>> In __dwc3_gadget_ep_disable all started, pending and cancelled
>>> lists for this endpoint will call dwc3_gadget_giveback in
>>> dwc3_remove_requests.
>>>
>>> After that no list containing the afterwards dequed request,
>>> therefor it is not necessary to run the dequeue routine.
>>>
>>> Signed-off-by: Michael Grzeschik <[email protected]>
>>> ---
>>> @Lars-Peter Clausen:
>>>
>>> This patch addresses the case that not queued requests get dequeued.
>>> The only case that this happens seems on disabling the gadget.
>>
>> I don't believe it does. Calling usb_ep_dequeue() is not limited to be
>> called after the endpoint has been disabled. It is part of the API and can
>> be called at any time. E.g. with function_fs you can abort a queued transfer
>> from userspace at any time.
> OK, can you give me an Userspace example how to simply trigger the
> issue. I tried to figure your mentioned function stack but it would
> be much easier if it could be reproduced.
>
> The patch on the other hand can stand on itself, as it then
> fixes another code path that is much more common.

Hi,

I don't have a standalone example. But the issue occurs if you submit a
request when using function_fs through the AIO API and then call
io_cancel() to abort it. At the same time there must be traffic on the
USB bus so that the URB has a chance to complete.

This is a race condition between the IRQ handler running on one CPU and
the usb_ep_dequeue() running on another CPU. As such it might take a
while of stress testing before it is triggered. The more CPUs your
system has the higher the chance of trigger the issue.

You can find the code which triggers the issue below.

Submission of the request:

https://github.com/analogdevicesinc/libiio/blob/master/iiod/ops.c#L139-L156

Canceling it:

https://github.com/analogdevicesinc/libiio/blob/master/iiod/ops.c#L193

2020-03-27 11:16:49

by Andy Shevchenko

[permalink] [raw]

Subject: Re: [PATCH] usb: dwc3: gadget: don't dequeue requests on already disabled endpoints

On Fri, Mar 27, 2020 at 10:54 AM Sergei Shtylyov
<[email protected]> wrote:
> On 27.03.2020 11:43, Michael Grzeschik wrote:

...

> > therefor it is not necessary to run the dequeue routine.
>
> Therefore?

Original as good on its own.

--
With Best Regards,
Andy Shevchenko

2020-03-28 08:29:38

[permalink] [raw]

Subject: Re: [PATCH] usb: dwc3: gadget: don't dequeue requests on already disabled endpoints

Hi,

Michael Grzeschik <[email protected]> writes:
> dwc3_gadget_ep_disable gets called before the last request gets
> dequeued.
>
> In __dwc3_gadget_ep_disable all started, pending and cancelled
> lists for this endpoint will call dwc3_gadget_giveback in
> dwc3_remove_requests.
>
> After that no list containing the afterwards dequed request,
> therefor it is not necessary to run the dequeue routine.
>
> Signed-off-by: Michael Grzeschik <[email protected]>
> ---
> @Lars-Peter Clausen:
>
> This patch addresses the case that not queued requests get dequeued.
> The only case that this happens seems on disabling the gadget.
>
> drivers/usb/dwc3/gadget.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
> index 9a6f741d1db0dc..5d4fa8d6c93e49 100644
> --- a/drivers/usb/dwc3/gadget.c
> +++ b/drivers/usb/dwc3/gadget.c
> @@ -1609,6 +1609,9 @@ static int dwc3_gadget_ep_dequeue(struct usb_ep *ep,
>
> trace_dwc3_ep_dequeue(req);
>
> + if (!(dep->flags & DWC3_EP_ENABLED))
> + return 0;

which driver is trying to call dequeue after the endpoint is disabled?
Got some tracepoints of the problem happening?

--
balbi

Attachments:

signature.asc (847.00 B)

2020-03-29 19:07:43

[permalink] [raw]

Subject: Re: [PATCH] usb: dwc3: gadget: don't dequeue requests on already disabled endpoints

On Sat, Mar 28, 2020 at 10:27:49AM +0200, Felipe Balbi wrote:
>
> Hi,
>
> Michael Grzeschik <[email protected]> writes:
> > dwc3_gadget_ep_disable gets called before the last request gets
> > dequeued.
> >
> > In __dwc3_gadget_ep_disable all started, pending and cancelled
> > lists for this endpoint will call dwc3_gadget_giveback in
> > dwc3_remove_requests.
> >
> > After that no list containing the afterwards dequed request,
> > therefor it is not necessary to run the dequeue routine.
> >
> > Signed-off-by: Michael Grzeschik <[email protected]>
> > ---
> > @Lars-Peter Clausen:
> >
> > This patch addresses the case that not queued requests get dequeued.
> > The only case that this happens seems on disabling the gadget.
> >
> > drivers/usb/dwc3/gadget.c | 3 +++
> > 1 file changed, 3 insertions(+)
> >
> > diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
> > index 9a6f741d1db0dc..5d4fa8d6c93e49 100644
> > --- a/drivers/usb/dwc3/gadget.c
> > +++ b/drivers/usb/dwc3/gadget.c
> > @@ -1609,6 +1609,9 @@ static int dwc3_gadget_ep_dequeue(struct usb_ep *ep,
> >
> > trace_dwc3_ep_dequeue(req);
> >
> > + if (!(dep->flags & DWC3_EP_ENABLED))
> > + return 0;
>
> which driver is trying to call dequeue after the endpoint is disabled?
> Got some tracepoints of the problem happening?

I see the case when using uvc-gadget.

Look into uvc_v4l2_release in uvc_v4l2.c:

uvc_function_disconnect
composite_disconnect
reset_config
uvc_function_disable->usb_ep_disable

uvcg_video_enable
usb_ep_dequeue
dwc3_gadget_ep_dequeue

Regards,
Michael

--
Pengutronix e.K. | |
Steuerwalder Str. 21 | http://www.pengutronix.de/ |
31137 Hildesheim, Germany | Phone: +49-5121-206917-0 |
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |

Attachments:

(No filename) (1.93 kB)
signature.asc (849.00 B)
Download all attachments

2020-03-30 07:19:48

[permalink] [raw]

Subject: Re: [PATCH] usb: dwc3: gadget: don't dequeue requests on already disabled endpoints

Hi,

Michael Grzeschik <[email protected]> writes:
>> > dwc3_gadget_ep_disable gets called before the last request gets
>> > dequeued.
>> >
>> > In __dwc3_gadget_ep_disable all started, pending and cancelled
>> > lists for this endpoint will call dwc3_gadget_giveback in
>> > dwc3_remove_requests.
>> >
>> > After that no list containing the afterwards dequed request,
>> > therefor it is not necessary to run the dequeue routine.
>> >
>> > Signed-off-by: Michael Grzeschik <[email protected]>
>> > ---
>> > @Lars-Peter Clausen:
>> >
>> > This patch addresses the case that not queued requests get dequeued.
>> > The only case that this happens seems on disabling the gadget.
>> >
>> > drivers/usb/dwc3/gadget.c | 3 +++
>> > 1 file changed, 3 insertions(+)
>> >
>> > diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
>> > index 9a6f741d1db0dc..5d4fa8d6c93e49 100644
>> > --- a/drivers/usb/dwc3/gadget.c
>> > +++ b/drivers/usb/dwc3/gadget.c
>> > @@ -1609,6 +1609,9 @@ static int dwc3_gadget_ep_dequeue(struct usb_ep *ep,
>> >
>> > trace_dwc3_ep_dequeue(req);
>> >
>> > + if (!(dep->flags & DWC3_EP_ENABLED))
>> > + return 0;
>>
>> which driver is trying to call dequeue after the endpoint is disabled?
>> Got some tracepoints of the problem happening?
>
> I see the case when using uvc-gadget.
>
> Look into uvc_v4l2_release in uvc_v4l2.c:
>
> uvc_function_disconnect
> composite_disconnect
> reset_config
> uvc_function_disable->usb_ep_disable
>
> uvcg_video_enable
> usb_ep_dequeue
> dwc3_gadget_ep_dequeue

Oh, I see what you mean. We get a disconnect, which disables the
endpoints, which forces all requests to be dequeued. Now I remember why
this exists: we giveback the requests from disconnect because not all
gadget drivers will call usb_ep_dequeue() if simply told about the
disconnect. Then UDC drivers have to be a little more careful and make
sure that all requests are givenback.

In any case, why is it a problem to call usb_ep_dequeue()? Is it only
because of that dev_err()? We could just remove that message,
really. Eventually, I want to move more of this logic into UDC core so
udc drivers can be simplified. For that work, though, first we would
have to add a "generic" struct usb_ep_hw implementation and manage list
of requests as part of UDC core as well.

--
balbi

Attachments:

signature.asc (847.00 B)

2020-03-30 08:27:00

[permalink] [raw]

Subject: Re: [PATCH] usb: dwc3: gadget: don't dequeue requests on already disabled endpoints

On Mon, Mar 30, 2020 at 10:18:57AM +0300, Felipe Balbi wrote:
>
> Hi,
>
> Michael Grzeschik <[email protected]> writes:
> >> > dwc3_gadget_ep_disable gets called before the last request gets
> >> > dequeued.
> >> >
> >> > In __dwc3_gadget_ep_disable all started, pending and cancelled
> >> > lists for this endpoint will call dwc3_gadget_giveback in
> >> > dwc3_remove_requests.
> >> >
> >> > After that no list containing the afterwards dequed request,
> >> > therefor it is not necessary to run the dequeue routine.
> >> >
> >> > Signed-off-by: Michael Grzeschik <[email protected]>
> >> > ---
> >> > @Lars-Peter Clausen:
> >> >
> >> > This patch addresses the case that not queued requests get dequeued.
> >> > The only case that this happens seems on disabling the gadget.
> >> >
> >> > drivers/usb/dwc3/gadget.c | 3 +++
> >> > 1 file changed, 3 insertions(+)
> >> >
> >> > diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
> >> > index 9a6f741d1db0dc..5d4fa8d6c93e49 100644
> >> > --- a/drivers/usb/dwc3/gadget.c
> >> > +++ b/drivers/usb/dwc3/gadget.c
> >> > @@ -1609,6 +1609,9 @@ static int dwc3_gadget_ep_dequeue(struct usb_ep *ep,
> >> >
> >> > trace_dwc3_ep_dequeue(req);
> >> >
> >> > + if (!(dep->flags & DWC3_EP_ENABLED))
> >> > + return 0;
> >>
> >> which driver is trying to call dequeue after the endpoint is disabled?
> >> Got some tracepoints of the problem happening?
> >
> > I see the case when using uvc-gadget.
> >
> > Look into uvc_v4l2_release in uvc_v4l2.c:
> >
> > uvc_function_disconnect
> > composite_disconnect
> > reset_config
> > uvc_function_disable->usb_ep_disable
> >
> > uvcg_video_enable
> > usb_ep_dequeue
> > dwc3_gadget_ep_dequeue
>
> Oh, I see what you mean. We get a disconnect, which disables the
> endpoints, which forces all requests to be dequeued. Now I remember why
> this exists: we giveback the requests from disconnect because not all
> gadget drivers will call usb_ep_dequeue() if simply told about the
> disconnect. Then UDC drivers have to be a little more careful and make
> sure that all requests are givenback.
>
> In any case, why is it a problem to call usb_ep_dequeue()? Is it only
> because of that dev_err()? We could just remove that message,
> really.

In my case, it is not a problem removing the dev_err. The ep_dequeue
will only be called once for each request at the stream end. I don't
know about the case Lars has mentioned.

If we have to search all lists for the request every n times while in
traffic, only to find out that it was not enqueued, I think it would be
worth it to keep the dev_err and let these cases trigger so we have an
option to find and avoid/fix these.

> Eventually, I want to move more of this logic into UDC core so
> udc drivers can be simplified. For that work, though, first we would
> have to add a "generic" struct usb_ep_hw implementation and manage list
> of requests as part of UDC core as well.

I don't know about the cases you plan to abstract but it sounds
like a good idea to get some gadget logic out of the drivers.

Thanks,
Michael

--
Pengutronix e.K. | |
Steuerwalder Str. 21 | http://www.pengutronix.de/ |
31137 Hildesheim, Germany | Phone: +49-5121-206917-0 |
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |

Attachments:

(No filename) (3.42 kB)
signature.asc (849.00 B)
Download all attachments

2020-03-30 10:08:53