Received-SPF: pass (google.com: domain of linux-kernel+bounces-168849-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Sun, 5 May 2024 01:44:58 +0200
From: Michael Grzeschik <mgr@pengutronix.de>
To: Thinh Nguyen <Thinh.Nguyen@synopsys.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	"linux-usb@vger.kernel.org" <linux-usb@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v2] usb: dwc3: gadget: check drained isoc ep
Message-ID: <ZjbIeib2UMta7FbY@pengutronix.de>
References: <20240307-dwc3-gadget-complete-irq-v2-1-8c5e9b35f7b9@pengutronix.de>
 <20240402230555.xgt5uilc42diyr4m@synopsys.com>
 <20240402231848.4hzzrxegjrcmdab2@synopsys.com>
 <20240404002906.wk6xbz2wp2tf2xwn@synopsys.com>
 <Zie5sN473m2rgNTK@pengutronix.de>
 <20240424015059.w7hsee4tt2ixkp5y@synopsys.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="pLoVn6qsK0dxz8GE"
Content-Disposition: inline
In-Reply-To: <20240424015059.w7hsee4tt2ixkp5y@synopsys.com>


--pLoVn6qsK0dxz8GE
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Wed, Apr 24, 2024 at 01:51:01AM +0000, Thinh Nguyen wrote:
>On Tue, Apr 23, 2024, Michael Grzeschik wrote:
>> Hi Thinh,
>>
>> On Thu, Apr 04, 2024 at 12:29:14AM +0000, Thinh Nguyen wrote:
>> > On Tue, Apr 02, 2024, Thinh Nguyen wrote:
>> > > On Tue, Apr 02, 2024, Thinh Nguyen wrote:
>> > > > My concern here is for the case where transfer_in_flight =3D=3D tr=
ue and
>> > >
>> > > I mean transfer_in_flight =3D=3D false
>> > >
>> > > > list_empty(started_list) =3D=3D false. That means that the request=
s in the
>> > > > started_list are completed but are not given back to the gadget dr=
iver.
>> > > >
>> > > > Since they remained in the started_list, they will be resubmitted =
again
>> > > > on the next usb_ep_queue. We may send duplicate transfers right?
>> >
>> > Actually, since the requests are completed, the HWO bits are cleared,
>> > nothing is submitted and no duplicate. But since the requests are not
>> > given back yet from the started_list, then the next Start_Transfer
>> > command will begin with the TRB address of the completed request
>> > (HWO=3D0), the controller may not process the next TRBs. Have you test=
ed
>> > this scenario?
>> >
>> > > >
>> > > > You can try to cleanup requests in the started_list, but you need =
to be
>> > > > careful to make sure you're not out of sync with the transfer comp=
letion
>> > > > events and new requests from gadget driver.
>> > > >
>> >
>> > Was the problem you encounter due to no_interrupt settings where the
>> > it was set to the last request of the uvc data pump?
>> >
>> > if that's the case, can UVC function driver make sure to not set
>> > no_interrupt to the last request of the data pump from the UVC?
>>
>> Actually no. What I want to do is to ensure that the dwc3 stream
>> is stopped when the hardware was drained. Which is a valid point
>> in my case. Since we are actually potentially enqueueing new request
>> in the complete handler, be it zero length or real transfers.
>>
>> Calling kick_transfer on an drained hw will absolutely run into
>> missed isocs if the irq thread was called late. We saw this on real hard=
ware,
>> where another irq_thread was scheduled with the same priority as the
>> dwc3 irq_thread but was running so long that the HW was running dry in
>> between the hw irq and the actual dwc3_irq_thread run.
>>
>
>Right. Unfortunately, dwc3 can only "guess" when UVC function stops
>pumping more request or whether it's due to some large latency. The
>logic to workaround this underrun issue will not be foolproof. Perhaps
>we can improve upon it, but the solution is better implement at the UVC
>function driver.

Yes, the best way to solve this is in the uvc driver.

>I thought we have the mechanism in UVC function now to ensure queuing
>enough zero-length requests to account for underrun/latency issue?
>What's the issue now?

This is actually only partially true. Even with the zero-length packages
it is possible that we run into underruns. This is why we implemented
this patch. This has happened because another interrupt thread with the
same prio on the same CPU as this interrupt thread was keeping the CPU
busy. As the dwc3 interrupt thread get to its call, the time was already
over and the hw was already drained, although the started list was not
yet empty, which was causing the next queued requests to be queued to
late. (zero length or not)

Yes, this needed to be solved on the upper level first, by moving the
long running work of the other interrupt thread to another thread or
even into the userspace.

However I thought it would be great if we could somehow find out in
the dwc3 core and make the pump mechanism more robust against such
late enqueues.

This all started with that series.

https://lore.kernel.org/all/20240307-dwc3-gadget-complete-irq-v1-0-4fe9ac0b=
a2b7@pengutronix.de/

And patch 2 of this series did work well so far. The next move was this
patch.

Since the last week debugging we found out that it got other issues.
It is not allways save to read the HWO bit, from the driver.

Turns out that after an new TRB was prepared with the HWO bit set
it is not save to read immideatly back from that value as the hw
will be doing some operations on that exactly new prepared TRB.

We ran into this problem when applying this patch. The trb buffer list
was actually filled but we hit a false positive where the latest HWO bit
was 0 (probably due to the hw action in the background) and therefor
went into end transfer.

Michael

--=20
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

--pLoVn6qsK0dxz8GE
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEElXvEUs6VPX6mDPT8C+njFXoeLGQFAmY2yHgACgkQC+njFXoe
LGRSaxAAxhBtE6gNRWYO1/UABS9K/sdIsmrSKNvVz9fOIYkEbrulPeJkOG/jJal4
rWsVidfcLmgcOIJqn1kQG/K6qxWnojSURVSnhYueHPfIim/lakNyDspMfUwnf3nh
gm3vaoGR7J3bX0wBIIIHAq8v3Js1qoeMHGgq/sw54OLqkIMn1s8nF2AID6PUxU7H
CZtXz3YtcO5uWd4BK6krSrZd8+4jDRmab40iSMfOHYX79/a9hsy85gXuSOUV+wuz
XVgeujs0iQnK8CY1hY3M9lAU6iE/ftUiQOqu70fmqASz6x7v3OYiwxrdc8bnjfKO
5w44jXALoTjiyXDXsHH6/SwjdsT1zcgta0Xf6wKZSEp075c/LJ4w1dlb1Syh4k5H
c7P10eO7xEogcd0uvYhCTDYAM1HlYU5MHt33+5KX88zlpIrbcy0t/mXiWnB8Hfdf
BomDWOUskARk5S9EH7hobUEQSHYpEQT66WoJlhNVsf57GsAlxhoxraHbJ6ByDr9W
fYgAA71tG9agpcFWWwxtQ9RDQ/kg3qp5euA8HbC5TGkAXKXYy6TfsrVpLIlk+tB8
maR/dm2UBWQ9Wrlgn2F9Xb3DXhaszNkHi8SlBgMN/mvNcXg7luxYMg6yuCl+XLTK
sLxDy5gdxqpqdbuIeWo591ZCrRyIjeh6F2aF/QuHIzgRTqFmwAM=
=Uzn2
-----END PGP SIGNATURE-----

--pLoVn6qsK0dxz8GE--