Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Felipe Balbi <balbi@kernel.org>
To:     John Stultz <john.stultz@linaro.org>
Cc:     Josh Gao <jmgao@google.com>, YongQin Liu <yongqin.liu@linaro.org>,
        Anurag Kumar Vulisha <anurag.kumar.vulisha@xilinx.com>,
        Yang Fei <fei.yang@intel.com>,
        Thinh Nguyen <thinhn@synopsys.com>,
        Tejas Joglekar <tejas.joglekar@synopsys.com>,
        Andrzej Pietrasiewicz <andrzej.p@collabora.com>,
        Jack Pham <jackp@codeaurora.org>, Todd Kjos <tkjos@google.com>,
        Greg KH <gregkh@linuxfoundation.org>,
        Linux USB List <linux-usb@vger.kernel.org>,
        lkml <linux-kernel@vger.kernel.org>
Subject: Re: More dwc3 gadget issues with adb
In-Reply-To: <CALAqxLXkY2veX7DKAhXn-uxtMYygfKrJQaPiSKKLbnDvQnHinA@mail.gmail.com>
References: <CALAqxLW2R4d=Zm=TKbFprN-uYrerL1oCYsVC3VedEKtW0gCsyA@mail.gmail.com> <877dyfsv00.fsf@kernel.org> <CALAqxLUdzKRV6nrcLpWsykK+WPnqhUK4iwRe4_Xmo-TvEV5KOg@mail.gmail.com> <87blnkcb6i.fsf@kernel.org> <CALAqxLXkY2veX7DKAhXn-uxtMYygfKrJQaPiSKKLbnDvQnHinA@mail.gmail.com>
Date:   Thu, 23 Apr 2020 12:22:32 +0300
Message-ID: <87k1261rqf.fsf@kernel.org>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
        micalg=pgp-sha256; protocol="application/pgp-signature"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

--=-=-=
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable


Hi,

John Stultz <john.stultz@linaro.org> writes:
> On Tue, Apr 21, 2020 at 11:00 PM Felipe Balbi <balbi@kernel.org> wrote:
>> John Stultz <john.stultz@linaro.org> writes:
>> > On Thu, Apr 16, 2020 at 1:19 AM Felipe Balbi <balbi@kernel.org> wrote:
>> >> One thing I noticed is that we're missing a giveback on ep1out. Here'=
s a
>> >> working case:
>> >>
>> >
>> > Hey Felipe,
>> >   So I found some time to dig around on this today and I started
>> > trying to understand this issue that you've pointed out about missing
>> > the giveback.
>> >
>> > It seems part of the issue is we get to a point where we have some req
>> > where pending_sgs is more than one.
>> >
>> > We call dwc3_prepare_one_trb_sg() on it:
>> >   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/t=
ree/drivers/usb/dwc3/gadget.c?h=3Dv5.7-rc2#n1068
>> >
>> > And we process the sg list incrementing req->num_queued_sgs for each o=
ne.
>> >
>> > then later, dwc3_gadget_ep_cleanup_completed_request() is called on th=
e request:
>> >   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/t=
ree/drivers/usb/dwc3/gadget.c?h=3Dv5.7-rc2#n2522
>> >
>> > We call dwc3_gadget_ep_reclaim_trb_sg()
>> >   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/t=
ree/drivers/usb/dwc3/gadget.c?h=3Dv5.7-rc2#n2470
>> >
>> > Where we iterate over the req->sg, ideally decrementing
>> > num_pending_sgs each time and return.
>> >
>> > But back in dwc3_gadget_ep_cleanup_completed_request()  and there
>> > we're hitting the:
>> >   if (!dwc3_gadget_ep_request_completed(req) ||
>> >       req->num_pending_sgs) {
>> > case which causes us to skip the call to dwc3_gadget_giveback().
>>
>> This logic was modified recently. Can you check if today's linus/master
>> works for you?
>
> I was testing this against 5.7-rc2, but I updated to linus/master and
> I'm not seeing any change.

good

>> > Looking as to why the num_pending_sgs is non zero, that's because in
>> > dwc3_gadget_ep_reclaim_trb_sg we're hitting the case where the trb has
>> > the DWC3_TRB_CTRL_HWO ctrl flag set, which breaks us out of the loop
>> > early before we decrement num_pending_sgs.
>> >
>> > For that trb, we're setting the HWO flag in __dwc3_prepare_one_trb()
>> > (called from dwc3_prepare_one_trb_sg() back at the beginning):
>> >   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/t=
ree/drivers/usb/dwc3/gadget.c?h=3Dv5.7-rc2#n921
>> >
>> > I added logic showing every time we set or clear that flag, and it
>> > seems like we're always setting it but never clearing it. And often
>>
>> HW clears HWO flag. We only have to manually clear in one or two cases.
>
> I guess the bit that worries me with the current code is in
> dwc3_gadget_ep_reclaim_completed_trb():
>   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree=
/drivers/usb/dwc3/gadget.c?h=3Dv5.7-rc2#n2406
>
> There is the logic:
>   if (chain && (trb->ctrl & DWC3_TRB_CTRL_HWO))
>            trb->ctrl &=3D ~DWC3_TRB_CTRL_HWO;
>
> But that will never trip, as the only time we call
> dwc3_gadget_ep_reclaim_completed_trb() with chain=3D=3Dtrue is from
> dwc3_gadget_ep_reclaim_trb_sg():
>   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree=
/drivers/usb/dwc3/gadget.c?h=3Dv5.7-rc2#n2470
>
> Where if the trb->ctrl has the HWO flag set, we break out before
> calling  dwc3_gadget_ep_reclaim_completed_trb().
>
> So the logic quoted above seems to be effectively dead code (despite
> the big comment as to why we need it)

There is another situation when we use CHN bit. If we're starting an OUT
transfer and the length is not aligned to wMaxPacketSize.

> That's why I suspected the HWO check in
> dwc3_gadget_ep_reclaim_trb_sg() is problematic.

okay

>> > that's not an issue as we only have one sg entry. But if its set on a
>> > trb in a request with multiple sgs, that's where it seems to be
>> > causing the issue.
>>
>> The issue is completing with HWO set, which should never happen. Can you
>> collect tracepoints with linus/master of this particular situation?
>
> Will do, though again, its a little tough as often we hit the stall
> state pretty quickly at bootup, before I can even try to enable
> tracing, so it may take a few tries.

IIRC there's a way to configure tracepoints from cmdline, I never tested
it though.

>> >> One interesting thing is that TRB addresses are "odd". I can't find a
>> >> proper lifetime for these TRBs. Do you have IOMMU enabled? Can you run
>> >> without it? For example, nowhere in the log can I find the place where
>> >> trb 0000000092deef41 was first enqueue. I'm assuming the log to be
>> >> ordered, which means that trb is the same as 00000000f3db4076. But why
>> >> are the addresses different?
>> >>
>> >> Another weird thing is that even though we CHN bit being set in
>> >> 0000000092deef41, we don't see where the second trb (the one its chai=
ned
>> >> to) was prepared. It seems like it was *never* prepared, what gives?
>> >
>> > I suspect these bits were due to the tracing happening after some
>> > minor amount of initial adb traffic began at bootup? So the trace
>> > isn't capturing all the events.
>>
>> No, no. That's more likely to be IOMMU mucking up the addresses. ADB is
>> very sequential in its behavior and USB transfers requests in
>> order. Please run linus/master without IOMMU.
>
> So I didn't have any IOMMU support enabled in the config I was testing
> with. I went through and added IOMMU options in my config with
> linus/master as well and that didn't seem to change the behavior
> either.

pretty odd that the TRB addresses change like that.

> I'll get back to you with further trace logs.

thanks

=2D-=20
balbi

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEElLzh7wn96CXwjh2IzL64meEamQYFAl6hXlsACgkQzL64meEa
mQZk3xAAv30RrB564Z2Wt3UlrD5nQVIKXRIkgaINWSlDMiESJH4Oi8j1pxmVO4A2
46UDXELuwxLWr5S8glicqGJ2evU3M3zRVFsByGguFsLGgZiD2RT6vuSwZJ17QBvK
yEu1J6SttHHIuFGv7UdfH2Q6Z+Rqa4c+HBrFCZL5OcYK2688ks/TB/DFyXLo0+cL
MwIZk/7vg51wrJZr7cEJsMPMXb2hOt/MgvW0e+KtstMF98zPotGLZHZxZL5wQEXQ
xPlW/XlLrr6I5A8PXM0i5zJ331UW3lH/ztCOZkfk7T4avk1ulWzEgHy2HR/1Ka/J
eJosIh9EQMi5ewsRw3d0RH3/cxS70mTlk/GrlhvVMCzVzZAe3pLwgIKR4gB0MQ0Y
OCejw/pSNcz2fzH5YmujwPEzdlQ3CX2343C1fVspwrsWAzAGdiiNBjDKb/5vksgL
qOYAeUAdohIJBXaPK8xpEfzdaKBPdRbr1OEDNGfArLr/6797otuUOuWKamPK4MLg
MlKUXXVYDazJ8FCSdSww3BxxMUZjgbZK7v+qRczbxJ1leC26b977kvVDxXWsaIKI
N427yeZ2cY+eWKqRrQ8aItNhZv/Ezu6qeFczY3PAkSd9qWTxM0DE4UqTpSNdT6V4
48WRrS7M3BmOmMNrA77NsTf3oHz790JulLMdq8qxX0NmFe9YLh4=
=HbNo
-----END PGP SIGNATURE-----
--=-=-=--