Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <CAA85sZukiFq4A+b9+en_G85eVDNXMQsnGc4o-4NZ9SfWKqaULA@mail.gmail.com>
 <CAA85sZvm1dL3oGO85k4R+TaqBiJsggUTpZmGpH1+dqdC+U_s1w@mail.gmail.com>
 <e7e49ed5-09e2-da48-002d-c7eccc9f9451@intel.com> <CAA85sZtyM+X_oHcpOBNSgF=kmB6k32bpB8FCJN5cVE14YCba+A@mail.gmail.com>
 <22aad588-47d6-6441-45b2-0e685ed84c8d@intel.com>
In-Reply-To: <22aad588-47d6-6441-45b2-0e685ed84c8d@intel.com>
From:   Ian Kumlien <ian.kumlien@gmail.com>
Date:   Mon, 26 Jun 2023 19:24:32 +0200
Message-ID: <CAA85sZti1=ET=Tc3MoqCX0FqthHLf6MSxGNAhJUNiMms1TfoKA@mail.gmail.com>
Subject: Re: [Intel-wired-lan] bug with rx-udp-gro-forwarding offloading?
To:     Alexander Lobakin <aleksander.lobakin@intel.com>
Cc:     intel-wired-lan <intel-wired-lan@lists.osuosl.org>,
        Jakub Kicinski <kuba@kernel.org>,
        Eric Dumazet <edumazet@google.com>,
        Paolo Abeni <pabeni@redhat.com>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Mon, Jun 26, 2023 at 7:15=E2=80=AFPM Alexander Lobakin
<aleksander.lobakin@intel.com> wrote:
>
> From: Ian Kumlien <ian.kumlien@gmail.com>
> Date: Mon, 26 Jun 2023 16:25:24 +0200
>
> > On Mon, Jun 26, 2023 at 4:18=E2=80=AFPM Alexander Lobakin
> > <aleksander.lobakin@intel.com> wrote:
> >>
> >> From: Ian Kumlien <ian.kumlien@gmail.com>
> >> Date: Sun, 25 Jun 2023 12:59:54 +0200
> >>
> >>> It could actually be that it's related to: rx-gro-list but
> >>> rx-udp-gro-forwarding makes it trigger quicker...  I have yet to
> >>> trigger it on igb
> >>
> >> Hi, the rx-udp-gro-forwarding author here.
> >>
> >> (good thing this appeared on IWL, which I read time to time, but pleas=
e
> >>  Cc netdev next time)
> >> (thus +Cc Jakub, Eric, and netdev)
> >
> > Well, two things, it seems like rx-udp-gro-forwarding accelerates it
> > but the issue is actually in: rx-gro-list
>
> Do you enable them simultaneously? I remember, when I was adding
> gro-fwd, it was working (and working good) as follows:
>
> 1. gro-fwd on, gro-list off: gro-fwd
> 2. gro-fwd off, gro-list on: gro-list
> 3. gro-fwd on, gro-list on: gro-list
>
> Note that their receive paths are independent[0]: skb_gro_receive_list()
> vs skb_gro_receive(), thus I'm still not really sure how gro-fwd can
> trigger gro-list's bug.

Neither am I... I have enabled sol via ipmitool now, will try to get a
better capture

> > And since i've only been able to trigger it in ixgbe i thought it
> > might be a driver issue =3D)
>
> Your screenshot says "__udp_gso_segment", which means that the
> problematic UDP GRO packet hits the Tx path. Rx is in general
> driver-independent. Tx has separate netdev feature ("tx-gso-list"), but
> it's not supported by any driver, just software stack. It might be that
> your traffic goes through a bridge or tunnel or anything else that
> triggers GSO and software segmentation then booms for some reason.
> BTW, __udp_gso_segment() is one-liner when the passed skb was
> gro-listed[1], so having it in the bug splat could mean the skb didn't
> take that route. But hard to say with no full stacktrace.

I do have a UDP tunnel, in wireguard, will disable it.

Beyond that some bridges and veth interfaces, but lets wait for a full trac=
e

> [...]
>
> >>>> But correlating that with the source is beyond me, it could be gener=
ic
> >>>> but i thought i'd send it you first since it's part of the redhat
> >>>> guide to speeding up udp traffic
> >> [0]
> >> https://lore.kernel.org/netdev/f83d79d6-f8d7-a229-941a-7d7427975160@nv=
idia.com
> >>
> >> Thanks,
> >> Olek
>
> [0]
> https://elixir.bootlin.com/linux/latest/source/net/ipv4/udp_offload.c#L51=
8
> [1]
> https://elixir.bootlin.com/linux/latest/source/net/ipv4/udp_offload.c#L27=
7
>
> Thanks,
> Olek