Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <a766b45e26b465224fa5f0be721af147a8599fa7.camel@redhat.com>
Subject: Re: [Intel-wired-lan] bug with rx-udp-gro-forwarding offloading?
From:   Paolo Abeni <pabeni@redhat.com>
To:     Ian Kumlien <ian.kumlien@gmail.com>,
        Alexander Lobakin <aleksander.lobakin@intel.com>
Cc:     intel-wired-lan <intel-wired-lan@lists.osuosl.org>,
        Jakub Kicinski <kuba@kernel.org>,
        Eric Dumazet <edumazet@google.com>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Date:   Mon, 26 Jun 2023 18:42:04 +0200
In-Reply-To: <CAA85sZtyM+X_oHcpOBNSgF=kmB6k32bpB8FCJN5cVE14YCba+A@mail.gmail.com>
References: <CAA85sZukiFq4A+b9+en_G85eVDNXMQsnGc4o-4NZ9SfWKqaULA@mail.gmail.com>
         <CAA85sZvm1dL3oGO85k4R+TaqBiJsggUTpZmGpH1+dqdC+U_s1w@mail.gmail.com>
         <e7e49ed5-09e2-da48-002d-c7eccc9f9451@intel.com>
         <CAA85sZtyM+X_oHcpOBNSgF=kmB6k32bpB8FCJN5cVE14YCba+A@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.46.4 (3.46.4-1.fc37) 
MIME-Version: 1.0
Precedence: bulk

On Mon, 2023-06-26 at 16:25 +0200, Ian Kumlien wrote:
> On Mon, Jun 26, 2023 at 4:18=E2=80=AFPM Alexander Lobakin
> <aleksander.lobakin@intel.com> wrote:
> >=20
> > From: Ian Kumlien <ian.kumlien@gmail.com>
> > Date: Sun, 25 Jun 2023 12:59:54 +0200
> >=20
> > > It could actually be that it's related to: rx-gro-list but
> > > rx-udp-gro-forwarding makes it trigger quicker...  I have yet to
> > > trigger it on igb
> >=20
> > Hi, the rx-udp-gro-forwarding author here.
> >=20
> > (good thing this appeared on IWL, which I read time to time, but please
> >  Cc netdev next time)
> > (thus +Cc Jakub, Eric, and netdev)
>=20
> Well, two things, it seems like rx-udp-gro-forwarding accelerates it
> but the issue is actually in: rx-gro-list
>=20
> And since i've only been able to trigger it in ixgbe i thought it
> might be a driver issue =3D)
>=20
> > > On Sat, Jun 24, 2023 at 10:03=E2=80=AFPM Ian Kumlien <ian.kumlien@gma=
il.com> wrote:
> > > >=20
> > > > Hi again,
> > > >=20
> > > > I suspect that I have rounded this down to the rx-udp-gro-forwardin=
g
> > > > option... I know it's not on by default but....
> > > >=20
> > > > So, I have a machine with four nics, all using ixgbe, they are all:
> > > > 06:00.0 Ethernet controller: Intel Corporation Ethernet Connection
> > > > X553 1GbE (rev 11)
> > > > 06:00.1 Ethernet controller: Intel Corporation Ethernet Connection
> > > > X553 1GbE (rev 11)
> > > > 07:00.0 Ethernet controller: Intel Corporation Ethernet Connection
> > > > X553 1GbE (rev 11)
> > > > 07:00.1 Ethernet controller: Intel Corporation Ethernet Connection
> > > > X553 1GbE (rev 11)
> > > >=20
> > > > But I have been playing with various... currently i do:
> > > > for interface in eno1 eno2 eno3 eno4 ; do
> > > >   for offload in ntuple hw-tc-offload rx-gro-list ; do
> > > >     ethtool -K $interface $offload on > /dev/null
> > > >   done
> > > >   ethtool -G $interface rx 8192 tx 8192 > /devYnull
> > > > done
> > > >=20
> > > > And it all seems to work just fine for my little firewall
> > > >=20
> > > > However, enabling rx-udp-gro-forwarding results in the attached ooo=
ops
> > > > (sorry, can't see more, been recreating by watching shows on HBO
> > > > max... )
> >=20
> > Where's the mentioned oops? Where's the original message?
>=20
> Held by the mailing list since i can only get a screenshot of it...
> Will attach the latest one to this email

That image is not very useful/does not provide a lot of relevant
information. Could you please use kdump/crash to collect a (decoded)
full stack trace?

> (I wish that i could easily get a larger backtrace but i haven't
> looked in further atm)
>=20
> > Can't this be related to [0]?
>=20
> Don't know, my main test has been running video streams in the
> background - eventually they cause a oops (within 40 minutes or so)
> But i doubt it's counted as tunnel data ;)

I read the above as you don't have UDP tunnels in your setup. Am I
correct?

Thanks,

Paolo