Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <CAA85sZukiFq4A+b9+en_G85eVDNXMQsnGc4o-4NZ9SfWKqaULA@mail.gmail.com>
 <CAA85sZvm1dL3oGO85k4R+TaqBiJsggUTpZmGpH1+dqdC+U_s1w@mail.gmail.com>
 <e7e49ed5-09e2-da48-002d-c7eccc9f9451@intel.com> <CAA85sZtyM+X_oHcpOBNSgF=kmB6k32bpB8FCJN5cVE14YCba+A@mail.gmail.com>
 <22aad588-47d6-6441-45b2-0e685ed84c8d@intel.com> <CAA85sZti1=ET=Tc3MoqCX0FqthHLf6MSxGNAhJUNiMms1TfoKA@mail.gmail.com>
 <CAA85sZvn04k7=oiTQ=4_C8x7pNEXRWzeEStcaXvi3v63ah7OUQ@mail.gmail.com>
 <ffb554bfa4739381d928406ad24697a4dbbbe4a2.camel@redhat.com>
 <CAA85sZunA=tf0FgLH=MNVYq3Edewb1j58oBAoXE1Tyuy3GJObg@mail.gmail.com>
 <CAA85sZsH1tMwLtL=VDa5=GBdVNWgifvhK+eG-hQg69PeSxBWkg@mail.gmail.com>
 <CAA85sZu=CzJx9QD87-vehOStzO9qHUSWk6DXZg3TzJeqOV5-aw@mail.gmail.com> <0a040331995c072c56fce58794848f5e9853c44f.camel@redhat.com>
In-Reply-To: <0a040331995c072c56fce58794848f5e9853c44f.camel@redhat.com>
From:   Ian Kumlien <ian.kumlien@gmail.com>
Date:   Tue, 27 Jun 2023 14:31:13 +0200
Message-ID: <CAA85sZuuwxtAQcMe3LHpFVeF7y-bVoHtO1nukAa2+NyJw3zcyg@mail.gmail.com>
Subject: Re: [Intel-wired-lan] bug with rx-udp-gro-forwarding offloading?
To:     Paolo Abeni <pabeni@redhat.com>
Cc:     Alexander Lobakin <aleksander.lobakin@intel.com>,
        intel-wired-lan <intel-wired-lan@lists.osuosl.org>,
        Jakub Kicinski <kuba@kernel.org>,
        Eric Dumazet <edumazet@google.com>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Tue, Jun 27, 2023 at 11:19=E2=80=AFAM Paolo Abeni <pabeni@redhat.com> wr=
ote:
>
> On Mon, 2023-06-26 at 20:59 +0200, Ian Kumlien wrote:
> > On Mon, Jun 26, 2023 at 8:20=E2=80=AFPM Ian Kumlien <ian.kumlien@gmail.=
com> wrote:
> > >
> > > Nevermind, I think I found it, I will loop this thing until I have a
> > > proper trace....
> >
> > Still some question marks, but much better
>
> Thanks!
> >
> > cat bug.txt | ./scripts/decode_stacktrace.sh vmlinux
> > [   62.624003] BUG: kernel NULL pointer dereference, address: 000000000=
00000c0
> > [   62.631083] #PF: supervisor read access in kernel mode
> > [   62.636312] #PF: error_code(0x0000) - not-present page
> > [   62.641541] PGD 0 P4D 0
> > [   62.644174] Oops: 0000 [#1] PREEMPT SMP NOPTI
> > [   62.648629] CPU: 1 PID: 913 Comm: napi/eno2-79 Not tainted 6.4.0 #36=
4
> > [   62.655162] Hardware name: Supermicro Super Server/A2SDi-12C-HLN4F,
> > BIOS 1.7a 10/13/2022
> > [   62.663344] RIP: 0010:__udp_gso_segment
> > (./include/linux/skbuff.h:2858 ./include/linux/udp.h:23
> > net/ipv4/udp_offload.c:228 net/ipv4/udp_offload.c:261
> > net/ipv4/udp_offload.c:277)
>
> So it's faulting here:
>
> static struct sk_buff *__udpv4_gso_segment_list_csum(struct sk_buff *segs=
)
> {
>         struct sk_buff *seg;
>         struct udphdr *uh, *uh2;
>         struct iphdr *iph, *iph2;
>
>         seg =3D segs;
>         uh =3D udp_hdr(seg);
>         iph =3D ip_hdr(seg);
>
>         if ((udp_hdr(seg)->dest =3D=3D udp_hdr(seg->next)->dest) &&
>         // ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> The GSO segment has been assembled by skb_gro_receive_list()
> I guess seg->next is NULL, which is somewhat unexpected as
> napi_gro_complete() clears the gso_size when sending up the stack a
> single frame.
>
> On the flip side, AFAICS, nothing prevents the stack from changing the
> aggregated packet layout (e.g. pulling data and/or linearizing the
> skb).
>
> In any case this looks more related to rx-gro-list then rx-udp-gro-
> forwarding. I understand you have both feature enabled in your env?
>
> Side questions: do you have any non trivial nf/br filter rule?
>
> The following could possibly validate the above and avoid the issue,
> but it's a bit papering over it. Could you please try it in your env?

Will do as soon as i get home =3D)

> Thanks!
>
> Paolo
> ---
> diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> index 6c5915efbc17..75531686bfdf 100644
> --- a/net/core/skbuff.c
> +++ b/net/core/skbuff.c
> @@ -4319,6 +4319,9 @@ struct sk_buff *skb_segment_list(struct sk_buff *sk=
b,
>
>         skb->prev =3D tail;
>
> +       if (WARN_ON_ONCE(!skb->next))
> +               goto err_linearize;
> +
>         if (skb_needs_linearize(skb, features) &&
>             __skb_linearize(skb))
>                 goto err_linearize;
>