Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <CAM=1FV3ODgP1+iST6zVh4EFY9WLf=Us8PTTmbH=8KF1Xc7zmvA@mail.gmail.com>
In-Reply-To: <CAM=1FV3ODgP1+iST6zVh4EFY9WLf=Us8PTTmbH=8KF1Xc7zmvA@mail.gmail.com>
From:   Tomas Hruby <tomas@tigera.io>
Date:   Wed, 19 Oct 2022 12:10:30 -0700
Message-ID: <CAM=1FV1Nt9a5-d7LneS=-o0S2=FnDeCxLPXrB==XYJvjJ8v=+A@mail.gmail.com>
Subject: Re: kernel BUG at net/core/skbuff.c:4219
To:     eric.dumazet@gmail.com
Cc:     edumazet@google.com, herbert@gondor.apana.org.au,
        jpiotrowski@linux.microsoft.com, linux-kernel@vger.kernel.org,
        netdev@vger.kernel.org, seh@panix.com
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

> > Hi,
> >
> > One of our Flatcar users has been hitting the kernel BUG in the subject line
> > for the past year (https://github.com/flatcar/Flatcar/issues/378). This was
> > first reported when on 5.10.25, but has been happening across kernel updates,
> > most recently with 5.15.63. The nodes where this happens are AWS EC2 instances,
> > using ENA and calico networking in eBPF mode with VXLAN encapsulation. When
> > GRO/GSO is enabled, the host hits this bug and prints the following stacktrace:
>
>
> I suspect eBPF code lowers gso_size ?
>
> gso stack is not able to arbitrarily segment a GRO packet after gso_size
> being changed.

Based on the stack trace, it happens for a tcp packet. Since it seems
like it is on egress from a host, I suspect that it is encapsulating a
tcp packet into a vxlan tunnel and useds bpf_skb_adjust_room() here
https://github.com/projectcalico/calico/blob/master/felix/bpf-gpl/nat.h#L77-L80
Not sure if it should use BPF_F_ADJ_ROOM_FIXED_GSO in that case.

Calico uses the flag when decapsulating packets here
https://github.com/projectcalico/calico/blob/master/felix/bpf-gpl/nat.h#L143
and it uses the flag as the packet is UDP.

Any of that could cause the BUG?


Tomas