Received-SPF: pass (google.com: domain of linux-kernel+bounces-154964-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Precedence: bulk
MIME-Version: 1.0
References: <20240423094117.93206-1-nbd@nbd.name> <CANn89i+6xRe4V6aDmD-9EM0uD7A87f6rzg3S7Xq6-NaB_Mb4nw@mail.gmail.com>
 <63abfa26-d990-46c3-8982-3eaf7b8f8ee5@nbd.name>
In-Reply-To: <63abfa26-d990-46c3-8982-3eaf7b8f8ee5@nbd.name>
From: Eric Dumazet <edumazet@google.com>
Date: Tue, 23 Apr 2024 13:17:40 +0200
Message-ID: <CANn89iJZvoKVB+AK1_44gki2pHyigyMLXFkyevSQpH3iDbnCvw@mail.gmail.com>
Subject: Re: [RFC] net: add TCP fraglist GRO support
To: Felix Fietkau <nbd@nbd.name>
Cc: netdev@vger.kernel.org, "David S. Miller" <davem@davemloft.net>, 
	Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>, David Ahern <dsahern@kernel.org>, 
	linux-kernel@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Tue, Apr 23, 2024 at 12:25=E2=80=AFPM Felix Fietkau <nbd@nbd.name> wrote=
:
>
> On 23.04.24 12:15, Eric Dumazet wrote:
> > On Tue, Apr 23, 2024 at 11:41=E2=80=AFAM Felix Fietkau <nbd@nbd.name> w=
rote:
> >>
> >> When forwarding TCP after GRO, software segmentation is very expensive=
,
> >> especially when the checksum needs to be recalculated.
> >> One case where that's currently unavoidable is when routing packets ov=
er
> >> PPPoE. Performance improves significantly when using fraglist GRO
> >> implemented in the same way as for UDP.
> >>
> >> Here's a measurement of running 2 TCP streams through a MediaTek MT762=
2
> >> device (2-core Cortex-A53), which runs NAT with flow offload enabled f=
rom
> >> one ethernet port to PPPoE on another ethernet port + cake qdisc set t=
o
> >> 1Gbps.
> >>
> >> rx-gro-list off: 630 Mbit/s, CPU 35% idle
> >> rx-gro-list on:  770 Mbit/s, CPU 40% idle
> >
> > Hi Felix
> >
> > changelog is a bit terse, and patch complex.
> >
> > Could you elaborate why this issue
> > seems to be related to a specific driver ?
> >
> > I think we should push hard to not use frag_list in drivers :/
> >
> > And GRO itself could avoid building frag_list skbs
> > in hosts where forwarding is enabled.
> >
> > (Note that we also can increase MAX_SKB_FRAGS to 45 these days)
>
> The issue is not related to a specific driver at all. Here's how traffic
> flows: TCP packets are received on the SoC ethernet driver, the network
> stack performs regular GRO. The packet gets forwarded by flow offloading
> until it reaches the PPPoE device. PPPoE does not support GSO packets,
> so the packets need to be segmented again.
> This is *very* expensive, since data needs to be copied and checksummed.

gso segmentation does not copy the payload, unless the device has no
SG capability.

I guess something should be done about that, regardless of your GRO work,
since most ethernet devices support SG these days.

Some drivers use header split for RX, so forwarding to  PPPoE
would require a linearization anyway, if SG is not properly handled.

>
> So in my patch, I changed the code to build fraglist GRO instead of
> regular GRO packets, whenever there is no local socket to receive the
> packets. This makes segmenting very cheap, since the original skbs are
> preserved on the trip through the stack. The only cost is an extra
> socket lookup whenever NETIF_F_FRAGLIST_GRO is enabled.

A socket lookup in multi-net-namespace world is not going to work generical=
ly,
but I get the idea now.