Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Tue, 3 Nov 2020 17:26:46 -0800
From:   Jakub Kicinski <kuba@kernel.org>
To:     Alexander Lobakin <alobakin@pm.me>
Cc:     "David S. Miller" <davem@davemloft.net>,
        Jay Vosburgh <j.vosburgh@gmail.com>,
        Veaceslav Falico <vfalico@gmail.com>,
        Andy Gospodarek <andy@greyhouse.net>,
        Jiri Pirko <jiri@resnulli.us>,
        Steffen Klassert <steffen.klassert@secunet.com>,
        Willem de Bruijn <willemb@google.com>,
        Miaohe Lin <linmiaohe@huawei.com>,
        Antoine Tenart <atenart@kernel.org>,
        Mauro Carvalho Chehab <mchehab+huawei@kernel.org>,
        netdev@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2 net-next 0/2] net: allow virtual netdevs to forward
 UDP L4 and fraglist GSO skbs
Message-ID: <20201103172646.30dc9502@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>
In-Reply-To: <NysZRGMkuWq0KPTCJ1Dz2FTjRkeJXDH3edVrsEeJkQI@cp4-web-036.plabs.ch>
References: <NysZRGMkuWq0KPTCJ1Dz2FTjRkeJXDH3edVrsEeJkQI@cp4-web-036.plabs.ch>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Precedence: bulk

On Sun, 01 Nov 2020 13:16:32 +0000 Alexander Lobakin wrote:
> NETIF_F_GSO_UDP_L4 and NETIF_F_GSO_FRAGLIST allow drivers to offload
> GSO UDP L4. This works well on simple setups, but when any logical
> netdev (e.g. VLAN) is present, kernel stack always performs software
> resegmentation which actually kills the performance.
> 
> The full path in such cases is like:
> 1. Our NIC driver advertises a support for fraglists, GSO UDP L4, GSO
>    fraglists.
> 2. User enables fraglisted GRO via Ethtool.
> 3. GRO subsystem receives UDP frames from driver and merges the packets
>    into fraglisted GSO skb(s).
> 4. Networking stack queues it up for xmitting.
> 5. Virtual device like VLAN doesn't advertise a support for GSO UDP L4
>    and GSO fraglists, so skb_gso_check() doesn't allow to pass this skb
>    as is to the real driver.
> 6. Kernel then has to form a bunch of regular UDP skbs from that one and
>    pass it to the driver instead. This fallback is *extremely* slow for
>    any GSO types, but especially for GSO fraglists.
> 7. All further processing performs with a series of plain UDP skbs, and
>    the driver gets it one-by-one, despite that it supports UDP L4 and
>    fraglisted GSO.
> 
> That's not OK because:
> a) logical/virtual netdevs like VLANs, bridges etc. should pass GSO skbs
>    as is;
> b) even if the final driver doesn't support such type of GSO, this
>    software resegmenting should be performed right before it, not in the
>    middle of processing -- I think I even saw that note somewhere in
>    kernel documentation, and it's totally reasonable in terms of
>    performance.
> 
> Despite the fact that no mainline drivers currently supports fraglist
> GSO, this should and can be easily fixed by adding UDP L4 and fraglist
> GSO to the list of GSO types that can be passed-through the logical
> interfaces (NETIF_F_GSO_SOFTWARE). After this change, no resegmentation
> occurs (if a particular driver supports and advertises this), and the
> performance goes on par with e.g. 1:1 forwarding.
> The only logical netdevs that seem to be unaffected to this are bridge
> interfaces, as their code uses full NETIF_F_GSO_MASK.
> 
> Tested on MIPS32 R2 router board with a WIP NIC driver in VLAN NAT:
> 20 Mbps baseline, 1 Gbps / link speed with this patch.
> 
> Since v1 [1]:
>  - handle bonding and team drivers as suggested by Willem de Bruijn;
>  - reword and expand the introduction with the particular example. 
> 
> [1] https://lore.kernel.org/netdev/Mx3BWGop6fGORN6Cpo4mHIHz2b1bb0eLxeMG8vsijnk@cp3-web-020.plabs.ch


Applied, thanks!