Commit 9fd1ff5d2ac7 ("udp: Support UDP fraglist GRO/GSO.") added a support
for fraglist UDP L4 and fraglist GSO not only for local traffic, but also
for forwarding. This works well on simple setups, but when any logical
netdev (e.g. VLAN) is present, kernel stack always performs software
resegmentation which actually kills the performance.
Despite the fact that no mainline drivers currently supports fraglist GSO,
this should and can be easily fixed by adding UDP L4 and fraglist GSO to
the list of GSO types that can be passed-through the logical interfaces
(NETIF_F_GSO_SOFTWARE). After this change, no resegmentation occurs (if
a particular driver supports and advertises this), and the performance
goes on par with e.g. 1:1 forwarding.
The only logical netdevs that seem to be unaffected to this are bridge
interfaces, as their code uses full NETIF_F_GSO_MASK.
Tested on MIPS32 R2 router board with a WIP NIC driver in VLAN NAT:
20 Mbps baseline, 1 Gbps / link speed with this patch.
Signed-off-by: Alexander Lobakin <[email protected]>
---
include/linux/netdev_features.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/include/linux/netdev_features.h b/include/linux/netdev_features.h
index 0b17c4322b09..934de56644e7 100644
--- a/include/linux/netdev_features.h
+++ b/include/linux/netdev_features.h
@@ -207,8 +207,8 @@ static inline int find_next_netdev_feature(u64 feature, unsigned long start)
NETIF_F_FSO)
/* List of features with software fallbacks. */
-#define NETIF_F_GSO_SOFTWARE (NETIF_F_ALL_TSO | \
- NETIF_F_GSO_SCTP)
+#define NETIF_F_GSO_SOFTWARE (NETIF_F_ALL_TSO | NETIF_F_GSO_SCTP | \
+ NETIF_F_GSO_UDP_L4 | NETIF_F_GSO_FRAGLIST)
/*
* If one device supports one of these features, then enable them
--
2.29.2
On Fri, Oct 30, 2020 at 2:33 PM Alexander Lobakin <[email protected]> wrote:
>
> Commit 9fd1ff5d2ac7 ("udp: Support UDP fraglist GRO/GSO.") added a support
> for fraglist UDP L4 and fraglist GSO not only for local traffic, but also
> for forwarding. This works well on simple setups, but when any logical
> netdev (e.g. VLAN) is present, kernel stack always performs software
> resegmentation which actually kills the performance.
> Despite the fact that no mainline drivers currently supports fraglist GSO,
> this should and can be easily fixed by adding UDP L4 and fraglist GSO to
> the list of GSO types that can be passed-through the logical interfaces
> (NETIF_F_GSO_SOFTWARE). After this change, no resegmentation occurs (if
> a particular driver supports and advertises this), and the performance
> goes on par with e.g. 1:1 forwarding.
> The only logical netdevs that seem to be unaffected to this are bridge
> interfaces, as their code uses full NETIF_F_GSO_MASK.
>
> Tested on MIPS32 R2 router board with a WIP NIC driver in VLAN NAT:
> 20 Mbps baseline, 1 Gbps / link speed with this patch.
>
> Signed-off-by: Alexander Lobakin <[email protected]>
> ---
> include/linux/netdev_features.h | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/include/linux/netdev_features.h b/include/linux/netdev_features.h
> index 0b17c4322b09..934de56644e7 100644
> --- a/include/linux/netdev_features.h
> +++ b/include/linux/netdev_features.h
> @@ -207,8 +207,8 @@ static inline int find_next_netdev_feature(u64 feature, unsigned long start)
> NETIF_F_FSO)
>
> /* List of features with software fallbacks. */
> -#define NETIF_F_GSO_SOFTWARE (NETIF_F_ALL_TSO | \
> - NETIF_F_GSO_SCTP)
> +#define NETIF_F_GSO_SOFTWARE (NETIF_F_ALL_TSO | NETIF_F_GSO_SCTP | \
> + NETIF_F_GSO_UDP_L4 | NETIF_F_GSO_FRAGLIST)
What exactly do you mean by *re*segmenting?
I think it is fine to reenable this again, now that UDP sockets will
segment unexpected UDP GSO packets that may have looped. We previously
added general software support in commit 83aa025f535f ("udp: add gso
support to virtual devices"). Then reduced its scope to egress only in
8eea1ca82be9 ("gso: limit udp gso to egress-only virtual devices") to
handle that edge case.
If we can enable for all virtual devices again, we could revert those
device specific options.
On Saturday, 31 October 2020, 2:12, Willem de Bruijn <[email protected]> wrote:
Hi Willem,
> On Fri, Oct 30, 2020 at 2:33 PM Alexander Lobakin [email protected] wrote:
>
> > Commit 9fd1ff5d2ac7 ("udp: Support UDP fraglist GRO/GSO.") added a support
> > for fraglist UDP L4 and fraglist GSO not only for local traffic, but also
> > for forwarding. This works well on simple setups, but when any logical
> > netdev (e.g. VLAN) is present, kernel stack always performs software
> > resegmentation which actually kills the performance.
> > Despite the fact that no mainline drivers currently supports fraglist GSO,
> > this should and can be easily fixed by adding UDP L4 and fraglist GSO to
> > the list of GSO types that can be passed-through the logical interfaces
> > (NETIF_F_GSO_SOFTWARE). After this change, no resegmentation occurs (if
> > a particular driver supports and advertises this), and the performance
> > goes on par with e.g. 1:1 forwarding.
> > The only logical netdevs that seem to be unaffected to this are bridge
> > interfaces, as their code uses full NETIF_F_GSO_MASK.
> > Tested on MIPS32 R2 router board with a WIP NIC driver in VLAN NAT:
> > 20 Mbps baseline, 1 Gbps / link speed with this patch.
> >
> > Signed-off-by: Alexander Lobakin [email protected]
> >
> > ------------------------------------------------
> >
> > include/linux/netdev_features.h | 4 ++--
> > 1 file changed, 2 insertions(+), 2 deletions(-)
> > diff --git a/include/linux/netdev_features.h b/include/linux/netdev_features.h
> > index 0b17c4322b09..934de56644e7 100644
> > --- a/include/linux/netdev_features.h
> > +++ b/include/linux/netdev_features.h
> > @@ -207,8 +207,8 @@ static inline int find_next_netdev_feature(u64 feature, unsigned long start)
> > NETIF_F_FSO)
> > /* List of features with software fallbacks. */
> > -#define NETIF_F_GSO_SOFTWARE (NETIF_F_ALL_TSO | \
> >
> > - NETIF_F_GSO_SCTP)
> >
> >
> >
> > +#define NETIF_F_GSO_SOFTWARE (NETIF_F_ALL_TSO | NETIF_F_GSO_SCTP | \
> >
> > - NETIF_F_GSO_UDP_L4 | NETIF_F_GSO_FRAGLIST)
> >
> >
>
> What exactly do you mean by resegmenting?
I mean pts 5-6 from the full path:
1. Our NIC driver advertises a support for fraglists, GSO UDP L4, GSO fraglists.
2. User enables fraglisted GRO via Ethtool.
3. GRO subsystem receives UDP frames from driver and merges the packets into
fraglisted GSO skb(s).
4. Networking stack queues it up for xmitting.
5. Virtual device like VLAN doesn't advertise a support for GSO UDP L4 and
GSO fraglists, so skb_gso_check() doesn't allow to pass this skb as is to
the real driver.
6. Kernel then has to form a bunch of regular UDP skbs from that one and pass
it to the driver instead. This fallback is *extremely* slow for any GSO types,
but especially for GSO fraglists.
7. All further processing performs with a series of plain UDP skbs, and the
driver gets it one-by-one, despite that it supports UDP L4 and fraglisted GSO.
That's not OK because:
a) logical/virtual netdevs like VLANs, bridges etc. should pass GSO skbs as is;
b) even if the final driver doesn't support such type of GSO, this software
resegmenting should be performed right before it, not in the middle of
processing -- I think I even saw that note somewhere in kernel documentation,
and it's totally reasonable in terms of performance.
> I think it is fine to reenable this again, now that UDP sockets will
> segment unexpected UDP GSO packets that may have looped. We previously
> added general software support in commit 83aa025f535f ("udp: add gso
> support to virtual devices"). Then reduced its scope to egress only in
> 8eea1ca82be9 ("gso: limit udp gso to egress-only virtual devices") to
> handle that edge case.
>
> If we can enable for all virtual devices again, we could revert those
> device specific options.
Thanks,
Al
From: Alexander Lobakin <[email protected]>
Date: Sat, 31 Oct 2020 10:31:31 +0000
> On Saturday, 31 October 2020, 2:12, Willem de Bruijn <[email protected]> wrote:
>
> Hi Willem,
>
>> On Fri, Oct 30, 2020 at 2:33 PM Alexander Lobakin [email protected] wrote:
>>
>>> Commit 9fd1ff5d2ac7 ("udp: Support UDP fraglist GRO/GSO.") added a support
>>> for fraglist UDP L4 and fraglist GSO not only for local traffic, but also
>>> for forwarding. This works well on simple setups, but when any logical
>>> netdev (e.g. VLAN) is present, kernel stack always performs software
>>> resegmentation which actually kills the performance.
>>> Despite the fact that no mainline drivers currently supports fraglist GSO,
>>> this should and can be easily fixed by adding UDP L4 and fraglist GSO to
>>> the list of GSO types that can be passed-through the logical interfaces
>>> (NETIF_F_GSO_SOFTWARE). After this change, no resegmentation occurs (if
>>> a particular driver supports and advertises this), and the performance
>>> goes on par with e.g. 1:1 forwarding.
>>> The only logical netdevs that seem to be unaffected to this are bridge
>>> interfaces, as their code uses full NETIF_F_GSO_MASK.
>>>
>>> Tested on MIPS32 R2 router board with a WIP NIC driver in VLAN NAT:
>>> 20 Mbps baseline, 1 Gbps / link speed with this patch.
>>>
>>> Signed-off-by: Alexander Lobakin [email protected]
>>> ------------------------------------------------
>>> include/linux/netdev_features.h | 4 ++--
>>> 1 file changed, 2 insertions(+), 2 deletions(-)
>>> diff --git a/include/linux/netdev_features.h b/include/linux/netdev_feautres.h
>>> index 0b17c4322b09..934de56644e7 100644
>>> --- a/include/linux/netdev_features.h
>>> +++ b/include/linux/netdev_features.h
>>> @@ -207,8 +207,8 @@ static inline int find_next_netdev_feature(u64 feature, unsigned long start)
>>> NETIF_F_FSO)
>>> /* List of features with software fallbacks. */
>>> -#define NETIF_F_GSO_SOFTWARE (NETIF_F_ALL_TSO | \
>>> - NETIF_F_GSO_SCTP)
>>> +#define NETIF_F_GSO_SOFTWARE (NETIF_F_ALL_TSO | NETIF_F_GSO_SCTP | \
>>> - NETIF_F_GSO_UDP_L4 | NETIF_F_GSO_FRAGLIST)
>>
>> What exactly do you mean by resegmenting?
>
> I mean pts 5-6 from the full path:
> 1. Our NIC driver advertises a support for fraglists, GSO UDP L4, GSO fraglists.
> 2. User enables fraglisted GRO via Ethtool.
> 3. GRO subsystem receives UDP frames from driver and merges the packets into
> fraglisted GSO skb(s).
> 4. Networking stack queues it up for xmitting.
> 5. Virtual device like VLAN doesn't advertise a support for GSO UDP L4 and
> GSO fraglists, so skb_gso_check() doesn't allow to pass this skb as is to
> the real driver.
> 6. Kernel then has to form a bunch of regular UDP skbs from that one and pass
> it to the driver instead. This fallback is *extremely* slow for any GSO types,
> but especially for GSO fraglists.
> 7. All further processing performs with a series of plain UDP skbs, and the
> driver gets it one-by-one, despite that it supports UDP L4 and fraglisted GSO.
>
> That's not OK because:
> a) logical/virtual netdevs like VLANs, bridges etc. should pass GSO skbs as is;
> b) even if the final driver doesn't support such type of GSO, this software
> resegmenting should be performed right before it, not in the middle of
> processing -- I think I even saw that note somewhere in kernel documentation,
> and it's totally reasonable in terms of performance.
>
>> I think it is fine to reenable this again, now that UDP sockets will
>> segment unexpected UDP GSO packets that may have looped. We previously
>> added general software support in commit 83aa025f535f ("udp: add gso
>> support to virtual devices"). Then reduced its scope to egress only in
>> 8eea1ca82be9 ("gso: limit udp gso to egress-only virtual devices") to
>> handle that edge case.
Regarding bonding and teaming: I think they should also use
NETIF_F_GSO_SOFTWARE mask, not NETIF_F_ALL_TSO, as SCTP also has
a software fallback. This way we could also remove a separate
advertising of NETIF_F_GSO_UDP_L4, as it will be included in the first.
So, if this one:
1. Add NETIF_F_GSO_UDP_L4 and NETIF_F_GSO_FRAGLIST to
NETIF_F_GSO_SOFTWARE;
2. Change bonding and teaming features mask from NETIF_F_ALL_TSO |
NETIF_F_GSO_UDP_L4 to NETIF_F_GSO_SOFTWARE;
3. Check that every virtual netdev has NETIF_F_GSO_SOFTWARE _or_
NETIF_F_GSO_MASK in its advertising.
is fine for everyone, I'll publish more appropriate and polished v2 soon.
>> If we can enable for all virtual devices again, we could revert those
>> device specific options.
>
> Thanks,
> Al
Al
On Sat, Oct 31, 2020 at 6:31 AM Alexander Lobakin <[email protected]> wrote:
>
> On Saturday, 31 October 2020, 2:12, Willem de Bruijn <[email protected]> wrote:
>
> Hi Willem,
>
> > On Fri, Oct 30, 2020 at 2:33 PM Alexander Lobakin [email protected] wrote:
> >
> > > Commit 9fd1ff5d2ac7 ("udp: Support UDP fraglist GRO/GSO.") added a support
> > > for fraglist UDP L4 and fraglist GSO not only for local traffic, but also
> > > for forwarding. This works well on simple setups, but when any logical
> > > netdev (e.g. VLAN) is present, kernel stack always performs software
> > > resegmentation which actually kills the performance.
> > > Despite the fact that no mainline drivers currently supports fraglist GSO,
> > > this should and can be easily fixed by adding UDP L4 and fraglist GSO to
> > > the list of GSO types that can be passed-through the logical interfaces
> > > (NETIF_F_GSO_SOFTWARE). After this change, no resegmentation occurs (if
> > > a particular driver supports and advertises this), and the performance
> > > goes on par with e.g. 1:1 forwarding.
> > > The only logical netdevs that seem to be unaffected to this are bridge
> > > interfaces, as their code uses full NETIF_F_GSO_MASK.
> > > Tested on MIPS32 R2 router board with a WIP NIC driver in VLAN NAT:
> > > 20 Mbps baseline, 1 Gbps / link speed with this patch.
> > >
> > > Signed-off-by: Alexander Lobakin [email protected]
> > >
> > > ------------------------------------------------
> > >
> > > include/linux/netdev_features.h | 4 ++--
> > > 1 file changed, 2 insertions(+), 2 deletions(-)
> > > diff --git a/include/linux/netdev_features.h b/include/linux/netdev_features.h
> > > index 0b17c4322b09..934de56644e7 100644
> > > --- a/include/linux/netdev_features.h
> > > +++ b/include/linux/netdev_features.h
> > > @@ -207,8 +207,8 @@ static inline int find_next_netdev_feature(u64 feature, unsigned long start)
> > > NETIF_F_FSO)
> > > /* List of features with software fallbacks. */
> > > -#define NETIF_F_GSO_SOFTWARE (NETIF_F_ALL_TSO | \
> > >
> > > - NETIF_F_GSO_SCTP)
> > >
> > >
> > >
> > > +#define NETIF_F_GSO_SOFTWARE (NETIF_F_ALL_TSO | NETIF_F_GSO_SCTP | \
> > >
> > > - NETIF_F_GSO_UDP_L4 | NETIF_F_GSO_FRAGLIST)
> > >
> > >
> >
> > What exactly do you mean by resegmenting?
>
> I mean pts 5-6 from the full path:
> 1. Our NIC driver advertises a support for fraglists, GSO UDP L4, GSO fraglists.
I see. This was the part I missed. The commit message mentions that no
mainline driver advertises h/w offload support. I had missed that
there may be non-mainline drivers that do ;)
Yes, then the use case is perfectly clear.
Great to see these features getting offload support.
> >> I think it is fine to reenable this again, now that UDP sockets will
> >> segment unexpected UDP GSO packets that may have looped. We previously
> >> added general software support in commit 83aa025f535f ("udp: add gso
> >> support to virtual devices"). Then reduced its scope to egress only in
> >> 8eea1ca82be9 ("gso: limit udp gso to egress-only virtual devices") to
> >> handle that edge case.
>
> Regarding bonding and teaming: I think they should also use
> NETIF_F_GSO_SOFTWARE mask, not NETIF_F_ALL_TSO, as SCTP also has
> a software fallback. This way we could also remove a separate
> advertising of NETIF_F_GSO_UDP_L4, as it will be included in the first.
>
> So, if this one:
> 1. Add NETIF_F_GSO_UDP_L4 and NETIF_F_GSO_FRAGLIST to
> NETIF_F_GSO_SOFTWARE;
> 2. Change bonding and teaming features mask from NETIF_F_ALL_TSO |
> NETIF_F_GSO_UDP_L4 to NETIF_F_GSO_SOFTWARE;
> 3. Check that every virtual netdev has NETIF_F_GSO_SOFTWARE _or_
> NETIF_F_GSO_MASK in its advertising.
>
> is fine for everyone, I'll publish more appropriate and polished v2 soon.
I think we can revert 8eea1ca82be9. Except for the part where it
defines the feature in NETIF_F_GSO_ENCAP_ALL instead of
NETIF_F_GSO_SOFTWARE. That appears to have been a peculiar choice. I
can't recall exactly why I chose that. Most likely because that was
(at the time) the only macro that covered all the devices I wanted to
capture.
As for SCTP: that has the same concern that prompted that commit for
UDP: is it safe to forward those packets to the ingress path today?
From: Alexander Lobakin <[email protected]>
Date: Sat, 31 Oct 2020 14:17:03 +0000
> From: Alexander Lobakin <[email protected]>
> Date: Sat, 31 Oct 2020 10:31:31 +0000
>
>> On Saturday, 31 October 2020, 2:12, Willem de Bruijn <[email protected]> wrote:
>>
>> Hi Willem,
>>
>>> On Fri, Oct 30, 2020 at 2:33 PM Alexander Lobakin [email protected] wrote:
>>>
>>>> Commit 9fd1ff5d2ac7 ("udp: Support UDP fraglist GRO/GSO.") added a support
>>>> for fraglist UDP L4 and fraglist GSO not only for local traffic, but also
>>>> for forwarding. This works well on simple setups, but when any logical
>>>> netdev (e.g. VLAN) is present, kernel stack always performs software
>>>> resegmentation which actually kills the performance.
>>>> Despite the fact that no mainline drivers currently supports fraglist GSO,
>>>> this should and can be easily fixed by adding UDP L4 and fraglist GSO to
>>>> the list of GSO types that can be passed-through the logical interfaces
>>>> (NETIF_F_GSO_SOFTWARE). After this change, no resegmentation occurs (if
>>>> a particular driver supports and advertises this), and the performance
>>>> goes on par with e.g. 1:1 forwarding.
>>>> The only logical netdevs that seem to be unaffected to this are bridge
>>>> interfaces, as their code uses full NETIF_F_GSO_MASK.
>>>>
>>>> Tested on MIPS32 R2 router board with a WIP NIC driver in VLAN NAT:
>>>> 20 Mbps baseline, 1 Gbps / link speed with this patch.
>>>>
>>>> Signed-off-by: Alexander Lobakin <[email protected]>
>>>> ---
>>>> include/linux/netdev_features.h | 4 ++--
>>>> 1 file changed, 2 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/include/linux/netdev_features.h b/include/linux/netdev_features.h
>>>> index 0b17c4322b09..934de56644e7 100644
>>>> --- a/include/linux/netdev_features.h
>>>> +++ b/include/linux/netdev_features.h
>>>> @@ -207,8 +207,8 @@ static inline int find_next_netdev_feature(u64 feature, unsigned long start)
>>>> NETIF_F_FSO)
>>>>
>>>> /* List of features with software fallbacks. */
>>>> -#define NETIF_F_GSO_SOFTWARE (NETIF_F_ALL_TSO | \
>>>> - NETIF_F_GSO_SCTP)
>>>> +#define NETIF_F_GSO_SOFTWARE (NETIF_F_ALL_TSO | NETIF_F_GSO_SCTP | \
>>>> + NETIF_F_GSO_UDP_L4 | NETIF_F_GSO_FRAGLIST)
>>>>
>>>> /*
>>>> * If one device supports one of these features, then enable them
>>>> --
>>>> 2.29.2
>>>
>>> What exactly do you mean by resegmenting?
>>
>> I mean pts 5-6 from the full path:
>> 1. Our NIC driver advertises a support for fraglists, GSO UDP L4, GSO fraglists.
>> 2. User enables fraglisted GRO via Ethtool.
>> 3. GRO subsystem receives UDP frames from driver and merges the packets into
>> fraglisted GSO skb(s).
>> 4. Networking stack queues it up for xmitting.
>> 5. Virtual device like VLAN doesn't advertise a support for GSO UDP L4 and
>> GSO fraglists, so skb_gso_check() doesn't allow to pass this skb as is to
>> the real driver.
>> 6. Kernel then has to form a bunch of regular UDP skbs from that one and pass
>> it to the driver instead. This fallback is *extremely* slow for any GSO types,
>> but especially for GSO fraglists.
>> 7. All further processing performs with a series of plain UDP skbs, and the
>> driver gets it one-by-one, despite that it supports UDP L4 and fraglisted GSO.
>>
>> That's not OK because:
>> a) logical/virtual netdevs like VLANs, bridges etc. should pass GSO skbs as is;
>> b) even if the final driver doesn't support such type of GSO, this software
>> resegmenting should be performed right before it, not in the middle of
>> processing -- I think I even saw that note somewhere in kernel documentation,
>> and it's totally reasonable in terms of performance.
>>
>>> I think it is fine to reenable this again, now that UDP sockets will
>>> segment unexpected UDP GSO packets that may have looped. We previously
>>> added general software support in commit 83aa025f535f ("udp: add gso
>>> support to virtual devices"). Then reduced its scope to egress only in
>>> 8eea1ca82be9 ("gso: limit udp gso to egress-only virtual devices") to
>>> handle that edge case.
>
> Regarding bonding and teaming: I think they should also use
> NETIF_F_GSO_SOFTWARE mask, not NETIF_F_ALL_TSO, as SCTP also has
> a software fallback. This way we could also remove a separate
> advertising of NETIF_F_GSO_UDP_L4, as it will be included in the first.
>
> So, if this one:
> 1. Add NETIF_F_GSO_UDP_L4 and NETIF_F_GSO_FRAGLIST to
> NETIF_F_GSO_SOFTWARE;
> 2. Change bonding and teaming features mask from NETIF_F_ALL_TSO |
> NETIF_F_GSO_UDP_L4 to NETIF_F_GSO_SOFTWARE;
> 3. Check that every virtual netdev has NETIF_F_GSO_SOFTWARE _or_
> NETIF_F_GSO_MASK in its advertising.
>
> is fine for everyone, I'll publish more appropriate and polished v2 soon.
>
>>> If we can enable for all virtual devices again, we could revert those
>>> device specific options.
Just for reference: commit 8eea1ca82be9 ("gso: limit udp gso to
egress-only virtual devices") from May 2018 (4.18-rc1) says:
"Until the udp receive stack supports large packets (UDP GRO), GSO
packets must not loop from the egress to the ingress path".
UDP GRO was actually added in commit e20cf8d3f1f7 ("udp: implement GRO
for plain UDP sockets.") (Nov 2018, 5.0-rc1) and then expanded with
fraglists with commit 9fd1ff5d2ac7 ("udp: Support UDP fraglist GRO/GSO.")
(Jan 2020, v5.6-rc1), so there should be no problems with
macvlan/veth/etc. anymore.
>> Thanks,
>> Al
>
> Al
Al
From: Willem de Bruijn <[email protected]>
Date: Sat, 31 Oct 2020 11:26:24 -0400
>>>> I think it is fine to reenable this again, now that UDP sockets will
>>>> segment unexpected UDP GSO packets that may have looped. We previously
>>>> added general software support in commit 83aa025f535f ("udp: add gso
>>>> support to virtual devices"). Then reduced its scope to egress only in
>>>> 8eea1ca82be9 ("gso: limit udp gso to egress-only virtual devices") to
>>>> handle that edge case.
>>
>> Regarding bonding and teaming: I think they should also use
>> NETIF_F_GSO_SOFTWARE mask, not NETIF_F_ALL_TSO, as SCTP also has
>> a software fallback. This way we could also remove a separate
>> advertising of NETIF_F_GSO_UDP_L4, as it will be included in the first.
>>
>> So, if this one:
>> 1. Add NETIF_F_GSO_UDP_L4 and NETIF_F_GSO_FRAGLIST to
>> NETIF_F_GSO_SOFTWARE;
>> 2. Change bonding and teaming features mask from NETIF_F_ALL_TSO |
>> NETIF_F_GSO_UDP_L4 to NETIF_F_GSO_SOFTWARE;
>> 3. Check that every virtual netdev has NETIF_F_GSO_SOFTWARE _or_
>> NETIF_F_GSO_MASK in its advertising.
>>
>> is fine for everyone, I'll publish more appropriate and polished v2 soon.
>
> I think we can revert 8eea1ca82be9. Except for the part where it
> defines the feature in NETIF_F_GSO_ENCAP_ALL instead of
> NETIF_F_GSO_SOFTWARE. That appears to have been a peculiar choice. I
> can't recall exactly why I chose that. Most likely because that was
> (at the time) the only macro that covered all the devices I wanted to
> capture.
>
> As for SCTP: that has the same concern that prompted that commit for
> UDP: is it safe to forward those packets to the ingress path today?
Oh well. I just looked up into net/sctp/offload.c and see no GRO
receiving callbacks, only GSO ones. On the other hand,
NETIF_F_GSO_SOFTWARE includes GSO_SCTP and is used in almost every
virtual netdev driver, including macvlan and veth mentioned earlier,
so that seems to be fine.
> I had missed that there may be non-mainline drivers that do ;)
> Great to see these features getting offload support.
It will be mainlined sooner or later depending on my workload :)
UDP fraglists *really* boosted the things up for me, so I don't quite
understand why not a single mainline driver has a support for them.
Al
On Sat, Oct 31, 2020 at 11:56 AM Alexander Lobakin <[email protected]> wrote:
>
> From: Willem de Bruijn <[email protected]>
> Date: Sat, 31 Oct 2020 11:26:24 -0400
>
> >>>> I think it is fine to reenable this again, now that UDP sockets will
> >>>> segment unexpected UDP GSO packets that may have looped. We previously
> >>>> added general software support in commit 83aa025f535f ("udp: add gso
> >>>> support to virtual devices"). Then reduced its scope to egress only in
> >>>> 8eea1ca82be9 ("gso: limit udp gso to egress-only virtual devices") to
> >>>> handle that edge case.
> >>
> >> Regarding bonding and teaming: I think they should also use
> >> NETIF_F_GSO_SOFTWARE mask, not NETIF_F_ALL_TSO, as SCTP also has
> >> a software fallback. This way we could also remove a separate
> >> advertising of NETIF_F_GSO_UDP_L4, as it will be included in the first.
> >>
> >> So, if this one:
> >> 1. Add NETIF_F_GSO_UDP_L4 and NETIF_F_GSO_FRAGLIST to
> >> NETIF_F_GSO_SOFTWARE;
> >> 2. Change bonding and teaming features mask from NETIF_F_ALL_TSO |
> >> NETIF_F_GSO_UDP_L4 to NETIF_F_GSO_SOFTWARE;
> >> 3. Check that every virtual netdev has NETIF_F_GSO_SOFTWARE _or_
> >> NETIF_F_GSO_MASK in its advertising.
> >>
> >> is fine for everyone, I'll publish more appropriate and polished v2 soon.
> >
> > I think we can revert 8eea1ca82be9. Except for the part where it
> > defines the feature in NETIF_F_GSO_ENCAP_ALL instead of
> > NETIF_F_GSO_SOFTWARE. That appears to have been a peculiar choice. I
> > can't recall exactly why I chose that. Most likely because that was
> > (at the time) the only macro that covered all the devices I wanted to
> > capture.
> >
> > As for SCTP: that has the same concern that prompted that commit for
> > UDP: is it safe to forward those packets to the ingress path today?
>
> Oh well. I just looked up into net/sctp/offload.c and see no GRO
> receiving callbacks, only GSO ones. On the other hand,
> NETIF_F_GSO_SOFTWARE includes GSO_SCTP and is used in almost every
> virtual netdev driver, including macvlan and veth mentioned earlier,
> so that seems to be fine.
To follow up: SCTP sockets can handle such packets. So both local
reception and forwarding are fine. This was expressly added to the
second revision of the SCTP GSO commit.