We see a performance problem when the slaves of the bond
don't support checksum offload features. What we see is
tcp_sendmsg's skb_add_data_nocache ending up not using the
csum_and_copy_from_user which would have computed the
checksum while copying from user buffer to kernel buffer.
Instead it computes later in dev_hard_start_xmit when it
figures out the slave doesn't support checksum offload and
ends up expensive .
The bonding interface's "features" has ?NETIF_F_HW_CSUM
(or NETIF_F_NO_CSUM in 2.6.39) set which makes the
stack think checksum need not be computed in software.
/*
* Check whether we can use HW checksum.
*/
if (sk->sk_route_caps & NETIF_F_ALL_CSUM)
???? skb->ip_summed = CHECKSUM_PARTIAL;
But later in dev_hard_start_xmit it finds out the slave does not
support checksumming and decides to compute in software.
/* If packet is not checksummed and device does not
* support checksumming for this protocol, complete
* checksumming here.
*/
if (skb->ip_summed == CHECKSUM_PARTIAL) {
?????? ??skb_set_transport_header(skb,
?????? ???????skb_checksum_start_offset(skb));
????????if (!(features & NETIF_F_ALL_CSUM) &&
??????? ?????skb_checksum_help(skb))
???????? ??????goto out_kfree_skb;
}
We see this problem after this commit:
commit 1742f183fc218798dab6fcf0ded25b6608fc0a48
Author: Micha?<82> Miros?<82>aw <[email protected]>
Date:?? Fri Apr 22 06:31:16 2011 +0000
??? net: fix netdev_increment_features()
??? Simplify and fix netdev_increment_features() to conform to what is
??? stated in netdevice.h comments about NETIF_F_ONE_FOR_ALL.
??? Include FCoE segmentation and VLAN-challedged flags in computation.
??? Signed-off-by: Micha?<82> Miros?<82>aw <[email protected]>
??? Signed-off-by: David S. Miller <[email protected]>
Prior to that the below code in netdev_increment_features was helping in
turning off NETIF_F_NO_CSUM on bond when the slaves don't support it:
/* If device needs checksumming, downgrade to it. */
if (all & NETIF_F_NO_CSUM && !(one & NETIF_F_NO_CSUM))
??????? all ^= NETIF_F_NO_CSUM | (one & NETIF_F_ALL_CSUM);
The slaves are Mellanox IB adapters. This is on x86_64 platform.
Please let us know if you need any additional information.
Thanks.
Venkat
On Mon, 2014-04-21 at 07:23 -0700, Venkat Venkatsubra wrote:
> We see a performance problem when the slaves of the bond
> don't support checksum offload features. What we see is
> tcp_sendmsg's skb_add_data_nocache ending up not using the
> csum_and_copy_from_user which would have computed the
> checksum while copying from user buffer to kernel buffer.
> Instead it computes later in dev_hard_start_xmit when it
> figures out the slave doesn't support checksum offload and
> ends up expensive .
>
> The bonding interface's "features" has NETIF_F_HW_CSUM
> (or NETIF_F_NO_CSUM in 2.6.39) set which makes the
> stack think checksum need not be computed in software.
> /*
> * Check whether we can use HW checksum.
> */
> if (sk->sk_route_caps & NETIF_F_ALL_CSUM)
> skb->ip_summed = CHECKSUM_PARTIAL;
>
> But later in dev_hard_start_xmit it finds out the slave does not
> support checksumming and decides to compute in software.
>
> /* If packet is not checksummed and device does not
> * support checksumming for this protocol, complete
> * checksumming here.
> */
> if (skb->ip_summed == CHECKSUM_PARTIAL) {
> skb_set_transport_header(skb,
> skb_checksum_start_offset(skb));
> if (!(features & NETIF_F_ALL_CSUM) &&
> skb_checksum_help(skb))
> goto out_kfree_skb;
> }
>
> We see this problem after this commit:
> commit 1742f183fc218798dab6fcf0ded25b6608fc0a48
> Author: MichaÅ<82> MirosÅ<82>aw <[email protected]>
> Date: Fri Apr 22 06:31:16 2011 +0000
>
> net: fix netdev_increment_features()
>
> Simplify and fix netdev_increment_features() to conform to what is
> stated in netdevice.h comments about NETIF_F_ONE_FOR_ALL.
> Include FCoE segmentation and VLAN-challedged flags in computation.
>
> Signed-off-by: MichaÅ<82> MirosÅ<82>aw <[email protected]>
> Signed-off-by: David S. Miller <[email protected]>
>
> Prior to that the below code in netdev_increment_features was helping in
> turning off NETIF_F_NO_CSUM on bond when the slaves don't support it:
> /* If device needs checksumming, downgrade to it. */
> if (all & NETIF_F_NO_CSUM && !(one & NETIF_F_NO_CSUM))
> all ^= NETIF_F_NO_CSUM | (one & NETIF_F_ALL_CSUM);
>
> The slaves are Mellanox IB adapters. This is on x86_64 platform.
>
> Please let us know if you need any additional information.
Please CC patch author (I did), instead of sending this to hundred of
people (linux-kernel ??? netdev is more appropriate...)
Do these NIC really not support TX checksum ?
You did not provide kernel version you use.
Please also provide : (using a recent ethtool to get extended offload
info)
ethtool -k bond0 # or the bonding device name
ethtool -k eth1 # or the slave name
ethtool -i eth1
Thanks !