2014-04-29 10:44:24

by Zhangjie (HZ)

Subject: [PATCH net-next] virtio-net: Set needed_headroom for virtio-net when VIRTIO_F_ANY_LAYOUT is true

This is a small supplement for commit e7428e95a06fb516fac1308bd0e176e27c0b9287
("virtio-net: put virtio-net header inline with data"). TCP packages have
enough room to put virtio-net header in, but UDP packages do not. By
setting dev->needed_headroom for virtio-net device, UDP packages could have
enough room.

For UDP packages, sk_buff is alloced in fun __ip_append_data. The size is
"alloclen + hh_len + 15", and "hh_len = LL_RESERVED_SPACE(rt-dst.dev);".
The Macro is defined as follows:
#define LL_RESERVED_SPACE(dev) \
((((dev)->hard_header_len+(dev)->needed_headroom)\
&~(HH_DATA_MOD - 1)) + HH_DATA_MOD)
By default, for UDP packages, after skb is allocated, only 16 bytes
reserved. And 2 bytes remained after mac header is set. That is not enough
to put virtio-net header in. If we set dev->needed_headroom to 12 or 10
(according to mergeable_rx_bufs is on or off ), more room can be reserved.
Then there is enough room for UDP packages to put the header in.

test result list as below:
guest and host: suse11sp3, netperf, intel 2.4GHz
+-------+---------+---------+---------+---------+
| | old | new |
+-------+---------+---------+---------+---------+
| UDP | Gbit/s | pps | Gbit/s | pps |
| 64 | 0.57 | 692232 | 0.61 | 742420 |
| 256 | 1.60 | 686860 | 1.71 | 733331 |
| 512 | 2.92 | 674576 | 3.07 | 710446 |
| 1024 | 4.99 | 598977 | 5.17 | 620821 |
| 1460 | 5.68 | 483757 | 7.16 | 610519 |
| 4096 | 6.98 | 637468 | 7.21 | 658471 |
+-------+---------+---------+---------+---------+

Signed-off-by: Zhang Jie <[email protected]>
---
drivers/net/virtio_net.c | 7 +++++++
1 file changed, 7 insertions(+)

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 8a852b5..b852bb3 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -1724,6 +1724,13 @@ static int virtnet_probe(struct virtio_device *vdev)
if (virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_VQ))
vi->has_cvq = true;

+ if (vi->any_header_sg) {
+ if (vi->mergeable_rx_bufs)
+ dev->needed_headroom = sizeof(struct virtio_net_hdr_mrg_rxbuf);
+ else
+ dev->needed_headroom = sizeof(struct virtio_net_hdr);
+ }
+
/* Use single tx/rx queue pair as default */
vi->curr_queue_pairs = 1;
vi->max_queue_pairs = max_queue_pairs;
--
1.9.0.msysgit.0

2014-04-30 07:11:24

by Rusty Russell

Subject: Re: [PATCH net-next] virtio-net: Set needed_headroom for virtio-net when VIRTIO_F_ANY_LAYOUT is true

"Zhangjie (HZ)" <[email protected]> writes:
> This is a small supplement for commit e7428e95a06fb516fac1308bd0e176e27c0b9287
> ("virtio-net: put virtio-net header inline with data"). TCP packages have
> enough room to put virtio-net header in, but UDP packages do not. By
> setting dev->needed_headroom for virtio-net device, UDP packages could have
> enough room.
>
> For UDP packages, sk_buff is alloced in fun __ip_append_data. The size is
> "alloclen + hh_len + 15", and "hh_len = LL_RESERVED_SPACE(rt-dst.dev);".
> The Macro is defined as follows:
> #define LL_RESERVED_SPACE(dev) \
> ((((dev)->hard_header_len+(dev)->needed_headroom)\
> &~(HH_DATA_MOD - 1)) + HH_DATA_MOD)
> By default, for UDP packages, after skb is allocated, only 16 bytes
> reserved. And 2 bytes remained after mac header is set. That is not enough
> to put virtio-net header in. If we set dev->needed_headroom to 12 or 10
> (according to mergeable_rx_bufs is on or off ), more room can be reserved.
> Then there is enough room for UDP packages to put the header in.

Seems like a no-brainer. Good find!

Acked-by: Rusty Russell <[email protected]>

Thanks,
Rusty.



>
> test result list as below:
> guest and host: suse11sp3, netperf, intel 2.4GHz
> +-------+---------+---------+---------+---------+
> | | old | new |
> +-------+---------+---------+---------+---------+
> | UDP | Gbit/s | pps | Gbit/s | pps |
> | 64 | 0.57 | 692232 | 0.61 | 742420 |
> | 256 | 1.60 | 686860 | 1.71 | 733331 |
> | 512 | 2.92 | 674576 | 3.07 | 710446 |
> | 1024 | 4.99 | 598977 | 5.17 | 620821 |
> | 1460 | 5.68 | 483757 | 7.16 | 610519 |
> | 4096 | 6.98 | 637468 | 7.21 | 658471 |
> +-------+---------+---------+---------+---------+
>
> Signed-off-by: Zhang Jie <[email protected]>
> ---
> drivers/net/virtio_net.c | 7 +++++++
> 1 file changed, 7 insertions(+)
>
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index 8a852b5..b852bb3 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -1724,6 +1724,13 @@ static int virtnet_probe(struct virtio_device *vdev)
> if (virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_VQ))
> vi->has_cvq = true;
>
> + if (vi->any_header_sg) {
> + if (vi->mergeable_rx_bufs)
> + dev->needed_headroom = sizeof(struct virtio_net_hdr_mrg_rxbuf);
> + else
> + dev->needed_headroom = sizeof(struct virtio_net_hdr);
> + }
> +
> /* Use single tx/rx queue pair as default */
> vi->curr_queue_pairs = 1;
> vi->max_queue_pairs = max_queue_pairs;
> --
> 1.9.0.msysgit.0
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/

2014-04-30 09:29:01

by Michael S. Tsirkin

Subject: Re: [PATCH net-next] virtio-net: Set needed_headroom for virtio-net when VIRTIO_F_ANY_LAYOUT is true

On Tue, Apr 29, 2014 at 06:43:22PM +0800, Zhangjie (HZ) wrote:
> This is a small supplement for commit e7428e95a06fb516fac1308bd0e176e27c0b9287
> ("virtio-net: put virtio-net header inline with data"). TCP packages have
> enough room to put virtio-net header in, but UDP packages do not. By
> setting dev->needed_headroom for virtio-net device, UDP packages could have
> enough room.
>
> For UDP packages, sk_buff is alloced in fun __ip_append_data. The size is
> "alloclen + hh_len + 15", and "hh_len = LL_RESERVED_SPACE(rt-dst.dev);".
> The Macro is defined as follows:
> #define LL_RESERVED_SPACE(dev) \
> ((((dev)->hard_header_len+(dev)->needed_headroom)\
> &~(HH_DATA_MOD - 1)) + HH_DATA_MOD)
> By default, for UDP packages, after skb is allocated, only 16 bytes
> reserved. And 2 bytes remained after mac header is set. That is not enough
> to put virtio-net header in. If we set dev->needed_headroom to 12 or 10
> (according to mergeable_rx_bufs is on or off ), more room can be reserved.
> Then there is enough room for UDP packages to put the header in.
>
> test result list as below:
> guest and host: suse11sp3, netperf, intel 2.4GHz
> +-------+---------+---------+---------+---------+
> | | old | new |
> +-------+---------+---------+---------+---------+
> | UDP | Gbit/s | pps | Gbit/s | pps |
> | 64 | 0.57 | 692232 | 0.61 | 742420 |
> | 256 | 1.60 | 686860 | 1.71 | 733331 |
> | 512 | 2.92 | 674576 | 3.07 | 710446 |
> | 1024 | 4.99 | 598977 | 5.17 | 620821 |
> | 1460 | 5.68 | 483757 | 7.16 | 610519 |
> | 4096 | 6.98 | 637468 | 7.21 | 658471 |
> +-------+---------+---------+---------+---------+
>
> Signed-off-by: Zhang Jie <[email protected]>

Thanks for your patience!

Acked-by: Michael S. Tsirkin <[email protected]>


> ---
> drivers/net/virtio_net.c | 7 +++++++
> 1 file changed, 7 insertions(+)
>
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index 8a852b5..b852bb3 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -1724,6 +1724,13 @@ static int virtnet_probe(struct virtio_device *vdev)
> if (virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_VQ))
> vi->has_cvq = true;
>
> + if (vi->any_header_sg) {
> + if (vi->mergeable_rx_bufs)
> + dev->needed_headroom = sizeof(struct virtio_net_hdr_mrg_rxbuf);
> + else
> + dev->needed_headroom = sizeof(struct virtio_net_hdr);
> + }
> +
> /* Use single tx/rx queue pair as default */
> vi->curr_queue_pairs = 1;
> vi->max_queue_pairs = max_queue_pairs;
> --
> 1.9.0.msysgit.0
>

2014-04-30 09:38:32

by Jason Wang

Subject: Re: [PATCH net-next] virtio-net: Set needed_headroom for virtio-net when VIRTIO_F_ANY_LAYOUT is true

On 04/29/2014 06:43 PM, Zhangjie (HZ) wrote:
> This is a small supplement for commit e7428e95a06fb516fac1308bd0e176e27c0b9287
> ("virtio-net: put virtio-net header inline with data"). TCP packages have
> enough room to put virtio-net header in, but UDP packages do not. By
> setting dev->needed_headroom for virtio-net device, UDP packages could have
> enough room.
>
> For UDP packages, sk_buff is alloced in fun __ip_append_data. The size is
> "alloclen + hh_len + 15", and "hh_len = LL_RESERVED_SPACE(rt-dst.dev);".
> The Macro is defined as follows:
> #define LL_RESERVED_SPACE(dev) \
> ((((dev)->hard_header_len+(dev)->needed_headroom)\
> &~(HH_DATA_MOD - 1)) + HH_DATA_MOD)
> By default, for UDP packages, after skb is allocated, only 16 bytes
> reserved. And 2 bytes remained after mac header is set. That is not enough
> to put virtio-net header in. If we set dev->needed_headroom to 12 or 10
> (according to mergeable_rx_bufs is on or off ), more room can be reserved.
> Then there is enough room for UDP packages to put the header in.
>
> test result list as below:
> guest and host: suse11sp3, netperf, intel 2.4GHz
> +-------+---------+---------+---------+---------+
> | | old | new |
> +-------+---------+---------+---------+---------+
> | UDP | Gbit/s | pps | Gbit/s | pps |
> | 64 | 0.57 | 692232 | 0.61 | 742420 |
> | 256 | 1.60 | 686860 | 1.71 | 733331 |
> | 512 | 2.92 | 674576 | 3.07 | 710446 |
> | 1024 | 4.99 | 598977 | 5.17 | 620821 |
> | 1460 | 5.68 | 483757 | 7.16 | 610519 |
> | 4096 | 6.98 | 637468 | 7.21 | 658471 |
> +-------+---------+---------+---------+---------+
>
> Signed-off-by: Zhang Jie <[email protected]>
> ---
> drivers/net/virtio_net.c | 7 +++++++
> 1 file changed, 7 insertions(+)
>
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index 8a852b5..b852bb3 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -1724,6 +1724,13 @@ static int virtnet_probe(struct virtio_device *vdev)
> if (virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_VQ))
> vi->has_cvq = true;
>
> + if (vi->any_header_sg) {
> + if (vi->mergeable_rx_bufs)
> + dev->needed_headroom = sizeof(struct virtio_net_hdr_mrg_rxbuf);
> + else
> + dev->needed_headroom = sizeof(struct virtio_net_hdr);
> + }
> +
> /* Use single tx/rx queue pair as default */
> vi->curr_queue_pairs = 1;
> vi->max_queue_pairs = max_queue_pairs;

Acked-by: Jason Wang <[email protected]>

2014-04-30 17:32:32

by David Miller

Subject: Re: [PATCH net-next] virtio-net: Set needed_headroom for virtio-net when VIRTIO_F_ANY_LAYOUT is true

From: "Zhangjie (HZ)" <[email protected]>
Date: Tue, 29 Apr 2014 18:43:22 +0800

> This is a small supplement for commit e7428e95a06fb516fac1308bd0e176e27c0b9287
> ("virtio-net: put virtio-net header inline with data"). TCP packages have
> enough room to put virtio-net header in, but UDP packages do not. By
> setting dev->needed_headroom for virtio-net device, UDP packages could have
> enough room.
>
> For UDP packages, sk_buff is alloced in fun __ip_append_data. The size is
> "alloclen + hh_len + 15", and "hh_len = LL_RESERVED_SPACE(rt-dst.dev);".
> The Macro is defined as follows:
> #define LL_RESERVED_SPACE(dev) \
> ((((dev)->hard_header_len+(dev)->needed_headroom)\
> &~(HH_DATA_MOD - 1)) + HH_DATA_MOD)
> By default, for UDP packages, after skb is allocated, only 16 bytes
> reserved. And 2 bytes remained after mac header is set. That is not enough
> to put virtio-net header in. If we set dev->needed_headroom to 12 or 10
> (according to mergeable_rx_bufs is on or off ), more room can be reserved.
> Then there is enough room for UDP packages to put the header in.
>
> test result list as below:
> guest and host: suse11sp3, netperf, intel 2.4GHz
> +-------+---------+---------+---------+---------+
> | | old | new |
> +-------+---------+---------+---------+---------+
> | UDP | Gbit/s | pps | Gbit/s | pps |
> | 64 | 0.57 | 692232 | 0.61 | 742420 |
> | 256 | 1.60 | 686860 | 1.71 | 733331 |
> | 512 | 2.92 | 674576 | 3.07 | 710446 |
> | 1024 | 4.99 | 598977 | 5.17 | 620821 |
> | 1460 | 5.68 | 483757 | 7.16 | 610519 |
> | 4096 | 6.98 | 637468 | 7.21 | 658471 |
> +-------+---------+---------+---------+---------+
>
> Signed-off-by: Zhang Jie <[email protected]>

Applied.