Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <a144aa6351412e25bbdf866c0d31b550e6ff3e8a.camel@redhat.com>
Subject: Re: [RFC Optimizing veth xsk performance 00/10]
From:   Paolo Abeni <pabeni@redhat.com>
To:     "huangjie.albert" <huangjie.albert@bytedance.com>,
        davem@davemloft.net, edumazet@google.com, kuba@kernel.org
Cc:     Alexei Starovoitov <ast@kernel.org>,
        Daniel Borkmann <daniel@iogearbox.net>,
        Jesper Dangaard Brouer <hawk@kernel.org>,
        John Fastabend <john.fastabend@gmail.com>,
        =?ISO-8859-1?Q?Bj=F6rn_T=F6pel?= <bjorn@kernel.org>,
        Magnus Karlsson <magnus.karlsson@intel.com>,
        Maciej Fijalkowski <maciej.fijalkowski@intel.com>,
        Jonathan Lemon <jonathan.lemon@gmail.com>,
        Pavel Begunkov <asml.silence@gmail.com>,
        Yunsheng Lin <linyunsheng@huawei.com>,
        Kees Cook <keescook@chromium.org>,
        Richard Gobert <richardbgobert@gmail.com>,
        "open list:NETWORKING DRIVERS" <netdev@vger.kernel.org>,
        open list <linux-kernel@vger.kernel.org>,
        "open list:XDP (eXpress Data Path)" <bpf@vger.kernel.org>
Date:   Thu, 03 Aug 2023 16:20:06 +0200
In-Reply-To: <20230803140441.53596-1-huangjie.albert@bytedance.com>
References: <20230803140441.53596-1-huangjie.albert@bytedance.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.46.4 (3.46.4-1.fc37) 
MIME-Version: 1.0
Precedence: bulk

On Thu, 2023-08-03 at 22:04 +0800, huangjie.albert wrote:
> AF_XDP is a kernel bypass technology that can greatly improve performance=
.
> However, for virtual devices like veth, even with the use of AF_XDP socke=
ts,
> there are still many additional software paths that consume CPU resources=
.=20
> This patch series focuses on optimizing the performance of AF_XDP sockets=
=20
> for veth virtual devices. Patches 1 to 4 mainly involve preparatory work.=
=20
> Patch 5 introduces tx queue and tx napi for packet transmission, while=
=20
> patch 9 primarily implements zero-copy, and patch 10 adds support for=20
> batch sending of IPv4 UDP packets. These optimizations significantly redu=
ce=20
> the software path and support checksum offload.
>=20
> I tested those feature with
> A typical topology is shown below:
> veth<-->veth-peer                                    veth1-peer<--->veth1
> 	1       |                                                  |   7
> 	        |2                                                6|
> 	        |                                                  |
> 	      bridge<------->eth0(mlnx5)- switch -eth1(mlnx5)<--->bridge1
>                   3                    4                 5   =20
>              (machine1)                              (machine2)   =20
> AF_XDP socket is attach to veth and veth1. and send packets to physical N=
IC(eth0)
> veth:(172.17.0.2/24)
> bridge:(172.17.0.1/24)
> eth0:(192.168.156.66/24)
>=20
> eth1(172.17.0.2/24)
> bridge1:(172.17.0.1/24)
> eth0:(192.168.156.88/24)
>=20
> after set default route=EF=BF=BD=EF=BF=BD?snat=EF=BF=BD=EF=BF=BD?dnat. we=
 can have a tests
> to get the performance results.
>=20
> packets send from veth to veth1:
> af_xdp test tool:
> link:https://github.com/cclinuxer/libxudp
> send:(veth)
> ./objs/xudpperf send --dst 192.168.156.88:6002 -l 1300
> recv:(veth1)
> ./objs/xudpperf recv --src 172.17.0.2:6002
>=20
> udp test tool:iperf3
> send:(veth)
> iperf3 -c 192.168.156.88 -p 6002 -l 1300 -b 60G -u

Should be: '-b 0' otherwise you will experience additional overhead.

And you would likely pin processes and irqs to ensure BH and US run on
different cores of the same numa node.

Cheers,

Paolo