Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Wei Fang <wei.fang@nxp.com>
To:     Jesper Dangaard Brouer <hawk@kernel.org>,
        Jesper Dangaard Brouer <jbrouer@redhat.com>,
        Jakub Kicinski <kuba@kernel.org>
CC:     "brouer@redhat.com" <brouer@redhat.com>,
        "davem@davemloft.net" <davem@davemloft.net>,
        "edumazet@google.com" <edumazet@google.com>,
        "pabeni@redhat.com" <pabeni@redhat.com>,
        Shenwei Wang <shenwei.wang@nxp.com>,
        Clark Wang <xiaoning.wang@nxp.com>,
        "ast@kernel.org" <ast@kernel.org>,
        "daniel@iogearbox.net" <daniel@iogearbox.net>,
        "john.fastabend@gmail.com" <john.fastabend@gmail.com>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        dl-linux-imx <linux-imx@nxp.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "bpf@vger.kernel.org" <bpf@vger.kernel.org>,
        Andrew Lunn <andrew@lunn.ch>
Subject: RE: [PATCH V3 net-next] net: fec: add XDP_TX feature support
Thread-Topic: [PATCH V3 net-next] net: fec: add XDP_TX feature support
Thread-Index: AQHZw3VFF5NSBizeREOxLxVc18wksa/XS8kAgACl6SCAAEHAAIAAOPLAgAAgSoCAAOhtQA==
Date:   Fri, 4 Aug 2023 03:06:11 +0000
Message-ID: <AM5PR04MB3139D8AAAB6B96B58425BBA08809A@AM5PR04MB3139.eurprd04.prod.outlook.com>
References: <20230731060025.3117343-1-wei.fang@nxp.com>
 <20230802104706.5ce541e9@kernel.org>
 <AM5PR04MB313985C61D92E183238809138808A@AM5PR04MB3139.eurprd04.prod.outlook.com>
 <1bf41ea8-5131-7d54-c373-00c1fbcac095@redhat.com>
 <AM5PR04MB31398ABF941EBDD0907E845B8808A@AM5PR04MB3139.eurprd04.prod.outlook.com>
 <cc24e860-7d6f-7ec8-49cb-a49cb066f618@kernel.org>
In-Reply-To: <cc24e860-7d6f-7ec8-49cb-a49cb066f618@kernel.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?YdLhissh+3QIYrLkQ7jUMV1eMLq1Q1yDGsI39Ds0nr2zxRS/ruTFHlCHY7b+?=
 =?us-ascii?Q?XjdQ3/j8L5EVdKgySMxSAo08p7yeQbjTJ/8h24Ss0ELpq2iriDfObHLDRRdw?=
 =?us-ascii?Q?TC7Vyjm3j6I/j+BiyXIGAj1qFRGXnHBG+rMyGpabIHMa8WWmlczy3Mre1KWq?=
 =?us-ascii?Q?r4YtzMDwx72d0tg/WAfiwUil0GgVUbdWHkIQjTa+2hoG/S1xv1WAoGtu5Jsf?=
 =?us-ascii?Q?/9o7MpwbC9mK0dktQVjnwlZtPl6G5jne2ZCgDMqLtuSEBwIkulKzLQJkpVsb?=
 =?us-ascii?Q?KL51nMibyRjI0s6d/asGMOfg0t7Y+FT1G+KS5PPPy1SdA6vb9iGd0QLg+msZ?=
 =?us-ascii?Q?k4jzYwb9wFCjNeUCQl74CfCOd9tdvkOgu5mQ0dec0fh0jHCVXtF3aCr8CXsY?=
 =?us-ascii?Q?d1vzz4KzJrMLKz7frCyZiNMO2LlIJeLHsIzcQVTI4VpZxT6u2Ro0VQWuCx3L?=
 =?us-ascii?Q?DxbANDo93ZFV5SBzKVFf8UJ0GS5g+qj30hxvFK1C11Rn9pISO1Wwi+kCtWjK?=
 =?us-ascii?Q?dhnTLfpKho19sS1X4DJ4Zgj8g5ZUD+JPsweWJ6drrJnzF3/R+bkhG7cj+S6J?=
 =?us-ascii?Q?Pk5FJz/MjE4xdS2GbIwR9Fz4pX/cyrgnFzVw46Q6J38NcRJ9mdySXxwHZAhh?=
 =?us-ascii?Q?KZugJIgx3xxPgcScjcImUEu0Wd0Yil90YclblXmpcvz6/QJLWVhpt7rTrvIT?=
 =?us-ascii?Q?4iHw1F688UF/OaHD/zjWR8cdc5wTeUBLIGWWN3SMknP79qP9lmi7P1kJaa9Y?=
 =?us-ascii?Q?f6TbvJAZTrWFmR+ZUSgZBawSHdpRPukQsXilXKbAJRX3jDSQG2VDUVML/btB?=
 =?us-ascii?Q?CZocEhJgISCeM6odffMaJjv5WJ6WMetbr4s2njer5xmN5TppEBwPLAY66APh?=
 =?us-ascii?Q?rRi8hscbYdzWJDKd2Fc5umQHFoqA72Gkn3EcdeDYXT4vWgqocCgo1LD4ZIME?=
 =?us-ascii?Q?9HV58Gy1J5h/+KSTcQEcomZvJFFFPJjPLbnB+5/UgzBwGkddIyY0EKI3ssUg?=
 =?us-ascii?Q?GIjHExGE8sgbL56Q9xFBjEcOw+fPkEX19sF1PQ1RURv/GKb9V73Z2LZAFTd/?=
 =?us-ascii?Q?rWmIperSTMVFYXRDqHddJTv2YOhfXelVhDspVxVPqw5kfcglMWPPuiIWf+7n?=
 =?us-ascii?Q?wa9WQvbrjDN3Y9DMM3B127e2BO8rII4zW0g4VFdCHucoMItpHkU4KEhOflcI?=
 =?us-ascii?Q?IW3ScBOTfpEeZDUtqTQN+rd2z2cNAURiLNchpr0Nyh7xp2iI1Dw9BnDV/RW+?=
 =?us-ascii?Q?qXE1HEpMZzr4KJr3qdydlbLDlkf+038YQfrKGJGAHngSGOT2Z5V2kt0SfizQ?=
 =?us-ascii?Q?x3rw48llH3UfEgPmtDcYutUQkTLqBKsS+JYrQQmO0QBcLWfeZr65sTCsPNT7?=
 =?us-ascii?Q?NS+iNr9G9wReglYLgLJmg6kYqtnD3n4154SaHnW6rv8AmnY9a1qZKv4GZrFW?=
 =?us-ascii?Q?kKBbh0eIsEYNJzT2QEVo8L9CFgROMvusD5Oa+pR6vIziyQmet0FjauWN0wKV?=
 =?us-ascii?Q?x9zMSc6b3lxg/eZPLTiNFTyIAMydHPu872zZ94J1Pa5HzPBqrdcVk4pxalhr?=
 =?us-ascii?Q?FBsxAFR0WU1mQHZhN4U=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: nxp.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: AM5PR04MB3139.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 921fb525-46e6-4071-c6d6-08db9497c1ee
X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Aug 2023 03:06:11.7387
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: ICpL3GuoD7Do5XrJi4kOhMzJvYlzfT+yJoHgmXrSc/VxFctnzFEMlVpo5teQjWoRVFzknyIz7KB0TGostyvOIw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB8PR04MB6779
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS,T_SCC_BODY_TEXT_LINE,
        URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> > The FEC of i.MX8MP-EVK has dma_coherent=3Dfalse, and as I mentioned
> > above, I did not see an obvious difference in the performance. :(
>=20
> That is surprising - given the results.
>=20
> (see below, lack of perf/diff might be caused by Ethernet flow-control).
>=20
> >
> >>> The result of the current modification.
> >>> root@imx8mpevk:~# ./xdp2 eth0
> >>> proto 17:     260180 pkt/s
> >>
> >> These results are*significantly*  better than reported in patch-1.
> >> What happened?!?
> >>
> > The test environment is slightly different, in patch-1, the FEC port
> > was directly connected to the port of another board. But in the latest
> > test, the ports of the two boards were connected to a switch, so the
> > ports of the two boards are not directly connected.
> >
>=20
> Hmm, I've seen this kind of perf behavior of direct-connected or via swit=
ch
> before. The mistake I made was, that I had not disabled Ethernet flow-con=
trol.
> The xdp2 XDP_TX program will swap the mac addresses, and send the packet
> back to the packet generator (running pktgen), which will get overloaded
> itself and starts sending Ethernet flow-control pause frames.
>=20
> Command line to disable:
>   # ethtool -A eth0 rx off tx off
>=20
> Can I ask/get you to make sure that Ethernet flow-control is disabled (on
> both generator and DUT (to be on safe-side)) and run the test again?
>=20
The flow-control was not disabled before, so according to your suggestion,
I disable the flow-control on the both boards and run the test again, the
performance is slightly improved, but still can not see a clear difference
between the two methods. Below are the results.

Result: use "sync_dma_len" method
root@imx8mpevk:~# ./xdp2 eth0
proto 17:     258886 pkt/s
proto 17:     258879 pkt/s
proto 17:     258872 pkt/s
proto 17:     258312 pkt/s
proto 17:     258926 pkt/s
proto 17:     259057 pkt/s
proto 17:     258437 pkt/s
proto 17:     259242 pkt/s
proto 17:     258665 pkt/s
proto 17:     258779 pkt/s
proto 17:     259209 pkt/s

Result: use the current method
root@imx8mpevk:~# ./xdp2 eth0
proto 17:     258752 pkt/s
proto 17:     258672 pkt/s
proto 17:     258317 pkt/s
proto 17:     258787 pkt/s
proto 17:     258757 pkt/s
proto 17:     258542 pkt/s
proto 17:     258829 pkt/s
proto 17:     258480 pkt/s
proto 17:     258859 pkt/s
proto 17:     258918 pkt/s
proto 17:     258782 pkt/s
proto 17:     259086 pkt/s
proto 17:     258337 pkt/s