Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 20 Apr 2023 22:52:45 +0200
From:   Maciej Fijalkowski <maciej.fijalkowski@intel.com>
To:     Horatiu Vultur <horatiu.vultur@microchip.com>
CC:     <netdev@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        <bpf@vger.kernel.org>, <davem@davemloft.net>,
        <edumazet@google.com>, <kuba@kernel.org>, <pabeni@redhat.com>,
        <daniel@iogearbox.net>, <hawk@kernel.org>,
        <john.fastabend@gmail.com>, <richardcochran@gmail.com>,
        <UNGLinuxDriver@microchip.com>, <alexandr.lobakin@intel.com>
Subject: Re: [PATCH net-next] net: lan966x: Don't use xdp_frame when action
 is XDP_TX
Message-ID: <ZEGmHe2pyxwWiYRL@boxer>
References: <20230420121152.2737625-1-horatiu.vultur@microchip.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230420121152.2737625-1-horatiu.vultur@microchip.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?AWM82YGNhHZCK5vNKq5Ezf8bhg8TDRX5B6cTQKNt9s3wL/TLw9vhOdL6lfsn?=
 =?us-ascii?Q?EU41fcQhVz9s/NVNcllR3XiLuK78Glg+xmPjY7Sa7jkXQE59rKihxjKaWRLr?=
 =?us-ascii?Q?bSxcr4ETcIG8w8B1R7oag4mTJ3rwNGhAs54F8BesYEqOn52R0Ikthv8qKzZq?=
 =?us-ascii?Q?e4XDapYwMA4DgLw+tn6Ppqhg8wwtYlx1psf07L33a8FHvcNxBeO0r1HvhSWH?=
 =?us-ascii?Q?9P8dCG6nRKwvvtjSCgdNwexI5Cv22CZMWgvaXL+xRYT8dxYv1Nb19t2U8pn8?=
 =?us-ascii?Q?tc8J50ahT6JV3ekf7UUcDM/CD0GYtgifX3RE+ePtyNzdRoGJmEjrYCFlhjKa?=
 =?us-ascii?Q?H+ad/WXPcsLOZFcWE9xSnEtaO5EtCEqoAoNKUrGvZBK58M7Q3sWGAdjy5uC/?=
 =?us-ascii?Q?wmg860Hsmpgd1HYYBtF6U9HiHiiV0ZvNa6+fwks9v4kAw+HcoMZv/dX9Ku5N?=
 =?us-ascii?Q?d6QvTAbeqzzrc6J2XNbudfraSwbfobBoBpGTlnwPVj5WhSZk4dlzNppK7Qob?=
 =?us-ascii?Q?llKWS2BjABO5l2+kjear6fYR++q/ouQHRsKP6naGvp6HS6bRoaZ4zevsxR+s?=
 =?us-ascii?Q?NXvUBRsRbDWWKf1jXdEHENs3Lqa9DivubIE88NTLbufqR24ijsNjKi42luoi?=
 =?us-ascii?Q?MJBMT3h8izVdOdKZqPuDnY8jTSW8AY4RUzRIvr98kFq4qvLiZMK9034RaBsG?=
 =?us-ascii?Q?dHDPapJmkR4wTx+CNnQ+7acH6LsPQKFw+9iLnoSU7BEASnuB04pNmoSDYxKp?=
 =?us-ascii?Q?xIKSs3lqRisYay4gRqIo5zdrXIU4VXKgelrJWLse8xU+75yaFCJhlLsBymKr?=
 =?us-ascii?Q?7rCnNJfgOFVBrctPnN20e94WIAF+ew5F5JVuWkFEmNlWG+Gbdfwd0NZOC8bK?=
 =?us-ascii?Q?GYV9Ub2b9MDe31PFdcYB2LqFnR3JDYWBTPyFvefho4uuQdccytVeiyHfzQWq?=
 =?us-ascii?Q?Omhe9BbQJ9lB/VeBOY+Ovm5P/7h+TlH0dzJKSEj2wjtHUQP9emK++K4qzlOD?=
 =?us-ascii?Q?OIBODvjF57sMdzDcoCJoIJktjW8lU4EpGnEFqd8TphiMqqlR4knArs3zLhBi?=
 =?us-ascii?Q?HCQ4BWHg5pOg/tvYMuknltCSkqz/96DGeK1zhNy+SEcQFte0p9NqdHpwunR5?=
 =?us-ascii?Q?4Ae5J4u6FD381najs2E5Lmh/gV08nAzJ+aRVwu//rJBFY3M42Sg8VL8II+XR?=
 =?us-ascii?Q?QJ9yyRRaWx2IlzN3KrCKKZep/bsCVxoOtiA8AZ8YiyTcFn6XyaQqsZtGjpD5?=
 =?us-ascii?Q?JtUYkn0gGlD1S8hgkSh+L+EGa91/ArTcjYM3Sr1/H7QQrMcXwT0MipgeNRCu?=
 =?us-ascii?Q?XMgcVNovGE3DRcbuytVM1dLU3R+a8nKnAPTUsAwHxbNrKZ4Ehvg2dIrUC1V4?=
 =?us-ascii?Q?CJcL5ccZjINGSPBwQ2E1Kyl5D7qKA3Vemkgl/dcC8Rs4c4k03hrzJlnR5Lz3?=
 =?us-ascii?Q?HAHQ2mLz4+uMselmr9Ygq54zxryqOK7dJPrMojT/0DO6ZRXS+7OPYOrr3kyZ?=
 =?us-ascii?Q?KGV4YwfBp95C6iUjdCqNh0I8jv5XtqbWfJHHzzX8+6wJd88/LvMcFEXJr72H?=
 =?us-ascii?Q?Z4lOPaKO9hpLKgBe01OHlkPDfpWr8R5qVOUUVzUjA+2JpzZ3+XTmhNVGr/ET?=
 =?us-ascii?Q?DA=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: c6d26caf-bfb1-4bf0-9d90-08db41e13954
X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB6117.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Apr 2023 20:52:58.9832
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: rjt+sXYPAn676HtjqAi3Wt3HgYdiBx1P1ycU2xHxQ+GKqJkHDxmQuv9PEGVZEBwL6JtMw6LDE+0d4r5FjC0J7Irv6Znbl+vqclswF0uwPTQ=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR11MB5027
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Apr 20, 2023 at 02:11:52PM +0200, Horatiu Vultur wrote:

'net: ' in patch subject is excessive to me

> When the action of an xdp program was XDP_TX, lan966x was creating
> a xdp_frame and use this one to send the frame back. But it is also
> possible to send back the frame without needing a xdp_frame, because
> it possible to send it back using the page.

s/it/it is

> And then once the frame is transmitted is possible to use directly
> page_pool_recycle_direct as lan966x is using page pools.
> This would save some CPU usage on this path.

i remember this optimization gave me noticeable perf improvement, would
you mind sharing it in % on your side?

> 
> Signed-off-by: Horatiu Vultur <horatiu.vultur@microchip.com>
> ---
>  .../ethernet/microchip/lan966x/lan966x_fdma.c | 35 +++++++++++--------
>  .../ethernet/microchip/lan966x/lan966x_main.h |  2 ++
>  .../ethernet/microchip/lan966x/lan966x_xdp.c  | 11 +++---
>  3 files changed, 27 insertions(+), 21 deletions(-)
> 
> diff --git a/drivers/net/ethernet/microchip/lan966x/lan966x_fdma.c b/drivers/net/ethernet/microchip/lan966x/lan966x_fdma.c
> index 2ed76bb61a731..7947259e67e4e 100644
> --- a/drivers/net/ethernet/microchip/lan966x/lan966x_fdma.c
> +++ b/drivers/net/ethernet/microchip/lan966x/lan966x_fdma.c
> @@ -390,6 +390,7 @@ static void lan966x_fdma_stop_netdev(struct lan966x *lan966x)
>  static void lan966x_fdma_tx_clear_buf(struct lan966x *lan966x, int weight)
>  {
>  	struct lan966x_tx *tx = &lan966x->tx;
> +	struct lan966x_rx *rx = &lan966x->rx;
>  	struct lan966x_tx_dcb_buf *dcb_buf;
>  	struct xdp_frame_bulk bq;
>  	struct lan966x_db *db;
> @@ -432,7 +433,8 @@ static void lan966x_fdma_tx_clear_buf(struct lan966x *lan966x, int weight)
>  			if (dcb_buf->xdp_ndo)
>  				xdp_return_frame_bulk(dcb_buf->data.xdpf, &bq);
>  			else
> -				xdp_return_frame_rx_napi(dcb_buf->data.xdpf);
> +				page_pool_recycle_direct(rx->page_pool,
> +							 dcb_buf->data.page);
>  		}
>  
>  		clear = true;
> @@ -702,6 +704,7 @@ static void lan966x_fdma_tx_start(struct lan966x_tx *tx, int next_to_use)
>  int lan966x_fdma_xmit_xdpf(struct lan966x_port *port,
>  			   struct xdp_frame *xdpf,
>  			   struct page *page,
> +			   u32 len,

agreed with Olek regarding arguments reduction here

>  			   bool dma_map)
>  {
>  	struct lan966x *lan966x = port->lan966x;
> @@ -722,6 +725,15 @@ int lan966x_fdma_xmit_xdpf(struct lan966x_port *port,
>  		goto out;
>  	}
>  
> +	/* Fill up the buffer */
> +	next_dcb_buf = &tx->dcbs_buf[next_to_use];
> +	next_dcb_buf->use_skb = false;
> +	next_dcb_buf->xdp_ndo = dma_map;

a bit misleading that xdp_ndo is a bool :P

> +	next_dcb_buf->len = len + IFH_LEN_BYTES;
> +	next_dcb_buf->used = true;
> +	next_dcb_buf->ptp = false;
> +	next_dcb_buf->dev = port->dev;
> +
>  	/* Generate new IFH */
>  	if (dma_map) {
>  		if (xdpf->headroom < IFH_LEN_BYTES) {
> @@ -736,16 +748,18 @@ int lan966x_fdma_xmit_xdpf(struct lan966x_port *port,
>  
>  		dma_addr = dma_map_single(lan966x->dev,
>  					  xdpf->data - IFH_LEN_BYTES,
> -					  xdpf->len + IFH_LEN_BYTES,
> +					  len + IFH_LEN_BYTES,
>  					  DMA_TO_DEVICE);
>  		if (dma_mapping_error(lan966x->dev, dma_addr)) {
>  			ret = NETDEV_TX_OK;
>  			goto out;
>  		}
>  
> +		next_dcb_buf->data.xdpf = xdpf;
> +
>  		/* Setup next dcb */
>  		lan966x_fdma_tx_setup_dcb(tx, next_to_use,
> -					  xdpf->len + IFH_LEN_BYTES,
> +					  len + IFH_LEN_BYTES,
>  					  dma_addr);
>  	} else {
>  		ifh = page_address(page) + XDP_PACKET_HEADROOM;
> @@ -756,25 +770,18 @@ int lan966x_fdma_xmit_xdpf(struct lan966x_port *port,
>  		dma_addr = page_pool_get_dma_addr(page);
>  		dma_sync_single_for_device(lan966x->dev,
>  					   dma_addr + XDP_PACKET_HEADROOM,
> -					   xdpf->len + IFH_LEN_BYTES,
> +					   len + IFH_LEN_BYTES,
>  					   DMA_TO_DEVICE);
>  
> +		next_dcb_buf->data.page = page;
> +
>  		/* Setup next dcb */
>  		lan966x_fdma_tx_setup_dcb(tx, next_to_use,
> -					  xdpf->len + IFH_LEN_BYTES,
> +					  len + IFH_LEN_BYTES,
>  					  dma_addr + XDP_PACKET_HEADROOM);
>  	}
>  
> -	/* Fill up the buffer */
> -	next_dcb_buf = &tx->dcbs_buf[next_to_use];
> -	next_dcb_buf->use_skb = false;
> -	next_dcb_buf->data.xdpf = xdpf;
> -	next_dcb_buf->xdp_ndo = dma_map;
> -	next_dcb_buf->len = xdpf->len + IFH_LEN_BYTES;
>  	next_dcb_buf->dma_addr = dma_addr;
> -	next_dcb_buf->used = true;
> -	next_dcb_buf->ptp = false;
> -	next_dcb_buf->dev = port->dev;
>  
>  	/* Start the transmission */
>  	lan966x_fdma_tx_start(tx, next_to_use);
> diff --git a/drivers/net/ethernet/microchip/lan966x/lan966x_main.h b/drivers/net/ethernet/microchip/lan966x/lan966x_main.h
> index 851afb0166b19..59da35a2c93d4 100644
> --- a/drivers/net/ethernet/microchip/lan966x/lan966x_main.h
> +++ b/drivers/net/ethernet/microchip/lan966x/lan966x_main.h
> @@ -243,6 +243,7 @@ struct lan966x_tx_dcb_buf {
>  	union {
>  		struct sk_buff *skb;
>  		struct xdp_frame *xdpf;
> +		struct page *page;
>  	} data;
>  	u32 len;
>  	u32 used : 1;
> @@ -544,6 +545,7 @@ int lan966x_fdma_xmit(struct sk_buff *skb, __be32 *ifh, struct net_device *dev);
>  int lan966x_fdma_xmit_xdpf(struct lan966x_port *port,
>  			   struct xdp_frame *frame,
>  			   struct page *page,
> +			   u32 len,
>  			   bool dma_map);
>  int lan966x_fdma_change_mtu(struct lan966x *lan966x);
>  void lan966x_fdma_netdev_init(struct lan966x *lan966x, struct net_device *dev);
> diff --git a/drivers/net/ethernet/microchip/lan966x/lan966x_xdp.c b/drivers/net/ethernet/microchip/lan966x/lan966x_xdp.c
> index 2e6f486ec67d7..a8ad1f4e431cb 100644
> --- a/drivers/net/ethernet/microchip/lan966x/lan966x_xdp.c
> +++ b/drivers/net/ethernet/microchip/lan966x/lan966x_xdp.c
> @@ -62,7 +62,7 @@ int lan966x_xdp_xmit(struct net_device *dev,
>  		struct xdp_frame *xdpf = frames[i];
>  		int err;
>  
> -		err = lan966x_fdma_xmit_xdpf(port, xdpf, NULL, true);
> +		err = lan966x_fdma_xmit_xdpf(port, xdpf, NULL, xdpf->len, true);
>  		if (err)
>  			break;
>  
> @@ -76,7 +76,6 @@ int lan966x_xdp_run(struct lan966x_port *port, struct page *page, u32 data_len)
>  {
>  	struct bpf_prog *xdp_prog = port->xdp_prog;
>  	struct lan966x *lan966x = port->lan966x;
> -	struct xdp_frame *xdpf;
>  	struct xdp_buff xdp;
>  	u32 act;
>  
> @@ -90,11 +89,9 @@ int lan966x_xdp_run(struct lan966x_port *port, struct page *page, u32 data_len)
>  	case XDP_PASS:
>  		return FDMA_PASS;
>  	case XDP_TX:
> -		xdpf = xdp_convert_buff_to_frame(&xdp);
> -		if (!xdpf)
> -			return FDMA_DROP;
> -
> -		return lan966x_fdma_xmit_xdpf(port, xdpf, page, false) ?
> +		return lan966x_fdma_xmit_xdpf(port, NULL, page,
> +					      data_len - IFH_LEN_BYTES,
> +					      false) ?
>  		       FDMA_DROP : FDMA_TX;
>  	case XDP_REDIRECT:
>  		if (xdp_do_redirect(port->dev, &xdp, xdp_prog))
> -- 
> 2.38.0
>