Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <422e4a51-4cec-5772-70f5-1019789acd18@intel.com>
Date:   Tue, 11 Apr 2023 16:10:51 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.9.1
Subject: Re: [PATCH V2,net-next, 2/3] net: mana: Enable RX path to handle
 various MTU sizes
Content-Language: en-US
To:     Haiyang Zhang <haiyangz@microsoft.com>,
        <linux-hyperv@vger.kernel.org>, <netdev@vger.kernel.org>
CC:     <decui@microsoft.com>, <kys@microsoft.com>,
        <paulros@microsoft.com>, <olaf@aepfle.de>, <vkuznets@redhat.com>,
        <davem@davemloft.net>, <wei.liu@kernel.org>, <edumazet@google.com>,
        <kuba@kernel.org>, <pabeni@redhat.com>, <leon@kernel.org>,
        <longli@microsoft.com>, <ssengar@linux.microsoft.com>,
        <linux-rdma@vger.kernel.org>, <daniel@iogearbox.net>,
        <john.fastabend@gmail.com>, <bpf@vger.kernel.org>,
        <ast@kernel.org>, <sharmaajay@microsoft.com>, <hawk@kernel.org>,
        <linux-kernel@vger.kernel.org>
References: <1680901196-20643-1-git-send-email-haiyangz@microsoft.com>
 <1680901196-20643-3-git-send-email-haiyangz@microsoft.com>
From:   Jacob Keller <jacob.e.keller@intel.com>
In-Reply-To: <1680901196-20643-3-git-send-email-haiyangz@microsoft.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?eisreDNzT2lKZ1ZrMSsvMFFuQVJCeURueURKRjZqU1hBRUs4aTh0bjZiQlBm?=
 =?utf-8?B?QURSNzlNaUJFVGZZeEVnbjA5YW4xYjJMSXlwTmx4ejl6aFlWN0NZQXprb2dm?=
 =?utf-8?B?Y3hBT29DWVdkc0k3R3RIa2swMDdBMmRNdUoyWUVobTdRUjRnTlh3SThVNTdV?=
 =?utf-8?B?RHNpWTgyVytYb3drOEIzNGNsRmp3bG5abVR3dUgzT1RSTFcwQ3dvZFoyNERs?=
 =?utf-8?B?T0FLZCtJbjEwWXVzRitUQ2o2c3Z1T0k0RGREU3NRejZQM29qZXpwTi9HYXRt?=
 =?utf-8?B?cEM4Vzg3aHBONXptVzk1c0hxUWNtdGt1NVNCZlNzQUUxQWgwSG16RmFCWHFk?=
 =?utf-8?B?bWFsWHpWUGxLakRBMWV2cVA3UVo4eWFCK2dMRm1zSjhlbGhKdDRJbGI5TEg2?=
 =?utf-8?B?VVlGRVhvYUk2d09NUFFqbXVHSmk3TUk2Y2JJZktmeXR1aDdhMFNyVHAycHdP?=
 =?utf-8?B?dVlIZk91ZFErbG5CeHF4bjdnNkJocU5yQVBIeWRydVY5SzY0WDI3NFo3eTNu?=
 =?utf-8?B?c2xPVHNYNzdVRnMva3dkcjcxVTZqRHZKTUt0Zm1iV3BySmNWMkJLcURNejhn?=
 =?utf-8?B?ZU5XSGErNHNXU3FSL2FyU0tVNDZCS1o2R3J5bzdoQmFsZkxMeEdPY2dRcEdp?=
 =?utf-8?B?UncyY2YrcS9uUGpaZGNPbk1WTU1JWENEbWZWTVdQQmZjbWs4bE1QeDJHMVBm?=
 =?utf-8?B?US9JOW9YbDdSQjJld01SVHJlNzI5cUhrcEp5cExNdHpoOFlOWnNLYlBxMWtU?=
 =?utf-8?B?MStQdGZsZ1g5RWI5VXRnWkZjMjFGY0Z4SW50WGJKYlJRTVhpVHQ1c2cySU9E?=
 =?utf-8?B?ZmdUQnZNSjdEZ3l4a3lkdzU3NmJZaktXak9SVHdjYzJDeXppWE4xaTN4cERx?=
 =?utf-8?B?QVR5VDBUMW9nc2k1N01RS2pJSHpQTzk4YzVhRHQ5OTJxL09FcG5BcGx5M1p4?=
 =?utf-8?B?UmxFWVNrYXBXZlBYb0EyaURRZzlSSnJ6TTdqUmdVdE9aMWxzVHkzRTJFY0xL?=
 =?utf-8?B?QkYvd3JTTUl3REtndzRTN1dzc0FVWUdvT3kwYXZKazJneDdjZmFXd2YrREIw?=
 =?utf-8?B?R3B2bEgzZEFYSzBBRGFId0Uya0pUMzZ0T2ZMSDNzSXM2Y3pHYm1VaUp1dkNH?=
 =?utf-8?B?MHZ5dXkva3pxNUpxYVY0K2FMb3E1dG85K3VXU1BWS2pKbVVlUVJNKzdQUUJr?=
 =?utf-8?B?M2NJMjlid1lQL1NrczZzWkt3djZyYWp3SzAreW1KUnZIbWhhV0c2OW45dGtF?=
 =?utf-8?B?emNLZWlncVNuWUNTNFVXa2JSUHZMQ1g4Wk1MV3BOYTRCT25XWTFtdGFCS1lz?=
 =?utf-8?B?WmJUdlRvVURBMEw1RkRVT0pOcjJGZHk5aEYxdGVaUTJTdDVBYitMZWx1cnpi?=
 =?utf-8?B?Sm1NWVFtVUEzNjdLd013d0NyRitKM1ExZHpLaW1qUkRqYVZWcytiUmY3ZSs4?=
 =?utf-8?B?Vkx2bWRxR2FoZkFlZUhITk5iQVg1L1Y0RzlzSmVZMDJPNE5YaTZDdUdLeUcr?=
 =?utf-8?B?a0ZMSWIzQkp5TmZRY0FUTFBTRUF1UEZ6eU9HWWlNV2hSZUF6em03SlVCUmVT?=
 =?utf-8?B?aTJHbmxrUEZvMXUybTErOEM0elhvaWt5OWVqVlVwMXdJOHBNcktjNzRJQS9H?=
 =?utf-8?B?clRIMW9zekZuNlRxbFh2VXR4cUg4VjhKUHdsRDNmQUZyeDhHUnIvVStNWW5O?=
 =?utf-8?B?NWtTSjB5NURjb1RPVDZDWG9zUzZSckRXcjRqZnc1VkE2WHlSb05Vc0JmcFVj?=
 =?utf-8?B?Q081SmZSN0pCY2tIM3dqVkVBQ1NxTXNuVk5RR0laQ24zQ2FBd1F4ZTVLTU83?=
 =?utf-8?B?cW0wS2FndmtGQTYyTmJFQld2cXlQdGNyWEh4SWpDMEpvR2JJd2FVdmhVajlk?=
 =?utf-8?B?OUQ2dzJoU0ZnK1ZsYlpTV3lEVlUwUVR1Q01aanU3Wk1Gb2xuSk9oVUtCcEtz?=
 =?utf-8?B?TDRmdFNlVDBhemhqWTBiSE0relgrdUVaYTNFeEl6YVpOUk9HMWI4VWhZOFp4?=
 =?utf-8?B?NG81bE9rZTlhL1BsSmJVVVloZUNBSUdmSGNqMVRkcHk3QWd1dlUvdmc5bXds?=
 =?utf-8?B?QStyZENyK0ZLWWFBOWFwelIrQlFac29WS1l2UHlEZFc0ZHBTbzJwdUs0WU83?=
 =?utf-8?B?d0xBcnBOYVN2cE93STVxaXJpUHZJZkptREsxK1VVa29TNWJhb3paRGlEMmRO?=
 =?utf-8?B?Ymc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 1148a386-1d66-4302-2da4-08db3ae1fbdb
X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB5089.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Apr 2023 23:10:47.3908
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 9n1kCzci4bdyURBhgAHK2XCMvsw3d7zxI2VZR+J+fsmGBS95LKd4DXklCug5yEZh97rQeM25npZzYODM1ppOReMV424fAlBFHMm4tkRETEU=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB8223
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-6.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 4/7/2023 1:59 PM, Haiyang Zhang wrote:
> Update RX data path to allocate and use RX queue DMA buffers with
> proper size based on potentially various MTU sizes.
> 
> Signed-off-by: Haiyang Zhang <haiyangz@microsoft.com>
> 
> ---
> V2:
> Refectored to multiple patches for readability. Suggested by Yunsheng Lin.
> 
> ---
>  drivers/net/ethernet/microsoft/mana/mana_en.c | 188 +++++++++++-------
>  include/net/mana/mana.h                       |  13 +-
>  2 files changed, 124 insertions(+), 77 deletions(-)
> 
> diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c
> index 112c642dc89b..e5d5dea763f2 100644
> --- a/drivers/net/ethernet/microsoft/mana/mana_en.c
> +++ b/drivers/net/ethernet/microsoft/mana/mana_en.c
> @@ -1185,10 +1185,10 @@ static void mana_post_pkt_rxq(struct mana_rxq *rxq)
>  	WARN_ON_ONCE(recv_buf_oob->wqe_inf.wqe_size_in_bu != 1);
>  }
>  
> -static struct sk_buff *mana_build_skb(void *buf_va, uint pkt_len,
> -				      struct xdp_buff *xdp)
> +static struct sk_buff *mana_build_skb(struct mana_rxq *rxq, void *buf_va,
> +				      uint pkt_len, struct xdp_buff *xdp)
>  {
> -	struct sk_buff *skb = napi_build_skb(buf_va, PAGE_SIZE);
> +	struct sk_buff *skb = napi_build_skb(buf_va, rxq->alloc_size);
>  
>  	if (!skb)
>  		return NULL;
> @@ -1196,11 +1196,12 @@ static struct sk_buff *mana_build_skb(void *buf_va, uint pkt_len,
>  	if (xdp->data_hard_start) {
>  		skb_reserve(skb, xdp->data - xdp->data_hard_start);
>  		skb_put(skb, xdp->data_end - xdp->data);
> -	} else {
> -		skb_reserve(skb, XDP_PACKET_HEADROOM);
> -		skb_put(skb, pkt_len);
> +		return skb;
>  	}
>  
> +	skb_reserve(skb, rxq->headroom);
> +	skb_put(skb, pkt_len);
> +
>  	return skb;
>  }
>  
> @@ -1233,7 +1234,7 @@ static void mana_rx_skb(void *buf_va, struct mana_rxcomp_oob *cqe,
>  	if (act != XDP_PASS && act != XDP_TX)
>  		goto drop_xdp;
>  
> -	skb = mana_build_skb(buf_va, pkt_len, &xdp);
> +	skb = mana_build_skb(rxq, buf_va, pkt_len, &xdp);
>  
>  	if (!skb)
>  		goto drop;
> @@ -1282,14 +1283,72 @@ static void mana_rx_skb(void *buf_va, struct mana_rxcomp_oob *cqe,
>  	u64_stats_update_end(&rx_stats->syncp);
>  
>  drop:
> -	WARN_ON_ONCE(rxq->xdp_save_page);
> -	rxq->xdp_save_page = virt_to_page(buf_va);
> +	WARN_ON_ONCE(rxq->xdp_save_va);
> +	/* Save for reuse */
> +	rxq->xdp_save_va = buf_va;
>  
>  	++ndev->stats.rx_dropped;
>  
>  	return;
>  }
>  
> +static void *mana_get_rxfrag(struct mana_rxq *rxq, struct device *dev,
> +			     dma_addr_t *da, bool is_napi)
> +{
> +	struct page *page;
> +	void *va;
> +
> +	/* Reuse XDP dropped page if available */
> +	if (rxq->xdp_save_va) {
> +		va = rxq->xdp_save_va;
> +		rxq->xdp_save_va = NULL;
> +	} else if (rxq->alloc_size > PAGE_SIZE) {
> +		if (is_napi)
> +			va = napi_alloc_frag(rxq->alloc_size);
> +		else
> +			va = netdev_alloc_frag(rxq->alloc_size);
> +
> +		if (!va)
> +			return NULL;
> +	} else {
> +		page = dev_alloc_page();
> +		if (!page)
> +			return NULL;
> +
> +		va = page_to_virt(page);
> +	}
> +
> +	*da = dma_map_single(dev, va + rxq->headroom, rxq->datasize,
> +			     DMA_FROM_DEVICE);
> +
> +	if (dma_mapping_error(dev, *da)) {
> +		put_page(virt_to_head_page(va));
> +		return NULL;
> +	}
> +
> +	return va;
> +}
> +
> +/* Allocate frag for rx buffer, and save the old buf */
> +static void mana_refill_rxoob(struct device *dev, struct mana_rxq *rxq,
> +			      struct mana_recv_buf_oob *rxoob, void **old_buf)
> +{
> +	dma_addr_t da;
> +	void *va;
> +
> +	va = mana_get_rxfrag(rxq, dev, &da, true);
> +
> +	if (!va)
> +		return;
> +
> +	dma_unmap_single(dev, rxoob->sgl[0].address, rxq->datasize,
> +			 DMA_FROM_DEVICE);
> +	*old_buf = rxoob->buf_va;
> +
> +	rxoob->buf_va = va;
> +	rxoob->sgl[0].address = da;
> +}
> +

So you're pulling out these functions from the code below, which is
good, but it makes it hard to tell what code actually changed.

>  static void mana_process_rx_cqe(struct mana_rxq *rxq, struct mana_cq *cq,
>  				struct gdma_comp *cqe)
>  {
> @@ -1299,10 +1358,8 @@ static void mana_process_rx_cqe(struct mana_rxq *rxq, struct mana_cq *cq,
>  	struct mana_recv_buf_oob *rxbuf_oob;
>  	struct mana_port_context *apc;
>  	struct device *dev = gc->dev;
> -	void *new_buf, *old_buf;
> -	struct page *new_page;
> +	void *old_buf = NULL;
>  	u32 curr, pktlen;
> -	dma_addr_t da;
>  
>  	apc = netdev_priv(ndev);
>  
> @@ -1345,40 +1402,11 @@ static void mana_process_rx_cqe(struct mana_rxq *rxq, struct mana_cq *cq,
>  	rxbuf_oob = &rxq->rx_oobs[curr];
>  	WARN_ON_ONCE(rxbuf_oob->wqe_inf.wqe_size_in_bu != 1);
>  
> -	/* Reuse XDP dropped page if available */
> -	if (rxq->xdp_save_page) {
> -		new_page = rxq->xdp_save_page;
> -		rxq->xdp_save_page = NULL;
> -	} else {
> -		new_page = alloc_page(GFP_ATOMIC);
> -	}
> -
> -	if (new_page) {
> -		da = dma_map_page(dev, new_page, XDP_PACKET_HEADROOM, rxq->datasize,
> -				  DMA_FROM_DEVICE);
> -
> -		if (dma_mapping_error(dev, da)) {
> -			__free_page(new_page);
> -			new_page = NULL;
> -		}
> -	}
> -
> -	new_buf = new_page ? page_to_virt(new_page) : NULL;
> -
> -	if (new_buf) {
> -		dma_unmap_page(dev, rxbuf_oob->buf_dma_addr, rxq->datasize,
> -			       DMA_FROM_DEVICE);
> -
> -		old_buf = rxbuf_oob->buf_va;
> -
> -		/* refresh the rxbuf_oob with the new page */
> -		rxbuf_oob->buf_va = new_buf;
> -		rxbuf_oob->buf_dma_addr = da;
> -		rxbuf_oob->sgl[0].address = rxbuf_oob->buf_dma_addr;
> -	} else {
> -		old_buf = NULL; /* drop the packet if no memory */
> -	}

Could you do this split into helper functions first in a separate change
before adding support for handling various MTU size?

Doing it that way would make it much easier to review what actually
changes in that block of code.

Thanks,
Jake