Received-SPF: pass (google.com: domain of linux-kernel+bounces-132833-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Message-ID: <45eb2bf1-e7b0-4045-82b3-93b9f81b7988@intel.com>
Date: Fri, 5 Apr 2024 12:32:55 +0200
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH net-next v9 7/9] libeth: add Rx buffer management
Content-Language: en-US
To: Alexander Lobakin <aleksander.lobakin@intel.com>, "David S. Miller"
	<davem@davemloft.net>, Eric Dumazet <edumazet@google.com>, Jakub Kicinski
	<kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>
CC: Alexander Duyck <alexanderduyck@fb.com>, Yunsheng Lin
	<linyunsheng@huawei.com>, Jesper Dangaard Brouer <hawk@kernel.org>, "Ilias
 Apalodimas" <ilias.apalodimas@linaro.org>, Christoph Lameter <cl@linux.com>,
	Vlastimil Babka <vbabka@suse.cz>, Andrew Morton <akpm@linux-foundation.org>,
	<nex.sw.ncis.osdt.itp.upstreaming@intel.com>, <netdev@vger.kernel.org>,
	<intel-wired-lan@lists.osuosl.org>, <linux-mm@kvack.org>,
	<linux-kernel@vger.kernel.org>
References: <20240404154402.3581254-1-aleksander.lobakin@intel.com>
 <20240404154402.3581254-8-aleksander.lobakin@intel.com>
From: Przemek Kitszel <przemyslaw.kitszel@intel.com>
In-Reply-To: <20240404154402.3581254-8-aleksander.lobakin@intel.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?VXFSZ3ZhNHFwb3piaDBhdWtLWkxOdTd4d3BLaWdocjl6Z3VQRlBUUlEzemtR?=
 =?utf-8?B?VWJXVElFd2g2Sndpa1NCU0VrQ1NzbThERmtFZnVPWFFlMG9naHFldzNJOUdC?=
 =?utf-8?B?VHlvb2xQVTBJYms5Skd0RUxLUlVqcjdxNExuYTNnNzBrNzFNKzlRckJ1eThW?=
 =?utf-8?B?Mk9ERzg1QWpuYzhuYnlhcitSZ1pvemkwSjhCUm1uM2tVMzFtWmRPdU5XNVgv?=
 =?utf-8?B?bHJra1VkUVJ1WDk2eGxLVHNySjZ5bVlhS3p2MWVzcllsM0pwdm9Ec2lPNDVU?=
 =?utf-8?B?YjFKcGQxMkwwVlFiVnZQZUxudHBPTzRGSUptejRJOHYvTjl5QllzVXJzdVFl?=
 =?utf-8?B?bU5NbUc4MDBoV1VZTTMrWW9oeW8xR0dnY2J6SU1MTVJUR3lwb2tUT05LUjR6?=
 =?utf-8?B?SXpnOGRZdDgyU28rWENLOTFGTzZreWV1eE9nR0lvK2dVWTVaUE94TkJra3Rl?=
 =?utf-8?B?MFlsQzlUd09vRjRBZUlob3hmZEVlVEIyZVFBOXFRQjAyN1Q5SEpkOUgxa3Js?=
 =?utf-8?B?KzZ4ZmoreGh0V3RvZHhxSGtHU3R2S3Q3dnlBcGhML0NIRFBBNE9jekRlOU1Y?=
 =?utf-8?B?Q1QvUnUrQ3E5alNoVm9RTm9rR2Y0MVJXMTRvbkowTEtRcTV0SUVzTTd1VUM4?=
 =?utf-8?B?KzFzcnI4SEx0VEpJVDRZNXd5RkV3cnphajlqZ2Y1cTh6a3JSeUdmaUVyakxC?=
 =?utf-8?B?RWh6cTNaVkhOeEtLK2R2MEpTQWxGWW01dFJnOWxMYjJyeTZiUFFxYkU1dkJR?=
 =?utf-8?B?WDJXUEtXdWlLOFJGNGExaVJ6Q0xjTXNPU3BZc2R4VjdrUnFHSlp5VGFiNXcw?=
 =?utf-8?B?WkdOTjhGTzZudDIrYjVRUm1mM2NmTnNtLzZlb2pXaTBQME84UmpISGN4N0pq?=
 =?utf-8?B?NnVrU0tBK05yS1ZkQmE3YnRzdUZCQyt5NTFkWTdld0txOCtydTRSWlppWEFL?=
 =?utf-8?B?M1BjKy9aOXVySXlOQlZoT1hhYXNNc09Cb3N4R1VXRjg0bWo2Y0MvZnMzcmFD?=
 =?utf-8?B?SG9TQ3JVT1pET3NEbk00SGQxTnAvWCtmWWxNTndaSzNOcldmdW5BWXl4T3dB?=
 =?utf-8?B?Q1BCVjBvQ3d6aGZKZWQ5OGVTeHVnQi8zWktWK1dwNzlDNHFzaUJTdkltclNa?=
 =?utf-8?B?ZFE5dlZmeWJ1TXJVNktYZFpPK0xEcXh1NEdpL3prK2cwdS9UdDBFUGJvTk04?=
 =?utf-8?B?WWtMVm82NVRxNWhEM3pKYThNUGdtdG44WW1ESTF0cGlUTVd0ODliUHhvNHNu?=
 =?utf-8?B?OFNLcG44V0pUSGJybkN2ZzdUVERvMTBLQ2tjdHdkVmZQSW5CMUFENC9SMjlx?=
 =?utf-8?B?Ynlub2dYMnA3dzVQbGNxYy95RlZqSnFFc051dmRLSTBtWENMN1Q5MjdLZ0tx?=
 =?utf-8?B?UndBR0RWejUxNWl6VTg4ZENCQnRrMXN4SVYvd3UyWURoRFQwdElRQ3dEMDBs?=
 =?utf-8?B?dUVHR3Q5U3RBRlo4K2N4UmlvcGNDQk1udzhGOFFpY0xMQXA5Q01ZNlF2WURJ?=
 =?utf-8?B?V3JvUGRpYUJJWWFZZ25raTkvK1djWEx3UVVoMnBZQXVqN1BXR09FWkRGUXFH?=
 =?utf-8?B?VTc0bkZZYVY5eXJmQ0YzZnVuRTI2QmFtUXhGcnI1Y1V4aFd5a2swYUsvYXVC?=
 =?utf-8?B?b2k1RkVRZzdFMjlFS2F1NnN0RXdPNFR6dVFFNDBpbldPbTVYTE4xVk9BeG5j?=
 =?utf-8?B?V3VzTHBqbUgyajdYMkxkOEo3aXFKU1RqWmJxTG1Cc1lJNktTL2FOVnc5S3pM?=
 =?utf-8?B?RldUdDdpWGpkNkJ4SE53eFhnVTVtYTltdzVXK3d2azNMOWpEajNqMTBOKytC?=
 =?utf-8?B?ME1DL0txRWV0Q0JMV3pOMUZUazRCdHMvb2owaWRDdU5ubHJ4ZlZDVWlaRmJ1?=
 =?utf-8?B?QkZzcWc2SWhyMjNjWExuUWVmeFZPU2ZZR21nYkRhaVk3TkRZNUswZEZSekw0?=
 =?utf-8?B?VjdvZ2czbVVleDRqNVVRYk50M1JIQnh1RGo4SHM1cWYrVjRHdis3WUVSczlq?=
 =?utf-8?B?dnlFWm96WlRZME5sNCtSdHlObHBEcjE2TUpXNlpVT0tPUEhkcGVOdHhlUFEz?=
 =?utf-8?B?VGE2SHA0OS9yVnlEMllXcjkrMjVVVllYY0lXTzYralU5R1RENDZKOStZWk5B?=
 =?utf-8?B?QmFQaFdBbzRtTFlNRFJTR1cvVGxZWERhUEdmQThvTEhMbklGdzdLSDZra0Vk?=
 =?utf-8?B?Wmc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 6af062ad-526f-49e4-c7fb-08dc555bc463
X-MS-Exchange-CrossTenant-AuthSource: MN6PR11MB8102.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Apr 2024 10:33:00.6613
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: QGKESn/lpiroR7xcpBtALyEOnhzU8HHw+gy7Ab5Oi9FPESE+Cu6U7q/uvwlhzMSA/zouKQEJi0BYJQtiBLSnAaNkjnMXRkixTJxkaMWuV40=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYXPR11MB8662
X-OriginatorOrg: intel.com

On 4/4/24 17:44, Alexander Lobakin wrote:
> Add a couple intuitive helpers to hide Rx buffer implementation details
> in the library and not multiplicate it between drivers. The settings are
> sorta optimized for 100G+ NICs, but nothing really HW-specific here.
> Use the new page_pool_dev_alloc() to dynamically switch between
> split-page and full-page modes depending on MTU, page size, required
> headroom etc. For example, on x86_64 with the default driver settings
> each page is shared between 2 buffers. Turning on XDP (not in this
> series) -> increasing headroom requirement pushes truesize out of 2048
> boundary, leading to that each buffer starts getting a full page.
> The "ceiling" limit is %PAGE_SIZE, as only order-0 pages are used to
> avoid compound overhead. For the above architecture, this means maximum
> linear frame size of 3712 w/o XDP.
> Not that &libeth_buf_queue is not a complete queue/ring structure for
> now, rather a shim, but eventually the libeth-enabled drivers will move
> to it, with iavf being the first one.
> 
> Signed-off-by: Alexander Lobakin <aleksander.lobakin@intel.com>
> ---
>   drivers/net/ethernet/intel/libeth/Kconfig |   1 +
>   include/net/libeth/rx.h                   | 117 ++++++++++++++++++++++
>   drivers/net/ethernet/intel/libeth/rx.c    |  98 ++++++++++++++++++
>   3 files changed, 216 insertions(+)
>
[...]

> +/**
> + * struct libeth_fqe - structure representing an Rx buffer
> + * @page: page holding the buffer
> + * @offset: offset from the page start (to the headroom)
> + * @truesize: total space occupied by the buffer (w/ headroom and tailroom)
> + *
> + * Depending on the MTU, API switches between one-page-per-frame and shared
> + * page model (to conserve memory on bigger-page platforms). In case of the
> + * former, @offset is always 0 and @truesize is always ```PAGE_SIZE```.
> + */
> +struct libeth_fqe {
> +	struct page		*page;
> +	u32			offset;
> +	u32			truesize;
> +} __aligned_largest;
> +
> +/**
> + * struct libeth_fq - structure representing a buffer queue
> + * @fp: hotpath part of the structure
> + * @pp: &page_pool for buffer management
> + * @fqes: array of Rx buffers
> + * @truesize: size to allocate per buffer, w/overhead
> + * @count: number of descriptors/buffers the queue has
> + * @buf_len: HW-writeable length per each buffer
> + * @nid: ID of the closest NUMA node with memory
> + */
> +struct libeth_fq {
> +	struct_group_tagged(libeth_fq_fp, fp,
> +		struct page_pool	*pp;
> +		struct libeth_fqe	*fqes;
> +
> +		u32			truesize;
> +		u32			count;
> +	);
> +
> +	/* Cold fields */
> +	u32			buf_len;
> +	int			nid;
> +};

[...]

Could you please unpack the meaning of `fq` and `fqe` acronyms here?

otherwise the whole series is very good for me, thank you very much!