Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34;
Message-ID: <03d7e8b0-8766-4f59-afd4-15b592693a83@intel.com>
Date:   Mon, 11 Dec 2023 11:16:20 +0100
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH net-next v6 08/12] libie: add Rx buffer management (via
 Page Pool)
Content-Language: en-US
To:     Yunsheng Lin <linyunsheng@huawei.com>
CC:     "David S. Miller" <davem@davemloft.net>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>,
        Paolo Abeni <pabeni@redhat.com>,
        Maciej Fijalkowski <maciej.fijalkowski@intel.com>,
        Michal Kubiak <michal.kubiak@intel.com>,
        Larysa Zaremba <larysa.zaremba@intel.com>,
        Alexander Duyck <alexanderduyck@fb.com>,
        "David Christensen" <drc@linux.vnet.ibm.com>,
        Jesper Dangaard Brouer <hawk@kernel.org>,
        Ilias Apalodimas <ilias.apalodimas@linaro.org>,
        "Paul Menzel" <pmenzel@molgen.mpg.de>, <netdev@vger.kernel.org>,
        <intel-wired-lan@lists.osuosl.org>, <linux-kernel@vger.kernel.org>
References: <20231207172010.1441468-1-aleksander.lobakin@intel.com>
 <20231207172010.1441468-9-aleksander.lobakin@intel.com>
 <1103fe8f-04c8-8cc4-8f1b-ff45cea22b54@huawei.com>
From:   Alexander Lobakin <aleksander.lobakin@intel.com>
In-Reply-To: <1103fe8f-04c8-8cc4-8f1b-ff45cea22b54@huawei.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UzFlMlQzbFd0MUhkUDROTUE1MzhvbHZ2dnoybWZ3VGF1dDRiSXlSMEJJL0FR?=
 =?utf-8?B?dEFkTHNuRUo4cjlNOGJ6amJNdzY5bUlTckJOblQrMTg3eVZQQlpFcmpuZmlU?=
 =?utf-8?B?STFJdFpWVDRwVVorbFYyM2NqUGhGbDBuVlBwd2xiZ25MYjNFK2xZdkNCQ1Zp?=
 =?utf-8?B?aU10VHcwTHpoNk8yQmIvZGxDRVg0aWE4eTJMRDk5UUhWdEQ5TGk2UytGMFN3?=
 =?utf-8?B?MGQwU0NlWElsMVRraWlHZStoMjZRQzQ0TUd0K1FQZzYyOWNDMXNsYkhXa25C?=
 =?utf-8?B?bzh5dGxaZVJjNnZ4TUM3NHl0VnBhcFBLRGxJbitJb0lhSjR2ZEEzQ1EvcTAw?=
 =?utf-8?B?Z0JYT0xzZ1dORmQ3NG1FbTJXNkZUdXE1NklLYk0wZkJqUnVZV3FmNHIvOTlk?=
 =?utf-8?B?eGNveG9SLzhOMG5vdkk3d0dkZ3ZuNWhURTZUV2pVK0JZNGxOK0NYdVNNQ2Ev?=
 =?utf-8?B?citVRXh4dkRUNVlMSStzdWUvYU9KdHJ4eEhCem0vQ3Z5T1dZWjJibHVvVmNv?=
 =?utf-8?B?eHp6MCs1OGRjanBpQ3B6YkFveVkvVlllN1R4MFJRL3lvR1RtRVlZWHZyM0dS?=
 =?utf-8?B?SU1aTU01ci9uWmZBa3I3N05yKy9mckRhRXBtUDV0dWZxeGN0WTU5NVNvdjZZ?=
 =?utf-8?B?L0owbzZNdUN0WnVHSWp6R0U5eGJVY2JSZFM0ZzQxamdXeTlxYTlxM0pkM1dG?=
 =?utf-8?B?M2N3bXpvM2RYRDBDWmpxdUd3NHZqQUJrQ2ttclM0b3RXc1d3L1AvVmtPV21B?=
 =?utf-8?B?WlFsS0FzN3hEdU1FaXlsYUNLTGZwL1lENi9LaU15bVNaaThGL21zTCt2Z3Jm?=
 =?utf-8?B?aWJEbzBmZnF2YzZOcFE4a2tzZUhIRHAvaUp4NHpobUc3aE9HNGpzV0loR3NO?=
 =?utf-8?B?REtFeEJBTXV2ZjF5OTFJVEhwNE5QVlNaOFJUazg5OFBuR2hLWG1nT3pVOWQ1?=
 =?utf-8?B?U1dGTkJVOWNORS83MHNjSytMdDlkamREd0N2bVZVZmxQY0tIS1dvUnNHVDJx?=
 =?utf-8?B?eGRacGx6M041dXBqelFxaGdXOHJqQ0xycE5vWmlBd3JkQzN3Z0ZzZzRwYUdZ?=
 =?utf-8?B?aTd6S1dJVWxYNFIwYVEyeGpZZksvN0Vpc1E3Vm1IR1hDWk5nWE9WWnk4NkJt?=
 =?utf-8?B?UHAyTGFOaXk1UWJBd0tXWU56UnlLQVYvbHM3aGdoMlVuL3hjUFY1dzMzWnE2?=
 =?utf-8?B?MDJWa045S3BzSy9QcEhXbkU1MjRpM0lUN0gzay9jSG9OT0xaMUxaU1NXektq?=
 =?utf-8?B?NzJ5R2dFcXRrb0tKTkFBbmJWZWVXZWNUdDAya2ZjOTAveTB4eUdONC9Fd3Jj?=
 =?utf-8?B?Q0tTTUhHRVRNT3phREh1NHRxWFVmYnhxL24zN21SREUxbzVmOFVPUFdBWW9x?=
 =?utf-8?B?VWFQWERjNUlvb2hFOXFiSktPRHJGLytBa1lpOW84MXlXRkUrV0tWU01TQmNz?=
 =?utf-8?B?Q0ZZSjZzdWxReVovazVsTHA1aTE1NG5qTDJuV2w0dzVWMEVSUHRwSlkrelc3?=
 =?utf-8?B?enk0SDFXNFg1Q3JTOHFWUXp0K295L292N3pSTWZWYUw3SzZodXpRckRTOVBo?=
 =?utf-8?B?ZFU1QyszajNubVBwQjlkRy8xR0dYK3dCd281SEhqK2dRZGNVeGlSeGc4L2Vo?=
 =?utf-8?B?bkYvVkFWMk8zM0wrcC9sNHJQY2sxT1UvSSt5dk5wcjFwQzBqTUpBTVhjNXdQ?=
 =?utf-8?B?UzVNMWZ4eS93eFN2eTFnSmVwUjhpMUkyTnN3Z3pPdXN6NURrdVZXUlE5L2pF?=
 =?utf-8?B?L210VUNMZ2hkYURtYm5TaGgzTWhERTJkZllqN29ZQW5sMzdZY3RKT0ZtK3Er?=
 =?utf-8?B?OTBzRDN6Umdtbkl1Vk1oUG5UekJwWUw1VWVEZUNsb3RVNEp5N0Ntak8zY08v?=
 =?utf-8?B?eUczNGdIY0Y4T1pueVN5VGsvOUlnNEVZc0RFY0RYNGVpQkRpV3dpY2JtaWZS?=
 =?utf-8?B?UC8wYWhDOUFEcHcwaUxOOWNFUVhhWDNINXV2K0NQQ3VnRFQxeW1TTmdtamM0?=
 =?utf-8?B?SWNMRHRpdSsxeDFOZ0pnN2UxbnhsTEtoNXpnZTh4eWVZWXAweEF1STE5QnZD?=
 =?utf-8?B?enlyK2hMTU5JNXlHR1J5SFpLNkR4U2wxbWNhSm02ZHJuZzJsRW1JMWFqaUFS?=
 =?utf-8?B?VnhFVmRRWHdIc0c4V0l6eEVPTjlTT1g3T2d4U2JRV1l1eGpySWxCTkF3YWlp?=
 =?utf-8?B?Wnc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 05237e04-53c1-4a1d-0609-08dbfa326393
X-MS-Exchange-CrossTenant-AuthSource: DM6PR11MB3625.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Dec 2023 10:17:33.1376
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 5HeqXPFRpiH8gezsKiuwNgtVJBTjvoRZ5P0BZPTqc/OmTZqsXqpzvVvnm6DJ7Aqw6ITgGQGpjPUENc6KNTKhDaI+sQAlxoA8kf8W/t9UwFo=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB6520
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 11 Dec 2023 02:17:52 -0800 (PST)

From: Yunsheng Lin <linyunsheng@huawei.com>
Date: Fri, 8 Dec 2023 17:28:21 +0800

> On 2023/12/8 1:20, Alexander Lobakin wrote:
> ...
> 
>> +
>> +/**
>> + * libie_rx_page_pool_create - create a PP with the default libie settings
>> + * @bq: buffer queue struct to fill
>> + * @napi: &napi_struct covering this PP (no usage outside its poll loops)
>> + *
>> + * Return: 0 on success, -errno on failure.
>> + */
>> +int libie_rx_page_pool_create(struct libie_buf_queue *bq,
>> +			      struct napi_struct *napi)
>> +{
>> +	struct page_pool_params pp = {
>> +		.flags		= PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV,
>> +		.order		= LIBIE_RX_PAGE_ORDER,
>> +		.pool_size	= bq->count,
>> +		.nid		= NUMA_NO_NODE,
> 
> Is there a reason the NUMA_NO_NODE is used here instead of
> dev_to_node(napi->dev->dev.parent)?

NUMA_NO_NODE creates a "dynamic" page_pool and makes sure the pages are
local to the CPU where PP allocation functions are called. Setting ::nid
to a "static" value pins the PP to a particular node.
But the main reason is that Rx queues can be distributed across several
nodes and in that case NUMA_NO_NODE will make sure each page_pool is
local to the queue it's running on. dev_to_node() will return the same
value, thus forcing some PPs to allocate remote pages.

Ideally, I'd like to pass a CPU ID this queue will be run on and use
cpu_to_node(), but currently there's no NUMA-aware allocations in the
Intel drivers and Rx queues don't get the corresponding CPU ID when
configuring. I may revisit this later, but for now NUMA_NO_NODE is the
most optimal solution here.

[...]

Thanks,
Olek