Received-SPF: pass (google.com: domain of linux-kernel+bounces-66846-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Message-ID: <c7d4902b-593e-46d6-9ecf-bc24986c244c@intel.com>
Date: Thu, 15 Feb 2024 12:57:09 +0100
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH bpf-next] bpf/test_run: increase Page Pool's ptr_ring size
 in live frames mode
Content-Language: en-US
To: =?UTF-8?Q?Toke_H=C3=B8iland-J=C3=B8rgensen?= <toke@redhat.com>
CC: Alexei Starovoitov <ast@kernel.org>, Daniel Borkmann
	<daniel@iogearbox.net>, Andrii Nakryiko <andrii@kernel.org>, Martin KaFai Lau
	<martin.lau@linux.dev>, Jakub Kicinski <kuba@kernel.org>, Maciej Fijalkowski
	<maciej.fijalkowski@intel.com>, <bpf@vger.kernel.org>,
	<netdev@vger.kernel.org>, <linux-kernel@vger.kernel.org>
References: <20240214153838.4159970-1-aleksander.lobakin@intel.com>
 <87cyszdnrz.fsf@toke.dk> <87y1bmd4zg.fsf@toke.dk>
From: Alexander Lobakin <aleksander.lobakin@intel.com>
In-Reply-To: <87y1bmd4zg.fsf@toke.dk>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?L1QrZnJXMmo3Z2lBc0huQ0N2SVNjeFYrdU9Hdi9CR3doWmt6dmowblM0Q3FN?=
 =?utf-8?B?NHg4Z3RYei9tWXRKbkJoQUpYRDA5S1NkSVFKK3lHNnJPS3dZbkp1WHJLd3kw?=
 =?utf-8?B?c0FGR2Z1STY1aVo0bUhUd3pRb2NEVTVnUkRHdHNPa2JpeWptNk0wOGRMUHRn?=
 =?utf-8?B?UmFRS3ErSHR6dWk3NmVHK0R4Vzl0Z25rZ2dCYnZyWVBLT25sQ25ZTFY2Zy8w?=
 =?utf-8?B?bE53MnFZdU03RGJvNjlPaXhrRy9FWHVjdFp6anpTMGIrcTlaZ2l2S2pMWVNH?=
 =?utf-8?B?Q3lvSU1CemdDMUtMeWpKQkdtV3BWc25IaWt1dXlQVXZ3T2tHYXRDUzA1eCs2?=
 =?utf-8?B?NEdvRUNzTkhzeW4yZFJCdWhmdEMyNXhPTUdDeFV0VWNVNnFiRVd0MHVIK0l1?=
 =?utf-8?B?UDgwTWowMUh4eTkzall6UW1kNkpnUEdHYjJITG1YQk5kK1BHYTlScVd6SWNs?=
 =?utf-8?B?clFlaXAyZElGUmhWbHBPRkFEcnVCNnNCaFlJN2IrSCs5cStWcFdjd3YxS25V?=
 =?utf-8?B?MW5PTmRKdjA3dm9HdG8yTk5XcEZiZVVJL3VVZTY3U09vWDFFT2U4amZQYmJ3?=
 =?utf-8?B?eGN2TE0yQ2d0empma3loY1gyUlR6a2w4ZlFBc0UrdEREYmh0TEswWWNBczNr?=
 =?utf-8?B?TEU0VDFvMlhRRHBlb09sLzRyeXpNMnhmVVljdkFqVjlFeFpYU295d2xZYnoy?=
 =?utf-8?B?Sm9RdStXcHJpZXZYajlrK2Q5SVB1OVVnYW5sbmJkeGRZSVJBaHdCaDNpOEsy?=
 =?utf-8?B?bmVnMENvY2ZtcmZYem5XcWluNi8xUGpSNTJyc2tQbSs0TFNYekNrQjNvZmkz?=
 =?utf-8?B?aTNSMm5vR1BwWEMzaExrZCszZG95TGx3RTlLYys2RDgwUFRBcjlpUHNKT3RZ?=
 =?utf-8?B?TFdWbFdIV1FFWGd3S0hYOHBUYkZBTlJybTdKdis0VDNDTmg2NXNHWiswaTA1?=
 =?utf-8?B?aDJ3c3BOMUNoSkdzenk2SXR6RS9mL21CUEVqUWtvUGk2a2pvTlNscHVvM3R4?=
 =?utf-8?B?WGFNQmpOSml6U3Rmd3VKR0Y0bDduWXQ0WVh4NUFKRzhWbzhHaUV4ak5UNUNP?=
 =?utf-8?B?QzhpelNNRDdCaEdaQWpRYmRzR2dRMWtHQmZQQzJEdnhjeW1PTWlHMXU0MzlH?=
 =?utf-8?B?OGtDWjFLWTlZMUxFZ0ZoaVo4M0Jpc1NweklqVlp1WjErRWt2SFRYejU2UTV6?=
 =?utf-8?B?V2tFT3JZbEYrQzJ2bzhic1doZDZvMmc0NmdLMXJWS0NNNExNMVMzOGw2Q3pi?=
 =?utf-8?B?d0F6Z1k3NnJQSktTV0M1Z1d6QUxQYUJFUTVjQXU4QUJwbUhCcW1lV3YveWJE?=
 =?utf-8?B?QlhldmhpY0NkUFZDdDI4Q2lzV2V4cTZVblpsMzEyU1NGNmg2NmdPTFNjZDdK?=
 =?utf-8?B?TGk3SXBOeXphUDMwVGsxVVB2elhRalNBMXFlYWYrV3lEei9DM3REN1Rhamhx?=
 =?utf-8?B?RGtjU29GcURkcWhSYmtQREY4a3E2SnhBa1pkU3N3TWhSWUFTbTUwdGdMdkxk?=
 =?utf-8?B?a1ZYdXRZaml1LzQ2eGhKNEphQTgvQm9JeDhmQTZSMWZIQ2RzLzNnR2ROMHc3?=
 =?utf-8?B?Nk9OOUR5TDZPdlJJNHVJT2ZjRW9YZ3RuQUNSS0VxdEljekRNaVZlS0xRTUVU?=
 =?utf-8?B?a2ZqZVZncERheWJlc0Z5SFE3dks4VU44WFo3dkNMMTNvQVlFNGxLRXlVblNJ?=
 =?utf-8?B?SGhWZHJIMURjdjEwVWtWS0UrMkYva1RiYzhlVWM0QmRCcDFkZHRGRndEdmM3?=
 =?utf-8?B?UnQxM3VpL2hibUMrb1RocW9rREQyOURiZDk0TTI0UTFERElPd3ErMHNaM2Rr?=
 =?utf-8?B?YWwvcFZyYVpnSiswanZLOXNUMVdTUkZaSkJLSEFBTTNPVFYyWGducWR0bjFE?=
 =?utf-8?B?OWE3LzgrNElycktSaDVVcW1WandTai9KOS9pZVFWeHE4TGJQSG5GYUM2VTlm?=
 =?utf-8?B?MmtRZE5JVTBIZmF4VFJGVXhaSm5ORHZWMTJ2MlhTMUZUTVVoRU1hektnN05z?=
 =?utf-8?B?U0U3V0ZSTGVnOU5tMXFZSHFaV0YwSmo3YkpUeDYvdVNRWGdhWHV4d2Z2Sy9S?=
 =?utf-8?B?dTZ5b2MrSUt1OHMxK2F1eVhXcU84VWUzcVJIc2sxejJub3ZFRGk5YzAvL29y?=
 =?utf-8?B?T2drS0dlN3NQREltU0pEM0pKaElaV2RFdjlyMWIzT1hzMitkTXZFWTNDcFdQ?=
 =?utf-8?B?eXc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 79929fbc-0200-4841-1c52-08dc2e1d3fbc
X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB8718.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Feb 2024 11:57:14.0341
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 8N4gH5YExIBruza+p6NgeKabJWv0iswPDwgqmAdmm8zbtRSR5jFDl6QeKsiP3zHAlelp5PnTPFGXAPxeYGHDyHnXv832ndDMIpGBnBcdOtg=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB6446
X-OriginatorOrg: intel.com

From: Toke Høiland-Jørgensen <toke@redhat.com>
Date: Thu, 15 Feb 2024 00:02:27 +0100

> Toke Høiland-Jørgensen <toke@redhat.com> writes:
> 
>> Alexander Lobakin <aleksander.lobakin@intel.com> writes:
>>
>>> Currently, when running xdp-trafficgen, test_run creates page_pools with
>>> the ptr_ring size of %NAPI_POLL_WEIGHT (64).
>>> This might work fine if XDP Tx queues are polled with the budget
>>> limitation. However, we often clear them with no limitation to ensure
>>> maximum free space when sending.
>>> For example, in ice and idpf (upcoming), we use "lazy" cleaning, i.e. we
>>> clean XDP Tx queue only when the free space there is less than 1/4 of
>>> the queue size. Let's take the ring size of 512 just as an example. 3/4
>>> of the ring is 384 and often times, when we're entering the cleaning
>>> function, we have this whole amount ready (or 256 or 192, doesn't
>>> matter).
>>> Then we're calling xdp_return_frame_bulk() and after 64th frame,
>>> page_pool_put_page_bulk() starts returning pages to the page allocator
>>> due to that the ptr_ring is already full. put_page(), alloc_page() et at
>>> starts consuming a ton of CPU time and leading the board of the perf top
>>> output.
>>>
>>> Let's not limit ptr_ring to 64 for no real reason and allow more pages
>>> to be recycled. Just don't put anything to page_pool_params::size and
>>> let the Page Pool core pick the default of 1024 entries (I don't believe
>>> there are real use cases to clean more than that amount of descriptors).
>>> After the change, the MM layer disappears from the perf top output and
>>> all pages get recycled to the PP. On my test setup on idpf with the
>>> default ring size (512), this gives +80% of Tx performance with no
>>> visible memory consumption increase.
>>>
>>> Signed-off-by: Alexander Lobakin <aleksander.lobakin@intel.com>
>>
>> Hmm, so my original idea with keeping this low was to avoid having a lot
>> of large rings lying around if it is used by multiple processes at once.
>> But we need to move away from the per-syscall allocation anyway, and
>> with Lorenzo's patches introducing a global system page pool we have an
>> avenue for that. So in the meantime, I have no objection to this...
>>
>> Reviewed-by: Toke Høiland-Jørgensen <toke@redhat.com>
> 
> Actually, since Lorenzo's patches already landed in net-next, let's just
> move to using those straight away. I'll send a patch for this tomorrow :)

Keep in mind that system page_pools do direct recycling based on cpuid
and for now, memory leaks are possible. Pls see my patch[0] for the
details :D

> 
> -Toke
> 

[0]
https://lore.kernel.org/netdev/20240215113905.96817-1-aleksander.lobakin@intel.com

Olek