Received-SPF: pass (google.com: domain of linux-wireless-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <b7253d36-26cc-e5a3-e34a-d28d6fd8fde0@intel.com>
Date:   Thu, 15 Jun 2023 15:59:39 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.11.1
Subject: Re: [PATCH net-next v4 4/5] page_pool: remove PP_FLAG_PAGE_FRAG flag
To:     Jakub Kicinski <kuba@kernel.org>,
        Yunsheng Lin <linyunsheng@huawei.com>
CC:     <davem@davemloft.net>, <pabeni@redhat.com>,
        <netdev@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        Lorenzo Bianconi <lorenzo@kernel.org>,
        Alexander Duyck <alexander.duyck@gmail.com>,
        Yisen Zhuang <yisen.zhuang@huawei.com>,
        Salil Mehta <salil.mehta@huawei.com>,
        Eric Dumazet <edumazet@google.com>,
        Sunil Goutham <sgoutham@marvell.com>,
        Geetha sowjanya <gakula@marvell.com>,
        Subbaraya Sundeep <sbhatta@marvell.com>,
        hariprasad <hkelam@marvell.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Leon Romanovsky <leon@kernel.org>,
        Felix Fietkau <nbd@nbd.name>,
        Ryder Lee <ryder.lee@mediatek.com>,
        Shayne Chen <shayne.chen@mediatek.com>,
        Sean Wang <sean.wang@mediatek.com>,
        Kalle Valo <kvalo@kernel.org>,
        Matthias Brugger <matthias.bgg@gmail.com>,
        David Christensen <drc@linux.vnet.ibm.com>,
        AngeloGioacchino Del Regno 
        <angelogioacchino.delregno@collabora.com>,
        "Jesper Dangaard Brouer" <hawk@kernel.org>,
        Ilias Apalodimas <ilias.apalodimas@linaro.org>,
        <linux-rdma@vger.kernel.org>, <linux-wireless@vger.kernel.org>,
        <linux-arm-kernel@lists.infradead.org>,
        <linux-mediatek@lists.infradead.org>
References: <20230612130256.4572-1-linyunsheng@huawei.com>
 <20230612130256.4572-5-linyunsheng@huawei.com>
 <20230614101954.30112d6e@kernel.org>
Content-Language: en-US
From:   Alexander Lobakin <aleksander.lobakin@intel.com>
In-Reply-To: <20230614101954.30112d6e@kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZG1EaEJ5Y1RQMHdWc0gvd3Jua2tXQWhLcFN4QUd5RkpaKzhaL0M1bGZweXFI?=
 =?utf-8?B?OXpTT09kSUFHWmdlZ0Z3N1l4c2tXL0o3dnRzVmk3K3o5bDIzaG45c2hpakRR?=
 =?utf-8?B?ejB0OWJvZUdRVElFQVJReTE5NmNmUjlaQksvQTJFVm1qZ3FURXRZZUlobHRs?=
 =?utf-8?B?R205QXFaVFNOeHRRYzFwVlR0Zmo4MERGT3Z4SHoyM0RmeUZTTGN0ZDh3Nmxl?=
 =?utf-8?B?bFFOY0VPL25sSUhVcTcrUkY1L2xKTlFHV0NGMzR2TEZmSTAva2puK0pKODU4?=
 =?utf-8?B?VWUxb3lZbGNmb3NXMkpDbzB2LzFxckpCdUg4WTRURzdEN3I0SFdwbUVNT016?=
 =?utf-8?B?WXRNVFNnaHhzb3lnYnozV0xkcU1VV3F2ZlA1TDZiTTRCcXNwUlREVjU4enZl?=
 =?utf-8?B?aDhTUlp4enZrQUpvMEZjYVlheUVOTkFSZWtOcnJLeWJnTmE4UU1BT1FJVGFl?=
 =?utf-8?B?NjF3TWM4NVFISHVheFFnM1lZQ2grNHdqRWRXQXdSUlZlaUpkZVFzejJFZWZT?=
 =?utf-8?B?NVN6RGk0OEpRUWZMenIwRktrSS9OMXJjdG5wL01Jb3ZEaUttOXBGN2FPL2FT?=
 =?utf-8?B?S1pqeG4vREIvd2t0NHI2ZEJnZ3dXak5PeGhkNHR4cnhrV1dlZVZJUDU4Y0ZN?=
 =?utf-8?B?VGkzSEFBQXc0SFk2Ym5wV3oraEhPbnk0SjZ0WHlHTnVkWVZFWXJhYjNzRUh4?=
 =?utf-8?B?aXZuWjBwMjdHbE1pQllSYWJvd3RIWmV1TjgraDhuT0ZuQTFHNzJ4UnY3Wms5?=
 =?utf-8?B?SVY5anF2dUloK2MxTS9CRUowY1d4WUlYemF5ZnlibmhYUXc5UXErNlNMNWQ4?=
 =?utf-8?B?b2kzSWFNV2VUd3pRMThENEhSU1MvRGVIZjhoT2xtWEoySCttR0RPUmYrTC91?=
 =?utf-8?B?SFlkN2M3UXV2NmF3MXhjU3p1NGkxN2wxT0Y1Q1VuOHhDNi9VMHh5TEhCTVk0?=
 =?utf-8?B?L2RwR2J6SlhzTlBqdzFHb1JzTzZUTzhMczFNb2JhZTBxcmdjZ05iV1U4NlFW?=
 =?utf-8?B?RUV0Zk5taGpNUncxNml0RUNCUnhtblNFbTJHZ2pvV2dkYlZpSnVSUzBvRFJG?=
 =?utf-8?B?cVRlSHRPWEVOU09XZFFPQ2RzMjdobHI2VnI4SlNNNUxtWk9jK0VJT25hcVZC?=
 =?utf-8?B?azZUaSsvWUZxeDdTWUdDSWNVK1N5WW9mNnU3VHpxQlFLKzIrY2s0ZllVQXY1?=
 =?utf-8?B?TlU5eFd0a1YyamRQRFFxOCtlcTZBUnlEdFRYWUhMQ0VSdTBmTHhtODZIRFR3?=
 =?utf-8?B?UXBaYk5jcVBYUkpKWkxNREVtaG83Mm5uejk3aS9PdndnZzdjWkdzaCs4Mlpm?=
 =?utf-8?B?a0F0QW01ellscXN4dFRrZURWYkRyL0hCUU1qNXpuSFNQNlZueWlRTGhqbGNm?=
 =?utf-8?B?Mm1YdmV5ZlRmbm5oczFETU5QOXAxN3RZeGNHQ2ZEYmlKVW94dnMxUkdhTWJ1?=
 =?utf-8?B?eHJuSkk1ekhJNXVsdWlINURvMVpvbUZkNFo3enBjcmRVU0hNcnVCNjZhSDh6?=
 =?utf-8?B?VExLTEQ4VTVZVTFyVlNPQTNyVk82NXZOcE1jalg1UGN0R1Q1eXRJRmR5cGZZ?=
 =?utf-8?B?dkViRlN5eEQxSVNMOHlQRDdqOTgybEwrWjVtaTlPb2lxbXE2ZXZjUkFpZHF1?=
 =?utf-8?B?eXU1NkhjQlBUUU43RFlFOE5YVEwwd1VKMy9md2x2WXVTUkFJa0dYdTVndER0?=
 =?utf-8?B?ZTVXekU0R3VBQktrUTkvSm95d1dQQ2h4dzJiVEtHbk5qSEdoMmN2T0dJbW5L?=
 =?utf-8?B?MUNXVWNwZW03ZTBZb0dZdk5QMVp1aS9uMzErMTJCU3FTQW1VZThwZkg0dFJm?=
 =?utf-8?B?eUtTQ3d0WUFvVGtGMFduWDJFNC9OdGMvb1A0cTBsaWlkS3BvUTF2Z3FyS1RX?=
 =?utf-8?B?Z1FObW9MR2JndFpyM3liRExWYWNVTzIxTkdPZm9iWUcrQ3FCZTF1Q1gwZjBW?=
 =?utf-8?B?Tk9KQzFwZERNdUczRUxMYU5lOXJVTXZLQ25OUnVORy9BTnNUMVBsMEFuTjBH?=
 =?utf-8?B?MWF5a2JzQ2paNy9TekdiNlI3a0RWRG1aM1ZDaWdET3p5YVhrMWJGYzcvdVdW?=
 =?utf-8?B?U2phNHlPK1FBd2I0L2dEcmpXRzdxeGxaMzVoS0VmU0xqdmtScDQ4ZjlHSmVI?=
 =?utf-8?B?YXNKM3FGNTVDN0N4R3EwcjdWZW5mTHR1OE5FTWdTdXFuOXpoeFk5dWd6M1J2?=
 =?utf-8?B?Y0E9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 2427c4bd-1ead-4a08-a366-08db6da8c8bb
X-MS-Exchange-CrossTenant-AuthSource: DM6PR11MB3625.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jun 2023 13:59:49.5277
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Zf3/FqU1/RcGgweR0e7M6UbRN5uJhfexIYo064dpExFoFEVubR2Up3jhQ76q9FDWZHqCcRnPUz0e8fBhI7o6kxaFqVNsDYZK9/y69j2gHAs=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN6PR11MB8172
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-wireless.vger.kernel.org>
X-Mailing-List: linux-wireless@vger.kernel.org

From: Jakub Kicinski <kuba@kernel.org>
Date: Wed, 14 Jun 2023 10:19:54 -0700

> On Mon, 12 Jun 2023 21:02:55 +0800 Yunsheng Lin wrote:
>>  	struct page_pool_params pp_params = {
>> -		.flags = PP_FLAG_DMA_MAP | PP_FLAG_PAGE_FRAG |
>> -				PP_FLAG_DMA_SYNC_DEV,
>> +		.flags = PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV,
>>  		.order = hns3_page_order(ring),
> 
> Does hns3_page_order() set a good example for the users?
> 
> static inline unsigned int hns3_page_order(struct hns3_enet_ring *ring)
> {
> #if (PAGE_SIZE < 8192)
> 	if (ring->buf_size > (PAGE_SIZE / 2))
> 		return 1;
> #endif
> 	return 0;
> }

Oh lol, just what Intel drivers do. They don't have a pool to keep some
bunch of pages (they can recycle a page only within its buffer), so in
order to still recycle them, they allocate order-1 pages to be able to
flip the halves >_<

> 
> Why allocate order 1 pages for buffers which would fit in a single page?
> I feel like this soft of heuristic should be built into the API itself.

Offtop:

I tested this series with IAVF: very little perf regression* (almost
stddev) comparing to just 1-page-per-frame Page Pool series, but 21 Mb
less RAM taken comparing to both "old" PP series and baseline, nice :D

(+Cc David Christensen, he'll be glad to hear we're stopping eating 64Kb
 pages)

* this might be caused by that in the previous version I was hardcoding
truesize, but now it depends on what page_pool_alloc() returns. Same for
Rx offset: it was always 0 previously, as every frame was placed at the
start of page, now depends on how PP places** it.
With MTU of 1500 and no XDP, two frames fit into one 4k page. With XDP
on (increased headroom) or increased MTU, PP starts effectively do
1-frame-per-page with literally no changes in performance (increased RAM
usage obviously -- I mean, it gets restored to the baseline numbers).

** BTW, instead of 2048 + 2048, I'm getting 1920 + 2176. Maybe the stack
would be happier to see more consistent truesize for cache purposes.
I'll try to play with it.

Thanks,
Olek