Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Subject: Re: [PATCH net-next v2 2/3] page_pool: support non-frag page for
 page_pool_alloc_frag()
To:     Alexander Duyck <alexander.duyck@gmail.com>,
        Yunsheng Lin <linyunsheng@huawei.com>
Cc:     davem@davemloft.net, kuba@kernel.org, pabeni@redhat.com,
        netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
        Lorenzo Bianconi <lorenzo@kernel.org>,
        Jesper Dangaard Brouer <hawk@kernel.org>,
        Ilias Apalodimas <ilias.apalodimas@linaro.org>,
        Eric Dumazet <edumazet@google.com>
References: <20230529092840.40413-1-linyunsheng@huawei.com>
 <20230529092840.40413-3-linyunsheng@huawei.com>
 <977d55210bfcb4f454b9d740fcbe6c451079a086.camel@gmail.com>
 <2e4f0359-151a-5cff-6d31-0ea0f014ef9a@huawei.com>
 <CAKgT0UcGYXstFP_H8VQtUooYEaYgDpG_crkodYOEyX4q0D58LQ@mail.gmail.com>
 <8c9d5dd8-b654-2d50-039d-9b7732e7746f@huawei.com>
 <CAKgT0UchHBO+kyPZMYJR7JHfqYsk+qSeuvXzA-H9w3VH-9Tfrg@mail.gmail.com>
From:   Yunsheng Lin <yunshenglin0825@gmail.com>
Message-ID: <f5e372ca-e637-4873-0bea-b1b19c623124@gmail.com>
Date:   Sat, 3 Jun 2023 12:20:15 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.10.2
MIME-Version: 1.0
In-Reply-To: <CAKgT0UchHBO+kyPZMYJR7JHfqYsk+qSeuvXzA-H9w3VH-9Tfrg@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-GB
Content-Transfer-Encoding: 8bit
Precedence: bulk

On 2023/6/2 23:57, Alexander Duyck wrote:
> On Fri, Jun 2, 2023 at 5:23 AM Yunsheng Lin <linyunsheng@huawei.com> wrote:

...

>>
>> According to my defination in this patchset:
>> frag page: page alloced from page_pool_alloc_frag() with page->pp_frag_count
>>            being greater than one.
>> non-frag page:page alloced return from both page_pool_alloc_frag() and
>>               page_pool_alloc_pages() with page->pp_frag_count being one.
>>
>> I assume the above 'non-page pool pages' refer to what I call as 'non-frag
>> page' alloced return from both page_pool_alloc_frag(), right? And it is
>> still about doing the (size << 1 > max_size)' checking at the begin instead
>> of at the middle right now to avoid extra steps for 'non-frag page' case?
> 
> Yeah, the non-page I was referring to were you mono-frag pages.

I was using 'frag page' and 'non-frag page' per the defination above,
and you were using 'mono-frag' mostly and 'non-page' sometimes.
I am really confused by them as I felt like I got what they meant and
then I was lost when you used them in the next comment. I really hope
that you could describe what do you mean in more detailed by using
'mono-frag pages' and 'non-page', so that we can choose the right
naming to continue the discussion without further misunderstanding
and confusion.

> 
>>>
>>>>>
>>>>>> -    if (page && *offset + size > max_size) {
>>>>>> +    if (page) {
>>>>>> +            *offset = pool->frag_offset;
>>>>>> +
>>>>>> +            if (*offset + size <= max_size) {
>>>>>> +                    pool->frag_users++;
>>>>>> +                    pool->frag_offset = *offset + size;
>>>>>> +                    alloc_stat_inc(pool, fast);
>>>>>> +                    return page;
>>>>
>>>> Note that we still allow frag page here when '(size << 1 > max_size)'.
>>
>> This is the optimization I was taking about: suppose we start
>> from a clean state with 64K page size, if page_pool_alloc_frag()
>> is called with size being 2K and then 34K, we only need one page
>> to satisfy caller's need as we do the '*offset + size > max_size'
>> checking before the '(size << 1 > max_size)' checking.
> 
> The issue is the unaccounted for waste. We are supposed to know the
> general size of the frags being used so we can compute truesize. If

Note, for case of veth and virtio_net, the driver may only know the
current frag size when calling page_pool_alloc_frag(), it does not
konw what is the size of the frags will be used next time, how exactly
are we going to compute the truesize for cases with different frag
size?  As far as I can tell, we may only do something like virtio_net
is doing with 'page_frag' for the last frag as below, for other frags,
the truesize may need to take accounting to the aligning requirement:
https://elixir.bootlin.com/linux/v6.3.5/source/drivers/net/virtio_net.c#L1638

> for example you are using an order 3 page and you are splitting it
> between a 2K and a 17K fragment the 2K fragments will have a massive
> truesize underestimate that can lead to memory issues if those smaller
> fragments end up holding onto the pages.
> 
> As such we should try to keep the small fragments away from anything
> larger than half of the page.

IMHO, doing the above only alleviate the problem. How is above splitting
different from splitting it evently with 16 2K fragments, and when 15 frag
is released, we still have the last 2K fragment holding onto 32K memory,
doesn't that also cause massive truesize underestimate? Not to mention that
for system with 64K page size.

In RFC patch below, 'page_pool_frag' is used to report the truesize, but
I was thinking both 'page_frag' and 'page_frag_cache' both have a similiar
problem, so I dropped it in V1 and left that as a future improvement.
 
I can pick it up again if 'truesize' is really the concern, but we have to
align on how to compute the truesize here first.

https://patchwork.kernel.org/project/netdevbpf/patch/20230516124801.2465-4-linyunsheng@huawei.com/