Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Tue, 25 Jan 2022 09:34:58 +0000
From:   Hyeonggon Yoo <42.hyeyoo@gmail.com>
To:     Shay Agroskin <shayagr@amazon.com>
Cc:     Julian Wiedmann <jwiedmann.dev@gmail.com>, netdev@vger.kernel.org,
        Arthur Kiyanovski <akiyano@amazon.com>,
        David Arinzon <darinzon@amazon.com>,
        Noam Dagan <ndagan@amazon.com>,
        Saeed Bishara <saeedb@amazon.com>,
        "David S . Miller" <davem@davemloft.net>,
        Jakub Kicinski <kuba@kernel.org>,
        Daniel Borkmann <daniel@iogearbox.net>,
        Sameeh Jubran <sameehj@amazon.com>,
        Wei Yongjun <weiyongjun1@huawei.com>,
        Lorenzo Bianconi <lorenzo@kernel.org>,
        linux-kernel@vger.kernel.org
Subject: Re: [PATCH] net: ena: Do not waste napi skb cache
Message-ID: <Ye/EQgqCBogZR87T@ip-172-31-19-208.ap-northeast-1.compute.internal>
References: <20220123115623.94843-1-42.hyeyoo@gmail.com>
 <f835cbb3-a028-1daf-c038-516dd47ce47c@gmail.com>
 <5cca8bdd-bed0-f26a-6c96-d18947d3a50b@gmail.com>
 <pj41zlmtjk7t9a.fsf@u570694869fb251.ant.amazon.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <pj41zlmtjk7t9a.fsf@u570694869fb251.ant.amazon.com>
Precedence: bulk

On Mon, Jan 24, 2022 at 10:50:05PM +0200, Shay Agroskin wrote:
> 
> Julian Wiedmann <jwiedmann.dev@gmail.com> writes:
> 
> > On 24.01.22 10:57, Julian Wiedmann wrote:
> > > On 23.01.22 13:56, Hyeonggon Yoo wrote:
> > > > By profiling, discovered that ena device driver allocates skb by
> > > > build_skb() and frees by napi_skb_cache_put(). Because the
> > > > driver
> > > > does not use napi skb cache in allocation path, napi skb cache
> > > > is
> > > > periodically filled and flushed. This is waste of napi skb
> > > > cache.
> > > > 
> > > > As ena_alloc_skb() is called only in napi, Use napi_build_skb()
> > > > instead of build_skb() to when allocating skb.
> > > > 
> > > > This patch was tested on aws a1.metal instance.
> > > > 
> > > > Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
> > > > ---
> > > >  drivers/net/ethernet/amazon/ena/ena_netdev.c | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > 
> > > > diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c
> > > > b/drivers/net/ethernet/amazon/ena/ena_netdev.c
> > > > index c72f0c7ff4aa..2c67fb1703c5 100644
> > > > --- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
> > > > +++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
> > > > @@ -1407,7 +1407,7 @@ static struct sk_buff
> > > > *ena_alloc_skb(struct ena_ring *rx_ring, void *first_frag)
> > > >  		skb = netdev_alloc_skb_ip_align(rx_ring->netdev,
> > > >  						rx_ring->rx_copybreak);
> > > 
> > > To keep things consistent, this should then also be
> > > napi_alloc_skb().
> > >

Right. I missed this. Thank you for pointing this!

> > 
> > And on closer look, this copybreak path also looks buggy. If
> > rx_copybreak
> > gets reduced _while_ receiving a frame, the allocated skb can end up too
> > small to take all the data.
> > 
> > @ ena maintainers: can you please fix this?
> > 
> 

Shay and Julian, Thank you for reviewing this.

> Updating the copybreak value is done through ena_ethtool.c
> (ena_set_tunable()) which updates `adapter->rx_copybreak`.
> The adapter->rx_copybreak value is "propagated back" to the ring local
> attributes (rx_ring->rx_copybreak) only after an interface toggle which
> stops the napi routine first.
> 
> Unless I'm missing something here I don't think the bug you're describing
> exists.
> 
> I agree that the netdev_alloc_skb_ip_align() can become napi_alloc_skb().
> Hyeonggon Yoo, can you please apply this change as well to this patch?
> 

Okay. I'll update and test it again.

BTW, It seems netdev_alloc_skb_ip_align() is used to make some fields
be aligned. It's okay to just ignore this?

Thanks,
Hyeonggon.

> Thanks,
> Shay
> 
> 
> > > >  	else
> > > > -		skb = build_skb(first_frag, ENA_PAGE_SIZE);
> > > > +		skb = napi_build_skb(first_frag, ENA_PAGE_SIZE);
> > > >  	if (unlikely(!skb)) {
> > > >  		ena_increase_stat(&rx_ring->rx_stats.skb_alloc_fail,  1,
> > >