Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Tue, 23 Mar 2021 19:01:52 +0200
From:   Ilias Apalodimas <ilias.apalodimas@linaro.org>
To:     Alexander Lobakin <alobakin@pm.me>
Cc:     Matteo Croce <mcroce@linux.microsoft.com>,
        Jesper Dangaard Brouer <jbrouer@redhat.com>,
        netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
        Jonathan Lemon <jonathan.lemon@gmail.com>,
        "David S. Miller" <davem@davemloft.net>,
        Jesper Dangaard Brouer <hawk@kernel.org>,
        Lorenzo Bianconi <lorenzo@kernel.org>,
        Saeed Mahameed <saeedm@nvidia.com>,
        David Ahern <dsahern@gmail.com>,
        Saeed Mahameed <saeed@kernel.org>, Andrew Lunn <andrew@lunn.ch>
Subject: Re: [PATCH net-next 0/6] page_pool: recycle buffers
Message-ID: <YFofANKiR3tD9zgm@enceladus>
References: <20210322170301.26017-1-mcroce@linux.microsoft.com>
 <20210323154112.131110-1-alobakin@pm.me>
 <YFoNoohTULmcpeCr@enceladus>
 <20210323170447.78d65d05@carbon>
 <YFoTBm0mJ4GyuHb6@enceladus>
 <CAFnufp1K+t76n9shfOZB_scV7myUWCTXbB+yf5sr-8ORYQxCEQ@mail.gmail.com>
 <20210323165523.187134-1-alobakin@pm.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20210323165523.187134-1-alobakin@pm.me>
Precedence: bulk

On Tue, Mar 23, 2021 at 04:55:31PM +0000, Alexander Lobakin wrote:
> > > > > >

[...]

> > > > >
> > > > > Thanks for the testing!
> > > > > Any chance you can get a perf measurement on this?
> > > >
> > > > I guess you mean perf-report (--stdio) output, right?
> > > >
> > >
> > > Yea,
> > > As hinted below, I am just trying to figure out if on Alexander's platform the
> > > cost of syncing, is bigger that free-allocate. I remember one armv7 were that
> > > was the case.
> > >
> > > > > Is DMA syncing taking a substantial amount of your cpu usage?
> > > >
> > > > (+1 this is an important question)
> 
> Sure, I'll drop perf tools to my test env and share the results,
> maybe tomorrow or in a few days.
> From what I know for sure about MIPS and my platform,
> post-Rx synching (dma_sync_single_for_cpu()) is a no-op, and
> pre-Rx (dma_sync_single_for_device() etc.) is a bit expensive.
> I always have sane page_pool->pp.max_len value (smth about 1668
> for MTU of 1500) to minimize the overhead.
> 
> By the word, IIRC, all machines shipped with mvpp2 have hardware
> cache coherency units and don't suffer from sync routines at all.
> That may be the reason why mvpp2 wins the most from this series.

Yep exactly. It's also the reason why you explicitly have to opt-in using the
recycling (by marking the skb for it), instead of hiding the feature in the
page pool internals 

Cheers
/Ilias

> 
> > > > > >
> > > > > > [0] https://lore.kernel.org/netdev/20210323153550.130385-1-alobakin@pm.me
> > > > > >
> > > >
> >
> > That would be the same as for mvneta:
> >
> > Overhead  Shared Object     Symbol
> >   24.10%  [kernel]          [k] __pi___inval_dcache_area
> >   23.02%  [mvneta]          [k] mvneta_rx_swbm
> >    7.19%  [kernel]          [k] kmem_cache_alloc
> >
> > Anyway, I tried to use the recycling *and* napi_build_skb on mvpp2,
> > and I get lower packet rate than recycling alone.
> > I don't know why, we should investigate it.
> 
> mvpp2 driver doesn't use napi_consume_skb() on its Tx completion path.
> As a result, NAPI percpu caches get refilled only through
> kmem_cache_alloc_bulk(), and most of skbuff_head recycling
> doesn't work.
> 
> > Regards,
> > --
> > per aspera ad upstream
> 
> Oh, I love that one!
> 
> Al
>