Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20230725074148.2936402-1-wei.fang@nxp.com> <70b71e7bb8a7dff2dacab99b0746e7bf2bee9344.camel@gmail.com>
 <AM5PR04MB31390FCD7DB9F905FCB599108800A@AM5PR04MB3139.eurprd04.prod.outlook.com>
In-Reply-To: <AM5PR04MB31390FCD7DB9F905FCB599108800A@AM5PR04MB3139.eurprd04.prod.outlook.com>
From:   Alexander Duyck <alexander.duyck@gmail.com>
Date:   Wed, 26 Jul 2023 08:53:47 -0700
Message-ID: <CAKgT0Ufo8exTv1783Ud7EUg_1ei90Eb4ZoiHFd49zAbfhLgAsQ@mail.gmail.com>
Subject: Re: [PATCH net] net: fec: tx processing does not call XDP APIs if
 budget is 0
To:     Wei Fang <wei.fang@nxp.com>
Cc:     dl-linux-imx <linux-imx@nxp.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "davem@davemloft.net" <davem@davemloft.net>,
        "edumazet@google.com" <edumazet@google.com>,
        "kuba@kernel.org" <kuba@kernel.org>,
        Shenwei Wang <shenwei.wang@nxp.com>,
        Clark Wang <xiaoning.wang@nxp.com>,
        "pabeni@redhat.com" <pabeni@redhat.com>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Tue, Jul 25, 2023 at 8:40=E2=80=AFPM Wei Fang <wei.fang@nxp.com> wrote:
>
> Hi Alexander,
>
> > > @@ -1416,6 +1416,14 @@ fec_enet_tx_queue(struct net_device *ndev,
> > u16 queue_id)
> > >                     if (!skb)
> > >                             goto tx_buf_done;
> > >             } else {
> > > +                   /* Tx processing cannot call any XDP (or page poo=
l) APIs if
> > > +                    * the "budget" is 0. Because NAPI is called with=
 budget of
> > > +                    * 0 (such as netpoll) indicates we may be in an =
IRQ context,
> > > +                    * however, we can't use the page pool from IRQ c=
ontext.
> > > +                    */
> > > +                   if (unlikely(!budget))
> > > +                           break;
> > > +
> > >                     xdpf =3D txq->tx_buf[index].xdp;
> > >                     if (bdp->cbd_bufaddr)
> > >                             dma_unmap_single(&fep->pdev->dev,
> >
> > This statement isn't correct. There are napi enabled and non-napi
> > versions of these calls. This is the reason for things like the
> > "allow_direct" parameter in page_pool_put_full_page and the
> > "napi_direct" parameter in __xdp_return.
> >
> > By blocking on these cases you can end up hanging the Tx queue which is
> > going to break netpoll as you are going to stall the ring on XDP
> > packets if they are already in the queue.
> >
> > From what I can tell your driver is using xdp_return_frame in the case
> > of an XDP frame which doesn't make use of the NAPI optimizations in
> > freeing from what I can tell. The NAPI optimized version is
> > xdp_return_frame_rx.
> >
> So you mean it is safe to use xdp_return_frame no matter in NAPI context
> or non-NAPI context? And xdp_return_frame_rx_napi must be used in NAPI
> context? If so, I think I must have misunderstood, then this patch is not=
 necessary.

Actually after talking with Jakub a bit more there is an issue here,
but not freeing the frames isn't the solution. We likely need to just
fix the page pool code so that it doesn't attempt to recycle the
frames if operating in IRQ context.

The way this is dealt with for skbs is that we queue skbs if we are in
IRQ context so that it can be deferred to be freed by the
net_tx_action. We likely need to look at doing something similar for
page_pool pages or XDP frames.

> > > @@ -1508,14 +1516,14 @@ fec_enet_tx_queue(struct net_device *ndev,
> > u16 queue_id)
> > >             writel(0, txq->bd.reg_desc_active);
> > >  }
> > >
> > > -static void fec_enet_tx(struct net_device *ndev)
> > > +static void fec_enet_tx(struct net_device *ndev, int budget)
> > >  {
> > >     struct fec_enet_private *fep =3D netdev_priv(ndev);
> > >     int i;
> > >
> > >     /* Make sure that AVB queues are processed first. */
> > >     for (i =3D fep->num_tx_queues - 1; i >=3D 0; i--)
> > > -           fec_enet_tx_queue(ndev, i);
> > > +           fec_enet_tx_queue(ndev, i, budget);
> > >  }
> > >
> > >  static void fec_enet_update_cbd(struct fec_enet_priv_rx_q *rxq,
> > > @@ -1858,7 +1866,7 @@ static int fec_enet_rx_napi(struct napi_struct
> > *napi, int budget)
> > >
> > >     do {
> > >             done +=3D fec_enet_rx(ndev, budget - done);
> > > -           fec_enet_tx(ndev);
> > > +           fec_enet_tx(ndev, budget);
> > >     } while ((done < budget) && fec_enet_collect_events(fep));
> > >
> > >     if (done < budget) {
> >
> > Since you are passing budget, one optimization you could make use of
> > would be napi_consume_skb in your Tx path instead of dev_kfree_skb_any.
> That's good suggestion, I think I can add this optimization in my XDP_TX =
support
> patch. Thanks!