Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20230321085953.24949-1-huangjie.albert@bytedance.com> <CACGkMEvx_-3XbnBk1PakqODhL+C0Oy-BVORm=FsMxvzVcBbrnA@mail.gmail.com>
In-Reply-To: <CACGkMEvx_-3XbnBk1PakqODhL+C0Oy-BVORm=FsMxvzVcBbrnA@mail.gmail.com>
From:   =?UTF-8?B?6buE5p2w?= <huangjie.albert@bytedance.com>
Date:   Thu, 23 Mar 2023 16:01:17 +0800
Message-ID: <CABKxMyNSp1-pJW11B3YuDm39mg=eT48JspDsrEePjKFrHNK8NQ@mail.gmail.com>
Subject: Re: [External] Re: [PATCH] virtio_ring: Suppress tx interrupt when
 napi_tx disable
To:     Jason Wang <jasowang@redhat.com>
Cc:     "Michael S . Tsirkin" <mst@redhat.com>,
        "open list:VIRTIO CORE AND NET DRIVERS" 
        <virtualization@lists.linux-foundation.org>,
        open list <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

Jason Wang <jasowang@redhat.com> =E4=BA=8E2023=E5=B9=B43=E6=9C=8822=E6=97=
=A5=E5=91=A8=E4=B8=89 10:37=E5=86=99=E9=81=93=EF=BC=9A
>
> On Tue, Mar 21, 2023 at 5:00=E2=80=AFPM Albert Huang
> <huangjie.albert@bytedance.com> wrote:
> >
> > From: "huangjie.albert" <huangjie.albert@bytedance.com>
> >
> > fix commit 8d622d21d248 ("virtio: fix up virtio_disable_cb")
> >
> > if we disable the napi_tx. when we triger a tx interrupt, the
>
> typo should be "trigger"
>

OK, thanks for this. I will correct it in the next version

> > vq->event_triggered will be set to true. It will no longer be
> > set to false. Unless we explicitly call virtqueue_enable_cb_delayed
> > or virtqueue_enable_cb_prepare
> >
> > if we disable the napi_tx, It will only be called when the tx ring
> > buffer is relatively small:
> > virtio_net->start_xmit:
> >         if (sq->vq->num_free < 2+MAX_SKB_FRAGS) {
> >                 netif_stop_subqueue(dev, qnum);
> >                 if (!use_napi &&
> >                     unlikely(!virtqueue_enable_cb_delayed(sq->vq))) {
> >                         /* More just got used, free them then recheck. =
*/
> >                         free_old_xmit_skbs(sq, false);
> >                         if (sq->vq->num_free >=3D 2+MAX_SKB_FRAGS) {
> >                                 netif_start_subqueue(dev, qnum);
> >                                 virtqueue_disable_cb(sq->vq);
> >                         }
>
> The code example here is out of date, make sure your tree has this:

also, I will correct it in the next version=EF=BC=8Cthis is from kernel 5.1=
5.

>
> commit d71ebe8114b4bf622804b810f5e274069060a174
> Author: Jason Wang <jasowang@redhat.com>
> Date:   Tue Jan 17 11:47:07 2023 +0800
>
>     virtio-net: correctly enable callback during start_xmit
>
> >                 }
> >         }
> > Because event_triggered is true.Therefore, VRING_AVAIL_F_NO_INTERRUPT o=
r
> > VRING_PACKED_EVENT_FLAG_DISABLE will not be set.So we update
> > vring_used_event(&vq->split.vring) or vq->packed.vring.driver->off_wrap
> > every time we call virtqueue_get_buf_ctx.This will bring more interrupt=
ions.
>
> Can you please post how to test with the performance numbers?
>

iperf3 tcp stream:
vm1 -----------------> vm2
vm2 just receive tcp data stream from vm1, and send the ack to vm1,
there are so
many tx interruptions  in vm2.

but without event_triggered there are just a few tx interruptions.

> >
> > if event_triggered is set to true, do not update vring_used_event(&vq->=
split.vring)
> > or vq->packed.vring.driver->off_wrap
> >
> > Signed-off-by: huangjie.albert <huangjie.albert@bytedance.com>
> > ---
> >  drivers/virtio/virtio_ring.c | 6 ++++--
> >  1 file changed, 4 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.=
c
> > index 307e139cb11d..f486cccadbeb 100644
> > --- a/drivers/virtio/virtio_ring.c
> > +++ b/drivers/virtio/virtio_ring.c
> > @@ -795,7 +795,8 @@ static void *virtqueue_get_buf_ctx_split(struct vir=
tqueue *_vq,
> >         /* If we expect an interrupt for the next entry, tell host
> >          * by writing event index and flush out the write before
> >          * the read in the next get_buf call. */
> > -       if (!(vq->split.avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT=
))
> > +       if (!(vq->split.avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT=
)
> > +                       && (vq->event_triggered =3D=3D false))
>
> I'm not sure this can work, when event_triggered is true it means
> we've got an interrupt, in this case if we want another interrupt for
> the next entry, we should update used_event otherwise we will lose
> that interrupt?
>
> Thanks

Normally, if we receive an interrupt, we should disable the interrupt
in the interrupt callback handler.
But because of the introduction of event_triggered, here,
virtqueue_get_buf_ctx_split  cannot be recognized
that the interrupt has been turned off.

if we want  another interrupt for the next entry, We should probably
call virtqueue_enable_cb=EF=BC=9F

Thanks

>
> >                 virtio_store_mb(vq->weak_barriers,
> >                                 &vring_used_event(&vq->split.vring),
> >                                 cpu_to_virtio16(_vq->vdev, vq->last_use=
d_idx));
> > @@ -1529,7 +1530,8 @@ static void *virtqueue_get_buf_ctx_packed(struct =
virtqueue *_vq,
> >          * by writing event index and flush out the write before
> >          * the read in the next get_buf call.
> >          */
> > -       if (vq->packed.event_flags_shadow =3D=3D VRING_PACKED_EVENT_FLA=
G_DESC)
> > +       if (vq->packed.event_flags_shadow =3D=3D VRING_PACKED_EVENT_FLA=
G_DESC
> > +                       && (vq->event_triggered =3D=3D false))
> >                 virtio_store_mb(vq->weak_barriers,
> >                                 &vq->packed.vring.driver->off_wrap,
> >                                 cpu_to_le16(vq->last_used_idx));
> > --
> > 2.31.1
> >
>