Message-ID: <82084397a98d55e17657b32a09c2909912498b41.camel@redhat.com>
Subject: Re: [PATCH v3] virtio-net: Fix probe of virtio-net on kvmtool
From:   Paolo Abeni <pabeni@redhat.com>
To:     Jason Wang <jasowang@redhat.com>,
        "Michael S. Tsirkin" <mst@redhat.com>
Cc:     rbradford@rivosinc.com, "David S. Miller" <davem@davemloft.net>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>,
        virtualization@lists.linux-foundation.org, netdev@vger.kernel.org,
        linux-kernel@vger.kernel.org
Date:   Thu, 02 Mar 2023 10:45:00 +0100
In-Reply-To: <CACGkMEsG10CWigz+S6JgSVK8XfbpT=L=30hZ8LDvohtaanAiZQ@mail.gmail.com>
References: <20230223-virtio-net-kvmtool-v3-1-e038660624de@rivosinc.com>
         <20230301093054-mutt-send-email-mst@kernel.org>
         <CACGkMEsG10CWigz+S6JgSVK8XfbpT=L=30hZ8LDvohtaanAiZQ@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.46.4 (3.46.4-1.fc37) 
MIME-Version: 1.0
Precedence: bulk

On Thu, 2023-03-02 at 16:10 +0800, Jason Wang wrote:
> On Wed, Mar 1, 2023 at 10:44=E2=80=AFPM Michael S. Tsirkin <mst@redhat.co=
m> wrote:
> >=20
> > On Wed, Mar 01, 2023 at 01:59:52PM +0000, Rob Bradford via B4 Relay wro=
te:
> > > From: Rob Bradford <rbradford@rivosinc.com>
> > >=20
> > > Since the following commit virtio-net on kvmtool has printed a warnin=
g
> > > during the probe:
> > >=20
> > > commit dbcf24d153884439dad30484a0e3f02350692e4c
> > > Author: Jason Wang <jasowang@redhat.com>
> > > Date:   Tue Aug 17 16:06:59 2021 +0800
> > >=20
> > >     virtio-net: use NETIF_F_GRO_HW instead of NETIF_F_LRO
> > >=20
> > > [    1.865992] net eth0: Fail to set guest offload.
> > > [    1.872491] virtio_net virtio2 eth0: set_features() failed (-22); =
wanted 0x0000000000134829, left 0x0080000000134829
> > >=20
> > > This is because during the probing the underlying netdev device has
> > > identified that the netdev features on the device has changed and
> > > attempts to update the virtio-net offloads through the virtio-net
> > > control queue. kvmtool however does not have a control queue that sup=
ports
> > > offload changing (VIRTIO_NET_F_CTRL_GUEST_OFFLOADS is not advertised)
> > >=20
> > > The netdev features have changed due to validation checks in
> > > netdev_fix_features():
> > >=20
> > > if (!(features & NETIF_F_RXCSUM)) {
> > >       /* NETIF_F_GRO_HW implies doing RXCSUM since every packet
> > >        * successfully merged by hardware must also have the
> > >        * checksum verified by hardware.  If the user does not
> > >        * want to enable RXCSUM, logically, we should disable GRO_HW.
> > >        */
> > >       if (features & NETIF_F_GRO_HW) {
> > >               netdev_dbg(dev, "Dropping NETIF_F_GRO_HW since no RXCSU=
M feature.\n");
> > >               features &=3D ~NETIF_F_GRO_HW;
> > >       }
> > > }
> > >=20
> > > Since kvmtool does not advertise the VIRTIO_NET_F_GUEST_CSUM feature =
the
> > > NETIF_F_RXCSUM bit is not present and so the NETIF_F_GRO_HW bit is
> > > cleared. This results in the netdev features changing, which triggers
> > > the attempt to reprogram the virtio-net offloads which then fails.
> > >=20
> > > This commit prevents that set of netdev features from changing by
> > > preemptively applying the same validation and only setting
> > > NETIF_F_GRO_HW if NETIF_F_RXCSUM is set because the device supports b=
oth
> > > VIRTIO_NET_F_GUEST_CSUM and VIRTIO_NET_F_GUEST_TSO{4,6}
> > >=20
> > > Signed-off-by: Rob Bradford <rbradford@rivosinc.com>
> > > ---
> > > Changes in v3:
> > > - Identified root-cause of feature bit changing and updated condition=
s
> > >   check
> > > - Link to v2: https://lore.kernel.org/r/20230223-virtio-net-kvmtool-v=
2-1-8ec93511e67f@rivosinc.com
> > >=20
> > > Changes in v2:
> > > - Use parentheses to group logical OR of features
> > > - Link to v1:
> > >   https://lore.kernel.org/r/20230223-virtio-net-kvmtool-v1-1-fc23d29b=
9d7a@rivosinc.com
> > > ---
> > >  drivers/net/virtio_net.c | 10 ++++++----
> > >  1 file changed, 6 insertions(+), 4 deletions(-)
> > >=20
> > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> > > index 61e33e4dd0cd..2e7705142ca5 100644
> > > --- a/drivers/net/virtio_net.c
> > > +++ b/drivers/net/virtio_net.c
> > > @@ -3778,11 +3778,13 @@ static int virtnet_probe(struct virtio_device=
 *vdev)
> > >                       dev->features |=3D dev->hw_features & NETIF_F_A=
LL_TSO;
> > >               /* (!csum && gso) case will be fixed by register_netdev=
() */
> > >       }
> > > -     if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_CSUM))
> > > +     if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_CSUM)) {
> > >               dev->features |=3D NETIF_F_RXCSUM;
> > > -     if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
> > > -         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6))
> > > -             dev->features |=3D NETIF_F_GRO_HW;
> > > +             /* This dependency is enforced by netdev_fix_features *=
/
> > > +             if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) |=
|
> > > +                 virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6))
> > > +                     dev->features |=3D NETIF_F_GRO_HW;
> > > +     }
> > >       if (virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_GUEST_OFFLOADS))
> > >               dev->hw_features |=3D NETIF_F_GRO_HW;
>=20
> Should we move this also under the check of RXCSUM, otherwise we may
> end up the following case:
>=20
> when CSUM is not negotiated but GUEST_OFFLOADS, can still try to
> enable-or-disable guest offloads? Or do we need to fail the probe in
> the case via virtnet_validate_features()?
>=20
> > >=20
> >=20
> > I see. It is annoying that we are duplicating the logic from
> > netdev_fix_features here though :(
> > Maybe we should call netdev_update_features, in the callback check
> > the flags and decide what to set and what to clear?
> > Or export netdev_fix_features to modules?
>=20
> There's a ndo_fix_features() that might be used here.

I agree with Jason: I think a virtio_net specific ndo_fix_features()
should be the right place to implement the above logic.

Cheers,

Paolo