Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Thu, 27 Sep 2018 12:23:55 +0200
From:   Stephen Hemminger <stephen@networkplumber.org>
To:     Mohammed Gamal <mgamal@redhat.com>
Cc:     Haiyang Zhang <haiyangz@microsoft.com>,
        Stephen Hemminger <sthemmin@microsoft.com>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        "otubo@redhat.com" <otubo@redhat.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "devel@linuxdriverproject.org" <devel@linuxdriverproject.org>,
        vkuznets <vkuznets@redhat.com>
Subject: Re: [PATCH] hv_netvsc: Make sure out channel is fully opened on
 send
Message-ID: <20180927122355.470df119@shemminger-XPS-13-9360>
In-Reply-To: <1538038625.19334.2.camel@redhat.com>
References: <1537979659-26979-1-git-send-email-mgamal@redhat.com>
        <BN6PR21MB01612C60B5421AF2BB09742CCA150@BN6PR21MB0161.namprd21.prod.outlook.com>
        <1538038625.19334.2.camel@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Thu, 27 Sep 2018 10:57:05 +0200
Mohammed Gamal <mgamal@redhat.com> wrote:

> On Wed, 2018-09-26 at 17:13 +0000, Haiyang Zhang wrote:
> > > -----Original Message-----
> > > From: Mohammed Gamal <mgamal@redhat.com>
> > > Sent: Wednesday, September 26, 2018 12:34 PM
> > > To: Stephen Hemminger <sthemmin@microsoft.com>; netdev@vger.kernel.
> > > org
> > > Cc: KY Srinivasan <kys@microsoft.com>; Haiyang Zhang
> > > <haiyangz@microsoft.com>; vkuznets <vkuznets@redhat.com>;
> > > otubo@redhat.com; cavery <cavery@redhat.com>; linux-
> > > kernel@vger.kernel.org; devel@linuxdriverproject.org; Mohammed
> > > Gamal
> > > <mgamal@redhat.com>
> > > Subject: [PATCH] hv_netvsc: Make sure out channel is fully opened
> > > on send
> > >=20
> > > Dring high network traffic changes to network interface parameters
> > > such as
> > > number of channels or MTU can cause a kernel panic with a NULL
> > > pointer
> > > dereference. This is due to netvsc_device_remove() being called and
> > > deallocating the channel ring buffers, which can then be accessed
> > > by
> > > netvsc_send_pkt() before they're allocated on calling
> > > netvsc_device_add()
> > >=20
> > > The patch fixes this problem by checking the channel state and
> > > returning
> > > ENODEV if not yet opened. We also move the call to
> > > hv_ringbuf_avail_percent()
> > > which may access the uninitialized ring buffer.
> > >=20
> > > Signed-off-by: Mohammed Gamal <mgamal@redhat.com>
> > > ---
> > > =C2=A0drivers/net/hyperv/netvsc.c | 7 ++++++-
> > > =C2=A01 file changed, 6 insertions(+), 1 deletion(-)
> > >=20
> > > diff --git a/drivers/net/hyperv/netvsc.c
> > > b/drivers/net/hyperv/netvsc.c index
> > > fe01e14..75f1b31 100644
> > > --- a/drivers/net/hyperv/netvsc.c
> > > +++ b/drivers/net/hyperv/netvsc.c
> > > @@ -825,7 +825,12 @@ static inline int netvsc_send_pkt(
> > > =C2=A0	struct netdev_queue *txq =3D netdev_get_tx_queue(ndev,
> > > packet->q_idx);
> > > =C2=A0	u64 req_id;
> > > =C2=A0	int ret;
> > > -	u32 ring_avail =3D
> > > hv_get_avail_to_write_percent(&out_channel- =20
> > > > outbound); =20
> > >=20
> > > +	u32 ring_avail;
> > > +
> > > +	if (out_channel->state !=3D CHANNEL_OPENED_STATE)
> > > +		return -ENODEV;
> > > +
> > > +	ring_avail =3D hv_get_avail_to_write_percent(&out_channel- =20
> > > >outbound); =20
> >=20
> > When you reproducing the NULL ptr panic, does your kernel include the
> > following patch?
> > hv_netvsc: common detach logic
> > https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/c
> > ommit/?id=3D7b2ee50c0cd513a176a26a71f2989facdd75bfea
> >  =20
> Yes it is included. And the commit did reduce the occurrence of this
> race condition, but it still nevertheless occurs albeit rarely.
>=20
> > We call netif_tx_disable(ndev) and netif_device_detach(ndev) before
> > doing the changes=C2=A0
> > on MTU or #channels. So there should be no call to start_xmit() when
> > channel is not ready.
> >=20
> > If you see the check for CHANNEL_OPENED_STATE is still necessary on
> > upstream kernel (including=C2=A0
> > the patch " common detach logic "), we should debug further on the
> > code and find out the=C2=A0
> > root cause.
> >=20
> > Thanks,
> > - Haiyang
> >  =20
> _______________________________________________
> devel mailing list
> devel@linuxdriverproject.org
> http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel

Is there some workload, that can be used to reproduce this?
The stress test from Vitaly with changing parameters while running network =
traffic
passes now.

Can you reproduce this with the upstream current kernel?

Adding the check in start xmit is still racy, and won't cure the problem.

Another solution would be to add a grace period in the netvsc detach logic.