Received: by 2002:a05:6a10:f347:0:0:0:0 with SMTP id d7csp4350641pxu;
        Wed, 9 Dec 2020 15:05:05 -0800 (PST)
X-Google-Smtp-Source: ABdhPJyUOSpo724O2G2WMfsI2Non8WvcKPnuWmrjVOPE4Nqo7aHp1Pnm5yLGsTtiL+jPLbkO+mI2
X-Received: by 2002:a05:6402:22b4:: with SMTP id cx20mr4108610edb.262.1607555105495;
        Wed, 09 Dec 2020 15:05:05 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; t=1607555105; cv=none;
        d=google.com; s=arc-20160816;
        b=lF9sIdgc8VXyEdaGcpKOEl19NwznkNRELKtb2zHjnd6qoBPdEyKGaUm2GvVh8K/ld2
         iOLbQj3sqEHduuUztVOY4yz5uN/5aVD18hVG554mH5D4vML43r4lrCGIMgTjDSvsgDpS
         PlbgSqa3HgXfVwUF219PJnP8gxwXm/Obv0nKAJDRd2LQ6kVQTdEHbCE85gmZNHlbp3qa
         vEeruIyDwROv7oUmK1Exve15jnGJymncs8iBwjOUPYw9ma5KOGELiFfdHzOKuey1Lqnh
         ApohH40+I2tBs2ovvvVUNayWFxv3zUlSQn8dbtp4BbnQXud7KMmik443gIg3C/7FUvXU
         IvoQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
        h=list-id:precedence:dkim-signature:user-agent:in-reply-to
         :content-disposition:mime-version:references:message-id:subject:cc
         :to:from:date;
        bh=Z5+c0f4H/JFqW0cesUT+R7QjXdwvYDH7MFyx2zflVVY=;
        b=e9mM8jQDqrS1MPTCbiI5J1G4CEE+2taKpJmGMzWd+XHN6HBdYcb4pU1meVnay5B8bX
         ZRvMzwdxjmIOKqYUiHk/4QhqKukDLpJs9Ew3Fr+EPQfbr4beWcKCrMDT7FlRMJxdji6n
         slQvt7bDihd50JfzY+bPlgwmZihDzpsz2lmP4wWhS2ogtFprdDbnscCjo3eapMaz9Mt9
         zz4bWAt6yVzDF7zqo4Z9iHgrenVvsc7Af0gUwG2c1unBH29t2J91jxDjdl54C89XlhE7
         HXJXzvIv3Z2LWv04Ts6q5DxZv2cje9iVD8QtEb3aFfF8f8ud/BREVW+38Rno6AIaK2p2
         b71w==
ARC-Authentication-Results: i=1; mx.google.com;
       dkim=pass header.i=@nvidia.com header.s=n1 header.b=MBsFEGzt;
       spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org;
       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18])
        by mx.google.com with ESMTP id j24si1570198ejs.60.2020.12.09.15.04.41;
        Wed, 09 Dec 2020 15:05:05 -0800 (PST)
Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Authentication-Results: mx.google.com;
       dkim=pass header.i=@nvidia.com header.s=n1 header.b=MBsFEGzt;
       spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org;
       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1729218AbgLIJjW (ORCPT <rfc822;ketanpantkp@gmail.com>
        + 99 others); Wed, 9 Dec 2020 04:39:22 -0500
Received: from hqnvemgate24.nvidia.com ([216.228.121.143]:15552 "EHLO
        hqnvemgate24.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1728311AbgLIJjV (ORCPT
        <rfc822;linux-kernel@vger.kernel.org>);
        Wed, 9 Dec 2020 04:39:21 -0500
Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate24.nvidia.com (using TLS: TLSv1.2, AES256-SHA)
        id <B5fd09b210001>; Wed, 09 Dec 2020 01:38:41 -0800
Received: from mtl-vdi-166.wap.labs.mlnx (10.124.1.5) by HQMAIL107.nvidia.com
 (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Wed, 9 Dec
 2020 09:38:39 +0000
Date:   Wed, 9 Dec 2020 11:38:36 +0200
From:   Eli Cohen <elic@nvidia.com>
To:     "Michael S. Tsirkin" <mst@redhat.com>
CC:     <jasowang@redhat.com>, <virtualization@lists.linux-foundation.org>,
        <linux-kernel@vger.kernel.org>, <lulu@redhat.com>
Subject: Re: [PATCH] vdpa/mlx5: Use write memory barrier after updating CQ
 index
Message-ID: <20201209093836.GA62204@mtl-vdi-166.wap.labs.mlnx>
References: <20201206105719.123753-1-elic@nvidia.com>
 <20201208164356-mutt-send-email-mst@kernel.org>
 <20201209060230.GA57362@mtl-vdi-166.wap.labs.mlnx>
 <20201209014547-mutt-send-email-mst@kernel.org>
 <20201209065846.GA59515@mtl-vdi-166.wap.labs.mlnx>
 <20201209025712-mutt-send-email-mst@kernel.org>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20201209025712-mutt-send-email-mst@kernel.org>
User-Agent: Mutt/1.9.5 (bf161cf53efb) (2018-04-13)
X-Originating-IP: [10.124.1.5]
X-ClientProxiedBy: HQMAIL105.nvidia.com (172.20.187.12) To
 HQMAIL107.nvidia.com (172.20.187.13)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1;
        t=1607506721; bh=Z5+c0f4H/JFqW0cesUT+R7QjXdwvYDH7MFyx2zflVVY=;
        h=Date:From:To:CC:Subject:Message-ID:References:MIME-Version:
         Content-Type:Content-Disposition:In-Reply-To:User-Agent:
         X-Originating-IP:X-ClientProxiedBy;
        b=MBsFEGztIAQ4P97fvFVRaezB7DegQ3S5VRh7/JtpXMA7w/tyviXvT7VDevwQBKWZ3
         rgOXLdSBtrZBjHQk9Bv0q+u1ZS4HU4nrCyhpYYseykP0LK7WFyPSUDQW8XveoC/mwm
         7HwPSiYLtzRYfRCJZsZhzKVtz6tcibIsjadBFVleTf84T7yxbX4IcICuQk9yD15FoD
         93oZ8eYBVwIyZAzQ472LKICaeCrfOMkLJHcctZmJDMOep8/hTDc1HLkCSAXpFgYWcO
         ZbNwbsQVCvnYFFbFobv8aXHHQtPxr2JqNRVog13dfi+9Mgiuhc39ZGnuxNRhwrCibN
         HE6va6U9hyTOA==
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Dec 09, 2020 at 03:05:42AM -0500, Michael S. Tsirkin wrote:
> On Wed, Dec 09, 2020 at 08:58:46AM +0200, Eli Cohen wrote:
> > On Wed, Dec 09, 2020 at 01:46:22AM -0500, Michael S. Tsirkin wrote:
> > > On Wed, Dec 09, 2020 at 08:02:30AM +0200, Eli Cohen wrote:
> > > > On Tue, Dec 08, 2020 at 04:45:04PM -0500, Michael S. Tsirkin wrote:
> > > > > On Sun, Dec 06, 2020 at 12:57:19PM +0200, Eli Cohen wrote:
> > > > > > Make sure to put write memory barrier after updating CQ consumer index
> > > > > > so the hardware knows that there are available CQE slots in the queue.
> > > > > > 
> > > > > > Failure to do this can cause the update of the RX doorbell record to get
> > > > > > updated before the CQ consumer index resulting in CQ overrun.
> > > > > > 
> > > > > > Change-Id: Ib0ae4c118cce524c9f492b32569179f3c1f04cc1
> > > > > > Fixes: 1a86b377aa21 ("vdpa/mlx5: Add VDPA driver for supported mlx5 devices")
> > > > > > Signed-off-by: Eli Cohen <elic@nvidia.com>
> > > > > 
> > > > > Aren't both memory writes?
> > > > 
> > > > Not sure what exactly you mean here.
> > > 
> > > Both updates are CPU writes into RAM that hardware then reads
> > > using DMA.
> > > 
> > 
> > You mean why I did not put a memory barrier right after updating the
> > recieve doorbell record?
> 
> Sorry about being unclear.  I just tried to give justification for why
> dma_wmb seems more appropriate than wmb here. If you need to
> order memory writes wrt writes to card, that is different, but generally
> writeX and friends will handle the ordering for you, except when
> using relaxed memory mappings - then wmb is generally necessary.
> 

Bear in mind, we're writing to memory (not io memory). In this case, we
want this write to be visible my the DMA device.

https://www.kernel.org/doc/Documentation/memory-barriers.txt gives a
similar example using dma_wmb() to flush updates to make them visible
by the hardware before notifying the hardware to come and inspect this
memory.


> > I thought about this and I think it is not required. Suppose it takes a
> > very long time till the hardware can actually see this update. The worst
> > effect would be that the hardware will drop received packets if it does
> > sees none available due to the delayed update. Eventually it will see
> > the update and will continue working.
> > 
> > If I put a memory barrier, I put some delay waiting for the CPU to flush
> > the write before continuing. I tried both options while checking packet
> > rate on couldn't see noticable difference in either case.
> 
> 
> makes sense.
> 
> > > > > And given that, isn't dma_wmb() sufficient here?
> > > > 
> > > > I agree that dma_wmb() is more appropriate here.
> > > > 
> > > > > 
> > > > > 
> > > > > > ---
> > > > > >  drivers/vdpa/mlx5/net/mlx5_vnet.c | 5 +++++
> > > > > >  1 file changed, 5 insertions(+)
> > > > > > 
> > > > > > diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > > > > index 1f4089c6f9d7..295f46eea2a5 100644
> > > > > > --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > > > > +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > > > > @@ -478,6 +478,11 @@ static int mlx5_vdpa_poll_one(struct mlx5_vdpa_cq *vcq)
> > > > > >  static void mlx5_vdpa_handle_completions(struct mlx5_vdpa_virtqueue *mvq, int num)
> > > > > >  {
> > > > > >  	mlx5_cq_set_ci(&mvq->cq.mcq);
> > > > > > +
> > > > > > +	/* make sure CQ cosumer update is visible to the hardware before updating
> > > > > > +	 * RX doorbell record.
> > > > > > +	 */
> > > > > > +	wmb();
> > > > > >  	rx_post(&mvq->vqqp, num);
> > > > > >  	if (mvq->event_cb.callback)
> > > > > >  		mvq->event_cb.callback(mvq->event_cb.private);
> > > > > > -- 
> > > > > > 2.27.0
> > > > > 
> > > 
>