Received: by 2002:a05:6a10:8c0a:0:0:0:0 with SMTP id go10csp40365pxb; Wed, 20 Jan 2021 00:14:58 -0800 (PST) X-Google-Smtp-Source: ABdhPJz1BBWe4TAbza42y8fRe42wqKOFqay0nvspwaCcJTQ2zoyhxXJ+GEqvCBx/5ZJQdOzLi37m X-Received: by 2002:a05:6402:3508:: with SMTP id b8mr6554336edd.341.1611130498521; Wed, 20 Jan 2021 00:14:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1611130498; cv=none; d=google.com; s=arc-20160816; b=MUehKGFd1Noh7v8uiwYyvV7tKibDz7FWGVjzIZGHLP5X3DCMZxor/jbeW6iGKeuckc qxWTrdGsvjC6tLv7kL60brDD7L8n+x5fUuTRhrS13PROhtKwXpw8ft+wLuOx5AnHY+p5 WfZZS6cVRj1VlkyiuVuDQnon+MwvxY1KSeAWzP3b0H7dvJiqY/625s3eiCczWeCwvsmM HEWzzszoCEfdctkwxEx/obzO4Ormh5R52GY1xC4ktQ/w7T0IlTNcXES0Gm4xwB2fcmKA axwdYbjzT1uh0oSZdNRwtfEjWRWT5axLWni+hEd6zVB3L3bjZOfpaXcpHkP6ZHmumDEj CY0A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:dkim-signature:user-agent:in-reply-to :content-transfer-encoding:content-disposition:mime-version :references:message-id:subject:cc:to:from:date; bh=MVk+azMlkit5BNEnIYzuX/Z7yw0qeR4a9+6r4m/MV+Q=; b=HFEc2Y2B9sO5g4Otu3tz3ZOGoGKBiFWrg5DAt854xS1trzcPIa8Qn4WXwzoWlmpZYk 6U4qRSzgVNHD85gh+ETPmxof4rWIRSSzTw9a6LKElhLgmOF0imrdY75LsqRXRB0WKSBp IebsYtnkDen7Wnt0cJHBWw3bM/7fv6Z7bDgbDChSpaKRnYhLh2Z7LaMoOAhVnbjcWJtP ICPAvOEGbEcqP77lfEonP3568hqo5CjdfwqgSJr4QEW4Tb4tx4NBWiOar+Xi8E06lw2c Uvbsn5l54gTdSQcAwadBcucH3amKRPv2tExD4JELBP5i2TPrUPDlg7xa1DCg8fIAYuB5 VZvQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@nvidia.com header.s=n1 header.b=b5LhqIAD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id 20si581714edw.131.2021.01.20.00.14.34; Wed, 20 Jan 2021 00:14:58 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@nvidia.com header.s=n1 header.b=b5LhqIAD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730659AbhATIN6 (ORCPT + 99 others); Wed, 20 Jan 2021 03:13:58 -0500 Received: from hqnvemgate25.nvidia.com ([216.228.121.64]:16222 "EHLO hqnvemgate25.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730421AbhATIMk (ORCPT ); Wed, 20 Jan 2021 03:12:40 -0500 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate25.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Wed, 20 Jan 2021 00:12:00 -0800 Received: from mtl-vdi-166.wap.labs.mlnx (172.20.145.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Wed, 20 Jan 2021 08:11:58 +0000 Date: Wed, 20 Jan 2021 10:11:54 +0200 From: Eli Cohen To: "Michael S. Tsirkin" CC: Jason Wang , , , , Subject: Re: [PATCH v1] vdpa/mlx5: Fix memory key MTT population Message-ID: <20210120081154.GA132136@mtl-vdi-166.wap.labs.mlnx> References: <20210107071845.GA224876@mtl-vdi-166.wap.labs.mlnx> <07d336a3-7fc2-5e4a-667a-495b5bb755da@redhat.com> <20210120053619.GA126435@mtl-vdi-166.wap.labs.mlnx> <20210120025651-mutt-send-email-mst@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: <20210120025651-mutt-send-email-mst@kernel.org> User-Agent: Mutt/1.9.5 (bf161cf53efb) (2018-04-13) X-Originating-IP: [172.20.145.6] X-ClientProxiedBy: HQMAIL105.nvidia.com (172.20.187.12) To HQMAIL107.nvidia.com (172.20.187.13) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1611130320; bh=MVk+azMlkit5BNEnIYzuX/Z7yw0qeR4a9+6r4m/MV+Q=; h=Date:From:To:CC:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:Content-Transfer-Encoding: In-Reply-To:User-Agent:X-Originating-IP:X-ClientProxiedBy; b=b5LhqIADVb8tj+wzPULi6LXHqkISBQNqhucSSxTaTaPkT9t0F9WhwqLH7kYjavYpP TzFhnOisgZXW7ZbNwJfyBmqcUd+Gg74TXngfzckRI+W1rmAb4JVk7YB1ZWVIDmuEEf UmolAxD55NjVYwbMpEQwvSRMaWD3UpIyb8c8ThrL5dJq3YN7ikjeuLWXvfpLxmIOKM Np03E/vMj++Yq53lzDav0hlmp9anysEWtPTbgtMZ9063mHFYsE/FbTGL5q/GbYfrKv zNXqFK6PgpYcI2/dzn2XXv7znstS2lhMfgKduia7e4yua2aj2BBcRjj8dl9vkLMY1g Y0Lq8BIXZLJqQ== Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Jan 20, 2021 at 02:57:05AM -0500, Michael S. Tsirkin wrote: > On Wed, Jan 20, 2021 at 07:36:19AM +0200, Eli Cohen wrote: > > On Fri, Jan 08, 2021 at 04:38:55PM +0800, Jason Wang wrote: > >=20 > > Hi Michael, > > this patch is a fix. Are you going to merge it? >=20 > yes - in the next pull request. >=20 Great thanks. Can you send the path to your git tree where you keep the patches you intend to merge? > > >=20 > > > On 2021/1/7 =E4=B8=8B=E5=8D=883:18, Eli Cohen wrote: > > > > map_direct_mr() assumed that the number of scatter/gather entries > > > > returned by dma_map_sg_attrs() was equal to the number of segments = in > > > > the sgl list. This led to wrong population of the mkey object. Fix = this > > > > by properly referring to the returned value. > > > >=20 > > > > The hardware expects each MTT entry to contain the DMA address of a > > > > contiguous block of memory of size (1 << mr->log_size) bytes. > > > > dma_map_sg_attrs() can coalesce several sg entries into a single > > > > scatter/gather entry of contiguous DMA range so we need to scan the= list > > > > and refer to the size of each s/g entry. > > > >=20 > > > > In addition, get rid of fill_sg() which effect is overwritten by > > > > populate_mtts(). > > > >=20 > > > > Fixes: 94abbccdf291 ("vdpa/mlx5: Add shared memory registration cod= e") > > > > Signed-off-by: Eli Cohen > > > > --- > > > > V0->V1: > > > > 1. Fix typos > > > > 2. Improve changelog > > >=20 > > >=20 > > > Acked-by: Jason Wang > > >=20 > > >=20 > > > >=20 > > > > drivers/vdpa/mlx5/core/mlx5_vdpa.h | 1 + > > > > drivers/vdpa/mlx5/core/mr.c | 28 ++++++++++++-------------= --- > > > > 2 files changed, 13 insertions(+), 16 deletions(-) > > > >=20 > > > > diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5= /core/mlx5_vdpa.h > > > > index 5c92a576edae..08f742fd2409 100644 > > > > --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h > > > > +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h > > > > @@ -15,6 +15,7 @@ struct mlx5_vdpa_direct_mr { > > > > struct sg_table sg_head; > > > > int log_size; > > > > int nsg; > > > > + int nent; > > > > struct list_head list; > > > > u64 offset; > > > > }; > > > > diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/m= r.c > > > > index 4b6195666c58..d300f799efcd 100644 > > > > --- a/drivers/vdpa/mlx5/core/mr.c > > > > +++ b/drivers/vdpa/mlx5/core/mr.c > > > > @@ -25,17 +25,6 @@ static int get_octo_len(u64 len, int page_shift) > > > > return (npages + 1) / 2; > > > > } > > > > -static void fill_sg(struct mlx5_vdpa_direct_mr *mr, void *in) > > > > -{ > > > > - struct scatterlist *sg; > > > > - __be64 *pas; > > > > - int i; > > > > - > > > > - pas =3D MLX5_ADDR_OF(create_mkey_in, in, klm_pas_mtt); > > > > - for_each_sg(mr->sg_head.sgl, sg, mr->nsg, i) > > > > - (*pas) =3D cpu_to_be64(sg_dma_address(sg)); > > > > -} > > > > - > > > > static void mlx5_set_access_mode(void *mkc, int mode) > > > > { > > > > MLX5_SET(mkc, mkc, access_mode_1_0, mode & 0x3); > > > > @@ -45,10 +34,18 @@ static void mlx5_set_access_mode(void *mkc, int= mode) > > > > static void populate_mtts(struct mlx5_vdpa_direct_mr *mr, __be64 = *mtt) > > > > { > > > > struct scatterlist *sg; > > > > + int nsg =3D mr->nsg; > > > > + u64 dma_addr; > > > > + u64 dma_len; > > > > + int j =3D 0; > > > > int i; > > > > - for_each_sg(mr->sg_head.sgl, sg, mr->nsg, i) > > > > - mtt[i] =3D cpu_to_be64(sg_dma_address(sg)); > > > > + for_each_sg(mr->sg_head.sgl, sg, mr->nent, i) { > > > > + for (dma_addr =3D sg_dma_address(sg), dma_len =3D sg_dma_len(sg)= ; > > > > + nsg && dma_len; > > > > + nsg--, dma_addr +=3D BIT(mr->log_size), dma_len -=3D BIT(mr= ->log_size)) > > > > + mtt[j++] =3D cpu_to_be64(dma_addr); > > > > + } > > > > } > > > > static int create_direct_mr(struct mlx5_vdpa_dev *mvdev, struct m= lx5_vdpa_direct_mr *mr) > > > > @@ -64,7 +61,6 @@ static int create_direct_mr(struct mlx5_vdpa_dev = *mvdev, struct mlx5_vdpa_direct > > > > return -ENOMEM; > > > > MLX5_SET(create_mkey_in, in, uid, mvdev->res.uid); > > > > - fill_sg(mr, in); > > > > mkc =3D MLX5_ADDR_OF(create_mkey_in, in, memory_key_mkey_entry); > > > > MLX5_SET(mkc, mkc, lw, !!(mr->perm & VHOST_MAP_WO)); > > > > MLX5_SET(mkc, mkc, lr, !!(mr->perm & VHOST_MAP_RO)); > > > > @@ -276,8 +272,8 @@ static int map_direct_mr(struct mlx5_vdpa_dev *= mvdev, struct mlx5_vdpa_direct_mr > > > > done: > > > > mr->log_size =3D log_entity_size; > > > > mr->nsg =3D nsg; > > > > - err =3D dma_map_sg_attrs(dma, mr->sg_head.sgl, mr->nsg, DMA_BIDIR= ECTIONAL, 0); > > > > - if (!err) > > > > + mr->nent =3D dma_map_sg_attrs(dma, mr->sg_head.sgl, mr->nsg, DMA_= BIDIRECTIONAL, 0); > > > > + if (!mr->nent) > > > > goto err_map; > > > > err =3D create_direct_mr(mvdev, mr); > > >=20 >=20