Received-SPF: pass (google.com: domain of linux-kernel+bounces-89316-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
From: Long Li <longli@microsoft.com>
To: Konstantin Taranov <kotaranov@linux.microsoft.com>, Konstantin Taranov
	<kotaranov@microsoft.com>, "sharmaajay@microsoft.com"
	<sharmaajay@microsoft.com>, "jgg@ziepe.ca" <jgg@ziepe.ca>, "leon@kernel.org"
	<leon@kernel.org>
CC: "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH rdma-next v2 2/2] RDMA/mana_ib: Use virtual address in dma
 regions for MRs
Thread-Topic: [PATCH rdma-next v2 2/2] RDMA/mana_ib: Use virtual address in
 dma regions for MRs
Thread-Index: AQHaaIUJDd9nmAvQZky1S7+zCOe6Q7EjyPqw
Date: Sat, 2 Mar 2024 02:57:59 +0000
Message-ID:
 <SJ1PR21MB3457D7E2DE4C06C9C5D2C625CE5D2@SJ1PR21MB3457.namprd21.prod.outlook.com>
References: <1708932339-27914-1-git-send-email-kotaranov@linux.microsoft.com>
 <1708932339-27914-3-git-send-email-kotaranov@linux.microsoft.com>
In-Reply-To: <1708932339-27914-3-git-send-email-kotaranov@linux.microsoft.com>
Accept-Language: en-US
Content-Language: en-US
msip_labels:
 MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=f03f0774-6a73-4eaa-8790-40d8d41ec645;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2024-03-02T02:54:56Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47;
authentication-results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=microsoft.com;
x-ms-publictraffictype: Email
x-ms-traffictypediagnostic: SJ1PR21MB3457:EE_|SJ1PR21MB3528:EE_
x-ms-office365-filtering-correlation-id: 5075939c-b060-47d4-96b6-08dc3a6491e8
x-ms-exchange-senderadcheck: 1
x-ms-exchange-antispam-relay: 0
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info:
 YKn8s44/+b0KO8WeF3sMMEfUokN9VPsk4WNvkVvT3iae2oFOrVqZMMuhVPVap6osY4Nuhg4Fw5x+yeZ5TR+tsIIQSwTOgqxgYlGriDfLm0yUV2iktpb1gcNOHGkNiV0UBnSucb+dtiLovkh7vdQbyj0LtPlRRXxQ9nMbEnMb51EfDp1ClF2I5eGKu3aQm/LP4YTXNHv7DayomJJOk2nIFak3f9ZTcBwVMWAkjQmYnGNxEhBSFAEU19sYfc9hGZFj6+BJsS78roqnpZxJSQaRt+L0dAbOqON53yOy2mgcrwOaJhzekfpntzrWHa3W9letmXyoytiFdNulbP3KwD/9Jx7GIBaTRY3Fs+3HA5h/yijfIAcOVdeuXTyYmjOXSXmE8JUx9eNvsiuXunm8mcwq0kUorIxh1rs4xfbjIKuEtMvdbndpK2ibhm2XP3Hz+rPvpvJH+CP4lseq/DkalcKgrM5P/lQrrdAvIgeKVhRcLTcaBMWBJAVU1zNIU3fRREfH1gMWwwdLG8mejq3qYXXOPiJgYyEGn4PP2WxfHm+lxbndf+jP/se1O0gmlM8zpuDPJCySe1Jj1BOl6IzrTpiZsPZLw3shdPs/Gylait2D9b7sLZaMmGeA/0XWMwPro1aoVyivMRBO01ZNXyCYq9vTKkLu56GcojyQBWwryuB4YvUHVcf0Ps7KoBqIrKxxfEiTsSVoR453HFGpttiDhfgBSw==
x-forefront-antispam-report:
 CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SJ1PR21MB3457.namprd21.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(38070700009);DIR:OUT;SFP:1102;
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0:
 =?us-ascii?Q?blvMDYEeNUwTu//+LqidbHL6J9bbKUgQ+g4qKmmBjpZtr5vYiuO1I77KEKCZ?=
 =?us-ascii?Q?uNrA/TzSeFJWnvgWCYe79GcSauWKRRILeIr5kUMjaxhgJX3thHsHWn86WZnl?=
 =?us-ascii?Q?3hi+liFY1ylgchWKKQZKljN4EB/iTUpYYPC6ySLQL+LbUOM2n8w/8S582hgu?=
 =?us-ascii?Q?gAizu4xSuXzEWxrlHzwNNijaUnoUqMX8WfbWRgQz3+IKPVgs1hr8bnSgACeD?=
 =?us-ascii?Q?rwyEXTMm/+tpG4wdViAqhU8/O8bbxlgLdVrxOtGpAHRTnVQ61LjUdK2Hbsn2?=
 =?us-ascii?Q?4PyD3e9RnJE5xJ+GA8JnXXU4/DDqEIBq85w37lA95m9YOEjrtLd+Jut/cL+K?=
 =?us-ascii?Q?D3VXpHlHuetFIFImljFtin8/778PfnI/WWkuh4yAriyAQ/Zto1VvzwYFUW1+?=
 =?us-ascii?Q?QtQ79FtHvGat3FMCBoiXKqfRefthHwELMS/luo91WcYHu7R0OZl10SL6EGzu?=
 =?us-ascii?Q?3Sj0m6rMP0Vw63vv4mZ4HYIf51Hkqu8WTScokkdeRS/gQSlWAU3myJluGDzD?=
 =?us-ascii?Q?jdTVnBiIk1v7OUufsyyTBUdrwmtG2xxG5P7kBpCEBATanojnFTwi65AbKqux?=
 =?us-ascii?Q?As05ZWxK/NB53j2pr/j8AFNzHfl5+K1uNPwwcg86LNpot4F21vscUmPYIbuC?=
 =?us-ascii?Q?fjXdRXb3Z/hJn3+Ouh/H+3hGsc+258E6BzGo+b5SFsc4KEouC1BeJCu8LIMk?=
 =?us-ascii?Q?ifekLcTFS1SesrfN9W6Mp7vE1zYROjcSpDTrzHZeyBO+AoCaUykHuwjOKPqv?=
 =?us-ascii?Q?2caKYMTS4kVtkat5LJBqU1q8HnGqwAPvj09zUEqQyDgr7HunVEBH+a1Iv702?=
 =?us-ascii?Q?AKTalx4J4pBKTGRT24Ugl6dRgdiqLzvnkn+6YDisYeXCerDX5fP5lSRyVAFc?=
 =?us-ascii?Q?N5pFmXD0O7eOs+QyAvx89dxtB82V/OOxSqD9Vv8oEIy1q8IcK8dgJolZJp6a?=
 =?us-ascii?Q?EyQeLrdnw5SjjfRZdemaoqcugxVL+jmorPViwPonsN3bO0kQX9sNrtnzfehM?=
 =?us-ascii?Q?fDMn4SAxMZEBjPI2mYPd/KUboxHBvGx09s/Y5rgyzEbfNx/bzrGLXzn/iKpa?=
 =?us-ascii?Q?VLMx3LuF+yMnSfcpyPqExKtq0rRCMF4niFdjc+t9usxS0z7UmYv2xwQWzagv?=
 =?us-ascii?Q?aWAWBm6GLmQGRRFPUxtlBOkdML/et3bBu0V1WJCfkG9PbCplocbGm0SezUao?=
 =?us-ascii?Q?9FhAcANL20oDUbv7l9O6Sq1FqwfNgwoma7YAkj9gtuJW0hsh9c8JkeM3bHH3?=
 =?us-ascii?Q?B3QNhpX7b/rYEI4MLDRz+1BOLuGKgXN/fuYJIiSSC4Q2ibI9W7rS+sJExa1G?=
 =?us-ascii?Q?QLX1j/wy3vJGBhi4BXR2vIM+6Sk5mklmv4/lXofnKh9APeJAMyOG3MZ50l+K?=
 =?us-ascii?Q?hbDjZ8MtKY+XRfqmp2GXpEJxAd9dqFu7FtEMWytBHl42LZuDzWsVvuAjSnf6?=
 =?us-ascii?Q?D5o5FjfkYj5Jdofa3hzmwrDzD9wcHXrpdgeo7im0mZ1C27nzx+42BO3b0R4a?=
 =?us-ascii?Q?BAH9cpqQNq6niEquHyDVMLmuxo0powIPb7vz6XVpLmRZo54iw7HXKzztumgO?=
 =?us-ascii?Q?Yn8Q85liEMRriNkmz+LkkkPceb8lAsXcNVmy+K4o?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-OriginatorOrg: microsoft.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SJ1PR21MB3457.namprd21.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5075939c-b060-47d4-96b6-08dc3a6491e8
X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Mar 2024 02:57:59.8537
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: RzsnEdPNicnhfoytuZgCacsP/tN4NJEZLJ49wyHdjlcCN3Wi8hCpvgD4xsF8hBfGKKXGNiOHSRm5TaIayWZJgg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ1PR21MB3528

> Subject: [PATCH rdma-next v2 2/2] RDMA/mana_ib: Use virtual address in
> dma regions for MRs
>=20
> From: Konstantin Taranov <kotaranov@microsoft.com>
>=20
> Introduce mana_ib_create_dma_region() to create dma regions with iova for
> MRs.
>=20
> For dma regions that must have a zero dma offset (e.g., for queues),
> mana_ib_create_zero_offset_dma_region() is added.
> To get the zero offset, ib_umem_find_best_pgoff() is used with zero
> pgoff_bitmask.
>=20
> Signed-off-by: Konstantin Taranov <kotaranov@microsoft.com>

Do you need a "Fixes:" for this patch?

Also, you need to include a "change log" in the cover letter 0/2 since this=
 is a v2 seires.

Long

> ---
>  drivers/infiniband/hw/mana/cq.c      |  4 +--
>  drivers/infiniband/hw/mana/main.c    | 40 +++++++++++++++++++++-------
>  drivers/infiniband/hw/mana/mana_ib.h |  7 +++--
>  drivers/infiniband/hw/mana/mr.c      |  4 +--
>  drivers/infiniband/hw/mana/qp.c      |  6 ++---
>  drivers/infiniband/hw/mana/wq.c      |  4 +--
>  6 files changed, 45 insertions(+), 20 deletions(-)
>=20
> diff --git a/drivers/infiniband/hw/mana/cq.c
> b/drivers/infiniband/hw/mana/cq.c index 83d20c3f0..4a71e678d 100644
> --- a/drivers/infiniband/hw/mana/cq.c
> +++ b/drivers/infiniband/hw/mana/cq.c
> @@ -48,7 +48,7 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const struct
> ib_cq_init_attr *attr,
>  		return err;
>  	}
>=20
> -	err =3D mana_ib_gd_create_dma_region(mdev, cq->umem, &cq-
> >gdma_region);
> +	err =3D mana_ib_create_zero_offset_dma_region(mdev, cq->umem,
> +&cq->gdma_region);
>  	if (err) {
>  		ibdev_dbg(ibdev,
>  			  "Failed to create dma region for create cq, %d\n",
> @@ -57,7 +57,7 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const struct
> ib_cq_init_attr *attr,
>  	}
>=20
>  	ibdev_dbg(ibdev,
> -		  "mana_ib_gd_create_dma_region ret %d gdma_region
> 0x%llx\n",
> +		  "create_dma_region ret %d gdma_region 0x%llx\n",
>  		  err, cq->gdma_region);
>=20
>  	/*
> diff --git a/drivers/infiniband/hw/mana/main.c
> b/drivers/infiniband/hw/mana/main.c
> index dd570832d..30b874938 100644
> --- a/drivers/infiniband/hw/mana/main.c
> +++ b/drivers/infiniband/hw/mana/main.c
> @@ -301,8 +301,8 @@ mana_ib_gd_add_dma_region(struct mana_ib_dev
> *dev, struct gdma_context *gc,
>  	return 0;
>  }
>=20
> -int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct
> ib_umem *umem,
> -				 mana_handle_t *gdma_region)
> +static int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct
> ib_umem *umem,
> +					mana_handle_t *gdma_region,
> unsigned long page_sz)
>  {
>  	struct gdma_dma_region_add_pages_req *add_req =3D NULL;
>  	size_t num_pages_processed =3D 0, num_pages_to_handle; @@ -314,7
> +314,6 @@ int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev,
> struct ib_umem *umem,
>  	size_t max_pgs_create_cmd;
>  	struct gdma_context *gc;
>  	size_t num_pages_total;
> -	unsigned long page_sz;
>  	unsigned int tail =3D 0;
>  	u64 *page_addr_list;
>  	void *request_buf;
> @@ -323,12 +322,6 @@ int mana_ib_gd_create_dma_region(struct
> mana_ib_dev *dev, struct ib_umem *umem,
>  	gc =3D mdev_to_gc(dev);
>  	hwc =3D gc->hwc.driver_data;
>=20
> -	/* Hardware requires dma region to align to chosen page size */
> -	page_sz =3D ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, 0);
> -	if (!page_sz) {
> -		ibdev_dbg(&dev->ib_dev, "failed to find page size.\n");
> -		return -ENOMEM;
> -	}
>  	num_pages_total =3D ib_umem_num_dma_blocks(umem, page_sz);
>=20
>  	max_pgs_create_cmd =3D
> @@ -414,6 +407,35 @@ int mana_ib_gd_create_dma_region(struct
> mana_ib_dev *dev, struct ib_umem *umem,
>  	return err;
>  }
>=20
> +int mana_ib_create_dma_region(struct mana_ib_dev *dev, struct ib_umem
> *umem,
> +			      mana_handle_t *gdma_region, u64 virt) {
> +	unsigned long page_sz;
> +
> +	page_sz =3D ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, virt);
> +	if (!page_sz) {
> +		ibdev_dbg(&dev->ib_dev, "Failed to find page size.\n");
> +		return -EINVAL;
> +	}
> +
> +	return mana_ib_gd_create_dma_region(dev, umem, gdma_region,
> page_sz);
> +}
> +
> +int mana_ib_create_zero_offset_dma_region(struct mana_ib_dev *dev,
> struct ib_umem *umem,
> +					  mana_handle_t *gdma_region)
> +{
> +	unsigned long page_sz;
> +
> +	/* Hardware requires dma region to align to chosen page size */
> +	page_sz =3D ib_umem_find_best_pgoff(umem, PAGE_SZ_BM, 0);
> +	if (!page_sz) {
> +		ibdev_dbg(&dev->ib_dev, "Failed to find page size.\n");
> +		return -ENOMEM;
> +	}
> +
> +	return mana_ib_gd_create_dma_region(dev, umem, gdma_region,
> page_sz);
> +}
> +
>  int mana_ib_gd_destroy_dma_region(struct mana_ib_dev *dev, u64
> gdma_region)  {
>  	struct gdma_context *gc =3D mdev_to_gc(dev); diff --git
> a/drivers/infiniband/hw/mana/mana_ib.h
> b/drivers/infiniband/hw/mana/mana_ib.h
> index 6a03ae645..f83390eeb 100644
> --- a/drivers/infiniband/hw/mana/mana_ib.h
> +++ b/drivers/infiniband/hw/mana/mana_ib.h
> @@ -160,8 +160,11 @@ static inline struct net_device
> *mana_ib_get_netdev(struct ib_device *ibdev, u32
>=20
>  int mana_ib_install_cq_cb(struct mana_ib_dev *mdev, struct mana_ib_cq
> *cq);
>=20
> -int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct
> ib_umem *umem,
> -				 mana_handle_t *gdma_region);
> +int mana_ib_create_zero_offset_dma_region(struct mana_ib_dev *dev,
> struct ib_umem *umem,
> +					  mana_handle_t *gdma_region);
> +
> +int mana_ib_create_dma_region(struct mana_ib_dev *dev, struct ib_umem
> *umem,
> +			      mana_handle_t *gdma_region, u64 virt);
>=20
>  int mana_ib_gd_destroy_dma_region(struct mana_ib_dev *dev,
>  				  mana_handle_t gdma_region);
> diff --git a/drivers/infiniband/hw/mana/mr.c
> b/drivers/infiniband/hw/mana/mr.c index ee4d4f834..b70b13484 100644
> --- a/drivers/infiniband/hw/mana/mr.c
> +++ b/drivers/infiniband/hw/mana/mr.c
> @@ -127,7 +127,7 @@ struct ib_mr *mana_ib_reg_user_mr(struct ib_pd
> *ibpd, u64 start, u64 length,
>  		goto err_free;
>  	}
>=20
> -	err =3D mana_ib_gd_create_dma_region(dev, mr->umem,
> &dma_region_handle);
> +	err =3D mana_ib_create_dma_region(dev, mr->umem,
> &dma_region_handle,
> +iova);
>  	if (err) {
>  		ibdev_dbg(ibdev, "Failed create dma region for user-mr,
> %d\n",
>  			  err);
> @@ -135,7 +135,7 @@ struct ib_mr *mana_ib_reg_user_mr(struct ib_pd
> *ibpd, u64 start, u64 length,
>  	}
>=20
>  	ibdev_dbg(ibdev,
> -		  "mana_ib_gd_create_dma_region ret %d gdma_region
> %llx\n", err,
> +		  "create_dma_region ret %d gdma_region %llx\n", err,
>  		  dma_region_handle);
>=20
>  	mr_params.pd_handle =3D pd->pd_handle;
> diff --git a/drivers/infiniband/hw/mana/qp.c
> b/drivers/infiniband/hw/mana/qp.c index 5d4c05dcd..6e7627745 100644
> --- a/drivers/infiniband/hw/mana/qp.c
> +++ b/drivers/infiniband/hw/mana/qp.c
> @@ -357,8 +357,8 @@ static int mana_ib_create_qp_raw(struct ib_qp *ibqp,
> struct ib_pd *ibpd,
>  	}
>  	qp->sq_umem =3D umem;
>=20
> -	err =3D mana_ib_gd_create_dma_region(mdev, qp->sq_umem,
> -					   &qp->sq_gdma_region);
> +	err =3D mana_ib_create_zero_offset_dma_region(mdev, qp->sq_umem,
> +						    &qp->sq_gdma_region);
>  	if (err) {
>  		ibdev_dbg(&mdev->ib_dev,
>  			  "Failed to create dma region for create qp-raw,
> %d\n", @@ -367,7 +367,7 @@ static int mana_ib_create_qp_raw(struct ib_qp
> *ibqp, struct ib_pd *ibpd,
>  	}
>=20
>  	ibdev_dbg(&mdev->ib_dev,
> -		  "mana_ib_gd_create_dma_region ret %d gdma_region
> 0x%llx\n",
> +		  "create_dma_region ret %d gdma_region 0x%llx\n",
>  		  err, qp->sq_gdma_region);
>=20
>  	/* Create a WQ on the same port handle used by the Ethernet */ diff -
> -git a/drivers/infiniband/hw/mana/wq.c b/drivers/infiniband/hw/mana/wq.c
> index 372d36151..7c9c69962 100644
> --- a/drivers/infiniband/hw/mana/wq.c
> +++ b/drivers/infiniband/hw/mana/wq.c
> @@ -46,7 +46,7 @@ struct ib_wq *mana_ib_create_wq(struct ib_pd *pd,
>  	wq->wq_buf_size =3D ucmd.wq_buf_size;
>  	wq->rx_object =3D INVALID_MANA_HANDLE;
>=20
> -	err =3D mana_ib_gd_create_dma_region(mdev, wq->umem, &wq-
> >gdma_region);
> +	err =3D mana_ib_create_zero_offset_dma_region(mdev, wq->umem,
> +&wq->gdma_region);
>  	if (err) {
>  		ibdev_dbg(&mdev->ib_dev,
>  			  "Failed to create dma region for create wq, %d\n",
> @@ -55,7 +55,7 @@ struct ib_wq *mana_ib_create_wq(struct ib_pd *pd,
>  	}
>=20
>  	ibdev_dbg(&mdev->ib_dev,
> -		  "mana_ib_gd_create_dma_region ret %d gdma_region
> 0x%llx\n",
> +		  "create_dma_region ret %d gdma_region 0x%llx\n",
>  		  err, wq->gdma_region);
>=20
>  	/* WQ ID is returned at wq_create time, doesn't know the value yet */
> --
> 2.43.0