Received-SPF: pass (google.com: domain of linux-kernel+bounces-210421-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
From: "Zeng, Oak" <oak.zeng@intel.com>
To: Leon Romanovsky <leon@kernel.org>
CC: Jason Gunthorpe <jgg@ziepe.ca>, Christoph Hellwig <hch@lst.de>, "Robin
 Murphy" <robin.murphy@arm.com>, Marek Szyprowski <m.szyprowski@samsung.com>,
	Joerg Roedel <joro@8bytes.org>, Will Deacon <will@kernel.org>, "Chaitanya
 Kulkarni" <chaitanyak@nvidia.com>, "Brost, Matthew"
	<matthew.brost@intel.com>, "Hellstrom, Thomas" <thomas.hellstrom@intel.com>,
	Jonathan Corbet <corbet@lwn.net>, Jens Axboe <axboe@kernel.dk>, Keith Busch
	<kbusch@kernel.org>, Sagi Grimberg <sagi@grimberg.me>, Yishai Hadas
	<yishaih@nvidia.com>, Shameer Kolothum
	<shameerali.kolothum.thodi@huawei.com>, "Tian, Kevin" <kevin.tian@intel.com>,
	Alex Williamson <alex.williamson@redhat.com>,
	=?iso-8859-1?Q?J=E9r=F4me_Glisse?= <jglisse@redhat.com>, Andrew Morton
	<akpm@linux-foundation.org>, "linux-doc@vger.kernel.org"
	<linux-doc@vger.kernel.org>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, "linux-block@vger.kernel.org"
	<linux-block@vger.kernel.org>, "linux-rdma@vger.kernel.org"
	<linux-rdma@vger.kernel.org>, "iommu@lists.linux.dev"
	<iommu@lists.linux.dev>, "linux-nvme@lists.infradead.org"
	<linux-nvme@lists.infradead.org>, "kvm@vger.kernel.org"
	<kvm@vger.kernel.org>, "linux-mm@kvack.org" <linux-mm@kvack.org>, "Bart Van
 Assche" <bvanassche@acm.org>, Damien Le Moal
	<damien.lemoal@opensource.wdc.com>, Amir Goldstein <amir73il@gmail.com>,
	"josef@toxicpanda.com" <josef@toxicpanda.com>, "Martin K. Petersen"
	<martin.petersen@oracle.com>, "daniel@iogearbox.net" <daniel@iogearbox.net>,
	"Williams, Dan J" <dan.j.williams@intel.com>, "jack@suse.com"
	<jack@suse.com>, Zhu Yanjun <zyjzyj2000@gmail.com>, "Bommu, Krishnaiah"
	<krishnaiah.bommu@intel.com>, "Ghimiray, Himal Prasad"
	<himal.prasad.ghimiray@intel.com>
Subject: RE: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to two steps
Thread-Topic: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to two
 steps
Thread-Index: AQHanOEtbi0VHTnLn0mNwgftYB7xRbGEmFqggAEftYCAO5ykoIAAFS8AgAAFauCAAA0xgIAAMSmggAFFTICAACZPQA==
Date: Tue, 11 Jun 2024 18:26:23 +0000
Message-ID: <SA1PR11MB69915818E4DC6FFAD620D62C92C72@SA1PR11MB6991.namprd11.prod.outlook.com>
References: <cover.1709635535.git.leon@kernel.org>
 <SA1PR11MB6991CB2B1398948F4241E51992182@SA1PR11MB6991.namprd11.prod.outlook.com>
 <20240503164239.GB901876@ziepe.ca>
 <PH7PR11MB70047236290DC1CFF9150B8592C62@PH7PR11MB7004.namprd11.prod.outlook.com>
 <20240610161826.GA4966@unreal>
 <PH7PR11MB7004A071F27B4CF45740B87E92C62@PH7PR11MB7004.namprd11.prod.outlook.com>
 <20240610172501.GJ791043@ziepe.ca>
 <PH7PR11MB7004DDE9816D92F690A5C0B692C62@PH7PR11MB7004.namprd11.prod.outlook.com>
 <20240611154515.GC4966@unreal>
In-Reply-To: <20240611154515.GC4966@unreal>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?MmR3hg8IsWJYIvQ71iVEWnAWp1lQ/tQaNJImq1Ka4QEmv9pFZDBum7n0Sg?=
 =?iso-8859-1?Q?SjVYcmWUL2yhK/CrSxRmoxJa9+koYjxmS7ytmHyZP2fHEqwtpcF+WyZQEJ?=
 =?iso-8859-1?Q?weG2Lpm3zFVh0Pqubww/wq5aPRIoHl6+2YeTEgpFxH7v6CX6PvINhwxAiL?=
 =?iso-8859-1?Q?RWKmB/kAsP4PPVuFBsGB40eryd7/szxpmqD1Yhdnwa+WSMU8QfdDoQTxan?=
 =?iso-8859-1?Q?kyQFZ29d56xRFhxfr1jscs3D4Zk7AZkrAjxJCiJrLq9xmiYahMByhGGg6l?=
 =?iso-8859-1?Q?QshucYdvJiyGInAVRT2fFFzp3u4CQW9HnktgYpTdwpmsnwZqk1B1qP8S8c?=
 =?iso-8859-1?Q?fomewPW/NV5G+igi0iDWZEi/iquK3C/X4bkD+FQyM8dOlVbt3mRnHzSQly?=
 =?iso-8859-1?Q?spHXC278Jr9c7qTrzNRCJxsyTv4FsS5g3VQHSPK9X/hcbCM6zxGf9yDoGE?=
 =?iso-8859-1?Q?7f+CqCN1GbwEFMCQmkELuE3jixXCrdKxrMvBaMgGBj2XL8zpyadn6VgMAF?=
 =?iso-8859-1?Q?vWf4bwE2SK3TbdDscWX+vYmQjNd7RoMfqt4C6vv+b7jTQfBRVgQOGfXMPs?=
 =?iso-8859-1?Q?bCgVe0Se+gh5BEVjp6lePn/+/4/K1CdPhQieX9DGf7AsG8d8ca5CDBlnal?=
 =?iso-8859-1?Q?qC//XOERJw2ZMhwA0+EA/RYo7jOlJseoazzJEnShZFxQqU9cV/SCQ/x9lA?=
 =?iso-8859-1?Q?YUTtV/uGf451O1njCQQIIFXOqbO6ZuRnP0+Cxnwjsg8SmQf929zcSHs+Fd?=
 =?iso-8859-1?Q?1I51nV/26LtnTU8cjch1nzfWbpikH1CuBcvv6Nj1gK3f9BAVtPH7ULjYRB?=
 =?iso-8859-1?Q?lYHAn/Xv5VX9YrMSoocBZTVm6qI08PUZXP9jVANEvyhtBr9iAKwlj6hyG8?=
 =?iso-8859-1?Q?REECAtmaHevYU+fcqotsdpuPthoedaAOev2DCfZ4WxR79PxwyCORHNYgKl?=
 =?iso-8859-1?Q?s1yfr31tNX8Ag9LrqKDGz7JKjGHW17VDzCuiUP57K4tEJE//yzT1MeyOYJ?=
 =?iso-8859-1?Q?oOl7zo7F7V4Pk9zTIL8NdyNrj5mNN0kNVT+hWHjVhaZ24lh5yKYueAs687?=
 =?iso-8859-1?Q?bHd0aAEeMvgYUYDjxb2ZZVv626SW/lP8IOyH8FdsApAVw53or5cHJCeqo9?=
 =?iso-8859-1?Q?eLSj8PQ87XrAORUeFrk+XkfuPcBJUndMViWNblnfkMZ/+cm23t3odH412c?=
 =?iso-8859-1?Q?tZ4/6n8IIBS+pXYQu7zbZ/yAwW1zfLY0dh3WvY5meXgm8uSLLwEyhmMcw2?=
 =?iso-8859-1?Q?PhWMrMdmz/M3r8i0/+PONUhJbJTGEIIV7ESymSghoCT3hkJpiYGwUI/ttp?=
 =?iso-8859-1?Q?kytiPnamQSmMuxfi3M0kls2xXJUG5YWbsdspnfzRpv2pJs9pfLjOAplFFQ?=
 =?iso-8859-1?Q?vxkWLpe+uJNoCslaUcxNL8HyC8Sl3rfKWCPLcFrxS0AnmiTk3USMutHh/m?=
 =?iso-8859-1?Q?lPk03XdFWBsjX6NbXK4WercrpLk6XYFNqdFCq0gWHKg0fMqGqlB3Fo7H/D?=
 =?iso-8859-1?Q?eLjzA/V0OY4tgovG1dBbeZ9+qoGPscmBdFzC8NSF4nCm+qXtJMxKGHDkca?=
 =?iso-8859-1?Q?SbfBhM+2ioUqXW7hdt36l/OA3dGI5rEPcxdB+YIb+fhyN+BFP2ahXvbdTf?=
 =?iso-8859-1?Q?tVSemmqX+Fs9c=3D?=
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SA1PR11MB6991.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f329519a-1364-4c08-8771-08dc8a43ffc2
X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Jun 2024 18:26:23.7564
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: BbxlFgIYjKb36+W3Oaxtw7TpQeUlpP4aGP9T1TE1PBZPfEdv2KV9/iMJrS9nY6aFMfVagEs85QKNRrxe2UgDsQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB7115
X-OriginatorOrg: intel.com

Thank you Leon. That is helpful.

I also have another very na=EFve question. I don't understand what is the i=
ova address. I previously thought the iova address space is the same as the=
 dma_address space when iommu is involved. I thought the dma_alloc_iova wou=
ld allocate some contiguous iova address range and later dma_link_range fun=
ction would link a physical page to the iova address and return the iova ad=
dress. In other words, I thought the dma_address is iova address, and the i=
ommu page table translate a dma_address or iova address to the physical add=
ress.

But from my print below, my above understanding is obviously wrong: the iov=
a.dma_addr is 0 and the dma_address returned from dma_link_range is none ze=
ro... Can you help me what is iova address? Is iova address iommu related? =
Since dma_link_range returns a non-iova address, does this function allocat=
e the dma-address itself? Is dma-address correlated with iova address?

Oak=20

> -----Original Message-----
> From: Leon Romanovsky <leon@kernel.org>
> Sent: Tuesday, June 11, 2024 11:45 AM
> To: Zeng, Oak <oak.zeng@intel.com>
> Cc: Jason Gunthorpe <jgg@ziepe.ca>; Christoph Hellwig <hch@lst.de>; Robin
> Murphy <robin.murphy@arm.com>; Marek Szyprowski
> <m.szyprowski@samsung.com>; Joerg Roedel <joro@8bytes.org>; Will
> Deacon <will@kernel.org>; Chaitanya Kulkarni <chaitanyak@nvidia.com>;
> Brost, Matthew <matthew.brost@intel.com>; Hellstrom, Thomas
> <thomas.hellstrom@intel.com>; Jonathan Corbet <corbet@lwn.net>; Jens
> Axboe <axboe@kernel.dk>; Keith Busch <kbusch@kernel.org>; Sagi
> Grimberg <sagi@grimberg.me>; Yishai Hadas <yishaih@nvidia.com>;
> Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>; Tian, Kevin
> <kevin.tian@intel.com>; Alex Williamson <alex.williamson@redhat.com>;
> J=E9r=F4me Glisse <jglisse@redhat.com>; Andrew Morton <akpm@linux-
> foundation.org>; linux-doc@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-block@vger.kernel.org; linux-rdma@vger.kernel.org;
> iommu@lists.linux.dev; linux-nvme@lists.infradead.org;
> kvm@vger.kernel.org; linux-mm@kvack.org; Bart Van Assche
> <bvanassche@acm.org>; Damien Le Moal
> <damien.lemoal@opensource.wdc.com>; Amir Goldstein
> <amir73il@gmail.com>; josef@toxicpanda.com; Martin K. Petersen
> <martin.petersen@oracle.com>; daniel@iogearbox.net; Williams, Dan J
> <dan.j.williams@intel.com>; jack@suse.com; Zhu Yanjun
> <zyjzyj2000@gmail.com>; Bommu, Krishnaiah
> <krishnaiah.bommu@intel.com>; Ghimiray, Himal Prasad
> <himal.prasad.ghimiray@intel.com>
> Subject: Re: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to
> two steps
>=20
> On Mon, Jun 10, 2024 at 09:28:04PM +0000, Zeng, Oak wrote:
> > Hi Jason, Leon,
> >
> > I was able to fix the issue from my side. Things work fine now. I got t=
wo
> questions though:
> >
> > 1) The value returned from dma_link_range function is not contiguous, s=
ee
> below print. The "linked pa" is the function return.
> > I think dma_map_sgtable API would return some contiguous dma address.
> Is the dma-map_sgtable api is more efficient regarding the iommu page tab=
le?
> i.e., try to use bigger page size, such as use 2M page size when it is po=
ssible.
> With your new API, does it also have such consideration? I vaguely
> remembered Jason mentioned such thing, but my print below doesn't look
> like so. Maybe I need to test bigger range (only 16 pages range in the te=
st of
> below printing). Comment?
>=20
> My API gives you the flexibility to use any page size you want. You can
> use 2M pages instead of 4K pages. The API doesn't enforce any page size.
>=20
> >
> > [17584.665126] drm_svm_hmmptr_map_dma_pages iova.dma_addr =3D 0x0,
> linked pa =3D 18ef3f000
> > [17584.665146] drm_svm_hmmptr_map_dma_pages iova.dma_addr =3D 0x0,
> linked pa =3D 190d00000
> > [17584.665150] drm_svm_hmmptr_map_dma_pages iova.dma_addr =3D 0x0,
> linked pa =3D 190024000
> > [17584.665153] drm_svm_hmmptr_map_dma_pages iova.dma_addr =3D 0x0,
> linked pa =3D 178e89000
> >
> > 2) in the comment of dma_link_range function, it is said: " @dma_offset
> needs to be advanced by the caller with the size of previous page that wa=
s
> linked + DMA address returned for the previous page".
> > Is this description correct? I don't understand the part "+ DMA address
> returned for the previous page ".
> > In my codes, let's say I call this function to link 10 pages, the first
> dma_offset is 0, second is 4k, third 8k. This worked for me. I didn't add=
 the
> previously returned dma address.
> > Maybe I need more test. But any comment?
>=20
> You did it perfectly right. This is the correct way to advance dma_offset=
.
>=20
> Thanks
>=20
> >
> > Thanks,
> > Oak
> >
> > > -----Original Message-----
> > > From: Jason Gunthorpe <jgg@ziepe.ca>
> > > Sent: Monday, June 10, 2024 1:25 PM
> > > To: Zeng, Oak <oak.zeng@intel.com>
> > > Cc: Leon Romanovsky <leon@kernel.org>; Christoph Hellwig
> <hch@lst.de>;
> > > Robin Murphy <robin.murphy@arm.com>; Marek Szyprowski
> > > <m.szyprowski@samsung.com>; Joerg Roedel <joro@8bytes.org>; Will
> > > Deacon <will@kernel.org>; Chaitanya Kulkarni <chaitanyak@nvidia.com>;
> > > Brost, Matthew <matthew.brost@intel.com>; Hellstrom, Thomas
> > > <thomas.hellstrom@intel.com>; Jonathan Corbet <corbet@lwn.net>;
> Jens
> > > Axboe <axboe@kernel.dk>; Keith Busch <kbusch@kernel.org>; Sagi
> > > Grimberg <sagi@grimberg.me>; Yishai Hadas <yishaih@nvidia.com>;
> > > Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>; Tian,
> Kevin
> > > <kevin.tian@intel.com>; Alex Williamson <alex.williamson@redhat.com>;
> > > J=E9r=F4me Glisse <jglisse@redhat.com>; Andrew Morton <akpm@linux-
> > > foundation.org>; linux-doc@vger.kernel.org; linux-
> kernel@vger.kernel.org;
> > > linux-block@vger.kernel.org; linux-rdma@vger.kernel.org;
> > > iommu@lists.linux.dev; linux-nvme@lists.infradead.org;
> > > kvm@vger.kernel.org; linux-mm@kvack.org; Bart Van Assche
> > > <bvanassche@acm.org>; Damien Le Moal
> > > <damien.lemoal@opensource.wdc.com>; Amir Goldstein
> > > <amir73il@gmail.com>; josef@toxicpanda.com; Martin K. Petersen
> > > <martin.petersen@oracle.com>; daniel@iogearbox.net; Williams, Dan J
> > > <dan.j.williams@intel.com>; jack@suse.com; Zhu Yanjun
> > > <zyjzyj2000@gmail.com>; Bommu, Krishnaiah
> > > <krishnaiah.bommu@intel.com>; Ghimiray, Himal Prasad
> > > <himal.prasad.ghimiray@intel.com>
> > > Subject: Re: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to
> > > two steps
> > >
> > > On Mon, Jun 10, 2024 at 04:40:19PM +0000, Zeng, Oak wrote:
> > > > Thanks Leon and Yanjun for the reply!
> > > >
> > > > Based on the reply, we will continue use the current version for
> > > > test (as it is tested for vfio and rdma). We will switch to v1 once
> > > > it is fully tested/reviewed.
> > >
> > > I'm glad you are finding it useful, one of my interests with this wor=
k
> > > is to improve all the HMM users.
> > >
> > > Jason