Received-SPF: pass (google.com: domain of linux-kernel+bounces-208563-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
From: "Zeng, Oak" <oak.zeng@intel.com>
To: Leon Romanovsky <leon@kernel.org>
CC: Jason Gunthorpe <jgg@ziepe.ca>, Christoph Hellwig <hch@lst.de>, "Robin
 Murphy" <robin.murphy@arm.com>, Marek Szyprowski <m.szyprowski@samsung.com>,
	Joerg Roedel <joro@8bytes.org>, Will Deacon <will@kernel.org>, "Chaitanya
 Kulkarni" <chaitanyak@nvidia.com>, "Brost, Matthew"
	<matthew.brost@intel.com>, "Hellstrom, Thomas" <thomas.hellstrom@intel.com>,
	Jonathan Corbet <corbet@lwn.net>, Jens Axboe <axboe@kernel.dk>, Keith Busch
	<kbusch@kernel.org>, Sagi Grimberg <sagi@grimberg.me>, Yishai Hadas
	<yishaih@nvidia.com>, Shameer Kolothum
	<shameerali.kolothum.thodi@huawei.com>, "Tian, Kevin" <kevin.tian@intel.com>,
	Alex Williamson <alex.williamson@redhat.com>,
	=?iso-8859-1?Q?J=E9r=F4me_Glisse?= <jglisse@redhat.com>, Andrew Morton
	<akpm@linux-foundation.org>, "linux-doc@vger.kernel.org"
	<linux-doc@vger.kernel.org>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, "linux-block@vger.kernel.org"
	<linux-block@vger.kernel.org>, "linux-rdma@vger.kernel.org"
	<linux-rdma@vger.kernel.org>, "iommu@lists.linux.dev"
	<iommu@lists.linux.dev>, "linux-nvme@lists.infradead.org"
	<linux-nvme@lists.infradead.org>, "kvm@vger.kernel.org"
	<kvm@vger.kernel.org>, "linux-mm@kvack.org" <linux-mm@kvack.org>, "Bart Van
 Assche" <bvanassche@acm.org>, Damien Le Moal
	<damien.lemoal@opensource.wdc.com>, Amir Goldstein <amir73il@gmail.com>,
	"josef@toxicpanda.com" <josef@toxicpanda.com>, "Martin K. Petersen"
	<martin.petersen@oracle.com>, "daniel@iogearbox.net" <daniel@iogearbox.net>,
	"Williams, Dan J" <dan.j.williams@intel.com>, "jack@suse.com"
	<jack@suse.com>, Zhu Yanjun <zyjzyj2000@gmail.com>, "Bommu, Krishnaiah"
	<krishnaiah.bommu@intel.com>, "Ghimiray, Himal Prasad"
	<himal.prasad.ghimiray@intel.com>
Subject: RE: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to two steps
Thread-Topic: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to two
 steps
Thread-Index: AQHanOEtbi0VHTnLn0mNwgftYB7xRbGEmFqggAEftYCAO5ykoIAAFS8AgAAFauA=
Date: Mon, 10 Jun 2024 16:40:19 +0000
Message-ID: <PH7PR11MB7004A071F27B4CF45740B87E92C62@PH7PR11MB7004.namprd11.prod.outlook.com>
References: <cover.1709635535.git.leon@kernel.org>
 <SA1PR11MB6991CB2B1398948F4241E51992182@SA1PR11MB6991.namprd11.prod.outlook.com>
 <20240503164239.GB901876@ziepe.ca>
 <PH7PR11MB70047236290DC1CFF9150B8592C62@PH7PR11MB7004.namprd11.prod.outlook.com>
 <20240610161826.GA4966@unreal>
In-Reply-To: <20240610161826.GA4966@unreal>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?G/zvuWMRqrPRdrGretmKlmNaSdayUA4/COSS0ONc56FbVGs2tRskrM3nHd?=
 =?iso-8859-1?Q?fxParQLHb72XuMbbaw/nduQl5PzrHq52I4Jo37i3SPhBQ0re0r9M72PC6y?=
 =?iso-8859-1?Q?58rot3dzsjHaHLLhc0B4oOFjVSnIbzazG33fnwLQCWbLQoCI3mBKKpJTGb?=
 =?iso-8859-1?Q?co1hZ6jBEvgW7SGOtCH9Z8O7t4NtPpy6Fc3lu8zE99D0LRTM9LPUYrvGXE?=
 =?iso-8859-1?Q?FmyvR/rlZHIotu2i0H6QLg4xHPV+fKqs1N/6Ks2ioMnqoySfs5cUFKfceJ?=
 =?iso-8859-1?Q?qW4UcT4yu1eiFaZGnY/k4AWoPkHvh/R+biGyZYksumuRpQ+PEvnqFzpZxF?=
 =?iso-8859-1?Q?MhRTCS3bAkJlJudaN4SKHv8/M8X+mFHdcyCVT1gEgTz+Kn7SQhUI46J661?=
 =?iso-8859-1?Q?u5ethACWP/m6F0a4RTuLVVGyBLzN3GxIllSiTXQWNYcyr8O99w+T+EwK0O?=
 =?iso-8859-1?Q?iFmETe8l6C2eo4v/q2xxzBqhvpcuCgPuICG7SjJthur/VaBO2jKllNhBr0?=
 =?iso-8859-1?Q?BlZUiD7n/GC12T/DPWt+Zf1mWD5xdUMmdQQzKELOtOLeYBRSo7wDksqyJW?=
 =?iso-8859-1?Q?kVP2tlWPLOfYo9haP6KvBZNGsGhwUbuKcP58/VRfWwJjRZYJdke/A7MDB0?=
 =?iso-8859-1?Q?VNFIHEDPAc2pMntKPFdX+q8SNLW/9rdZ1DE9u+XgIFdqLzoCzYQy7oQQta?=
 =?iso-8859-1?Q?B8k3uHTWSD4Qye0A+tL30jo0JBHSuyj/Mw1RCgHA7XQTJjXfIj52oL3PeD?=
 =?iso-8859-1?Q?+PYj8THw/FdT2ZyfYCvebQd/kGm9aL3TkVX3dQ8J2Eclpto6/tw0o4vcPG?=
 =?iso-8859-1?Q?XuQU8Kf81++E7cz2gIOZUEIAO1GsNHQR7JD36KXh8AlEO/9YvypkEBufIq?=
 =?iso-8859-1?Q?Px0XP/JBgD8+si+L0EwnLjiOgrUwnziLRenrdrXrt1MTYQmo3fg1D/QoYE?=
 =?iso-8859-1?Q?V3qcD21XMHVCTtQhkyv++SeDLLsHn+76QDYSQbJqjNeBOExSQATp6cVYeg?=
 =?iso-8859-1?Q?OL7nNx8Wddd5APV2VNXeSd64UFZvbovU0DcSmXzg/gyg9TmnPuuxG8Vb3/?=
 =?iso-8859-1?Q?VNiGtNZkcNSvbt0bp8XdgTY4NNozAmsq22NDJO867258HBwi3Y8o5ueHBo?=
 =?iso-8859-1?Q?OHz5SvNdcfgQgApnjP3UvdWW6rwXDsk00XjSSU/L++0srcB/YhmRijjubX?=
 =?iso-8859-1?Q?cgBErOhjIuBNJj0/36Ygfc9oc/UOl+1tnCRn+qOnp3bPYfiWYM+wqsnsux?=
 =?iso-8859-1?Q?fCcEIcWEh6OTOfYXES1uDSffQVEcyQSyZ1ciJr4c/Fs3GEMn0VRqHGLSKo?=
 =?iso-8859-1?Q?e+7CatOOp585gWDVgsVtJCo75gQGbHhtXFgn8zdSVmLDxYHy5VCNEnr/c4?=
 =?iso-8859-1?Q?TKIMVsPDD0MbtJ13KxzxryEtGiTEpS9zydOd9B0h+BSzppqqhoQN608pSr?=
 =?iso-8859-1?Q?m1Vb+u2ti4H+jYUXVDDqQ0yqdTBLfszH/2YpOBp8WpFy5UDGhzjja3fpla?=
 =?iso-8859-1?Q?k98KelzHpQ5Oqoadrt7PaJ0o3mErzDBhZm7OQi2gZABlAxgjakrFgfa3bx?=
 =?iso-8859-1?Q?x1wMjDEibiDRQzU1pgTpMMqowiZAcqEE/d+SNEz3Kg14iXfKn8wXPnGPrz?=
 =?iso-8859-1?Q?WDg4+9iXpBL/A=3D?=
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PH7PR11MB7004.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 38376e52-048e-47bb-c16b-08dc896c0417
X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Jun 2024 16:40:19.7359
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: l7gsPJnsNHM6Wuxcr27UOd+Puqge5USKlT6zG/AwWIWGJBBNUFhPpn1PZiNbMmXleMbgpnXy4KDBhoQDR24p+w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR11MB8159
X-OriginatorOrg: intel.com

Thanks Leon and Yanjun for the reply!

Based on the reply, we will continue use the current version for test (as i=
t is tested for vfio and rdma). We will switch to v1 once it is fully teste=
d/reviewed.

Thanks,
Oak

> -----Original Message-----
> From: Leon Romanovsky <leon@kernel.org>
> Sent: Monday, June 10, 2024 12:18 PM
> To: Zeng, Oak <oak.zeng@intel.com>
> Cc: Jason Gunthorpe <jgg@ziepe.ca>; Christoph Hellwig <hch@lst.de>; Robin
> Murphy <robin.murphy@arm.com>; Marek Szyprowski
> <m.szyprowski@samsung.com>; Joerg Roedel <joro@8bytes.org>; Will
> Deacon <will@kernel.org>; Chaitanya Kulkarni <chaitanyak@nvidia.com>;
> Brost, Matthew <matthew.brost@intel.com>; Hellstrom, Thomas
> <thomas.hellstrom@intel.com>; Jonathan Corbet <corbet@lwn.net>; Jens
> Axboe <axboe@kernel.dk>; Keith Busch <kbusch@kernel.org>; Sagi
> Grimberg <sagi@grimberg.me>; Yishai Hadas <yishaih@nvidia.com>;
> Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>; Tian, Kevin
> <kevin.tian@intel.com>; Alex Williamson <alex.williamson@redhat.com>;
> J=E9r=F4me Glisse <jglisse@redhat.com>; Andrew Morton <akpm@linux-
> foundation.org>; linux-doc@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-block@vger.kernel.org; linux-rdma@vger.kernel.org;
> iommu@lists.linux.dev; linux-nvme@lists.infradead.org;
> kvm@vger.kernel.org; linux-mm@kvack.org; Bart Van Assche
> <bvanassche@acm.org>; Damien Le Moal
> <damien.lemoal@opensource.wdc.com>; Amir Goldstein
> <amir73il@gmail.com>; josef@toxicpanda.com; Martin K. Petersen
> <martin.petersen@oracle.com>; daniel@iogearbox.net; Williams, Dan J
> <dan.j.williams@intel.com>; jack@suse.com; Zhu Yanjun
> <zyjzyj2000@gmail.com>; Bommu, Krishnaiah
> <krishnaiah.bommu@intel.com>; Ghimiray, Himal Prasad
> <himal.prasad.ghimiray@intel.com>
> Subject: Re: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to
> two steps
>=20
> On Mon, Jun 10, 2024 at 03:12:25PM +0000, Zeng, Oak wrote:
> > Hi Jason, Leon,
> >
> > I come back to this thread to ask a question. Per the discussion in ano=
ther
> thread, I have integrated the new dma-mapping API (the first 6 patches of
> this series) to DRM subsystem. The new API seems fit pretty good to our
> purpose, better than scatter-gather dma-mapping. So we want to continue
> work with you to adopt this new API.
>=20
> Sounds great, thanks for the feedback.
>=20
> >
> > Did you test the new API in RDMA subsystem?
>=20
> This version was tested in our regression tests, but there is a chance
> that you are hitting flows that were not relevant for RDMA case.
>=20
> > Or this RFC series was just some untested codes sending out to get
> people's design feedback?
>=20
> RFC was fully tested in VFIO and RDMA paths, but not NVMe patch.
>=20
> > Do you have refined version for us to try? I ask because we are seeing
> some issues but not sure whether it is caused by the new API. We are
> debugging but it would be good to also ask at the same time.
>=20
> Yes, as an outcome of the feedback in this thread, I implemented a new
> version. Unfortunately, there are some personal matters that are preventi=
ng
> from me to send it right away.
> https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-
> rdma.git/log/?h=3Ddma-split-v1
>=20
> There are some differences in the API, but the main idea is the same.
> This version is not fully tested yet.
>=20
> Thanks
>=20
> >
> > Cc Himal/Krishna who are also working/testing the new API.
> >
> > Thanks,
> > Oak
> >
> > > -----Original Message-----
> > > From: Jason Gunthorpe <jgg@ziepe.ca>
> > > Sent: Friday, May 3, 2024 12:43 PM
> > > To: Zeng, Oak <oak.zeng@intel.com>
> > > Cc: leon@kernel.org; Christoph Hellwig <hch@lst.de>; Robin Murphy
> > > <robin.murphy@arm.com>; Marek Szyprowski
> > > <m.szyprowski@samsung.com>; Joerg Roedel <joro@8bytes.org>; Will
> > > Deacon <will@kernel.org>; Chaitanya Kulkarni <chaitanyak@nvidia.com>;
> > > Brost, Matthew <matthew.brost@intel.com>; Hellstrom, Thomas
> > > <thomas.hellstrom@intel.com>; Jonathan Corbet <corbet@lwn.net>;
> Jens
> > > Axboe <axboe@kernel.dk>; Keith Busch <kbusch@kernel.org>; Sagi
> > > Grimberg <sagi@grimberg.me>; Yishai Hadas <yishaih@nvidia.com>;
> > > Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>; Tian,
> Kevin
> > > <kevin.tian@intel.com>; Alex Williamson <alex.williamson@redhat.com>;
> > > J=E9r=F4me Glisse <jglisse@redhat.com>; Andrew Morton <akpm@linux-
> > > foundation.org>; linux-doc@vger.kernel.org; linux-
> kernel@vger.kernel.org;
> > > linux-block@vger.kernel.org; linux-rdma@vger.kernel.org;
> > > iommu@lists.linux.dev; linux-nvme@lists.infradead.org;
> > > kvm@vger.kernel.org; linux-mm@kvack.org; Bart Van Assche
> > > <bvanassche@acm.org>; Damien Le Moal
> > > <damien.lemoal@opensource.wdc.com>; Amir Goldstein
> > > <amir73il@gmail.com>; josef@toxicpanda.com; Martin K. Petersen
> > > <martin.petersen@oracle.com>; daniel@iogearbox.net; Williams, Dan J
> > > <dan.j.williams@intel.com>; jack@suse.com; Leon Romanovsky
> > > <leonro@nvidia.com>; Zhu Yanjun <zyjzyj2000@gmail.com>
> > > Subject: Re: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to
> > > two steps
> > >
> > > On Thu, May 02, 2024 at 11:32:55PM +0000, Zeng, Oak wrote:
> > >
> > > > > Instead of teaching DMA to know these specific datatypes, let's
> separate
> > > > > existing DMA mapping routine to two steps and give an option to
> > > advanced
> > > > > callers (subsystems) perform all calculations internally in advan=
ce and
> > > > > map pages later when it is needed.
> > > >
> > > > I looked into how this scheme can be applied to DRM subsystem and
> GPU
> > > drivers.
> > > >
> > > > I figured RDMA can apply this scheme because RDMA can calculate the
> > > > iova size. Per my limited knowledge of rdma, user can register a
> > > > memory region (the reg_user_mr vfunc) and memory region's sized is
> > > > used to pre-allocate iova space. And in the RDMA use case, it seems
> > > > the user registered region can be very big, e.g., 512MiB or even Gi=
B
> > >
> > > In RDMA the iova would be linked to the SVA granual we discussed
> > > previously.
> > >
> > > > In GPU driver, we have a few use cases where we need dma-mapping.
> Just
> > > name two:
> > > >
> > > > 1) userptr: it is user malloc'ed/mmap'ed memory and registers to gp=
u
> > > > (in Intel's driver it is through a vm_bind api, similar to mmap). A
> > > > userptr can be of any random size, depending on user malloc
> > > > size. Today we use dma-map-sg for this use case. The down side of
> > > > our approach is, during userptr invalidation, even if user only
> > > > munmap partially of an userptr, we invalidate the whole userptr fro=
m
> > > > gpu page table, because there is no way for us to partially
> > > > dma-unmap the whole sg list. I think we can try your new API in thi=
s
> > > > case. The main benefit of the new approach is the partial munmap
> > > > case.
> > >
> > > Yes, this is one of the main things it will improve.
> > >
> > > > We will have to pre-allocate iova for each userptr, and we have man=
y
> > > > userptrs of random size... So we might be not as efficient as RDMA
> > > > case where I assume user register a few big memory regions.
> > >
> > > You are already doing this. dma_map_sg() does exactly the same IOVA
> > > allocation under the covers.
> > >
> > > > 2) system allocator: it is malloc'ed/mmap'ed memory be used for GPU
> > > > program directly, without any other extra driver API call. We call
> > > > this use case system allocator.
> > >
> > > > For system allocator, driver have no knowledge of which virtual
> > > > address range is valid in advance. So when GPU access a
> > > > malloc'ed/mmap'ed address, we have a page fault. We then look up a
> > > > CPU vma which contains the fault address. I guess we can use the CP=
U
> > > > vma size to allocate the iova space of the same size?
> > >
> > > No. You'd follow what we discussed in the other thread.
> > >
> > > If you do a full SVA then you'd split your MM space into granuals and
> > > when a fault hits a granual you'd allocate the IOVA for the whole
> > > granual. RDMA ODP is using a 512M granual currently.
> > >
> > > If you are doing sub ranges then you'd probably allocate the IOVA for
> > > the well defined sub range (assuming the typical use case isn't huge)
> > >
> > > > But there will be a true difficulty to apply your scheme to this us=
e
> > > > case. It is related to the STICKY flag. As I understand it, the
> > > > sticky flag is designed for driver to mark "this page/pfn has been
> > > > populated, no need to re-populate again", roughly...Unlike userptr
> > > > and RDMA use cases where the backing store of a buffer is always in
> > > > system memory, in the system allocator use case, the backing store
> > > > can be changing b/t system memory and GPU's device private
> > > > memory. Even worse, we have to assume the data migration b/t
> system
> > > > and GPU is dynamic. When data is migrated to GPU, we don't need
> > > > dma-map. And when migration happens to a pfn with STICKY flag, we
> > > > still need to repopulate this pfn. So you can see, it is not easy t=
o
> > > > apply this scheme to this use case. At least I can't see an obvious
> > > > way.
> > >
> > > You are already doing this today, you are keeping the sg list around
> > > until you unmap it.
> > >
> > > Instead of keeping the sg list you'd keep a much smaller datastructur=
e
> > > per-granual. The sticky bit is simply a convient way for ODP to manag=
e
> > > the smaller data structure, you don't have to use it.
> > >
> > > But you do need to keep track of what pages in the granual have been
> > > DMA mapped - sg list was doing this before. This could be a simple
> > > bitmap array matching the granual size.
> > >
> > > Looking (far) forward we may be able to have a "replace" API that
> > > allows installing a new page unconditionally regardless of what is
> > > already there.
> > >
> > > Jason