Received-SPF: pass (google.com: domain of linux-kernel+bounces-208432-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
From: "Zeng, Oak" <oak.zeng@intel.com>
To: Jason Gunthorpe <jgg@ziepe.ca>
CC: "leon@kernel.org" <leon@kernel.org>, Christoph Hellwig <hch@lst.de>,
	"Robin Murphy" <robin.murphy@arm.com>, Marek Szyprowski
	<m.szyprowski@samsung.com>, Joerg Roedel <joro@8bytes.org>, Will Deacon
	<will@kernel.org>, "Chaitanya Kulkarni" <chaitanyak@nvidia.com>, "Brost,
 Matthew" <matthew.brost@intel.com>, "Hellstrom, Thomas"
	<thomas.hellstrom@intel.com>, Jonathan Corbet <corbet@lwn.net>, Jens Axboe
	<axboe@kernel.dk>, Keith Busch <kbusch@kernel.org>, Sagi Grimberg
	<sagi@grimberg.me>, Yishai Hadas <yishaih@nvidia.com>, Shameer Kolothum
	<shameerali.kolothum.thodi@huawei.com>, "Tian, Kevin" <kevin.tian@intel.com>,
	Alex Williamson <alex.williamson@redhat.com>,
	=?iso-8859-1?Q?J=E9r=F4me_Glisse?= <jglisse@redhat.com>, Andrew Morton
	<akpm@linux-foundation.org>, "linux-doc@vger.kernel.org"
	<linux-doc@vger.kernel.org>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, "linux-block@vger.kernel.org"
	<linux-block@vger.kernel.org>, "linux-rdma@vger.kernel.org"
	<linux-rdma@vger.kernel.org>, "iommu@lists.linux.dev"
	<iommu@lists.linux.dev>, "linux-nvme@lists.infradead.org"
	<linux-nvme@lists.infradead.org>, "kvm@vger.kernel.org"
	<kvm@vger.kernel.org>, "linux-mm@kvack.org" <linux-mm@kvack.org>, "Bart Van
 Assche" <bvanassche@acm.org>, Damien Le Moal
	<damien.lemoal@opensource.wdc.com>, Amir Goldstein <amir73il@gmail.com>,
	"josef@toxicpanda.com" <josef@toxicpanda.com>, "Martin K. Petersen"
	<martin.petersen@oracle.com>, "daniel@iogearbox.net" <daniel@iogearbox.net>,
	"Williams, Dan J" <dan.j.williams@intel.com>, "jack@suse.com"
	<jack@suse.com>, Leon Romanovsky <leonro@nvidia.com>, Zhu Yanjun
	<zyjzyj2000@gmail.com>, "Bommu, Krishnaiah" <krishnaiah.bommu@intel.com>,
	"Ghimiray, Himal Prasad" <himal.prasad.ghimiray@intel.com>
Subject: RE: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to two steps
Thread-Topic: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to two
 steps
Thread-Index: AQHanOEtbi0VHTnLn0mNwgftYB7xRbGEmFqggAEftYCAO5ykoA==
Date: Mon, 10 Jun 2024 15:12:25 +0000
Message-ID: <PH7PR11MB70047236290DC1CFF9150B8592C62@PH7PR11MB7004.namprd11.prod.outlook.com>
References: <cover.1709635535.git.leon@kernel.org>
 <SA1PR11MB6991CB2B1398948F4241E51992182@SA1PR11MB6991.namprd11.prod.outlook.com>
 <20240503164239.GB901876@ziepe.ca>
In-Reply-To: <20240503164239.GB901876@ziepe.ca>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?2X+emzaduGjmW3aGIKukkqNS8/Zm1DeplWqDIiloB5HL9vSpLFnljzz849?=
 =?iso-8859-1?Q?8KPMzFtA2f9AenlsQuikry1a9MRWdIufdIX8E+2rAnxeT5W+zSvvgzndJH?=
 =?iso-8859-1?Q?WVkCWZGJcqI6CZ+2oQDuSL/35yYstkLwOmcPlcoJkiCXNUGteuhqhbwhSm?=
 =?iso-8859-1?Q?63n1r+blIgOnHXklZFKM6msjOF+A6VlJh2XaHkCGWnW7xFBcT6aq5Goj6i?=
 =?iso-8859-1?Q?ZgtJSHc+Z9378AlcmyC+bD4KOkQffAlocVt+mJs0RR1rfinIJkiqC3FnPa?=
 =?iso-8859-1?Q?Lf5BHupUoDb0WlW7MlLZSwRDc5ScYaNse1NUqY3NJrp9Pfaa5y5N0/Xx0T?=
 =?iso-8859-1?Q?9dvUZu8n5j755VkCYHYPaTG/dXBqBhayR3XSk1v7B3CfYOyizC6sRhyRIX?=
 =?iso-8859-1?Q?FA83SCyJNUjjd8k9EjSFW1+/WXPjb1zursbdGOh4qCzB9ft8tHHCI9/NKI?=
 =?iso-8859-1?Q?sHzW4/fQINDtvXEmuYMMNfFvOvEinq86FRT/2s+OwauOfLZyPl04x9pOLM?=
 =?iso-8859-1?Q?flBDKKsiyMqhg7cDvCBZ1ZDNL4qlvtKc+UlXDXfy1wo1h0pQmDrPZD3xEG?=
 =?iso-8859-1?Q?OYen63P3gej9NDOn1en16lateFCMkfjuV5L1JhJpvRi+HLbNqaJgg6CS7f?=
 =?iso-8859-1?Q?2uvX/0Re8iKjgapk/Qi66J62mlU6jE4JlJxXNLXogps5Eh8CTkgA2PA7Dd?=
 =?iso-8859-1?Q?JccG4fUdcK3bOjVQ4qxdLlwLQ7+NXgttAXmAs8LDOdf/gRDAXDcy/DiJ+6?=
 =?iso-8859-1?Q?ddx0ytXHMaYC4KHAvQNQLbDaQCcCp3C5rhVFRhMsGl2xOYsER0SF6Qvgt6?=
 =?iso-8859-1?Q?ocX8ZDgstCUTfa/GIFR9rfMguZ7dJaFUNNXv7PaU1d15Pnfswjlz+v1K+C?=
 =?iso-8859-1?Q?rQdTqo7iDtfzgzTvtm6EzSxqWVqyPGO+vgn6KtINJCuBPaMQWWjiSu6s8X?=
 =?iso-8859-1?Q?pa0VSKwtIoaihNzgU0L/bjRHtlPePFkJYLbRZz2DhKlHmQWsXZEoCEuyhu?=
 =?iso-8859-1?Q?TElmnZT9ZsqrOzhdG7DySmOwY5hD/uAU+cMaRZ5RogsJUQOvpmKshCzQxI?=
 =?iso-8859-1?Q?ZjESh87rO3V++jZxgT55Fc8h83lAGzyymzBG353uR3MyxRiRS7JFmVfkzy?=
 =?iso-8859-1?Q?WZFdGnDDnRhc7EXJGGMwyTCK5yd0VLRBRAIQ0dhvFx6S/JwGd/ewL06TIY?=
 =?iso-8859-1?Q?gRArwUT6jFpY4qV4l82FPWQSljdDpFdQbeeVC2CGQ7dSKEyCf7LlEdwRTv?=
 =?iso-8859-1?Q?NlypSd8a/+vNyZE4VMXW/It+b/AATjosGKUgCVuX+lg3hRij6eQyTZyhxD?=
 =?iso-8859-1?Q?GTFaRitHX1J5cadgAj33PM4XtpK1aJDI65f3xmRAlaMSQ7YBQ5FJ2sYzig?=
 =?iso-8859-1?Q?JajEh2Ub9SUZBT/zOCNaXm73IBAc5T8ILQKPUYY/12IdErUH/Ld+KnSrrd?=
 =?iso-8859-1?Q?azaYYC3VVQ5HXSt4JNRoK3Kl2FQeTeeFWLE08his9bMsvbPvMgxf0zsNfO?=
 =?iso-8859-1?Q?bvueJ7dQZLOew44vSuk+HPqBF+TV3Rm7wSIWGPLGM5KYmTcC0ySA+5lj2N?=
 =?iso-8859-1?Q?nox2w2I8mxJKDV4cImCd/c/b/OpSyYFERHqOLI+NtJomEot92YclhaJnPI?=
 =?iso-8859-1?Q?8bgUU//K4xFyM=3D?=
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PH7PR11MB7004.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 415cb717-016e-45cd-c3dc-08dc895fbc81
X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Jun 2024 15:12:25.7198
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: dPoMx2tlf098Vc0YA+SzVf+jQQYG+ICixPVfQt+fWbALhBsVt2YFGBnt5FAtkcZdJANKtMcyNPwdRM3Y6QrJDg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR11MB5026
X-OriginatorOrg: intel.com

Hi Jason, Leon,

I come back to this thread to ask a question. Per the discussion in another=
 thread, I have integrated the new dma-mapping API (the first 6 patches of =
this series) to DRM subsystem. The new API seems fit pretty good to our pur=
pose, better than scatter-gather dma-mapping. So we want to continue work w=
ith you to adopt this new API.

Did you test the new API in RDMA subsystem? Or this RFC series was just som=
e untested codes sending out to get people's design feedback? Do you have r=
efined version for us to try? I ask because we are seeing some issues but n=
ot sure whether it is caused by the new API. We are debugging but it would =
be good to also ask at the same time.

Cc Himal/Krishna who are also working/testing the new API.

Thanks,
Oak

> -----Original Message-----
> From: Jason Gunthorpe <jgg@ziepe.ca>
> Sent: Friday, May 3, 2024 12:43 PM
> To: Zeng, Oak <oak.zeng@intel.com>
> Cc: leon@kernel.org; Christoph Hellwig <hch@lst.de>; Robin Murphy
> <robin.murphy@arm.com>; Marek Szyprowski
> <m.szyprowski@samsung.com>; Joerg Roedel <joro@8bytes.org>; Will
> Deacon <will@kernel.org>; Chaitanya Kulkarni <chaitanyak@nvidia.com>;
> Brost, Matthew <matthew.brost@intel.com>; Hellstrom, Thomas
> <thomas.hellstrom@intel.com>; Jonathan Corbet <corbet@lwn.net>; Jens
> Axboe <axboe@kernel.dk>; Keith Busch <kbusch@kernel.org>; Sagi
> Grimberg <sagi@grimberg.me>; Yishai Hadas <yishaih@nvidia.com>;
> Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>; Tian, Kevin
> <kevin.tian@intel.com>; Alex Williamson <alex.williamson@redhat.com>;
> J=E9r=F4me Glisse <jglisse@redhat.com>; Andrew Morton <akpm@linux-
> foundation.org>; linux-doc@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-block@vger.kernel.org; linux-rdma@vger.kernel.org;
> iommu@lists.linux.dev; linux-nvme@lists.infradead.org;
> kvm@vger.kernel.org; linux-mm@kvack.org; Bart Van Assche
> <bvanassche@acm.org>; Damien Le Moal
> <damien.lemoal@opensource.wdc.com>; Amir Goldstein
> <amir73il@gmail.com>; josef@toxicpanda.com; Martin K. Petersen
> <martin.petersen@oracle.com>; daniel@iogearbox.net; Williams, Dan J
> <dan.j.williams@intel.com>; jack@suse.com; Leon Romanovsky
> <leonro@nvidia.com>; Zhu Yanjun <zyjzyj2000@gmail.com>
> Subject: Re: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to
> two steps
>=20
> On Thu, May 02, 2024 at 11:32:55PM +0000, Zeng, Oak wrote:
>=20
> > > Instead of teaching DMA to know these specific datatypes, let's separ=
ate
> > > existing DMA mapping routine to two steps and give an option to
> advanced
> > > callers (subsystems) perform all calculations internally in advance a=
nd
> > > map pages later when it is needed.
> >
> > I looked into how this scheme can be applied to DRM subsystem and GPU
> drivers.
> >
> > I figured RDMA can apply this scheme because RDMA can calculate the
> > iova size. Per my limited knowledge of rdma, user can register a
> > memory region (the reg_user_mr vfunc) and memory region's sized is
> > used to pre-allocate iova space. And in the RDMA use case, it seems
> > the user registered region can be very big, e.g., 512MiB or even GiB
>=20
> In RDMA the iova would be linked to the SVA granual we discussed
> previously.
>=20
> > In GPU driver, we have a few use cases where we need dma-mapping. Just
> name two:
> >
> > 1) userptr: it is user malloc'ed/mmap'ed memory and registers to gpu
> > (in Intel's driver it is through a vm_bind api, similar to mmap). A
> > userptr can be of any random size, depending on user malloc
> > size. Today we use dma-map-sg for this use case. The down side of
> > our approach is, during userptr invalidation, even if user only
> > munmap partially of an userptr, we invalidate the whole userptr from
> > gpu page table, because there is no way for us to partially
> > dma-unmap the whole sg list. I think we can try your new API in this
> > case. The main benefit of the new approach is the partial munmap
> > case.
>=20
> Yes, this is one of the main things it will improve.
>=20
> > We will have to pre-allocate iova for each userptr, and we have many
> > userptrs of random size... So we might be not as efficient as RDMA
> > case where I assume user register a few big memory regions.
>=20
> You are already doing this. dma_map_sg() does exactly the same IOVA
> allocation under the covers.
>=20
> > 2) system allocator: it is malloc'ed/mmap'ed memory be used for GPU
> > program directly, without any other extra driver API call. We call
> > this use case system allocator.
>=20
> > For system allocator, driver have no knowledge of which virtual
> > address range is valid in advance. So when GPU access a
> > malloc'ed/mmap'ed address, we have a page fault. We then look up a
> > CPU vma which contains the fault address. I guess we can use the CPU
> > vma size to allocate the iova space of the same size?
>=20
> No. You'd follow what we discussed in the other thread.
>=20
> If you do a full SVA then you'd split your MM space into granuals and
> when a fault hits a granual you'd allocate the IOVA for the whole
> granual. RDMA ODP is using a 512M granual currently.
>=20
> If you are doing sub ranges then you'd probably allocate the IOVA for
> the well defined sub range (assuming the typical use case isn't huge)
>=20
> > But there will be a true difficulty to apply your scheme to this use
> > case. It is related to the STICKY flag. As I understand it, the
> > sticky flag is designed for driver to mark "this page/pfn has been
> > populated, no need to re-populate again", roughly...Unlike userptr
> > and RDMA use cases where the backing store of a buffer is always in
> > system memory, in the system allocator use case, the backing store
> > can be changing b/t system memory and GPU's device private
> > memory. Even worse, we have to assume the data migration b/t system
> > and GPU is dynamic. When data is migrated to GPU, we don't need
> > dma-map. And when migration happens to a pfn with STICKY flag, we
> > still need to repopulate this pfn. So you can see, it is not easy to
> > apply this scheme to this use case. At least I can't see an obvious
> > way.
>=20
> You are already doing this today, you are keeping the sg list around
> until you unmap it.
>=20
> Instead of keeping the sg list you'd keep a much smaller datastructure
> per-granual. The sticky bit is simply a convient way for ODP to manage
> the smaller data structure, you don't have to use it.
>=20
> But you do need to keep track of what pages in the granual have been
> DMA mapped - sg list was doing this before. This could be a simple
> bitmap array matching the granual size.
>=20
> Looking (far) forward we may be able to have a "replace" API that
> allows installing a new page unconditionally regardless of what is
> already there.
>=20
> Jason