Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S936594AbcKXBZ3 (ORCPT ); Wed, 23 Nov 2016 20:25:29 -0500 Received: from ale.deltatee.com ([207.54.116.67]:36574 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S934630AbcKXBZZ (ORCPT ); Wed, 23 Nov 2016 20:25:25 -0500 To: Jason Gunthorpe References: <75a1f44f-c495-7d1e-7e1c-17e89555edba@amd.com> <45c6e878-bece-7987-aee7-0e940044158c@deltatee.com> <20161123190515.GA12146@obsidianresearch.com> <7bc38037-b6ab-943f-59db-6280e16901ab@amd.com> <20161123193228.GC12146@obsidianresearch.com> <20161123203332.GA15062@obsidianresearch.com> <20161123215510.GA16311@obsidianresearch.com> Cc: Serguei Sagalovitch , Dan Williams , "Deucher, Alexander" , "linux-nvdimm@lists.01.org" , "linux-rdma@vger.kernel.org" , "linux-pci@vger.kernel.org" , "Kuehling, Felix" , "Bridgman, John" , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , "Koenig, Christian" , "Sander, Ben" , "Suthikulpanit, Suravee" , "Blinzer, Paul" , "Linux-media@vger.kernel.org" , Haggai Eran From: Logan Gunthorpe Message-ID: <91d28749-bc64-622f-56a1-26c00e6b462a@deltatee.com> Date: Wed, 23 Nov 2016 18:25:21 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Icedove/45.4.0 MIME-Version: 1.0 In-Reply-To: <20161123215510.GA16311@obsidianresearch.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 172.16.1.111 X-SA-Exim-Rcpt-To: haggaie@mellanox.com, Linux-media@vger.kernel.org, Paul.Blinzer@amd.com, Suravee.Suthikulpanit@amd.com, ben.sander@amd.com, Christian.Koenig@amd.com, dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org, John.Bridgman@amd.com, Felix.Kuehling@amd.com, linux-pci@vger.kernel.org, linux-rdma@vger.kernel.org, linux-nvdimm@ml01.01.org, Alexander.Deucher@amd.com, dan.j.williams@intel.com, serguei.sagalovitch@amd.com, jgunthorpe@obsidianresearch.com X-SA-Exim-Mail-From: logang@deltatee.com Subject: Re: Enabling peer to peer device transactions for PCIe devices X-SA-Exim-Version: 4.2.1 (built Mon, 26 Dec 2011 16:24:06 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1462 Lines: 37 On 23/11/16 02:55 PM, Jason Gunthorpe wrote: >>> Only ODP hardware allows changing the DMA address on the fly, and it >>> works at the page table level. We do not need special handling for >>> RDMA. >> >> I am aware of ODP but, noted by others, it doesn't provide a general >> solution to the points above. > > How do you mean? I was only saying it wasn't general in that it wouldn't work for IB hardware that doesn't support ODP or other hardware that doesn't do similar things (like an NVMe drive). It makes sense for hardware that supports ODP to allow MRs to not pin the underlying memory and provide for migrations that the hardware can follow. But most DMA engines will require the memory to be pinned and any complex allocators (GPU or otherwise) should respect that. And that seems like it should be the default way most of this works -- and I think it wouldn't actually take too much effort to make it all work now as is. (Our iopmem work is actually quite small and simple.) >> It's also worth noting that #4 makes use of ZONE_DEVICE (#2) so they are >> really the same option. iopmem is really just one way to get BAR >> addresses to user-space while inside the kernel it's ZONE_DEVICE. > > Seems fine for RDMA? Yeah, we've had RDMA and O_DIRECT transfers to PCIe backed ZONE_DEVICE memory working for some time. I'd say it's a good fit. The main question we've had is how to expose PCIe bars to userspace to be used as MRs and such. Logan