Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756794AbcKWVLg (ORCPT ); Wed, 23 Nov 2016 16:11:36 -0500 Received: from ale.deltatee.com ([207.54.116.67]:35959 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755788AbcKWVLe (ORCPT ); Wed, 23 Nov 2016 16:11:34 -0500 To: Jason Gunthorpe , Serguei Sagalovitch References: <75a1f44f-c495-7d1e-7e1c-17e89555edba@amd.com> <45c6e878-bece-7987-aee7-0e940044158c@deltatee.com> <20161123190515.GA12146@obsidianresearch.com> <7bc38037-b6ab-943f-59db-6280e16901ab@amd.com> <20161123193228.GC12146@obsidianresearch.com> <20161123203332.GA15062@obsidianresearch.com> Cc: Dan Williams , "Deucher, Alexander" , "linux-nvdimm@lists.01.org" , "linux-rdma@vger.kernel.org" , "linux-pci@vger.kernel.org" , "Kuehling, Felix" , "Bridgman, John" , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , "Koenig, Christian" , "Sander, Ben" , "Suthikulpanit, Suravee" , "Blinzer, Paul" , "Linux-media@vger.kernel.org" , Haggai Eran From: Logan Gunthorpe Message-ID: Date: Wed, 23 Nov 2016 14:11:29 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Icedove/45.4.0 MIME-Version: 1.0 In-Reply-To: <20161123203332.GA15062@obsidianresearch.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 172.16.1.111 X-SA-Exim-Rcpt-To: haggaie@mellanox.com, Linux-media@vger.kernel.org, Paul.Blinzer@amd.com, Suravee.Suthikulpanit@amd.com, ben.sander@amd.com, Christian.Koenig@amd.com, dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org, John.Bridgman@amd.com, Felix.Kuehling@amd.com, linux-pci@vger.kernel.org, linux-rdma@vger.kernel.org, linux-nvdimm@ml01.01.org, Alexander.Deucher@amd.com, dan.j.williams@intel.com, serguei.sagalovitch@amd.com, jgunthorpe@obsidianresearch.com X-SA-Exim-Mail-From: logang@deltatee.com Subject: Re: Enabling peer to peer device transactions for PCIe devices X-SA-Exim-Version: 4.2.1 (built Mon, 26 Dec 2011 16:24:06 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1631 Lines: 41 On 23/11/16 01:33 PM, Jason Gunthorpe wrote: > On Wed, Nov 23, 2016 at 02:58:38PM -0500, Serguei Sagalovitch wrote: > >> We do not want to have "highly" dynamic translation due to >> performance cost. We need to support "overcommit" but would >> like to minimize impact. To support RDMA MRs for GPU/VRAM/PCIe >> device memory (which is must) we need either globally force >> pinning for the scope of "get_user_pages() / "put_pages" or have >> special handling for RDMA MRs and similar cases. > > As I said, there is no possible special handling. Standard IB hardware > does not support changing the DMA address once a MR is created. Forget > about doing that. Yeah, that's essentially the point I was trying to make. Not to mention all the other unrelated hardware that can't DMA to an address that might disappear mid-transfer. > Only ODP hardware allows changing the DMA address on the fly, and it > works at the page table level. We do not need special handling for > RDMA. I am aware of ODP but, noted by others, it doesn't provide a general solution to the points above. > Like I said, this is the direction the industry seems to be moving in, > so any solution here should focus on VMAs/page tables as the way to link > the peer-peer devices. Yes, this was the appeal to us of using ZONE_DEVICE. > To me this means at least items #1 and #3 should be removed from > Alexander's list. It's also worth noting that #4 makes use of ZONE_DEVICE (#2) so they are really the same option. iopmem is really just one way to get BAR addresses to user-space while inside the kernel it's ZONE_DEVICE. Logan