Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753176AbcK1SUr (ORCPT ); Mon, 28 Nov 2016 13:20:47 -0500 Received: from ale.deltatee.com ([207.54.116.67]:51367 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751120AbcK1SUh (ORCPT ); Mon, 28 Nov 2016 13:20:37 -0500 To: Jason Gunthorpe , Haggai Eran References: <20161123193228.GC12146@obsidianresearch.com> <20161123203332.GA15062@obsidianresearch.com> <20161123215510.GA16311@obsidianresearch.com> <91d28749-bc64-622f-56a1-26c00e6b462a@deltatee.com> <20161124164249.GD20818@obsidianresearch.com> <3f2d2db3-fb75-2422-2a18-a8497fd5d70e@amd.com> <20161125193252.GC16504@obsidianresearch.com> <20161128165751.GB28381@obsidianresearch.com> Cc: =?UTF-8?Q?Christian_K=c3=b6nig?= , Serguei Sagalovitch , Dan Williams , "Deucher, Alexander" , "linux-nvdimm@lists.01.org" , "linux-rdma@vger.kernel.org" , "linux-pci@vger.kernel.org" , "Kuehling, Felix" , "Bridgman, John" , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , "Sander, Ben" , "Suthikulpanit, Suravee" , "Blinzer, Paul" , "Linux-media@vger.kernel.org" , Max Gurtovoy From: Logan Gunthorpe Message-ID: Date: Mon, 28 Nov 2016 11:20:27 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Icedove/45.4.0 MIME-Version: 1.0 In-Reply-To: <20161128165751.GB28381@obsidianresearch.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 172.16.1.111 X-SA-Exim-Rcpt-To: maxg@mellanox.com, Linux-media@vger.kernel.org, Paul.Blinzer@amd.com, Suravee.Suthikulpanit@amd.com, ben.sander@amd.com, dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org, John.Bridgman@amd.com, Felix.Kuehling@amd.com, linux-pci@vger.kernel.org, linux-rdma@vger.kernel.org, linux-nvdimm@ml01.01.org, Alexander.Deucher@amd.com, dan.j.williams@intel.com, serguei.sagalovitch@amd.com, christian.koenig@amd.com, haggaie@mellanox.com, jgunthorpe@obsidianresearch.com X-SA-Exim-Mail-From: logang@deltatee.com Subject: Re: Enabling peer to peer device transactions for PCIe devices X-SA-Exim-Version: 4.2.1 (built Mon, 26 Dec 2011 16:24:06 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 968 Lines: 21 On 28/11/16 09:57 AM, Jason Gunthorpe wrote: >> On PeerDirect, we have some kind of a middle-ground solution for pinning >> GPU memory. We create a non-ODP MR pointing to VRAM but rely on >> user-space and the GPU not to migrate it. If they do, the MR gets >> destroyed immediately. > > That sounds horrible. How can that possibly work? What if the MR is > being used when the GPU decides to migrate? I would not support that > upstream without a lot more explanation.. Yup, this was our experience when playing around with PeerDirect. There was nothing we could do if the GPU decided to invalidate the P2P mapping. It just meant the application would fail or need complicated logic to detect this and redo just about everything. And given that it was a reasonably rare occurrence during development it probably means not a lot of applications will be developed to handle it and most would end up being randomly broken in environments with memory pressure. Logan