Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933604AbcKWSkv (ORCPT ); Wed, 23 Nov 2016 13:40:51 -0500 Received: from mail-oi0-f43.google.com ([209.85.218.43]:34353 "EHLO mail-oi0-f43.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S933392AbcKWSkt (ORCPT ); Wed, 23 Nov 2016 13:40:49 -0500 MIME-Version: 1.0 In-Reply-To: References: <75a1f44f-c495-7d1e-7e1c-17e89555edba@amd.com> <45c6e878-bece-7987-aee7-0e940044158c@deltatee.com> From: Dan Williams Date: Wed, 23 Nov 2016 10:40:47 -0800 Message-ID: Subject: Re: Enabling peer to peer device transactions for PCIe devices To: Bart Van Assche Cc: Logan Gunthorpe , Serguei Sagalovitch , "Deucher, Alexander" , "linux-nvdimm@lists.01.org" , "linux-rdma@vger.kernel.org" , "linux-pci@vger.kernel.org" , "Kuehling, Felix" , "Bridgman, John" , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , "Koenig, Christian" , "Sander, Ben" , "Suthikulpanit, Suravee" , "Blinzer, Paul" , "Linux-media@vger.kernel.org" Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1414 Lines: 29 On Wed, Nov 23, 2016 at 9:27 AM, Bart Van Assche wrote: > On 11/23/2016 09:13 AM, Logan Gunthorpe wrote: >> >> IMO any memory that has been registered for a P2P transaction should be >> locked from being evicted. So if there's a get_user_pages call it needs >> to be pinned until the put_page. The main issue being with the RDMA >> case: handling an eviction when a chunk of memory has been registered as >> an MR would be very tricky. The MR may be relied upon by another host >> and the kernel would have to inform user-space the MR was invalid then >> user-space would have to tell the remote application. > > > Hello Logan, > > Are you aware that the Linux kernel already supports ODP (On Demand Paging)? > See also the output of git grep -nHi on.demand.paging. See also > https://www.openfabrics.org/images/eventpresos/workshops2014/DevWorkshop/presos/Tuesday/pdf/04_ODP_update.pdf. > I don't think that was designed for the case where the backing memory is a special/static physical address range rather than anonymous "System RAM", right? I think we should handle the graphics P2P concerns separately from the general P2P-DMA case since the latter does not require the higher order memory management facilities. Using ZONE_DEVICE/DAX mappings to avoid changes to every driver that wants to support P2P-DMA separately from typical DMA still seems the path of least resistance.