Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753675AbdDRFXA (ORCPT ); Tue, 18 Apr 2017 01:23:00 -0400 Received: from ale.deltatee.com ([207.54.116.67]:52048 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752734AbdDRFW4 (ORCPT ); Tue, 18 Apr 2017 01:22:56 -0400 To: Dan Williams References: <6e732d6a-9baf-1768-3e9c-f6c887a836b2@deltatee.com> <1492381958.25766.50.camel@kernel.crashing.org> <6149ab5e-c981-6881-8c5a-22349561c3e8@deltatee.com> <1492413640.25766.52.camel@kernel.crashing.org> Cc: Benjamin Herrenschmidt , Bjorn Helgaas , Jason Gunthorpe , Christoph Hellwig , Sagi Grimberg , "James E.J. Bottomley" , "Martin K. Petersen" , Jens Axboe , Steve Wise , Stephen Bates , Max Gurtovoy , Keith Busch , linux-pci@vger.kernel.org, linux-scsi , linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm , "linux-kernel@vger.kernel.org" , Jerome Glisse From: Logan Gunthorpe Message-ID: Date: Mon, 17 Apr 2017 23:22:30 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Icedove/45.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 50.66.97.235 X-SA-Exim-Rcpt-To: jglisse@redhat.com, linux-kernel@vger.kernel.org, linux-nvdimm@ml01.01.org, linux-rdma@vger.kernel.org, linux-nvme@lists.infradead.org, linux-scsi@vger.kernel.org, linux-pci@vger.kernel.org, keith.busch@intel.com, maxg@mellanox.com, sbates@raithlin.com, swise@opengridcomputing.com, axboe@kernel.dk, martin.petersen@oracle.com, jejb@linux.vnet.ibm.com, sagi@grimberg.me, hch@lst.de, jgunthorpe@obsidianresearch.com, helgaas@kernel.org, benh@kernel.crashing.org, dan.j.williams@intel.com X-SA-Exim-Mail-From: logang@deltatee.com Subject: Re: [RFC 0/8] Copy Offload with Peer-to-Peer PCI Memory X-SA-Exim-Version: 4.2.1 (built Mon, 26 Dec 2011 16:24:06 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1119 Lines: 26 On 17/04/17 11:04 AM, Dan Williams wrote: >> Yes, in this scheme, it needs an additional p2pmem child. Why is that an >> issue? It certainly makes it a lot easier for the user to understand the >> p2pmem memory in the system (through the sysfs tree) and reason about >> the topology and when to use it. This is important. > > I think you want to go the other way in the hierarchy and find a > shared *parent* to land the p2pmem capability. Because that same agent > is going to be responsible handling address translation for the peers. > > Peer-dma is always going to be a property of the bus and not the end > devices. Requiring each bus implementation to explicitly enable > peer-to-peer support is a feature not a bug. > > We shouldn't design for some future possible use case. Solve it for > pci and when / if another bus comes along then look at a more generic > abstraction. Thanks Dan, these are some good points. Wedding it closer to the PCI code makes more sense to me now. I'd still think you'd want some struct device though to appear in the device hierarchy and allow reasoning about topology. Logan