Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751207AbcLDNre (ORCPT ); Sun, 4 Dec 2016 08:47:34 -0500 Received: from gateway20.websitewelcome.com ([192.185.66.3]:47393 "EHLO gateway20.websitewelcome.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750804AbcLDNrd (ORCPT ); Sun, 4 Dec 2016 08:47:33 -0500 X-Greylist: delayed 1424 seconds by postgrey-1.27 at vger.kernel.org; Sun, 04 Dec 2016 08:47:33 EST Message-ID: <61a2fb07344aacd81111449d222de66e.squirrel@webmail.raithlin.com> In-Reply-To: References: <20161123215510.GA16311@obsidianresearch.com> <91d28749-bc64-622f-56a1-26c00e6b462a@deltatee.com> <20161124164249.GD20818@obsidianresearch.com> <3f2d2db3-fb75-2422-2a18-a8497fd5d70e@amd.com> <20161125193252.GC16504@obsidianresearch.com> <20161128165751.GB28381@obsidianresearch.com> <1480357179.19407.13.camel@mellanox.com> <20161128190244.GA21975@obsidianresearch.com> <20161130162353.GA24639@obsidianresearch.com> <5f5b7989-84f5-737e-47c8-831f752d6280@deltatee.com> Date: Sun, 4 Dec 2016 07:23:00 -0600 Subject: Re: Enabling peer to peer device transactions for PCIe devices From: "Stephen Bates" To: "Haggai Eran" Cc: "Logan Gunthorpe" , "Jason Gunthorpe" , "linux-kernel@vger.kernel.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@ml01.01.org" , "christian.koenig@amd.com" , "Suravee.Suthikulpanit@amd.com" , "John.Bridgman@amd.com" , "Alexander.Deucher@amd.com" , "Linux-media@vger.kernel.org" , "dan.j.williams@intel.com" , "dri-devel@lists.freedesktop.org" , "Max Gurtovoy" , "linux-pci@vger.kernel.org" , "serguei.sagalovitch@amd.com" , "Paul.Blinzer@amd.com" , "Felix.Kuehling@amd.com" , "ben.sander@amd.com" User-Agent: SquirrelMail/1.5.2 [SVN] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7BIT X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - estate.websitewelcome.com X-AntiAbuse: Original Domain - vger.kernel.org X-AntiAbuse: Originator/Caller UID/GID - [1547 32008] / [47 12] X-AntiAbuse: Sender Address Domain - raithlin.com X-BWhitelist: no X-Source-IP: X-Exim-ID: 1cDWku-0009ga-DH X-Source: X-Source-Args: /usr/local/cpanel/3rdparty/php/54/bin/php-cgi /usr/local/cpanel/base/3rdparty/squirrelmail/src/compose.php X-Source-Dir: /usr/local/cpanel/base/3rdparty/squirrelmail/src X-Source-Sender: X-Source-Auth: raithlin X-Email-Count: 35 X-Source-Cap: cmFpdGhsaW47c2NvdHQ7ZXN0YXRlLndlYnNpdGV3ZWxjb21lLmNvbQ== Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1892 Lines: 44 Hi All This has been a great thread (thanks to Alex for kicking it off) and I wanted to jump in and maybe try and put some summary around the discussion. I also wanted to propose we include this as a topic for LFS/MM because I think we need more discussion on the best way to add this functionality to the kernel. As far as I can tell the people looking for P2P support in the kernel fall into two main camps: 1. Those who simply want to expose static BARs on PCIe devices that can be used as the source/destination for DMAs from another PCIe device. This group has no need for memory invalidation and are happy to use physical/bus addresses and not virtual addresses. 2. Those who want to support devices that suffer from occasional memory pressure and need to invalidate memory regions from time to time. This camp also would like to use virtual addresses rather than physical ones to allow for things like migration. I am wondering if people agree with this assessment? I think something like the iopmem patches Logan and I submitted recently come close to addressing use case 1. There are some issues around routability but based on feedback to date that does not seem to be a show-stopper for an initial inclusion. For use-case 2 it looks like there are several options and some of them (like HMM) have been around for quite some time without gaining acceptance. I think there needs to be more discussion on this usecase and it could be some time before we get something upstreamable. I for one, would really like to see use case 1 get addressed soon because we have consumers for it coming soon in the form of CMBs for NVMe devices. Long term I think Jason summed it up really well. CPU vendors will put high-speed, open, switchable, coherent buses on their processors and all these problems will vanish. But I ain't holding my breathe for that to happen ;-). Cheers Stephen