Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752704AbcLFI2p (ORCPT ); Tue, 6 Dec 2016 03:28:45 -0500 Received: from gateway20.websitewelcome.com ([192.185.44.20]:59695 "EHLO gateway20.websitewelcome.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752113AbcLFI2n (ORCPT ); Tue, 6 Dec 2016 03:28:43 -0500 Message-ID: In-Reply-To: References: <20161128165751.GB28381@obsidianresearch.com> <1480357179.19407.13.camel@mellanox.com> <20161128190244.GA21975@obsidianresearch.com> <20161130162353.GA24639@obsidianresearch.com> <5f5b7989-84f5-737e-47c8-831f752d6280@deltatee.com> <61a2fb07344aacd81111449d222de66e.squirrel@webmail.raithlin.com> <20161205171830.GB27784@obsidianresearch.com> <20161205180231.GA28133@obsidianresearch.com> Date: Tue, 6 Dec 2016 02:06:14 -0600 Subject: Re: Enabling peer to peer device transactions for PCIe devices From: "Stephen Bates" To: "Dan Williams" Cc: "Logan Gunthorpe" , "Jason Gunthorpe" , "Haggai Eran" , "linux-kernel@vger.kernel.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@ml01.01.org" , "christian.koenig@amd.com" , "Suravee.Suthikulpanit@amd.com" , "John.Bridgman@amd.com" , "Alexander.Deucher@amd.com" , "Linux-media@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , "Max Gurtovoy" , "linux-pci@vger.kernel.org" , "serguei.sagalovitch@amd.com" , "Paul.Blinzer@amd.com" , "Felix.Kuehling@amd.com" , "ben.sander@amd.com" User-Agent: SquirrelMail/1.5.2 [SVN] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7BIT X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - estate.websitewelcome.com X-AntiAbuse: Original Domain - vger.kernel.org X-AntiAbuse: Originator/Caller UID/GID - [1547 32008] / [47 12] X-AntiAbuse: Sender Address Domain - raithlin.com X-BWhitelist: no X-Source-IP: X-Exim-ID: 1cEAlS-000M3A-Ol X-Source: X-Source-Args: /usr/local/cpanel/3rdparty/php/54/bin/php-cgi /usr/local/cpanel/base/3rdparty/squirrelmail/src/compose.php X-Source-Dir: /usr/local/cpanel/base/3rdparty/squirrelmail/src X-Source-Sender: X-Source-Auth: raithlin X-Email-Count: 15 X-Source-Cap: cmFpdGhsaW47c2NvdHQ7ZXN0YXRlLndlYnNpdGV3ZWxjb21lLmNvbQ== Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1800 Lines: 38 >>> I've already recommended that iopmem not be a block device and >>> instead be a device-dax instance. I also don't think it should claim >>> the PCI ID, rather the driver that wants to map one of its bars this >>> way can register the memory region with the device-dax core. >>> >>> I'm not sure there are enough device drivers that want to do this to >>> have it be a generic /sys/.../resource_dmableX capability. It still >>> seems to be an exotic one-off type of configuration. >> >> >> Yes, this is essentially my thinking. Except I think the userspace >> interface should really depend on the device itself. Device dax is a >> good choice for many and I agree the block device approach wouldn't be >> ideal. I tend to agree here. The block device interface has seen quite a bit of resistance and /dev/dax looks like a better approach for most. We can look at doing it that way in v2. >> >> Specifically for NVME CMB: I think it would make a lot of sense to just >> hand out these mappings with an mmap call on /dev/nvmeX. I expect CMB >> buffers would be volatile and thus you wouldn't need to keep track of >> where in the BAR the region came from. Thus, the mmap call would just be >> an allocator from BAR memory. If device-dax were used, userspace would >> need to lookup which device-dax instance corresponds to which nvme >> drive. >> > > I'm not opposed to mapping /dev/nvmeX. However, the lookup is trivial > to accomplish in sysfs through /sys/dev/char to find the sysfs path of the > device-dax instance under the nvme device, or if you already have the nvme > sysfs path the dax instance(s) will appear under the "dax" sub-directory. > Personally I think mapping the dax resource in the sysfs tree is a nice way to do this and a bit more intuitive than mapping a /dev/nvmeX.