Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752674AbcLESja (ORCPT ); Mon, 5 Dec 2016 13:39:30 -0500 Received: from ale.deltatee.com ([207.54.116.67]:45687 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752581AbcLESj0 (ORCPT ); Mon, 5 Dec 2016 13:39:26 -0500 To: Dan Williams , Jason Gunthorpe References: <20161128165751.GB28381@obsidianresearch.com> <1480357179.19407.13.camel@mellanox.com> <20161128190244.GA21975@obsidianresearch.com> <20161130162353.GA24639@obsidianresearch.com> <5f5b7989-84f5-737e-47c8-831f752d6280@deltatee.com> <61a2fb07344aacd81111449d222de66e.squirrel@webmail.raithlin.com> <20161205171830.GB27784@obsidianresearch.com> <20161205180231.GA28133@obsidianresearch.com> Cc: Stephen Bates , Haggai Eran , "linux-kernel@vger.kernel.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@ml01.01.org" , "christian.koenig@amd.com" , "Suravee.Suthikulpanit@amd.com" , "John.Bridgman@amd.com" , "Alexander.Deucher@amd.com" , "Linux-media@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , Max Gurtovoy , "linux-pci@vger.kernel.org" , "serguei.sagalovitch@amd.com" , "Paul.Blinzer@amd.com" , "Felix.Kuehling@amd.com" , "ben.sander@amd.com" From: Logan Gunthorpe Message-ID: Date: Mon, 5 Dec 2016 11:39:13 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Icedove/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 50.66.97.235 X-SA-Exim-Rcpt-To: ben.sander@amd.com, felix.kuehling@amd.com, paul.blinzer@amd.com, serguei.sagalovitch@amd.com, linux-pci@vger.kernel.org, maxg@mellanox.com, dri-devel@lists.freedesktop.org, linux-media@vger.kernel.org, alexander.deucher@amd.com, john.bridgman@amd.com, suravee.suthikulpanit@amd.com, christian.koenig@amd.com, linux-nvdimm@ml01.01.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, haggaie@mellanox.com, sbates@raithlin.com, jgunthorpe@obsidianresearch.com, dan.j.williams@intel.com X-SA-Exim-Mail-From: logang@deltatee.com Subject: Re: Enabling peer to peer device transactions for PCIe devices X-SA-Exim-Version: 4.2.1 (built Mon, 26 Dec 2011 16:24:06 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1173 Lines: 24 On 05/12/16 11:08 AM, Dan Williams wrote: > I've already recommended that iopmem not be a block device and instead > be a device-dax instance. I also don't think it should claim the PCI > ID, rather the driver that wants to map one of its bars this way can > register the memory region with the device-dax core. > > I'm not sure there are enough device drivers that want to do this to > have it be a generic /sys/.../resource_dmableX capability. It still > seems to be an exotic one-off type of configuration. Yes, this is essentially my thinking. Except I think the userspace interface should really depend on the device itself. Device dax is a good choice for many and I agree the block device approach wouldn't be ideal. Specifically for NVME CMB: I think it would make a lot of sense to just hand out these mappings with an mmap call on /dev/nvmeX. I expect CMB buffers would be volatile and thus you wouldn't need to keep track of where in the BAR the region came from. Thus, the mmap call would just be an allocator from BAR memory. If device-dax were used, userspace would need to lookup which device-dax instance corresponds to which nvme drive. Logan