Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752015AbcLESIo (ORCPT ); Mon, 5 Dec 2016 13:08:44 -0500 Received: from mail-oi0-f51.google.com ([209.85.218.51]:36388 "EHLO mail-oi0-f51.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751503AbcLESIk (ORCPT ); Mon, 5 Dec 2016 13:08:40 -0500 MIME-Version: 1.0 In-Reply-To: <20161205180231.GA28133@obsidianresearch.com> References: <20161128165751.GB28381@obsidianresearch.com> <1480357179.19407.13.camel@mellanox.com> <20161128190244.GA21975@obsidianresearch.com> <20161130162353.GA24639@obsidianresearch.com> <5f5b7989-84f5-737e-47c8-831f752d6280@deltatee.com> <61a2fb07344aacd81111449d222de66e.squirrel@webmail.raithlin.com> <20161205171830.GB27784@obsidianresearch.com> <20161205180231.GA28133@obsidianresearch.com> From: Dan Williams Date: Mon, 5 Dec 2016 10:08:37 -0800 Message-ID: Subject: Re: Enabling peer to peer device transactions for PCIe devices To: Jason Gunthorpe Cc: Stephen Bates , Haggai Eran , Logan Gunthorpe , "linux-kernel@vger.kernel.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@ml01.01.org" , "christian.koenig@amd.com" , "Suravee.Suthikulpanit@amd.com" , "John.Bridgman@amd.com" , "Alexander.Deucher@amd.com" , "Linux-media@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , Max Gurtovoy , "linux-pci@vger.kernel.org" , "serguei.sagalovitch@amd.com" , "Paul.Blinzer@amd.com" , "Felix.Kuehling@amd.com" , "ben.sander@amd.com" Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1929 Lines: 41 On Mon, Dec 5, 2016 at 10:02 AM, Jason Gunthorpe wrote: > On Mon, Dec 05, 2016 at 09:40:38AM -0800, Dan Williams wrote: > >> > If it is kernel only with physical addresess we don't need a uAPI for >> > it, so I'm not sure #1 is at all related to iopmem. >> > >> > Most people who want #1 probably can just mmap >> > /sys/../pci/../resourceX to get a user handle to it, or pass around >> > __iomem pointers in the kernel. This has been asked for before with >> > RDMA. >> > >> > I'm still not really clear what iopmem is for, or why DAX should ever >> > be involved in this.. >> >> Right, by default remap_pfn_range() does not establish DMA capable >> mappings. You can think of iopmem as remap_pfn_range() converted to >> use devm_memremap_pages(). Given the extra constraints of >> devm_memremap_pages() it seems reasonable to have those DMA capable >> mappings be optionally established via a separate driver. > > Except the iopmem driver claims the PCI ID, and presents a block > interface which is really *NOT* what people who have asked for this in > the past have wanted. IIRC it was embedded stuff eg RDMA video > directly out of a capture card or a similar kind of thinking. > > It is a good point about devm_memremap_pages limitations, but maybe > that just says to create a /sys/.../resource_dmableX ? > > Or is there some reason why people want a filesystem on top of BAR > memory? That does not seem to have been covered yet.. > I've already recommended that iopmem not be a block device and instead be a device-dax instance. I also don't think it should claim the PCI ID, rather the driver that wants to map one of its bars this way can register the memory region with the device-dax core. I'm not sure there are enough device drivers that want to do this to have it be a generic /sys/.../resource_dmableX capability. It still seems to be an exotic one-off type of configuration.