Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751736AbcKYIVm (ORCPT ); Fri, 25 Nov 2016 03:21:42 -0500 Received: from bombadil.infradead.org ([198.137.202.9]:59519 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750726AbcKYIVb (ORCPT ); Fri, 25 Nov 2016 03:21:31 -0500 Date: Thu, 24 Nov 2016 23:58:17 -0800 From: Christoph Hellwig To: Logan Gunthorpe Cc: Jason Gunthorpe , Serguei Sagalovitch , Dan Williams , "Deucher, Alexander" , "linux-nvdimm@lists.01.org" , "linux-rdma@vger.kernel.org" , "linux-pci@vger.kernel.org" , "Kuehling, Felix" , "Bridgman, John" , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , "Koenig, Christian" , "Sander, Ben" , "Suthikulpanit, Suravee" , "Blinzer, Paul" , "Linux-media@vger.kernel.org" , Haggai Eran Subject: Re: Enabling peer to peer device transactions for PCIe devices Message-ID: <20161125075817.GA18428@infradead.org> References: <20161123190515.GA12146@obsidianresearch.com> <7bc38037-b6ab-943f-59db-6280e16901ab@amd.com> <20161123193228.GC12146@obsidianresearch.com> <20161123203332.GA15062@obsidianresearch.com> <20161123215510.GA16311@obsidianresearch.com> <91d28749-bc64-622f-56a1-26c00e6b462a@deltatee.com> <20161124164249.GD20818@obsidianresearch.com> <9cc22068-ede8-c1bc-5d8b-cf6224a7ce05@deltatee.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <9cc22068-ede8-c1bc-5d8b-cf6224a7ce05@deltatee.com> User-Agent: Mutt/1.6.1 (2016-04-27) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 871 Lines: 14 On Thu, Nov 24, 2016 at 11:11:34AM -0700, Logan Gunthorpe wrote: > * Regular DAX in the FS doesn't work at this time because the FS can > move the file you think your transfer to out from under you. Though I > understand there's been some work with XFS to solve that issue. The file system will never move anything under locked down pages, locking down pages is used exactly to protect against that. So as long as we page structures available RDMA to/from device memory _from kernel space_ is trivial, although for file systems to work properly you really want a notification to the consumer if the file systems wants to remove the mapping. We have implemented that using FL_LAYOUTS locks for NFSD, but only XFS supports it so far. Without that a long term locked down region of memory (e.g. a kernel MR) would prevent various file operations that would simply hang.