Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755403AbdDRTCT (ORCPT ); Tue, 18 Apr 2017 15:02:19 -0400 Received: from quartz.orcorp.ca ([184.70.90.242]:58034 "EHLO quartz.orcorp.ca" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753293AbdDRTCP (ORCPT ); Tue, 18 Apr 2017 15:02:15 -0400 Date: Tue, 18 Apr 2017 13:01:38 -0600 From: Jason Gunthorpe To: Logan Gunthorpe Cc: Benjamin Herrenschmidt , Dan Williams , Bjorn Helgaas , Christoph Hellwig , Sagi Grimberg , "James E.J. Bottomley" , "Martin K. Petersen" , Jens Axboe , Steve Wise , Stephen Bates , Max Gurtovoy , Keith Busch , linux-pci@vger.kernel.org, linux-scsi , linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm , "linux-kernel@vger.kernel.org" , Jerome Glisse Subject: Re: [RFC 0/8] Copy Offload with Peer-to-Peer PCI Memory Message-ID: <20170418190138.GH7181@obsidianresearch.com> References: <1492381396.25766.43.camel@kernel.crashing.org> <20170418164557.GA7181@obsidianresearch.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Broken-Reverse-DNS: no host name found for IP address 10.0.0.156 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1622 Lines: 42 On Tue, Apr 18, 2017 at 12:30:59PM -0600, Logan Gunthorpe wrote: > > - The dma ops provider must be able to tell if source memory is bar > > mapped and recover the pci device backing the mapping. > > Do you mean to say that every dma-ops provider needs to be taught about > p2p backed pages? I was hoping we could have dma_map_* just use special > p2p dma-ops if it was passed p2p pages (though there are some > complications to this too). I think that is how it will end up working out if this is the path.. Ultimately every dma_ops will need special code to support P2P with the special hardware that ops is controlling, so it makes some sense to start by pushing the check down there in the first place. This advice is partially motivated by how dma_map_sg is just a small wrapper around the function pointer call... Something like: foo_dma_map_sg(...) { for (every page in sg) if (page is p2p) dma_addr[I] = p2p_same_segment_map_page(...); } Where p2p_same_segment_map_page checks if the two devices are on the 'same switch' and if so returns the address translated to match the bus address programmed into the BAR or fails. We knows this case is required to work by the PCI spec, so it makes sense to use it as the first canned helper. This also proves out the basic idea that the dma ops can recover the pci device and perform an inspection of the traversed fabric path. >From there every arch would have to expand the implementation to support a wider range of things. Eg x86 with no iommu and no offset could allow every address to be used based on a host bridge white list. Jason