Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S969115AbdDSSd2 (ORCPT ); Wed, 19 Apr 2017 14:33:28 -0400 Received: from quartz.orcorp.ca ([184.70.90.242]:49426 "EHLO quartz.orcorp.ca" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1030752AbdDSSdX (ORCPT ); Wed, 19 Apr 2017 14:33:23 -0400 Date: Wed, 19 Apr 2017 12:32:47 -0600 From: Jason Gunthorpe To: Logan Gunthorpe Cc: Benjamin Herrenschmidt , Dan Williams , Bjorn Helgaas , Christoph Hellwig , Sagi Grimberg , "James E.J. Bottomley" , "Martin K. Petersen" , Jens Axboe , Steve Wise , Stephen Bates , Max Gurtovoy , Keith Busch , linux-pci@vger.kernel.org, linux-scsi , linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm , "linux-kernel@vger.kernel.org" , Jerome Glisse Subject: Re: [RFC 0/8] Copy Offload with Peer-to-Peer PCI Memory Message-ID: <20170419183247.GA13716@obsidianresearch.com> References: <20170418190138.GH7181@obsidianresearch.com> <20170418210339.GA24257@obsidianresearch.com> <1492564806.25766.124.camel@kernel.crashing.org> <20170419155557.GA8497@obsidianresearch.com> <4899b011-bdfb-18d8-ef00-33a1516216a6@deltatee.com> <20170419171451.GA10020@obsidianresearch.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Broken-Reverse-DNS: no host name found for IP address 10.0.0.156 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2595 Lines: 82 On Wed, Apr 19, 2017 at 12:01:39PM -0600, Logan Gunthorpe wrote: > I'm just spit balling here but if HMM wanted to use unaddressable memory > as a DMA target, it could set that function to create a window ine gpu > memory, then call the pci_p2p_same_segment and return the result as the > dma address. Not entirely, it would have to call through the whole process including the arch_p2p_cross_segment().. Maybe we can start down the road of using ops for more iommu steps with something like this as the helper: dma_addr_t dma_map_pa(struct device *initiator, struct page *page, void *data) { struct device *completer = get_p2p_completer(page); dma_addr_t pa; if (IS_ERR(completer)) return SYSTEM_MEMORY; // Or maybe ? return init_ops->dma_map_pa(..); // Try the generic method pa = pci_p2p_same_segment(dev, p2p_src, page); if (pa != ERROR) goto out; // Try the arch specific helper const struct dma_map_ops *comp_ops = get_dma_ops(completer); const struct dma_map_ops *init_ops = get_dma_ops(initiator); /* FUTURE: Let something translate a HMM page into a DMA'ble page, eg by mapping it into a GPU window. Maybe this callback lives in devmap ? */ page = comp_ops->translate_dma_page(completer, page); /* New dma_map_op is the same as arch_p2p_cross_segment in prior version. Any arch specific data needed to program the iommu flows through data */ pa = init_ops->p2p_cross_segment_map(completer, inititator, page, data); out: device_put(completer); return pa; } // map_sg op: for (each sgl) { struct page *page = sg_page(s); struct arch_iommu_data data = {}; // pass through to ops->p2p_cross_segment dma_addr_t pa; pa = dma_map_pa(dev, page, &data) if (pa == ERROR) // fail if (!data.needs_iommu) { // Insert PA directly into the result SGL sg++; continue; } // Else pa & data describe how to setup the iommu } > > dma_addr_t pci_p2p_same_segment(struct device *initator, > > struct device *completer, > > struct page *page) > > I'm not sure I like the name pci_p2p_same_segment. It reads as though > it's only checking if the devices are not the same segment. Well, that is exactly what it is doing. If it succeeds then the caller knows the DMA will not flow outside the segment and no iommu setup/etc is required. That function cannot be expanded to include generic cross-segment traffic, a new function would be needed.. Jason