Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756305AbcKVUan (ORCPT ); Tue, 22 Nov 2016 15:30:43 -0500 Received: from mail-io0-f194.google.com ([209.85.223.194]:33483 "EHLO mail-io0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755365AbcKVUam (ORCPT ); Tue, 22 Nov 2016 15:30:42 -0500 MIME-Version: 1.0 X-Originating-IP: [2a02:168:56b5:0:ac27:b86c:7764:9429] In-Reply-To: References: <75a1f44f-c495-7d1e-7e1c-17e89555edba@amd.com> From: Daniel Vetter Date: Tue, 22 Nov 2016 21:10:43 +0100 X-Google-Sender-Auth: 83FMdqvW8ivEXNbeg5aENtmUvLM Message-ID: Subject: Re: Enabling peer to peer device transactions for PCIe devices To: Dan Williams Cc: Serguei Sagalovitch , Dave Hansen , "linux-nvdimm@lists.01.org" , "linux-rdma@vger.kernel.org" , "linux-pci@vger.kernel.org" , "Kuehling, Felix" , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , "Koenig, Christian" , "Sander, Ben" , "Suthikulpanit, Suravee" , "Deucher, Alexander" , "Blinzer, Paul" , "Linux-media@vger.kernel.org" Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1833 Lines: 37 On Tue, Nov 22, 2016 at 9:01 PM, Dan Williams wrote: > On Tue, Nov 22, 2016 at 10:59 AM, Serguei Sagalovitch > wrote: >> I personally like "device-DAX" idea but my concerns are: >> >> - How well it will co-exists with the DRM infrastructure / implementations >> in part dealing with CPU pointers? > > Inside the kernel a device-DAX range is "just memory" in the sense > that you can perform pfn_to_page() on it and issue I/O, but the vma is > not migratable. To be honest I do not know how well that co-exists > with drm infrastructure. > >> - How well we will be able to handle case when we need to "move"/"evict" >> memory/data to the new location so CPU pointer should point to the new >> physical location/address >> (and may be not in PCI device memory at all)? > > So, device-DAX deliberately avoids support for in-kernel migration or > overcommit. Those cases are left to the core mm or drm. The device-dax > interface is for cases where all that is needed is a direct-mapping to > a statically-allocated physical-address range be it persistent memory > or some other special reserved memory range. For some of the fancy use-cases (e.g. to be comparable to what HMM can pull off) I think we want all the magic in core mm, i.e. migration and overcommit. At least that seems to be the very strong drive in all general-purpose gpu abstractions and implementations, where memory is allocated with malloc, and then mapped/moved into vram/gpu address space through some magic, but still visible on both the cpu and gpu side in some form. Special device to allocate memory, and not being able to migrate stuff around sound like misfeatures from that pov. -Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch