Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S934630AbcKWT7G (ORCPT ); Wed, 23 Nov 2016 14:59:06 -0500 Received: from mail-sn1nam01on0045.outbound.protection.outlook.com ([104.47.32.45]:16256 "EHLO NAM01-SN1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S932754AbcKWT7A (ORCPT ); Wed, 23 Nov 2016 14:59:00 -0500 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Serguei.Sagalovitch@amd.com; Subject: Re: Enabling peer to peer device transactions for PCIe devices To: Jason Gunthorpe , Dan Williams References: <75a1f44f-c495-7d1e-7e1c-17e89555edba@amd.com> <45c6e878-bece-7987-aee7-0e940044158c@deltatee.com> <20161123191215.GB12146@obsidianresearch.com> CC: Bart Van Assche , Logan Gunthorpe , "Deucher, Alexander" , "linux-nvdimm@lists.01.org" , "linux-rdma@vger.kernel.org" , "linux-pci@vger.kernel.org" , "Kuehling, Felix" , "Bridgman, John" , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , "Koenig, Christian" , "Sander, Ben" , "Suthikulpanit, Suravee" , "Blinzer, Paul" , "Linux-media@vger.kernel.org" From: Serguei Sagalovitch Message-ID: <1dffffdf-55e2-0842-60f0-ffbfbd70aa2d@amd.com> Date: Wed, 23 Nov 2016 14:24:33 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <20161123191215.GB12146@obsidianresearch.com> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: BN3PR16CA0045.namprd16.prod.outlook.com (10.165.72.141) To CY1PR12MB0697.namprd12.prod.outlook.com (10.163.238.158) X-MS-Office365-Filtering-Correlation-Id: 2222146b-01d7-42b2-7b31-08d413d65fda X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:CY1PR12MB0697; X-Microsoft-Exchange-Diagnostics: 1;CY1PR12MB0697;3:XqQCYOdbX3XPjI0/9LwE6AMImaMcGnwkm8YvYR+p1NomVoFLwQiO7IQ0i/t8juX/9oamRc0VUDZYw6rxO3QBL1bUQYakqAYmZCag8JGr1Eh8vTYnwLCsGA52ctNrNrrpDfNNiYtBs7wBfRvp16WwHMtMRouU7eT55q0ZuhSyWP7nHrJSmsBpVsuKX2lurQhCrft9RlJXXGIP04OiCSf5S1mjkyDQgKYtNSkPl+e+tKa8s9UFGuk5oqY+24s7FIV5RIOMqrAqWT+w+A9jMmxnAg== X-Microsoft-Exchange-Diagnostics: 1;CY1PR12MB0697;25:RTavwUsTX4V5jybixFHLFk4kZUmYeie1aSRylEFFA36dkgBABy9cfHld9kbelXl4cMl7Y6BsANnfADuBJsXWv3dK2igOAhQhfgy6Ma9FpdMfxZ26imMWSa2oahcix6cMDdcGaWzGf7JfeEdzH49EmHUZwRecbtOo2gdTx9B49Zo0YqNnsHB9VIUYQDEQ1iRwnNt2ZERWc8srh9tzH54UsgzapIkU9/85GoUNtIWCbwZuJrqem9NRU4Xke1MVNdHr8Pj2notiBrI+//8rozYM6+NvfkSuVRNSW6g0lJlJPztLqx+Tv7kBAC7qXNhjxUdNaEnGvDX747ZrOItR0YpF3kb368JIALyVYopgORQgL2UslveMfXJ2glxa6THNinW+0o+tyYwwAdS57PWf5wQPTFxRXnXjd4ZJAjUawg6r50qy8Cyna4HpYt1e9deYkSlNUFdDviNm0F1UFTwfIKykp4BeF6d1kuFvPNOxrVQ2MoLhwIhastmoELodGgNqCPPeDIUGN+rklKGv8BKEON11LAupUgvrr3tm/p3pmvpbNaHYghZ08unluNfYQ59rHmP9Tfw6/lD42YmIiVhiU7sPxM82JsXuU79QFiCLuV5WH6qWg7KENXbmKAOkiKOgWolh4NpDUBDOC4LjLc1j/Hz1S5qeevbCvHE5/fenDdwP1rU00tAbmD8BLnOUo1wPAf3LhPOCH9F/eHtFpes9MMKApIH9tYKVTP9PJKo2AT8dyHoT0p56NtNpXKZ4jMYXIL24lPfZdQUH0d8kmnUSjK6wJg== X-Microsoft-Exchange-Diagnostics: 1;CY1PR12MB0697;31:mv/BNqq8zKrHvjb/MPkXjzl3WUMzfia/l+H6CQvRm///bWGDn2TgUWsjrzSG0+S3DHzFvPRUFlSkrfv3NWxHqfymreq43GJCQZntdyVFL5gvTGOZWCUT6UCw45JR6tu2kxF6/3mOCS6MNLz7vzPMP2Afy77vaR5zeB33/pOO10W8/BXfIWoE8dK0WyNOL9pKuOdBIkcimFASbnXzsJOZd3bxZGw1ynAYWZx3zTsGkRUUn0RVT/ehu1mb2P1hOhcqyNbFsDu+E0LOjnXRMcZXUfJdMxBu7Z2BShn5VF0eHI0=;20:+aknLFkE2V1hcw2nMbaWoVhc4y+gfcT1fo7M7T1WX337OnBIzilm4cT5yhKD1RUL5EDBv6QQqhTMpgnM+qA4Ok2SN+ci9bs6ojKYQVFR/cl+guSb0DPZULZmYm+I8VKIKH8bRGZESXEkTR4RVcml5X7YNq/ajYAteON6lofZB4Bfp4FFsHdxJNhdyXoyK81SSlpT3F75vFnb47a25D9RICmoPbURAHNc56u6QlDAK2Ql2qJq2k5JttOPTG5uusEzWLp1tfRU7/NeeZ8U+HgrP14b8Ovn4dXjzY0QIBNTVUpNdf9C/rSnZ7Fs3x/qVi4nzHBAm9F/HWKLzIQt+bTxXuNEe9kHTAq5YXN/OcndgKrF+JAMr3UK63slOlAAbW2rfSTzrzJP10h0jKrETUW5oNwXCYn7mOHkbaAZPrJXHMmUHZK//yxvmihT/WAbPKZ25bbRbzER6KlLVHRHwobELKuC2RytixxFcM8wKRt6vur/YIlcYPnOhzuGdxyNTZlV X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(278428928389397); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040307)(6060326)(6045199)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6055026)(6061324)(6041248);SRVR:CY1PR12MB0697;BCL:0;PCL:0;RULEID:;SRVR:CY1PR12MB0697; X-Microsoft-Exchange-Diagnostics: 1;CY1PR12MB0697;4:eoTlAcXMvnXfAup3PDLPA1DjRnW6QgL7d6QAzckmT6ar4U8h91GUhy9inAkMcikmwEjoi4qBEwWOnDl5/B0aaeAbQ2y8NVvZUWGmGQMVMeI1dCmbno/6eWS+K7sdSYjTWu4RyKFQgQ7SHbeGmfZq91JAv3pgR90HCBizPXsW8rEohRsqOmTHSd9ATlNj+6Ogi/TK0PdnFDbKAKYSzgv+SNkRUK8UscJRbKpmd6Gs0WD+daL/VYzuapgTgnMLQVTy3naF42dIuLf6vkFro+bXA6GvPN4TR6NRJKd5w1aTWu7qa3R7uALTyDdHbsC/ERPi8/OBl7wMMYPTONlxnXQteoo1V6JgN+CcQK8tN5v7M6zGBYeFYqlLVulOvW1gh0gbPeL6AV7mkK53b8uOdtM8zjjQq564YzuwdTi6HXX4B3UeJttY2k9jEJQBt56OgeJMkbKUj9SeTI1aiLntH6Xe2BPrS+R1h8KZfe4pcXapyqK0m9H5qT4vw+zxbM4+KGQPdZkq5faUh5ByVB7djV03ytYrv/ArUe3sYDhS5hIYCZY= X-Forefront-PRVS: 013568035E X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6049001)(6009001)(7916002)(24454002)(199003)(189002)(377424004)(377454003)(229853002)(2950100002)(7416002)(64126003)(83506001)(23746002)(31696002)(105586002)(92566002)(5660300001)(7846002)(106356001)(65826007)(2906002)(86362001)(65806001)(7736002)(47776003)(6666003)(305945005)(65956001)(66066001)(8666005)(3846002)(230700001)(101416001)(5001770100001)(54356999)(77096005)(4001350100001)(33646002)(97736004)(76176999)(50466002)(4326007)(189998001)(50986999)(36756003)(4001150100001)(31686004)(8676002)(38730400001)(93886004)(81166006)(42186005)(68736007)(81156014)(6116002)(7059030);DIR:OUT;SFP:1101;SCL:1;SRVR:CY1PR12MB0697;H:[172.27.224.67];FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;CY1PR12MB0697;23:Kb0bokafYKDxQSLz6frn9sYx46TeccGyCb3co?= =?Windows-1252?Q?9GzAzONcwWK9g3FISZyN7ROTnd9a9Zhx/6uyPFzvB/FW1Ty/PwkS8Li4?= =?Windows-1252?Q?rudzsOq8ybKA3ZyP/oLBLDinYbQlLYDPrLHIP0EDpga2PUQ0WZo8xGKZ?= =?Windows-1252?Q?R7GdBFh/HcigOrFzydUjOpN05j/3O8BBZqGEIENhPJwyMFFl6vqwiniB?= =?Windows-1252?Q?nHfYrXnXYI4UjX6E+df8TIvLHtpeaYHMF/FYROaA2Bl9Y5DM2+V8SAst?= =?Windows-1252?Q?XURrY9Y3E9+w0j3Bm7KQ7oKDhJY+WbLZRXwErdHffNWlM2ekT3sNHUSu?= =?Windows-1252?Q?pUop8iC/yJ9g6s2ftjYQZOJVTNZ/iWyWnru+oKeHvbeQ/SdOTwHcBlYe?= =?Windows-1252?Q?PS7ds2i7tlC0lGaLxiw95ogwwFiqpgaj8Zn5yxl+B62QL4xzscewa3Xi?= =?Windows-1252?Q?v+mGsy+hT5O1Z07RVBAMoEoMX/ha4EcPTeQtdmqfNqtbX5LBHu62sea9?= =?Windows-1252?Q?xlo4WBfDbvBSqC5B1yQEtUkGpZHz3MhwT0Uo1oGrT0pQU0s3oyHe7o3b?= =?Windows-1252?Q?xwgGiWyj0Gl5OEWjVNxTA6ljVFDVGoCkwJfy3wCJgFhTzji4HdPBw5Og?= =?Windows-1252?Q?q+/kQ1ZCgER5ylalUOyx0JnNk6oHoMZQAGoyzCobgwWDEBNTCxMiOJej?= =?Windows-1252?Q?KbaEPd5CV9TBxHHlrKaGMD/UXm5SzgiTdFMjcPC7YGXKTarDpuWaVbej?= =?Windows-1252?Q?IzB9n8QVa+DCmYfmSnwkXPF3VeqatNIzJtKBIkgIarhmwhgBOfSQ5Owo?= =?Windows-1252?Q?/onkWR/OM82eCgrgECb8stScYXmCXbEPsa+JuGSrYxQSu9ckrkFGXeyc?= =?Windows-1252?Q?DKgwMDbUHoWsNPBLUJXlAvlS8Kdj/XwCPsH9Q8921alko5MqKusdmMtP?= =?Windows-1252?Q?l5NZbCUpwWr+/zuIjC8tUpAgYInRD8ZEM7Up6yRXjMuhv5nEdrj2unjF?= =?Windows-1252?Q?PX30qfhNrSCzQTMcCMsykp7qDB1a/mm6tRbQ1UAZlQNzzGKmbJAtmic7?= =?Windows-1252?Q?Wcvn+TcwusmnwgIeUl0qeXuR66UVwL0ZqIpzMG8ceKnxhoGRWMCatNhp?= =?Windows-1252?Q?2uJEl81XzMWpP70Ukl4s05TjYyOqdAtyhtzge0TGqqVmVFwNC2grBpL7?= =?Windows-1252?Q?Ib6RiCn/QRfRE7xSpfsOg/2ib3X2ePjWHnFn2LBY7CskicIOKFVl8t8D?= =?Windows-1252?Q?x7sg1BzlUPD0c56cSkZlxG0VQF/H6Rv8L/pqG37tfQHUFGLa3xMMn//z?= =?Windows-1252?Q?dVxDK0RK0DZ66vwwqtTTbqwvGB16rc2GgAdYY3NFlVUTmVxLS/NnernB?= =?Windows-1252?Q?aMAZrb9AzbJVPrS3vUCeVOZ/WaRE2WxHgvyPStExs2bEPJ6fM3X/pHlm?= =?Windows-1252?Q?z4LMdXNrvjQglxIXJoWk88kGQiREJpwFSePXm5x9pRNqCGZRWPsbCYHl?= =?Windows-1252?Q?NF8FzAaSe28QsWonTVWf4ish0di?= X-Microsoft-Exchange-Diagnostics: 1;CY1PR12MB0697;6:Xc25ltUOPZRDSa1ISdZvrLeJ+hSa6CECQO3fwAchnTNaChKQ9NNOPcT5wEpMvJDd0xoRfttUUsrMErtbWHFnDxaBeaUZ/y0V3HVqesbZgj8K8EGav4hT62ajIdn2m9rVdMwjhESVBAEP8YnbZdJzIi/aqb04cNlZ3mU/sSjeqiaprf2SEXQuSmscwSWm6KuRV0Jo9wFQCsEICIgd0Bw/XhGVDN4F0bfpcfAKuNeFxHnebMlQO5i+3lecwcJxzX0WorAFBHMI3owj/Lm/5U4XrjNzb5znOi9ANLGRPeYBIWTi6cfP3/mzapIiVGBL4AqVSNkaybutKgLSNBauuP1cn8VY1Ph72OW65qQE8EMBJcojDBkifm3QnSqN9mfWcV5Y7a0liyfoqwJ+8/U3qYvCBD3k4WofBt3O0btdMa8DrKhiy3hK8YYTgazsxqOJ0TotK5n+to3IUmjs060sEbVniZKdm2nBx2OIpbiNhtCrLHiQ1L42VHHfAHT0/XDpKScg;5:t5v4eVHDoMrpF03pJACF6nijVKu/8A95khJ5oQHOSHMlLuvd+HQEkMngu7Fcgz4BN6+N9yUaQA/NTHekdY0Lw5BjbqhjGN/UM88VSEH2yGMLezHsRQge+2Uy2uurH26b112yU3xU5bxFx6n1lf+vgQ==;24:lmcmSoTNoQZPKQ/KHGMkSTUMoZ/IrccCkaMzHuyZcC8JRSSOmdWj08ZudO4Mu7DTEMp0ATKR7tdKnzmu8Njb4JHTx5dYlxeaMn0pfw1YgBw= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;CY1PR12MB0697;7:AcetJ7f3LDbrc1j+fj2gu2CKMVVfjnAqgoYcEQ7wYazvLyTBpiXPkG8Y3XFukSrIakps8WvRIwtvUuEusnJz5BkVK+clUk7kOg/zhdB5QxnEzH/8Emn/6UhPdIqXybAGJLYowgcI8qWgeK2h+7lGlUJTloZ7FKEIDuS6NULF55hDsFZI8aKKV9mWGs/o3rqflL1Own+zAx+86rOZaJtiBXug1M1j5tx3YiPVAhnE66cY3QGPjvQkAKhQVm+VwNwVTj+vfhMGj95T43cBG0XRFFJwQR1FclSDnjctHGlxKij+BsSNg38xdnfzYbGpX6mK203kZPlChgO62hp8+QqyPfh2WFNLvbOpzxExB2PGgn8zRYb8CgcOI7JJvJPb38UFiaECEBeyq4XMYk8BwQ0+Um2S9kFNXBIbcgmgiswZSghCPUV+8zczpoRMSQOKoMmStGS1JcbHL6a2nmj56i0n4g==;20:qoHcW9LzlYiHGb+vOndu8S+bNUYAsCWxK+anTZG/+Vog/ZKYSvuw0EEkxyyC9BIT6CCNjo83V8KKVaklgufjgFU8soSfGEURQI64e2gwyCngAmUeNskh2w+HBUWAYE9IvdOsJo3cdOk4uunL3h1zwUh/qnXVM5SWJ1A813A7TyTKuJOE90kYrCJ9bRK1TyOkzrcWu2rBAr+rVU1Q2PYOE7u+joi2GKdBRyYB5fxPUVI7Nam9aRYU3MMFqas3QGKS X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Nov 2016 19:24:38.3096 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR12MB0697 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1231 Lines: 29 On 2016-11-23 02:12 PM, Jason Gunthorpe wrote: > On Wed, Nov 23, 2016 at 10:40:47AM -0800, Dan Williams wrote: > >> I don't think that was designed for the case where the backing memory >> is a special/static physical address range rather than anonymous >> "System RAM", right? > The hardware doesn't care where the memory is. ODP is just a generic > mechanism to provide demand-fault behavior for a mirrored page table. > > ODP has the same issue as everything else, it needs to translate a > page table entry into a DMA address, and we have no API to do that > when the page table points to peer-peer memory. > > Jason I would like to note that for graphics applications (especially for VR support) we should avoid ODP case at any cost during graphics commands execution due to requirement to have smooth and predictable playback. We want to load / "pin" all required resources before graphics processor begin to touch them. This is not so critical for compute applications. Because only graphics / compute stack knows which resource will be in used as well as all statistics accordingly only graphics stack is capable to make the correct decision when and _where_ evict as well as when and _where_ to put memory back.