Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752618AbcJZIkN (ORCPT ); Wed, 26 Oct 2016 04:40:13 -0400 Received: from mail-db5eur01on0054.outbound.protection.outlook.com ([104.47.2.54]:13184 "EHLO EUR01-DB5-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1750967AbcJZIjV (ORCPT ); Wed, 26 Oct 2016 04:39:21 -0400 Authentication-Results: spf=pass (sender IP is 193.47.165.134) smtp.mailfrom=mellanox.com; intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=pass action=none header.from=mellanox.com; Subject: Re: [PATCH 0/3] iopmem : A block device for PCIe memory To: Dan Williams , Stephen Bates References: <1476826937-20665-1-git-send-email-sbates@raithlin.com> CC: "linux-kernel@vger.kernel.org" , "linux-nvdimm@lists.01.org" , , , Linux MM , Ross Zwisler , "Matthew Wilcox" , , "Christoph Hellwig" , Jens Axboe , Jonathan Corbet , , , "Logan Gunthorpe" , David Woodhouse , "Raj, Ashok" From: Haggai Eran Message-ID: Date: Wed, 26 Oct 2016 11:24:51 +0300 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [10.0.52.254] X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:193.47.165.134;IPV:NLI;CTRY:IL;EFV:NLI;SFV:NSPM;SFS:(10009020)(6009001)(7916002)(2980300002)(438002)(3190300001)(199003)(54094003)(189002)(377454003)(24454002)(23676002)(7846002)(36756003)(19580405001)(4326007)(5660300001)(626004)(7416002)(50466002)(19580395003)(87936001)(31686004)(8936002)(230700001)(64126003)(33646002)(11100500001)(83506001)(6116002)(3846002)(47776003)(31696002)(305945005)(77096005)(76176999)(356003)(8666005)(65956001)(50986999)(54356999)(586003)(5001770100001)(65806001)(4001350100001)(189998001)(81156014)(92566002)(65826007)(2950100002)(81166006)(8676002)(86362001)(7736002)(2906002)(106466001)(7059030)(3940600001);DIR:OUT;SFP:1101;SCL:1;SRVR:VI1PR0501MB2366;H:mtlcas13.mtl.com;FPR:;SPF:Pass;PTR:mail13.mellanox.com;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: 1;AM1FFO11FD039;1:INIjs5Du+TLn30FUQHr7kFerVB7jX6Qit/kwUhc5TSIMmCtdhHW2SWwiPTIk46HvEViAEaQ1uGUhoB01vhCMeoT9AIw9JeTyHRkzuCCUbkI4G/LiEm6Pjp605sEvA4/savHoVn0Kt8V2GfLWEKP4fr7f3G1EmsmDyOuebH5hiozvybmOcHXZC+vdMsswUZYkQVD/gzMPFpqxP30om0j1UGWFMSQf+W8DC9rWDDjBm+b2r8I5t+piEj20DRG+sYJDMipaYd1bJEANdu6j9HF+gwYmqvFdKIR1RbfR5ROMC99/zMPO3KdI/yvAUvtmYT+LYlIGDe72PJ8w+cnWAE46ffRi2ZXZhqd3WOr+fPiU/ij9huYoUPUfNxvZUnge2MBnHJEi1iTJyDMvNXOP8ZoE6WARx54z26609m3ztV2NC0WxR8JT0yG9v5NbIXgqOkdnA/a3B3niiUvqdcM5N6wEQfI4jt2XfcAKRICiWzobpiZf+XBK5mDBYkH9U9k4dOccfBKG2iu9I1bBs+bAcKDH5epHBmYohQ4pqjviH7BhvIz/bhepu0fdKcr5ZYJX7NaJcIQFgdB64ymnAvYD3p/gxg== X-MS-Office365-Filtering-Correlation-Id: 0476e481-32c9-4461-8740-08d3fd799699 X-Microsoft-Exchange-Diagnostics: 1;VI1PR0501MB2366;2:Mrb5pKRmIJwqWqzByq7p8ICeffdKm3jvHFgVyIZ9k+dVhi0DAP3ZsMlxbb35QwCV5kFd6Zrcxh/27ZyBTVX7vHJ1xhNjfy3fgM2KEYV/T6nd5Rkl7Kvz+H2ZmzPQ2thTU+xkd3l1OnEOuG/k9jOkR/8TRvICTcNwXnhqDoIDpz/qyDPBeHQPKkJbw9Q4jFwiq/yKwMF1v6qzKFO4kyNnlw==;3:fsBb7xnxWSstfxGc0HCFNMnBXk8+lP8YRoS9uOCtCAMSVDq5hgyaru+d5UUCON3alVyXGDaikb+o7MkAEz0XCTl2szH7UMNyvt6Gn8bMzdIA4nuZfm5wmlxi1HXeQKmmaOay4mJcAx07uO6UWRQIs4cXahl/O95i+8vxF8Kk5YNopxxOxTC/FN30ds8Z+C4zYvxx5WtrFTqB1SrID/coI5bi+IFqxdX/T4hBu4dRm5o+WF7JFFBfvupgceFDrkaOEkTGJGUp6z2Rq/Ua1bNBIVyBaYf1/+y58PWZl17TXUM= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(8251501002);SRVR:VI1PR0501MB2366; X-Microsoft-Exchange-Diagnostics: 1;VI1PR0501MB2366;25:HYgiTDd3kyq2zySInDhFCDEVc6nWCOzBG3qFZLC4CbvwVU+p3OoOkyY0LSJdqahNLHYuRzHbFWEPnJjvCzYcxXlNBKOHoqFAMfVxhf/vPUf62636wRZI1lpN226Y5Btx4j13PJTMCmtA0WynElXw1RZM9Xm7ONBYApFj6yuNwAoSZhsQPEY0DY6ZRGMX4cRB7iZ6UG6eQQFuw834M3oqZPH17h++vvEcBM2lCty9AFfSw1QYyd3sDqenIBgPrGcuQB5iYlLs7GQtCzPULGjACImstN4MkEqEgaPpg2r6x6mD4U2klFY/zH4Wa7J9orjsjIgS/FyJSum/Lim4jH7aVE71WJXYjpSpNzZf8n8k9OYJMDAR806PoQS/u0VZmS5r8F1ET/Jyfx20R8b+46LyTbdtocAxy4hjQ6KZT40elSV1UC7XtocskfFsfG+ST0nVN8VTICi3TEWPX6g9ZC4JpXzi9VQlIQsV6p6c5DxPGxYxJEOfbsrou8DbYJ8BdTYLU8b9HgKO6barIHaoHmqiI8YUtTctj6szhAZuS1CbYJa+Ww9jASR9gyOjLaYtkP7feu0ev5OYQc1ttXpyG936j/XuoAqnSpv0vVcMP3g7E99TNamgtrJW9xJemZeS+rIGHT1TyPdUj2Caehkil5jS/YgZ7h3jTS/8RUWNU96S/MDefnha47q02UknhVrHcPYtZ5+NTnbbl2huTQaZw0Axz2+adTliSUb5GpCAl9vOGfWWaw6SdJJaN1/J8Ny3emuekl4GKlGEXskwwM+zZ0SsccSz9z7w3+KLF+1hdLvQi08= X-Microsoft-Exchange-Diagnostics: 1;VI1PR0501MB2366;31:GFGLDp0azvjbeFnCZOADmq2qaeIqbax5kcnUitAXuHEuK32alHB/DdVC0IeNJBzxG3ObKtAXeMm+LwhBqOHuN+8flUFrI1vxvnKBwnZ+s/5SH4Zb5oNDef8M1WwtohwCkAvx9+ca89mCP745u1jgiDZTaNgCnXRzfm4cJilnUULCgx8HrWOXB6Kc8vICkt47xQA3wHEQnNOUBz6VRCzEUB04LRRZp3vvcsFzJJkWjY/HFpKHNKpU2VOwDojhmPujRrkUq0cTp2C98leOQf848w==;20:nDWz88n8gQmuVA6T1wYSR0OU7PN7Tx/JZ0L+3lVnFTpP/YPIo9mp80TNHelr+UzyphZwyCZf3+kcAqTGRJJO7ITvHQHeu6LcKeRcRQyJZOXsziYl5gKe7Z+VlI4T3aLRnCtI5oFkOFz/imBOYTR3Js/Ka+nW18LSSIPTcfTSOvLiFtoZ7HYHJYobC4pYiDKwOHSGTywUDgpDtSnfL69IGRXIaDqMXs1qe4RDjBIKX79PqETTU3hNJTWqr7r7U3ZXEM6yAVkYvh2CfvSxw/T+rywA0ecN8J3cMhKMfGyi9HPCoMpwtkiE2OyU/l3d3IWLuXlp3RQANwEErZVegFkouDa97w6Icg6VtmjTuFjZJLq8IxKJv0pd6X1HMm0SVj6ubAbL/VeuHcGhZ953+JIN4EMfT1DUY/lnU223ejayauahb0S3c2feNnnWUuSJXnZRO5dY2s+VoB4M3QUZGU0b2+umWjHLmm6/p1VKon9KziRsmh8ih6hbGB9eIvY6FaB8 X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040176)(601004)(2401047)(8121501046)(5005006)(13016025)(13018025)(3002001)(10201501046)(6055026);SRVR:VI1PR0501MB2366;BCL:0;PCL:0;RULEID:;SRVR:VI1PR0501MB2366; X-Microsoft-Exchange-Diagnostics: 1;VI1PR0501MB2366;4:BwWrkqjelVWCMgNGQn4R7S+HBiMRU++SCGyA8aTU3Bdz0bCys7/zw0gYQUWWMWVCbhNQdkvc7VaeLf3aV+f/aiRMcYpyoQ/S1XJNmt2HhT3uELHmA9Il7It8DWYxnLb4cVMNVEHiB9llQR5jTnQETkDtBuwzytd/BUpFYbrxIqNXhsAiFHx6aYXWthw7/Z0yrXyzh6Q/ixDXcbtblIebxOcrw6jDZckrxGYVp0uJEqnMweDZHKlTsSoh48vnd5wDKA8tCql1P/vxaUYyzUY/fdX6NUCLJtWPft1lxmkmxA9l72fYf/34mUr2WQnN56NCRZtRcijT/fY1FRADNZh9GwVshHkQfXeVFNixV19GDMdFJbzTyIsSdMziDcmLnFRClQxcxl8TlKVjxbyPzNLkecuIQO1A32/D77yYgc2BtGdQSWlf7yo0jGWlRn+Ffxc+fOeHNU0ndFd69I6+M6lbjg== X-Forefront-PRVS: 0107098B6C X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtWSTFQUjA1MDFNQjIzNjY7MjM6MU5KaXhVbFd2T1FVbzlMR29uZk1aWlIr?= =?utf-8?B?OGtqYTRKZlhhVUFJaFBVWjkrTEkzak1RVCtnKzNLRHVBbE1FZ2gvZGI0c3l2?= =?utf-8?B?ckxNZ2ZhU2NHcFVZYTNnWGtGUmt2UVVFTjIwZVlYNWlwbmVKMHR0U283NGtN?= =?utf-8?B?OFAvNUltUCs2L3dqVitvK3h2VkdKZjBLWTJQelZEQmVBNWxWU3A1cncwYVhI?= =?utf-8?B?VFByTnNrWmtwVWlyM2FXY1pYNFdhQkowZWtkeHdVNzhVV0ZMQzBBSlZWeDRL?= =?utf-8?B?b0ZYMSt6MUdWa2g1Mmx6eHZpU010REtHdC9OeWFETi96dEFwcU5Ed2E3T0Fk?= =?utf-8?B?QnRURFZHZDlnR3NqOGhkZ1NQOC9MbXZZUUgwWkRSNHJWZXNFeVpWNXhaS3lj?= =?utf-8?B?NVRVRzdHb1k1bThlbFhmeUZmRlJHZHhMNnRlWmoyc2VSQWhXMEZZSDNscGcy?= =?utf-8?B?Vld2cmhtNXF2bW0vak9ISHZrMDJXbXVjUkwrRnlVbTJVRzYyY0x4bG9Xd2F1?= =?utf-8?B?M1lMSDBXbkMrdVB5aUdQcENCQy9pUmtJaHBEVFVUSXlMQ2pnaWcyNHcvVi9E?= =?utf-8?B?UHpPU3dGUmx6SDVhL254QjhZTlFLTFVKUnZBNmtESlZLVmZGVGVHY0QxNlVD?= =?utf-8?B?KzJOS2xyVVFOdERpWXBxN1l4ajNKN3JBaVhCeDhXOVJzaUxYSktUTWJIYTN5?= =?utf-8?B?RENGbnorQzlBeHl0Q0JLdkFMMWxxSWh0MnN4dmlIZ3ZQYVV2ejc0ZEk5amMw?= =?utf-8?B?aEpHS1NoN1Y1U01VcEM5K0IwZEgzVmRvOXdxUUNkWW9LK0lWNVZHc01ValVD?= =?utf-8?B?NTBJeWFBYitlUkdTMVdTUUhtem13TDV2MDdkREpBRDhDTkFaZUZaeWJpdTJt?= =?utf-8?B?b09nYkorZXpGVXR1dlpXc3A3em0velNtdkFJRUJoU0JmZFc5d2pmZTNZbXEy?= =?utf-8?B?UEw5d1J1andEUTFRMWlWblZrM3c2SmFud2xBWWU0MnNIVzE1S1RHdWltWFY4?= =?utf-8?B?TFBkUlpNUEYxdUVQc0s4VXY5bzNDdGRVNDcvbHJKcm1ZOWtJdko1dGVvUW44?= =?utf-8?B?bDkwakFORFVqRFM5aEdjNGN4M09qWFQyY1dYUHRvaUNZNXluQ25ZaCtzOTBD?= =?utf-8?B?RlVPd3RwN0tmUFVmaUNsV0VMRG10OWc0b1lwZkZ1eHh2Wk8yaFkyNi9JdmF4?= =?utf-8?B?NVNxQm1pVXgzYStsTWJkQVQ1UVpoNUs0VGIwQjF0djdVODZyd3lhYWVFaXFG?= =?utf-8?B?VmhwSjdGa25wVkR2VGlFcDQ5WmlkWUFSOWIzL1VwR2ZCY0FyN2U0YVl4U0t5?= =?utf-8?B?Z1RnK05ndW9tN3MrcG5DMjJ2UWk2eWNvYnpHYm9wYk5yNWxNYlFrWVlZV1Y5?= =?utf-8?B?dHgwTE1QNlAvbFYvbk16UWFjb2NkU0FiUi91blUwWnE0UksrN0lCR2llc0JM?= =?utf-8?B?blp0N0EvUDNHUFpiaXhsNGcvSGtjcFNtYjBUbTllT2JIenlkQ1l5TUtzS3Vm?= =?utf-8?B?em5yNW8wVWpUZGtURk9PUS85NUpJYXNPQWtlS1NTSVU5Q0dUT3FnSkFWRE9B?= =?utf-8?B?aHBndWtLR05Oc3hNSGtSZzJqUEhHMVJHak9Dd3g4Zmk4dHF5NHJFWHREYWpD?= =?utf-8?B?YS85ZkY4RE9JQTdLMVZRTjRIVm5XU1A3NGRqTkQ3SldmSjUxUTRPYlVjNmdt?= =?utf-8?B?OU0yc3NQc1grZWJKbUZJb1RwYlNMbTdvRXYyTUFZZHJXa2xCUEp4b2V2WFlu?= =?utf-8?B?VVpLbkJmL2wyWHhoNHNuU0tkTklYT08zRkp5a1docnpJRzJPRExSVkJkZk5Z?= =?utf-8?B?dUE1M0hQcHI5c3BwVWtXSmtiS0FObHRPeDdHa3lnY1gzcjQyemhjZnEzc2xh?= =?utf-8?Q?P3WpTjggdKAY2Upp0necLTmT+t4hroBjf8?= X-Microsoft-Exchange-Diagnostics: 1;VI1PR0501MB2366;6:2UvVQIkuB+htYg34OIpPzetsQrhZADHkHcYopkA9rdik7qsNbbQZ8PhqlgkCiejhpGfJ3kIVQyLYwIlQSU57JQNiR2ulrC3nr7NkKY/tBTypKwISxZqHeDmfa83hQZ/v6axLsnZJ2uJmmq0veNKpUPZYZw+OurIFAypbN/YJf90DA/xtvEAKhK/bFvZ+LcYs82o3W3vN1ws2N8DQHJUw40m5Z7wJ9sBQZ63BKb9jrMtBj+UPFaxsYZwU8/mFYo11HGhdjQUlo1iXV8RaLuyA0yi8tarwIDlWTH3e8mL/pJrFuQhFTRuQGVohF91AfuMT0piVUIfNFMGFzeJ1Z/jiQA==;5:mMBk5oEh1acPPSS/tgBoOfGGX+fQGGWBxUN61e12o3aa2UGSWo9U4iI3M4w9yF7qJE5qOAzEHBISLkEjD3KsDARgMu38pOjlDF36RV2sNU7z1gyS3pMEoodDq+CjSW/gtpov11R4HmRfOvyceMPBafgb3Sg85BeqWE2VMTd/Bk0=;24:93rVyVU4QAzaDO36ndYbl88DuHeYILMgHgNi708vMu7YJ2i6um+qj5ieECwI1EQdJfZ7itZ4YXSncieJZr117T3jIf/VStyMDCLeCb7nX4E= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;VI1PR0501MB2366;7:aiJZkYaXXU7cQu+aeYCALcTgexVo6qRB3UL1NhdtHBBTJWGvWs/ZjEuNGkEnMFNIbfKmN25ILJMivvUu8qp9xnUqaD95Z/ysCbHrNUKfEqDTk/KvtoBc5qi+5HEpRJ2hgSogOWsfdXbphofJyhMd2vcRsLfxwSYwiz0WPuBSGEnwUwYiEMoy6RnCZEg5LttQys6icofLbTdLuPGF0xlvPIAdYgQgPFQ9+LXI6uTdoub6BC8/JWhSY2IXySnO+X7FFyNpNkRAAI3HD3jyFoutHSpCxPGLcKfOb3pVvfX6egeaTVHO2UEGtHiorEMUUhu5TlII4+jqvuH9dcUelf1nR4pfKkXZEAzRWFf7tViGQPA= X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Oct 2016 08:25:04.7014 (UTC) X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=a652971c-7d2e-4d9b-a6a4-d149256f461b;Ip=[193.47.165.134];Helo=[mtlcas13.mtl.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0501MB2366 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1402 Lines: 29 On 10/19/2016 6:51 AM, Dan Williams wrote: > On Tue, Oct 18, 2016 at 2:42 PM, Stephen Bates wrote: >> 1. Address Translation. Suggestions have been made that in certain >> architectures and topologies the dma_addr_t passed to the DMA master >> in a peer-2-peer transfer will not correctly route to the IO memory >> intended. However in our testing to date we have not seen this to be >> an issue, even in systems with IOMMUs and PCIe switches. It is our >> understanding that an IOMMU only maps system memory and would not >> interfere with device memory regions. I'm not sure that's the case. I think it works because with ZONE_DEVICE, the iommu driver will simply treat a dma_map_page call as any other PFN, and create a mapping as it does for any memory page. >> (It certainly has no opportunity >> to do so if the transfer gets routed through a switch). It can still go through the IOMMU if you enable ACS upstream forwarding. > There may still be platforms where peer-to-peer cycles are routed up > through the root bridge and then back down to target device, but we > can address that when / if it happens. I agree. > I wonder if we could (ab)use a > software-defined 'pasid' as the requester id for a peer-to-peer > mapping that needs address translation. Why would you need that? Isn't it enough to map the peer-to-peer addresses correctly in the iommu driver? Haggai