Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757378AbcKXRAf (ORCPT ); Thu, 24 Nov 2016 12:00:35 -0500 Received: from mail-dm3nam03on0059.outbound.protection.outlook.com ([104.47.41.59]:31360 "EHLO NAM03-DM3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1756664AbcKXRAZ (ORCPT ); Thu, 24 Nov 2016 12:00:25 -0500 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Serguei.Sagalovitch@amd.com; Subject: Re: Enabling peer to peer device transactions for PCIe devices To: Jason Gunthorpe , =?UTF-8?Q?Christian_K=c3=b6nig?= References: <20161123190515.GA12146@obsidianresearch.com> <7bc38037-b6ab-943f-59db-6280e16901ab@amd.com> <20161123193228.GC12146@obsidianresearch.com> <20161123203332.GA15062@obsidianresearch.com> <20161123215510.GA16311@obsidianresearch.com> <20161123232503.GA13965@obsidianresearch.com> <20161124162620.GC20818@obsidianresearch.com> CC: Dan Williams , Logan Gunthorpe , "Deucher, Alexander" , "linux-nvdimm@lists.01.org" , "linux-rdma@vger.kernel.org" , "linux-pci@vger.kernel.org" , "Kuehling, Felix" , "Bridgman, John" , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , "Sander, Ben" , "Suthikulpanit, Suravee" , "Blinzer, Paul" , "Linux-media@vger.kernel.org" , Haggai Eran From: Serguei Sagalovitch Message-ID: <87e6cfce-c65d-2d9f-a3c6-e4224cb94b2c@amd.com> Date: Thu, 24 Nov 2016 12:00:14 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <20161124162620.GC20818@obsidianresearch.com> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 8bit X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: SN1PR05CA0004.namprd05.prod.outlook.com (10.163.68.142) To CY1PR12MB0697.namprd12.prod.outlook.com (10.163.238.158) X-MS-Office365-Filtering-Correlation-Id: d86b25a9-49ea-420d-6f5d-08d4148b6010 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:CY1PR12MB0697; X-Microsoft-Exchange-Diagnostics: 1;CY1PR12MB0697;3:KJUJL/KWpoyvaM4e9njzGdAXCTDJHY09VR8tUZ4QKWrXBJ+5imtdSWBHCtfH2mf5qlDM4gWiPVT9EOtStCCBrPep2wmIZlWHcTTfmkkNCaWrWjidwLMcqZzIl8bITTQ3EcPJWRIjcA0IGfme+wKyxAr8U5N9v+GnXREJ9QIbeAr4XWwyPgblHc4JKoFVe7XH2OMh9ORhmLikb4Kis49EmpsQ0gZONJTpEh/lBbmWTMqRSHvZ+o5nFX9jZSSPomVuEWW7RsuU6mVj5cXP250Ulg== X-Microsoft-Exchange-Diagnostics: 1;CY1PR12MB0697;25:jcuxOSwNXAEHTMve6FBiu+C3eFoUEAxXtOxV30dOJrla2qvD4Is8k1WYz60tvFdsJfqX4rXruLw8kKCZZosdx3NJ7BWI3mkGfzuxKNm9daiFrOnUdfh+Dqb9a8VBHX8dCC5XXOhdUgEBgfAYnRF5aOc8Wf/rCaTmhIdVOqLmcGDXUxKMmtJ1ZcHNha2A+easCnX/OT/iqTwW6DCD5T1yF2+vOzBQYmopspTRSJOhPCNUVvgP7L7kvIx9ZkCZQiDc3SsSTDdQ5fGiUmtlnmv+x73evf3elUkJM4qN26Jm5bgAQbnHf6t0FecQlDdqV1OZbJLbvrNL2e1Cj4bJ8uIR/ayfLtHgiJYBqq7oh5D5Yld8Er8hF5cz3FJvdSPTCJ2BT59PkTkdhGrEhx46WVTscelHrv60tAcTAmVPVE0YLeOjO7Wb0ljm4iVbiECTFla6rFYhZIyJtIXThE+PG+LNDc3JNChbzELkfDOiM4d0sk20L96N2Ob3bFn2z6o0gT11v7CDOXLRo6nMuD2A+iuNgoMV6MTUed8K19ITxoEcyUkEHnd1JLPQMvBT2fHVKpIoSzXptatpsta0lsl2EmckvE4a24JI+E9r13xpUlWproa/3/x7ULVv2hwi4FTtDB3NaBYaGGWPmy5rAKyZfvdqVu8d+aEirb0zVP6g3A2n5fqySxacSvVpFDqTo7MkQNNCGk7tog5TbpahOQvt24Sx4Mc5TfLvY96uY5TwCdkVwE/k/cgUpVHPERNTZVejk6Knu0iSR+H5ooQmzVYMa0AiFw== X-Microsoft-Exchange-Diagnostics: 1;CY1PR12MB0697;31:Cz7Ec07wJPfqQQr8QniQBLQda1rN4pkkC+hK3bu8tOH5/T5fFPHOOms+bmq00Q8YvUiA8CWwlUVQhmfg0nIHjqMHvMlxu27UogdWFjOd6XsGqSvsmibdlJQCwWAVgLCHFNkugXYo/Fyq/P+ZdxQ6E+QEKps5rM2cBZTYU5nr9o+7v1ZGxN2YZpbREy/TlCbrHXWte8xgVhJ+ncb04ufuSAYNZn+t5KX5EuH7wH9D6KGZHpcmYjZwHwYQv/ZXBorb5GrUE3XGkfBk4WLm6Fcb6w==;20:Iier86Dl70f6ZV4pksYXRWItHWPYqnNQRH8dXwkH7n5HvzqYGirFPBE1+c1IswJO+honBfDixY2H9w+3SbYttkniW1LlXmzP1YXF/csm/EYvqJ4r/R45yYzCo/LEoNtzOK2dwFkQCOjXrV0DSefSReWFmc7arCBoa5JbPO1rl7F+cMaP5e/wjG56wh5XtxCLyoW+mPmbQgvzpOu789vrFgtWEp9N4j1PS/DactVFiZiSDeUCG7C89tCGq0aYVo8G27/0G+AXH0P2hhA9sM4Rv7Ejvo2EQ+2qvpuU5xTv9FlQ4kwIMxbGZRbZaEBmhUS0R156UKp+kL6WKzmSSYIYsv8oxFMNK50c6Oo6+n/mQ78YszjRdjnZ0N4UhZTLj/NOld8k13xTVn1M9m+9CPsVXJJbvAsQEPPDn3DYnQ5dPwEgoat/FD0CCQLfH11r8CLqbPqHP063gUaqryvjQ/AxRb3p+wCSOCClaI3l9/aeAJfHtYxlK23v3YLModzEP2xI X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6060326)(6045199)(6040361)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001)(6055026)(6061324)(6041248)(20161123555025)(20161123564025)(20161123560025)(20161123562025);SRVR:CY1PR12MB0697;BCL:0;PCL:0;RULEID:;SRVR:CY1PR12MB0697; X-Microsoft-Exchange-Diagnostics: 1;CY1PR12MB0697;4:+SpUAZaoWBGH/UNz2UwlkNLJRYQkYJavZWXrsWSFbIgSTtodrwGQ6nDSpIx8s6LMCq5eU4t8AJWoOpR+7yxlPzNOw5XgJ4uZbBTfp6sQ3kKIIaJr7DKrEBt2MTfCy/bYVfGtGdAXsGBejDuepG56bb1vFxU8cu9aSmpg7SAE7ZzImEYSZHcKJhk2DzoCYKN+GxSYEb4EMTi0oBGQZE3pBJtPAHhgY4sbmTvu3DG8RK26aeizmPqLcsfAWb13+vU2BcWOCbMnedHApEgnAK3heTYxKJ0S2jMstYq4rtDjf1V4XsPMdFVkCKM8ngagEEERQiygCvTDX24wxt+xrlDE3V/ywRgjYoEHVHCxKBJxcj14A6H/GpKFMxAHwq2F0ANY7zaqHGFwKeipM7g+8/bDNhblPW9VJLnqy42SvrkVJpKDfOnXwYaGokIhlyPqk9Zc94/xcuuBaaITmrm1PWHf5occEHvSL0Zv2YVa5VJj5FODktcwqvpMMWquJWmyZ5+vQCebDtShugzztytGNYia3oFse6OwQX7taUF3R/QKxzjMzzs+sC6IqA6i/dEJr5Brrq8sqYvzKdiaMHJbwD458Slb2VshR34GiL/GDMtCQBwRRnDtXWB4aFYuZokTdFYI X-Forefront-PRVS: 0136C1DDA4 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6009001)(6049001)(7916002)(199003)(377424004)(24454002)(377454003)(189002)(33646002)(39380400001)(39410400001)(77096005)(6636002)(36756003)(50986999)(93886004)(101416001)(50466002)(54356999)(76176999)(65826007)(7846002)(7736002)(305945005)(7416002)(31686004)(42186005)(92566002)(2950100002)(64126003)(6666003)(105586002)(106356001)(5660300001)(23746002)(2906002)(189998001)(2870700001)(68736007)(4326007)(5001770100001)(81156014)(38730400001)(8676002)(66066001)(65806001)(229853002)(83506001)(4001150100001)(65956001)(47776003)(3846002)(6116002)(81166006)(97736004)(31696002)(86362001)(4001350100001)(39400400001);DIR:OUT;SFP:1101;SCL:1;SRVR:CY1PR12MB0697;H:[172.27.224.67];FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;CY1PR12MB0697;23:zyK1nVuATkBFue5AslmwmyKOxVFqNMvZNwWhX?= =?Windows-1252?Q?2hqysPFu7+JLe6mUyFt8aGCTTGSfWaqAZ9RJ36KoP8q3hj/lcZDRB8Le?= =?Windows-1252?Q?NPGZO9S2duanCND1i0CXM2HFlXaiZwsImLQgIaoBjYugS/u2xR5VbSOG?= =?Windows-1252?Q?ESHhQSC5UiUXWtthgTbwxJ3F/FrRzig+BNZC+l/1R/YTaAX7lS3NKCb9?= =?Windows-1252?Q?VEMkfwtp51IjQwN2NWfgB0Zq2CZ7Sk+r/6ZyTSzVZmP598s0AcbFGRls?= =?Windows-1252?Q?TZJM9wugv6G2ReHgOjuONlsJS3CLJxH4JiF9ORdnb5fXJDJy5x8Igh+Q?= =?Windows-1252?Q?YVL+MzHeFGTCC/bTZvBy+vviJFP/WpX7aIfzMsW8WGD7uRWPJQv6mP+k?= =?Windows-1252?Q?2RoGuJkwY6+gFfBraoqbFxKcPYzlleWI9fCjml6RKOw8q6u7FpD0yIkl?= =?Windows-1252?Q?OyPZpeOxsJGYDf7hSfILLT83aZMd2vdTsV7en6i7G2oXvzfpYgIVh6RZ?= =?Windows-1252?Q?N/LvdnzjqW27wNAKxie4ioOgbAKVleL7Os15Y8cYU9ZuIqC08uotRSWF?= =?Windows-1252?Q?zfQABXMqA3ACOl7uNq9U+CM6L8tqoAREN1er0g9YQ5AhlfMwrqFx3fZL?= =?Windows-1252?Q?UBqldcDN8ta+jzJeJkjXBqYoTM0Ll8ys85xTG71GFZncntqWCou9zr/A?= =?Windows-1252?Q?TKLE6g33AHfBjiOlOQM3RJaefmjn2+5ouoAH8+gmNtOAg5Lk8jr0gYrz?= =?Windows-1252?Q?RFq+yH5WZFaHUrwuskjqUPiTsXgW6fkMurqGPfQ6txOy27LaZMD5ueRI?= =?Windows-1252?Q?1DCcW5+2V/N4eCpiQd9xhHG1Zfu4QQsOFd5a9gI/rPzRngEC5llIHkiT?= =?Windows-1252?Q?4UxxDFhjYU6+uWg3F0ZSdDg0N+5wvZPod1Tl+O08LnGcBkqCmZjQGPtt?= =?Windows-1252?Q?fTN842gKsmMeaocuJf/LdCQbSQ8OYlb9SUTngUs+7tVX3PwWzseGMRtl?= =?Windows-1252?Q?B68KrcPznYNROOZbD2MZZfT40JvE/QnSl8f4lVyxuLMSBJ1OwE7HeL6p?= =?Windows-1252?Q?rgk3tTYgIPMvykgaeJiBmFspnmiPLr1nh9s6QDTk6/P8y/oudktxHU8h?= =?Windows-1252?Q?3r7p1zHbUFgsb75ezDqOnT7wY26e4XYKJVCu3zF5pjI8bA8PXacSkqTt?= =?Windows-1252?Q?MIzElYkEocUA8/MvPIsLcL7dnE1M9hcpLUgP3Lo/oMkI6qxmRZMWM1fY?= =?Windows-1252?Q?6atV0+hfSJhJ44JIODWeVfENc2F6n8DQ0pyFhZCdEotbzgsq7h2bXBQ+?= =?Windows-1252?Q?V2QKIa6ZP9qxtUJwV3F3FkvwoovS7ouqhJOtf/5Sr40965wepF83VT0O?= =?Windows-1252?Q?kncPg0WSN3k1IkgYNQyxPUclI2HCM34U2HymatyGShUBxQaQQvq1yCaY?= =?Windows-1252?Q?ks8d7RNu3Ux3zvZK7xEvgISVgw1EKvs7lbBTRWAe0mBTHgahxRaaDOXg?= =?Windows-1252?Q?1wfwdYi2qdIdhcrWwiaga7yiQtvCLD4BYepbvs/Xm7HuSBJ/52HT3apb?= =?Windows-1252?Q?RypWjoA2eN+pHu/jXrF+8wGCU/IV7+szdOs?= X-Microsoft-Exchange-Diagnostics: 1;CY1PR12MB0697;6:pqpkU07KO3mo7533XCOz8qBYR8WgNSj2w9OByyKpnTSrcvNndRjRFJkH3KW+Mpybjq6T1alWMKUrQQZ50aPER0Kd6hA8zwQlS5Gb2z5v5I30sI5DkZEDjSO5DPJRr/BOX0DxDCft7Ht9yGrfukgdmZxmsM7uzo0BmG7w8uZnuVdEksOrP8CYai7uGgJW4lldjMtuowS9pPltKG2Jjp+lODwz3tkEx2MzR49JK82isQs/nJnit5cc7oE1csq/m96IOIdBwn0s6/dOqKfjcugiI2ALXe3xZJoEjtKTUQr9Zkc7wuHtx15Xy0wpd38UEAduPEEQvrEMofZ05KS/jA5kPsYwEXKjUa1IxdocuMuGtwFxDoPa5srN7QSUIoUWRk7hdFO3PQLkoBsLhUIPoC3cuPSK1M75Lo2/jHIoirQYw1NGEnA+aZ2mUPb/3yITD15Et0Awt3lJdlX8dztzhs5oUl7LptKxWa67Ohr3uS5YsnQ=;5:/wgH2q1N+HtwasGJ5bDqKp2pA7ZYF9EphraNA1F3WHxe1G9sZlGa7h9iInzE6QECvErYGsRQER7xCIAo4MlIa4wyb932jPwQcTOFNdiclB4GSZGz4A9k1CXS/LJUQ1BqfvtEa3MBD4Iz2ll11xpPi3yKldwZqbf7//hH4VcRtb0=;24:c0S3DljMG5eeX/BIiQq2FnhT+WH7XAQfmp/dldu5qUioJGSHLqIqNifmZe6u6gGJLLtrq4mlkFE+ORJfH+ckLUMWRaMR8dMilHHNvw1NGmc= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;CY1PR12MB0697;7:Z7uM+MMPzb/WV9IyZkIbpjPjP0+NfY8h1ggcCCPCao4mdbAepRTGQ0/0xGOWKgfjEozeIIWd77fOn+XEgtzZQtnAXKSIxaaoxs1B5opK1g03hgDLzp0H4apZ5ncb0+CtY45v0Gzav7tABVZ/J98MkZnQKBuny/iRE3z66kqzm9peOqI/wjKqmIAyqRnsbOvS1DO3vXUr5FK9pV3ZKg7DzoIhAOw5uFOrhD0aKsa7KaXEHQ6ZIszp7iIC3Wi4dkdJSfoQFp637/EUNqWQVeTxl1afMgUH0L/NPAZM6aG4h2vR/psKLjOQR/jzBpVcgie1H0ehd2Ek5gpHkILJ20aNQoWyqIO3aYALIa7IZvECtnG+Lu75g6YSCk1uqJGPaiD2sZHJj6HCqP093GXagoed6smrG38PcfdvVyuYG4kjNsIg8S8lcApb+FzGxRtrrMJqa1LH4B0NnNQdWSxlkADbpA==;20:Xa1ANTZ9qC11Y5CM+QBLNst1TYkXr0QqNYTQDoInN7FrkCOO++hHOBU5JqnSrxjzUcB07DYI9yzYNg4zcuBWG0RTFt1BYdlVcpgWnvTChigGQ2NlM05pI47hPCmNS9oS+IChi5MiP011U5X0FlHYWZiEFvNMMXRkxqP4eEVM6KjbxiXq+S7YdrDbWGNBTX68nxOnt6/tem0mUFzIkmGauU+nrcIkRjoGq1Z05/znPeHtGndYFJZF4uSKjGtYiVeI X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Nov 2016 17:00:18.5451 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR12MB0697 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1251 Lines: 27 On 2016-11-24 11:26 AM, Jason Gunthorpe wrote: > On Thu, Nov 24, 2016 at 10:45:18AM +0100, Christian K?nig wrote: >> Am 24.11.2016 um 00:25 schrieb Jason Gunthorpe: >>> There is certainly nothing about the hardware that cares >>> about ZONE_DEVICE vs System memory. >> Well that is clearly not so simple. When your ZONE_DEVICE pages describe a >> PCI BAR and another PCI device initiates a DMA to this address the DMA >> subsystem must be able to check if the interconnection really works. > I said the hardware doesn't care.. You are right, we still have an > outstanding problem in Linux of how to generically DMA map a P2P > address - which is a different issue from getting the P2P address from > a __user pointer... > > Jason I agreed but the problem is that one issue immediately introduce another one to solve and so on (if we do not want to cut corners). I would think that a lot of them interconnected because the way how one problem could be solved may impact solution for another. btw: about "DMA map a p2p address": Right now to enable p2p between devices it is required/recommended to disable iommu support (e.g. intel iommu driver has special logic for graphics and comment "Reserve all PCI MMIO to avoid peer-to-peer access").