Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1161074AbcK3R3g (ORCPT ); Wed, 30 Nov 2016 12:29:36 -0500 Received: from mail-by2nam01on0069.outbound.protection.outlook.com ([104.47.34.69]:12384 "EHLO NAM01-BY2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1758464AbcK3R3C (ORCPT ); Wed, 30 Nov 2016 12:29:02 -0500 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Serguei.Sagalovitch@amd.com; Subject: Re: Enabling peer to peer device transactions for PCIe devices To: Jason Gunthorpe , Haggai Eran References: <20161123215510.GA16311@obsidianresearch.com> <91d28749-bc64-622f-56a1-26c00e6b462a@deltatee.com> <20161124164249.GD20818@obsidianresearch.com> <3f2d2db3-fb75-2422-2a18-a8497fd5d70e@amd.com> <20161125193252.GC16504@obsidianresearch.com> <20161128165751.GB28381@obsidianresearch.com> <1480357179.19407.13.camel@mellanox.com> <20161128190244.GA21975@obsidianresearch.com> <20161130162353.GA24639@obsidianresearch.com> CC: "linux-kernel@vger.kernel.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@ml01.01.org" , "christian.koenig@amd.com" , "Suravee.Suthikulpanit@amd.com" , "John.Bridgman@amd.com" , "Alexander.Deucher@amd.com" , "Linux-media@vger.kernel.org" , "dan.j.williams@intel.com" , "logang@deltatee.com" , "dri-devel@lists.freedesktop.org" , Max Gurtovoy , "linux-pci@vger.kernel.org" , "Paul.Blinzer@amd.com" , "Felix.Kuehling@amd.com" , "ben.sander@amd.com" From: Serguei Sagalovitch Message-ID: <2560aab2-426c-6e58-cb4f-77ec76e0c941@amd.com> Date: Wed, 30 Nov 2016 12:28:24 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <20161130162353.GA24639@obsidianresearch.com> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: DM5PR03CA0004.namprd03.prod.outlook.com (10.175.104.14) To BLUPR12MB0690.namprd12.prod.outlook.com (10.163.218.140) X-MS-Office365-Filtering-Correlation-Id: d982d43d-9736-4ab0-6f2b-08d419464e4d X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:BLUPR12MB0690; X-Microsoft-Exchange-Diagnostics: 1;BLUPR12MB0690;3:etjEucnXG7N+AY5qQ/DwdHGOgZHTx/RfVFN5b8H7++viRgLmrEGwk0pnDFp51bp3x8X1Nm5WhW3ckGktUz5pH9WAIBF+v7kKTtE4cNppx9jytBDZZiQJ1+coat0CPnOVD9aiFk/2B3b43zBpHJooLQlGnnVz6t7LsNgiYwUtOn6cCeAAqo27AZKethxQj1eIXk4PhSjt8kkSV1YY1MGJYzPgGFCT/9A7EPq7rnOpQV0F22DDLx+hyb2bKeC7nKjtfJr+ZLzjXkMwrp0s2qJTGA== X-Microsoft-Exchange-Diagnostics: 1;BLUPR12MB0690;25:jUAhpbmmHgMLcC8qtJqmuBIkIiU6HkZs4HjQOfXfdBNAAhC+tcJsK+ib9yAAu5YTWpDt15/KPZq5h5O5ck25wVW4oyRFPoClTCIbFBP+VuMwl3/pamP8nQ2CAuLannSJMVghBRLYVL/tEe8AyNUHkJaxOw+MOdKAM3WeotWl8RBaOHcuCs2rN4ixZ8jCOeBLygocUDpcOvY97IaeHOl3fqjv3wO1jgeEXlH1JU3P/laDJcYH+BNTuEQ90MdU5jSTNuKhXCZzJ2VELPQ9lnLe4z/9YYhZnbbajTzichDrubMM2MLGThGoy/nosQGQER/zDx4ARHXlem9xwL/2yEx6CddWfvI6CMSB7nRHKVkrGmyVLjHJgz/o5PNvNhbvSVTERUQnPvztavQBuEmmOVwf28LD5TmHh0qroSfDkrsHxvhTJl+4Bbbi2fk4yJlCkQR2HpZSqLfAiZHap+HkYGpIGBFrEPGB95XC5XV85YaBJm+b7N1gxrtIGVX2kX+YjaZG1JWiAOPBEnVukmi0+MjSP4Rl1WqhJUvbW1i8lhaW5uJDN4dCZKAWlrk18/D7ayHQimZ5qbinBdN8dMTRXghTtKN+ZyQgPFKcmvMPvTs0tNiL5njyRUBq8rKLLMgdg4eeD3KcQyaqljrZGnU0bUlj7MYkAGb+Pr2hwUWW1UScxACjdSRc+BrHWNQyxUd7p7jReb+mrntWI3v/5rk64XZsdsclwnR3C7YoEspxxDe54H98xACYf8cdLV22FTLXKmkFP8N6CMsDO2ZnQ9GW9DG64SsS5BE7jAWGLxLmLu9F97g= X-Microsoft-Exchange-Diagnostics: 1;BLUPR12MB0690;31:ZQCjfWFgGkGX1n57+1PG3IwpWCIKzr0rcEOCNevjDtF1ttluxvh2WaEhg7V6ii1WS28wgJDZb4QF/U4J5H03v4kjvocbqS0YQmb/qYWxOAQ8MdECGrlUaAzURIeDf0PaIrE8Xb6+pQIZyHd6XXGs75HnZHIKsf114fLE9Rb1OYb1CUiDEnJPO0hkr61mb/uNTudry/i0jMT2ySyPsVW/v+LRWvW26kSQmIxYrbYatiySIyEHURx9YG5QdjhxzZ74IuweAgiXRpDK+6NwLkKsoqjOTEHzbD/YYV7gWK9z8IQ=;20:jltuJbpMXHAi8ZJLnCkILWQiYZM5HYlPyVnU6/ogTlr5O6a7knEvAS4fC+YSxWbnG/Y7Vk5AO2YKj/NVkjo5Q63q9ERCFvrAPBasrkdvqpNckgrOONJfBIcNaXzUPIi7rQV/kmitIttOgyJET7ERZgGqt1T6sfuvwd+DnPCTXl9ABQyardb/QqvY4HcU91E7RqZ4dfoo4XOQ5AjKRQhDNrigQLbROxNFwzMcQjuCfcd2RcsS/iJ7WLhTQbSGFnVVmZHz2lnIbaCIrSSF40IraCyusceashACO+Qv0QUggvNVUkCwW+nextQdss9jKAYrS38w+jbqg1X3PF21xHsBS2emjjKq2G+XqXbj5htQO1Zauc/TNw6UgGP+FkW5LF2PjyLCog74Xd7qrIH3ND1Znft5K6Xm2wGQi2h3EpBhyjsY8WSdxQVg2NGZka5vz4q5uE7REK15GT5SW+j3CxzoqnhPfc23rgdE2t70LNk9q5F3SS4Sh8fVmu1pKYnKPALI X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040375)(601004)(2401047)(5005006)(8121501046)(3002001)(10201501046)(6055026)(6041248)(20161123564025)(20161123562025)(20161123555025)(20161123560025)(6072148);SRVR:BLUPR12MB0690;BCL:0;PCL:0;RULEID:;SRVR:BLUPR12MB0690; X-Microsoft-Exchange-Diagnostics: 1;BLUPR12MB0690;4:u+ih358+bNFOuhk/9dVvUua7FPWVZ7RxS6JkF/I/21abBslFD3f9CakrGohYScYFC/L35VeG2TgZTlGor3dgbRfw+uetai9pM0cN/W63aZIs7wjGFHwePubI6cb+TggLnyGtIUo4hb6fKTV6ypSQaChJhpjIxC875OgoCKdc07JKSQeXa5mF/Htil6yQxscHNLpexgQAAnYM99uptJFQCHJ9XOOAnjrrrm4g85zL5QwoV+7urX1KbNPMw21g2INy2sxP5akM1FrZQ8PieKUr7vtiSaIF9J3kE78Bcn67yffY1NU+LKZVhUjEdeRJip/GKIW08ZLHEGSM6iKxz2XBfj2STDnj+WiAW2yZXWj4NCnuZ+Ja7Lifu1d+ySbFkjssbZCvnLM0unSn+FKKv6ed/MtjVOIGNJx0LRBd8p0hrNAptWU1BVl//iQ4agMmI3bT1bP1M3l1fpdAe9UQqb36FgmxjpJiVlTYz9fBGB38DrvsG6L7UveC4Cn5DtxaQiGdBYow7yhl09I6MyfY/gXHueHm9eDg6wuIoP2yHuT4PpXAkW9i1jN6PsF4hJwqNzdGLSld3E4GdF9mjaIMSaCiBA== X-Forefront-PRVS: 0142F22657 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6049001)(6009001)(7916002)(377454003)(24454002)(199003)(189002)(377424004)(57704003)(39450400002)(5001770100001)(39410400001)(33646002)(93886004)(305945005)(189998001)(7736002)(7846002)(101416001)(229853002)(7416002)(2950100002)(6666003)(38730400001)(54356999)(50986999)(76176999)(8676002)(81166006)(5660300001)(47776003)(81156014)(65806001)(65956001)(66066001)(65826007)(68736007)(64126003)(106356001)(105586002)(50466002)(31686004)(4001150100001)(42186005)(36756003)(97736004)(4001350100001)(92566002)(23746002)(3846002)(2906002)(4326007)(6486002)(83506001)(6116002)(230700001)(86362001)(77096006)(31696002)(733004);DIR:OUT;SFP:1101;SCL:1;SRVR:BLUPR12MB0690;H:[172.27.224.67];FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;BLUPR12MB0690;23:TzcktJK6BmOjrxgQMXEFIdZGUO5+dQXyoTYzM?= =?Windows-1252?Q?urYmcriowG2YfAabwiEaZq4woI1J8omFEyfwfNEPkEZ0w5IioXbLGBSE?= =?Windows-1252?Q?LwohMjjKY7zveikQbELEA3UEU5fmxdAJKKbJMmWM5AwAojQJalNtzxS9?= =?Windows-1252?Q?4N2vlrF4Kxgielj9+SU40Hi7zz7bOK6iXrTlD/f7bK6BJ8Nsi81erwcY?= =?Windows-1252?Q?t01Xu+IiFMlZlwT8oMFMJ8K+cQb6809DlTFHeBk0SfSkj4XHJNFqG/Sr?= =?Windows-1252?Q?s5q7Hxh2tzDpEs7ISwJRdpxNeGvjYIosRE+ebq/58Ie3roUkdH1dPc6l?= =?Windows-1252?Q?5/zjAQaCC2RRmCXReX9xHg4+vM/zDEm/NRxV7h3z3w28sF2OUb08kqdb?= =?Windows-1252?Q?feortwbZRLm+tTVtZevz90THDaLQdHPexP5nJK81PfxlOT+/nsJNAARK?= =?Windows-1252?Q?4EjM1dN+xregfxdP1efM3Fgx4/qvEmnmUO5X5KGp7EkNlGwEp2rpQlB8?= =?Windows-1252?Q?7U26Erc4iOhRS3EEVfrZiJw195XSi5QvJpzrs7TxGBJawa1oDNzTkeDn?= =?Windows-1252?Q?NJSJ55pvbhpmMDQKzuVuABfmQ6v/ikBmda7RSgvI4NbfExq1pLl6d+th?= =?Windows-1252?Q?LKZWfiK07DRyFHzv0hhR8KQIAME47yOervvDnbvSazJf+6DjlKWhcU16?= =?Windows-1252?Q?62bK6er7rCZaueIJQ2vD9QpPJzkT8MRn4NGhHPUo6lRhKKOTwMUwYiFu?= =?Windows-1252?Q?9dq7z4ky9X/WgiC/tJj0R3lzW+qSmdf9jnl8/S0sdAG1IMzTb1F1TPP4?= =?Windows-1252?Q?dZJmWWDAqAcGS20eACCHl8uI44O0PaRV2EtCM5cTL6klJzI3VoeOpiNl?= =?Windows-1252?Q?/LVjxmzUyFAcx+vuf/imXQiKEnJeQCxSmmiGcFyYDM2ZTX5clrw4MGEn?= =?Windows-1252?Q?DjDr4Ws4JYgINfdq9Ll0dDet1NR5xc7KjSVC8vSiw0BM+FsQiZEOveL2?= =?Windows-1252?Q?euhPGvIApwMnvab4fbUxVKeFfjSXUPjdUK/TCLNZ0DuM19t8AwllLllA?= =?Windows-1252?Q?E4HBA1eipCUCgZnz/PxxyVyRnOqLq3HfIKC1uAXy+YWswDYWGWvybfhF?= =?Windows-1252?Q?aHbVEieGwYvtGXj+pqzS8tML7ExBa8ZsS9Jdcf5BKvuUCTdkNdCpgeDe?= =?Windows-1252?Q?YSt39dm2J5nITyrgBzZqbFx+OLp2I1wELuQW3Yx52ljHGIqnqaGyeCLM?= =?Windows-1252?Q?Poi7JfG5J/ser2UVOICreZFsRyv4jMOZQqrsEhmCIhMswS24G26w3keU?= =?Windows-1252?Q?Gxfvw2KbIB0TN/xnXeL2OfTIMePJXuz/xvfsyiixuvnNxciOhh6AAnSL?= =?Windows-1252?Q?EYp8N2UyL+v0lH5MhUY4CWBFzcvPsJ3+aD75XVIe1Pm/sN57LOXvn1Gc?= =?Windows-1252?Q?Z2BgjugQfcK4DcjhJvb/QOEh7xq8cP7dOb87ZuW9S18HEyFASpCUXcIM?= =?Windows-1252?Q?aW+T4b9PqoHcEFHnI+yw1sj9mzBi5dh8IHHqkU4Ipr1DycBOzl2SoXgw?= =?Windows-1252?Q?Gj0B31l+qVtsKyEVKERWGYDjm6IagT6l+sQ?= X-Microsoft-Exchange-Diagnostics: 1;BLUPR12MB0690;6:EK35usi455R1v7X/Pc+qDq2zlm14HQ4Au5hJVI98cEweOfB5w7+GmNYrqXyjUMcbNK62OO6MzfiRAMcsv9skDMqvXNk3y+tfyoQMTshqEFPs/FWEWu0hob6IszqV0UeSxD8DIhEx4G4KfGXz5NNk1gvoomd8LdRMmKV/lGeDoDVgaLaM3GoY2nhcMyjIIduVbz1qyYZ3pDjYJtXTR2bWdShGowV9IFxOJc9Qqth1ufRs9Vhf5dhVQQ8mue5C5gV9aaXDOQ1cAq+u3DprAFHQtbwn3WJVdocxjjH0+vfCzsNJDQ92cWLWIV2kKqO04Ltt9vsiQ3HVenW3TUBnusAY1Yi49Wzu34AQU6/RDmEzFkJygPi50zq+lwDtwF07Ic+yOpjaGhYmF+1koL03L15FKCLeOLtwG7YHmxBxWUme2dJSivzRkUJVYaXoeUHaSsF3mJQr8XNY8WuBPv2tD/yf3w3OBDMOdeQprOGBwdozJyUKt0z8TjioxQpWuQAP7HAA;5:kQ6w2pMkw3RAy0xwiqnO46qrUR8aqYzs1xyBMB25J5nP3nMd9PiXgE9qmJCJ2neRj7S5jvrkEeRHqV10l39WpqsDM1DB0NwCdbWuQBLNPWnZhQVijeYIfyIxOv1ISBa8hLK5eoE5KtbxwdhmpG1onw==;24:wTSEpdujHtBiZ6IH1h1dPTXd0SyPmkjYvAtWbsis/S7n/RB+D8k6IyqYANaC2imH1Lgd97QmpBfgF3pIlCHTIqOZXoZR0ZHWm+yRWa1NwqU= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;BLUPR12MB0690;7:wAcGa/fXq87qiVLoywW3fHiTjFDJXddQSeUxDsVEgdHGA3iWbUO0L5vOD5M9KcxRTnBvJa8gGuykArsUUI+BfUBy09heySGOz/Kt3503UaMDVk0uM16/4Q5Xq8GQui3WjSI5/eSHyYY7fUSPv2mm/v8l10c7fv4DlBQaoK2cDZ4aDLtksOeSrnLskkXTxxiQcdgvOP+0+wxIh5v6By9/hfOlB4Zw7Xwy2d9Um+nblq5TjAQTmdv2Nxbm4L1YYYKeOFamkIjMzWqQOf4zFOCxLmegYeFVX75D3zQrqEEGJw2WvKh31fTtlRa95B5LmJyZAyWdkjANrCW2bkCSWPjl/0rTiJ9slZ8ZxN0kB9PU85hRXRIt1N4/eGv+Moet76AmGNCovSyUm2YL05yVLcTWIuHn/RLAxqfHfpS1lnAR0bKam1EMcm6ImbxK/RdNnh1NCoXgVL1cWvZXQjaFHmbJdw==;20:/wIhtpvjEaSykuPhYuwfEjP5D/bH6NnJ5jq5jjAXrKTYTlxk2d9EPKO0YTVNYIcv2sDjS/0hOzGETCt7CrBUtSmWs8u6kcZwLiEoRQHKdf+OXgOWgo1Q7+wFsXzfAamK94XPlIfBnQcOEecLgMXv9vrar9PjqqBCE7/itTUD9CA6SPfdaHICE/2p9D59l5vCjLJBClqbhJcnBXehF+fvLKEo1O9XEtI0vhEaRCwtrxBSWSeRhkGgEwnvz+U9LbHM X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Nov 2016 17:28:28.5475 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLUPR12MB0690 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1433 Lines: 32 On 2016-11-30 11:23 AM, Jason Gunthorpe wrote: >> Yes, that sounds fine. Can we simply kill the process from the GPU driver? >> Or do we need to extend the OOM killer to manage GPU pages? > I don't know.. We could use send_sig_info to send signal from kernel to user space. So theoretically GPU driver could issue KILL signal to some process. > On Wed, Nov 30, 2016 at 12:45:58PM +0200, Haggai Eran wrote: >> I think we can achieve the kernel's needs with ZONE_DEVICE and DMA-API support >> for peer to peer. I'm not sure we need vmap. We need a way to have a scatterlist >> of MMIO pfns, and ZONE_DEVICE allows that. I do not think that using DMA-API as it is is the best solution (at least in the current form): - It deals with handles/fd for the whole allocation but client could/will use sub-allocation as well as theoretically possible to "merge" several allocations in one from GPU perspective. - It require knowledge to export but because "sharing" is controlled from user space it means that we must "export" all allocation by default - It deals with 'fd'/handles but user application may work with addresses/pointers. Also current DMA-API force each time to do all DMA table programming unrelated if location was changed or not. With vma / mmu we are able to install notifier to intercept changes in location and update translation tables only as needed (we do not need to keep get_user_pages() lock).