Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933085AbcK1PDn (ORCPT ); Mon, 28 Nov 2016 10:03:43 -0500 Received: from mail-cys01nam02on0047.outbound.protection.outlook.com ([104.47.37.47]:55552 "EHLO NAM02-CY1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S932532AbcK1PDk (ORCPT ); Mon, 28 Nov 2016 10:03:40 -0500 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Serguei.Sagalovitch@amd.com; Subject: Re: Enabling peer to peer device transactions for PCIe devices To: Haggai Eran , Jason Gunthorpe , =?UTF-8?Q?Christian_K=c3=b6nig?= References: <20161123190515.GA12146@obsidianresearch.com> <7bc38037-b6ab-943f-59db-6280e16901ab@amd.com> <20161123193228.GC12146@obsidianresearch.com> <20161123203332.GA15062@obsidianresearch.com> <20161123215510.GA16311@obsidianresearch.com> <91d28749-bc64-622f-56a1-26c00e6b462a@deltatee.com> <20161124164249.GD20818@obsidianresearch.com> <3f2d2db3-fb75-2422-2a18-a8497fd5d70e@amd.com> <20161125193252.GC16504@obsidianresearch.com> CC: Logan Gunthorpe , Dan Williams , "Deucher, Alexander" , "linux-nvdimm@lists.01.org" , "linux-rdma@vger.kernel.org" , "linux-pci@vger.kernel.org" , "Kuehling, Felix" , "Bridgman, John" , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , "Sander, Ben" , "Suthikulpanit, Suravee" , "Blinzer, Paul" , "Linux-media@vger.kernel.org" , Max Gurtovoy From: Serguei Sagalovitch Message-ID: <314e9ef7-f60e-bf6b-d488-c585f1ea60e8@amd.com> Date: Mon, 28 Nov 2016 09:48:34 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: BN6PR20CA0031.namprd20.prod.outlook.com (10.171.173.17) To BY2PR12MB0695.namprd12.prod.outlook.com (10.163.113.17) X-MS-Office365-Filtering-Correlation-Id: bf03ff35-5367-4f93-338a-08d4179da6cd X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:BY2PR12MB0695; X-Microsoft-Exchange-Diagnostics: 1;BY2PR12MB0695;3:fT5wLX9HDhuvh2olF10toZuNfIDRA+jNtKeKzXspLBGwg3dgwTSH8DpWHOiYDh6Wty2SpS+gA4oaGaviW2VEOuiupsrS+FLuucHwzbAWNI/AL45QSCGoU84KWNB1AeJXC1ivF9dMb6o3jRTSiKxNpyA/2Y+aMDgLSCrBKpAhkwpxqiiklqKNxKgE2qvYYX6SYM5s+hPq/0EdUlhUAvRfqinTjnjpXGsHG0F+hXBY8yiWHReiZYm4OwZxfQKxE8qdq4dvI08NLDqf1j5/lk2RMg==;25:d4OUqbj6ohfEyRVUZK0Ri8+tGEtJzCqVBOCKYykekjPgC3c+B4cfU9A38KGcFMwAsBeuAzLPHIKvKxI6ZcGAR84ze4JyblBiQd1b7ehhpdngtgyhdXJZo1QTaAjKltC71HG3uEy6Ujez/rRotczXpqV05tUQX0HWJmeCv3Vvqz/e33ciQMDeJaT+t7denYFp0kcYufGlfe85WMpeHNNcjzqlO7ujeKthEu/Sm6c5jUB4iQaXSE/weaID5o6J+D7ZSJD/7Cb+iKkDiPD0fUWWBn0t0QQeQKgsaNXSglP5T65NRJZERFETHXDht+80NeQ0Sewxu9mjaIoiDwhbhPSTzLOK+WQp/aDzYUn4qBYFnRsBmA8XkiMBcps1maWtmDY84IC9B2Ny+fHI6BhGswSy6XnYzpbELYmkHQSVC3gkoYupruD+XhuXu6LTpQXWb6R5GStxiWLqNBWAmxVhPJgzfA== X-Microsoft-Exchange-Diagnostics: 1;BY2PR12MB0695;31:h1GjbL3G4sx2QOlnu83ZbXUUmL9H8rw1ZDIHNprjtS9kYb8B9+JmOjkyM2RrEdtVWGe0OXuc8YtbPiFoZ/amv7qGdzBQUI8kGUEv6GOeD1yP4aN6sBJpAFGqJW7fruOKDaVoM0j+qJP0F9bilmazkjpQlQXmekbC+Juw/gpcVwSOV4TzCoIOgaJo7+r2wKdqi5fMHp+42d9/jIedBecmNxRbAB8g1/s7/yzLj+F+iufh2Qc5wykcq34qF3Yml0FHy3RU01q9wihYC501FS3s1g==;20:pixCRm3aRqar/7hER6cDKespeuftCciUYbsYiElZTn1JLVPQmLZqNYg3mw1tZ3/m4T8LiOmCpAPGdbKKtusSXyfMnoYW+DASdCI8f7hi5C9W9otSEn4Zl+U1yLoDhfBr+ULipPmLZz76SB5tWKikMcOIb4eHexl6qz3Sy+M6DLp5RIJoYY6PbwA3hHjzn0cgt94RKriOZDMFimfNZ1Gyd5pPlCQl3aKN7GAls8dmPi3ULHNP3OmQHFaAQKY+tnAoJH1l9cE1HCXmjviizkSrqQZLr5ThitPyZ5FMo6ScltQQp/OPyEyxH6Ksur4wq9kTu1e3E8F+SwyksPay7SKBnDu6wyjdYmXP5bv2F2zb11O71ESWVCF2GFqwVqhKHawHYS4OPQvatlxvgXB1M5ATq4AROx4tPdN9HRuD+cEJ3Prk4fzC0CBKCS08qIr7Dm4Vdhrtq3lGLncqA6YdJ0N8mwt2vK6lQwGNuPL9THbkf56gl5vLlWJHcCqA/eOytSj/ X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(278428928389397); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6060326)(6040361)(6045199)(601004)(2401047)(5005006)(8121501046)(3002001)(10201501046)(6055026)(6041248)(6061324)(20161123560025)(20161123555025)(20161123564025)(20161123562025)(6072148);SRVR:BY2PR12MB0695;BCL:0;PCL:0;RULEID:;SRVR:BY2PR12MB0695; X-Microsoft-Exchange-Diagnostics: 1;BY2PR12MB0695;4:nyH8US/QFq0bzp7BNbnUknP9VZrEq0Mk6B9SAquy3jyIttZA4IcTDc8cqm7G5y2b5kyckAZGs7nQKNdXQrY+U81QF1iKn0Q17EDfg9KHhDy8CTZaL8BjXhB0fLG8mII6F4xR6YOnsJe8VpQBQQ8SNWc3bXGjAE8Rf+pGizJ29nDXSpSnu/2lHRrO8osBUJy8GsFoxzSwkCFrVed8WTs7T60GDvFEYL9NTeslvwO9/r0t0nQFRGIJyMLp99q1eQX13J3Qr9CirSwRlyJFB2gYTTOqKTm/itKQmPKqznn7GGtnwBvRa9wUPJI8PSxe+DkBVxzoFrEq4aCT1iNvtbMPackRbSll1S9Q8Fo7YtDN47XG3/zyFTxjpYs612T5us/OeHGyZicmV4SMZ2BPQ9pnegXR22NSFPRgwrw2UXZsxrmEqUoMXK9/x0VuxZqZO4SqyoCDstVDeuWdZK0r06Dy7ABhbOFwKTrJ1Hr5J0WfEfzRF7wkvlzducNe9BL1qU6S0RPWJTJCxl3lbPdpenho0hCCoxO2VMKJXMftmA1sfnmp/Qc+q+ihoHWdDmz7ACAdM6rgAexpi8k1wDR4VcSLSulEDLusSHia5R0KSPUoEuAYgvlfNmTtmurjYekhJe4ePL84uHFUF/DPEJ6A8OgTaxL+vVbhENZbda6VEEoJqXj1lniovhjraLRXHnxxjCNA X-Forefront-PRVS: 01401330D1 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6009001)(6049001)(7916002)(199003)(377454003)(189002)(377424004)(229853002)(5660300001)(39410400001)(39380400001)(39400400001)(7736002)(305945005)(77096006)(39450400002)(189998001)(733004)(47776003)(6486002)(8676002)(7416002)(68736007)(54356999)(4326007)(65826007)(50466002)(38730400001)(81166006)(81156014)(230700001)(76176999)(86362001)(50986999)(64126003)(31696002)(101416001)(7846002)(2950100002)(65806001)(65956001)(66066001)(5001770100001)(4001350100001)(105586002)(97736004)(23746002)(6666003)(6636002)(106356001)(36756003)(31686004)(83506001)(92566002)(4001150100001)(42186005)(33646002)(6116002)(93886004)(3846002)(2906002);DIR:OUT;SFP:1101;SCL:1;SRVR:BY2PR12MB0695;H:[172.27.224.67];FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;BY2PR12MB0695;23:xNvlu3ddxrUXigzCSKI3s1iCBMDni3TnywVs9?= =?Windows-1252?Q?8x6iuB5Fw8YVpohOayUJefQrH7Isfan/xJ+IoBQADpRiAelfltOojWGN?= =?Windows-1252?Q?FrZ5+LbgqjzDgD8lifGmxPyKyGcPdIH7EYnjcawsnPIlQYbG+srH1ron?= =?Windows-1252?Q?Ixj0YnK8b2aVvkS/awUAtn3vd9lQ1xNlZxKakZn3AxzwXKRWZaHX5hjH?= =?Windows-1252?Q?5rGmDBFyd6BRlp7pZvrW1LfYjpnFQzUQKixOddulY2I4ZG0ROCmII6ED?= =?Windows-1252?Q?6h/nMJoGAsCVwBhP3nlrWzM8NtUO8FVsrmduvTx5DFomdL9NV4AYUe/N?= =?Windows-1252?Q?ioQSpzO42Gw+3oaTG1JOqqj5d0LH+fvjIh1/00go/Ha1TlZA3xR5LTAY?= =?Windows-1252?Q?E4gDjZuBHHal7Pzf7BC+IOo4GxGSp8UoqEMChpbSol//ar9LMmTSmoVC?= =?Windows-1252?Q?Q5eAbRsECo9l8LWFTolisHhl77UjPyAopKww1OvWMUvHJ81IPWK2dH6y?= =?Windows-1252?Q?EpqXNFPBpSQrW9TqMBRBs8gTgwTMzQQbSnFOm2Yy+JuBAF5Z3GNAkNS3?= =?Windows-1252?Q?4UiJAbZ6ar57U4zq2S68iqAeir9xQOzub/bL3whzof5zMEDzTtrDJRkH?= =?Windows-1252?Q?LO9YZibzdP5cCWzTvr8nvyZ8sNqP8V5ct/GqGYnyEoUKuEXID5X7IT49?= =?Windows-1252?Q?+Prz3hYODUilAPtCPHOY/hr1Vj305s87FHzn4W8yApHHYJh/XdRQztxc?= =?Windows-1252?Q?sXC2QKHPbo4uRZLZIGiQwV1FFhO1zwe0qEdrrBdAEi/kXqTFwpwOGIF1?= =?Windows-1252?Q?av0znfsXNmnqMGndp/tJMX9QrxyGEuo0DUzAug2uU9mhx6BQq90CA8hn?= =?Windows-1252?Q?ArEc2nSA9SPu0xpMGfceRTZGk2DKVHZjxgqUP7+yHBwqOXqw3MBeW3fH?= =?Windows-1252?Q?zibQ7cPe/TjYucUpoT1YOGRIGcBcz76uea5cvHPfUIGRnqBj9cmbU6WH?= =?Windows-1252?Q?fZASVEeHxwy0fBrz9INH2De9dn4w2iOP+MUSS6oEytQ0uJibRXtNrzSz?= =?Windows-1252?Q?HON+lG2vxIc+Mjbh1+1a1zkz08ZzO863HOraw2VbcgfOetb9iztk/jyv?= =?Windows-1252?Q?B9FOT2mm/S/D6qxt5BeAs8FsphrnDI0djtL0TMdQdEg/INkj7lUPHLK/?= =?Windows-1252?Q?U1rnk2w4yAcL2gj488Mi57DwZNU385Wt5H1lXWufJOpo2ri+kxkpaJb1?= =?Windows-1252?Q?djXds0UTraxpj/1dmwa+Cjip4uvTTb3kVcCckW/7frGSEmVzKt37Nh4y?= =?Windows-1252?Q?/wbQmS4o8RMQFnwG7Qvhx5gXU+HH+xUHui8iJZ1DMJDxNt2u3BhHom6q?= =?Windows-1252?Q?/OPbnSKIicqkLauab5HLt2MWj9SVcpJLRxdsBN/+v6LEa7Tl/WHeTfi3?= =?Windows-1252?Q?ldiU6h2b55SpXvuOTjk3CArCRxWzKpgYIz7APk8YfwDLAFUG65s2zXgt?= =?Windows-1252?Q?Gbh3rW0a2sKyaMs8hogStbWrWJTRS2JjiAHVOWXHPM6GKsHy69gIlGoX?= =?Windows-1252?Q?v0ueZlAzG8H76w8Ua0gU2ae7EKWRTl901GPfXTkz+yiLpV+ekr7LkDwv?= =?Windows-1252?Q?G25l7+Hfu2yFI//I+jSMDs=3D?= X-Microsoft-Exchange-Diagnostics: 1;BY2PR12MB0695;6:E2o5Q5+VO1VeHUvOSm5vcxBNmTaUa6DSQEWRwqvdeCQxJM/pHJo3J1jtYgVeMTC7U+AWVk1HNzc3nv1YtWsjE6NP2qBQMU4TMMQpqBItJomsecrk7zjW+LmiZiZWyHMrFiuL89tM3OZlG7d/D2ZRmcOb2fS9v9zjN03nHlQfF8UBSRoseKDyhKcVwHslauM+026TqroPkcYgQrg1f3524El7neViU7CX+6fA1Bqp/8xoOqyMNuMn67veubjuG6Ev7fOVtGOhyfNk6W/59pEobosOIRxJgPTGQ9vaSIbj19ZZeMVpCgEGdOqAWEtYMu2kyQC6EuGhZ+cBuCYYyq158xnNW1nde7l+3ZyzzwgcFXRYLvKmWnls/pqYUE3/unqXqLGCn5ATjuF7/XbuA+US+Blhce9soi5D1wRbbNCpT1Ps4fkG6NYxboOSgZmtLsypYR/wft+CqIuCZ8EXJ2ijOBSSuedm8aUe0WkYC5EJTy9xI2N58/XTCLzoxcv47Cc8;5:UevJhghAUV+ZBOFtJmaozV3brQZRvgjtMtuiWHuNbRQjq91Maoiorw8C7Bf08DhaLS4Kree4m2slx1m6D+CLeSuHiKWUTRWM6Nd06ioF82+o5Xc2j3dtBsNrSDULSQS8Quvb9UKXHZQPouJvMq+W5bnl/otjp3bTWs7OYM0ZEI0=;24:8va9S8LZwR6PJ4KJjZEB/xNI7Rujs6m5HJRBlnW5oPh0GaRPn5kLkgYNaBbdapvCwpv05h2LRYdP45pE1NyFfafeAdm06AZpE2KpF0l43uI= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;BY2PR12MB0695;7:lEPCXnZDSEWCvtScHUtOqCZBwfVme0wpRU91w6SMQTbjv7Cy8IxJczk3JA4X8BO7aK1K3LuYwC2Mnmw0cb6v+BbytMknRJYhUxtgc5DIceoLxwYnhlRk7OMxP6W2LHQQ0SItlgR4HwJwHI7f4WpocO7qGbFPYTPj6uXszPXnLWQsIFMB84KTNak+q09sEz4ENy21Mv8RrhlAXjlKJMYtcdD4ur4Xl9q2/e4co//D2WrYx2+8fB/uuvYzy8XZ+pvLiLb8hCL7WX/AUhCuGH7tbxCqXBmxioRudEyDpY9Njbm9XQALblEtjDXa2RNBp+Ql5BFLQxLzcnFDmK+kTTxKwROzofSSRMXNRWEmCUDIykjjz3TzoTzlEfxM7ORWntPqAGzee1OkLJI/r/ebv6k+CRdx87V1maxJKFd/zveeqJjigF5gjEdrhfgCbsgroPbX7zkqfB1bpZDYCqZcKyT9sg==;20:5wGbvB0xwWRzJyzF+x7i2KO107CayWl6SH4iaQ/PyqEcUpKicrcJd28a87I1JXfcaocXAoQI9RuvL1kfjP2s/t6WfWFcEsw4+RMkqClSgrUwe67VhYqSfWVVWutsv+u8bp/CKqb1XL0VE5sBeT39BeVWbxbPoYKBHPUfwji5Ch7mPEXnJ9B63tLdjLWYKFBcICaAe+U5iws5tyouerU/QqjSotava6s7/hzwpTkX7VNmicKZ20/4He4bIwUT+kSw X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Nov 2016 14:48:39.9408 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY2PR12MB0695 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2078 Lines: 36 On 2016-11-27 09:02 AM, Haggai Eran wrote > On PeerDirect, we have some kind of a middle-ground solution for pinning > GPU memory. We create a non-ODP MR pointing to VRAM but rely on > user-space and the GPU not to migrate it. If they do, the MR gets > destroyed immediately. This should work on legacy devices without ODP > support, and allows the system to safely terminate a process that > misbehaves. The downside of course is that it cannot transparently > migrate memory but I think for user-space RDMA doing that transparently > requires hardware support for paging, via something like HMM. > > ... May be I am wrong but my understanding is that PeerDirect logic basically follow "RDMA register MR" logic so basically nothing prevent to "terminate" process for "MMU notifier" case when we are very low on memory not making it similar (not worse) then PeerDirect case. >> I'm hearing most people say ZONE_DEVICE is the way to handle this, >> which means the missing remaing piece for RDMA is some kind of DMA >> core support for p2p address translation.. > Yes, this is definitely something we need. I think Will Davis's patches > are a good start. > > Another thing I think is that while HMM is good for user-space > applications, for kernel p2p use there is no need for that. About HMM: I do not think that in the current form HMM would fit in requirement for generic P2P transfer case. My understanding is that at the current stage HMM is good for "caching" system memory in device memory for fast GPU access but in RDMA MR non-ODP case it will not work because the location of memory should not be changed so memory should be allocated directly in PCIe memory. > Using ZONE_DEVICE with or without something like DMA-BUF to pin and unpin > pages for the short duration as you wrote above could work fine for > kernel uses in which we can guarantee they are short. Potentially there is another issue related to pin/unpin. If memory could be used a lot of time then there is no sense to rebuild and program s/g tables each time if location of memory was not changed.