Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933557AbcKYUvX (ORCPT ); Fri, 25 Nov 2016 15:51:23 -0500 Received: from mail-co1nam03on0079.outbound.protection.outlook.com ([104.47.40.79]:15384 "EHLO NAM03-CO1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S932466AbcKYUvM (ORCPT ); Fri, 25 Nov 2016 15:51:12 -0500 X-Greylist: delayed 1469 seconds by postgrey-1.27 at vger.kernel.org; Fri, 25 Nov 2016 15:51:11 EST Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Felix.Kuehling@amd.com; Subject: Re: Enabling peer to peer device transactions for PCIe devices To: =?UTF-8?Q?Christian_K=c3=b6nig?= , Jason Gunthorpe , =?UTF-8?Q?Christian_K=c3=b6nig?= References: <20161123190515.GA12146@obsidianresearch.com> <7bc38037-b6ab-943f-59db-6280e16901ab@amd.com> <20161123193228.GC12146@obsidianresearch.com> <20161123203332.GA15062@obsidianresearch.com> <20161123215510.GA16311@obsidianresearch.com> <91d28749-bc64-622f-56a1-26c00e6b462a@deltatee.com> <20161124164249.GD20818@obsidianresearch.com> <3f2d2db3-fb75-2422-2a18-a8497fd5d70e@amd.com> <20161125193252.GC16504@obsidianresearch.com> CC: Haggai Eran , "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , Serguei Sagalovitch , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , "Blinzer, Paul" , "Suthikulpanit, Suravee" , "linux-pci@vger.kernel.org" , "Deucher, Alexander" , Dan Williams , Logan Gunthorpe , "Sander, Ben" , "Linux-media@vger.kernel.org" From: Felix Kuehling Organization: AMD Inc. Message-ID: Date: Fri, 25 Nov 2016 15:51:15 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: BN3PR16CA0073.namprd16.prod.outlook.com (10.165.72.169) To DM5PR12MB1353.namprd12.prod.outlook.com (10.168.238.16) X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1353;2:nkSkJ2A/Je6A57f+Bsw8Go8rVY9+S+U+ckquB43k7EEIZJate29whYRldEypCZpFWmkUOYM2RozlxgbmhnbiNx+qjraaSCMRoaor7oUIHcdXSOhNya5OazOL14nqS5/A1kKUrpGspHLc4x2za+7PUQ9zizpWOKLQTXLidOMJfsY=;3:wPDRACzXM7Tse1DkdYMPWn3P/RvEoCKAlkD/owYV8by+EIRzRyoQsRf65OQ791ZpBrm2vEZuQGaqq7ep+7NYZO6I6e8NYQGBlJP+0CKFShqHd7VbiGiqI/ars7yapzzO7u6IemppV9vZ8Q4sbE3/6K3z/9YjUpk4w8lt6UZ5VSM= X-MS-Office365-Filtering-Correlation-Id: 6ff157cf-d476-4045-38d5-08d41574c8ff X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:DM5PR12MB1353; X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1353;25:zrGCh8lW3DEXzZNlOdDz/uXDWra8pChy/xastWwyjmkfGMW5Ay9PywRCTeK2pDUdsCENn+nuszhDY5Uf9TX1xqqa354AmQHAZIPxSsBawG6KUp1E/Ib4JuxEGNJ5xY3KRHLRRnXC0sbHrjc1JhoP/C7xfUhcmWxLAKXmyBObVg29joKDhrmNg7p0bsNPmwAC4Bi5YUj7S2HblHfriVYAQUWXfYuXjM6lDlwPXSJs7T4MgvzBBsreeYZv8nSVT+scD/zcaOQX8dK1F1Qs20tw55Yn06aMXB6Kba6gJ1C/BSw28bZ9aW+42U2nSF47nhwAQOINAIrIby35vik2GdLfeDYGuSPDvdRkd3zIf69HW2+3x5M0i21ejspm0iSzrGJ4O+Lvulr0VmGWC4EeiIBW2LZltwoJQYhkjB6sHK0VH0hWNWYU9JUKKVfzokT3DBlvcoT1yRCbaVXqNcN1Q/pxhOALhrKWmbagtBOOMUd+aQzBTpGJ2/eVws1oqFAMTtp1fwddCDHYKm18xaZWjA+BBt0UZI0tSCQHFsVgJN4DoEMrmDxmgUWylvplvZK94ytGYtJ8eIUSbGnfg62Um0KFQ/CjFGReqDiSquScUyCqAzoxcXBM7TbDMiFlKDccCSjdT+yeRuFtostUmzOkDXSmGVIaRKdB3MfhNNLBn/sv+T6nNIznx5VYE+mKINk+c3SEu0r2Uy9eVp7ezz/ZycIm0O3trk4ieIUxbDET8N1L2E9DLXqkwI2rjUBSCVVFI2kpQI76a3RLTMz4nfwxDTrSvQ== X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1353;31:QvFiDrngQhBz0k9NDsambDxknWwcQbAYbN44Qtmh+ZgdINrsxQKhoHW+JjhY9dhZh1AL6wVJ7/p0IyEK2CQ+OnTtusKFowvml3Fu2jxcYlG9gC8eABEh8MfmOuYOtfSCgbhNe0Vrw9vMQBRNqqeFDZNqBxhQgi+4j8STMJLSxETdl/ez/widgYTTgfBIXeyACmYEpQ9ymx+WR4vlF89g1Db1WLeSYajtq0tDsQTPgtUYZynP50jzFA0nSErG7AYoHIygboKjqw3+4uSfVGN4w0/wg30MyIJkIxlkPKv2ng13PHJOtfD7UKDVvAfFU8Sm;20:t9chHDXw3Scr1doxbWoHMI2B9N48XgGGmKUk8yJOU6qxU8WAgHu8oApUx1n2AUuthdFz90q0vrBJnUBaV09KOESsQrTIgNYjfn/5unQYpJ8IaKnXPMkoelkubumY9JO751n7nE1JlN5akYyGrTDNuBjMx8/cLSRs+kJHczjWSXg+GjsJSz707UZSppiSIryLYPdjrf36Djsikno0EDbXCo9xzqag0YDwZPHBcMiX5YIdcIpDuaxylBjFyxUx0uztuX8sp09yrGVo2sWfr/UmAt4ngqVk+rRRDPiRVF8Fh4n0Ieyy18bSvAEVbcaoi9ZDBWwiIIfyFc+ovM4xLRAao+e5LzI/jNb+zr7TyjRWMU9FH0Wkb0TxQn1uCPlMTgPPeyXiPRexkyCuN07ufDGw5jwor+dnqJb8KmZsisOzLxDksqVTcc9t4CEoiF8LpsmIiViLApIt/UdTiKwQlQtQszg4c0ilWxrnE7hStrnH3uLtj5ZYduyylh8aJdI4E1z9 X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6045199)(6060326)(6040361)(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001)(6055026)(6061324)(6041248)(20161123564025)(20161123562025)(20161123555025)(20161123560025)(20161123558021);SRVR:DM5PR12MB1353;BCL:0;PCL:0;RULEID:;SRVR:DM5PR12MB1353; X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1353;4:nh/v6cdTLJRBc0oaE6ST6I/QRvN9etQk1tWMaX+rjT9aOxpVe/0y3YOXuZNahlYFnOLmdcb+P6sSB8lzCGpBAM8Kx4taKQTE/9hA5d6odFH8X0OIpZ6/KWwOPvCIKCSERwdM5L1b7QHjA0SUYtUsbcIls4NhnYYdLd1yB1Fh72HzwBPIagzwkLz8nRgSUMFJOR9JW4vGU95kPXylFFT73SqvaKKDbBPuc3Pz47XR1zpnbzUMVbqVBgPIL9bxy4ErFoK9dhC3SFC64JF43wRY2b/EOzcA956RrKgE3GKVxffNcENbzsigUAi28tQxHoyESnG+DH+IB1oco9FEPduvEoUe2nSf9apdK3ZTDDg0C1pbbwUvRP4LSKqhKT+rU69/ysjFRqzdDgEO9+7Dd5pNQMp6utvh0XF6Sb5IYWQ4Sza9bVdw35WgpJXmGWoMC/S7qLUObj83AYQI0gVIs+OYACpLWgbhi1Kn/PiwPEXoMEMyH0zMwkw52uqIjfZzZHd/wVLt8D1LrKsPJWxiTctFFsG/goN8GJFGO2TgIaOKgzG8ReEvZ8vyBmW+L0S0+QlSdiWkE6R76WT9mkokp4H6KpJe1Ok3+fss74n1IPjhBf8GCWTGm21ttbIvTfFBs0b7xlpzml5n317W8sVOeNKAK8USyS/cRHvHfDuko5Oya5w= X-Forefront-PRVS: 01371B902F X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6049001)(6009001)(7916002)(377424004)(199003)(189002)(24454002)(377454003)(39410400001)(39380400001)(64126003)(68736007)(36756003)(3846002)(5660300001)(23676002)(65826007)(106356001)(2950100002)(105586002)(81156014)(77096006)(7736002)(305945005)(81166006)(8676002)(6666003)(7846002)(42186005)(39400400001)(65956001)(66066001)(93886004)(6636002)(76176999)(54356999)(31696002)(4326007)(39450400002)(38730400001)(65806001)(4001350100001)(7416002)(50986999)(33646002)(31686004)(229853002)(189998001)(86362001)(47776003)(50466002)(92566002)(5001770100001)(2870700001)(101416001)(6116002)(83506001)(97736004)(2906002)(733004);DIR:OUT;SFP:1101;SCL:1;SRVR:DM5PR12MB1353;H:[172.27.225.16];FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtETTVQUjEyTUIxMzUzOzIzOkdIZlo4NEtpeUV4WWRKZzRaYlVZYjlHcnZS?= =?utf-8?B?amRLQ2ZvUWVQN3Yzd1VBdUp3blhOWGp0REowbzZPZHRaNlJaQ1hvTDVxV3d1?= =?utf-8?B?TGVKRmVRdFYxUGF0U3BycGZTdGxzY0dYNi9tczFwb2VROFM3U296TjZidGlW?= =?utf-8?B?UUYrd09yYWFhKzR2V2ZsQ0ZjU0pJZi9YTGFQMjFmVmFvcUVTM3c5aGE1WVpF?= =?utf-8?B?RWZhZUsvVWhxNGxiR0xQVk91WnppYzl0YjVEV2dxMys3dUhKVXY0cjAyNVhU?= =?utf-8?B?SkprblhKWVJreHF5bWtZcVRCS09XNTl1STlsVkJ6Q1Y1U3M5UjRWRWoraXhT?= =?utf-8?B?a2UwcE9ndDBzR3p1RzdQclNvaEc5TGxYanlQRmhpdkNhZUErdjFQaHZYWlA3?= =?utf-8?B?ZmQwcllVNkNEWVBpNWFjK3dwK2NYbnNvcnpncnc3WnpFUjFwT0tsak5TQnFk?= =?utf-8?B?TDMyTlR6TjYyZTJhUkZaZmJ1dWJjeWJzMlNJdUZidmR5SklzU3pGbTczR3lG?= =?utf-8?B?QmVCYjN1c242RTVoVW9sTU5SemZOVHpTOVlvdmtEWTM1aUdQTWhrQktLdXVk?= =?utf-8?B?eWtPYncxQXZqSk9XYnVLajhueHV1aGlxTjJzMGttaURwWUp3bFhNYi90VldX?= =?utf-8?B?WCtBZGp0eGNsU1dCV0VwUnhGcTFDVUtmSWZMWE5uT01VdzRaZWFkUGxIT1R3?= =?utf-8?B?a1V1Wk9rV0t5dHVpNE15RzR2V3Fzc0xxWGtzb1IvYm5FbUl0U1YxRjNjNDE1?= =?utf-8?B?UjdWcnFMMG0wUmMwR250OXgyV1prd0lteVREa1AvL3U3eHc3a202RDBtdmZv?= =?utf-8?B?Wk05MXExL1ZFWjJTTHlPL2xGdjNkV25vVTVpcW40cFY4QkVNOU90OWsrQnBk?= =?utf-8?B?blMwckc0aEJkSkVVeUJoWHJBMmJiOTVWeGFSVmxlQ2tmN21WbWgyNFdiUjRx?= =?utf-8?B?a3lmSyt5SjZmZ0lEbDdvZ1V0M2lyazdRczZtSmlxZ2tBaXl0eWh3OWMzanZK?= =?utf-8?B?SGtIT2ZEakRLak1DVnRoU0JNMlFBMkFtZFJmcWpjYWRoTzhSLzkrWHNvYVhi?= =?utf-8?B?OE5hQ1dGdEJWQXc4azM0TlJtV1VFRFdhZUhNR1dRRWJFcEtUeWwwMzlhUFk1?= =?utf-8?B?eFhUQTRvZC9ub1RrcGJTTnFMcXl3QXdSQkZaZUJ6NUZIS2tsVTZpakpDQ0Ry?= =?utf-8?B?S0VWK0VOclB5NlYvNnl6SEo1L3B4Y3BMU2RtVk1nV1NOTFdhNmc5eEhybVZx?= =?utf-8?B?SnlpcFFUWHFOeG12bENnMXJJYS8rd1BIaTRxYkFsWkx4NzNkcUp1dU9lVzdi?= =?utf-8?B?aXFGNnZQdW50YXdOYmZqekdVUDVsRmFMbWI3V0hKdUVsU29jblhiMDcrMStE?= =?utf-8?B?TEVXbFR2SUdPQXlheTVDeHZQZzZGNWQ3OEtYMHlTbENZMGpET2JIemlWS3BR?= =?utf-8?B?TkZvay9QeHI3bTRXemlrcEx1azVTUjJVV2xOdGNoRlJsS0tiTThBNTdibS9i?= =?utf-8?B?c2ZmanJBalJXd0JtaDZKNG5Eb2F3WXZLdHB0d0drdTZUM2dWTmxGZVFEUlNN?= =?utf-8?B?djhJalN3bnhIQzV5ZlpCeEVhRXVWTGU3cUsyM2kzT3Ezc3drNW1XcjdNNXl0?= =?utf-8?B?NDFGQmN0dEJIa2s1QitWZ0xWYjZnbDlXMENtWDZSdEh1eStOZVJqK2pEWGJC?= =?utf-8?B?cjZxTURQT1E0aGtzOXBteHBUUWVLano3QUViQXRKY0hvSGZldjhaUFRXNGtU?= =?utf-8?B?MEphMkJpcnlaMWxQSnYzdkhEK3l6c0VqaUZPMWo0enY5WjA3cHdldW5ERElt?= =?utf-8?B?a2kxd2NtVUJ6blBobU9ZQ0FtN1ErTGl1Mm5uVWRweTFCTG5qRS9LMkRJamZy?= =?utf-8?B?UGxzUEtTZ3h4b1R3K0dhVFNFWldCdEdZZWpyS2xUak1IRWVsTHFLVHFXQzNE?= =?utf-8?B?b2g2UzhrN0lNM1VZSWRXYUVMSU9GbmRsajQ2SEVJN1FIZGEyNjVadUJkZnBG?= =?utf-8?B?Q2ZYV0JEeU14am5EeVhqYzRmYkVkYWowTDY1Y0h5RVRtQ0Q3c3BLM0k1RlZC?= =?utf-8?Q?W+d8=3D?= X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1353;6:fOT359g0W8afwQseyuE4/LZ6l9ScWVHp256VhbPrXdL+Zn0oLNeF4iAo3lDqKRJoGPgnrgF8sKNaVtMySyQKySHAP89QpcB7sx5lhTPRON+RpiihmX0g5bKG4Nt66Dx101DO/G2i2DGyxYKt1OJjwhAIo4Tbbc5qFheiTKaA+6NpSOGHKL9UYhi0Ijx5DNX5hN64Nm3TS047cK0kZoK6hlAyHnr1SPP3PhXHdHDNaZpT+TY1uNXoK8W/F/XSxb+2aypynJf/emtKEGjDEFo8d/m/1hi4qxFsSNbj9Lg3bXkp4Q0U8lFAM/LBx4UoIBh7exJ8vxzAA0p5HfaKCcRpHWusxdH3PCRs0aebf53VvO0DjMk9gF4q5eZIwEPxVbXw;5:f9w/+ncI8v/Jk+hLKSScGyRKX92yzH8MZHFQ+0OowuEhTUfoggoY58TTsvT5AGy/+UsLV4nB9XeWq1wqctVRyIxC7ZTg5hbyVUGhLsHcF5w0ODIRb+Gjq3UfedATEr9mepOTl93np7VqAYh+oBnfndAam+OmZIej/NBZaAX1rHI=;24:r0+AqmyTB016Nykce6pB2Pumj+cNsvAr4YZDd/AWAL29U14lCLc8IodeUvh+oU0cs+s/tmJkG4wtm0pexR1QSjm4kvrJJ+ZSA+YEY6b0eyc= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1353;7:T7RGcMjMlMxXH8G/4KPls5MpULfKr3iEw1WidREV5PnDpGPxBandxbIV1zNZmXo+zkBbYM0ag+81w/JGF3YvVColegAG7176pTyi2VEwul1dfZ8AZeKe30a3XtFD28PMUGcc/jaQefA/+SbPGCTI245foU+/dfYK06Sk8f83uGWjkGl9PHz5p9tB124uhg9wzvakz3XIS4X+rnQjmZCrTaTZvKhA4zOb1qLSEGnJ87qoKcYVzMezFDpybEtv6EVlnLrW2CAOeFM58dYaq2tLLELzpN3Ovi9ge+WtFjdXF2GDV5Ksz9TlzWuQIyj+CPImsJ68aNVF9T7MWQ60Mc9yeAqk+FBrsVsao6Gfag3d7aE=;20:F3CX0PSjajPQO0eTr1S5NpW148KW4YuGww7F3AkE2V+pO71P/UXtJUuvyhCaR6MN0NACfzqv9HGhUBVNnEqV3rpofavu3qszEbYOog7Dhp+35kGkwq7B51bgQlJ0dDKpASS29/qCBdaEsAmSrivvad/kWW5GsBK72ntob21ZZ3LcB0ap76cGWJRkXqSkjNMNOgN632BGStKDBxsJwyKFGd9wYKmXnC3JxeqlytvjXiq31BXa4dlfB9/8JFhbSRAE X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Nov 2016 20:51:06.5974 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1353 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1160 Lines: 27 On 16-11-25 03:40 PM, Christian König wrote: > Am 25.11.2016 um 20:32 schrieb Jason Gunthorpe: >> This assumes the commands are fairly short lived of course, the >> expectation of the mmu notifiers is that a flush is reasonably prompt > > Correct, this is another problem. GFX command submissions usually > don't take longer than a few milliseconds, but compute command > submission can easily take multiple hours. > > I can easily imagine what would happen when kswapd is blocked by a GPU > command submission for an hour or so while the system is under memory > pressure :) > > I'm thinking on this problem for about a year now and going in circles > for quite a while. So if you have ideas on this even if they sound > totally crazy, feel free to come up. Our GPUs (at least starting with VI) support compute-wave-save-restore and can swap out compute queues with fairly low latency. Yes, there is some overhead (both memory usage and time), but it's a fairly regular thing with our hardware scheduler (firmware, actually) when we need to preempt running compute queues to update runlists or we overcommit the hardware queue resources. Regards, Felix