Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753177AbaGVH2q (ORCPT ); Tue, 22 Jul 2014 03:28:46 -0400 Received: from mail-wg0-f50.google.com ([74.125.82.50]:49118 "EHLO mail-wg0-f50.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752818AbaGVH2o (ORCPT ); Tue, 22 Jul 2014 03:28:44 -0400 Date: Tue, 22 Jul 2014 09:28:51 +0200 From: Daniel Vetter To: Jerome Glisse Cc: Oded Gabbay , Daniel Vetter , Christian =?iso-8859-1?Q?K=F6nig?= , David Airlie , Alex Deucher , Andrew Morton , John Bridgman , Joerg Roedel , Andrew Lewycky , Michel =?iso-8859-1?Q?D=E4nzer?= , Ben Goz , Alexey Skidanov , Evgeny Pinchuk , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , linux-mm Subject: Re: [PATCH v2 00/25] AMDKFD kernel driver Message-ID: <20140722072851.GH15237@phenom.ffwll.local> Mail-Followup-To: Jerome Glisse , Oded Gabbay , Christian =?iso-8859-1?Q?K=F6nig?= , David Airlie , Alex Deucher , Andrew Morton , John Bridgman , Joerg Roedel , Andrew Lewycky , Michel =?iso-8859-1?Q?D=E4nzer?= , Ben Goz , Alexey Skidanov , Evgeny Pinchuk , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , linux-mm References: <20140720174652.GE3068@gmail.com> <53CD0961.4070505@amd.com> <53CD17FD.3000908@vodafone.de> <20140721152511.GW15237@phenom.ffwll.local> <20140721155851.GB4519@gmail.com> <20140721170546.GB15237@phenom.ffwll.local> <53CD4DD2.10906@amd.com> <53CD5ED9.2040600@amd.com> <20140721190306.GB5278@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140721190306.GB5278@gmail.com> X-Operating-System: Linux phenom 3.15.0-rc3+ User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Jul 21, 2014 at 03:03:07PM -0400, Jerome Glisse wrote: > On Mon, Jul 21, 2014 at 09:41:29PM +0300, Oded Gabbay wrote: > > On 21/07/14 21:22, Daniel Vetter wrote: > > > On Mon, Jul 21, 2014 at 7:28 PM, Oded Gabbay wrote: > > >>> I'm not sure whether we can do the same trick with the hw scheduler. But > > >>> then unpinning hw contexts will drain the pipeline anyway, so I guess we > > >>> can just stop feeding the hw scheduler until it runs dry. And then unpin > > >>> and evict. > > >> So, I'm afraid but we can't do this for AMD Kaveri because: > > > > > > Well as long as you can drain the hw scheduler queue (and you can do > > > that, worst case you have to unmap all the doorbells and other stuff > > > to intercept further submission from userspace) you can evict stuff. > > > > I can't drain the hw scheduler queue, as I can't do mid-wave preemption. > > Moreover, if I use the dequeue request register to preempt a queue > > during a dispatch it may be that some waves (wave groups actually) of > > the dispatch have not yet been created, and when I reactivate the mqd, > > they should be created but are not. However, this works fine if you use > > the HIQ. the CP ucode correctly saves and restores the state of an > > outstanding dispatch. I don't think we have access to the state from > > software at all, so it's not a bug, it is "as designed". > > > > I think here Daniel is suggesting to unmapp the doorbell page, and track > each write made by userspace to it and while unmapped wait for the gpu to > drain or use some kind of fence on a special queue. Once GPU is drain we > can move pinned buffer, then remap the doorbell and update it to the last > value written by userspace which will resume execution to the next job. Exactly, just prevent userspace from submitting more. And if you have misbehaving userspace that submits too much, reset the gpu and tell it that you're sorry but won't schedule any more work. We have this already in i915 (since like all other gpus we're not preempting right now) and it works. There's some code floating around to even restrict the reset to _just_ the offending submission context, with nothing else getting corrupted. You can do all this with the doorbells and unmapping them, but it's a pain. Much easier if you have a real ioctl, and I haven't seen anyone with perf data indicating that an ioctl would be too much overhead on linux. Neither in this thread nor internally here at intel. -Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/