Received: by 2002:a05:6358:d09b:b0:dc:cd0c:909e with SMTP id jc27csp2178150rwb; Wed, 30 Nov 2022 03:30:28 -0800 (PST) X-Google-Smtp-Source: AA0mqf5JcomlJ/w0h+GTVmFQkfkGbiTaPrOmerMKq4gY9tA+7XDpQ0a4SPLt4V2BJgpwaGohFu/7 X-Received: by 2002:a17:902:d50e:b0:186:944d:47b with SMTP id b14-20020a170902d50e00b00186944d047bmr45124434plg.42.1669807827928; Wed, 30 Nov 2022 03:30:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669807827; cv=none; d=google.com; s=arc-20160816; b=ldVA/h0mR1k4u08cVxzBWQWMZBZEik/CVtfQyveJKPrR3bT9HCpsG0XuPbhQl7Zv1E sn5IAvU00tE5Uqs+hP7pn5C+UYPP9rMJwDZ+5w0m1mU/LmIklKd3n8nIG1R1Fto+lDp3 cRzDa50ftAcqWQezUcQpTb0kfUk3NdhdtWo/pj+A0OG9vqcLxmnDNmWRlTbeFSczzTAZ wVKhKzmlo5H09xIpKEeyg4AF4eLbuiyAC+Kbo4UL9y40zNh4LWsTvUB5DJQTuQ54em/i 1nk2d1/BXahJFJqdKE2jC3QHYHEzCqIFoMOYbRPONxRRsYe4csN5pDz8s/GmLr+R7NWL Et8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-transfer-encoding :content-disposition:mime-version:references:mail-followup-to :message-id:subject:cc:to:from:date:dkim-signature; bh=juKWxF5oTD0FcdVr6vmRKJx9lD1N7VVH5XonHNDYvKI=; b=uNk6tpDq003wwTMFnvfHF1XzhJ5WiPe9qrBq6FVB00JSNdRh0GeHP2C9XD/r2oBCfq a13SzK4OxJnwJnkI0Bq0UY2RinXAc2WjapAxv26fGlan9ZGDaEwSZnYDkS6cJk+bk3vG 9ElF7AK6XviwRGA7sYVYmOEJXRGOBxOIukLQ1lPlbCeXF/pfC3VKfxBEDUce/lNdIkLd cU5IUViJraIjaze0INVYvcsBl7INbbs0s20nGA0Nv3BA5VFbz6nrwN0duntKlklCmIQm mXJWRBcfPMiAm/pKfI5AJ/QKSZWk45X0Lzha78bK0s7281nA1m1A/XZdrRcuFqIaJ+3O 6O1A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ffwll.ch header.s=google header.b=ZeLJpRgv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a17-20020a17090a481100b002130d0f6df8si1253347pjh.30.2022.11.30.03.30.16; Wed, 30 Nov 2022 03:30:27 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@ffwll.ch header.s=google header.b=ZeLJpRgv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233192AbiK3LLs (ORCPT + 84 others); Wed, 30 Nov 2022 06:11:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35004 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231550AbiK3LLp (ORCPT ); Wed, 30 Nov 2022 06:11:45 -0500 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 15C6474AAC for ; Wed, 30 Nov 2022 03:11:44 -0800 (PST) Received: by mail-wr1-x42d.google.com with SMTP id g12so26489718wrs.10 for ; Wed, 30 Nov 2022 03:11:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:mail-followup-to:message-id:subject:cc:to :from:date:from:to:cc:subject:date:message-id:reply-to; bh=juKWxF5oTD0FcdVr6vmRKJx9lD1N7VVH5XonHNDYvKI=; b=ZeLJpRgvKOGWEOW4EgtAkpVPc8pF30xRPEQDkJvUfV4oWOggtksYkh0wrqehizoYQZ qgWms415U1t3zr+nXd6FDdXnOZKbAeNH8D8m9SSNLLkDeAk8GOm4Yz+uPZhYeJ7LPqV4 1EhJdMtb76hFaUYi5iCkoy5P1q4EZZB5DFFlw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:mail-followup-to:message-id:subject:cc:to :from:date:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=juKWxF5oTD0FcdVr6vmRKJx9lD1N7VVH5XonHNDYvKI=; b=mApVluLxFpVuz6SprBqu4u4Nt0azTXwGhC2nFZqE6sXAxIljeR7gS1F3evj+OoiqKC 394qw8E3ebEIWe9OpxpP1mxwzZ949mi7dKyj81tb1+IdzcAfeVrhcCCNVCa05bdeVCB3 Q7/WuPcHp709TKUQGZdBTykLO4zer4iScaqOW6uISs/cVJrGY3MBw9L6p5lw06IHK0wY QCL7nXcrVayHrAU87n3b+iQAKdzVc5Qqq0K35NsBfrOp5gpT7WZELEhMJXMiuFh2MiLs gTMFQzLvwOJrRt8bbZK9f57AgJ0BwATNt9izz4nKAsO+lvaiBFo7qQapqrTpnyL6x/oU GHFA== X-Gm-Message-State: ANoB5pkaZU9XX+uUvY2OKqGSLOAHtieXDPQHtPwc2VZyamQyER486Nq+ S3rD+etufzjjGRcX0qTR3v+1bQ== X-Received: by 2002:adf:ec4b:0:b0:242:803:538c with SMTP id w11-20020adfec4b000000b002420803538cmr15569253wrn.656.1669806702599; Wed, 30 Nov 2022 03:11:42 -0800 (PST) Received: from phenom.ffwll.local (212-51-149-33.fiber7.init7.net. [212.51.149.33]) by smtp.gmail.com with ESMTPSA id w12-20020a05600c474c00b003b435c41103sm6798614wmo.0.2022.11.30.03.11.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Nov 2022 03:11:40 -0800 (PST) Date: Wed, 30 Nov 2022 12:11:38 +0100 From: Daniel Vetter To: =?iso-8859-1?Q?Andr=E9?= Almeida Cc: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, linux-kernel@vger.kernel.org, kernel-dev@igalia.com, alexander.deucher@amd.com, contactshashanksharma@gmail.com, amaranath.somalapuram@amd.com, christian.koenig@amd.com, pierre-eric.pelloux-prayer@amd.com, Simon Ser , Rob Clark , Andrey Grodzovsky , Pekka Paalanen , Daniel Vetter , Daniel Stone , 'Marek =?utf-8?B?T2zFocOhayc=?= , Dave Airlie , "Pierre-Loup A . Griffais" Subject: Re: [PATCH v3 0/2] drm: Add GPU reset sysfs Message-ID: Mail-Followup-To: =?iso-8859-1?Q?Andr=E9?= Almeida , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, linux-kernel@vger.kernel.org, kernel-dev@igalia.com, alexander.deucher@amd.com, contactshashanksharma@gmail.com, amaranath.somalapuram@amd.com, christian.koenig@amd.com, pierre-eric.pelloux-prayer@amd.com, Simon Ser , Rob Clark , Andrey Grodzovsky , Pekka Paalanen , Daniel Stone , 'Marek =?utf-8?B?T2zFocOhayc=?= , Dave Airlie , "Pierre-Loup A . Griffais" References: <20221125175203.52481-1-andrealmeid@igalia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20221125175203.52481-1-andrealmeid@igalia.com> X-Operating-System: Linux phenom 5.19.0-2-amd64 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Nov 25, 2022 at 02:52:01PM -0300, Andr? Almeida wrote: > This patchset adds a udev event for DRM device's resets. > > Userspace apps can trigger GPU resets by misuse of graphical APIs or driver > bugs. Either way, the GPU reset might lead the system to a broken state[1], that > might be recovered if user has access to a tty or a remote shell. Arguably, this > recovery could happen automatically by the system itself, thus this is the goal > of this patchset. > > For debugging and report purposes, device coredump support was already added > for amdgpu[2], but it's not suitable for programmatic usage like this one given > the uAPI not being stable and the need for parsing. > > GL/VK is out of scope for this use, giving that we are dealing with device > resets regardless of API. > > A basic userspace daemon is provided at [3] showing how the interface is used > to recovery from resets. > > [1] A search for "reset" in DRM/AMD issue tracker shows reports of resets > making the system unusable: > https://gitlab.freedesktop.org/drm/amd/-/issues/?search=reset > > [2] https://lore.kernel.org/amd-gfx/20220602081538.1652842-2-Amaranath.Somalapuram@amd.com/ > > [3] https://gitlab.freedesktop.org/andrealmeid/gpu-resetd > > v2: https://lore.kernel.org/dri-devel/20220308180403.75566-1-contactshashanksharma@gmail.com/ > > Andr? Almeida (1): > drm/amdgpu: Add work function for GPU reset event > > Shashank Sharma (1): > drm: Add GPU reset sysfs event This seems a bit much amd specific, and a bit much like an ad-hoc stopgap. On the amd specific piece: - amd's gpus suck the most for gpu hangs, because aside from the shader unblock, there's only device reset, which thrashes vram and display and absolutely everything. Which is terrible. Everyone else has engine only reset since years (i.e. doesn't thrash display or vram), and very often even just context reset (i.e. unless the driver is busted somehow or hw bug, malicious userspace will _only_ ever impact itself). - robustness extensions for gl/vk already have very clear specifications of all cases of reset, and this work here just ignores that. Yes on amd you only have device reset, but this is drm infra, so you need to be able to cope with ctx reset or reset which only affected a limited set of context. If this is for compute and compute apis lack robustness extensions, then those apis need to be fixed to fill that gap. - the entire deamon thing feels a bit like overkill and I'm not sure why it exists. I think for a start it would be much simpler if we just have a (per-device maybe) sysfs knob to enable automatic killing of process that die and which don't have arb robustness enabled (for gl case, for vk case the assumption is that _every_ app supports VK_DEVICE_LOST and can recover). Now onto the ad-hoc part: - Everyone hand-rolls ad-hoc gpu context structures and how to associate them with a pid. I think we need to stop doing that, because it's just endless pain and prevents us from building useful management stuff like cgroups for drivers that work across drivers (and driver/vendor specific cgroup wont be accepted by upstream cgroup maintainers). Or gpu reset events and dumps like here. This is going to be some work unforutnately. - I think the best starting point is the context structure drm/scheduler already has, but it needs some work: * untangling it from the scheduler part, so it can be used also for compute context that are directly scheduled by hw * (amd specific) moving amdkfd over to that context structure, at least internally * tracking the pid in there - I think the error dump facility should also be integrated into this. Userspace needs to know which dump is associated with which reset event, so that remote crash reporting works correctly. - ideally this framework can keep track of impacted context so that drivers don't have to reinvent the "which context are impacted" robustness ioctl book-keeping all on their own. For amd gpus it's kinda easy, since the impact is "everything", but for other gpus the impact can be all the way from "only one context" to "only contexts actively running on $set_of_engines" to "all the context actively running" to "we thrashed vram, everything is gone" - i915 has a bunch of this already, but I have honestly no idea whether it's any use because i915-gem is terminally not switching over to drm/scheduler (it needs a full rewrite, which is happening somewhere). So might only be useful to look at to make sure we're not building something which only works for full device reset gpus and nothing else. Over the various generations i915 has pretty much every possible gpu reset options you can think of, with resulting different reporting requirements to make sure robustness extensions work correctly. - pid isn't enough once you have engine/context reset, you need pid (well drm_file really, but I guess we can bind those to pid somehow) and gpu ctx id. Both gl and vk allow you to allocate limitless gpu context on the same device, and so this matters. - igt for this stuff. Probably needs some work to generalize the i915 infra for endless batchbuffers so that you can make very controlled gpu hangs. Cheers, Daniel > drivers/gpu/drm/amd/amdgpu/amdgpu.h | 4 +++ > drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 30 ++++++++++++++++++++++ > drivers/gpu/drm/drm_sysfs.c | 26 +++++++++++++++++++ > include/drm/drm_sysfs.h | 13 ++++++++++ > 4 files changed, 73 insertions(+) > > -- > 2.38.1 > -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch