Message-ID: <37e78013-5ed6-e80f-4a1c-1e61bed59735@amd.com>
Date:   Mon, 23 Jan 2023 21:38:11 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.4.2
Subject: Re: [RFC PATCH] drm: Create documentation about device resets
Content-Language: en-US
To:     =?UTF-8?Q?Andr=c3=a9_Almeida?= <andrealmeid@igalia.com>,
        dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org,
        linux-kernel@vger.kernel.org
Cc:     kernel-dev@igalia.com, alexander.deucher@amd.com,
        contactshashanksharma@gmail.com, amaranath.somalapuram@amd.com,
        pierre-eric.pelloux-prayer@amd.com,
        Simon Ser <contact@emersion.fr>,
        Rob Clark <robdclark@gmail.com>,
        Pekka Paalanen <ppaalanen@gmail.com>,
        Daniel Vetter <daniel@ffwll.ch>,
        Daniel Stone <daniel@fooishbar.org>,
        =?UTF-8?B?J01hcmVrIE9sxaHDoWsn?= <maraeo@gmail.com>,
        Dave Airlie <airlied@gmail.com>,
        "Pierre-Loup A . Griffais" <pgriffais@valvesoftware.com>
References: <20230123202646.356592-1-andrealmeid@igalia.com>
 <20230123202646.356592-2-andrealmeid@igalia.com>
From:   =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>
In-Reply-To: <20230123202646.356592-2-andrealmeid@igalia.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RTJMeGcrb2pJbmdyR05PWWxyRHFLcVV6SndwVUlFLzNod2d6NlhyN2hybWcr?=
 =?utf-8?B?Mm8wM1ZNVG12WVdsQ2RUZXkrQi92Qm50NllOSnNGWGJaQTY5OEdZNjFiK2Zt?=
 =?utf-8?B?ZzRqWm5zR0c0K3RUc0orbHlWL21SWkVySXNUVFIveWpFN0o5YmQxbXJYR29U?=
 =?utf-8?B?VTJsaW5LTU9qamFYaUVycTIxQyt5Uy9lOEd0MTN2cWJvWmNvWkxGOGtjZ3hP?=
 =?utf-8?B?SVVjRkRLakFBbTJ1aEtHaG4wbml6NlMwaUJpUTNiSE5pZlB6TWtJaE13MXk1?=
 =?utf-8?B?ZmdTdlcwZXZRZVJaUk92TS96Wldyc0U5bmxycjNvZEJOdy85eHZiRWlDK3N5?=
 =?utf-8?B?eWpwWGFtb0Jzd1ZDbjZ6Q09na1lRR25Mb1dhZjFZcU4vcWtRZER5ZDhxZGhM?=
 =?utf-8?B?NU9oakhNdXVlNy9pVk1hMHIxT3NnOGZEc1pYUjRsSXBhRVE2bWN6d2p1S1dM?=
 =?utf-8?B?VlpDTGpId2dIYVhrTEk0U0ZqT243blVveHUwV3VkU0lLUlVhdzBIUWRxWG1y?=
 =?utf-8?B?TUFKbHQxYTRjOFlUUzN6QkR2a3g0SSsxTzhEeFFzQ1cvZFQveFQveFdGRHEx?=
 =?utf-8?B?MlJxSk5XVlZXMnZtVEd2MFRTLzIvZmYrWXFWZEQ0dTJwVFZCcTVDZ1RXYzZ0?=
 =?utf-8?B?dFR1SFlnU2NrL0Jjek9obEdoV1BKcXRWS3JkOE42VkFTMk9JVXE1ZnVIcEx2?=
 =?utf-8?B?eGRPeTJJUzhkNnRURWdEemRrQzZ3QTFvbEtDOXhQYTc5dmplWGZNZFBMU1ds?=
 =?utf-8?B?VlZnaS9KcnYvT2NKaVowMS9KVFVVYjhaRVRmTlllNkFkRmZ2VFJOTnBDSDRh?=
 =?utf-8?B?bTF6VEN5RjBSRDJWY0dhaWl6Y2l1VE5HWFN3MlZ5c2xWS2tzUG1haTUwQ2R2?=
 =?utf-8?B?WEYzcG9QWUx5RWtOSGhINDF6bG9KQzJmNFRoL3RxL3g4emc1WWp3N3kyRmVT?=
 =?utf-8?B?NVoxbE5lcXpMT3ZSSHBsK0t0TWFVYkN3SW16UHp6c3EyeWdnU2dxM0pmNTF3?=
 =?utf-8?B?b2s1MWJMaTBrY0tyOFJGVTh6SXlDVmFGbHZJRkRsRktqS2VwVENwRlBMc3Iy?=
 =?utf-8?B?a2VaTW9jUktuM3hxSUNCbksrSE5MSUNDOEY0VjN5dFRoOHJWWWlmdDR5SHo3?=
 =?utf-8?B?L2VjVlVwaDErdUl5YXlRdFoyaU5iN2hOWmpDeU42dThlQVdOanpHSUxvMTY1?=
 =?utf-8?B?aThuU2JqdGNsV1lwQ3RCNC80Njg2QlJiaktDWHdtdUpMU3BzdVFPdEZsblFN?=
 =?utf-8?B?cWJKUG9YUE9qTXVxcGIzODR4MHFqSDRZUHg3Z0ZCN3JqaE1TWXlKYXpvOC82?=
 =?utf-8?B?c0JLS3NieFBQZW4vQi9qcXlGVlJicDRBTkxJL1VRcExQVG9hczYycjcvWDNt?=
 =?utf-8?B?WlhaL3VqMFliSWRYbU9iRmhMVFM2VmJQNkJBcEJVOHJTMUVZRlE0SktQSXQx?=
 =?utf-8?B?RnRFY1ZJVlEyU3BWVHY0cStZbGx4dkxsN1dQU0VCdjlFSDhzMTVkcWtOajlE?=
 =?utf-8?B?RWUzYzNKemU1ZlZrenlkY045eE9lOXVhTm9zcFlMOG9DbVZNUGQ3T2RhT2Ni?=
 =?utf-8?B?WVhTNUJGaDRwUTNXaVJqMnkrenlKWWIyYlQ1YTBOUmo2TlhPU3F1ZzduVXB1?=
 =?utf-8?B?a3JHRnlTOHF0eGF4R0ltS1JkV244RmN4WUF4TFo0LzVjYW4zWVJPOWE0L0c3?=
 =?utf-8?B?M0x0RW1sU2hZS0prZTJuVWFJdjlsRTNIT24xR0pkRWJVa2xIL1BoSUtPTmhu?=
 =?utf-8?B?YVF6UzRJcDdjM2F2eUxyeFB5Tjh6SDFqNXQyYiswYVRvMlRxRjZZc2xvSGdL?=
 =?utf-8?B?NDFnc1NXbkNJNUlnNi9mRlRzODJ4a1diZVAwZDQ4bWlYdHBteldSWFRQaHJk?=
 =?utf-8?B?R2lXYVJwbjk1eXNhWng0T0xvRFJrV3BSRmFLT3drenZqQzRxOEc4TkI3dWM0?=
 =?utf-8?B?UDRtUWhUUDQrbWRjZ1dLaHdGMDN0eWRIMDhrQ2VYTko5NU1VMGdremdPcTlP?=
 =?utf-8?B?RXB1czVnaENMNE5PL2dYcHBSWVcvZjZ1YndubEtkeGZpUzVEbkhOUFU0T2Jq?=
 =?utf-8?B?TzIxS0RhVlFTZWJJYlFIaFJDVGZYUmpaNjFERncrc1VDRHlheklJRHlLenYz?=
 =?utf-8?B?QWpYQjVKK2RIdkROSW1NL3dvMDNxKzBUbUFxeXl2RmkyRmFrbmhNNWFsMnN0?=
 =?utf-8?Q?v28sSs8gwF4WuvHI/eYSCC94KygRORZuvT4Gg0zOhx9h?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e819c14f-3f70-49ab-8f13-08dafd81c29a
X-MS-Exchange-CrossTenant-AuthSource: BN8PR12MB3587.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Jan 2023 20:38:18.6363
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Bc1fScc7hckxR5BEdUdmqvjBnZp/7/waAlDcKeADcEN8blbTSZR5ze+xmiCKQbsO
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4186
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Am 23.01.23 um 21:26 schrieb André Almeida:
> Create a document that specifies how to deal with DRM device resets for
> kernel and userspace drivers.
>
> Signed-off-by: André Almeida <andrealmeid@igalia.com>
> ---
>   Documentation/gpu/drm-reset.rst | 51 +++++++++++++++++++++++++++++++++
>   Documentation/gpu/index.rst     |  1 +
>   2 files changed, 52 insertions(+)
>   create mode 100644 Documentation/gpu/drm-reset.rst
>
> diff --git a/Documentation/gpu/drm-reset.rst b/Documentation/gpu/drm-reset.rst
> new file mode 100644
> index 000000000000..0dd11a469cf9
> --- /dev/null
> +++ b/Documentation/gpu/drm-reset.rst
> @@ -0,0 +1,51 @@
> +================
> +DRM Device Reset
> +================
> +
> +The GPU stack is really complex and is prone to errors, from hardware bugs,
> +faulty applications and everything in the many layers in between. To recover
> +from this kind of state, sometimes is needed to reset the GPU. Unproper handling
> +of GPU resets can lead to an unstable userspace. This page describes what's the
> +expected behaviour from DRM drivers to do in those situations, from usermode
> +drivers and compositors as well.
> +
> +Robustness
> +----------
> +
> +First of all, application robust APIs, when available, should be used. This
> +allows the application to correctly recover and continue to run after a reset.
> +Apps that doesn't use this should be promptly killed when the kernel driver
> +detects that it's in broken state. Specifically guidelines for some APIs:
> +

> +- OpenGL: During a reset, KMD kill processes that haven't ARB Robustness
> +  enabled, assuming they can't recover.

This is a pretty clear NAK from my side to this approach. The KMD should 
never mess with an userspace process directly in such a way.

Instead use something like this "OpenGL: KMD signals the abortion of 
submitted commands and the UMD should then react accordingly and abort 
the application.".

> +- Vulkan: Assumes that every app is able to deal with ``VK_ERROR_DEVICE_LOST``,
> +  so KMD doesn't kill any. If it doesn't do it right, it's considered a broken
> +  application and UMD will deal with it.

Again, pleas remove the "KMD kill" reference.

> +
> +Kernel mode driver
> +------------------
> +
> +The KMD should be able to detect that something is wrong with the application

Please replace *should* with *must* here, this is mandatory or otherwise 
core memory management can run into deadlocks during reclaim.

Regards,
Christian.

> +and that a reset is needed to take place to recover the device (e.g. an endless
> +wait). It needs to properly track the context that is broken and mark it as
> +dead, so any other syscalls to that context should be further rejected. The
> +other contexts should be preserved when possible, avoid crashing the rest of
> +userspace. KMD can ban a file descriptor that keeps causing resets, as it's
> +likely in a broken loop.
> +
> +User mode driver
> +----------------
> +
> +During a reset, UMD should be aware that rejected syscalls indicates that the
> +context is broken and for robust apps the recovery should happen for the
> +context. Non-robust apps would be already terminated by KMD. If no new context
> +is created for some time, it is assumed that the recovery didn't work, so UMD
> +should terminate it.
> +
> +Compositors
> +-----------
> +
> +(In the long term) compositors should be robust as well to properly deal with it
> +errors. Init systems should be aware of the compositor status and reset it if is
> +broken.
> diff --git a/Documentation/gpu/index.rst b/Documentation/gpu/index.rst
> index b99dede9a5b1..300b2529bd39 100644
> --- a/Documentation/gpu/index.rst
> +++ b/Documentation/gpu/index.rst
> @@ -9,6 +9,7 @@ Linux GPU Driver Developer's Guide
>      drm-mm
>      drm-kms
>      drm-kms-helpers
> +   drm-reset
>      drm-uapi
>      drm-usage-stats
>      driver-uapi