Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <d7264c5a-29b4-0fb3-153b-673a8a73d635@amd.com>
Date:   Mon, 24 Apr 2023 09:03:13 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.10.0
Subject: Re: [PATCH] drm/amdgpu: Mark contexts guilty for any reset type
Content-Language: en-US
To:     =?UTF-8?Q?Andr=c3=a9_Almeida?= <andrealmeid@igalia.com>,
        dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org,
        linux-kernel@vger.kernel.org
Cc:     kernel-dev@igalia.com, alexander.deucher@amd.com,
        pierre-eric.pelloux-prayer@amd.com,
        =?UTF-8?B?J01hcmVrIE9sxaHDoWsn?= <maraeo@gmail.com>,
        "Tuikov, Luben" <Luben.Tuikov@amd.com>
References: <20230424014324.218531-1-andrealmeid@igalia.com>
From:   =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>
In-Reply-To: <20230424014324.218531-1-andrealmeid@igalia.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cGdkM3BxMm5wZkZISkdjVjdyK0JISmc5MlFCSGFCcWJKOHcwU3dYMnkvcm8z?=
 =?utf-8?B?eUhOS2o0czY5SEJaVTNSWkpCV0xWd0hkSXBCckxyUktHVEppUVF5S0tVWkRk?=
 =?utf-8?B?ZVJZSEc3V3NYWURlRnp4aUdISHM0TXN5UVNjTG9GL2EwUmVuWDhKMlBSSnJn?=
 =?utf-8?B?WVNQNDh4RitRN3RPL2tlNE1tdS9YdGhsK2NHNHBKT01FWW9udWhCWmEraG56?=
 =?utf-8?B?cmYzNTlCUHpXRHR6bmYzSklqcEhEYTlncHBpM3FEcjhMa2ZCWE5pd2NmQTJj?=
 =?utf-8?B?OFlOUGtpa2srTDRXVWR5UzhYamVjUGtBS3hJTTdIVlozUTZ3cDFhdDdWTjk2?=
 =?utf-8?B?dzE1VDYvbi9EczVzY1RKVy96Y0I3ZFlGUnV2Rzh2ZlNXdTM3TFZ1eHhxOWE4?=
 =?utf-8?B?WXo0RzN2UDExR0xISTlsWFB6TmJXVk54NTgzdDZpL3RTdWsvZURSTDdSUVVU?=
 =?utf-8?B?TGkvSVFTS2xRTVpZUWo5eTJzdDM5RzNJcW1IekdSWERwM3M3V1RSY3Fvd2FP?=
 =?utf-8?B?UFM4c053dVNJOHB1SEpsWmdyanNjY1dBazI5aVhLSXkwR09uRWJSMURsOUFP?=
 =?utf-8?B?Umw0dzlVaGZjL0kvUTA1OWxNQzVESTJGdDNFcmpONEk4QUxtNTFPUUZSYjNw?=
 =?utf-8?B?YTVybXJlMHYzdTFoZWc2R2lRbXRaWWlMTHp2NWJxU0lDMWNXN1pCSzZYbFJ0?=
 =?utf-8?B?V05mMkt6S3laM1dqQnR2d1I0ZG9BV2Z4RFkyWE45UXFYcGhiMmVpNlRpRXJG?=
 =?utf-8?B?dU05YktMbHFtQ1hlTFpqZEEwdHp6dXVuQ1NNais4ZXJpeWt2c1hVWlc4V1ph?=
 =?utf-8?B?SWZYdFFaNjhBS1pFNDFFSS9EbCswTXgzLy9ha01nRENXSklYQmNuQzk3UVhQ?=
 =?utf-8?B?Smo5TklCeEw0aXJtMVh0aHRFdWIxNFRRRjJjRnNwL242L1JMNWpjM1dyNlRt?=
 =?utf-8?B?STRvN25NbGF2R2l2SUJmTEloR3FhSktwcGxBWkJrWnNlTmJvTnovWVpCbTJ3?=
 =?utf-8?B?M3pYOXdNNnhMYURVSUl5VGpFakN5OWpqTHRYS0tOMTBscWs2ZXV0dXpLbVhh?=
 =?utf-8?B?UnlsOW5NK1VMbW1zcVRrNGg2MXNMbk9jYmtSRXIzTW1mc3BTOCtncjBreGJE?=
 =?utf-8?B?ditDWUVMbGhZb3F6K0ViTi9EeXM4OXYycGNCSG80b243bVVyR09nK1dldTJ4?=
 =?utf-8?B?L012NmN2ajkydDd4MXl1MGFCMVNYdzh3Z1B3OGZYdXcxN2NkWmtJYTJZUDg0?=
 =?utf-8?B?MkQ3VFBTT2VSRUMwSkZ1Y0tCeG9OVkRtN0FOREhUOURyQ0drMmEzbEhmYVh5?=
 =?utf-8?B?Vk4vcEZwYXkvb2wwTTcxajVjL0xwVkZxZENIWCtuVmFOSlJuV2JjTFJiNFZa?=
 =?utf-8?B?N0tJejY1cUdBYmJ4ZWRDOWt3MWMxR3NBWisyZE9pYjZBMldGUmZkK2pXSGlw?=
 =?utf-8?B?WTBYTm9IYStJQjNWcng0eFZ1Z3NiTTJnMFBMUXBPdEphQ2xvbm5rbyt3VjdT?=
 =?utf-8?B?ZXVXKy9xZFBJZ3QwZnVVMTlVS0dFRmR0MCtLc2xBQWNHVEl3RlA5S3pRR1l3?=
 =?utf-8?B?OUd3dFlZUytYaUJ4bGJQS3lSSGQxYzl5T1o5UnpUUDRMbVlNaThrTzNNcWEv?=
 =?utf-8?B?YWVVMURYOEtmZ1Y3V0w4b0wwVksvalFMbXFXSmxENkk3MVlvZHIvMDFmUEtX?=
 =?utf-8?B?WWl0ZXhSSHZEL21ydlQ2SnZDTU9zd2ZLbmhKN0hFNFRhdkMrU25NWEdQbFIx?=
 =?utf-8?B?WXlXcDYzMUJsODdmeEFTamlYOUhjU3p6TWZSZExzeW1ZNUpESkdlUjNLL1V5?=
 =?utf-8?B?S2JmNlB4RGNSR0N2S3c5NGFDZS9RZm1XcjR3K0hRMkhHTUpxNFNDalh2NUMx?=
 =?utf-8?B?SGpwbzc2OTVHMmhvcENYSXFsb2g2UEdlNkpDUURqaEZXaVlJMEJEdlJ1NWg0?=
 =?utf-8?B?VDlyNW1wUzFKVE11aWJNVlJIL1FVY3ZxdXdlNEpGN2RPOUdrc3F6TWR3cVZp?=
 =?utf-8?B?TGhsZlJpVzVyNXhsV1RWMHhmbm1NUFVGQzBMRk41bDF1c21XalBMaFZINWF1?=
 =?utf-8?B?ejJGbDRuL0JoUGlnb255a2FvSWNWcFFUOUpFVnF2Z3Q0ZU0zRU53akF6djlS?=
 =?utf-8?Q?Icis=3D?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5674ea33-98ba-4dca-7339-08db4491fc46
X-MS-Exchange-CrossTenant-AuthSource: BN8PR12MB3587.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Apr 2023 07:03:20.0877
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: I4EaMrF5zVT9/jm/455hUAjl2N0ydtOhVM4ed48Pw+sNi5b2cfZHDGDJH7SR1hfz
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB6744
X-Spam-Status: No, score=-3.9 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,NICE_REPLY_A,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Am 24.04.23 um 03:43 schrieb André Almeida:
> When a DRM job timeout, the GPU is probably hang and amdgpu have some
> ways to deal with that, ranging from soft recoveries to full device
> reset. Anyway, when userspace ask the kernel the state of the context
> (via AMDGPU_CTX_OP_QUERY_STATE), the kernel reports that the device was
> reset, regardless if a full reset happened or not.
>
> However, amdgpu only marks a context guilty in the ASIC reset path. This
> makes the userspace report incomplete, given that on soft recovery path
> the guilty context is not told that it's the guilty one.
>
> Fix this by marking the context guilty for every type of reset when a
> job timeouts.

The guilty handling is pretty much broken by design and only works 
because we go through multiple hops of validating the entity after the 
job has already been pushed to the hw.

I think we should probably just remove that completely and use an 
approach where we check the in flight submissions in the query state 
IOCTL. See my other patch on the mailing list regarding that.

Additional to that I currently didn't considered soft-recovered 
submissions as fatal and continue accepting submissions from that 
context, but already wanted to talk with Marek about that behavior.

Regards,
Christian.

>
> Signed-off-by: André Almeida <andrealmeid@igalia.com>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 3 ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_job.c    | 8 +++++++-
>   2 files changed, 7 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index ac78caa7cba8..ea169d1689e2 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -4771,9 +4771,6 @@ int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
>   
>   	amdgpu_fence_driver_isr_toggle(adev, false);
>   
> -	if (job && job->vm)
> -		drm_sched_increase_karma(&job->base);
> -
>   	r = amdgpu_reset_prepare_hwcontext(adev, reset_context);
>   	/* If reset handler not implemented, continue; otherwise return */
>   	if (r == -ENOSYS)
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
> index c3d9d75143f4..097ed8f06865 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
> @@ -51,6 +51,13 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
>   	memset(&ti, 0, sizeof(struct amdgpu_task_info));
>   	adev->job_hang = true;
>   
> +	amdgpu_vm_get_task_info(ring->adev, job->pasid, &ti);
> +
> +	if (job && job->vm) {
> +		DRM_INFO("marking %s context as guilty", ti.process_name);
> +		drm_sched_increase_karma(&job->base);
> +	}
> +
>   	if (amdgpu_gpu_recovery &&
>   	    amdgpu_ring_soft_recovery(ring, job->vmid, s_job->s_fence->parent)) {
>   		DRM_ERROR("ring %s timeout, but soft recovered\n",
> @@ -58,7 +65,6 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
>   		goto exit;
>   	}
>   
> -	amdgpu_vm_get_task_info(ring->adev, job->pasid, &ti);
>   	DRM_ERROR("ring %s timeout, signaled seq=%u, emitted seq=%u\n",
>   		  job->base.sched->name, atomic_read(&ring->fence_drv.last_seq),
>   		  ring->fence_drv.sync_seq);