Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <3764d627-d632-5754-0bcc-a150c157d9f9@amd.com>
Date:   Wed, 12 Jul 2023 12:39:47 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Subject: Re: [PATCH 3/6] drm/amdgpu: Rework coredump to use memory dynamically
Content-Language: en-US
To:     Lucas Stach <l.stach@pengutronix.de>,
        =?UTF-8?Q?Andr=c3=a9_Almeida?= <andrealmeid@igalia.com>,
        dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org,
        linux-kernel@vger.kernel.org
Cc:     pierre-eric.pelloux-prayer@amd.com,
        =?UTF-8?B?J01hcmVrIE9sxaHDoWsn?= <maraeo@gmail.com>,
        =?UTF-8?Q?Timur_Krist=c3=b3f?= <timur.kristof@gmail.com>,
        michel.daenzer@mailbox.org,
        Samuel Pitoiset <samuel.pitoiset@gmail.com>,
        kernel-dev@igalia.com, alexander.deucher@amd.com
References: <20230711213501.526237-1-andrealmeid@igalia.com>
 <20230711213501.526237-4-andrealmeid@igalia.com>
 <e488da74-af52-62eb-d601-0e8a13cf0e87@amd.com>
 <0e7f2b0cc29ac77d4a55d0de6a66c477d867fbf7.camel@pengutronix.de>
From:   =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>
In-Reply-To: <0e7f2b0cc29ac77d4a55d0de6a66c477d867fbf7.camel@pengutronix.de>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bDMrWjM1cDJzMnhxMVVhNCtWU0ZKNzhOZGttNmhrY0FnUzNyakRyK3Nld1Vh?=
 =?utf-8?B?NnJFVUV5Zk9FdjVic0JLZlJkbnRjSmNyOGg5L0VXVlNIY3AxRjB1U0l3M3Bz?=
 =?utf-8?B?RnlNMm1LUmVzalZDUC8wRHNDb2djcmsrZUxXRDFnNE1SdDBhckFVQXhvUDZy?=
 =?utf-8?B?NXNTaHFZNitaUDlUYk50TS9sdlRDQm5KbXhTUTdMSWpzR2hxTWhMWWRndFp6?=
 =?utf-8?B?T1lmalpBM2xQRlYvdW9vTlVaWi9WZlZnZTZGcXZUTDcwSDJjbGZoczVFQUhQ?=
 =?utf-8?B?TTRhOUo2K09qcFY3eEhnL2hsNzdacE5sYmdlR1R6WkYwSThsMHYrK25VVGk5?=
 =?utf-8?B?RERJVkFmMUpENzZRR1crdWFUYXgxSU1Yb2NFbmZ0b3ljNlRPNnB3dnJFMGhv?=
 =?utf-8?B?bVF5NnBoaGZzUjdEMStEK29PZnF1TDVYQll3d3ZIeDc3K20vbkFNRDZRMHdl?=
 =?utf-8?B?SVRQbEt6dlBBZ3VLbFdIS011Y1ZXZEdtMURIVjJ3ZGRqZC9HbDFLQ3JjZk5q?=
 =?utf-8?B?UzFUVE9zYy9MSkxxWW9qYUxtTEd5V2pqQ1kzZzYwc1BsUzlDYmtnMUpFVW1R?=
 =?utf-8?B?TXNmZE5sOW94eThEMkYxTW8vZmYrTEozanNpdHg0aWIyYWx6SjZjek1WaDdw?=
 =?utf-8?B?SGZXRkI5TlpwRlZ2ZDRoMVlVRk04bVZ6bVJLcFdOeHZvV3lkaDBmRno2dEMr?=
 =?utf-8?B?QThoWW1iUCsyLzY4SGx3S3BZNFhheXBITEZySDcrNW10YTVJNzE2c0FUZ08v?=
 =?utf-8?B?VnhHNE9SUitER0JtQjJ5K3ljSVdXQjFQVW5UdnRiVkdrdW1FR0JwQ0hoaXUz?=
 =?utf-8?B?aXQrOUp6M2s3cldaRjJPeUM1bmJpeElvOWdjWWZKTHBibkVsRjBRWitnanVk?=
 =?utf-8?B?UFJENkNRUGpERkF4M0hxM1p0eVVMczBpR0dFdzREQnBlTnJ4djQ0Wk1weVZO?=
 =?utf-8?B?QlU3WXZLSmF2OTRVaEQwenpWUDNHMzJielJXdmUrK1RlQlh5QTg1TDlzWVVo?=
 =?utf-8?B?RGp1Z3pSRTdWK3BlOUJCTGE2akJIbW5qQlkwVGhyNnBFQnlid2g4SVR1NlpJ?=
 =?utf-8?B?aWpmWE96R3hEeFpWbEF1RzF5MVRMWE1tNDlHZEczWHFDS1NsMHZEdDlrUFla?=
 =?utf-8?B?MVdUUmkwa29GdVJpbmpTTkJBQ0RwZ3pYdTNhLzdWRVczZTZHSzcydzRtMitO?=
 =?utf-8?B?WFVtbHRLSFY4NTBoVEhMbmVELzl3aERlSGlYNFRaQWpkekhwMHJKdEVObzl1?=
 =?utf-8?B?cHlVbzJsaUtuNGtLdCt6cjZDNks4SXV1K2JMcVZhT1htRmtTUG56UkRseDdk?=
 =?utf-8?B?ZmJLWHFLeXhlSW5wQnRjVGVnbDMySDlpWUlhbmdLQk9jOFZxd0NkeG1tLy9D?=
 =?utf-8?B?TG1SUzRxVmowbXNhTkhITURXQjdRV2ZKb2lGSmM4dTNhdlpqbXNpQ3FqNDBt?=
 =?utf-8?B?aDJ0YjBiOGpENWlOYW1OY09yNXVIbXA5UXRaTXdxdk1oS1pvYzg3bEhuMjR0?=
 =?utf-8?B?QUloOFNUSW00WVc1RzdwWkFQMmpuRVhqQ2Jxa0psUk9oMi9nU0E0RlZOeWxR?=
 =?utf-8?B?MUM1bWl1YkZodkoyQkZKZ2JGdE1QSnZ3S3ZTVmZnNExXcHdRWWhSUldOTWdq?=
 =?utf-8?B?OU1LcWR3dVdCSlRBeUx3bUd6a290Sm9mMXJ1dVQ2QWJOdlpQclh4eEtyUW5B?=
 =?utf-8?B?YksxVjBSeHl1SHNsNThsOXVjV0czVVIxczNueWFMWFFlVkxGellTYzJIaVJD?=
 =?utf-8?B?MTY1OU1WNzk1MDJPV0QvRnM2clNBajh4akpGUXFwaDFaelgwaXRRbzdyaTlk?=
 =?utf-8?B?bHV3NnNFRHg2bnNvamVRdm5SVzU1NTVVaEk0Y1JwcjlBNHcvS0RKQWJvU2U3?=
 =?utf-8?B?dmI4K0RXaVBmNWJLQk5tYkloOUJJZnRvTFBqa3llbEg0U2Z2aXY4RnNwcnZR?=
 =?utf-8?B?amhEK2lDTytXdmR1VGRXRm9oTnJBcSttUnBJSEhUWjRTMHpDd2xPZkVOUWxn?=
 =?utf-8?B?NFVQSE1QUDgvaWxXUTNrVVZCTTdCN2IyNnNXclQ5bGg4U3dQWGFoRjJzS2Ny?=
 =?utf-8?B?Mk5RWjdFbmJnbmJMVXdBNWR5cXNUaVYzWDlHdGxTZHkwWWlDdFRnNXJuWHhQ?=
 =?utf-8?B?VVB2V3ZFVFViNDN1dGJUQ0d3ZTByMnVFV1RuVHdFNXA2QjJCR0xBZXlBbFFH?=
 =?utf-8?Q?YP3VONpSmiAJqH7f6V8FSYbQPGEeLv5kGYXS//IesaiG?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f558ac75-a070-4b78-ae85-08db82c45379
X-MS-Exchange-CrossTenant-AuthSource: BN8PR12MB3587.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jul 2023 10:39:53.0300
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: +1dHCtmXUHWn/YG0z8Uhw4HPBp7hx9XHXlwfGWXsPn4CrM7PeylHzXcEEoeT7jEV
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5349
X-Spam-Status: No, score=-1.2 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,NICE_REPLY_A,
        RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=no autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Am 12.07.23 um 10:59 schrieb Lucas Stach:
> Am Mittwoch, dem 12.07.2023 um 10:37 +0200 schrieb Christian König:
>> Am 11.07.23 um 23:34 schrieb André Almeida:
>>> Instead of storing coredump information inside amdgpu_device struct,
>>> move if to a proper separated struct and allocate it dynamically. This
>>> will make it easier to further expand the logged information.
>> Verry big NAK to this. The problem is that memory allocation isn't
>> allowed during a GPU reset.
>>
>> What you could do is to allocate the memory with GFP_ATOMIC or similar,
>> but for a large structure that might not be possible.
>>
> I'm still not fully clear on what the rules are here. In etnaviv we do
> devcoredump allocation in the GPU reset path with __GFP_NOWARN |
> __GFP_NORETRY, which means the allocation will kick memory reclaim if
> necessary, but will just give up if no memory could be made available
> easily. This satisfies the forward progress guarantee in the absence of
> successful memory allocation, which is the most important property in
> this path, I think.
>
> However, I'm not sure if the reclaim could lead to locking issues or
> something like that with the more complex use-cases with MMU notifiers
> and stuff like that. Christian, do you have any experience or
> information that would be good to share in this regard?

Yeah, very good question.

__GFP_NORETRY isn't sufficient as far as I know. Reclaim must be 
completely suppressed to be able to allocate in a GPU reset handler.

Daniel added lockdep annotation to some of the dma-fence signaling paths 
and this yielded quite a bunch of potential deadlocks.

It's not even that reclaim itself waits for dma_fences (that can happen, 
but is quite unlikely), but rather that reclaim needs locks and under 
those locks we then wait for dma_fences.

We should probably add a define somewhere which documents that 
(GFP_ATOMIC | __NO_WARN) should be used in the GPU reset handlers when 
allocating memory for coredumps.

Regards,
Christian.

>
> Regards,
> Lucas
>
>> Regards,
>> Christian.
>>
>>> Signed-off-by: André Almeida <andrealmeid@igalia.com>
>>> ---
>>>    drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 14 +++--
>>>    drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 65 ++++++++++++++--------
>>>    2 files changed, 51 insertions(+), 28 deletions(-)
>>>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
>>> index dbe062a087c5..e1cc83a89d46 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
>>> @@ -1068,11 +1068,6 @@ struct amdgpu_device {
>>>    	uint32_t                        *reset_dump_reg_list;
>>>    	uint32_t			*reset_dump_reg_value;
>>>    	int                             num_regs;
>>> -#ifdef CONFIG_DEV_COREDUMP
>>> -	struct amdgpu_task_info         reset_task_info;
>>> -	bool                            reset_vram_lost;
>>> -	struct timespec64               reset_time;
>>> -#endif
>>>    
>>>    	bool                            scpm_enabled;
>>>    	uint32_t                        scpm_status;
>>> @@ -1085,6 +1080,15 @@ struct amdgpu_device {
>>>    	uint32_t			aid_mask;
>>>    };
>>>    
>>> +#ifdef CONFIG_DEV_COREDUMP
>>> +struct amdgpu_coredump_info {
>>> +	struct amdgpu_device		*adev;
>>> +	struct amdgpu_task_info         reset_task_info;
>>> +	struct timespec64               reset_time;
>>> +	bool                            reset_vram_lost;
>>> +};
>>> +#endif
>>> +
>>>    static inline struct amdgpu_device *drm_to_adev(struct drm_device *ddev)
>>>    {
>>>    	return container_of(ddev, struct amdgpu_device, ddev);
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>> index e25f085ee886..23b9784e9787 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>> @@ -4963,12 +4963,17 @@ static int amdgpu_reset_reg_dumps(struct amdgpu_device *adev)
>>>    	return 0;
>>>    }
>>>    
>>> -#ifdef CONFIG_DEV_COREDUMP
>>> +#ifndef CONFIG_DEV_COREDUMP
>>> +static void amdgpu_coredump(struct amdgpu_device *adev, bool vram_lost,
>>> +			    struct amdgpu_reset_context *reset_context)
>>> +{
>>> +}
>>> +#else
>>>    static ssize_t amdgpu_devcoredump_read(char *buffer, loff_t offset,
>>>    		size_t count, void *data, size_t datalen)
>>>    {
>>>    	struct drm_printer p;
>>> -	struct amdgpu_device *adev = data;
>>> +	struct amdgpu_coredump_info *coredump = data;
>>>    	struct drm_print_iterator iter;
>>>    	int i;
>>>    
>>> @@ -4982,21 +4987,21 @@ static ssize_t amdgpu_devcoredump_read(char *buffer, loff_t offset,
>>>    	drm_printf(&p, "**** AMDGPU Device Coredump ****\n");
>>>    	drm_printf(&p, "kernel: " UTS_RELEASE "\n");
>>>    	drm_printf(&p, "module: " KBUILD_MODNAME "\n");
>>> -	drm_printf(&p, "time: %lld.%09ld\n", adev->reset_time.tv_sec, adev->reset_time.tv_nsec);
>>> -	if (adev->reset_task_info.pid)
>>> +	drm_printf(&p, "time: %lld.%09ld\n", coredump->reset_time.tv_sec, coredump->reset_time.tv_nsec);
>>> +	if (coredump->reset_task_info.pid)
>>>    		drm_printf(&p, "process_name: %s PID: %d\n",
>>> -			   adev->reset_task_info.process_name,
>>> -			   adev->reset_task_info.pid);
>>> +			   coredump->reset_task_info.process_name,
>>> +			   coredump->reset_task_info.pid);
>>>    
>>> -	if (adev->reset_vram_lost)
>>> +	if (coredump->reset_vram_lost)
>>>    		drm_printf(&p, "VRAM is lost due to GPU reset!\n");
>>> -	if (adev->num_regs) {
>>> +	if (coredump->adev->num_regs) {
>>>    		drm_printf(&p, "AMDGPU register dumps:\nOffset:     Value:\n");
>>>    
>>> -		for (i = 0; i < adev->num_regs; i++)
>>> +		for (i = 0; i < coredump->adev->num_regs; i++)
>>>    			drm_printf(&p, "0x%08x: 0x%08x\n",
>>> -				   adev->reset_dump_reg_list[i],
>>> -				   adev->reset_dump_reg_value[i]);
>>> +				   coredump->adev->reset_dump_reg_list[i],
>>> +				   coredump->adev->reset_dump_reg_value[i]);
>>>    	}
>>>    
>>>    	return count - iter.remain;
>>> @@ -5004,14 +5009,34 @@ static ssize_t amdgpu_devcoredump_read(char *buffer, loff_t offset,
>>>    
>>>    static void amdgpu_devcoredump_free(void *data)
>>>    {
>>> +	kfree(data);
>>>    }
>>>    
>>> -static void amdgpu_reset_capture_coredumpm(struct amdgpu_device *adev)
>>> +static void amdgpu_coredump(struct amdgpu_device *adev, bool vram_lost,
>>> +			    struct amdgpu_reset_context *reset_context)
>>>    {
>>> +	struct amdgpu_coredump_info *coredump;
>>>    	struct drm_device *dev = adev_to_drm(adev);
>>>    
>>> -	ktime_get_ts64(&adev->reset_time);
>>> -	dev_coredumpm(dev->dev, THIS_MODULE, adev, 0, GFP_KERNEL,
>>> +	coredump = kmalloc(sizeof(*coredump), GFP_KERNEL);
>>> +
>>> +	if (!coredump) {
>>> +		DRM_ERROR("%s: failed to allocate memory for coredump\n", __func__);
>>> +		return;
>>> +	}
>>> +
>>> +	memset(coredump, 0, sizeof(*coredump));
>>> +
>>> +	coredump->reset_vram_lost = vram_lost;
>>> +
>>> +	if (reset_context->job && reset_context->job->vm)
>>> +		coredump->reset_task_info = reset_context->job->vm->task_info;
>>> +
>>> +	coredump->adev = adev;
>>> +
>>> +	ktime_get_ts64(&coredump->reset_time);
>>> +
>>> +	dev_coredumpm(dev->dev, THIS_MODULE, coredump, 0, GFP_KERNEL,
>>>    		      amdgpu_devcoredump_read, amdgpu_devcoredump_free);
>>>    }
>>>    #endif
>>> @@ -5119,15 +5144,9 @@ int amdgpu_do_asic_reset(struct list_head *device_list_handle,
>>>    					goto out;
>>>    
>>>    				vram_lost = amdgpu_device_check_vram_lost(tmp_adev);
>>> -#ifdef CONFIG_DEV_COREDUMP
>>> -				tmp_adev->reset_vram_lost = vram_lost;
>>> -				memset(&tmp_adev->reset_task_info, 0,
>>> -						sizeof(tmp_adev->reset_task_info));
>>> -				if (reset_context->job && reset_context->job->vm)
>>> -					tmp_adev->reset_task_info =
>>> -						reset_context->job->vm->task_info;
>>> -				amdgpu_reset_capture_coredumpm(tmp_adev);
>>> -#endif
>>> +
>>> +				amdgpu_coredump(tmp_adev, vram_lost, reset_context);
>>> +
>>>    				if (vram_lost) {
>>>    					DRM_INFO("VRAM is lost due to GPU reset!\n");
>>>    					amdgpu_inc_vram_lost(tmp_adev);