Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <420fd11e-8a3e-54d1-8e9f-119507f7af48@amd.com>
Date:   Tue, 28 Mar 2023 12:29:44 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.8.0
Subject: Re: [Regression] drm/scheduler: track GPU active time per entity
Content-Language: en-US
To:     Danilo Krummrich <danilokrummrich@yahoo.de>, daniel@ffwll.ch,
        Dave Airlie <airlied@gmail.com>, luben.tuikov@amd.com,
        l.stach@pengutronix.de, Bagas Sanjaya <bagasdotme@gmail.com>,
        andrey.grodzovsky@amd.com, dri-devel@lists.freedesktop.org,
        linux-kernel@vger.kernel.org
References: <e3e07282-136f-5239-96ae-4aeb2d3c95d7.ref@yahoo.de>
 <e3e07282-136f-5239-96ae-4aeb2d3c95d7@yahoo.de>
From:   =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>
In-Reply-To: <e3e07282-136f-5239-96ae-4aeb2d3c95d7@yahoo.de>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MEN5eXdCc0JoUHIxdEgzTW9TeCtJK1Z3OExQQkp4SjFPWEJqeGxKUVl2aFJN?=
 =?utf-8?B?L0JxRnR2WjgxV3U3K1IyN0JpSjNVOFBteFVjRjI1M0toSXBzRFdjamRyOVhI?=
 =?utf-8?B?UnZjVXhyT2RYd01hQk1ZM1dCMHBna0xWMmwzcXlkbFRobWo4azY0Y3BpKzU2?=
 =?utf-8?B?bjJ6WDNiWkpWY0J6cGQweVpmQVZOclR6MHJaWUczZ1ZHUm9uTHQ4QjkyQXhI?=
 =?utf-8?B?U1NzNmhrUUtka09tK0hERytaVGlHalNibU0wL3pjSE8rUUUzWG1wSUR6eURB?=
 =?utf-8?B?cEtzQVVFTWlhK1crM3F2anJtNytKQ0Q4M0dqcTIxUTF2aGR4cjNHZE1wWkts?=
 =?utf-8?B?OTd4T08zYVdhdU4wMjlROU8xRVAwSGhLaXM0U2lGak5ETWNuSjNGSVRjZmVB?=
 =?utf-8?B?RTFKUzdZL3BsUmtYMDdxYVJOUU9HVGpGKzQ1OFFENWRrQVZoZEFaR2dRVFVr?=
 =?utf-8?B?SXN4RFhSZ1pDUHAyMEhNTmZ1NTFRZ0dJOG9JK3hzNTVJWlI5UVZEZWw1TGdI?=
 =?utf-8?B?dDMwZThUdm9YVWtEeitKT2FSalkwTzBlRHFmZDVwSEYzTTBOOXpOZHNpVDlP?=
 =?utf-8?B?WGlkL0dCWnVtblVzRytlRGYzY2E1UzlZTm1CeStCOVRiM2tJaWlZQy9lSlcy?=
 =?utf-8?B?WTNQRmlJWDNvY1Qrby9PbllURE5jV010d2dRMXZOaSs5M3EwamZpUFJsNW1L?=
 =?utf-8?B?d0NmMVFKU3lsNG93aVhGTlRRd0ZaS1NUV21UclJqTTFWRVh4QklJU1BFdUIz?=
 =?utf-8?B?aDluN2E1UXl4TUVyY2hxZzF5V3RQUGhvNWNwRncvekt1Z3YrS3B3cVR2Rkhp?=
 =?utf-8?B?a3dNWFl0dElud1ZvQ2VNY01wREhpeW5NWXpra09GR2Q3eHF4dnh2SFJGVnBZ?=
 =?utf-8?B?V2h4YXVsY1p1SDNrU3A1MlBMQkNYYkZVOHo1cStsTzRLODV4WHRvQTIvR1h5?=
 =?utf-8?B?enk0aCt1V3JTdFpWZEJRT0JLMXE0ZXVoK1hKODZwbzFrWXFjY3RJakdocmV2?=
 =?utf-8?B?Z2dxOGhVTSsxRVFCM3N2YlFzUXVGZFVwaytaeVJUd2FFYVllZkpHTWd1TzFw?=
 =?utf-8?B?Sld6V1dtVm8zNHFETFgxMk9kSHI1dmR1VVhnVVdJMklRWms0Z21jV1ZvMC9T?=
 =?utf-8?B?TWg1a0lteXVYUHJoV2gySlVzRnZrWkdJMHZ3WlU2dkRmS0tNRklYUnp3OGV4?=
 =?utf-8?B?QWZEdHNJb3JWWjc1WEdJT2tMaS9sdEFjeDNGbEpuYXVDSUdqREl6SXZnUWNx?=
 =?utf-8?B?dUNQak53U0laQ2RBZU1TdEJlMjByODNvVXlQRzFrWGc5N1JoUzFab0FIeGlY?=
 =?utf-8?B?M1ZTS1dlekFpNC9PZitxQVlqNTZzKzFnMGgvM2NTVUt6aXNLcnVhY2p5L2tJ?=
 =?utf-8?B?QnRMNTE0TDM3UGdtRGJCb1J2OERiQ2hNS25QNDBiZWswR2prOG90cHhYWlJE?=
 =?utf-8?B?MGNEQk84SGpqNDc4YWtTWnNwWHNQaTdhRTBRMnpIanQzVUZYc25JUzN6S2NZ?=
 =?utf-8?B?Qk8xNi93encvdC9TNjdmS1JpUjBTNHFhR05ydXZYTnFnU1ZGOTJUajNzRm1T?=
 =?utf-8?B?ZHlOUXlRcEFIYW9CS1V2QS9jcm8yVmxoOHIyT0tkM1c4dW82ekNDMTlFczd5?=
 =?utf-8?B?OUY2ZjYwaGJUdmJ5NmhjU2lXUGR4SUtyS0R2L05IUmZlQ2J4SjQ4M3p5SlUy?=
 =?utf-8?B?Snd6VytucXpJbkFXOTdmMmp4QWFtNDdOTUZkN0RMKzlBZzY4SFFVVFlqZnpQ?=
 =?utf-8?B?NVFXN1BxNnhWME9ZbkpLQmZraWttZVk1NkVGR2dGUFVYL2dudmxrSTlyN0x5?=
 =?utf-8?B?MEUvOFBnQjB1NDlmMUMyWmkxNlc2Ym80SmxpRURtdVNObEZKZStYK1NQQUli?=
 =?utf-8?B?d2Z0UjNsOW1URUhQN1BvUU5jaFg5T0FPRGJpMW03VC9jUHhzdnh4WUJqQUFk?=
 =?utf-8?B?Nnl6VHVSTThkZzBicmZoVTljcC9pZWNkTlNsYmM2TUdYN3V6dU5vUmdTYWJJ?=
 =?utf-8?B?RDJqaFRBYmwvWjVmVWhLOXJIZ3p3amw4MHBCTDB6TXJWczZHSFFmdE1weE0v?=
 =?utf-8?B?cGFsRzJDdU5GanU2MG1vQVJFeGh4ZGwvSFNXM3VQWGpSTkxOMm04RHpFYjNC?=
 =?utf-8?Q?yxKA=3D?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0af6818a-5a53-4de2-75e0-08db2f775bcb
X-MS-Exchange-CrossTenant-AuthSource: BN8PR12MB3587.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Mar 2023 10:29:49.3033
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: QRRxKv0zHyUzB/KHRLcAvupPWsV8rmF/C8oywd13Ra6K+173sdWUcw202hEt3In8
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB7360
X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
        DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,NICE_REPLY_A,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE
        autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

The same strategy was suggested before for amdgpu and reverted as well 
because of running into the same problems.

No idea how that slipped, Andrew reviewed it and IIRC he was also the 
one who reverted the initial approach for amdgpu.

Instead of letting the scheduler sum that stuff up proactively we should 
take care of that re-actively when the fdinfo data is queried like 
amdgpu does as well.

We should probably revert that patch for now and then discuss how we can 
do this in a generalized helper.

Regards,
Christian.

Am 28.03.23 um 02:54 schrieb Danilo Krummrich:
> Hi all,
>
> Commit df622729ddbf ("drm/scheduler: track GPU active time per 
> entity") tries to track the accumulated time that a job was active on 
> the GPU writing it to the entity through which the job was deployed to 
> the scheduler originally. This is done within 
> drm_sched_get_cleanup_job() which fetches a job from the schedulers 
> pending_list.
>
> Doing this can result in a race condition where the entity is already 
> freed, but the entity's newly added elapsed_ns field is still accessed 
> once the job is fetched from the pending_list.
>
> After drm_sched_entity_destroy() being called it should be safe to 
> free the structure that embeds the entity. However, a job originally 
> handed over to the scheduler by this entity might still reside in the 
> schedulers pending_list for cleanup after drm_sched_entity_destroy() 
> already being called and the entity being freed. Hence, we can run 
> into a UAF.
>
> In my case it happened that a job, as explained above, was just picked 
> from the schedulers pending_list after the entity was freed due to the 
> client application exiting. Meanwhile this freed up memory was already 
> allocated for a subsequent client applications job structure again. 
> Hence, the new jobs memory got corrupted. Luckily, I was able to 
> reproduce the same corruption over and over again by just using 
> deqp-runner to run a specific set of VK test cases in parallel.
>
> Fixing this issue doesn't seem to be very straightforward though 
> (unless I miss something), which is why I'm writing this mail instead 
> of sending a fix directly.
>
> Spontaneously, I see three options to fix it:
>
> 1. Rather than embedding the entity into driver specific structures 
> (e.g. tied to file_priv) we could allocate the entity separately and 
> reference count it, such that it's only freed up once all jobs that 
> were deployed through this entity are fetched from the schedulers 
> pending list.
>
> 2. Somehow make sure drm_sched_entity_destroy() does block until all 
> jobs deployed through this entity were fetched from the schedulers 
> pending list. Though, I'm pretty sure that this is not really desirable.
>
> 3. Just revert the change and let drivers implement tracking of GPU 
> active times themselves.
>
> In the case of just reverting the change I'd propose to also set a 
> jobs entity pointer to NULL  once the job was taken from the entity, 
> such that in case of a future issue we fail where the actual issue 
> resides and to make it more obvious that the field shouldn't be used 
> anymore after the job was taken from the entity.
>
> I'm happy to implement the solution we agree on. However, it might 
> also make sense to revert the change until we have a solution in 
> place. I'm also happy to send a revert with a proper description of 
> the problem. Please let me know what you think.
>
> - Danilo