Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <21c789de-e381-cccd-9093-741dcd7a2dc5@amd.com>
Date:   Tue, 11 Apr 2023 16:06:30 -0400
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.9.1
Content-Language: en-CA
To:     Danilo Krummrich <dakr@redhat.com>,
        =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>,
        airlied@gmail.com, daniel@ffwll.ch, l.stach@pengutronix.de,
        "Prosyak, Vitaly" <Vitaly.Prosyak@amd.com>
Cc:     dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org
References: <20230331000622.4156-1-dakr@redhat.com>
 <6ad72a7f-302f-4be1-0d53-00ff9dc37ef7@amd.com>
 <ecef210b-dc7d-e385-f9b2-927d55a6777e@amd.com>
 <32c28b47-2a3b-db1d-e927-ae44d52cae0b@redhat.com>
From:   Luben Tuikov <luben.tuikov@amd.com>
Subject: Re: [PATCH] drm/scheduler: set entity to NULL in
 drm_sched_entity_pop_job()
In-Reply-To: <32c28b47-2a3b-db1d-e927-ae44d52cae0b@redhat.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?OTdDeXRLWG1ZQUxKNytkdnhyQ2R5NjlxbmVKdHpXUzBWMHI5YTZOdGs1S1ZL?=
 =?utf-8?B?bDJIL2U4WnYrQkZlNklrblJGOVFDbzJaTFc3U1NCVUlHc3lHZktGcmxlT2RT?=
 =?utf-8?B?TEhqdjNJNElFY3ZGd0RieFU0SGt1bVZINit6SmlaOXBtVFMzV3Aybk9PRnpS?=
 =?utf-8?B?NVVVQzgxNnUvUitteXBDYksza2dXRFpmblhzWFdBZWFtRGw4ZUJDbEUrVi9u?=
 =?utf-8?B?MDc2b3MrZCtMbEo3eGovN2FxOVU4VHM5eFlYSUl2d1lyNkdXYnBVdWgvbElS?=
 =?utf-8?B?bmJFeVVUVlVEZ3l2allkMktwQW40eHAyb2l6aXNVaUVndEVFTzEzTndXanQ3?=
 =?utf-8?B?WXh1QmR3VGo1aEU1QWxmZktkKy9KV2RZUzZIeTV4QnRiZHhMUjEvbGRPR0RN?=
 =?utf-8?B?aE1BT3RBQW1ZdTdWb3IwYXBYWkh6dTRUbWpoRVYwbUVURDVLS3hYZnZLTzlh?=
 =?utf-8?B?ZmZEZm9qSWV1Rjc5NzViREZCeGVkZWpRMmk3OXBnbjZVd3pzYkhieVdqSUdN?=
 =?utf-8?B?WnRGVFZVSTBwVVBhaGx0bzNSYmFJSi9SM21nSVVYVVNMa0lZSE5iS210SkFI?=
 =?utf-8?B?TkFJZ3R1TTRtR00wU3cxdm94V3V5M1FvREM4Uk1GRWc1a0xRa2EzR1FFSjJv?=
 =?utf-8?B?VmpCVWFmTDUvYUFBbCtITmZ4TWxNK01VK0hranNmR1B5QkNndWs2UHc0dlcr?=
 =?utf-8?B?dXppMlVpRHV2SHhGN2NvWVhsVlZJZDZwSmZrNUZWanhZRWRwUUdTdFdYZmxS?=
 =?utf-8?B?cFcxQm5lWTU4RWFUSFZaM0FBbmZoMkxUc0o0QU45cnVNMmcwLzdGZlErbzlQ?=
 =?utf-8?B?M2JpbWlIQTRwL2VWQ2lvMkt0MEFLK1RNUWY1R0UrTzJ4QmpRQ0w3Tk5XUHA3?=
 =?utf-8?B?bE5JZjdPemxnbTN5MWJ2ZTVQQjFYUmlLNDJaN24zM0ZhQXZ0VzFwVytLMUhw?=
 =?utf-8?B?S0FzUE1hOUZmUWF1K3Q1QXZXWFBtc3pmd09CMk5FdnVXS2lJZXoyRllsMXo4?=
 =?utf-8?B?a09JNmtUaWFVdmtDcWQwVHRWMzB5d3pLQTdzUjBVYmtKVldOa3BBL3FFQ0li?=
 =?utf-8?B?VWpPVVhhVWduWVBUckY0aG5uQkNwbTk4ZlYxcUJSSjhDakZLRE92dGRTcDFH?=
 =?utf-8?B?Tk5rNXFmUVhPdi92VGJncXkveC9oMUFkSjRvUmxNQmcxQm1XMVpRY3h2Vmp0?=
 =?utf-8?B?WElVaE1qQ2NpV2Z5S3o5MW1waEZQQllXYjRQbnNYUnMzSCtaUDVrRWdKdDJS?=
 =?utf-8?B?L2MrVEVoeHhycUxmN1VYVWdwTE1UZllCSEVISjN0aU5SZGMzTFM0M1BzZXVV?=
 =?utf-8?B?Z1ZEUmZtWWdhUGROUThFMW50YjRtS3hvcUV5b3NkSTNCdkNoOWJkQU1QYlFJ?=
 =?utf-8?B?QzdzR2dJc1ErSlRNNXY4L3lCQVdWeWhOdzNwbk9lYkg4V0F6SUI2VVVqRG1E?=
 =?utf-8?B?di9mL0tpWnFXbVB2VHdoVTlXQXpYeEllQWlQUEpPeWh1QlM2YUoxcHhvaHlD?=
 =?utf-8?B?cCtqZE50UXk4OFgrNTg1cFJ6K1p2cllzQ0lBblR5bWNFUGg4UHFUOGlEU0Jx?=
 =?utf-8?B?aTFxOFhialExRW1kWFo5WFFvTUhSbzBFZTYwWWZQeXdmdWhENFpsQzFpSHhu?=
 =?utf-8?B?bjc0TU1TSDVSTFFSd3RTYzRYbmJETDlQV2FBQWxLWThLV1k5OTl2ME9yMnNY?=
 =?utf-8?B?MXV1RzUzU2hDZUFKQTA2RW1pU0xYcjNOSmt6SjVCcTlKSUVkOVZpeFgrRlNp?=
 =?utf-8?B?L3dsMTE1dWsxcUVXU1FoV0UxaXYwdk8vQ1F2Q0NReVVxTnhaalJFbEFWTWhL?=
 =?utf-8?B?aW8rc29sQWI3T1pmQnhrczdwZndEc0I4TWRrQjJOT2tIV0I3SUNLR2wySEZK?=
 =?utf-8?B?RDZHM0pEY1VzdjJjSis4eGp4aVpNRTYxUGtGeTQ2WlNuTlE1UUdwZzNuQTZz?=
 =?utf-8?B?NWl0TEozQnlmZ1dpUGdGaVpqZy9UNzIyUWVjNVhMdHpOK3R0WmZ5aHh4ajkv?=
 =?utf-8?B?OGVjbDJTVWNVQ0g1cXp5NDdEWStuMmJMdStWNkp6bVJrV2NqVnJWdUtlTmNR?=
 =?utf-8?B?R2NHSUsxL2pvSjV0YjJXTEVVb2thVmRwbHRlSnFRUXAzczFlajFMMC8wdDJu?=
 =?utf-8?Q?tv9+ksukbjlOgU3C82cfUJUFk?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6cb6f583-604f-432b-7221-08db3ac83f48
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3370.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Apr 2023 20:06:33.5004
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: /vb+MWhGFkJDv3SAbdXnv+UOM60ejVOlH+TmYAGuanD4vJi3SdIu5/9Dq1p2gLcK
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6305
X-Spam-Status: No, score=-3.3 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,NICE_REPLY_A,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 2023-04-11 14:13, Danilo Krummrich wrote:
> On 4/5/23 19:39, Luben Tuikov wrote:
>> On 2023-03-31 01:59, Christian König wrote:
>>> Am 31.03.23 um 02:06 schrieb Danilo Krummrich:
>>>> It already happend a few times that patches slipped through which
>>>> implemented access to an entity through a job that was already removed
>>>> from the entities queue. Since jobs and entities might have different
>>>> lifecycles, this can potentially cause UAF bugs.
>>>>
>>>> In order to make it obvious that a jobs entity pointer shouldn't be
>>>> accessed after drm_sched_entity_pop_job() was called successfully, set
>>>> the jobs entity pointer to NULL once the job is removed from the entity
>>>> queue.
>>>>
>>>> Moreover, debugging a potential NULL pointer dereference is way easier
>>>> than potentially corrupted memory through a UAF.
>>>>
>>>> Signed-off-by: Danilo Krummrich <dakr@redhat.com>
>>>
>>> In general "YES PLEASE!", but I fear that this will break amdgpus reset
>>> sequence.
>>>
>>> On the other hand when amdgpu still relies on that pointer it's clearly
>>> a bug (which I pointed out tons of times before).
>>>
>>> Luben any opinion on that? Could you drive cleaning that up as well?
>>
>> I didn't find any references to scheduling entity after the job
>> is submitted to the hardware. (I commented the same in the other
>> thread, we just need to decide which way to go.)
> 
> AFAICS from the other mail thread it seems to be consensus to not 
> ref-count entities and handle job statistics differently.
> 
> Should we go ahead and take this patch then? Maybe it also makes sense 
> to send a V2 additionally adding a comment to the drm_sched_job 
> structure mentioning that .entity must not be used after the job was 
> taken from the entities queue.

Yes, please send a v2, but instead of mentioning (or in addition to)
that the job was taken from the "entities queue", mention that
once the job is pushed to the hardware, i.e. the pending queue,
from then on, the "entity" should not be referenced anymore. IOW, we
want to mention that it is going from X to Y, as opposed to just
that it's taken from X, because the latter begs the question "Where
is it going to then?".

For the record, I think that using kref would give us the most
stability, even if we skipped "entity" and let the scheduler, or
even the controller keep a kref on submitted commands down to
the GPU. On reset, we block command submission to the GPU from
the outermost boundary, and then start peeling the layers from
the innermost boundary.

Using kref also forces us to think objectively and set explicit
(clear) dependencies from the outset--i.e. who gets freed and when
and under what circumstances.

And using kref might even expose the wrong dependencies, thus
prompting a redesign and thus a fix.
-- 
Regards,
Luben