Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <332e031c-c04e-998c-e401-685c817ea2a1@amd.com>
Date:   Fri, 14 Jul 2023 10:40:03 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Subject: Re: [PATCH 1/3] drm/scheduler: Add more documentation
Content-Language: en-US
To:     Asahi Lina <lina@asahilina.net>,
        Luben Tuikov <luben.tuikov@amd.com>,
        David Airlie <airlied@gmail.com>,
        Daniel Vetter <daniel@ffwll.ch>,
        Sumit Semwal <sumit.semwal@linaro.org>
Cc:     Faith Ekstrand <faith.ekstrand@collabora.com>,
        Alyssa Rosenzweig <alyssa@rosenzweig.io>,
        dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org,
        linux-media@vger.kernel.org, asahi@lists.linux.dev
References: <20230714-drm-sched-fixes-v1-0-c567249709f7@asahilina.net>
 <20230714-drm-sched-fixes-v1-1-c567249709f7@asahilina.net>
From:   =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>
In-Reply-To: <20230714-drm-sched-fixes-v1-1-c567249709f7@asahilina.net>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Skt4SEsydHY3M29zNUQvUkIzcjNVb0c4TzJVckNYQmNCcGo1bnVxL0E1bDR5?=
 =?utf-8?B?cWV6cTFaeDBBaW5Ea2FJOVNwUFdaOXdjM1VJdU9rZlNsUVh2K1QwQnNXa0Z6?=
 =?utf-8?B?dzZacTA2MzJublczN0VJZXNTTjJZeEIrdUZRS2s0RGhONkNTT0F2L2lxTFZh?=
 =?utf-8?B?Yy8zaE1yZk1OWmhUYmUwMlprSHJZbTBTbG9sbmtaTzF5bzNjMzJzd1R1S1N2?=
 =?utf-8?B?NUpYNXNVVzNpbVMzVG1CaU1pcDU0THFuT2tTSXM4Tk5ad05tOWJDT3dMQkVp?=
 =?utf-8?B?TFZpQld4dS9BZkVzWGJRWWVkVjZRWWFOVzUyTjVVZDhIK3R1STdQdlpkc1J3?=
 =?utf-8?B?c25SMlNzdjRISXNtRWc2a1BEVU9pRVFzcUxsZ0UwVnlXVFhENStRZkpyQnJ6?=
 =?utf-8?B?bjBoMTFXUGwxVnJYNG5GTVVtQVhUR21XcWwxUmlyWWc2R1NpZ0gzRUM4Z2t3?=
 =?utf-8?B?MkZjMk9GeitsTFdWeXowZHRnT3RwSTdLYlIrZnRpTU52RC9iYUtVTHRsMHcr?=
 =?utf-8?B?MU0yYlF3VldtdmUrUU9zbTkxOG9nSzAzTUFoa0MvKzdpenlhQllKMEwzU2Uv?=
 =?utf-8?B?clhEQk9pYlR6aHlieUxRWHdrK1NXZjhxZXpTS2FXbkFnZ3lvT01iT3FvaUpT?=
 =?utf-8?B?K09YNGo0Wm5mWktvLzJaSW9hV0ZBTERPa2xnNHdmaWpFWCtUbk5FVWg4MCsy?=
 =?utf-8?B?K0NkMHJzV0pUM01tQU9qTkROaXNUTDFjNklOMytnb2dNWU9ETlNnV0VCRWJF?=
 =?utf-8?B?MmF5UHdMZWpOTDBOcHpSRzF3aEJZQmZIWTA0QVBFaUYvRXhzcURVQ042SGZw?=
 =?utf-8?B?SEMwekN6RFBnZWYwZmNVMHJxTm1NNi80d1Z3aFByQ3drQVprRitnMENtL3BR?=
 =?utf-8?B?dHQ3NFVEYWNvMVNvZmMyTmtPMWd4eW95NnlRYklyaTh2NEUxZDZ4Zzl1U0Qy?=
 =?utf-8?B?cGIrbTBKUmlvQ1RxdUoxRG9SektycFFScEFUWDFzOFNqaUk1MXNIUERvcjcy?=
 =?utf-8?B?WEEzTTR4aHhNb0d3YmxEV3NmNXQrdVI4elRQQXBTVGY5bURJYzF0QnMxUEdv?=
 =?utf-8?B?N2hGY0I0MWtaYlJIMjAyWHZPS2t3cDVKS0h0a2lmR09IdElvU3JYNEw3aDY2?=
 =?utf-8?B?Y1ZTajh2dFQ1bUR1bjJsbE1YenhOWWFCOFZIaXI3ZFpmQkE0cmIraC9aTkhE?=
 =?utf-8?B?U0JvR1pGL1F6U3JIS0lvSVFYUmVIVEtYR1VaN3RhaFJBZ3c1UnhQdStQcDBn?=
 =?utf-8?B?RVFnSmNkNjdJOWFMY25VZ2ZGa1IyY3lYekMzSGVJZXlWTW1CdG5mQzJRSGRp?=
 =?utf-8?B?bXhPcXc5bWQ3L0pkdnhSZkdJcDRRTjNIQWY1cmtLWTRFSW9kZ1ZRd3lFaFI0?=
 =?utf-8?B?U1llQ1ROMmdldW5iOEFOWVplYS9RZVdHd0o2REZCa0paYmQzaFNOckxjMThY?=
 =?utf-8?B?RVQzMjJiUkZZY2hTY0xsem42QWs4TEVSb2F6OXZWbW5jdVlnOG1paEYyRmtj?=
 =?utf-8?B?UE5HOVBHMW9pdElZR0V4RlFKSUxWLzVybnlHTW9lRSthMEpyMVRyeFZlVkI0?=
 =?utf-8?B?cUI3eDVoc2puZzdyMmxmMXgwNStxMXYrYWRFQ2ZZeFhJMmVRdWtzT2x4UGZ1?=
 =?utf-8?B?c3h0SC9lZ2lvbnBLSDdPTUc4ZEc3RTd4M3lJNzUwQllzbGNYdGdQRXlWV3lL?=
 =?utf-8?B?UnFSVHdjbzBwTW50TDdDaTNzZGN6cWFlMWRnc0t5TnJ3MGhUa1Y1b0RmcEVV?=
 =?utf-8?B?VGN1eStxVUcyNjJza2h0NUYrRGxvVWZHMzdmQThBazRPQzhlT3JQeUNBQ2NL?=
 =?utf-8?B?U3VvYXFlTmttWDFETzU3YWM0K1R1MDBaa1hNbXQ1c3M5S2xySVR2eTRGTEMy?=
 =?utf-8?B?TGgrZEtNTVB6NDZQR05RUEJPRUg3RVBaWDdrKzFHUS9BTm4xY29WbGJwYUhF?=
 =?utf-8?B?MWFHZ01GTFRUTHp2R05IeE5ERW1hV0I1b1JWS2lVWWNPSndFYlpkRHhKOThy?=
 =?utf-8?B?TjhXSnBjT2REamJKalU2cUpRTzY5cjNmVDBWVTNQTGNka3psM3pHVldKUWI1?=
 =?utf-8?B?U0YxbXNZWVNDWkc4WXQwT1J2TlhJVEpPQS9RUEZsWnRreUl6NFhaN3ZHR29l?=
 =?utf-8?B?ak56bjk2VlIyVjlURkNJbmROQldGcXdHaXVla25JdzkrL29VQVNLSkV6NGxm?=
 =?utf-8?Q?EewElrMaYzywUc45pOllCfPrQXc/5GXS/FhCEbDLYZiY?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c5b3807b-9f9b-4a9f-6f5e-08db8445ee33
X-MS-Exchange-CrossTenant-AuthSource: BN8PR12MB3587.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jul 2023 08:40:08.8933
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 5hVQ9NPwQ+tywLj/r9/X/Co8XUUaic1b8BNZlqUqcEfX0qKrgBH+LvW+NMTBP6FM
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW5PR12MB5684
X-Spam-Status: No, score=-1.2 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,NICE_REPLY_A,
        RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=no autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Am 14.07.23 um 10:21 schrieb Asahi Lina:
> Document the implied lifetime rules of the scheduler (or at least the
> intended ones), as well as the expectations of how resource acquisition
> should be handled.
>
> Signed-off-by: Asahi Lina <lina@asahilina.net>
> ---
>   drivers/gpu/drm/scheduler/sched_main.c | 58 ++++++++++++++++++++++++++++++++--
>   1 file changed, 55 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c
> index 7b2bfc10c1a5..1f3bc3606239 100644
> --- a/drivers/gpu/drm/scheduler/sched_main.c
> +++ b/drivers/gpu/drm/scheduler/sched_main.c
> @@ -43,9 +43,61 @@
>    *
>    * The jobs in a entity are always scheduled in the order that they were pushed.
>    *
> - * Note that once a job was taken from the entities queue and pushed to the
> - * hardware, i.e. the pending queue, the entity must not be referenced anymore
> - * through the jobs entity pointer.
> + * Lifetime rules
> + * --------------
> + *
> + * Getting object lifetimes right across the stack is critical to avoid UAF
> + * issues. The DRM scheduler has the following lifetime rules:
> + *
> + * - The scheduler must outlive all of its entities.
> + * - Jobs pushed to the scheduler are owned by it, and must only be freed
> + *   after the free_job() callback is called.
> + * - Scheduler fences are reference-counted and may outlive the scheduler.

> + * - The scheduler *may* be destroyed while jobs are still in flight.

That's not correct. The scheduler can only be destroyed after all the 
entities serving it have been destroyed as well as all the jobs already 
pushed to the hw finished.

What might be possible to add is that the hw is still working on the 
already pushed jobs, but so far that was rejected as undesirable.

> + * - There is no guarantee that all jobs have been freed when all entities
> + *   and the scheduled have been destroyed. Jobs may be freed asynchronously
> + *   after this point.
> + * - Once a job is taken from the entity's queue and pushed to the hardware,
> + *   i.e. the pending queue, the entity must not be referenced any more
> + *   through the job's entity pointer. In other words, entities are not
> + *   required to outlive job execution.
> + *
> + * If the scheduler is destroyed with jobs in flight, the following
> + * happens:
> + *
> + * - Jobs that were pushed but have not yet run will be destroyed as part
> + *   of the entity cleanup (which must happen before the scheduler itself
> + *   is destroyed, per the first rule above). This signals the job
> + *   finished fence with an error flag. This process runs asynchronously
> + *   after drm_sched_entity_destroy() returns.
> + * - Jobs that are in-flight on the hardware are "detached" from their
> + *   driver fence (the fence returned from the run_job() callback). In
> + *   this case, it is up to the driver to ensure that any bookkeeping or
> + *   internal data structures have separately managed lifetimes and that
> + *   the hardware either cancels the jobs or runs them to completion.
> + *   The DRM scheduler itself will immediately signal the job complete
> + *   fence (with an error flag) and then call free_job() as part of the
> + *   cleanup process.
> + *
> + * After the scheduler is destroyed, drivers *may* (but are not required to)
> + * skip signaling their remaining driver fences, as long as they have only ever
> + * been returned to the scheduler being destroyed as the return value from
> + * run_job() and not passed anywhere else.

This is an outright NAK to this. Fences must always be cleanly signaled.

IIRC Daniel documented this as mandatory on the dma_fence behavior.

Regards,
Christian.

>   If these fences are used in any other
> + * context, then the driver *must* signal them, per the usual fence signaling
> + * rules.
> + *
> + * Resource management
> + * -------------------
> + *
> + * Drivers may need to acquire certain hardware resources (e.g. VM IDs) in order
> + * to run a job. This process must happen during the job's prepare() callback,
> + * not in the run() callback. If any resource is unavailable at job prepare time,
> + * the driver must return a suitable fence that can be waited on to wait for the
> + * resource to (potentially) become available.
> + *
> + * In order to avoid deadlocks, drivers must always acquire resources in the
> + * same order, and release them in opposite order when a job completes or if
> + * resource acquisition fails.
>    */
>   
>   #include <linux/kthread.h>
>