LinuxLists.cc - [PATCH v1] drm/scheduler: Fix lockup in drm_sched_entity

2022-11-23 00:59:18

Subject: [PATCH v1] drm/scheduler: Fix lockup in drm_sched_entity_kill()

The drm_sched_entity_kill() is invoked twice by drm_sched_entity_destroy()
while userspace process is exiting or being killed. First time it's invoked
when sched entity is flushed and second time when entity is released. This
causes a lockup within wait_for_completion(entity_idle) due to how completion
API works.

Calling wait_for_completion() more times than complete() was invoked is a
error condition that causes lockup because completion internally uses
counter for complete/wait calls. The complete_all() must be used instead
in such cases.

This patch fixes lockup of Panfrost driver that is reproducible by killing
any application in a middle of 3d drawing operation.

Fixes: 2fdb8a8f07c2 ("drm/scheduler: rework entity flush, kill and fini")
Signed-off-by: Dmitry Osipenko <[email protected]>
---
drivers/gpu/drm/scheduler/sched_entity.c | 2 +-
drivers/gpu/drm/scheduler/sched_main.c | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c
index fe09e5be79bd..15d04a0ec623 100644
--- a/drivers/gpu/drm/scheduler/sched_entity.c
+++ b/drivers/gpu/drm/scheduler/sched_entity.c
@@ -81,7 +81,7 @@ int drm_sched_entity_init(struct drm_sched_entity *entity,
init_completion(&entity->entity_idle);

/* We start in an idle state. */
- complete(&entity->entity_idle);
+ complete_all(&entity->entity_idle);

spin_lock_init(&entity->rq_lock);
spsc_queue_init(&entity->job_queue);
diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c
index 6ce04c2e90c0..857ec20be9e8 100644
--- a/drivers/gpu/drm/scheduler/sched_main.c
+++ b/drivers/gpu/drm/scheduler/sched_main.c
@@ -1026,7 +1026,7 @@ static int drm_sched_main(void *param)
sched_job = drm_sched_entity_pop_job(entity);

if (!sched_job) {
- complete(&entity->entity_idle);
+ complete_all(&entity->entity_idle);
continue;
}

@@ -1037,7 +1037,7 @@ static int drm_sched_main(void *param)

trace_drm_run_job(sched_job, entity);
fence = sched->ops->run_job(sched_job);
- complete(&entity->entity_idle);
+ complete_all(&entity->entity_idle);
drm_sched_fence_scheduled(s_fence);

if (!IS_ERR_OR_NULL(fence)) {
--
2.38.1

2022-11-23 06:49:52

by Christian König

[permalink] [raw]

Subject: Re: [PATCH v1] drm/scheduler: Fix lockup in drm_sched_entity_kill()

Am 23.11.22 um 01:13 schrieb Dmitry Osipenko:
> The drm_sched_entity_kill() is invoked twice by drm_sched_entity_destroy()
> while userspace process is exiting or being killed. First time it's invoked
> when sched entity is flushed and second time when entity is released. This
> causes a lockup within wait_for_completion(entity_idle) due to how completion
> API works.
>
> Calling wait_for_completion() more times than complete() was invoked is a
> error condition that causes lockup because completion internally uses
> counter for complete/wait calls. The complete_all() must be used instead
> in such cases.
>
> This patch fixes lockup of Panfrost driver that is reproducible by killing
> any application in a middle of 3d drawing operation.
>
> Fixes: 2fdb8a8f07c2 ("drm/scheduler: rework entity flush, kill and fini")
> Signed-off-by: Dmitry Osipenko <[email protected]>

Oh, good point. Reviewed-by: Christian König <[email protected]>

> ---
> drivers/gpu/drm/scheduler/sched_entity.c | 2 +-
> drivers/gpu/drm/scheduler/sched_main.c | 4 ++--
> 2 files changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c
> index fe09e5be79bd..15d04a0ec623 100644
> --- a/drivers/gpu/drm/scheduler/sched_entity.c
> +++ b/drivers/gpu/drm/scheduler/sched_entity.c
> @@ -81,7 +81,7 @@ int drm_sched_entity_init(struct drm_sched_entity *entity,
> init_completion(&entity->entity_idle);
>
> /* We start in an idle state. */
> - complete(&entity->entity_idle);
> + complete_all(&entity->entity_idle);
>
> spin_lock_init(&entity->rq_lock);
> spsc_queue_init(&entity->job_queue);
> diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c
> index 6ce04c2e90c0..857ec20be9e8 100644
> --- a/drivers/gpu/drm/scheduler/sched_main.c
> +++ b/drivers/gpu/drm/scheduler/sched_main.c
> @@ -1026,7 +1026,7 @@ static int drm_sched_main(void *param)
> sched_job = drm_sched_entity_pop_job(entity);
>
> if (!sched_job) {
> - complete(&entity->entity_idle);
> + complete_all(&entity->entity_idle);
> continue;
> }
>
> @@ -1037,7 +1037,7 @@ static int drm_sched_main(void *param)
>
> trace_drm_run_job(sched_job, entity);
> fence = sched->ops->run_job(sched_job);
> - complete(&entity->entity_idle);
> + complete_all(&entity->entity_idle);
> drm_sched_fence_scheduled(s_fence);
>
> if (!IS_ERR_OR_NULL(fence)) {

2022-11-23 13:51:08

by Dmitry Osipenko

[permalink] [raw]

Subject: Re: [PATCH v1] drm/scheduler: Fix lockup in drm_sched_entity_kill()

On 11/23/22 03:13, Dmitry Osipenko wrote:
> The drm_sched_entity_kill() is invoked twice by drm_sched_entity_destroy()
> while userspace process is exiting or being killed. First time it's invoked
> when sched entity is flushed and second time when entity is released. This
> causes a lockup within wait_for_completion(entity_idle) due to how completion
> API works.
>
> Calling wait_for_completion() more times than complete() was invoked is a
> error condition that causes lockup because completion internally uses
> counter for complete/wait calls. The complete_all() must be used instead
> in such cases.
>
> This patch fixes lockup of Panfrost driver that is reproducible by killing
> any application in a middle of 3d drawing operation.
>
> Fixes: 2fdb8a8f07c2 ("drm/scheduler: rework entity flush, kill and fini")
> Signed-off-by: Dmitry Osipenko <[email protected]>
> ---
> drivers/gpu/drm/scheduler/sched_entity.c | 2 +-
> drivers/gpu/drm/scheduler/sched_main.c | 4 ++--
> 2 files changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c
> index fe09e5be79bd..15d04a0ec623 100644
> --- a/drivers/gpu/drm/scheduler/sched_entity.c
> +++ b/drivers/gpu/drm/scheduler/sched_entity.c
> @@ -81,7 +81,7 @@ int drm_sched_entity_init(struct drm_sched_entity *entity,
> init_completion(&entity->entity_idle);
>
> /* We start in an idle state. */
> - complete(&entity->entity_idle);
> + complete_all(&entity->entity_idle);
>
> spin_lock_init(&entity->rq_lock);
> spsc_queue_init(&entity->job_queue);
> diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c
> index 6ce04c2e90c0..857ec20be9e8 100644
> --- a/drivers/gpu/drm/scheduler/sched_main.c
> +++ b/drivers/gpu/drm/scheduler/sched_main.c
> @@ -1026,7 +1026,7 @@ static int drm_sched_main(void *param)
> sched_job = drm_sched_entity_pop_job(entity);
>
> if (!sched_job) {
> - complete(&entity->entity_idle);
> + complete_all(&entity->entity_idle);
> continue;
> }
>
> @@ -1037,7 +1037,7 @@ static int drm_sched_main(void *param)
>
> trace_drm_run_job(sched_job, entity);
> fence = sched->ops->run_job(sched_job);
> - complete(&entity->entity_idle);
> + complete_all(&entity->entity_idle);
> drm_sched_fence_scheduled(s_fence);
>
> if (!IS_ERR_OR_NULL(fence)) {

Applied to drm-misc-next

--
Best regards,
Dmitry

2022-12-27 19:56:21

by Guilherme G. Piccoli

[permalink] [raw]

Subject: Re: [PATCH v1] drm/scheduler: Fix lockup in drm_sched_entity_kill()

Hi Dmitry / Christian, thanks for the fix!

(And thanks Melissa for pointing that, saving me lots of time in
research heh)

Is this fix planned to be released on 6.2-rc cycle? I've just tested it
on Steam Deck, and it resolved a lockup observed (since v6.2-rc1) -
exactly the same thing mentioned in the commit message.

FWIW:
Tested-By: Guilherme G. Piccoli <[email protected]> # Steam Deck

Cheers,

Guilherme

2022-12-29 10:33:15

by Dmitry Osipenko

[permalink] [raw]

Subject: Re: [PATCH v1] drm/scheduler: Fix lockup in drm_sched_entity_kill()

Hi,

On 12/27/22 22:28, Guilherme G. Piccoli wrote:
> Hi Dmitry / Christian, thanks for the fix!
>
> (And thanks Melissa for pointing that, saving me lots of time in
> research heh)
>
> Is this fix planned to be released on 6.2-rc cycle? I've just tested it
> on Steam Deck, and it resolved a lockup observed (since v6.2-rc1) -
> exactly the same thing mentioned in the commit message.
>
> FWIW:
> Tested-By: Guilherme G. Piccoli <[email protected]> # Steam Deck
I'll push the patch to misc-fixes as soon as it will be rebased on
6.2-rc. Thanks!

Best regards,
Dmitry

2022-12-29 19:19:54

by Guilherme G. Piccoli

[permalink] [raw]

Subject: Re: [PATCH v1] drm/scheduler: Fix lockup in drm_sched_entity_kill()

On 29/12/2022 07:15, Dmitry Osipenko wrote:
> [...]
> I'll push the patch to misc-fixes as soon as it will be rebased on
> 6.2-rc. Thanks!
>
> Best regards,
> Dmitry
>

Thank you Dmitry, much appreciated!
Cheers,

Guilherme

2023-01-02 15:37:23

by Dmitry Osipenko

[permalink] [raw]