Received-SPF: pass (google.com: domain of linux-kernel+bounces-10055-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Precedence: bulk
MIME-Version: 1.0
References: <20231218155927.368881-1-robdclark@gmail.com> <k5yte4b36j23p3pfqmflhpkfesyiwoczi3avs2vavvzb3i2oct@zhssj7u5a6og>
In-Reply-To: <k5yte4b36j23p3pfqmflhpkfesyiwoczi3avs2vavvzb3i2oct@zhssj7u5a6og>
From: Rob Clark <robdclark@gmail.com>
Date: Fri, 22 Dec 2023 12:15:34 -0800
Message-ID: <CAF6AEGvGwXU5Qf-bgPfoiwDf9AdFSUERger_gk-pogOo4=hWSA@mail.gmail.com>
Subject: Re: [PATCH] drm/msm/a6xx: Fix recovery vs runpm race
To: Akhil P Oommen <quic_akhilpo@quicinc.com>
Cc: dri-devel@lists.freedesktop.org, freedreno@lists.freedesktop.org, 
	linux-arm-msm@vger.kernel.org, Rob Clark <robdclark@chromium.org>, 
	David Heidelberg <david.heidelberg@collabora.com>, Abhinav Kumar <quic_abhinavk@quicinc.com>, 
	Dmitry Baryshkov <dmitry.baryshkov@linaro.org>, Sean Paul <sean@poorly.run>, 
	Marijn Suijten <marijn.suijten@somainline.org>, David Airlie <airlied@gmail.com>, 
	Daniel Vetter <daniel@ffwll.ch>, Konrad Dybcio <konrad.dybcio@linaro.org>, 
	Danylo Piliaiev <dpiliaiev@igalia.com>, Bjorn Andersson <andersson@kernel.org>, 
	open list <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Fri, Dec 22, 2023 at 11:58=E2=80=AFAM Akhil P Oommen
<quic_akhilpo@quicinc.com> wrote:
>
> On Mon, Dec 18, 2023 at 07:59:24AM -0800, Rob Clark wrote:
> >
> > From: Rob Clark <robdclark@chromium.org>
> >
> > a6xx_recover() is relying on the gpu lock to serialize against incoming
> > submits doing a runpm get, as it tries to temporarily balance out the
> > runpm gets with puts in order to power off the GPU.  Unfortunately this
> > gets worse when we (in a later patch) will move the runpm get out of th=
e
> > scheduler thread/work to move it out of the fence signaling path.
> >
> > Instead we can just simplify the whole thing by using force_suspend() /
> > force_resume() instead of trying to be clever.
>
> At some places, we take a pm_runtime vote and access the gpu
> registers assuming it will be powered until we drop the vote.  a6xx_get_t=
imestamp()
> is an example. If we do a force suspend, it may cause bus errors from
> those threads. Now you have to serialize every place we do runtime_get/pu=
t with a
> mutex. Or is there a better way to handle the 'later patch' you
> mentioned?

So I was running into issues, when I started adding an igt test to
stress test recovery vs multi-threaded submit, with cxpd not always
suspending and getting "cx gdsc did not collapse", which may be
related.

I was considering using force_suspend() on the gmu and cxpd if
gpu->hang=3D=3Dtrue, I'm not sure.  I ran out of time to play with this
when I was in the office.

The issue the 'later patch' is trying to deal with is getting memory
allocations out of the "fence signaling path", ie. out from the
drm/sched kthread/worker.  One way to do that, without dragging all of
runpm/device-link/etc into it is to do the runpm get in the submit
ioctl before enqueuing the job to the scheduler.  But then we can hold
a lock to protect against racing with recovery.

BR,
-R

> -Akhil.
>
> >
> > Reported-by: David Heidelberg <david.heidelberg@collabora.com>
> > Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/10272
> > Fixes: abe2023b4cea ("drm/msm/gpu: Push gpu lock down past runpm")
> > Signed-off-by: Rob Clark <robdclark@chromium.org>
> > ---
> >  drivers/gpu/drm/msm/adreno/a6xx_gpu.c | 12 ++----------
> >  1 file changed, 2 insertions(+), 10 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/msm/adreno/a6xx_gpu.c b/drivers/gpu/drm/ms=
m/adreno/a6xx_gpu.c
> > index 268737e59131..a5660d63535b 100644
> > --- a/drivers/gpu/drm/msm/adreno/a6xx_gpu.c
> > +++ b/drivers/gpu/drm/msm/adreno/a6xx_gpu.c
> > @@ -1244,12 +1244,7 @@ static void a6xx_recover(struct msm_gpu *gpu)
> >       dev_pm_genpd_add_notifier(gmu->cxpd, &gmu->pd_nb);
> >       dev_pm_genpd_synced_poweroff(gmu->cxpd);
> >
> > -     /* Drop the rpm refcount from active submits */
> > -     if (active_submits)
> > -             pm_runtime_put(&gpu->pdev->dev);
> > -
> > -     /* And the final one from recover worker */
> > -     pm_runtime_put_sync(&gpu->pdev->dev);
> > +     pm_runtime_force_suspend(&gpu->pdev->dev);
> >
> >       if (!wait_for_completion_timeout(&gmu->pd_gate, msecs_to_jiffies(=
1000)))
> >               DRM_DEV_ERROR(&gpu->pdev->dev, "cx gdsc didn't collapse\n=
");
> > @@ -1258,10 +1253,7 @@ static void a6xx_recover(struct msm_gpu *gpu)
> >
> >       pm_runtime_use_autosuspend(&gpu->pdev->dev);
> >
> > -     if (active_submits)
> > -             pm_runtime_get(&gpu->pdev->dev);
> > -
> > -     pm_runtime_get_sync(&gpu->pdev->dev);
> > +     pm_runtime_force_resume(&gpu->pdev->dev);
> >
> >       gpu->active_submits =3D active_submits;
> >       mutex_unlock(&gpu->active_lock);
> > --
> > 2.43.0
> >