Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 13 Oct 2022 12:03:34 -0600
From:   Mathieu Poirier <mathieu.poirier@linaro.org>
To:     "Aiqun(Maria) Yu" <quic_aiquny@quicinc.com>
Cc:     linux-remoteproc@vger.kernel.org, linux-arm-msm@vger.kernel.org,
        linux-kernel@vger.kernel.org, quic_clew@quicinc.com
Subject: Re: [PATCH v4] remoteproc: core: do pm relax when in RPROC_OFFLINE
Message-ID: <20221013180334.GB1279972@p14s>
References: <128dc161-8949-1146-bf8b-310aa33c06a8@quicinc.com>
 <1663312351-28476-1-git-send-email-quic_aiquny@quicinc.com>
 <20221012204344.GA1178915@p14s>
 <792f05fc-995e-9a87-ab7d-bee03f15bc79@quicinc.com>
 <20221013173442.GA1279972@p14s>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20221013173442.GA1279972@p14s>
Precedence: bulk

On Thu, Oct 13, 2022 at 11:34:42AM -0600, Mathieu Poirier wrote:
> On Thu, Oct 13, 2022 at 09:40:09AM +0800, Aiqun(Maria) Yu wrote:
> > Hi Mathieu,
> > 
> > On 10/13/2022 4:43 AM, Mathieu Poirier wrote:
> > > Please add what has changed from one version to another, either in a cover
> > > letter or after the "Signed-off-by".  There are many examples on how to do that
> > > on the mailing list.
> > > 
> > Thx for the information, will take a note and benefit for next time.
> > 
> > > On Fri, Sep 16, 2022 at 03:12:31PM +0800, Maria Yu wrote:
> > > > RPROC_OFFLINE state indicate there is no recovery process
> > > > is in progress and no chance to do the pm_relax.
> > > > Because when recovering from crash, rproc->lock is held and
> > > > state is RPROC_CRASHED -> RPROC_OFFLINE -> RPROC_RUNNING,
> > > > and then unlock rproc->lock.
> > > 
> > > You are correct - because the lock is held rproc->state should be set to RPROC_RUNNING
> > > when rproc_trigger_recovery() returns.  If that is not the case then something
> > > went wrong.
> > > 
> > > Function rproc_stop() sets rproc->state to RPROC_OFFLINE just before returning,
> > > so we know the remote processor was stopped.  Therefore if rproc->state is set
> > > to RPROC_OFFLINE something went wrong in either request_firmware() or
> > > rproc_start().  Either way the remote processor is offline and the system probably
> > > in an unknown/unstable.  As such I don't see how calling pm_relax() can help
> > > things along.
> > > 
> > PROC_OFFLINE is possible that rproc_shutdown is triggered and successfully
> > finished.
> > Even if it is multi crash rproc_crash_handler_work contention issue, and
> > last rproc_trigger_recovery bailed out with only
> > rproc->state==RPROC_OFFLINE, it is still worth to do pm_relax in pair.
> > Since the subsystem may still can be recovered with customer's next trigger
> > of rproc_start, and we can make each error out path clean with pm resources.
> > 
> > > I suggest spending time understanding what leads to the failure when recovering
> > > from a crash and address that problem(s).
> > > 
> > In current case, the customer's information is that the issue happened when
> > rproc_shutdown is triggered at similar time. So not an issue from error out
> > of rproc_trigger_recovery.
> 
> That is a very important element to consider and should have been mentioned from
> the beginning.  What I see happening is the following:
> 
> rproc_report_crash()
>         pm_stay_awake()
>         queue_work() // current thread is suspended
> 
> rproc_shutdown()
>         rproc_stop()
>                 rproc->state = RPROC_OFFLINE;
> 
> rproc_crash_handler_work()
>         if (rproc->state == RPROC_OFFLINE)
>                 return // pm_relax() is not called
> 
> The right way to fix this is to add a pm_relax() in rproc_shutdown() and
> rproc_detach(), along with a very descriptive comment as to why it is needed.

Thinking about this further there are more ramifications to consider.  Please
confirm the above scenario is what you are facing.  I will advise on how to move
forward if that is the case.

> 
> 
> > > Thanks,
> > > Mathieu
> > > 
> > > 
> > > > When the state is in RPROC_OFFLINE it means separate request
> > > > of rproc_stop was done and no need to hold the wakeup source
> > > > in crash handler to recover any more.
> > > > 
> > > > Signed-off-by: Maria Yu <quic_aiquny@quicinc.com>
> > > > ---
> > > >   drivers/remoteproc/remoteproc_core.c | 11 +++++++++++
> > > >   1 file changed, 11 insertions(+)
> > > > 
> > > > diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
> > > > index e5279ed9a8d7..6bc7b8b7d01e 100644
> > > > --- a/drivers/remoteproc/remoteproc_core.c
> > > > +++ b/drivers/remoteproc/remoteproc_core.c
> > > > @@ -1956,6 +1956,17 @@ static void rproc_crash_handler_work(struct work_struct *work)
> > > >   	if (rproc->state == RPROC_CRASHED || rproc->state == RPROC_OFFLINE) {
> > > >   		/* handle only the first crash detected */
> > > >   		mutex_unlock(&rproc->lock);
> > > > +		/*
> > > > +		 * RPROC_OFFLINE state indicate there is no recovery process
> > > > +		 * is in progress and no chance to have pm_relax in place.
> > > > +		 * Because when recovering from crash, rproc->lock is held and
> > > > +		 * state is RPROC_CRASHED -> RPROC_OFFLINE -> RPROC_RUNNING,
> > > > +		 * and then unlock rproc->lock.
> > > > +		 * RPROC_OFFLINE is only an intermediate state in recovery
> > > > +		 * process.
> > > > +		 */
> > > > +		if (rproc->state == RPROC_OFFLINE)
> > > > +			pm_relax(rproc->dev.parent);
> > > >   		return;
> > > >   	}
> > > > -- 
> > > > 2.7.4
> > > > 
> > 
> > 
> > -- 
> > Thx and BRs,
> > Aiqun(Maria) Yu