Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 14 Nov 2022 14:09:04 +0000
From:   Will Deacon <will@kernel.org>
To:     Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
Cc:     Catalin Marinas <catalin.marinas@arm.com>,
        Amit Pundir <amit.pundir@linaro.org>,
        Robin Murphy <robin.murphy@arm.com>,
        Bjorn Andersson <andersson@kernel.org>,
        Sibi Sankar <quic_sibis@quicinc.com>,
        Linus Torvalds <torvalds@linux-foundation.org>,
        linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org,
        Dmitry Baryshkov <dmitry.baryshkov@linaro.org>
Subject: Re: [GIT PULL] arm64 updates for 6.1-rc1
Message-ID: <20221114140903.GF30263@willie-the-truck>
References: <20221005144116.2256580-1-catalin.marinas@arm.com>
 <CAMi1Hd38YB2m_r=m0wuDz+wErii37fUq1BJnvN9=y6opDzo_Fw@mail.gmail.com>
 <Y0aq8y5mEZi14lJ/@arm.com>
 <CAMi1Hd3Y9AibeVnh9_KYJ2EXar7bBSypKm4Tixj47htM7ZOURw@mail.gmail.com>
 <CAMi1Hd1VBCFhf7+EXWHQWcGy4k=tcyLa7RGiFdprtRnegSG0Mw@mail.gmail.com>
 <Y24uvyDJU3CL1jOi@arm.com>
 <20221111173952.GB5393@thinkpad>
 <20221111191820.GC5393@thinkpad>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20221111191820.GC5393@thinkpad>
User-Agent: Mutt/1.10.1 (2018-07-13)
Precedence: bulk

On Sat, Nov 12, 2022 at 12:48:20AM +0530, Manivannan Sadhasivam wrote:
> On Fri, Nov 11, 2022 at 11:10:01PM +0530, Manivannan Sadhasivam wrote:
> > On Fri, Nov 11, 2022 at 11:15:11AM +0000, Catalin Marinas wrote:
> > > On Tue, Nov 08, 2022 at 10:58:16PM +0530, Amit Pundir wrote:
> > > > On Tue, 25 Oct 2022 at 18:08, Amit Pundir <amit.pundir@linaro.org> wrote:
> > > > > On Wed, 12 Oct 2022 at 17:24, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > > > > > On Sat, Oct 08, 2022 at 08:28:26PM +0530, Amit Pundir wrote:
> > > > > > > On Wed, 5 Oct 2022 at 20:11, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > > > > > > > Will Deacon (2):
> > > > > > > >       arm64: dma: Drop cache invalidation from arch_dma_prep_coherent()
> > > > > > >
> > > > > > > This patch broke AOSP on Dragonboard 845c (SDM845). I don't see any
> > > > > > > relevant crash in the attached log and device silently reboots into
> > > > > > > USB crash dump mode. The crash is fairly reproducible on db845c. I
> > > > > > > could trigger it twice in 5 reboots and it always crash at the same
> > > > > > > point during the boot process. Reverting this patch fixes the crash.
> > > > > > >
> > > > > > > I'm happy to test run any debug patche(s), that would help narrow
> > > > > > > down this breakage.
> > > [...]
> > > > > Further narrowed down the breakage to the userspace daemon rmtfs
> > > > > https://github.com/andersson/rmtfs. Is there anything specific in the
> > > > > userspace code that I should be paying attention to?
> > > 
> > > Since you don't see anything in the logs like a crash and the system
> > > restarts, I suspect it's some deadlock and that's triggering the
> > > watchdog. We have an erratum (826319) but that's for Cortex-A53. IIUC
> > > SDM845 has Kryo 3xx series which based on some random google searches is
> > > derived from A75/A55. Unfortunately the MIDR_EL1 register doesn't match
> > > the Arm Ltd numbering, so I have no idea what CPUs these are by looking
> > > at the boot log.
> > > 
> > > I wouldn't be surprised if you hit a similar bug, though I couldn't find
> > > anything close in the A55 errata notice.
> > > 
> > > While we could revert commit c44094eee32f ("arm64: dma: Drop cache
> > > invalidation from arch_dma_prep_coherent()"), if you hit a real hardware
> > > issue it may trigger in other scenario where we only do cache cleaning
> > > (without invalidate), like arch_sync_dma_for_device(). So I'd rather get
> > > to the bottom of this and potentially enable the workaround for this
> > > chipset.
> > > 
> > > You could give it a quick try to by adding the MIDR ranges for SDM845 to
> > > struct midr_range workaround_clean_cache[].
> > > 
> > 
> > I gave it a shot and indeed it fixes the crash on DB845.
> > 
> > > After that I suggest you raise it with Qualcomm to investigate. Normally
> > > we ask for an erratum number to enable a workaround and it's only
> > > Qualcomm that can provide one here.
> > > 
> > 
> > I will check with Qualcomm folks and update.
> > 
> 
> I digged a little further and found that the crash was due to the secure
> processor (XPU) violation. It happens because, CPU tried acccessing the memory
> after sharing it with the modem for firmware metadata validation.

Can you share more details about this violation, please? For example, is it
s read or a write, what size is it, how is it detected?

> Sibi tried fixing this problem earlier by using a hack in the remoteproc driver
> [1], but I guess that got negated due to c44094eee32f?

Performing a clean rather than a clean+invalidate when the buffer is
allocated (which is what is achieved by c44094eee32f) shouldn't affect
this afaict.

> This is a common issue for other Qcom remoteproc drivers as well where CPU
> shares a chunk of memory with the modem. There is one more hack in place where
> the a chunk of memory is reserved and the driver will do memremap/copy the
> data/memunmap using it and share it with modem.
> 
> But is there a better solution overall that you could advise?

I think we need a better understanding of what Qualcomm's SCM firmware is
expecting about the state of the buffer pages being shared with the modem
before we can suggest other solutions.

Will