Received-SPF: pass (google.com: domain of linux-kernel+bounces-70937-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Mon, 19 Feb 2024 14:02:20 +0530
From: Manivannan Sadhasivam <mani@kernel.org>
To: Manivannan Sadhasivam <mani@kernel.org>
Cc: Konrad Dybcio <konrad.dybcio@linaro.org>,
	Mrinmay Sarkar <quic_msarkar@quicinc.com>, agross@kernel.org,
	andersson@kernel.org, krzysztof.kozlowski+dt@linaro.org,
	conor+dt@kernel.org, robh+dt@kernel.org, quic_shazhuss@quicinc.com,
	quic_nitegupt@quicinc.com, quic_ramkri@quicinc.com,
	quic_nayiluri@quicinc.com, dmitry.baryshkov@linaro.org,
	robh@kernel.org, quic_krichai@quicinc.com,
	quic_vbadigan@quicinc.com, quic_parass@quicinc.com,
	quic_schintav@quicinc.com, quic_shijjose@quicinc.com,
	Lorenzo Pieralisi <lpieralisi@kernel.org>,
	Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= <kw@linux.com>,
	Bjorn Helgaas <bhelgaas@google.com>, linux-arm-msm@vger.kernel.org,
	devicetree@vger.kernel.org, linux-kernel@vger.kernel.org,
	linux-pci@vger.kernel.org
Subject: Re: [PATCH v4 1/3] PCI: qcom: Enable cache coherency for SA8775P RC\
Message-ID: <20240219083220.GA3281@thinkpad>
References: <1700577493-18538-1-git-send-email-quic_msarkar@quicinc.com>
 <1700577493-18538-2-git-send-email-quic_msarkar@quicinc.com>
 <20231130052116.GA3043@thinkpad>
 <a9c2532a-eaa6-4019-8ce9-5a58b1b720b2@linaro.org>
 <20231130110909.GQ3043@thinkpad>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <20231130110909.GQ3043@thinkpad>

On Thu, Nov 30, 2023 at 04:39:09PM +0530, Manivannan Sadhasivam wrote:
> On Thu, Nov 30, 2023 at 11:09:59AM +0100, Konrad Dybcio wrote:
> > On 30.11.2023 06:21, Manivannan Sadhasivam wrote:
> > > On Tue, Nov 21, 2023 at 08:08:11PM +0530, Mrinmay Sarkar wrote:
> > >> In a multiprocessor system cache snooping maintains the consistency
> > >> of caches. Snooping logic is disabled from HW on this platform.
> > >> Cache coherency doesn’t work without enabling this logic.
> > >>
> > >> 8775 has IP version 1.34.0 so intruduce a new cfg(cfg_1_34_0) for this
> > >> platform. Assign no_snoop_override flag into struct qcom_pcie_cfg and
> > >> set it true in cfg_1_34_0 and enable cache snooping if this particular
> > >> flag is true.
> > >>
> > > 
> > > I just happen to check the internal register details of other platforms and I
> > > see this PCIE_PARF_NO_SNOOP_OVERIDE register with the reset value of 0x0. So
> > > going by the logic of this patch, this register needs to be configured for other
> > > platforms as well to enable cache coherency, but it seems like not the case as
> > > we never did and all are working fine (so far no issues reported).
> > 
> > Guess we know that already [1]
> > 
> 
> Bummer! I didn't look close into that reply :/
> 
> > The question is whether this override is necessary, or the default
> > internal state is OK on other platforms
> > 
> 
> I digged into it further...
> 
> The register description says "Enable this bit x to override no_snoop". So
> NO_SNOOP is the default behavior unless bit x is set in this register.
> 
> This means if bit x is set, MRd and MWd TLPs originating from the desired PCIe
> controller (Requester) will have the NO_SNOOP bit set in the header. So the
> completer will not do any cache management for the transaction. But this also
> requires that the address referenced by the TLP is not cacheable.
> 
> My guess here is that, hw designers have enabled the NO_SNOOP logic by default
> and running into coherency issues on the completer side. Maybe due to the
> addresses are cacheable always (?).
> 
> And the default value of this register has no impact on the NO_SNOOP attribute
> unless specific bits are set.
> 
> But I need to confirm my above observations with HW team. Until then, I will
> hold on to my Nack.
> 

I had some discussions with the hardware folks and clarified my concerns with
them. Here is the summary:

Due to some hardware changes, SA8775P has set the NO_SNOOP attribute in its TLP
for all the PCIe controllers. NO_SNOOP attribute when set, the requester is
indicating that there no cache coherency issues exit for the addressed memory
on the host i.e., memory is not cached. But in reality, requester cannot assume
this unless there is a complete control/visibility over the addressed memory on
the host.

And worst case, if the memory is cached on the host, it may lead to memory
corruption issues. It should be noted that the caching of memory on the host is
not solely dependent on the NO_SNOOP attribute in TLP.

So to avoid the corruption, this patch overrides the NO_SNOOP attribute by
setting the PCIE_PARF_NO_SNOOP_OVERIDE register. This patch is not needed for
other upstream supported platforms since they do not set NO_SNOOP attribute by
default.

Mrinmay, please add above information in the commit message while sending v2.
I'm taking by NACK back.

- Mani

-- 
மணிவண்ணன் சதாசிவம்