Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 31 May 2023 17:28:56 +0100
From:   Conor Dooley <conor@kernel.org>
To:     Jisheng Zhang <jszhang@kernel.org>
Cc:     Conor Dooley <conor.dooley@microchip.com>,
        Paul Walmsley <paul.walmsley@sifive.com>,
        Palmer Dabbelt <palmer@dabbelt.com>,
        Albert Ou <aou@eecs.berkeley.edu>,
        linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org,
        Catalin Marinas <catalin.marinas@arm.com>
Subject: Re: [PATCH 4/6] riscv: mm: pass noncoherent or not to
 riscv_noncoherent_supported()
Message-ID: <20230531-applied-antacid-77abfb5b2e55@spud>
References: <20230526165958.908-1-jszhang@kernel.org>
 <20230526165958.908-5-jszhang@kernel.org>
 <20230529-gainfully-ribbon-48520d25ef6e@wendy>
 <ZHdmnfJvF1N5rgvx@xhacker>
 <ZHdnliDU7ikKaFoX@xhacker>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
        protocol="application/pgp-signature"; boundary="BjOrWNtSKlv6yyL3"
Content-Disposition: inline
In-Reply-To: <ZHdnliDU7ikKaFoX@xhacker>
Precedence: bulk


--BjOrWNtSKlv6yyL3
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Wed, May 31, 2023 at 11:28:22PM +0800, Jisheng Zhang wrote:
> On Wed, May 31, 2023 at 11:24:19PM +0800, Jisheng Zhang wrote:
> > On Mon, May 29, 2023 at 12:13:10PM +0100, Conor Dooley wrote:
> > > On Sat, May 27, 2023 at 12:59:56AM +0800, Jisheng Zhang wrote:
> > > > We will soon take different actions by checking the HW is noncohere=
nt
> > > > or not, I.E ZICBOM/ERRATA_THEAD_CMO or not.
> > > >=20
> > > > Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
> > > > ---
> > > >  arch/riscv/errata/thead/errata.c    | 19 +++++++++++--------
> > > >  arch/riscv/include/asm/cacheflush.h |  4 ++--
> > > >  arch/riscv/kernel/setup.c           |  6 +++++-
> > > >  arch/riscv/mm/dma-noncoherent.c     | 10 ++++++----
> > > >  4 files changed, 24 insertions(+), 15 deletions(-)
> > > >=20
> > > > diff --git a/arch/riscv/errata/thead/errata.c b/arch/riscv/errata/t=
head/errata.c
> > > > index be84b14f0118..c192b80a5166 100644
> > > > --- a/arch/riscv/errata/thead/errata.c
> > > > +++ b/arch/riscv/errata/thead/errata.c
> > > > @@ -36,21 +36,24 @@ static bool errata_probe_pbmt(unsigned int stag=
e,
> > > >  static bool errata_probe_cmo(unsigned int stage,
> > > >  			     unsigned long arch_id, unsigned long impid)
> > > >  {
> > > > -	if (!IS_ENABLED(CONFIG_ERRATA_THEAD_CMO))
> > > > -		return false;
> > > > -
> > > > -	if (arch_id !=3D 0 || impid !=3D 0)
> > > > -		return false;
> > > > +	bool cmo;
> > > > =20
> > > >  	if (stage =3D=3D RISCV_ALTERNATIVES_EARLY_BOOT)
> > > >  		return false;
> > > > =20
> > > > +	if (IS_ENABLED(CONFIG_ERRATA_THEAD_CMO) &&
> > > > +	    (arch_id =3D=3D 0 && impid =3D=3D 0))
> > > > +		cmo =3D true;
> > > > +	else
> > > > +		cmo =3D false;
> > > > +
> > > >  	if (stage =3D=3D RISCV_ALTERNATIVES_BOOT) {
> > > > -		riscv_cbom_block_size =3D L1_CACHE_BYTES;
> > > > -		riscv_noncoherent_supported();
> > > > +		if (cmo)
> > > > +			riscv_cbom_block_size =3D L1_CACHE_BYTES;
> > > > +		riscv_noncoherent_supported(cmo);
> > > >  	}
> > > > =20
> > > > -	return true;
> > > > +	return cmo;
> > >=20
> > > I don't really understand the changes that you are making to this
> > > function, so that is tries really hard to call
> > > riscv_noncoherent_supported(). Why do we need to always call the func=
tion
> > > in the erratum's probe function, if the erratum is not detected, given
> >=20
> > In one unified kernel Image, to support both coherent and noncoherent
> > platforms(currently, either T-HEAD CMO or ZICBOM), we need to let the
> > kmalloc meet both cases, specifically, ARCH_DMA_MINALIGN aligned.
>=20
> seems adding three words can make it better:
>=20
> kmalloc meet both cases at the beginning, specifically ...
>=20
> > Once we know the underlying HW is coherent, I.E neither T-HEAD CMO nor
> > ZICBOM, we need to notice kmalloc we are safe to reduce the alignment
> > to 1. The notice action is done in patch 5:
> >=20
> > +       } else {
> > +               dma_cache_alignment =3D 1;
> >=20
> >=20
> > > that riscv_noncoherent_supported() is called immediately after
> > > apply_boot_alternatives() in setup_arch()?

This bit here is the key part of my confusion. You try really hard in
the errata stuff to call riscv_noncoherent_supported(), which I do
understand is because of the other branch that you add to the function
later in the series.

What I do not understand is why we are not able to rely on the call to
it in setup_arch() to trigger it when we do not have T-HEAD CMOs or
Zicbom.
You've explained why you want to make sure it always gets called during
boot, but my question is about why it looks like it is being called more
than once.

Actually, now that I think of it, what happens on a T-HEAD system where
there is no T-HEAD CMOs, but there is Zicbom. In theory, this could
exist.
Bear with me here a moment in case I am completely wrong, snippet is
=66rom setup_arch()
	apply_boot_alternatives();
On my example system, this will trigger, eventually sending us into
errata_probe_cmo(), where we will call riscv_noncoherent_supported()
with false, setting dma_cache_alignment to 1.

	if (IS_ENABLED(CONFIG_RISCV_ISA_ZICBOM) &&
	    riscv_isa_extension_available(NULL, ZICBOM))
		cmo =3D true;

On this system, this will be true.

	else
		cmo =3D false;
	riscv_noncoherent_supported(cmo);

now riscv_noncoherent_supported() is called with true, and we have
dma_cache_alignment =3D 1 still. Is that not problematic? Or the inverse,
where the T-HEAD system has its custom CMOs and there is no Zicbom, it
gets called twice with different args too.

There's clearly something fundamental that I am missing here, this seems
like it should be immediately obvious why this either cannot happen or
is not a problem, but I can't see it.

Sorry,
Conor.

--BjOrWNtSKlv6yyL3
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iHUEABYIAB0WIQRh246EGq/8RLhDjO14tDGHoIJi0gUCZHd1tQAKCRB4tDGHoIJi
0pGNAPwLSwfmm5OuyEL8BRsFeMVHyDtHAo2DcjggAIKEVSHDngEA0jAajERdcZXM
dZ5aJkHDIEmfSvibu/CfaGJlDgFknQY=
=AxKn
-----END PGP SIGNATURE-----

--BjOrWNtSKlv6yyL3--