MIME-Version: 1.0
References: <20230314085433.4078119-1-chenhuacai@loongson.cn> <21f5aba50b7732b48d2610a5015186adf6bdfb38.camel@xry111.site>
In-Reply-To: <21f5aba50b7732b48d2610a5015186adf6bdfb38.camel@xry111.site>
From:   Huacai Chen <chenhuacai@kernel.org>
Date:   Tue, 14 Mar 2023 18:02:21 +0800
Message-ID: <CAAhV-H4TpQuq9sE2H=zbO8cvwn3Aer_pu1a-DU9L9OSR1bQX=g@mail.gmail.com>
Subject: Re: [PATCH] LoongArch: Make WriteCombine configurable for ioremap()
To:     Xi Ruoyao <xry111@xry111.site>
Cc:     Huacai Chen <chenhuacai@loongson.cn>, loongarch@lists.linux.dev,
        Xuefeng Li <lixuefeng@loongson.cn>,
        Guo Ren <guoren@kernel.org>, Xuerui Wang <kernel@xen0n.name>,
        Jiaxun Yang <jiaxun.yang@flygoat.com>,
        linux-kernel@vger.kernel.org, loongson-kernel@lists.loongnix.cn
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

Hi, Ruoyao,

On Tue, Mar 14, 2023 at 5:41=E2=80=AFPM Xi Ruoyao <xry111@xry111.site> wrot=
e:
>
> On Tue, 2023-03-14 at 16:54 +0800, Huacai Chen wrote:
> > LoongArch maintains cache coherency in hardware, but when works with
> > LS7A chipsets the WUC attribute (Weak-ordered UnCached, which is simila=
r
> > to WriteCombine) is out of the scope of cache coherency machanism for
> > PCIe devices (this is a PCIe protocol violation, may be fixed in newer
> > chipsets).
>
> IIUC all launched LS7A models (7A1000 and 7A2000) suffers this issue?
Yes, very unfortunately, but this issue is only observed in the amdgpu
driver now.

>
> > This means WUC can only used for write-only memory regions now, so this
> > option is disabled by default (means WUC falls back to SUC for ioremap)=
.
> > You can enable this option if the kernel is ensured to run on bug-free
> > hardwares.
>
> Hmm, is it possible to make a PCI quirk so SUC/WUC will be decided
> automatically from the vendor:device ID of the PCI root controller?
> Then we don't need to rely on the user or distro maintainer to select an
> option.  I see there is already many architecture-dependant #if
> directives in drivers/pci/quirks.c so I guess such a quirk is acceptable
> in PCI tree...
Not a good idea, pci quirks need too long a time to review, and we
don't know when this issue can be fixed in hardware.

>
> If a PCI quirk is not possible, then is it possible to make a kernel
> command line option, leaving this CONFIG as the default value of the
> option?  I guess in the future many LoongArch users will just install a
> binary distro, then it would be much easier to edit grub.cfg than
> rebuilding the kernel when they finally buy a compliant PCIe controller.
If we use command line parameter, we can remove this Kconfig option.

Huacai
>
> > Suggested-by: WANG Xuerui <kernel@xen0n.name>
> > Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
> > ---
> >  arch/loongarch/Kconfig          | 14 ++++++++++++++
> >  arch/loongarch/include/asm/io.h |  5 +++++
> >  2 files changed, 19 insertions(+)
> >
> > diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig
> > index 0d11738a861a..e3f5c422636f 100644
> > --- a/arch/loongarch/Kconfig
> > +++ b/arch/loongarch/Kconfig
> > @@ -446,6 +446,20 @@ config ARCH_IOREMAP
> >           protection support. However, you can enable LoongArch DMW-bas=
ed
> >           ioremap() for better performance.
> >
> > +config ARCH_WRITECOMBINE
> > +       bool "Enable WriteCombine (WUC) for ioremap()"
> > +       help
> > +         LoongArch maintains cache coherency in hardware, but with LS7=
A
> > +         chipsets the WUC attribute (Weak-ordered UnCached, which is s=
imilar
> > +         to WriteCombine) is out of the scope of cache coherency macha=
nism
> > +         for PCIe devices (this is a PCIe protocol violation, may be f=
ixed
> > +         in newer chipsets).
> > +
> > +         This means WUC can only used for write-only memory regions no=
w, so
> > +         this option is disabled by default (means WUC falls back to S=
UC for
> > +         ioremap). You can enable this option if the kernel is ensured=
 to run
> > +         on bug-free hardwares.
> > +
> >  config ARCH_STRICT_ALIGN
> >         bool "Enable -mstrict-align to prevent unaligned accesses" if E=
XPERT
> >         default y
> > diff --git a/arch/loongarch/include/asm/io.h b/arch/loongarch/include/a=
sm/io.h
> > index 402a7d9e3a53..079ef897ed1a 100644
> > --- a/arch/loongarch/include/asm/io.h
> > +++ b/arch/loongarch/include/asm/io.h
> > @@ -54,8 +54,13 @@ static inline void __iomem *ioremap_prot(phys_addr_t=
 offset, unsigned long size,
> >   * @offset:    bus address of the memory
> >   * @size:      size of the resource to map
> >   */
> > +#ifdef CONFIG_ARCH_WRITECOMBINE
> >  #define ioremap_wc(offset, size)       \
> >         ioremap_prot((offset), (size), pgprot_val(PAGE_KERNEL_WUC))
> > +#else
> > +#define ioremap_wc(offset, size)       \
> > +       ioremap_prot((offset), (size), pgprot_val(PAGE_KERNEL_SUC))
> > +#endif
> >
> >  #define ioremap_cache(offset, size)    \
> >         ioremap_prot((offset), (size), pgprot_val(PAGE_KERNEL))
>
> --
> Xi Ruoyao <xry111@xry111.site>
> School of Aerospace Science and Technology, Xidian University