Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
Date:   Tue, 5 Apr 2022 13:51:34 -0300
From:   Jason Gunthorpe <jgg@ziepe.ca>
To:     Marc Zyngier <maz@kernel.org>
Cc:     xieming <xieming@kylinos.cn>, sashal@kernel.org,
        catalin.marinas@arm.com, linux@armlinux.org.uk,
        linux-kernel@vger.kernel.org, alex.williamson@redhat.com,
        will@kernel.org, kvmarm@lists.cs.columbia.edu,
        linux-arm-kernel@lists.infradead.org
Subject: Re: [PATCH v2] kvm/arm64: fixed passthrough gpu into vm on arm64
Message-ID: <20220405165134.GS64706@ziepe.ca>
References: <20220401090828.614167-1-xieming@kylinos.cn>
 <87tubcbvgk.wl-maz@kernel.org>
 <20220404132405.GQ64706@ziepe.ca>
 <87o81gc3dc.wl-maz@kernel.org>
 <20220404170202.GR64706@ziepe.ca>
 <87mtgzblez.wl-maz@kernel.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87mtgzblez.wl-maz@kernel.org>
Precedence: bulk

On Tue, Apr 05, 2022 at 04:27:16PM +0100, Marc Zyngier wrote:
> On Mon, 04 Apr 2022 18:02:02 +0100,
> Jason Gunthorpe <jgg@ziepe.ca> wrote:
> > 
> > On Mon, Apr 04, 2022 at 03:47:11PM +0100, Marc Zyngier wrote:
> > > > I'm guessing it will turn into a SBSA like thing where the ARM ARM is
> > > > kind of vauge but a SOC has to implement Normal-NC in a certain way to
> > > > be functional for the server market.
> > > 
> > > The main issue is that this equivalence isn't architected, so people
> > > can build whatever they want. SBSA means nothing to KVM (or Linux at
> > > large), and there is currently no way to describe which devices are
> > > safe to map as Normal-NC vs Device.
> > 
> > And people have, we know of some ARM SOC's that don't work fully with
> > NORMAL_NC for this usage. That is already a problem for baremetal
> > Linux, let alone KVM..
> > 
> > That is why I likened it to SBSA - if you want to build a server SOC
> > that works with existing server software, you have to support
> > NORMAL_NC in this way. Even if it isn't architected.
> 
> I see it the other way around. If it isn't architected (and in this
> case not even detectable in a scalable way), it simply isn't
> supportable by SW.

Except the software already supports it. Catalin decided NORMAL_NC
would be how Linux works in 2014 in commit de2db7432917 ("arm64: Make
DMA coherent and strongly ordered mappings not executable")

There are 47 places under drivers/ that call pgprot_writecombine()
already, and if you make a "server" kind of chip you are likely to
encounter these drivers and must support them. Linux has created a
de-facto spec here.

While I agree the current situation in ARM64 is not nice and could be
improved, it has been supported by SW this way for a long time
already.

> > I didn't quite understand your other remarks though - is there a
> > problem here? It seems like yes from the other thread you pointed at?
> 
> The main issue is that we have no idea what the behaviour is on a
> given implementation, and no way to even detect that for a given
> device, NORMAL_NC is a memory type that won't cause any issue.

I agree with this, but that is a driver problem for calling
pgprot_writecombine() not a KVM problem. vfio is just another driver
in this sense.

We already have arch_can_pci_mmap_wc() which is a half attempt to
solve this problem, but ARM64 doesn't wire it up.

We've also gone far enough down this path for long enough that we
can't break all the existing systems that are working this way
already. So I expect any future accomodation would be some FW
indication that NORMAL_NC doesn't work for pgprot_writecombine(),
probably in DT and probably for an embedded focused chip. Maybe
combined with a quirk list of non-working CPU IDs or something.

Wire it up to arch_can_pci_mmap_wc() and you hvae something reasonable -
except that none of the 47 drivers actually use this call
today. Sigh.

Thanks,
Jason