Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Date:   Mon, 4 Apr 2022 14:02:02 -0300
From:   Jason Gunthorpe <jgg@ziepe.ca>
To:     Marc Zyngier <maz@kernel.org>
Cc:     xieming <xieming@kylinos.cn>, sashal@kernel.org,
        catalin.marinas@arm.com, linux@armlinux.org.uk,
        linux-kernel@vger.kernel.org, alex.williamson@redhat.com,
        will@kernel.org, kvmarm@lists.cs.columbia.edu,
        linux-arm-kernel@lists.infradead.org
Subject: Re: [PATCH v2] kvm/arm64: fixed passthrough gpu into vm on arm64
Message-ID: <20220404170202.GR64706@ziepe.ca>
References: <20220401090828.614167-1-xieming@kylinos.cn>
 <87tubcbvgk.wl-maz@kernel.org>
 <20220404132405.GQ64706@ziepe.ca>
 <87o81gc3dc.wl-maz@kernel.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87o81gc3dc.wl-maz@kernel.org>
Precedence: bulk

On Mon, Apr 04, 2022 at 03:47:11PM +0100, Marc Zyngier wrote:
> > I'm guessing it will turn into a SBSA like thing where the ARM ARM is
> > kind of vauge but a SOC has to implement Normal-NC in a certain way to
> > be functional for the server market.
> 
> The main issue is that this equivalence isn't architected, so people
> can build whatever they want. SBSA means nothing to KVM (or Linux at
> large), and there is currently no way to describe which devices are
> safe to map as Normal-NC vs Device.

And people have, we know of some ARM SOC's that don't work fully with
NORMAL_NC for this usage. That is already a problem for baremetal
Linux, let alone KVM..

That is why I likened it to SBSA - if you want to build a server SOC
that works with existing server software, you have to support
NORMAL_NC in this way. Even if it isn't architected.

The KVM challenge, at least, is to support a CPU with working
NORMAL_NC to create VM that emulates the same CPU with working
NORMAL_NC.

I didn't quite understand your other remarks though - is there a
problem here? It seems like yes from the other thread you pointed at?

I would think that KVM should mirror the process page table
configuration into the KVM page table and make this into a userspace
problem?

That turns it into a VFIO problem to negotiate with userspace and set
the proper pgprot. At least VFIO has a better chance than KVM to
consult DT or something to learn about the specific device's
properties.

I don't know how VFIO/qemu/etc can make this all work automatically
correctly 100% of the time. It seems to me it is the same problem as
just basic baremetal "WC" is troubled on ARM in general today. Maybe
some tables and a command line option in qemu is the best we can hope
for.

Long ago I asked that the ARM folks could come with some Linux
definition of all the WC-like modes and some arch calls to indicate
which one(s) should be used. Nobody seemed interested in doing that,
so the above SOC was left non-working in mainline Linux..

> We either have to take userspace's word for it, or rely on some other
> heuristics (do this for PCIe, but not anything else). None of which
> are entirely safe. Not to mention that no currently available CPU
> implements FEAT_DGH.

DHG is an optimization, not a functional requirement. Currently
available CPUs use one of the more expensive barriers that are
architected to include DHG behavior.

In any event, this is an important optimization. It is why ARMv9 is
introducing a new instruction specifically to optmize it.

Jason