Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Fri, 11 Dec 2020 15:14:31 +0100
From:   Cornelia Huck <cohuck@redhat.com>
To:     Niklas Schnelle <schnelle@linux.ibm.com>
Cc:     Matthew Rosato <mjrosato@linux.ibm.com>,
        alex.williamson@redhat.com, pmorel@linux.ibm.com,
        borntraeger@de.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com,
        gerald.schaefer@linux.ibm.com, linux-s390@vger.kernel.org,
        kvm@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [RFC 0/4] vfio-pci/zdev: Fixing s390 vfio-pci ISM support
Message-ID: <20201211151431.75a4a3f4.cohuck@redhat.com>
In-Reply-To: <7bce88b2-8c7d-c0f4-89a0-b1e8f511ad0b@linux.ibm.com>
References: <1607545670-1557-1-git-send-email-mjrosato@linux.ibm.com>
        <20201210133306.70d1a556.cohuck@redhat.com>
        <ce9d4ef2-2629-59b7-99ed-4c8212cb004f@linux.ibm.com>
        <7bce88b2-8c7d-c0f4-89a0-b1e8f511ad0b@linux.ibm.com>
Organization: Red Hat GmbH
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Thu, 10 Dec 2020 17:14:24 +0100
Niklas Schnelle <schnelle@linux.ibm.com> wrote:

> On 12/10/20 4:51 PM, Matthew Rosato wrote:
> > On 12/10/20 7:33 AM, Cornelia Huck wrote: =20
> >> On Wed,=C2=A0 9 Dec 2020 15:27:46 -0500
> >> Matthew Rosato <mjrosato@linux.ibm.com> wrote:
> >> =20
> >>> Today, ISM devices are completely disallowed for vfio-pci passthrough=
 as
> >>> QEMU will reject the device due to an (inappropriate) MSI-X check.
> >>> However, in an effort to enable ISM device passthrough, I realized th=
at the
> >>> manner in which ISM performs block write operations is highly incompa=
tible
> >>> with the way that QEMU s390 PCI instruction interception and
> >>> vfio_pci_bar_rw break up I/O operations into 8B and 4B operations -- =
ISM
> >>> devices have particular requirements in regards to the alignment, siz=
e and
> >>> order of writes performed.=C2=A0 Furthermore, they require that legac=
y/non-MIO
> >>> s390 PCI instructions are used, which is also not guaranteed when the=
 I/O
> >>> is passed through the typical userspace channels. =20
> >>
> >> The part about the non-MIO instructions confuses me. How can MIO
> >> instructions be generated with the current code, and why does changing=
 =20
> >=20
> > So to be clear, they are not being generated at all in the guest as the=
 necessary facility is reported as unavailable.
> >=20
> > Let's talk about Linux in LPAR / the host kernel:=C2=A0 When hardware t=
hat supports MIO instructions is available, all userspace I/O traffic is go=
ing to be routed through the MIO variants of the s390 PCI instructions.=C2=
=A0 This is working well for other device types, but does not work for ISM =
which does not support these variants.=C2=A0 However, the ISM driver also d=
oes not invoke the userspace I/O routines for the kernel, it invokes the s3=
90 PCI layer directly, which in turn ensures the proper PCI instructions ar=
e used -- This approach falls apart when the guest ISM driver invokes those=
 routines in the guest -- we (qemu) pass those non-MIO instructions from th=
e guest as memory operations through vfio-pci, traversing through the vfio =
I/O layer in the guest (vfio_pci_bar_rw and friends), where we then arrive =
in the host s390 PCI layer -- where the MIO variant is used because the fac=
ility is available. =20
>=20
> Slight clarification since I think the word "userspace" is a bit overload=
ed as
> KVM folks often use it to talk about the guest even when that calls throu=
gh vfio.
> Application userspace (i.e. things like DPDK) can use PCI MIO Load/Stores
> directly on mmap()ed/ioremap()ed memory these don't go through the Kernel=
 at
> all.
> QEMU while also in userspace on the other hand goes through the vfio_bar_=
rw()
> region which uses the common code _Kernel_ ioread()/iowrite() API. This K=
ernel
> ioread()/iowrite() API uses PCI MIO Load/Stores by default on machines th=
at
> support them (z15 currently).  The ISM driver, knowing that its device do=
es not
> support MIO, goes around this API and directly calls zpci_store()/zpci_lo=
ad().

Ok, thanks for the explanation.

>=20
>=20
> >=20
> > Per conversations with Niklas (on CC), it's not trivial to decide by th=
e time we reach the s390 PCI I/O layer to switch gears and use the non-MIO =
instruction set. =20
>=20
> Yes, we have some ideas about dynamically switching to legacy PCI stores =
in
> ioread()/iowrite() for devices that are set up for it but since that only=
 gets
> an ioremap()ed address, a value and a size it would evolve such nasty thi=
ngs as
> looking at this virtual address to determine if it includes a ZPCI_ADDR()
> cookie that we use to get to the function handle needed for the legacy PCI
> Load/Stores, while MIO PCI Load/Stores directly work on virtual addresses.
>=20
> Now purely for the Kernel API we think this could work since that always
> allocates between VMALLOC_START and VMALLOC_END and we control where we p=
ut the
> ZPCI_ADDR() cookie but I'm very hesitant to add something like that.
>=20
> As for application userspace (DPDK) we do have a syscall
> (arch/s390/pci/pci_mmio.c) API that had a similar problem but we could ma=
ke use
> of the fact that our Architecture is pretty nifty with address spaces and=
 just
> execute the MIO PCI Load/Store in the syscall _as if_ by the calling user=
space
> application.

Is ISM (currently) the only device that needs to use the non-MIO
instructions, or are there others as well? Is there any characteristic
that a meta driver like vfio could discover, or is it a device quirk
you just need to know about?