Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <d2e13511df130c3d4824a78ed0aa24c49e7137e5.camel@linux.ibm.com>
Subject: Re: [PATCH 1/1] vfio/type1: Respect IOMMU reserved regions in
 vfio_test_domain_fgsp()
From:   Niklas Schnelle <schnelle@linux.ibm.com>
To:     Jason Gunthorpe <jgg@ziepe.ca>
Cc:     Alex Williamson <alex.williamson@redhat.com>,
        Cornelia Huck <cohuck@redhat.com>, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org,
        Matthew Rosato <mjrosato@linux.ibm.com>,
        Pierre Morel <pmorel@linux.ibm.com>,
        Christian =?ISO-8859-1?Q?Borntr=E4ger?= 
        <borntraeger@linux.ibm.com>
Date:   Wed, 04 Jan 2023 14:08:42 +0100
In-Reply-To: <Y7VuFJFUHtkqA9ZM@ziepe.ca>
References: <20230102093452.761185-1-schnelle@linux.ibm.com>
         <20230102093452.761185-2-schnelle@linux.ibm.com>
         <Y7S8loyvHyjAmNdh@ziepe.ca>
         <4e8d3f53af2f73a464e4ffc4a9a28c8d31692369.camel@linux.ibm.com>
         <Y7VuFJFUHtkqA9ZM@ziepe.ca>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.46.2 (3.46.2-1.fc37) 
MIME-Version: 1.0
Precedence: bulk

On Wed, 2023-01-04 at 08:16 -0400, Jason Gunthorpe wrote:
> On Wed, Jan 04, 2023 at 10:52:55AM +0100, Niklas Schnelle wrote:
>=20
> > The problem manifests only with ISM devices which are a special s390
> > virtual PCI device that is implemented in the machine hypervisor. This
> > device is used for high speed cross-LPAR (Logical Partition)
> > communication, basically it allows two LPARs that previously exchanged
> > an authentication token to memcpy between their partitioned memory
> > using the virtual device. For copying a receiving LPAR will IOMMU map a
> > region of memory for the ISM device that it will allow DMAing into
> > (memcpy by the hypervisor). All other regions remain unmapped and thus
> > inaccessible. In preparation the device  emulation in the machine
> > hypervisor intercepts the IOTLB flush and looks at the IOMMU
> > translation tables performing e.g. size and alignment checks I presume,
> > one of these checks against the start/end DMA boundaries. This check
> > fails which leads to the virtual ISM device being put into an error
> > state. Being in an error state it then fails to be initialized by the
> > guest driver later on.
>=20
> You could rephrase this as saying that the S390 map operation doesn't
> check for bounds so mapping in a reserved region doesn't fail, but
> errors the HW.
>=20
> Which seems reasonable to me
>=20
> Jason

Kind of yes, before the recent IOMMU changes the IOMMU code did check
on map failing early but now handles the limits via reserved regions.
The IOMMU hardware would only check the limits once an actual DMA uses
them but of course no DMA will be triggered for this test mapping. For
this specific virtual device though there is an extra check as part of
an intercepted IOTLB flush (RPCIT instruction in S390).