Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <0da03411190c004e85cc856965b0aca901fd78fb.camel@linux.ibm.com>
Subject: Re: [PATCH v4 5/5] iommu/s390: Fix incorrect pgsize_bitmap
From:   Niklas Schnelle <schnelle@linux.ibm.com>
To:     Robin Murphy <robin.murphy@arm.com>,
        Matthew Rosato <mjrosato@linux.ibm.com>,
        Pierre Morel <pmorel@linux.ibm.com>, iommu@lists.linux.dev
Cc:     linux-s390@vger.kernel.org, borntraeger@linux.ibm.com,
        hca@linux.ibm.com, gor@linux.ibm.com,
        gerald.schaefer@linux.ibm.com, agordeev@linux.ibm.com,
        svens@linux.ibm.com, joro@8bytes.org, will@kernel.org,
        jgg@nvidia.com, linux-kernel@vger.kernel.org
Date:   Tue, 04 Oct 2022 18:13:09 +0200
In-Reply-To: <a02a3cee-773a-da00-616f-04652c0905ce@arm.com>
References: <20221004120706.2957492-1-schnelle@linux.ibm.com>
         <20221004120706.2957492-6-schnelle@linux.ibm.com>
         <eb1955e4-a618-ebf0-562d-17f9dd58b0da@arm.com>
         <b3c6fd2b-74d4-91d7-dc53-ba526c41b067@linux.ibm.com>
         <a02a3cee-773a-da00-616f-04652c0905ce@arm.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Precedence: bulk

On Tue, 2022-10-04 at 16:31 +0100, Robin Murphy wrote:
> On 2022-10-04 16:12, Matthew Rosato wrote:
> > On 10/4/22 11:02 AM, Robin Murphy wrote:
> > > On 2022-10-04 13:07, Niklas Schnelle wrote:
> > > > The .pgsize_bitmap property of struct iommu_ops is not a page mask but
> > > > rather has a bit set for each size of pages the IOMMU supports. As the
> > > > comment correctly pointed out at this moment the code only support 4K
> > > > pages so simply use SZ_4K here.
> > > 
> > > Unless it's already been done somewhere else, you'll want to switch over to the {map,unmap}_pages() interfaces as well to avoid taking a hit on efficiency here. The "page mask" thing was an old hack to trick the core API into making fewer map/unmap calls where the driver could map arbitrary numbers of pages at once anyway. The multi-page interfaces now do that more honestly and generally better (since they work for non-power-of-two sizes as well).
> > 
> > Thanks for the heads up -- Niklas has some additional series coming soon as described here:
> > 
> > https://lore.kernel.org/linux-iommu/a10424adbe01a0fd40372cbd0736d11e517951a1.camel@linux.ibm.com/
> > 
> > So implementing the _pages() interfaces is soon up on the roadmap.  But given what you say I wonder if this patch should just wait until the series that implements {map,unmap}_pages().
> 
> Perhaps, although the full change should be trivial enough that there's 
> probably just as much argument for doing the whole thing in its own 
> right for the sake of this cleanup. The main point is that 
> S390_IOMMU_PGSIZES is not incorrect as such, it's just not spelling out 
> the deliberate trick that it's achieving - everyone copied it from 
> intel-iommu, but since that got converted to the new interfaces the 
> original explanation is now gone. The only effect of "fixing" it in 
> isolation right now will be to make large VFIO mappings slower.
> 
> Robin.

The patch changing to map_pages()/unmap_pages() is currently part of a
larger series of improvements, some of which are less trivial. So I'm
planning to send those as RFC first. Those include changing the
spin_lock protected list to RCU so the map/unmap can paralellize
better. Another one is atomic updates to the IOMMU tables to do away
with locks in map/unmap. So I think pulling that whole
series into this one isn't ideal. I could pull just the
map_pages()/unmap_pages() change though.

> 
> > > > Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
> > > > Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
> > > > ---
> > > >    drivers/iommu/s390-iommu.c | 9 +--------
> > > >    1 file changed, 1 insertion(+), 8 deletions(-)
> > > > 
> > > > diff --git a/drivers/iommu/s390-iommu.c b/drivers/iommu/s390-iommu.c
> > > > index 94c444b909bd..6bf23e7830a2 100644
> > > > --- a/drivers/iommu/s390-iommu.c
> > > > +++ b/drivers/iommu/s390-iommu.c
> > > > @@ -12,13 +12,6 @@
> > > >    #include <linux/sizes.h>
> > > >    #include <asm/pci_dma.h>
> > > >    -/*
> > > > - * Physically contiguous memory regions can be mapped with 4 KiB alignment,
> > > > - * we allow all page sizes that are an order of 4KiB (no special large page
> > > > - * support so far).
> > > > - */
> > > > -#define S390_IOMMU_PGSIZES    (~0xFFFUL)
> > > > -
> > > >    static const struct iommu_ops s390_iommu_ops;
> > > >      struct s390_domain {
> > > > @@ -350,7 +343,7 @@ static const struct iommu_ops s390_iommu_ops = {
> > > >        .probe_device = s390_iommu_probe_device,
> > > >        .release_device = s390_iommu_release_device,
> > > >        .device_group = generic_device_group,
> > > > -    .pgsize_bitmap = S390_IOMMU_PGSIZES,
> > > > +    .pgsize_bitmap = SZ_4K,
> > > >        .get_resv_regions = s390_iommu_get_resv_regions,
> > > >        .default_domain_ops = &(const struct iommu_domain_ops) {
> > > >            .attach_dev    = s390_iommu_attach_device,