Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <3e363da787126a4e8f779988ced92ae4624e3ec3.camel@linux.ibm.com>
Subject: Re: [PATCH v3 2/7] iommu: Allow .iotlb_sync_map to fail and handle
 s390's -ENOMEM return
From:   Niklas Schnelle <schnelle@linux.ibm.com>
To:     Alexandra Winter <wintera@linux.ibm.com>,
        Joerg Roedel <joro@8bytes.org>, Will Deacon <will@kernel.org>,
        Robin Murphy <robin.murphy@arm.com>,
        Jason Gunthorpe <jgg@nvidia.com>,
        Wenjia Zhang <wenjia@linux.ibm.com>
Cc:     Matthew Rosato <mjrosato@linux.ibm.com>,
        Gerd Bayer <gbayer@linux.ibm.com>,
        Pierre Morel <pmorel@linux.ibm.com>, iommu@lists.linux.dev,
        linux-s390@vger.kernel.org, borntraeger@linux.ibm.com,
        hca@linux.ibm.com, gor@linux.ibm.com,
        gerald.schaefer@linux.ibm.com, agordeev@linux.ibm.com,
        svens@linux.ibm.com, linux-kernel@vger.kernel.org,
        Julian Ruess <julianr@linux.ibm.com>
Date:   Tue, 03 Jan 2023 09:16:22 +0100
In-Reply-To: <2f1beb15-e9e4-d8ab-1b68-c83f1a53c5c5@linux.ibm.com>
References: <20230102115619.2088685-1-schnelle@linux.ibm.com>
         <20230102115619.2088685-3-schnelle@linux.ibm.com>
         <2f1beb15-e9e4-d8ab-1b68-c83f1a53c5c5@linux.ibm.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.46.2 (3.46.2-1.fc37) 
MIME-Version: 1.0
Precedence: bulk

On Mon, 2023-01-02 at 19:25 +0100, Alexandra Winter wrote:
>=20
> On 02.01.23 12:56, Niklas Schnelle wrote:
> > On s390 .iotlb_sync_map is used to sync mappings to an underlying
> > hypervisor by letting the hypervisor inspect the synced IOVA range and
> > updating its shadow table. This however means that it can fail as the
> > hypervisor may run out of resources. This can be due to the hypervisor
> > being unable to pin guest pages, due to a limit on concurrently mapped
> > addresses such as vfio_iommu_type1.dma_entry_limit or other resources.
> > Either way such a failure to sync a mapping should result in
> > a DMA_MAPPING_EROR.
> >=20
> > Now especially when running with batched IOTLB flushes for unmap it may
> > be that some IOVAs have already been invalidated but not yet synced via
> > .iotlb_sync_map. Thus if the hypervisor indicates running out of
> > resources, first do a global flush allowing the hypervisor to free
> > resources associated with these mappings and only if that also fails
> > report this error to callers.
> >=20
> > Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
> > ---
> Just a small typo, I noticed
> [...]

You mean the misspelled DMA_MAPPING_ERROR, right? Either way I did edit
the commit message for a bit more clarity on some of the details:

On s390 when using a paging hypervisor, .iotlb_sync_map is used to sync
mappings by letting the hypervisor inspect the synced IOVA range and
updating a shadow table. This however means that .iotlb_sync_map can
fail as the hypervisor may run out of resources while doing the sync.
This can be due to the hypervisor being unable to pin guest pages, due
to a limit on mapped addresses such as vfio_iommu_type1.dma_entry_limit
or lack of other resources. Either way such a failure to sync a mapping
should result in a DMA_MAPPING_ERROR.

Now especially when running with batched IOTLB flushes for unmap it may
be that some IOVAs have already been invalidated but not yet synced via
.iotlb_sync_map. Thus if the hypervisor indicates running out of
resources, first do a global flush allowing the hypervisor to free
resources associated with these mappings as well a retry creating the
new mappings and only if that also fails report this error to callers.


> > diff --git a/drivers/iommu/s390-iommu.c b/drivers/iommu/s390-iommu.c
> > index ed33c6cce083..6ba38b4f5b37 100644
> > --- a/drivers/iommu/s390-iommu.c
> > +++ b/drivers/iommu/s390-iommu.c
> > @@ -210,6 +210,14 @@ static void s390_iommu_release_device(struct devic=
e *dev)
> >  		__s390_iommu_detach_device(zdev);
> >  }
> > =20
> > +
> > +static int zpci_refresh_all(struct zpci_dev *zdev)
> > +{
> > +	return zpci_refresh_trans((u64)zdev->fh << 32, zdev->start_dma,
> > +				  zdev->end_dma - zdev->start_dma + 1);
> > +
> > +}
> > +
> >  static void s390_iommu_flush_iotlb_all(struct iommu_domain *domain)
> >  {
> >  	struct s390_domain *s390_domain =3D to_s390_domain(domain);
> > @@ -217,8 +225,7 @@ static void s390_iommu_flush_iotlb_all(struct iommu=
_domain *domain)
> > =20
> >  	rcu_read_lock();
> >  	list_for_each_entry_rcu(zdev, &s390_domain->devices, iommu_list) {
> > -		zpci_refresh_trans((u64)zdev->fh << 32, zdev->start_dma,
> > -				   zdev->end_dma - zdev->start_dma + 1);
> > +		zpci_refresh_all(zdev);
> >  	}
> >  	rcu_read_unlock();
> >  }
> > @@ -242,20 +249,32 @@ static void s390_iommu_iotlb_sync(struct iommu_do=
main *domain,
> >  	rcu_read_unlock();
> >  }
> > =20
> > -static void s390_iommu_iotlb_sync_map(struct iommu_domain *domain,
> > +static int s390_iommu_iotlb_sync_map(struct iommu_domain *domain,
> >  				      unsigned long iova, size_t size)
> >  {
> >  	struct s390_domain *s390_domain =3D to_s390_domain(domain);
> >  	struct zpci_dev *zdev;
> > +	int ret =3D 0;
> > =20
> >  	rcu_read_lock();
> >  	list_for_each_entry_rcu(zdev, &s390_domain->devices, iommu_list) {
> >  		if (!zdev->tlb_refresh)
> >  			continue;
> > -		zpci_refresh_trans((u64)zdev->fh << 32,
> > -				   iova, size);
> > +		ret =3D zpci_refresh_trans((u64)zdev->fh << 32,
> > +					 iova, size);
> > +		/*
> > +		 * let the hypervisor disover invalidated entries
> 			typo: s/disover/discover/g
> > +		 * allowing it to free IOVAs and unpin pages
> > +		 */
> > +		if (ret =3D=3D -ENOMEM) {
> > +			ret =3D zpci_refresh_all(zdev);
> > +			if (ret)
> > +				break;
> > +		}
> >  	}
> >  	rcu_read_unlock();
> > +
> > +	return ret;
> >  }
> > =20
> >  static int s390_iommu_validate_trans(struct s390_domain *s390_domain,
> [...]