Hi All,
This is v5 of a follow up to Matt's recent series[0] where he tackled
a race that turned out to be outside of the s390 IOMMU driver itself as
well as duplicate device attachments. After an internal discussion we came
up with what I believe is a cleaner fix. Instead of actively checking for
duplicates we instead detach from any previous domain on attach. From my
cursory reading of the code this seems to be what the Intel IOMMU driver is
doing as well.
Moreover we drop the attempt to re-attach the device to its previous IOMMU
domain on failure. This was fragile, unlikely to help and unexpected for
calling code. Thanks Jason for the suggestion.
We can also get rid of struct s390_domain_device entirely if we instead
thread the list through the attached struct zpci_devs. This saves us from
having to allocate during attach and gets rid of one level of indirection
during IOMMU operations.
Additionally 3 more fixes have been added in v3 that weren't in v2 of this
series. One is for a potential situation where the aperture of a domain
could shrink and leave invalid translations. The next one fixes an off by
one in checking validity of an IOVA and the last one fixes a wrong value
for pgsize_bitmap.
In v4 we also add a patch changing to the map_pages()/unmap_pages()
interface in order to prevent a performance regression due to the
pgsize_bitmap change.
*Note*:
This series is against the s390 features branch[1] which already contains
the bus_next field removal that was part of v2.
It is also available as a branch with the GPG signed tag
s390_iommu_fixes_v5 on my niks/linux.git on git.kernel.org[2].
*Open Question*:
Which tree should this go via?
Best regards,
Niklas
Changes since v4:
- Add patch to change to the map_pages()/unmap_pages() API to prevent
a performance regression from the pgsize_bitmap change (Robin)
- In patch 1 unregister IOAT on error (Matt)
- Turn the aperture check in attach into a WARN_ON() in patch 3 (Jason)
Changes since v3:
- Drop s390_domain from __s390_iommu_detach_device() (Jason)
- WARN_ON() mismatched domain in s390_iommu_detach_device() (Jason)
- Use __s390_iommu_detach_device() in s390_iommu_release_device() (Jason)
- Make aperture check resistant against overflow (Jason)
Changes since v2:
- The patch removing the unused bus_next field has been spun out and
already made it into the s390 feature branch on git.kernel.org
- Make __s390_iommu_detach_device() return void (Jason)
- Remove the re-attach on failure dance as it is unlikely to help
and complicates debug and recovery (Jason)
- Ignore attempts to detach from domain that is not the active one
- Add patch to fix potential shrinking of the aperture and use
reserved ranges per device instead of the aperture to respect
IOVA range restrictions (Jason)
- Add a fix for an off by one error on checking an IOVA against
the aperture
- Add a fix for wrong pgsize_bitmap
Changes since v1:
- After patch 3 we don't have to search in the devices list on detach as
we alreadz have hold of the zpci_dev (Jason)
- Add a WARN_ON() if somehow ended up detaching a device from a domain that
isn't the device's current domain.
- Removed the iteration and list delete from s390_domain_free() instead
just WARN_ON() when we're freeing without having detached
- The last two points should help catching sequencing errors much more
quickly in the future.
[0] https://lore.kernel.org/linux-iommu/[email protected]/
[1] https://git.kernel.org/pub/scm/linux/kernel/git/s390/linux.git/
[2] https://git.kernel.org/pub/scm/linux/kernel/git/niks/linux.git
Niklas Schnelle (6):
iommu/s390: Fix duplicate domain attachments
iommu/s390: Get rid of s390_domain_device
iommu/s390: Fix potential s390_domain aperture shrinking
iommu/s390: Fix incorrect aperture check
iommu/s390: Fix incorrect pgsize_bitmap
iommu/s390: Implement map_pages()/unmap_pages() instead of
map()/unmap()
arch/s390/include/asm/pci.h | 1 +
drivers/iommu/s390-iommu.c | 221 +++++++++++++++++-------------------
2 files changed, 107 insertions(+), 115 deletions(-)
--
2.34.1
The .pgsize_bitmap property of struct iommu_ops is not a page mask but
rather has a bit set for each size of pages the IOMMU supports. As the
comment correctly pointed out at this moment the code only support 4K
pages so simply use SZ_4K here.
Reviewed-by: Matthew Rosato <[email protected]>
Reviewed-by: Jason Gunthorpe <[email protected]>
Signed-off-by: Niklas Schnelle <[email protected]>
---
drivers/iommu/s390-iommu.c | 9 +--------
1 file changed, 1 insertion(+), 8 deletions(-)
diff --git a/drivers/iommu/s390-iommu.c b/drivers/iommu/s390-iommu.c
index a89fd0256f99..ac200f0b81fa 100644
--- a/drivers/iommu/s390-iommu.c
+++ b/drivers/iommu/s390-iommu.c
@@ -12,13 +12,6 @@
#include <linux/sizes.h>
#include <asm/pci_dma.h>
-/*
- * Physically contiguous memory regions can be mapped with 4 KiB alignment,
- * we allow all page sizes that are an order of 4KiB (no special large page
- * support so far).
- */
-#define S390_IOMMU_PGSIZES (~0xFFFUL)
-
static const struct iommu_ops s390_iommu_ops;
struct s390_domain {
@@ -356,7 +349,7 @@ static const struct iommu_ops s390_iommu_ops = {
.probe_device = s390_iommu_probe_device,
.release_device = s390_iommu_release_device,
.device_group = generic_device_group,
- .pgsize_bitmap = S390_IOMMU_PGSIZES,
+ .pgsize_bitmap = SZ_4K,
.get_resv_regions = s390_iommu_get_resv_regions,
.default_domain_ops = &(const struct iommu_domain_ops) {
.attach_dev = s390_iommu_attach_device,
--
2.34.1
Since commit fa7e9ecc5e1c ("iommu/s390: Tolerate repeat attach_dev
calls") we can end up with duplicates in the list of devices attached to
a domain. This is inefficient and confusing since only one domain can
actually be in control of the IOMMU translations for a device. Fix this
by detaching the device from the previous domain, if any, on attach.
Add a WARN_ON() in case we still have attached devices on freeing the
domain. While here remove the re-attach on failure dance as it was
determined to be unlikely to help and may confuse debug and recovery.
Fixes: fa7e9ecc5e1c ("iommu/s390: Tolerate repeat attach_dev calls")
Signed-off-by: Niklas Schnelle <[email protected]>
---
v4->v5:
- Unregister IOAT and set zdev->dma_table on error (Matt)
drivers/iommu/s390-iommu.c | 102 ++++++++++++++++---------------------
1 file changed, 43 insertions(+), 59 deletions(-)
diff --git a/drivers/iommu/s390-iommu.c b/drivers/iommu/s390-iommu.c
index c898bcbbce11..938998c46bd3 100644
--- a/drivers/iommu/s390-iommu.c
+++ b/drivers/iommu/s390-iommu.c
@@ -79,10 +79,36 @@ static void s390_domain_free(struct iommu_domain *domain)
{
struct s390_domain *s390_domain = to_s390_domain(domain);
+ WARN_ON(!list_empty(&s390_domain->devices));
dma_cleanup_tables(s390_domain->dma_table);
kfree(s390_domain);
}
+static void __s390_iommu_detach_device(struct zpci_dev *zdev)
+{
+ struct s390_domain *s390_domain = zdev->s390_domain;
+ struct s390_domain_device *domain_device, *tmp;
+ unsigned long flags;
+
+ if (!s390_domain)
+ return;
+
+ spin_lock_irqsave(&s390_domain->list_lock, flags);
+ list_for_each_entry_safe(domain_device, tmp, &s390_domain->devices,
+ list) {
+ if (domain_device->zdev == zdev) {
+ list_del(&domain_device->list);
+ kfree(domain_device);
+ break;
+ }
+ }
+ spin_unlock_irqrestore(&s390_domain->list_lock, flags);
+
+ zpci_unregister_ioat(zdev, 0);
+ zdev->s390_domain = NULL;
+ zdev->dma_table = NULL;
+}
+
static int s390_iommu_attach_device(struct iommu_domain *domain,
struct device *dev)
{
@@ -90,7 +116,7 @@ static int s390_iommu_attach_device(struct iommu_domain *domain,
struct zpci_dev *zdev = to_zpci_dev(dev);
struct s390_domain_device *domain_device;
unsigned long flags;
- int cc, rc;
+ int cc, rc = 0;
if (!zdev)
return -ENODEV;
@@ -99,23 +125,17 @@ static int s390_iommu_attach_device(struct iommu_domain *domain,
if (!domain_device)
return -ENOMEM;
- if (zdev->dma_table && !zdev->s390_domain) {
- cc = zpci_dma_exit_device(zdev);
- if (cc) {
- rc = -EIO;
- goto out_free;
- }
- }
-
if (zdev->s390_domain)
- zpci_unregister_ioat(zdev, 0);
+ __s390_iommu_detach_device(zdev);
+ else if (zdev->dma_table)
+ zpci_dma_exit_device(zdev);
zdev->dma_table = s390_domain->dma_table;
cc = zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
virt_to_phys(zdev->dma_table));
if (cc) {
rc = -EIO;
- goto out_restore;
+ goto out_free;
}
spin_lock_irqsave(&s390_domain->list_lock, flags);
@@ -127,9 +147,9 @@ static int s390_iommu_attach_device(struct iommu_domain *domain,
/* Allow only devices with identical DMA range limits */
} else if (domain->geometry.aperture_start != zdev->start_dma ||
domain->geometry.aperture_end != zdev->end_dma) {
- rc = -EINVAL;
spin_unlock_irqrestore(&s390_domain->list_lock, flags);
- goto out_restore;
+ rc = -EINVAL;
+ goto out_unregister;
}
domain_device->zdev = zdev;
zdev->s390_domain = s390_domain;
@@ -138,14 +158,9 @@ static int s390_iommu_attach_device(struct iommu_domain *domain,
return 0;
-out_restore:
- if (!zdev->s390_domain) {
- zpci_dma_init_device(zdev);
- } else {
- zdev->dma_table = zdev->s390_domain->dma_table;
- zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
- virt_to_phys(zdev->dma_table));
- }
+out_unregister:
+ zpci_unregister_ioat(zdev, 0);
+ zdev->dma_table = NULL;
out_free:
kfree(domain_device);
@@ -155,32 +170,12 @@ static int s390_iommu_attach_device(struct iommu_domain *domain,
static void s390_iommu_detach_device(struct iommu_domain *domain,
struct device *dev)
{
- struct s390_domain *s390_domain = to_s390_domain(domain);
struct zpci_dev *zdev = to_zpci_dev(dev);
- struct s390_domain_device *domain_device, *tmp;
- unsigned long flags;
- int found = 0;
- if (!zdev)
- return;
-
- spin_lock_irqsave(&s390_domain->list_lock, flags);
- list_for_each_entry_safe(domain_device, tmp, &s390_domain->devices,
- list) {
- if (domain_device->zdev == zdev) {
- list_del(&domain_device->list);
- kfree(domain_device);
- found = 1;
- break;
- }
- }
- spin_unlock_irqrestore(&s390_domain->list_lock, flags);
+ WARN_ON(zdev->s390_domain != to_s390_domain(domain));
- if (found && (zdev->s390_domain == s390_domain)) {
- zdev->s390_domain = NULL;
- zpci_unregister_ioat(zdev, 0);
- zpci_dma_init_device(zdev);
- }
+ __s390_iommu_detach_device(zdev);
+ zpci_dma_init_device(zdev);
}
static struct iommu_device *s390_iommu_probe_device(struct device *dev)
@@ -193,24 +188,13 @@ static struct iommu_device *s390_iommu_probe_device(struct device *dev)
static void s390_iommu_release_device(struct device *dev)
{
struct zpci_dev *zdev = to_zpci_dev(dev);
- struct iommu_domain *domain;
/*
- * This is a workaround for a scenario where the IOMMU API common code
- * "forgets" to call the detach_dev callback: After binding a device
- * to vfio-pci and completing the VFIO_SET_IOMMU ioctl (which triggers
- * the attach_dev), removing the device via
- * "echo 1 > /sys/bus/pci/devices/.../remove" won't trigger detach_dev,
- * only release_device will be called via the BUS_NOTIFY_REMOVED_DEVICE
- * notifier.
- *
- * So let's call detach_dev from here if it hasn't been called before.
+ * release_device is expected to detach any domain currently attached
+ * to the device, but keep it attached to other devices in the group.
*/
- if (zdev && zdev->s390_domain) {
- domain = iommu_get_domain_for_dev(dev);
- if (domain)
- s390_iommu_detach_device(domain, dev);
- }
+ if (zdev)
+ __s390_iommu_detach_device(zdev);
}
static int s390_iommu_update_trans(struct s390_domain *s390_domain,
--
2.34.1
On 10/6/22 10:46 AM, Niklas Schnelle wrote:
> Since commit fa7e9ecc5e1c ("iommu/s390: Tolerate repeat attach_dev
> calls") we can end up with duplicates in the list of devices attached to
> a domain. This is inefficient and confusing since only one domain can
> actually be in control of the IOMMU translations for a device. Fix this
> by detaching the device from the previous domain, if any, on attach.
> Add a WARN_ON() in case we still have attached devices on freeing the
> domain. While here remove the re-attach on failure dance as it was
> determined to be unlikely to help and may confuse debug and recovery.
>
> Fixes: fa7e9ecc5e1c ("iommu/s390: Tolerate repeat attach_dev calls")
> Signed-off-by: Niklas Schnelle <[email protected]>
> ---
> v4->v5:
> - Unregister IOAT and set zdev->dma_table on error (Matt)
>
...
> static int s390_iommu_attach_device(struct iommu_domain *domain,
> struct device *dev)
> {
> @@ -90,7 +116,7 @@ static int s390_iommu_attach_device(struct iommu_domain *domain,
> struct zpci_dev *zdev = to_zpci_dev(dev);
> struct s390_domain_device *domain_device;
> unsigned long flags;
> - int cc, rc;
> + int cc, rc = 0;
>
> if (!zdev)
> return -ENODEV;
> @@ -99,23 +125,17 @@ static int s390_iommu_attach_device(struct iommu_domain *domain,
> if (!domain_device)
> return -ENOMEM;
>
> - if (zdev->dma_table && !zdev->s390_domain) {
> - cc = zpci_dma_exit_device(zdev);
> - if (cc) {
> - rc = -EIO;
> - goto out_free;
> - }
> - }
> -
> if (zdev->s390_domain)
> - zpci_unregister_ioat(zdev, 0);
> + __s390_iommu_detach_device(zdev);
> + else if (zdev->dma_table)
> + zpci_dma_exit_device(zdev);
>
> zdev->dma_table = s390_domain->dma_table;
> cc = zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
> virt_to_phys(zdev->dma_table));
> if (cc) {
> rc = -EIO;
> - goto out_restore;
> + goto out_free;
> }
Hmm, with this we will leave attach_dev with a zdev->dma_table associated with this domain (not one generated via zpci_dma_init_device) and zdev->s390_domain == 0. Won't this cause both s390_domain_free and zpci_dma_exit_device() to try and free the same dma table?
I think we also have to leave with a NULL zdev->dma_table in this case too (you technically could skip the zpci_unregister_ioat)
>
> spin_lock_irqsave(&s390_domain->list_lock, flags);
> @@ -127,9 +147,9 @@ static int s390_iommu_attach_device(struct iommu_domain *domain,
> /* Allow only devices with identical DMA range limits */
> } else if (domain->geometry.aperture_start != zdev->start_dma ||
> domain->geometry.aperture_end != zdev->end_dma) {
> - rc = -EINVAL;
> spin_unlock_irqrestore(&s390_domain->list_lock, flags);
> - goto out_restore;
> + rc = -EINVAL;
> + goto out_unregister;
> }
> domain_device->zdev = zdev;
> zdev->s390_domain = s390_domain;
> @@ -138,14 +158,9 @@ static int s390_iommu_attach_device(struct iommu_domain *domain,
>
> return 0;
>
> -out_restore:
> - if (!zdev->s390_domain) {
> - zpci_dma_init_device(zdev);
> - } else {
> - zdev->dma_table = zdev->s390_domain->dma_table;
> - zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
> - virt_to_phys(zdev->dma_table));
> - }
> +out_unregister:
> + zpci_unregister_ioat(zdev, 0);
> + zdev->dma_table = NULL;
> out_free:
> kfree(domain_device);
>
On Thu, 2022-10-06 at 17:02 -0400, Matthew Rosato wrote:
> On 10/6/22 10:46 AM, Niklas Schnelle wrote:
> > Since commit fa7e9ecc5e1c ("iommu/s390: Tolerate repeat attach_dev
> > calls") we can end up with duplicates in the list of devices attached to
> > a domain. This is inefficient and confusing since only one domain can
> > actually be in control of the IOMMU translations for a device. Fix this
> > by detaching the device from the previous domain, if any, on attach.
> > Add a WARN_ON() in case we still have attached devices on freeing the
> > domain. While here remove the re-attach on failure dance as it was
> > determined to be unlikely to help and may confuse debug and recovery.
> >
> > Fixes: fa7e9ecc5e1c ("iommu/s390: Tolerate repeat attach_dev calls")
> > Signed-off-by: Niklas Schnelle <[email protected]>
> > ---
> > v4->v5:
> > - Unregister IOAT and set zdev->dma_table on error (Matt)
> >
> ...
>
> > static int s390_iommu_attach_device(struct iommu_domain *domain,
> > struct device *dev)
> > {
> > @@ -90,7 +116,7 @@ static int s390_iommu_attach_device(struct iommu_domain *domain,
> > struct zpci_dev *zdev = to_zpci_dev(dev);
> > struct s390_domain_device *domain_device;
> > unsigned long flags;
> > - int cc, rc;
> > + int cc, rc = 0;
> >
> > if (!zdev)
> > return -ENODEV;
> > @@ -99,23 +125,17 @@ static int s390_iommu_attach_device(struct iommu_domain *domain,
> > if (!domain_device)
> > return -ENOMEM;
> >
> > - if (zdev->dma_table && !zdev->s390_domain) {
> > - cc = zpci_dma_exit_device(zdev);
> > - if (cc) {
> > - rc = -EIO;
> > - goto out_free;
> > - }
> > - }
> > -
> > if (zdev->s390_domain)
> > - zpci_unregister_ioat(zdev, 0);
> > + __s390_iommu_detach_device(zdev);
> > + else if (zdev->dma_table)
> > + zpci_dma_exit_device(zdev);
> >
> > zdev->dma_table = s390_domain->dma_table;
> > cc = zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
> > virt_to_phys(zdev->dma_table));
> > if (cc) {
> > rc = -EIO;
> > - goto out_restore;
> > + goto out_free;
> > }
>
> Hmm, with this we will leave attach_dev with a zdev->dma_table associated with this domain (not one generated via zpci_dma_init_device) and zdev->s390_domain == 0. Won't this cause both s390_domain_free and zpci_dma_exit_device() to try and free the same dma table?
>
> I think we also have to leave with a NULL zdev->dma_table in this case too (you technically could skip the zpci_unregister_ioat)
Argh you're right. This is I think a a bad rebase, in v4 I had the
zpci_register_ioat() use s390_domain->dma_table and only set zdev-
>dma_table after that succeeded. I seem to have lost that part
somewhere along the way. With that we zdev->dma_table would be NULL and
all would be good.
>
> >
> > spin_lock_irqsave(&s390_domain->list_lock, flags);
> > @@ -127,9 +147,9 @@ static int s390_iommu_attach_device(struct iommu_domain *domain,
> > /* Allow only devices with identical DMA range limits */
> > } else if (domain->geometry.aperture_start != zdev->start_dma ||
> > domain->geometry.aperture_end != zdev->end_dma) {
> > - rc = -EINVAL;
> > spin_unlock_irqrestore(&s390_domain->list_lock, flags);
> > - goto out_restore;
> > + rc = -EINVAL;
> > + goto out_unregister;
> > }
> > domain_device->zdev = zdev;
> > zdev->s390_domain = s390_domain;
> > @@ -138,14 +158,9 @@ static int s390_iommu_attach_device(struct iommu_domain *domain,
> >
> > return 0;
> >
> > -out_restore:
> > - if (!zdev->s390_domain) {
> > - zpci_dma_init_device(zdev);
> > - } else {
> > - zdev->dma_table = zdev->s390_domain->dma_table;
> > - zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
> > - virt_to_phys(zdev->dma_table));
> > - }
> > +out_unregister:
> > + zpci_unregister_ioat(zdev, 0);
> > + zdev->dma_table = NULL;
> > out_free:
> > kfree(domain_device);
> >
On Fri, 2022-10-07 at 08:55 +0200, Niklas Schnelle wrote:
> On Thu, 2022-10-06 at 17:02 -0400, Matthew Rosato wrote:
> > On 10/6/22 10:46 AM, Niklas Schnelle wrote:
> > > Since commit fa7e9ecc5e1c ("iommu/s390: Tolerate repeat attach_dev
> > > calls") we can end up with duplicates in the list of devices attached to
> > > a domain. This is inefficient and confusing since only one domain can
> > > actually be in control of the IOMMU translations for a device. Fix this
> > > by detaching the device from the previous domain, if any, on attach.
> > > Add a WARN_ON() in case we still have attached devices on freeing the
> > > domain. While here remove the re-attach on failure dance as it was
> > > determined to be unlikely to help and may confuse debug and recovery.
> > >
> > > Fixes: fa7e9ecc5e1c ("iommu/s390: Tolerate repeat attach_dev calls")
> > > Signed-off-by: Niklas Schnelle <[email protected]>
> > > ---
> > > v4->v5:
> > > - Unregister IOAT and set zdev->dma_table on error (Matt)
> > >
> > ...
> >
> > > static int s390_iommu_attach_device(struct iommu_domain *domain,
> > > struct device *dev)
> > > {
> > > @@ -90,7 +116,7 @@ static int s390_iommu_attach_device(struct iommu_domain *domain,
> > > struct zpci_dev *zdev = to_zpci_dev(dev);
> > > struct s390_domain_device *domain_device;
> > > unsigned long flags;
> > > - int cc, rc;
> > > + int cc, rc = 0;
> > >
> > > if (!zdev)
> > > return -ENODEV;
> > > @@ -99,23 +125,17 @@ static int s390_iommu_attach_device(struct iommu_domain *domain,
> > > if (!domain_device)
> > > return -ENOMEM;
> > >
> > > - if (zdev->dma_table && !zdev->s390_domain) {
> > > - cc = zpci_dma_exit_device(zdev);
> > > - if (cc) {
> > > - rc = -EIO;
> > > - goto out_free;
> > > - }
> > > - }
> > > -
> > > if (zdev->s390_domain)
> > > - zpci_unregister_ioat(zdev, 0);
> > > + __s390_iommu_detach_device(zdev);
> > > + else if (zdev->dma_table)
> > > + zpci_dma_exit_device(zdev);
> > >
> > > zdev->dma_table = s390_domain->dma_table;
> > > cc = zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
> > > virt_to_phys(zdev->dma_table));
> > > if (cc) {
> > > rc = -EIO;
> > > - goto out_restore;
> > > + goto out_free;
> > > }
> >
> > Hmm, with this we will leave attach_dev with a zdev->dma_table associated with this domain (not one generated via zpci_dma_init_device) and zdev->s390_domain == 0. Won't this cause both s390_domain_free and zpci_dma_exit_device() to try and free the same dma table?
> >
> > I think we also have to leave with a NULL zdev->dma_table in this case too (you technically could skip the zpci_unregister_ioat)
>
> Argh you're right. This is I think a a bad rebase, in v4 I had the
> zpci_register_ioat() use s390_domain->dma_table and only set zdev-
> > dma_table after that succeeded. I seem to have lost that part
> somewhere along the way. With that we zdev->dma_table would be NULL and
> all would be good.
>
Went back to the way I did it in v4 for v6. I think I was simply an
idiot and when comparing to the state prior to the commit forgot why I
did it this way and thought it was an unneeded change..
> > >
> > > spin_lock_irqsave(&s390_domain->list_lock, flags);
> > > @@ -127,9 +147,9 @@ static int s390_iommu_attach_device(struct iommu_domain *domain,
> > > /* Allow only devices with identical DMA range limits */
> > > } else if (domain->geometry.aperture_start != zdev->start_dma ||
> > > domain->geometry.aperture_end != zdev->end_dma) {
> > > - rc = -EINVAL;
> > > spin_unlock_irqrestore(&s390_domain->list_lock, flags);
> > > - goto out_restore;
> > > + rc = -EINVAL;
> > > + goto out_unregister;
> > > }
> > > domain_device->zdev = zdev;
> > > zdev->s390_domain = s390_domain;
> > > @@ -138,14 +158,9 @@ static int s390_iommu_attach_device(struct iommu_domain *domain,
> > >
> > > return 0;
> > >
> > > -out_restore:
> > > - if (!zdev->s390_domain) {
> > > - zpci_dma_init_device(zdev);
> > > - } else {
> > > - zdev->dma_table = zdev->s390_domain->dma_table;
> > > - zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
> > > - virt_to_phys(zdev->dma_table));
> > > - }
> > > +out_unregister:
> > > + zpci_unregister_ioat(zdev, 0);
> > > + zdev->dma_table = NULL;
> > > out_free:
> > > kfree(domain_device);
> > >
>
>