Received: by 2002:a05:6358:1087:b0:cb:c9d3:cd90 with SMTP id j7csp7138371rwi; Mon, 24 Oct 2022 10:15:55 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4FikeXrGsC+o3fw8kPlZV/a1gZuMtLQqB+nvJ1ZpwhHXV66TiKIkY+Hk3fv/TOt5/beW5G X-Received: by 2002:a17:90a:6845:b0:210:7ef5:ab99 with SMTP id e5-20020a17090a684500b002107ef5ab99mr30677410pjm.80.1666631754825; Mon, 24 Oct 2022 10:15:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666631754; cv=none; d=google.com; s=arc-20160816; b=Ystw/JXNN5ehWQlw2z1gtSv1o4gFJUo6cze29G1+uywVGcQ62WgYEob5661FKep0Lr 7tkc0KRCltixf7///5ULzkcPcz96GcIRZYX0AndC7Jovhh+EiPuRrs5k99DNEpUePMnw xl4wmNMU6xY8CICN6UoZ3Lhb6MMYCoo9YW9jNlqd/1dS1u6h0xSH70qP4HfPfbob7r4P axLbWlU3/MhjD69scuY8EhiM14gfHxOGjquuHqC2Jd5CVMxwO7oWbKg+zhJfXzWI9NuC jXUC2XOJLzVs8DIq5+67PLzt8Y0Jbi+Ht34H9xb4bpzBhgEykCd8rNNLTt61sG4auiCR JBYw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:date:cc:to:from:subject:message-id :dkim-signature; bh=CXbPk6KfVbEUj4mvUy75NRKeO6rjMSSdAQ2/7/QtbBo=; b=bEs/e7QQXOdDXtJvOG+FQRMU4XeXQOwezAFcUKVGQVMws7JqwzBCPlaXwALFXBg158 TX8eRBQ+k8XliA+rq1c8S739NWiZZCWBetGMnPQWP95V9dZA0XZ26HjpW8ZXzyFUo5GS w4NyDjh5Gj0tj5kfhUXgNVyV62nlM500Ia6w/OG6GTrrYDK+QWH3gLUO60NxLdjXCpqZ wA/ClglzTD/5SOh165k8HXB6SZ7vtw8zlc4NzrAA9IbIravifB7ha7JA7BGKh2bzcU4Y i+apbDIH4OH7JpF1qyIm/e2CrEOYQjphxGaTar1Q6Z5nYl9TWEx2YeTz4tzwliwteFhO Cicw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b=ZZmSvkhL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j22-20020a170902759600b0017f685b47b8si57912pll.151.2022.10.24.10.15.42; Mon, 24 Oct 2022 10:15:54 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b=ZZmSvkhL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234586AbiJXQmn (ORCPT + 99 others); Mon, 24 Oct 2022 12:42:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35446 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231974AbiJXQlt (ORCPT ); Mon, 24 Oct 2022 12:41:49 -0400 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5246226C9; Mon, 24 Oct 2022 08:28:47 -0700 (PDT) Received: from pps.filterd (m0127361.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 29OEijmo002466; Mon, 24 Oct 2022 15:07:18 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=message-id : subject : from : to : cc : date : in-reply-to : references : content-type : mime-version : content-transfer-encoding; s=pp1; bh=CXbPk6KfVbEUj4mvUy75NRKeO6rjMSSdAQ2/7/QtbBo=; b=ZZmSvkhLj5SSBrPdoeQWpzZWaAldgYJ0njL5k3kby2GdmItkePmhuW7YERbZk0V+HFz1 ZTryhdspaoIHlwMdwnSZBKOByuctSIexpOWpEUy3t53QRKQLEhMj4fBnFgdsK2MbCnNY zVGkR7Nc7LgLQwPOYmLMkkBOSlaaUTZH/qwoEPoITXJ0ckOK9IK8dZFpqtGcBlrOcfCB HqoeFbkci+wi3G5rg4HpGwTiDnejnIFKWDHCBY9EYpAzvt1YXILNTfyINbeTmevH9WTz TR8lv+Ssx8Fy/b0fTnNWMgQpOYB4e2J+jlM7yzJ+RgiGhpizk2CvY0a9CQACnfeaXN7l aQ== Received: from ppma06fra.de.ibm.com (48.49.7a9f.ip4.static.sl-reverse.com [159.122.73.72]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3kdvqy106m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Oct 2022 15:07:18 +0000 Received: from pps.filterd (ppma06fra.de.ibm.com [127.0.0.1]) by ppma06fra.de.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 29OEoQSe019281; Mon, 24 Oct 2022 15:02:16 GMT Received: from b06cxnps4075.portsmouth.uk.ibm.com (d06relay12.portsmouth.uk.ibm.com [9.149.109.197]) by ppma06fra.de.ibm.com with ESMTP id 3kc7sj2kqc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Oct 2022 15:02:16 +0000 Received: from b06wcsmtp001.portsmouth.uk.ibm.com (b06wcsmtp001.portsmouth.uk.ibm.com [9.149.105.160]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 29OF2Cst35586708 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 24 Oct 2022 15:02:12 GMT Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8E21AA4054; Mon, 24 Oct 2022 15:02:12 +0000 (GMT) Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C7C1DA405F; Mon, 24 Oct 2022 15:02:11 +0000 (GMT) Received: from sig-9-145-54-55.uk.ibm.com (unknown [9.145.54.55]) by b06wcsmtp001.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 24 Oct 2022 15:02:11 +0000 (GMT) Message-ID: Subject: Re: [PATCH v7 3/6] iommu/s390: Fix potential s390_domain aperture shrinking From: Niklas Schnelle To: iommu@lists.linux.dev, Joerg Roedel , Will Deacon , Robin Murphy , Jason Gunthorpe Cc: Matthew Rosato , Gerd Bayer , Pierre Morel , linux-s390@vger.kernel.org, borntraeger@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, gerald.schaefer@linux.ibm.com, agordeev@linux.ibm.com, svens@linux.ibm.com, linux-kernel@vger.kernel.org Date: Mon, 24 Oct 2022 17:02:11 +0200 In-Reply-To: <20221017124558.1386337-4-schnelle@linux.ibm.com> References: <20221017124558.1386337-1-schnelle@linux.ibm.com> <20221017124558.1386337-4-schnelle@linux.ibm.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.28.5 (3.28.5-18.el8) Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: mp3qej8oda2Ka77tiT84HKTasKin6PS0 X-Proofpoint-GUID: mp3qej8oda2Ka77tiT84HKTasKin6PS0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-10-24_04,2022-10-21_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 mlxlogscore=999 clxscore=1015 phishscore=0 mlxscore=0 impostorscore=0 priorityscore=1501 malwarescore=0 adultscore=0 spamscore=0 bulkscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000 definitions=main-2210240092 X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 2022-10-17 at 14:45 +0200, Niklas Schnelle wrote: > The s390 IOMMU driver currently sets the IOMMU domain's aperture to > match the device specific DMA address range of the device that is first > attached. This is not ideal. For one if the domain has no device > attached in the meantime the aperture could be shrunk allowing > translations outside the aperture to exist in the translation tables. > Also this is a bit of a misuse of the aperture which really should > describe what addresses can be translated and not some device specific > limitations. > > Instead of misusing the aperture like this we can instead create > reserved ranges for the ranges inaccessible to the attached devices > allowing devices with overlapping ranges to still share an IOMMU domain. > This also significantly simplifies s390_iommu_attach_device() allowing > us to move the aperture check to the beginning of the function and > removing the need to hold the device list's lock to check the aperture. > > As we then use the same aperture for all domains and it only depends on > the table properties we can already check zdev->start_dma/end_dma at > probe time and turn the check on attach into a WARN_ON(). > > Suggested-by: Jason Gunthorpe > Reviewed-by: Matthew Rosato > Signed-off-by: Niklas Schnelle > --- > v5->v6: > - Return -EINVAL after WARN_ON() in attach > v4->v5: > - Make aperture check in attach a WARN_ON() and fail in probe if > zdev->start_dma/end_dma doesn't git in aperture (Jason) > > drivers/iommu/s390-iommu.c | 63 ++++++++++++++++++++++++++------------ > 1 file changed, 43 insertions(+), 20 deletions(-) > > diff --git a/drivers/iommu/s390-iommu.c b/drivers/iommu/s390-iommu.c > index af83ccde16a4..c4203a37faa4 100644 > --- a/drivers/iommu/s390-iommu.c > +++ b/drivers/iommu/s390-iommu.c > @@ -62,6 +62,9 @@ static struct iommu_domain *s390_domain_alloc(unsigned domain_type) > kfree(s390_domain); > return NULL; > } > + s390_domain->domain.geometry.force_aperture = true; > + s390_domain->domain.geometry.aperture_start = 0; > + s390_domain->domain.geometry.aperture_end = ZPCI_TABLE_SIZE_RT - 1; > > spin_lock_init(&s390_domain->dma_table_lock); > spin_lock_init(&s390_domain->list_lock); > @@ -102,11 +105,15 @@ static int s390_iommu_attach_device(struct iommu_domain *domain, > struct s390_domain *s390_domain = to_s390_domain(domain); > struct zpci_dev *zdev = to_zpci_dev(dev); > unsigned long flags; > - int cc, rc = 0; > + int cc; > > if (!zdev) > return -ENODEV; > > + if (WARN_ON(domain->geometry.aperture_start > zdev->end_dma || > + domain->geometry.aperture_end < zdev->start_dma)) > + return -EINVAL; > + > if (zdev->s390_domain) > __s390_iommu_detach_device(zdev); > else if (zdev->dma_table) > @@ -118,30 +125,14 @@ static int s390_iommu_attach_device(struct iommu_domain *domain, > return -EIO; > zdev->dma_table = s390_domain->dma_table; > > - spin_lock_irqsave(&s390_domain->list_lock, flags); > - /* First device defines the DMA range limits */ > - if (list_empty(&s390_domain->devices)) { > - domain->geometry.aperture_start = zdev->start_dma; > - domain->geometry.aperture_end = zdev->end_dma; > - domain->geometry.force_aperture = true; > - /* Allow only devices with identical DMA range limits */ > - } else if (domain->geometry.aperture_start != zdev->start_dma || > - domain->geometry.aperture_end != zdev->end_dma) { > - spin_unlock_irqrestore(&s390_domain->list_lock, flags); > - rc = -EINVAL; > - goto out_unregister; > - } > + zdev->dma_table = s390_domain->dma_table; > zdev->s390_domain = s390_domain; > + > + spin_lock_irqsave(&s390_domain->list_lock, flags); > list_add(&zdev->iommu_list, &s390_domain->devices); > spin_unlock_irqrestore(&s390_domain->list_lock, flags); > > return 0; > - > -out_unregister: > - zpci_unregister_ioat(zdev, 0); > - zdev->dma_table = NULL; > - > - return rc; > } > > static void s390_iommu_detach_device(struct iommu_domain *domain, > @@ -155,6 +146,30 @@ static void s390_iommu_detach_device(struct iommu_domain *domain, > zpci_dma_init_device(zdev); > } > > +static void s390_iommu_get_resv_regions(struct device *dev, > + struct list_head *list) > +{ > + struct zpci_dev *zdev = to_zpci_dev(dev); > + struct iommu_resv_region *region; > + > + if (zdev->start_dma) { > + region = iommu_alloc_resv_region(0, zdev->start_dma, 0, > + IOMMU_RESV_RESERVED); Heads up! The iommu_alloc_resv_region() function gained a gfp parameter from v6.1-rc1 to v6.1-rc2 so the above needs "…, GFP_KERNEL);". @Joerg, @Will if you don't mind I'll rebase on v6.1-rc2 and resend. @Jason if you want to re-add your R-b this would be good time. > + if (!region) > + return; > + list_add_tail(®ion->list, list); > + } > + > + if (zdev->end_dma < ZPCI_TABLE_SIZE_RT - 1) { > + region = iommu_alloc_resv_region(zdev->end_dma + 1, > + ZPCI_TABLE_SIZE_RT - zdev->end_dma - 1, > + 0, IOMMU_RESV_RESERVED); Same as above. > + if (!region) > + return; > + list_add_tail(®ion->list, list); > + } > +} > + > static struct iommu_device *s390_iommu_probe_device(struct device *dev) > { > struct zpci_dev *zdev; > @@ -164,6 +179,13 @@ static struct iommu_device *s390_iommu_probe_device(struct device *dev) > > zdev = to_zpci_dev(dev); > > + if (zdev->start_dma > zdev->end_dma || > + zdev->start_dma > ZPCI_TABLE_SIZE_RT - 1) > + return ERR_PTR(-EINVAL); > + > + if (zdev->end_dma > ZPCI_TABLE_SIZE_RT - 1) > + zdev->end_dma = ZPCI_TABLE_SIZE_RT - 1; > + > return &zdev->iommu_dev; > } > > @@ -342,6 +364,7 @@ static const struct iommu_ops s390_iommu_ops = { > .release_device = s390_iommu_release_device, > .device_group = generic_device_group, > .pgsize_bitmap = S390_IOMMU_PGSIZES, > + .get_resv_regions = s390_iommu_get_resv_regions, > .default_domain_ops = &(const struct iommu_domain_ops) { > .attach_dev = s390_iommu_attach_device, > .detach_dev = s390_iommu_detach_device,