Received: by 2002:a25:ef43:0:0:0:0:0 with SMTP id w3csp145197ybm; Wed, 27 May 2020 22:18:17 -0700 (PDT) X-Google-Smtp-Source: ABdhPJw1wEl+TWNfPWAqYr69ia6y9ASlDBWGXmPImf+ZXVJSP50LOlMEmEXxJMcizmJB9sDHz/iE X-Received: by 2002:a17:906:4353:: with SMTP id z19mr1508912ejm.371.1590643097380; Wed, 27 May 2020 22:18:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1590643097; cv=none; d=google.com; s=arc-20160816; b=qqHuwsmw7fJq+itwWa1kGMcBiiom9cp8KD8tQXLEtFoKV4OO3d1E62jr4PA4RfNgHQ Ez8DCNX6hOBavtHSfDIOA6cLuIdM/q+IrNno3xXyUT6E9UkRHmBcSDi6ztssr0IiYGsE R269brGlnf/vA7KyyBQ0pOAZI3va/Ew0+YBssGtWVD/aA+I/2b8tsBFF35ZbcNA1LRWi y15bbEijKAohyJnkX/uNKIgD7RjMIYpxTzZkXN7NVKyIJwqov8FS2gwnJIW0YZnmZJj8 rRyq50STJMjppwum8yrU8OB0n/vvn1QOZTgjxEvIeW4GQ0+9atpS8dnMALxhNJFXrHeL D/OA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:dkim-signature; bh=9ogbDNvgcnvbjK2u+CfCd7F7vOweOB0DcAaASJSQleM=; b=pj9lC3P/VQGOc52TRQtFa25r7mD9t9XgliJVnDpKg/pQ15opKI02KkQlKAz/7fYjeD 6NP746ybClOOqeduXU2yD+ZwJNQJaW4D53BhUtk2vK25dsY7GjyvIRDRJRywFqJBl69i iZJLpXyx1p4ZsLwX35SzBtgATxzqcHyTfrfdNLcc74k3tsYNICNZ1U1o8h8TobFZ5gxw gYG3RETpF6AsRqx6P31wvFISMBk04FRBPnElhmVB43haiIcjCvEG9Radd7kINCW+kvp/ 4YVPJKw0937uatTp5KL4WYJB+fXF8L84JVNynIp91ltMhVomBw1Wcwl6vJPcpEBXaf+i TJBA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=gK1Wa8Ql; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id v18si3392764ejb.705.2020.05.27.22.17.54; Wed, 27 May 2020 22:18:17 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=gK1Wa8Ql; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727798AbgE1FPb (ORCPT + 99 others); Thu, 28 May 2020 01:15:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40758 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727047AbgE1FP2 (ORCPT ); Thu, 28 May 2020 01:15:28 -0400 Received: from mail-wm1-x344.google.com (mail-wm1-x344.google.com [IPv6:2a00:1450:4864:20::344]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31FCBC05BD1E for ; Wed, 27 May 2020 22:15:28 -0700 (PDT) Received: by mail-wm1-x344.google.com with SMTP id n5so1848379wmd.0 for ; Wed, 27 May 2020 22:15:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=9ogbDNvgcnvbjK2u+CfCd7F7vOweOB0DcAaASJSQleM=; b=gK1Wa8QlQn0RzLYnPm0DOAuWIsb3ZgQVejc8of4HcZHM9bTmfb7ku5IuobyHBWLV+A 4pfX2QFACm9JWcZbeVXnuhuEeCC7j9gvmyRWHbderhxw2Wo93egg9/eMdIOKPW2vJ+Hi sdBTJqihazq9u6DGx72wef17BzJOJywp3pC7M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=9ogbDNvgcnvbjK2u+CfCd7F7vOweOB0DcAaASJSQleM=; b=btBzYYhx8LnebZ1I1Qndye39GRygrhwf1n/58R2HlyI5i9UvaKTD4wYDSTVxqnN52U kkpp3T8AKyska0BfObu4PgVr6TcbgfA+86S61jITjaMw+dqwSRP7Kmt8JMa8ldK8AS6q BZO68uAl6kbdu6peaVm5d/0DbKrDtwTFskNo86yfAT+wqK5EP5mWRXJ/CznW8SNY3ZNW Of0jygnk22OcMXgG7XtrUVNharkrRDzOCvX02GskTICJ6AARdnGHq5wJNLi3TWnVq3l/ qaoubP+FiC6bIKGb/ZQpYIWqSbLsy5o3oJsBKVgPtfdEDHElaHsvvZ48A8qkaTrlUH9I LNtA== X-Gm-Message-State: AOAM5334K3xu9/SQEQMi0ELIlHy+8DHa6A8X3Z6HnKCIsR2uNc9X+fZ5 SpXuvEWpbXmWi999WkUFjhy5vgPJtgSPzkvNTpDsiQ== X-Received: by 2002:a1c:740e:: with SMTP id p14mr1592366wmc.155.1590642926688; Wed, 27 May 2020 22:15:26 -0700 (PDT) MIME-Version: 1.0 References: <1590595398-4217-1-git-send-email-srinath.mannam@broadcom.com> In-Reply-To: From: Srinath Mannam Date: Thu, 28 May 2020 10:45:14 +0530 Message-ID: Subject: Re: [RFC PATCH] iommu/arm-smmu: Add module parameter to set msi iova address To: Robin Murphy Cc: Will Deacon , Joerg Roedel , BCM Kernel Feedback , Linux ARM , iommu@lists.linux-foundation.org, Linux Kernel Mailing List , Eric Auger , Alex Williamson , Jean-Philippe Brucker Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, May 27, 2020 at 11:00 PM Robin Murphy wrote: > Thanks Robin for your quick response. > On 2020-05-27 17:03, Srinath Mannam wrote: > > This patch gives the provision to change default value of MSI IOVA base > > to platform's suitable IOVA using module parameter. The present > > hardcoded MSI IOVA base may not be the accessible IOVA ranges of platform. > > That in itself doesn't seem entirely unreasonable; IIRC the current > address is just an arbitrary choice to fit nicely into Qemu's memory > map, and there was always the possibility that it wouldn't suit everything. > > > Since commit aadad097cd46 ("iommu/dma: Reserve IOVA for PCIe inaccessible > > DMA address"), inaccessible IOVA address ranges parsed from dma-ranges > > property are reserved. > > That, however, doesn't seem to fit here; iommu-dma maps MSI doorbells > dynamically, so they aren't affected by reserved regions any more than > regular DMA pages are. In fact, it explicitly ignores the software MSI > region, since as the comment says, it *is* the software that manages those. Yes you are right, we don't see any issues with kernel drivers(PCI EP) because MSI IOVA allocated dynamically by honouring reserved regions same as DMA pages. > > The MSI_IOVA_BASE region exists for VFIO, precisely because in that case > the kernel *doesn't* control the address space, but still needs some way > to steal a bit of it for MSIs that the guest doesn't necessarily know > about, and give userspace a fighting chance of knowing what it's taken. > I think at the time we discussed the idea of adding something to the > VFIO uapi such that userspace could move this around if it wanted or > needed to, but decided we could live without that initially. Perhaps now > the time has come? Yes, we see issues only with user-space drivers(DPDK) in which MSI_IOVA_BASE region is considered to map MSI registers. This patch helps us to fix the issue. Thanks, Srinath. > > Robin. > > > If any platform has the limitaion to access default MSI IOVA, then it can > > be changed using "arm-smmu.msi_iova_base=0xa0000000" command line argument. > > > > Signed-off-by: Srinath Mannam > > --- > > drivers/iommu/arm-smmu.c | 5 ++++- > > 1 file changed, 4 insertions(+), 1 deletion(-) > > > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c > > index 4f1a350..5e59c9d 100644 > > --- a/drivers/iommu/arm-smmu.c > > +++ b/drivers/iommu/arm-smmu.c > > @@ -72,6 +72,9 @@ static bool disable_bypass = > > module_param(disable_bypass, bool, S_IRUGO); > > MODULE_PARM_DESC(disable_bypass, > > "Disable bypass streams such that incoming transactions from devices that are not attached to an iommu domain will report an abort back to the device and will not be allowed to pass through the SMMU."); > > +static unsigned long msi_iova_base = MSI_IOVA_BASE; > > +module_param(msi_iova_base, ulong, S_IRUGO); > > +MODULE_PARM_DESC(msi_iova_base, "msi iova base address."); > > > > struct arm_smmu_s2cr { > > struct iommu_group *group; > > @@ -1566,7 +1569,7 @@ static void arm_smmu_get_resv_regions(struct device *dev, > > struct iommu_resv_region *region; > > int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO; > > > > - region = iommu_alloc_resv_region(MSI_IOVA_BASE, MSI_IOVA_LENGTH, > > + region = iommu_alloc_resv_region(msi_iova_base, MSI_IOVA_LENGTH, > > prot, IOMMU_RESV_SW_MSI); > > if (!region) > > return; > >