Received: by 2002:a25:ef43:0:0:0:0:0 with SMTP id w3csp251598ybm; Thu, 28 May 2020 01:41:33 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxHM31Fen8xNPf10eYHNI/PwgdJil0fDy1P0bX/t/UEzP3O24gohpX+djP++PjIq2oIAHij X-Received: by 2002:a17:906:858b:: with SMTP id v11mr1908671ejx.348.1590655293149; Thu, 28 May 2020 01:41:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1590655293; cv=none; d=google.com; s=arc-20160816; b=g6ZdwCvqg9VyM/wl1EZ6rYaTi5I8hjEnh6Ixp+HUbbhfUf7DgSf6eyxIVWwsd/4Sqz D9pVTV7XxUuOpiWSgEW/L/nr0WBFxwilNb9xNIer42+LObEnLSTZ4wLhfAL9U9NJ4e2T 68+o9QLcloM1WP8Kj5nQ9pTSWl8QhX+gS1DwUCYnOzrHnW+3UDKXzJCrt3PSbjlrDpkv vDyPxVAAYlAQAjKhCG+J9BuConzoT5neVhhzL2nL9sD35QYjh2L7X/RqhY9Jy4sKrExQ ur9kZKIDIDasMhi6O4XBYRhd25ERnGRRo4LHynnN2YDCKi80e9+Qv6Lw1YzxIP1q0iWn oPig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:in-reply-to:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :dkim-signature; bh=YVkNlO8rWjI0H/buRx03yHhJ40oletHMhbdGkngmX0U=; b=vgvCDnpfVBsPcNWZZFg8kwXQKOcANw9mJIncgP0MjrwRuRHk7+tr0r/KuKqwJhK0sF 33N6BdER/KmbJND+x8Dj6OuGCFwuWZKjE1kg7CoQlqAFRAChjL3u/dzYGSybm7V4r3Y/ qGJiOf9beaq9NnhQXlVTWK6ACuw4ssC+x0cpO6Ep6K44b+t7VEeQImp4/NaVq6GErnvM Ly465lRys9wTDg+Q0n2gIhoc2NQd1bavXHa6FDWwsyFIW0ECrz2cXzPDeaO7xpaqlnNB h2A13O49icayQO7Dq90snBUXzIrA9aT0oN7kIvZZbB0MpE0AOPth86j1FT3NHobX6xeh /X0g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GsCNBPNo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id cm13si2569205edb.333.2020.05.28.01.41.10; Thu, 28 May 2020 01:41:33 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GsCNBPNo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727816AbgE1IjF (ORCPT + 99 others); Thu, 28 May 2020 04:39:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43992 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727803AbgE1IjE (ORCPT ); Thu, 28 May 2020 04:39:04 -0400 Received: from mail-wm1-x343.google.com (mail-wm1-x343.google.com [IPv6:2a00:1450:4864:20::343]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1D9E0C05BD1E for ; Thu, 28 May 2020 01:39:04 -0700 (PDT) Received: by mail-wm1-x343.google.com with SMTP id u26so3983658wmn.1 for ; Thu, 28 May 2020 01:39:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=YVkNlO8rWjI0H/buRx03yHhJ40oletHMhbdGkngmX0U=; b=GsCNBPNoC/NhrVuCzx9gpMv+W9gaKaX98tVY9sKPAP0z0eHQJG3ETMcWY8AnmIGF/s cNcbP7ItPaKM8kTRVYPal6ftk9RaU/3h09LilWQgVRtegLVoRs4tX24qyhnRupWqAZna X1ImFR6pCjpMR5Id9jAly/eXEgbqNXPH8crjj2/nd2C/inEgdQkv9Yk8vLYkDwykFU8x 4DG76XIuQpBnHI2hEXDXnh49RSGT2d1a2t9IvpPLOD6UKYOkuF6SY190L80w1Nmx5NxH 6r856JkQlwz1iub7xlLCGTNZIU+dakL4Mx7PgxLhyFysvqAXDQ2sVqMRB279lGnm3JdJ yrRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=YVkNlO8rWjI0H/buRx03yHhJ40oletHMhbdGkngmX0U=; b=sNTe3Bl+KshoFIVPX13CX9OIxIS+HDM75BEkX0t3odquPj775GNVRJZBTPuG4EJgPV SnL2JcvEpTeYndwyUFdrl7j/GKP/yEqK++i6V+XWgczp1aQF/A9IEUYMTwigxF1Zl5yk 4bsh6mVAC2JoJtShZ4gYNDqld5MUVuE9mBJ+WBHsQ3UTBiNeDittK6VSYnkZSxDF4e7X RJ1st3K6c2MLxc4ModGS3vbwes+mgkknBOJSF9KPxwotrMN5O6zuVJyh421JwIVsA8aZ 6sF5XeKhBQxnrcr0+Gv7eNQR/phFw5PIcJht2vxuyKMmzgjs8ky4Gm3PtUc/tI8zGXiD ULkQ== X-Gm-Message-State: AOAM533+/N9QRqaHDtKKEj7a93SUIcZVtjBR+xxfKWqkKt/QQUv7ppEo rue6AtiYxftwZixIiKBQmolHyA== X-Received: by 2002:a1c:3dd6:: with SMTP id k205mr1309305wma.87.1590655142697; Thu, 28 May 2020 01:39:02 -0700 (PDT) Received: from myrica ([2001:171b:226e:c200:c43b:ef78:d083:b355]) by smtp.gmail.com with ESMTPSA id q4sm5775850wma.47.2020.05.28.01.39.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 May 2020 01:39:02 -0700 (PDT) Date: Thu, 28 May 2020 10:38:51 +0200 From: Jean-Philippe Brucker To: Auger Eric Cc: Srinath Mannam , Alex Williamson , Robin Murphy , Joerg Roedel , Linux Kernel Mailing List , iommu@lists.linux-foundation.org, BCM Kernel Feedback , Will Deacon , Linux ARM , shameerali.kolothum.thodi@huawei.com Subject: Re: [RFC PATCH] iommu/arm-smmu: Add module parameter to set msi iova address Message-ID: <20200528083851.GB414784@myrica> References: <1590595398-4217-1-git-send-email-srinath.mannam@broadcom.com> <20200528072308.GA414784@myrica> <527f25a4-ca5a-10da-150f-0b4ea3839635@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <527f25a4-ca5a-10da-150f-0b4ea3839635@redhat.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org [+ Shameer] On Thu, May 28, 2020 at 09:43:46AM +0200, Auger Eric wrote: > Hi, > > On 5/28/20 9:23 AM, Jean-Philippe Brucker wrote: > > On Thu, May 28, 2020 at 10:45:14AM +0530, Srinath Mannam wrote: > >> On Wed, May 27, 2020 at 11:00 PM Robin Murphy wrote: > >>> > >> Thanks Robin for your quick response. > >>> On 2020-05-27 17:03, Srinath Mannam wrote: > >>>> This patch gives the provision to change default value of MSI IOVA base > >>>> to platform's suitable IOVA using module parameter. The present > >>>> hardcoded MSI IOVA base may not be the accessible IOVA ranges of platform. > >>> > >>> That in itself doesn't seem entirely unreasonable; IIRC the current > >>> address is just an arbitrary choice to fit nicely into Qemu's memory > >>> map, and there was always the possibility that it wouldn't suit everything. > >>> > >>>> Since commit aadad097cd46 ("iommu/dma: Reserve IOVA for PCIe inaccessible > >>>> DMA address"), inaccessible IOVA address ranges parsed from dma-ranges > >>>> property are reserved. > > > > I don't understand why we only reserve the PCIe windows for DMA domains. > > Shouldn't VFIO also prevent userspace from mapping them? > > VFIO prevents userspace from DMA mapping iovas within reserved regions: > 9b77e5c79840 vfio/type1: check dma map request is within a valid iova range Right but I was asking specifically about the IOVA reservation introduced by commit aadad097cd46. They are not registered as reserved regions within the IOMMU core, they are only taken into account by dma-iommu.c when creating a DMA domain. As VFIO uses UNMANAGED domains, it isn't aware of those regions and they won't be seen by vfio_iommu_resv_exclude(). It looks like the PCIe regions used to be common until cd2c9fcf5c66 ("iommu/dma: Move PCI window region reservation back into dma specific path.") But I couldn't find the justification for this commit. The thing is, if VFIO isn't aware of the reserved PCIe windows, then allowing VFIO or userspace to choose MSI_IOVA_BASE won't solve the problem reported by Srinath, because they could well choose an IOVA within the PCIe window... Thanks, Jean > but it does not prevent the SW MSI region chosen by the kernel from > colliding with other reserved regions (esp. PCIe host bridge windows). > > If they were > > part of the common reserved regions then we could have VFIO choose a > > SW_MSI region among the remaining free space. > As Robin said this was the initial chosen approach > [PATCH 10/10] vfio: allow the user to register reserved iova range for > MSI mapping > https://patchwork.kernel.org/patch/8121641/ > > Some additional background about why the static SW MSI region chosen by > the kernel was later chosen: > Summary of LPC guest MSI discussion in Santa Fe (was: Re: [RFC 0/8] KVM > PCIe/MSI passthrough on ARM/ARM64 (Alt II)) > https://lists.linuxfoundation.org/pipermail/iommu/2016-November/019060.html > > Thanks > > Eric > > > It would just need a > > different way of asking the IOMMU driver if a SW_MSI is needed, for > > example with a domain attribute. > > > > Thanks, > > Jean > > > >>> > >>> That, however, doesn't seem to fit here; iommu-dma maps MSI doorbells > >>> dynamically, so they aren't affected by reserved regions any more than > >>> regular DMA pages are. In fact, it explicitly ignores the software MSI > >>> region, since as the comment says, it *is* the software that manages those. > >> Yes you are right, we don't see any issues with kernel drivers(PCI EP) because > >> MSI IOVA allocated dynamically by honouring reserved regions same as DMA pages. > >>> > >>> The MSI_IOVA_BASE region exists for VFIO, precisely because in that case > >>> the kernel *doesn't* control the address space, but still needs some way > >>> to steal a bit of it for MSIs that the guest doesn't necessarily know > >>> about, and give userspace a fighting chance of knowing what it's taken. > >>> I think at the time we discussed the idea of adding something to the > >>> VFIO uapi such that userspace could move this around if it wanted or > >>> needed to, but decided we could live without that initially. Perhaps now > >>> the time has come? > >> Yes, we see issues only with user-space drivers(DPDK) in which MSI_IOVA_BASE > >> region is considered to map MSI registers. This patch helps us to fix the issue. > >> > >> Thanks, > >> Srinath. > >>> > >>> Robin. > >>> > >>>> If any platform has the limitaion to access default MSI IOVA, then it can > >>>> be changed using "arm-smmu.msi_iova_base=0xa0000000" command line argument. > >>>> > >>>> Signed-off-by: Srinath Mannam > >>>> --- > >>>> drivers/iommu/arm-smmu.c | 5 ++++- > >>>> 1 file changed, 4 insertions(+), 1 deletion(-) > >>>> > >>>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c > >>>> index 4f1a350..5e59c9d 100644 > >>>> --- a/drivers/iommu/arm-smmu.c > >>>> +++ b/drivers/iommu/arm-smmu.c > >>>> @@ -72,6 +72,9 @@ static bool disable_bypass = > >>>> module_param(disable_bypass, bool, S_IRUGO); > >>>> MODULE_PARM_DESC(disable_bypass, > >>>> "Disable bypass streams such that incoming transactions from devices that are not attached to an iommu domain will report an abort back to the device and will not be allowed to pass through the SMMU."); > >>>> +static unsigned long msi_iova_base = MSI_IOVA_BASE; > >>>> +module_param(msi_iova_base, ulong, S_IRUGO); > >>>> +MODULE_PARM_DESC(msi_iova_base, "msi iova base address."); > >>>> > >>>> struct arm_smmu_s2cr { > >>>> struct iommu_group *group; > >>>> @@ -1566,7 +1569,7 @@ static void arm_smmu_get_resv_regions(struct device *dev, > >>>> struct iommu_resv_region *region; > >>>> int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO; > >>>> > >>>> - region = iommu_alloc_resv_region(MSI_IOVA_BASE, MSI_IOVA_LENGTH, > >>>> + region = iommu_alloc_resv_region(msi_iova_base, MSI_IOVA_LENGTH, > >>>> prot, IOMMU_RESV_SW_MSI); > >>>> if (!region) > >>>> return; > >>>> > > > > _______________________________________________ > > linux-arm-kernel mailing list > > linux-arm-kernel@lists.infradead.org > > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel > > >