Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <CAKgT0UezciLjHacOx372+v8MZkDf22D5Thn82n-07xxKy_0FTQ@mail.gmail.com>
 <CAKgT0UfMeVOz6AOqSvVvzpsedGDiXCNQrjM+4KDv7qJJ1orpsw@mail.gmail.com> <a1cff65b-b390-3872-25b5-dd6bbfb3524c@linux.intel.com>
In-Reply-To: <a1cff65b-b390-3872-25b5-dd6bbfb3524c@linux.intel.com>
From:   Alexander Duyck <alexander.duyck@gmail.com>
Date:   Thu, 8 Jun 2023 07:33:31 -0700
Message-ID: <CAKgT0UcE5bUe7ChytSyUWEkyqdwnNR1k_rcfyykPPWJ=ZzsdRg@mail.gmail.com>
Subject: Re: Question about reserved_regions w/ Intel IOMMU
To:     Baolu Lu <baolu.lu@linux.intel.com>
Cc:     LKML <linux-kernel@vger.kernel.org>,
        linux-pci <linux-pci@vger.kernel.org>, iommu@lists.linux.dev
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Wed, Jun 7, 2023 at 8:05=E2=80=AFPM Baolu Lu <baolu.lu@linux.intel.com> =
wrote:
>
> On 6/8/23 7:03 AM, Alexander Duyck wrote:
> > On Wed, Jun 7, 2023 at 3:40=E2=80=AFPM Alexander Duyck
> > <alexander.duyck@gmail.com> wrote:
> >>
> >> I am running into a DMA issue that appears to be a conflict between
> >> ACS and IOMMU. As per the documentation I can find, the IOMMU is
> >> supposed to create reserved regions for MSI and the memory window
> >> behind the root port. However looking at reserved_regions I am not
> >> seeing that. I only see the reservation for the MSI.
> >>
> >> So for example with an enabled NIC and iommu enabled w/o passthru I am=
 seeing:
> >> # cat /sys/bus/pci/devices/0000\:83\:00.0/iommu_group/reserved_regions
> >> 0x00000000fee00000 0x00000000feefffff msi
> >>
> >> Shouldn't there also be a memory window for the region behind the root
> >> port to prevent any possible peer-to-peer access?
> >
> > Since the iommu portion of the email bounced I figured I would fix
> > that and provide some additional info.
> >
> > I added some instrumentation to the kernel to dump the resources found
> > in iova_reserve_pci_windows. From what I can tell it is finding the
> > correct resources for the Memory and Prefetchable regions behind the
> > root port. It seems to be calling reserve_iova which is successfully
> > allocating an iova to reserve the region.
> >
> > However still no luck on why it isn't showing up in reserved_regions.
>
> Perhaps I can ask the opposite question, why it should show up in
> reserve_regions? Why does the iommu subsystem block any possible peer-
> to-peer DMA access? Isn't that a decision of the device driver.
>
> The iova_reserve_pci_windows() you've seen is for kernel DMA interfaces
> which is not related to peer-to-peer accesses.

The problem is if the IOVA overlaps with the physical addresses of
other devices that can be routed to via ACS redirect. As such if ACS
redirect is enabled a host IOVA could be directed to another device on
the switch instead. To prevent that we need to reserve those addresses
to avoid address space collisions.

From what I can tell it looks like the IOVA should be reserved, but I
don't see it showing up anywhere in reserved_regions. What I am
wondering is if iova_reserve_pci_windows() should be taking some steps
so that it will appear, or if  intel_iommu_get_resv_regions() needs to
have some code similar to iova_reserve_pci_windows() to get the ranges
and verify they are reserved in the IOVA.