Date:   Tue, 31 Jan 2023 17:50:52 -0600
From:   Bjorn Helgaas <helgaas@kernel.org>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Baolu Lu <baolu.lu@linux.intel.com>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Joerg Roedel <jroedel@suse.de>,
        Matt Fagnani <matt.fagnani@bell.net>,
        Christian =?iso-8859-1?Q?K=F6nig?= <christian.koenig@amd.com>,
        Kevin Tian <kevin.tian@intel.com>,
        Vasant Hegde <vasant.hegde@amd.com>,
        Tony Zhu <tony.zhu@intel.com>, linux-pci@vger.kernel.org,
        iommu@lists.linux.dev, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v3 1/1] PCI: Add translated request only flag for
 pci_enable_pasid()
Message-ID: <20230131235052.GA1811430@bhelgaas>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Y9gQxCmzqq6WXZK4@nvidia.com>
Precedence: bulk

On Mon, Jan 30, 2023 at 02:47:32PM -0400, Jason Gunthorpe wrote:
> On Mon, Jan 30, 2023 at 12:38:10PM -0600, Bjorn Helgaas wrote:
> 
> > Sorry, I'm still confused.  PCI_PASID_XLATED_REQ_ONLY is a
> > device-specific property, and you want to opt-in AMD graphics devices.
> > Where's the AMD graphics-specific change?  The current patch does
> > this:
> > 
> >   pdev_pri_ats_enable
> >     pci_enable_pasid(pdev, 0, PCI_PASID_XLATED_REQ_ONLY)
> > 
> > which looks like it does it for *all* devices below an AMD IOMMU,
> > without any device or driver input.
> 
> AMD GPU has a private interface to AMD IOMMU to support PASID support
> that only it uses.

What is it that makes this a private interface?  It seems like we will
end up enabling PASID on any device below an AMD v2 IOMMU.

I do see the DRM amdgpu_amdkfd_device_init() path that ends up at
domain_enable_v2().  Maybe that's it?

But amd_iommu_domain_alloc() also leads to domain_enable_v2(), and
that's pretty generic, so it looks like we set PD_IOMMUV2_MASK
whenever the IOMMU supports it.

And it seems like we call pci_enable_pasid() for all endpoints below
a v2 IOMMU.  Of course, if the endpoint doesn't have a PASID cap,
pci_enable_pasid() fails (and pdev_pri_ats_enable() probably splats
useless warnings when we try to disable PRI and PASID which haven't
been enabled).

I guess I'm trying to convince myself that no harm in enabling PASID
for any device below an AMD v2 IOMMU.  But I don't think a device is
*required* to use translated addresses with PASID, and if it uses
untranslated addresses with PASID, don't we need ACS to avoid
misrouting?

Bjorn