Date:   Wed, 1 Feb 2023 10:58:52 -0600
From:   Bjorn Helgaas <helgaas@kernel.org>
To:     Baolu Lu <baolu.lu@linux.intel.com>
Cc:     Bjorn Helgaas <bhelgaas@google.com>,
        Joerg Roedel <jroedel@suse.de>,
        Matt Fagnani <matt.fagnani@bell.net>,
        Christian =?iso-8859-1?Q?K=F6nig?= <christian.koenig@amd.com>,
        Jason Gunthorpe <jgg@nvidia.com>,
        Kevin Tian <kevin.tian@intel.com>,
        Vasant Hegde <vasant.hegde@amd.com>,
        Tony Zhu <tony.zhu@intel.com>, linux-pci@vger.kernel.org,
        iommu@lists.linux.dev, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v3 1/1] PCI: Add translated request only flag for
 pci_enable_pasid()
Message-ID: <20230201165852.GA1775862@bhelgaas>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <8cb478bf-db44-0a8f-8521-771529ca0706@linux.intel.com>
Precedence: bulk

On Tue, Jan 31, 2023 at 08:25:05PM +0800, Baolu Lu wrote:
> On 2023/1/31 2:38, Bjorn Helgaas wrote:
> > PCIe r6.0, sec 6.20.1:
> > 
> >    A Function is not permitted to generate Requests using Translated
> >    Addresses and a PASID unless both PASID Enable and Translated
> >    Requests with PASID Enable are Set.
> > 
> > You want AMD graphics devices to do DMA with translated addresses and
> > PASID, right?  pci_enable_pasid() sets PASID Enable
> > (PCI_PASID_CTRL_ENABLE), but I don't see where "Translated Requests
> > with PASID Enable" is set.  We don't even have a #define for it.
> > 
> > I would think we should check "Translated Requests with PASID
> > Supported" before setting "Translated Requests with PASID Enable",
> > too?
> 
> This seems to be an ECN for PCIe 5.x:
> 
> https://members.pcisig.com/wg/PCI-SIG/document/14929
> 
> What I read from this ECN is that,
> 
> With this ECN, translated memory requests for PASIDs are not allowed to
> carry a PASID prefix if "Translated Requests with PASID Enabled" is not
> set. It does not mean whether the device can generate translated memory
> requests for PASID, but whether the memory request can carry a PASID
> prefix.

My assumption that "you want AMD graphics devices to do DMA with
translated addresses and PASID" was wrong.

Per Jason [1], it sounds like the AMD GPU generates Translation
Requests (sec 10.2.2) with a PASID.  The GPU will cache the translated
address from the Translation Completion in its local ATC, and will do
DMA (MemRd/Wr) with that translated address but *without* PASID
prefixes.

That makes sense because (PASID, IOVA) maps to a translated address,
e.g., a a CPU physical address, and the GPU can DMA to that address
directly without needing the PASID.

Bjorn

[1] https://lore.kernel.org/r/Y9nQK9P3HOxEeZ4U@nvidia.com