Date:   Fri, 3 Feb 2023 14:52:36 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Bjorn Helgaas <helgaas@kernel.org>
Cc:     Baolu Lu <baolu.lu@linux.intel.com>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Joerg Roedel <jroedel@suse.de>,
        Matt Fagnani <matt.fagnani@bell.net>,
        Christian =?utf-8?B?S8O2bmln?= <christian.koenig@amd.com>,
        Kevin Tian <kevin.tian@intel.com>,
        Vasant Hegde <vasant.hegde@amd.com>,
        Tony Zhu <tony.zhu@intel.com>, linux-pci@vger.kernel.org,
        iommu@lists.linux.dev, linux-kernel@vger.kernel.org,
        Alex Williamson <alex.williamson@redhat.com>
Subject: Re: [PATCH v3 1/1] PCI: Add translated request only flag for
 pci_enable_pasid()
Message-ID: <Y91X9MeCOsa67CC6@nvidia.com>
References: <Y9wg0Znc0tRWj4O9@nvidia.com>
 <20230203182045.GA1972366@bhelgaas>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20230203182045.GA1972366@bhelgaas>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?3zIZj4ZlyFLldWk7lXqfboLT9+oIEEJpSRAeoQVlvddEdLt3hoxwPF4TTvZT?=
 =?us-ascii?Q?BzwgomjIQ/FrrK8Rg/HSdDAxLg1alEd8NstWtQmTqJV5A0gX3PlIFoFVV19H?=
 =?us-ascii?Q?UMVnEq1uWs6DO4GSJwSJ8ChTz3yqTNN+g/z4TkcAnjWk2UlZvDFPLinntP4z?=
 =?us-ascii?Q?C9N107JWFPcmevUJwch7iVY6CdZpxq522ib//SJgyTSt7krabqcO1TQ2ag1p?=
 =?us-ascii?Q?IGU8pmhPvh/xQQOc3wayvMBhHHX3NJPTdsnteytDsGU0XOTqq4VebpBhPggx?=
 =?us-ascii?Q?RQhESSFGDlX6TT3BoCf4Tc9O/KqbbWZc9DJKKltEmr6w9rIOVjFCgAXyrt/Y?=
 =?us-ascii?Q?6Ln/a7bNEve0J6aS//yv/MkcxoAuBsoqC3Mrn77pyjcNL1k6mJmMqmd0mCAx?=
 =?us-ascii?Q?QuflN4jDO/FRWTgK1cCy5m/hMGlfx62kV0DelnoZisZKI8lA0YJ3TNIC6ZwN?=
 =?us-ascii?Q?7VByAfEwW9sHv7Mv3+yWagJG1gWbxE9MMmo3vDYh+yFHhjgElubywHB0Zp5I?=
 =?us-ascii?Q?2HPtQG5ipDV9tP5H2Tm4ZJZTSvPHNn4CBC2L4DL2EUVlxPxpkgNtSVpQ3njt?=
 =?us-ascii?Q?SOhF8Y9NM7rhVP6C3ChEnzVSin5sgYMXxQryYbQPmLlSgfrGYUiS0vq8BJEE?=
 =?us-ascii?Q?O8OjHvd3rMF/QtNS8V9E3pW1MXjUKvyZE3+Ujzt+HdNH3nxjhQx488FlQPZy?=
 =?us-ascii?Q?9EFW5+FN5jhY1JqMchucnSj3OhAA8NsJFTUfkjxKcna+g3B4SPU3NI0ueUTd?=
 =?us-ascii?Q?yYViPo4YeVCSY0MIOV8LLBMp/M0aH4iv35Ks9/ilcsUJSMEiLMnluf3A1X1R?=
 =?us-ascii?Q?G7Hl0kbMgDrHNCrJPGJUgMbmR1Mv+g9vO3gKlFnVISq8ec3H7eKeCl0+4czK?=
 =?us-ascii?Q?ubnL6AaWDK6vsALlXDQFcsoqrXpaBmZDe8z+FJv3O4F6YZ4eWKiZq/eBrEJO?=
 =?us-ascii?Q?YaXdQxSgJNYNHmm1qeH0UJSNl+UXan5deZvpyQIftftr3WFLmNm6Z0wCgG3z?=
 =?us-ascii?Q?k9GdMaSZ+F/5l3SRSIdaZrOn3nzSOsBd+chwrMgvOCA1npUTRc+9wa2uv9sr?=
 =?us-ascii?Q?+WlSlI+TOJnhp+u6hOfm3YyFfFzoNmUpqtZI69wGKjyaFuDdfNMzP6ukO7DP?=
 =?us-ascii?Q?m2yHcp93zTGdICy0h9b34kMX0K6ea5wqb7pjd8RMEkXh80z4mSRijPxdDvGP?=
 =?us-ascii?Q?a7mgRM8G3pOLie1kIcAf/eCCVGIwvgz/zBj2ywm0vF+72ZaqwKDce9hzgBPE?=
 =?us-ascii?Q?ElTHLvD8V6qom3i8xCd2+0QKf89qr63dvJZdzphqCi3xYDl/ZmPfJVCS3T3w?=
 =?us-ascii?Q?Dfh1dgm5lKCDNwJNq3IXDU9l4/GYRopDU8DH4iqx6/Awvkq6DkPmI4fKC5ue?=
 =?us-ascii?Q?ob2sMwarkmlpQ/+YhUSLgDFeKhDQ/83T/5EtWhTTr8zA6RY1K8ftwJ3MpBUJ?=
 =?us-ascii?Q?7/dlXymCfMrViqrEP4t9SJWPh4OSL0/PW891hXCbqTH4iVeEL1knYEnDhIAZ?=
 =?us-ascii?Q?H7A3CKzO/htdGCBlyWUWnh/8kSL/EXKjQdrWgSrBP7ymkHUUf8GQBrnYXuWE?=
 =?us-ascii?Q?sLLvyKoVd1kYmi8mxBJbMrh8qlCxt1mOm1M/Dkas?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0ba00048-1bb7-4a71-85f0-08db0617d1d5
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Feb 2023 18:52:37.9534
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: zEtzNHRmoAEfbakMHv9DFoGB+BcSdpLsRQmRqHLwjgUYTBzgsWSmH+iVbRUAhMSe
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN6PR12MB8589
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Feb 03, 2023 at 12:20:45PM -0600, Bjorn Helgaas wrote:
> > The PCI rules are a bit complicated:
> >  - A simple MemRd/Wr with a PASID will be routed according to the
> >    address. This can be mis-routed
> >  - A ATS translation request with a PASID is always routed to the host
> >    bridge
> 
> From a PCIe point of view, I think these cases are equivalent because
> a PASID prefix doesn't affect routing (sec 2.2.10.4).  A Translation
> Request includes an Untranslated Address, and if that happens to look
> like a PCI bus address, I think it will be mis-routed just like a
> MemRd/Wr would be.

So, I trusted AMD when they said this is why their platform worked, I
didn't check carefully the ATS spec language.

But looking now, I agree. I don't see any language that says ATS is
routed differently, if anything it says it is routed the same way as
MemRd.

AMD folks?

If yes then this has all been the wrong direction.

> > > Do IOMMUs allocate (PASID, Untranslated Addresses) that look like
> > > PCI bus addresses?
> > 
> > Yes, because it is mapped to a mm_struct userspace can use any mmap
> > to access any valid address as an IOVA and thus PASID tagged
> > translation must never become confused with bus addresses.
> 
> If PCI bus addresses are carved out of the Translated Address arena,
> the MemRd/Wr TLPs should be fine,

Yes, that is my point that Translated requests are fine because they
already carry a Translated address, and by its nature a Translated
address cannot be mis-routed.

The spec actually talks about this whole issue see the NOTE at the end
of 10.1.1

Per that note, for Linux, the "implementation constraint on the TA" is
that the TA requires ACS so that Untranslated always routes to the TA.

The AMD defect is some BIOS's on some of their SOC's don't report ACS
for the MFD, even though it does presumably implement ACS.

> Are we on track to fix this before v6.2?  

AMD? Did the patches you were talking about to fix the oops in the AMD
driver land? Can we rely on that to fix the black screen for v6.2?

Otherwise I think the PCI core is correct here and I'm loath to change
it to work around a GPU driver bug. The GPU driver should understand
that PASID is not supported because the PCI core says so. It shouldn't
crash and blackscreen.

Keep in mind, from a PCI perspective, this protection is a security
senstive check, described in the spec. If we drop it we expose
platforms using SVA to a potential security issue upon
mis-configuration.

So, I'd much rather leave the PCI alone and see that the AMD driver is
fixed.

Jason