Date:   Wed, 15 Feb 2023 20:44:02 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Felix Kuehling <felix.kuehling@amd.com>
Cc:     Bjorn Helgaas <helgaas@kernel.org>,
        Baolu Lu <baolu.lu@linux.intel.com>,
        "Huang, Shimmer" <Shimmer.Huang@amd.com>,
        "Liu, Aaron" <Aaron.Liu@amd.com>, Joerg Roedel <jroedel@suse.de>,
        "regressions@lists.linux.dev" <regressions@lists.linux.dev>,
        Thorsten Leemhuis <regressions@leemhuis.info>,
        Linux PCI <linux-pci@vger.kernel.org>,
        "Pan, Xinhui" <Xinhui.Pan@amd.com>,
        "Hegde, Vasant" <Vasant.Hegde@amd.com>,
        amd-gfx@lists.freedesktop.org, LKML <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        Matt Fagnani <matt.fagnani@bell.net>,
        "Deucher, Alexander" <Alexander.Deucher@amd.com>,
        Christian =?utf-8?B?S8O2bmln?= <christian.koenig@amd.com>
Subject: Re: [regression, bisected, =?utf-8?Q?pci?=
 =?utf-8?Q?=2Fiommu=5D_Bug=C2=A0216865_-_Black_screen_whe?= =?utf-8?Q?n?=
 amdgpu started during 6.2-rc1 boot with AMD IOMMU enabled
Message-ID: <Y+18UuVTKIshk8EF@nvidia.com>
References: <20230215153913.GA3189407@bhelgaas>
 <e3b866eb-830c-9037-39c7-978714aaf4d2@amd.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <e3b866eb-830c-9037-39c7-978714aaf4d2@amd.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?eGNQIPMdQ4xrC9eMhoRn4fQTXVQs2scvff20VfVeX++6CV9wFe/fRRar9IB5?=
 =?us-ascii?Q?UcDTBCk6M85/BxRqiBSil96vRNsYXbdC6eZ9Z6RbGQ5zxjIjbBDguur0rByN?=
 =?us-ascii?Q?6BukQU5b4tlty4iEMNIgGOk7hXj42C6BdW/gdEJSGQ0BiN7n05ZO/FfD7kwv?=
 =?us-ascii?Q?9UMyfl3oONDAs6p/34y2CMeMJIbJ95f4MFLajxQDgUH2+3uxlg/TKspd4UuR?=
 =?us-ascii?Q?TRuHJz856Qqf8A5T+MNAFFtSaNar0JnJI8q1I5TgMkAzYLTih/F2raV/Tz2k?=
 =?us-ascii?Q?nbYjg8eZkc+2cpCSZZKHj3SBvPzrDC5Wypxjg2y+3soXlGJ9q9y/Su+bk30G?=
 =?us-ascii?Q?FgSCk9bQolQR3gb0PoOonRwpao0uJA3A8E3WOIxgR3uJLhA6G3c5xMKWTm7X?=
 =?us-ascii?Q?CkEC2/7EtbL1dUjVqRrVVINNiCF0PcUnYtWeat8Ju3V+50XR0CTqTdpCwVQA?=
 =?us-ascii?Q?I/fBT95NVNM+A70Szqra5PiLGw6B2zYjIafjMpe2uLrP/QN4a7uaV3SYHNPo?=
 =?us-ascii?Q?PEzqpQiMjQKDhBofVtSEuKvSw0nxMYOtg3v6FlGW1GjiIyyYXxHRC1E6xwIh?=
 =?us-ascii?Q?9p+kpa+QSmreGRBeemQTuvJ3eEjf/5NfbJfDrA4eA/OqmpnY+0y0OhbVFddX?=
 =?us-ascii?Q?nlF4xBsvjGO0OWpNFM8Nd2jCZ1qjavYF3a/xzSWKVbrgleNnd3FqbdU4Ygl4?=
 =?us-ascii?Q?w6eHIQsXi0M09UTPtQ4lU/X9UdfTV35Q0WeVqDWKvVwDX+e6ywksNzgWbOFJ?=
 =?us-ascii?Q?K1Tm+kOLlS68hgG491htgE+nAoA49QnZNsvAxGxg7WjSSz2eGfQKIhchcf6N?=
 =?us-ascii?Q?JYxtjYdPbkVYDVH6RXKrleIHPhNcY+uiHrNQ+xW/Lw1eGygIxDQIwXbTtBXM?=
 =?us-ascii?Q?97hINWxxSl6VS6InTruyvegTX6eU7Z1dPhMVItHGF7PYUuzltzI6wDzM+2z6?=
 =?us-ascii?Q?yofzumzrZNtasaSCE//0XCI2UI/PAx0FK0GVli1rx1MViSXoj1Mz1GDLhQdi?=
 =?us-ascii?Q?KixIK/nTTHoUBHV0EouDFWQ+Jer2F034vEfaBm0+8Ml45eCBOynt1G7ptXFS?=
 =?us-ascii?Q?ihCEgcP42ZqBDObFpOgl12WqRcED579xgzxx88SelB7BLfvKf/OsJFCuhX7p?=
 =?us-ascii?Q?Oa54NeEDQn6GJTkSxyRrb/M2ai9PCGWZdiNxQSZ/y4oY19M7n2FKtJLAYECE?=
 =?us-ascii?Q?oR5AZNLQR6Hs2KYAg6JKNF002h7AIG8PDVFm2U7FZJKxgWJaJOK2ZS0bTRpA?=
 =?us-ascii?Q?rjsL3Zt/o+t6WJg4zonn/Tsi+ffrz+L8X3BHYfULdVnChDTYKagJmmzD6xAV?=
 =?us-ascii?Q?+hfS6+Dr+yMuhU4hO0d8sQ2xi80wn/tP3uoAzrExHh9L8jXinz3CZG88oqaB?=
 =?us-ascii?Q?MazEx03AcvA1G+p+0OCX+uysT904hNuPgiL32DhBZE9Zyl93fpz4Sq7M7EeL?=
 =?us-ascii?Q?/CbEyMkw6bzSKIbME8N/RU9L3Wosn1H7f6T7vfgGLx+AijenhBOy1+Q8Ezca?=
 =?us-ascii?Q?2Ebrtpz6BYvv95/oR0Wg+OAMLzMJ+NYDEbOXsPMps1gE4TXLyjwnulaSZiwG?=
 =?us-ascii?Q?TXI9jsXHwsPbYr+87IWq7KO2e9hyaJmJkQ4goMr3?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 323a7478-d51e-4ef8-e55a-08db0fb6e734
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Feb 2023 00:44:04.2087
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Mwp6GwrEFp/UGyIjsZAddWnILK412xjX455mU5IilZyB0hEsJ132zaQJGsA9I7fY
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4470
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Feb 15, 2023 at 07:35:45PM -0500, Felix Kuehling wrote:
> 
> If I understand this correctly, the HW or the BIOS is doing something wrong
> about reporting ACS. I don't know what the GPU driver can do other than add
> some quirk to stop using AMD IOMMUv2 on this HW/BIOS.

How about this:

diff --git a/drivers/iommu/amd/iommu_v2.c b/drivers/iommu/amd/iommu_v2.c
index 864e4ffb6aa94e..cc027ce9a6e86f 100644
--- a/drivers/iommu/amd/iommu_v2.c
+++ b/drivers/iommu/amd/iommu_v2.c
@@ -732,6 +732,7 @@ EXPORT_SYMBOL(amd_iommu_unbind_pasid);
 
 int amd_iommu_init_device(struct pci_dev *pdev, int pasids)
 {
+	struct iommu_dev_data *dev_data = dev_iommu_priv_get(&pdev->dev);
 	struct device_state *dev_state;
 	struct iommu_group *group;
 	unsigned long flags;
@@ -740,6 +741,9 @@ int amd_iommu_init_device(struct pci_dev *pdev, int pasids)
 
 	might_sleep();
 
+	if (!dev_data->ats.enabled)
+		return -EINVAL;
+
 	/*
 	 * When memory encryption is active the device is likely not in a
 	 * direct-mapped domain. Forbid using IOMMUv2 functionality for now.