Message-ID: <9103b537-62c3-d6b2-b576-713406635455@amd.com>
Date:   Thu, 16 Feb 2023 11:07:06 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.8.1
Subject: =?UTF-8?Q?Re=3a_=5bregression=2c_bisected=2c_pci/iommu=5d_Bug=c2=a0?=
 =?UTF-8?Q?216865_-_Black_screen_when_amdgpu_started_during_6=2e2-rc1_boot_w?=
 =?UTF-8?Q?ith_AMD_IOMMU_enabled?=
Content-Language: en-US
To:     Jason Gunthorpe <jgg@nvidia.com>,
        Felix Kuehling <felix.kuehling@amd.com>
Cc:     Bjorn Helgaas <helgaas@kernel.org>,
        Baolu Lu <baolu.lu@linux.intel.com>,
        "Huang, Shimmer" <Shimmer.Huang@amd.com>,
        "Liu, Aaron" <Aaron.Liu@amd.com>, Joerg Roedel <jroedel@suse.de>,
        "regressions@lists.linux.dev" <regressions@lists.linux.dev>,
        Thorsten Leemhuis <regressions@leemhuis.info>,
        Linux PCI <linux-pci@vger.kernel.org>,
        "Pan, Xinhui" <Xinhui.Pan@amd.com>, amd-gfx@lists.freedesktop.org,
        LKML <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        Matt Fagnani <matt.fagnani@bell.net>,
        "Deucher, Alexander" <Alexander.Deucher@amd.com>,
        =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>
References: <20230215153913.GA3189407@bhelgaas>
 <e3b866eb-830c-9037-39c7-978714aaf4d2@amd.com> <Y+18UuVTKIshk8EF@nvidia.com>
From:   Vasant Hegde <vasant.hegde@amd.com>
In-Reply-To: <Y+18UuVTKIshk8EF@nvidia.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?aDVPSjRqaEpsV0ppcGVneHMyUzV5VGZta3NBUWJsazJ0dTlJZEVzSVZpSTU3?=
 =?utf-8?B?aG1kV3l0bkExbGRNSERIN0xWdXFrVWVOaEhsWC9DRDFENXRjYXF3OWtVcXRQ?=
 =?utf-8?B?MkxCRyt0Z29PUXpOdTlOZ0ZSM3g2RWU4RTMxcmRYTjJkbm1ibkhaWHFaVHNO?=
 =?utf-8?B?RWlEYWp6V25CLzlxdzh6NHpBNGZxMGJqSVM2Vjd5QXN1U1NVY1BEY0hwWm9L?=
 =?utf-8?B?UlYrT05hUFdKZUJRbitXcjhnWG1nUjNJQmN4ZlNub2tXVkh6VjdiZVlmWm5h?=
 =?utf-8?B?Rk4wd0h0SExrazB5YmRDZVJjNUZpMEVuU09JeC9PTTAwU1ZvU2NjcWxoK0Ix?=
 =?utf-8?B?Z0xKUGphdkF3TGY0VGg2dWFteDIwOHN0dVN5NjFYOHlvbDZjWlllR3JKK2hS?=
 =?utf-8?B?akZLY1JYTnZSMEZHK0NMelZnaGpIRk9EclhENlBWOUV2YXQybitjL2J5Tm5C?=
 =?utf-8?B?ZU42TFdUZk9WRDYvMmlSMTJUTGZqZmlqUndyS1Q1c0h6NWZicnQ3dnVXbDR3?=
 =?utf-8?B?VndwNVJmZ0EvM3d5L1ZDV0hMa3RJQkVFdkFCS2dnT2w5UkRZQTc3S2xRS0dn?=
 =?utf-8?B?TU1IeVlXWk85R0UyT3NtV0RYWm1oTE5jTHVrTnBYbFpac0NvU0JLNFFqVitZ?=
 =?utf-8?B?MnFCbTd2b2dNWG1sa2d0K3dqeTV3RGZiUmRncFJoRWlBUkVuMGhHSkdpQzNj?=
 =?utf-8?B?azg5NElJaDd6ZkJxK3d4Y3h2MjZGTUw4alhvQWIxVjBaV2I1WmE0Wml3QVM5?=
 =?utf-8?B?djVpdWtzbWprUDVYZFNpYjk2U09iQkVkT2hoeWRINHFlVnB5ZHRBUFpzREto?=
 =?utf-8?B?eFBsLyt4WjdMSVF0YmpqcUUxV0xLaURoT0lRWnVmOFl3N1k4T2FMbU84VmM5?=
 =?utf-8?B?cFI4U0VHYk11RW1MZjJnZDZIMXRmemh6c3R3a1BpY1QybytRNHV5TEVGOHNI?=
 =?utf-8?B?aW9zM2E5WTd5R2FYNzBRTThyYWxMVlFXdzF5NWFhQVVGL1dMZzJ6QWcxTzlw?=
 =?utf-8?B?WTJYc0x5Mlg4Q0U5cElwcG4rZzlnc21VeHR0RTVpT25WV1piY1JaWlRGclJN?=
 =?utf-8?B?cWx0UHR6aUNHTHFHZ0pHekozZXpSdGpWZ29MeXRRRVR4ajVtWHZ5MTAxK0JW?=
 =?utf-8?B?SStPYk9rM2tmeHVLZXRIZUt5QUNhM29QSXc0OURvWWhULzFaMG9ZVXJnV2E2?=
 =?utf-8?B?VFNEakN2SGJRR3JkVHNqRkFlSldDNFBINmZ5UXp2SlhJRXdQNDMzWE1wRHFu?=
 =?utf-8?B?TmhoSlRVRDU3ZDM3R2YxWlc1dXRwMkNOSEZWQ05DZ3ZRdEFGU2xKSFVOcDhS?=
 =?utf-8?B?YU5TUjh5S0x1YnBSREw3a2t3RzJCd3N2SEp3YmZaNEZiSW9lSDUzTVp4Tk1l?=
 =?utf-8?B?QkRhR2k4V1huVWdyTzVVZjlkT0M3RjRBeUN0L0pxK3FiMnJNVmZzS1J6a0py?=
 =?utf-8?B?T3N6Zi9QR2ovZ1M5RVVFVlJtSHFJb1hNSUFBV1N1aU0zRGpxZG1lK3l4YVY5?=
 =?utf-8?B?VWFkRG1PdmVrWEZoVmMza1VEMzBIVVNHSm1TaDl4L0xTak0wbWhOR25lRTJS?=
 =?utf-8?B?QzdlczJtUnlWd2tOb1BEbHorM0xVUmpPR0UzNmFvTll1bjRtdlJiRVF2ZGtr?=
 =?utf-8?B?Zi9mRU82RzlRWlJCRGRwcHZBZjRFczFWQ3JxVC9ybXNVUEJQYmZuVVVHR2Nh?=
 =?utf-8?B?VnUxWnk3cnBSTkRQUC9xVTRZcFpSTjEwZU9xcGtXU3lzU2hMNy8zV3drZUxN?=
 =?utf-8?B?citDWjFzZ2hOa24xUzRHaTRmR3R6bXZPR0FHQ3RSYUppY05YckZRMFBISG4r?=
 =?utf-8?B?MHM2ZlJZSmhaRzQ2QTlBS0FBaGVzYTQ4RDNJM1UrT3haQXhKUWQ4UU9LSXJs?=
 =?utf-8?B?NERZdThuR29FVzBLRkZzMWxKT3RtNllWc0E3S0NtYWR5bUlTWmZVSGJJQlpz?=
 =?utf-8?B?TGE1WUdSbGVTN0U2N2Z0djhNOU9zKzNVSXdTbUlTVm9NcXU4WHBvK0VxOEpo?=
 =?utf-8?B?UlF0M0ZnbGFsV3FGbzRMSFN5OTBvb0krMUVGODFhVlY4UFFGNjJ4M2hJQi9Z?=
 =?utf-8?B?UVIvNFV6dEdXU0t6UHBBOE5EQ3ZSa3Q0c2ZtSzZwMlNTbWFrVDdkNWVwODV1?=
 =?utf-8?Q?g63BEMN5WSG8ZdzF44rVxM68T?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6d0fecd4-5528-4c54-a5aa-08db0fdfdfea
X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB6048.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Feb 2023 05:37:21.5627
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: hnJKC/9FAv/7hmTAiOQxWIk/w3mjplnXjHBq+9iqEh/T6ecr01SG5sjX1CTv2bv77FPsgrjj8+Jvcvyd83Zxnw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB8557
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Jason,


On 2/16/2023 6:14 AM, Jason Gunthorpe wrote:
> On Wed, Feb 15, 2023 at 07:35:45PM -0500, Felix Kuehling wrote:
>>
>> If I understand this correctly, the HW or the BIOS is doing something wrong
>> about reporting ACS. I don't know what the GPU driver can do other than add
>> some quirk to stop using AMD IOMMUv2 on this HW/BIOS.
> 
> How about this:
> 
> diff --git a/drivers/iommu/amd/iommu_v2.c b/drivers/iommu/amd/iommu_v2.c
> index 864e4ffb6aa94e..cc027ce9a6e86f 100644
> --- a/drivers/iommu/amd/iommu_v2.c
> +++ b/drivers/iommu/amd/iommu_v2.c
> @@ -732,6 +732,7 @@ EXPORT_SYMBOL(amd_iommu_unbind_pasid);
>  
>  int amd_iommu_init_device(struct pci_dev *pdev, int pasids)
>  {
> +	struct iommu_dev_data *dev_data = dev_iommu_priv_get(&pdev->dev);
>  	struct device_state *dev_state;
>  	struct iommu_group *group;
>  	unsigned long flags;
> @@ -740,6 +741,9 @@ int amd_iommu_init_device(struct pci_dev *pdev, int pasids)
>  
>  	might_sleep();
>  
> +	if (!dev_data->ats.enabled)
> +		return -EINVAL;
> +

Thanks for the proposed fix. But aactually this will not solve the issue because
current flow is :
  - in this function it tries to allocate new domain
  - Calls iommu_attach_group() which will call attach_device. In that path
    it will try to enable ATS/PASID and hitting error.

As I mentioned in other reply I think even current code returns error from
amd_iommu_init_device() to GPU. But the issue is, in __iommu_attach_group() path
it detached device from current domain, failed to attach to new domain and
returned error. We didn't put the device back to old domain thats causing the
issue. Below series should fix this issue.

https://lore.kernel.org/linux-iommu/20230215052642.6016-1-vasant.hegde@amd.com/

-Vasant