Message-ID: <33d2ac92-3be0-f653-59ef-58a04c744ba9@amd.com>
Date:   Thu, 16 Feb 2023 09:55:18 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.7.1
Subject: =?UTF-8?Q?Re=3a_=5bregression=2c_bisected=2c_pci/iommu=5d_Bug=c2=a0?=
 =?UTF-8?Q?216865_-_Black_screen_when_amdgpu_started_during_6=2e2-rc1_boot_w?=
 =?UTF-8?Q?ith_AMD_IOMMU_enabled?=
Content-Language: en-US
To:     Vasant Hegde <vasant.hegde@amd.com>,
        Jason Gunthorpe <jgg@nvidia.com>,
        "Suthikulpanit, Suravee" <Suravee.Suthikulpanit@amd.com>
Cc:     Bjorn Helgaas <helgaas@kernel.org>,
        Baolu Lu <baolu.lu@linux.intel.com>,
        "Huang, Shimmer" <Shimmer.Huang@amd.com>,
        "Liu, Aaron" <Aaron.Liu@amd.com>, Joerg Roedel <jroedel@suse.de>,
        "regressions@lists.linux.dev" <regressions@lists.linux.dev>,
        Thorsten Leemhuis <regressions@leemhuis.info>,
        Linux PCI <linux-pci@vger.kernel.org>,
        "Pan, Xinhui" <Xinhui.Pan@amd.com>, amd-gfx@lists.freedesktop.org,
        LKML <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        Matt Fagnani <matt.fagnani@bell.net>,
        "Deucher, Alexander" <Alexander.Deucher@amd.com>,
        =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>
References: <20230215153913.GA3189407@bhelgaas>
 <e3b866eb-830c-9037-39c7-978714aaf4d2@amd.com> <Y+18UuVTKIshk8EF@nvidia.com>
 <9103b537-62c3-d6b2-b576-713406635455@amd.com>
From:   Felix Kuehling <felix.kuehling@amd.com>
In-Reply-To: <9103b537-62c3-d6b2-b576-713406635455@amd.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MWpXZnVsZi85cUttWE41aC93Mi9HU2F1ajloMko0ek1MLzVQdmZ2STFNYVdB?=
 =?utf-8?B?N2loT2hGaUMrUnVGTGRRN2c1NXo0Mi9KU2hhNmMvaGE0Q1prcTg3VUpqK0tv?=
 =?utf-8?B?cWtFMzZPamptcWtqWENpWE5CeGhZSUJTNTJPYURiSlNKcEJwYmdLNk9SZVVu?=
 =?utf-8?B?a3lFMmtXNmUzenFodVlvRzFJbUJXd0Q4NUUrNk1tbDFIbWlFaUh1MWZwTkFx?=
 =?utf-8?B?L3hCVE1ERm4rRlVEUWdqRkV6RTdLWlNVdEd4RUhvZFZEcjZVUVgzQ3M0OUVu?=
 =?utf-8?B?ZHVYQnFrYmx2Y3N2c0psOUxTUlB5V2xXTWFYVnkzVUZlY2F0NkxTMmFqOW90?=
 =?utf-8?B?TEN4emMxZ0V5UXFVU2NQU0kyRWtEMmVlOG04bmFLc0wvZUJobWlEMXFGU0lt?=
 =?utf-8?B?VUZ5OTlWYmZXZ2toYlNDN05lT0RNN2EwbEVhNzNSSVgxRDBSaDFNUXM5bW5B?=
 =?utf-8?B?Nmd2TlhKME9OM2UvbjVRcmdPZStnUDh0ODVZanFMVXRDeWxtUzNQMW5TRERu?=
 =?utf-8?B?a0VXZW9VR3dGOGJBZnVlekE0LzQrSzVrdHgyeDlmdm80Z1JvdXJKSEhoa0Nt?=
 =?utf-8?B?Mm5mN091ajNYZ1F2aXJlS3VTWjRFTWtkRVdHTEtDaXFBaXBUVlBHMG56Q3BL?=
 =?utf-8?B?L1FwVTlQSG1hUDhjSE5NWS9zK3Jwd2pwU3dYYy9GRHlTU0VMalR4aDRielBz?=
 =?utf-8?B?TFJ2a3psSmFBdkdESGEwMm1lNXlRM2JRUGwyR1dOMnpwNG53TU5FNEVzMmIw?=
 =?utf-8?B?aUx4R2JpTDZXN2RBNVN4NDl1Q1FPTkd5Z2RkVUw0WDhtekwyNmFwS010aEoy?=
 =?utf-8?B?L2kvazQwczFDOWh4REpLak5rRFNWRDJST1ozcG1HbkZ1Q0ZDZVpUbHZndWEr?=
 =?utf-8?B?d0V0UHhRNmJSUi9LSHZPSXpTaE5XMHBFOU5obSt6a3VFd1YxS2J6eHg5cWFI?=
 =?utf-8?B?NFlSUTY5Q1ZGT1djMllDTzkwNEZUcXZIcEhURkdST3hnYlNOZHpMWW55Ymcy?=
 =?utf-8?B?bHN5S2J5TWZOTHpHVjFocjZNbXVIR0FtOGFyckE4V0ZDdnRiTE5zZ1IzQ25o?=
 =?utf-8?B?a3RrbllsMUxlNkc4RGxBK0NyeXhvTE5wWk4vQnc5WUkwcE1DVC9CMTFUakg2?=
 =?utf-8?B?bFpUWXlxY2ZRK3piQ25iWkhvTXZKZy9haGFGTkozNW9QSXJObGp4YTZYbURs?=
 =?utf-8?B?Wjd2MDg0cEhVUjM4UHowQVR3b2tNZ2pRQlJqdVhFU0oxL2lxbnBHbS9EK2dk?=
 =?utf-8?B?ZXlLRHN5RHRpcEFNNDlPaGZtQU1ETTFIR2VidDN2U2VId3JVUEtRRHZwUlVt?=
 =?utf-8?B?WTg5OEgrRnBMUmZlUytsYnFyUi8wT1JrRTVTblh2VnhWSk5iR2UrTHZiN0Ex?=
 =?utf-8?B?b0gwRUl4RVlzMkxRSDdhQVVkYkFTMllBMmVNOWEzL0ozNzJxOFdJeWtad1By?=
 =?utf-8?B?eDh3TUpTQkJQMm56c3VmdHo5YlUvZU1RR1cvMjhqOEk4ZHVRUG1tZEU1VU9L?=
 =?utf-8?B?ZFFTUG41M2hKY1dvUG5PbXhOYXVNRVdvNUhkSmhKT0pYc29kUHZUNlQ1VEFo?=
 =?utf-8?B?TzNPaFlEK1FyNlNtc3dHZmNVWHZGK1RPdWQ3UUU2ZTF4Y2tHcnB5Q3IwRWdI?=
 =?utf-8?B?QWh6NEpTM2w4SVFWZm9Ec2JEeUhiZ01RbnhwY21yMjFPUG92SU56anpwRCty?=
 =?utf-8?B?d3RPNzF1WUYrQ1ZtTFdLV2FqWEJSN3NkMmV6NnZXNTh2NlQwQWxJQVVOdEtT?=
 =?utf-8?B?TlEyWnBMSGdnK1lTdDh1dHp5OE9xa0IyT2tpUGpZanQvWnFyZ2lLRmNQVmlZ?=
 =?utf-8?B?ZlFxbC9CL2VmbnlBNk1malExb0htSEVPK1FIVEY1TU1Sd1l2NEpEbW5oMG5Q?=
 =?utf-8?B?RlVjRmlQY3FWZ0xrQ2ljL1gzc2NwRTZGekxTc1lOSVlxaFdJQ3JyU2dSYTdk?=
 =?utf-8?B?bE1EU0piNDZoM0wxZlRCem9ubWxuSi9nZW5QcmIyZ3N6N3F4eWlHZzhqUlhW?=
 =?utf-8?B?ZFhSblZLbVF1Z3prbkZGbUUvOEFPQTlHS1pPbTIwcEh5eUl1SHBkUTBxVmNN?=
 =?utf-8?B?ckpkaGxCclN1eW9abkJiUnBDN1htc3JwNllWRkxNalBaVndDSm5BM205Z3VD?=
 =?utf-8?Q?OpUYsR5HRvItQyY8SEpPsNF2Y?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 84599b1e-9b4c-425f-8670-08db102dd402
X-MS-Exchange-CrossTenant-AuthSource: BN9PR12MB5115.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Feb 2023 14:55:22.1752
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: vosDTrqoFUKBQNngmVuL49QxGKXh8c3nrAX/ayXPgMqCWMGujlot868YjEebOzBMVo83nGYmkzPlR4P2W01ZXA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB8717
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

[+Suravee]

Am 2023-02-16 um 00:37 schrieb Vasant Hegde:
> Hi Jason,
>
>
> On 2/16/2023 6:14 AM, Jason Gunthorpe wrote:
>> On Wed, Feb 15, 2023 at 07:35:45PM -0500, Felix Kuehling wrote:
>>> If I understand this correctly, the HW or the BIOS is doing something wrong
>>> about reporting ACS. I don't know what the GPU driver can do other than add
>>> some quirk to stop using AMD IOMMUv2 on this HW/BIOS.
>> How about this:
>>
>> diff --git a/drivers/iommu/amd/iommu_v2.c b/drivers/iommu/amd/iommu_v2.c
>> index 864e4ffb6aa94e..cc027ce9a6e86f 100644
>> --- a/drivers/iommu/amd/iommu_v2.c
>> +++ b/drivers/iommu/amd/iommu_v2.c
>> @@ -732,6 +732,7 @@ EXPORT_SYMBOL(amd_iommu_unbind_pasid);
>>   
>>   int amd_iommu_init_device(struct pci_dev *pdev, int pasids)
>>   {
>> +	struct iommu_dev_data *dev_data = dev_iommu_priv_get(&pdev->dev);
>>   	struct device_state *dev_state;
>>   	struct iommu_group *group;
>>   	unsigned long flags;
>> @@ -740,6 +741,9 @@ int amd_iommu_init_device(struct pci_dev *pdev, int pasids)
>>   
>>   	might_sleep();
>>   
>> +	if (!dev_data->ats.enabled)
>> +		return -EINVAL;
>> +
> Thanks for the proposed fix. But aactually this will not solve the issue because
> current flow is :
>    - in this function it tries to allocate new domain
>    - Calls iommu_attach_group() which will call attach_device. In that path
>      it will try to enable ATS/PASID and hitting error.
>
> As I mentioned in other reply I think even current code returns error from
> amd_iommu_init_device() to GPU. But the issue is, in __iommu_attach_group() path
> it detached device from current domain, failed to attach to new domain and
> returned error. We didn't put the device back to old domain thats causing the
> issue. Below series should fix this issue.
>
> https://lore.kernel.org/linux-iommu/20230215052642.6016-1-vasant.hegde@amd.com/
>
> -Vasant
>