Received-SPF: pass (google.com: domain of linux-kernel+bounces-184807-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <b0ee8466-8668-415f-80f5-b1c65dcb0950@amd.com>
Date: Tue, 21 May 2024 16:03:13 +0530
User-Agent: Mozilla Thunderbird
Subject: Re: 6.10/regression/bisected commit c4cb23111103 causes sleeping
 function called from invalid context at kernel/locking/mutex.c:585
To: Mikhail Gavrilov <mikhail.v.gavrilov@gmail.com>,
 suravee.suthikulpanit@amd.com, jgg@nvidia.com, jroedel@suse.de,
 "Deucher, Alexander" <Alexander.Deucher@amd.com>,
 amd-gfx list <amd-gfx@lists.freedesktop.org>,
 dri-devel <dri-devel@lists.freedesktop.org>,
 Linux List Kernel Mailing <linux-kernel@vger.kernel.org>
References: <CABXGCsN1z2gj99zSdhQWynpTXBymrqHejDfF8axxxoiZ_0g_-g@mail.gmail.com>
Content-Language: en-US
From: Vasant Hegde <vasant.hegde@amd.com>
In-Reply-To: <CABXGCsN1z2gj99zSdhQWynpTXBymrqHejDfF8axxxoiZ_0g_-g@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?WHNYZTNUMVh3ckZTNk1MaEJzYWY2QUROTkhDU0ZBMXBtVFVlVGV0cXpmajd2?=
 =?utf-8?B?cGg0RmI5UU1NRmZ2NGNJaVdPVWNuZHhrbmV1UENIYTd6bmlldnQveThRaUFU?=
 =?utf-8?B?T3IzSTNQRUw2b0RRajVDNStuNGh3REJSWVgvMmNOQUxOUTBqbk0vaHBrMnhM?=
 =?utf-8?B?OTd0b1U0dnloT1VoeHY2LytOUjFEZXFPTmRwQWJzeTFHekR2UFI4ZDhwZW10?=
 =?utf-8?B?QXdDTjlWQVNHNHNaQUJIV1BsSXVJdUhrSUpKZy9RNlFqVHdQc2VhTUlieXBZ?=
 =?utf-8?B?RVllR1hZdXZndzZUOXJaOFFlVGplUEtzSXVRMVpQQmRTSFRhSSswVWIvM3ZX?=
 =?utf-8?B?cyt0cHNpSVlQSjNuRUViaUZQSWtKY1RUQ0c2OVc1M2ZMSUJENFVUQ1dRWGhW?=
 =?utf-8?B?OVdLbWZ1UE5sbmJyVzZyNDg3WVRDL1FVcjRIa2ZTYmNwcXlqY1V3TTFCLzk4?=
 =?utf-8?B?RklLZHIwdEdaVDNiak03b1JXZXRPdjBjZGZ2N1ZtbE9SU3BoaXNmTlZjcUZi?=
 =?utf-8?B?L2w3ck9nMTZTSGVuYnpNSGViT040MkNwVmwzb2lBb1lya1FEdTJCYWJMakIr?=
 =?utf-8?B?VWp0d3Y1R0Z3YXFCSEx0WEwzbVMrbkFtZkpPM084Y1FnRGZlcjhtZzdock5Y?=
 =?utf-8?B?ZXBGeVBYVmxMZEovK0FHeFprWU5JbnFzMUd5Ykhub0dRdjl6OTBDZHRmRDB2?=
 =?utf-8?B?d0xSS3FDcVNvTWFSdE5xUFpJeE9NRWhWelNIZ2hkVlgvclEwVnBKMGNBZGZD?=
 =?utf-8?B?akZUcy9kM3NLdWNDSnVROXlQbHJZNmVNWDFKM3NXZU5EaThKQzkyV2NYYUxX?=
 =?utf-8?B?eGdjZ1YyYTJ5bDBEOGZVSWwwS0k4OXlZVjF0NzI1YTQ2dysrNVlSTzhwWkwv?=
 =?utf-8?B?SGV2Y2JxdzFhQmdjdzk1dEdOVTV3VTBqK1ZESlYrWnliRjhET1cyNTF0Zldy?=
 =?utf-8?B?UkJ1c0l2bHdBQTY2ZllHS0Vnd2YraVRCdi9jc2tDYjNucjNEa3c2WUVPOUtw?=
 =?utf-8?B?dUlwUzk3WlIwQ3RRNDV6VzQwcGxiME5ldkJqNWliVkIyeVgzWG1xYy8zcXd6?=
 =?utf-8?B?TVpYTXl5UkloVHVta09OZlREbXRxNDVJSU1pREJBa2RBRGdSQWZUb0JBSDVx?=
 =?utf-8?B?R1ZXTGhVYU1CY2o2SWNnRVlQdUhseDVuMzlDcTRCU2c3Z3dRNXM0bkVQaFBO?=
 =?utf-8?B?RkRNU1FzMDFUYzdUaGVPbzhMbFNneExPWXA1U1lkRHpEbGNlUjBzL08xT2RL?=
 =?utf-8?B?azliUlNDYXIwTlJsYjVKQUJQTXJZRWNhTHJjci9TYWU2ZXM2VmRMOUFyTDVO?=
 =?utf-8?B?bWRSYlVqaHQ0V2VTbjYxZ0tCWFR4aVkvaVIxVmR1Um1Za1I4dVV5Tm5qd1hq?=
 =?utf-8?B?VGJTME10NG9YNnpUSGRzVjhUVXlJY29GSm1SYkR0ZW80OCszd2pERG9OaTNw?=
 =?utf-8?B?ZVQ0YklNSExNYVlUN1JRZWJhMnJmRFJ6aWpoZVAxL3JUZEdlbjdTVEoxNGE0?=
 =?utf-8?B?U0JaQXFnVUNYWVBoUjNQZlRNVDA2aVVPT1VIQW9qWGlRUFdGbVBQUXJQVnZq?=
 =?utf-8?B?SW5yUkhyR3J6bXB0eG8wSGpzTEhOOWFaTGluM3BuVmFnZDYvbTRpMHIzT29n?=
 =?utf-8?B?KzVHbDdEK2UrS2RlS2FvL0FtdUZTUHlHNWhFR1RGcmErSEpuSU9qa0NiMTJJ?=
 =?utf-8?B?STNTSGI0N3RBdzhVd3NSSFk3SExjbzg2a282RnN6NlVsN0NiR01YZnk4cnB6?=
 =?utf-8?B?Y2J6Z0NtVmxKWDZVeENUS05nYy96MUIyemtzODh4dEN3NXl2aWh0WG10VCtx?=
 =?utf-8?B?OTNHZmtqU1FGaWh5MzdtOVQyOUhYVDFKaDlvK08zT01Pc2FJZTkzbk1aeVRk?=
 =?utf-8?B?WDRVMm5MWHFZVWQyS2prL2FlcXRyT3Y4ZnRJcGdkeWZCSWJONUk3cXA4bXM3?=
 =?utf-8?B?WnA2SzRiSEtTcURvNTdWTXo2QzJ0UjdWMEJ6am8vaWdQcHBzM3Qybm1kTnQw?=
 =?utf-8?B?V0JwVm11Z3JvSDlPRnRQMWw4VmdRcFdEblpJdUp5YVpRWFRwQVJXSU4rYzhK?=
 =?utf-8?B?Vkp6Q2Uwd08vdE1wWHlsdWNLNlJCREhoY2tKbjF6UERTYUt5M0ZqRjZIM21j?=
 =?utf-8?Q?BdTkvyRA97ZplxaMxtRmSHk51?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d1df44ef-ff24-4ae3-7cf1-08dc79817021
X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB6048.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 May 2024 10:33:22.2258
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: AcPr3pzTsPzstrGKMPXqjDuCP9d5yonqlrxy3d6R770edKDJk5vQ7BXqOCTdcI9TnRGAiQ4EPP0mIm2B9beMVw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB7797

Hi Mikhail,

Thanks for the report. I will look into it this week and get back to you.

-Vasant


On 5/21/2024 3:09 PM, Mikhail Gavrilov wrote:
> Hi,
> Yesterday on the fresh kernel snapshot
> I spotted a new bug message with follow stacktrace:
> [    4.307097] BUG: sleeping function called from invalid context at
> kernel/locking/mutex.c:585
> [    4.307135] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid:
> 1, name: swapper/0
> [    4.307150] preempt_count: 3, expected: 0
> [    4.307159] RCU nest depth: 0, expected: 0
> [    4.307168] 4 locks held by swapper/0/1:
> [    4.307176]  #0: ffff8881080ba920 (&group->mutex){+.+.}-{3:3}, at:
> bus_iommu_probe+0xf6/0x4c0
> [    4.307203]  #1: ffff88811654c1b8 (&domain->lock){....}-{2:2}, at:
> amd_iommu_attach_device+0x1ad/0x1e80
> [    4.307227]  #2: ffff888113518c18 (&dev_data->lock){....}-{2:2},
> at: amd_iommu_attach_device+0x213/0x1e80
> [    4.307243]  #3: ffff888108393030 (&iommu->lock){....}-{2:2}, at:
> amd_iommu_iopf_add_device+0x69/0x140
> [    4.307243] irq event stamp: 1021718
> [    4.307243] hardirqs last  enabled at (1021717):
> [<ffffffff9cc35f2e>] kasan_quarantine_put+0x12e/0x250
> [    4.307243] hardirqs last disabled at (1021718):
> [<ffffffff9f46895c>] _raw_spin_lock_irqsave+0x7c/0xa0
> [    4.307243] softirqs last  enabled at (1020154):
> [<ffffffff9c2721fb>] __irq_exit_rcu+0xbb/0x1c0
> [    4.307243] softirqs last disabled at (1020147):
> [<ffffffff9c2721fb>] __irq_exit_rcu+0xbb/0x1c0
> [    4.307243] Preemption disabled at:
> [    4.307243] [<0000000000000000>] 0x0
> [    4.307243] CPU: 0 PID: 1 Comm: swapper/0 Not tainted
> 6.10.0-0.rc0.20240520giteb6a9339efeb.9.fc41.x86_64+debug #1
> [    4.307243] Hardware name: ASUS System Product Name/ROG STRIX
> B650E-I GAMING WIFI, BIOS 2611 04/07/2024
> [    4.307243] Call Trace:
> [    4.307243]  <TASK>
> [    4.307243]  dump_stack_lvl+0x84/0xd0
> [    4.307243]  __might_resched.cold+0x1f7/0x23d
> [    4.307243]  ? __pfx___might_resched+0x10/0x10
> [    4.307243]  __mutex_lock+0xf3/0x13f0
> [    4.307243]  ? iopf_queue_add_device+0xd2/0x5d0
> [    4.307243]  ? __pfx___mutex_lock+0x10/0x10
> [    4.307243]  ? find_held_lock+0x34/0x120
> [    4.307243]  ? __pfx_lock_acquired+0x10/0x10
> [    4.307243]  ? iopf_queue_add_device+0xd2/0x5d0
> [    4.307243]  iopf_queue_add_device+0xd2/0x5d0
> [    4.307243]  amd_iommu_iopf_add_device+0xcd/0x140
> [    4.307243]  amd_iommu_attach_device+0xdc8/0x1e80
> [    4.307243]  ? iommu_create_device_direct_mappings+0x571/0x7d0
> [    4.307243]  __iommu_attach_device+0x64/0x250
> [    4.307243]  __iommu_device_set_domain+0x122/0x1c0
> [    4.307243]  __iommu_group_set_domain_internal+0xfa/0x2d0
> [    4.307243]  iommu_setup_default_domain+0x918/0xcd0
> [    4.307243]  bus_iommu_probe+0x1ad/0x4c0
> [    4.307243]  ? __pfx_bus_iommu_probe+0x10/0x10
> [    4.307243]  iommu_device_register+0x184/0x230
> [    4.307243]  ? amd_iommu_iopf_init+0xfd/0x170
> [    4.307243]  iommu_go_to_state+0xf87/0x3890
> [    4.307243]  ? __pfx_iommu_go_to_state+0x10/0x10
> [    4.307243]  ? lockdep_hardirqs_on+0x7c/0x100
> [    4.307243]  ? _raw_spin_unlock_irqrestore+0x4f/0x80
> [    4.307243]  ? add_device_randomness+0xb8/0xf0
> [    4.307243]  ? __pfx_add_device_randomness+0x10/0x10
> [    4.307243]  ? __pfx_pci_iommu_init+0x10/0x10
> [    4.307243]  amd_iommu_init+0x21/0x60
> [    4.307243]  ? __pfx_pci_iommu_init+0x10/0x10
> [    4.307243]  pci_iommu_init+0x38/0x60
> [    4.307243]  do_one_initcall+0xd6/0x460
> [    4.307243]  ? __pfx_do_one_initcall+0x10/0x10
> [    4.307243]  ? kernel_init_freeable+0x4cb/0x750
> [    4.307243]  ? kasan_unpoison+0x44/0x70
> [    4.307243]  kernel_init_freeable+0x6b4/0x750
> [    4.307243]  ? __pfx_kernel_init_freeable+0x10/0x10
> [    4.307243]  ? __pfx_kernel_init+0x10/0x10
> [    4.307243]  ? __pfx_kernel_init+0x10/0x10
> [    4.307243]  kernel_init+0x1c/0x150
> [    4.307243]  ? __pfx_kernel_init+0x10/0x10
> [    4.307243]  ret_from_fork+0x31/0x70
> [    4.307243]  ? __pfx_kernel_init+0x10/0x10
> [    4.307243]  ret_from_fork_asm+0x1a/0x30
> [    4.307243]  </TASK>
> 
> [    4.307243] =============================
> [    4.307243] [ BUG: Invalid wait context ]
> [    4.307243] 6.10.0-0.rc0.20240520giteb6a9339efeb.9.fc41.x86_64+debug
> #1 Tainted: G        W         -------  ---
> [    4.307243] -----------------------------
> [    4.307243] swapper/0/1 is trying to lock:
> [    4.307243] ffff88810de2fa88 (&queue->lock){....}-{3:3}, at:
> iopf_queue_add_device+0xd2/0x5d0
> [    4.307243] other info that might help us debug this:
> [    4.307243] context-{4:4}
> [    4.307243] 4 locks held by swapper/0/1:
> [    4.307243]  #0: ffff8881080ba920 (&group->mutex){+.+.}-{3:3}, at:
> bus_iommu_probe+0xf6/0x4c0
> [    4.307243]  #1: ffff88811654c1b8 (&domain->lock){....}-{2:2}, at:
> amd_iommu_attach_device+0x1ad/0x1e80
> [    4.307243]  #2: ffff888113518c18 (&dev_data->lock){....}-{2:2},
> at: amd_iommu_attach_device+0x213/0x1e80
> [    4.307243]  #3: ffff888108393030 (&iommu->lock){....}-{2:2}, at:
> amd_iommu_iopf_add_device+0x69/0x140
> [    4.307243] stack backtrace:
> [    4.307243] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G        W
>   -------  ---
> 6.10.0-0.rc0.20240520giteb6a9339efeb.9.fc41.x86_64+debug #1
> [    4.307243] Hardware name: ASUS System Product Name/ROG STRIX
> B650E-I GAMING WIFI, BIOS 2611 04/07/2024
> [    4.307243] Call Trace:
> [    4.307243]  <TASK>
> [    4.307243]  dump_stack_lvl+0x84/0xd0
> [    4.307243]  __lock_acquire.cold+0x1fe/0x2a0
> [    4.307243]  ? __pfx___lock_acquire+0x10/0x10
> [    4.307243]  ? ret_from_fork_asm+0x1a/0x30
> [    4.307243]  lock_acquire+0x1ae/0x540
> [    4.307243]  ? iopf_queue_add_device+0xd2/0x5d0
> [    4.307243]  ? __pfx_lock_acquire+0x10/0x10
> [    4.307243]  ? __printk_cpu_sync_put+0x35/0x60
> [    4.307243]  ? add_taint+0x2a/0x70
> [    4.307243]  ? __might_resched.cold+0x203/0x23d
> [    4.307243]  ? __pfx___might_resched+0x10/0x10
> [    4.307243]  __mutex_lock+0x189/0x13f0
> [    4.307243]  ? iopf_queue_add_device+0xd2/0x5d0
> [    4.307243]  ? iopf_queue_add_device+0xd2/0x5d0
> [    4.307243]  ? __pfx___mutex_lock+0x10/0x10
> [    4.307243]  ? find_held_lock+0x34/0x120
> [    4.307243]  ? __pfx_lock_acquired+0x10/0x10
> [    4.307243]  ? iopf_queue_add_device+0xd2/0x5d0
> [    4.307243]  iopf_queue_add_device+0xd2/0x5d0
> [    4.307243]  amd_iommu_iopf_add_device+0xcd/0x140
> [    4.307243]  amd_iommu_attach_device+0xdc8/0x1e80
> [    4.307243]  ? iommu_create_device_direct_mappings+0x571/0x7d0
> [    4.307243]  __iommu_attach_device+0x64/0x250
> [    4.307243]  __iommu_device_set_domain+0x122/0x1c0
> [    4.307243]  __iommu_group_set_domain_internal+0xfa/0x2d0
> [    4.307243]  iommu_setup_default_domain+0x918/0xcd0
> [    4.307243]  bus_iommu_probe+0x1ad/0x4c0
> [    4.307243]  ? __pfx_bus_iommu_probe+0x10/0x10
> [    4.307243]  iommu_device_register+0x184/0x230
> [    4.307243]  ? amd_iommu_iopf_init+0xfd/0x170
> [    4.307243]  iommu_go_to_state+0xf87/0x3890
> [    4.307243]  ? __pfx_iommu_go_to_state+0x10/0x10
> [    4.307243]  ? lockdep_hardirqs_on+0x7c/0x100
> [    4.307243]  ? _raw_spin_unlock_irqrestore+0x4f/0x80
> [    4.307243]  ? add_device_randomness+0xb8/0xf0
> [    4.307243]  ? __pfx_add_device_randomness+0x10/0x10
> [    4.307243]  ? __pfx_pci_iommu_init+0x10/0x10
> [    4.307243]  amd_iommu_init+0x21/0x60
> [    4.307243]  ? __pfx_pci_iommu_init+0x10/0x10
> [    4.307243]  pci_iommu_init+0x38/0x60
> [    4.307243]  do_one_initcall+0xd6/0x460
> [    4.307243]  ? __pfx_do_one_initcall+0x10/0x10
> [    4.307243]  ? kernel_init_freeable+0x4cb/0x750
> [    4.307243]  ? kasan_unpoison+0x44/0x70
> [    4.307243]  kernel_init_freeable+0x6b4/0x750
> [    4.307243]  ? __pfx_kernel_init_freeable+0x10/0x10
> [    4.307243]  ? __pfx_kernel_init+0x10/0x10
> [    4.307243]  ? __pfx_kernel_init+0x10/0x10
> [    4.307243]  kernel_init+0x1c/0x150
> [    4.307243]  ? __pfx_kernel_init+0x10/0x10
> [    4.307243]  ret_from_fork+0x31/0x70
> [    4.307243]  ? __pfx_kernel_init+0x10/0x10
> [    4.307243]  ret_from_fork_asm+0x1a/0x30
> [    4.307243]  </TASK>
> [    4.311628] AMD-Vi: Extended features (0x246577efa2254afa, 0x0):
> PPR NX GT [5] IA GA PC GA_vAPIC
> [    4.311639] AMD-Vi: Interrupt remapping enabled
> [    4.366191] AMD-Vi: Virtual APIC enabled
> 
> Bisect pointed to commit:
> commit c4cb23111103a841c2df30058597398443bcad5f (HEAD)
> Author: Vasant Hegde <vasant.hegde@amd.com>
> Date:   Thu Apr 18 10:33:57 2024 +0000
> 
>     iommu/amd: Add support for enable/disable IOPF
> 
>     Return success from enable_feature(IOPF) path as this interface is going
>     away. Instead we will enable/disable IOPF support in attach/detach device
>     path.
> 
>     In attach device path, if device is capable of PRI, then we will add it to
>     per IOMMU IOPF queue and enable PPR support in IOMMU. Also it will
>     attach device to domain even if it fails to enable PRI or add device to
>     IOPF queue as device can continue to work without PRI support.
> 
>     In detach device patch it follows following sequence:
>       - Flush the queue for the given device
>       - Disable PPR support in DTE[devid]
>       - Remove device from IOPF queue
>       - Disable device PRI
> 
>     Also add IOMMU_IOPF as dependency to AMD_IOMMU driver.
> 
>     Co-developed-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
>     Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
>     Signed-off-by: Vasant Hegde <vasant.hegde@amd.com>
>     Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
>     Link: https://lore.kernel.org/r/20240418103400.6229-13-vasant.hegde@amd.com
>     Signed-off-by: Joerg Roedel <jroedel@suse.de>
> 
>  drivers/iommu/amd/Kconfig     |  1 +
>  drivers/iommu/amd/amd_iommu.h |  4 ++++
>  drivers/iommu/amd/iommu.c     | 39 +++++++++++++++++++++++++++++++--------
>  drivers/iommu/amd/ppr.c       | 41 +++++++++++++++++++++++++++++++++++++++++
>  4 files changed, 77 insertions(+), 8 deletions(-)
> 
> 
> Unfortunately I can't check revert c4cb23111103 due to conflicts:
>> git revert -n c4cb23111103a841c2df30058597398443bcad5f
> Auto-merging drivers/iommu/amd/Kconfig
> CONFLICT (content): Merge conflict in drivers/iommu/amd/Kconfig
> Auto-merging drivers/iommu/amd/amd_iommu.h
> Auto-merging drivers/iommu/amd/iommu.c
> CONFLICT (content): Merge conflict in drivers/iommu/amd/iommu.c
> Auto-merging drivers/iommu/amd/ppr.c
> error: could not revert c4cb23111103... iommu/amd: Add support for
> enable/disable IOPF
> 
> 
> I also attach below a full kernel log and build config.
> 
> My hardware specs: https://linux-hardware.org/?probe=ca80d0a307
> 
> Vasant or anyone else from the AMD team can you look please.
>