Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1;
Received-SPF: Pass (protection.outlook.com: domain of amd.com designates
 165.204.84.17 as permitted sender) receiver=protection.outlook.com;
 client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C
From:   Brett Creeley <brett.creeley@amd.com>
To:     <jgg@ziepe.ca>, <yishaih@nvidia.com>,
        <shameerali.kolothum.thodi@huawei.com>, <kevin.tian@intel.com>,
        <alex.williamson@redhat.com>, <dan.carpenter@linaro.org>
CC:     <kvm@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        <brett.creeley@amd.com>, <shannon.nelson@amd.com>
Subject: [PATCH vfio 2/3] pds/vfio: Fix mutex lock->magic != lock warning
Date:   Thu, 14 Sep 2023 12:15:39 -0700
Message-ID: <20230914191540.54946-3-brett.creeley@amd.com>
In-Reply-To: <20230914191540.54946-1-brett.creeley@amd.com>
References: <20230914191540.54946-1-brett.creeley@amd.com>
MIME-Version: 1.0
Content-Type: text/plain
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Sep 2023 19:15:58.1265
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 12e20de4-aaaa-4dc2-c25a-08dbb55706ad
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com]
X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002636B.namprd02.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW3PR12MB4569
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Thu, 14 Sep 2023 12:16:36 -0700 (PDT)
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email

The following BUG was found when running on a kernel with
CONFIG_DEBUG_MUTEXES=y set:

[  502.794510] DEBUG_LOCKS_WARN_ON(lock->magic != lock)
[  502.794519] WARNING: CPU: 16 PID: 2571 at kernel/locking/mutex.c:1085 mutex_trylock+0x10d/0x120
[  502.794537] Modules linked in: pds_vfio_pci(OE) pds_core xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nft_compat
nf_nat_tftp nf_conntrack_tftp bridge stp llc nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4
nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink
intel_rapl_msr intel_rapl_common isst_if_common nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl mei_me ses intel_cstate ipmi_ssif sunrpc enclosure mei nvme
intel_uncore nvme_core hpwdt hpilo ioatdma acpi_tad acpi_ipmi pcspkr lpc_ich intel_pch_thermal acpi_power_meter ipmi_si xfs
libcrc32c sd_mod t10_pi crc64_rocksoft crc64 sg mgag200 drm_kms_helper drm_shmem_helper crc32c_intel serio_raw igb smartpqi
dca drm scsi_transport_sas i2c_algo_bit ionic wmi dm_mirror dm_region_hash dm_log dm_mod ipmi_devintf ipmi_msghandler
fuse [last unloaded: pds_core]
[  502.794673] CPU: 16 PID: 2571 Comm: bash Tainted: G S         OE      6.6.0-rc1-next-20230911 #1
[  502.794676] Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 01/23/2021
[  502.794678] RIP: 0010:mutex_trylock+0x10d/0x120
[  502.794681] Code: ff 85 c0 0f 84 40 ff ff ff 8b 3d f2 35 eb 00 85 ff 0f 85 32 ff ff ff 48 c7 c6 47 ff d3 bc 48 c7 c7 80 69 d3 bc e8 33 c7 4a ff <0f> 0b e9 18 ff ff ff b8 01 00 00 00 e9 5c ff ff ff 66 90 90 90 90
[  502.794684] RSP: 0018:ffffabadc750fda8 EFLAGS: 00010282
[  502.794686] RAX: 0000000000000000 RBX: ffff930000451000 RCX: 0000000000000027
[  502.794688] RDX: 0000000000000027 RSI: ffffabadc750fca0 RDI: ffff930f7f920948
[  502.794690] RBP: ffff930000451708 R08: 0000000000000000 R09: c0000000ffff7fff
[  502.794692] R10: 0000000000000001 R11: ffffabadc750fc40 R12: 0000000000000000
[  502.794694] R13: fffffffffffffff2 R14: ffffabadc750fea0 R15: ffff92ffc85dc520
[  502.794695] FS:  00007f3d8e322740(0000) GS:ffff930f7f900000(0000) knlGS:0000000000000000
[  502.794697] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  502.794698] CR2: 000055cc4f6d9310 CR3: 0000000109894005 CR4: 00000000007706e0
[  502.794700] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  502.794701] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  502.794703] PKRU: 55555554
[  502.794704] Call Trace:
[  502.794707]  <TASK>
[  502.794710]  ? __warn+0x85/0x140
[  502.794718]  ? mutex_trylock+0x10d/0x120
[  502.794720]  ? report_bug+0xfc/0x1e0
[  502.794727]  ? handle_bug+0x3f/0x70
[  502.794732]  ? exc_invalid_op+0x17/0x70
[  502.794735]  ? asm_exc_invalid_op+0x1a/0x20
[  502.794742]  ? mutex_trylock+0x10d/0x120
[  502.794744]  ? mutex_trylock+0x10d/0x120
[  502.794748]  pds_vfio_reset+0x3a/0x60 [pds_vfio_pci]
[  502.794756]  pci_reset_function+0x4b/0x70
[  502.794763]  reset_store+0x5b/0xa0
[  502.794770]  kernfs_fop_write_iter+0x137/0x1d0
[  502.794776]  vfs_write+0x2de/0x410
[  502.794784]  ksys_write+0x5d/0xd0
[  502.794787]  do_syscall_64+0x3b/0x90
[  502.794790]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  502.794794] RIP: 0033:0x7f3d8d51fa28
[  502.794796] Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 15 4d 2a 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4 55
[  502.794799] RSP: 002b:00007ffc7b6c76f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[  502.794801] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d8d51fa28
[  502.794803] RDX: 0000000000000002 RSI: 000055cc4f742aa0 RDI: 0000000000000001
[  502.794804] RBP: 000055cc4f742aa0 R08: 000000000000000a R09: 00007f3d8d57fae0
[  502.794805] R10: 000000000000000a R11: 0000000000000246 R12: 00007f3d8d7c06e0
[  502.794807] R13: 0000000000000002 R14: 00007f3d8d7bb860 R15: 0000000000000002
[  502.794809]  </TASK>

As shown, lock->magic != lock. This is because
mutex_init(&pds_vfio->state_mutex) is called in the VFIO open path. So,
if a reset is initiated before the VFIO device is opened the mutex will
have never been initialized. Fix this by calling
mutex_init(&pds_vfio->state_mutex) in the VFIO init path.

Signed-off-by: Brett Creeley <brett.creeley@amd.com>
Reviewed-by: Shannon Nelson <shannon.nelson@amd.com>
---
 drivers/vfio/pci/pds/vfio_dev.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/vfio/pci/pds/vfio_dev.c b/drivers/vfio/pci/pds/vfio_dev.c
index 147a543a7c39..9db5f2c8f1ea 100644
--- a/drivers/vfio/pci/pds/vfio_dev.c
+++ b/drivers/vfio/pci/pds/vfio_dev.c
@@ -155,6 +155,7 @@ static int pds_vfio_init_device(struct vfio_device *vdev)
 
 	pds_vfio->vf_id = vf_id;
 
+	mutex_init(&pds_vfio->state_mutex);
 	spin_lock_init(&pds_vfio->reset_lock);
 
 	vdev->migration_flags = VFIO_MIGRATION_STOP_COPY | VFIO_MIGRATION_P2P;
@@ -181,7 +182,6 @@ static int pds_vfio_open_device(struct vfio_device *vdev)
 	if (err)
 		return err;
 
-	mutex_init(&pds_vfio->state_mutex);
 	pds_vfio->state = VFIO_DEVICE_STATE_RUNNING;
 	pds_vfio->deferred_reset_state = VFIO_DEVICE_STATE_RUNNING;
 
-- 
2.17.1