2020-07-27 21:23:38

by Abhishek Pandit-Subedi

[permalink] [raw]
Subject: [PATCH] Bluetooth: Fix suspend notifier race

Unregister from suspend notifications and cancel suspend preparations
before running hci_dev_do_close. Otherwise, the suspend notifier may
race with unregister and cause cmd_timeout even after hdev has been
freed.

Signed-off-by: Abhishek Pandit-Subedi <[email protected]>
Reviewed-by: Miao-chen Chou <[email protected]>
---
Hi Marcel,

This fixes a race between hci_unregister_dev and the suspend notifier.
Without these changes, we encountered the following kernel panic when
a USB disconnect (with btusb) occurred on resume:

[ 832.578518] Bluetooth: hci_core.c:hci_cmd_timeout() hci0: command 0x0c05 tx timeout
[ 832.586200] BUG: kernel NULL pointer dereference, address: 0000000000000000
[ 832.586203] #PF: supervisor read access in kernel mode
[ 832.586205] #PF: error_code(0x0000) - not-present page
[ 832.586206] PGD 0 P4D 0
[ 832.586210] PM: suspend exit
[ 832.608870] Oops: 0000 [#1] PREEMPT SMP NOPTI
[ 832.613232] CPU: 3 PID: 10755 Comm: kworker/3:7 Not tainted 5.4.44-04894-g1e9dbb96a161 #1
[ 832.630036] Workqueue: events hci_cmd_timeout [bluetooth]
[ 832.630046] RIP: 0010:__queue_work+0xf0/0x374
[ 832.630051] RSP: 0018:ffff9b5285f1fdf8 EFLAGS: 00010046
[ 832.674033] RAX: ffff8a97681bac00 RBX: 0000000000000000 RCX: ffff8a976a000600
[ 832.681162] RDX: 0000000000000000 RSI: 0000000000000009 RDI: ffff8a976a000748
[ 832.688289] RBP: ffff9b5285f1fe38 R08: 0000000000000000 R09: ffff8a97681bac00
[ 832.695418] R10: 0000000000000002 R11: ffff8a976a0006d8 R12: ffff8a9745107600
[ 832.698045] usb 1-6: new full-speed USB device number 119 using xhci_hcd
[ 832.702547] R13: ffff8a9673658850 R14: 0000000000000040 R15: 000000000000001e
[ 832.702549] FS: 0000000000000000(0000) GS:ffff8a976af80000(0000) knlGS:0000000000000000
[ 832.702550] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 832.702550] CR2: 0000000000000000 CR3: 000000010415a000 CR4: 00000000003406e0
[ 832.702551] Call Trace:
[ 832.702558] queue_work_on+0x3f/0x68
[ 832.702562] process_one_work+0x1db/0x396
[ 832.747397] worker_thread+0x216/0x375
[ 832.751147] kthread+0x138/0x140
[ 832.754377] ? pr_cont_work+0x58/0x58
[ 832.758037] ? kthread_blkcg+0x2e/0x2e
[ 832.761787] ret_from_fork+0x22/0x40
[ 832.846191] ---[ end trace fa93f466da517212 ]---

The suspend notifier handler seemed to be scheduling commands even after
it was cleaned up and this was resulting in a panic in cmd_timeout (when
it tries to requeue the cmd_timer).

This was tested on 5.4 kernel with a suspend+resume stress test for 500+
iterations. I also confirmed that after a usb disconnect, the suspend
notifier times out before the USB device is probed again (fixing the
original race between the usb_disconnect + probe and the notifier).

Thanks
Abhishek


net/bluetooth/hci_core.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/net/bluetooth/hci_core.c b/net/bluetooth/hci_core.c
index 6509f785dd1481..97221d1fa883d1 100644
--- a/net/bluetooth/hci_core.c
+++ b/net/bluetooth/hci_core.c
@@ -3765,9 +3765,10 @@ void hci_unregister_dev(struct hci_dev *hdev)

cancel_work_sync(&hdev->power_on);

- hci_dev_do_close(hdev);
-
unregister_pm_notifier(&hdev->suspend_notifier);
+ cancel_work_sync(&hdev->suspend_prepare);
+
+ hci_dev_do_close(hdev);

if (!test_bit(HCI_INIT, &hdev->flags) &&
!hci_dev_test_flag(hdev, HCI_SETUP) &&
--
2.28.0.rc0.142.g3c755180ce-goog


2020-07-28 07:06:43

by Marcel Holtmann

[permalink] [raw]
Subject: Re: [PATCH] Bluetooth: Fix suspend notifier race

Hi Abhishek,

> Unregister from suspend notifications and cancel suspend preparations
> before running hci_dev_do_close. Otherwise, the suspend notifier may
> race with unregister and cause cmd_timeout even after hdev has been
> freed.
>
> Signed-off-by: Abhishek Pandit-Subedi <[email protected]>
> Reviewed-by: Miao-chen Chou <[email protected]>
> ---
> Hi Marcel,
>
> This fixes a race between hci_unregister_dev and the suspend notifier.
> Without these changes, we encountered the following kernel panic when
> a USB disconnect (with btusb) occurred on resume:
>
> [ 832.578518] Bluetooth: hci_core.c:hci_cmd_timeout() hci0: command 0x0c05 tx timeout
> [ 832.586200] BUG: kernel NULL pointer dereference, address: 0000000000000000
> [ 832.586203] #PF: supervisor read access in kernel mode
> [ 832.586205] #PF: error_code(0x0000) - not-present page
> [ 832.586206] PGD 0 P4D 0
> [ 832.586210] PM: suspend exit
> [ 832.608870] Oops: 0000 [#1] PREEMPT SMP NOPTI
> [ 832.613232] CPU: 3 PID: 10755 Comm: kworker/3:7 Not tainted 5.4.44-04894-g1e9dbb96a161 #1
> [ 832.630036] Workqueue: events hci_cmd_timeout [bluetooth]
> [ 832.630046] RIP: 0010:__queue_work+0xf0/0x374
> [ 832.630051] RSP: 0018:ffff9b5285f1fdf8 EFLAGS: 00010046
> [ 832.674033] RAX: ffff8a97681bac00 RBX: 0000000000000000 RCX: ffff8a976a000600
> [ 832.681162] RDX: 0000000000000000 RSI: 0000000000000009 RDI: ffff8a976a000748
> [ 832.688289] RBP: ffff9b5285f1fe38 R08: 0000000000000000 R09: ffff8a97681bac00
> [ 832.695418] R10: 0000000000000002 R11: ffff8a976a0006d8 R12: ffff8a9745107600
> [ 832.698045] usb 1-6: new full-speed USB device number 119 using xhci_hcd
> [ 832.702547] R13: ffff8a9673658850 R14: 0000000000000040 R15: 000000000000001e
> [ 832.702549] FS: 0000000000000000(0000) GS:ffff8a976af80000(0000) knlGS:0000000000000000
> [ 832.702550] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 832.702550] CR2: 0000000000000000 CR3: 000000010415a000 CR4: 00000000003406e0
> [ 832.702551] Call Trace:
> [ 832.702558] queue_work_on+0x3f/0x68
> [ 832.702562] process_one_work+0x1db/0x396
> [ 832.747397] worker_thread+0x216/0x375
> [ 832.751147] kthread+0x138/0x140
> [ 832.754377] ? pr_cont_work+0x58/0x58
> [ 832.758037] ? kthread_blkcg+0x2e/0x2e
> [ 832.761787] ret_from_fork+0x22/0x40
> [ 832.846191] ---[ end trace fa93f466da517212 ]---
>
> The suspend notifier handler seemed to be scheduling commands even after
> it was cleaned up and this was resulting in a panic in cmd_timeout (when
> it tries to requeue the cmd_timer).
>
> This was tested on 5.4 kernel with a suspend+resume stress test for 500+
> iterations. I also confirmed that after a usb disconnect, the suspend
> notifier times out before the USB device is probed again (fixing the
> original race between the usb_disconnect + probe and the notifier).

Can you please structure the commit message so that the oops is included. It is valuable information Everything after --- is a personal note to the maintainer. And we might want to include a Fixes tag as well.

Regards

Marcel