2018-09-07 15:51:52

by Olaf Hering

[permalink] [raw]
Subject: [PATCH v4] xen: avoid crash in disable_hotplug_cpu

The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:

BUG: unable to handle kernel NULL pointer dereference at 00000000000002d8
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP NOPTI
CPU: 7 PID: 65 Comm: xenwatch Not tainted 4.19.0-rc2-1.ga9462db-default #1 openSUSE Tumbleweed (unreleased)
Hardware name: Intel Corporation S5520UR/S5520UR, BIOS S5500.86B.01.00.0050.050620101605 05/06/2010
RIP: e030:device_offline+0x9/0xb0
Code: 77 24 00 e9 ce fe ff ff 48 8b 13 e9 68 ff ff ff 48 8b 13 e9 29 ff ff ff 48 8b 13 e9 ea fe ff ff 90 66 66 66 66 90 41 54 55 53 <f6> 87 d8 02 00 00 01 0f 85 88 00 00 00 48 c7 c2 20 09 60 81 31 f6
RSP: e02b:ffffc90040f27e80 EFLAGS: 00010203
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ffff8801f3800000 RSI: ffffc90040f27e70 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffffffff820e47b3 R09: 0000000000000000
R10: 0000000000007ff0 R11: 0000000000000000 R12: ffffffff822e6d30
R13: dead000000000200 R14: dead000000000100 R15: ffffffff8158b4e0
FS: 00007ffa595158c0(0000) GS:ffff8801f39c0000(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000002d8 CR3: 00000001d9602000 CR4: 0000000000002660
Call Trace:
handle_vcpu_hotplug_event+0xb5/0xc0
xenwatch_thread+0x80/0x140
? wait_woken+0x80/0x80
kthread+0x112/0x130
? kthread_create_worker_on_cpu+0x40/0x40
ret_from_fork+0x3a/0x50

This happens because handle_vcpu_hotplug_event is called twice. In the
first iteration cpu_present is still true, in the second iteration
cpu_present is false which causes get_cpu_device to return NULL.
In case of cpu#0, cpu_online is apparently always true.

Fix this crash by checking if the cpu can be hotplugged, which is false
for a cpu that was just removed.

Also check if the cpu was actually offlined by device_remove, otherwise
leave the cpu_present state as it is.

Rearrange to code to do all work with device_hotplug_lock held.

Signed-off-by: Olaf Hering <[email protected]>
---
drivers/xen/cpu_hotplug.c | 15 ++++++++-------
1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/drivers/xen/cpu_hotplug.c b/drivers/xen/cpu_hotplug.c
index d4265c8ebb22..b1357aa4bc55 100644
--- a/drivers/xen/cpu_hotplug.c
+++ b/drivers/xen/cpu_hotplug.c
@@ -19,15 +19,16 @@ static void enable_hotplug_cpu(int cpu)

static void disable_hotplug_cpu(int cpu)
{
- if (cpu_online(cpu)) {
- lock_device_hotplug();
+ if (!cpu_is_hotpluggable(cpu))
+ return;
+ lock_device_hotplug();
+ if (cpu_online(cpu))
device_offline(get_cpu_device(cpu));
- unlock_device_hotplug();
- }
- if (cpu_present(cpu))
+ if (!cpu_online(cpu) && cpu_present(cpu)) {
xen_arch_unregister_cpu(cpu);
-
- set_cpu_present(cpu, false);
+ set_cpu_present(cpu, false);
+ }
+ unlock_device_hotplug();
}

static int vcpu_online(unsigned int cpu)


2018-09-07 15:50:51

by Jürgen Groß

[permalink] [raw]
Subject: Re: [PATCH v4] xen: avoid crash in disable_hotplug_cpu

On 07/09/18 16:31, Olaf Hering wrote:
> The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:
>
> BUG: unable to handle kernel NULL pointer dereference at 00000000000002d8
> PGD 0 P4D 0
> Oops: 0000 [#1] PREEMPT SMP NOPTI
> CPU: 7 PID: 65 Comm: xenwatch Not tainted 4.19.0-rc2-1.ga9462db-default #1 openSUSE Tumbleweed (unreleased)
> Hardware name: Intel Corporation S5520UR/S5520UR, BIOS S5500.86B.01.00.0050.050620101605 05/06/2010
> RIP: e030:device_offline+0x9/0xb0
> Code: 77 24 00 e9 ce fe ff ff 48 8b 13 e9 68 ff ff ff 48 8b 13 e9 29 ff ff ff 48 8b 13 e9 ea fe ff ff 90 66 66 66 66 90 41 54 55 53 <f6> 87 d8 02 00 00 01 0f 85 88 00 00 00 48 c7 c2 20 09 60 81 31 f6
> RSP: e02b:ffffc90040f27e80 EFLAGS: 00010203
> RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
> RDX: ffff8801f3800000 RSI: ffffc90040f27e70 RDI: 0000000000000000
> RBP: 0000000000000000 R08: ffffffff820e47b3 R09: 0000000000000000
> R10: 0000000000007ff0 R11: 0000000000000000 R12: ffffffff822e6d30
> R13: dead000000000200 R14: dead000000000100 R15: ffffffff8158b4e0
> FS: 00007ffa595158c0(0000) GS:ffff8801f39c0000(0000) knlGS:0000000000000000
> CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00000000000002d8 CR3: 00000001d9602000 CR4: 0000000000002660
> Call Trace:
> handle_vcpu_hotplug_event+0xb5/0xc0
> xenwatch_thread+0x80/0x140
> ? wait_woken+0x80/0x80
> kthread+0x112/0x130
> ? kthread_create_worker_on_cpu+0x40/0x40
> ret_from_fork+0x3a/0x50
>
> This happens because handle_vcpu_hotplug_event is called twice. In the
> first iteration cpu_present is still true, in the second iteration
> cpu_present is false which causes get_cpu_device to return NULL.
> In case of cpu#0, cpu_online is apparently always true.
>
> Fix this crash by checking if the cpu can be hotplugged, which is false
> for a cpu that was just removed.
>
> Also check if the cpu was actually offlined by device_remove, otherwise
> leave the cpu_present state as it is.
>
> Rearrange to code to do all work with device_hotplug_lock held.
>
> Signed-off-by: Olaf Hering <[email protected]>

Reviewed-by: Juergen Gross <[email protected]>


Juergen

2018-09-07 16:57:15

by Boris Ostrovsky

[permalink] [raw]
Subject: Re: [PATCH v4] xen: avoid crash in disable_hotplug_cpu

On 09/07/2018 10:31 AM, Olaf Hering wrote:
> The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:
>
> BUG: unable to handle kernel NULL pointer dereference at 00000000000002d8
> PGD 0 P4D 0
> Oops: 0000 [#1] PREEMPT SMP NOPTI
> CPU: 7 PID: 65 Comm: xenwatch Not tainted 4.19.0-rc2-1.ga9462db-default #1 openSUSE Tumbleweed (unreleased)
> Hardware name: Intel Corporation S5520UR/S5520UR, BIOS S5500.86B.01.00.0050.050620101605 05/06/2010
> RIP: e030:device_offline+0x9/0xb0
> Code: 77 24 00 e9 ce fe ff ff 48 8b 13 e9 68 ff ff ff 48 8b 13 e9 29 ff ff ff 48 8b 13 e9 ea fe ff ff 90 66 66 66 66 90 41 54 55 53 <f6> 87 d8 02 00 00 01 0f 85 88 00 00 00 48 c7 c2 20 09 60 81 31 f6
> RSP: e02b:ffffc90040f27e80 EFLAGS: 00010203
> RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
> RDX: ffff8801f3800000 RSI: ffffc90040f27e70 RDI: 0000000000000000
> RBP: 0000000000000000 R08: ffffffff820e47b3 R09: 0000000000000000
> R10: 0000000000007ff0 R11: 0000000000000000 R12: ffffffff822e6d30
> R13: dead000000000200 R14: dead000000000100 R15: ffffffff8158b4e0
> FS: 00007ffa595158c0(0000) GS:ffff8801f39c0000(0000) knlGS:0000000000000000
> CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00000000000002d8 CR3: 00000001d9602000 CR4: 0000000000002660
> Call Trace:
> handle_vcpu_hotplug_event+0xb5/0xc0
> xenwatch_thread+0x80/0x140
> ? wait_woken+0x80/0x80
> kthread+0x112/0x130
> ? kthread_create_worker_on_cpu+0x40/0x40
> ret_from_fork+0x3a/0x50
>
> This happens because handle_vcpu_hotplug_event is called twice. In the
> first iteration cpu_present is still true, in the second iteration
> cpu_present is false which causes get_cpu_device to return NULL.
> In case of cpu#0, cpu_online is apparently always true.
>
> Fix this crash by checking if the cpu can be hotplugged, which is false
> for a cpu that was just removed.
>
> Also check if the cpu was actually offlined by device_remove, otherwise
> leave the cpu_present state as it is.
>
> Rearrange to code to do all work with device_hotplug_lock held.
>
> Signed-off-by: Olaf Hering <[email protected]>
> ---
> drivers/xen/cpu_hotplug.c | 15 ++++++++-------
> 1 file changed, 8 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/xen/cpu_hotplug.c b/drivers/xen/cpu_hotplug.c
> index d4265c8ebb22..b1357aa4bc55 100644
> --- a/drivers/xen/cpu_hotplug.c
> +++ b/drivers/xen/cpu_hotplug.c
> @@ -19,15 +19,16 @@ static void enable_hotplug_cpu(int cpu)
>
> static void disable_hotplug_cpu(int cpu)
> {
> - if (cpu_online(cpu)) {
> - lock_device_hotplug();
> + if (!cpu_is_hotpluggable(cpu))
> + return;
> + lock_device_hotplug();
> + if (cpu_online(cpu))
> device_offline(get_cpu_device(cpu));
> - unlock_device_hotplug();
> - }
> - if (cpu_present(cpu))
> + if (!cpu_online(cpu) && cpu_present(cpu)) {
> xen_arch_unregister_cpu(cpu);
> -
> - set_cpu_present(cpu, false);
> + set_cpu_present(cpu, false);
> + }
> + unlock_device_hotplug();
> }
>
> static int vcpu_online(unsigned int cpu)


I was hoping you'd respond to my question about warning.

root@haswell> xl vcpu-set 3 0


and in the guest

[root@vm-0238 ~]# [   32.866955] ------------[ cut here ]------------
[   32.866963] spinlock on CPU0 exists on IRQ1!
[   32.866984] WARNING: CPU: 0 PID: 14 at arch/x86/xen/spinlock.c:90
xen_init_lock_cpu+0xbf/0xd0
[   32.866990] Modules linked in:
[   32.866995] CPU: 0 PID: 14 Comm: cpuhp/0 Not tainted 4.19.0-rc2 #31
[   32.867001] RIP: e030:xen_init_lock_cpu+0xbf/0xd0
[   32.867005] Code: 4a 8b 0c e5 00 c7 14 82 48 c7 c2 90 4f 01 00 4c 89
2c 11 e9 85 00 00 00 8b 14 02 44 89 e6 48 c7 c7 a0 0f 08 82 e8 ab e3 05
00 <0f> 0b e9 7a ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 80 3d 59 02 20
[   32.867015] RSP: e02b:ffffc900401ffe40 EFLAGS: 00010286
[   32.867019] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
0000000000000006
[   32.867024] RDX: 0000000000000007 RSI: 0000000000000001 RDI:
ffff88003d8168b0
[   32.867039] RBP: 0000000000014f98 R08: ffffffff81eb04a0 R09:
0000000000007f9b
[   32.867045] R10: 0000000000000065 R11: ffffffff82a9b7cd R12:
0000000000000000
[   32.867050] R13: ffffffff8101a820 R14: ffff88003d401280 R15:
ffffffff810aec10
[   32.867061] FS:  0000000000000000(0000) GS:ffff88003d800000(0000)
knlGS:0000000000000000
[   32.867066] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[   32.867081] CR2: 00005569b64e72b8 CR3: 000000002e902000 CR4:
0000000000042660
[   32.867089] Call Trace:
[   32.867096]  ? cstate_cleanup+0x47/0x47
[   32.867101]  xen_cpu_up_online+0xa/0x10
[   32.867107]  cpuhp_invoke_callback+0x8d/0x500
[   32.867113]  ? sort_range+0x20/0x20
[   32.867117]  cpuhp_thread_fun+0xb0/0x110
[   32.867121]  smpboot_thread_fn+0xc5/0x160
[   32.867126]  kthread+0x112/0x130
[   32.867131]  ? kthread_bind+0x30/0x30
[   32.867136]  ret_from_fork+0x35/0x40
[   32.867141] ---[ end trace 15d4d7112a1b1cea ]---
[   32.867148] genirq: Flags mismatch irq 1. 0002cc00 (spinlock0) vs.
0002cc00 (spinlock0)
[   32.867154] CPU: 0 PID: 14 Comm: cpuhp/0 Tainted: G        W        
4.19.0-rc2 #31
[   32.867160] Call Trace:
[   32.867165]  dump_stack+0x5c/0x80
[   32.867171]  __setup_irq.cold.51+0x4e/0x9e
[   32.867177]  request_threaded_irq+0xf5/0x160
[   32.867182]  ? xen_qlock_wait+0x40/0x40
[   32.867188]  bind_ipi_to_irqhandler+0xae/0x1d0
[   32.867194]  ? sort_range+0x20/0x20
[   32.867198]  xen_init_lock_cpu+0x74/0xd0
[   32.867202]  ? cstate_cleanup+0x47/0x47
[   32.867206]  xen_cpu_up_online+0xa/0x10
[   32.867210]  cpuhp_invoke_callback+0x8d/0x500
[   32.867215]  ? sort_range+0x20/0x20
[   32.867219]  cpuhp_thread_fun+0xb0/0x110
[   32.867223]  smpboot_thread_fn+0xc5/0x160
[   32.867227]  kthread+0x112/0x130
[   32.867231]  ? kthread_bind+0x30/0x30
[   32.867235]  ret_from_fork+0x35/0x40
[   32.867249] cpu 0 spinlock event irq -16
[   32.880877] IRQ 16: no longer affine to CPU1
[   32.880879] IRQ 17: no longer affine to CPU1
[   32.880881] IRQ 18: no longer affine to CPU1
[   32.880882] IRQ 19: no longer affine to CPU1
[   32.880884] IRQ 20: no longer affine to CPU1
[   32.880885] IRQ 21: no longer affine to CPU1
[   32.880886] IRQ 22: no longer affine to CPU1
[   32.880888] IRQ 23: no longer affine to CPU1
[   32.880889] IRQ 24: no longer affine to CPU1
[   32.882202] smpboot: CPU 1 is now offline


2018-09-10 16:00:26

by Olaf Hering

[permalink] [raw]
Subject: Re: [PATCH v4] xen: avoid crash in disable_hotplug_cpu

Am Fri, 7 Sep 2018 12:56:37 -0400
schrieb Boris Ostrovsky <[email protected]>:

> I was hoping you'd respond to my question about warning.
>
> root@haswell> xl vcpu-set 3 0
> and in the guest
>
> [root@vm-0238 ~]# [   32.866955] ------------[ cut here ]------------
> [   32.866963] spinlock on CPU0 exists on IRQ1!
> [   32.866984] WARNING: CPU: 0 PID: 14 at arch/x86/xen/spinlock.c:90
> xen_init_lock_cpu+0xbf/0xd0

This happens to work for me, on X5550. Please send your .config.

Olaf


Attachments:
(No filename) (201.00 B)
Digitale Signatur von OpenPGP

2018-09-10 17:55:40

by Boris Ostrovsky

[permalink] [raw]
Subject: Re: [PATCH v4] xen: avoid crash in disable_hotplug_cpu

On 09/10/2018 11:58 AM, Olaf Hering wrote:
> Am Fri, 7 Sep 2018 12:56:37 -0400
> schrieb Boris Ostrovsky <[email protected]>:
>
>> I was hoping you'd respond to my question about warning.
>>
>> root@haswell> xl vcpu-set 3 0
>> and in the guest
>>
>> [root@vm-0238 ~]# [   32.866955] ------------[ cut here ]------------
>> [   32.866963] spinlock on CPU0 exists on IRQ1!
>> [   32.866984] WARNING: CPU: 0 PID: 14 at arch/x86/xen/spinlock.c:90
>> xen_init_lock_cpu+0xbf/0xd0
> This happens to work for me, on X5550. Please send your .config.
>


Attached. I suspect it may be CONFIG_PARAVIRT_SPINLOCKS.

-boris


Attachments:
config (116.70 kB)
signature.asc (849.00 B)
OpenPGP digital signature
Download all attachments

2018-09-11 07:53:35

by Olaf Hering

[permalink] [raw]
Subject: Re: [PATCH v4] xen: avoid crash in disable_hotplug_cpu

Am Fri, 7 Sep 2018 12:56:37 -0400
schrieb Boris Ostrovsky <[email protected]>:

> I was hoping you'd respond to my question about warning.

It looks like CONFIG_BOOTPARAM_HOTPLUG_CPU0=y is the reason for the warning.
As Jürgen suggested in another mail, Xen should probably disable hotplugging
for cpu#0 in the generic setup code. Then cpu_is_hotpluggable(cpu) would
do the right thing.

Do you want me to merge that extra change into this patch?

Olaf


Attachments:
(No filename) (201.00 B)
Digitale Signatur von OpenPGP

2018-09-11 07:55:35

by Jürgen Groß

[permalink] [raw]
Subject: Re: [PATCH v4] xen: avoid crash in disable_hotplug_cpu

On 11/09/18 09:52, Olaf Hering wrote:
> Am Fri, 7 Sep 2018 12:56:37 -0400
> schrieb Boris Ostrovsky <[email protected]>:
>
>> I was hoping you'd respond to my question about warning.
>
> It looks like CONFIG_BOOTPARAM_HOTPLUG_CPU0=y is the reason for the warning.
> As Jürgen suggested in another mail, Xen should probably disable hotplugging
> for cpu#0 in the generic setup code. Then cpu_is_hotpluggable(cpu) would
> do the right thing.
>
> Do you want me to merge that extra change into this patch?

I think an extra patch would be better, as it is a different issue.


Juergen

2018-09-11 10:48:45

by Olaf Hering

[permalink] [raw]
Subject: Re: [PATCH v4] xen: avoid crash in disable_hotplug_cpu

Am Tue, 11 Sep 2018 09:52:58 +0200
schrieb Olaf Hering <[email protected]>:

> As Jürgen suggested in another mail, Xen should probably disable hotplugging
> for cpu#0 in the generic setup code. Then cpu_is_hotpluggable(cpu) would
> do the right thing.

The relevant code is all private to arch/x86/kernel/topology.c.
There is no way to set .hotpluggable prior to register_cpu.
I guess you need to work that out with the x86 maintainers.

Olaf


Attachments:
(No filename) (201.00 B)
Digitale Signatur von OpenPGP

2018-09-11 10:51:45

by Jürgen Groß

[permalink] [raw]
Subject: Re: [PATCH v4] xen: avoid crash in disable_hotplug_cpu

On 11/09/18 12:48, Olaf Hering wrote:
> Am Tue, 11 Sep 2018 09:52:58 +0200
> schrieb Olaf Hering <[email protected]>:
>
>> As Jürgen suggested in another mail, Xen should probably disable hotplugging
>> for cpu#0 in the generic setup code. Then cpu_is_hotpluggable(cpu) would
>> do the right thing.
>
> The relevant code is all private to arch/x86/kernel/topology.c.
> There is no way to set .hotpluggable prior to register_cpu.
> I guess you need to work that out with the x86 maintainers.

Okay, I'll do that.


Juergen

2018-09-11 19:38:42

by Boris Ostrovsky

[permalink] [raw]
Subject: Re: [PATCH v4] xen: avoid crash in disable_hotplug_cpu

On 9/7/18 10:31 AM, Olaf Hering wrote:
> The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:
>
> BUG: unable to handle kernel NULL pointer dereference at 00000000000002d8
> PGD 0 P4D 0
> Oops: 0000 [#1] PREEMPT SMP NOPTI
> CPU: 7 PID: 65 Comm: xenwatch Not tainted 4.19.0-rc2-1.ga9462db-default #1 openSUSE Tumbleweed (unreleased)
> Hardware name: Intel Corporation S5520UR/S5520UR, BIOS S5500.86B.01.00.0050.050620101605 05/06/2010
> RIP: e030:device_offline+0x9/0xb0
> Code: 77 24 00 e9 ce fe ff ff 48 8b 13 e9 68 ff ff ff 48 8b 13 e9 29 ff ff ff 48 8b 13 e9 ea fe ff ff 90 66 66 66 66 90 41 54 55 53 <f6> 87 d8 02 00 00 01 0f 85 88 00 00 00 48 c7 c2 20 09 60 81 31 f6
> RSP: e02b:ffffc90040f27e80 EFLAGS: 00010203
> RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
> RDX: ffff8801f3800000 RSI: ffffc90040f27e70 RDI: 0000000000000000
> RBP: 0000000000000000 R08: ffffffff820e47b3 R09: 0000000000000000
> R10: 0000000000007ff0 R11: 0000000000000000 R12: ffffffff822e6d30
> R13: dead000000000200 R14: dead000000000100 R15: ffffffff8158b4e0
> FS: 00007ffa595158c0(0000) GS:ffff8801f39c0000(0000) knlGS:0000000000000000
> CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00000000000002d8 CR3: 00000001d9602000 CR4: 0000000000002660
> Call Trace:
> handle_vcpu_hotplug_event+0xb5/0xc0
> xenwatch_thread+0x80/0x140
> ? wait_woken+0x80/0x80
> kthread+0x112/0x130
> ? kthread_create_worker_on_cpu+0x40/0x40
> ret_from_fork+0x3a/0x50
>
> This happens because handle_vcpu_hotplug_event is called twice. In the
> first iteration cpu_present is still true, in the second iteration
> cpu_present is false which causes get_cpu_device to return NULL.
> In case of cpu#0, cpu_online is apparently always true.
>
> Fix this crash by checking if the cpu can be hotplugged, which is false
> for a cpu that was just removed.
>
> Also check if the cpu was actually offlined by device_remove, otherwise
> leave the cpu_present state as it is.
>
> Rearrange to code to do all work with device_hotplug_lock held.
>
> Signed-off-by: Olaf Hering <[email protected]>


Applied to for-linus-4.19b

-boris