LinuxLists.cc - [PATCH v3] xen: avoid crash in disable_hotplug

2018-09-07 06:31:53

Subject: [PATCH v3] xen: avoid crash in disable_hotplug_cpu

The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:

BUG: unable to handle kernel NULL pointer dereference at 00000000000002d8
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP NOPTI
CPU: 7 PID: 65 Comm: xenwatch Not tainted 4.19.0-rc2-1.ga9462db-default #1 openSUSE Tumbleweed (unreleased)
Hardware name: Intel Corporation S5520UR/S5520UR, BIOS S5500.86B.01.00.0050.050620101605 05/06/2010
RIP: e030:device_offline+0x9/0xb0
Code: 77 24 00 e9 ce fe ff ff 48 8b 13 e9 68 ff ff ff 48 8b 13 e9 29 ff ff ff 48 8b 13 e9 ea fe ff ff 90 66 66 66 66 90 41 54 55 53 <f6> 87 d8 02 00 00 01 0f 85 88 00 00 00 48 c7 c2 20 09 60 81 31 f6
RSP: e02b:ffffc90040f27e80 EFLAGS: 00010203
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ffff8801f3800000 RSI: ffffc90040f27e70 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffffffff820e47b3 R09: 0000000000000000
R10: 0000000000007ff0 R11: 0000000000000000 R12: ffffffff822e6d30
R13: dead000000000200 R14: dead000000000100 R15: ffffffff8158b4e0
FS: 00007ffa595158c0(0000) GS:ffff8801f39c0000(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000002d8 CR3: 00000001d9602000 CR4: 0000000000002660
Call Trace:
handle_vcpu_hotplug_event+0xb5/0xc0
xenwatch_thread+0x80/0x140
? wait_woken+0x80/0x80
kthread+0x112/0x130
? kthread_create_worker_on_cpu+0x40/0x40
ret_from_fork+0x3a/0x50

This happens because handle_vcpu_hotplug_event is called twice. In the
first iteration cpu_present is still true, in the second iteration
cpu_present is false which causes get_cpu_device to return NULL.
In case of cpu#0, cpu_online is apparently always true.

Fix this crash by checking if the cpu can be hotplugged, which is false
for a cpu that was just removed.

Also check if the cpu was actually offlined by device_remove, otherwise
leave the cpu_present state as it is.

Signed-off-by: Olaf Hering <[email protected]>
---
drivers/xen/cpu_hotplug.c | 4 ++++
1 file changed, 4 insertions(+)

diff --git a/drivers/xen/cpu_hotplug.c b/drivers/xen/cpu_hotplug.c
index d4265c8ebb22..68f9f663da08 100644
--- a/drivers/xen/cpu_hotplug.c
+++ b/drivers/xen/cpu_hotplug.c
@@ -19,11 +19,15 @@ static void enable_hotplug_cpu(int cpu)

static void disable_hotplug_cpu(int cpu)
{
+ if (!cpu_is_hotpluggable(cpu))
+ return;
if (cpu_online(cpu)) {
lock_device_hotplug();
device_offline(get_cpu_device(cpu));
unlock_device_hotplug();
}
+ if (cpu_online(cpu))
+ return;
if (cpu_present(cpu))
xen_arch_unregister_cpu(cpu);

2018-09-07 09:26:56

by Olaf Hering

[permalink] [raw]

Subject: Re: [PATCH v3] xen: avoid crash in disable_hotplug_cpu

Am Fri, 7 Sep 2018 09:48:28 +0200
schrieb Juergen Gross <[email protected]>:

> On 07/09/18 08:30, Olaf Hering wrote:
> > + if (cpu_online(cpu))
> > + return;
> > if (cpu_present(cpu))
> > xen_arch_unregister_cpu(cpu);

> Could you merge the two if conditions?
> if (!cpu_online(cpu) && cpu_present(cpu))
> xen_arch_unregister_cpu(cpu);

Is that any different, beside being wrong, from what the patch actually does?
It would still clear the present bit later on.

Olaf

Attachments:

(No filename) (201.00 B)
Digitale Signatur von OpenPGP

2018-09-07 09:37:11

by Juergen Gross

[permalink] [raw]

Subject: Re: [PATCH v3] xen: avoid crash in disable_hotplug_cpu

On 07/09/18 08:30, Olaf Hering wrote:
> The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:
>
> BUG: unable to handle kernel NULL pointer dereference at 00000000000002d8
> PGD 0 P4D 0
> Oops: 0000 [#1] PREEMPT SMP NOPTI
> CPU: 7 PID: 65 Comm: xenwatch Not tainted 4.19.0-rc2-1.ga9462db-default #1 openSUSE Tumbleweed (unreleased)
> Hardware name: Intel Corporation S5520UR/S5520UR, BIOS S5500.86B.01.00.0050.050620101605 05/06/2010
> RIP: e030:device_offline+0x9/0xb0
> Code: 77 24 00 e9 ce fe ff ff 48 8b 13 e9 68 ff ff ff 48 8b 13 e9 29 ff ff ff 48 8b 13 e9 ea fe ff ff 90 66 66 66 66 90 41 54 55 53 <f6> 87 d8 02 00 00 01 0f 85 88 00 00 00 48 c7 c2 20 09 60 81 31 f6
> RSP: e02b:ffffc90040f27e80 EFLAGS: 00010203
> RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
> RDX: ffff8801f3800000 RSI: ffffc90040f27e70 RDI: 0000000000000000
> RBP: 0000000000000000 R08: ffffffff820e47b3 R09: 0000000000000000
> R10: 0000000000007ff0 R11: 0000000000000000 R12: ffffffff822e6d30
> R13: dead000000000200 R14: dead000000000100 R15: ffffffff8158b4e0
> FS: 00007ffa595158c0(0000) GS:ffff8801f39c0000(0000) knlGS:0000000000000000
> CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00000000000002d8 CR3: 00000001d9602000 CR4: 0000000000002660
> Call Trace:
> handle_vcpu_hotplug_event+0xb5/0xc0
> xenwatch_thread+0x80/0x140
> ? wait_woken+0x80/0x80
> kthread+0x112/0x130
> ? kthread_create_worker_on_cpu+0x40/0x40
> ret_from_fork+0x3a/0x50
>
> This happens because handle_vcpu_hotplug_event is called twice. In the
> first iteration cpu_present is still true, in the second iteration
> cpu_present is false which causes get_cpu_device to return NULL.
> In case of cpu#0, cpu_online is apparently always true.
>
> Fix this crash by checking if the cpu can be hotplugged, which is false
> for a cpu that was just removed.
>
> Also check if the cpu was actually offlined by device_remove, otherwise
> leave the cpu_present state as it is.
>
> Signed-off-by: Olaf Hering <[email protected]>
> ---
> drivers/xen/cpu_hotplug.c | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/drivers/xen/cpu_hotplug.c b/drivers/xen/cpu_hotplug.c
> index d4265c8ebb22..68f9f663da08 100644
> --- a/drivers/xen/cpu_hotplug.c
> +++ b/drivers/xen/cpu_hotplug.c
> @@ -19,11 +19,15 @@ static void enable_hotplug_cpu(int cpu)
>
> static void disable_hotplug_cpu(int cpu)
> {
> + if (!cpu_is_hotpluggable(cpu))
> + return;
> if (cpu_online(cpu)) {
> lock_device_hotplug();
> device_offline(get_cpu_device(cpu));
> unlock_device_hotplug();
> }
> + if (cpu_online(cpu))
> + return;
> if (cpu_present(cpu))
> xen_arch_unregister_cpu(cpu);

Could you merge the two if conditions?

if (!cpu_online(cpu) && cpu_present(cpu))
xen_arch_unregister_cpu(cpu);

And while not really important, as we are called in the xenstore watch
thread only, it might be a good idea to move the first cpu_online()
test into the lock, i.e.:

lock_device_hotplug();
if (cpu_online(cpu))
device_offline(get_cpu_device(cpu));
unlock_device_hotplug();

This will make the code robust against reentry.

Juergen

2018-09-07 10:10:21

by Juergen Gross

[permalink] [raw]

Subject: Re: [PATCH v3] xen: avoid crash in disable_hotplug_cpu

On 07/09/18 11:24, Olaf Hering wrote:
> Am Fri, 7 Sep 2018 09:48:28 +0200
> schrieb Juergen Gross <[email protected]>:
>
>> On 07/09/18 08:30, Olaf Hering wrote:
>>> + if (cpu_online(cpu))
>>> + return;
>>> if (cpu_present(cpu))
>>> xen_arch_unregister_cpu(cpu);
>
>> Could you merge the two if conditions?
>> if (!cpu_online(cpu) && cpu_present(cpu))
>> xen_arch_unregister_cpu(cpu);
>
> Is that any different, beside being wrong, from what the patch actually does?

No. Just a matter of style.

> It would still clear the present bit later on.

This should be part of the if clause, of course.

Juergen

2018-09-07 15:44:14

by Boris Ostrovsky

[permalink] [raw]

Subject: Re: [PATCH v3] xen: avoid crash in disable_hotplug_cpu

On 09/07/2018 02:30 AM, Olaf Hering wrote:
> The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:
>
> BUG: unable to handle kernel NULL pointer dereference at 00000000000002d8
> PGD 0 P4D 0
> Oops: 0000 [#1] PREEMPT SMP NOPTI
> CPU: 7 PID: 65 Comm: xenwatch Not tainted 4.19.0-rc2-1.ga9462db-default #1 openSUSE Tumbleweed (unreleased)
> Hardware name: Intel Corporation S5520UR/S5520UR, BIOS S5500.86B.01.00.0050.050620101605 05/06/2010
> RIP: e030:device_offline+0x9/0xb0
> Code: 77 24 00 e9 ce fe ff ff 48 8b 13 e9 68 ff ff ff 48 8b 13 e9 29 ff ff ff 48 8b 13 e9 ea fe ff ff 90 66 66 66 66 90 41 54 55 53 <f6> 87 d8 02 00 00 01 0f 85 88 00 00 00 48 c7 c2 20 09 60 81 31 f6
> RSP: e02b:ffffc90040f27e80 EFLAGS: 00010203
> RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
> RDX: ffff8801f3800000 RSI: ffffc90040f27e70 RDI: 0000000000000000
> RBP: 0000000000000000 R08: ffffffff820e47b3 R09: 0000000000000000
> R10: 0000000000007ff0 R11: 0000000000000000 R12: ffffffff822e6d30
> R13: dead000000000200 R14: dead000000000100 R15: ffffffff8158b4e0
> FS: 00007ffa595158c0(0000) GS:ffff8801f39c0000(0000) knlGS:0000000000000000
> CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00000000000002d8 CR3: 00000001d9602000 CR4: 0000000000002660
> Call Trace:
> handle_vcpu_hotplug_event+0xb5/0xc0
> xenwatch_thread+0x80/0x140
> ? wait_woken+0x80/0x80
> kthread+0x112/0x130
> ? kthread_create_worker_on_cpu+0x40/0x40
> ret_from_fork+0x3a/0x50
>
> This happens because handle_vcpu_hotplug_event is called twice. In the
> first iteration cpu_present is still true, in the second iteration
> cpu_present is false which causes get_cpu_device to return NULL.
> In case of cpu#0, cpu_online is apparently always true.
>
> Fix this crash by checking if the cpu can be hotplugged, which is false
> for a cpu that was just removed.
>
> Also check if the cpu was actually offlined by device_remove, otherwise
> leave the cpu_present state as it is.
>
> Signed-off-by: Olaf Hering <[email protected]>
> ---
> drivers/xen/cpu_hotplug.c | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/drivers/xen/cpu_hotplug.c b/drivers/xen/cpu_hotplug.c
> index d4265c8ebb22..68f9f663da08 100644
> --- a/drivers/xen/cpu_hotplug.c
> +++ b/drivers/xen/cpu_hotplug.c
> @@ -19,11 +19,15 @@ static void enable_hotplug_cpu(int cpu)
>
> static void disable_hotplug_cpu(int cpu)
> {
> + if (!cpu_is_hotpluggable(cpu))
> + return;
> if (cpu_online(cpu)) {
> lock_device_hotplug();
> device_offline(get_cpu_device(cpu));
> unlock_device_hotplug();
> }
> + if (cpu_online(cpu))
> + return;
> if (cpu_present(cpu))
> xen_arch_unregister_cpu(cpu);
>

You don't get a warning in xen_init_lock_cpu() for CPU0?

Also, as a side question --- what is the purpose of 'xl vcpu-set <domid> 0'?

-boris

2018-09-07 15:45:59

by Olaf Hering

[permalink] [raw]

Subject: Re: [PATCH v3] xen: avoid crash in disable_hotplug_cpu

Am Fri, 7 Sep 2018 09:33:20 -0400
schrieb Boris Ostrovsky <[email protected]>:

> what is the purpose of 'xl vcpu-set <domid> 0'?

Likely just a script that went wrong. But that command should not break the dom0.

Olaf

Attachments:

(No filename) (201.00 B)
Digitale Signatur von OpenPGP

2018-09-11 11:27:01

by Juergen Gross

[permalink] [raw]

Subject: Re: [PATCH v3] xen: avoid crash in disable_hotplug_cpu

2018-09-11 14:28:20

by Boris Ostrovsky

[permalink] [raw]

Subject: Re: [PATCH v3] xen: avoid crash in disable_hotplug_cpu

On 9/11/18 7:25 AM, Juergen Gross wrote:
>
> Reviewed-by: Juergen Gross <[email protected]>

So is it v3 or v4, you gave R-b for both. (I slightly prefer v3, but
either is fine).

-boris

2018-09-11 14:35:03

by Juergen Gross

[permalink] [raw]

Subject: Re: [PATCH v3] xen: avoid crash in disable_hotplug_cpu

On 11/09/18 16:29, Boris Ostrovsky wrote:
> On 9/11/18 7:25 AM, Juergen Gross wrote:
>>
>> Reviewed-by: Juergen Gross <[email protected]>
>
> So is it v3 or v4, you gave R-b for both. (I slightly prefer v3, but
> either is fine).

V4 is better as it does the online test only after acquiring the
hotplug lock.

Juergen