2020-04-21 16:37:51

by Daniel Jordan

[permalink] [raw]
Subject: [PATCH] padata: add separate cpuhp node for CPUHP_PADATA_DEAD

Removing the pcrypt module triggers this:

general protection fault, probably for non-canonical
address 0xdead000000000122
CPU: 5 PID: 264 Comm: modprobe Not tainted 5.6.0+ #2
Hardware name: QEMU Standard PC
RIP: 0010:__cpuhp_state_remove_instance+0xcc/0x120
Call Trace:
padata_sysfs_release+0x74/0xce
kobject_put+0x81/0xd0
padata_free+0x12/0x20
pcrypt_exit+0x43/0x8ee [pcrypt]

padata instances wrongly use the same hlist node for the online and dead
states, so __padata_free()'s second cpuhp remove call chokes on the node
that the first poisoned.

cpuhp multi-instance callbacks only walk forward in cpuhp_step->list and
the same node is linked in both the online and dead lists, so the list
corruption that results from padata_alloc() adding the node to a second
list without removing it from the first doesn't cause problems as long
as no instances are freed.

Avoid the issue by giving each state its own node.

Fixes: 894c9ef9780c ("padata: validate cpumask without removed CPU during offline")
Signed-off-by: Daniel Jordan <[email protected]>
Cc: Herbert Xu <[email protected]>
Cc: Steffen Klassert <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected] # v5.4+
---
include/linux/padata.h | 6 ++++--
kernel/padata.c | 14 ++++++++------
2 files changed, 12 insertions(+), 8 deletions(-)

diff --git a/include/linux/padata.h b/include/linux/padata.h
index a0d8b41850b2..693cae9bfe66 100644
--- a/include/linux/padata.h
+++ b/include/linux/padata.h
@@ -139,7 +139,8 @@ struct padata_shell {
/**
* struct padata_instance - The overall control structure.
*
- * @node: Used by CPU hotplug.
+ * @cpu_online_node: Linkage for CPU online callback.
+ * @cpu_dead_node: Linkage for CPU offline callback.
* @parallel_wq: The workqueue used for parallel work.
* @serial_wq: The workqueue used for serial work.
* @pslist: List of padata_shell objects attached to this instance.
@@ -150,7 +151,8 @@ struct padata_shell {
* @flags: padata flags.
*/
struct padata_instance {
- struct hlist_node node;
+ struct hlist_node cpu_online_node;
+ struct hlist_node cpu_dead_node;
struct workqueue_struct *parallel_wq;
struct workqueue_struct *serial_wq;
struct list_head pslist;
diff --git a/kernel/padata.c b/kernel/padata.c
index a6afa12fb75e..aae789896616 100644
--- a/kernel/padata.c
+++ b/kernel/padata.c
@@ -703,7 +703,7 @@ static int padata_cpu_online(unsigned int cpu, struct hlist_node *node)
struct padata_instance *pinst;
int ret;

- pinst = hlist_entry_safe(node, struct padata_instance, node);
+ pinst = hlist_entry_safe(node, struct padata_instance, cpu_online_node);
if (!pinst_has_cpu(pinst, cpu))
return 0;

@@ -718,7 +718,7 @@ static int padata_cpu_dead(unsigned int cpu, struct hlist_node *node)
struct padata_instance *pinst;
int ret;

- pinst = hlist_entry_safe(node, struct padata_instance, node);
+ pinst = hlist_entry_safe(node, struct padata_instance, cpu_dead_node);
if (!pinst_has_cpu(pinst, cpu))
return 0;

@@ -734,8 +734,9 @@ static enum cpuhp_state hp_online;
static void __padata_free(struct padata_instance *pinst)
{
#ifdef CONFIG_HOTPLUG_CPU
- cpuhp_state_remove_instance_nocalls(CPUHP_PADATA_DEAD, &pinst->node);
- cpuhp_state_remove_instance_nocalls(hp_online, &pinst->node);
+ cpuhp_state_remove_instance_nocalls(CPUHP_PADATA_DEAD,
+ &pinst->cpu_dead_node);
+ cpuhp_state_remove_instance_nocalls(hp_online, &pinst->cpu_online_node);
#endif

WARN_ON(!list_empty(&pinst->pslist));
@@ -939,9 +940,10 @@ static struct padata_instance *padata_alloc(const char *name,
mutex_init(&pinst->lock);

#ifdef CONFIG_HOTPLUG_CPU
- cpuhp_state_add_instance_nocalls_cpuslocked(hp_online, &pinst->node);
+ cpuhp_state_add_instance_nocalls_cpuslocked(hp_online,
+ &pinst->cpu_online_node);
cpuhp_state_add_instance_nocalls_cpuslocked(CPUHP_PADATA_DEAD,
- &pinst->node);
+ &pinst->cpu_dead_node);
#endif

put_online_cpus();

base-commit: ae83d0b416db002fe95601e7f97f64b59514d936
--
2.26.0


2020-04-22 13:27:48

by Sasha Levin

[permalink] [raw]
Subject: Re: [PATCH] padata: add separate cpuhp node for CPUHP_PADATA_DEAD

Hi

[This is an automated email]

This commit has been processed because it contains a "Fixes:" tag
fixing commit: 894c9ef9780c ("padata: validate cpumask without removed CPU during offline").

The bot has tested the following trees: v5.6.5, v5.4.33.

v5.6.5: Build OK!
v5.4.33: Failed to apply! Possible dependencies:
bfcdcef8c8e3 ("padata: update documentation")


NOTE: The patch will not be queued to stable trees until it is upstream.

How should we proceed with this patch?

--
Thanks
Sasha

2020-04-30 05:31:40

by Herbert Xu

[permalink] [raw]
Subject: Re: [PATCH] padata: add separate cpuhp node for CPUHP_PADATA_DEAD

On Tue, Apr 21, 2020 at 12:34:55PM -0400, Daniel Jordan wrote:
> Removing the pcrypt module triggers this:
>
> general protection fault, probably for non-canonical
> address 0xdead000000000122
> CPU: 5 PID: 264 Comm: modprobe Not tainted 5.6.0+ #2
> Hardware name: QEMU Standard PC
> RIP: 0010:__cpuhp_state_remove_instance+0xcc/0x120
> Call Trace:
> padata_sysfs_release+0x74/0xce
> kobject_put+0x81/0xd0
> padata_free+0x12/0x20
> pcrypt_exit+0x43/0x8ee [pcrypt]
>
> padata instances wrongly use the same hlist node for the online and dead
> states, so __padata_free()'s second cpuhp remove call chokes on the node
> that the first poisoned.
>
> cpuhp multi-instance callbacks only walk forward in cpuhp_step->list and
> the same node is linked in both the online and dead lists, so the list
> corruption that results from padata_alloc() adding the node to a second
> list without removing it from the first doesn't cause problems as long
> as no instances are freed.
>
> Avoid the issue by giving each state its own node.
>
> Fixes: 894c9ef9780c ("padata: validate cpumask without removed CPU during offline")
> Signed-off-by: Daniel Jordan <[email protected]>
> Cc: Herbert Xu <[email protected]>
> Cc: Steffen Klassert <[email protected]>
> Cc: [email protected]
> Cc: [email protected]
> Cc: [email protected] # v5.4+
> ---
> include/linux/padata.h | 6 ++++--
> kernel/padata.c | 14 ++++++++------
> 2 files changed, 12 insertions(+), 8 deletions(-)

Patch applied. Thanks.
--
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt