try_offline_node() checks that all cpus related with removed node have been
removed by using cpu_present_bits. If all cpus related with removed node have
been removed, try_offline_node() clears the node information.
But try_offline_node() called from acpi_processor_remove() never clears
the node information. For disabling cpu_present_bits, acpi_unmap_lsapic()
need be called. But acpi_unmap_lsapic() is called after try_offline_node()
runs. So when try_offline_node() runs, the cpu's cpu_present_bits is always
set.
This patch moves try_offline_node() after acpi_unmap_lsapic().
Signed-off-by: Yasuaki Ishimatsu <[email protected]>
---
drivers/acpi/acpi_processor.c | 3 ++-
1 files changed, 2 insertions(+), 1 deletions(-)
diff --git a/drivers/acpi/acpi_processor.c b/drivers/acpi/acpi_processor.c
index fd6c51c..5a74a9c 100644
--- a/drivers/acpi/acpi_processor.c
+++ b/drivers/acpi/acpi_processor.c
@@ -451,7 +451,6 @@ static void acpi_processor_remove(struct acpi_device *device)
/* Clean up. */
per_cpu(processor_device_array, pr->id) = NULL;
per_cpu(processors, pr->id) = NULL;
- try_offline_node(cpu_to_node(pr->id));
/* Remove the CPU. */
get_online_cpus();
@@ -459,6 +458,8 @@ static void acpi_processor_remove(struct acpi_device *device)
acpi_unmap_lsapic(pr->id);
put_online_cpus();
+ try_offline_node(cpu_to_node(pr->id));
+
out:
free_cpumask_var(pr->throttling.shared_cpu_map);
kfree(pr);
On Tue, 2013-08-06 at 19:11 +0900, Yasuaki Ishimatsu wrote:
> try_offline_node() checks that all cpus related with removed node have been
> removed by using cpu_present_bits. If all cpus related with removed node have
> been removed, try_offline_node() clears the node information.
>
> But try_offline_node() called from acpi_processor_remove() never clears
> the node information. For disabling cpu_present_bits, acpi_unmap_lsapic()
> need be called. But acpi_unmap_lsapic() is called after try_offline_node()
> runs. So when try_offline_node() runs, the cpu's cpu_present_bits is always
> set.
>
> This patch moves try_offline_node() after acpi_unmap_lsapic().
>
> Signed-off-by: Yasuaki Ishimatsu <[email protected]>
The change looks good to me.
Acked-by: Toshi Kani <[email protected]>
BTW, do you know why try_offline_node() has to use stop_machine()?
Thanks,
-Toshi
(2013/08/07 1:56), Toshi Kani wrote:
> On Tue, 2013-08-06 at 19:11 +0900, Yasuaki Ishimatsu wrote:
>> try_offline_node() checks that all cpus related with removed node have been
>> removed by using cpu_present_bits. If all cpus related with removed node have
>> been removed, try_offline_node() clears the node information.
>>
>> But try_offline_node() called from acpi_processor_remove() never clears
>> the node information. For disabling cpu_present_bits, acpi_unmap_lsapic()
>> need be called. But acpi_unmap_lsapic() is called after try_offline_node()
>> runs. So when try_offline_node() runs, the cpu's cpu_present_bits is always
>> set.
>>
>> This patch moves try_offline_node() after acpi_unmap_lsapic().
>>
>> Signed-off-by: Yasuaki Ishimatsu <[email protected]>
>
> The change looks good to me.
>
> Acked-by: Toshi Kani <[email protected]>
Thanks, Toshi!
> BTW, do you know why try_offline_node() has to use stop_machine()?
I don't know the reason.
Thanks,
Yasuaki Ishimatsu
>
> Thanks,
> -Toshi
>
Hi Ishimatsu-san,
On 08/06/2013 06:11 PM, Yasuaki Ishimatsu wrote:
> try_offline_node() checks that all cpus related with removed node have been
> removed by using cpu_present_bits. If all cpus related with removed node have
> been removed, try_offline_node() clears the node information.
>
> But try_offline_node() called from acpi_processor_remove() never clears
> the node information. For disabling cpu_present_bits, acpi_unmap_lsapic()
> need be called. But acpi_unmap_lsapic() is called after try_offline_node()
> runs. So when try_offline_node() runs, the cpu's cpu_present_bits is always
> set.
>
> This patch moves try_offline_node() after acpi_unmap_lsapic().
It works well.
>
> Signed-off-by: Yasuaki Ishimatsu <[email protected]>
Tested-by: Gu Zheng <[email protected]>
Thanks,
Gu
> ---
> drivers/acpi/acpi_processor.c | 3 ++-
> 1 files changed, 2 insertions(+), 1 deletions(-)
>
> diff --git a/drivers/acpi/acpi_processor.c b/drivers/acpi/acpi_processor.c
> index fd6c51c..5a74a9c 100644
> --- a/drivers/acpi/acpi_processor.c
> +++ b/drivers/acpi/acpi_processor.c
> @@ -451,7 +451,6 @@ static void acpi_processor_remove(struct acpi_device *device)
> /* Clean up. */
> per_cpu(processor_device_array, pr->id) = NULL;
> per_cpu(processors, pr->id) = NULL;
> - try_offline_node(cpu_to_node(pr->id));
>
> /* Remove the CPU. */
> get_online_cpus();
> @@ -459,6 +458,8 @@ static void acpi_processor_remove(struct acpi_device *device)
> acpi_unmap_lsapic(pr->id);
> put_online_cpus();
>
> + try_offline_node(cpu_to_node(pr->id));
> +
> out:
> free_cpumask_var(pr->throttling.shared_cpu_map);
> kfree(pr);
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
(2013/08/09 11:43), Gu Zheng wrote:
> Hi Ishimatsu-san,
> On 08/06/2013 06:11 PM, Yasuaki Ishimatsu wrote:
>
>> try_offline_node() checks that all cpus related with removed node have been
>> removed by using cpu_present_bits. If all cpus related with removed node have
>> been removed, try_offline_node() clears the node information.
>>
>> But try_offline_node() called from acpi_processor_remove() never clears
>> the node information. For disabling cpu_present_bits, acpi_unmap_lsapic()
>> need be called. But acpi_unmap_lsapic() is called after try_offline_node()
>> runs. So when try_offline_node() runs, the cpu's cpu_present_bits is always
>> set.
>>
>> This patch moves try_offline_node() after acpi_unmap_lsapic().
>
> It works well.
>
>>
>> Signed-off-by: Yasuaki Ishimatsu <[email protected]>
>
>
> Tested-by: Gu Zheng <[email protected]>
Thank you for testing.
Thanks,
Yasuaki Ishimatsu
>
> Thanks,
> Gu
>
>> ---
>> drivers/acpi/acpi_processor.c | 3 ++-
>> 1 files changed, 2 insertions(+), 1 deletions(-)
>>
>> diff --git a/drivers/acpi/acpi_processor.c b/drivers/acpi/acpi_processor.c
>> index fd6c51c..5a74a9c 100644
>> --- a/drivers/acpi/acpi_processor.c
>> +++ b/drivers/acpi/acpi_processor.c
>> @@ -451,7 +451,6 @@ static void acpi_processor_remove(struct acpi_device *device)
>> /* Clean up. */
>> per_cpu(processor_device_array, pr->id) = NULL;
>> per_cpu(processors, pr->id) = NULL;
>> - try_offline_node(cpu_to_node(pr->id));
>>
>> /* Remove the CPU. */
>> get_online_cpus();
>> @@ -459,6 +458,8 @@ static void acpi_processor_remove(struct acpi_device *device)
>> acpi_unmap_lsapic(pr->id);
>> put_online_cpus();
>>
>> + try_offline_node(cpu_to_node(pr->id));
>> +
>> out:
>> free_cpumask_var(pr->throttling.shared_cpu_map);
>> kfree(pr);
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to [email protected]
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at http://www.tux.org/lkml/
>>
>
>
On 08/07/2013 12:56 AM, Toshi Kani wrote:
> On Tue, 2013-08-06 at 19:11 +0900, Yasuaki Ishimatsu wrote:
>> try_offline_node() checks that all cpus related with removed node have been
>> removed by using cpu_present_bits. If all cpus related with removed node have
>> been removed, try_offline_node() clears the node information.
>>
>> But try_offline_node() called from acpi_processor_remove() never clears
>> the node information. For disabling cpu_present_bits, acpi_unmap_lsapic()
>> need be called. But acpi_unmap_lsapic() is called after try_offline_node()
>> runs. So when try_offline_node() runs, the cpu's cpu_present_bits is always
>> set.
>>
>> This patch moves try_offline_node() after acpi_unmap_lsapic().
>>
>> Signed-off-by: Yasuaki Ishimatsu<[email protected]>
>
> The change looks good to me.
>
> Acked-by: Toshi Kani<[email protected]>
>
> BTW, do you know why try_offline_node() has to use stop_machine()?
try_offline_node() is used to check if the node could be hot-removed
after each memory or cpu hot-remove operation.
In memory hot-remove path, we have lock_memory_hotplug() to series all
the memory hot-remove options.
But when doing cpu hot-remove,
acpi_processor_remove()
|->try_offline_node()
There is no lock to protect it. I think, when we are going to hot-remove
a node, others should not do any memory or cpu hotplug operation. In memory
hotplug path, we have lock_memory_hotplug(). But in cpu hotplug path, I
didn't find any lock. So we used stop_machine() to call check_cpu_on_node().
If we find any cpu still present, we return and do not remove the node.
Thanks.
On Fri, 2013-08-09 at 15:28 +0800, Tang Chen wrote:
> On 08/07/2013 12:56 AM, Toshi Kani wrote:
> > On Tue, 2013-08-06 at 19:11 +0900, Yasuaki Ishimatsu wrote:
> >> try_offline_node() checks that all cpus related with removed node have been
> >> removed by using cpu_present_bits. If all cpus related with removed node have
> >> been removed, try_offline_node() clears the node information.
> >>
> >> But try_offline_node() called from acpi_processor_remove() never clears
> >> the node information. For disabling cpu_present_bits, acpi_unmap_lsapic()
> >> need be called. But acpi_unmap_lsapic() is called after try_offline_node()
> >> runs. So when try_offline_node() runs, the cpu's cpu_present_bits is always
> >> set.
> >>
> >> This patch moves try_offline_node() after acpi_unmap_lsapic().
> >>
> >> Signed-off-by: Yasuaki Ishimatsu<[email protected]>
> >
> > The change looks good to me.
> >
> > Acked-by: Toshi Kani<[email protected]>
> >
> > BTW, do you know why try_offline_node() has to use stop_machine()?
>
> try_offline_node() is used to check if the node could be hot-removed
> after each memory or cpu hot-remove operation.
>
> In memory hot-remove path, we have lock_memory_hotplug() to series all
> the memory hot-remove options.
>
> But when doing cpu hot-remove,
>
> acpi_processor_remove()
> |->try_offline_node()
>
> There is no lock to protect it. I think, when we are going to hot-remove
> a node, others should not do any memory or cpu hotplug operation. In memory
> hotplug path, we have lock_memory_hotplug(). But in cpu hotplug path, I
> didn't find any lock. So we used stop_machine() to call check_cpu_on_node().
> If we find any cpu still present, we return and do not remove the node.
CPU/Memory hotplug operations and sysfs eject are serialized with
acpi_os_hotplug_execute(). CPU online/offline is protected by
cpu_hotplug_[begin|done]() and [get|put]_online_cpus(). But, yes,
online/offline and hotplug operations are not serialized. I tried to
serialize them before, but that framework was not well received.
Anyway, it looks to me that cpu_up()->mem_online_node() path can race
with try_offline_node(). And I do not think stop_machine() protects
this case, either. try_offline_node() stops the execution of
num_online_node() during stop_machine(), but then lets it continues to
run after that. I think they need to be protected by a lock.
I will look further to see if I can come up with some solution.
Thanks,
-Toshi
On Friday, August 09, 2013 04:16:56 PM Toshi Kani wrote:
> On Fri, 2013-08-09 at 15:28 +0800, Tang Chen wrote:
> > On 08/07/2013 12:56 AM, Toshi Kani wrote:
> > > On Tue, 2013-08-06 at 19:11 +0900, Yasuaki Ishimatsu wrote:
> > >> try_offline_node() checks that all cpus related with removed node have been
> > >> removed by using cpu_present_bits. If all cpus related with removed node have
> > >> been removed, try_offline_node() clears the node information.
> > >>
> > >> But try_offline_node() called from acpi_processor_remove() never clears
> > >> the node information. For disabling cpu_present_bits, acpi_unmap_lsapic()
> > >> need be called. But acpi_unmap_lsapic() is called after try_offline_node()
> > >> runs. So when try_offline_node() runs, the cpu's cpu_present_bits is always
> > >> set.
> > >>
> > >> This patch moves try_offline_node() after acpi_unmap_lsapic().
> > >>
> > >> Signed-off-by: Yasuaki Ishimatsu<[email protected]>
> > >
> > > The change looks good to me.
> > >
> > > Acked-by: Toshi Kani<[email protected]>
> > >
> > > BTW, do you know why try_offline_node() has to use stop_machine()?
> >
> > try_offline_node() is used to check if the node could be hot-removed
> > after each memory or cpu hot-remove operation.
> >
> > In memory hot-remove path, we have lock_memory_hotplug() to series all
> > the memory hot-remove options.
> >
> > But when doing cpu hot-remove,
> >
> > acpi_processor_remove()
> > |->try_offline_node()
> >
> > There is no lock to protect it. I think, when we are going to hot-remove
> > a node, others should not do any memory or cpu hotplug operation. In memory
> > hotplug path, we have lock_memory_hotplug(). But in cpu hotplug path, I
> > didn't find any lock. So we used stop_machine() to call check_cpu_on_node().
> > If we find any cpu still present, we return and do not remove the node.
>
> CPU/Memory hotplug operations and sysfs eject are serialized with
> acpi_os_hotplug_execute(). CPU online/offline is protected by
> cpu_hotplug_[begin|done]() and [get|put]_online_cpus(). But, yes,
> online/offline and hotplug operations are not serialized. I tried to
> serialize them before, but that framework was not well received.
What about lock_device_hotplug()? It is taken by both online/offline and
the ACPI hotplug code, isn't it?
> Anyway, it looks to me that cpu_up()->mem_online_node() path can race
> with try_offline_node().
It can in principle, but I'm not sure if there's a way to trigger that
race. Do you have an example?
Rafael
--
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.
On Sat, 2013-08-10 at 01:29 +0200, Rafael J. Wysocki wrote:
> On Friday, August 09, 2013 04:16:56 PM Toshi Kani wrote:
> > On Fri, 2013-08-09 at 15:28 +0800, Tang Chen wrote:
> > > On 08/07/2013 12:56 AM, Toshi Kani wrote:
> > > > On Tue, 2013-08-06 at 19:11 +0900, Yasuaki Ishimatsu wrote:
> > > >> try_offline_node() checks that all cpus related with removed node have been
> > > >> removed by using cpu_present_bits. If all cpus related with removed node have
> > > >> been removed, try_offline_node() clears the node information.
> > > >>
> > > >> But try_offline_node() called from acpi_processor_remove() never clears
> > > >> the node information. For disabling cpu_present_bits, acpi_unmap_lsapic()
> > > >> need be called. But acpi_unmap_lsapic() is called after try_offline_node()
> > > >> runs. So when try_offline_node() runs, the cpu's cpu_present_bits is always
> > > >> set.
> > > >>
> > > >> This patch moves try_offline_node() after acpi_unmap_lsapic().
> > > >>
> > > >> Signed-off-by: Yasuaki Ishimatsu<[email protected]>
> > > >
> > > > The change looks good to me.
> > > >
> > > > Acked-by: Toshi Kani<[email protected]>
> > > >
> > > > BTW, do you know why try_offline_node() has to use stop_machine()?
> > >
> > > try_offline_node() is used to check if the node could be hot-removed
> > > after each memory or cpu hot-remove operation.
> > >
> > > In memory hot-remove path, we have lock_memory_hotplug() to series all
> > > the memory hot-remove options.
> > >
> > > But when doing cpu hot-remove,
> > >
> > > acpi_processor_remove()
> > > |->try_offline_node()
> > >
> > > There is no lock to protect it. I think, when we are going to hot-remove
> > > a node, others should not do any memory or cpu hotplug operation. In memory
> > > hotplug path, we have lock_memory_hotplug(). But in cpu hotplug path, I
> > > didn't find any lock. So we used stop_machine() to call check_cpu_on_node().
> > > If we find any cpu still present, we return and do not remove the node.
> >
> > CPU/Memory hotplug operations and sysfs eject are serialized with
> > acpi_os_hotplug_execute(). CPU online/offline is protected by
> > cpu_hotplug_[begin|done]() and [get|put]_online_cpus(). But, yes,
> > online/offline and hotplug operations are not serialized. I tried to
> > serialize them before, but that framework was not well received.
>
> What about lock_device_hotplug()? It is taken by both online/offline and
> the ACPI hotplug code, isn't it?
Oh, that's right! I forgot about this one. Yes, lock_device_hotplug()
nicely protects online/offline and hotplug operations. :-)
> > Anyway, it looks to me that cpu_up()->mem_online_node() path can race
> > with try_offline_node().
>
> It can in principle, but I'm not sure if there's a way to trigger that
> race. Do you have an example?
With lock_device_hotplug(), I agree that we do not have this race
condition -- cpu_up() may not run while other hotplug is running.
store_online() will be blocked at lock_device_hotplug() in such case.
When store_online() acquired the lock, this CPU may have been deleted.
So, we still need to make sure that this case is handled properly. I
suppose sysfs keeps *dev valid with ref_count (Is that right?). I think
cpu_up() needs to check with cpu_present(), not cpu_possible(), at the
top. Otherwise, cpu_to_node(cpu) may return NUMA_NO_NODE (-1), which is
probably not a good value for node_online(nid).
Thanks,
-Toshi
On Friday, August 09, 2013 08:11:19 PM Toshi Kani wrote:
> On Sat, 2013-08-10 at 01:29 +0200, Rafael J. Wysocki wrote:
> > On Friday, August 09, 2013 04:16:56 PM Toshi Kani wrote:
> > > On Fri, 2013-08-09 at 15:28 +0800, Tang Chen wrote:
> > > > On 08/07/2013 12:56 AM, Toshi Kani wrote:
> > > > > On Tue, 2013-08-06 at 19:11 +0900, Yasuaki Ishimatsu wrote:
> > > > >> try_offline_node() checks that all cpus related with removed node have been
> > > > >> removed by using cpu_present_bits. If all cpus related with removed node have
> > > > >> been removed, try_offline_node() clears the node information.
> > > > >>
> > > > >> But try_offline_node() called from acpi_processor_remove() never clears
> > > > >> the node information. For disabling cpu_present_bits, acpi_unmap_lsapic()
> > > > >> need be called. But acpi_unmap_lsapic() is called after try_offline_node()
> > > > >> runs. So when try_offline_node() runs, the cpu's cpu_present_bits is always
> > > > >> set.
> > > > >>
> > > > >> This patch moves try_offline_node() after acpi_unmap_lsapic().
> > > > >>
> > > > >> Signed-off-by: Yasuaki Ishimatsu<[email protected]>
> > > > >
> > > > > The change looks good to me.
> > > > >
> > > > > Acked-by: Toshi Kani<[email protected]>
> > > > >
> > > > > BTW, do you know why try_offline_node() has to use stop_machine()?
> > > >
> > > > try_offline_node() is used to check if the node could be hot-removed
> > > > after each memory or cpu hot-remove operation.
> > > >
> > > > In memory hot-remove path, we have lock_memory_hotplug() to series all
> > > > the memory hot-remove options.
> > > >
> > > > But when doing cpu hot-remove,
> > > >
> > > > acpi_processor_remove()
> > > > |->try_offline_node()
> > > >
> > > > There is no lock to protect it. I think, when we are going to hot-remove
> > > > a node, others should not do any memory or cpu hotplug operation. In memory
> > > > hotplug path, we have lock_memory_hotplug(). But in cpu hotplug path, I
> > > > didn't find any lock. So we used stop_machine() to call check_cpu_on_node().
> > > > If we find any cpu still present, we return and do not remove the node.
> > >
> > > CPU/Memory hotplug operations and sysfs eject are serialized with
> > > acpi_os_hotplug_execute(). CPU online/offline is protected by
> > > cpu_hotplug_[begin|done]() and [get|put]_online_cpus(). But, yes,
> > > online/offline and hotplug operations are not serialized. I tried to
> > > serialize them before, but that framework was not well received.
> >
> > What about lock_device_hotplug()? It is taken by both online/offline and
> > the ACPI hotplug code, isn't it?
>
> Oh, that's right! I forgot about this one. Yes, lock_device_hotplug()
> nicely protects online/offline and hotplug operations. :-)
>
> > > Anyway, it looks to me that cpu_up()->mem_online_node() path can race
> > > with try_offline_node().
> >
> > It can in principle, but I'm not sure if there's a way to trigger that
> > race. Do you have an example?
>
> With lock_device_hotplug(), I agree that we do not have this race
> condition -- cpu_up() may not run while other hotplug is running.
> store_online() will be blocked at lock_device_hotplug() in such case.
> When store_online() acquired the lock, this CPU may have been deleted.
> So, we still need to make sure that this case is handled properly.
Yes.
> I suppose sysfs keeps *dev valid with ref_count (Is that right?).
Yes, it does.
> I think cpu_up() needs to check with cpu_present(), not cpu_possible(), at
> the top. Otherwise, cpu_to_node(cpu) may return NUMA_NO_NODE (-1), which is
> probably not a good value for node_online(nid).
We do cpu_to_node(cpuid) in cpu_subsys_online() before that, so maybe
it's better to check the result already there and bail out if that's
negative?
Something like the patch below.
Thanks,
Rafael
---
drivers/base/cpu.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
Index: linux-pm/drivers/base/cpu.c
===================================================================
--- linux-pm.orig/drivers/base/cpu.c
+++ linux-pm/drivers/base/cpu.c
@@ -43,11 +43,14 @@ static int __ref cpu_subsys_online(struc
struct cpu *cpu = container_of(dev, struct cpu, dev);
int cpuid = dev->id;
int from_nid, to_nid;
- int ret;
+ int ret = -ENODEV;
cpu_hotplug_driver_lock();
from_nid = cpu_to_node(cpuid);
+ if (from_nid == NUMA_NO_NODE)
+ goto out;
+
ret = cpu_up(cpuid);
/*
* When hot adding memory to memoryless node and enabling a cpu
@@ -57,6 +60,7 @@ static int __ref cpu_subsys_online(struc
if (from_nid != to_nid)
change_cpu_under_node(cpu, from_nid, to_nid);
+ out:
cpu_hotplug_driver_unlock();
return ret;
}
On Sun, 2013-08-11 at 22:42 +0200, Rafael J. Wysocki wrote:
> On Friday, August 09, 2013 08:11:19 PM Toshi Kani wrote:
> > On Sat, 2013-08-10 at 01:29 +0200, Rafael J. Wysocki wrote:
> > > On Friday, August 09, 2013 04:16:56 PM Toshi Kani wrote:
> > > > On Fri, 2013-08-09 at 15:28 +0800, Tang Chen wrote:
> > > > > On 08/07/2013 12:56 AM, Toshi Kani wrote:
:
> > > > > > BTW, do you know why try_offline_node() has to use stop_machine()?
> > > > >
> > > > > try_offline_node() is used to check if the node could be hot-removed
> > > > > after each memory or cpu hot-remove operation.
> > > > >
> > > > > In memory hot-remove path, we have lock_memory_hotplug() to series all
> > > > > the memory hot-remove options.
> > > > >
> > > > > But when doing cpu hot-remove,
> > > > >
> > > > > acpi_processor_remove()
> > > > > |->try_offline_node()
> > > > >
> > > > > There is no lock to protect it. I think, when we are going to hot-remove
> > > > > a node, others should not do any memory or cpu hotplug operation. In memory
> > > > > hotplug path, we have lock_memory_hotplug(). But in cpu hotplug path, I
> > > > > didn't find any lock. So we used stop_machine() to call check_cpu_on_node().
> > > > > If we find any cpu still present, we return and do not remove the node.
> > > >
> > > > CPU/Memory hotplug operations and sysfs eject are serialized with
> > > > acpi_os_hotplug_execute(). CPU online/offline is protected by
> > > > cpu_hotplug_[begin|done]() and [get|put]_online_cpus(). But, yes,
> > > > online/offline and hotplug operations are not serialized. I tried to
> > > > serialize them before, but that framework was not well received.
> > >
> > > What about lock_device_hotplug()? It is taken by both online/offline and
> > > the ACPI hotplug code, isn't it?
> >
> > Oh, that's right! I forgot about this one. Yes, lock_device_hotplug()
> > nicely protects online/offline and hotplug operations. :-)
> >
> > > > Anyway, it looks to me that cpu_up()->mem_online_node() path can race
> > > > with try_offline_node().
> > >
> > > It can in principle, but I'm not sure if there's a way to trigger that
> > > race. Do you have an example?
> >
> > With lock_device_hotplug(), I agree that we do not have this race
> > condition -- cpu_up() may not run while other hotplug is running.
> > store_online() will be blocked at lock_device_hotplug() in such case.
> > When store_online() acquired the lock, this CPU may have been deleted.
> > So, we still need to make sure that this case is handled properly.
>
> Yes.
>
> > I suppose sysfs keeps *dev valid with ref_count (Is that right?).
>
> Yes, it does.
>
> > I think cpu_up() needs to check with cpu_present(), not cpu_possible(), at
> > the top. Otherwise, cpu_to_node(cpu) may return NUMA_NO_NODE (-1), which is
> > probably not a good value for node_online(nid).
>
> We do cpu_to_node(cpuid) in cpu_subsys_online() before that, so maybe
> it's better to check the result already there and bail out if that's
> negative?
>
> Something like the patch below.
That looks good to me.
Thanks,
-Toshi
From: Rafael J. Wysocki <[email protected]>
There is a potential race condition between cpu_subsys_online()
and either acpi_processor_remove() or remove_memory() that execute
try_offline_node(). Namely, it is possible that cpu_subsys_online()
will run right after the CPUs NUMA node has been put offline and
cpu_to_node() executed by it will return NUMA_NO_NODE (-1). In
that case the CPU is gone and it doesn't make sense to call cpu_up()
for it, so make cpu_subsys_online() return -ENODEV then.
Signed-off-by: Rafael J. Wysocki <[email protected]>
Acked-by: Toshi Kani <[email protected]>
---
Hi Greg,
Can you please kindly queue up this one for 3.12?
I've added the Toshi's ACK as he said it looked good to him. :-)
Thanks,
Rafael
---
drivers/base/cpu.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
Index: linux-pm/drivers/base/cpu.c
===================================================================
--- linux-pm.orig/drivers/base/cpu.c
+++ linux-pm/drivers/base/cpu.c
@@ -43,11 +43,14 @@ static int __ref cpu_subsys_online(struc
struct cpu *cpu = container_of(dev, struct cpu, dev);
int cpuid = dev->id;
int from_nid, to_nid;
- int ret;
+ int ret = -ENODEV;
cpu_hotplug_driver_lock();
from_nid = cpu_to_node(cpuid);
+ if (from_nid == NUMA_NO_NODE)
+ goto out;
+
ret = cpu_up(cpuid);
/*
* When hot adding memory to memoryless node and enabling a cpu
@@ -57,6 +60,7 @@ static int __ref cpu_subsys_online(struc
if (from_nid != to_nid)
change_cpu_under_node(cpu, from_nid, to_nid);
+ out:
cpu_hotplug_driver_unlock();
return ret;
}
On Tue, 2013-08-13 at 02:39 +0200, Rafael J. Wysocki wrote:
> From: Rafael J. Wysocki <[email protected]>
>
> There is a potential race condition between cpu_subsys_online()
> and either acpi_processor_remove() or remove_memory() that execute
> try_offline_node(). Namely, it is possible that cpu_subsys_online()
> will run right after the CPUs NUMA node has been put offline and
> cpu_to_node() executed by it will return NUMA_NO_NODE (-1). In
> that case the CPU is gone and it doesn't make sense to call cpu_up()
> for it, so make cpu_subsys_online() return -ENODEV then.
>
> Signed-off-by: Rafael J. Wysocki <[email protected]>
> Acked-by: Toshi Kani <[email protected]>
> ---
>
> Hi Greg,
>
> Can you please kindly queue up this one for 3.12?
>
> I've added the Toshi's ACK as he said it looked good to him. :-)
That's fine by me.
-Toshi