CPU hotremove will migrate tasks and redirect interrupts off dead cpu.
To remove multiple CPUs, we should iteratively do single cpu removal.
If tasks and interrupts are migrated to a cpu which will be soon
removed, then we will trash tasks and interrupts again. The following
patches allow remove several cpus one time. It's fast and avoids
unnecessary repeated trash tasks and interrupts. This will help NUMA
style hardware removal and SMP suspend/resume. Comments and suggestions
are appreciated.
Thanks,
Shaohua
On Mon, May 08, 2006 at 01:45:37PM +0800, Shaohua Li wrote:
> CPU hotremove will migrate tasks and redirect interrupts off dead cpu.
> To remove multiple CPUs, we should iteratively do single cpu removal.
> If tasks and interrupts are migrated to a cpu which will be soon
> removed, then we will trash tasks and interrupts again. The following
> patches allow remove several cpus one time. It's fast and avoids
> unnecessary repeated trash tasks and interrupts. This will help NUMA
> style hardware removal and SMP suspend/resume. Comments and suggestions
> are appreciated.
>
Thanks shaohua, since the initial code has been collecting dust for
some several months now, and I never got time to finish it.
Think you forgot to cc the ia64 crowd. We tried to do it a little bit more
generic way, but some required arch assist due to some differences.
Andrew: please help stage in -mm since we would need more extensive testing
and help feedback to fix any before becomming prime time.
It would also help to check if other archs have any compile issues etc..
--
Cheers,
Ashok Raj
- Open Source Technology Center
Shaohua Li wrote:
> CPU hotremove will migrate tasks and redirect interrupts off dead cpu.
> To remove multiple CPUs, we should iteratively do single cpu removal.
> If tasks and interrupts are migrated to a cpu which will be soon
> removed, then we will trash tasks and interrupts again. The following
> patches allow remove several cpus one time. It's fast and avoids
> unnecessary repeated trash tasks and interrupts. This will help NUMA
> style hardware removal and SMP suspend/resume. Comments and suggestions
> are appreciated.
Some quantification of the benefits of adding this complexity would be
appreciated. Like, how long does it take to offline all the cpus in a
node serially, and how much faster is the bulk method?
On Mon, 2006-05-08 at 01:29 -0500, Nathan Lynch wrote:
> Shaohua Li wrote:
> > CPU hotremove will migrate tasks and redirect interrupts off dead cpu.
> > To remove multiple CPUs, we should iteratively do single cpu removal.
> > If tasks and interrupts are migrated to a cpu which will be soon
> > removed, then we will trash tasks and interrupts again. The following
> > patches allow remove several cpus one time. It's fast and avoids
> > unnecessary repeated trash tasks and interrupts. This will help NUMA
> > style hardware removal and SMP suspend/resume. Comments and suggestions
> > are appreciated.
>
> Some quantification of the benefits of adding this complexity would be
> appreciated. Like, how long does it take to offline all the cpus in a
> node serially, and how much faster is the bulk method?
This depends on the workload and I haven't a data at hand, sorry. Fast
(less tasks/timers/softirqs migration) is one benefit. The method also
means less time in stop_machine_run and less interrupt redirection, so
smaller impact to the whole system.
Thanks,
Shaohua
Shaohua Li <[email protected]> wrote:
>
> CPU hotremove will migrate tasks and redirect interrupts off dead cpu.
> To remove multiple CPUs, we should iteratively do single cpu removal.
> If tasks and interrupts are migrated to a cpu which will be soon
> removed, then we will trash tasks and interrupts again. The following
> patches allow remove several cpus one time. It's fast and avoids
> unnecessary repeated trash tasks and interrupts. This will help NUMA
> style hardware removal and SMP suspend/resume. Comments and suggestions
> are appreciated.
This seems an awful lot of code for something which happens so infrequently.
How big is the problem you're fixing here, and what are the
user-observeable effects of these changes?
On Wed, May 10, 2006 at 11:06:06PM -0700, Andrew Morton wrote:
Hi Andrew,
> Shaohua Li <[email protected]> wrote:
> >
> > CPU hotremove will migrate tasks and redirect interrupts off dead cpu.
>
> This seems an awful lot of code for something which happens so infrequently.
>
> How big is the problem you're fixing here, and what are the
> user-observeable effects of these changes?
This is useful when say a NUMA node is being removed. With new multi-core
CPUs comming up, considering a 2 core with HT, we could have up to 4 logical
per socket. On NUMA node with 4 sockets, a node removal will mean we
do 16 single cpu offlines. Each time the process and interrupts could
end up on a CPU that might be removed just immediatly.
The same is also useful for SMP Suspend/resume cases since the logical offline
is same here as well.
Even thought the code changes seem a lot, most of it is just preparation of
functions ready to accept a cpumask_t instead of a single cpu like earlier.
The reason we split them to smaller chunks so the scope of change is well
understood with each patch.
The major changes are
- stop machine to run cpu offline functions on each cpu going offline
- prepare offline functions in offline path to take cpumask_t
- Some task migrate dead lock removal consideration that we ran into
during stress test.
I know Shaohua ran tests for more than 20+ hrs with the patch, both on i386
and x86_64.
once we get some time deltas on a bigger machine it will help a lot.
Iam also trying to check with some OEM';s who have such large machines for
some data.. keep posted.
ashokr
--
Cheers,
Ashok Raj
- Open Source Technology Center
Ashok Raj <[email protected]> wrote:
>
> On Wed, May 10, 2006 at 11:06:06PM -0700, Andrew Morton wrote:
> Hi Andrew,
>
> > Shaohua Li <[email protected]> wrote:
> > >
> > > CPU hotremove will migrate tasks and redirect interrupts off dead cpu.
> >
> > This seems an awful lot of code for something which happens so infrequently.
> >
> > How big is the problem you're fixing here, and what are the
> > user-observeable effects of these changes?
>
> This is useful when say a NUMA node is being removed. With new multi-core
> CPUs comming up, considering a 2 core with HT, we could have up to 4 logical
> per socket. On NUMA node with 4 sockets, a node removal will mean we
> do 16 single cpu offlines. Each time the process and interrupts could
> end up on a CPU that might be removed just immediatly.
>
> The same is also useful for SMP Suspend/resume cases since the logical offline
> is same here as well.
>
> Even thought the code changes seem a lot, most of it is just preparation of
> functions ready to accept a cpumask_t instead of a single cpu like earlier.
> The reason we split them to smaller chunks so the scope of change is well
> understood with each patch.
>
> The major changes are
>
> - stop machine to run cpu offline functions on each cpu going offline
> - prepare offline functions in offline path to take cpumask_t
> - Some task migrate dead lock removal consideration that we ran into
> during stress test.
>
> I know Shaohua ran tests for more than 20+ hrs with the patch, both on i386
> and x86_64.
>
> once we get some time deltas on a bigger machine it will help a lot.
> Iam also trying to check with some OEM';s who have such large machines for
> some data.. keep posted.
>
OK, thanks. I'm a little surprised that this patch wasn't accompanied by a
problem description, really. I mean, if a single CPU offlining takes three
milliseconds then why bother?
I assume it must take much longer, else you wouldn't have written the code.
Have you any ballpark numbers for how long it _does_ take?
Ashok Raj wrote:
> On Wed, May 10, 2006 at 11:06:06PM -0700, Andrew Morton wrote:
> Hi Andrew,
>
> > Shaohua Li <[email protected]> wrote:
> > >
> > > CPU hotremove will migrate tasks and redirect interrupts off dead cpu.
> >
> > This seems an awful lot of code for something which happens so infrequently.
> >
> > How big is the problem you're fixing here, and what are the
> > user-observeable effects of these changes?
>
> This is useful when say a NUMA node is being removed. With new multi-core
> CPUs comming up, considering a 2 core with HT, we could have up to 4 logical
> per socket. On NUMA node with 4 sockets, a node removal will mean we
> do 16 single cpu offlines. Each time the process and interrupts could
> end up on a CPU that might be removed just immediatly.
But offlining all the cpus in a node is already something that just
works. If the user is all that concerned about not thrashing the
tasks running on that node, they would have a workload manager that
migrates the tasks off the node before shooting down cpus. Similar
argument applies to interrupt affinity.
I really haven't seen a compelling argument for why this is needed,
just a bunch of handwaving so far, sorry.
On Thu, May 11, 2006 at 10:02:15AM -0700, Andrew Morton wrote:
>
> OK, thanks. I'm a little surprised that this patch wasn't accompanied by a
> problem description, really. I mean, if a single CPU offlining takes three
> milliseconds then why bother?
It depends on whats running at the time... with some light load, i measured
wall clock time, i remember seeing 2 secs at times, but its been a long time
i did that.. so take that with a pinch :-)_
i will try to get those idle and load times worked out again... the best
i have is a 16 way, if i get help from big system oems i will send the
numbers out
>
> I assume it must take much longer, else you wouldn't have written the code.
> Have you any ballpark numbers for how long it _does_ take?
--
Cheers,
Ashok Raj
- Open Source Technology Center
On Thu, May 11, 2006 at 12:19:20PM -0500, Nathan Lynch wrote:
>
> But offlining all the cpus in a node is already something that just
> works. If the user is all that concerned about not thrashing the
> tasks running on that node, they would have a workload manager that
> migrates the tasks off the node before shooting down cpus. Similar
> argument applies to interrupt affinity.
>
> I really haven't seen a compelling argument for why this is needed,
> just a bunch of handwaving so far, sorry.
Hand waving? Dont think that was intensional though.. i think we are trying
to address a real problem, if there is a reasonable alternate already
that we are not aware of, no problemo...
1. Regarding process migration, someone needs to make sure they run
something like a taskset away from all the cpus that are planned to be
removed upfront. This needs to be done on all processes on the system.
[nick, is there an easier way to do this in today's sched infrastructure or
otherwise]
2. For interrrupt migration, today when we take a cpu offline, we pick
a random online cpu today. So if you have a cpu going offline, and the
next logical cpu is also part of the same package, or node, we have
no smarts today to keep migration away from those "to be offlined" cpus.
If we have a solution to these already, or a simpler alternative, we are
open to those... and iam getting help on large system validation.. it might
not be easy right away, but its comming.
Cheers,
ashok
--
Cheers,
Ashok Raj
- Open Source Technology Center
Ashok Raj wrote:
> On Thu, May 11, 2006 at 12:19:20PM -0500, Nathan Lynch wrote:
> >
> > But offlining all the cpus in a node is already something that just
> > works. If the user is all that concerned about not thrashing the
> > tasks running on that node, they would have a workload manager that
> > migrates the tasks off the node before shooting down cpus. Similar
> > argument applies to interrupt affinity.
> >
> > I really haven't seen a compelling argument for why this is needed,
> > just a bunch of handwaving so far, sorry.
>
> Hand waving? Dont think that was intensional though.. i think we are trying
> to address a real problem, if there is a reasonable alternate already
> that we are not aware of, no problemo...
If the motivation for these patches is to minimize disruption of the
workload when offlining a group of cpus, then I think the reasonable
alternative is for the admin (or a script) to migrate sensitive
tasks and interrupts to cpus that are not going to be offlined --
before offlining any cpus.
On the other hand, I'm getting the feeling that the problem you're
really trying to address is that offlining lots of cpus takes a long
time (on the order of a couple seconds per cpu) on your architectures.
So is it one of these two things, or a combination of both, or what?
> 1. Regarding process migration, someone needs to make sure they run
> something like a taskset away from all the cpus that are planned to be
> removed upfront. This needs to be done on all processes on the system.
I guess I don't understand what you're getting at here. Are you
agreeing with me? Or are you saying that doing this is a problem?
> 2. For interrrupt migration, today when we take a cpu offline, we pick
> a random online cpu today.
That's an architecture-specific behavior. On powerpc, interrupts
bound to a dying cpu have their affinity reset to the default (all cpus).
> So if you have a cpu going offline, and the
> next logical cpu is also part of the same package, or node, we have
> no smarts today to keep migration away from those "to be offlined"
> cpus.
Yes, and the kernel really shouldn't have to be smart about this.
This is a matter of policy. The admin knows which cpus are going to
be online. The admin is free to reassign the irqs before offlining
cpus for optimal behavior. (Of course, the kernel should still have a
sane fallback behavior, and it does.)
So, I still don't see the benefit of adding this change which works
around the kernel's scheduling behavior etc. when the admin easily has
the ability to mitigate the disruption by taking some preliminary
measures.
Ashok Raj wrote:
> On Thu, May 11, 2006 at 10:02:15AM -0700, Andrew Morton wrote:
>
>>OK, thanks. I'm a little surprised that this patch wasn't accompanied by a
>>problem description, really. I mean, if a single CPU offlining takes three
>>milliseconds then why bother?
>
>
> It depends on whats running at the time... with some light load, i measured
> wall clock time, i remember seeing 2 secs at times, but its been a long time
> i did that.. so take that with a pinch :-)_
>
> i will try to get those idle and load times worked out again... the best
> i have is a 16 way, if i get help from big system oems i will send the
> numbers out
Why is taking 30s to offline CPUs a problem?
M.
On Thu, May 11, 2006 at 01:42:47PM -0700, Martin Bligh wrote:
> Ashok Raj wrote:
> >
> >
> > It depends on whats running at the time... with some light load, i measured
> > wall clock time, i remember seeing 2 secs at times, but its been a long time
> > i did that.. so take that with a pinch :-)_
> >
> > i will try to get those idle and load times worked out again... the best
> > i have is a 16 way, if i get help from big system oems i will send the
> > numbers out
>
> Why is taking 30s to offline CPUs a problem?
>
Well, the real problem is that for each cpu offline we schedule a RT thread
kstopmachine() on each cpu, then turn off interrupts until this one cpu has
removed. stop_machine_run() is a big enough sledge hammer during cpu offline
and doing this repeatedly... say on a 4 socket system, where each socket=16
logical cpus.
the system would tend to get hick ups 64 times, since we do the stopmachine
thread once for each logical cpu. When we want to replace a node for
reliability reasons, its not clear if this hick ups is a good thing.
Doing kstopmachine() on a single system is in itself noticable, what we heard
from some OEM's is this would have other app level impact as well.
With the bulk removal, we do stop machine just once, but all the 16 cpus
get removed once hence there is just one hickup, instead of 64.
Less time to offline, avoid process and interrupt bouncing on and off a cpu
which is just about to be offlined are almost extra fringe benefits you get
with the bulk removal approach.
--
Cheers,
Ashok Raj
- Open Source Technology Center
On Thu, May 11, 2006 at 02:19:39PM -0500, Nathan Lynch wrote:
> > > just a bunch of handwaving so far, sorry.
> >
> > Hand waving? Dont think that was intensional though.. i think we are trying
> > to address a real problem, if there is a reasonable alternate already
> > that we are not aware of, no problemo...
>
> If the motivation for these patches is to minimize disruption of the
> workload when offlining a group of cpus, then I think the reasonable
> alternative is for the admin (or a script) to migrate sensitive
> tasks and interrupts to cpus that are not going to be offlined --
> before offlining any cpus.
>
> On the other hand, I'm getting the feeling that the problem you're
> really trying to address is that offlining lots of cpus takes a long
Think i relied to the other thread... we can invent lots of goodies to keep
process and interrupt bouncing... but nothing is going to stop the
kstopmachine() hammer for each logical cpu offline in any architecture:-(
Ashok Raj wrote:
> On Thu, May 11, 2006 at 01:42:47PM -0700, Martin Bligh wrote:
> > Ashok Raj wrote:
> > >
> > >
> > > It depends on whats running at the time... with some light load, i measured
> > > wall clock time, i remember seeing 2 secs at times, but its been a long time
> > > i did that.. so take that with a pinch :-)_
> > >
> > > i will try to get those idle and load times worked out again... the best
> > > i have is a 16 way, if i get help from big system oems i will send the
> > > numbers out
> >
> > Why is taking 30s to offline CPUs a problem?
> >
>
> Well, the real problem is that for each cpu offline we schedule a RT thread
> kstopmachine() on each cpu, then turn off interrupts until this one cpu has
> removed. stop_machine_run() is a big enough sledge hammer during cpu offline
> and doing this repeatedly... say on a 4 socket system, where each socket=16
> logical cpus.
>
> the system would tend to get hick ups 64 times, since we do the stopmachine
> thread once for each logical cpu. When we want to replace a node for
> reliability reasons, its not clear if this hick ups is a good thing.
Can you provide more detail on what you mean by hiccups?
> Doing kstopmachine() on a single system is in itself noticable, what we heard
> from some OEM's is this would have other app level impact as well.
What "other app level impact"?
> With the bulk removal, we do stop machine just once, but all the 16 cpus
> get removed once hence there is just one hickup, instead of 64.
Have you done any profiling or other instrumentation that identifies
stopmachine as the real culprit here? I mean, it's a reasonable
assumption to make, but are you sure there's not something else
causing the hiccups? Perhaps contention on the cpu hotplug lock, or
something wrong in the architecture cpu_disable code?
Module unload also uses stop_machine_run, iirc. Do you see hiccups
with that, too?
> Less time to offline, avoid process and interrupt bouncing on and off a cpu
> which is just about to be offlined are almost extra fringe benefits you get
> with the bulk removal approach.
Ok, so that's not the primary motivation for these patches?
Ashok Raj wrote:
> > On Thu, May 11, 2006 at 01:42:47PM -0700, Martin Bligh wrote:
> > > Ashok Raj wrote:
> > > >
> > > >
> > > > It depends on whats running at the time... with some
> light load, i
> > > > measured wall clock time, i remember seeing 2 secs at
> times, but
> > > > its been a long time i did that.. so take that with a pinch :-)_
> > > >
> > > > i will try to get those idle and load times worked out again...
> > > > the best i have is a 16 way, if i get help from big
> system oems i
> > > > will send the numbers out
> > >
> > > Why is taking 30s to offline CPUs a problem?
> > >
> >
> > Well, the real problem is that for each cpu offline we
> schedule a RT
> > thread
> > kstopmachine() on each cpu, then turn off interrupts until this one
> > cpu has removed. stop_machine_run() is a big enough sledge hammer
> > during cpu offline and doing this repeatedly... say on a 4 socket
> > system, where each socket=16 logical cpus.
> >
> > the system would tend to get hick ups 64 times, since we do the
> > stopmachine thread once for each logical cpu. When we want
> to replace
> > a node for reliability reasons, its not clear if this hick
> ups is a good thing.
>
> Can you provide more detail on what you mean by hiccups?
>
>
> > Doing kstopmachine() on a single system is in itself
> noticable, what
> > we heard from some OEM's is this would have other app level
> impact as well.
>
Per Ashok's request here is some data on offlining with cpu_bulk_remove
vs. sequentially with a shell script.
I had 64x system (physical CPU) and 128 (those 64 hyperthreaded). The
system was idle.
Elapsed times are not strikingly different but system/user times are:
64 CPU 128 CPU
(64 physical hyperthreaded)
cpu_bulk_remove shell script
cpu_bulk_remove shell script
all offline real 0m11.231s real 0m10.775s real
0m26.973s real 0m16.484s
user 0m0.000s user 0m0.056 user
0m0.000s user 0m0.068s
sys 0m0.072s sys 0m0.448 sys
0m0.132s sys 0m1.312s
real 0m11.977s real 0m10.550s real
0m28.978s real 0m14.259s
user 0m0.000s user 0m0.064s user
0m0.000s user 0m0.060s
sys 0m0.032s sys 0m0.464s sys
0m0.152s sys 0m1.432s
32 offline real 0m1.320s real 0m2.422s real
0m1.647s real 0m1.896s
user 0m0.000s user 0m0.000s user
0m0.000s user 0m0.020s
sys 0m0.076s sys 0m0.232s sys
0m0.096s sys 0m0.456s
real 0m1.407s real 0m3.348s real
0m0.418s real 0m1.198s
user 0m0.000s user 0m0.012s user
0m0.000s user 0m0.008s
sys 0m0.072s sys 0m0.276s sys
0m0.044s sys 0m0.244s
groups of 16 real 0m5.877s real 0m11.403s
user 0m0.000s user 0m0.024s
sys 0m0.140s sys 0m0.408s
groups of 8 real 0m8.847s real 0m12.078s real
0m12.311s real 0m12.736s
user 0m0.004s user 0m0.028s user
0m0.004s user 0m0.076s
sys 0m0.232s sys 0m0.536s sys
0m0.448s sys 0m1.448s
real
0m11.968s real 0m14.314s
user
0m0.008s user 0m0.084s
sys
0m0.400s sys 0m1.492s
With smaller "bulks" cpu_bulk_remove is always better, but with large
ones shell script mostly wins, especially with hyperthreading (despite
of much better system and user times!)
Thanks,
--Natalie
> Per Ashok's request here is some data on offlining with
> cpu_bulk_remove vs. sequentially with a shell script.
> I had 64x system (physical CPU) and 128 (those 64
> hyperthreaded). The system was idle.
> Elapsed times are not strikingly different but system/user times are:
>
> 64 CPU
> 128 CPU
> (64 physical hyperthreaded)
> cpu_bulk_remove shell script
> cpu_bulk_remove shell script
> all offline real 0m11.231s real 0m10.775s real
> 0m26.973s real 0m16.484s
> user 0m0.000s user 0m0.056 user
> 0m0.000s user 0m0.068s
> sys 0m0.072s sys 0m0.448 sys
> 0m0.132s sys 0m1.312s
>
> real 0m11.977s real 0m10.550s real
> 0m28.978s real 0m14.259s
> user 0m0.000s user 0m0.064s user
> 0m0.000s user 0m0.060s
> sys 0m0.032s sys 0m0.464s sys
> 0m0.152s sys 0m1.432s
>
> 32 offline real 0m1.320s real 0m2.422s real
> 0m1.647s real 0m1.896s
> user 0m0.000s user 0m0.000s user
> 0m0.000s user 0m0.020s
> sys 0m0.076s sys 0m0.232s sys
> 0m0.096s sys 0m0.456s
>
> real 0m1.407s real 0m3.348s real
> 0m0.418s real 0m1.198s
> user 0m0.000s user 0m0.012s user
> 0m0.000s user 0m0.008s
> sys 0m0.072s sys 0m0.276s sys
> 0m0.044s sys 0m0.244s
>
> groups of 16 real 0m5.877s real 0m11.403s
> user 0m0.000s user 0m0.024s
> sys 0m0.140s sys 0m0.408s
>
> groups of 8 real 0m8.847s real 0m12.078s real
> 0m12.311s real 0m12.736s
> user 0m0.004s user 0m0.028s user
> 0m0.004s user 0m0.076s
> sys 0m0.232s sys 0m0.536s sys
> 0m0.448s sys 0m1.448s
>
> real
> 0m11.968s real 0m14.314s
> user
> 0m0.008s user 0m0.084s
> sys
> 0m0.400s sys 0m1.492s
>
> With smaller "bulks" cpu_bulk_remove is always better, but
> with large ones shell script mostly wins, especially with
> hyperthreading (despite of much better system and user times!)
>
Oops it all wrapped :O
I'm attaching the data file instead, sorry about that.
--Natalie
>