Increase the generic ia64 config from its current max of 512 CPUs to a
new max of 1024 CPUs.
The principal file change that implements this is arch/ia64/defconfig,
which currently defines CONFIG_NR_CPUS=512. The change to
arch/ia64/Kconfig brings that file's declaration of the NR_CPUS default
up to the new reality of 1024. That Kconfig currently says
default "64"
which is bogus -- it is currently 512.
The recent change to arch/ia64/configs/sn2_defconfig just raised that
platform-specific defconfig to ==1024.
The rationale for increasing the generic ia64 defconfig to 1024 is that
the generic config needs to support the new upcoming max CPU count for
ia64 platforms, which for ia64/sn ("Altix") platform is 1024, just as
the current max of 512 supports the current max CPU count.
The downside is that the ia64 cpumask increases from 8 words to 16.
I have tried various heavy workloads and have seen no significant
measurable performance regression from this increase. The potential
extra cachemiss seems to be lost in the noise. The for_each_*cpu()
macros are relatively efficient in skipping past zeroed cpumask bits.
Workloads that impose higher loads on the CPU Scheduler tend to
bottleneck on non-Scheduler parts of the kernel, and it's the Scheduler
which makes the principal use of the cpumask_t, so these extra
cachemiss inefficiencies and extra CPU cycles to scan zero mask words
just get lost in the general system overhead.
Signed-off-by: John Hawkes <[email protected]>
Acked-by: Jack Steiner <[email protected]>
Acked-by: Greg Edwards <[email protected]>
Index: linux/arch/ia64/Kconfig
===================================================================
--- linux.orig/arch/ia64/Kconfig 2006-01-02 19:21:10.000000000 -0800
+++ linux/arch/ia64/Kconfig 2006-01-04 10:32:50.000000000 -0800
@@ -245,7 +245,7 @@
int "Maximum number of CPUs (2-1024)"
range 2 1024
depends on SMP
- default "64"
+ default "1024"
help
You should set this to the number of CPUs in your system, but
keep in mind that a kernel compiled for, e.g., 2 CPUs will boot but
Index: linux/arch/ia64/defconfig
===================================================================
--- linux.orig/arch/ia64/defconfig 2006-01-02 19:21:10.000000000 -0800
+++ linux/arch/ia64/defconfig 2006-01-04 10:32:50.000000000 -0800
@@ -98,7 +98,7 @@
# CONFIG_IA64_SGI_SN_XP is not set
CONFIG_FORCE_MAX_ZONEORDER=18
CONFIG_SMP=y
-CONFIG_NR_CPUS=512
+CONFIG_NR_CPUS=1024
CONFIG_HOTPLUG_CPU=y
# CONFIG_SCHED_SMT is not set
# CONFIG_PREEMPT is not set
[email protected] wrote on Thursday, January 05, 2006 1:40 PM
> The downside is that the ia64 cpumask increases from 8 words to 16.
> I have tried various heavy workloads and have seen no significant
> measurable performance regression from this increase.
What type of heavy workloads have you measured? Including db transaction
processing and decision making workloads?
> The potential
> extra cachemiss seems to be lost in the noise. The for_each_*cpu()
> macros are relatively efficient in skipping past zeroed cpumask bits.
> Workloads that impose higher loads on the CPU Scheduler tend to
> bottleneck on non-Scheduler parts of the kernel, and it's the Scheduler
> which makes the principal use of the cpumask_t, so these extra
> cachemiss inefficiencies and extra CPU cycles to scan zero mask words
> just get lost in the general system overhead.
I found above claims are generally false for workload that puts tons
of pressure on CPU cache, especially with db workload. Typically
for db workload, the working set in user space is so large that making
a trip into the kernel has far large secondary effect then the primary
cache miss occurred in the kernel. In other word, cache lines evicted
by the kernel code have far larger impact to the overall application
performance and leads to lower overall lower system performance. So
when you say "get lost in the general system overhead", did you consider
the secondary effect it does to the application performance?
What we found is going from NR_CPU = 64 to 128, it has small performance
impact to db transaction processing workload. Though I have not measured
difference between 128 to 1024.
- Ken
On Thu, 2006-01-05 at 13:39 -0800, [email protected] wrote:
> Increase the generic ia64 config from its current max of 512 CPUs to a
> new max of 1024 CPUs.
I wonder why this would be seen as a sane thing...
Very few people have 1024 cpus, and the ones that do sure would know how
to set 1024 in their kernel config. In fact I would argue to lower it. I
can see the point of keeping it over 64, to give the
more-cpus-than-a-long code more testing, but 512 is too high already..
most people who have ia64 will not have 512 cpus.
(and please don't say "but distributions.." because distributions don't
use the defconfigs anyway)
From: "Chen, Kenneth W" <[email protected]>
> What type of heavy workloads have you measured? Including db transaction
> processing and decision making workloads?
I haven't used a db transaction processing benchmark, but I have used other
workloads with large process counts and high context-switch rates.
> > The potential
> > extra cachemiss seems to be lost in the noise. The for_each_*cpu()
> > macros are relatively efficient in skipping past zeroed cpumask bits.
> > Workloads that impose higher loads on the CPU Scheduler tend to
> > bottleneck on non-Scheduler parts of the kernel, and it's the Scheduler
> > which makes the principal use of the cpumask_t, so these extra
> > cachemiss inefficiencies and extra CPU cycles to scan zero mask words
> > just get lost in the general system overhead.
>
> I found above claims are generally false for workload that puts tons
> of pressure on CPU cache, especially with db workload. Typically
> for db workload, the working set in user space is so large that making
> a trip into the kernel has far large secondary effect then the primary
> cache miss occurred in the kernel. In other word, cache lines evicted
> by the kernel code have far larger impact to the overall application
> performance and leads to lower overall lower system performance. So
> when you say "get lost in the general system overhead", did you consider
> the secondary effect it does to the application performance?
The current default is 512p, which is 8 words -- a cacheline. Increasing to
1024p adds an additional 8 words -- one cacheline -- to the cpumask_t. I
doubt you're going to see a performance regression on your db transaction
processing benchmark because of an additional cachemiss during active or
passive load-balancing.
I agree that throughout the kernel we ought to be aware of increasing
cachemisses and the lengthening code paths, but I don't believe this
particular one is some evil that needs to be suppressed. We have far more
micro-performance-impacting algorithms and data structures in the kernel right
now that we ought to consider -- e.g., cache coloring conflicts with the
struct runqueue -- as well as the obvious algorithm tweaks that greatly affect
processor assignments -- e.g., whether or not to call wake_idle().
> What we found is going from NR_CPU = 64 to 128, it has small performance
> impact to db transaction processing workload. Though I have not measured
> difference between 128 to 1024.
Going from 64 (one word) to >64 (an array of words) produces a qualitative
change to the emitted code in how the cpumask_t is passed in calling sequences
and how it is manipulated. I completely understand that you can detect a
small performance regression between 64 and 128. I just don't believe you can
conclude that going from 512 to 1024 will exhibit a similar measurable
regression.
John Hawkes
>> Increase the generic ia64 config from its current max of 512 CPUs to a
>> new max of 1024 CPUs.
>
>I wonder why this would be seen as a sane thing...
>Very few people have 1024 cpus, and the ones that do sure would know how
>to set 1024 in their kernel config. In fact I would argue to lower it. I
>can see the point of keeping it over 64, to give the
>more-cpus-than-a-long code more testing, but 512 is too high already..
>most people who have ia64 will not have 512 cpus.
Would it be impossibly hard to make the >64 cpus case (when the code
switches from a single word to an array) be dependent on a boot-time
variable? If we could, then the defconfig could just say 128, and
users of monster machines would just boot with "maxcpus=4096" to increase
the size of the bitmask arrays.
-Tony
On Fri, 2006-01-06 at 09:19 -0800, Luck, Tony wrote:
> >> Increase the generic ia64 config from its current max of 512 CPUs to a
> >> new max of 1024 CPUs.
> >
> >I wonder why this would be seen as a sane thing...
> >Very few people have 1024 cpus, and the ones that do sure would know how
> >to set 1024 in their kernel config. In fact I would argue to lower it. I
> >can see the point of keeping it over 64, to give the
> >more-cpus-than-a-long code more testing, but 512 is too high already..
> >most people who have ia64 will not have 512 cpus.
>
> Would it be impossibly hard to make the >64 cpus case (when the code
> switches from a single word to an array) be dependent on a boot-time
> variable? If we could, then the defconfig could just say 128, and
> users of monster machines would just boot with "maxcpus=4096" to increase
> the size of the bitmask arrays.
variable size runtime cpu count.. that's beside the point for this
change though. The *default* shouldn't be insanely high, but more
represent a "typical" setup. The *maximum* is of course an entirely
different thing. So.. I'd say lower the default from 512 to 128 (just to
get the default to be the "complex case", which has benefits). Going to
1024 for a per user default is silly. Just about everyone will change it
anyway to something lower. And distros set their own regardless.
On Fri, Jan 06, 2006 at 09:19:28AM -0800, Luck, Tony wrote:
> Would it be impossibly hard to make the >64 cpus case (when the code
> switches from a single word to an array) be dependent on a boot-time
> variable? If we could, then the defconfig could just say 128, and
> users of monster machines would just boot with "maxcpus=4096" to increase
> the size of the bitmask arrays.
Why can't we keep the default below 64? Surely the 0.1% of the market
which needs more than 64 cpus can recompile their kernel ...
>Why can't we keep the default below 64? Surely the 0.1% of the market
>which needs more than 64 cpus can recompile their kernel ...
I suppose that depends on your expectations from defconfig. In my
mind its the one that builds into a kernel that will boot and run
on just about any box. People who want to get a bit of extra performance
will do the re-compilation to strip out the bits that they don't want
and tune down limits that are set higher than they need. I only
ever boot a defconfig kernel to check that it still works, my systems
all run tiger_defconfig/zx1_defconfig based most of the time. But
perhaps I'm the weird one?
There are quite a few >16 socket boxes out there, which will give
you >64 cpus with Montecito ... so I don't think that the >64 cpu
system is going to remain in the noise for long.
-Tony
On Fri, Jan 06, 2006 at 09:45:20AM -0800, Luck, Tony wrote:
> I suppose that depends on your expectations from defconfig. In my
> mind its the one that builds into a kernel that will boot and run
> on just about any box. People who want to get a bit of extra performance
> will do the re-compilation to strip out the bits that they don't want
> and tune down limits that are set higher than they need. I only
You can use that argument to set the CPU limit low too -- since a kernel
with a CPU limit lower than the number of CPUs in the box will just ignore
the additional ones, people who want to get the additional performance
will tune limits that are set lower than they need ;-)
> ever boot a defconfig kernel to check that it still works, my systems
> all run tiger_defconfig/zx1_defconfig based most of the time. But
> perhaps I'm the weird one?
>
> There are quite a few >16 socket boxes out there, which will give
> you >64 cpus with Montecito ... so I don't think that the >64 cpu
> system is going to remain in the noise for long.
I bet the number of 32-way+ boxes is lost in the noise compared to the
number of 1-, 2- and 4-way boxes sold. Not that HP trust me with that
kind of sales data ;-)
On Fri, 6 Jan 2006, Matthew Wilcox wrote:
> On Fri, Jan 06, 2006 at 09:45:20AM -0800, Luck, Tony wrote:
> > I suppose that depends on your expectations from defconfig. In my
> > mind its the one that builds into a kernel that will boot and run
> > on just about any box. People who want to get a bit of extra performance
> > will do the re-compilation to strip out the bits that they don't want
> > and tune down limits that are set higher than they need. I only
>
> You can use that argument to set the CPU limit low too -- since a kernel
> with a CPU limit lower than the number of CPUs in the box will just ignore
> the additional ones, people who want to get the additional performance
> will tune limits that are set lower than they need ;-)
The dicey thing in all of this is that the generic kernels will be used
for the certification of applications. If the cpu limit is too low then
applications will simply not be certified for these high processor counts.
One may encounter problems if the app is then run with a higher processor
count.
> > There are quite a few >16 socket boxes out there, which will give
> > you >64 cpus with Montecito ... so I don't think that the >64 cpu
> > system is going to remain in the noise for long.
>
> I bet the number of 32-way+ boxes is lost in the noise compared to the
> number of 1-, 2- and 4-way boxes sold. Not that HP trust me with that
> kind of sales data ;-)
I think it is wrong to count the number of boxes. It is the number of
users that is of major interest. These small boxes typically have one
user. The larger boxes have whole communities working with them. The
number of users of very large systems can potentially go into the 10
thousands of users per installation.
Moreover these larger boxes significantly impact the technological
progress of humanity on the planet. These large boxes run hurricane
simulations, do complex visualization necessary for bio technological
advances, cosmological simulations, other physic stuff etc etc.
On Fri, Jan 06, 2006 at 10:04:56AM -0800, Christoph Lameter wrote:
> On Fri, 6 Jan 2006, Matthew Wilcox wrote:
> > You can use that argument to set the CPU limit low too -- since a kernel
> > with a CPU limit lower than the number of CPUs in the box will just ignore
> > the additional ones, people who want to get the additional performance
> > will tune limits that are set lower than they need ;-)
>
> The dicey thing in all of this is that the generic kernels will be used
> for the certification of applications. If the cpu limit is too low then
Why on earth would somebody do that?
On Fri, 6 Jan 2006, Christoph Lameter wrote:
> On Fri, 6 Jan 2006, Matthew Wilcox wrote:
>
> > On Fri, Jan 06, 2006 at 09:45:20AM -0800, Luck, Tony wrote:
> > > I suppose that depends on your expectations from defconfig. In my
> > > mind its the one that builds into a kernel that will boot and run
> > > on just about any box. People who want to get a bit of extra performance
> > > will do the re-compilation to strip out the bits that they don't want
> > > and tune down limits that are set higher than they need. I only
> >
> > You can use that argument to set the CPU limit low too -- since a kernel
> > with a CPU limit lower than the number of CPUs in the box will just ignore
> > the additional ones, people who want to get the additional performance
> > will tune limits that are set lower than they need ;-)
>
> The dicey thing in all of this is that the generic kernels will be used
> for the certification of applications. If the cpu limit is too low then
> applications will simply not be certified for these high processor counts.
> One may encounter problems if the app is then run with a higher processor
> count.
Do you equate a 'defconfig' kernel with a generic kernel?
I would expect certs to be done on vendor kernels, and as
Arjan has suggested, they will have their own configs,
not defconfig.
> > > There are quite a few >16 socket boxes out there, which will give
> > > you >64 cpus with Montecito ... so I don't think that the >64 cpu
> > > system is going to remain in the noise for long.
> >
> > I bet the number of 32-way+ boxes is lost in the noise compared to the
> > number of 1-, 2- and 4-way boxes sold. Not that HP trust me with that
> > kind of sales data ;-)
>
> I think it is wrong to count the number of boxes. It is the number of
> users that is of major interest. These small boxes typically have one
> user. The larger boxes have whole communities working with them. The
> number of users of very large systems can potentially go into the 10
> thousands of users per installation.
>
> Moreover these larger boxes significantly impact the technological
> progress of humanity on the planet. These large boxes run hurricane
> simulations, do complex visualization necessary for bio technological
> advances, cosmological simulations, other physic stuff etc etc.
--
~Randy
On Fri, Jan 06, 2006 at 09:45:20AM -0800, Luck, Tony wrote:
> >Why can't we keep the default below 64? Surely the 0.1% of the market
> >which needs more than 64 cpus can recompile their kernel ...
>
> I suppose that depends on your expectations from defconfig. In my
> mind its the one that builds into a kernel that will boot and run
> on just about any box. People who want to get a bit of extra performance
> will do the re-compilation to strip out the bits that they don't want
> and tune down limits that are set higher than they need. I only
> ever boot a defconfig kernel to check that it still works, my systems
> all run tiger_defconfig/zx1_defconfig based most of the time. But
> perhaps I'm the weird one?
>...
defconfig's are usually not intended for running on all supported
machines, they are more a base for compile-tests and a starting point
for building an own configuration.
If you intend to use the ia64 defconfig in a different way I don't
see any strong point against it.
But if you seriously want a defconfig "that builds into a kernel that
will boot and run on just about any box", please change your defconfig
to ITANIUM=y,MCKINLEY=n. "People who want to get a bit of extra
performance" can still change their configuration to omit support for
the original Itanium.
> -Tony
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
On Fri, 6 Jan 2006, Randy.Dunlap wrote:
> > The dicey thing in all of this is that the generic kernels will be used
> > for the certification of applications. If the cpu limit is too low then
> > applications will simply not be certified for these high processor counts.
> > One may encounter problems if the app is then run with a higher processor
> > count.
>
> Do you equate a 'defconfig' kernel with a generic kernel?
>
> I would expect certs to be done on vendor kernels, and as
> Arjan has suggested, they will have their own configs,
> not defconfig.
Vendors look for the upstream defaults and orient themselves on the
defconfig. It is best to have as much common code and configurations as
possible.
> Vendors look for the upstream defaults and orient themselves on the
> defconfig.
they do? That's news to me. I've worked at a vendor for almost 5 years,
3 1/2 years of which I was the person who decided on the configs (with
external input of course). In those 3 1/2 years I *never* looked at
defconfig. *never*. And I don't expect other vendor kernel owners to do
things differently; when a config option needs deciding you look at the
description and pick a good value. That's it. Defconfig doesn't matter.
On Gwe, 2006-01-06 at 19:59 +0100, Arjan van de Ven wrote:
> things differently; when a config option needs deciding you look at the
> description and pick a good value. That's it. Defconfig doesn't matter.
defconfig is 'Linus config', and he's stated as much before along with
polite hints that this wasn't going to change. The vendor default config
s are actually far saner for most users.
Do vendors look at defconfig - not really, much more interesting is
every other vendors config choices 8)
Alan
On Fri, 6 Jan 2006, Alan Cox wrote:
> On Gwe, 2006-01-06 at 19:59 +0100, Arjan van de Ven wrote:
> > things differently; when a config option needs deciding you look at the
> > description and pick a good value. That's it. Defconfig doesn't matter.
>
> defconfig is 'Linus config', and he's stated as much before along with
> polite hints that this wasn't going to change. The vendor default config
> s are actually far saner for most users.
Right (Linus's config) for i386 (& ppc64 ?).
Probably not his for ia64 or Arm...
> Do vendors look at defconfig - not really, much more interesting is
> every other vendors config choices 8)
--
~Randy
On Fri, 2006-01-06 at 10:59 -0800, Arjan van de Ven wrote:
>
> > Vendors look for the upstream defaults and orient themselves on the
> > defconfig.
>
> they do? That's news to me. I've worked at a vendor for almost 5
> years,
> 3 1/2 years of which I was the person who decided on the configs (with
> external input of course). In those 3 1/2 years I *never* looked at
> defconfig. *never*. And I don't expect other vendor kernel owners to
> do
> things differently; when a config option needs deciding you look at
> the
> description and pick a good value. That's it. Defconfig doesn't
> matter.
>
IMO something like 128 is a good number as a default for most IA-64
machines. As Arjan said above, OSVs almost always have their own
reasons and input to choose this number. And lately there is a trend of
having at least two kernels, one for mostly used platforms and the other
one for bigger configurations.
-rohit
On Fri, Jan 06, 2006 at 10:37:33AM -0800, Christoph Lameter wrote:
> On Fri, 6 Jan 2006, Randy.Dunlap wrote:
>
> > > The dicey thing in all of this is that the generic kernels will be used
> > > for the certification of applications. If the cpu limit is too low then
> > > applications will simply not be certified for these high processor counts.
> > > One may encounter problems if the app is then run with a higher processor
> > > count.
> >
> > Do you equate a 'defconfig' kernel with a generic kernel?
> >
> > I would expect certs to be done on vendor kernels, and as
> > Arjan has suggested, they will have their own configs,
> > not defconfig.
>
> Vendors look for the upstream defaults and orient themselves on the
> defconfig. It is best to have as much common code and configurations as
> possible.
As someone who builds vendor kernels, I can say this isn't true (from my
experience at least). When a new config option appears, I look at the
Kconfig, and make a decision. I *never* even look at the defconfig, as
a lot of the time, they are either out of date, or irrelevant.
Dave
When I tried to build a 2.6.15-mm2 kernel with NR_CPUS 1024, and turned
on some of the LOCK DEBUG options:
1160c1160
< # CONFIG_DEBUG_SLAB is not set
---
> CONFIG_DEBUG_SLAB=y
1163,1164c1163,1164
< # CONFIG_DEBUG_SPINLOCK is not set
< # CONFIG_DEBUG_SPINLOCK_SLEEP is not set
---
> CONFIG_DEBUG_SPINLOCK=y
> CONFIG_DEBUG_SPINLOCK_SLEEP=y
then the build failed:
LD .tmp_vmlinux1
arch/ia64/kernel/built-in.o(.init.text+0x68b2): In function `topology_init':
arch/ia64/kernel/ivt.S:1465: undefined reference to `__you_cannot_kmalloc_that_much'
make: *** [.tmp_vmlinux1] Error 1
Backing off to 512 CPUs built ok.
There are a couple of kmalloc() calls in topology_init(). I didn't try
to figure out which one blew up, nor did I try to investigate further.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401
On Wed, 11 Jan 2006, Paul Jackson wrote:
> LD .tmp_vmlinux1
> arch/ia64/kernel/built-in.o(.init.text+0x68b2): In function `topology_init':
> arch/ia64/kernel/ivt.S:1465: undefined reference to `__you_cannot_kmalloc_that_much'
> make: *** [.tmp_vmlinux1] Error 1
>
> Backing off to 512 CPUs built ok.
>
> There are a couple of kmalloc() calls in topology_init(). I didn't try
> to figure out which one blew up, nor did I try to investigate further.
Here are two patches that might to cure it (used to be in
2.6.15-rc5-mm1 but the cure accepted for the workqueues was to use
alloc_percpu):
From: Christoph Lameter <[email protected]>
The workqueue structure can grow larger than 128k under 2.6.14-rc2 (with
all debugging features enabled on 64 bit platforms) which will make kzalloc
for workqueue structure entries fail. This patch increases the maximum
slab entry size to 256K.
Signed-off-by: Christoph Lameter <[email protected]>
Cc: Manfred Spraul <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
---
include/linux/kmalloc_sizes.h | 2 +-
1 files changed, 1 insertion(+), 1 deletion(-)
diff -puN include/linux/kmalloc_sizes.h~increase-maximum-kmalloc-size-to-256k include/linux/kmalloc_sizes.h
--- 25/include/linux/kmalloc_sizes.h~increase-maximum-kmalloc-size-to-256k Thu Sep 22 15:09:57 2005
+++ 25-akpm/include/linux/kmalloc_sizes.h Thu Sep 22 15:09:57 2005
@@ -19,8 +19,8 @@
CACHE(32768)
CACHE(65536)
CACHE(131072)
-#ifndef CONFIG_MMU
CACHE(262144)
+#ifndef CONFIG_MMU
CACHE(524288)
CACHE(1048576)
#ifdef CONFIG_LARGE_ALLOCS
_
From: Andrew Morton <[email protected]>
Fix instaoops.
Cc: Christoph Lameter <[email protected]>
Cc: Manfred Spraul <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
---
mm/slab.c | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
diff -puN mm/slab.c~increase-maximum-kmalloc-size-to-256k-fix mm/slab.c
--- devel/mm/slab.c~increase-maximum-kmalloc-size-to-256k-fix 2005-09-22 21:19:27.000000000 -0700
+++ devel-akpm/mm/slab.c 2005-09-22 21:19:27.000000000 -0700
@@ -551,8 +551,8 @@ static void **dbg_userword(kmem_cache_t
#define MAX_OBJ_ORDER 13 /* up to 32Mb */
#define MAX_GFP_ORDER 13 /* up to 32Mb */
#elif defined(CONFIG_MMU)
-#define MAX_OBJ_ORDER 5 /* 32 pages */
-#define MAX_GFP_ORDER 5 /* 32 pages */
+#define MAX_OBJ_ORDER 6 /* 64 pages */
+#define MAX_GFP_ORDER 6 /* 64 pages */
#else
#define MAX_OBJ_ORDER 8 /* up to 1Mb */
#define MAX_GFP_ORDER 8 /* up to 1Mb */
_