This is quite odd and I've no idea where to start looking for the
cause, but let me describe what I'm seeing and maybe someone can point
me in the right direction.
I'm running SMP 2.6.x kernels on a Athlon 64 X2 4400+
When I build new kernels I use 'make -j' to get both CPU cores busy
and minimize build time.
I've observed that when I use 'make -j 2', 'make -j 3' or 'make -j 4'
only ~half of my CPU resources get used during the build and when I
look at the output it looks exactely like output from plain 'make' for
something like 95% of the build - that is, files get build
sequentially, not in parallel.
However, if I run 'make -j 5' or higher, then both cores get lots of
work to do and utilization of both cores stay close to 100% for almost
the entire build, and the output during the build shows lots of files
from different directories intermixed, so it's clearly building stuf
in parallel.
I find this quite strange since anything from 'make -j 2' and up
should be able to keep both cores resonably busy, but there seems to
be a huge difference between j <= 4 and j > 4.
Another datapoint: This is most pronounced when I am also running the
make process nice'd. Which I usually do in order to be able to use the
machine for other tasks while the build is in progress.
If I don't run the build nice'd then it starts to be resonably
parallel at j >= 3, but still doesn't *really* load both cores before
j >= 5.
Just to be completely clear - a few examples:
This is what I usually run to load both cores well :
nice make -j 5 2>&1 | tee build.log
this however gives me a more or less serial build with only half the
system resources used :
nice make -j 4 2>&1 | tee build.log
Without nice this loads the box somewhat OK (but not completely):
make -j 3 2>&1 | tee build.log
and this pretty much gives me a serial build:
make -j 2 2>&1 | tee build.log
Any good explanations for this behaviour ?
--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html
On 2/21/06, Jesper Juhl <[email protected]> wrote:
> This is quite odd and I've no idea where to start looking for the
> cause, but let me describe what I'm seeing and maybe someone can point
> me in the right direction.
>
> I'm running SMP 2.6.x kernels on a Athlon 64 X2 4400+
>
I should probably mention that the kernel I'm currently running and
observing this behaviour with is 2.6.16-rc4-mm1.
> When I build new kernels I use 'make -j' to get both CPU cores busy
> and minimize build time.
>
> I've observed that when I use 'make -j 2', 'make -j 3' or 'make -j 4'
> only ~half of my CPU resources get used during the build and when I
> look at the output it looks exactely like output from plain 'make' for
> something like 95% of the build - that is, files get build
> sequentially, not in parallel.
> However, if I run 'make -j 5' or higher, then both cores get lots of
> work to do and utilization of both cores stay close to 100% for almost
> the entire build, and the output during the build shows lots of files
> from different directories intermixed, so it's clearly building stuf
> in parallel.
>
> I find this quite strange since anything from 'make -j 2' and up
> should be able to keep both cores resonably busy, but there seems to
> be a huge difference between j <= 4 and j > 4.
>
> Another datapoint: This is most pronounced when I am also running the
> make process nice'd. Which I usually do in order to be able to use the
> machine for other tasks while the build is in progress.
> If I don't run the build nice'd then it starts to be resonably
> parallel at j >= 3, but still doesn't *really* load both cores before
> j >= 5.
>
>
> Just to be completely clear - a few examples:
>
> This is what I usually run to load both cores well :
>
> nice make -j 5 2>&1 | tee build.log
>
> this however gives me a more or less serial build with only half the
> system resources used :
>
> nice make -j 4 2>&1 | tee build.log
>
> Without nice this loads the box somewhat OK (but not completely):
>
> make -j 3 2>&1 | tee build.log
>
> and this pretty much gives me a serial build:
>
> make -j 2 2>&1 | tee build.log
>
>
> Any good explanations for this behaviour ?
>
>
--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html
On Tue, 2006-02-21 at 22:10 +0100, Jesper Juhl wrote:
> I should probably mention that the kernel I'm currently running and
> observing this behaviour with is 2.6.16-rc4-mm1.
...
> > I find this quite strange since anything from 'make -j 2' and up
> > should be able to keep both cores resonably busy, but there seems to
> > be a huge difference between j <= 4 and j > 4.
I've seen the same thing (on Athlon 64x2 64 bit)
but was not sure if it was a problem.
The break point for me seems to be between -j 2 and -j 3
-j 2 = serialized (or the appearance of)
-j 3 = both cores mostly busy
I'm pretty sure with an earlier 2.6 kernel source (but same environment)
I did not see this. I'll start back tracking to earlier kernels
to see if I can identify when this started.
--
Paul Fulghum
Microgate Systems, Ltd
On Tue, 2006-02-21 at 15:42 -0600, Paul Fulghum wrote:
> I'm pretty sure with an earlier 2.6 kernel source (but same environment)
> I did not see this. I'll start back tracking to earlier kernels
> to see if I can identify when this started.
2.6.14 and 2.6.15 keeps both cores busy with -j 2
2.6.16 series appears to build serially with -j 2
--
Paul Fulghum
Microgate Systems, Ltd
On 2/21/06, Paul Fulghum <[email protected]> wrote:
> On Tue, 2006-02-21 at 22:10 +0100, Jesper Juhl wrote:
>
> > I should probably mention that the kernel I'm currently running and
> > observing this behaviour with is 2.6.16-rc4-mm1.
> ...
> > > I find this quite strange since anything from 'make -j 2' and up
> > > should be able to keep both cores resonably busy, but there seems to
> > > be a huge difference between j <= 4 and j > 4.
>
> I've seen the same thing (on Athlon 64x2 64 bit)
> but was not sure if it was a problem.
>
> The break point for me seems to be between -j 2 and -j 3
> -j 2 = serialized (or the appearance of)
> -j 3 = both cores mostly busy
>
> I'm pretty sure with an earlier 2.6 kernel source (but same environment)
> I did not see this. I'll start back tracking to earlier kernels
> to see if I can identify when this started.
>
I know positively that I've seen this with previous 2.6.16-<something>
kernels, but not sure which ones exactely. I just dismissed it as
something that would probably be fixed soon and then today when I
build a few test kernels I noticed it again and thought "ohh, so it
didn't get fixed, better report it".
I don't recall seeing it with 2.6.15 and earlier kernels, but I'm not
at all sure - especially since I only got this Athlon X2 box recently
and the first kernels I ever ran on it were 2.6.15-<something>.
--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html
On Tue, 21 Feb 2006, Jesper Juhl wrote:
> On 2/21/06, Paul Fulghum <[email protected]> wrote:
>> On Tue, 2006-02-21 at 22:10 +0100, Jesper Juhl wrote:
>>
>>> I should probably mention that the kernel I'm currently running and
>>> observing this behaviour with is 2.6.16-rc4-mm1.
>> ...
>>>> I find this quite strange since anything from 'make -j 2' and up
>>>> should be able to keep both cores resonably busy, but there seems to
>>>> be a huge difference between j <= 4 and j > 4.
>>
>> I've seen the same thing (on Athlon 64x2 64 bit)
>> but was not sure if it was a problem.
>>
>> The break point for me seems to be between -j 2 and -j 3
>> -j 2 = serialized (or the appearance of)
>> -j 3 = both cores mostly busy
>>
>> I'm pretty sure with an earlier 2.6 kernel source (but same environment)
>> I did not see this. I'll start back tracking to earlier kernels
>> to see if I can identify when this started.
>>
>
> I know positively that I've seen this with previous 2.6.16-<something>
> kernels, but not sure which ones exactely. I just dismissed it as
> something that would probably be fixed soon and then today when I
> build a few test kernels I noticed it again and thought "ohh, so it
> didn't get fixed, better report it".
>
> I don't recall seeing it with 2.6.15 and earlier kernels, but I'm not
> at all sure - especially since I only got this Athlon X2 box recently
> and the first kernels I ever ran on it were 2.6.15-<something>.
>
> --
> Jesper Juhl <[email protected]>
> Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
> Plain text mails only, please http://www.expita.com/nomime.html
Could it be, simply, that you are now I/O bound with nothing
for these CPUs to do in user-space, being busy handling the
kernel I/O?
Cheers,
Dick Johnson
Penguin : Linux version 2.6.13.4 on an i686 machine (5589.49 BogoMips).
Warning : 98.36% of all statistics are fiction.
_
****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.
Thank you.
On 2/21/06, linux-os (Dick Johnson) <[email protected]> wrote:
>
> On Tue, 21 Feb 2006, Jesper Juhl wrote:
>
> > On 2/21/06, Paul Fulghum <[email protected]> wrote:
> >> On Tue, 2006-02-21 at 22:10 +0100, Jesper Juhl wrote:
> >>
> >>> I should probably mention that the kernel I'm currently running and
> >>> observing this behaviour with is 2.6.16-rc4-mm1.
> >> ...
> >>>> I find this quite strange since anything from 'make -j 2' and up
> >>>> should be able to keep both cores resonably busy, but there seems to
> >>>> be a huge difference between j <= 4 and j > 4.
> >>
> >> I've seen the same thing (on Athlon 64x2 64 bit)
> >> but was not sure if it was a problem.
> >>
> >> The break point for me seems to be between -j 2 and -j 3
> >> -j 2 = serialized (or the appearance of)
> >> -j 3 = both cores mostly busy
> >>
> >> I'm pretty sure with an earlier 2.6 kernel source (but same environment)
> >> I did not see this. I'll start back tracking to earlier kernels
> >> to see if I can identify when this started.
> >>
> >
> > I know positively that I've seen this with previous 2.6.16-<something>
> > kernels, but not sure which ones exactely. I just dismissed it as
> > something that would probably be fixed soon and then today when I
> > build a few test kernels I noticed it again and thought "ohh, so it
> > didn't get fixed, better report it".
> >
> > I don't recall seeing it with 2.6.15 and earlier kernels, but I'm not
> > at all sure - especially since I only got this Athlon X2 box recently
> > and the first kernels I ever ran on it were 2.6.15-<something>.
> >
> > --
> > Jesper Juhl <[email protected]>
> > Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
> > Plain text mails only, please http://www.expita.com/nomime.html
>
> Could it be, simply, that you are now I/O bound with nothing
> for these CPUs to do in user-space, being busy handling the
> kernel I/O?
>
I doubt it.
If that was the case then I should be I/O bound with 'make -j 4' and
even more so with 'make -j 5'. But what I'm seeing is that with 'make
-j 4' (and less) only ~50% of my CPU resources are being used but with
'make -j 5' I get close to 100% CPU utilization.
--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html
On 2/21/06, Paul Fulghum <[email protected]> wrote:
> On Tue, 2006-02-21 at 22:10 +0100, Jesper Juhl wrote:
>
> > I should probably mention that the kernel I'm currently running and
> > observing this behaviour with is 2.6.16-rc4-mm1.
> ...
> > > I find this quite strange since anything from 'make -j 2' and up
> > > should be able to keep both cores resonably busy, but there seems to
> > > be a huge difference between j <= 4 and j > 4.
>
> I've seen the same thing (on Athlon 64x2 64 bit)
> but was not sure if it was a problem.
>
> The break point for me seems to be between -j 2 and -j 3
That's the break point for me as well when I'm not using 'nice'. When
I use 'nice' the break point moves to -j 4/5 .
> -j 2 = serialized (or the appearance of)
> -j 3 = both cores mostly busy
>
> I'm pretty sure with an earlier 2.6 kernel source (but same environment)
> I did not see this. I'll start back tracking to earlier kernels
> to see if I can identify when this started.
>
--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html
On Tue, Feb 21, 2006 at 10:10:46PM +0100, Jesper Juhl wrote:
> On 2/21/06, Jesper Juhl <[email protected]> wrote:
> > This is quite odd and I've no idea where to start looking for the
> > cause, but let me describe what I'm seeing and maybe someone can point
> > me in the right direction.
> >
> > I'm running SMP 2.6.x kernels on a Athlon 64 X2 4400+
> >
>
> I should probably mention that the kernel I'm currently running and
> observing this behaviour with is 2.6.16-rc4-mm1.
Hi Jesper.
Could you please double check that patch below has been applied to your
tree.
Sam
diff-tree 36cbbe5eb9857730768aa5f54ad94d69e0b2133d (from 9f672004ab1a8094bec1785b39ac683ab9eebebc)
Author: Benjamin LaHaise <[email protected]>
Date: Wed Feb 15 15:17:35 2006 -0800
[PATCH] kbuild: revert "fix make -jN with multiple targets with O=..."
Commit 296e0855b0f9a4ec9be17106ac541745a55b2ce1:
"kbuild: fix make -jN with multiple targets with O=..."
causes a ~95% increase in build time for the kernel. Before: 4m21s
after: 8m1.403s. Can we revert this until another approach is found?
Cc: Sam Ravnborg <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
diff --git a/Makefile b/Makefile
index 74d67b2..48d569d 100644
--- a/Makefile
+++ b/Makefile
@@ -104,17 +104,16 @@ ifneq ($(KBUILD_OUTPUT),)
saved-output := $(KBUILD_OUTPUT)
KBUILD_OUTPUT := $(shell cd $(KBUILD_OUTPUT) && /bin/pwd)
$(if $(KBUILD_OUTPUT),, \
$(error output directory "$(saved-output)" does not exist))
-.PHONY: $(MAKECMDGOALS) cdbuilddir
-$(MAKECMDGOALS) _all: cdbuilddir
+.PHONY: $(MAKECMDGOALS)
-cdbuilddir:
+$(filter-out _all,$(MAKECMDGOALS)) _all:
$(if $(KBUILD_VERBOSE:1=),@)$(MAKE) -C $(KBUILD_OUTPUT) \
KBUILD_SRC=$(CURDIR) \
- KBUILD_EXTMOD="$(KBUILD_EXTMOD)" -f $(CURDIR)/Makefile $(MAKECMDGOALS)
+ KBUILD_EXTMOD="$(KBUILD_EXTMOD)" -f $(CURDIR)/Makefile $@
# Leave processing to above invocation of make
skip-makefile := 1
endif # ifneq ($(KBUILD_OUTPUT),)
endif # ifeq ($(KBUILD_SRC),)
On 2/21/06, Sam Ravnborg <[email protected]> wrote:
> On Tue, Feb 21, 2006 at 10:10:46PM +0100, Jesper Juhl wrote:
> > On 2/21/06, Jesper Juhl <[email protected]> wrote:
> > > This is quite odd and I've no idea where to start looking for the
> > > cause, but let me describe what I'm seeing and maybe someone can point
> > > me in the right direction.
> > >
> > > I'm running SMP 2.6.x kernels on a Athlon 64 X2 4400+
> > >
> >
> > I should probably mention that the kernel I'm currently running and
> > observing this behaviour with is 2.6.16-rc4-mm1.
>
> Hi Jesper.
> Could you please double check that patch below has been applied to your
> tree.
>
Hi Sam,
I just checked and that patch is indeed in the tree I'm building
(which is also the one I'm running).
--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html
>
>This is quite odd and I've no idea where to start looking for the
>cause, but let me describe what I'm seeing and maybe someone can point
>me in the right direction.
>When I build new kernels I use 'make -j' to get both CPU cores busy
>and minimize build time.
>
I have seen something similar, but at a different point.
Often, `make -j2` makes make run in a way that only one instance of gcc is
active at a time.
Jan Engelhardt
--