Hi David,
I've been experiencing a performance regression when running a parallel
compilation (eg, make -j72) on recent kernels.
I bisected it to this commit:
commit b667b867344301e24f21d4a4c844675ff61d89e1
Author: David Howells <[email protected]>
Date: Tue Sep 24 16:09:04 2019 +0100
pipe: Advance tail pointer inside of wait spinlock in pipe_read()
Advance the pipe ring tail pointer inside of wait spinlock in pipe_read()
so that the pipe can be written into with kernel notifications from
contexts where pipe->mutex cannot be taken.
Prior to this commit I got 70% or so thread saturation of cc1 and after
it rarely gets above 15% and would often drop down to 1-2 threads. It
doesn't look like a clean revert either. Looking at upstream, it seems
that some later code changed the wakeup. I'm not really sure how this
all fits into parallelized make.
Best
Jon
On Thu, Jun 18, 2020 at 11:52:55PM +0000, Derrick, Jonathan wrote:
> Hi David,
>
> I've been experiencing a performance regression when running a parallel
> compilation (eg, make -j72) on recent kernels.
I bet you're using a version of make which predates 4.3:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0ddad21d3e99c743a3aa473121dc5561679e26bb
On Thu, 2020-06-18 at 18:05 -0700, Matthew Wilcox wrote:
> On Thu, Jun 18, 2020 at 11:52:55PM +0000, Derrick, Jonathan wrote:
> > Hi David,
> >
> > I've been experiencing a performance regression when running a parallel
> > compilation (eg, make -j72) on recent kernels.
>
> I bet you're using a version of make which predates 4.3:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0ddad21d3e99c743a3aa473121dc5561679e26bb
>
I am!
# make --version
GNU Make 4.2.1
Thank you Matthew!
On Sat, Jun 20, 2020 at 5:53 AM Derrick, Jonathan
<[email protected]> wrote:
>
> On Thu, 2020-06-18 at 18:05 -0700, Matthew Wilcox wrote:
> > On Thu, Jun 18, 2020 at 11:52:55PM +0000, Derrick, Jonathan wrote:
> > > Hi David,
> > >
> > > I've been experiencing a performance regression when running a parallel
> > > compilation (eg, make -j72) on recent kernels.
> >
> > I bet you're using a version of make which predates 4.3:
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0ddad21d3e99c743a3aa473121dc5561679e26bb
> >
>
> I am!
>
> # make --version
> GNU Make 4.2.1
>
>
> Thank you Matthew!
Check your distribution and included patches on top of a vanilla make v4.2.
Debian had some important ones I described in [1].
When proposing "make version 4.3" for tc-build - an opportunity to
build a llvm-toolchain the easy way - we saw different numbers.
I pointed in [1] to the Linus patch Matthew did here.
Personally, with switching to Debian's make version 4.3-3 I have seen
no big differences when using "make -j3" to build Linux v5.7+.
That might be different with "make -j72"...
...can I have SSH access to this machine, please :-),
You forgot to tell which Linux version you use.
If you are interested please look at closed tc-build issue #72 for our analysis.
For tc-build it did not matter - (Debian's) make v4.2 had some
slightly better performance.
BTW, with that pipe improvements in Linux v5.7 I see some better
numbers when using pipebench to benchmark my devices:
Example: SanDisk iSSD 16GB
root# cat /dev/sdb | pipebench > /dev/null
Summary:
Piped 14.91 GB in 00h01m25.20s: 179.23 MB/second
Before: approx. 100MB/s
BTW, I heard of hyperfine benchmark tool the first time when dealing
with make performance (see [2]) in ClangBuiltLinux.
I would like to see some benchmark numbers with hyperfine from you if
you do not mind.
Stay OPEN minded and curious.
Thanks.
- Sedat -
[1] https://github.com/ClangBuiltLinux/tc-build/issues/72
[2] https://github.com/sharkdp/hyperfine
[3] https://github.com/sharkdp/hyperfine/releases