From: Austin S. Hemmelgarn <[email protected]>
This patch adds Kconfig options to allow optimization for AMD family
10h, AMD Bulldozer, and AMD Piledriver derived CPU's in version
3.12-rc2. These provide noticeable improvements over the K8 config
option, and allow the kernel to take full advantage of AMD specific
instruction set extensions, such as ABM, LZCNT, and POPCNT.
Signed-Off-By: Austin S. Hemmelgarn <[email protected]>
---
diff -uprN linux/arch/x86/Kconfig.cpu linux-new/arch/x86/Kconfig.cpu
--- linux/arch/x86/Kconfig.cpu 2013-09-23 18:41:09.000000000 -0400
+++ linux-new/arch/x86/Kconfig.cpu 2013-09-28 21:31:35.782173832 -0400
@@ -155,9 +155,9 @@ config MK7
flags to GCC.
config MK8
- bool "Opteron/Athlon64/Hammer/K8"
+ bool "Old Opteron/Athlon64/Hammer/K8"
---help---
- Select this for an AMD Opteron or Athlon64 Hammer-family processor.
+ Select this for an AMD old 64-bit Opteron or Athlon64 Hammer-family processor.
Enables use of some extended instructions, and passes appropriate
optimization flags to GCC.
@@ -269,6 +269,57 @@ config MATOM
accordingly optimized code. Use a recent GCC with specific Atom
support in order to fully benefit from selecting this option.
+config MAMDFAM10
+ bool "AMD Family 10h (Athlon II, Phenom II, and Opteron)"
+ depends on X86_64
+ ---help---
+
+ Select this for AMD Family 10h processors.
+ This includes Athlon II, Phenom II, early third-generation
+ Opterons, and a number of other Socket AM2, AM2+, AM3, and
+ Socket F processors. CPU's in this series show cpu family
+ 16 in /proc/cpuinfo. Note that a kernel compiled with this
+ option may not run correctly on non-AMD processors.
+
+config MBULLDOZER
+ bool "AMD Bulldozer (FX and Opteron)"
+ depends on X86_64
+ ---help---
+
+ Select this for AMD Buldozer microarchitecture processors.
+ This includes the following CPUs:
+ FX-41x0
+ FX-61x0
+ FX-6200
+ FX-81x0
+ Opteron 32xx
+ Opteron 42xx
+ Opteron 62xx
+
+ Note that a kernel compiled with this option may not run
+ correctly on non-Bulldozer derived processors, if it runs at all.
+
+config MPILEDRIVER
+ bool "AMD Piledriver (FX, APU, and Opteron)"
+ depends on X86_64
+ ---help---
+
+ Select this for AMD Piledriver microarchitecture processors.
+ This includes the Following CPUs:
+ 'Trinity' APUs
+ 'Richland' APUs
+ FX-43xx
+ FX-63xx
+ FX-83xx
+ FX-9370
+ FX-9590
+ Opteron 33xx
+ Opteron 43xx
+ Opteron 63xx
+
+ Note that a kernel compiled with this option may not run
+ correctly on non-Piledriver derived processors, if it runs at all.
+
config GENERIC_CPU
bool "Generic-x86-64"
depends on X86_64
@@ -300,7 +351,7 @@ config X86_INTERNODE_CACHE_SHIFT
config X86_L1_CACHE_SHIFT
int
default "7" if MPENTIUM4 || MPSC
- default "6" if MK7 || MK8 || MPENTIUMM || MCORE2 || MATOM || MVIAC7 || X86_GENERIC || GENERIC_CPU
+ default "6" if MK7 || MK8 || MPENTIUMM || MCORE2 || MATOM || MVIAC7 || MAMDFAM10 || MBULLDOZER || MPILEDRIVER || X86_GENERIC || GENERIC_CPU
default "4" if MELAN || M486 || MGEODEGX1
default "5" if MWINCHIP3D || MWINCHIPC6 || MCRUSOE || MEFFICEON || MCYRIXIII || MK6 || MPENTIUMIII || MPENTIUMII || M686 || M586MMX || M586TSC || M586 || MVIAC3_2 || MGEODE_LX
diff -uprN linux/arch/x86/Makefile linux-new/arch/x86/Makefile
--- linux/arch/x86/Makefile 2013-09-23 18:41:09.000000000 -0400
+++ linux-new/arch/x86/Makefile 2013-09-28 21:20:41.869697978 -0400
@@ -68,6 +68,12 @@ else
$(call cc-option,-march=core2,$(call cc-option,-mtune=generic))
cflags-$(CONFIG_MATOM) += $(call cc-option,-march=atom) \
$(call cc-option,-mtune=atom,$(call cc-option,-mtune=generic))
+ cflags-$(CONFIG_AMDFAM10) += $(call cc-option,-march=amdfam10) \
+ $(call cc-option,-mtune=amdfam10,$(call cc-option,-mtune=generic))
+ cflags-$(CONFIG_MBULLCOZER) += $(call cc-option,-march=bdver1) \
+ $(call cc-option,-mtune=bdver1,$(call cc-option,-mtune=generic))
+ cflags-$(CONFIG_MPILEDRIVER) += $(call cc-option,-march=bdver2) \
+ $(call cc-option,-mtune=bdver2,$(call cc-option,-mtune=generic))
cflags-$(CONFIG_GENERIC_CPU) += $(call cc-option,-mtune=generic)
KBUILD_CFLAGS += $(cflags-y)
On Sun, Sep 29, 2013 at 01:54:00PM -0400, Austin S Hemmelgarn wrote:
> From: Austin S. Hemmelgarn <[email protected]>
>
> This patch adds Kconfig options to allow optimization for AMD family
> 10h, AMD Bulldozer, and AMD Piledriver derived CPU's in version
> 3.12-rc2. These provide noticeable improvements over the K8 config
> option, and allow the kernel to take full advantage of AMD specific
> instruction set extensions, such as ABM, LZCNT, and POPCNT.
A patch like that keeps popping up every couple of months. Please show
us those noticeable improvements because the guy last time failed to do
so.
And also, keep in mind that distro kernels are built with
CONFIG_GENERIC_CPU so CPU-specific kernels are used only by the small
percentage of people who actually do build their own kernels.
Thanks.
--
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--
On 09/29/2013 02:01 PM, Borislav Petkov wrote:
> On Sun, Sep 29, 2013 at 01:54:00PM -0400, Austin S Hemmelgarn
> wrote:
>> From: Austin S. Hemmelgarn <[email protected]>
>>
>> This patch adds Kconfig options to allow optimization for AMD
>> family 10h, AMD Bulldozer, and AMD Piledriver derived CPU's in
>> version 3.12-rc2. These provide noticeable improvements over the
>> K8 config option, and allow the kernel to take full advantage of
>> AMD specific instruction set extensions, such as ABM, LZCNT, and
>> POPCNT.
>
> A patch like that keeps popping up every couple of months. Please
> show us those noticeable improvements because the guy last time
> failed to do so.
>
> And also, keep in mind that distro kernels are built with
> CONFIG_GENERIC_CPU so CPU-specific kernels are used only by the
> small percentage of people who actually do build their own
> kernels.
>
> Thanks.
>
While I understand that you want decisive proof that it provides an
improvement, does it specifically matter if the option is unused by
most people and doesn't result in a negative performance hit when used?
I will, however, do some intensive profiling over the next week and
get back to you about exact improvements. Just in the past few days
of using it, I have noticed much better response time in time
sensitive user-space code (mostly emulators and multimedia apps).
On Sun, Sep 29, 2013 at 04:41:02PM -0400, Austin S Hemmelgarn wrote:
> While I understand that you want decisive proof that it provides an
> improvement, does it specifically matter if the option is unused by
> most people and doesn't result in a negative performance hit when
> used?
Just having the option for no good reason at all is a no-no.
> I will, however, do some intensive profiling over the next week and
> get back to you about exact improvements. Just in the past few days of
> using it, I have noticed much better response time in time sensitive
> user-space code (mostly emulators and multimedia apps).
Yes, we need concrete benchmark numbers from a kernel built with MK8
or even GENERIC vs one built with those new options, most likely
with the one for the box you'll be running the benchmarks on. And no
microbenchmarks please.
--
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--
On 09/29/2013 04:50 PM, Borislav Petkov wrote:
> On Sun, Sep 29, 2013 at 04:41:02PM -0400, Austin S Hemmelgarn
> wrote:
>> While I understand that you want decisive proof that it provides
>> an improvement, does it specifically matter if the option is
>> unused by most people and doesn't result in a negative
>> performance hit when used?
>
> Just having the option for no good reason at all is a no-no.
I'm not saying that should just be included without substantiation, I
simply mean that the reason to include it (as far as I am concerned)
is that it doesn't break anything and provides something useful that
isn't in the kernel already.
Despite this I am still happy to benchmark it to provide some more
concrete proof.
On Sun, Sep 29, 2013 at 05:23:37PM -0400, Austin S Hemmelgarn wrote:
> I'm not saying that should just be included without substantiation, I
> simply mean that the reason to include it (as far as I am concerned)
> is that it doesn't break anything and provides something useful that
> isn't in the kernel already.
If it doesn't bring any performance improvement - and I don't want to
rain on your parade but I think it won't, at least not enough to warrant
a serious look - there's absolutely no reason to add it.
But hey, I'm always open to surprises! So surprise me!
:-)
--
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--
On 2013-09-29 17:30, Borislav Petkov wrote:
> On Sun, Sep 29, 2013 at 05:23:37PM -0400, Austin S Hemmelgarn wrote:
>> I'm not saying that should just be included without substantiation,
>> I simply mean that the reason to include it (as far as I am
>> concerned) is that it doesn't break anything and provides something
>> useful that isn't in the kernel already.
>
> If it doesn't bring any performance improvement - and I don't want
> to rain on your parade but I think it won't, at least not enough to
> warrant a serious look - there's absolutely no reason to add it.
>
> But hey, I'm always open to surprises! So surprise me!
>
> :-)
>
Sorry it took me so long to get back to you about this.
After doing some intensive testing, it appears that there is some
improvement, but it is mostly in the scheduling code and syscalls. Most
sysbench benchmarks only show improvement when running with
more threads than processor cores; however, build jobs appear to be much
improved. Building kernel 3.12-rc2 with allmodconfig using 8 jobs on a
FX-8320 takes 22 minutes and 57 seconds on a kernel with CONFIG_MK8, 21
minutes and 35 seconds on a kernel with CONFIG_GENERIC, and 19 minutes
and 11 seconds on a kernel with CONFIG_PILEDRIVER. I see similar
results for a build of GCC 4.7 (45m1s, 41m39s, and 38m56s(. I would
love to test it on my Athlon II system, but I can't afford to have any
appreciable load on that system because I'm using it as a
router/firewall, based on a look at the low level stuff though, I
believe that it would still be at least a bit better with
CONFIG_MAMDFAM10 than with CONFIG_GENERIC, and certainly better than
CONFIG_MK8 (there are some big differences in the execution pipeline
between K8 and family 10h).
I don't know about you, but that sure seems to be a worthwhile
performance increase to me.
On Thu, Oct 3, 2013 at 5:06 AM, Austin S Hemmelgarn
<[email protected]> wrote:
> improved. Building kernel 3.12-rc2 with allmodconfig using 8 jobs on a FX-8320 takes
>
> 22 minutes and 57 seconds on a kernel with CONFIG_MK8,
> 21 minutes and 35 seconds on a kernel with CONFIG_GENERIC, and
> 19 minutes and 11 seconds on a kernel with CONFIG_PILEDRIVER.
That's certainly noticeable. Surprisingly so. What makes MK8 so bad in
particular, I wonder?
Just out of interest, have you done any profiles on the kernel cost
here to see what it is that makes such a big difference. Because
normally on a kernel build, I see most of the overhead in path lookup.
But that's only true for otherwise optimized builds that don't have
system call auditing etc debugging that spreads the costs out over
everything..
Linus
On Thu, Oct 03, 2013 at 09:27:45AM -0700, Linus Torvalds wrote:
> On Thu, Oct 3, 2013 at 5:06 AM, Austin S Hemmelgarn
> <[email protected]> wrote:
> > improved. Building kernel 3.12-rc2 with allmodconfig using 8 jobs on a FX-8320 takes
> >
> > 22 minutes and 57 seconds on a kernel with CONFIG_MK8,
> > 21 minutes and 35 seconds on a kernel with CONFIG_GENERIC, and
> > 19 minutes and 11 seconds on a kernel with CONFIG_PILEDRIVER.
>
> That's certainly noticeable. Surprisingly so. What makes MK8 so bad in
> particular, I wonder?
>
> Just out of interest, have you done any profiles on the kernel cost
> here to see what it is that makes such a big difference. Because
> normally on a kernel build, I see most of the overhead in path lookup.
> But that's only true for otherwise optimized builds that don't have
> system call auditing etc debugging that spreads the costs out over
> everything..
Yeah, I was having some doubts about the numbers above so I ran my own
benchmarking, machine is a Piledriver box:
vendor_id : AuthenticAMD
cpu family : 21
model : 2
model name : AMD FX(tm)-8350 Eight-Core Processor
stepping : 0
and I don't really see any of those improvements above. Actually,
-march=bdver2 is even slightly worse in comparison to mk8.
And the workload is of building a config specific to that machine but
allmodconfig looks very similar, the numbers being simply higher.
$ zgrep MK8 /proc/config.gz
CONFIG_MK8=y
/home/boris/bin/perf stat --repeat 10 -a --sync --pre /home/boris/kernel/pre-build-kernel.sh make -s -j64 bzImage
Performance counter stats for 'make -s -j64 bzImage' (10 runs):
1081808.628840 task-clock # 7.996 CPUs utilized ( +- 0.06% ) [100.00%]
1,203,753 context-switches # 0.001 M/sec ( +- 0.04% ) [100.00%]
48,748 cpu-migrations # 0.045 K/sec ( +- 0.59% ) [100.00%]
31,145,439 page-faults # 0.029 M/sec ( +- 0.00% )
3,836,736,801,500 cycles # 3.547 GHz ( +- 0.03% ) [100.00%]
957,386,966,493 stalled-cycles-frontend # 24.95% frontend cycles idle ( +- 0.06% ) [100.00%]
218,581,249,251 stalled-cycles-backend # 5.70% backend cycles idle ( +- 0.06% ) [100.00%]
2,466,632,641,972 instructions # 0.64 insns per cycle
# 0.39 stalled cycles per insn ( +- 0.00% ) [100.00%]
537,749,333,838 branches # 497.084 M/sec ( +- 0.00% ) [100.00%]
27,802,940,176 branch-misses # 5.17% of all branches ( +- 0.00% )
135.292843025 seconds time elapsed ( +- 0.06% )
$ zgrep PILEDRIVER /proc/config.gz
CONFIG_MPILEDRIVER=y
/home/boris/bin/perf stat --repeat 10 -a --sync --pre /home/boris/kernel/pre-build-kernel.sh make -s -j64 bzImage
Performance counter stats for 'make -s -j64 bzImage' (10 runs):
1085723.230470 task-clock # 7.996 CPUs utilized ( +- 0.10% ) [100.00%]
1,204,355 context-switches # 0.001 M/sec ( +- 0.10% ) [100.00%]
49,143 cpu-migrations # 0.045 K/sec ( +- 0.76% ) [100.00%]
31,196,575 page-faults # 0.029 M/sec ( +- 0.00% )
3,851,255,065,133 cycles # 3.547 GHz ( +- 0.02% ) [100.00%]
958,840,197,117 stalled-cycles-frontend # 24.90% frontend cycles idle ( +- 0.09% ) [100.00%]
220,260,399,411 stalled-cycles-backend # 5.72% backend cycles idle ( +- 0.04% ) [100.00%]
2,466,701,295,156 instructions # 0.64 insns per cycle
# 0.39 stalled cycles per insn ( +- 0.00% ) [100.00%]
537,992,040,195 branches # 495.515 M/sec ( +- 0.00% ) [100.00%]
27,860,290,286 branch-misses # 5.18% of all branches ( +- 0.00% )
135.784111961 seconds time elapsed ( +- 0.10% )
--
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--
On 2013-10-03 12:57, Borislav Petkov wrote:> On Thu, Oct 03, 2013 at 09:27:45AM -0700, Linus Torvalds wrote:
>> On Thu, Oct 3, 2013 at 5:06 AM, Austin S Hemmelgarn
>> <[email protected]> wrote:
>>> improved. Building kernel 3.12-rc2 with allmodconfig using 8 jobs on a FX-8320 takes
>>>
>>> 22 minutes and 57 seconds on a kernel with CONFIG_MK8,
>>> 21 minutes and 35 seconds on a kernel with CONFIG_GENERIC, and
>>> 19 minutes and 11 seconds on a kernel with CONFIG_PILEDRIVER.
>>
>> That's certainly noticeable. Surprisingly so. What makes MK8 so bad in
>> particular, I wonder?
>>
>> Just out of interest, have you done any profiles on the kernel cost
>> here to see what it is that makes such a big difference. Because
>> normally on a kernel build, I see most of the overhead in path lookup.
>> But that's only true for otherwise optimized builds that don't have
>> system call auditing etc debugging that spreads the costs out over
>> everything..
>
> Yeah, I was having some doubts about the numbers above so I ran my own
> benchmarking, machine is a Piledriver box:
>
> vendor_id : AuthenticAMD
> cpu family : 21
> model : 2
> model name : AMD FX(tm)-8350 Eight-Core Processor
> stepping : 0
>
> and I don't really see any of those improvements above. Actually,
> -march=bdver2 is even slightly worse in comparison to mk8.
>
> And the workload is of building a config specific to that machine but
> allmodconfig looks very similar, the numbers being simply higher.
>
> $ zgrep MK8 /proc/config.gz
> CONFIG_MK8=y
>
> /home/boris/bin/perf stat --repeat 10 -a --sync --pre /home/boris/kernel/pre-build-kernel.sh make -s -j64 bzImage
>
> Performance counter stats for 'make -s -j64 bzImage' (10 runs):
>
> 1081808.628840 task-clock # 7.996 CPUs utilized ( +- 0.06% ) [100.00%]
> 1,203,753 context-switches # 0.001 M/sec ( +- 0.04% ) [100.00%]
> 48,748 cpu-migrations # 0.045 K/sec ( +- 0.59% ) [100.00%]
> 31,145,439 page-faults # 0.029 M/sec ( +- 0.00% )
> 3,836,736,801,500 cycles # 3.547 GHz ( +- 0.03% ) [100.00%]
> 957,386,966,493 stalled-cycles-frontend # 24.95% frontend cycles idle ( +- 0.06% ) [100.00%]
> 218,581,249,251 stalled-cycles-backend # 5.70% backend cycles idle ( +- 0.06% ) [100.00%]
> 2,466,632,641,972 instructions # 0.64 insns per cycle
> # 0.39 stalled cycles per insn ( +- 0.00% ) [100.00%]
> 537,749,333,838 branches # 497.084 M/sec ( +- 0.00% ) [100.00%]
> 27,802,940,176 branch-misses # 5.17% of all branches ( +- 0.00% )
>
> 135.292843025 seconds time elapsed ( +- 0.06% )
>
>
> $ zgrep PILEDRIVER /proc/config.gz
> CONFIG_MPILEDRIVER=y
>
> /home/boris/bin/perf stat --repeat 10 -a --sync --pre /home/boris/kernel/pre-build-kernel.sh make -s -j64 bzImage
>
> Performance counter stats for 'make -s -j64 bzImage' (10 runs):
>
> 1085723.230470 task-clock # 7.996 CPUs utilized ( +- 0.10% ) [100.00%]
> 1,204,355 context-switches # 0.001 M/sec ( +- 0.10% ) [100.00%]
> 49,143 cpu-migrations # 0.045 K/sec ( +- 0.76% ) [100.00%]
> 31,196,575 page-faults # 0.029 M/sec ( +- 0.00% )
> 3,851,255,065,133 cycles # 3.547 GHz ( +- 0.02% ) [100.00%]
> 958,840,197,117 stalled-cycles-frontend # 24.90% frontend cycles idle ( +- 0.09% ) [100.00%]
> 220,260,399,411 stalled-cycles-backend # 5.72% backend cycles idle ( +- 0.04% ) [100.00%]
> 2,466,701,295,156 instructions # 0.64 insns per cycle
> # 0.39 stalled cycles per insn ( +- 0.00% ) [100.00%]
> 537,992,040,195 branches # 495.515 M/sec ( +- 0.00% ) [100.00%]
> 27,860,290,286 branch-misses # 5.18% of all branches ( +- 0.00% )
>
> 135.784111961 seconds time elapsed ( +- 0.10% )
>
Part of the difference between our results may be that I have my entire userspace built with -mtune=bdver2, so less of the time is spent in userspace. Also, the part about using many more threads than cpu cores was with regards to sysbench, not the kernel build, for that I just used 8 jobs in make.
With regards to the differences shown above relative to CONFIG_MK8, that does actually make sense; with CONFIG_MK8, gcc makes very minimal use of extension instructions (afaik, only MMX, SSE, and 3Dnow!), this improves performance slightly on bulldozer derivatives because there are only half as many SSE and FP units as CPU cores (and the scheduler isn't as smart as it could be with regards to that, but that is something for another patch as far as I am concerned).
On Thu, Oct 03, 2013 at 02:12:07PM -0400, Austin S Hemmelgarn wrote:
> Part of the difference between our results may be that I have my
> entire userspace built with -mtune=bdver2, so less of the time is
> spent in userspace.
That still cannot explain the huge difference between building on a
mk8 vs bdver2 kernel. Provided your userspace is the same and only the
kernels are different, I don't see how that happens.
Btw, you could run my workload on your machine, once on an mk8 kernel
and once on a bdver2 one.
The contents of the pre-build-kernel.sh script are:
--
#!/bin/bash
make -s clean
echo 3 > /proc/sys/vm/drop_caches
--
Of course, you need to run this as root.
> Also, the part about using many more threads than cpu cores was with
> regards to sysbench, not the kernel build, for that I just used 8 jobs
> in make.
Overcommitting gcc threads does not bring a really a big difference with
a kernel build.
> With regards to the differences shown above relative to CONFIG_MK8,
> that does actually make sense; with CONFIG_MK8, gcc makes very minimal
> use of extension instructions (afaik, only MMX, SSE, and 3Dnow!), this
> improves performance slightly on bulldozer derivatives because there
> are only half as many SSE and FP units as CPU cores (and the
yes, the FPU is shared between the cores of a compute unit on BD.
> scheduler isn't as smart as it could be with regards to that, but that
> is something for another patch as far as I am concerned).
Again, with unchanging userspace and only differing kernels, you
shouldn't be seeing such a big difference. Besides, the kernel itself
makes very little, if at all, use of FPU instructions.
--
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--