Hi,
the Kconfig help for PREEMPT_DYNAMIC states the following:
The runtime overhead is negligible with HAVE_STATIC_CALL_INLINE enabled
but if runtime patching is not available for the specific architecture
then the potential overhead should be considered.
I've tried evaluating the practical runtime overhead of PREEMPT_DYNAMIC
on my Ryzen 7 x86 workstation and it does actually seem quite
significant, for certain workloads.
I tried four different workloads: the first two are userspace-heavy (ie: the
CPU spends most of its time executing at the user level, as reported by sar
%user). There is *no* significant performance difference between a kernel with
PREEMPT_DYNAMIC=y and a kernel with PREEMPT_DYNAMIC=n for those:
1) compressing data with `xz -T 0` (less than 1% system)
dd if=/dev/urandom of=/dev/stdout bs=1M count=1024 | xz --stdout -T 0 > /dev/null
2) building a kernel (10% system)
The third and fourth benchmarks are kernel-heavy (ie: most of the CPU time is
at the system level, as reported by sar %system). The performance difference
between PREEMPT_DYNAMIC=y and PREEMPT_DYNAMIC=n is quite visible for those:
3) load testing nginx (35% system)
4) tbench (90% system)
see https://www.samba.org/ftp/tridge/dbench/README
Details of the nginx benchmark:
- 'hey' (https://github.com/rakyll/hey) HTTP load generator version 0.1.4-2
- nginx 1.24.0 serving the default 'Welcome to nginx!' page
- logging to disk disabled with `access_log off` in nginx.conf
- hey -n 2000000 http://localhost/ | grep Requests
Results of the nginx benchmark:
- PREEMPT_DYNAMIC=n 102K rps
- PREEMPT_DYNAMIC=y 96K rps
- 5% performance degradation with PREEMPT_DYNAMIC
Details of the tbench benchmark:
- tbench version 4.0 https://www.samba.org/ftp/tridge/dbench/
- Run /usr/bin/tbench_srv in a window
- Run /usr/bin/tbench `nproc` in another window
Results of the tbench benchmark:
- PREEMPT_DYNAMIC=n Throughput 3831.82 MB/sec
- PREEMPT_DYNAMIC=y Throughput 3006.54 MB/sec
- 20% performance degradation with PREEMPT_DYNAMIC
All the tests above were performed using Linux 6.5.6 on a AMD Ryzen 7 3700X
8-Core CPU.
Could someone please confirm whether the approach above seems sane and perhaps
try and reproduce the results?
Thanks,
Emanuele
On Fri, 2023-10-20 at 11:14 +0200, Emanuele Rocca wrote:
>
> Results of the tbench benchmark:
>
> - PREEMPT_DYNAMIC=n Throughput 3831.82 MB/sec
> - PREEMPT_DYNAMIC=y Throughput 3006.54 MB/sec
> - 20% performance degradation with PREEMPT_DYNAMIC
>
> All the tests above were performed using Linux 6.5.6 on a AMD Ryzen 7 3700X
> 8-Core CPU.
>
> Could someone please confirm whether the approach above seems sane and perhaps
> try and reproduce the results?
FWIW I can't reproduce anything like those crazy tbench numbers. It's
neither free nor crazy expensive here. It's kinda hard to imagine the
author's mailbox surviving submission day were it _that_ horrid, surely
something is amiss.
i7-4790 (quad+smt) avg cmdline
6.5.8-voluntary 3685.08 3679.93 3704.98 3689.99 1.000
6.5.8-dynamic 3571.62 3568.61 3550.55 3563.59 .965
3651.37 3599.87 3615.18 3622.14 .981 preempt=none
3459.58 3514.09 3539.88 3504.51 .949 preempt=full
voluntary (my usual)
CONFIG_PREEMPT_VOLUNTARY_BUILD=y
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
# CONFIG_PREEMPT_DYNAMIC is not set
dynamic (same config, just flip dynamic switch)
CONFIG_PREEMPT_BUILD=y
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y
CONFIG_PREEMPT_DYNAMIC=y
Hi Mike,
On 2023-10-20 02:42, Mike Galbraith wrote:
> It's kinda hard to imagine the author's mailbox surviving submission
> day were it _that_ horrid, surely something is amiss.
Thanks for trying to reproduce! Let's see if we can find out what's
wrong in my setup then.
> voluntary (my usual)
> CONFIG_PREEMPT_VOLUNTARY_BUILD=y
> # CONFIG_PREEMPT_NONE is not set
> CONFIG_PREEMPT_VOLUNTARY=y
> # CONFIG_PREEMPT is not set
> # CONFIG_PREEMPT_DYNAMIC is not set
>
> dynamic (same config, just flip dynamic switch)
> CONFIG_PREEMPT_BUILD=y
> # CONFIG_PREEMPT_NONE is not set
> CONFIG_PREEMPT_VOLUNTARY=y
> # CONFIG_PREEMPT is not set
> CONFIG_PREEMPT_COUNT=y
> CONFIG_PREEMPTION=y
> CONFIG_PREEMPT_DYNAMIC=y
Here is the full diff between the kernel configurations I used. The only
change I made was setting CONFIG_PREEMPT_DYNAMIC=n, everything else was
a consequence of that AFAICT, but please do let me know if you see
anything that shouldn't be there.
--- config-6.5.0-0.preempt-dynamic-amd64 2023-10-11 15:30:02.000000000 +0200
+++ config-6.5.0-0.a.test-amd64 2023-10-11 14:30:02.000000000 +0200
@@ -125,13 +125,11 @@
CONFIG_BPF_LSM=y
# end of BPF subsystem
-CONFIG_PREEMPT_BUILD=y
+CONFIG_PREEMPT_VOLUNTARY_BUILD=y
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
-CONFIG_PREEMPT_COUNT=y
-CONFIG_PREEMPTION=y
-CONFIG_PREEMPT_DYNAMIC=y
+# CONFIG_PREEMPT_DYNAMIC is not set
# CONFIG_SCHED_CORE is not set
#
@@ -156,11 +154,9 @@
# RCU Subsystem
#
CONFIG_TREE_RCU=y
-CONFIG_PREEMPT_RCU=y
# CONFIG_RCU_EXPERT is not set
CONFIG_TREE_SRCU=y
CONFIG_TASKS_RCU_GENERIC=y
-CONFIG_TASKS_RCU=y
CONFIG_TASKS_RUDE_RCU=y
CONFIG_TASKS_TRACE_RCU=y
CONFIG_RCU_STALL_COMMON=y
@@ -1009,7 +1005,11 @@
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_PADATA=y
CONFIG_ASN1=y
-CONFIG_UNINLINE_SPIN_UNLOCK=y
+CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
+CONFIG_INLINE_READ_UNLOCK=y
+CONFIG_INLINE_READ_UNLOCK_IRQ=y
+CONFIG_INLINE_WRITE_UNLOCK=y
+CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
CONFIG_ARCH_SUPPORTS_ATOMIC_RMW=y
CONFIG_MUTEX_SPIN_ON_OWNER=y
CONFIG_RWSEM_SPIN_ON_OWNER=y
@@ -5395,7 +5395,6 @@
CONFIG_MEDIA_CEC_SUPPORT=y
# CONFIG_CEC_CH7322 is not set
# CONFIG_CEC_CROS_EC is not set
-# CONFIG_CEC_GPIO is not set
CONFIG_CEC_SECO=m
# CONFIG_CEC_SECO_RC is not set
CONFIG_USB_PULSE8_CEC=m
@@ -10597,7 +10596,6 @@
# end of Scheduler Debugging
# CONFIG_DEBUG_TIMEKEEPING is not set
-CONFIG_DEBUG_PREEMPT=y
#
# Lock Debugging (spinlocks, mutexes, etc...)
@@ -10696,7 +10694,6 @@
# CONFIG_FUNCTION_PROFILER is not set
CONFIG_STACK_TRACER=y
# CONFIG_IRQSOFF_TRACER is not set
-# CONFIG_PREEMPT_TRACER is not set
# CONFIG_SCHED_TRACER is not set
# CONFIG_HWLAT_TRACER is not set
# CONFIG_OSNOISE_TRACER is not set
On Fri, 2023-10-20 at 15:40 +0200, Emanuele Rocca wrote:
> Hi Mike,
>
> Here is the full diff between the kernel configurations I used. The only
> change I made was setting CONFIG_PREEMPT_DYNAMIC=n, everything else was
> a consequence of that AFAICT, but please do let me know if you see
> anything that shouldn't be there.
>
> --- config-6.5.0-0.preempt-dynamic-amd64 2023-10-11 15:30:02.000000000 +0200
> +++ config-6.5.0-0.a.test-amd64 2023-10-11 14:30:02.000000000 +0200
>
> @@ -10597,7 +10596,6 @@
> # end of Scheduler Debugging
>
> # CONFIG_DEBUG_TIMEKEEPING is not set
> -CONFIG_DEBUG_PREEMPT=y
>
> #
> # Lock Debugging (spinlocks, mutexes, etc...)
Seems you had also turned on DEBUG_PREEMPT in the dynamic setup, which
adds some overhead.. but not a metric ton.
-Mike
On Fri, 2023-10-20 at 16:11 +0200, Mike Galbraith wrote:
> On Fri, 2023-10-20 at 15:40 +0200, Emanuele Rocca wrote:
> > Hi Mike,
> >
> > Here is the full diff between the kernel configurations I used. The only
> > change I made was setting CONFIG_PREEMPT_DYNAMIC=n, everything else was
> > a consequence of that AFAICT, but please do let me know if you see
> > anything that shouldn't be there.
> >
> > --- config-6.5.0-0.preempt-dynamic-amd64 2023-10-11 15:30:02.000000000 +0200
> > +++ config-6.5.0-0.a.test-amd64 2023-10-11 14:30:02.000000000 +0200
> >
> > @@ -10597,7 +10596,6 @@
> > # end of Scheduler Debugging
> >
> > # CONFIG_DEBUG_TIMEKEEPING is not set
> > -CONFIG_DEBUG_PREEMPT=y
> >
> > #
> > # Lock Debugging (spinlocks, mutexes, etc...)
>
> Seems you had also turned on DEBUG_PREEMPT in the dynamic setup, which
> adds some overhead.. but not a metric ton.
Hm, I don't recall the overhead as being that bad, but thar she blows.
i7-4790 avg cmdline
6.5.8-voluntary 3685.08 3679.93 3704.98 3689.99 1.000
6.5.8-dynamic 3571.62 3568.61 3550.55 3563.59 .965
3052.06 3032.74 3019.93 3034.91 .822 +DEBUG_PREEMPT
3651.37 3599.87 3615.18 3622.14 .981 preempt=none
3459.58 3514.09 3539.88 3504.51 .949 preempt=full
On 2023-10-20 04:34, Mike Galbraith wrote:
> Hm, I don't recall the overhead as being that bad, but thar she blows.
>
> i7-4790 avg cmdline
> 6.5.8-voluntary 3685.08 3679.93 3704.98 3689.99 1.000
> 6.5.8-dynamic 3571.62 3568.61 3550.55 3563.59 .965
> 3052.06 3032.74 3019.93 3034.91 .822 +DEBUG_PREEMPT
> 3651.37 3599.87 3615.18 3622.14 .981 preempt=none
> 3459.58 3514.09 3539.88 3504.51 .949 preempt=full
Perfect, that's in line with my measurements (~20% overhead). Debian
kernels have CONFIG_DEBUG_PREEMPT=y, it seems wise to set it to 'n'
instead. Thank you.