Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7;
Sender: Emanuele Rocca <emarocca@gmail.com>
Date:   Fri, 20 Oct 2023 11:14:43 +0200
From:   Emanuele Rocca <ema@debian.org>
To:     linux-kernel@vger.kernel.org
Cc:     Michal Hocko <mhocko@suse.com>
Subject: Runtime overhead of PREEMPT_DYNAMIC
Message-ID: <ZTJFA_Ac6nWawIHb@ariel>
Reply-To: ema@debian.org, linux-kernel@vger.kernel.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Precedence: bulk

Hi,

the Kconfig help for PREEMPT_DYNAMIC states the following:

  The runtime overhead is negligible with HAVE_STATIC_CALL_INLINE enabled
  but if runtime patching is not available for the specific architecture
  then the potential overhead should be considered.

I've tried evaluating the practical runtime overhead of PREEMPT_DYNAMIC
on my Ryzen 7 x86 workstation and it does actually seem quite
significant, for certain workloads.

I tried four different workloads: the first two are userspace-heavy (ie: the
CPU spends most of its time executing at the user level, as reported by sar
%user). There is *no* significant performance difference between a kernel with
PREEMPT_DYNAMIC=y and a kernel with PREEMPT_DYNAMIC=n for those:

1) compressing data with `xz -T 0` (less than 1% system)
   dd if=/dev/urandom of=/dev/stdout bs=1M count=1024 | xz --stdout -T 0 > /dev/null
2) building a kernel (10% system) 

The third and fourth benchmarks are kernel-heavy (ie: most of the CPU time is
at the system level, as reported by sar %system). The performance difference
between PREEMPT_DYNAMIC=y and PREEMPT_DYNAMIC=n is quite visible for those:

3) load testing nginx (35% system)
4) tbench (90% system)
   see https://www.samba.org/ftp/tridge/dbench/README

Details of the nginx benchmark:

- 'hey' (https://github.com/rakyll/hey) HTTP load generator version 0.1.4-2
- nginx 1.24.0 serving the default 'Welcome to nginx!' page
- logging to disk disabled with `access_log off` in nginx.conf
- hey -n 2000000 http://localhost/ | grep Requests

Results of the nginx benchmark:

- PREEMPT_DYNAMIC=n 102K rps
- PREEMPT_DYNAMIC=y 96K  rps
- 5% performance degradation with PREEMPT_DYNAMIC

Details of the tbench benchmark:

- tbench version 4.0 https://www.samba.org/ftp/tridge/dbench/
- Run /usr/bin/tbench_srv in a window
- Run /usr/bin/tbench `nproc` in another window

Results of the tbench benchmark:

- PREEMPT_DYNAMIC=n Throughput 3831.82 MB/sec
- PREEMPT_DYNAMIC=y Throughput 3006.54 MB/sec
- 20% performance degradation with PREEMPT_DYNAMIC

All the tests above were performed using Linux 6.5.6 on a AMD Ryzen 7 3700X
8-Core CPU.

Could someone please confirm whether the approach above seems sane and perhaps
try and reproduce the results?

Thanks,
  Emanuele