2023-08-08 18:55:28

by Mateusz Guzik

[permalink] [raw]
Subject: hardening and other opts in kernel config used for benchmarking?

Hello,

I have no idea who should be in To: or Cc:, I grabbed names from an
e-mail I got regarding one of previous changes.

Recently I benchmarked a change which added unconditional file
position locking and found a minor regression from it (with profiler
output to justify it):
https://lore.kernel.org/linux-fsdevel/CAHk-=whJtLkYwEFTS9LcRiMjSqq_xswDeXo7hYNWT0Em6nL4Sw@mail.gmail.com/T/#m7c0cd6e913c6295732daea3c88f502bd4724ffb3

However, according to Christian the change was benchmarked by your
machinery and no difference was found.

I briefly poked around and found that used configs have:
CONFIG_RANDOMIZE_KSTACK_OFFSET=y

This is an optional and very expensive hardening feature, my question
is if it was enabled on purpose. The cost comes from adding rdtsc to
every syscall.

Looking at the rest of the config you have a mixed bag (e.g., hardened
usercopy but *no* init_on_alloc) so I genuinely don't know.

Given the high cost of the opt I would suggest removing it, as it
avoidably muddles the waters for single-threaded changes (one way or
the other -- slowdowns can hide and speed ups go unnoticed).

I did not review the whole config.

Any comments? :)

Thanks,
--
Mateusz Guzik <mjguzik gmail.com>


2023-08-10 06:24:25

by Yin, Fengwei

[permalink] [raw]
Subject: Re: hardening and other opts in kernel config used for benchmarking?

Add more LKP people.

On 8/8/23 23:25, Mateusz Guzik wrote:
> Hello,
>
> I have no idea who should be in To: or Cc:, I grabbed names from an
> e-mail I got regarding one of previous changes.
>
> Recently I benchmarked a change which added unconditional file
> position locking and found a minor regression from it (with profiler
> output to justify it):
> https://lore.kernel.org/linux-fsdevel/CAHk-=whJtLkYwEFTS9LcRiMjSqq_xswDeXo7hYNWT0Em6nL4Sw@mail.gmail.com/T/#m7c0cd6e913c6295732daea3c88f502bd4724ffb3
>
> However, according to Christian the change was benchmarked by your
> machinery and no difference was found.
>
> I briefly poked around and found that used configs have:
> CONFIG_RANDOMIZE_KSTACK_OFFSET=y
>
> This is an optional and very expensive hardening feature, my question
> is if it was enabled on purpose. The cost comes from adding rdtsc to
> every syscall.
>
> Looking at the rest of the config you have a mixed bag (e.g., hardened
> usercopy but *no* init_on_alloc) so I genuinely don't know.
We used a kernel config from Redhat Enterprise Linux distribution as
base config (it was from redhat kernel-4.18.0-187). And then use the
default value of new configs.


>
> Given the high cost of the opt I would suggest removing it, as it
From arch/Kconfig:

config RANDOMIZE_KSTACK_OFFSET
bool "Support for randomizing kernel stack offset on syscall entry" if EXPERT
default y
depends on HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET
depends on INIT_STACK_NONE || !CC_IS_CLANG || CLANG_VERSION >= 140000
help
The kernel stack offset can be randomized (after pt_regs) by
roughly 5 bits of entropy, frustrating memory corruption
attacks that depend on stack address determinism or
cross-syscall address exposures.

The feature is controlled via the "randomize_kstack_offset=on/off"
kernel boot param, and if turned off has zero overhead due to its use
of static branches (see JUMP_LABEL).

If unsure, say Y.

I don't think we want to disable this kernel config in LKP.


> avoidably muddles the waters for single-threaded changes (one way or
> the other -- slowdowns can hide and speed ups go unnoticed).
>
> I did not review the whole config.
>
> Any comments? :)
IMHO, LKP cares more about the possible performance downgrade users could
observe and assume users are using distribution. So LKP sticks to a distribution
config and using the default value for new configs.


I did test the will-it-scale.read1 against the commit
20ea1e7d13c1b544fe67c4a8dc3943bb1ab33e6f
and its parent commit:
9e0ee0c7545c7ec012a53878e7687e05b87abc75
by using "read1_process -s 4 -t 1" on a IceLake platform with 48C/96T + 192G
memory. I don't have available Sapphire Rapid to run same test.

The config is with the CONFIG_RANDOMIZE_KSTACK_OFFSET and CONFIG_HARDENED_USERCOPY
disabled/enabled.


I found the result is not stable. stddev is around 7% ~ 8%. larger or close
the the performance difference. For average: there is no regression with
these two kernel configs disabled. Around 9% regression with these two kernel
configs enabled.

But from LKP perspective (Oliver, Yujie and Philip, correct me if I am wrong
here), this test result is not reported as regression because the stddev is
very close to the regression.


The specific result is as following:
parent without random/hardened_usercopy: child without random/hardened_usercopy:
4050865 3570609
3732989 3971604
4043509 3577899
3644475 4106721
4156399 3432816
3633719 4207917
3508474 3581210
3363729 3548164
3794146 3490865
3449113 4088112

avg: 3737741 avg: 3757591
stddev: 7% stddev: 7%


parent with random/hardened_usercopy: child with random/hardened_usercopy:
3770330 4279949
3824055 3377960
3497524 3359246
4244985 3462981
3442711 3520535
4418889 3436251
4255512 3884103
3605148 3480065
3965194 3230780
3654892 3372904

avg: 3867924 avg: 3540477 9% regression. But with 8.7% stddev
stddev: 8.8% stddev: 8.7%


Regards
Yin, Fengwei

>
> Thanks,