Hi all,
I’ve two identical Linux systems with only kernel differences.
While doing kernel profiling with perf, I got the below mentioned
metrics for Scheduler benchmarks.
1st system (older kernel version compared to the other system) benchmark result:
$ perf bench sched messaging -g 64
# Running 'sched/messaging' benchmark:
# 20 sender and receiver processes per group
# 64 groups == 2560 processes run
Total time: 2.936 [sec]
2nd system benchmark result:
$ perf bench sched messaging -g 64
# Running 'sched/messaging' benchmark:
# 20 sender and receiver processes per group
# 64 groups == 2560 processes run
Total time: 10.074 [sec]
So as per scheduler benchmark results, clearly a huge difference
between two systems.
Can anyone suggest to me how to dive deeper to know the root cause for
it. Also are there any tunable kernel parameters related to this one?
--
Thanks,
Sekhar
On Tue, Aug 18, 2020 at 08:00:11PM +0530, Muni Sekhar wrote:
> Hi all,
>
> I’ve two identical Linux systems with only kernel differences.
What are the differences in the kernels?
> While doing kernel profiling with perf, I got the below mentioned
> metrics for Scheduler benchmarks.
>
> 1st system (older kernel version compared to the other system) benchmark result:
>
> $ perf bench sched messaging -g 64
> # Running 'sched/messaging' benchmark:
> # 20 sender and receiver processes per group
> # 64 groups == 2560 processes run
>
> Total time: 2.936 [sec]
>
>
> 2nd system benchmark result:
>
> $ perf bench sched messaging -g 64
> # Running 'sched/messaging' benchmark:
> # 20 sender and receiver processes per group
> # 64 groups == 2560 processes run
>
> Total time: 10.074 [sec]
>
>
> So as per scheduler benchmark results, clearly a huge difference
> between two systems.
> Can anyone suggest to me how to dive deeper to know the root cause for
> it.
Look a the differences between your different kernels, that would be a
great start :)
good luck!
greg k-h
On Tue, Aug 18, 2020 at 8:06 PM Greg KH <[email protected]> wrote:
>
> On Tue, Aug 18, 2020 at 08:00:11PM +0530, Muni Sekhar wrote:
> > Hi all,
> >
> > I’ve two identical Linux systems with only kernel differences.
>
> What are the differences in the kernels?
>
> > While doing kernel profiling with perf, I got the below mentioned
> > metrics for Scheduler benchmarks.
> >
> > 1st system (older kernel version compared to the other system) benchmark result:
> >
> > $ perf bench sched messaging -g 64
> > # Running 'sched/messaging' benchmark:
> > # 20 sender and receiver processes per group
> > # 64 groups == 2560 processes run
> >
> > Total time: 2.936 [sec]
> >
> >
> > 2nd system benchmark result:
> >
> > $ perf bench sched messaging -g 64
> > # Running 'sched/messaging' benchmark:
> > # 20 sender and receiver processes per group
> > # 64 groups == 2560 processes run
> >
> > Total time: 10.074 [sec]
> >
> >
> > So as per scheduler benchmark results, clearly a huge difference
> > between two systems.
> > Can anyone suggest to me how to dive deeper to know the root cause for
> > it.
>
> Look a the differences between your different kernels, that would be a
> great start :)
I created the difference between two kernel config files(please find
it in the attachment: config.patch) and then try to spot the
CONFIG*SCHED* differences.
Interestingly I see the difference in I/O scheduler config, 1st system
is set to “deadline” and other one is set to “cfq”. So I made it equal
by echoing to “/sys/block/<disk device>/queue/scheduler" but still no
change in scheduler benchmark.
Is it the correct way to find the differences between kernels? If so,
what other important CONFIG_* variables need to consider?
$ cat config.patch | grep -i sched
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
# CONFIG_RT_GROUP_SCHED is not set
# IO Schedulers
@@ -369,10 +434,14 @@ CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_CFQ_GROUP_IOSCHED=y
-CONFIG_DEFAULT_IOSCHED="deadline"
+CONFIG_DEFAULT_IOSCHED="cfq"
+CONFIG_MQ_IOSCHED_DEADLINE=m
+CONFIG_MQ_IOSCHED_KYBER=m
+CONFIG_IOSCHED_BFQ=m
+CONFIG_BFQ_GROUP_IOSCHED=y
CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
+CONFIG_SCHED_MC_PRIO=y
+# CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL is not set
+CONFIG_CPU_FREQ_GOV_SCHEDUTIL=y
>
> good luck!
>
> greg k-h
--
Thanks,
Sekhar
On Tue, Aug 18, 2020 at 8:06 PM Greg KH <[email protected]> wrote:
>
> On Tue, Aug 18, 2020 at 08:00:11PM +0530, Muni Sekhar wrote:
> > Hi all,
> >
> > I’ve two identical Linux systems with only kernel differences.
>
> What are the differences in the kernels?
>
> > While doing kernel profiling with perf, I got the below mentioned
> > metrics for Scheduler benchmarks.
> >
> > 1st system (older kernel version compared to the other system) benchmark result:
> >
> > $ perf bench sched messaging -g 64
> > # Running 'sched/messaging' benchmark:
> > # 20 sender and receiver processes per group
> > # 64 groups == 2560 processes run
> >
> > Total time: 2.936 [sec]
> >
> >
> > 2nd system benchmark result:
> >
> > $ perf bench sched messaging -g 64
> > # Running 'sched/messaging' benchmark:
> > # 20 sender and receiver processes per group
> > # 64 groups == 2560 processes run
> >
> > Total time: 10.074 [sec]
> >
> >
> > So as per scheduler benchmark results, clearly a huge difference
> > between two systems.
> > Can anyone suggest to me how to dive deeper to know the root cause for
> > it.
>
> Look a the differences between your different kernels, that would be a
> great start :)
I created the difference between two kernel config files(please find
it in the attachment: config.patch) and then try to spot the
CONFIG*SCHED* differences.
Interestingly I see the difference in I/O scheduler config, 1st system
is set to “deadline” and other one is set to “cfq”. So I made it equal
by echoing to “/sys/block/<disk device>/queue/scheduler" but still no
change in scheduler benchmark.
Is it the correct way to find the differences between kernels? If so,
what other important CONFIG_* variables need to consider?
$ cat config.patch | grep -i sched
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
# CONFIG_RT_GROUP_SCHED is not set
# IO Schedulers
@@ -369,10 +434,14 @@ CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_CFQ_GROUP_IOSCHED=y
-CONFIG_DEFAULT_IOSCHED="deadline"
+CONFIG_DEFAULT_IOSCHED="cfq"
+CONFIG_MQ_IOSCHED_DEADLINE=m
+CONFIG_MQ_IOSCHED_KYBER=m
+CONFIG_IOSCHED_BFQ=m
+CONFIG_BFQ_GROUP_IOSCHED=y
CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
+CONFIG_SCHED_MC_PRIO=y
+# CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL is not set
+CONFIG_CPU_FREQ_GOV_SCHEDUTIL=y
>
> good luck!
>
> greg k-h
--
Thanks,
Sekhar
On Tue, Aug 18, 2020 at 8:06 PM Greg KH <[email protected]> wrote:
>
> On Tue, Aug 18, 2020 at 08:00:11PM +0530, Muni Sekhar wrote:
> > Hi all,
> >
> > I’ve two identical Linux systems with only kernel differences.
>
> What are the differences in the kernels?
>
> > While doing kernel profiling with perf, I got the below mentioned
> > metrics for Scheduler benchmarks.
> >
> > 1st system (older kernel version compared to the other system) benchmark result:
> >
> > $ perf bench sched messaging -g 64
> > # Running 'sched/messaging' benchmark:
> > # 20 sender and receiver processes per group
> > # 64 groups == 2560 processes run
> >
> > Total time: 2.936 [sec]
> >
> >
> > 2nd system benchmark result:
> >
> > $ perf bench sched messaging -g 64
> > # Running 'sched/messaging' benchmark:
> > # 20 sender and receiver processes per group
> > # 64 groups == 2560 processes run
> >
> > Total time: 10.074 [sec]
> >
> >
> > So as per scheduler benchmark results, clearly a huge difference
> > between two systems.
> > Can anyone suggest to me how to dive deeper to know the root cause for
> > it.
>
> Look a the differences between your different kernels, that would be a
> great start :)
I created the difference between two kernel config files and then
tried to spot the CONFIG*SCHED* differences.
Interestingly I see the difference in I/O scheduler config, 1st system
is set to “deadline” and other one is set to “cfq”. So I made it equal
by echoing to “/sys/block/<disk device>/queue/scheduler" but still no
change in scheduler benchmark metrics.
Is it the correct way to find the differences between kernels? If so,
what other important CONFIG_* variables need to consider?
$ cat config.patch | grep -i sched
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
# CONFIG_RT_GROUP_SCHED is not set
# IO Schedulers
@@ -369,10 +434,14 @@ CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_CFQ_GROUP_IOSCHED=y
-CONFIG_DEFAULT_IOSCHED="deadline"
+CONFIG_DEFAULT_IOSCHED="cfq"
+CONFIG_MQ_IOSCHED_DEADLINE=m
+CONFIG_MQ_IOSCHED_KYBER=m
+CONFIG_IOSCHED_BFQ=m
+CONFIG_BFQ_GROUP_IOSCHED=y
CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
+CONFIG_SCHED_MC_PRIO=y
+# CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL is not set
+CONFIG_CPU_FREQ_GOV_SCHEDUTIL=y
>
> good luck!
>
> greg k-h
--
Thanks,
Sekhar
On Tue, Aug 18, 2020 at 10:24:13PM +0530, Muni Sekhar wrote:
> On Tue, Aug 18, 2020 at 8:06 PM Greg KH <[email protected]> wrote:
> >
> > On Tue, Aug 18, 2020 at 08:00:11PM +0530, Muni Sekhar wrote:
> > > Hi all,
> > >
> > > I’ve two identical Linux systems with only kernel differences.
> >
> > What are the differences in the kernels?
You didn't answer this question, is this the same kernel source being
compared here? Same version? Same compiler? Everything identical?
> > > While doing kernel profiling with perf, I got the below mentioned
> > > metrics for Scheduler benchmarks.
> > >
> > > 1st system (older kernel version compared to the other system) benchmark result:
> > >
> > > $ perf bench sched messaging -g 64
> > > # Running 'sched/messaging' benchmark:
> > > # 20 sender and receiver processes per group
> > > # 64 groups == 2560 processes run
> > >
> > > Total time: 2.936 [sec]
> > >
> > >
> > > 2nd system benchmark result:
> > >
> > > $ perf bench sched messaging -g 64
> > > # Running 'sched/messaging' benchmark:
> > > # 20 sender and receiver processes per group
> > > # 64 groups == 2560 processes run
> > >
> > > Total time: 10.074 [sec]
> > >
> > >
> > > So as per scheduler benchmark results, clearly a huge difference
> > > between two systems.
> > > Can anyone suggest to me how to dive deeper to know the root cause for
> > > it.
> >
> > Look a the differences between your different kernels, that would be a
> > great start :)
> I created the difference between two kernel config files and then
> tried to spot the CONFIG*SCHED* differences.
> Interestingly I see the difference in I/O scheduler config, 1st system
> is set to “deadline” and other one is set to “cfq”. So I made it equal
> by echoing to “/sys/block/<disk device>/queue/scheduler" but still no
> change in scheduler benchmark metrics.
>
> Is it the correct way to find the differences between kernels? If so,
> what other important CONFIG_* variables need to consider?
>
>
> $ cat config.patch | grep -i sched
>
> CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
> CONFIG_CGROUP_SCHED=y
> CONFIG_FAIR_GROUP_SCHED=y
> # CONFIG_RT_GROUP_SCHED is not set
> # IO Schedulers
> @@ -369,10 +434,14 @@ CONFIG_IOSCHED_NOOP=y
> CONFIG_IOSCHED_DEADLINE=y
> CONFIG_IOSCHED_CFQ=y
> CONFIG_CFQ_GROUP_IOSCHED=y
> -CONFIG_DEFAULT_IOSCHED="deadline"
> +CONFIG_DEFAULT_IOSCHED="cfq"
> +CONFIG_MQ_IOSCHED_DEADLINE=m
> +CONFIG_MQ_IOSCHED_KYBER=m
> +CONFIG_IOSCHED_BFQ=m
> +CONFIG_BFQ_GROUP_IOSCHED=y
> CONFIG_SCHED_SMT=y
> CONFIG_SCHED_MC=y
> +CONFIG_SCHED_MC_PRIO=y
> +# CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL is not set
> +CONFIG_CPU_FREQ_GOV_SCHEDUTIL=y
There's lots of other options that affect performance, depending on your
specific benchmark, other than these.
good luck!
greg k-h
On Tue, Aug 18, 2020 at 10:44 PM Greg KH <[email protected]> wrote:
>
> On Tue, Aug 18, 2020 at 10:24:13PM +0530, Muni Sekhar wrote:
> > On Tue, Aug 18, 2020 at 8:06 PM Greg KH <[email protected]> wrote:
> > >
> > > On Tue, Aug 18, 2020 at 08:00:11PM +0530, Muni Sekhar wrote:
> > > > Hi all,
> > > >
> > > > I’ve two identical Linux systems with only kernel differences.
> > >
> > > What are the differences in the kernels?
>
> You didn't answer this question, is this the same kernel source being
> compared here? Same version? Same compiler? Everything identical?
Both systems are having exactly the same hardware configuration.
Compiler and kernel versions are different. One system has Ubuntu
16.04.4 LTS(4.4.0-66-generic kernel with gcc version 5.4.0) kernel and
the other one has Ubuntu 18.04.4 LTS(4.15.0-91-generic kernel with gcc
version 7.5.0).
>
> > > > While doing kernel profiling with perf, I got the below mentioned
> > > > metrics for Scheduler benchmarks.
> > > >
> > > > 1st system (older kernel version compared to the other system) benchmark result:
> > > >
> > > > $ perf bench sched messaging -g 64
> > > > # Running 'sched/messaging' benchmark:
> > > > # 20 sender and receiver processes per group
> > > > # 64 groups == 2560 processes run
> > > >
> > > > Total time: 2.936 [sec]
> > > >
> > > >
> > > > 2nd system benchmark result:
> > > >
> > > > $ perf bench sched messaging -g 64
> > > > # Running 'sched/messaging' benchmark:
> > > > # 20 sender and receiver processes per group
> > > > # 64 groups == 2560 processes run
> > > >
> > > > Total time: 10.074 [sec]
> > > >
> > > >
> > > > So as per scheduler benchmark results, clearly a huge difference
> > > > between two systems.
> > > > Can anyone suggest to me how to dive deeper to know the root cause for
> > > > it.
> > >
> > > Look a the differences between your different kernels, that would be a
> > > great start :)
> > I created the difference between two kernel config files and then
> > tried to spot the CONFIG*SCHED* differences.
> > Interestingly I see the difference in I/O scheduler config, 1st system
> > is set to “deadline” and other one is set to “cfq”. So I made it equal
> > by echoing to “/sys/block/<disk device>/queue/scheduler" but still no
> > change in scheduler benchmark metrics.
> >
> > Is it the correct way to find the differences between kernels? If so,
> > what other important CONFIG_* variables need to consider?
> >
> >
> > $ cat config.patch | grep -i sched
> >
> > CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
> > CONFIG_CGROUP_SCHED=y
> > CONFIG_FAIR_GROUP_SCHED=y
> > # CONFIG_RT_GROUP_SCHED is not set
> > # IO Schedulers
> > @@ -369,10 +434,14 @@ CONFIG_IOSCHED_NOOP=y
> > CONFIG_IOSCHED_DEADLINE=y
> > CONFIG_IOSCHED_CFQ=y
> > CONFIG_CFQ_GROUP_IOSCHED=y
> > -CONFIG_DEFAULT_IOSCHED="deadline"
> > +CONFIG_DEFAULT_IOSCHED="cfq"
> > +CONFIG_MQ_IOSCHED_DEADLINE=m
> > +CONFIG_MQ_IOSCHED_KYBER=m
> > +CONFIG_IOSCHED_BFQ=m
> > +CONFIG_BFQ_GROUP_IOSCHED=y
> > CONFIG_SCHED_SMT=y
> > CONFIG_SCHED_MC=y
> > +CONFIG_SCHED_MC_PRIO=y
> > +# CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL is not set
> > +CONFIG_CPU_FREQ_GOV_SCHEDUTIL=y
>
> There's lots of other options that affect performance, depending on your
> specific benchmark, other than these.
>
> good luck!
>
> greg k-h
--
Thanks,
Sekhar
On Tue, Aug 18, 2020 at 11:01:35PM +0530, Muni Sekhar wrote:
> On Tue, Aug 18, 2020 at 10:44 PM Greg KH <[email protected]> wrote:
> >
> > On Tue, Aug 18, 2020 at 10:24:13PM +0530, Muni Sekhar wrote:
> > > On Tue, Aug 18, 2020 at 8:06 PM Greg KH <[email protected]> wrote:
> > > >
> > > > On Tue, Aug 18, 2020 at 08:00:11PM +0530, Muni Sekhar wrote:
> > > > > Hi all,
> > > > >
> > > > > I’ve two identical Linux systems with only kernel differences.
> > > >
> > > > What are the differences in the kernels?
> >
> > You didn't answer this question, is this the same kernel source being
> > compared here? Same version? Same compiler? Everything identical?
> Both systems are having exactly the same hardware configuration.
> Compiler and kernel versions are different. One system has Ubuntu
> 16.04.4 LTS(4.4.0-66-generic kernel with gcc version 5.4.0) kernel and
> the other one has Ubuntu 18.04.4 LTS(4.15.0-91-generic kernel with gcc
> version 7.5.0).
Those are _very_ different kernel versions, with many years and tens of
thousands of different changes between them.
Hopefully the newer kernel is faster, so just stick with that :)
greg k-h
On Tue, Aug 18, 2020 at 11:06 PM Greg KH <[email protected]> wrote:
>
> On Tue, Aug 18, 2020 at 11:01:35PM +0530, Muni Sekhar wrote:
> > On Tue, Aug 18, 2020 at 10:44 PM Greg KH <[email protected]> wrote:
> > >
> > > On Tue, Aug 18, 2020 at 10:24:13PM +0530, Muni Sekhar wrote:
> > > > On Tue, Aug 18, 2020 at 8:06 PM Greg KH <[email protected]> wrote:
> > > > >
> > > > > On Tue, Aug 18, 2020 at 08:00:11PM +0530, Muni Sekhar wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > I’ve two identical Linux systems with only kernel differences.
> > > > >
> > > > > What are the differences in the kernels?
> > >
> > > You didn't answer this question, is this the same kernel source being
> > > compared here? Same version? Same compiler? Everything identical?
> > Both systems are having exactly the same hardware configuration.
> > Compiler and kernel versions are different. One system has Ubuntu
> > 16.04.4 LTS(4.4.0-66-generic kernel with gcc version 5.4.0) kernel and
> > the other one has Ubuntu 18.04.4 LTS(4.15.0-91-generic kernel with gcc
> > version 7.5.0).
>
> Those are _very_ different kernel versions, with many years and tens of
> thousands of different changes between them.
>
> Hopefully the newer kernel is faster, so just stick with that :)
But unfortunately the newer kernel is very slow, that is the reason
for starting this investigation :)
Any type of help, and guidelines to dive deeper will be highly appreciated.
>
> greg k-h
--
Thanks,
Sekhar
On 8/18/20 7:53 PM, Muni Sekhar wrote:
> On Tue, Aug 18, 2020 at 11:06 PM Greg KH <[email protected]> wrote:
>> On Tue, Aug 18, 2020 at 11:01:35PM +0530, Muni Sekhar wrote:
>>> On Tue, Aug 18, 2020 at 10:44 PM Greg KH <[email protected]> wrote:
>>>> On Tue, Aug 18, 2020 at 10:24:13PM +0530, Muni Sekhar wrote:
>>>>> On Tue, Aug 18, 2020 at 8:06 PM Greg KH <[email protected]> wrote:
>>>>>> On Tue, Aug 18, 2020 at 08:00:11PM +0530, Muni Sekhar wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I’ve two identical Linux systems with only kernel differences.
>>>>>> What are the differences in the kernels?
>>>> You didn't answer this question, is this the same kernel source being
>>>> compared here? Same version? Same compiler? Everything identical?
>>> Both systems are having exactly the same hardware configuration.
>>> Compiler and kernel versions are different. One system has Ubuntu
>>> 16.04.4 LTS(4.4.0-66-generic kernel with gcc version 5.4.0) kernel and
>>> the other one has Ubuntu 18.04.4 LTS(4.15.0-91-generic kernel with gcc
>>> version 7.5.0).
>> Those are _very_ different kernel versions, with many years and tens of
>> thousands of different changes between them.
>>
>> Hopefully the newer kernel is faster, so just stick with that :)
> But unfortunately the newer kernel is very slow, that is the reason
> for starting this investigation :)
> Any type of help, and guidelines to dive deeper will be highly appreciated.
On the 4.4 kernel you dont have
+CONFIG_RETPOLINE=y
+CONFIG_INTEL_RDT=y
And your base is very different two.
Try to use mainline on both system and see.
You can also use the same base kernel version from ubuntu and
run your test.
>> greg k-h
>
>
On Tue, Aug 18, 2020 at 11:45 PM peter enderborg
<[email protected]> wrote:
>
> On 8/18/20 7:53 PM, Muni Sekhar wrote:
> > On Tue, Aug 18, 2020 at 11:06 PM Greg KH <[email protected]> wrote:
> >> On Tue, Aug 18, 2020 at 11:01:35PM +0530, Muni Sekhar wrote:
> >>> On Tue, Aug 18, 2020 at 10:44 PM Greg KH <[email protected]> wrote:
> >>>> On Tue, Aug 18, 2020 at 10:24:13PM +0530, Muni Sekhar wrote:
> >>>>> On Tue, Aug 18, 2020 at 8:06 PM Greg KH <[email protected]> wrote:
> >>>>>> On Tue, Aug 18, 2020 at 08:00:11PM +0530, Muni Sekhar wrote:
> >>>>>>> Hi all,
> >>>>>>>
> >>>>>>> I’ve two identical Linux systems with only kernel differences.
> >>>>>> What are the differences in the kernels?
> >>>> You didn't answer this question, is this the same kernel source being
> >>>> compared here? Same version? Same compiler? Everything identical?
> >>> Both systems are having exactly the same hardware configuration.
> >>> Compiler and kernel versions are different. One system has Ubuntu
> >>> 16.04.4 LTS(4.4.0-66-generic kernel with gcc version 5.4.0) kernel and
> >>> the other one has Ubuntu 18.04.4 LTS(4.15.0-91-generic kernel with gcc
> >>> version 7.5.0).
> >> Those are _very_ different kernel versions, with many years and tens of
> >> thousands of different changes between them.
> >>
> >> Hopefully the newer kernel is faster, so just stick with that :)
> > But unfortunately the newer kernel is very slow, that is the reason
> > for starting this investigation :)
> > Any type of help, and guidelines to dive deeper will be highly appreciated.
>
> On the 4.4 kernel you dont have
>
> +CONFIG_RETPOLINE=y
> +CONFIG_INTEL_RDT=y
Thanks! That is helpful. Yes, I see 4.4 kernel don't have the above
two config options.
What analysis can be done to narrow down the root cause?
Any example of reference could be helpful to understand.
>
> And your base is very different two.
>
> Try to use mainline on both system and see.
>
> You can also use the same base kernel version from ubuntu and
>
> run your test.
>
>
> >> greg k-h
> >
> >
>
--
Thanks,
Sekhar
On Wed, Aug 19, 2020 at 03:46:06PM +0530, Muni Sekhar wrote:
> On Tue, Aug 18, 2020 at 11:45 PM peter enderborg
> <[email protected]> wrote:
> >
> > On 8/18/20 7:53 PM, Muni Sekhar wrote:
> > > On Tue, Aug 18, 2020 at 11:06 PM Greg KH <[email protected]> wrote:
> > >> On Tue, Aug 18, 2020 at 11:01:35PM +0530, Muni Sekhar wrote:
> > >>> On Tue, Aug 18, 2020 at 10:44 PM Greg KH <[email protected]> wrote:
> > >>>> On Tue, Aug 18, 2020 at 10:24:13PM +0530, Muni Sekhar wrote:
> > >>>>> On Tue, Aug 18, 2020 at 8:06 PM Greg KH <[email protected]> wrote:
> > >>>>>> On Tue, Aug 18, 2020 at 08:00:11PM +0530, Muni Sekhar wrote:
> > >>>>>>> Hi all,
> > >>>>>>>
> > >>>>>>> I’ve two identical Linux systems with only kernel differences.
> > >>>>>> What are the differences in the kernels?
> > >>>> You didn't answer this question, is this the same kernel source being
> > >>>> compared here? Same version? Same compiler? Everything identical?
> > >>> Both systems are having exactly the same hardware configuration.
> > >>> Compiler and kernel versions are different. One system has Ubuntu
> > >>> 16.04.4 LTS(4.4.0-66-generic kernel with gcc version 5.4.0) kernel and
> > >>> the other one has Ubuntu 18.04.4 LTS(4.15.0-91-generic kernel with gcc
> > >>> version 7.5.0).
> > >> Those are _very_ different kernel versions, with many years and tens of
> > >> thousands of different changes between them.
> > >>
> > >> Hopefully the newer kernel is faster, so just stick with that :)
> > > But unfortunately the newer kernel is very slow, that is the reason
> > > for starting this investigation :)
> > > Any type of help, and guidelines to dive deeper will be highly appreciated.
> >
> > On the 4.4 kernel you dont have
> >
> > +CONFIG_RETPOLINE=y
> > +CONFIG_INTEL_RDT=y
> Thanks! That is helpful. Yes, I see 4.4 kernel don't have the above
> two config options.
> What analysis can be done to narrow down the root cause?
"root cause" is you are comparing a kernel without any spectre/meltdown
mitigations, with one that has it.
Those mitigations are known to have a huge slowdown on some
benchmarks/workloads, as you have shown here.
If you don't like this, you can disable these changes on newer kernels,
if you really really know what you are doing, but I strongly do not
recommend it.
Also you can complain to your hardware vendors for selling you broken
hardware :)
> Any example of reference could be helpful to understand.
Look up Spectre and Meltdown for many many examples of what happened and
what went wrong with chip designs and how we had to fix these things in
the kernel a few years ago.
greg k-h
On 19/08/2020 10:16, Muni Sekhar wrote:
> On Tue, Aug 18, 2020 at 11:45 PM peter enderborg
> <[email protected]> wrote:
[...]
>> On the 4.4 kernel you dont have
>>
>> +CONFIG_RETPOLINE=y
>> +CONFIG_INTEL_RDT=y
> Thanks! That is helpful. Yes, I see 4.4 kernel don't have the above
> two config options.
> What analysis can be done to narrow down the root cause?
> Any example of reference could be helpful to understand.
I haven't checked the date of the older kernel but Spectre+
Meltdown mitigation costs a lot (20%-30% speed IIRC, out
of the top of my head).
MfG,
Bernd
--
There is no cloud, just other people computers.
-- https://static.fsf.org/nosvn/stickers/thereisnocloud.svg
From: Bernd Petrovitsch
> Sent: 19 August 2020 11:22
>
> On 19/08/2020 10:16, Muni Sekhar wrote:
> > On Tue, Aug 18, 2020 at 11:45 PM peter enderborg
> > <[email protected]> wrote:
> [...]
> >> On the 4.4 kernel you dont have
> >>
> >> +CONFIG_RETPOLINE=y
> >> +CONFIG_INTEL_RDT=y
> > Thanks! That is helpful. Yes, I see 4.4 kernel don't have the above
> > two config options.
> > What analysis can be done to narrow down the root cause?
> > Any example of reference could be helpful to understand.
>
> I haven't checked the date of the older kernel but Spectre+
> Meltdown mitigation costs a lot (20%-30% speed IIRC, out
> of the top of my head).
It depends greatly on the benchmark.
Page table separation has a massive effect on anything that
does a lot of short system calls.
There may be some mitigation in very recent hardware.
And I can't remember if it gets disabled for 'root' processes
(which can read kernel memory anyway).
It is definitely worth building a kernel with the mitigations
disabled - if only to check that they are responsible.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)
On Wed, 19 Aug 2020 12:42:54 +0200, Greg KH said:
> Look up Spectre and Meltdown for many many examples of what happened and
> what went wrong with chip designs and how we had to fix these things in
> the kernel a few years ago.
And I'm sure that nobody sane thinks we're done with security holes
caused by speculative execution... :)
On Wed, Aug 19, 2020 at 12:43:52PM -0400, Valdis Klētnieks wrote:
> On Wed, 19 Aug 2020 12:42:54 +0200, Greg KH said:
> > Look up Spectre and Meltdown for many many examples of what happened and
> > what went wrong with chip designs and how we had to fix these things in
> > the kernel a few years ago.
>
> And I'm sure that nobody sane thinks we're done with security holes
> caused by speculative execution... :)
Hah, not at all, the patches keep flowing in, as do the microcode
updates...
Em Wed, Aug 19, 2020 at 03:46:06PM +0530, Muni Sekhar escreveu:
> On Tue, Aug 18, 2020 at 11:45 PM peter enderborg <[email protected]> wrote:
> > On 8/18/20 7:53 PM, Muni Sekhar wrote:
> > > On Tue, Aug 18, 2020 at 11:06 PM Greg KH <[email protected]> wrote:
> > >> On Tue, Aug 18, 2020 at 11:01:35PM +0530, Muni Sekhar wrote:
> > >>> On Tue, Aug 18, 2020 at 10:44 PM Greg KH <[email protected]> wrote:
> > >> Those are _very_ different kernel versions, with many years and tens of
> > >> thousands of different changes between them.
> > >>
> > >> Hopefully the newer kernel is faster, so just stick with that :)
> > > But unfortunately the newer kernel is very slow, that is the reason
> > > for starting this investigation :)
> > > Any type of help, and guidelines to dive deeper will be highly appreciated.
> >
> > On the 4.4 kernel you dont have
> >
> > +CONFIG_RETPOLINE=y
> > +CONFIG_INTEL_RDT=y
> Thanks! That is helpful. Yes, I see 4.4 kernel don't have the above
> two config options.
> What analysis can be done to narrow down the root cause?
> Any example of reference could be helpful to understand.
This made me remember Lisbon:
http://vger.kernel.org/~acme/perf/bpf-is-eating-the-world-dont-you-see-lpc-lisbon-2019/#/8/2
Greg was there when this slide was first shown... ;-)
Look at the icache misses when the kernel is entered (syscall entry).
Then add this to the kernel command line: "nospectre_v1 nospectre_v2"
And see what happens:
http://vger.kernel.org/~acme/perf/bpf-is-eating-the-world-dont-you-see-lpc-lisbon-2019/#/8/3
So using these (and any new ones related to disabling hardware flaw
mitigation) maybe a way to analyse this, if you haven't figured this out
yet, sorry for the huge delay in replying 8-)
- Arnaldo
> >
> > And your base is very different two.
> >
> > Try to use mainline on both system and see.
> >
> > You can also use the same base kernel version from ubuntu and
> >
> > run your test.
> >
> >
> > >> greg k-h
> > >
> > >
> >
>
>
> --
> Thanks,
> Sekhar
--
- Arnaldo
