Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <CAHhAz+inPwKYx_4qaujQ=bGG9twashiuqLhQQ-+vgHWF7FLhRA@mail.gmail.com>
 <20200818143633.GA628293@kroah.com>
In-Reply-To: <20200818143633.GA628293@kroah.com>
From:   Muni Sekhar <munisekharrms@gmail.com>
Date:   Tue, 18 Aug 2020 22:24:13 +0530
Message-ID: <CAHhAz+hG5kS5rhph4SaSLOEc5PgcSOTPWpiANLNpwotY9Zy6qQ@mail.gmail.com>
Subject: Re: Scheduler benchmarks
To:     Greg KH <greg@kroah.com>
Cc:     kernelnewbies <kernelnewbies@kernelnewbies.org>,
        linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Tue, Aug 18, 2020 at 8:06 PM Greg KH <greg@kroah.com> wrote:
>
> On Tue, Aug 18, 2020 at 08:00:11PM +0530, Muni Sekhar wrote:
> > Hi all,
> >
> > I=E2=80=99ve two identical Linux systems with only kernel differences.
>
> What are the differences in the kernels?
>
> > While doing kernel profiling with perf, I got the below mentioned
> > metrics for Scheduler benchmarks.
> >
> > 1st system (older kernel version compared to the other system) benchmar=
k result:
> >
> > $ perf bench sched messaging -g 64
> > # Running 'sched/messaging' benchmark:
> > # 20 sender and receiver processes per group
> > # 64 groups =3D=3D 2560 processes run
> >
> >      Total time: 2.936 [sec]
> >
> >
> > 2nd system benchmark result:
> >
> > $ perf bench sched messaging -g 64
> > # Running 'sched/messaging' benchmark:
> > # 20 sender and receiver processes per group
> > # 64 groups =3D=3D 2560 processes run
> >
> >      Total time: 10.074 [sec]
> >
> >
> > So as per scheduler benchmark results, clearly a huge difference
> > between two systems.
> > Can anyone suggest to me how to dive deeper to know the root cause for
> > it.
>
> Look a the differences between your different kernels, that would be a
> great start :)
I created the difference between two kernel config files and then
tried to spot the CONFIG*SCHED* differences.
Interestingly I see the difference in I/O scheduler config, 1st system
is set to =E2=80=9Cdeadline=E2=80=9D and other one is set to =E2=80=9Ccfq=
=E2=80=9D. So I made it equal
by echoing to =E2=80=9C/sys/block/<disk device>/queue/scheduler" but still =
no
change in scheduler benchmark metrics.

Is it the correct way to find the differences between kernels? If so,
what other important CONFIG_* variables need to consider?


$ cat config.patch | grep -i sched

 CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=3Dy
 CONFIG_CGROUP_SCHED=3Dy
 CONFIG_FAIR_GROUP_SCHED=3Dy
 # CONFIG_RT_GROUP_SCHED is not set
 # IO Schedulers
@@ -369,10 +434,14 @@ CONFIG_IOSCHED_NOOP=3Dy
 CONFIG_IOSCHED_DEADLINE=3Dy
 CONFIG_IOSCHED_CFQ=3Dy
 CONFIG_CFQ_GROUP_IOSCHED=3Dy
-CONFIG_DEFAULT_IOSCHED=3D"deadline"
+CONFIG_DEFAULT_IOSCHED=3D"cfq"
+CONFIG_MQ_IOSCHED_DEADLINE=3Dm
+CONFIG_MQ_IOSCHED_KYBER=3Dm
+CONFIG_IOSCHED_BFQ=3Dm
+CONFIG_BFQ_GROUP_IOSCHED=3Dy
 CONFIG_SCHED_SMT=3Dy
 CONFIG_SCHED_MC=3Dy
+CONFIG_SCHED_MC_PRIO=3Dy
+# CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL is not set
+CONFIG_CPU_FREQ_GOV_SCHEDUTIL=3Dy


>
> good luck!
>
> greg k-h


--=20
Thanks,
Sekhar