Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Message-ID: <1516628254.7500.19.camel@gmx.de>
Subject: Re: unixbench context switch perfomance & cpu topology
From:   Mike Galbraith <efault@gmx.de>
To:     Wanpeng Li <kernellwp@gmail.com>
Cc:     linux-kernel@vger.kernel.org, kvm <kvm@vger.kernel.org>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Radim Krcmar <rkrcmar@redhat.com>,
        Frederic Weisbecker <fweisbec@gmail.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@kernel.org>
Date:   Mon, 22 Jan 2018 14:37:34 +0100
In-Reply-To: <CANRm+CyXVkQKbHFMQG4v3fjUuPOHpY15_c6gU+gaUzf_67xnFg@mail.gmail.com>
References: <CANRm+CwpGoSTLNG3D8KN+fgfc+20-gFND7xGAYo2b5EuwzPeOg@mail.gmail.com>
         <1516622939.24679.5.camel@gmx.de>
         <CANRm+CyXVkQKbHFMQG4v3fjUuPOHpY15_c6gU+gaUzf_67xnFg@mail.gmail.com>
Content-Type: text/plain; charset="ISO-8859-15"
Mime-Version: 1.0
Content-Transfer-Encoding: 8BIT
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Mon, 2018-01-22 at 20:27 +0800, Wanpeng Li wrote:
> 2018-01-22 20:08 GMT+08:00 Mike Galbraith <efault@gmx.de>:
> > On Mon, 2018-01-22 at 19:47 +0800, Wanpeng Li wrote:
> >> Hi all,
> >>
> >> We can observe unixbench context switch performance is heavily
> >> influenced by cpu topology which is exposed to the guest. the score is
> >> posted below, bigger is better, both the guest and the host kernel are
> >> 3.15-rc3(we can also reproduce against centos 7.4 693 guest/host), LLC
> >> is exposed to the guest, kvm adaptive halt-polling is default enabled,
> >> then start a guest w/ 8 logical cpus.
> >>
> >>
> >>
> >> unixbench context switch
> >> -smp 8, sockets=8, cores=1, threads=1    382036
> >> -smp 8, sockets=4, cores=2, threads=1    132480
> >> -smp 8, sockets=2, cores=4, threads=1    128032
> >> -smp 8, sockets=2, cores=2, threads=2    131767
> >> -smp 8, sockets=1, cores=4, threads=2    132742
> >> -smp 8, sockets=1, cores=4, threads=2 (guest w/ nohz=off idle=poll)    331471
> >>
> >> I can observe there are a lot of reschedule IPIs sent from one vCPU to
> >> another vCPU, the context switch workload switches between running and
> >> idle frequently which results in HLT instruction in the idle path, I
> >> use idle=poll to avoid vmexit due to HLT and to avoid reschedule IPIs
> >> since idle task checks TIF_NEED_RESCHED flags in a loop, nohz=off can
> >> stop to program lapic timer/other nohz stuffs. Any idea why sockets=8
> >> can get best performance?
> >
> > Probably because with that topology, there is no shared llc, thus no
> > cross-core scheduling, micro-benchmark waker/wakee are stacked.  If
> > your benchmark does nothing but schedule, stacking makes beautiful (but
> > utterly meaningless) numbers.
> 
> The waker and wakee are just sporadic on the same logical cpu in the
> guest(-smp 8, sockets=8, cores=1, threads=1) during the testing, in
> addition, binding the waker/wakee to one logical cpu in the guest(-smp
> 8, sockets=1, cores=4, threads=2) also can get the performance as
> better as 8 sockets setup.

Here, with tip.today and that topology, context1 does stack up on one core.

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ P COMMAND                                               
 4218 root      20   0    4048    808    732 R 52.16 0.022   0:12.77 4 context1                                              
 4219 root      20   0    4048     80      0 S 47.18 0.002   0:11.96 4 context1

There's a bit of bouncing, but the two stack right back up. ?But
whatever, what Peter said, the benchmark should pin itself to do this.

	-Mike