Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Content-Type: text/plain;
        charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\))
Subject: Re: [RFC PATCH 0/5] x86: dynamic indirect call promotion
From:   Nadav Amit <nadav.amit@gmail.com>
In-Reply-To: <7182c1db-2b23-5d69-7f4c-856a6ad43f8e@intel.com>
Date:   Tue, 23 Oct 2018 13:32:21 -0700
Cc:     Andy Lutomirski <luto@kernel.org>,
        Peter Zijlstra <peterz@infradead.org>,
        "H . Peter Anvin" <hpa@zytor.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        linux-kernel@vger.kernel.org, x86@kernel.org,
        Borislav Petkov <bp@alien8.de>,
        David Woodhouse <dwmw@amazon.co.uk>
Content-Transfer-Encoding: quoted-printable
Message-Id: <FBC2D350-E001-48C2-A4B7-0532FFD54531@gmail.com>
References: <20181018005420.82993-1-namit@vmware.com>
 <7182c1db-2b23-5d69-7f4c-856a6ad43f8e@intel.com>
To:     Dave Hansen <dave.hansen@intel.com>, Nadav Amit <namit@vmware.com>,
        Ingo Molnar <mingo@redhat.com>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

at 11:36 AM, Dave Hansen <dave.hansen@intel.com> wrote:

> On 10/17/18 5:54 PM, Nadav Amit wrote:
>> base		relpoline
>> 		----		---------
>> nginx 	22898 		25178 (+10%)
>> redis-ycsb	24523		25486 (+4%)
>> dbench	2144		2103 (+2%)
>=20
> Just out of curiosity, which indirect branches are the culprits here =
for
> causing the slowdowns?

So I didn=E2=80=99t try to measure exactly which one. There are roughly =
500 that
actually =E2=80=9Crun=E2=80=9D in my tests. Initially, I took the silly =
approach of trying
to patch the C source-code using semi automatically-generated Coccinelle
scripts, so I can tell you it is not just few branches but many. The
network stack is full of function pointers (e.g., tcp_congestion_ops,
tcp_sock_af_ops, dst_ops). The file-system also uses many function =
pointers
(file_operations specifically). Compound-pages have d=E2=80=99tor and so =
on.

If you want, you can rebuild the kernel without retpolines and run
=09
  perf record -e br_inst_exec.taken_indirect_near_call:k (your workload)

For some reason I didn=E2=80=99t manage to use PEBS (:ppp) from either =
the guest or
the host, so my results are a bit skewed (i.e., the sampled location is
usually after the call was taken). Running dbench in the VM gives me the
following =E2=80=9Chot-spots=E2=80=9D:

# Samples: 304  of event 'br_inst_exec.taken_indirect_near_call'
# Event count (approx.): 60800912
#
# Overhead  Command  Shared Object            Symbol                     =
                 =20
# ........  .......  .......................  =
.............................................
#
     5.26%  :197970  [guest.kernel.kallsyms]  [g] __fget_light
     4.28%  :197969  [guest.kernel.kallsyms]  [g] __fget_light
     3.95%  :197969  [guest.kernel.kallsyms]  [g] dcache_readdir
     3.29%  :197970  [guest.kernel.kallsyms]  [g] next_positive.isra.14
     2.96%  :197970  [guest.kernel.kallsyms]  [g] __do_sys_kill
     2.30%  :197970  [guest.kernel.kallsyms]  [g] apparmor_file_open
     1.97%  :197969  [guest.kernel.kallsyms]  [g] __do_sys_kill
     1.97%  :197969  [guest.kernel.kallsyms]  [g] next_positive.isra.14
     1.97%  :197970  [guest.kernel.kallsyms]  [g] _raw_spin_lock
     1.64%  :197969  [guest.kernel.kallsyms]  [g] __alloc_file
     1.64%  :197969  [guest.kernel.kallsyms]  [g] common_file_perm
     1.64%  :197969  [guest.kernel.kallsyms]  [g] filldir
     1.64%  :197970  [guest.kernel.kallsyms]  [g] do_dentry_open
     1.64%  :197970  [guest.kernel.kallsyms]  [g] kmem_cache_free
     1.32%  :197969  [guest.kernel.kallsyms]  [g] =
__raw_callee_save___pv_queued_spin_unlock
     1.32%  :197969  [guest.kernel.kallsyms]  [g] __slab_free

Regards,
Nadav=