Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Thomas Gleixner <tglx@linutronix.de>
To:     Alexander Graf <graf@amazon.com>,
        LKML <linux-kernel@vger.kernel.org>
Cc:     Andy Lutomirski <luto@kernel.org>,
        Andrew Cooper <andrew.cooper3@citrix.com>,
        X86 ML <x86@kernel.org>,
        "Paul E. McKenney" <paulmck@kernel.org>,
        Alexandre Chartre <alexandre.chartre@oracle.com>,
        Frederic Weisbecker <frederic@kernel.org>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Sean Christopherson <sean.j.christopherson@intel.com>,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Petr Mladek <pmladek@suse.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Joel Fernandes <joel@joelfernandes.org>,
        Boris Ostrovsky <boris.ostrovsky@oracle.com>,
        Juergen Gross <jgross@suse.com>,
        Brian Gerst <brgerst@gmail.com>,
        Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
        Josh Poimboeuf <jpoimboe@redhat.com>,
        Will Deacon <will@kernel.org>,
        Tom Lendacky <thomas.lendacky@amd.com>,
        Wei Liu <wei.liu@kernel.org>,
        Michael Kelley <mikelley@microsoft.com>,
        Jason Chen CJ <jason.cj.chen@intel.com>,
        Zhao Yakui <yakui.zhao@intel.com>,
        "Peter Zijlstra \(Intel\)" <peterz@infradead.org>,
        avi@scylladb.com, "Herrenschmidt\, Benjamin" <benh@amazon.com>,
        robketr@amazon.de, amos@scylladb.com
Subject: Re: [patch V9 21/39] x86/irq: Convey vector as argument and not in ptregs
In-Reply-To: <f0079706-4cb3-b3e3-9a5e-97aaba0aa0eb@amazon.com>
References: <20200521200513.656533920@linutronix.de> <20200521202118.796915981@linutronix.de> <f0079706-4cb3-b3e3-9a5e-97aaba0aa0eb@amazon.com>
Date:   Tue, 25 Aug 2020 12:28:24 +0200
Message-ID: <87a6yj58af.fsf@nanos.tec.linutronix.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

Alex,

On Mon, Aug 24 2020 at 19:29, Alexander Graf wrote:
> I'm currently trying to understand a performance regression with=20
> ScyllaDB on i3en.3xlarge (KVM based VM on Skylake) which we reliably=20
> bisected down to this commit:
>
>    https://github.com/scylladb/scylla/issues/7036
>
> What we're seeing is that syscalls such as membarrier() take forever=20
> (0-10 =C2=B5s would be normal):
...
> That again seems to stem from a vastly slowed down=20
> smp_call_function_many_cond():
>
> Samples: 218K of event 'cpu-clock', 4000 Hz
> Overhead  Shared Object        Symbol
>    94.51%  [kernel]             [k] smp_call_function_many_cond
>     0.76%  [kernel]             [k] __do_softirq
>     0.32%  [kernel]             [k] native_queued_spin_lock_slowpath
> [...]
>
> which is stuck in
>
>         =E2=94=82     csd_lock_wait():
>         =E2=94=82             smp_cond_load_acquire(&csd->flags, !(VAL &
>    0.00 =E2=94=82       mov    0x8(%rcx),%edx
>    0.00 =E2=94=82       and    $0x1,%edx
>         =E2=94=82     =E2=86=93 je     2b9
>         =E2=94=82     rep_nop():
>    0.70 =E2=94=822af:   pause
>         =E2=94=82     csd_lock_wait():
>   92.82 =E2=94=82       mov    0x8(%rcx),%edx
>    6.48 =E2=94=82       and    $0x1,%edx
>    0.00 =E2=94=82     =E2=86=91 jne    2af
>    0.00 =E2=94=822b9: =E2=86=91 jmp    282
>
>
> Given the patch at hand I was expecting lost IPIs, but I can't quite see=
=20
> anything getting lost.

I have no idea how that patch should be related to IPI and smp function
calls. It's changing the way how regular device interrupts and their
spurious counterpart are handled and not the way how IPIs are
handled. They are handled by direct vectors and do not go through
do_IRQ() at all.

Aside of that the commit just changes the way how the interrupt vector
of a regular device interrupt is stored and conveyed. The extra read and
write on the cache hot stack is hardly related to anything IPI.

Thanks,

        tglx