Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Thu, 24 Sep 2020 10:42:15 +0200
From:   peterz@infradead.org
To:     George Prekas <prekageo@amazon.com>
Cc:     Ingo Molnar <mingo@kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
        Peter Xu <peterx@redhat.com>,
        Kaitao Cheng <pilgrimtao@gmail.com>,
        <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] latency improvement in __smp_call_single_queue
Message-ID: <20200924084215.GD1362448@hirez.programming.kicks-ass.net>
References: <281da382-4511-e1df-6917-154a5914dd43@amazon.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <281da382-4511-e1df-6917-154a5914dd43@amazon.com>
Precedence: bulk

On Wed, Sep 23, 2020 at 10:00:41AM -0500, George Prekas wrote:
> If an interrupt arrives between llist_add and
> send_call_function_single_ipi in the following code snippet, then the
> remote CPU will not receive the IPI in a timely manner and subsequent
> SMP calls even from other CPUs for other functions will be delayed:
> 
> ??? if (llist_add(node, &per_cpu(call_single_queue, cpu)))
> ??????? send_call_function_single_ipi(cpu);
> 
> Note: llist_add returns 1 if it was empty before the operation.
> 
> CPU 0?????????????????????????? | CPU 1???????????????????? | CPU 2
> __smp_call_single_q(2,f1)?????? | __smp_call_single_q(2,f2) |
> ? llist_add returns 1?????????? |?????????????????????????? |
> ? interrupted?????????????????? |?? llist_add returns 0???? |
> ????? ...?????????????????????? |?? branch not taken??????? |
> ????? ...?????????????????????? |?????????????????????????? |
> ? resumed?????????????????????? |?????????????????????????? |
> ? send_call_function_single_ipi |?????????????????????????? |
> ??????????????????????????????? |?????????????????????????? | f1
> ??????????????????????????????? |?????????????????????????? | f2
> 
> The call from CPU 1 for function f2 will be delayed because CPU 0 was
> interrupted.

Do you happen to have any actual numbers and a use-case where this was
relevant?