Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [PATCH] latency improvement in __smp_call_single_queue
To:     <peterz@infradead.org>
CC:     Ingo Molnar <mingo@kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
        Peter Xu <peterx@redhat.com>,
        Kaitao Cheng <pilgrimtao@gmail.com>,
        <linux-kernel@vger.kernel.org>
References: <281da382-4511-e1df-6917-154a5914dd43@amazon.com>
 <20200924084215.GD1362448@hirez.programming.kicks-ass.net>
From:   George Prekas <prekageo@amazon.com>
Message-ID: <85f7c7c5-2ab0-d5df-45a7-989be1286ef9@amazon.com>
Date:   Thu, 24 Sep 2020 10:04:43 -0500
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101
 Thunderbird/68.12.0
MIME-Version: 1.0
In-Reply-To: <20200924084215.GD1362448@hirez.programming.kicks-ass.net>
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-US
Precedence: bulk

On 9/24/2020 3:42 AM, peterz@infradead.org wrote:
 > On Wed, Sep 23, 2020 at 10:00:41AM -0500, George Prekas wrote:
 >> If an interrupt arrives between llist_add and
 >> send_call_function_single_ipi in the following code snippet, then the
 >> remote CPU will not receive the IPI in a timely manner and subsequent
 >> SMP calls even from other CPUs for other functions will be delayed:
 >>
 >>     if (llist_add(node, &per_cpu(call_single_queue, cpu)))
 >>         send_call_function_single_ipi(cpu);
 >>
 >> Note: llist_add returns 1 if it was empty before the operation.
 >>
 >> CPU 0                           | CPU 1 | CPU 2
 >> __smp_call_single_q(2,f1)       | __smp_call_single_q(2,f2) |
 >>   llist_add returns 1           | |
 >>   interrupted                   |   llist_add returns 0 |
 >>       ...                       |   branch not taken |
 >>       ...                       | |
 >>   resumed                       | |
 >>   send_call_function_single_ipi | |
 >>                                 | | f1
 >>                                 | | f2
 >>
 >> The call from CPU 1 for function f2 will be delayed because CPU 0 was
 >> interrupted.
 >
 > Do you happen to have any actual numbers and a use-case where this was
 > relevant?

Hi Peter,

I encountered this problem while developing a device driver that used 
smp_call_function_single to communicate with other cores. I observed 
latency spikes and after investigation I figured out the problem 
described above.

I have written a simple device driver to validate the above fix. It does 
smp_call_function_single and measures the latency. I can post it here if 
it is appropriate. The latency impact is equal to the duration of the 
CPU 0's interruption.

--
George