Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: [RFC PATCH 1/2] x86/ibpb: Skip IBPB when we switch back to same
 user process
To:     Peter Zijlstra <peterz@infradead.org>
Cc:     Tim Chen <tim.c.chen@linux.intel.com>,
        linux-kernel@vger.kernel.org,
        KarimAllah Ahmed <karahmed@amazon.de>,
        Andi Kleen <ak@linux.intel.com>,
        Andrea Arcangeli <aarcange@redhat.com>,
        Andy Lutomirski <luto@kernel.org>,
        Ashok Raj <ashok.raj@intel.com>,
        Asit Mallick <asit.k.mallick@intel.com>,
        Borislav Petkov <bp@suse.de>,
        Dan Williams <dan.j.williams@intel.com>,
        Dave Hansen <dave.hansen@intel.com>,
        David Woodhouse <dwmw@amazon.co.uk>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        "H . Peter Anvin" <hpa@zytor.com>, Ingo Molnar <mingo@redhat.com>,
        Janakarajan Natarajan <Janakarajan.Natarajan@amd.com>,
        Joerg Roedel <joro@8bytes.org>,
        Jun Nakajima <jun.nakajima@intel.com>,
        Laura Abbott <labbott@redhat.com>,
        Linus Torvalds <torvalds@linux-foundation.org>,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Paolo Bonzini <pbonzini@redhat.com>, rkrcmar@redhat.com,
        Thomas Gleixner <tglx@linutronix.de>,
        Tom Lendacky <thomas.lendacky@amd.com>, x86@kernel.org
References: <c5cc429013c2248ddebe0a8480b172ae70b29733.1516840211.git.tim.c.chen@linux.intel.com>
 <20180125085820.GV2228@hirez.programming.kicks-ass.net>
 <20180125092233.GE2295@hirez.programming.kicks-ass.net>
 <86541aca-8de7-163d-b620-083dddf29184@linux.intel.com>
 <20180125135055.GK2249@hirez.programming.kicks-ass.net>
From:   Arjan van de Ven <arjan@linux.intel.com>
Message-ID: <aeef4a08-7dd3-3d3a-02f4-9b8b4a1461bf@linux.intel.com>
Date:   Thu, 25 Jan 2018 06:07:07 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101
 Thunderbird/52.5.2
MIME-Version: 1.0
In-Reply-To: <20180125135055.GK2249@hirez.programming.kicks-ass.net>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On 1/25/2018 5:50 AM, Peter Zijlstra wrote:
> On Thu, Jan 25, 2018 at 05:21:30AM -0800, Arjan van de Ven wrote:
>>>
>>> This means that 'A -> idle -> A' should never pass through switch_mm to
>>> begin with.
>>>
>>> Please clarify how you think it does.
>>>
>>
>> the idle code does leave_mm() to avoid having to IPI CPUs in deep sleep states
>> for a tlb flush.
> 
> The intel_idle code does, not the idle code. This is squirreled away in
> some driver :/

afaik (but haven't looked in a while) acpi drivers did too
> 
>> (trust me, that you really want, sequentially IPI's a pile of cores in a deep sleep
>> state to just flush a tlb that's empty, the performance of that is horrific)
> 
> Hurmph. I'd rather fix that some other way than leave_mm(), this is
> piling special on special.
> 
the problem was tricky. but of course if something better is possible lets figure this out

problem is that an IPI to an idle cpu is both power inefficient and will take time,
exit of a deep C state can be, say 50 to 100 usec range of time (it varies by many things, but
for abstractly thinking about the problem one should generally round up to nice round numbers)

if you have say 64 cores that had the mm at some point, but 63 are in idle, the 64th
really does not want to IPI each of those 63 serially (technically this is does not need
to be serial but IPI code is tricky, some things end up serializing this a bit)
to get the 100 usec hit 63 times. Actually, even if it's not serialized, even ONE hit of 100 usec
is unpleasant.

so a CPU that goes idle wants to "unsubscribe" itself from those IPIs as general objective.

but not getting flush IPIs is only safe if the TLBs in the CPU have nothing that such IPI would
want to flush, so the TLB needs to be empty of those things.

the only way to do THAT is to switch to an mm that is safe; a leave_mm() does this, but I'm sure other
options exist.

note: While a CPU that is in a deeper C state will itself flush the TLB, you don't know if you will actually
enter that deep at the time of making OS decisions (if an interrupt comes in the cycle before mwait, mwait
becomes a nop for example). In addition, once you wake up, you don't want the CPU to go start filling
the TLBs with invalid data so you can't really just set a bit and flush after leaving idle