Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751432AbdGQMxA (ORCPT ); Mon, 17 Jul 2017 08:53:00 -0400 Received: from mail-oi0-f67.google.com ([209.85.218.67]:33923 "EHLO mail-oi0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751349AbdGQMw5 (ORCPT ); Mon, 17 Jul 2017 08:52:57 -0400 Subject: Re: [PATCH 2/2] x86/idle: use dynamic halt poll To: Alexander Graf , =?UTF-8?B?UmFkaW0gS3LEjW3DocWZ?= References: <4444ffc8-9e7b-5bd2-20da-af422fe834cc@redhat.com> <2245bef7-b668-9265-f3f8-3b63d71b1033@gmail.com> <7d085956-2573-212f-44f4-86104beba9bb@gmail.com> <05ec7efc-fb9c-ae24-5770-66fc472545a4@redhat.com> <20170627134043.GA1487@potion> <2771f905-d1b0-b118-9ae9-db5fb87f877c@redhat.com> <20170627142251.GB1487@potion> <20170704141322.GC30880@potion> Cc: Paolo Bonzini , Wanpeng Li , Thomas Gleixner , Ingo Molnar , "H. Peter Anvin" , the arch/x86 maintainers , Jonathan Corbet , tony.luck@intel.com, Borislav Petkov , Peter Zijlstra , mchehab@kernel.org, Andrew Morton , krzk@kernel.org, jpoimboe@redhat.com, Andy Lutomirski , Christian Borntraeger , Thomas Garnier , Robert Gerst , Mathias Krause , douly.fnst@cn.fujitsu.com, Nicolai Stange , Frederic Weisbecker , dvlasenk@redhat.com, Daniel Bristot de Oliveira , yamada.masahiro@socionext.com, mika.westerberg@linux.intel.com, Chen Yu , aaron.lu@intel.com, Steven Rostedt , Kyle Huey , Len Brown , Prarit Bhargava , hidehiro.kawai.ez@hitachi.com, fengtiantian@huawei.com, pmladek@suse.com, jeyu@redhat.com, Larry.Finger@lwfinger.net, zijun_hu@htc.com, luisbg@osg.samsung.com, johannes.berg@intel.com, niklas.soderlund+renesas@ragnatech.se, zlpnobody@gmail.com, Alexey Dobriyan , fgao@48lvckh6395k16k5.yundunddos.com, ebiederm@xmission.com, Subash Abhinov Kasiviswanathan , Arnd Bergmann , Matt Fleming , Mel Gorman , "linux-kernel@vger.kernel.org" , linux-doc@vger.kernel.org, linux-edac@vger.kernel.org, kvm From: Yang Zhang Message-ID: Date: Mon, 17 Jul 2017 20:50:03 +0800 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 5391 Lines: 139 On 2017/7/17 17:54, Alexander Graf wrote: > > > On 17.07.17 11:26, Yang Zhang wrote: >> On 2017/7/14 17:37, Alexander Graf wrote: >>> >>> >>> On 13.07.17 13:49, Yang Zhang wrote: >>>> On 2017/7/4 22:13, Radim Krčmář wrote: >>>>> 2017-07-03 17:28+0800, Yang Zhang: >>>>>> The background is that we(Alibaba Cloud) do get more and more >>>>>> complaints >>>>>> from our customers in both KVM and Xen compare to bare-mental.After >>>>>> investigations, the root cause is known to us: big cost in message >>>>>> passing >>>>>> workload(David show it in KVM forum 2015) >>>>>> >>>>>> A typical message workload like below: >>>>>> vcpu 0 vcpu 1 >>>>>> 1. send ipi 2. doing hlt >>>>>> 3. go into idle 4. receive ipi and wake up from hlt >>>>>> 5. write APIC time twice 6. write APIC time twice to >>>>>> to stop sched timer reprogram sched timer >>>>> >>>>> One write is enough to disable/re-enable the APIC timer -- why does >>>>> Linux use two? >>>> >>>> One is to remove the timer and another one is to reprogram the timer. >>>> Normally, only one write to remove the timer.But in some cases, it >>>> will reprogram it. >>>> >>>>> >>>>>> 7. doing hlt 8. handle task and send ipi to >>>>>> vcpu 0 >>>>>> 9. same to 4. 10. same to 3 >>>>>> >>>>>> One transaction will introduce about 12 vmexits(2 hlt and 10 msr >>>>>> write). The >>>>>> cost of such vmexits will degrades performance severely. >>>>> >>>>> Yeah, sounds like too much ... I understood that there are >>>>> >>>>> IPI from 1 to 2 >>>>> 4 * APIC timer >>>>> IPI from 2 to 1 >>>>> >>>>> which adds to 6 MSR writes -- what are the other 4? >>>> >>>> In the worst case, each timer will touch APIC timer twice.So it will >>>> add additional 4 msr writse. But this is not always true. >>>> >>>>> >>>>>> Linux kernel >>>>>> already provide idle=poll to mitigate the trend. But it only >>>>>> eliminates the >>>>>> IPI and hlt vmexit. It has nothing to do with start/stop sched >>>>>> timer. A >>>>>> compromise would be to turn off NOHZ kernel, but it is not the >>>>>> default >>>>>> config for new distributions. Same for halt-poll in KVM, it only >>>>>> solve the >>>>>> cost from schedule in/out in host and can not help such workload >>>>>> much. >>>>>> >>>>>> The purpose of this patch we want to improve current idle=poll >>>>>> mechanism to >>>>> >>>>> Please aim to allow MWAIT instead of idle=poll -- MWAIT doesn't slow >>>>> down the sibling hyperthread. MWAIT solves the IPI problem, but >>>>> doesn't >>>>> get rid of the timer one. >>>> >>>> Yes, i can try it. But MWAIT will not yield CPU, it only helps the >>>> sibling hyperthread as you mentioned. >>> >>> If you implement proper MWAIT emulation that conditionally gets en- or >>> disabled depending on the same halt poll dynamics that we already have >>> for in-host HLT handling, it will also yield the CPU. >> >> It is hard to do . If we not intercept MWAIT instruction, there is no >> chance to wake up the CPU unless an interrupt arrived or a store to >> the address armed by MONITOR which is the same with idle=polling. > > Yes, but you can reconfigure the VMCS/VMCB to trap on MWAIT or not trap > on it. That's something that idle=polling does not give you at all - a > guest vcpu will always use 100% CPU. There are two things we need to figure out: 1. How and when to reconfigure the VMCS? Currently, all the knowledge are from guest, we don't know when to reconfigure it. Also, we cannot prevent guest from using MWAIT in other place if it see the feature. 2. If guest execute MWAIT without trap, since there is no way to set timeout for it, that would be a waste of CPU too. > > The only really tricky part is how to limit the effect of MONITOR on > nested page table maintenance. But if we just set the MONITOR cache size > to 4k, well behaved guests should ideally always give us the one same > page for wakeup - which we can then leave marked as trapping. > >> >>> >>> As for the timer - are you sure the problem is really the overhead of >>> the timer configuration, not the latency that it takes to actually fire >>> the guest timer? >> >> No, the main cost is introduced by vmexit, includes IPIs, Timer >> program, HLT. David detailed it in KVM forum, you can search "Message >> Passing Workloads in KVM" in google and the first link give the whole >> analysis of the problem. > > During time critical message passing you want to keep both vCPUs inside > the guest, yes. That again is something that guest exposed MWAIT would > buy you. I think MWAIT only helps sibling hyper-threading case. But in real Cloud, hyper-threading is not always turning on, i.e. most products of Azure and some products of Alibaba Cloud. So it shouldn't be a big problem. > > The problem is that overcommitting CPU is very expensive with anything > that does not set the guests idle at all. And not everyone can afford to > throw more CPUs at problems :). Agree, that's the reason why we choose dynamically halt polling. But on other side, the cloud vendor has the knowledge to control whether turn on it or not. The only problem is that there is no such way for us to do currently. > > > Alex -- Yang Alibaba Cloud Computing