Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754688Ab0DNDZR (ORCPT ); Tue, 13 Apr 2010 23:25:17 -0400 Received: from mga14.intel.com ([143.182.124.37]:4534 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754634Ab0DNDZO convert rfc822-to-8bit (ORCPT ); Tue, 13 Apr 2010 23:25:14 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.52,202,1270450800"; d="scan'208";a="265388566" From: "Zhang, Xiantao" To: Avi Kivity CC: "kvm@vger.kernel.org" , Marcelo Tosatti , "Yang, Xiaowei" , "Dong, Eddie" , "Li, Xin" , Ingo Molnar , Peter Zijlstra , Mike Galbraith , Linux Kernel Mailing List Date: Wed, 14 Apr 2010 11:24:26 +0800 Subject: RE: VM performance issue in KVM guests. Thread-Topic: VM performance issue in KVM guests. Thread-Index: Acra1RJBa7/WsUXHSmCNX+Wx4nKFQQAqjs2Q Message-ID: References: <4BC0D125.9050108@redhat.com> <4BC2C07B.4040607@redhat.com> <4BC41338.80206@redhat.com> In-Reply-To: <4BC41338.80206@redhat.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 8BIT MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2729 Lines: 56 Avi Kivity wrote: > On 04/13/2010 03:50 AM, Zhang, Xiantao wrote: >> Avi Kivity wrote: >> >>> On 04/12/2010 05:04 AM, Zhang, Xiantao wrote: >>> >>>> >>>>> What was the performance hit? What was your I/O setup (image >>>>> format, using aio?) >>>>> >>>>> >>>> The issue only happens when vcpu number is over-committed(e.g. >>>> vcpu/pcpu>2) and physical cpus are saturated. For example, when >>>> run webbench in windows OS in this case, its performance drops by >>>> 80%. In our experiment, we are using image file through virtio, >>>> and I think aio should be used by default also. >>>> >>>> >>> Is this on a machine that does pause-loop exits? The current >>> handing of PLE is very suboptimal. With proper directed yield we >>> should be much better there. >>> >>> Without PLE, we need paravirtualized spinlocks, no way around it. >>> >> PLE has the ability to eliminate the issue at some extent, and pv >> solution should be helpful also. But for windows guests running on >> machines without PLE, we still needs to enhance host side to resolve >> the issue. >> > > Well, was this on a machine with PLE or without PLE? I am saying the machine has no PLE feature support. Even with PLE feature support, there is still performance loss due to PLE's cost. >>> Spin loops need to be addressed first, they are known to kill >>> performance in overcommit situations. >>> >> Even in overcommit case, if vcpu threads of one qemu are not >> scheduled or pulled to the same logical processor, the performance >> drop is tolerant like Xen's case today. But for KVM, it has to >> suffer from additional performance loss, since host's scheduler >> actively pulls these vcpu threads together. >> >> > Can you quantify this loss? Give examples of what happens? For example, one machine is configured with 2 pCPUs and there are two Windows guests running on the machine, and each guest is cconfigured with 2 vcpus and one webbench server runs in it. If use host's default scheduler, webbench's performance is very bad, but if pin each geust's vCPU0 to pCPU0 and vCPU1 to pCPU1, we can see 5-10X performance improvement with same CPU utilization. In addition, we also see kvm's perf scalability is also impacted in large systems, for some performance experiments, kvm's perf begins to drop when vCPU is overcommitted and pCPU are saturated, but once the wake_up_affine feature is switched off in scheduler, kvm's perf can keep rising in this case. Xiantao -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/