Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757263Ab0DPC2Z (ORCPT ); Thu, 15 Apr 2010 22:28:25 -0400 Received: from mga14.intel.com ([143.182.124.37]:12602 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753516Ab0DPC2X convert rfc822-to-8bit (ORCPT ); Thu, 15 Apr 2010 22:28:23 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.52,216,1270450800"; d="scan'208";a="266356384" From: "Zhang, Xiantao" To: Avi Kivity CC: "kvm@vger.kernel.org" , Marcelo Tosatti , "Yang, Xiaowei" , "Dong, Eddie" , "Li, Xin" , Ingo Molnar , Peter Zijlstra , Mike Galbraith , Linux Kernel Mailing List Date: Fri, 16 Apr 2010 10:27:14 +0800 Subject: RE: VM performance issue in KVM guests. Thread-Topic: VM performance issue in KVM guests. Thread-Index: AcrbqoCEuPequYdhSdScPLAFwf/1ogBYXdZw Message-ID: References: <4BC0D125.9050108@redhat.com> <4BC2C07B.4040607@redhat.com> <4BC41338.80206@redhat.com> <4BC57949.2090003@redhat.com> In-Reply-To: <4BC57949.2090003@redhat.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 8BIT MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2716 Lines: 60 Avi Kivity wrote: > On 04/14/2010 06:24 AM, Zhang, Xiantao wrote: >> >>>>> Spin loops need to be addressed first, they are known to kill >>>>> performance in overcommit situations. >>>>> >>>>> >>>> Even in overcommit case, if vcpu threads of one qemu are not >>>> scheduled or pulled to the same logical processor, the performance >>>> drop is tolerant like Xen's case today. But for KVM, it has to >>>> suffer from additional performance loss, since host's scheduler >>>> actively pulls these vcpu threads together. >>>> >>>> >>>> >>> Can you quantify this loss? Give examples of what happens? >>> >> For example, one machine is configured with 2 pCPUs and there are >> two Windows guests running on the machine, and each guest is >> cconfigured with 2 vcpus and one webbench server runs in it. >> If use host's default scheduler, webbench's performance is very bad, >> but if pin each geust's vCPU0 to pCPU0 and vCPU1 to pCPU1, we can >> see 5-10X performance improvement with same CPU utilization. >> In addition, we also see kvm's perf scalability is also impacted in >> large systems, for some performance experiments, kvm's perf begins >> to drop when vCPU is overcommitted and pCPU are saturated, but once >> the wake_up_affine feature is switched off in scheduler, kvm's perf >> can keep rising in this case. >> > > Ok. This is probably due to spinlock contention. Yes, exactly. > When vcpus are pinned to pcpus, there is a 50% chance that a guest's > vcpus will be co-scheduled and spinlocks will perform will. > > When vcpus are not pinned, but affine wakeups are disabled, there is a > 33% chance that vcpus will be co-scheduled. > > When vcpus are not pinned and affine wakeups are enabled there is a 0% > chance that vcpus will be co-scheduled. > > Keeping both vcpus on the same core actually makes sense since they > can communicate through the local cache faster than across cores. > What we need is to make sure that they don't spin. > > Windows 2008 can report spinlock spinning through a hypercall. Can > you hook to that interface and see if it happens regularly? > Altenatively use a PLE capable host and trace the kvm_vcpu_on_spin() > function. We only tried windows 2003 for the experiments, and have no data related to windows 2008. But maybe we can have a try later. Anyway, the key point is we have to enhance the scheduler to let it Know which threads are vcpu threads to avoid perf loss in this case. Xiantao -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/