Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751922AbaJ0S4j (ORCPT ); Mon, 27 Oct 2014 14:56:39 -0400 Received: from userp1040.oracle.com ([156.151.31.81]:50059 "EHLO userp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751163AbaJ0S4i (ORCPT ); Mon, 27 Oct 2014 14:56:38 -0400 Message-ID: <544E95D5.1020000@oracle.com> Date: Mon, 27 Oct 2014 14:58:29 -0400 From: Boris Ostrovsky User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Konrad Rzeszutek Wilk CC: david.vrabel@citrix.com, stefano.stabellini@eu.citrix.com, jun.nakajima@intel.com, linux-kernel@vger.kernel.org, xen-devel@lists.xen.org Subject: Re: [PATCH v2 2/2] xen/pci: Use APIC directly when APIC virtualization is supported by hardware References: <1414425953-13987-1-git-send-email-boris.ostrovsky@oracle.com> <1414425953-13987-3-git-send-email-boris.ostrovsky@oracle.com> <20141027183939.GB13854@laptop.dumpdata.com> In-Reply-To: <20141027183939.GB13854@laptop.dumpdata.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Source-IP: ucsinet21.oracle.com [156.151.31.93] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 10/27/2014 02:39 PM, Konrad Rzeszutek Wilk wrote: > On Mon, Oct 27, 2014 at 12:05:53PM -0400, Boris Ostrovsky wrote: >> When hardware supports APIC/x2APIC virtualization we don't need to use pirqs >> for MSI handling and instead use APIC since most APIC accesses (MMIO or MSR) >> will now be processed without VMEXITs. >> >> As an example, netperf on the original code produces this profile >> (collected wih 'xentrace -e 0x0008ffff -T 5'): >> >> 342 cpu_change >> 260 CPUID >> 34638 HLT >> 64067 INJ_VIRQ >> 28374 INTR >> 82733 INTR_WINDOW >> 10 NPF >> 24337 TRAP >> 370610 vlapic_accept_pic_intr >> 307528 VMENTRY >> 307527 VMEXIT >> 140998 VMMCALL >> 127 wrap_buffer >> >> After applying this patch the same test shows >> >> 230 cpu_change >> 260 CPUID >> 36542 HLT >> 174 INJ_VIRQ >> 27250 INTR >> 222 INTR_WINDOW >> 20 NPF >> 24999 TRAP >> 381812 vlapic_accept_pic_intr >> 166480 VMENTRY >> 166479 VMEXIT >> 77208 VMMCALL >> 81 wrap_buffer >> >> ApacheBench results (ab -n 10000 -c 200) improve by about 10% > What type of NIC did you use for this? Intel I350 card, igb driver. I probably should have mentioned this in the commit message. -boris > > Thanks! >> Signed-off-by: Boris Ostrovsky >> --- >> arch/x86/include/asm/xen/cpuid.h | 91 ++++++++++++++++++++++++++++++++++++++ >> arch/x86/pci/xen.c | 9 ++++ >> 2 files changed, 100 insertions(+), 0 deletions(-) >> create mode 100644 arch/x86/include/asm/xen/cpuid.h >> >> diff --git a/arch/x86/include/asm/xen/cpuid.h b/arch/x86/include/asm/xen/cpuid.h >> new file mode 100644 >> index 0000000..0d809e9 >> --- /dev/null >> +++ b/arch/x86/include/asm/xen/cpuid.h >> @@ -0,0 +1,91 @@ >> +/****************************************************************************** >> + * arch-x86/cpuid.h >> + * >> + * CPUID interface to Xen. >> + * >> + * Permission is hereby granted, free of charge, to any person obtaining a copy >> + * of this software and associated documentation files (the "Software"), to >> + * deal in the Software without restriction, including without limitation the >> + * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or >> + * sell copies of the Software, and to permit persons to whom the Software is >> + * furnished to do so, subject to the following conditions: >> + * >> + * The above copyright notice and this permission notice shall be included in >> + * all copies or substantial portions of the Software. >> + * >> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR >> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, >> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE >> + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER >> + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING >> + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER >> + * DEALINGS IN THE SOFTWARE. >> + * >> + * Copyright (c) 2007 Citrix Systems, Inc. >> + * >> + * Authors: >> + * Keir Fraser >> + */ >> + >> +#ifndef __XEN_PUBLIC_ARCH_X86_CPUID_H__ >> +#define __XEN_PUBLIC_ARCH_X86_CPUID_H__ >> + >> +/* >> + * For compatibility with other hypervisor interfaces, the Xen cpuid leaves >> + * can be found at the first otherwise unused 0x100 aligned boundary starting >> + * from 0x40000000. >> + * >> + * e.g If viridian extensions are enabled for an HVM domain, the Xen cpuid >> + * leaves will start at 0x40000100 >> + */ >> + >> +#define XEN_CPUID_FIRST_LEAF 0x40000000 >> +#define XEN_CPUID_LEAF(i) (XEN_CPUID_FIRST_LEAF + (i)) >> + >> +/* >> + * Leaf 1 (0x40000x00) >> + * EAX: Largest Xen-information leaf. All leaves up to an including @EAX >> + * are supported by the Xen host. >> + * EBX-EDX: "XenVMMXenVMM" signature, allowing positive identification >> + * of a Xen host. >> + */ >> +#define XEN_CPUID_SIGNATURE_EBX 0x566e6558 /* "XenV" */ >> +#define XEN_CPUID_SIGNATURE_ECX 0x65584d4d /* "MMXe" */ >> +#define XEN_CPUID_SIGNATURE_EDX 0x4d4d566e /* "nVMM" */ >> + >> +/* >> + * Leaf 2 (0x40000x01) >> + * EAX[31:16]: Xen major version. >> + * EAX[15: 0]: Xen minor version. >> + * EBX-EDX: Reserved (currently all zeroes). >> + */ >> + >> +/* >> + * Leaf 3 (0x40000x02) >> + * EAX: Number of hypercall transfer pages. This register is always guaranteed >> + * to specify one hypercall page. >> + * EBX: Base address of Xen-specific MSRs. >> + * ECX: Features 1. Unused bits are set to zero. >> + * EDX: Features 2. Unused bits are set to zero. >> + */ >> + >> +/* Does the host support MMU_PT_UPDATE_PRESERVE_AD for this guest? */ >> +#define _XEN_CPUID_FEAT1_MMU_PT_UPDATE_PRESERVE_AD 0 >> +#define XEN_CPUID_FEAT1_MMU_PT_UPDATE_PRESERVE_AD (1u<<0) >> + >> +/* >> + * Leaf 5 (0x40000x04) >> + * HVM-specific features >> + */ >> + >> +/* EAX Features */ >> +/* Virtualized APIC registers */ >> +#define XEN_HVM_CPUID_APIC_ACCESS_VIRT (1u << 0) >> +/* Virtualized x2APIC accesses */ >> +#define XEN_HVM_CPUID_X2APIC_VIRT (1u << 1) >> +/* Memory mapped from other domains has valid IOMMU entries */ >> +#define XEN_HVM_CPUID_IOMMU_MAPPINGS (1u << 2) >> + >> +#define XEN_CPUID_MAX_NUM_LEAVES 4 >> + >> +#endif /* __XEN_PUBLIC_ARCH_X86_CPUID_H__ */ >> diff --git a/arch/x86/pci/xen.c b/arch/x86/pci/xen.c >> index 1370716..d768371 100644 >> --- a/arch/x86/pci/xen.c >> +++ b/arch/x86/pci/xen.c >> @@ -23,6 +23,7 @@ >> #include >> #include >> #include >> +#include >> #include >> >> static int xen_pcifront_enable_irq(struct pci_dev *dev) >> @@ -434,6 +435,14 @@ int __init pci_xen_init(void) >> #ifdef CONFIG_PCI_MSI >> void __init xen_msi_init(void) >> { >> + if (!disable_apic) { >> + uint32_t eax = cpuid_eax(xen_cpuid_base() + 4); >> + >> + if (((eax & XEN_HVM_CPUID_X2APIC_VIRT) && x2apic_mode) || >> + ((eax & XEN_HVM_CPUID_APIC_ACCESS_VIRT) && cpu_has_apic)) >> + return; >> + } >> + >> x86_msi.setup_msi_irqs = xen_hvm_setup_msi_irqs; >> x86_msi.teardown_msi_irq = xen_teardown_msi_irq; >> } >> -- >> 1.7.1 >> -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/