Received: by 2002:ac0:a5a7:0:0:0:0:0 with SMTP id m36-v6csp2236728imm; Thu, 19 Jul 2018 16:07:17 -0700 (PDT) X-Google-Smtp-Source: AAOMgpeQ5vEF/Aq0GLFKz5MdNyIL2MISmz5S1vqa4AxzIIU+IpzYOdFOAiPkPKEdCTVQmvUf7rZJ X-Received: by 2002:a65:6699:: with SMTP id b25-v6mr11888166pgw.426.1532041637878; Thu, 19 Jul 2018 16:07:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1532041637; cv=none; d=google.com; s=arc-20160816; b=M/hDgPe5oE5RQixZ0W3A5y4N10jmB33tJQOH65ba53aal76lZe5XUNzejxWnCleCkl PKbzFEptL+XoFv6Hb3nWrgasL33zDpGBR2HdQKGNL2YljG9r0enZ/pDQFF3IIVU4euvM fY51J884sB7yPpB978LLQ84aXb4WYldy+CSX/VI7kVQBkYufkfylyg9W4RPJBjy21fE9 Z1F71oJW5A4VPQMJ4HSWb5PTDhC9a0JsdBPDWXmdMu3U5FlKulrIqGrcupUgxywuB7ib yDUVmmmN05EKu6lyjLEdTXwJ6FQXx4RPhWesJub9+GbbWEGl4nO/jOO26p28TcSXi8VO jpDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:cc:to:subject :message-id:date:from:in-reply-to:references:mime-version :dkim-signature:arc-authentication-results; bh=GNDXFinDuqjRE/pBPRFb1LMrWIfFGzOYlNzCBu0aQ6M=; b=W4DiwPdMise91NCkycre6UfoWR9V0hBx9l+koltz1hvoLjo2UUjnnJ3UsaG/XToxcF jj1tIpDVnIFB8zyCu3ls8pBb8Eo86FrAHHPykLgnFsEzToFFJUfWcPbnuX4fYMj9mN3d xPSL4U+J6yYbegkP/xQUVogFPFAdw+0oQVqObHQ7GiPIN9ZEYvzYRSpXWhGzMTZ90JBG GL/zv+N3PewKjykWoLVuMk7Ml8tdd3EwX7bH8MkglueQoY4ora2sJSlPCE7DSLut9OR3 ABV/go7paxyqy6182nHgfG5M3tfhCAM92LCGU07p/BH+t42QgXBj6O2oLXpNQJyXFZ6q yMuA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=XHcU4HRv; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id m20-v6si341965pgb.392.2018.07.19.16.07.02; Thu, 19 Jul 2018 16:07:17 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=XHcU4HRv; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731213AbeGSXu7 (ORCPT + 99 others); Thu, 19 Jul 2018 19:50:59 -0400 Received: from mail-lf1-f68.google.com ([209.85.167.68]:43890 "EHLO mail-lf1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730998AbeGSXu7 (ORCPT ); Thu, 19 Jul 2018 19:50:59 -0400 Received: by mail-lf1-f68.google.com with SMTP id m12-v6so496837lfc.10 for ; Thu, 19 Jul 2018 16:05:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=GNDXFinDuqjRE/pBPRFb1LMrWIfFGzOYlNzCBu0aQ6M=; b=XHcU4HRvU0othR0WoD+9hFqHY+US/7KzIT+UR1p/cKdo8tPWFOQET8nI/Ak3/7akLT k9kMo9YnJLIq4pA7KWjvQXiacrxHmLFlRUiy1MFnKBNMAo5+HZBk5gynv4GtZQuILVF0 vQ+h62RX8EtAo0pQYMk82y5tkf0Ig04b9Ut7NJijOup/858Tb01slmv/BqOJ2y18iaaa m0wDJyDaN6ak5kTsyTLUP+sROqBePyLXuoY025+eTLbniHakCWch2yKsbg+t+CcyM/v/ va2UiCz5aFhDzYyutt3UomxrZGPzO4xRQRqp7MBsHYaEYSVYoGbvSwBGri1WHjy6cW+F EVqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=GNDXFinDuqjRE/pBPRFb1LMrWIfFGzOYlNzCBu0aQ6M=; b=A2yW8gZ3VO8Qhxxlzfyvr0iLkyjKcjC5Rjf15ObQeDhaQ1BniJNKoM23fGEq3FWplZ gCX1mw+Dh9onjyoAUlXmCXWjxDW7JRv7R0x9uPrXn4GzUbi6pQ6GdouEQlpZ+ui1pR6F dVLjWTPcZhLuviYumrzRwwNYHfOiewgU6VQedySuprF3MBZmiOPJFUb3Z6waGYZevr0w GIKeM0OZXO6S32l0PLBV3x/YLcn6OO37vtGuzlXtKXUYhaN8EtGazhS/ygMRR2z7MmiH iH8HQLIX+mClZTZBbEb5aY8pcCV8fqV5CH2lpg6WPGhmeM3yguoovC/QcFvuOnvxMMm7 jsjQ== X-Gm-Message-State: AOUpUlF45h7AkE5uvyALg25qwqU02gIR0E8aAInJN2f13clkNWf6/EpT d2mkWNmr7M2WahTE98labROArMnO8Vt6fBg3tamRryZX X-Received: by 2002:a19:9d92:: with SMTP id g140-v6mr8193506lfe.85.1532041540279; Thu, 19 Jul 2018 16:05:40 -0700 (PDT) MIME-Version: 1.0 References: <1530598891-21370-1-git-send-email-wanpengli@tencent.com> <1530598891-21370-3-git-send-email-wanpengli@tencent.com> In-Reply-To: <1530598891-21370-3-git-send-email-wanpengli@tencent.com> From: David Matlack Date: Thu, 19 Jul 2018 16:05:14 -0700 Message-ID: Subject: Re: [PATCH v3 2/6] KVM: X86: Implement PV IPIs in linux guest To: Wanpeng Li Cc: linux-kernel@vger.kernel.org, kvm list , Paolo Bonzini , =?UTF-8?B?UmFkaW0gS3LEjW3DocWZ?= , vkuznets@redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Jul 2, 2018 at 11:23 PM Wanpeng Li wrote: > > From: Wanpeng Li > > Implement paravirtual apic hooks to enable PV IPIs. Very cool. Thanks for working on this! > > apic->send_IPI_mask > apic->send_IPI_mask_allbutself > apic->send_IPI_allbutself > apic->send_IPI_all > > The PV IPIs supports maximal 128 vCPUs VM, it is big enough for cloud > environment currently, From the Cloud perspective, 128 vCPUs is already obsolete. GCE's n1-utlramem-160 VMs have 160 vCPUs where the maximum APIC ID is 231. I'd definitely prefer an approach that scales to higher APIC IDs, like Paolo's offset idea. To Radim's point of real world performance testing, do you know what is the primary source of multi-target IPIs? If it's TLB shootdowns we might get a bigger bang for our buck with a PV TLB Shootdown. > supporting more vCPUs needs to introduce more > complex logic, in the future this might be extended if needed. > > Cc: Paolo Bonzini > Cc: Radim Kr=C4=8Dm=C3=A1=C5=99 > Cc: Vitaly Kuznetsov > Signed-off-by: Wanpeng Li > --- > arch/x86/include/uapi/asm/kvm_para.h | 1 + > arch/x86/kernel/kvm.c | 70 ++++++++++++++++++++++++++++++= ++++++ > include/uapi/linux/kvm_para.h | 1 + > 3 files changed, 72 insertions(+) > > diff --git a/arch/x86/include/uapi/asm/kvm_para.h b/arch/x86/include/uapi= /asm/kvm_para.h > index 0ede697..19980ec 100644 > --- a/arch/x86/include/uapi/asm/kvm_para.h > +++ b/arch/x86/include/uapi/asm/kvm_para.h > @@ -28,6 +28,7 @@ > #define KVM_FEATURE_PV_UNHALT 7 > #define KVM_FEATURE_PV_TLB_FLUSH 9 > #define KVM_FEATURE_ASYNC_PF_VMEXIT 10 > +#define KVM_FEATURE_PV_SEND_IPI 11 > > #define KVM_HINTS_REALTIME 0 > > diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c > index 591bcf2..2fe1420 100644 > --- a/arch/x86/kernel/kvm.c > +++ b/arch/x86/kernel/kvm.c > @@ -454,6 +454,71 @@ static void __init sev_map_percpu_data(void) > } > > #ifdef CONFIG_SMP > + > +#ifdef CONFIG_X86_64 > +static void __send_ipi_mask(const struct cpumask *mask, int vector) > +{ > + unsigned long flags, ipi_bitmap_low =3D 0, ipi_bitmap_high =3D 0; > + int cpu, apic_id; > + > + if (cpumask_empty(mask)) > + return; > + > + local_irq_save(flags); > + > + for_each_cpu(cpu, mask) { > + apic_id =3D per_cpu(x86_cpu_to_apicid, cpu); > + if (apic_id < BITS_PER_LONG) > + __set_bit(apic_id, &ipi_bitmap_low); > + else if (apic_id < 2 * BITS_PER_LONG) > + __set_bit(apic_id - BITS_PER_LONG, &ipi_bitmap_hi= gh); > + } > + > + kvm_hypercall3(KVM_HC_SEND_IPI, ipi_bitmap_low, ipi_bitmap_high, = vector); > + > + local_irq_restore(flags); > +} > + > +static void kvm_send_ipi_mask(const struct cpumask *mask, int vector) > +{ > + __send_ipi_mask(mask, vector); > +} > + > +static void kvm_send_ipi_mask_allbutself(const struct cpumask *mask, int= vector) > +{ > + unsigned int this_cpu =3D smp_processor_id(); > + struct cpumask new_mask; > + const struct cpumask *local_mask; > + > + cpumask_copy(&new_mask, mask); > + cpumask_clear_cpu(this_cpu, &new_mask); > + local_mask =3D &new_mask; > + __send_ipi_mask(local_mask, vector); > +} > + > +static void kvm_send_ipi_allbutself(int vector) > +{ > + kvm_send_ipi_mask_allbutself(cpu_online_mask, vector); > +} > + > +static void kvm_send_ipi_all(int vector) > +{ > + __send_ipi_mask(cpu_online_mask, vector); > +} > + > +/* > + * Set the IPI entry points > + */ > +static void kvm_setup_pv_ipi(void) > +{ > + apic->send_IPI_mask =3D kvm_send_ipi_mask; > + apic->send_IPI_mask_allbutself =3D kvm_send_ipi_mask_allbutself; > + apic->send_IPI_allbutself =3D kvm_send_ipi_allbutself; > + apic->send_IPI_all =3D kvm_send_ipi_all; > + pr_info("KVM setup pv IPIs\n"); > +} > +#endif > + > static void __init kvm_smp_prepare_cpus(unsigned int max_cpus) > { > native_smp_prepare_cpus(max_cpus); > @@ -626,6 +691,11 @@ static uint32_t __init kvm_detect(void) > > static void __init kvm_apic_init(void) > { > +#if defined(CONFIG_SMP) && defined(CONFIG_X86_64) > + if (kvm_para_has_feature(KVM_FEATURE_PV_SEND_IPI) && > + num_possible_cpus() <=3D 2 * BITS_PER_LONG) > + kvm_setup_pv_ipi(); > +#endif > } > > static void __init kvm_init_platform(void) > diff --git a/include/uapi/linux/kvm_para.h b/include/uapi/linux/kvm_para.= h > index dcf629d..84f8fe3 100644 > --- a/include/uapi/linux/kvm_para.h > +++ b/include/uapi/linux/kvm_para.h > @@ -26,6 +26,7 @@ > #define KVM_HC_MIPS_EXIT_VM 7 > #define KVM_HC_MIPS_CONSOLE_OUTPUT 8 > #define KVM_HC_CLOCK_PAIRING 9 > +#define KVM_HC_SEND_IPI 10 > > /* > * hypercalls use architecture specific > -- > 2.7.4 >