Received: by 2002:a25:f815:0:0:0:0:0 with SMTP id u21csp3027637ybd; Fri, 28 Jun 2019 01:28:11 -0700 (PDT) X-Google-Smtp-Source: APXvYqys4h218wh9IGDtYUkG+bpiRCmhS9dp/LSV/5WyAY6PcYDIRECqDEBaXo70s6FhdxgecpFy X-Received: by 2002:a17:90a:3548:: with SMTP id q66mr11734612pjb.17.1561710491841; Fri, 28 Jun 2019 01:28:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561710491; cv=none; d=google.com; s=arc-20160816; b=VuKT+3Fd6a/RoCIOwUPfL+HnVJAw4ombOlmhYFvfMKMQU0f6bdTOiaBMmRI8bCupRZ C/VHKUGFb8nkm+jeQ50vxtHhb3CDrf3ubQKyxh2EpVlDkR/UbBnPnfbiVgQ/NdXn6oCG 60SsrIZdNDtEx9W3Mhfd9cxsuY89QWi97UFETe84kqYbJJJHA94yOBPb4WI+bT/l3gEF 8RBP0bIrLxslvO0gqONhXcNfwLKKIv6/1pzy9yaSu5IuwMeOhfm5Cvgqr0GKVLcYAsVs QOT1gGcvu52I4kL6uNgnpkKmHsTQAgh2QwJozyo2lLIEhS/G34rGiiCirYpDJsN93maL 4B9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:cc:to:subject :message-id:date:from:in-reply-to:references:mime-version :dkim-signature; bh=yv6CgnWPC6xWm8QD/szBDMm8993i38/ihT9u27h0IHg=; b=uqp/geAvSgtLA8safEl1uRxp4jvWQpS8iyVYmXaa+P3p8wH8ARw4DbYCWC1w0piISO Z7rnc9HjrHTVmZZIX7YeiFelxMVbp2MAjRGUYE4wO9Qy0cGBeKhwPNQS0Apk+QXm8ARG iQznwJ7OI+bsQgVuQTeOBjnHt8dR0pm+O2VZCJxgAsPrTpjc74s+N53Gq9vjh7fEkEhX FVZ/ORIc0fM6OLphz/zY+i61QJA4zMpnCp0WXgj7QEzZZHuhQA+ZGeLEBuvj8BprB+wp kg/Vm2ulVi/gk5XYfutAW3QxxF6QOL4Z3mAibuallSiuMhlbnhfYYdA/kKN5ArgxpYaV S2Mg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="H/GoLEao"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id t186si1786292pfb.19.2019.06.28.01.27.56; Fri, 28 Jun 2019 01:28:11 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="H/GoLEao"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726653AbfF1I0d (ORCPT + 99 others); Fri, 28 Jun 2019 04:26:33 -0400 Received: from mail-ot1-f67.google.com ([209.85.210.67]:41617 "EHLO mail-ot1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726056AbfF1I0d (ORCPT ); Fri, 28 Jun 2019 04:26:33 -0400 Received: by mail-ot1-f67.google.com with SMTP id o101so4858887ota.8; Fri, 28 Jun 2019 01:26:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=yv6CgnWPC6xWm8QD/szBDMm8993i38/ihT9u27h0IHg=; b=H/GoLEaonyeEZGb/WdSNNwT0RKBkwJ+epaDaWfsOguW1ydhvWbSbwKtl33NMN9i2m8 4YgucQRpRpAj11NGAPXL1N6Upj0r4fa4YR6t0s5wOFoljLN+EmakXd1tfahz1hCskGtv d7xiJM8RVgBagmJlykCsk+u67mBH37AiwXILVK83OKJGYlFGuyQ+1DkSGU/2cGs9S5FK KxxmtUL5bRKoGEoaz8tSikz12ZSqU84znM7sD62zx2lL3V+VqRCP232O1crpiURARA9r l1BuLi3QAgDiuXPbyUpZcCqFVO/FhwNpXYyD1DSxHzknKJSGlebq6X4RuPzP0nzmNkDZ 04Xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=yv6CgnWPC6xWm8QD/szBDMm8993i38/ihT9u27h0IHg=; b=TWj+2af4txb6fz1QktR4XaQE11ffW9bVY0n5kdV12hXjyaBwSZATPGeChqHeajAvea Pe/O9hpbe0di7WWLwqS1UHcEIeFt/kPrcW89xkekAWm7mSWgZ32Ze53cKnAxHedmzFqQ VjlEl+QsaSgyulErDqKmQLomCvVHF78wSVYNwkmgOXiwUDAkkTqFYF8ud5nXjNrXm5V3 wP5BmNVKL3AW+EcTcj++rD+2roKghjEOFONvT9mDJnnIlm4S4cHP7fTRttyd9+r7exmM 2fri1uDrpo4qJdWRJG7K9LzL3vnlusjLkoX3R0Q8eCHvKSBObrxQD+YLcSi6GmekouyD xPag== X-Gm-Message-State: APjAAAUO9RXaXr+wjUkMFeGSGcihYBeqUbFsaKKiDYnXo4PRcUQ5dc0g M05deLYsskoTq2f3V0vovxFS9S81OMplydWeiHisKsU7q2I= X-Received: by 2002:a9d:62c4:: with SMTP id z4mr6806096otk.56.1561710392254; Fri, 28 Jun 2019 01:26:32 -0700 (PDT) MIME-Version: 1.0 References: <1560770687-23227-1-git-send-email-wanpengli@tencent.com> <1560770687-23227-3-git-send-email-wanpengli@tencent.com> <20190618133541.GA3932@amt.cnet> <20190619210346.GA13033@amt.cnet> <20190621214205.GA4751@amt.cnet> <20190625190010.GA3377@amt.cnet> <20190626164401.GA2211@amt.cnet> In-Reply-To: <20190626164401.GA2211@amt.cnet> From: Wanpeng Li Date: Fri, 28 Jun 2019 16:26:20 +0800 Message-ID: Subject: Re: [PATCH v4 2/5] KVM: LAPIC: inject lapic timer interrupt by posted interrupt To: Marcelo Tosatti Cc: Paolo Bonzini , LKML , kvm , =?UTF-8?B?UmFkaW0gS3LEjW3DocWZ?= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, 27 Jun 2019 at 00:44, Marcelo Tosatti wrote: > > On Wed, Jun 26, 2019 at 07:02:13PM +0800, Wanpeng Li wrote: > > On Wed, 26 Jun 2019 at 03:03, Marcelo Tosatti wro= te: > > > > > > On Mon, Jun 24, 2019 at 04:53:53PM +0800, Wanpeng Li wrote: > > > > On Sat, 22 Jun 2019 at 06:11, Marcelo Tosatti = wrote: > > > > > > > > > > On Fri, Jun 21, 2019 at 09:42:39AM +0800, Wanpeng Li wrote: > > > > > > On Thu, 20 Jun 2019 at 05:04, Marcelo Tosatti wrote: > > > > > > > > > > > > > > Hi Li, > > > > > > > > > > > > > > On Wed, Jun 19, 2019 at 08:36:06AM +0800, Wanpeng Li wrote: > > > > > > > > On Tue, 18 Jun 2019 at 21:36, Marcelo Tosatti wrote: > > > > > > > > > > > > > > > > > > On Mon, Jun 17, 2019 at 07:24:44PM +0800, Wanpeng Li wrot= e: > > > > > > > > > > From: Wanpeng Li > > > > > > > > > > > > > > > > > > > > Dedicated instances are currently disturbed by unnecess= ary jitter due > > > > > > > > > > to the emulated lapic timers fire on the same pCPUs whi= ch vCPUs resident. > > > > > > > > > > There is no hardware virtual timer on Intel for guest l= ike ARM. Both > > > > > > > > > > programming timer in guest and the emulated timer fires= incur vmexits. > > > > > > > > > > This patch tries to avoid vmexit which is incurred by t= he emulated > > > > > > > > > > timer fires in dedicated instance scenario. > > > > > > > > > > > > > > > > > > > > When nohz_full is enabled in dedicated instances scenar= io, the emulated > > > > > > > > > > timers can be offload to the nearest busy housekeeping = cpus since APICv > > > > > > > > > > is really common in recent years. The guest timer inter= rupt is injected > > > > > > > > > > by posted-interrupt which is delivered by housekeeping = cpu once the emulated > > > > > > > > > > timer fires. > > > > > > > > > > > > > > > > > > > > The host admin should fine tuned, e.g. dedicated instan= ces scenario w/ > > > > > > > > > > nohz_full cover the pCPUs which vCPUs resident, several= pCPUs surplus > > > > > > > > > > for busy housekeeping, disable mwait/hlt/pause vmexits = to keep in non-root > > > > > > > > > > mode, ~3% redis performance benefit can be observed on = Skylake server. > > > > > > > > > > > > > > > > > > > > w/o patch: > > > > > > > > > > > > > > > > > > > > VM-EXIT Samples Samples% Time% Min Tim= e Max Time Avg time > > > > > > > > > > > > > > > > > > > > EXTERNAL_INTERRUPT 42916 49.43% 39.30% 0.47us= 106.09us 0.71us ( +- 1.09% ) > > > > > > > > > > > > > > > > > > > > w/ patch: > > > > > > > > > > > > > > > > > > > > VM-EXIT Samples Samples% Time% Min Tim= e Max Time Avg time > > > > > > > > > > > > > > > > > > > > EXTERNAL_INTERRUPT 6871 9.29% 2.96% 0.44us= 57.88us 0.72us ( +- 4.02% ) > > > > > > > > > > > > > > > > > > > > Cc: Paolo Bonzini > > > > > > > > > > Cc: Radim Kr=C4=8Dm=C3=A1=C5=99 > > > > > > > > > > Cc: Marcelo Tosatti > > > > > > > > > > Signed-off-by: Wanpeng Li > > > > > > > > > > --- > > > > > > > > > > arch/x86/kvm/lapic.c | 33 +++++++++++++++++= +++++++++------- > > > > > > > > > > arch/x86/kvm/lapic.h | 1 + > > > > > > > > > > arch/x86/kvm/vmx/vmx.c | 3 ++- > > > > > > > > > > arch/x86/kvm/x86.c | 5 +++++ > > > > > > > > > > arch/x86/kvm/x86.h | 2 ++ > > > > > > > > > > include/linux/sched/isolation.h | 2 ++ > > > > > > > > > > kernel/sched/isolation.c | 6 ++++++ > > > > > > > > > > 7 files changed, 44 insertions(+), 8 deletions(-) > > > > > > > > > > > > > > > > > > > > diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.= c > > > > > > > > > > index 87ecb56..9ceeee5 100644 > > > > > > > > > > --- a/arch/x86/kvm/lapic.c > > > > > > > > > > +++ b/arch/x86/kvm/lapic.c > > > > > > > > > > @@ -122,6 +122,13 @@ static inline u32 kvm_x2apic_id(st= ruct kvm_lapic *apic) > > > > > > > > > > return apic->vcpu->vcpu_id; > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > +bool posted_interrupt_inject_timer(struct kvm_vcpu *vc= pu) > > > > > > > > > > +{ > > > > > > > > > > + return pi_inject_timer && kvm_vcpu_apicv_active(v= cpu) && > > > > > > > > > > + kvm_hlt_in_guest(vcpu->kvm); > > > > > > > > > > +} > > > > > > > > > > +EXPORT_SYMBOL_GPL(posted_interrupt_inject_timer); > > > > > > > > > > > > > > > > > > Paolo, can you explain the reasoning behind this? > > > > > > > > > > > > > > > > > > Should not be necessary... > > > > > > > > > > > > https://lkml.org/lkml/2019/6/5/436 "Here you need to check > > > > > > kvm_halt_in_guest, not kvm_mwait_in_guest, because you need to = go > > > > > > through kvm_apic_expired if the guest needs to be woken up from > > > > > > kvm_vcpu_block." > > > > > > > > > > Ah, i think he means that a sleeping vcpu (in kvm_vcpu_block) mus= t > > > > > be woken up, if it receives a timer interrupt. > > > > > > > > > > But your patch will go through: > > > > > > > > > > kvm_apic_inject_pending_timer_irqs > > > > > __apic_accept_irq -> > > > > > vmx_deliver_posted_interrupt -> > > > > > kvm_vcpu_trigger_posted_interrupt returns false > > > > > (because vcpu->mode !=3D IN_GUEST_MODE) -> > > > > > kvm_vcpu_kick > > > > > > > > > > Which will wakeup the vcpu. > > > > > > > > Hi Marcelo, > > > > > > > > > > > > > > Apart from this oops, which triggers when running: > > > > > taskset -c 1 ./cyclictest -D 3600 -p 99 -t 1 -h 30 -m -n -i 5000= 0 -b 40 > > > > > > > > I try both host and guest use latest kvm/queue w/ CONFIG_PREEMPT > > > > enabled, and expose mwait as your config, however, there is no oops= . > > > > Can you reproduce steadily or encounter casually? Can you reproduce > > > > w/o the patchset? > > > > > > Hi Li, > > > > Hi Marcelo, > > > > > > > > Steadily. > > > > > > Do you have this as well: > > > > w/ or w/o below diff, testing on both SKX and HSW servers on hand, I > > didn't see any oops. Could you observe the oops disappear when w/o > > below diff? If the answer is yes, then the oops will not block to > > merge the patchset since Paolo prefers to add the kvm_hlt_in_guest() > > condition to guarantee be woken up from kvm_vcpu_block(). > > He agreed that its not necessary. Removing the HLT in guest widens > the scope of the patch greatly. > > > For the > > exitless injection if the guest is running(DPDK style workloads that > > busy-spin on network card) scenarios, we can find a solution later. > > What is the use-case for HLT in guest again? Together w/ mwait/hlt/pause no vmexits for dedicated instances. In addition, hlt in guest will disable pv qspinlock which is not optimal for dedicated instance. Refer to commit b2798ba0b876 (KVM: X86: Choose qspinlock when dedicated physical CPUs are available) and commit caa057a2cad64 (KVM: X86: Provide a capability to disable HLT intercepts). > > I'll find the source for the oops (or confirm can't reproduce with > kvm/queue RSN). Looking forward to it, thanks Marcelo, hope we can catch the upcoming merge window. :) Regards, Wanpeng Li