Received: by 2002:a05:6a10:af89:0:0:0:0 with SMTP id iu9csp3380355pxb; Mon, 17 Jan 2022 19:07:01 -0800 (PST) X-Google-Smtp-Source: ABdhPJzxeLarCa5BUs0hSjQ59XbtIelIolOI2EoD/lgekuMw4To5BAsj399EyEx0EpMEuIpEWn83 X-Received: by 2002:a17:902:d484:b0:14a:5895:b680 with SMTP id c4-20020a170902d48400b0014a5895b680mr25815739plg.20.1642475221654; Mon, 17 Jan 2022 19:07:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1642475221; cv=none; d=google.com; s=arc-20160816; b=JwY2FSsWGX+mVK2YJtuekCJ84GLReFkj+Dhicx7w/RowhY3W4BgkNM2HUlrT88o/oT focJR+yuItF0k6/j9vTk1Ac1Pd3WBxRqWwjCPXNl+gKqEohmb1N70Uvrvagm/Om8Cu1C Pvlkx6j2lJBNdbO+03bmscDgFucW5gMJIt6Bxfe6ffnz3Bs7XmsXl0e70YcDl9X5zJzE HhLaAa7cdTPwBsqMq4GK51I5/+6kvf+FueFZ/5dYOLuicETEK63YCkjfdh4Hvwuw/d08 aT33ZRUNoRmdzjqsotpZB2Too3iKUMmOz1F5E9jbYpeG75TGBcd4cCA98vbxzmjtZD5T yEMQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:user-agent:in-reply-to:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :dkim-signature; bh=CJsGgXKRZGr8S6yNAa5i5WpaEkqxBZdeL51I15ePCB4=; b=MJUHa6cfMJrQfCfcMdGmBnEiau60/h9REFC4wqGQnJIKdWXWd0V0TRXaZFAVwdWpAN Z3uKGicU25K6vZQsvXLRzWkqtLgj09SVBfB44IvAiuFUQ6MbmA1CI8WKujdMSvod43F2 L4qEVJ5oKHxiDFhXRh4R1vMXbvfTHlvXg+rumE6kZWl/peakqjw6Kc72QPtFrTXh0u55 e27mRC4u2i2+oCMOPWOLTdLXcDGg3Fls5yplit40QflAYCZPMyc5ul7JIhJtMbqQFpsR A9QFzr6an6VJJ6x80OvtLXYVmz2wYOIKcN8Wk4qvx4kCAeoYDg1PJ2rnJt5Yr1QYxt2A 8YXQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=ndLbYCbQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id p7si15520158plf.144.2022.01.17.19.06.48; Mon, 17 Jan 2022 19:07:01 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=ndLbYCbQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236469AbiARAoN (ORCPT + 99 others); Mon, 17 Jan 2022 19:44:13 -0500 Received: from mga07.intel.com ([134.134.136.100]:25747 "EHLO mga07.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229632AbiARAoM (ORCPT ); Mon, 17 Jan 2022 19:44:12 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1642466652; x=1674002652; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=fC5OmCNrq2on5Z2DCLXH2zztCSuT3R3WK2oNnnsL7Lw=; b=ndLbYCbQ3hZj+z85O85HxWHY4DwKP4INDAZbUvegdVc8fPyUFmmb/cmc JBIVODSwe3UtGGCnUE4zZ8ppoqxtCS62+JvQDhXy9YlMWiEHwtvK9pJmy mXKsi8iH9RtdfA1E+YYZlliwcb290cvkHi6QDLsMnZQKd+SQf5H+V/Klm mBmwyLE+88v7pt9kpGiI0i7/Xo87WgmsUi8+FOK5z+KFnW/wx+0kK6Moa 86cXvs0Go2QTACyH7T1gUbaMTGbJBOix9jF9sgSFl9tvM4hFqoHzsJ3gU YJa2plb4H+YFZivH9LPsejpXRpdmeBXnC5tyxq00JIeYiNU61OFj7dG5N A==; X-IronPort-AV: E=McAfee;i="6200,9189,10230"; a="308050191" X-IronPort-AV: E=Sophos;i="5.88,296,1635231600"; d="scan'208";a="308050191" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Jan 2022 16:44:11 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,296,1635231600"; d="scan'208";a="531529613" Received: from yy-desk-7060.sh.intel.com (HELO localhost) ([10.239.159.76]) by orsmga008.jf.intel.com with ESMTP; 17 Jan 2022 16:44:06 -0800 Date: Tue, 18 Jan 2022 08:44:05 +0800 From: Yuan Yao To: Zeng Guang Cc: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , "kvm@vger.kernel.org" , Dave Hansen , "Luck, Tony" , Kan Liang , Thomas Gleixner , Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , Kim Phillips , Jarkko Sakkinen , Jethro Beekman , "Huang, Kai" , "x86@kernel.org" , "linux-kernel@vger.kernel.org" , "Hu, Robert" , "Gao, Chao" Subject: Re: [PATCH v5 5/8] KVM: x86: Support interrupt dispatch in x2APIC mode with APIC-write VM exit Message-ID: <20220118004405.po36x3lxi26mkwsz@yy-desk-7060> References: <20211231142849.611-1-guang.zeng@intel.com> <20211231142849.611-6-guang.zeng@intel.com> <8ab5f976-1f3e-e2a5-87f6-e6cf376ead2f@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <8ab5f976-1f3e-e2a5-87f6-e6cf376ead2f@intel.com> User-Agent: NeoMutt/20171215 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sat, Jan 15, 2022 at 10:08:10AM +0800, Zeng Guang wrote: > On 1/15/2022 1:34 AM, Sean Christopherson wrote: > > On Fri, Jan 14, 2022, Zeng Guang wrote: > > > kvm_lapic_reg_read() is limited to read up to 4 bytes. It needs extension to > > > support 64bit read. > > Ah, right. > > > > > And another concern is here getting reg value only specific from vICR(no > > > other regs need take care), going through whole path on kvm_lapic_reg_read() > > > could be time-consuming unnecessarily. Is it proper that calling > > > kvm_lapic_get_reg64() to retrieve vICR value directly? > > Hmm, no, I don't think that's proper. Retrieving a 64-bit value really is unique > > to vICR. Yes, the code does WARN on that, but if future architectural extensions > > even generate APIC-write exits on other registers, then using kvm_lapic_get_reg64() > > would be wrong and this code would need to be updated again. > Split on x2apic and WARN on (offset != APIC_ICR) already limit register read > to vICR only. Actually > we just need consider to deal with 64bit data specific to vICR in APIC-write > exits. From this point of > view, previous design can be compatible on handling other registers even if > future architectural > extensions changes. :) > > > > What about tweaking my prep patch from before to the below? That would yield: > > > > if (apic_x2apic_mode(apic)) { > > if (WARN_ON_ONCE(offset != APIC_ICR)) > > return 1; > > > > kvm_lapic_msr_read(apic, offset, &val); > > I think it's problematic to use kvm_lapic_msr_read() in this case. It > premises the high 32bit value > already valid at APIC_ICR2, while in handling "nodecode" x2APIC writes we > need get continuous 64bit > data from offset 300H first and prepare emulation of APIC_ICR2 write. At > this time, APIC_ICR2 is not > ready yet. How about combine them, then you can handle the ICR write vmexit for IPI virtualization and Sean's patch can still work with code reusing, like below: if (apic_x2apic_mode(apic)) { if (WARN_ON_ONCE(offset != APIC_ICR)) kvm_lapic_msr_read(apic, offset, &val); else kvm_lapic_get_reg64(apic, offset, &val); kvm_lapic_msr_write(apic, offset, val); } else { kvm_lapic_reg_read(apic, offset, 4, &val); kvm_lapic_reg_write(apic, offset, val); } > > > kvm_lapic_msr_write(apic, offset, val); > > } else { > > kvm_lapic_reg_read(apic, offset, 4, &val); > > kvm_lapic_reg_write(apic, offset, val); > > } > > > > I like that the above has "msr" in the low level x2apic helpers, and it maximizes > > code reuse. Compile tested only... > > > > From: Sean Christopherson > > Date: Fri, 14 Jan 2022 09:29:34 -0800 > > Subject: [PATCH] KVM: x86: Add helpers to handle 64-bit APIC MSR read/writes > > > > Add helpers to handle 64-bit APIC read/writes via MSRs to deduplicate the > > x2APIC and Hyper-V code needed to service reads/writes to ICR. Future > > support for IPI virtualization will add yet another path where KVM must > > handle 64-bit APIC MSR reads/write (to ICR). > > > > Opportunistically fix the comment in the write path; ICR2 holds the > > destination (if there's no shorthand), not the vector. > > > > No functional change intended. > > > > Signed-off-by: Sean Christopherson > > --- > > arch/x86/kvm/lapic.c | 59 ++++++++++++++++++++++---------------------- > > 1 file changed, 29 insertions(+), 30 deletions(-) > > > > diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c > > index f206fc35deff..cc4531eb448f 100644 > > --- a/arch/x86/kvm/lapic.c > > +++ b/arch/x86/kvm/lapic.c > > @@ -2787,6 +2787,30 @@ int kvm_lapic_set_vapic_addr(struct kvm_vcpu *vcpu, gpa_t vapic_addr) > > return 0; > > } > > > > +static int kvm_lapic_msr_read(struct kvm_lapic *apic, u32 reg, u64 *data) > > +{ > > + u32 low, high = 0; > > + > > + if (kvm_lapic_reg_read(apic, reg, 4, &low)) > > + return 1; > > + > > + if (reg == APIC_ICR && > > + WARN_ON_ONCE(kvm_lapic_reg_read(apic, APIC_ICR2, 4, &high))) > > + return 1; > > + > > + *data = (((u64)high) << 32) | low; > > + > > + return 0; > > +} > > + > > +static int kvm_lapic_msr_write(struct kvm_lapic *apic, u32 reg, u64 data) > > +{ > > + /* For 64-bit ICR writes, set ICR2 (dest) before ICR (command). */ > > + if (reg == APIC_ICR) > > + kvm_lapic_reg_write(apic, APIC_ICR2, (u32)(data >> 32)); > > + return kvm_lapic_reg_write(apic, reg, (u32)data); > > +} > > + > > int kvm_x2apic_msr_write(struct kvm_vcpu *vcpu, u32 msr, u64 data) > > { > > struct kvm_lapic *apic = vcpu->arch.apic; > > @@ -2798,16 +2822,13 @@ int kvm_x2apic_msr_write(struct kvm_vcpu *vcpu, u32 msr, u64 data) > > if (reg == APIC_ICR2) > > return 1; > > > > - /* if this is ICR write vector before command */ > > - if (reg == APIC_ICR) > > - kvm_lapic_reg_write(apic, APIC_ICR2, (u32)(data >> 32)); > > - return kvm_lapic_reg_write(apic, reg, (u32)data); > > + return kvm_lapic_msr_write(apic, reg, data); > > } > > > > int kvm_x2apic_msr_read(struct kvm_vcpu *vcpu, u32 msr, u64 *data) > > { > > struct kvm_lapic *apic = vcpu->arch.apic; > > - u32 reg = (msr - APIC_BASE_MSR) << 4, low, high = 0; > > + u32 reg = (msr - APIC_BASE_MSR) << 4; > > > > if (!lapic_in_kernel(vcpu) || !apic_x2apic_mode(apic)) > > return 1; > > @@ -2815,45 +2836,23 @@ int kvm_x2apic_msr_read(struct kvm_vcpu *vcpu, u32 msr, u64 *data) > > if (reg == APIC_DFR || reg == APIC_ICR2) > > return 1; > > > > - if (kvm_lapic_reg_read(apic, reg, 4, &low)) > > - return 1; > > - if (reg == APIC_ICR) > > - kvm_lapic_reg_read(apic, APIC_ICR2, 4, &high); > > - > > - *data = (((u64)high) << 32) | low; > > - > > - return 0; > > + return kvm_lapic_msr_read(apic, reg, data); > > } > > > > int kvm_hv_vapic_msr_write(struct kvm_vcpu *vcpu, u32 reg, u64 data) > > { > > - struct kvm_lapic *apic = vcpu->arch.apic; > > - > > if (!lapic_in_kernel(vcpu)) > > return 1; > > > > - /* if this is ICR write vector before command */ > > - if (reg == APIC_ICR) > > - kvm_lapic_reg_write(apic, APIC_ICR2, (u32)(data >> 32)); > > - return kvm_lapic_reg_write(apic, reg, (u32)data); > > + return kvm_lapic_msr_write(vcpu->arch.apic, reg, data); > > } > > > > int kvm_hv_vapic_msr_read(struct kvm_vcpu *vcpu, u32 reg, u64 *data) > > { > > - struct kvm_lapic *apic = vcpu->arch.apic; > > - u32 low, high = 0; > > - > > if (!lapic_in_kernel(vcpu)) > > return 1; > > > > - if (kvm_lapic_reg_read(apic, reg, 4, &low)) > > - return 1; > > - if (reg == APIC_ICR) > > - kvm_lapic_reg_read(apic, APIC_ICR2, 4, &high); > > - > > - *data = (((u64)high) << 32) | low; > > - > > - return 0; > > + return kvm_lapic_msr_read(vcpu->arch.apic, reg, data); > > } > > > > int kvm_lapic_set_pv_eoi(struct kvm_vcpu *vcpu, u64 data, unsigned long len) > > --