Received: by 2002:a25:4158:0:0:0:0:0 with SMTP id o85csp124495yba; Mon, 1 Apr 2019 03:09:38 -0700 (PDT) X-Google-Smtp-Source: APXvYqwdvAfFZPUEWA0CbQOXOPRkuaxWcd4J9mUosb3gQVd6dQUG7bMGsmkMB6rblp64hZHn0cxU X-Received: by 2002:a63:c944:: with SMTP id y4mr12926719pgg.257.1554113378819; Mon, 01 Apr 2019 03:09:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554113378; cv=none; d=google.com; s=arc-20160816; b=WQ0AvW/2BFf2+Dm7rTwS1ffEGe+Z+CElZ96pYxI+Z9swagtzXVvuBurJawJ5jQ1bkH dudeN7xwHoSoaVBaJAClisqN3/mE/OiIDs1ayunpnTlc7shVg2LhMegufO92a9WQRAOC c34U147Sigv8e5zkAeM6imRhk6jsZ/NiYVkcl1l87C4Y4rCRbVfxqe9SMpmwfXrX/dZQ R5xAUVKo0/ClXwg/uU9gxvz0BJCiNtGVckomj7sLIhDqQNtk5c99BJg1TTyIO03cOba6 wLgBwspjiF5yBfuknB7A0sK3DWbvtqYFya9N+/C+S6j/Tz8VK2fO8PZw+/zEg86EBqLv HvdQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:to:references:message-id :content-transfer-encoding:cc:date:in-reply-to:from:subject :mime-version:dkim-signature; bh=X+rIyJ0QQfWxb3wvNs8B9usSAXkS7iVp+Q4mCPq9nC4=; b=yqKsMF2yD8Prhxcs7L9UAmg2iS1slCqBrCFQDceZS6GNVGZ/kFwlznVQsttzYoVYwG WSf5lonGmENXLa6T2DnBslIYRbACzmKyWM4+nJFlaL7be2hUoJ7513QvPHeCVyIuaJLB 8U96wfp37tYYmlKqVsckOWmtZmvNyTX+5JB21xFvirYmESTpogcAsXZh5eAsLrmmYDRz rzTuBNN16QUg6OsJfqDMmCT0GOSj1Pt3bfzTRyMEtS2Z1PeudPb0DxSRqUkWYywBNg40 k6dscDn2JKcUdgaOIlFb89RT5VV+2j9HQSGM5a2xwu1aUOblYx2SdinLATu0oirKXUmh bSgg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b="VuIE/q9x"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id a17si7100408pgk.572.2019.04.01.03.09.23; Mon, 01 Apr 2019 03:09:38 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b="VuIE/q9x"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726183AbfDAKIs (ORCPT + 99 others); Mon, 1 Apr 2019 06:08:48 -0400 Received: from userp2130.oracle.com ([156.151.31.86]:60830 "EHLO userp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725867AbfDAKIr (ORCPT ); Mon, 1 Apr 2019 06:08:47 -0400 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x31A604V141939; Mon, 1 Apr 2019 10:08:43 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=content-type : mime-version : subject : from : in-reply-to : date : cc : content-transfer-encoding : message-id : references : to; s=corp-2018-07-02; bh=X+rIyJ0QQfWxb3wvNs8B9usSAXkS7iVp+Q4mCPq9nC4=; b=VuIE/q9x4k0gCyzhYisTtWbYI9CoYj7OfsayAiPhlwXdmZsK/6PI3fzV2rFEqH6TOACi nNJ0v8eiNC11EeIAWUHOZCibHWZNmpyY6unVmvlsY8t2srCktVHmhwvME6EbS5l/KPBK UxI9A7is1OTN9hr46xXSUh1728T8E2MFp9H2Y8bvWjVrhzy/AxdHfXMc24BdPcJx0zJM JLMrUnH1VjDllIrPTF8B5KelfFMQLitHzd1DFZNfoVvyuTj2/oGl3m2BCUhzw/l9PbgD 4yJS3OVafaBjKNvE2XQr4ywlWLn+OkHYGKK9ZjXtr/zXUTIIOAXenrmcQZ6WvqgF7cpT Zw== Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp2130.oracle.com with ESMTP id 2rhyvsx1b8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 01 Apr 2019 10:08:43 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x31A8Wsm024998 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 1 Apr 2019 10:08:32 GMT Received: from abhmp0019.oracle.com (abhmp0019.oracle.com [141.146.116.25]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x31A8W1a025590; Mon, 1 Apr 2019 10:08:32 GMT Received: from [192.168.14.112] (/109.65.229.28) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 01 Apr 2019 03:08:32 -0700 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.1 \(3445.4.7\)) Subject: Re: [PATCH RFC] KVM: x86: vmx: throttle immediate exit through preemtion timer to assist buggy guests From: Liran Alon In-Reply-To: <87v9zy15mc.fsf@vitty.brq.redhat.com> Date: Mon, 1 Apr 2019 13:08:27 +0300 Cc: Paolo Bonzini , kvm@vger.kernel.org, =?utf-8?B?UmFkaW0gS3LEjW3DocWZ?= , Sean Christopherson , linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <20190328203110.20655-1-vkuznets@redhat.com> <87d0m93frp.fsf@vitty.brq.redhat.com> <89d4189b-de6a-7634-de8b-29a044a86e12@redhat.com> <63F1B6AE-F7A0-40D9-9582-558723800682@oracle.com> <32296de3-8ce0-c20a-5948-a7eef267a328@redhat.com> <87v9zy15mc.fsf@vitty.brq.redhat.com> To: Vitaly Kuznetsov X-Mailer: Apple Mail (2.3445.4.7) X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9213 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=874 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904010070 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org > On 1 Apr 2019, at 11:39, Vitaly Kuznetsov wrote: >=20 > Paolo Bonzini writes: >=20 >> On 29/03/19 16:32, Liran Alon wrote: >>> Paolo I am not sure this is the case here. Please read my other >>> replies in this email thread. >>>=20 >>> I think this is just a standard issue of a level-triggered interrupt >>> handler in L1 (Hyper-V) that performs EOI before it lowers the >>> irq-line. I don=E2=80=99t think vector 96 is even related to the = issue at >>> hand here. This is why after it was already handled, the loop of >>> EXTERNAL_INTERRUPT happens on vector 80 and not vector 96. >>=20 >> Hmm... Vitaly, what machine were you testing on---does it have = APIC-v? >> If not, then you should have seen either an EOI for irq 96 or a TPR >> below threshold vmexit. However, if it has APIC-v then you wouldn't >> have seen any of this (you only see the EOI for irq 80 because it's >> level triggered) and Liran is probably right. >>=20 >=20 > It does, however, the issue is reproducible with and without > it. Moreover, I think the second simultaneous IRQ is just a red = herring; > Here is another trace (enable_apicv). Posting it non-stripped and hope > your eyes will catch something I'm missing: >=20 > [001] 513675.736316: kvm_exit: reason VMRESUME rip = 0xfffff80002cae115 info 0 0 > [001] 513675.736321: kvm_entry: vcpu 0 > [001] 513675.736565: kvm_exit: reason EXTERNAL_INTERRUPT = rip 0xfffff80362dcd26d info 0 800000ec > [001] 513675.736566: kvm_nested_vmexit: rip fffff80362dcd26d reason = EXTERNAL_INTERRUPT info1 0 info2 0 int_info 800000ec int_info_err 0 > [001] 513675.736568: kvm_entry: vcpu 0 > [001] 513675.736650: kvm_exit: reason EPT_VIOLATION rip = 0xfffff80362dcd230 info 182 0 > [001] 513675.736651: kvm_nested_vmexit: rip fffff80362dcd230 reason = EPT_VIOLATION info1 182 info2 0 int_info 0 int_info_err 0 > [001] 513675.736651: kvm_page_fault: address 261200000 = error_code 182 >=20 > -> injecting >=20 > [008] 513675.737059: kvm_set_irq: gsi 23 level 1 source 0 > [008] 513675.737061: kvm_msi_set_irq: dst 0 vec 80 = (Fixed|physical|level) > [008] 513675.737062: kvm_apic_accept_irq: apicid 0 vec 80 = (Fixed|edge) > [001] 513675.737233: kvm_nested_vmexit_inject: reason = EXTERNAL_INTERRUPT info1 0 info2 0 int_info 80000050 int_info_err 0 > [001] 513675.737239: kvm_entry: vcpu 0 > [001] 513675.737243: kvm_exit: reason EOI_INDUCED rip = 0xfffff80002c85e1a info 50 0 >=20 > -> immediate EOI causing re-injection (even preemption timer is not > involved here). >=20 > [001] 513675.737244: kvm_eoi: apicid 0 vector 80 > [001] 513675.737245: kvm_fpu: unload > [001] 513675.737246: kvm_userspace_exit: reason KVM_EXIT_IOAPIC_EOI = (26) > [001] 513675.737256: kvm_set_irq: gsi 23 level 1 source 0 > [001] 513675.737259: kvm_msi_set_irq: dst 0 vec 80 = (Fixed|physical|level) > [001] 513675.737260: kvm_apic_accept_irq: apicid 0 vec 80 = (Fixed|edge) > [001] 513675.737264: kvm_fpu: load > [001] 513675.737265: kvm_entry: vcpu 0 > [001] 513675.737271: kvm_exit: reason VMRESUME rip = 0xfffff80002cae115 info 0 0 > [001] 513675.737278: kvm_entry: vcpu 0 > [001] 513675.737282: kvm_exit: reason PREEMPTION_TIMER rip = 0xfffff80362dcc2d0 info 0 0 > [001] 513675.737283: kvm_nested_vmexit: rip fffff80362dcc2d0 reason = PREEMPTION_TIMER info1 0 info2 0 int_info 0 int_info_err 0 > [001] 513675.737285: kvm_nested_vmexit_inject: reason = EXTERNAL_INTERRUPT info1 0 info2 0 int_info 80000050 int_info_err 0 > [001] 513675.737289: kvm_entry: vcpu 0 > [001] 513675.737293: kvm_exit: reason EOI_INDUCED rip = 0xfffff80002c85e1a info 50 0 > [001] 513675.737293: kvm_eoi: apicid 0 vector 80 > [001] 513675.737294: kvm_fpu: unload > [001] 513675.737295: kvm_userspace_exit: reason KVM_EXIT_IOAPIC_EOI = (26) > [001] 513675.737299: kvm_set_irq: gsi 23 level 1 source 0 > [001] 513675.737299: kvm_msi_set_irq: dst 0 vec 80 = (Fixed|physical|level) > [001] 513675.737300: kvm_apic_accept_irq: apicid 0 vec 80 = (Fixed|edge) > [001] 513675.737302: kvm_fpu: load > [001] 513675.737303: kvm_entry: vcpu 0 > [001] 513675.737307: kvm_exit: reason VMRESUME rip = 0xfffff80002cae115 info 0 0 >=20 > ... >=20 > --=20 > Vitaly So to sum-up: This matches what I mentioned in my previous emails right? That vector 96 is not related, and the only issue here is that = level-triggered interrupt handler for vector 80 is doing EOI before = lowering the irq-line. Which cause vector 80 to be injected in infinite loop. And this is not even related to being a nested virtualization workload. = It=E2=80=99s just an issue in Hyper-V (L1) interrupt handler for vector = 80. Therefore the only action-items are: 1) Microsoft to fix Hyper-V vector 80 interrupt handler to lower = irq-line before EOI. 2) Patch QEMU IOAPIC implementation to have a mechanism similar to KVM = to delay injection of level-triggered interrupt in case we are injecting the same interrupt for X times in a row. -Liran