Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Content-Type: text/plain;
        charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 11.1 \(3445.4.7\))
Subject: Re: [PATCH RFC] KVM: x86: vmx: throttle immediate exit through
 preemtion timer to assist buggy guests
From:   Liran Alon <liran.alon@oracle.com>
In-Reply-To: <89d4189b-de6a-7634-de8b-29a044a86e12@redhat.com>
Date:   Fri, 29 Mar 2019 18:32:32 +0300
Cc:     Vitaly Kuznetsov <vkuznets@redhat.com>, kvm@vger.kernel.org,
        =?utf-8?B?UmFkaW0gS3LEjW3DocWZ?= <rkrcmar@redhat.com>,
        Sean Christopherson <sean.j.christopherson@intel.com>,
        linux-kernel@vger.kernel.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <63F1B6AE-F7A0-40D9-9582-558723800682@oracle.com>
References: <20190328203110.20655-1-vkuznets@redhat.com>
 <f64742ef-c67d-db0f-d812-2b6f5048d7ee@redhat.com>
 <87d0m93frp.fsf@vitty.brq.redhat.com>
 <89d4189b-de6a-7634-de8b-29a044a86e12@redhat.com>
To:     Paolo Bonzini <pbonzini@redhat.com>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


> On 29 Mar 2019, at 18:01, Paolo Bonzini <pbonzini@redhat.com> wrote:
>=20
> On 29/03/19 15:40, Vitaly Kuznetsov wrote:
>> Paolo Bonzini <pbonzini@redhat.com> writes:
>>=20
>>> On 28/03/19 21:31, Vitaly Kuznetsov wrote:
>>>>=20
>>>> The 'hang' scenario develops like this:
>>>> 1) Hyper-V boots and QEMU is trying to inject two irq =
simultaneously. One
>>>> of them is level-triggered. KVM injects the edge-triggered one and
>>>> requests immediate exit to inject the level-triggered:
>>>>=20
>>>> kvm_set_irq:          gsi 23 level 1 source 0
>>>> kvm_msi_set_irq:      dst 0 vec 80 (Fixed|physical|level)
>>>> kvm_apic_accept_irq:  apicid 0 vec 80 (Fixed|edge)
>>>> kvm_msi_set_irq:      dst 0 vec 96 (Fixed|physical|edge)
>>>> kvm_apic_accept_irq:  apicid 0 vec 96 (Fixed|edge)
>>>> kvm_nested_vmexit_inject: reason EXTERNAL_INTERRUPT info1 0 info2 0 =
int_info 80000060 int_info_err 0
>>>>=20
>>>> 2) Hyper-V requires one of its VMs to run to handle the situation =
but
>>>> immediate exit happens:
>>>>=20
>>>> kvm_entry:            vcpu 0
>>>> kvm_exit:             reason VMRESUME rip 0xfffff80006a40115 info 0 =
0
>>>> kvm_entry:            vcpu 0
>>>> kvm_exit:             reason PREEMPTION_TIMER rip =
0xfffff8022f3d8350 info 0 0
>>>> kvm_nested_vmexit:    rip fffff8022f3d8350 reason PREEMPTION_TIMER =
info1 0 info2 0 int_info 0 int_info_err 0
>>>> kvm_nested_vmexit_inject: reason EXTERNAL_INTERRUPT info1 0 info2 0 =
int_info 80000050 int_info_err 0
>>>=20
>>> I supposed before this there was an eoi for vector 96?
>>=20
>> AFAIR: no, it seems that it is actually the VM it is trying to resume
>> (Windows partition?) which needs to do some work and with the =
preemtion
>> timer of 0 we don't allow it to.
>=20
> kvm_apic_accept_irq placed IRQ 96 in IRR, and Hyper-V should be =
running
> with "acknowledge interrupt on exit" since int_info is nonzero in
> kvm_nested_vmexit_inject.
>=20
> Therefore, at the kvm_nested_vmexit_inject tracepoint KVM should have
> set bit 96 in ISR; and because PPR is now 96, interrupt 80 should have
> never been delivered.  Unless 96 is an auto-EOI interrupt, in which =
case
> this comment would apply
>=20
>          /*
>           * For auto-EOI interrupts, there might be another pending
>           * interrupt above PPR, so check whether to raise another
>           * KVM_REQ_EVENT.
>           */
>=20
> IIRC there was an enlightenment to tell Windows "I support auto-EOI =
but
> please don't use it".  If this is what's happening, that would also =
fix it.
>=20
> Thanks,
>=20
> Paolo

Paolo I am not sure this is the case here.
Please read my other replies in this email thread.

I think this is just a standard issue of a level-triggered interrupt =
handler in L1 (Hyper-V) that performs EOI before it lowers the irq-line.
I don=E2=80=99t think vector 96 is even related to the issue at hand =
here. This is why after it was already handled, the loop of =
EXTERNAL_INTERRUPT
happens on vector 80 and not vector 96.

In addition, there is a missing optimisation from Hyper-V that after it =
handles an EXTERNAL_INTERRUPT exit, it doesn=E2=80=99t enable interrupts
to receive other pending host interrupts (In our case, the pending =
vector 80) and will therefore only receive it once it enters back to L2
which will cause another EXTERNAL_INTERRUPT exit but this time on vector =
80.

-Liran