Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
From:   Andy Lutomirski <luto@amacapital.net>
Mime-Version: 1.0 (1.0)
Subject: Re: [PATCH v2] x86/kvm: Disable KVM_ASYNC_PF_SEND_ALWAYS
Date:   Tue, 7 Apr 2020 14:41:02 -0700
Message-Id: <B85606B0-71B5-4B7D-A892-293CB9C1B434@amacapital.net>
References: <87eeszjbe6.fsf@nanos.tec.linutronix.de>
Cc:     Vivek Goyal <vgoyal@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Andy Lutomirski <luto@kernel.org>,
        Paolo Bonzini <pbonzini@redhat.com>,
        LKML <linux-kernel@vger.kernel.org>, X86 ML <x86@kernel.org>,
        kvm list <kvm@vger.kernel.org>, stable <stable@vger.kernel.org>
In-Reply-To: <87eeszjbe6.fsf@nanos.tec.linutronix.de>
To:     Thomas Gleixner <tglx@linutronix.de>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


> On Apr 7, 2020, at 1:20 PM, Thomas Gleixner <tglx@linutronix.de> wrote:
>=20
> =EF=BB=BFAndy Lutomirski <luto@amacapital.net> writes:
>>>> On Apr 7, 2020, at 10:21 AM, Vivek Goyal <vgoyal@redhat.com> wrote:
>>> Whether interrupts are enabled or not check only happens before we decid=
e
>>> if async pf protocol should be followed or not. Once we decide to
>>> send PAGE_NOT_PRESENT, later notification PAGE_READY does not check
>>> if interrupts are enabled or not. And it kind of makes sense otherwise
>>> guest process will wait infinitely to receive PAGE_READY.
>>>=20
>>> I modified the code a bit to disable interrupt and wait 10 seconds (afte=
r
>>> getting PAGE_NOT_PRESENT message). And I noticed that error async pf
>>> got delivered after 10 seconds after enabling interrupts. So error
>>> async pf was not lost because interrupts were disabled.
>=20
> Async PF is not the same as a real #PF. It just hijacked the #PF vector
> because someone thought this is a brilliant idea.
>=20
>>> Havind said that, I thought disabling interrupts does not mask exception=
s.
>>> So page fault exception should have been delivered even with interrupts
>>> disabled. Is that correct? May be there was no vm exit/entry during
>>> those 10 seconds and that's why.
>=20
> No. Async PF is not a real exception. It has interrupt semantics and it
> can only be injected when the guest has interrupts enabled. It's bad
> design.
>=20
>> My point is that the entire async pf is nonsense. There are two types of e=
vents right now:
>>=20
>> =E2=80=9CPage not ready=E2=80=9D: normally this isn=E2=80=99t even visibl=
e to the guest =E2=80=94 the
>> guest just waits. With async pf, the idea is to try to tell the guest
>> that a particular instruction would block and the guest should do
>> something else instead. Sending a normal exception is a poor design,
>> though: the guest may not expect this instruction to cause an
>> exception. I think KVM should try to deliver an *interrupt* and, if it
>> can=E2=80=99t, then just block the guest.
>=20
> That's pretty much what it does, just that it runs this through #PF and
> has the checks for interrupts disabled - i.e can't right now' around
> that. If it can't then KVM schedules the guest out until the situation
> has been resolved.
>=20
>> =E2=80=9CPage ready=E2=80=9D: this is a regular asynchronous notification=
 just like,
>> say, a virtio completion. It should be an ordinary interrupt.  Some in
>> memory data structure should indicate which pages are ready.
>>=20
>> =E2=80=9CPage is malfunctioning=E2=80=9D is tricky because you *must* del=
iver the
>> event. x86=E2=80=99s #MC is not exactly a masterpiece, but it does kind o=
f
>> work.
>=20
> Nooooo. This does not need #MC at all. Don't even think about it.

Yessssssssssss.  Please do think about it. :)

>=20
> The point is that the access to such a page is either happening in user
> space or in kernel space with a proper exception table fixup.
>=20
> That means a real #PF is perfectly fine. That can be injected any time
> and does not have the interrupt semantics of async PF.

The hypervisor has no way to distinguish between MOV-and-has-valid-stack-and=
-extable-entry and MOV-definitely-can=E2=80=99t-fault-here.  Or, for that ma=
tter, MOV-in-do_page_fault()-will-recurve-if-it-faults.

>=20
> So now lets assume we distangled async PF from #PF and made it a regular
> interrupt, then the following situation still needs to be dealt with:
>=20
>   guest -> access faults
>=20
> host -> injects async fault
>=20
>   guest -> handles and blocks the task
>=20
> host figures out that the page does not exist anymore and now needs to
> fixup the situation.
>=20
> host -> injects async wakeup
>=20
>   guest -> returns from aysnc PF interrupt and retries the instruction
>            which faults again.
>=20
> host -> knows by now that this is a real fault and injects a proper #PF
>=20
>   guest -> #PF runs and either sends signal to user space or runs
>            the exception table fixup for a kernel fault.

Or guest blows up because the fault could not be recovered using #PF.

I can see two somewhat sane ways to make this work.

1. Access to bad memory results in an async-page-not-present, except that,  i=
t=E2=80=99s not deliverable, the guest is killed. Either that async-page-not=
-present has a special flag saying =E2=80=9Cmemory failure=E2=80=9D or the e=
ventual wakeup says =E2=80=9Cmemory failure=E2=80=9D.

2. Access to bad memory results in #MC.  Sure, #MC is a turd, but it=E2=80=99=
s an *architectural* turd. By all means, have a nice simple PV mechanism to t=
ell the #MC code exactly what went wrong, but keep the overall flow the same=
 as in the native case.

I think I like #2 much better. It has another nice effect: a good implementa=
tion will serve as a way to exercise the #MC code without needing to muck wi=
th EINJ or with whatever magic Tony uses. The average kernel developer does n=
ot have access to a box with testable memory failure reporting.

>=20
> Thanks,
>=20
>        tglx
>=20
>=20
>=20
>=20