Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20220223062412.22334-1-chenyi.qiang@intel.com>
 <CALMp9eT50LjXYSwfWENjmfg=XxT4Bx3RzOYubKty8kr_APXCEw@mail.gmail.com>
 <88eb9a9a-fbe3-8e2c-02bd-4bdfc855b67f@intel.com> <6a839b88-392d-886d-836d-ca04cf700dce@intel.com>
 <7859e03f-10fa-dbc2-ed3c-5c09e62f9016@redhat.com> <bcc83b3d-31fe-949a-6bbf-4615bb982f0c@intel.com>
 <CALMp9eT1NRudtVqPuHU8Y8LpFYWZsAB_MnE2BAbg5NY0jR823w@mail.gmail.com>
In-Reply-To: <CALMp9eT1NRudtVqPuHU8Y8LpFYWZsAB_MnE2BAbg5NY0jR823w@mail.gmail.com>
From:   Jim Mattson <jmattson@google.com>
Date:   Fri, 25 Feb 2022 20:53:48 -0800
Message-ID: <CALMp9eS6cBDuax8O=woSdkNH2e2Y2EodE-7EfUTFfzBvCWCmcg@mail.gmail.com>
Subject: Re: [PATCH v3] KVM: VMX: Enable Notify VM exit
To:     Xiaoyao Li <xiaoyao.li@intel.com>
Cc:     Paolo Bonzini <pbonzini@redhat.com>,
        Chenyi Qiang <chenyi.qiang@intel.com>,
        Sean Christopherson <seanjc@google.com>,
        Vitaly Kuznetsov <vkuznets@redhat.com>,
        Wanpeng Li <wanpengli@tencent.com>,
        Joerg Roedel <joro@8bytes.org>, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Fri, Feb 25, 2022 at 8:25 PM Jim Mattson <jmattson@google.com> wrote:
>
> On Fri, Feb 25, 2022 at 8:07 PM Xiaoyao Li <xiaoyao.li@intel.com> wrote:
> >
> > On 2/25/2022 11:13 PM, Paolo Bonzini wrote:
> > > On 2/25/22 16:12, Xiaoyao Li wrote:
> > >>>>>
> > >>>>
> > >>>> I don't like the idea of making things up without notifying userspace
> > >>>> that this is fictional. How is my customer running nested VMs supposed
> > >>>> to know that L2 didn't actually shutdown, but L0 killed it because the
> > >>>> notify window was exceeded? If this information isn't reported to
> > >>>> userspace, I have no way of getting the information to the customer.
> > >>>
> > >>> Then, maybe a dedicated software define VM exit for it instead of
> > >>> reusing triple fault?
> > >>>
> > >>
> > >> Second thought, we can even just return Notify VM exit to L1 to tell
> > >> L2 causes Notify VM exit, even thought Notify VM exit is not exposed
> > >> to L1.
> > >
> > > That might cause NULL pointer dereferences or other nasty occurrences.
> >
> > IMO, a well written VMM (in L1) should handle it correctly.
> >
> > L0 KVM reports no Notify VM Exit support to L1, so L1 runs without
> > setting Notify VM exit. If a L2 causes notify_vm_exit with
> > invalid_vm_context, L0 just reflects it to L1. In L1's view, there is no
> > support of Notify VM Exit from VMX MSR capability. Following L1 handler
> > is possible:
> >
> > a)      if (notify_vm_exit available & notify_vm_exit enabled) {
> >                 handle in b)
> >         } else {
> >                 report unexpected vm exit reason to userspace;
> >         }
> >
> > b)      similar handler like we implement in KVM:
> >         if (!vm_context_invalid)
> >                 re-enter guest;
> >         else
> >                 report to userspace;
> >
> > c)      no Notify VM Exit related code (e.g. old KVM), it's treated as
> > unsupported exit reason
> >
> > As long as it belongs to any case above, I think L1 can handle it
> > correctly. Any nasty occurrence should be caused by incorrect handler in
> > L1 VMM, in my opinion.
>
> Please test some common hypervisors (e.g. ESXi and Hyper-V).

I took a look at KVM in Linux v4.9 (one of our more popular guests),
and it will not handle this case well:

        if (exit_reason < kvm_vmx_max_exit_handlers
            && kvm_vmx_exit_handlers[exit_reason])
                return kvm_vmx_exit_handlers[exit_reason](vcpu);
        else {
                WARN_ONCE(1, "vmx: unexpected exit reason 0x%x\n", exit_reason);
                kvm_queue_exception(vcpu, UD_VECTOR);
                return 1;
        }

At least there's an L1 kernel log message for the first unexpected
NOTIFY VM-exit, but after that, there is silence. Just a completely
inexplicable #UD in L2, assuming that L2 is resumable at this point.