Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <20190917085304.16987-1-weijiang.yang@intel.com> <20190917085304.16987-2-weijiang.yang@intel.com>
In-Reply-To: <20190917085304.16987-2-weijiang.yang@intel.com>
From:   Jim Mattson <jmattson@google.com>
Date:   Fri, 11 Oct 2019 13:31:08 -0700
Message-ID: <CALMp9eT+P5QTGy=LfZzMozkTC7jdEhbupbfza2tTE3U1grtZkQ@mail.gmail.com>
Subject: Re: [PATCH v5 1/9] Documentation: Introduce EPT based Subpage Protection
To:     Yang Weijiang <weijiang.yang@intel.com>
Cc:     kvm list <kvm@vger.kernel.org>,
        LKML <linux-kernel@vger.kernel.org>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Sean Christopherson <sean.j.christopherson@intel.com>,
        "Michael S. Tsirkin" <mst@redhat.com>,
        =?UTF-8?B?UmFkaW0gS3LEjW3DocWZ?= <rkrcmar@redhat.com>,
        yu.c.zhang@intel.com, alazar@bitdefender.com
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Tue, Sep 17, 2019 at 1:52 AM Yang Weijiang <weijiang.yang@intel.com> wrote:
>
> Co-developed-by: yi.z.zhang@linux.intel.com
> Signed-off-by: yi.z.zhang@linux.intel.com
> Signed-off-by: Yang Weijiang <weijiang.yang@intel.com>
> ---
>  Documentation/virtual/kvm/spp_kvm.txt | 178 ++++++++++++++++++++++++++
>  1 file changed, 178 insertions(+)
>  create mode 100644 Documentation/virtual/kvm/spp_kvm.txt
>
> diff --git a/Documentation/virtual/kvm/spp_kvm.txt b/Documentation/virtual/kvm/spp_kvm.txt
> new file mode 100644
> index 000000000000..1bd1c11d0a99
> --- /dev/null
> +++ b/Documentation/virtual/kvm/spp_kvm.txt
> @@ -0,0 +1,178 @@
> +EPT-Based Sub-Page Protection (SPP) for KVM
> +====================================================
> +
> +1.Overview
> +  EPT-based Sub-Page Protection(SPP) allows VMM to specify
> +  fine-grained(128byte per sub-page) write-protection for guest physical
> +  memory. When it's enabled, the CPU enforces write-access permission
> +  for the sub-pages within a 4KB page, if corresponding bit is set in
> +  permission vector, write to sub-page region is allowed, otherwise,
> +  it's prevented with a EPT violation.
> +
> +  *Note*: In current implementation, SPP is exclusive with nested flag,
> +  if it's on, SPP feature won't work.
> +
> +2.SPP Operation
> +  Sub-Page Protection Table (SPPT) is introduced to manage sub-page
> +  write-access permission.
> +
> +  It is active when:
> +  a) nested flag is turned off.
> +  b) "sub-page write protection" VM-execution control is 1.
> +  c) SPP is initialized with KVM_INIT_SPP ioctl.
> +  d) Sub-page permissions are set with KVM_SUBPAGES_SET_ACCESS ioctl.
> +     see below sections for details.
> +
> +  __________________________________________________________________________
> +
> +  How SPP hardware works:
> +  __________________________________________________________________________
> +
> +  Guest write access --> GPA --> Walk EPT --> EPT leaf entry -----|
> +  |---------------------------------------------------------------|
> +  |-> if VMexec_control.spp && ept_leaf_entry.spp_bit (bit 61)
> +       |
> +       |-> <false> --> EPT legacy behavior
> +       |
> +       |
> +       |-> <true>  --> if ept_leaf_entry.writable
> +                        |
> +                        |-> <true>  --> Ignore SPP
> +                        |
> +                        |-> <false> --> GPA --> Walk SPP 4-level table--|
> +                                                                        |
> +  |------------<----------get-the-SPPT-point-from-VMCS-filed-----<------|
/filed/field/
> +  |
> +  Walk SPP L4E table
> +  |
> +  |---> if-entry-misconfiguration ------------>-------|-------<---------|
> +   |                                                  |                 |
> +  else                                                |                 |
> +   |                                                  |                 |
> +   |   |------------------SPP VMexit<-----------------|                 |
> +   |   |                                                                |
> +   |   |-> exit_qualification & sppt_misconfig --> sppt misconfig       |
> +   |   |                                                                |
> +   |   |-> exit_qualification & sppt_miss --> sppt miss                 |
> +   |---|                                                                |
> +       |                                                                |
> +  walk SPPT L3E--|--> if-entry-misconfiguration------------>------------|
> +                 |                                                      |
> +                else                                                    |
> +                 |                                                      |
> +                 |                                                      |
> +          walk SPPT L2E --|--> if-entry-misconfiguration-------->-------|
> +                          |                                             |
> +                         else                                           |
> +                          |                                             |
> +                          |                                             |
> +                   walk SPPT L1E --|-> if-entry-misconfiguration--->----|
> +                                   |
> +                                 else
> +                                   |
> +                                   |-> if sub-page writable
> +                                   |-> <true>  allow, write access
> +                                   |-> <false> disallow, EPT violation
> +  ______________________________________________________________________________
> +
> +3.IOCTL Interfaces
> +
> +    KVM_INIT_SPP:
> +    Allocate storage for sub-page permission vectors and SPPT root page.
> +
> +    KVM_SUBPAGES_GET_ACCESS:
> +    Get sub-page write permission vectors for given continuous guest pages.
/continuous/contiguous/
> +
> +    KVM_SUBPAGES_SET_ACCESS
> +    Set SPP bit in EPT leaf entries for given continuous guest pages. The
/continuous/contiguous/
> +    actual SPPT setup is triggered when SPP miss vm-exit is handled.
> +
> +    /* for KVM_SUBPAGES_GET_ACCESS and KVM_SUBPAGES_SET_ACCESS */
> +    struct kvm_subpage_info {
> +       __u64 gfn; /* the first page gfn of the continuous pages */
/continuous/contiguous/
> +       __u64 npages; /* number of 4K pages */
> +       __u64 *access_map; /* sub-page write-access bitmap array */
> +    };
> +
> +    #define KVM_SUBPAGES_GET_ACCESS   _IOR(KVMIO,  0x49, __u64)
> +    #define KVM_SUBPAGES_SET_ACCESS   _IOW(KVMIO,  0x4a, __u64)
> +    #define KVM_INIT_SPP              _IOW(KVMIO,  0x4b, __u64)

The ioctls should be documented in api.txt.

> +4.Set Sub-Page Permission
> +
> +  * To enable SPP protection, system admin sets sub-page permission via
Why system admin? Can't any kvm user do this?
> +    KVM_SUBPAGES_SET_ACCESS ioctl:
> +    (1) It first stores the access permissions in bitmap array.
> +
> +    (2) Then, if the target 4KB page is mapped as PT_PAGE_TABLE_LEVEL entry in EPT,
/page is/pages are/
> +       it sets SPP bit of the corresponding entry to mark sub-page protection.
> +       If the 4KB page is mapped as PT_DIRECTORY_LEVEL or PT_PDPE_LEVEL, it
/page is/pages are/
> +       zapps the hugepage entry and let following memroy access to trigger EPT
/zapps/zaps/, /entry/enttries/, /memroy/memory/
> +       page fault, there the gfn is check against SPP permission bitmap and
/page fault/violation/
> +       proper level is selected to set up EPT entry.
> +
> +
> +   The SPPT paging structure format is as below:
> +
> +   Format of the SPPT L4E, L3E, L2E:
> +   | Bit    | Contents                                                                 |
> +   | :----- | :------------------------------------------------------------------------|
> +   | 0      | Valid entry when set; indicates whether the entry is present             |
> +   | 11:1   | Reserved (0)                                                             |
> +   | N-1:12 | Physical address of 4KB aligned SPPT LX-1 Table referenced by this entry |
> +   | 51:N   | Reserved (0)                                                             |
> +   | 63:52  | Reserved (0)                                                             |
> +   Note: N is the physical address width supported by the processor. X is the page level
> +
> +   Format of the SPPT L1E:
> +   | Bit   | Contents                                                          |
> +   | :---- | :---------------------------------------------------------------- |
> +   | 0+2i  | Write permission for i-th 128 byte sub-page region.               |
> +   | 1+2i  | Reserved (0).                                                     |
> +   Note: 0<=i<=31
> +
> +5.SPPT-induced VM exit
> +
> +  * SPPT miss and misconfiguration induced VM exit
> +
> +    A SPPT missing VM exit occurs when walk the SPPT, there is no SPPT
> +    misconfiguration but a paging-structure entry is not
> +    present in any of L4E/L3E/L2E entries.
> +
> +    A SPPT misconfiguration VM exit occurs when reserved bits or unsupported values
> +    are set in SPPT entry.
> +
> +    *NOTE* SPPT miss and SPPT misconfigurations can occur only due to an
> +    attempt to write memory with a guest physical address.

Can you clarify what this means? For instance, setting an A or D bit
in a PTE is an attempt to "write memory with a guest physical
address," but per the SDM, it is not an operation that is eligible for
sub-page write permissions.

> +  * SPP permission induced VM exit
> +    SPP sub-page permission induced violation is reported as EPT violation
> +    thesefore causes VM exit.
/thesefore/therefore/

> +
> +6.SPPT-induced VM exit handling
> +
> +  #define EXIT_REASON_SPP                 66
> +
> +  static int (*const kvm_vmx_exit_handlers[])(struct kvm_vcpu *vcpu) = {
> +    ...
> +    [EXIT_REASON_SPP]                     = handle_spp,
> +    ...
> +  };
> +
> +  New exit qualification for SPPT-induced vmexits.
> +
> +  | Bit   | Contents                                                          |
> +  | :---- | :---------------------------------------------------------------- |
> +  | 10:0  | Reserved (0).                                                     |
> +  | 11    | SPPT VM exit type. Set for SPPT Miss, cleared for SPPT Misconfig. |
> +  | 12    | NMI unblocking due to IRET                                        |
> +  | 63:13 | Reserved (0)                                                      |
> +
> +  In addition to the exit qualification, guest linear address and guest
> +  physical address fields will be reported.
> +
> +  * SPPT miss and misconfiguration induced VM exit
> +    Set up SPPT entries correctly.
> +
> +  * SPP permission induced VM exit
> +    This kind of VM exit is left to VMI tool to handle.
> --
> 2.17.2
>