Fix a handful of minor bugs in KVMGT, and overhaul KVM's page-track APIs
to provide a leaner and cleaner interface. The motivation for this
series is to (significantly) reduce the number of KVM APIs that KVMGT
uses, with a long-term goal of making all kvm_host.h headers KVM-internal.
If there are no objections or issues, my plan is to take this through the
KVM tree for 6.6 (I had it ready early last week, and then forgot to actually
post v4, /facepalm).
Thanks much for all the help!
v4:
- Collect tags. [Yongwei, Zhi, Yan]
- Add a patch to fix a benign (other than a WARN) bug where KVMGT would
attempt to unpin an empty range. [Yan]
- Move the check for an attached vGPU all the way up to shadow_ppgtt_mm(). [Zhi]
v3:
- https://lore.kernel.org/all/[email protected]
- Collect reviewed/tested tags (I apologize if I missed any, I manually
gathered them this time due to a goof in my workflow). [Yan]
- Drop check on max KVM paging size from KVMGT. [Yan]
- Drop the explicit change on THP pages, and instead validate that the
pfns (not struct page pointers) are contiguous. [Yan]
- Fix buggy intel_gvt_dma_map_guest_page() usage by eliminating a helper
for shadowing 2MiB GTT entries. [Yan]
- Move kvm_arch_flush_shadow_{all,memslot}() to mmu.c instead of exposing
kvm_mmu_zap_all_fast() outside of mmu.c. [Yan]
- Fix an alignment goof in hlist_for_each_entry_srcu() usage. [Yan]
- Wrap full definition of external page track structures with
CONFIG_KVM_EXTERNAL_WRITE_TRACKING. [Yan]
v2:
- https://lore.kernel.org/all/[email protected]
- Reuse vgpu_lock to protect gfn hash instead of introducing a new (and
buggy) mutext. [Yan]
- Remove a spurious return from kvm_page_track_init(). [Yan]
- Take @kvm directly in the inner __kvm_page_track_write(). [Yan]
- Delete the gfn sanity check that relies on kvm_is_visible_gfn() instead
of providing a dedicated interface. [Yan]
v1: https://lore.kernel.org/lkml/[email protected]
Sean Christopherson (24):
drm/i915/gvt: Verify pfn is "valid" before dereferencing "struct page"
drm/i915/gvt: Verify hugepages are contiguous in physical address
space
drm/i915/gvt: Put the page reference obtained by KVM's gfn_to_pfn()
drm/i915/gvt: Explicitly check that vGPU is attached before shadowing
drm/i915/gvt: Error out on an attempt to shadowing an unknown GTT
entry type
drm/i915/gvt: Don't rely on KVM's gfn_to_pfn() to query possible 2M
GTT
drm/i915/gvt: Use an "unsigned long" to iterate over memslot gfns
drm/i915/gvt: Drop unused helper intel_vgpu_reset_gtt()
drm/i915/gvt: Protect gfn hash table with vgpu_lock
KVM: x86/mmu: Move kvm_arch_flush_shadow_{all,memslot}() to mmu.c
KVM: x86/mmu: Don't rely on page-track mechanism to flush on memslot
change
KVM: x86/mmu: Don't bounce through page-track mechanism for guest PTEs
KVM: drm/i915/gvt: Drop @vcpu from KVM's ->track_write() hook
KVM: x86: Reject memslot MOVE operations if KVMGT is attached
drm/i915/gvt: Don't bother removing write-protection on to-be-deleted
slot
KVM: x86/mmu: Move KVM-only page-track declarations to internal header
KVM: x86/mmu: Use page-track notifiers iff there are external users
KVM: x86/mmu: Drop infrastructure for multiple page-track modes
KVM: x86/mmu: Rename page-track APIs to reflect the new reality
KVM: x86/mmu: Assert that correct locks are held for page
write-tracking
KVM: x86/mmu: Bug the VM if write-tracking is used but not enabled
KVM: x86/mmu: Drop @slot param from exported/external page-track APIs
KVM: x86/mmu: Handle KVM bookkeeping in page-track APIs, not callers
drm/i915/gvt: Drop final dependencies on KVM internal details
Yan Zhao (5):
drm/i915/gvt: remove interface intel_gvt_is_valid_gfn
drm/i915/gvt: Don't try to unpin an empty page range
KVM: x86: Add a new page-track hook to handle memslot deletion
drm/i915/gvt: switch from ->track_flush_slot() to
->track_remove_region()
KVM: x86: Remove the unused page-track hook track_flush_slot()
arch/x86/include/asm/kvm_host.h | 16 +-
arch/x86/include/asm/kvm_page_track.h | 73 +++-----
arch/x86/kvm/mmu.h | 2 +
arch/x86/kvm/mmu/mmu.c | 51 +++--
arch/x86/kvm/mmu/page_track.c | 256 +++++++++++++-------------
arch/x86/kvm/mmu/page_track.h | 58 ++++++
arch/x86/kvm/x86.c | 22 +--
drivers/gpu/drm/i915/gvt/gtt.c | 102 ++--------
drivers/gpu/drm/i915/gvt/gtt.h | 1 -
drivers/gpu/drm/i915/gvt/gvt.h | 3 +-
drivers/gpu/drm/i915/gvt/kvmgt.c | 120 +++++-------
drivers/gpu/drm/i915/gvt/page_track.c | 10 +-
12 files changed, 322 insertions(+), 392 deletions(-)
create mode 100644 arch/x86/kvm/mmu/page_track.h
base-commit: fdf0eaf11452d72945af31804e2a1048ee1b574c
--
2.41.0.487.g6d72f3e995-goog
Bug the VM if something attempts to write-track a gfn, but write-tracking
isn't enabled. The VM is doomed (and KVM has an egregious bug) if KVM or
KVMGT wants to shadow guest page tables but can't because write-tracking
isn't enabled.
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
arch/x86/kvm/mmu/page_track.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index 29ae61f1e303..eedb5889d73e 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -90,7 +90,7 @@ void kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
lockdep_assert_once(lockdep_is_held(&kvm->slots_lock) ||
srcu_read_lock_held(&kvm->srcu));
- if (WARN_ON(!kvm_page_track_write_tracking_enabled(kvm)))
+ if (KVM_BUG_ON(!kvm_page_track_write_tracking_enabled(kvm), kvm))
return;
update_gfn_write_track(slot, gfn, 1);
@@ -122,7 +122,7 @@ void kvm_write_track_remove_gfn(struct kvm *kvm,
lockdep_assert_once(lockdep_is_held(&kvm->slots_lock) ||
srcu_read_lock_held(&kvm->srcu));
- if (WARN_ON(!kvm_page_track_write_tracking_enabled(kvm)))
+ if (KVM_BUG_ON(!kvm_page_track_write_tracking_enabled(kvm), kvm))
return;
update_gfn_write_track(slot, gfn, -1);
--
2.41.0.487.g6d72f3e995-goog
Disallow moving memslots if the VM has external page-track users, i.e. if
KVMGT is being used to expose a virtual GPU to the guest, as KVMGT doesn't
correctly handle moving memory regions.
Note, this is potential ABI breakage! E.g. userspace could move regions
that aren't shadowed by KVMGT without harming the guest. However, the
only known user of KVMGT is QEMU, and QEMU doesn't move generic memory
regions. KVM's own support for moving memory regions was also broken for
multiple years (albeit for an edge case, but arguably moving RAM is
itself an edge case), e.g. see commit edd4fa37baa6 ("KVM: x86: Allocate
new rmap and large page tracking when moving memslot").
Reviewed-by: Yan Zhao <[email protected]>
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
arch/x86/include/asm/kvm_page_track.h | 3 +++
arch/x86/kvm/mmu/page_track.c | 5 +++++
arch/x86/kvm/x86.c | 7 +++++++
3 files changed, 15 insertions(+)
diff --git a/arch/x86/include/asm/kvm_page_track.h b/arch/x86/include/asm/kvm_page_track.h
index 8c4d216e3b2b..f744682648e7 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -75,4 +75,7 @@ kvm_page_track_unregister_notifier(struct kvm *kvm,
void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
int bytes);
void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
+
+bool kvm_page_track_has_external_user(struct kvm *kvm);
+
#endif
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index 891e5cc52b45..e6de9638e560 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -303,3 +303,8 @@ void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot)
n->track_flush_slot(kvm, slot, n);
srcu_read_unlock(&head->track_srcu, idx);
}
+
+bool kvm_page_track_has_external_user(struct kvm *kvm)
+{
+ return hlist_empty(&kvm->arch.track_notifier_head.track_notifier_list);
+}
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 059571d5abed..4394bb49051f 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -12606,6 +12606,13 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,
struct kvm_memory_slot *new,
enum kvm_mr_change change)
{
+ /*
+ * KVM doesn't support moving memslots when there are external page
+ * trackers attached to the VM, i.e. if KVMGT is in use.
+ */
+ if (change == KVM_MR_MOVE && kvm_page_track_has_external_user(kvm))
+ return -EINVAL;
+
if (change == KVM_MR_CREATE || change == KVM_MR_MOVE) {
if ((new->base_gfn + new->npages - 1) > kvm_mmu_max_gfn())
return -EINVAL;
--
2.41.0.487.g6d72f3e995-goog
Rename the page-track APIs to capture that they're all about tracking
writes, now that the facade of supporting multiple modes is gone.
Opportunstically replace "slot" with "gfn" in anticipation of removing
the @slot param from the external APIs.
No functional change intended.
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
arch/x86/include/asm/kvm_page_track.h | 8 ++++----
arch/x86/kvm/mmu/mmu.c | 8 ++++----
arch/x86/kvm/mmu/page_track.c | 21 +++++++++------------
arch/x86/kvm/mmu/page_track.h | 4 ++--
drivers/gpu/drm/i915/gvt/kvmgt.c | 4 ++--
5 files changed, 21 insertions(+), 24 deletions(-)
diff --git a/arch/x86/include/asm/kvm_page_track.h b/arch/x86/include/asm/kvm_page_track.h
index 9e4ee26d1779..f5c1db36cdb7 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -4,10 +4,10 @@
#include <linux/kvm_types.h>
-void kvm_slot_page_track_add_page(struct kvm *kvm,
- struct kvm_memory_slot *slot, gfn_t gfn);
-void kvm_slot_page_track_remove_page(struct kvm *kvm,
- struct kvm_memory_slot *slot, gfn_t gfn);
+void kvm_write_track_add_gfn(struct kvm *kvm,
+ struct kvm_memory_slot *slot, gfn_t gfn);
+void kvm_write_track_remove_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
+ gfn_t gfn);
#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
/*
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index b8dce17bffdc..a0309fde3549 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -840,7 +840,7 @@ static void account_shadowed(struct kvm *kvm, struct kvm_mmu_page *sp)
/* the non-leaf shadow pages are keeping readonly. */
if (sp->role.level > PG_LEVEL_4K)
- return kvm_slot_page_track_add_page(kvm, slot, gfn);
+ return kvm_write_track_add_gfn(kvm, slot, gfn);
kvm_mmu_gfn_disallow_lpage(slot, gfn);
@@ -886,7 +886,7 @@ static void unaccount_shadowed(struct kvm *kvm, struct kvm_mmu_page *sp)
slots = kvm_memslots_for_spte_role(kvm, sp->role);
slot = __gfn_to_memslot(slots, gfn);
if (sp->role.level > PG_LEVEL_4K)
- return kvm_slot_page_track_remove_page(kvm, slot, gfn);
+ return kvm_write_track_remove_gfn(kvm, slot, gfn);
kvm_mmu_gfn_allow_lpage(slot, gfn);
}
@@ -2830,7 +2830,7 @@ int mmu_try_to_unsync_pages(struct kvm *kvm, const struct kvm_memory_slot *slot,
* track machinery is used to write-protect upper-level shadow pages,
* i.e. this guards the role.level == 4K assertion below!
*/
- if (kvm_slot_page_track_is_active(kvm, slot, gfn))
+ if (kvm_gfn_is_write_tracked(kvm, slot, gfn))
return -EPERM;
/*
@@ -4231,7 +4231,7 @@ static bool page_fault_handle_page_track(struct kvm_vcpu *vcpu,
* guest is writing the page which is write tracked which can
* not be fixed by page fault handler.
*/
- if (kvm_slot_page_track_is_active(vcpu->kvm, fault->slot, fault->gfn))
+ if (kvm_gfn_is_write_tracked(vcpu->kvm, fault->slot, fault->gfn))
return true;
return false;
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index cdc6069b8caf..b835ba7f325c 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -84,10 +84,9 @@ static void update_gfn_write_track(struct kvm_memory_slot *slot, gfn_t gfn,
* @slot: the @gfn belongs to.
* @gfn: the guest page.
*/
-void kvm_slot_page_track_add_page(struct kvm *kvm,
- struct kvm_memory_slot *slot, gfn_t gfn)
+void kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
+ gfn_t gfn)
{
-
if (WARN_ON(!kvm_page_track_write_tracking_enabled(kvm)))
return;
@@ -102,12 +101,11 @@ void kvm_slot_page_track_add_page(struct kvm *kvm,
if (kvm_mmu_slot_gfn_write_protect(kvm, slot, gfn, PG_LEVEL_4K))
kvm_flush_remote_tlbs(kvm);
}
-EXPORT_SYMBOL_GPL(kvm_slot_page_track_add_page);
+EXPORT_SYMBOL_GPL(kvm_write_track_add_gfn);
/*
* remove the guest page from the tracking pool which stops the interception
- * of corresponding access on that page. It is the opposed operation of
- * kvm_slot_page_track_add_page().
+ * of corresponding access on that page.
*
* It should be called under the protection both of mmu-lock and kvm->srcu
* or kvm->slots_lock.
@@ -116,8 +114,8 @@ EXPORT_SYMBOL_GPL(kvm_slot_page_track_add_page);
* @slot: the @gfn belongs to.
* @gfn: the guest page.
*/
-void kvm_slot_page_track_remove_page(struct kvm *kvm,
- struct kvm_memory_slot *slot, gfn_t gfn)
+void kvm_write_track_remove_gfn(struct kvm *kvm,
+ struct kvm_memory_slot *slot, gfn_t gfn)
{
if (WARN_ON(!kvm_page_track_write_tracking_enabled(kvm)))
return;
@@ -130,14 +128,13 @@ void kvm_slot_page_track_remove_page(struct kvm *kvm,
*/
kvm_mmu_gfn_allow_lpage(slot, gfn);
}
-EXPORT_SYMBOL_GPL(kvm_slot_page_track_remove_page);
+EXPORT_SYMBOL_GPL(kvm_write_track_remove_gfn);
/*
* check if the corresponding access on the specified guest page is tracked.
*/
-bool kvm_slot_page_track_is_active(struct kvm *kvm,
- const struct kvm_memory_slot *slot,
- gfn_t gfn)
+bool kvm_gfn_is_write_tracked(struct kvm *kvm,
+ const struct kvm_memory_slot *slot, gfn_t gfn)
{
int index;
diff --git a/arch/x86/kvm/mmu/page_track.h b/arch/x86/kvm/mmu/page_track.h
index 789d0c479519..50d3278e8c69 100644
--- a/arch/x86/kvm/mmu/page_track.h
+++ b/arch/x86/kvm/mmu/page_track.h
@@ -15,8 +15,8 @@ int kvm_page_track_create_memslot(struct kvm *kvm,
struct kvm_memory_slot *slot,
unsigned long npages);
-bool kvm_slot_page_track_is_active(struct kvm *kvm,
- const struct kvm_memory_slot *slot, gfn_t gfn);
+bool kvm_gfn_is_write_tracked(struct kvm *kvm,
+ const struct kvm_memory_slot *slot, gfn_t gfn);
#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
int kvm_page_track_init(struct kvm *kvm);
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index e71182b8a3f2..05a7e614ead0 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -1564,7 +1564,7 @@ int intel_gvt_page_track_add(struct intel_vgpu *info, u64 gfn)
}
write_lock(&kvm->mmu_lock);
- kvm_slot_page_track_add_page(kvm, slot, gfn);
+ kvm_write_track_add_gfn(kvm, slot, gfn);
write_unlock(&kvm->mmu_lock);
srcu_read_unlock(&kvm->srcu, idx);
@@ -1593,7 +1593,7 @@ int intel_gvt_page_track_remove(struct intel_vgpu *info, u64 gfn)
}
write_lock(&kvm->mmu_lock);
- kvm_slot_page_track_remove_page(kvm, slot, gfn);
+ kvm_write_track_remove_gfn(kvm, slot, gfn);
write_unlock(&kvm->mmu_lock);
srcu_read_unlock(&kvm->srcu, idx);
--
2.41.0.487.g6d72f3e995-goog
Bail from ppgtt_populate_shadow_entry() if an unexpected GTT entry type
is encountered instead of subtly falling through to the common "direct
shadow" path. Eliminating the default/error path's reliance on the common
handling will allow hoisting intel_gvt_dma_map_guest_page() into the case
statements so that the 2MiB case can try intel_gvt_dma_map_guest_page()
and fallback to splitting the entry on failure.
Reviewed-by: Zhi Wang <[email protected]>
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
drivers/gpu/drm/i915/gvt/gtt.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c
index 2aed31b497c9..61e38acee2d5 100644
--- a/drivers/gpu/drm/i915/gvt/gtt.c
+++ b/drivers/gpu/drm/i915/gvt/gtt.c
@@ -1306,6 +1306,7 @@ static int ppgtt_populate_shadow_entry(struct intel_vgpu *vgpu,
return -EINVAL;
default:
GEM_BUG_ON(1);
+ return -EINVAL;
}
/* direct shadow */
--
2.41.0.487.g6d72f3e995-goog
Drop intel_vgpu_reset_gtt() as it no longer has any callers. In addition
to eliminating dead code, this eliminates the last possible scenario where
__kvmgt_protect_table_find() can be reached without holding vgpu_lock.
Requiring vgpu_lock to be held when calling __kvmgt_protect_table_find()
will allow a protecting the gfn hash with vgpu_lock without too much fuss.
No functional change intended.
Fixes: ba25d977571e ("drm/i915/gvt: Do not destroy ppgtt_mm during vGPU D3->D0.")
Reviewed-by: Yan Zhao <[email protected]>
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
drivers/gpu/drm/i915/gvt/gtt.c | 18 ------------------
drivers/gpu/drm/i915/gvt/gtt.h | 1 -
2 files changed, 19 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c
index f505be9e647a..c3c623b929ce 100644
--- a/drivers/gpu/drm/i915/gvt/gtt.c
+++ b/drivers/gpu/drm/i915/gvt/gtt.c
@@ -2817,24 +2817,6 @@ void intel_vgpu_reset_ggtt(struct intel_vgpu *vgpu, bool invalidate_old)
ggtt_invalidate(gvt->gt);
}
-/**
- * intel_vgpu_reset_gtt - reset the all GTT related status
- * @vgpu: a vGPU
- *
- * This function is called from vfio core to reset reset all
- * GTT related status, including GGTT, PPGTT, scratch page.
- *
- */
-void intel_vgpu_reset_gtt(struct intel_vgpu *vgpu)
-{
- /* Shadow pages are only created when there is no page
- * table tracking data, so remove page tracking data after
- * removing the shadow pages.
- */
- intel_vgpu_destroy_all_ppgtt_mm(vgpu);
- intel_vgpu_reset_ggtt(vgpu, true);
-}
-
/**
* intel_gvt_restore_ggtt - restore all vGPU's ggtt entries
* @gvt: intel gvt device
diff --git a/drivers/gpu/drm/i915/gvt/gtt.h b/drivers/gpu/drm/i915/gvt/gtt.h
index a3b0f59ec8bd..4cb183e06e95 100644
--- a/drivers/gpu/drm/i915/gvt/gtt.h
+++ b/drivers/gpu/drm/i915/gvt/gtt.h
@@ -224,7 +224,6 @@ void intel_vgpu_reset_ggtt(struct intel_vgpu *vgpu, bool invalidate_old);
void intel_vgpu_invalidate_ppgtt(struct intel_vgpu *vgpu);
int intel_gvt_init_gtt(struct intel_gvt *gvt);
-void intel_vgpu_reset_gtt(struct intel_vgpu *vgpu);
void intel_gvt_clean_gtt(struct intel_gvt *gvt);
struct intel_vgpu_mm *intel_gvt_find_ppgtt_mm(struct intel_vgpu *vgpu,
--
2.41.0.487.g6d72f3e995-goog
From: Yan Zhao <[email protected]>
Remove ->track_remove_slot(), there are no longer any users and it's
unlikely a "flush" hook will ever be the correct API to provide to an
external page-track user.
Cc: Zhenyu Wang <[email protected]>
Suggested-by: Sean Christopherson <[email protected]>
Signed-off-by: Yan Zhao <[email protected]>
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
arch/x86/include/asm/kvm_page_track.h | 11 -----------
arch/x86/kvm/mmu/mmu.c | 2 --
arch/x86/kvm/mmu/page_track.c | 26 --------------------------
3 files changed, 39 deletions(-)
diff --git a/arch/x86/include/asm/kvm_page_track.h b/arch/x86/include/asm/kvm_page_track.h
index cfd36c22b467..5c348ffdc194 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -33,16 +33,6 @@ struct kvm_page_track_notifier_node {
*/
void (*track_write)(gpa_t gpa, const u8 *new, int bytes,
struct kvm_page_track_notifier_node *node);
- /*
- * It is called when memory slot is being moved or removed
- * users can drop write-protection for the pages in that memory slot
- *
- * @kvm: the kvm where memory slot being moved or removed
- * @slot: the memory slot being moved or removed
- * @node: this node
- */
- void (*track_flush_slot)(struct kvm *kvm, struct kvm_memory_slot *slot,
- struct kvm_page_track_notifier_node *node);
/*
* Invoked when a memory region is removed from the guest. Or in KVM
@@ -85,7 +75,6 @@ kvm_page_track_unregister_notifier(struct kvm *kvm,
struct kvm_page_track_notifier_node *n);
void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
int bytes);
-void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
bool kvm_page_track_has_external_user(struct kvm *kvm);
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index c404264f8de5..c1d3ac303964 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -6749,8 +6749,6 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
struct kvm_memory_slot *slot)
{
kvm_mmu_zap_all_fast(kvm);
-
- kvm_page_track_flush_slot(kvm, slot);
}
void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen)
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index d971c28be99d..2a6ab7c455c0 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -278,32 +278,6 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
kvm_mmu_track_write(vcpu, gpa, new, bytes);
}
-/*
- * Notify the node that memory slot is being removed or moved so that it can
- * drop write-protection for the pages in the memory slot.
- *
- * The node should figure out it has any write-protected pages in this slot
- * by itself.
- */
-void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot)
-{
- struct kvm_page_track_notifier_head *head;
- struct kvm_page_track_notifier_node *n;
- int idx;
-
- head = &kvm->arch.track_notifier_head;
-
- if (hlist_empty(&head->track_notifier_list))
- return;
-
- idx = srcu_read_lock(&head->track_srcu);
- hlist_for_each_entry_srcu(n, &head->track_notifier_list, node,
- srcu_read_lock_held(&head->track_srcu))
- if (n->track_flush_slot)
- n->track_flush_slot(kvm, slot, n);
- srcu_read_unlock(&head->track_srcu, idx);
-}
-
/*
* Notify external page track nodes that a memory region is being removed from
* the VM, e.g. so that users can free any associated metadata.
--
2.41.0.487.g6d72f3e995-goog
From: Yan Zhao <[email protected]>
Currently intel_gvt_is_valid_gfn() is called in two places:
(1) shadowing guest GGTT entry
(2) shadowing guest PPGTT leaf entry,
which was introduced in commit cc753fbe1ac4
("drm/i915/gvt: validate gfn before set shadow page entry").
However, now it's not necessary to call this interface any more, because
a. GGTT partial write issue has been fixed by
commit bc0686ff5fad
("drm/i915/gvt: support inconsecutive partial gtt entry write")
commit 510fe10b6180
("drm/i915/gvt: fix a bug of partially write ggtt enties")
b. PPGTT resides in normal guest RAM and we only treat 8-byte writes
as valid page table writes. Any invalid GPA found is regarded as
an error, either due to guest misbehavior/attack or bug in host
shadow code.
So,rather than do GFN pre-checking and replace invalid GFNs with
scratch GFN and continue silently, just remove the pre-checking and
abort PPGTT shadowing on error detected.
c. GFN validity check is still performed in
intel_gvt_dma_map_guest_page() --> gvt_pin_guest_page().
It's more desirable to call VFIO interface to do both validity check
and mapping.
Calling intel_gvt_is_valid_gfn() to do GFN validity check from KVM side
while later mapping the GFN through VFIO interface is unnecessarily
fragile and confusing for unaware readers.
Signed-off-by: Yan Zhao <[email protected]>
[sean: remove now-unused local variables]
Acked-by: Zhi Wang <[email protected]>
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
drivers/gpu/drm/i915/gvt/gtt.c | 36 +---------------------------------
1 file changed, 1 insertion(+), 35 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c
index 58b9b316ae46..f30922c55a0c 100644
--- a/drivers/gpu/drm/i915/gvt/gtt.c
+++ b/drivers/gpu/drm/i915/gvt/gtt.c
@@ -49,22 +49,6 @@
static bool enable_out_of_sync = false;
static int preallocated_oos_pages = 8192;
-static bool intel_gvt_is_valid_gfn(struct intel_vgpu *vgpu, unsigned long gfn)
-{
- struct kvm *kvm = vgpu->vfio_device.kvm;
- int idx;
- bool ret;
-
- if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, vgpu->status))
- return false;
-
- idx = srcu_read_lock(&kvm->srcu);
- ret = kvm_is_visible_gfn(kvm, gfn);
- srcu_read_unlock(&kvm->srcu, idx);
-
- return ret;
-}
-
/*
* validate a gm address and related range size,
* translate it to host gm address
@@ -1333,11 +1317,9 @@ static int ppgtt_populate_shadow_entry(struct intel_vgpu *vgpu,
static int ppgtt_populate_spt(struct intel_vgpu_ppgtt_spt *spt)
{
struct intel_vgpu *vgpu = spt->vgpu;
- struct intel_gvt *gvt = vgpu->gvt;
- const struct intel_gvt_gtt_pte_ops *ops = gvt->gtt.pte_ops;
struct intel_vgpu_ppgtt_spt *s;
struct intel_gvt_gtt_entry se, ge;
- unsigned long gfn, i;
+ unsigned long i;
int ret;
trace_spt_change(spt->vgpu->id, "born", spt,
@@ -1354,13 +1336,6 @@ static int ppgtt_populate_spt(struct intel_vgpu_ppgtt_spt *spt)
ppgtt_generate_shadow_entry(&se, s, &ge);
ppgtt_set_shadow_entry(spt, &se, i);
} else {
- gfn = ops->get_pfn(&ge);
- if (!intel_gvt_is_valid_gfn(vgpu, gfn)) {
- ops->set_pfn(&se, gvt->gtt.scratch_mfn);
- ppgtt_set_shadow_entry(spt, &se, i);
- continue;
- }
-
ret = ppgtt_populate_shadow_entry(vgpu, spt, i, &ge);
if (ret)
goto fail;
@@ -2335,14 +2310,6 @@ static int emulate_ggtt_mmio_write(struct intel_vgpu *vgpu, unsigned int off,
m.val64 = e.val64;
m.type = e.type;
- /* one PTE update may be issued in multiple writes and the
- * first write may not construct a valid gfn
- */
- if (!intel_gvt_is_valid_gfn(vgpu, gfn)) {
- ops->set_pfn(&m, gvt->gtt.scratch_mfn);
- goto out;
- }
-
ret = intel_gvt_dma_map_guest_page(vgpu, gfn, PAGE_SIZE,
&dma_addr);
if (ret) {
@@ -2359,7 +2326,6 @@ static int emulate_ggtt_mmio_write(struct intel_vgpu *vgpu, unsigned int off,
ops->clear_present(&m);
}
-out:
ggtt_set_guest_entry(ggtt_mm, &e, g_gtt_index);
ggtt_get_host_entry(ggtt_mm, &e, g_gtt_index);
--
2.41.0.487.g6d72f3e995-goog
From: Yan Zhao <[email protected]>
Attempt to unpin pages in the error path of gvt_pin_guest_page() if and
only if at least one page was successfully pinned. Unpinning doesn't
cause functional problems, but vfio_device_container_unpin_pages()
rightfully warns about being asked to unpin zero pages.
Signed-off-by: Yan Zhao <[email protected]>
[sean: write changelog]
Signed-off-by: Sean Christopherson <[email protected]>
---
drivers/gpu/drm/i915/gvt/kvmgt.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 429f0f993a13..0366a699baf5 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -172,7 +172,8 @@ static int gvt_pin_guest_page(struct intel_vgpu *vgpu, unsigned long gfn,
*page = base_page;
return 0;
err:
- gvt_unpin_guest_page(vgpu, gfn, npage * PAGE_SIZE);
+ if (npage)
+ gvt_unpin_guest_page(vgpu, gfn, npage * PAGE_SIZE);
return ret;
}
--
2.41.0.487.g6d72f3e995-goog
Call kvm_mmu_zap_all_fast() directly when flushing a memslot instead of
bouncing through the page-track mechanism. KVM (unfortunately) needs to
zap and flush all page tables on memslot DELETE/MOVE irrespective of
whether KVM is shadowing guest page tables.
This will allow changing KVM to register a page-track notifier on the
first shadow root allocation, and will also allow deleting the misguided
kvm_page_track_flush_slot() hook itself once KVM-GT also moves to a
different method for reacting to memslot changes.
No functional change intended.
Cc: Yan Zhao <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Reviewed-by: Yan Zhao <[email protected]>
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
arch/x86/kvm/mmu/mmu.c | 10 ++--------
1 file changed, 2 insertions(+), 8 deletions(-)
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index c6dee659d592..79ea57396d97 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -6199,13 +6199,6 @@ static bool kvm_has_zapped_obsolete_pages(struct kvm *kvm)
return unlikely(!list_empty_careful(&kvm->arch.zapped_obsolete_pages));
}
-static void kvm_mmu_invalidate_zap_pages_in_memslot(struct kvm *kvm,
- struct kvm_memory_slot *slot,
- struct kvm_page_track_notifier_node *node)
-{
- kvm_mmu_zap_all_fast(kvm);
-}
-
int kvm_mmu_init_vm(struct kvm *kvm)
{
struct kvm_page_track_notifier_node *node = &kvm->arch.mmu_sp_tracker;
@@ -6223,7 +6216,6 @@ int kvm_mmu_init_vm(struct kvm *kvm)
}
node->track_write = kvm_mmu_pte_write;
- node->track_flush_slot = kvm_mmu_invalidate_zap_pages_in_memslot;
kvm_page_track_register_notifier(kvm, node);
kvm->arch.split_page_header_cache.kmem_cache = mmu_page_header_cache;
@@ -6765,6 +6757,8 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm)
void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
struct kvm_memory_slot *slot)
{
+ kvm_mmu_zap_all_fast(kvm);
+
kvm_page_track_flush_slot(kvm, slot);
}
--
2.41.0.487.g6d72f3e995-goog
From: Yan Zhao <[email protected]>
Add a new page-track hook, track_remove_region(), that is called when a
memslot DELETE operation is about to be committed. The "remove" hook
will be used by KVMGT and will effectively replace the existing
track_flush_slot() altogether now that KVM itself doesn't rely on the
"flush" hook either.
The "flush" hook is flawed as it's invoked before the memslot operation
is guaranteed to succeed, i.e. KVM might ultimately keep the existing
memslot without notifying external page track users, a.k.a. KVMGT. In
practice, this can't currently happen on x86, but there are no guarantees
that won't change in the future, not to mention that "flush" does a very
poor job of describing what is happening.
Pass in the gfn+nr_pages instead of the slot itself so external users,
i.e. KVMGT, don't need to exposed to KVM internals (memslots). This will
help set the stage for additional cleanups to the page-track APIs.
Opportunistically align the existing srcu_read_lock_held() usage so that
the new case doesn't stand out like a sore thumb (and not aligning the
new code makes bots unhappy).
Cc: Zhenyu Wang <[email protected]>
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Yan Zhao <[email protected]>
Co-developed-by: Sean Christopherson <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
arch/x86/include/asm/kvm_page_track.h | 12 ++++++++++++
arch/x86/kvm/mmu/page_track.c | 27 +++++++++++++++++++++++++--
arch/x86/kvm/x86.c | 3 +++
3 files changed, 40 insertions(+), 2 deletions(-)
diff --git a/arch/x86/include/asm/kvm_page_track.h b/arch/x86/include/asm/kvm_page_track.h
index f744682648e7..cfd36c22b467 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -43,6 +43,17 @@ struct kvm_page_track_notifier_node {
*/
void (*track_flush_slot)(struct kvm *kvm, struct kvm_memory_slot *slot,
struct kvm_page_track_notifier_node *node);
+
+ /*
+ * Invoked when a memory region is removed from the guest. Or in KVM
+ * terms, when a memslot is deleted.
+ *
+ * @gfn: base gfn of the region being removed
+ * @nr_pages: number of pages in the to-be-removed region
+ * @node: this node
+ */
+ void (*track_remove_region)(gfn_t gfn, unsigned long nr_pages,
+ struct kvm_page_track_notifier_node *node);
};
int kvm_page_track_init(struct kvm *kvm);
@@ -75,6 +86,7 @@ kvm_page_track_unregister_notifier(struct kvm *kvm,
void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
int bytes);
void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
+void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
bool kvm_page_track_has_external_user(struct kvm *kvm);
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index e6de9638e560..d971c28be99d 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -270,7 +270,7 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
idx = srcu_read_lock(&head->track_srcu);
hlist_for_each_entry_srcu(n, &head->track_notifier_list, node,
- srcu_read_lock_held(&head->track_srcu))
+ srcu_read_lock_held(&head->track_srcu))
if (n->track_write)
n->track_write(gpa, new, bytes, n);
srcu_read_unlock(&head->track_srcu, idx);
@@ -298,12 +298,35 @@ void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot)
idx = srcu_read_lock(&head->track_srcu);
hlist_for_each_entry_srcu(n, &head->track_notifier_list, node,
- srcu_read_lock_held(&head->track_srcu))
+ srcu_read_lock_held(&head->track_srcu))
if (n->track_flush_slot)
n->track_flush_slot(kvm, slot, n);
srcu_read_unlock(&head->track_srcu, idx);
}
+/*
+ * Notify external page track nodes that a memory region is being removed from
+ * the VM, e.g. so that users can free any associated metadata.
+ */
+void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot)
+{
+ struct kvm_page_track_notifier_head *head;
+ struct kvm_page_track_notifier_node *n;
+ int idx;
+
+ head = &kvm->arch.track_notifier_head;
+
+ if (hlist_empty(&head->track_notifier_list))
+ return;
+
+ idx = srcu_read_lock(&head->track_srcu);
+ hlist_for_each_entry_srcu(n, &head->track_notifier_list, node,
+ srcu_read_lock_held(&head->track_srcu))
+ if (n->track_remove_region)
+ n->track_remove_region(slot->base_gfn, slot->npages, n);
+ srcu_read_unlock(&head->track_srcu, idx);
+}
+
bool kvm_page_track_has_external_user(struct kvm *kvm)
{
return hlist_empty(&kvm->arch.track_notifier_head.track_notifier_list);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 4394bb49051f..e9ecdf5fbdac 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -12767,6 +12767,9 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
const struct kvm_memory_slot *new,
enum kvm_mr_change change)
{
+ if (change == KVM_MR_DELETE)
+ kvm_page_track_delete_slot(kvm, old);
+
if (!kvm->arch.n_requested_mmu_pages &&
(change == KVM_MR_CREATE || change == KVM_MR_DELETE)) {
unsigned long nr_mmu_pages;
--
2.41.0.487.g6d72f3e995-goog
Disable the page-track notifier code at compile time if there are no
external users, i.e. if CONFIG_KVM_EXTERNAL_WRITE_TRACKING=n. KVM itself
now hooks emulated writes directly instead of relying on the page-track
mechanism.
Provide a stub for "struct kvm_page_track_notifier_node" so that including
headers directly from the command line, e.g. for testing include guards,
doesn't fail due to a struct having an incomplete type.
Reviewed-by: Yan Zhao <[email protected]>
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
arch/x86/include/asm/kvm_host.h | 2 ++
arch/x86/include/asm/kvm_page_track.h | 22 +++++++++++++-------
arch/x86/kvm/mmu/page_track.c | 10 ++++-----
arch/x86/kvm/mmu/page_track.h | 29 +++++++++++++++++++++++----
4 files changed, 47 insertions(+), 16 deletions(-)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 85605f2497bb..33b1ceb30dd2 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1247,7 +1247,9 @@ struct kvm_arch {
* create an NX huge page (without hanging the guest).
*/
struct list_head possible_nx_huge_pages;
+#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
struct kvm_page_track_notifier_head track_notifier_head;
+#endif
/*
* Protects marking pages unsync during page faults, as TDP MMU page
* faults only take mmu_lock for read. For simplicity, the unsync
diff --git a/arch/x86/include/asm/kvm_page_track.h b/arch/x86/include/asm/kvm_page_track.h
index 76c0070dfe2a..61adb07b5927 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -9,6 +9,14 @@ enum kvm_page_track_mode {
KVM_PAGE_TRACK_MAX,
};
+void kvm_slot_page_track_add_page(struct kvm *kvm,
+ struct kvm_memory_slot *slot, gfn_t gfn,
+ enum kvm_page_track_mode mode);
+void kvm_slot_page_track_remove_page(struct kvm *kvm,
+ struct kvm_memory_slot *slot, gfn_t gfn,
+ enum kvm_page_track_mode mode);
+
+#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
/*
* The notifier represented by @kvm_page_track_notifier_node is linked into
* the head which will be notified when guest is triggering the track event.
@@ -48,18 +56,18 @@ struct kvm_page_track_notifier_node {
struct kvm_page_track_notifier_node *node);
};
-void kvm_slot_page_track_add_page(struct kvm *kvm,
- struct kvm_memory_slot *slot, gfn_t gfn,
- enum kvm_page_track_mode mode);
-void kvm_slot_page_track_remove_page(struct kvm *kvm,
- struct kvm_memory_slot *slot, gfn_t gfn,
- enum kvm_page_track_mode mode);
-
void
kvm_page_track_register_notifier(struct kvm *kvm,
struct kvm_page_track_notifier_node *n);
void
kvm_page_track_unregister_notifier(struct kvm *kvm,
struct kvm_page_track_notifier_node *n);
+#else
+/*
+ * Allow defining a node in a structure even if page tracking is disabled, e.g.
+ * to play nice with testing headers via direct inclusion from the command line.
+ */
+struct kvm_page_track_notifier_node {};
+#endif /* CONFIG_KVM_EXTERNAL_WRITE_TRACKING */
#endif
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index e15329d48f95..b20aad7ac3fe 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -194,6 +194,7 @@ bool kvm_slot_page_track_is_active(struct kvm *kvm,
return !!READ_ONCE(slot->arch.gfn_track[mode][index]);
}
+#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
void kvm_page_track_cleanup(struct kvm *kvm)
{
struct kvm_page_track_notifier_head *head;
@@ -255,14 +256,13 @@ EXPORT_SYMBOL_GPL(kvm_page_track_unregister_notifier);
* The node should figure out if the written page is the one that node is
* interested in by itself.
*/
-void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
- int bytes)
+void __kvm_page_track_write(struct kvm *kvm, gpa_t gpa, const u8 *new, int bytes)
{
struct kvm_page_track_notifier_head *head;
struct kvm_page_track_notifier_node *n;
int idx;
- head = &vcpu->kvm->arch.track_notifier_head;
+ head = &kvm->arch.track_notifier_head;
if (hlist_empty(&head->track_notifier_list))
return;
@@ -273,8 +273,6 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
if (n->track_write)
n->track_write(gpa, new, bytes, n);
srcu_read_unlock(&head->track_srcu, idx);
-
- kvm_mmu_track_write(vcpu, gpa, new, bytes);
}
/*
@@ -299,3 +297,5 @@ void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot)
n->track_remove_region(slot->base_gfn, slot->npages, n);
srcu_read_unlock(&head->track_srcu, idx);
}
+
+#endif
diff --git a/arch/x86/kvm/mmu/page_track.h b/arch/x86/kvm/mmu/page_track.h
index 89712f123ad3..931b26b8fc8f 100644
--- a/arch/x86/kvm/mmu/page_track.h
+++ b/arch/x86/kvm/mmu/page_track.h
@@ -6,8 +6,6 @@
#include <asm/kvm_page_track.h>
-int kvm_page_track_init(struct kvm *kvm);
-void kvm_page_track_cleanup(struct kvm *kvm);
bool kvm_page_track_write_tracking_enabled(struct kvm *kvm);
int kvm_page_track_write_tracking_alloc(struct kvm_memory_slot *slot);
@@ -21,13 +19,36 @@ bool kvm_slot_page_track_is_active(struct kvm *kvm,
const struct kvm_memory_slot *slot,
gfn_t gfn, enum kvm_page_track_mode mode);
-void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
- int bytes);
+#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
+int kvm_page_track_init(struct kvm *kvm);
+void kvm_page_track_cleanup(struct kvm *kvm);
+
+void __kvm_page_track_write(struct kvm *kvm, gpa_t gpa, const u8 *new, int bytes);
void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
static inline bool kvm_page_track_has_external_user(struct kvm *kvm)
{
return hlist_empty(&kvm->arch.track_notifier_head.track_notifier_list);
}
+#else
+static inline int kvm_page_track_init(struct kvm *kvm) { return 0; }
+static inline void kvm_page_track_cleanup(struct kvm *kvm) { }
+
+static inline void __kvm_page_track_write(struct kvm *kvm, gpa_t gpa,
+ const u8 *new, int bytes) { }
+static inline void kvm_page_track_delete_slot(struct kvm *kvm,
+ struct kvm_memory_slot *slot) { }
+
+static inline bool kvm_page_track_has_external_user(struct kvm *kvm) { return false; }
+
+#endif /* CONFIG_KVM_EXTERNAL_WRITE_TRACKING */
+
+static inline void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
+ const u8 *new, int bytes)
+{
+ __kvm_page_track_write(vcpu->kvm, gpa, new, bytes);
+
+ kvm_mmu_track_write(vcpu, gpa, new, bytes);
+}
#endif /* __KVM_X86_PAGE_TRACK_H */
--
2.41.0.487.g6d72f3e995-goog
Put the struct page reference acquired by gfn_to_pfn(), KVM's API is that
the caller is ultimately responsible for dropping any reference.
Note, kvm_release_pfn_clean() ensures the pfn is actually a refcounted
struct page before trying to put any references.
Fixes: b901b252b6cf ("drm/i915/gvt: Add 2M huge gtt support")
Reviewed-by: Yan Zhao <[email protected]>
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
drivers/gpu/drm/i915/gvt/gtt.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c
index f30922c55a0c..5426a27c1b71 100644
--- a/drivers/gpu/drm/i915/gvt/gtt.c
+++ b/drivers/gpu/drm/i915/gvt/gtt.c
@@ -1158,6 +1158,7 @@ static int is_2MB_gtt_possible(struct intel_vgpu *vgpu,
{
const struct intel_gvt_gtt_pte_ops *ops = vgpu->gvt->gtt.pte_ops;
kvm_pfn_t pfn;
+ int ret;
if (!HAS_PAGE_SIZES(vgpu->gvt->gt->i915, I915_GTT_PAGE_SIZE_2M))
return 0;
@@ -1171,7 +1172,9 @@ static int is_2MB_gtt_possible(struct intel_vgpu *vgpu,
if (!pfn_valid(pfn))
return -EINVAL;
- return PageTransHuge(pfn_to_page(pfn));
+ ret = PageTransHuge(pfn_to_page(pfn));
+ kvm_release_pfn_clean(pfn);
+ return ret;
}
static int split_2MB_gtt_entry(struct intel_vgpu *vgpu,
--
2.41.0.487.g6d72f3e995-goog
When adding/removing gfns to/from write-tracking, assert that mmu_lock
is held for write, and that either slots_lock or kvm->srcu is held.
mmu_lock must be held for write to protect gfn_write_track's refcount,
and SRCU or slots_lock must be held to protect the memslot itself.
Tested-by: Yan Zhao <[email protected]>
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
arch/x86/kvm/mmu/page_track.c | 17 +++++++++++------
1 file changed, 11 insertions(+), 6 deletions(-)
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index b835ba7f325c..29ae61f1e303 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -12,6 +12,7 @@
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include <linux/lockdep.h>
#include <linux/kvm_host.h>
#include <linux/rculist.h>
@@ -77,9 +78,6 @@ static void update_gfn_write_track(struct kvm_memory_slot *slot, gfn_t gfn,
* add guest page to the tracking pool so that corresponding access on that
* page will be intercepted.
*
- * It should be called under the protection both of mmu-lock and kvm->srcu
- * or kvm->slots_lock.
- *
* @kvm: the guest instance we are interested in.
* @slot: the @gfn belongs to.
* @gfn: the guest page.
@@ -87,6 +85,11 @@ static void update_gfn_write_track(struct kvm_memory_slot *slot, gfn_t gfn,
void kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
gfn_t gfn)
{
+ lockdep_assert_held_write(&kvm->mmu_lock);
+
+ lockdep_assert_once(lockdep_is_held(&kvm->slots_lock) ||
+ srcu_read_lock_held(&kvm->srcu));
+
if (WARN_ON(!kvm_page_track_write_tracking_enabled(kvm)))
return;
@@ -107,9 +110,6 @@ EXPORT_SYMBOL_GPL(kvm_write_track_add_gfn);
* remove the guest page from the tracking pool which stops the interception
* of corresponding access on that page.
*
- * It should be called under the protection both of mmu-lock and kvm->srcu
- * or kvm->slots_lock.
- *
* @kvm: the guest instance we are interested in.
* @slot: the @gfn belongs to.
* @gfn: the guest page.
@@ -117,6 +117,11 @@ EXPORT_SYMBOL_GPL(kvm_write_track_add_gfn);
void kvm_write_track_remove_gfn(struct kvm *kvm,
struct kvm_memory_slot *slot, gfn_t gfn)
{
+ lockdep_assert_held_write(&kvm->mmu_lock);
+
+ lockdep_assert_once(lockdep_is_held(&kvm->slots_lock) ||
+ srcu_read_lock_held(&kvm->srcu));
+
if (WARN_ON(!kvm_page_track_write_tracking_enabled(kvm)))
return;
--
2.41.0.487.g6d72f3e995-goog
Check that the pfn found by gfn_to_pfn() is actually backed by "struct
page" memory prior to retrieving and dereferencing the page. KVM
supports backing guest memory with VM_PFNMAP, VM_IO, etc., and so
there is no guarantee the pfn returned by gfn_to_pfn() has an associated
"struct page".
Fixes: b901b252b6cf ("drm/i915/gvt: Add 2M huge gtt support")
Reviewed-by: Yan Zhao <[email protected]>
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
drivers/gpu/drm/i915/gvt/gtt.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c
index 4ec85308379a..58b9b316ae46 100644
--- a/drivers/gpu/drm/i915/gvt/gtt.c
+++ b/drivers/gpu/drm/i915/gvt/gtt.c
@@ -1183,6 +1183,10 @@ static int is_2MB_gtt_possible(struct intel_vgpu *vgpu,
pfn = gfn_to_pfn(vgpu->vfio_device.kvm, ops->get_pfn(entry));
if (is_error_noslot_pfn(pfn))
return -EINVAL;
+
+ if (!pfn_valid(pfn))
+ return -EINVAL;
+
return PageTransHuge(pfn_to_page(pfn));
}
--
2.41.0.487.g6d72f3e995-goog
Refactor KVM's exported/external page-track, a.k.a. write-track, APIs
to take only the gfn and do the required memslot lookup in KVM proper.
Forcing users of the APIs to get the memslot unnecessarily bleeds
KVM internals into KVMGT and complicates usage of the APIs.
No functional change intended.
Reviewed-by: Yan Zhao <[email protected]>
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
arch/x86/include/asm/kvm_page_track.h | 7 +--
arch/x86/kvm/mmu/mmu.c | 4 +-
arch/x86/kvm/mmu/page_track.c | 85 ++++++++++++++++++++-------
arch/x86/kvm/mmu/page_track.h | 5 ++
drivers/gpu/drm/i915/gvt/kvmgt.c | 37 +++---------
5 files changed, 80 insertions(+), 58 deletions(-)
diff --git a/arch/x86/include/asm/kvm_page_track.h b/arch/x86/include/asm/kvm_page_track.h
index f5c1db36cdb7..4afab697e21c 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -4,11 +4,6 @@
#include <linux/kvm_types.h>
-void kvm_write_track_add_gfn(struct kvm *kvm,
- struct kvm_memory_slot *slot, gfn_t gfn);
-void kvm_write_track_remove_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
- gfn_t gfn);
-
#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
/*
* The notifier represented by @kvm_page_track_notifier_node is linked into
@@ -55,6 +50,8 @@ kvm_page_track_register_notifier(struct kvm *kvm,
void
kvm_page_track_unregister_notifier(struct kvm *kvm,
struct kvm_page_track_notifier_node *n);
+int kvm_write_track_add_gfn(struct kvm *kvm, gfn_t gfn);
+int kvm_write_track_remove_gfn(struct kvm *kvm, gfn_t gfn);
#else
/*
* Allow defining a node in a structure even if page tracking is disabled, e.g.
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index a0309fde3549..c6ae1885371c 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -840,7 +840,7 @@ static void account_shadowed(struct kvm *kvm, struct kvm_mmu_page *sp)
/* the non-leaf shadow pages are keeping readonly. */
if (sp->role.level > PG_LEVEL_4K)
- return kvm_write_track_add_gfn(kvm, slot, gfn);
+ return __kvm_write_track_add_gfn(kvm, slot, gfn);
kvm_mmu_gfn_disallow_lpage(slot, gfn);
@@ -886,7 +886,7 @@ static void unaccount_shadowed(struct kvm *kvm, struct kvm_mmu_page *sp)
slots = kvm_memslots_for_spte_role(kvm, sp->role);
slot = __gfn_to_memslot(slots, gfn);
if (sp->role.level > PG_LEVEL_4K)
- return kvm_write_track_remove_gfn(kvm, slot, gfn);
+ return __kvm_write_track_remove_gfn(kvm, slot, gfn);
kvm_mmu_gfn_allow_lpage(slot, gfn);
}
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index eedb5889d73e..2a64df38ccab 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -74,16 +74,8 @@ static void update_gfn_write_track(struct kvm_memory_slot *slot, gfn_t gfn,
slot->arch.gfn_write_track[index] += count;
}
-/*
- * add guest page to the tracking pool so that corresponding access on that
- * page will be intercepted.
- *
- * @kvm: the guest instance we are interested in.
- * @slot: the @gfn belongs to.
- * @gfn: the guest page.
- */
-void kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
- gfn_t gfn)
+void __kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
+ gfn_t gfn)
{
lockdep_assert_held_write(&kvm->mmu_lock);
@@ -104,18 +96,9 @@ void kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
if (kvm_mmu_slot_gfn_write_protect(kvm, slot, gfn, PG_LEVEL_4K))
kvm_flush_remote_tlbs(kvm);
}
-EXPORT_SYMBOL_GPL(kvm_write_track_add_gfn);
-/*
- * remove the guest page from the tracking pool which stops the interception
- * of corresponding access on that page.
- *
- * @kvm: the guest instance we are interested in.
- * @slot: the @gfn belongs to.
- * @gfn: the guest page.
- */
-void kvm_write_track_remove_gfn(struct kvm *kvm,
- struct kvm_memory_slot *slot, gfn_t gfn)
+void __kvm_write_track_remove_gfn(struct kvm *kvm,
+ struct kvm_memory_slot *slot, gfn_t gfn)
{
lockdep_assert_held_write(&kvm->mmu_lock);
@@ -133,7 +116,6 @@ void kvm_write_track_remove_gfn(struct kvm *kvm,
*/
kvm_mmu_gfn_allow_lpage(slot, gfn);
}
-EXPORT_SYMBOL_GPL(kvm_write_track_remove_gfn);
/*
* check if the corresponding access on the specified guest page is tracked.
@@ -257,4 +239,63 @@ void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot)
srcu_read_unlock(&head->track_srcu, idx);
}
+/*
+ * add guest page to the tracking pool so that corresponding access on that
+ * page will be intercepted.
+ *
+ * @kvm: the guest instance we are interested in.
+ * @gfn: the guest page.
+ */
+int kvm_write_track_add_gfn(struct kvm *kvm, gfn_t gfn)
+{
+ struct kvm_memory_slot *slot;
+ int idx;
+
+ idx = srcu_read_lock(&kvm->srcu);
+
+ slot = gfn_to_memslot(kvm, gfn);
+ if (!slot) {
+ srcu_read_unlock(&kvm->srcu, idx);
+ return -EINVAL;
+ }
+
+ write_lock(&kvm->mmu_lock);
+ __kvm_write_track_add_gfn(kvm, slot, gfn);
+ write_unlock(&kvm->mmu_lock);
+
+ srcu_read_unlock(&kvm->srcu, idx);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(kvm_write_track_add_gfn);
+
+/*
+ * remove the guest page from the tracking pool which stops the interception
+ * of corresponding access on that page.
+ *
+ * @kvm: the guest instance we are interested in.
+ * @gfn: the guest page.
+ */
+int kvm_write_track_remove_gfn(struct kvm *kvm, gfn_t gfn)
+{
+ struct kvm_memory_slot *slot;
+ int idx;
+
+ idx = srcu_read_lock(&kvm->srcu);
+
+ slot = gfn_to_memslot(kvm, gfn);
+ if (!slot) {
+ srcu_read_unlock(&kvm->srcu, idx);
+ return -EINVAL;
+ }
+
+ write_lock(&kvm->mmu_lock);
+ __kvm_write_track_remove_gfn(kvm, slot, gfn);
+ write_unlock(&kvm->mmu_lock);
+
+ srcu_read_unlock(&kvm->srcu, idx);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(kvm_write_track_remove_gfn);
#endif
diff --git a/arch/x86/kvm/mmu/page_track.h b/arch/x86/kvm/mmu/page_track.h
index 50d3278e8c69..62f98c6c5af3 100644
--- a/arch/x86/kvm/mmu/page_track.h
+++ b/arch/x86/kvm/mmu/page_track.h
@@ -15,6 +15,11 @@ int kvm_page_track_create_memslot(struct kvm *kvm,
struct kvm_memory_slot *slot,
unsigned long npages);
+void __kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot,
+ gfn_t gfn);
+void __kvm_write_track_remove_gfn(struct kvm *kvm,
+ struct kvm_memory_slot *slot, gfn_t gfn);
+
bool kvm_gfn_is_write_tracked(struct kvm *kvm,
const struct kvm_memory_slot *slot, gfn_t gfn);
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 05a7e614ead0..21342a93e418 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -1546,9 +1546,7 @@ static struct mdev_driver intel_vgpu_mdev_driver = {
int intel_gvt_page_track_add(struct intel_vgpu *info, u64 gfn)
{
- struct kvm *kvm = info->vfio_device.kvm;
- struct kvm_memory_slot *slot;
- int idx;
+ int r;
if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, info->status))
return -ESRCH;
@@ -1556,18 +1554,9 @@ int intel_gvt_page_track_add(struct intel_vgpu *info, u64 gfn)
if (kvmgt_gfn_is_write_protected(info, gfn))
return 0;
- idx = srcu_read_lock(&kvm->srcu);
- slot = gfn_to_memslot(kvm, gfn);
- if (!slot) {
- srcu_read_unlock(&kvm->srcu, idx);
- return -EINVAL;
- }
-
- write_lock(&kvm->mmu_lock);
- kvm_write_track_add_gfn(kvm, slot, gfn);
- write_unlock(&kvm->mmu_lock);
-
- srcu_read_unlock(&kvm->srcu, idx);
+ r = kvm_write_track_add_gfn(info->vfio_device.kvm, gfn);
+ if (r)
+ return r;
kvmgt_protect_table_add(info, gfn);
return 0;
@@ -1575,9 +1564,7 @@ int intel_gvt_page_track_add(struct intel_vgpu *info, u64 gfn)
int intel_gvt_page_track_remove(struct intel_vgpu *info, u64 gfn)
{
- struct kvm *kvm = info->vfio_device.kvm;
- struct kvm_memory_slot *slot;
- int idx;
+ int r;
if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, info->status))
return -ESRCH;
@@ -1585,17 +1572,9 @@ int intel_gvt_page_track_remove(struct intel_vgpu *info, u64 gfn)
if (!kvmgt_gfn_is_write_protected(info, gfn))
return 0;
- idx = srcu_read_lock(&kvm->srcu);
- slot = gfn_to_memslot(kvm, gfn);
- if (!slot) {
- srcu_read_unlock(&kvm->srcu, idx);
- return -EINVAL;
- }
-
- write_lock(&kvm->mmu_lock);
- kvm_write_track_remove_gfn(kvm, slot, gfn);
- write_unlock(&kvm->mmu_lock);
- srcu_read_unlock(&kvm->srcu, idx);
+ r = kvm_write_track_remove_gfn(info->vfio_device.kvm, gfn);
+ if (r)
+ return r;
kvmgt_protect_table_del(info, gfn);
return 0;
--
2.41.0.487.g6d72f3e995-goog
When shadowing a GTT entry with a 2M page, verify that the pfns are
contiguous, not just that the struct page pointers are contiguous. The
memory map is virtual contiguous if "CONFIG_FLATMEM=y ||
CONFIG_SPARSEMEM_VMEMMAP=y", but not for "CONFIG_SPARSEMEM=y &&
CONFIG_SPARSEMEM_VMEMMAP=n", so theoretically KVMGT could encounter struct
pages that are virtually contiguous, but not physically contiguous.
In practice, this flaw is likely a non-issue as it would cause functional
problems iff a section isn't 2M aligned _and_ is directly adjacent to
another section with discontiguous pfns.
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
drivers/gpu/drm/i915/gvt/kvmgt.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index de675d799c7d..429f0f993a13 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -161,7 +161,7 @@ static int gvt_pin_guest_page(struct intel_vgpu *vgpu, unsigned long gfn,
if (npage == 0)
base_page = cur_page;
- else if (base_page + npage != cur_page) {
+ else if (page_to_pfn(base_page) + npage != page_to_pfn(cur_page)) {
gvt_vgpu_err("The pages are not continuous\n");
ret = -EINVAL;
npage++;
--
2.41.0.487.g6d72f3e995-goog
Move x86's implementation of kvm_arch_flush_shadow_{all,memslot}() into
mmu.c, and make kvm_mmu_zap_all() static as it was globally visible only
for kvm_arch_flush_shadow_all(). This will allow refactoring
kvm_arch_flush_shadow_memslot() to call kvm_mmu_zap_all() directly without
having to expose kvm_mmu_zap_all_fast() outside of mmu.c. Keeping
everything in mmu.c will also likely simplify supporting TDX, which
intends to do zap only relevant SPTEs on memslot updates.
No functional change intended.
Suggested-by: Yan Zhao <[email protected]>
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
arch/x86/include/asm/kvm_host.h | 1 -
arch/x86/kvm/mmu/mmu.c | 13 ++++++++++++-
arch/x86/kvm/x86.c | 11 -----------
3 files changed, 12 insertions(+), 13 deletions(-)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 28bd38303d70..856ec22aceb6 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1832,7 +1832,6 @@ void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm,
const struct kvm_memory_slot *memslot);
void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm,
const struct kvm_memory_slot *memslot);
-void kvm_mmu_zap_all(struct kvm *kvm);
void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen);
void kvm_mmu_change_mmu_pages(struct kvm *kvm, unsigned long kvm_nr_mmu_pages);
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index ec169f5c7dce..c6dee659d592 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -6732,7 +6732,7 @@ void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm,
*/
}
-void kvm_mmu_zap_all(struct kvm *kvm)
+static void kvm_mmu_zap_all(struct kvm *kvm)
{
struct kvm_mmu_page *sp, *node;
LIST_HEAD(invalid_list);
@@ -6757,6 +6757,17 @@ void kvm_mmu_zap_all(struct kvm *kvm)
write_unlock(&kvm->mmu_lock);
}
+void kvm_arch_flush_shadow_all(struct kvm *kvm)
+{
+ kvm_mmu_zap_all(kvm);
+}
+
+void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
+ struct kvm_memory_slot *slot)
+{
+ kvm_page_track_flush_slot(kvm, slot);
+}
+
void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen)
{
WARN_ON(gen & KVM_MEMSLOT_GEN_UPDATE_IN_PROGRESS);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index a6b9bea62fb8..059571d5abed 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -12776,17 +12776,6 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
kvm_arch_free_memslot(kvm, old);
}
-void kvm_arch_flush_shadow_all(struct kvm *kvm)
-{
- kvm_mmu_zap_all(kvm);
-}
-
-void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
- struct kvm_memory_slot *slot)
-{
- kvm_page_track_flush_slot(kvm, slot);
-}
-
static inline bool kvm_guest_apic_has_interrupt(struct kvm_vcpu *vcpu)
{
return (is_guest_mode(vcpu) &&
--
2.41.0.487.g6d72f3e995-goog
Drop @vcpu from KVM's ->track_write() hook provided for external users of
the page-track APIs now that KVM itself doesn't use the page-track
mechanism.
Reviewed-by: Yan Zhao <[email protected]>
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
arch/x86/include/asm/kvm_page_track.h | 5 ++---
arch/x86/kvm/mmu/page_track.c | 2 +-
drivers/gpu/drm/i915/gvt/kvmgt.c | 10 ++++------
3 files changed, 7 insertions(+), 10 deletions(-)
diff --git a/arch/x86/include/asm/kvm_page_track.h b/arch/x86/include/asm/kvm_page_track.h
index eb186bc57f6a..8c4d216e3b2b 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -26,14 +26,13 @@ struct kvm_page_track_notifier_node {
* It is called when guest is writing the write-tracked page
* and write emulation is finished at that time.
*
- * @vcpu: the vcpu where the write access happened.
* @gpa: the physical address written by guest.
* @new: the data was written to the address.
* @bytes: the written length.
* @node: this node
*/
- void (*track_write)(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
- int bytes, struct kvm_page_track_notifier_node *node);
+ void (*track_write)(gpa_t gpa, const u8 *new, int bytes,
+ struct kvm_page_track_notifier_node *node);
/*
* It is called when memory slot is being moved or removed
* users can drop write-protection for the pages in that memory slot
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index 23088c90d2fd..891e5cc52b45 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -272,7 +272,7 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
hlist_for_each_entry_srcu(n, &head->track_notifier_list, node,
srcu_read_lock_held(&head->track_srcu))
if (n->track_write)
- n->track_write(vcpu, gpa, new, bytes, n);
+ n->track_write(gpa, new, bytes, n);
srcu_read_unlock(&head->track_srcu, idx);
kvm_mmu_track_write(vcpu, gpa, new, bytes);
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 034be0655daa..e9276500435d 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -106,9 +106,8 @@ struct gvt_dma {
#define vfio_dev_to_vgpu(vfio_dev) \
container_of((vfio_dev), struct intel_vgpu, vfio_device)
-static void kvmgt_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
- const u8 *val, int len,
- struct kvm_page_track_notifier_node *node);
+static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len,
+ struct kvm_page_track_notifier_node *node);
static void kvmgt_page_track_flush_slot(struct kvm *kvm,
struct kvm_memory_slot *slot,
struct kvm_page_track_notifier_node *node);
@@ -1603,9 +1602,8 @@ int intel_gvt_page_track_remove(struct intel_vgpu *info, u64 gfn)
return 0;
}
-static void kvmgt_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
- const u8 *val, int len,
- struct kvm_page_track_notifier_node *node)
+static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len,
+ struct kvm_page_track_notifier_node *node)
{
struct intel_vgpu *info =
container_of(node, struct intel_vgpu, track_node);
--
2.41.0.487.g6d72f3e995-goog
Use vgpu_lock instead of KVM's mmu_lock to protect accesses to the hash
table used to track which gfns are write-protected when shadowing the
guest's GTT, and hoist the acquisition of vgpu_lock from
intel_vgpu_page_track_handler() out to its sole caller,
kvmgt_page_track_write().
This fixes a bug where kvmgt_page_track_write(), which doesn't hold
kvm->mmu_lock, could race with intel_gvt_page_track_remove() and trigger
a use-after-free.
Fixing kvmgt_page_track_write() by taking kvm->mmu_lock is not an option
as mmu_lock is a r/w spinlock, and intel_vgpu_page_track_handler() might
sleep when acquiring vgpu->cache_lock deep down the callstack:
intel_vgpu_page_track_handler()
|
|-> page_track->handler / ppgtt_write_protection_handler()
|
|-> ppgtt_handle_guest_write_page_table_bytes()
|
|-> ppgtt_handle_guest_write_page_table()
|
|-> ppgtt_handle_guest_entry_removal()
|
|-> ppgtt_invalidate_pte()
|
|-> intel_gvt_dma_unmap_guest_page()
|
|-> mutex_lock(&vgpu->cache_lock);
Reviewed-by: Yan Zhao <[email protected]>
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
drivers/gpu/drm/i915/gvt/kvmgt.c | 55 +++++++++++++++------------
drivers/gpu/drm/i915/gvt/page_track.c | 10 +----
2 files changed, 33 insertions(+), 32 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 6f52886c4051..034be0655daa 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -352,6 +352,8 @@ __kvmgt_protect_table_find(struct intel_vgpu *info, gfn_t gfn)
{
struct kvmgt_pgfn *p, *res = NULL;
+ lockdep_assert_held(&info->vgpu_lock);
+
hash_for_each_possible(info->ptable, p, hnode, gfn) {
if (gfn == p->gfn) {
res = p;
@@ -1553,6 +1555,9 @@ int intel_gvt_page_track_add(struct intel_vgpu *info, u64 gfn)
if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, info->status))
return -ESRCH;
+ if (kvmgt_gfn_is_write_protected(info, gfn))
+ return 0;
+
idx = srcu_read_lock(&kvm->srcu);
slot = gfn_to_memslot(kvm, gfn);
if (!slot) {
@@ -1561,16 +1566,12 @@ int intel_gvt_page_track_add(struct intel_vgpu *info, u64 gfn)
}
write_lock(&kvm->mmu_lock);
-
- if (kvmgt_gfn_is_write_protected(info, gfn))
- goto out;
-
kvm_slot_page_track_add_page(kvm, slot, gfn, KVM_PAGE_TRACK_WRITE);
+ write_unlock(&kvm->mmu_lock);
+
+ srcu_read_unlock(&kvm->srcu, idx);
+
kvmgt_protect_table_add(info, gfn);
-
-out:
- write_unlock(&kvm->mmu_lock);
- srcu_read_unlock(&kvm->srcu, idx);
return 0;
}
@@ -1583,24 +1584,22 @@ int intel_gvt_page_track_remove(struct intel_vgpu *info, u64 gfn)
if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, info->status))
return -ESRCH;
- idx = srcu_read_lock(&kvm->srcu);
- slot = gfn_to_memslot(kvm, gfn);
- if (!slot) {
- srcu_read_unlock(&kvm->srcu, idx);
- return -EINVAL;
- }
-
- write_lock(&kvm->mmu_lock);
-
if (!kvmgt_gfn_is_write_protected(info, gfn))
- goto out;
+ return 0;
+ idx = srcu_read_lock(&kvm->srcu);
+ slot = gfn_to_memslot(kvm, gfn);
+ if (!slot) {
+ srcu_read_unlock(&kvm->srcu, idx);
+ return -EINVAL;
+ }
+
+ write_lock(&kvm->mmu_lock);
kvm_slot_page_track_remove_page(kvm, slot, gfn, KVM_PAGE_TRACK_WRITE);
+ write_unlock(&kvm->mmu_lock);
+ srcu_read_unlock(&kvm->srcu, idx);
+
kvmgt_protect_table_del(info, gfn);
-
-out:
- write_unlock(&kvm->mmu_lock);
- srcu_read_unlock(&kvm->srcu, idx);
return 0;
}
@@ -1611,9 +1610,13 @@ static void kvmgt_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
struct intel_vgpu *info =
container_of(node, struct intel_vgpu, track_node);
+ mutex_lock(&info->vgpu_lock);
+
if (kvmgt_gfn_is_write_protected(info, gpa_to_gfn(gpa)))
intel_vgpu_page_track_handler(info, gpa,
(void *)val, len);
+
+ mutex_unlock(&info->vgpu_lock);
}
static void kvmgt_page_track_flush_slot(struct kvm *kvm,
@@ -1625,16 +1628,20 @@ static void kvmgt_page_track_flush_slot(struct kvm *kvm,
struct intel_vgpu *info =
container_of(node, struct intel_vgpu, track_node);
- write_lock(&kvm->mmu_lock);
+ mutex_lock(&info->vgpu_lock);
+
for (i = 0; i < slot->npages; i++) {
gfn = slot->base_gfn + i;
if (kvmgt_gfn_is_write_protected(info, gfn)) {
+ write_lock(&kvm->mmu_lock);
kvm_slot_page_track_remove_page(kvm, slot, gfn,
KVM_PAGE_TRACK_WRITE);
+ write_unlock(&kvm->mmu_lock);
+
kvmgt_protect_table_del(info, gfn);
}
}
- write_unlock(&kvm->mmu_lock);
+ mutex_unlock(&info->vgpu_lock);
}
void intel_vgpu_detach_regions(struct intel_vgpu *vgpu)
diff --git a/drivers/gpu/drm/i915/gvt/page_track.c b/drivers/gpu/drm/i915/gvt/page_track.c
index df34e73cba41..60a65435556d 100644
--- a/drivers/gpu/drm/i915/gvt/page_track.c
+++ b/drivers/gpu/drm/i915/gvt/page_track.c
@@ -162,13 +162,9 @@ int intel_vgpu_page_track_handler(struct intel_vgpu *vgpu, u64 gpa,
struct intel_vgpu_page_track *page_track;
int ret = 0;
- mutex_lock(&vgpu->vgpu_lock);
-
page_track = intel_vgpu_find_page_track(vgpu, gpa >> PAGE_SHIFT);
- if (!page_track) {
- ret = -ENXIO;
- goto out;
- }
+ if (!page_track)
+ return -ENXIO;
if (unlikely(vgpu->failsafe)) {
/* Remove write protection to prevent furture traps. */
@@ -179,7 +175,5 @@ int intel_vgpu_page_track_handler(struct intel_vgpu *vgpu, u64 gpa,
gvt_err("guest page write error, gpa %llx\n", gpa);
}
-out:
- mutex_unlock(&vgpu->vgpu_lock);
return ret;
}
--
2.41.0.487.g6d72f3e995-goog
Open code gpa_to_gfn() in kvmgt_page_track_write() and drop KVMGT's
dependency on kvm_host.h, i.e. include only kvm_page_track.h. KVMGT
assumes "gfn == gpa >> PAGE_SHIFT" all over the place, including a few
lines below in the same function with the same gpa, i.e. there's no
reason to use KVM's helper for this one case.
No functional change intended.
Reviewed-by: Yan Zhao <[email protected]>
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
drivers/gpu/drm/i915/gvt/gvt.h | 3 ++-
drivers/gpu/drm/i915/gvt/kvmgt.c | 2 +-
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/gvt.h b/drivers/gpu/drm/i915/gvt/gvt.h
index 2d65800d8e93..53a0a42a50db 100644
--- a/drivers/gpu/drm/i915/gvt/gvt.h
+++ b/drivers/gpu/drm/i915/gvt/gvt.h
@@ -34,10 +34,11 @@
#define _GVT_H_
#include <uapi/linux/pci_regs.h>
-#include <linux/kvm_host.h>
#include <linux/vfio.h>
#include <linux/mdev.h>
+#include <asm/kvm_page_track.h>
+
#include "i915_drv.h"
#include "intel_gvt.h"
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index eb50997dd369..aaed3969f204 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -1585,7 +1585,7 @@ static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len,
mutex_lock(&info->vgpu_lock);
- if (kvmgt_gfn_is_write_protected(info, gpa_to_gfn(gpa)))
+ if (kvmgt_gfn_is_write_protected(info, gpa >> PAGE_SHIFT))
intel_vgpu_page_track_handler(info, gpa,
(void *)val, len);
--
2.41.0.487.g6d72f3e995-goog
When handling a slot "flush", don't call back into KVM to drop write
protection for gfns in the slot. Now that KVM rejects attempts to move
memory slots while KVMGT is attached, the only time a slot is "flushed"
is when it's being removed, i.e. the memslot and all its write-tracking
metadata is about to be deleted.
Reviewed-by: Yan Zhao <[email protected]>
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
drivers/gpu/drm/i915/gvt/kvmgt.c | 8 +-------
1 file changed, 1 insertion(+), 7 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index e9276500435d..3ea3cb9eb599 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -1630,14 +1630,8 @@ static void kvmgt_page_track_flush_slot(struct kvm *kvm,
for (i = 0; i < slot->npages; i++) {
gfn = slot->base_gfn + i;
- if (kvmgt_gfn_is_write_protected(info, gfn)) {
- write_lock(&kvm->mmu_lock);
- kvm_slot_page_track_remove_page(kvm, slot, gfn,
- KVM_PAGE_TRACK_WRITE);
- write_unlock(&kvm->mmu_lock);
-
+ if (kvmgt_gfn_is_write_protected(info, gfn))
kvmgt_protect_table_del(info, gfn);
- }
}
mutex_unlock(&info->vgpu_lock);
}
--
2.41.0.487.g6d72f3e995-goog
Bury the declaration of the page-track helpers that are intended only for
internal KVM use in a "private" header. In addition to guarding against
unwanted usage of the internal-only helpers, dropping their definitions
avoids exposing other structures that should be KVM-internal, e.g. for
memslots. This is a baby step toward making kvm_host.h a KVM-internal
header in the very distant future.
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
arch/x86/include/asm/kvm_page_track.h | 21 ++---------------
arch/x86/kvm/mmu/mmu.c | 3 ++-
arch/x86/kvm/mmu/page_track.c | 8 +------
arch/x86/kvm/mmu/page_track.h | 33 +++++++++++++++++++++++++++
arch/x86/kvm/x86.c | 1 +
5 files changed, 39 insertions(+), 27 deletions(-)
create mode 100644 arch/x86/kvm/mmu/page_track.h
diff --git a/arch/x86/include/asm/kvm_page_track.h b/arch/x86/include/asm/kvm_page_track.h
index 5c348ffdc194..76c0070dfe2a 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -2,6 +2,8 @@
#ifndef _ASM_X86_KVM_PAGE_TRACK_H
#define _ASM_X86_KVM_PAGE_TRACK_H
+#include <linux/kvm_types.h>
+
enum kvm_page_track_mode {
KVM_PAGE_TRACK_WRITE,
KVM_PAGE_TRACK_MAX,
@@ -46,26 +48,12 @@ struct kvm_page_track_notifier_node {
struct kvm_page_track_notifier_node *node);
};
-int kvm_page_track_init(struct kvm *kvm);
-void kvm_page_track_cleanup(struct kvm *kvm);
-
-bool kvm_page_track_write_tracking_enabled(struct kvm *kvm);
-int kvm_page_track_write_tracking_alloc(struct kvm_memory_slot *slot);
-
-void kvm_page_track_free_memslot(struct kvm_memory_slot *slot);
-int kvm_page_track_create_memslot(struct kvm *kvm,
- struct kvm_memory_slot *slot,
- unsigned long npages);
-
void kvm_slot_page_track_add_page(struct kvm *kvm,
struct kvm_memory_slot *slot, gfn_t gfn,
enum kvm_page_track_mode mode);
void kvm_slot_page_track_remove_page(struct kvm *kvm,
struct kvm_memory_slot *slot, gfn_t gfn,
enum kvm_page_track_mode mode);
-bool kvm_slot_page_track_is_active(struct kvm *kvm,
- const struct kvm_memory_slot *slot,
- gfn_t gfn, enum kvm_page_track_mode mode);
void
kvm_page_track_register_notifier(struct kvm *kvm,
@@ -73,10 +61,5 @@ kvm_page_track_register_notifier(struct kvm *kvm,
void
kvm_page_track_unregister_notifier(struct kvm *kvm,
struct kvm_page_track_notifier_node *n);
-void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
- int bytes);
-void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
-
-bool kvm_page_track_has_external_user(struct kvm *kvm);
#endif
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index c1d3ac303964..88923b1eb510 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -25,6 +25,7 @@
#include "kvm_cache_regs.h"
#include "smm.h"
#include "kvm_emulate.h"
+#include "page_track.h"
#include "cpuid.h"
#include "spte.h"
@@ -53,7 +54,7 @@
#include <asm/io.h>
#include <asm/set_memory.h>
#include <asm/vmx.h>
-#include <asm/kvm_page_track.h>
+
#include "trace.h"
extern bool itlb_multihit_kvm_mitigation;
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index 2a6ab7c455c0..e15329d48f95 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -15,10 +15,9 @@
#include <linux/kvm_host.h>
#include <linux/rculist.h>
-#include <asm/kvm_page_track.h>
-
#include "mmu.h"
#include "mmu_internal.h"
+#include "page_track.h"
bool kvm_page_track_write_tracking_enabled(struct kvm *kvm)
{
@@ -300,8 +299,3 @@ void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot)
n->track_remove_region(slot->base_gfn, slot->npages, n);
srcu_read_unlock(&head->track_srcu, idx);
}
-
-bool kvm_page_track_has_external_user(struct kvm *kvm)
-{
- return hlist_empty(&kvm->arch.track_notifier_head.track_notifier_list);
-}
diff --git a/arch/x86/kvm/mmu/page_track.h b/arch/x86/kvm/mmu/page_track.h
new file mode 100644
index 000000000000..89712f123ad3
--- /dev/null
+++ b/arch/x86/kvm/mmu/page_track.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __KVM_X86_PAGE_TRACK_H
+#define __KVM_X86_PAGE_TRACK_H
+
+#include <linux/kvm_host.h>
+
+#include <asm/kvm_page_track.h>
+
+int kvm_page_track_init(struct kvm *kvm);
+void kvm_page_track_cleanup(struct kvm *kvm);
+
+bool kvm_page_track_write_tracking_enabled(struct kvm *kvm);
+int kvm_page_track_write_tracking_alloc(struct kvm_memory_slot *slot);
+
+void kvm_page_track_free_memslot(struct kvm_memory_slot *slot);
+int kvm_page_track_create_memslot(struct kvm *kvm,
+ struct kvm_memory_slot *slot,
+ unsigned long npages);
+
+bool kvm_slot_page_track_is_active(struct kvm *kvm,
+ const struct kvm_memory_slot *slot,
+ gfn_t gfn, enum kvm_page_track_mode mode);
+
+void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
+ int bytes);
+void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
+
+static inline bool kvm_page_track_has_external_user(struct kvm *kvm)
+{
+ return hlist_empty(&kvm->arch.track_notifier_head.track_notifier_list);
+}
+
+#endif /* __KVM_X86_PAGE_TRACK_H */
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index e9ecdf5fbdac..05a68d7d99fe 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -25,6 +25,7 @@
#include "tss.h"
#include "kvm_cache_regs.h"
#include "kvm_emulate.h"
+#include "mmu/page_track.h"
#include "x86.h"
#include "cpuid.h"
#include "pmu.h"
--
2.41.0.487.g6d72f3e995-goog
Use an "unsigned long" instead of an "int" when iterating over the gfns
in a memslot. The number of pages in the memslot is tracked as an
"unsigned long", e.g. KVMGT could theoretically break if a KVM memslot
larger than 16TiB were deleted (2^32 * 4KiB).
Reviewed-by: Yan Zhao <[email protected]>
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
drivers/gpu/drm/i915/gvt/kvmgt.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 97c6d3c53710..6f52886c4051 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -1620,7 +1620,7 @@ static void kvmgt_page_track_flush_slot(struct kvm *kvm,
struct kvm_memory_slot *slot,
struct kvm_page_track_notifier_node *node)
{
- int i;
+ unsigned long i;
gfn_t gfn;
struct intel_vgpu *info =
container_of(node, struct intel_vgpu, track_node);
--
2.41.0.487.g6d72f3e995-goog
From: Yan Zhao <[email protected]>
Switch from the poorly named and flawed ->track_flush_slot() to the newly
introduced ->track_remove_region(). From KVMGT's perspective, the two
hooks are functionally equivalent, the only difference being that
->track_remove_region() is called only when KVM is 100% certain the
memory region will be removed, i.e. is invoked slightly later in KVM's
memslot modification flow.
Cc: Zhenyu Wang <[email protected]>
Suggested-by: Sean Christopherson <[email protected]>
Signed-off-by: Yan Zhao <[email protected]>
[sean: handle name change, massage changelog, rebase]
Tested-by: Yan Zhao <[email protected]>
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
drivers/gpu/drm/i915/gvt/kvmgt.c | 21 +++++++++------------
1 file changed, 9 insertions(+), 12 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 3ea3cb9eb599..3f2327455d85 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -108,9 +108,8 @@ struct gvt_dma {
static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len,
struct kvm_page_track_notifier_node *node);
-static void kvmgt_page_track_flush_slot(struct kvm *kvm,
- struct kvm_memory_slot *slot,
- struct kvm_page_track_notifier_node *node);
+static void kvmgt_page_track_remove_region(gfn_t gfn, unsigned long nr_pages,
+ struct kvm_page_track_notifier_node *node);
static ssize_t intel_vgpu_show_description(struct mdev_type *mtype, char *buf)
{
@@ -666,7 +665,7 @@ static int intel_vgpu_open_device(struct vfio_device *vfio_dev)
return -EEXIST;
vgpu->track_node.track_write = kvmgt_page_track_write;
- vgpu->track_node.track_flush_slot = kvmgt_page_track_flush_slot;
+ vgpu->track_node.track_remove_region = kvmgt_page_track_remove_region;
kvm_get_kvm(vgpu->vfio_device.kvm);
kvm_page_track_register_notifier(vgpu->vfio_device.kvm,
&vgpu->track_node);
@@ -1617,22 +1616,20 @@ static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len,
mutex_unlock(&info->vgpu_lock);
}
-static void kvmgt_page_track_flush_slot(struct kvm *kvm,
- struct kvm_memory_slot *slot,
- struct kvm_page_track_notifier_node *node)
+static void kvmgt_page_track_remove_region(gfn_t gfn, unsigned long nr_pages,
+ struct kvm_page_track_notifier_node *node)
{
unsigned long i;
- gfn_t gfn;
struct intel_vgpu *info =
container_of(node, struct intel_vgpu, track_node);
mutex_lock(&info->vgpu_lock);
- for (i = 0; i < slot->npages; i++) {
- gfn = slot->base_gfn + i;
- if (kvmgt_gfn_is_write_protected(info, gfn))
- kvmgt_protect_table_del(info, gfn);
+ for (i = 0; i < nr_pages; i++) {
+ if (kvmgt_gfn_is_write_protected(info, gfn + i))
+ kvmgt_protect_table_del(info, gfn + i);
}
+
mutex_unlock(&info->vgpu_lock);
}
--
2.41.0.487.g6d72f3e995-goog
Get/put references to KVM when a page-track notifier is (un)registered
instead of relying on the caller to do so. Forcing the caller to do the
bookkeeping is unnecessary and adds one more thing for users to get
wrong, e.g. see commit 9ed1fdee9ee3 ("drm/i915/gvt: Get reference to KVM
iff attachment to VM is successful").
Reviewed-by: Yan Zhao <[email protected]>
Tested-by: Yongwei Ma <[email protected]>
Signed-off-by: Sean Christopherson <[email protected]>
---
arch/x86/include/asm/kvm_page_track.h | 11 +++++------
arch/x86/kvm/mmu/page_track.c | 18 ++++++++++++------
drivers/gpu/drm/i915/gvt/kvmgt.c | 17 +++++++----------
3 files changed, 24 insertions(+), 22 deletions(-)
diff --git a/arch/x86/include/asm/kvm_page_track.h b/arch/x86/include/asm/kvm_page_track.h
index 4afab697e21c..3d040741044b 100644
--- a/arch/x86/include/asm/kvm_page_track.h
+++ b/arch/x86/include/asm/kvm_page_track.h
@@ -44,12 +44,11 @@ struct kvm_page_track_notifier_node {
struct kvm_page_track_notifier_node *node);
};
-void
-kvm_page_track_register_notifier(struct kvm *kvm,
- struct kvm_page_track_notifier_node *n);
-void
-kvm_page_track_unregister_notifier(struct kvm *kvm,
- struct kvm_page_track_notifier_node *n);
+int kvm_page_track_register_notifier(struct kvm *kvm,
+ struct kvm_page_track_notifier_node *n);
+void kvm_page_track_unregister_notifier(struct kvm *kvm,
+ struct kvm_page_track_notifier_node *n);
+
int kvm_write_track_add_gfn(struct kvm *kvm, gfn_t gfn);
int kvm_write_track_remove_gfn(struct kvm *kvm, gfn_t gfn);
#else
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index 2a64df38ccab..fd04e618ad2d 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -157,17 +157,22 @@ int kvm_page_track_init(struct kvm *kvm)
* register the notifier so that event interception for the tracked guest
* pages can be received.
*/
-void
-kvm_page_track_register_notifier(struct kvm *kvm,
- struct kvm_page_track_notifier_node *n)
+int kvm_page_track_register_notifier(struct kvm *kvm,
+ struct kvm_page_track_notifier_node *n)
{
struct kvm_page_track_notifier_head *head;
+ if (!kvm || kvm->mm != current->mm)
+ return -ESRCH;
+
+ kvm_get_kvm(kvm);
+
head = &kvm->arch.track_notifier_head;
write_lock(&kvm->mmu_lock);
hlist_add_head_rcu(&n->node, &head->track_notifier_list);
write_unlock(&kvm->mmu_lock);
+ return 0;
}
EXPORT_SYMBOL_GPL(kvm_page_track_register_notifier);
@@ -175,9 +180,8 @@ EXPORT_SYMBOL_GPL(kvm_page_track_register_notifier);
* stop receiving the event interception. It is the opposed operation of
* kvm_page_track_register_notifier().
*/
-void
-kvm_page_track_unregister_notifier(struct kvm *kvm,
- struct kvm_page_track_notifier_node *n)
+void kvm_page_track_unregister_notifier(struct kvm *kvm,
+ struct kvm_page_track_notifier_node *n)
{
struct kvm_page_track_notifier_head *head;
@@ -187,6 +191,8 @@ kvm_page_track_unregister_notifier(struct kvm *kvm,
hlist_del_rcu(&n->node);
write_unlock(&kvm->mmu_lock);
synchronize_srcu(&head->track_srcu);
+
+ kvm_put_kvm(kvm);
}
EXPORT_SYMBOL_GPL(kvm_page_track_unregister_notifier);
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 21342a93e418..eb50997dd369 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -654,21 +654,19 @@ static bool __kvmgt_vgpu_exist(struct intel_vgpu *vgpu)
static int intel_vgpu_open_device(struct vfio_device *vfio_dev)
{
struct intel_vgpu *vgpu = vfio_dev_to_vgpu(vfio_dev);
-
- if (!vgpu->vfio_device.kvm ||
- vgpu->vfio_device.kvm->mm != current->mm) {
- gvt_vgpu_err("KVM is required to use Intel vGPU\n");
- return -ESRCH;
- }
+ int ret;
if (__kvmgt_vgpu_exist(vgpu))
return -EEXIST;
vgpu->track_node.track_write = kvmgt_page_track_write;
vgpu->track_node.track_remove_region = kvmgt_page_track_remove_region;
- kvm_get_kvm(vgpu->vfio_device.kvm);
- kvm_page_track_register_notifier(vgpu->vfio_device.kvm,
- &vgpu->track_node);
+ ret = kvm_page_track_register_notifier(vgpu->vfio_device.kvm,
+ &vgpu->track_node);
+ if (ret) {
+ gvt_vgpu_err("KVM is required to use Intel vGPU\n");
+ return ret;
+ }
set_bit(INTEL_VGPU_STATUS_ATTACHED, vgpu->status);
@@ -703,7 +701,6 @@ static void intel_vgpu_close_device(struct vfio_device *vfio_dev)
kvm_page_track_unregister_notifier(vgpu->vfio_device.kvm,
&vgpu->track_node);
- kvm_put_kvm(vgpu->vfio_device.kvm);
kvmgt_protect_table_destroy(vgpu);
gvt_cache_destroy(vgpu);
--
2.41.0.487.g6d72f3e995-goog
Reviewed-by: Yan Zhao <[email protected]>
On Fri, Jul 28, 2023 at 06:35:13PM -0700, Sean Christopherson wrote:
> Bail from ppgtt_populate_shadow_entry() if an unexpected GTT entry type
> is encountered instead of subtly falling through to the common "direct
> shadow" path. Eliminating the default/error path's reliance on the common
> handling will allow hoisting intel_gvt_dma_map_guest_page() into the case
> statements so that the 2MiB case can try intel_gvt_dma_map_guest_page()
> and fallback to splitting the entry on failure.
>
> Reviewed-by: Zhi Wang <[email protected]>
> Tested-by: Yongwei Ma <[email protected]>
> Signed-off-by: Sean Christopherson <[email protected]>
> ---
> drivers/gpu/drm/i915/gvt/gtt.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c
> index 2aed31b497c9..61e38acee2d5 100644
> --- a/drivers/gpu/drm/i915/gvt/gtt.c
> +++ b/drivers/gpu/drm/i915/gvt/gtt.c
> @@ -1306,6 +1306,7 @@ static int ppgtt_populate_shadow_entry(struct intel_vgpu *vgpu,
> return -EINVAL;
> default:
> GEM_BUG_ON(1);
> + return -EINVAL;
> }
>
> /* direct shadow */
> --
> 2.41.0.487.g6d72f3e995-goog
>
Reviewed-by: Yan Zhao <[email protected]>
On Fri, Jul 28, 2023 at 06:35:09PM -0700, Sean Christopherson wrote:
> When shadowing a GTT entry with a 2M page, verify that the pfns are
> contiguous, not just that the struct page pointers are contiguous. The
> memory map is virtual contiguous if "CONFIG_FLATMEM=y ||
> CONFIG_SPARSEMEM_VMEMMAP=y", but not for "CONFIG_SPARSEMEM=y &&
> CONFIG_SPARSEMEM_VMEMMAP=n", so theoretically KVMGT could encounter struct
> pages that are virtually contiguous, but not physically contiguous.
>
> In practice, this flaw is likely a non-issue as it would cause functional
> problems iff a section isn't 2M aligned _and_ is directly adjacent to
> another section with discontiguous pfns.
>
> Tested-by: Yongwei Ma <[email protected]>
> Signed-off-by: Sean Christopherson <[email protected]>
> ---
> drivers/gpu/drm/i915/gvt/kvmgt.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
> index de675d799c7d..429f0f993a13 100644
> --- a/drivers/gpu/drm/i915/gvt/kvmgt.c
> +++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
> @@ -161,7 +161,7 @@ static int gvt_pin_guest_page(struct intel_vgpu *vgpu, unsigned long gfn,
>
> if (npage == 0)
> base_page = cur_page;
> - else if (base_page + npage != cur_page) {
> + else if (page_to_pfn(base_page) + npage != page_to_pfn(cur_page)) {
> gvt_vgpu_err("The pages are not continuous\n");
> ret = -EINVAL;
> npage++;
> --
> 2.41.0.487.g6d72f3e995-goog
>
On 7/29/2023 4:35 AM, Sean Christopherson wrote:
> From: Yan Zhao <[email protected]>
>
> Attempt to unpin pages in the error path of gvt_pin_guest_page() if and
> only if at least one page was successfully pinned. Unpinning doesn't
> cause functional problems, but vfio_device_container_unpin_pages()
> rightfully warns about being asked to unpin zero pages.
>
> Signed-off-by: Yan Zhao <[email protected]>
> [sean: write changelog]
> Signed-off-by: Sean Christopherson <[email protected]>
> ---
> drivers/gpu/drm/i915/gvt/kvmgt.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
> index 429f0f993a13..0366a699baf5 100644
> --- a/drivers/gpu/drm/i915/gvt/kvmgt.c
> +++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
> @@ -172,7 +172,8 @@ static int gvt_pin_guest_page(struct intel_vgpu *vgpu, unsigned long gfn,
> *page = base_page;
> return 0;
> err:
> - gvt_unpin_guest_page(vgpu, gfn, npage * PAGE_SIZE);
> + if (npage)
> + gvt_unpin_guest_page(vgpu, gfn, npage * PAGE_SIZE);
> return ret;
> }
>
Reviewed-by: Zhi Wang <[email protected]>
On 8/1/2023 4:47 AM, Yan Zhao wrote:
> Reviewed-by: Yan Zhao <[email protected]>
>
> On Fri, Jul 28, 2023 at 06:35:09PM -0700, Sean Christopherson wrote:
>> When shadowing a GTT entry with a 2M page, verify that the pfns are
>> contiguous, not just that the struct page pointers are contiguous. The
>> memory map is virtual contiguous if "CONFIG_FLATMEM=y ||
>> CONFIG_SPARSEMEM_VMEMMAP=y", but not for "CONFIG_SPARSEMEM=y &&
>> CONFIG_SPARSEMEM_VMEMMAP=n", so theoretically KVMGT could encounter struct
>> pages that are virtually contiguous, but not physically contiguous.
>>
>> In practice, this flaw is likely a non-issue as it would cause functional
>> problems iff a section isn't 2M aligned _and_ is directly adjacent to
>> another section with discontiguous pfns.
>>
>> Tested-by: Yongwei Ma <[email protected]>
>> Signed-off-by: Sean Christopherson <[email protected]>
>> ---
>> drivers/gpu/drm/i915/gvt/kvmgt.c | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
>> index de675d799c7d..429f0f993a13 100644
>> --- a/drivers/gpu/drm/i915/gvt/kvmgt.c
>> +++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
>> @@ -161,7 +161,7 @@ static int gvt_pin_guest_page(struct intel_vgpu *vgpu, unsigned long gfn,
>>
>> if (npage == 0)
>> base_page = cur_page;
>> - else if (base_page + npage != cur_page) {
>> + else if (page_to_pfn(base_page) + npage != page_to_pfn(cur_page)) {
>> gvt_vgpu_err("The pages are not continuous\n");
>> ret = -EINVAL;
>> npage++;
>> --
>> 2.41.0.487.g6d72f3e995-goog
>>
Reviewed-by: Zhi Wang <[email protected]>
On 7/29/2023 4:35 AM, Sean Christopherson wrote:
> Check that the pfn found by gfn_to_pfn() is actually backed by "struct
> page" memory prior to retrieving and dereferencing the page. KVM
> supports backing guest memory with VM_PFNMAP, VM_IO, etc., and so
> there is no guarantee the pfn returned by gfn_to_pfn() has an associated
> "struct page".
>
> Fixes: b901b252b6cf ("drm/i915/gvt: Add 2M huge gtt support")
> Reviewed-by: Yan Zhao <[email protected]>
> Tested-by: Yongwei Ma <[email protected]>
> Signed-off-by: Sean Christopherson <[email protected]>
> ---
> drivers/gpu/drm/i915/gvt/gtt.c | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c
> index 4ec85308379a..58b9b316ae46 100644
> --- a/drivers/gpu/drm/i915/gvt/gtt.c
> +++ b/drivers/gpu/drm/i915/gvt/gtt.c
> @@ -1183,6 +1183,10 @@ static int is_2MB_gtt_possible(struct intel_vgpu *vgpu,
> pfn = gfn_to_pfn(vgpu->vfio_device.kvm, ops->get_pfn(entry));
> if (is_error_noslot_pfn(pfn))
> return -EINVAL;
> +
> + if (!pfn_valid(pfn))
> + return -EINVAL;
> +
> return PageTransHuge(pfn_to_page(pfn));
> }
>
Reviewed-by: Zhi Wang <[email protected]>
On 7/29/2023 4:35 AM, Sean Christopherson wrote:
> Put the struct page reference acquired by gfn_to_pfn(), KVM's API is that
> the caller is ultimately responsible for dropping any reference.
>
> Note, kvm_release_pfn_clean() ensures the pfn is actually a refcounted
> struct page before trying to put any references.
>
> Fixes: b901b252b6cf ("drm/i915/gvt: Add 2M huge gtt support")
> Reviewed-by: Yan Zhao <[email protected]>
> Tested-by: Yongwei Ma <[email protected]>
> Signed-off-by: Sean Christopherson <[email protected]>
> ---
> drivers/gpu/drm/i915/gvt/gtt.c | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c
> index f30922c55a0c..5426a27c1b71 100644
> --- a/drivers/gpu/drm/i915/gvt/gtt.c
> +++ b/drivers/gpu/drm/i915/gvt/gtt.c
> @@ -1158,6 +1158,7 @@ static int is_2MB_gtt_possible(struct intel_vgpu *vgpu,
> {
> const struct intel_gvt_gtt_pte_ops *ops = vgpu->gvt->gtt.pte_ops;
> kvm_pfn_t pfn;
> + int ret;
>
> if (!HAS_PAGE_SIZES(vgpu->gvt->gt->i915, I915_GTT_PAGE_SIZE_2M))
> return 0;
> @@ -1171,7 +1172,9 @@ static int is_2MB_gtt_possible(struct intel_vgpu *vgpu,
> if (!pfn_valid(pfn))
> return -EINVAL;
>
> - return PageTransHuge(pfn_to_page(pfn));
> + ret = PageTransHuge(pfn_to_page(pfn));
> + kvm_release_pfn_clean(pfn);
> + return ret;
> }
>
> static int split_2MB_gtt_entry(struct intel_vgpu *vgpu,
Reviewed-by: Zhi Wang <[email protected]>
On 7/29/2023 4:35 AM, Sean Christopherson wrote:
> From: Yan Zhao <[email protected]>
>
> Switch from the poorly named and flawed ->track_flush_slot() to the newly
> introduced ->track_remove_region(). From KVMGT's perspective, the two
> hooks are functionally equivalent, the only difference being that
> ->track_remove_region() is called only when KVM is 100% certain the
> memory region will be removed, i.e. is invoked slightly later in KVM's
> memslot modification flow.
>
> Cc: Zhenyu Wang <[email protected]>
> Suggested-by: Sean Christopherson <[email protected]>
> Signed-off-by: Yan Zhao <[email protected]>
> [sean: handle name change, massage changelog, rebase]
> Tested-by: Yan Zhao <[email protected]>
> Tested-by: Yongwei Ma <[email protected]>
> Signed-off-by: Sean Christopherson <[email protected]>
> ---
> drivers/gpu/drm/i915/gvt/kvmgt.c | 21 +++++++++------------
> 1 file changed, 9 insertions(+), 12 deletions(-)
>
> diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
> index 3ea3cb9eb599..3f2327455d85 100644
> --- a/drivers/gpu/drm/i915/gvt/kvmgt.c
> +++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
> @@ -108,9 +108,8 @@ struct gvt_dma {
>
> static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len,
> struct kvm_page_track_notifier_node *node);
> -static void kvmgt_page_track_flush_slot(struct kvm *kvm,
> - struct kvm_memory_slot *slot,
> - struct kvm_page_track_notifier_node *node);
> +static void kvmgt_page_track_remove_region(gfn_t gfn, unsigned long nr_pages,
> + struct kvm_page_track_notifier_node *node);
>
> static ssize_t intel_vgpu_show_description(struct mdev_type *mtype, char *buf)
> {
> @@ -666,7 +665,7 @@ static int intel_vgpu_open_device(struct vfio_device *vfio_dev)
> return -EEXIST;
>
> vgpu->track_node.track_write = kvmgt_page_track_write;
> - vgpu->track_node.track_flush_slot = kvmgt_page_track_flush_slot;
> + vgpu->track_node.track_remove_region = kvmgt_page_track_remove_region;
> kvm_get_kvm(vgpu->vfio_device.kvm);
> kvm_page_track_register_notifier(vgpu->vfio_device.kvm,
> &vgpu->track_node);
> @@ -1617,22 +1616,20 @@ static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len,
> mutex_unlock(&info->vgpu_lock);
> }
>
> -static void kvmgt_page_track_flush_slot(struct kvm *kvm,
> - struct kvm_memory_slot *slot,
> - struct kvm_page_track_notifier_node *node)
> +static void kvmgt_page_track_remove_region(gfn_t gfn, unsigned long nr_pages,
> + struct kvm_page_track_notifier_node *node)
> {
> unsigned long i;
> - gfn_t gfn;
> struct intel_vgpu *info =
> container_of(node, struct intel_vgpu, track_node);
>
> mutex_lock(&info->vgpu_lock);
>
> - for (i = 0; i < slot->npages; i++) {
> - gfn = slot->base_gfn + i;
> - if (kvmgt_gfn_is_write_protected(info, gfn))
> - kvmgt_protect_table_del(info, gfn);
> + for (i = 0; i < nr_pages; i++) {
> + if (kvmgt_gfn_is_write_protected(info, gfn + i))
> + kvmgt_protect_table_del(info, gfn + i);
> }
> +
> mutex_unlock(&info->vgpu_lock);
> }
>
Reviewed-by: Zhi Wang <[email protected]>
On 7/29/2023 4:35 AM, Sean Christopherson wrote:
> Use an "unsigned long" instead of an "int" when iterating over the gfns
> in a memslot. The number of pages in the memslot is tracked as an
> "unsigned long", e.g. KVMGT could theoretically break if a KVM memslot
> larger than 16TiB were deleted (2^32 * 4KiB).
>
> Reviewed-by: Yan Zhao <[email protected]>
> Tested-by: Yongwei Ma <[email protected]>
> Signed-off-by: Sean Christopherson <[email protected]>
> ---
> drivers/gpu/drm/i915/gvt/kvmgt.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
> index 97c6d3c53710..6f52886c4051 100644
> --- a/drivers/gpu/drm/i915/gvt/kvmgt.c
> +++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
> @@ -1620,7 +1620,7 @@ static void kvmgt_page_track_flush_slot(struct kvm *kvm,
> struct kvm_memory_slot *slot,
> struct kvm_page_track_notifier_node *node)
> {
> - int i;
> + unsigned long i;
> gfn_t gfn;
> struct intel_vgpu *info =
> container_of(node, struct intel_vgpu, track_node);
Reviewed-by: Zhi Wang <[email protected]>
On 7/29/2023 4:35 AM, Sean Christopherson wrote:
> Drop intel_vgpu_reset_gtt() as it no longer has any callers. In addition
> to eliminating dead code, this eliminates the last possible scenario where
> __kvmgt_protect_table_find() can be reached without holding vgpu_lock.
> Requiring vgpu_lock to be held when calling __kvmgt_protect_table_find()
> will allow a protecting the gfn hash with vgpu_lock without too much fuss.
>
> No functional change intended.
>
> Fixes: ba25d977571e ("drm/i915/gvt: Do not destroy ppgtt_mm during vGPU D3->D0.")
> Reviewed-by: Yan Zhao <[email protected]>
> Tested-by: Yongwei Ma <[email protected]>
> Signed-off-by: Sean Christopherson <[email protected]>
> ---
> drivers/gpu/drm/i915/gvt/gtt.c | 18 ------------------
> drivers/gpu/drm/i915/gvt/gtt.h | 1 -
> 2 files changed, 19 deletions(-)
>
> diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c
> index f505be9e647a..c3c623b929ce 100644
> --- a/drivers/gpu/drm/i915/gvt/gtt.c
> +++ b/drivers/gpu/drm/i915/gvt/gtt.c
> @@ -2817,24 +2817,6 @@ void intel_vgpu_reset_ggtt(struct intel_vgpu *vgpu, bool invalidate_old)
> ggtt_invalidate(gvt->gt);
> }
>
> -/**
> - * intel_vgpu_reset_gtt - reset the all GTT related status
> - * @vgpu: a vGPU
> - *
> - * This function is called from vfio core to reset reset all
> - * GTT related status, including GGTT, PPGTT, scratch page.
> - *
> - */
> -void intel_vgpu_reset_gtt(struct intel_vgpu *vgpu)
> -{
> - /* Shadow pages are only created when there is no page
> - * table tracking data, so remove page tracking data after
> - * removing the shadow pages.
> - */
> - intel_vgpu_destroy_all_ppgtt_mm(vgpu);
> - intel_vgpu_reset_ggtt(vgpu, true);
> -}
> -
> /**
> * intel_gvt_restore_ggtt - restore all vGPU's ggtt entries
> * @gvt: intel gvt device
> diff --git a/drivers/gpu/drm/i915/gvt/gtt.h b/drivers/gpu/drm/i915/gvt/gtt.h
> index a3b0f59ec8bd..4cb183e06e95 100644
> --- a/drivers/gpu/drm/i915/gvt/gtt.h
> +++ b/drivers/gpu/drm/i915/gvt/gtt.h
> @@ -224,7 +224,6 @@ void intel_vgpu_reset_ggtt(struct intel_vgpu *vgpu, bool invalidate_old);
> void intel_vgpu_invalidate_ppgtt(struct intel_vgpu *vgpu);
>
> int intel_gvt_init_gtt(struct intel_gvt *gvt);
> -void intel_vgpu_reset_gtt(struct intel_vgpu *vgpu);
> void intel_gvt_clean_gtt(struct intel_gvt *gvt);
>
> struct intel_vgpu_mm *intel_gvt_find_ppgtt_mm(struct intel_vgpu *vgpu,
Reviewed-by: Zhi Wang <[email protected]>
On 7/29/2023 4:35 AM, Sean Christopherson wrote:
> Drop @vcpu from KVM's ->track_write() hook provided for external users of
> the page-track APIs now that KVM itself doesn't use the page-track
> mechanism.
>
> Reviewed-by: Yan Zhao <[email protected]>
> Tested-by: Yongwei Ma <[email protected]>
> Signed-off-by: Sean Christopherson <[email protected]>
> ---
> arch/x86/include/asm/kvm_page_track.h | 5 ++---
> arch/x86/kvm/mmu/page_track.c | 2 +-
> drivers/gpu/drm/i915/gvt/kvmgt.c | 10 ++++------
> 3 files changed, 7 insertions(+), 10 deletions(-)
>
> diff --git a/arch/x86/include/asm/kvm_page_track.h b/arch/x86/include/asm/kvm_page_track.h
> index eb186bc57f6a..8c4d216e3b2b 100644
> --- a/arch/x86/include/asm/kvm_page_track.h
> +++ b/arch/x86/include/asm/kvm_page_track.h
> @@ -26,14 +26,13 @@ struct kvm_page_track_notifier_node {
> * It is called when guest is writing the write-tracked page
> * and write emulation is finished at that time.
> *
> - * @vcpu: the vcpu where the write access happened.
> * @gpa: the physical address written by guest.
> * @new: the data was written to the address.
> * @bytes: the written length.
> * @node: this node
> */
> - void (*track_write)(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
> - int bytes, struct kvm_page_track_notifier_node *node);
> + void (*track_write)(gpa_t gpa, const u8 *new, int bytes,
> + struct kvm_page_track_notifier_node *node);
> /*
> * It is called when memory slot is being moved or removed
> * users can drop write-protection for the pages in that memory slot
> diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
> index 23088c90d2fd..891e5cc52b45 100644
> --- a/arch/x86/kvm/mmu/page_track.c
> +++ b/arch/x86/kvm/mmu/page_track.c
> @@ -272,7 +272,7 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
> hlist_for_each_entry_srcu(n, &head->track_notifier_list, node,
> srcu_read_lock_held(&head->track_srcu))
> if (n->track_write)
> - n->track_write(vcpu, gpa, new, bytes, n);
> + n->track_write(gpa, new, bytes, n);
> srcu_read_unlock(&head->track_srcu, idx);
>
> kvm_mmu_track_write(vcpu, gpa, new, bytes);
> diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
> index 034be0655daa..e9276500435d 100644
> --- a/drivers/gpu/drm/i915/gvt/kvmgt.c
> +++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
> @@ -106,9 +106,8 @@ struct gvt_dma {
> #define vfio_dev_to_vgpu(vfio_dev) \
> container_of((vfio_dev), struct intel_vgpu, vfio_device)
>
> -static void kvmgt_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
> - const u8 *val, int len,
> - struct kvm_page_track_notifier_node *node);
> +static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len,
> + struct kvm_page_track_notifier_node *node);
> static void kvmgt_page_track_flush_slot(struct kvm *kvm,
> struct kvm_memory_slot *slot,
> struct kvm_page_track_notifier_node *node);
> @@ -1603,9 +1602,8 @@ int intel_gvt_page_track_remove(struct intel_vgpu *info, u64 gfn)
> return 0;
> }
>
> -static void kvmgt_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
> - const u8 *val, int len,
> - struct kvm_page_track_notifier_node *node)
> +static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len,
> + struct kvm_page_track_notifier_node *node)
> {
> struct intel_vgpu *info =
> container_of(node, struct intel_vgpu, track_node);
Reviewed-by: Zhi Wang <[email protected]>
On 7/29/2023 4:35 AM, Sean Christopherson wrote:
> Use vgpu_lock instead of KVM's mmu_lock to protect accesses to the hash
> table used to track which gfns are write-protected when shadowing the
> guest's GTT, and hoist the acquisition of vgpu_lock from
> intel_vgpu_page_track_handler() out to its sole caller,
> kvmgt_page_track_write().
>
> This fixes a bug where kvmgt_page_track_write(), which doesn't hold
> kvm->mmu_lock, could race with intel_gvt_page_track_remove() and trigger
> a use-after-free.
>
> Fixing kvmgt_page_track_write() by taking kvm->mmu_lock is not an option
> as mmu_lock is a r/w spinlock, and intel_vgpu_page_track_handler() might
> sleep when acquiring vgpu->cache_lock deep down the callstack:
>
> intel_vgpu_page_track_handler()
> |
> |-> page_track->handler / ppgtt_write_protection_handler()
> |
> |-> ppgtt_handle_guest_write_page_table_bytes()
> |
> |-> ppgtt_handle_guest_write_page_table()
> |
> |-> ppgtt_handle_guest_entry_removal()
> |
> |-> ppgtt_invalidate_pte()
> |
> |-> intel_gvt_dma_unmap_guest_page()
> |
> |-> mutex_lock(&vgpu->cache_lock);
>
> Reviewed-by: Yan Zhao <[email protected]>
> Tested-by: Yongwei Ma <[email protected]>
> Signed-off-by: Sean Christopherson <[email protected]>
> ---
> drivers/gpu/drm/i915/gvt/kvmgt.c | 55 +++++++++++++++------------
> drivers/gpu/drm/i915/gvt/page_track.c | 10 +----
> 2 files changed, 33 insertions(+), 32 deletions(-)
>
> diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
> index 6f52886c4051..034be0655daa 100644
> --- a/drivers/gpu/drm/i915/gvt/kvmgt.c
> +++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
> @@ -352,6 +352,8 @@ __kvmgt_protect_table_find(struct intel_vgpu *info, gfn_t gfn)
> {
> struct kvmgt_pgfn *p, *res = NULL;
>
> + lockdep_assert_held(&info->vgpu_lock);
> +
> hash_for_each_possible(info->ptable, p, hnode, gfn) {
> if (gfn == p->gfn) {
> res = p;
> @@ -1553,6 +1555,9 @@ int intel_gvt_page_track_add(struct intel_vgpu *info, u64 gfn)
> if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, info->status))
> return -ESRCH;
>
> + if (kvmgt_gfn_is_write_protected(info, gfn))
> + return 0;
> +
> idx = srcu_read_lock(&kvm->srcu);
> slot = gfn_to_memslot(kvm, gfn);
> if (!slot) {
> @@ -1561,16 +1566,12 @@ int intel_gvt_page_track_add(struct intel_vgpu *info, u64 gfn)
> }
>
> write_lock(&kvm->mmu_lock);
> -
> - if (kvmgt_gfn_is_write_protected(info, gfn))
> - goto out;
> -
> kvm_slot_page_track_add_page(kvm, slot, gfn, KVM_PAGE_TRACK_WRITE);
> + write_unlock(&kvm->mmu_lock);
> +
> + srcu_read_unlock(&kvm->srcu, idx);
> +
> kvmgt_protect_table_add(info, gfn);
> -
> -out:
> - write_unlock(&kvm->mmu_lock);
> - srcu_read_unlock(&kvm->srcu, idx);
> return 0;
> }
>
> @@ -1583,24 +1584,22 @@ int intel_gvt_page_track_remove(struct intel_vgpu *info, u64 gfn)
> if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, info->status))
> return -ESRCH;
>
> - idx = srcu_read_lock(&kvm->srcu);
> - slot = gfn_to_memslot(kvm, gfn);
> - if (!slot) {
> - srcu_read_unlock(&kvm->srcu, idx);
> - return -EINVAL;
> - }
> -
> - write_lock(&kvm->mmu_lock);
> -
> if (!kvmgt_gfn_is_write_protected(info, gfn))
> - goto out;
> + return 0;
>
> + idx = srcu_read_lock(&kvm->srcu);
> + slot = gfn_to_memslot(kvm, gfn);
> + if (!slot) {
> + srcu_read_unlock(&kvm->srcu, idx);
> + return -EINVAL;
> + }
> +
> + write_lock(&kvm->mmu_lock);
> kvm_slot_page_track_remove_page(kvm, slot, gfn, KVM_PAGE_TRACK_WRITE);
> + write_unlock(&kvm->mmu_lock);
> + srcu_read_unlock(&kvm->srcu, idx);
> +
> kvmgt_protect_table_del(info, gfn);
> -
> -out:
> - write_unlock(&kvm->mmu_lock);
> - srcu_read_unlock(&kvm->srcu, idx);
> return 0;
> }
>
> @@ -1611,9 +1610,13 @@ static void kvmgt_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa,
> struct intel_vgpu *info =
> container_of(node, struct intel_vgpu, track_node);
>
> + mutex_lock(&info->vgpu_lock);
> +
> if (kvmgt_gfn_is_write_protected(info, gpa_to_gfn(gpa)))
> intel_vgpu_page_track_handler(info, gpa,
> (void *)val, len);
> +
> + mutex_unlock(&info->vgpu_lock);
> }
>
> static void kvmgt_page_track_flush_slot(struct kvm *kvm,
> @@ -1625,16 +1628,20 @@ static void kvmgt_page_track_flush_slot(struct kvm *kvm,
> struct intel_vgpu *info =
> container_of(node, struct intel_vgpu, track_node);
>
> - write_lock(&kvm->mmu_lock);
> + mutex_lock(&info->vgpu_lock);
> +
> for (i = 0; i < slot->npages; i++) {
> gfn = slot->base_gfn + i;
> if (kvmgt_gfn_is_write_protected(info, gfn)) {
> + write_lock(&kvm->mmu_lock);
> kvm_slot_page_track_remove_page(kvm, slot, gfn,
> KVM_PAGE_TRACK_WRITE);
> + write_unlock(&kvm->mmu_lock);
> +
> kvmgt_protect_table_del(info, gfn);
> }
> }
> - write_unlock(&kvm->mmu_lock);
> + mutex_unlock(&info->vgpu_lock);
> }
>
> void intel_vgpu_detach_regions(struct intel_vgpu *vgpu)
> diff --git a/drivers/gpu/drm/i915/gvt/page_track.c b/drivers/gpu/drm/i915/gvt/page_track.c
> index df34e73cba41..60a65435556d 100644
> --- a/drivers/gpu/drm/i915/gvt/page_track.c
> +++ b/drivers/gpu/drm/i915/gvt/page_track.c
> @@ -162,13 +162,9 @@ int intel_vgpu_page_track_handler(struct intel_vgpu *vgpu, u64 gpa,
> struct intel_vgpu_page_track *page_track;
> int ret = 0;
>
> - mutex_lock(&vgpu->vgpu_lock);
> -
> page_track = intel_vgpu_find_page_track(vgpu, gpa >> PAGE_SHIFT);
> - if (!page_track) {
> - ret = -ENXIO;
> - goto out;
> - }
> + if (!page_track)
> + return -ENXIO;
>
> if (unlikely(vgpu->failsafe)) {
> /* Remove write protection to prevent furture traps. */
> @@ -179,7 +175,5 @@ int intel_vgpu_page_track_handler(struct intel_vgpu *vgpu, u64 gpa,
> gvt_err("guest page write error, gpa %llx\n", gpa);
> }
>
> -out:
> - mutex_unlock(&vgpu->vgpu_lock);
> return ret;
> }
Reviewed-by: Zhi Wang <[email protected]>
On 7/29/2023 4:35 AM, Sean Christopherson wrote:
> Open code gpa_to_gfn() in kvmgt_page_track_write() and drop KVMGT's
> dependency on kvm_host.h, i.e. include only kvm_page_track.h. KVMGT
> assumes "gfn == gpa >> PAGE_SHIFT" all over the place, including a few
> lines below in the same function with the same gpa, i.e. there's no
> reason to use KVM's helper for this one case.
>
> No functional change intended.
>
> Reviewed-by: Yan Zhao <[email protected]>
> Tested-by: Yongwei Ma <[email protected]>
> Signed-off-by: Sean Christopherson <[email protected]>
> ---
> drivers/gpu/drm/i915/gvt/gvt.h | 3 ++-
> drivers/gpu/drm/i915/gvt/kvmgt.c | 2 +-
> 2 files changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/gpu/drm/i915/gvt/gvt.h b/drivers/gpu/drm/i915/gvt/gvt.h
> index 2d65800d8e93..53a0a42a50db 100644
> --- a/drivers/gpu/drm/i915/gvt/gvt.h
> +++ b/drivers/gpu/drm/i915/gvt/gvt.h
> @@ -34,10 +34,11 @@
> #define _GVT_H_
>
> #include <uapi/linux/pci_regs.h>
> -#include <linux/kvm_host.h>
> #include <linux/vfio.h>
> #include <linux/mdev.h>
>
> +#include <asm/kvm_page_track.h>
> +
> #include "i915_drv.h"
> #include "intel_gvt.h"
>
> diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
> index eb50997dd369..aaed3969f204 100644
> --- a/drivers/gpu/drm/i915/gvt/kvmgt.c
> +++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
> @@ -1585,7 +1585,7 @@ static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len,
>
> mutex_lock(&info->vgpu_lock);
>
> - if (kvmgt_gfn_is_write_protected(info, gpa_to_gfn(gpa)))
> + if (kvmgt_gfn_is_write_protected(info, gpa >> PAGE_SHIFT))
> intel_vgpu_page_track_handler(info, gpa,
> (void *)val, len);
>
Reviewed-by: Zhi Wang <[email protected]>
On 7/29/2023 4:35 AM, Sean Christopherson wrote:
> When handling a slot "flush", don't call back into KVM to drop write
> protection for gfns in the slot. Now that KVM rejects attempts to move
> memory slots while KVMGT is attached, the only time a slot is "flushed"
> is when it's being removed, i.e. the memslot and all its write-tracking
> metadata is about to be deleted.
>
> Reviewed-by: Yan Zhao <[email protected]>
> Tested-by: Yongwei Ma <[email protected]>
> Signed-off-by: Sean Christopherson <[email protected]>
> ---
> drivers/gpu/drm/i915/gvt/kvmgt.c | 8 +-------
> 1 file changed, 1 insertion(+), 7 deletions(-)
>
> diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
> index e9276500435d..3ea3cb9eb599 100644
> --- a/drivers/gpu/drm/i915/gvt/kvmgt.c
> +++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
> @@ -1630,14 +1630,8 @@ static void kvmgt_page_track_flush_slot(struct kvm *kvm,
>
> for (i = 0; i < slot->npages; i++) {
> gfn = slot->base_gfn + i;
> - if (kvmgt_gfn_is_write_protected(info, gfn)) {
> - write_lock(&kvm->mmu_lock);
> - kvm_slot_page_track_remove_page(kvm, slot, gfn,
> - KVM_PAGE_TRACK_WRITE);
> - write_unlock(&kvm->mmu_lock);
> -
> + if (kvmgt_gfn_is_write_protected(info, gfn))
> kvmgt_protect_table_del(info, gfn);
> - }
> }
> mutex_unlock(&info->vgpu_lock);
> }
Reviewed-by: Zhi Wang <[email protected]>
On Fri, Jul 28, 2023 at 06:35:18PM -0700,
Sean Christopherson <[email protected]> wrote:
> Move x86's implementation of kvm_arch_flush_shadow_{all,memslot}() into
> mmu.c, and make kvm_mmu_zap_all() static as it was globally visible only
> for kvm_arch_flush_shadow_all(). This will allow refactoring
> kvm_arch_flush_shadow_memslot() to call kvm_mmu_zap_all() directly without
> having to expose kvm_mmu_zap_all_fast() outside of mmu.c. Keeping
> everything in mmu.c will also likely simplify supporting TDX, which
> intends to do zap only relevant SPTEs on memslot updates.
Yes, it helps TDX code cleaner to move mmu related function under mmu.c.
Reviewed-by: Isaku Yamahata <[email protected]>
Thanks,
>
> No functional change intended.
>
> Suggested-by: Yan Zhao <[email protected]>
> Tested-by: Yongwei Ma <[email protected]>
> Signed-off-by: Sean Christopherson <[email protected]>
> ---
> arch/x86/include/asm/kvm_host.h | 1 -
> arch/x86/kvm/mmu/mmu.c | 13 ++++++++++++-
> arch/x86/kvm/x86.c | 11 -----------
> 3 files changed, 12 insertions(+), 13 deletions(-)
>
> diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
> index 28bd38303d70..856ec22aceb6 100644
> --- a/arch/x86/include/asm/kvm_host.h
> +++ b/arch/x86/include/asm/kvm_host.h
> @@ -1832,7 +1832,6 @@ void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm,
> const struct kvm_memory_slot *memslot);
> void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm,
> const struct kvm_memory_slot *memslot);
> -void kvm_mmu_zap_all(struct kvm *kvm);
> void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen);
> void kvm_mmu_change_mmu_pages(struct kvm *kvm, unsigned long kvm_nr_mmu_pages);
>
> diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
> index ec169f5c7dce..c6dee659d592 100644
> --- a/arch/x86/kvm/mmu/mmu.c
> +++ b/arch/x86/kvm/mmu/mmu.c
> @@ -6732,7 +6732,7 @@ void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm,
> */
> }
>
> -void kvm_mmu_zap_all(struct kvm *kvm)
> +static void kvm_mmu_zap_all(struct kvm *kvm)
> {
> struct kvm_mmu_page *sp, *node;
> LIST_HEAD(invalid_list);
> @@ -6757,6 +6757,17 @@ void kvm_mmu_zap_all(struct kvm *kvm)
> write_unlock(&kvm->mmu_lock);
> }
>
> +void kvm_arch_flush_shadow_all(struct kvm *kvm)
> +{
> + kvm_mmu_zap_all(kvm);
> +}
> +
> +void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
> + struct kvm_memory_slot *slot)
> +{
> + kvm_page_track_flush_slot(kvm, slot);
> +}
> +
> void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen)
> {
> WARN_ON(gen & KVM_MEMSLOT_GEN_UPDATE_IN_PROGRESS);
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index a6b9bea62fb8..059571d5abed 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -12776,17 +12776,6 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
> kvm_arch_free_memslot(kvm, old);
> }
>
> -void kvm_arch_flush_shadow_all(struct kvm *kvm)
> -{
> - kvm_mmu_zap_all(kvm);
> -}
> -
> -void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
> - struct kvm_memory_slot *slot)
> -{
> - kvm_page_track_flush_slot(kvm, slot);
> -}
> -
> static inline bool kvm_guest_apic_has_interrupt(struct kvm_vcpu *vcpu)
> {
> return (is_guest_mode(vcpu) &&
> --
> 2.41.0.487.g6d72f3e995-goog
>
--
Isaku Yamahata <[email protected]>
On Fri, 28 Jul 2023 18:35:06 -0700, Sean Christopherson wrote:
> Fix a handful of minor bugs in KVMGT, and overhaul KVM's page-track APIs
> to provide a leaner and cleaner interface. The motivation for this
> series is to (significantly) reduce the number of KVM APIs that KVMGT
> uses, with a long-term goal of making all kvm_host.h headers KVM-internal.
>
> If there are no objections or issues, my plan is to take this through the
> KVM tree for 6.6 (I had it ready early last week, and then forgot to actually
> post v4, /facepalm).
>
> [...]
Applied to kvm-x86 mmu, thanks!
[01/29] drm/i915/gvt: Verify pfn is "valid" before dereferencing "struct page"
https://github.com/kvm-x86/linux/commit/865327865164
[02/29] drm/i915/gvt: remove interface intel_gvt_is_valid_gfn
https://github.com/kvm-x86/linux/commit/823ab2ea8429
[03/29] drm/i915/gvt: Verify hugepages are contiguous in physical address space
https://github.com/kvm-x86/linux/commit/e27395fb1b87
[04/29] drm/i915/gvt: Don't try to unpin an empty page range
https://github.com/kvm-x86/linux/commit/6a718c54c2ee
[05/29] drm/i915/gvt: Put the page reference obtained by KVM's gfn_to_pfn()
https://github.com/kvm-x86/linux/commit/f969ecabe30b
[06/29] drm/i915/gvt: Explicitly check that vGPU is attached before shadowing
https://github.com/kvm-x86/linux/commit/537eef32e720
[07/29] drm/i915/gvt: Error out on an attempt to shadowing an unknown GTT entry type
https://github.com/kvm-x86/linux/commit/c94811471997
[08/29] drm/i915/gvt: Don't rely on KVM's gfn_to_pfn() to query possible 2M GTT
https://github.com/kvm-x86/linux/commit/f018c319cc2f
[09/29] drm/i915/gvt: Use an "unsigned long" to iterate over memslot gfns
https://github.com/kvm-x86/linux/commit/4879a4370304
[10/29] drm/i915/gvt: Drop unused helper intel_vgpu_reset_gtt()
https://github.com/kvm-x86/linux/commit/ac5e77621712
[11/29] drm/i915/gvt: Protect gfn hash table with vgpu_lock
https://github.com/kvm-x86/linux/commit/49a83e190b5b
[12/29] KVM: x86/mmu: Move kvm_arch_flush_shadow_{all,memslot}() to mmu.c
https://github.com/kvm-x86/linux/commit/2f502998b046
[13/29] KVM: x86/mmu: Don't rely on page-track mechanism to flush on memslot change
https://github.com/kvm-x86/linux/commit/e2fe84fb5eae
[14/29] KVM: x86/mmu: Don't bounce through page-track mechanism for guest PTEs
https://github.com/kvm-x86/linux/commit/f1c58cdb8e04
[15/29] KVM: drm/i915/gvt: Drop @vcpu from KVM's ->track_write() hook
https://github.com/kvm-x86/linux/commit/3f8eb1d7d3ee
[16/29] KVM: x86: Reject memslot MOVE operations if KVMGT is attached
https://github.com/kvm-x86/linux/commit/aa611a99adb4
[17/29] drm/i915/gvt: Don't bother removing write-protection on to-be-deleted slot
https://github.com/kvm-x86/linux/commit/a41e34b05da7
[18/29] KVM: x86: Add a new page-track hook to handle memslot deletion
https://github.com/kvm-x86/linux/commit/cc49e12d8d3b
[19/29] drm/i915/gvt: switch from ->track_flush_slot() to ->track_remove_region()
https://github.com/kvm-x86/linux/commit/b9ae8a09f357
[20/29] KVM: x86: Remove the unused page-track hook track_flush_slot()
https://github.com/kvm-x86/linux/commit/1265fb534fa1
[21/29] KVM: x86/mmu: Move KVM-only page-track declarations to internal header
https://github.com/kvm-x86/linux/commit/c87966b313cc
[22/29] KVM: x86/mmu: Use page-track notifiers iff there are external users
https://github.com/kvm-x86/linux/commit/b5e33f265acd
[23/29] KVM: x86/mmu: Drop infrastructure for multiple page-track modes
https://github.com/kvm-x86/linux/commit/2431c9ab231a
[24/29] KVM: x86/mmu: Rename page-track APIs to reflect the new reality
https://github.com/kvm-x86/linux/commit/4b42f39917c1
[25/29] KVM: x86/mmu: Assert that correct locks are held for page write-tracking
https://github.com/kvm-x86/linux/commit/21e0e1efd880
[26/29] KVM: x86/mmu: Bug the VM if write-tracking is used but not enabled
https://github.com/kvm-x86/linux/commit/6d6ff9e6db15
[27/29] KVM: x86/mmu: Drop @slot param from exported/external page-track APIs
https://github.com/kvm-x86/linux/commit/ca181aa50724
[28/29] KVM: x86/mmu: Handle KVM bookkeeping in page-track APIs, not callers
https://github.com/kvm-x86/linux/commit/eae2d71635a1
[29/29] drm/i915/gvt: Drop final dependencies on KVM internal details
https://github.com/kvm-x86/linux/commit/2d28b1230c1c
--
https://github.com/kvm-x86/linux/tree/next
https://github.com/kvm-x86/linux/tree/fixes
On Wed, Aug 30, 2023, Like Xu wrote:
> On 2023/7/29 09:35, Sean Christopherson wrote:
> > Disallow moving memslots if the VM has external page-track users, i.e. if
> > KVMGT is being used to expose a virtual GPU to the guest, as KVMGT doesn't
> > correctly handle moving memory regions.
> >
> > Note, this is potential ABI breakage! E.g. userspace could move regions
> > that aren't shadowed by KVMGT without harming the guest. However, the
> > only known user of KVMGT is QEMU, and QEMU doesn't move generic memory
>
> This change breaks two kvm selftests:
>
> - set_memory_region_test;
> - memslot_perf_test;
It shoudn't. As of this patch, KVM doesn't register itself as a page-track user,
i.e. KVMGT is the only remaining caller to kvm_page_track_register_notifier().
Unless I messed up, the only way kvm_page_track_has_external_user() can return
true is if KVMGT is attached to the VM. The selftests most definitely don't do
anything with KVMGT, so I don't see how they can fail.
Are you seeing actually failures?
> Please help confirm if the tests/doc needs to be updated,
> or if the assumption needs to be further clarified.
What assumption?
> > regions. KVM's own support for moving memory regions was also broken for
> > multiple years (albeit for an edge case, but arguably moving RAM is
> > itself an edge case), e.g. see commit edd4fa37baa6 ("KVM: x86: Allocate
> > new rmap and large page tracking when moving memslot").
> >
> > Reviewed-by: Yan Zhao <[email protected]>
> > Tested-by: Yongwei Ma <[email protected]>
> > Signed-off-by: Sean Christopherson <[email protected]>
> > ---
> > arch/x86/include/asm/kvm_page_track.h | 3 +++
> > arch/x86/kvm/mmu/page_track.c | 5 +++++
> > arch/x86/kvm/x86.c | 7 +++++++
> > 3 files changed, 15 insertions(+)
> >
> > diff --git a/arch/x86/include/asm/kvm_page_track.h b/arch/x86/include/asm/kvm_page_track.h
> > index 8c4d216e3b2b..f744682648e7 100644
> > --- a/arch/x86/include/asm/kvm_page_track.h
> > +++ b/arch/x86/include/asm/kvm_page_track.h
> > @@ -75,4 +75,7 @@ kvm_page_track_unregister_notifier(struct kvm *kvm,
> > void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
> > int bytes);
> > void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot);
> > +
> > +bool kvm_page_track_has_external_user(struct kvm *kvm);
> > +
> > #endif
> > diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
> > index 891e5cc52b45..e6de9638e560 100644
> > --- a/arch/x86/kvm/mmu/page_track.c
> > +++ b/arch/x86/kvm/mmu/page_track.c
> > @@ -303,3 +303,8 @@ void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot)
> > n->track_flush_slot(kvm, slot, n);
> > srcu_read_unlock(&head->track_srcu, idx);
> > }
> > +
> > +bool kvm_page_track_has_external_user(struct kvm *kvm)
> > +{
> > + return hlist_empty(&kvm->arch.track_notifier_head.track_notifier_list);
> > +}
> > diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> > index 059571d5abed..4394bb49051f 100644
> > --- a/arch/x86/kvm/x86.c
> > +++ b/arch/x86/kvm/x86.c
> > @@ -12606,6 +12606,13 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,
> > struct kvm_memory_slot *new,
> > enum kvm_mr_change change)
> > {
> > + /*
> > + * KVM doesn't support moving memslots when there are external page
> > + * trackers attached to the VM, i.e. if KVMGT is in use.
> > + */
> > + if (change == KVM_MR_MOVE && kvm_page_track_has_external_user(kvm))
> > + return -EINVAL;
> > +
> > if (change == KVM_MR_CREATE || change == KVM_MR_MOVE) {
> > if ((new->base_gfn + new->npages - 1) > kvm_mmu_max_gfn())
> > return -EINVAL;