Received: by 10.223.176.5 with SMTP id f5csp839453wra; Fri, 2 Feb 2018 07:00:40 -0800 (PST) X-Google-Smtp-Source: AH8x227vLDFPjkW/4D99EWDWvVmU0jNC3j/84APPdiwS1Bp0zWiWzhw/89M5dTtDzhCgyagU9vtI X-Received: by 10.98.64.9 with SMTP id n9mr40981802pfa.194.1517583640760; Fri, 02 Feb 2018 07:00:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1517583640; cv=none; d=google.com; s=arc-20160816; b=BdRNIHcKERZOmu/xcgAPzGhk3gBkYkN7Iz/7gohp3qOFduCwVbMLDVAFfb/n/JtxzW MHJVNprqTREuP5Pmt7TotpKfXIYhTnPv90rVSX2ixhHfi26wQ29qTkILr32h49Pf+j4T p7qVjbrcd+nUP6qBieAy62aDfbnkSKFfs6jY60PL+sn0ULH/eeKS5yFVd06pz4/9ksZW G3NvVNtzs6wZ/OKXKoTOqMNFnXAVAJDzqnlLE2mFqVihaOIiHiq8gGxgETcNpqiUb8IO QVkBUJSfKtWwGUzBRl4BvrrGpP7g/0nuia25FQNcDEpRbFb5MR16XTRDcbCuaoh2cSRU 6qTQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:date:subject:to:from :dkim-signature:arc-authentication-results; bh=uOY8BT+W6z84iwiuFlMqmoc6Uc8rdTVdxxwlmx+Cc4k=; b=qA9LBHDlVm1ITvGPo0g3urrLtjNk/VHJm4NoeDJ3qwdyPiCr07MniF3UMRS1tRlRp4 1k7qCNHVoeOABKWcuhl1Sh862bTlBLo4+EN326/4ZiupwkRl85HkX1+lt2A4V1mY0fTf mcEi1T/k11kuwKFsjPKvVk5kCmACYlfOfIcW+nLnZOvp/y1/LCv8kdVzKJselYDev9rP B9y4QZH799a9BqmbsmVfXaPROGJTmgndXI+msJfTemdKopQvOT5OAVhtZezKDENelNDn V9VDPmB4ZdYVYJ0H/Nq8EQnfw9Ggax4/yQAUOP0WiB6r2UAd/M/uwprYTuTCGA1A+8G/ y6DA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@amazon.co.uk header.s=amazon201209 header.b=aYwgaIEV; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.co.uk Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id k10-v6si2031676pln.378.2018.02.02.07.00.25; Fri, 02 Feb 2018 07:00:40 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@amazon.co.uk header.s=amazon201209 header.b=aYwgaIEV; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.co.uk Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752446AbeBBO7i (ORCPT + 99 others); Fri, 2 Feb 2018 09:59:38 -0500 Received: from smtp-fw-6002.amazon.com ([52.95.49.90]:5264 "EHLO smtp-fw-6002.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752069AbeBBO7c (ORCPT ); Fri, 2 Feb 2018 09:59:32 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.co.uk; i=@amazon.co.uk; q=dns/txt; s=amazon201209; t=1517583572; x=1549119572; h=from:to:subject:date:message-id; bh=uOY8BT+W6z84iwiuFlMqmoc6Uc8rdTVdxxwlmx+Cc4k=; b=aYwgaIEVHmHi2XtWxEP/3agwT8ccmx8pNnAgtdswYJ41UWQOkA3obQhA WJn3rv9TTwzPpRun8JysfN6ayNb2jFiiJFono716ikyievmh+qOZJvfcz cb5g/Gd/CS4mfZVRgakkxBHKgpO8s0MyYGFpNUoP8XAp3e34EhGYwy+OG A=; X-IronPort-AV: E=Sophos;i="5.46,448,1511827200"; d="scan'208";a="330329713" Received: from iad6-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2b-55156cd4.us-west-2.amazon.com) ([10.124.125.6]) by smtp-border-fw-out-6002.iad6.amazon.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 02 Feb 2018 14:59:30 +0000 Received: from uc8d3ff76b9bc5848a9cc.ant.amazon.com (pdx2-ws-svc-lb17-vlan3.amazon.com [10.247.140.70]) by email-inbound-relay-2b-55156cd4.us-west-2.amazon.com (8.14.7/8.14.7) with ESMTP id w12ExM7K028639 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 2 Feb 2018 14:59:24 GMT Received: from uc8d3ff76b9bc5848a9cc.ant.amazon.com (localhost [127.0.0.1]) by uc8d3ff76b9bc5848a9cc.ant.amazon.com (8.15.2/8.15.2/Debian-3) with ESMTP id w12ExLOl000814; Fri, 2 Feb 2018 14:59:21 GMT Received: (from dwmw@localhost) by uc8d3ff76b9bc5848a9cc.ant.amazon.com (8.15.2/8.15.2/Submit) id w12ExLww000809; Fri, 2 Feb 2018 14:59:21 GMT From: David Woodhouse To: tglx@linutronix.de, karahmed@amazon.de, sironi@amazon.de, x86@kernel.org, kvm@vger.kernel.org, torvalds@linux-foundation.org, pbonzini@redhat.com, linux-kernel@vger.kernel.org, bp@alien8.de, peterz@infradead.org Subject: [PATCH] KVM: x86: Reduce retpoline performance impact in slot_handle_level_range() Date: Fri, 2 Feb 2018 14:59:19 +0000 Message-Id: <1517583559-424-1-git-send-email-dwmw@amazon.co.uk> X-Mailer: git-send-email 2.7.4 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org With retpoline, tight loops of "call this function for every XXX" are very much pessimised by taking a prediction miss *every* time. This one showed up very high in our early testing, and it only has five things it'll ever call so make it take an 'op' enum instead of a function pointer and let's see how that works out... Signed-off-by: David Woodhouse --- Not sure I like this. Better suggestions welcomed... In the general case, we have a few things we can do with the calls that retpoline turns into bottlenecks. This is one of them. Another option, if there are just one or two "likely" functions, is something along the lines of if (func == likelyfunc) likelyfunc() else (*func)(); // GCC does retpoline for this For things like kvm_x86_ops we really could just turn *all* of those into direct calls at runtime, like pvops does. There are some which land somewhere in the middle, like the default dma_ops. We probably want something like the 'likelyfunc' version above, except that we *also* want to flip the likelyfunc between the Intel and AMD IOMMU ops functions, at early boot. I'll see what I can come up with... arch/x86/kvm/mmu.c | 72 ++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 48 insertions(+), 24 deletions(-) diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c index 2b8eb4d..44f9de7 100644 --- a/arch/x86/kvm/mmu.c +++ b/arch/x86/kvm/mmu.c @@ -5055,12 +5055,21 @@ void kvm_mmu_uninit_vm(struct kvm *kvm) } /* The return value indicates if tlb flush on all vcpus is needed. */ -typedef bool (*slot_level_handler) (struct kvm *kvm, struct kvm_rmap_head *rmap_head); +enum slot_handler_op { + SLOT_RMAP_CLEAR_DIRTY, + SLOT_RMAP_SET_DIRTY, + SLOT_RMAP_WRITE_PROTECT, + SLOT_ZAP_RMAPP, + SLOT_ZAP_COLLAPSIBLE_SPTE, +}; + +static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm, + struct kvm_rmap_head *rmap_head); /* The caller should hold mmu-lock before calling this function. */ static bool slot_handle_level_range(struct kvm *kvm, struct kvm_memory_slot *memslot, - slot_level_handler fn, int start_level, int end_level, + enum slot_handler_op op, int start_level, int end_level, gfn_t start_gfn, gfn_t end_gfn, bool lock_flush_tlb) { struct slot_rmap_walk_iterator iterator; @@ -5068,8 +5077,29 @@ slot_handle_level_range(struct kvm *kvm, struct kvm_memory_slot *memslot, for_each_slot_rmap_range(memslot, start_level, end_level, start_gfn, end_gfn, &iterator) { - if (iterator.rmap) - flush |= fn(kvm, iterator.rmap); + if (iterator.rmap) { + switch (op) { + case SLOT_RMAP_CLEAR_DIRTY: + flush |= __rmap_clear_dirty(kvm, iterator.rmap); + break; + + case SLOT_RMAP_SET_DIRTY: + flush |= __rmap_set_dirty(kvm, iterator.rmap); + break; + + case SLOT_RMAP_WRITE_PROTECT: + flush |= __rmap_write_protect(kvm, iterator.rmap, false); + break; + + case SLOT_ZAP_RMAPP: + flush |= kvm_zap_rmapp(kvm, iterator.rmap); + break; + + case SLOT_ZAP_COLLAPSIBLE_SPTE: + flush |= kvm_mmu_zap_collapsible_spte(kvm, iterator.rmap); + break; + } + } if (need_resched() || spin_needbreak(&kvm->mmu_lock)) { if (flush && lock_flush_tlb) { @@ -5090,10 +5120,10 @@ slot_handle_level_range(struct kvm *kvm, struct kvm_memory_slot *memslot, static bool slot_handle_level(struct kvm *kvm, struct kvm_memory_slot *memslot, - slot_level_handler fn, int start_level, int end_level, + enum slot_handler_op op, int start_level, int end_level, bool lock_flush_tlb) { - return slot_handle_level_range(kvm, memslot, fn, start_level, + return slot_handle_level_range(kvm, memslot, op, start_level, end_level, memslot->base_gfn, memslot->base_gfn + memslot->npages - 1, lock_flush_tlb); @@ -5101,25 +5131,25 @@ slot_handle_level(struct kvm *kvm, struct kvm_memory_slot *memslot, static bool slot_handle_all_level(struct kvm *kvm, struct kvm_memory_slot *memslot, - slot_level_handler fn, bool lock_flush_tlb) + enum slot_handler_op op, bool lock_flush_tlb) { - return slot_handle_level(kvm, memslot, fn, PT_PAGE_TABLE_LEVEL, + return slot_handle_level(kvm, memslot, op, PT_PAGE_TABLE_LEVEL, PT_MAX_HUGEPAGE_LEVEL, lock_flush_tlb); } static bool slot_handle_large_level(struct kvm *kvm, struct kvm_memory_slot *memslot, - slot_level_handler fn, bool lock_flush_tlb) + enum slot_handler_op op, bool lock_flush_tlb) { - return slot_handle_level(kvm, memslot, fn, PT_PAGE_TABLE_LEVEL + 1, + return slot_handle_level(kvm, memslot, op, PT_PAGE_TABLE_LEVEL + 1, PT_MAX_HUGEPAGE_LEVEL, lock_flush_tlb); } static bool slot_handle_leaf(struct kvm *kvm, struct kvm_memory_slot *memslot, - slot_level_handler fn, bool lock_flush_tlb) + enum slot_handler_op op, bool lock_flush_tlb) { - return slot_handle_level(kvm, memslot, fn, PT_PAGE_TABLE_LEVEL, + return slot_handle_level(kvm, memslot, op, PT_PAGE_TABLE_LEVEL, PT_PAGE_TABLE_LEVEL, lock_flush_tlb); } @@ -5140,7 +5170,7 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end) if (start >= end) continue; - slot_handle_level_range(kvm, memslot, kvm_zap_rmapp, + slot_handle_level_range(kvm, memslot, SLOT_ZAP_RMAPP, PT_PAGE_TABLE_LEVEL, PT_MAX_HUGEPAGE_LEVEL, start, end - 1, true); } @@ -5149,19 +5179,13 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end) spin_unlock(&kvm->mmu_lock); } -static bool slot_rmap_write_protect(struct kvm *kvm, - struct kvm_rmap_head *rmap_head) -{ - return __rmap_write_protect(kvm, rmap_head, false); -} - void kvm_mmu_slot_remove_write_access(struct kvm *kvm, struct kvm_memory_slot *memslot) { bool flush; spin_lock(&kvm->mmu_lock); - flush = slot_handle_all_level(kvm, memslot, slot_rmap_write_protect, + flush = slot_handle_all_level(kvm, memslot, SLOT_RMAP_WRITE_PROTECT, false); spin_unlock(&kvm->mmu_lock); @@ -5226,7 +5250,7 @@ void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm, /* FIXME: const-ify all uses of struct kvm_memory_slot. */ spin_lock(&kvm->mmu_lock); slot_handle_leaf(kvm, (struct kvm_memory_slot *)memslot, - kvm_mmu_zap_collapsible_spte, true); + SLOT_ZAP_COLLAPSIBLE_SPTE, true); spin_unlock(&kvm->mmu_lock); } @@ -5236,7 +5260,7 @@ void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm, bool flush; spin_lock(&kvm->mmu_lock); - flush = slot_handle_leaf(kvm, memslot, __rmap_clear_dirty, false); + flush = slot_handle_leaf(kvm, memslot, SLOT_RMAP_CLEAR_DIRTY, false); spin_unlock(&kvm->mmu_lock); lockdep_assert_held(&kvm->slots_lock); @@ -5258,7 +5282,7 @@ void kvm_mmu_slot_largepage_remove_write_access(struct kvm *kvm, bool flush; spin_lock(&kvm->mmu_lock); - flush = slot_handle_large_level(kvm, memslot, slot_rmap_write_protect, + flush = slot_handle_large_level(kvm, memslot, SLOT_RMAP_WRITE_PROTECT, false); spin_unlock(&kvm->mmu_lock); @@ -5276,7 +5300,7 @@ void kvm_mmu_slot_set_dirty(struct kvm *kvm, bool flush; spin_lock(&kvm->mmu_lock); - flush = slot_handle_all_level(kvm, memslot, __rmap_set_dirty, false); + flush = slot_handle_all_level(kvm, memslot, SLOT_RMAP_SET_DIRTY, false); spin_unlock(&kvm->mmu_lock); lockdep_assert_held(&kvm->slots_lock); -- 2.7.4