Received: by 2002:a05:7412:a9a2:b0:e2:908c:2ebd with SMTP id o34csp1246642rdh; Fri, 27 Oct 2023 08:35:32 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGC9ZKGEe0fUd9EOwAEkDSS8BsD9rawqVsXiDLPK42M1qeJkbxAH2wDfLEB2WSjWdXTGK7H X-Received: by 2002:a25:2044:0:b0:d9a:4692:817b with SMTP id g65-20020a252044000000b00d9a4692817bmr9763936ybg.25.1698420931669; Fri, 27 Oct 2023 08:35:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698420931; cv=none; d=google.com; s=arc-20160816; b=0nq9KKX5V9ltdq6hNtY8AVMLZE9GUsb4OSvVlZSgTVK/wNovEjqopVn1CmwYebKh9P HKZo6k7xzzYzD4xuV9xWS4pNYb3HIVVmXc2pf2Pec5j8Bdbw7TSdiQQPv7hOX7CPPq8a j0a+TgGOiLtx67Vsjeoldn5CzpYEscFvpzJEEiDru6UHPULoLVIsMeSalAFOBAgpdAft vAcA9+IB6SVHVydoGmq9JIASvTnp1bnoRCh7MGiNFBnSA5iW82hpfkIyDknISd5z8Xlk M/6xU6RkN7OwuFMbBMzc29qCcquzdsKJZ+FdMIL+V392c46AZ7OE5tJprRjEIfrDWb7v SeDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=XHOnqXTZrq1c88NmZSEuOQ4vGOEbUT+2leErliknXQk=; fh=LIAit08ByDgkRLqlEmM1nmhI6SEZnuCWjlP1SRPYJ5Q=; b=vFQt96D/LLkOF4ERoItuPTqmI0ah6vVYYuYpdT2HQhS6ehVpf4n/ZjF8NC0o7O47GW Jp60LVOnwhOXqRbTfUFHKmYi+Q8XGByNmbeySioL4B43o7eOY71ZTHKryd/tHSjqFpzf LMjztad/3ewZ0NbVjwj1rLtl5o8pH6mQGXOqScJ8kOnSxLYcsM9cFbsMQj8ZZRptz9pl ykjFx8FuWxxr5uoyGn5Tt0pj3dRUs264M+q/ylcvz2MsJWQUEDa4PQfhXeIZEg+X+/K5 deMP0FZ3WKSo++ZhZSLTaMlL21UiroqbErLhA+swJg2I+qSy9QEiscYnAijz9C2NqjSx LS7w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@alien8.de header.s=alien8 header.b="Ux4mmu6/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alien8.de Return-Path: Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id a12-20020a5b000c000000b00d9a6b331944si2965342ybp.304.2023.10.27.08.35.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Oct 2023 08:35:31 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@alien8.de header.s=alien8 header.b="Ux4mmu6/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alien8.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 4DCD280740DF; Fri, 27 Oct 2023 08:35:23 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346308AbjJ0PfL (ORCPT + 99 others); Fri, 27 Oct 2023 11:35:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37856 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346277AbjJ0PfJ (ORCPT ); Fri, 27 Oct 2023 11:35:09 -0400 Received: from mail.alien8.de (mail.alien8.de [65.109.113.108]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9CC40AF for ; Fri, 27 Oct 2023 08:35:07 -0700 (PDT) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.alien8.de (SuperMail on ZX Spectrum 128k) with ESMTP id 162DF40E01A4; Fri, 27 Oct 2023 15:35:06 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at mail.alien8.de Authentication-Results: mail.alien8.de (amavisd-new); dkim=pass (4096-bit key) header.d=alien8.de Received: from mail.alien8.de ([127.0.0.1]) by localhost (mail.alien8.de [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 4tkJqkoZ2WmC; Fri, 27 Oct 2023 15:35:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alien8.de; s=alien8; t=1698420903; bh=XHOnqXTZrq1c88NmZSEuOQ4vGOEbUT+2leErliknXQk=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=Ux4mmu6/0gNDVYw5Aukc0WfhM6+P+jTIFRk9yEliV2XJS6/6j9c1EL2CVVzZGLSMv uIfeVMF0XgJVdpzcCOJlki+bBSn9ieh+DXKS+Sh7kxAUCGngPAOf0aiykXkOV0gfYz CM1fDlBmIfeDoLHg/v21UyJ4h6YqQC+xBdho0QFv6xjlC4iW6dgzuOsbbI+75koysQ B6U2Kh4UpJqdBymcKcEmDCLrtKAzAhK8GJApir7bNRfGEhH4npjfHdGJj+LpZbhJ9+ FgGsnDeCh5OIMYjqUNVJVJ8TGsuj/3qaIL+O1rA01PTVCTQFdntESt5tK2H1rmA9sH NdNBrdvfKZy1l7vu+IhwRP0GqcyywzIp0YMf0/sfbpp8TCXQS6VqLKYiZ1lzKwaImM cXP9A+x8LRN2PZ0O6sTTLKEIsTPTmqAKrlb6xoOQhzq+UHE0RIxk20pJE35x3OikWG thTcD2N0/gx9ZUOLdqJAoCzDx47iX018OYViafDFWBSPxupBpNjxNZN3QseCc4U60u noQPe+DD+UTENAD6YC1kH/X9UvdAw/+Y+dVthHtGyYe4wItVqqk9L1AalJ0K/hpYAR 7LWRkZbKiWQdzYJTo0cu7bNJFJDkC1VAkVpdcrFHzkpcWee1/RTMiqxFBSwJWJ2NRW 74IxjQSEa5JfuRfHM72d06MQ= Received: from zn.tnic (pd95304da.dip0.t-ipconnect.de [217.83.4.218]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature ECDSA (P-256) server-digest SHA256) (No client certificate requested) by mail.alien8.de (SuperMail on ZX Spectrum 128k) with ESMTPSA id 83D4440E0173; Fri, 27 Oct 2023 15:34:59 +0000 (UTC) Date: Fri, 27 Oct 2023 17:34:58 +0200 From: Borislav Petkov To: Peter Zijlstra Cc: X86 ML , Kishon VijayAbraham , LKML Subject: [PATCH 2/2] x86/barrier: Do not serialize MSR accesses on AMD Message-ID: <20231027153458.GMZTvYou1tlK6HD8/Y@fat_crate.local> References: <20230622095212.20940-1-bp@alien8.de> <20230703125419.GJ4253@hirez.programming.kicks-ass.net> <20230704074631.GAZKPOV/9BfqP0aU8v@fat_crate.local> <20230704090132.GP4253@hirez.programming.kicks-ass.net> <20230704092222.GBZKPkzgdM8rbPe7zA@fat_crate.local> <20231027153327.GKZTvYR3qslaTUjtCT@fat_crate.local> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20231027153327.GKZTvYR3qslaTUjtCT@fat_crate.local> X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Fri, 27 Oct 2023 08:35:23 -0700 (PDT) From: "Borislav Petkov (AMD)" Date: Fri, 27 Oct 2023 14:24:16 +0200 AMD does not have the requirement for a synchronization barrier when acccessing a certain group of MSRs. Do not incur that unnecessary penalty there. While at it, move to processor.h to avoid include hell. Untangling that file properly is a matter for another day. Some notes on the performance aspect of why this is relevant, courtesy of Kishon VijayAbraham : On a AMD Zen4 system with 96 cores, a modified ipi-bench[1] on a VM shows x2AVIC IPI rate is 3% to 4% lower than AVIC IPI rate. The ipi-bench is modified so that the IPIs are sent between two vCPUs in the same CCX. This also requires to pin the vCPU to a physical core to prevent any latencies. This simulates the use case of pinning vCPUs to the thread of a single CCX to avoid interrupt IPI latency. In order to avoid run-to-run variance (for both x2AVIC and AVIC), the below configurations are done: 1) Disable Power States in BIOS (to prevent the system from going to lower power state) 2) Run the system at fixed frequency 2500MHz (to prevent the system from increasing the frequency when the load is more) With the above configuration: *) Performance measured using ipi-bench for AVIC: Average Latency: 1124.98ns [Time to send IPI from one vCPU to another vCPU] Cumulative throughput: 42.6759M/s [Total number of IPIs sent in a second from 48 vCPUs simultaneously] *) Performance measured using ipi-bench for x2AVIC: Average Latency: 1172.42ns [Time to send IPI from one vCPU to another vCPU] Cumulative throughput: 40.9432M/s [Total number of IPIs sent in a second from 48 vCPUs simultaneously] From above, x2AVIC latency is ~4% more than AVIC. However, the expectation is x2AVIC performance to be better or equivalent to AVIC. Upon analyzing the perf captures, it is observed significant time is spent in weak_wrmsr_fence() invoked by x2apic_send_IPI(). With the fix to skip weak_wrmsr_fence() *) Performance measured using ipi-bench for x2AVIC: Average Latency: 1117.44ns [Time to send IPI from one vCPU to another vCPU] Cumulative throughput: 42.9608M/s [Total number of IPIs sent in a second from 48 vCPUs simultaneously] Comparing the performance of x2AVIC with and without the fix, it can be seen the performance improves by ~4%. Performance captured using an unmodified ipi-bench using the 'mesh-ipi' option with and without weak_wrmsr_fence() on a Zen4 system also showed significant performance improvement without weak_wrmsr_fence(). The 'mesh-ipi' option ignores CCX or CCD and just picks random vCPU. Average throughput (10 iterations) with weak_wrmsr_fence(), Cumulative throughput: 4933374 IPI/s Average throughput (10 iterations) without weak_wrmsr_fence(), Cumulative throughput: 6355156 IPI/s [1] https://github.com/bytedance/kvm-utils/tree/master/microbenchmark/ipi-bench Signed-off-by: Borislav Petkov (AMD) --- arch/x86/include/asm/barrier.h | 18 ------------------ arch/x86/include/asm/processor.h | 18 ++++++++++++++++++ 2 files changed, 18 insertions(+), 18 deletions(-) diff --git a/arch/x86/include/asm/barrier.h b/arch/x86/include/asm/barrier.h index 35389b2af88e..0216f63a366b 100644 --- a/arch/x86/include/asm/barrier.h +++ b/arch/x86/include/asm/barrier.h @@ -81,22 +81,4 @@ do { \ #include -/* - * Make previous memory operations globally visible before - * a WRMSR. - * - * MFENCE makes writes visible, but only affects load/store - * instructions. WRMSR is unfortunately not a load/store - * instruction and is unaffected by MFENCE. The LFENCE ensures - * that the WRMSR is not reordered. - * - * Most WRMSRs are full serializing instructions themselves and - * do not require this barrier. This is only required for the - * IA32_TSC_DEADLINE and X2APIC MSRs. - */ -static inline void weak_wrmsr_fence(void) -{ - asm volatile("mfence; lfence" : : : "memory"); -} - #endif /* _ASM_X86_BARRIER_H */ diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index 4b130d894cb6..a43ff8d0ad24 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -752,4 +752,22 @@ enum mds_mitigations { extern bool gds_ucode_mitigated(void); +/* + * Make previous memory operations globally visible before + * a WRMSR. + * + * MFENCE makes writes visible, but only affects load/store + * instructions. WRMSR is unfortunately not a load/store + * instruction and is unaffected by MFENCE. The LFENCE ensures + * that the WRMSR is not reordered. + * + * Most WRMSRs are full serializing instructions themselves and + * do not require this barrier. This is only required for the + * IA32_TSC_DEADLINE and X2APIC MSRs. + */ +static inline void weak_wrmsr_fence(void) +{ + alternative("mfence; lfence", "", ALT_VENDOR(X86_VENDOR_AMD)); +} + #endif /* _ASM_X86_PROCESSOR_H */ -- 2.42.0.rc0.25.ga82fb66fed25 -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette