Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 31 Jul 2023 20:02:46 +0800
From:   Feng Tang <feng.tang@intel.com>
To:     kernel test robot <oliver.sang@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>
CC:     <x86@kernel.org>, Thomas Gleixner <tglx@linutronix.de>,
        <oe-lkp@lists.linux.dev>, <lkp@intel.com>,
        <linux-kernel@vger.kernel.org>, <ying.huang@intel.com>,
        <fengwei.yin@intel.com>
Subject: Re: [linus:master] [x86/fpu]  b81fac906a:
 stress-ng.af-alg.ops_per_sec -34.6% regression
Message-ID: <ZMei5uMRwOJo2r6Z@feng-clx>
References: <202307192135.203ac24e-oliver.sang@intel.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <202307192135.203ac24e-oliver.sang@intel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?HYK/r5E2sRL2z5wVnxQ/yAhXxv5JqaaVR4MxKvit1joMvSFFRHxK/18Vx5vJ?=
 =?us-ascii?Q?JsxX7VExNV02IO3F614K2KYOj3OOJjM27dotxSPFuvXZ48jpJKzzSiUu0JzG?=
 =?us-ascii?Q?vI+wGQbbv6n5EuSVha1DPZ0kKeBVw5pGQ4jC9k7LJ5dgufUThK0Ldy9rSTGo?=
 =?us-ascii?Q?OkP0Vkik9B9062llSMUDoubeh+UX19pg4dcNAZ0Wp/p+c1bq5DU5gBzy1SQg?=
 =?us-ascii?Q?/unK4zv9Mqb2Bq4EB1mqfciKuHCut4sfQYLLtYJTAWRf4CGLseyxNdb5z8Dl?=
 =?us-ascii?Q?9hc4mpD0y8le93yPO8mjbYiZ2Bygd0640llRlMzhka/3Vtcs3JbLPDdbqUVO?=
 =?us-ascii?Q?6KbM/ZypS20rPA/cY8Eo3uTJ+Cz5VdCeDWpxn6EDhO6lybAOfv7gxmk4nVaO?=
 =?us-ascii?Q?iCdhUEV5Ai2Zy68ZheIGcULPyxyt4P2RH4e7XOcLXZKDROyBrMKdWsTaiauV?=
 =?us-ascii?Q?3CjNF1xClSA8Cbw98WTueZcBDz9jyOuMiXaCnNm2TIYUhe6nIhl2pMmBo5Tk?=
 =?us-ascii?Q?+VZHnD5aDbak9mMyj93o/NMZNxjd8oJ0wA7dxM7MtaPv5QiWStWqQlWh57Ly?=
 =?us-ascii?Q?sBoucFUlrBkqFCv72bPwzlWUqgRa6faKTQ9Lhn3nPyfrgeqpay7iZizoR/JP?=
 =?us-ascii?Q?fkrlphQtwZuQniQSdIRwLdgJSyYLsUCTuGTrunQlcR+xMKCf2Dx5E//4bOr2?=
 =?us-ascii?Q?0J0oxVqjZX/MgVnXRtd97EYOYaZjg61Ezjn9IUHI/+4wbenrwyApDseEpCdl?=
 =?us-ascii?Q?GJmGHOAdMN6G/dOk19zqHbqKShTM0MtIig4MX769tvNqqe7YCJYFFbIiFNDw?=
 =?us-ascii?Q?YfGFut+ps7/GqdSKrUaRy/sU38m0heFhbjMKqA8iDOPi2Omb1mINuODoq4Ov?=
 =?us-ascii?Q?SdbVmj6N9EB4YoP/1Cqdlx262hwzVdgurLS0UJKaRj1FL403bEEXEy06EOZS?=
 =?us-ascii?Q?SNKdPhFLYAmxU88XGsnjNzR7xoerLLYsQgfdRMnrJePTEdpoDy0dX4/pE4wc?=
 =?us-ascii?Q?J6kfp23oOOkYDtL7O9568+4OzDIGscUhAEhxUvYwU3gYh85tnbD7XFGTRmGE?=
 =?us-ascii?Q?vjK/6PNSrzMUhlbHdSxQ3jXuKoCSf2KRwS0NaX+y6yXP2OsJiYGJXRDI/XQa?=
 =?us-ascii?Q?RiWlJCFyhx/CsntWfG5F5luSgPclU6qisqerwvyaxC/6IvpKUeXwlUeFcH1l?=
 =?us-ascii?Q?A/I8BKxP5f8HTJHnqHjjF8jT7SJE8sYfuMc23tBd9GW+cgb80/3L9nN+vDAD?=
 =?us-ascii?Q?a6zlIPWD3a7zoxPhsa3HGfw6+k1WXavNhpzRSpROyoKoE/vU6nqDnjEFk7Pt?=
 =?us-ascii?Q?rrTr2UWegKhPnzn/2utmVgJD5kgNfc1FV8RvipL6AwVAnef/HQuLfYeztTtE?=
 =?us-ascii?Q?CvgjR5GGTZh/M0fLpRx4ptRN1ncJZtWbRpGaQEZRJp+VY9wNzb3vSyOKv33/?=
 =?us-ascii?Q?QXZOgfztxGzxhfAAOLfGrDORjr3EwYqoNyZePrCTjF4Ax2NUf6xWbK98BdAq?=
 =?us-ascii?Q?CKbJmu+Ql+T+WpTRUXwZxw68u3gpPhsgXJoJ+N4Jb85ME/GIDOeQQG+sv0Yb?=
 =?us-ascii?Q?rWRrGNQv4Kp6JzgQTBYLKrFEMujFKNxAs8l31yI+?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 8f76da03-0de3-4ab2-799a-08db91bf250f
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6304.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Jul 2023 12:10:35.3861
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Gc0gI8soHj8T1Qn/VYbaHL0s1OFyEY6kBON8gsTxtq5TMifwCCKuKD7xdIk5/cc1rxo56AY5CP7nQ2yhIpzB3Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR11MB8433
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,
        RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Jul 19, 2023 at 10:07:52PM +0800, kernel test robot wrote:
> 
> 
> Hello,
> 
> kernel test robot noticed a -34.6% regression of stress-ng.af-alg.ops_per_sec on:
> 
> 
> commit: b81fac906a8f9e682e513ddd95697ec7a20878d4 ("x86/fpu: Move FPU initialization into arch_cpu_finalize_init()")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> 
> still regression on:
> fix commit: fe3e0a13e597c1c8617814bf9b42ab732db5c26e
> linus/master: ccff6d117d8dc8d8d86e8695a75e5f8b01e573bf
> 
> testcase: stress-ng
> test machine: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
> parameters:
> 
> 	nr_threads: 10%
> 	disk: 1HDD
> 	testtime: 60s
> 	fs: ext4
> 	class: os
> 	test: af-alg
> 	cpufreq_governor: performance
> 
> 
> besides, we noticed there are below information in dmesg (attached kmsg.xz),
> which we didn't see in dmesg for parent:
> 
> kern  :info  : [   65.660392] AVX or AES-NI instructions are not detected.
> user  :notice: [   65.661743] stress-ng: info:  [4738] af-alg: 9 cryptographic algorithms are internal and may be unused
> 
> kern  :info  : [   65.720205] AVX or AES-NI instructions are not detected.
> kern  :info  : [   65.820334] AVX2 instructions are not detected.
> kern  :info  : [   66.130644] AVX or AES-NI instructions are not detected.
> kern  :info  : [   66.167749] AVX or AES-NI instructions are not detected.
> kern  :info  : [   66.213178] AVX2 instructions are not detected.
> kern  :info  : [   66.318235] AVX or AES-NI instructions are not detected.
> kern  :info  : [   66.360766] AVX or AES-NI instructions are not detected.
> kern  :info  : [   66.437318] AVX2 instructions are not detected.
> kern  :info  : [   66.683666] AVX or AES-NI instructions are not detected.
> user  :notice: [   66.713976] BMC ARP Control         : ARP Responses Enabled, Gratuitous ARP Disabled
> 
> kern  :info  : [   66.738790] AVX or AES-NI instructions are not detected.
> kern  :info  : [   66.822332] AVX2 instructions are not detected.
> kern  :info  : [   67.083360] AVX or AES-NI instructions are not detected.
> kern  :info  : [   67.122804] AVX or AES-NI instructions are not detected.

I think the regression is related with the commit, that commit
b81fac906a8f changes the FPU init order, and cause the 'X86_FEATURE_OSXSAVE'
missed the cpu capability setup, thus all later check of
boot_cpu_has(X86_FEATURE_OSXSAVE) will return false.

   arch_cpu_finalize_init
       identify_boot_cpu
	   identify_cpu
	       generic_identify
                   get_cpu_cap --> setup cpu capability
       ...
       fpu__init_cpu
           fpu__init_cpu_xstate
               cr4_set_bits(X86_CR4_OSXSAVE);

Many security moduels in arch/x86 have dependency on X86_FEATURE_OSXSAVE,
and will fail to load later. While this 'af-alg' case involves secuirty
operation and tries to load those modules but fail, which cause the
regression.

Setting the 'OSXSVAE' feature bit after OSXSAVE enabling can restore
the regression, like the below RFC patch. pls help to review, thanks!

- Feng

---

From 62d05a24cae9198b0ed45a8857edbe4c9c4184f9 Mon Sep 17 00:00:00 2001
From: Feng Tang <feng.tang@intel.com>
Date: Mon, 31 Jul 2023 16:45:34 +0800
Subject: [PATCH RFC] x86/fpu: set X86_FEATURE_OSXSAVE feature after enabling
 OSXSAVE in CR4

Commit b81fac906a8f ("x86/fpu: Move FPU initialization into
arch_cpu_finalize_init()" optimized the FPU init order, and moves the
CR4_OSXSAVE enabling into a later flace.

   arch_cpu_finalize_init
       identify_boot_cpu
	   identify_cpu
	       generic_identify
                   get_cpu_cap --> setup cpu capability
       ...
       fpu__init_cpu
           fpu__init_cpu_xstate
               cr4_set_bits(X86_CR4_OSXSAVE);

This makes 'X86_FEATURE_OSXSAVE' missed in cpu capability setup. Many
security module like 'camellia_aesni_avx_x86_64' depends on this feature
and will fail to load after the commit. 0Day caught a 34% regression
in stress-ng's 'af-alg' test case [1].

So set X86_FEATURE_OSXSAVE feature after OSXSAVE enabling to fix it.

[1]. https://lore.kernel.org/lkml/202307192135.203ac24e-oliver.sang@intel.com/

Fixes: b81fac906a8f ("x86/fpu: Move FPU initialization into arch_cpu_finalize_init()")
Reported-by: kernel test robot <oliver.sang@intel.com>
Signed-off-by: Feng Tang <feng.tang@intel.com>
---
 arch/x86/kernel/fpu/xstate.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c
index 0bab497c9436..8ebea0d522d2 100644
--- a/arch/x86/kernel/fpu/xstate.c
+++ b/arch/x86/kernel/fpu/xstate.c
@@ -173,6 +173,9 @@ void fpu__init_cpu_xstate(void)
 
 	cr4_set_bits(X86_CR4_OSXSAVE);
 
+	if (!boot_cpu_has(X86_FEATURE_OSXSAVE))
+		setup_force_cpu_cap(X86_FEATURE_OSXSAVE);
+
 	/*
 	 * Must happen after CR4 setup and before xsetbv() to allow KVM
 	 * lazy passthrough.  Write independent of the dynamic state static
-- 
2.27.0