Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Mihai Carabas <mihai.carabas@oracle.com>
To:     linux-kernel@vger.kernel.org
Cc:     bp@alien8.de, ashok.raj@intel.com, boris.ostrovsky@oracle.com,
        konrad.wilk@oracle.com, patrick.colp@oracle.com,
        kanth.ghatraju@oracle.com, Jon.Grimm@amd.com,
        Thomas.Lendacky@amd.com, mihai.carabas@oracle.com
Subject: [PATCH] Parallel microcode update in Linux
Date:   Thu, 22 Aug 2019 23:43:46 +0300
Message-Id: <1566506627-16536-1-git-send-email-mihai.carabas@oracle.com>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

This patch enables parallel microcode loading. In order to measure the
improvements of parallel vs serial, we have used the following diff:

diff --git a/arch/x86/kernel/cpu/microcode/core.c b/arch/x86/kernel/cpu/microcode/core.c
index 577b223..1ea08d8 100644
--- a/arch/x86/kernel/cpu/microcode/core.c
+++ b/arch/x86/kernel/cpu/microcode/core.c
@@ -614,17 +614,25 @@ static int __reload_late(void *info)
  */
 static int microcode_reload_late(void)
 {
+       u64 p0, p1;
        int ret;

        atomic_set(&late_cpus_in,  0);
        atomic_set(&late_cpus_out, 0);

+       p0 = rdtsc_ordered();
+
        ret = stop_machine_cpuslocked(__reload_late, NULL, cpu_online_mask);
+
+       p1 = rdtsc_ordered();
+
        if (ret > 0)
                microcode_check();

        pr_info("Reload completed, microcode revision: 0x%x\n", boot_cpu_data.microcode);

+       pr_info("p0: %lld, p1: %lld, diff: %lld\n", p0, p1, p1 - p0);
+
        return ret;
 }

We have used a machine with a broken microcode in BIOS and no microcode in
initramfs (to bypass early loading).

Here are the results for parallel loading (we made two measurements):

[root@ovs108 ~]# uname -a
Linux ovs108 5.3.0-rc5.master.parallel.el7.dev.x86_64 #1 SMP Thu Aug 22 10:17:04 GMT 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@ovs108 ~]# dmesg | grep microcode
[    0.000000] Intel Spectre v2 broken microcode detected; disabling Speculation Control
[    0.197658] MDS: Vulnerable: Clear CPU buffers attempted, no microcode
[    2.114135] microcode: sig=0x50654, pf=0x80, revision=0x200003a
[    2.117555] microcode: Microcode Update Driver: v2.2.
[   18.197760] microcode: updated to revision 0x200005e, date = 2019-04-02
[   18.201225] x86/CPU: CPU features have changed after loading microcode, but might not take effect.
[   18.201230] microcode: Reload completed, microcode revision: 0x200005e
[   18.201232] microcode: p0: 118138123843052, p1: 118138153732656, diff: 29889604


[root@ovs108 ~]# dmesg | grep microcode
[    0.000000] Intel Spectre v2 broken microcode detected; disabling Speculation Control
[    0.195218] MDS: Vulnerable: Clear CPU buffers attempted, no microcode
[    2.111882] microcode: sig=0x50654, pf=0x80, revision=0x200003a
[    2.115265] microcode: Microcode Update Driver: v2.2.
[   18.033397] microcode: updated to revision 0x200005e, date = 2019-04-02
[   18.036590] x86/CPU: CPU features have changed after loading microcode, but might not take effect.
[   18.036595] microcode: Reload completed, microcode revision: 0x200005e
[   18.036597] microcode: p0: 118947162428414, p1: 118947191490162, diff: 29061748

Here are the results of serial loading:

[root@ovs108 ~]# uname -a
Linux ovs108 5.3.0-rc5.master.serial.el7.dev.x86_64 #1 SMP Thu Aug 22 12:22:18 GMT 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@ovs108 ~]# dmesg | grep microcode
[    0.000000] Intel Spectre v2 broken microcode detected; disabling Speculation Control
[    0.195158] MDS: Vulnerable: Clear CPU buffers attempted, no microcode
[    2.111353] microcode: sig=0x50654, pf=0x80, revision=0x200003a
[    2.114834] microcode: Microcode Update Driver: v2.2.
[   17.542518] microcode: updated to revision 0x200005e, date = 2019-04-02
[   17.898365] x86/CPU: CPU features have changed after loading microcode, but might not take effect.
[   17.898370] microcode: Reload completed, microcode revision: 0x200005e
[   17.898372] microcode: p0: 149220216047388, p1: 149221058945422, diff: 842898034

[root@ovs108 ~]# dmesg | grep microcode
[    0.000000] Intel Spectre v2 broken microcode detected; disabling Speculation Control
[    0.197158] MDS: Vulnerable: Clear CPU buffers attempted, no microcode
[    2.114005] microcode: sig=0x50654, pf=0x80, revision=0x200003a
[    2.117451] microcode: Microcode Update Driver: v2.2.
[   17.732026] microcode: updated to revision 0x200005e, date = 2019-04-02
[   18.041398] x86/CPU: CPU features have changed after loading microcode, but might not take effect.
[   18.041404] microcode: Reload completed, microcode revision: 0x200005e
[   18.041407] microcode: p0: 149835792698162, p1: 149836532930286, diff: 740232124


One can see that the difference is an order magnitude.

---

I also tested microcode loading with cpu hotplug.
- I unplugged the last two CPUs  (basically the last core with 2 hyperthreads)
[ 1077.756759] IRQ 324: no longer affine to CPU71
[ 1077.756889] IRQ 619: no longer affine to CPU71
[ 1077.756908] IRQ 645: no longer affine to CPU71
[ 1077.761213] smpboot: CPU 71 is now offline
[ 1082.702759] IRQ 289: no longer affine to CPU70
[ 1082.702771] IRQ 305: no longer affine to CPU70
[ 1082.702827] IRQ 521: no longer affine to CPU70
[ 1082.702860] IRQ 636: no longer affine to CPU70
[ 1082.702876] IRQ 679: no longer affine to CPU70
[ 1082.706897] smpboot: CPU 70 is now offline

- I did the microcode update:
[ 1123.818741] microcode: updated to revision 0x200005e, date = 2019-04-02
[ 1123.821013] x86/CPU: CPU features have changed after loading microcode, but might not take effect.
[ 1123.821014] x86/CPU: Please consider either early loading through initrd/built-in or a potential BIOS update.
[ 1123.821014] microcode: Reload completed, microcode revision: 0x200005e
[ 1123.821015] microcode: p0: 197460831869308, p1: 197460853607904, diff: 21738596

- Than I onlined CPU 70/71
[ 1151.170814] smpboot: Booting Node 1 Processor 70 APIC 0x75
[ 1151.177199] microcode: sig=0x50654, pf=0x80, revision=0x200005e
[ 1182.523811] smpboot: Booting Node 1 Processor 71 APIC 0x77

root@ovs108 ~]# cat /proc/cpuinfo | tr "\t" " " | grep -A 6 "processor : 70" | grep microcode
microcode : 0x200005e

[root@ovs108 ~]# cat /proc/cpuinfo | tr "\t" " " | grep -A 6 "processor : 71" | grep microcode
microcode : 0x200005e


We can see that both CPUs have been updated to the same microcode revision.

Thank you,
Mihai

Ashok Raj (1):
  x86/microcode: Update microcode for all cores in parallel

 arch/x86/kernel/cpu/microcode/core.c  | 44 ++++++++++++++++++++++++-----------
 arch/x86/kernel/cpu/microcode/intel.c | 14 ++++-------
 2 files changed, 36 insertions(+), 22 deletions(-)

-- 
1.8.3.1