Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Content-Type: text/plain;
        charset=us-ascii
Mime-Version: 1.0 (1.0)
Subject: Re: [PATCH] Parallel microcode update in Linux
From:   Mihai Carabas <mihai.carabas@oracle.com>
In-Reply-To: <20190901172547.GD1047@bug>
Date:   Mon, 2 Sep 2019 10:27:06 +0300
Cc:     linux-kernel@vger.kernel.org, bp@alien8.de, ashok.raj@intel.com,
        boris.ostrovsky@oracle.com, konrad.wilk@oracle.com,
        patrick.colp@oracle.com, kanth.ghatraju@oracle.com,
        Jon.Grimm@amd.com, Thomas.Lendacky@amd.com
Content-Transfer-Encoding: quoted-printable
Message-Id: <5BACA033-7613-49A9-801C-9F75F4306909@oracle.com>
References: <1566506627-16536-1-git-send-email-mihai.carabas@oracle.com> <20190901172547.GD1047@bug>
To:     Pavel Machek <pavel@ucw.cz>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


> On 1 Sep 2019, at 20:25, Pavel Machek <pavel@ucw.cz> wrote:
>=20
> Hi!
>=20
>> +       u64 p0, p1;
>>        int ret;
>>=20
>>        atomic_set(&late_cpus_in,  0);
>>        atomic_set(&late_cpus_out, 0);
>>=20
>> +       p0 =3D rdtsc_ordered();
>> +
>>        ret =3D stop_machine_cpuslocked(__reload_late, NULL, cpu_online_ma=
sk);
>> +
>> +       p1 =3D rdtsc_ordered();
>> +
>>        if (ret > 0)
>>                microcode_check();
>>=20
>>        pr_info("Reload completed, microcode revision: 0x%x\n", boot_cpu_d=
ata.microcode);
>>=20
>> +       pr_info("p0: %lld, p1: %lld, diff: %lld\n", p0, p1, p1 - p0);
>> +
>>        return ret;
>> }
>>=20
>> We have used a machine with a broken microcode in BIOS and no microcode i=
n
>> initramfs (to bypass early loading).
>>=20
>> Here are the results for parallel loading (we made two measurements):
>=20
>> [   18.197760] microcode: updated to revision 0x200005e, date =3D 2019-04=
-02
>> [   18.201225] x86/CPU: CPU features have changed after loading microcode=
, but might not take effect.
>> [   18.201230] microcode: Reload completed, microcode revision: 0x200005e=

>> [   18.201232] microcode: p0: 118138123843052, p1: 118138153732656, diff:=
 29889604
>=20
>> Here are the results of serial loading:
>>=20
>> [   17.542518] microcode: updated to revision 0x200005e, date =3D 2019-04=
-02
>> [   17.898365] x86/CPU: CPU features have changed after loading microcode=
, but might not take effect.
>> [   17.898370] microcode: Reload completed, microcode revision: 0x200005e=

>> [   17.898372] microcode: p0: 149220216047388, p1: 149221058945422, diff:=
 842898034
>>=20
>> One can see that the difference is an order magnitude.
>=20
> Well, that's impressive, but it seems to finish 300 msec later? Where does=
 that difference
> come from / how much real time do you gain by this?

The difference comes from the large amount of cores/threads the machine has:=
 72 in this case, but there are machines with more. As the commit message sa=
ys initially the microcode was applied serially one by one and now the micro=
code is updated in parallel on all cores.

300ms seems nothing but it is enough to cause disruption in some critical se=
rvices (e.g. storage) - 300ms in which we do not execute anything on CPUs. A=
lso this 300ms is increasing when the machine is fully loaded with guests.

Thanks,
Mihai

>=20
> Best regards,
>                                    Pavel
>=20
> --=20
> (english) https://urldefense.proofpoint.com/v2/url?u=3Dhttp-3A__www.livejo=
urnal.com_-7Epavelmachek&d=3DDwIBAg&c=3DRoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65e=
apI_JnE&r=3DIOMTUEJr06tE0LeEzvwr_907ba6u9S5iDf7M8ZYjbGY&m=3Dcz26YweqnHS4QvZB=
i-1jNR8t7o3n04-8UsSBZqEQHgA&s=3D-nEQbDyJrDjKxyrt496frey_aMJHXmgMcm-hH0ewO7M&=
e=3D=20
> (cesky, pictures) https://urldefense.proofpoint.com/v2/url?u=3Dhttp-3A__at=
rey.karlin.mff.cuni.cz_-7Epavel_picture_horses_blog.html&d=3DDwIBAg&c=3DRoP1=
YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=3DIOMTUEJr06tE0LeEzvwr_907ba6u9S5i=
Df7M8ZYjbGY&m=3Dcz26YweqnHS4QvZBi-1jNR8t7o3n04-8UsSBZqEQHgA&s=3D0L72IdzqTDn_=
8PmDVcNxLAFbcYG1jRDN9ob8SZ18XTE&e=3D=20