Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Fri, 6 Sep 2019 18:17:35 +0200
From:   Borislav Petkov <bp@alien8.de>
To:     Johannes Erdfelt <johannes@erdfelt.com>
Cc:     Thomas Gleixner <tglx@linutronix.de>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        Boris Ostrovsky <boris.ostrovsky@oracle.com>,
        Mihai Carabas <mihai.carabas@oracle.com>,
        "H. Peter Anvin" <hpa@zytor.com>, Ingo Molnar <mingo@redhat.com>,
        Jon Grimm <Jon.Grimm@amd.com>, kanth.ghatraju@oracle.com,
        konrad.wilk@oracle.com, patrick.colp@oracle.com,
        Tom Lendacky <thomas.lendacky@amd.com>,
        x86-ml <x86@kernel.org>, linux-kernel@vger.kernel.org
Subject: Re: [PATCH] x86/microcode: Add an option to reload microcode even if
 revision is unchanged
Message-ID: <20190906161735.GH19008@zn.tnic>
References: <D8A3D2BD-1FD4-4183-8663-3EF02A6099F3@alien8.de>
 <20190905002132.GA26568@otc-nc-03>
 <20190905072029.GB19246@zn.tnic>
 <20190905194044.GA3663@otc-nc-03>
 <alpine.DEB.2.21.1909052316130.1902@nanos.tec.linutronix.de>
 <20190905222706.GA4422@otc-nc-03>
 <alpine.DEB.2.21.1909061431330.1902@nanos.tec.linutronix.de>
 <20190906144039.GA29569@sventech.com>
 <20190906151617.GE19008@zn.tnic>
 <20190906154618.GB29569@sventech.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <20190906154618.GB29569@sventech.com>
User-Agent: Mutt/1.10.1 (2018-07-13)
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Fri, Sep 06, 2019 at 08:46:18AM -0700, Johannes Erdfelt wrote:
> That said, we very much rely on late microcode loading and it has helped
> us and our customers significantly.

You do realize that you rely on an update method which *won't* work in
all possible cases and then you *will* have to reboot if the microcode
patching *must* happen early, do you?

> It's really easy to say "fix your infrastructure" when you're not
> running that infrastructure.

I'm not saying you should fix your infrastructure now - I'm saying you
should keep that in mind when thinking whether to rely more on late
loading or not. Who knows, maybe newer generation machines in the fleet
could do load balancing, live migration, whatever fancy new cloud stuff
it is, to facilitate a proper reboot.

Or someone could rewrite arch/x86/ to rediscover new features upon a
microcode reload or a feature disabling. And do that in a clean way. Who
knows...

> Reboots suck. Customers hate it. Operations hates it. When you get into
> the number of hosts we have, you run into all kinds of weird failure
> scenarios. (What do you mean that the NIC that was working just fine
> before the reboot is no longer seen on the PCI bus?)

Yeah, I've heard all the stories.

> The more reboots we can avoid, the better it is for us and our
> customers.

So how do you update the kernels on those machines? Or you live-patch in
the new functionality too?

> I understand that it could be unsafe to late load some rare microcode
> updates (theoretical or not). However, that is certainly the exception.
> We have done this multiple times on our fleet and we plan to continue
> doing so in the future.

The fact that it has worked for you does not make it right. It won't
magically become safe, as tglx said.

But since you do custom development, you should be fine, it seems.

Practically speaking, late loading probably won't disappear as it is
being used apparently. Just don't expect that it will get "extended" if
that extension brings with itself fallout and duct tape fixes left and
right.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette