Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Mon, 2 Mar 2020 17:35:16 +0100
From:   Peter Zijlstra <peterz@infradead.org>
To:     Thomas Gleixner <tglx@linutronix.de>
Cc:     Jan Kiszka <jan.kiszka@siemens.com>, x86 <x86@kernel.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        "H. Peter Anvin" <hpa@zytor.com>
Subject: Re: x2apic_wrmsr_fence vs. Intel manual
Message-ID: <20200302163516.GB2579@hirez.programming.kicks-ass.net>
References: <783add60-f6c7-c8c6-b369-42e5ebfbf8c9@siemens.com>
 <87lfoienjp.fsf@nanos.tec.linutronix.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87lfoienjp.fsf@nanos.tec.linutronix.de>
User-Agent: Mutt/1.10.1 (2018-07-13)
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Mon, Mar 02, 2020 at 05:20:26PM +0100, Thomas Gleixner wrote:
> Jan Kiszka <jan.kiszka@siemens.com> writes:
> > as I generated a nice bug around fence vs. x2apic icr writes, I studied 
> > the kernel code and the Intel manual in this regard more closely. But 
> > there is a discrepancy:
> >
> > arch/x86/include/asm/apic.h:
> >
> > /*
> >  * Make previous memory operations globally visible before
> >  * sending the IPI through x2apic wrmsr. We need a serializing instruction or
> >  * mfence for this.
> >  */
> > static inline void x2apic_wrmsr_fence(void)
> > {
> >         asm volatile("mfence" : : : "memory");
> > }
> >
> > Intel SDM, 10.12.3 MSR Access in x2APIC Mode:
> >
> > "A WRMSR to an APIC register may complete before all preceding stores 
> > are globally visible; software can prevent this by inserting a 
> > serializing instruction or the sequence MFENCE;LFENCE before the WRMSR."
> >
> > The former dates back to ce4e240c279a, but that commit does not mention 
> > why lfence is not needed. Did the manual read differently back then? Or 
> > why are we safe? To my reading of lfence, it also has a certain 
> > instruction serializing effect that mfence does not have.
> 
> The 2011 SDM says:
> 
>   A WRMSR to an APIC register may complete before all preceding stores
>   are globally visible; software can prevent this by inserting a
>   serializing instruction, an SFENCE, or an MFENCE before the WRMSR.
> 
> Sigh....

*groan*, The only way I can explain this is...

... because we changed the definition of LFENCE from:

 - wait until completion of all prior LOADs

to

 - wait until completion of all prior instructions

Because Spectre (and because apparently it was implemented that way,
mostly). It could be that MFENCE, which is basically a completion
barrier for all prior LOADs *AND* STOREs, is no longer a stict superset
of LFENCE anymore...

Which makes the otherwise perverted sequence: MFENCE;LFENCE, actually
mean something :/

la-la-la

Would be good to have that clarified though.