Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: x2apic_wrmsr_fence vs. Intel manual
To:     Thomas Gleixner <tglx@linutronix.de>, x86 <x86@kernel.org>
Cc:     Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
References: <783add60-f6c7-c8c6-b369-42e5ebfbf8c9@siemens.com>
 <87lfoienjp.fsf@nanos.tec.linutronix.de>
From:   Jan Kiszka <jan.kiszka@siemens.com>
Message-ID: <9a7c60d5-bbb2-98a4-4564-844f161b212c@siemens.com>
Date:   Mon, 2 Mar 2020 17:33:50 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.5.0
MIME-Version: 1.0
In-Reply-To: <87lfoienjp.fsf@nanos.tec.linutronix.de>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On 02.03.20 17:20, Thomas Gleixner wrote:
> Jan Kiszka <jan.kiszka@siemens.com> writes:
>> as I generated a nice bug around fence vs. x2apic icr writes, I studied
>> the kernel code and the Intel manual in this regard more closely. But
>> there is a discrepancy:
>>
>> arch/x86/include/asm/apic.h:
>>
>> /*
>>   * Make previous memory operations globally visible before
>>   * sending the IPI through x2apic wrmsr. We need a serializing instruction or
>>   * mfence for this.
>>   */
>> static inline void x2apic_wrmsr_fence(void)
>> {
>>          asm volatile("mfence" : : : "memory");
>> }
>>
>> Intel SDM, 10.12.3 MSR Access in x2APIC Mode:
>>
>> "A WRMSR to an APIC register may complete before all preceding stores
>> are globally visible; software can prevent this by inserting a
>> serializing instruction or the sequence MFENCE;LFENCE before the WRMSR."
>>
>> The former dates back to ce4e240c279a, but that commit does not mention
>> why lfence is not needed. Did the manual read differently back then? Or
>> why are we safe? To my reading of lfence, it also has a certain
>> instruction serializing effect that mfence does not have.
> 
> The 2011 SDM says:
> 
>    A WRMSR to an APIC register may complete before all preceding stores
>    are globally visible; software can prevent this by inserting a
>    serializing instruction, an SFENCE, or an MFENCE before the WRMSR.
> 
> Sigh....
> 

OK, that explains it. Curious since when (date and CPU generation) we 
unknowingly started to play roulette here.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux