Date: Wed, 5 Aug 2015 09:25:46 -0700
From: Shaohua Li <shli@fb.com>
To: Ingo Molnar <mingo@kernel.org>
CC: Thomas Gleixner <tglx@linutronix.de>, <x86@kernel.org>,
        <linux-kernel@vger.kernel.org>, <Kernel-team@fb.com>,
        Suresh Siddha <suresh.b.siddha@intel.com>,
        "H. Peter Anvin" <hpa@zytor.com>, <stable@vger.kernel.org>,
        <ak@linux.intel.com>, <lenb@kernel.org>, <fenghua.yu@intel.com>
Subject: Re: [PATCH] x86: serialize LVTT and TSC_DEADLINE write
Message-ID: <20150805162535.GA2461245@devbig257.prn2.facebook.com>
References: <75ed9226b028a31e37861fbba51cdffbcfe04eda.1438300532.git.shli@fb.com>
 <alpine.DEB.2.11.1508011202080.3825@nanos>
 <20150802154948.GA1432307@devbig257.prn2.facebook.com>
 <alpine.DEB.2.11.1508022054080.3825@nanos>
 <20150803235801.GA1461084@devbig257.prn2.facebook.com>
 <20150805084424.GB21506@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <20150805084424.GB21506@gmail.com>
User-Agent: Mutt/1.5.20 (2009-12-10)
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 3306
Lines: 77

On Wed, Aug 05, 2015 at 10:44:24AM +0200, Ingo Molnar wrote:
> 
> * Shaohua Li <shli@fb.com> wrote:
> 
> > On Sun, Aug 02, 2015 at 09:41:08PM +0200, Thomas Gleixner wrote:
> > > On Sun, 2 Aug 2015, Shaohua Li wrote:
> > > 
> > > > On Sat, Aug 01, 2015 at 12:10:41PM +0200, Thomas Gleixner wrote:
> > > > > On Fri, 31 Jul 2015, Shaohua Li wrote:
> > > > > > @@ -336,6 +336,22 @@ static void __setup_APIC_LVTT(unsigned int clocks, int oneshot, int irqen)
> > > > > >  	apic_write(APIC_LVTT, lvtt_value);
> > > > > >  
> > > > > >  	if (lvtt_value & APIC_LVT_TIMER_TSCDEADLINE) {
> > > > > > +		u64 msr;
> > > > > > +
> > > > > > +		/*
> > > > > > +		 * See Intel SDM: TSC-Deadline Mode chapter. In xAPIC mode,
> > > > > > +		 * writing APIC LVTT and TSC_DEADLINE MSR isn't serialized.
> > > > > > +		 * This uses the algorithm described in Intel SDM to serialize
> > > > > > +		 * the two writes
> > > > > > +		 * */
> > > > > > +		while (1) {
> > > > > > +			wrmsrl(MSR_IA32_TSC_DEADLINE, -1L);
> > > > > > +			rdmsrl(MSR_IA32_TSC_DEADLINE, msr);
> > > > > > +			if (msr)
> > > > > > +				break;
> > > > > > +		}
> > > > > > +		wrmsrl(MSR_IA32_TSC_DEADLINE, 0);
> > > > > 
> > > > > 
> > > > > I think this is exceptionally silly. A proper fence after the
> > > > > apic_write() should have the same effect.
> > > > 
> > > > Not sure what happens in the hardware, I could have a try of fence, but
> > > > I'd prefer using the algorithm Intel described. This is not a fast path,
> > > 
> > > s/algorithm/voodoo/
> > > 
> > > > the loop will exit immediately regardless the issue occurs anyway.
> > > 
> > > Well, the SDM also says:
> > > 
> > >  "To allow for efficient access to the APIC registers in x2APIC mode,
> > >   the serializing semantics of WRMSR are relaxed when writing to the
> > >   APIC registers. Thus, system software should not use “WRMSR to APIC
> > >   registers in x2APIC mode” as a serializing instruction. Read and write
> > >   accesses to the APIC registers will occur in program order. A WRMSR to
> > >   an APIC register may complete before all preceding stores are globally
> > >   visible; software can prevent this by inserting a serializing
> > >   instruction, an SFENCE, or an MFENCE before the WRMSR."
> > > 
> > > And that's what happens here. The write to the LVT has not yet hit the
> > > APIC, so the WRMSR has no effect.
> > 
> > What you quoted is for x2APIC, I didn't see similar description for
> > xAPIC.
> > 
> > Tested mfence here, it does work. But I'm not convinced it's the right thing. 
> > the xAPIC access is memory mapped IO, mfence is nothing related to it. [...]
> 
> Huh?
> 
> Can you cite the SDM that says that MFENCE will have no effect on instructions 
> that deal with accesses that are not to system RAM (i.e. are memory mapped IO)?

Hmm, I didn't mean mfence can't serialize the instructions. For a true
IO, a serialization can't guarantee device finishes the IO, we generally
read some safe IO registers to wait IO finish. I completely don't know
if this case fits here though.

Thanks,
Shaohua
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/