Date: Thu, 28 Sep 2017 14:12:04 +0200 (CEST)
From: Thomas Gleixner <tglx@linutronix.de>
To: Peter Zijlstra <peterz@infradead.org>
cc: Dou Liyang <douly.fnst@cn.fujitsu.com>,
        Pasha Tatashin <pasha.tatashin@oracle.com>, linux@armlinux.org.uk,
        schwidefsky@de.ibm.com, heiko.carstens@de.ibm.com,
        john.stultz@linaro.org, sboyd@codeaurora.org, x86@kernel.org,
        linux-kernel@vger.kernel.org, mingo@redhat.com, hpa@zytor.com
Subject: Re: [PATCH v6 1/4] sched/clock: interface to allow timestamps early
 in boot
In-Reply-To: <20170928115814.hnpah2bgkuaegtct@hirez.programming.kicks-ass.net>
Message-ID: <alpine.DEB.2.20.1709281410310.1879@nanos>
References: <1504116205-355281-1-git-send-email-pasha.tatashin@oracle.com> <1504116205-355281-2-git-send-email-pasha.tatashin@oracle.com> <20170927125857.yvwefpejzskiduwu@hirez.programming.kicks-ass.net> <20170927131003.dxbvu7frcqgtiwaz@hirez.programming.kicks-ass.net>
 <aed54345-b752-e93a-8be7-96df9b083f6c@oracle.com> <4a62156c-ea0f-749f-e21b-37919dfda1df@cn.fujitsu.com> <20170927180548.GM17526@worktop.programming.kicks-ass.net> <20170927180950.GD439@worktop> <1dd60ace-c2aa-6c88-d4b0-cba934be4b79@cn.fujitsu.com>
 <20170928115814.hnpah2bgkuaegtct@hirez.programming.kicks-ass.net>
User-Agent: Alpine 2.20 (DEB 67 2015-01-07)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 2294
Lines: 52

On Thu, 28 Sep 2017, Peter Zijlstra wrote:

> On Thu, Sep 28, 2017 at 06:03:05PM +0800, Dou Liyang wrote:
> > At 09/28/2017 02:09 AM, Peter Zijlstra wrote:
> > > On Wed, Sep 27, 2017 at 08:05:48PM +0200, Peter Zijlstra wrote:
> > > > On Wed, Sep 27, 2017 at 09:52:36PM +0800, Dou Liyang wrote:
> > > > > We do not want to do that. Because, we use "notsc" to support Dynamic
> > > > > Reconfiguration[1].
> > > > > 
> > > > > AFAIK, this feature enables hot-add system board which contains CPUs
> > > > > and memories. But the CPUs in different board may have different TSCs
> > > > > which are not consistent with the TSC from the existing CPUs. If we hot-add
> > > > > a board directly, the machine may happen the inconsistency of
> > > > > TSC.
> > > > > 
> > > > > We make our effort to specify the same TSC value as existing one through
> > > > > hardware and firmware, but it is hard. So we recommend to specify
> > > > > "notsc" option in command line for users who want to use Dynamic
> > > > > Reconfiguration.
> > > > 
> > > > Oh gawd, that's horrific. And in my book a good reason to kill that
> > > > option.
> > > 
> > > That is, even with unsynchronized TSC we're better off using RDTSC. The
> > > whole mess in kernel/sched/clock.c is all about getting semi sensible
> > > results out of unsynchronized TSC.
> > > 
> > 
> > It will be best if we can support TSC sync capability in x86, but seems
> > is not easy.
> 
> Sure, your hardware achieving sync would be best, but even if it does
> not, we can still use TSC. Using notsc simple because you fail to sync
> TSCs is quite crazy.
> 
> The thing is, we need to support unsync'ed TSC in any case, because
> older chips (pre Nehalem) didn't have synchronized TSC in any case, and
> it still happens on recent chips if the BIOS mucks it up, which happens
> surprisingly often :-(
> 
> I would suggest you try your reconfigurable setup with "tsc=unstable"
> and see if that works for you. That marks the TSC unconditionally
> unstable at boot and avoids any further wobbles once the TSC watchdog
> notices (although that too _should_ more or less work).

That should do the trick nicely and we might just end up converting notsc
to tsc=unstable silently so we can avoid the bike shed discussions about
removing it.

Thanks,

	tglx