References: <1447099142-10220-1-git-send-email-gratian.crisan@ni.com> <20151109220232.GO17308@twins.programming.kicks-ass.net> <CAKA=qzarnUUmZb7DQE+u0Dei3F+FQNoL2bak_-dV9D9+3L=itQ@mail.gmail.com> <CAKA=qzbqN--3Ra+syiwM17z_Nue9aH9Ec1-pz-QwWt7OCHcbrA@mail.gmail.com> <87mvultz5f.fsf@spline.amer.corp.natinst.com> <CAKA=qzZNQqzL2sN5WFHog42ak3tsBmj6-Jy=nBoLC8EXn+aOdw@mail.gmail.com> <87io58tufu.fsf@spline.amer.corp.natinst.com> <20151113204338.GA9928@worktop.ger.corp.intel.com>
User-agent: mu4e 0.9.13; emacs 24.3.1
From: Gratian Crisan <gratian.crisan@ni.com>
To: Peter Zijlstra <peterz@infradead.org>
CC: Gratian Crisan <gratian.crisan@ni.com>, Josh Hunt <joshhunt00@gmail.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        LKML <linux-kernel@vger.kernel.org>, Ingo Molnar <mingo@redhat.com>,
        "H . Peter Anvin" <hpa@zytor.com>, <x86@kernel.org>,
        Borislav Petkov <bp@alien8.de>, Josh Cartwright <joshc@ni.com>,
        <gratian@gmail.com>
Subject: Re: [RFC PATCH] tsc: synchronize TSCs on buggy Intel Xeon E5 CPUs with offset error
In-Reply-To: <20151113204338.GA9928@worktop.ger.corp.intel.com>
Date: Tue, 17 Nov 2015 10:38:21 -0600
Message-ID: <87r3jofuo2.fsf@spline.amer.corp.natinst.com>
MIME-Version: 1.0
Content-Type: text/plain
SpamDiagnosticOutput: 1:23
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Nov 2015 16:38:26.3643 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL2PR04MB850
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 2310
Lines: 37


Peter Zijlstra writes:

> On Wed, Nov 11, 2015 at 09:41:25AM -0600, Gratian Crisan wrote:
>> I also wrote a small C utility[1], with a bit of code borrowed from the
>> kernel, for reading the TSC on all CPUs. It starts a high priority
>> thread per CPU, tries to synchronize them and prints out the TSC values
>> and their offset with regards to CPU0.
>> It can be called from a SysV init shell script[2] at the beginning of
>> the boot process and right before a reboot to save the values in a file.
>
> Could you also read and print TSC_ADJUST (msr 0x3b) ? This would tell us
> if for example your BIOS messed it up.

Good call on the TSC_ADJUST. The BIOS seems to set it for CPU0 but not
for any of the other ones. I'll bug our BIOS guys about it.
Here's how the data looks after a couple reboots:

stop      : 127385698358[0] 741784252175365[741656866477007] 741784252175432[741656866477074] 741784252175471[741656866477113] 741784252175349[741656866476991] 741784252175458[741656866477100] 741784252175285[741656866476927] 741784252175501[741656866477143] 
TSC_ADJUST: -741656866477048 0 0 0 0 0 0 0 

start     : 47601069657[0] 741849504816842[741801903747185] 741849504816884[741801903747227] 741849504817004[741801903747347] 741849504817113[741801903747456] 741849504817051[741801903747394] 741849504816746[741801903747089] 741849504816962[741801903747305] 
TSC_ADJUST: -741801903747272 0 0 0 0 0 0 0 

stop      : 127495422447[0] 741929399169793[741801903747346] 741929399169821[741801903747374] 741929399169739[741801903747292] 741929399169767[741801903747320] 741929399169657[741801903747210] 741929399169612[741801903747165] 741929399169679[741801903747232] 
TSC_ADJUST: -741801903747272 0 0 0 0 0 0 0 

start     : 47522880051[0] 741994508088208[741946985208157] 741994508088258[741946985208207] 741994508088305[741946985208254] 741994508088110[741946985208059] 741994508088052[741946985208001] 741994508088020[741946985207969] 741994508087930[741946985207879] 
TSC_ADJUST: -741946985208111 0 0 0 0 0 0 0

Thanks a lot for helping with this,
-Gratian
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/