Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754640AbcLNBg0 (ORCPT ); Tue, 13 Dec 2016 20:36:26 -0500 Received: from vie01a-dmta-ch01-1.mx.upcmail.net ([84.116.36.91]:49379 "EHLO vie01a-dmta-ch01-1.mx.upcmail.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754608AbcLNBgZ (ORCPT ); Tue, 13 Dec 2016 20:36:25 -0500 X-SourceIP: 77.56.147.151 X-Authenticated-Sender: rscheidegger_lists@hispeed.ch Subject: Re: [patch 0/2] tsc/adjust: Cure suspend/resume issues and prevent TSC deadline timer irq storm To: Thomas Gleixner References: <20161213131115.764824574@linutronix.de> <33d4286c-3f77-1274-34b7-bc62d2c146a4@hispeed.ch> Cc: LKML , x86@kernel.org, Peter Zijlstra , Borislav Petkov , Bruce Schlobohm , Kevin Stanton , Allen Hung From: Roland Scheidegger Message-ID: <357e0a0f-af6b-2a8e-2af0-b05652ccbb30@hispeed.ch> Date: Wed, 14 Dec 2016 02:36:21 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3246 Lines: 74 Am 13.12.2016 um 17:46 schrieb Thomas Gleixner: > On Tue, 13 Dec 2016, Roland Scheidegger wrote: > >> Am 13.12.2016 um 14:14 schrieb Thomas Gleixner: >>> Roland reported interesting TSC ADJUST register wreckage on his DELL >>> machine, which seems to populate that MSR with a random number generator. >> >> FWIW, I thought about the actual values some more and I don't actually >> think they are all that random any more: the behavior is consistent with >> the bios trying to zero the TSC of all cpus. If I understand this right, >> writing a zero to TSC would cause somewhat small negative values in the >> TSC_ADJ register at boot time, and larger negative values at suspend >> time (at least if the TSC just stops when suspended and isn't reset) - >> exactly what I'm seeing. >> (And of course the different TSC_ADJ values would be because the bios is >> writing TSC without any thoughts of synchronization, just one cpu after >> another). > > Yeah, that might be. Still it looks like random nonsense and definitely the > BIOS developers did not follow the secrit boot protocol. > >>> Deeper investagation into fixing this wreckage unearthed another special >>> feature which is designed by Intel: Negative TSC adjuste values cause >>> interrupt storms on the TSC deadline timer. Further details in patch 2/2 >> >> This actually looks like quite a serious hw bug to me, shouldn't there >> be an errata for such a bug? >> >> And I still don't quite understand why the lockup doesn't happen after a >> warm boot, there must be something different there... > > What are the adjust values after a warm boot? > So, after cold boot with a kernel which doesn't adjust TSCs, then warm boot I got: [ 0.000000] TSC ADJUST: CPU0: -602358264300 176072418728 [ 0.000000] TSC ADJUST: Boot CPU0: -602358264300 [ 0.172245] TSC ADJUST: CPU1: -602360207584 176587932558 [ 0.172245] TSC ADJUST differs: Reference CPU0: -602358264300 CPU1: -602360207584 [ 0.172246] TSC ADJUST synchronize: Reference CPU0: -602358264300 CPU1: -602360207584 [ 0.252663] TSC ADJUST: CPU2: -602359000822 176828627154 [ 0.252663] TSC ADJUST differs: Reference CPU0: -602358264300 CPU2: -602359000822 [ 0.252664] TSC ADJUST synchronize: Reference CPU0: -602358264300 CPU2: -602359000822 [ 0.337014] TSC ADJUST: CPU3: -602360177680 177081093132 [ 0.337014] TSC ADJUST differs: Reference CPU0: -602358264300 CPU3: -602360177680 [ 0.337015] TSC ADJUST synchronize: Reference CPU0: -602358264300 CPU3: -602360177680 and so on. Albeit after another reboot (some minutes later), it actually straight locked up again: TSC ADJUST: CPU1: -8257481427958 165112676430 TSC ADJUST differs: Reference CPU0: -8257479484330 CPU1: -8257481427958 TSC ADJUST synchronize: Reference CPU0: -8257479484330 CPU1: -8254781427958 TSC target sync skip ... smpboot: Target CPU is online So, actually I thought the TSC would get reset too on warm boot, but clearly looks like that isn't the case... But I don't know what's the difference between first and second reboot - the adjust values have just more magnitude, but otherwise even the direction of the adjustments and everything looks all the same (just like cold boot, which also looks all the same to me). Roland