Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 27 Mar 2023 10:19:54 +0800
From:   Feng Tang <feng.tang@intel.com>
To:     "Paul E. McKenney" <paulmck@kernel.org>
CC:     Thomas Gleixner <tglx@linutronix.de>,
        Waiman Long <longman@redhat.com>,
        <linux-kernel@vger.kernel.org>
Subject: Re: A couple of TSC questions
Message-ID: <ZCD9SnZSUCkE9Ss+@feng-clx>
References: <b2f07f18-b60b-403b-90ff-937ea32963ee@paulmck-laptop>
 <ZBqOyLByOgvdAve3@feng-clx>
 <acda3fb5-373a-48b1-b78e-eea18da4ccb0@paulmck-laptop>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <acda3fb5-373a-48b1-b78e-eea18da4ccb0@paulmck-laptop>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?FlUg5bzIdKa1vW92Hx1YQef8Dg9lANUY6cTDyBLSrzplFHpUzRWAGO0UN54K?=
 =?us-ascii?Q?5GfQ6slMYCoRUPpxQxXQZyvdyOAcWcqxTYD3qmVSAZKEJlxP5EppWD+qxe+o?=
 =?us-ascii?Q?d4qOGS2POg4pQf5qUEK458zUZ00clTh6N/4lDGwWVJH6qUCKLsoAaDpYMhwo?=
 =?us-ascii?Q?jnh+ZfvJFHMuUE9O1lDtazi6vFeICj/rAooMtQNeRyu81b7F3S/R8DwN3QRj?=
 =?us-ascii?Q?PWRAm5JY9JvrPYdTGoHvxRz95sBZ28oieprxsNtFWuwMa/fsrLAgkh9YcKCa?=
 =?us-ascii?Q?8FNl8kEGd2NBJRWJSVOGjJMgPqx4E1PrtYRFvGDiwVfCfWPstypNK0VHgIsL?=
 =?us-ascii?Q?KhgFx4MdWhPrX0eeXccmftjoi4jk3/3MqVUp42mib5TSAUQMjV3w0xZqgigX?=
 =?us-ascii?Q?kkGts2WRI391frdzOFwXS5xs6k/0MdeipO9LThCYyBjTJ0ElDio6DZR5RAb/?=
 =?us-ascii?Q?QS3iPPplMyYS5FV5p++1nKDK8tgqZ+GvcD6bT/lOodgOmcq9b+3CGRQXImzb?=
 =?us-ascii?Q?ocqEk56M++yx5rIedd//xImszgWNq1gtHHwOScnXEKpj448k9dEpib+Ak+4+?=
 =?us-ascii?Q?Idf9gimreXkevAFK7ucn/RLVj375VUS8NKBJkTgCM8rKgrPujBCR8CeiU0p0?=
 =?us-ascii?Q?lh9ItugamB5B6PyFxs74dxDFA19Tn7/txp+3sCwBZ+Xn3N+mvalSLTZj1GL0?=
 =?us-ascii?Q?PniXJsXwyqBgOVp/u+aoQ84MFmIjtxVeQ2+HTG6Qg7bbz/gWJCrrlbZEzfxr?=
 =?us-ascii?Q?BcIl3AqgFkZz47Uj3TNqWMgWKVAeVJ5EHeew8G1+/Ogc61HlPbdGdgbCfl8u?=
 =?us-ascii?Q?rRVEOWBXxpme2WzHeFo+v3eStiWSSFoAiY5wuYPg14gtm6sT52FtMTRqTiue?=
 =?us-ascii?Q?PSExuWYX34mZ9dZ+uCJmesITSDLBSe1EzqStI2JNwzjMSuBPA/wOIiLbZVL1?=
 =?us-ascii?Q?xfAn0Q6+39AgWEajSqRkjz2IEv4PZVVOFcwMnaRQdOWkLdSE42w6/HJfjggg?=
 =?us-ascii?Q?+TNEF9V6t1wxdkGdrCmHEyKBnHFhfDbf9nSAxBGxi7Zx/sebtONYUJyxXgi3?=
 =?us-ascii?Q?7YA50ytTEq7f4pivxgaejYgzhkJoe9iICZ8IORuVLhOb8Ikb5qnelzCNjTBk?=
 =?us-ascii?Q?Sz6Ar18y/4glzzydczmZQhSUmSzGbcFwSRl0QdQiCLQgmcKzOHanzBfs8n8i?=
 =?us-ascii?Q?6r5Mvk0dFaC9Gi+5V/m8WoZ1mAuOBCvRMl8urtGTrV70ydu8iOa84xHdfR00?=
 =?us-ascii?Q?a4csnno3si0zx77Y278DG+PLT5vtfDsFtNUOPJFwKCnDQhnxF3+StagREVE/?=
 =?us-ascii?Q?bMH6rmsVevdCWf3RpDmx3bigHxtsHkJK6LUiZ6gjt1Gokd4IMrg3w57P9oog?=
 =?us-ascii?Q?ltxWUoRmGTRh1sbV6C8FnNKdA1BkJq7pj040a975Ey1tTQqR7zbR7WMOvSxN?=
 =?us-ascii?Q?6T2FISwdpTa6LPnOhq20s/5ZofcVlaaTPmqqEB04qHzipz+dKQ1FRvi4jmJz?=
 =?us-ascii?Q?siKUXJJkYQukaP4xYQPOlHpx/hOZAbhrLdldHMV8GZ+Lsep4Fs8GXavSf1PS?=
 =?us-ascii?Q?MhTHWc8cipezNDPITpErrrpHjC+XoyRU7P7nU608?=
X-MS-Exchange-CrossTenant-Network-Message-Id: b6d6dd0e-8ac3-4b9c-2b1e-08db2e6a8772
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6304.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2023 02:25:27.8762
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: pdePvyfnl/CzNyx2bZ3Z7OCt1NUGj4ffg9olSX3LzWCt3KE4zUBuPX2Olwj15TBXHzfAl9KmfLoQdlgkL6cK3Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR11MB6861
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE,
        SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Mar 24, 2023 at 05:47:33PM -0700, Paul E. McKenney wrote:
> On Wed, Mar 22, 2023 at 01:14:48PM +0800, Feng Tang wrote:
> > Hi, Paul
> > 
> > On Tue, Mar 21, 2023 at 04:23:28PM -0700, Paul E. McKenney wrote:
> > > Hello, Feng!
> > > 
> > > I hope that things are going well for you and yours!
> > 
> > Thanks!
> > 
> > > First, given that the kernel can now kick out HPET instea of TSC in
> > > response to clock skew, does it make sense to permit recalibration of
> > > the still used TSC against the marked-unstable HPET?
> > 
> > Yes, it makes sense to me. I don't know the detail of the case, if
> > the TSC frequency comes from CPUID info, a recalibration against a
> > third party HW timer like ACPI_PM should help here. 
> > 
> > A further thought is if there are really quite some case that the
> > CPUID-provided TSC frequency info is not accurate, then we may need
> > to enable the recalibration by default, and give a warning message
> > when detecting any mismatch. 
> 
> Now that you mention it, it is quite hard to choose correctly within
> the kernel.  To do it right seems to require that NTP information be
> pushed into the kernel.

Yes, we need a 'always-right' reference, but the system have to has
network access.

I know there have been many different problems related to TSC, but
the real HW/FW related problems are only about the accuracy of
TSC frequency's calibration/calculation.

Before commit b50db7095fe0 ("x86/tsc: Disable clocksource watchdog
for TSC on qualified platorms"), if the TSC freq is calculated
from CPUID or MSR, the HPET/ACPI_PM_TIMER can detect the possible
calculation problem during clocksource watchdog check. For this
case, we may need to force the recalibration by HPET/ACPI_PM_TIMER.

Thanks,
Feng

> 
> > > Second, we are very occasionally running into console messages like this:
> > > 
> > > Measured 2 cycles TSC warp between CPUs, turning off TSC clock.
> > > 
> > > This comes from check_tsc_sync_source() and indicates that one CPU's
> > > TSC read produced a later time than a later read from some other CPU.
> > > I am beginning to suspect that these can be caused by unscheduled delays
> > > in the TSC synchronization code, but figured I should ask you if you have
> > > ever seen these.  And of course, if so, what the usual causes might be.
> > 
> > I haven't seen this error myself or got similar reports. Usually it
> > should be easy to detect once happened, as falling back to HPET
> > will trigger obvious performance degradation.
> 
> And that is exactly what happened.  ;-)
> 
> > Could you give more detail about when and how it happens, and the
> > HW info like how many sockets the platform has. 
> 
> We are in early days, so I am checking for other experiences.
> 
> > CC Thomas, Waiman, as they discussed simliar case here:
> > https://lore.kernel.org/lkml/87h76ew3sb.ffs@tglx/T/#md4d0a88fb708391654e78312ffa75b481690699f
> 
> Fun!  ;-)
> 
> 							Thanx, Paul