Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 11 Jan 2023 20:31:58 +0800
From:   Feng Tang <feng.tang@intel.com>
To:     Thomas Gleixner <tglx@linutronix.de>
CC:     "Paul E. McKenney" <paulmck@kernel.org>,
        <linux-kernel@vger.kernel.org>, <john.stultz@linaro.org>,
        <sboyd@kernel.org>, <corbet@lwn.net>, <Mark.Rutland@arm.com>,
        <maz@kernel.org>, <kernel-team@meta.com>, <neeraju@codeaurora.org>,
        <ak@linux.intel.com>, <zhengjun.xing@intel.com>,
        Waiman Long <longman@redhat.com>,
        John Stultz <jstultz@google.com>
Subject: Re: [PATCH clocksource 5/6] clocksource: Suspend the watchdog
 temporarily when high read latency detected
Message-ID: <Y76sPnnX/73LuGez@feng-clx>
References: <20230105010429.GA1773522@paulmck-ThinkPad-P17-Gen-1>
 <20230105010701.1773895-5-paulmck@kernel.org>
 <87r0w1ia65.ffs@tglx>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <87r0w1ia65.ffs@tglx>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?eYt2m4FoLjQ43eM682IbJB8/zpnnFmJQQA4Onq7sLpoG6Os/Hz97xmPLaBr1?=
 =?us-ascii?Q?ShGCjqX2ThIR76XwJ9KGiSSq/ey4H7uAsHYWWre2heukTbbDDkNWbykcaWvx?=
 =?us-ascii?Q?wmHaw2Q3cRUnfpt3PuibbqY3X3+0nEWpeKP/m3je/xtKykPUFQMP2a906Pha?=
 =?us-ascii?Q?HaxZxzTRj+oqZDAPxbv9wfYxfZVdzXZG2MonA/nhObnnRNBj7eMJxAPBQLfU?=
 =?us-ascii?Q?Jt0CAanTN9yPVVDUwACrrPRop1xTS5Duqykggjrtk10xd0zhs2F0l1iPyqHt?=
 =?us-ascii?Q?eGrsPguYzjB0NZoBVGAJ4VOKb8hrDQMZkhJePb1iTlb5pmWMV5RVRPIA/dej?=
 =?us-ascii?Q?53IOIMGpDoDxlFkDS+uVpocGjqQIAIXGLrH65TNIREN0Hs62Ssc1qOKW3bRt?=
 =?us-ascii?Q?zDWPaUY6WPLAhmo+MQ6GMR3YjMprrxbsj3t6vw4VNSW6UVKIwyRkcWiBtdrM?=
 =?us-ascii?Q?JSOL0y8SZLLhJxUHxitgIBBSWK2Rov1By5KC/7X7mzesRhmBWy1sxU/iGHi9?=
 =?us-ascii?Q?VbbQm3DAmGGxYAQnLNrqhEmczIbDxPwe7HuNpxB4cSbriq49stJrcrcMreCL?=
 =?us-ascii?Q?cas8b5r05YUdvBgT49kjixID/Gj/BOqn5dCPrpJbZydcIW/U0DJsrpv+VodD?=
 =?us-ascii?Q?8KQrRRFMJROBFfsIdz9SA4gb/i+U7TNI1X7TBlQBL1PN2ZsmFV9PelKL4/bo?=
 =?us-ascii?Q?Nm7PyvKQ5SDhraJxVUPcmIR8ad63Tu+Bx/abNoooFcDn2l1Ll5TKJrA5box3?=
 =?us-ascii?Q?yQM19UorNH5Iw4Z4i+/CjQsuKJGJUKvDphmjC02sENE733MSARsB62GbEcoN?=
 =?us-ascii?Q?1RzAeBwOkYmOjjW+/Jim+ne5AVR+Iqer5P6fLgYXQ79nF3W500AV3/136FDI?=
 =?us-ascii?Q?afwbfar+pXUIPQV/JVCuOUM+nHCs4+2apquVs5MB6WuPr1n3HaCM+QE3j899?=
 =?us-ascii?Q?Gc5xfRl9m7hqmva9IDMxVBN4iYExameS7JVWxK1c16NxcKAndE9MzXXGfnwF?=
 =?us-ascii?Q?Z1GoyxesSZjNQAgR4vm/46u6vrd+PgeGPvkJGwDseIhnKQ8s+dnE1sUSkBTD?=
 =?us-ascii?Q?Q58jenkHhXXA9Ifdn0kARHdpFSql+sulqMl5BMhP3xMi4+5AmIOWuPZV702j?=
 =?us-ascii?Q?Q0z+/hBLFfrD3NHnrriqdCOwJDEy95sDmF8n1mYBXJLqaFEleRX8Cy0QpQZQ?=
 =?us-ascii?Q?dLwZU21wkYMb3Duw/Urobl3wN5SxAeRpe66vcT3HZJwy7+QYkXm0ahi09BUR?=
 =?us-ascii?Q?qmsUsADqOeV7b5GoGhi6t8d3GzyXVTiN2gl9GUiwEi+7eTfrwjhFjoYeqbMm?=
 =?us-ascii?Q?e4SEct5C7B9I94uYerPVp4r/Xyrw95wM3fW8sQtVRT2ocOKGnL6iKHKgdTLi?=
 =?us-ascii?Q?fRHg3qtluiW2wBgHV3rt6eVMesqUPF7Ukkfoe74xUZI8QBOh0cUOo1cfol3f?=
 =?us-ascii?Q?xBxCblq4M1ovQMYjrOIZvdi39bPVKY5Bb9zVLFgCUW303bK9YgTbyF+fOJRO?=
 =?us-ascii?Q?oqBIfukyBNFTOSdYCuiWWFoRW7uGRR5luG1g7VUwIS3szHpLbnt+dhHWQ3nf?=
 =?us-ascii?Q?hJajwqWO5rNXb9hYZgDwPuMBwoBtTtfTsS6uNhyv?=
X-MS-Exchange-CrossTenant-Network-Message-Id: d63653ed-b6cc-4b4c-9142-08daf3d03631
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6304.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Jan 2023 12:34:41.5561
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: e8NNBb2gs3CR0yoi9B3W2+2m459Lbe/tKX2aUEbwN1mKOBbqokLfK1C41ekTqc2RZ2tIwy7YDQ+qrsHFv6SUiA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR11MB6986
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Jan 11, 2023 at 12:26:58PM +0100, Thomas Gleixner wrote:
> On Wed, Jan 04 2023 at 17:07, Paul E. McKenney wrote:
> > This can be reproduced by running memory intensive 'stream' tests,
> > or some of the stress-ng subcases such as 'ioport'.
> >
> > The reason for these issues is the when system is under heavy load, the
> > read latency of the clocksources can be very high.  Even lightweight TSC
> > reads can show high latencies, and latencies are much worse for external
> > clocksources such as HPET or the APIC PM timer.  These latencies can
> > result in false-positive clocksource-unstable determinations.
> >
> > Given that the clocksource watchdog is a continual diagnostic check with
> > frequency of twice a second, there is no need to rush it when the system
> > is under heavy load.  Therefore, when high clocksource read latencies
> > are detected, suspend the watchdog timer for 5 minutes.
> 
> We should have enough heuristics in place by now to qualify the output of
> the clocksource watchdog as a random number generator, right?

The issue was found on a 8 sockets machine (around 400 cores, 800 CPUs),
and seems with the bigger and bigger CPU numbers, the spark latency
of reading HPET or even TSC is very high, which does affect the
accuracy of clocksource watchdog check. And unfortunately, we can't
just disable the watchdog for these 8 sockets machine.

We tried a debug patch which disables interrupt and does consective
reads with 'rdtsc', and check the delta between these reads (when
system is running some heavy load), sometimes we can see up to
300 micro-seconds delta, on a 2-sockets Icelake machine. Similar
thing if we replace the 'rdtsc' with 'rdtscp' or 'lfence;rdtsc'.
And I was told the max latency is much higher on the 8 sockets
machine.

Thanks,
Feng