Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 30 Sep 2022 08:38:19 +0800
From:   Feng Tang <feng.tang@intel.com>
To:     Yu Liao <liaoyu15@huawei.com>
CC:     Zhang Rui <rui.zhang@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Bjorn Helgaas <helgaas@kernel.org>,
        Ingo Molnar <mingo@redhat.com>,
        "Borislav Petkov" <bp@alien8.de>, <x86@kernel.org>,
        <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Kai-Heng Feng <kai.heng.feng@canonical.com>,
        <len.brown@intel.com>, <wangxiongfeng2@huawei.com>,
        Xie XiuQi <xiexiuqi@huawei.com>
Subject: Re: [PATCH] x86/PCI: Convert force_disable_hpet() to standard quirk
Message-ID: <YzY6e0UwDAyU1GrX@feng-clx>
References: <20201119181904.149129-1-helgaas@kernel.org>
 <87v9dtk3j4.fsf@nanos.tec.linutronix.de>
 <20201126012421.GA92582@shbuild999.sh.intel.com>
 <87eekfk8bd.fsf@nanos.tec.linutronix.de>
 <20201127061131.GB105524@shbuild999.sh.intel.com>
 <87eekairc0.fsf@nanos.tec.linutronix.de>
 <bd5b97f89ab2887543fc262348d1c7cafcaae536.camel@intel.com>
 <9d3bf570-3108-0336-9c52-9bee15767d29@huawei.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <9d3bf570-3108-0336-9c52-9bee15767d29@huawei.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?4vGZTmu9V2JSSuTWUAZb8AUfQDB6Rm7ccXU8lS4XZ1aiefUN1tez91eXZHvV?=
 =?us-ascii?Q?OVHgHEc1YqDqodefRhj0f8oJtlXmLLnEN8AzBg+GPSgcJyKpmKbEzuaq2sx+?=
 =?us-ascii?Q?mn5BZxWY2D47CMUaJN/VcsReTR+GX2c7ItTMiARqcx20danE0J2KgeuDc+Yi?=
 =?us-ascii?Q?vwu3cB9fG+C4ozqUHEEJlGrqHajMBRYF80FwZgqYmZf8jWfdTmM4HAgFevcX?=
 =?us-ascii?Q?P8nHodhOp2sMJOQ5JcmGMBlInH4Jpehq8aUqXT9K3gqSY+qYm6zxSEwfdhKA?=
 =?us-ascii?Q?ml6SmHTsDnp4MQJJymg7+vFVBoUQqecazWHWR4tZNbYc1EaCeL+htZDRnB4I?=
 =?us-ascii?Q?wbYicf0SuQFIaIU9LaSrUPB1u8AZmhlzCbszI5aUjk2FSoS6J2ZXI7UNVToN?=
 =?us-ascii?Q?wl6FP66i841PkEsB4BUruJV8I4taZvrh0lrDm+8zPL0O/cSTDu5ukbO5mqne?=
 =?us-ascii?Q?Qf1JHEPlXXBucy+GPmzwiu93WvNND/SRl7FMxVPDMTCQExuo+Rojj1igituT?=
 =?us-ascii?Q?VVnTUu28qZ+1jOT0TA8Yga/Qk0i+U/vw9fXZr2V/7GfyDTBh8wKQLsc0sV0p?=
 =?us-ascii?Q?UpWK70I4acFb/O2k61iJP5DTeFzVuG1Xqt4RCP6GZWok9v2Y9frE6AH26jn3?=
 =?us-ascii?Q?dHRaLSjJDtsv0cS5VrjWoGkiJwvHc3j+T42J67Az5m/QNx964339W8UcbSUx?=
 =?us-ascii?Q?peOCBdb1Vx7g6CqYI7/bBeXNh8ZfScbeTjrIR58Ad2EGLP5ve7r62ml23wA8?=
 =?us-ascii?Q?gzN5pn4wVO/9FKZvURqpXL8bmosNrH/C744mi1PXWyRjawGyOqFSH3L+4kol?=
 =?us-ascii?Q?SQ5vhc+MmeAmJSefClc5OpehHrYBbOhI1JiWrqucHuWaT9yt2Bwn8qk1j8mS?=
 =?us-ascii?Q?aHmm0jC9WdtYlB/3e8kAqbmUW9EeYO1gBK7VTW6cx40uYzPVBjmd4oH6YJo5?=
 =?us-ascii?Q?v57pfGOiMS1gvBDOF5XurvRiW9TPDtjVqMW3N1yXdwcNgWhI/TZV0aOQonHv?=
 =?us-ascii?Q?svtRMxlqHB5u72t3VYxgfmrAs+JEfCwX85yBd9pjxOy9JuA368360eKvLorq?=
 =?us-ascii?Q?yQbvVx7YJd52Ltupp1tqegSMJFu2FK7IuXW57ivcYafvDP4DiqMwTgaPfLHo?=
 =?us-ascii?Q?WPJ04NIlVimAEv/LHuVtub2lYPeVF/ygcWhzkkqs7/NE/+OE9QZhjp2uF5R8?=
 =?us-ascii?Q?hvSjekNL8x1uobyRJPjoCQQa4m1JJcphkFQiIWtfKbmkRZJ+ZnCUgbTl9skV?=
 =?us-ascii?Q?vBa1PCDfKHAfQY/6SHFuuR/ds5vh2F9j+ErotGOO+oaEdjgbAbI2AjZNM2Qc?=
 =?us-ascii?Q?0REvtkv4trEN8mv+tuiJg3k2P7jhkRlyv52QD04M0hDYMPxqU1ifGSSZk6Pq?=
 =?us-ascii?Q?9fJBgy8+1hwv1rYR6wR9VtBnxnbOTtEhKAJIdgBXgUjvBoCdf9KWmx/lIXzk?=
 =?us-ascii?Q?h2vTvgu3X4WYlFDIeRPRStfRLcGudVzNcZm9lc19HTUwz/+6OxBNtLoBUPHa?=
 =?us-ascii?Q?ojK8UgSJwIsGSqzgQ5bOvT9KntgrW9ERnEW3bNMZkVhneLJAsc9jhhGDWDX7?=
 =?us-ascii?Q?sraNSZgawhGC4Kia+kc6gB7poJvvQImTDENSoHkD?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 8533615a-b603-4fde-fe43-08daa27c2051
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6304.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Sep 2022 00:38:42.9811
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 3EMrZvZ5ldFhhn4ynvx7/xH3kqGGANdvfvUTz71Cx/DadEVgzcPy+fNTLJudQnuBEk00qA2AZKbh1DUcrHo6sg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB7042
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Sep 29, 2022 at 11:52:28PM +0800, Yu Liao wrote:
> On 2020/12/2 15:28, Zhang Rui wrote:
> > On Mon, 2020-11-30 at 20:21 +0100, Thomas Gleixner wrote:
> >> Feng,
> >>
> >> On Fri, Nov 27 2020 at 14:11, Feng Tang wrote:
> >>> On Fri, Nov 27, 2020 at 12:27:34AM +0100, Thomas Gleixner wrote:
> >>>> On Thu, Nov 26 2020 at 09:24, Feng Tang wrote:
> >>>> Yes, that can happen. But OTOH, we should start to think about
> >>>> the
> >>>> requirements for using the TSC watchdog.
> > 
> > My original proposal is to disable jiffies and refined-jiffies as the
> > clocksource watchdog, because they are not reliable and it's better to
> > use clocksource that has a hardware counter as watchdog, like the patch
> > below, which I didn't sent out for upstream.
> > 
> >>From cf9ce0ecab8851a3745edcad92e072022af3dbd9 Mon Sep 17 00:00:00 2001
> > From: Zhang Rui <rui.zhang@intel.com>
> > Date: Fri, 19 Jun 2020 22:03:23 +0800
> > Subject: [RFC PATCH] time/clocksource: do not use refined-jiffies as watchdog
> > 
> > On IA platforms, if HPET is disabled, either via x86 early-quirks, or
> > via kernel commandline, refined-jiffies will be used as clocksource
> > watchdog in early boot phase, before acpi_pm timer registered.
> > 
> > This is not a problem if jiffies are accurate.
> > But in some cases, for example, when serial console is enabled, it may
> > take several milliseconds to write to the console, with irq disabled,
> > frequently. Thus many ticks may become longer than it should be.
> > 
> > Using refined-jiffies as watchdog in this case breaks the system because
> > a) duration calculated by refined-jiffies watchdog is always consistent
> >    with the watchdog timeout issued using add_timer(), say, around 500ms.
> > b) duration calculated by the running clocksource, usually TSC on IA
> >    platforms, reflects the real time cost, which may be much larger.
> > This results in the running clocksource being disabled erroneously.
> > 
> > This is reproduced on ICL because HPET is disabled in x86 early-quirks,
> > and also reproduced on a KBL and a WHL platform when HPET is disabled
> > via command line.
> > 
> > BTW, commit fd329f276eca
> > ("x86/mtrr: Skip cache flushes on CPUs with cache self-snooping") is
> > another example that refined-jiffies causes the same problem when ticks
> > become slow for some other reason.
> 
> Hi, Zhang Rui, we have met the same problem as you mentioned above. I have
> tested the following modification. It can solve the problem. Do you have plan
> to push it to upstream ?

Hi Liao Yu,

Could you provoide more details? Like, what ARCH is the platform (x86
or others), client or sever, if sever, how many sockets (2S/4S/8S)?

The error kernel log will also be helpful.

Thanks,
Feng

> Thanks,
> Liao Yu
> 
> > 
> > IMO, the right solution is to only use hardware clocksource as watchdog.
> > Then even if ticks are slow, both the running clocksource and the watchdog
> > returns real time cost, and they still match.
> > 
> > Signed-off-by: Zhang Rui <rui.zhang@intel.com>
> > ---
> >  kernel/time/clocksource.c | 4 ++++
> >  1 file changed, 4 insertions(+)
> > 
> > diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c
> > index 02441ead3c3b..e7e703858fa6 100644
> > --- a/kernel/time/clocksource.c
> > +++ b/kernel/time/clocksource.c
> > @@ -364,6 +364,10 @@ static void clocksource_select_watchdog(bool fallback)
> >  		watchdog = NULL;
> >  
> >  	list_for_each_entry(cs, &clocksource_list, list) {
> > +		/* Do not use refined-jiffies as clocksource watchdog */
> > +		if (cs->rating <= 2)
> > +			continue;
> > +
> >  		/* cs is a clocksource to be watched. */
> >  		if (cs->flags & CLOCK_SOURCE_MUST_VERIFY)
> >  			continue;
>