Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 30 Sep 2022 18:13:14 +0800
From:   Feng Tang <feng.tang@intel.com>
To:     Yu Liao <liaoyu15@huawei.com>
CC:     Xiongfeng Wang <wangxiongfeng2@huawei.com>,
        Zhang Rui <rui.zhang@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Bjorn Helgaas <helgaas@kernel.org>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        <x86@kernel.org>, <linux-kernel@vger.kernel.org>,
        "Bjorn Helgaas" <bhelgaas@google.com>,
        Kai-Heng Feng <kai.heng.feng@canonical.com>,
        <len.brown@intel.com>, Xie XiuQi <xiexiuqi@huawei.com>,
        Kefeng Wang <wangkefeng.wang@huawei.com>
Subject: Re: [PATCH] x86/PCI: Convert force_disable_hpet() to standard quirk
Message-ID: <YzbBOuX37XpUiP4y@feng-clx>
References: <20201126012421.GA92582@shbuild999.sh.intel.com>
 <87eekfk8bd.fsf@nanos.tec.linutronix.de>
 <20201127061131.GB105524@shbuild999.sh.intel.com>
 <87eekairc0.fsf@nanos.tec.linutronix.de>
 <bd5b97f89ab2887543fc262348d1c7cafcaae536.camel@intel.com>
 <9d3bf570-3108-0336-9c52-9bee15767d29@huawei.com>
 <YzY6e0UwDAyU1GrX@feng-clx>
 <44206484-620d-abaf-4fb9-fc4ef1c9184f@huawei.com>
 <YzZDLBKbDTbNr45b@feng-clx>
 <119b669e-aafb-4d73-e94e-ef119f909cfa@huawei.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <119b669e-aafb-4d73-e94e-ef119f909cfa@huawei.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?IYcieiyvVBCnd+D8+TW9sn6YBMZcDNmrXPMh6bKUqrarJVnM4PjetCYbDj14?=
 =?us-ascii?Q?/vPDGbO+2BUfcFtDqkTq6V4mHVJASZm+CePXkWBEqhX2QwIL6I/4V2vfBnhl?=
 =?us-ascii?Q?+VxRxosd+eqxYkxQtR8dMxk3NBR17AAgZ7vdP/iAS/AfjPR8MCHtdJUC26kU?=
 =?us-ascii?Q?r1qAVvx+W9as6vlXP2PPk5e+qhzvLJx00HGZCyzSd0pW2gCFqwcLnGW6UZll?=
 =?us-ascii?Q?CRhGt76bg+4DijMNbY9JNd7XOE1UEJUpe1eXadMMSXlD+uG8xPi1Ek8uAwpZ?=
 =?us-ascii?Q?t5A770Dx475WetYpAliu84JUWxbG0K6ikzX57A5o3/Wy/7W5h7M+2UNBthCw?=
 =?us-ascii?Q?oPTUIGFQ57ZvX1+xK7WJrQ5k/pG3xon1Sp0ZpkVPcznnqTWEJ1Px+5O36Usm?=
 =?us-ascii?Q?8SrIsfrEAFlyPxBehwP7KUj13SB/vgdqPYAeTiryHMUak9+Sbm/rU8uQqBGP?=
 =?us-ascii?Q?DqUGIMDK8NPv/kOSl4tyOMq404ujh8Kcc9KDgfxOuelWU98jiyWBE8LiCSXK?=
 =?us-ascii?Q?3WS5N74JLZ7N4ZxUpRAVaaPgpFeQgKzIDEeKPv+C1l1LqmvPLXx/pkTQjIeq?=
 =?us-ascii?Q?0B39M7ZdQ1jmA7OcbTuu252BHG0uQiT1l0Qlb1gQ/j4C4wnSlC2McmSzaBbr?=
 =?us-ascii?Q?beW62I4nmInb8+Pgnfbm1hXkJszao41IpM0OZzfDHbfOjTbwOWbanv35T4Cb?=
 =?us-ascii?Q?A1BFtpNbCNWUVv8Z6HnO4sEMHxBQLv5NLWI7EWYJ+h1uNOpOEHmnu55+HDLc?=
 =?us-ascii?Q?VyQB69JVcbEkD0RYobbYh64pmSrWFPwvUJg5Cr0Lf0hr/quxUPLopUQ2fPft?=
 =?us-ascii?Q?b2yhHuhyBopZRd8HaQ/zIBkWjnGWm5KGAdF4jxfQjPeip8Ro4jnyc/YWE4uV?=
 =?us-ascii?Q?D4/WRQkSXKwopKm8uPU3r0fD00sxu996My2O7tJfEpEyCzoJlgkdHkuHdL7/?=
 =?us-ascii?Q?BBJntVGZEoJbU3T4pLCO2k3uDNJmk1jPMX4zc+z8cyO/Fv36uCUh79IhGVJq?=
 =?us-ascii?Q?b2vRlaH9US7PKoDJ4Z54M215A40pGAPvrSwocpAS6CZ8AHBNT2QVF4woDuP5?=
 =?us-ascii?Q?KT2GNgrW66c4t5e/My/k2Yd1UY/TFZw4b0SZY4tMY5TZy60km9eAJvzWJ1pv?=
 =?us-ascii?Q?AVXXdB3fw25xTQ+kvCEZohNk++JozOp+iMc/ADHI+p3egsuDQz/Oz5dPm6fb?=
 =?us-ascii?Q?9Jpr4c50UQpmhqDHxaRLvMLxn2bsIwtvUkjju8c3Fms6UZyk8SsaCBvfcNIS?=
 =?us-ascii?Q?L7HYZ+yp7mWgjwe9Yk0D74cpCuh/opJRx4ukEXFJKKzg9psj4Yt+IYk2T+19?=
 =?us-ascii?Q?vQ4t5lRcBUYRC1VrAHQgU9yLMEvlRsNxcqjNmYFr2LmyIjIj4DrYSyiwKKeE?=
 =?us-ascii?Q?raG+uspGNmXI/riOQAudt/a6xH+gRSOD0pvrrgkL0Awgv+GEUmgPTWT/dqRk?=
 =?us-ascii?Q?80DC+6V0OqHkMTHt1coiwfVCADXKcdiwLamWpEllAeYOnxkt44Kp6eKvetzr?=
 =?us-ascii?Q?68ejLKK6DFTGo9gWRgEKpRuhtHJkKAJyIMK0EDZGgBwb3/1p/QgTo+NJllJw?=
 =?us-ascii?Q?hZWyQBGJUM2RkuMEWjEpVMIkZn9Mdg9nirRDJLzp?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 1a784013-c085-48d5-13b3-08daa2cc7014
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6304.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Sep 2022 10:13:36.5314
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 1ltfAwmL5+wMX9asRwd9NsWNVoTxID31TUgm3MimI2erZ+NJ3/ht+6dQQHm4Ed0tKVx9WYn1BxLxveygfk/6kQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB5184
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Sep 30, 2022 at 05:45:29PM +0800, Yu Liao wrote:
[...]
> >>>>
> >>>> Hi, Zhang Rui, we have met the same problem as you mentioned above. I have
> >>>> tested the following modification. It can solve the problem. Do you have plan
> >>>> to push it to upstream ?
> >>>
> >>> Hi Liao Yu,
> >>>
> >>> Could you provoide more details? Like, what ARCH is the platform (x86
> >>> or others), client or sever, if sever, how many sockets (2S/4S/8S)?
> >>>
> >>> The error kernel log will also be helpful.
> >>
> >> Hi, Feng Tang,
> >>
> >> It's a X86 Sever. lscpu print the following information:
> >>
> >> Architecture:                    x86_64
> >> CPU op-mode(s):                  32-bit, 64-bit
> >> Byte Order:                      Little Endian
> >> Address sizes:                   46 bits physical, 48 bits virtual
> >> CPU(s):                          224
> >> On-line CPU(s) list:             0-223
> >> Thread(s) per core:              2
> >> Core(s) per socket:              28
> >> Socket(s):                       4
> >> NUMA node(s):                    4
> >> Vendor ID:                       GenuineIntel
> >> CPU family:                      6
> >> Model:                           85
> >> Model name:                      Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz
> >> Stepping:                        4
> >> CPU MHz:                         3199.379
> >> CPU max MHz:                     3800.0000
> >> CPU min MHz:                     1000.0000
> >> BogoMIPS:                        5000.00
> >> Virtualization:                  VT-x
> >> L1d cache:                       3.5 MiB
> >> L1i cache:                       3.5 MiB
> >> L2 cache:                        112 MiB
> >> L3 cache:                        154 MiB
> >> NUMA node0 CPU(s):               0-27,112-139
> >> NUMA node1 CPU(s):               28-55,140-167
> >> NUMA node2 CPU(s):               56-83,168-195
> >> NUMA node3 CPU(s):               84-111,196-223
> >>
> >> Part of the kernel log is as follows.
> >>
> >> [    1.144402] smp: Brought up 4 nodes, 224 CPUs
> >> [    1.144402] smpboot: Max logical packages: 4
> >> [    1.144402] smpboot: Total of 224 processors activated (1121097.93 BogoMIPS)
> >> [    1.520003] clocksource: timekeeping watchdog on CPU2: Marking clocksource
> >> 'tsc-early' as unstable because the skew is too large:
> >> [    1.520010] clocksource:                       'refined-jiffies' wd_now:
> >> fffb7210 wd_last: fffb7018 mask: ffffffff
> >> [    1.520013] clocksource:                       'tsc-early' cs_now:
> >> 6606717afddd0 cs_last: 66065eff88ad4 mask: ffffffffffffffff
> >> [    1.520015] tsc: Marking TSC unstable due to clocksource watchdog
> >> [    5.164635] node 0 initialised, 98233092 pages in 4013ms
> >> [    5.209294] node 3 initialised, 98923232 pages in 4057ms
> >> [    5.220001] node 2 initialised, 99054870 pages in 4068ms
> >> [    5.222282] node 1 initialised, 99054870 pages in 4070ms
> > 
> > Thanks Xiaofeng for the info.
> > 
> > Could you try the below patch? It is kinda extension of 
> > 
> > b50db7095fe0 ("x86/tsc: Disable clocksource watchdog for TSC on qualified platorms") 
> > 
> > which I have run limited test on some 4 sockets Haswell and Cascadelake
> > AP x86 servers.
> > 
> > 
> > Thanks,
> > Feng
> > ---
> > diff --git a/arch/x86/kernel/tsc.c b/arch/x86/kernel/tsc.c
> > index cafacb2e58cc..b4ea79cb1d1a 100644
> > --- a/arch/x86/kernel/tsc.c
> > +++ b/arch/x86/kernel/tsc.c
> > @@ -1217,7 +1217,7 @@ static void __init check_system_tsc_reliable(void)
> >  	if (boot_cpu_has(X86_FEATURE_CONSTANT_TSC) &&
> >  	    boot_cpu_has(X86_FEATURE_NONSTOP_TSC) &&
> >  	    boot_cpu_has(X86_FEATURE_TSC_ADJUST) &&
> > -	    nr_online_nodes <= 2)
> > +	    nr_online_nodes <= 8)
> >  		tsc_disable_clocksource_watchdog();
> >  }
> >  
> > 
> Hi Feng,
> 
> I tested this patch on a previous server and it fixes the issue.
 
Thanks for the testing, please do let us know if there is any TSC
problem after long time or stress running.

Plan to send the patch for merging.

Thanks,
Feng

> Thanks,
> Yu
> 
>