Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Song Liu <songliubraving@meta.com>
To:     Peter Zijlstra <peterz@infradead.org>
CC:     Song Liu <song@kernel.org>, LKML <linux-kernel@vger.kernel.org>,
        Kernel Team <kernel-team@meta.com>,
        Andrew Morton <akpm@linux-foundation.org>
Subject: Re: [PATCH v4] watchdog: Allow nmi watchdog to use "ref-cycles" event
Thread-Topic: [PATCH v4] watchdog: Allow nmi watchdog to use "ref-cycles"
 event
Thread-Index: AQHZiR9WEaR3xoFFfEGCLaxGjWAWL694Nj4AgAAHooA=
Date:   Fri, 2 Jun 2023 23:15:14 +0000
Message-ID: <DCF269A8-4C64-4FDE-AFAC-92B6029EA3BA@fb.com>
References: <20230518002555.1114189-1-song@kernel.org>
 <20230602224744.GC695361@hirez.programming.kicks-ass.net>
In-Reply-To: <20230602224744.GC695361@hirez.programming.kicks-ass.net>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?u55c0nlbp327VE2llZlKGxGoGgec8wOjqWW5iURb4QehkF4pxmfqRZsuo1oj?=
 =?us-ascii?Q?YJd81Y1TdlRMYE5tgU27Vg1Ij39tU/M8Tm/9+KrtunYbHZB/z95fFBoS3kPK?=
 =?us-ascii?Q?GTrlAUh5gzBuwl+iG/wwe8xbt90CwbL4K7DQSPRzIU+QUF1q0xMerjQv3szd?=
 =?us-ascii?Q?tkPsx01M7/2yToUDZmw2D2lpQu6+PLJjluQZArYvkci+bbeGJCOqC/ElEhDr?=
 =?us-ascii?Q?Mwd0Lqkd6TChmnwHmUrqr7xrpZIt9TdAdKjzDDKH8DPt6Bo+2cRxe21N73P5?=
 =?us-ascii?Q?+B0PyIIrNoZKDos3G1uhmLP1zDnFI50HwE/EqdsfcIXc6G1/ZmC5q9o+Sk8C?=
 =?us-ascii?Q?qHo/WY0jclGC+NJg7WkB5UlRQFU1/1JKsCFHDCRQ3aySwn5oz6RESpiHM5I1?=
 =?us-ascii?Q?H9ReyzNvRWSP2P4KC/CuhOIRRODUXqC6GRg1ZCQTMSYuHP7x71CHPL8Q2Bkm?=
 =?us-ascii?Q?jYmqs0SJgvbJtj3XE6hyNpbnnUO0souMZxyvpHobOyoF8ZRKnqwNOFaO/n7S?=
 =?us-ascii?Q?PhcKGHVo9SMo/GkAhFAPaxKS5oaZF7cg6V71DJPnQgp13UVHaIlL6Kyxa9C1?=
 =?us-ascii?Q?vnHTs+7e8EvDyb/AagWA6z1U+cglM6Aiv+61ta3MoBxHk2werd6xuQcOEK8r?=
 =?us-ascii?Q?M7rRPeVI2ZK9xtM71eYZrSiXqcMkFYLQAAdgCl3V0o2b3hm3FWlIxRqhZuQt?=
 =?us-ascii?Q?9HJVuHw4n77+jByiBWLuH41unmWho5aWu+aYvWJcY2oM9iQIrHdhkwQKaw56?=
 =?us-ascii?Q?Y/638hNFaVvV7VRwktdf5dR9wFa7mZ910yPZwX3yjDeU/T++BzjHL9+qqsiv?=
 =?us-ascii?Q?BvPSCErkEm8UWHV44OJmnf8jKDljSEaQym19ZvSAFp6qWFc2HFqJHnGK1NLM?=
 =?us-ascii?Q?VkpHXqMtlgGp1ZXsZzkznMvyXCmNUswbMVpVaBINaQzX0KLNKlSF+3DKqtHY?=
 =?us-ascii?Q?owta5vKCF7MEFyVsC8aVDedlBb9IYwCW5sR+X+SQ/tJMxyF6+X8bYrtViHfo?=
 =?us-ascii?Q?jxxka5lpDNGblzqfpaeh671RxN26+xwtWJxZ/jzQRRgjQqxB+K7oU4PgHLNI?=
 =?us-ascii?Q?llW0cSCtJIyKCDemX7B0YKU5rZ8wIQUAQ+5lWKGKR0NGt7slJdk4u5CFt8ZY?=
 =?us-ascii?Q?MGgar0jXHQNqfVdkg1YRhsBrlWTqTne9QAeGst9C5EGz9UQu7wE1Vlqq+Czu?=
 =?us-ascii?Q?dG77kudflqG7XghYcYd8vGaCE8G3tfiCPZ+yfHbz98SRQgEqX+nnDRgVf3sN?=
 =?us-ascii?Q?9P2iRSK6SHXqNW3COO9PNmMfRwqn6R1fzz1HMmgDSA2KWyyqcBk5pJgL8vkp?=
 =?us-ascii?Q?Ky7eCt5zGa+CmVNrChbEPVKncYeyT15h6Y/PYnBamFEwGsBB524jz6R/QZI6?=
 =?us-ascii?Q?0ODHjTS4PdlNIib//ZIVAczW2ds0Y6YYIpsOE4TecPuQXwn7UPHh9HBQFS6E?=
 =?us-ascii?Q?L1bSDOYvH7LBm5tN1ahdtMoSbXkk0ILQuSFCE049hBTJ8RlexnVMlPp20ndz?=
 =?us-ascii?Q?7qRvNWQX2VkxLjZgM8tXg28S0luzWu2KibGKUxxebnKXCsv/DGYugpB/TRbW?=
 =?us-ascii?Q?rQ99MPKBsYjx1vBgkhXsIsz81dr5WIXZ12vsKw5au4EHra+FMV3/3I5D9ecQ?=
 =?us-ascii?Q?aLFJyhZmE7dBmMVXY2m0N37BTbZOGZCIpNB4mHTIftPN?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <B830001C016CE841BDDFBCA2E11D1E52@namprd15.prod.outlook.com>
MIME-Version: 1.0
X-OriginatorOrg: meta.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PH0PR15MB5117.namprd15.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2ed56d17-07fc-4507-8bd6-08db63bf389d
X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Jun 2023 23:15:14.2750
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: Ou/P0grCifkULDSgIwjtdSxFbtmjJfGV68SIfvWAQqPWLfayaBzpAi0KNw9SnusquB3w2BuYgsDQXgJwF4OGAw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR15MB5621
X-Proofpoint-GUID: X9ztUn_h22_Svi2dMpr_Rlv5V3Uu-5sN
X-Proofpoint-ORIG-GUID: X9ztUn_h22_Svi2dMpr_Rlv5V3Uu-5sN
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.573,FMLib:17.11.176.26
 definitions=2023-06-02_17,2023-06-02_02,2023-05-22_02
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_PASS,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


> On Jun 2, 2023, at 3:47 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> 
> On Wed, May 17, 2023 at 05:25:55PM -0700, Song Liu wrote:
>> NMI watchdog permanently consumes one hardware counters per CPU on the
>> system. For systems that use many hardware counters, this causes more
>> aggressive time multiplexing of perf events.
>> 
>> OTOH, some CPUs (mostly Intel) support "ref-cycles" event, which is rarely
>> used. Add kernel cmdline arg nmi_watchdog=ref-cycles to configure the
>> watchdog to use "ref-cycles" event instead of "cycles".
>> 
>> Cc: Andrew Morton <akpm@linux-foundation.org>
>> Cc: Peter Zijlstra <peterz@infradead.org>
>> Signed-off-by: Song Liu <song@kernel.org>
>> 
>> ---
>> Changes in v4:
>> Fix compile error for !CONFIG_HARDLOCKUP_DETECTOR_PERF. (kernel test bot)
>> 
>> Changes in v3:
>> 
>> Pivot the design to use kernel arg nmi_watchdog=ref-cycles (Peter)
>> ---
>> Documentation/admin-guide/kernel-parameters.txt | 5 +++--
>> include/linux/nmi.h                             | 2 ++
>> kernel/watchdog.c                               | 2 ++
>> kernel/watchdog_hld.c                           | 9 +++++++++
>> 4 files changed, 16 insertions(+), 2 deletions(-)
>> 
>> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
>> index 9e5bab29685f..d378e23dad7c 100644
>> --- a/Documentation/admin-guide/kernel-parameters.txt
>> +++ b/Documentation/admin-guide/kernel-parameters.txt
>> @@ -3593,10 +3593,12 @@
>> Format: [state][,regs][,debounce][,die]
>> 
>> nmi_watchdog= [KNL,BUGS=X86] Debugging features for SMP kernels
>> - Format: [panic,][nopanic,][num]
>> + Format: [panic,][nopanic,][ref-cycles][num]
>> Valid num: 0 or 1
>> 0 - turn hardlockup detector in nmi_watchdog off
>> 1 - turn hardlockup detector in nmi_watchdog on
>> + ref-cycles - configure the watchdog with perf event
>> +             "ref-cycles" instead of "cycles"
>> When panic is specified, panic when an NMI watchdog
>> timeout occurs (or 'nopanic' to not panic on an NMI
>> watchdog, if CONFIG_BOOTPARAM_HARDLOCKUP_PANIC is set)
> 
> I still hate the whole ref-cycles thing, at the very least powerpc also
> has HAVE_HARDLOCKUP_DETECTOR_PERF and they don't have ref-cycles, but
> perhaps them wants to use a different event when the moon is just so...
> 
> What again was wrong with the option of specifying a raw event value and
> falling back to cpu-cycles if that fails?

The same raw event number may mean different events on different hardware. 
So it is more likely to make mistakes in configurations. For example, r300 
means ref-cycles on Intel CPUs, but it also means something else on AMD 
CPUs. I need to be very careful which hosts to run with nmi_watchdog=r300, 
as it may cause surprises. OTOH, nmi_watchdog=ref-cycles won't have this 
issue. Of course, this won't work for powerpc. 

Does this make sense?

Thanks,
Song