Received: by 2002:a05:6358:d09b:b0:dc:cd0c:909e with SMTP id jc27csp7678834rwb; Wed, 23 Nov 2022 09:20:51 -0800 (PST) X-Google-Smtp-Source: AA0mqf4N9XqaqC16CWKG4DymqUO3hKz96wnbID90GByOI2nu44UvvZ9+2pDswC3KhZWYjfzjp+u/ X-Received: by 2002:a63:ff63:0:b0:477:8d4f:3dda with SMTP id s35-20020a63ff63000000b004778d4f3ddamr10604908pgk.552.1669224050898; Wed, 23 Nov 2022 09:20:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669224050; cv=none; d=google.com; s=arc-20160816; b=PWuG2rwlSlAns6PIOIQbtp+8gaP3PM7c6Ia+NW74kTK2cb958/dpxOvZHeEqiw71LA vpPf4bENgSjl0paRioqak7+Q5JJHORSn/FVW1mU+QVPnY2La27hN5Ha0MGdazmLisEQf F2/97+AY9hsYj0+9ryajAdR6vdZzba2a8kA4e5CslDpiUlVfbTwuoP9HP0/GnvjBeW6d joKs4aOIPOt+SLgSkE0dm4IqSl6i1/DnpRx7SFzxIzh/GN7NxLkP5kPO1t6zxykqgdM2 satscdJt3HFRQEzhNBBHqQXZngQdI/DiWROEz9aaJHX4ITXAbh/JHw3ZG3J7GS8kpJre GJ/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=aWg9iCmVtcgU19BjwQr7qHc7qmpPSCGL2/gExDLvHZM=; b=u+aX5tn/wROMUNPrJUsZPi+pBM/bd2ApNqp3wUUNilEp6qiMMP4F33yVzAG37weNLD ma5P58N4bl2nz2tbktzBxq7iEQi6orHP2iO3RIDok8hpbAaf7SRO1yazPi4HqCa8hlzx 5Eg858xbi7UEutieW9bD2uYNn8XO6ikZEppM7qusKGwH062B3n1isIzJN6gjC9bdJSVO tdb9L0jf0C6C4V8OXdfDz6VRuL84JHlnC42era4IZjCNnX9660coZRqzzEQY2yFaeNM4 rbdxYzlFMHik2ej7lC3NXwgW1N/clGBVb7skoX+6c6PlTcfy8k6gttolYNSCzYXYG3oX RFiQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Zpuzjps4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n1-20020a634d41000000b0046b2bef339csi17162619pgl.257.2022.11.23.09.20.39; Wed, 23 Nov 2022 09:20:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Zpuzjps4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239028AbiKWRHi (ORCPT + 88 others); Wed, 23 Nov 2022 12:07:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56680 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239024AbiKWRGp (ORCPT ); Wed, 23 Nov 2022 12:06:45 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0EE0BC4B5B; Wed, 23 Nov 2022 09:06:42 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id ACCF5B821BC; Wed, 23 Nov 2022 17:06:40 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8ABA2C4347C; Wed, 23 Nov 2022 17:06:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669223199; bh=UofQ2Om5TKKPZw19SG+8wKH8TntN+ojXWrOiFhpkAis=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Zpuzjps4rlDgFJHDwYeS5J9U2UQqY/xuJGczyIdL1b18H1liRMAp3/nDzPFYH4cDs oFxeCTPeCgFLw55Ltd74fC9YCrughkpeeRZWM/N4IJ7R96hByx9/Js7nDcf1Med+Ce sMMRWUCUMjBoraelsC500d4UZFobIBm/d3X8gDXucwOjzjB8YmE4rfQoNwWugurjrZ oU0b9KP0EgNI6zaFVn/ghRM1IX3Lx3d8+55m7VYUQLhbjj9m+C7ceYiWMT0E6zdXP2 kdBmF+xl43/9hyTlGlfuC0cCGCBAHAniTPnIregkApe8xkw8+6VLq4/nHBR1DvJNTD 3S7nCir1b+MPQ== From: Daniel Bristot de Oliveira To: Daniel Bristot de Oliveira , Steven Rostedt Cc: Masami Hiramatsu , Jonathan Corbet , Juri Lelli , Clark Williams , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 4/6] tracing/osnoise: Add PANIC_ON_STOP option Date: Wed, 23 Nov 2022 18:06:22 +0100 Message-Id: <31aa66b01dfc31af481bcadea04b4affed59430b.1669115208.git.bristot@kernel.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Often the latency observed in a CPU is not caused by the work being done in the CPU itself, but by work done on another CPU that causes the hardware to stall all CPUs. In this case, it is interesting to know what is happening on ALL CPUs, and the best way to do this is via crash dump analysis. Add the PANIC_ON_STOP option to osnoise/timerlat tracers. The default behavior is having this option off. When enabled by the user, the system will panic after hitting a stop tracing condition. This option was motivated by a real scenario that Juri Lelli and I were debugging. Cc: Juri Lelli Cc: Daniel Bristot de Oliveira Cc: Steven Rostedt Cc: Masami Hiramatsu Cc: Jonathan Corbet Signed-off-by: Daniel Bristot de Oliveira --- kernel/trace/trace_osnoise.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c index 3f10dd1f2f1c..801eba0b5cf8 100644 --- a/kernel/trace/trace_osnoise.c +++ b/kernel/trace/trace_osnoise.c @@ -54,10 +54,11 @@ enum osnoise_options_index { OSN_DEFAULTS = 0, OSN_WORKLOAD, + OSN_PANIC_ON_STOP, OSN_MAX }; -static const char * const osnoise_options_str[OSN_MAX] = { "DEFAULTS", "OSNOISE_WORKLOAD" }; +static const char * const osnoise_options_str[OSN_MAX] = { "DEFAULTS", "OSNOISE_WORKLOAD", "PANIC_ON_STOP" }; #define OSN_DEFAULT_OPTIONS 0x2 unsigned long osnoise_options = OSN_DEFAULT_OPTIONS; @@ -1270,6 +1271,9 @@ static __always_inline void osnoise_stop_tracing(void) trace_array_printk_buf(tr->array_buffer.buffer, _THIS_IP_, "stop tracing hit on cpu %d\n", smp_processor_id()); + if (test_bit(OSN_PANIC_ON_STOP, &osnoise_options)) + panic("tracer hit stop condition on CPU %d\n", smp_processor_id()); + tracer_tracing_off(tr); } rcu_read_unlock(); -- 2.32.0