Received: by 2002:a05:6a10:5bc5:0:0:0:0 with SMTP id os5csp3188003pxb; Mon, 18 Oct 2021 09:53:55 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzF8RuKy3rHxZDx3ac/L3vwCeqO5PH1ECTi9CzNv4lEEYqWOP31YJHARLkrZGwxVHX02WN7 X-Received: by 2002:a63:6e07:: with SMTP id j7mr25112450pgc.6.1634576035117; Mon, 18 Oct 2021 09:53:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1634576035; cv=none; d=google.com; s=arc-20160816; b=EsIblGL+A+cdsIPOFy51jK6oKUI/APtXqYsKencpDTupxNBnKmZxbbv5WJwZ0tWRBQ grzrKXDaeg7Ik9MvkbZaf4Y44G+mpLQO9VMDLg/H2qrh7MgzwkFEOEmSjDM4PTe4TJLZ UKXrxKSwRT4qTpbmLboniFq2eaf3X36MVLVa8ylYIretv7ldaK4pM7n44NCtZShhL+dm OuwaJgSrsO5qqF8E5k8T5C7QwnzN0usQpGvcARMdJr2lJMGF1UlxcKfYcgS3wKE4CheH gy4SfWjU5EuUIpE9CZoH1QFT07UBEQkSIKXgqPxDDNNU7sQIecRJxxdyidp5jqIhPXle 3P2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=/jICZVSsNH2yVEWo++2MUmYocuN7REnbVnFPmfIWv+U=; b=M+FQVKZnIBMJS0PdSsT8H0C4n1KICj0V9ZMfIyjudnPI8PbObNm3e9Edm/Fh8S4dEH pGPZUxP3mzK5+VfNrM54MFBake7okbybCi/NKo1s14PZ6jt0SfkBNY945EuqrifaLbQA SDRV2efFHAPfwQ/znM73PTj2cSbCsPvYukJ/61Wts490mvqVnl8+Brn9HLiX71BPt0Fi Ee3Tmr0rgbtC3lattxvz9AS3Kbqo9zuUioCYFOGHP+wAqzE0dacgPQO1/1IA1s90raHp IgvdBHAvHtkyFNq0MJooFf1mRC+XPfM40hn2tJfw7voSpRu02UkYxuX6sejliUddaWre hzHQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id k7si3001506pgn.289.2021.10.18.09.53.42; Mon, 18 Oct 2021 09:53:55 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234078AbhJRQxg convert rfc822-to-8bit (ORCPT + 99 others); Mon, 18 Oct 2021 12:53:36 -0400 Received: from us-smtp-delivery-44.mimecast.com ([207.211.30.44]:47860 "EHLO us-smtp-delivery-44.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234160AbhJRQxJ (ORCPT ); Mon, 18 Oct 2021 12:53:09 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-23-aJ1T7MqGNcmWQolwtWDlTg-1; Mon, 18 Oct 2021 12:50:54 -0400 X-MC-Unique: aJ1T7MqGNcmWQolwtWDlTg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8CF961006AA6; Mon, 18 Oct 2021 16:50:52 +0000 (UTC) Received: from x1.com (unknown [10.22.18.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9106F10016F4; Mon, 18 Oct 2021 16:50:49 +0000 (UTC) From: Daniel Bristot de Oliveira To: Steven Rostedt Cc: Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , Juri Lelli , Clark Williams , John Kacur , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , linux-rt-users@vger.kernel.org, linux-trace-devel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V3 19/19] rtla: Add rtla timerlat hist documentation Date: Mon, 18 Oct 2021 18:48:32 +0200 Message-Id: <79f9f1d1e6979f296edad78b59470310bbebba3b.1634574261.git.bristot@kernel.org> In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=bristot@kernel.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kernel.org Content-Transfer-Encoding: 8BIT Content-Type: text/plain; charset=WINDOWS-1252 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Man page for rtla timerlat hist mode. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Juri Lelli Cc: Clark Williams Cc: John Kacur Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Sebastian Andrzej Siewior Cc: Daniel Bristot de Oliveira Cc: linux-rt-users@vger.kernel.org Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Daniel Bristot de Oliveira --- .../rtla/Documentation/rtla-timerlat-hist.txt | 162 ++++++++++++++++++ 1 file changed, 162 insertions(+) create mode 100644 tools/tracing/rtla/Documentation/rtla-timerlat-hist.txt diff --git a/tools/tracing/rtla/Documentation/rtla-timerlat-hist.txt b/tools/tracing/rtla/Documentation/rtla-timerlat-hist.txt new file mode 100644 index 000000000000..6cc1e8d9abd4 --- /dev/null +++ b/tools/tracing/rtla/Documentation/rtla-timerlat-hist.txt @@ -0,0 +1,162 @@ +rtla-timerlat-hist(1) +===================== + +NAME +---- +rtla-timerlat-hist - Histograms of the operating system timer latency + +SYNOPSIS +-------- +*rtla timerlat hist* ['OPTIONS'] ... + +DESCRIPTION +----------- +The rtla-hist-hist(1) mode displays a histogram of each tracer event occurrence. + +The rtla-timerlat(1) tool is an interface for the timerlat tracer. The +timerlat tracer dispatches a kernel thread per-cpu. These threads set a +periodic timer to wake themselves up and go back to sleep. After the wakeup, +they collect and generate useful information for the debugging of operating +system timer latency. + +The timerlat tracer outputs information in two ways. It periodically prints +the timer latency at the timer 'IRQ' handler and the 'Thread' handler. It +also provides information for each noise via the osnoise tracepoints. This +tool uses the periodic information, and the osnoise tracepoints are enabled +when using the -T option. + +OPTIONS +------- +*-h*, *--help*:: +Print help menu. +*-p*, *--period* 'us':: +Set the timerlat tracer period in microseconds. +*-i*, *--irq* 'us':: +Stop trace if the irq latency is higher than the argument in us. +*-T*, *--thread* 'us':: +Stop trace if the thread latency is higher than the argument in us. +*-s*, *--stack* 'us':: +Save the stack trace at the IRQ if a thread latency is higher than the +argument in us. +*-c*, *--cpus* 'cpu-list':: +Set the timerlat tracer to run the sample threads in the cpu-list. +*-d*, *--duration* 'time[s|m|h|d]':: +Set the duration of the session. +*-T*, *--trace*['=file']:: +Save the stopped trace to ['file|timerlat_trace.txt']. +*-P*, *--priority* 'o:prio|r:prio|f:prio|d:runtime:period':: +Set scheduling parameters to the timerlat tracer threads, the format to +set the priority are: + - 'o:prio' - use SCHED_OTHER with 'prio'; + - 'r:prio' - use SCHED_RR with 'prio'; + - 'f:prio' - use SCHED_FIFO with 'prio'; + - 'd:runtime[us|ms|s]:period[us|ms|s]' - use SCHED_DEADLINE with 'runtime' and +'period' in nanoseconds. + +*-b*, *--bucket-size* 'N':: +Set the histogram bucket size (default 1). +*-e*, *--entries* 'N':: +Set the number of entries of the histogram (default 256). +*--no-irq*:: +Ignore IRQ latencies. +*--no-thread*:: +Ignore thread latencies. +*--no-header*:: +So not print header. +*--no-summary*:: +Do not print summary. +*--no-index*:: +So not print index. +*--with-zeros*:: +Print zero only entries. + +EXAMPLE +------- +In the example below, *rtla timerlat hist* is set to run for '10' minutes, +in the cpus '0-4', 'skipping zero' only lines. Moreover, *rtla timerlat +hist* will change the priority of the timelat threads to run under +'SCHED_DEADLINE' priority, with a '10 us' runtime every '1 ms' period. The +'1ms' period is also passed to the timerlat tracer. + +------------------------------------------ +[root@alien ~]# timerlat hist -d 10m -c 0-4 -P d:100us:1ms -p 1ms +# RTLA timerlat histogram +# Time unit is microseconds (us) +# Duration: 0 00:10:00 +Index IRQ-000 Thr-000 IRQ-001 Thr-001 IRQ-002 Thr-002 IRQ-003 Thr-003 IRQ-004 Thr-004 +0 276489 0 206089 0 466018 0 481102 0 205546 0 +1 318327 35487 388149 30024 94531 48382 83082 71078 388026 55730 +2 3282 122584 4019 126527 28231 109012 23311 89309 4568 98739 +3 940 11815 837 9863 6209 16227 6895 17196 910 9780 +4 444 17287 424 11574 2097 38443 2169 36736 462 13476 +5 206 43291 255 25581 1223 101908 1304 101137 236 28913 +6 132 101501 96 64584 635 213774 757 215471 99 73453 +7 74 169347 65 124758 350 57466 441 53639 69 148573 +8 53 85183 31 156751 229 9052 306 9026 39 139907 +9 22 10387 12 42762 161 2554 225 2689 19 26192 +10 13 1898 8 5770 114 1247 128 1405 13 3772 +11 9 560 9 924 71 686 76 765 8 713 +12 4 256 2 360 50 411 64 474 3 278 +13 2 167 2 172 43 256 53 350 4 180 +14 1 88 1 116 15 198 42 223 0 115 +15 2 63 3 94 11 139 20 150 0 58 +16 2 37 0 56 5 78 10 102 0 39 +17 0 18 0 28 4 57 8 80 0 15 +18 0 8 0 17 2 50 6 56 0 12 +19 0 9 0 5 0 19 0 48 0 18 +20 0 4 0 8 0 11 2 27 0 4 +21 0 2 0 3 1 9 1 18 0 6 +22 0 1 0 3 1 7 0 3 0 5 +23 0 2 0 4 0 2 0 7 0 2 +24 0 2 0 2 1 3 0 3 0 5 +25 0 0 0 1 0 1 0 1 0 3 +26 0 1 0 0 0 2 0 2 0 0 +27 0 0 0 3 0 1 0 0 0 1 +28 0 0 0 3 0 0 0 1 0 0 +29 0 0 0 2 0 2 0 1 0 3 +30 0 1 0 0 0 0 0 0 0 0 +31 0 1 0 0 0 0 0 2 0 2 +32 0 0 0 1 0 2 0 0 0 0 +33 0 0 0 2 0 0 0 0 0 1 +34 0 0 0 0 0 0 0 0 0 2 +35 0 1 0 1 0 0 0 0 0 1 +36 0 1 0 0 0 1 0 1 0 0 +37 0 0 0 1 0 0 0 0 0 0 +40 0 0 0 0 0 1 0 1 0 0 +41 0 0 0 0 0 0 0 0 0 1 +42 0 0 0 0 0 0 0 0 0 1 +44 0 0 0 0 0 1 0 0 0 0 +46 0 0 0 0 0 0 0 1 0 0 +47 0 0 0 0 0 0 0 0 0 1 +50 0 0 0 0 0 0 0 0 0 1 +54 0 0 0 1 0 0 0 0 0 0 +58 0 0 0 1 0 0 0 0 0 0 +over: 0 0 0 0 0 0 0 0 0 0 +count: 600002 600002 600002 600002 600002 600002 600002 600002 600002 600002 +min: 0 1 0 1 0 1 0 1 0 1 +avg: 0 5 0 5 0 4 0 4 0 5 +max: 16 36 15 58 24 44 21 46 13 50 +------------------------------------------ + +SEE ALSO +-------- +_rtla-timerlat(1)_, _rtla-timerlat-top(1)_ + +Timerlat tracer documentation: + +AUTHOR +------ +Written by Daniel Bristot de Oliveira + +REPORTING BUGS +-------------- +Report bugs to + +LICENSE +------- +rtla is Free Software licensed under the GNU GPLv2 + +COPYING +------- +Copyright \(C) 2021 Red Hat, Inc. Free use of this software is granted under +the terms of the GNU Public License (GPL). -- 2.31.1