Received: by 2002:a25:4158:0:0:0:0:0 with SMTP id o85csp5651169yba; Mon, 13 May 2019 14:51:19 -0700 (PDT) X-Google-Smtp-Source: APXvYqx9fwkL9b+6dSXw+terI60gu6C6j5qbxMsGhIAw8QaOqIrJmZ5LQCqABXqidVQb09TzncjK X-Received: by 2002:a17:902:f096:: with SMTP id go22mr35076872plb.49.1557784278971; Mon, 13 May 2019 14:51:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557784278; cv=none; d=google.com; s=arc-20160816; b=sXRX5RgAoXxLV+k9YDBealbPEVBOhN0ErkzKnQivYUByW8S54DMkwPLicHGqlpIBu/ FIgFvzvu+JnMWf2EtRNzFsjZfoVA13sg6MhIX1XErvaP9LtohEw0Rx8FsRFUb2e215FL 87kzeA1i7vdkuzfPPkXL9hK4kGKlHN6u8K3KdF+FBGLT4oIPmgL40Z6GCqca4JOYqfaO hds7pVYzgHy+b1RJQF0Fko+STXBGUQYtqFbu6E7mC5jcD1ECFNepEbVOXoAhbFeCEusf gSsulg1dsAO4oszcgJtVzpz8/nKP2O49NktZk+Ly6yNguboPfp4DJrB9eammT+bbTwkN 66hQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:date:subject:cc:to:from :dkim-signature; bh=Hov/pCeC1dnaeog3UO9V4+XwoPV+/3/2gv4pYq+t7W8=; b=n2q5lUJppiztu156d19uZn1tuUSHctTrEerZRZMUifNBBSytINarpGKL2Mnz0soKUo WjW31208Ft+uM0Yd94IubLXGvUm+Xx70P67H911g8NoOajmp3NDaWfKyYZNuUiApySAo GY0J2E9gNptOSUP7C3assGV5eVPf4cJUGKbSdXRlF+xDHCOKtlQVuDG2pNuUJq9Lwff1 z+FXI0KMg12q0QeUW8Kk/29vnAGKt6Ynixv/2ZyuOUrvGfM4E8fIhKj6XvshpFxZ8QE9 3P2+5lihSV/rIHIR9xxNMJOaMkY7lHlgMaXDoRUgKcP+YzZUbzJ8ejfXNKcn0I8pIjhy Mkow== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=j2PI4uQZ; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id h11si5024009pgq.170.2019.05.13.14.51.02; Mon, 13 May 2019 14:51:18 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=j2PI4uQZ; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726465AbfEMVuO (ORCPT + 99 others); Mon, 13 May 2019 17:50:14 -0400 Received: from mail-ed1-f66.google.com ([209.85.208.66]:46238 "EHLO mail-ed1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726339AbfEMVuO (ORCPT ); Mon, 13 May 2019 17:50:14 -0400 Received: by mail-ed1-f66.google.com with SMTP id f37so19637203edb.13 for ; Mon, 13 May 2019 14:50:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=Hov/pCeC1dnaeog3UO9V4+XwoPV+/3/2gv4pYq+t7W8=; b=j2PI4uQZAUXecuffmlcE1djUzaj9k8/4wV+sYCML2smEZm3KqEQc44/gjJ0iKsMkbu P9Uo9dA5qrag/GpYyxI+dU/ULJ8xJkWrw8XnvdL7outoIrgEXgST3LT/iuxxEEZXU+Ar C5S+toPXxlJijMjKAG3XCYgQ5w7va+uwfF+zlszaLFPaGK3Q2PFDRvyDZH9u003aUinZ 4k6qtiJIdETNzWiJe77pLFXuo9RYd6IH+FU3+3mPFLIYTXSZ/tEoiZlux05qtrBpcloo 1dY00x1apOcLxX12Hg3ALrlXKtR0hB//ciic6ddGpYfIOTyH2w1ILtWHRayFIVREFueu vKEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=Hov/pCeC1dnaeog3UO9V4+XwoPV+/3/2gv4pYq+t7W8=; b=s6tCin8CZUVQ1L5FfVCZsIuP5VLLrMKiOsXi41oNAZZNv706tCDJTFeqNuUelDjYOe y3VTeogzKvpc+jG1WwERqkvd3jWQm7AMp5ogV0B8gpbMl7uUh+pDSlHcgeBKlObXnNML /1FAGpLWtTQBK5410/t27yorzjR7IHCK0hYYNMxfarEXKSgDFBMq3WeiXtuiVYGCmoyq +ivwpfYeSqEHu2PZTEJlT5uolwQIHv84ugK08+dyd8z0wkL4Ez9dqMUP4Kme8ukWDNU9 2oBHQhkq+50UJh7K23LoZBSw0k917bQ0GDpiopc9oz+iZrt44yMfPGcJ+TQiRkjLbOce KY6w== X-Gm-Message-State: APjAAAWGapGQxywJWslFdyyx213kpGIXD9Artz0AHDKwhzzmLKGe0Tfx NnC34Ih+hRS4/+IUnB7fvQ== X-Received: by 2002:a50:a941:: with SMTP id m1mr31956525edc.157.1557784212192; Mon, 13 May 2019 14:50:12 -0700 (PDT) Received: from localhost.localdomain ([92.117.184.230]) by smtp.gmail.com with ESMTPSA id g11sm4040891eda.42.2019.05.13.14.50.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 13 May 2019 14:50:11 -0700 (PDT) From: Viktor Rosendahl To: Steven Rostedt , Ingo Molnar , linux-kernel@vger.kernel.org Cc: Joel Fernandes , Viktor Rosendahl Subject: [PATCH v3 0/4] Some new features for the latency tracers Date: Mon, 13 May 2019 23:50:04 +0200 Message-Id: <20190513215008.11256-1-viktor.rosendahl@gmail.com> X-Mailer: git-send-email 2.17.1 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hello all, Changes in v3: - [PATCH 1/4]: * I have generalized this to send the fsnotify() notifications for all tracers that use tracing_max_latency. * There are large additions of code to prevent queue_work() from being called while we are in __schedule() or do_idle(). This was a bug also in the previous version but had gone unnoticed. It became very visible when I tried to make it work with the wakeup tracers because they are always invoked from the sched_switch tracepoint. * The fsnotify notification is issued for tracing_max_latency, not trace. * I got rid of the kernel config option. The facility is always compiled when CONFIG_FSNOTIFY is enabled and any of the latency tracers are enabled. - [PATCH 2/4]: * No significant changes. - [PATCH 3/4]: * The latency-collector help messages have been tuned to the fact that it can be used also with wakeup and hwlat tracers. * I increased the size of the buffer for reading from /sys/kernel/debug/tracing/trace. * Adapted it to monitor tracing_max_latency instead of trace - [PATCH 4/4]: * I converted this from a kernel config option to a trace config option that can be changed at runtime. This series is meant to address two issues with the latency tracing. The first three patches provide a method to trace latencies that always occurs very close to each other and to differentiate between them, in spite of the fact that the latency tracers always work in overwrite mode. [PATCH 1/4] This implement fs notification for preempt/irqsoff. It makes it possible for userspace to detect when a new latency has been detected. [PATCH 2/4] This extends the preemptirq_delay_test module so that it can be used to generate a burst of closely occurring latencies. [PATCH 3/4] This adds a user space program to the tools directory that utilizes the fs notification feature and a randomized algorithm to print out any of the latencies in a burst with approximately equal probability. The last patch is not directly connected but doesn't apply cleanly on its own: [PATCH 4/4] This adds the option console-latency to the trace options. best regards, Viktor Viktor Rosendahl (4): ftrace: Implement fs notification for tracing_max_latency preemptirq_delay_test: Add the burst feature and a sysfs trigger Add the latency-collector to tools ftrace: Add an option for tracing console latencies include/linux/irqflags.h | 21 + include/trace/events/power.h | 40 + include/trace/events/sched.h | 40 + kernel/printk/printk.c | 6 +- kernel/sched/core.c | 2 + kernel/sched/idle.c | 2 + kernel/trace/Kconfig | 6 +- kernel/trace/preemptirq_delay_test.c | 145 ++- kernel/trace/trace.c | 160 +++- kernel/trace/trace.h | 11 + kernel/trace/trace_hwlat.c | 4 +- kernel/trace/trace_irqsoff.c | 12 + tools/Makefile | 14 +- tools/trace/Makefile | 20 + tools/trace/latency-collector.c | 1211 ++++++++++++++++++++++++++ 15 files changed, 1661 insertions(+), 33 deletions(-) create mode 100644 tools/trace/Makefile create mode 100644 tools/trace/latency-collector.c -- 2.17.1