Received: by 2002:a05:6358:45e:b0:b5:b6eb:e1f9 with SMTP id 30csp868303rwe; Thu, 1 Sep 2022 08:47:46 -0700 (PDT) X-Google-Smtp-Source: AA6agR402Wrzxw6Id5dveKpqaj5b3Fg/x5dXNE2oBfVqMHOW5CntfNnt1r/S/txh+XLN0In/IHL1 X-Received: by 2002:a17:906:cc0c:b0:73d:d6ce:5d6b with SMTP id ml12-20020a170906cc0c00b0073dd6ce5d6bmr23573818ejb.24.1662047266364; Thu, 01 Sep 2022 08:47:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1662047266; cv=none; d=google.com; s=arc-20160816; b=llEKJocDXGnPuJhbfZF6bQWdSBBvFBofx9piMj2q/qpUKwMmcn4ZV+dQWKnp+qBWtj 6rguMH4ZQuXAguaF9eIbLV+00JjM9gx3Ucduy685hq7pXofD8BWbluI4ReU5kiLC/30D +HpLsiZka567hFju7vTOHCRgi/2St6ap3uyLGOmEseCxkU9AmGCVkWUiSZ+ia890lEqT dNVLKDKyUMVtkzgipYi5DceBvltN7MDxM/93rkWqBSTd3aXEkYcvNQOjtuJHZZeiekaI 2QvdVVEC7doVLhd3NULRb0JYXivD8KBya4BVrWYU0f6LlvCH3BSEC/Q/GvTYma71QLNc m6rA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:dkim-signature:date; bh=nOHSW3FDpFxopIM9LbojW3h32sP5UuFpnfFE44cGJM8=; b=vUiFNYBuI5kQbsTTISuSigIfJ4y0gv4hiKaL4VbeoHVivJGGrvabgOlrzYFYIudX9n rhoVKw4BtvkKsNsSd3I6fa3ba/vBo3D5lMcUUrQd7qjxW5x/pUaGQtf7ZCpEl0Ttm8ma Rz1ugLQUxo52gr5GRDfKBeByJ4RoQhgwHgvTTWO3uiqYe487S9FN6aAinoOXF/mqLtdN FPNF5w41/fW7vChNg3Kwod4gYR+P/G0dOI5yZEvOwDelXTKze65wpkCaibODyT/1xnqy dMGubX+9ey3UcC3uPUnJKnOXr725Bc4pEfZInEES7omyQCRFeaa7I2USn9DMMnNCwe23 fQig== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=DYXiWVC0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id kk24-20020a170907767800b0073d80c9dc85si10862563ejc.459.2022.09.01.08.47.15; Thu, 01 Sep 2022 08:47:46 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=DYXiWVC0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234312AbiIAOnX (ORCPT + 99 others); Thu, 1 Sep 2022 10:43:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43568 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233597AbiIAOnV (ORCPT ); Thu, 1 Sep 2022 10:43:21 -0400 Received: from out2.migadu.com (out2.migadu.com [IPv6:2001:41d0:2:aacc::]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0AED76976; Thu, 1 Sep 2022 07:43:19 -0700 (PDT) Date: Thu, 1 Sep 2022 10:43:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1662043398; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=nOHSW3FDpFxopIM9LbojW3h32sP5UuFpnfFE44cGJM8=; b=DYXiWVC0vy4teLKZQztFxvW1KXEKgsv/W8GLVht+MMEppphvXl1beWtjobZtBs7ufDfyf1 /Az+HrU+UnzwRWndfnxlx9U/cWoea5qHWVDTW/7rGq5kcj9ODLnVIRzkPJn+RdzRKpoYMZ 5Z1XsUa8iU3hq/D237KbAKpwZVrQllY= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Kent Overstreet To: Peter Zijlstra Cc: Suren Baghdasaryan , akpm@linux-foundation.org, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, void@manifault.com, juri.lelli@redhat.com, ldufour@linux.ibm.com, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, changbin.du@intel.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, arnd@arndb.de, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, kernel-team@android.com, linux-mm@kvack.org, iommu@lists.linux.dev, kasan-dev@googlegroups.com, io-uring@vger.kernel.org, linux-arch@vger.kernel.org, xen-devel@lists.xenproject.org, linux-bcache@vger.kernel.org, linux-modules@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [RFC PATCH 27/30] Code tagging based latency tracking Message-ID: <20220901144311.ywhhdaigweyy7eo6@moria.home.lan> References: <20220830214919.53220-1-surenb@google.com> <20220830214919.53220-28-surenb@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,SPF_HELO_PASS, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Sep 01, 2022 at 09:11:17AM +0200, Peter Zijlstra wrote: > On Tue, Aug 30, 2022 at 02:49:16PM -0700, Suren Baghdasaryan wrote: > > From: Kent Overstreet > > > > This adds the ability to easily instrument code for measuring latency. > > To use, add the following to calls to your code, at the start and end of > > the event you wish to measure: > > > > code_tag_time_stats_start(start_time); > > code_tag_time_stats_finish(start_time); > > > > Stastistics will then show up in debugfs under > > /sys/kernel/debug/time_stats, listed by file and line number. > > > > Stastics measured include weighted averages of frequency, duration, max > > duration, as well as quantiles. > > > > This patch also instruments all calls to init_wait and finish_wait, > > which includes all calls to wait_event. Example debugfs output: > > How can't you do this with a simple eBPF script on top of > trace_sched_stat_* and friends? I know about those tracepoints, and I've never found them to be usable. I've never succesfully used them for debugging latency issues, or known anyone who has. And an eBPF script to do everything this does wouldn't be simple at all. Honesly, the time stats stuff looks _far_ simpler to me than anything involving tracing - and with tracing you have to correlate the start and end events after the fact.