Received: by 2002:a5d:9c59:0:0:0:0:0 with SMTP id 25csp2166235iof; Tue, 7 Jun 2022 21:57:23 -0700 (PDT) X-Google-Smtp-Source: ABdhPJx8DJw9bZMiu1XTWwdP968ZDv54W0kJQ8j9pzHcq4rgKHf7332JJwnGz3mlzZmDjSfJlYRM X-Received: by 2002:a17:903:2452:b0:166:4b6c:affb with SMTP id l18-20020a170903245200b001664b6caffbmr27491022pls.68.1654664243690; Tue, 07 Jun 2022 21:57:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1654664243; cv=none; d=google.com; s=arc-20160816; b=W5i31H8PkkS3ImhQlsQKWhGxigcd+ZWsZRshWHuhJ3IQhwJEPzAPLIWp/8SqMIanoK J7HmkUdW5ECLjIpeOgj/xA9strdVTNgla/eCOJGtxfy8BaRNCGvMkfLzJEYPfPVghTtR sU+XUGvtGBP93NJuYFZJwi1n9QgK7YAs8D6Cf+ISHgPj1ZMVKI8TyXI7q8iIIgK5XLSs B/zt3zMrEpeCm5O0hBrRGk1IApHz8fNQbXa7tG5wvwJmSbyNO9CxEP9/3QIakvxIHD+F zMTekk8YkJ6gi3g2fbnyjr5tOlA7s9G0Jalp3iaoC/icFDkKtPYAXJCJG4O/FkCizJbx j8JA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:subject:message-id:date:from:in-reply-to :references:mime-version:dkim-signature; bh=Sj24kgZtfXREXFgvbZGYuSQpYVOwbaKfG0SgT6LrWaE=; b=BKa3ayxOx2B2EygVWkt2H0EwI7fyGyoRlJkPXTFPyVocPvGpVVxpmQvNMXtJpFZ+wQ f6F+wWhp+LzoY4yq+QomfYJA0HO8EgbVLDqDkgGFX0eUxn9ZAoDCOoErXzehBldizyee au52XqKBPXMtHgh1Aq7CQ2gkkBfQwG/+Jx41JbNR4+CbAyx90BXnnW28/XZC2/cPkOsk hnhJVb7z5bXnG87IQORrgQytWDorw3nv1XuxVvJ+4bQ5BjwvmK9q8fYgVvGp8K4JknOh vTi1xC+1vcdEmt39oiByp/ffFLDsV6otMFXqqNXoHY8OgVrTegzg1qp7kSvEmZWTSRQ3 GCFA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=ftJs193n; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net. [2620:137:e000::1:18]) by mx.google.com with ESMTPS id d5-20020a170902728500b001624d97d474si24531734pll.183.2022.06.07.21.57.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jun 2022 21:57:23 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=ftJs193n; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id D85DD45E723; Tue, 7 Jun 2022 21:31:40 -0700 (PDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344704AbiFGQCI (ORCPT + 99 others); Tue, 7 Jun 2022 12:02:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53224 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344699AbiFGQCD (ORCPT ); Tue, 7 Jun 2022 12:02:03 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 44748DE301; Tue, 7 Jun 2022 09:02:00 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id j5-20020a05600c1c0500b0039c5dbbfa48so451406wms.5; Tue, 07 Jun 2022 09:02:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Sj24kgZtfXREXFgvbZGYuSQpYVOwbaKfG0SgT6LrWaE=; b=ftJs193nBakUNsSpferoDs0mNetkJfhPd+Av5tn8HvVLQPnQ/mfAjzz0pR30ZPtmlc E6Y6ZTLvFbWhQjY/brXMcv+vIQxAhOGZuCu3Wg31BDHZ7lfh29nINMKF+srY+P29g5GJ axVLYxIBk/KXfVSZ3/fYCobsKViM3MbgwtPSb9lcipwKfS4erGn64wg3QchUFIE1+EFQ vCcKYgyYWXgNEjxF+RZbElm0k6EkeLd8XsxYsSbvO2izDqpn2TnpG9UyuEG+JHiZE6AQ /kEnZBicwCJPOgeJTu3rRlV99PCZEUdW80MEZ03fKgWLpo+vkaKw2mm/Z42MRpwrPnfH dpXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Sj24kgZtfXREXFgvbZGYuSQpYVOwbaKfG0SgT6LrWaE=; b=Um/Rc6BFUu37bhuhHBAAFeCITuBByafBfSvsxOCsw6id1lkFECUlgYzYeNIGne3hfL 5/IcdmbrPLSYTfByP9jG6jKtByH+RW0JrIFZSO/0Kp8MlwrkCGo27Lz3SSyMyXtn8XLr z2n1QYUdUyTr6Db5Ork4hE23anVt0BDHL2OPKOtTeUeUgqLMXgj/f2MJ/zyQsSzapUeh 78VAbEzly/EgFGfQtiwb/FT6WAy/TycGsco5UAfZ2A//t9NWx27sWO8QJxy+ugIwWBR3 RVWYTy7lOq157pdmbI7jkB31l4WxENyVaWNGf0bVy/0dpunpWo8D2f5sLEAoIqLTvhEx 3ATQ== X-Gm-Message-State: AOAM531yFEz7bTzs6e3/8AsrXwwptrI5oYaYIIw62fCE8OUOM8BARZq7 EAVTaOA5mOHqbbTbIOc2juXn1G4plbF4Q3GIWzo= X-Received: by 2002:a05:600c:4f8e:b0:39c:52d6:3cc5 with SMTP id n14-20020a05600c4f8e00b0039c52d63cc5mr10311109wmq.84.1654617718464; Tue, 07 Jun 2022 09:01:58 -0700 (PDT) MIME-Version: 1.0 References: <20220606195432.1888346-1-robdclark@gmail.com> <20220606195432.1888346-2-robdclark@gmail.com> <8559b6f0-0322-9232-7000-534087e786fe@linux.intel.com> In-Reply-To: <8559b6f0-0322-9232-7000-534087e786fe@linux.intel.com> From: Rob Clark Date: Tue, 7 Jun 2022 09:02:00 -0700 Message-ID: Subject: Re: [PATCH v2 2/2] drm/msm: Expose client engine utilization via fdinfo To: Tvrtko Ursulin Cc: dri-devel , freedreno , linux-arm-msm , Rob Clark , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Daniel Vetter , Jonathan Corbet , Abhinav Kumar , Dmitry Baryshkov , Sean Paul , "open list:DOCUMENTATION" , open list Content-Type: text/plain; charset="UTF-8" X-Spam-Status: No, score=-1.7 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RDNS_NONE, SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Jun 7, 2022 at 1:56 AM Tvrtko Ursulin wrote: > > > On 06/06/2022 20:54, Rob Clark wrote: > > From: Rob Clark > > > > Similar to AMD commit > > 874442541133 ("drm/amdgpu: Add show_fdinfo() interface"), using the > > infrastructure added in previous patches, we add basic client info > > and GPU engine utilisation for msm. > > > > Example output: > > > > # cat /proc/`pgrep glmark2`/fdinfo/6 > > pos: 0 > > flags: 02400002 > > mnt_id: 21 > > ino: 162 > > drm-driver: msm > > drm-client-id: 7 > > drm-engine-gpu: 1734371319 ns > > drm-cycles-gpu: 1153645024 > > drm-maxfreq-gpu: 800000000 Hz > > > > See also: https://patchwork.freedesktop.org/patch/468505/ > > > > Signed-off-by: Rob Clark > > --- > > Documentation/gpu/drm-usage-stats.rst | 21 +++++++++++++++++++++ > > drivers/gpu/drm/msm/msm_drv.c | 19 ++++++++++++++++++- > > drivers/gpu/drm/msm/msm_gpu.c | 21 +++++++++++++++++++-- > > drivers/gpu/drm/msm/msm_gpu.h | 19 +++++++++++++++++++ > > 4 files changed, 77 insertions(+), 3 deletions(-) > > > > diff --git a/Documentation/gpu/drm-usage-stats.rst b/Documentation/gpu/drm-usage-stats.rst > > index 6c9f166a8d6f..60e5cc9c13ad 100644 > > --- a/Documentation/gpu/drm-usage-stats.rst > > +++ b/Documentation/gpu/drm-usage-stats.rst > > @@ -105,6 +105,27 @@ object belong to this client, in the respective memory region. > > Default unit shall be bytes with optional unit specifiers of 'KiB' or 'MiB' > > indicating kibi- or mebi-bytes. > > > > +- drm-cycles- > > + > > +Engine identifier string must be the same as the one specified in the > > +drm-engine- tag and shall contain the number of busy cycles for the given > > +engine. > > + > > +Values are not required to be constantly monotonic if it makes the driver > > +implementation easier, but are required to catch up with the previously reported > > +larger value within a reasonable period. Upon observing a value lower than what > > +was previously read, userspace is expected to stay with that larger previous > > +value until a monotonic update is seen. > > + > > +- drm-maxfreq- [Hz|MHz|KHz] > > + > > +Engine identifier string must be the same as the one specified in the > > +drm-engine- tag and shall contain the maxium frequence for the given > > maximum frequency > > > +engine. Taken together with drm-cycles-, this can be used to calculate > > +percentage utilization of the engine, whereas drm-engine- only refects > > reflects > > > +time active without considering what frequency the engine is operating as a > > +percentage of it's maximum frequency. > > Cycles vs max freq sounds very useful. My reservations is that how come > the idea hasn't happened in the CPU world. Or maybe it has and I am > un-informed? I do often pay attention to both where tasks get scheduled, and the individual CPU freq when I'm profiling CPU side stuff (eg. in perfetto) I could also report "always-count" cycles, I think, which could be used by gputop to derive freq. I'd have to think about that a bit, since keeping the result monotinic(ish) might be a bit tricky (the hw counter loses state across runtime suspend) > In any case, if going with this I think we need to clarify the text that > the value should reflect the current soft limit, where the driver > supports that, in case it has been set to lower than the maximum > frequency hardware can support. I am thinking about avoiding "my gpu > cannot hit 100%" support incidents in cases when user/admin lowered the > soft limit for some reason. Possibly does not apply to msm but can apply > to i915, if we decided to export the same data. Yes, with pm-qos thermal or userspace could limit the max freq.. but we also internally use a pm-qos constraint to reduce freq when the GPU is idle, and I don't think there is a good way to differentiate *which* constraint is which. I'll add something involving the word "recommended" ;-) BR, -R > > No other gotchas come to mind at the moment. > > Regards, > > Tvrtko > > > + > > =============================== > > Driver specific implementations > > =============================== > > diff --git a/drivers/gpu/drm/msm/msm_drv.c b/drivers/gpu/drm/msm/msm_drv.c > > index 14ab9a627d8b..57a66093e671 100644 > > --- a/drivers/gpu/drm/msm/msm_drv.c > > +++ b/drivers/gpu/drm/msm/msm_drv.c > > @@ -948,7 +948,24 @@ static const struct drm_ioctl_desc msm_ioctls[] = { > > DRM_IOCTL_DEF_DRV(MSM_SUBMITQUEUE_QUERY, msm_ioctl_submitqueue_query, DRM_RENDER_ALLOW), > > }; > > > > -DEFINE_DRM_GEM_FOPS(fops); > > +static void msm_fop_show_fdinfo(struct seq_file *m, struct file *f) > > +{ > > + struct drm_file *file = f->private_data; > > + struct drm_device *dev = file->minor->dev; > > + struct msm_drm_private *priv = dev->dev_private; > > + struct drm_printer p = drm_seq_file_printer(m); > > + > > + if (!priv->gpu) > > + return; > > + > > + msm_gpu_show_fdinfo(priv->gpu, file->driver_priv, &p); > > +} > > + > > +static const struct file_operations fops = { > > + .owner = THIS_MODULE, > > + DRM_GEM_FOPS, > > + .show_fdinfo = msm_fop_show_fdinfo, > > +}; > > > > static const struct drm_driver msm_driver = { > > .driver_features = DRIVER_GEM | > > diff --git a/drivers/gpu/drm/msm/msm_gpu.c b/drivers/gpu/drm/msm/msm_gpu.c > > index eb8a6663f309..333a9a299b41 100644 > > --- a/drivers/gpu/drm/msm/msm_gpu.c > > +++ b/drivers/gpu/drm/msm/msm_gpu.c > > @@ -4,6 +4,8 @@ > > * Author: Rob Clark > > */ > > > > +#include "drm/drm_drv.h" > > + > > #include "msm_gpu.h" > > #include "msm_gem.h" > > #include "msm_mmu.h" > > @@ -146,6 +148,16 @@ int msm_gpu_pm_suspend(struct msm_gpu *gpu) > > return 0; > > } > > > > +void msm_gpu_show_fdinfo(struct msm_gpu *gpu, struct msm_file_private *ctx, > > + struct drm_printer *p) > > +{ > > + drm_printf(p, "drm-driver:\t%s\n", gpu->dev->driver->name); > > + drm_printf(p, "drm-client-id:\t%u\n", ctx->seqno); > > + drm_printf(p, "drm-engine-gpu:\t%llu ns\n", ctx->elapsed_ns); > > + drm_printf(p, "drm-cycles-gpu:\t%llu\n", ctx->cycles); > > + drm_printf(p, "drm-maxfreq-gpu:\t%lu Hz\n", gpu->fast_rate); > > +} > > + > > int msm_gpu_hw_init(struct msm_gpu *gpu) > > { > > int ret; > > @@ -652,7 +664,7 @@ static void retire_submit(struct msm_gpu *gpu, struct msm_ringbuffer *ring, > > { > > int index = submit->seqno % MSM_GPU_SUBMIT_STATS_COUNT; > > volatile struct msm_gpu_submit_stats *stats; > > - u64 elapsed, clock = 0; > > + u64 elapsed, clock = 0, cycles; > > unsigned long flags; > > > > stats = &ring->memptrs->stats[index]; > > @@ -660,12 +672,17 @@ static void retire_submit(struct msm_gpu *gpu, struct msm_ringbuffer *ring, > > elapsed = (stats->alwayson_end - stats->alwayson_start) * 10000; > > do_div(elapsed, 192); > > > > + cycles = stats->cpcycles_end - stats->cpcycles_start; > > + > > /* Calculate the clock frequency from the number of CP cycles */ > > if (elapsed) { > > - clock = (stats->cpcycles_end - stats->cpcycles_start) * 1000; > > + clock = cycles * 1000; > > do_div(clock, elapsed); > > } > > > > + submit->queue->ctx->elapsed_ns += elapsed; > > + submit->queue->ctx->cycles += cycles; > > + > > trace_msm_gpu_submit_retired(submit, elapsed, clock, > > stats->alwayson_start, stats->alwayson_end); > > > > diff --git a/drivers/gpu/drm/msm/msm_gpu.h b/drivers/gpu/drm/msm/msm_gpu.h > > index 6def00883046..4911943ba53b 100644 > > --- a/drivers/gpu/drm/msm/msm_gpu.h > > +++ b/drivers/gpu/drm/msm/msm_gpu.h > > @@ -361,6 +361,22 @@ struct msm_file_private { > > /** cmdline: Overridden task cmdline, see MSM_PARAM_CMDLINE */ > > char *cmdline; > > > > + /** > > + * elapsed: > > + * > > + * The total (cumulative) elapsed time GPU was busy with rendering > > + * from this context in ns. > > + */ > > + uint64_t elapsed_ns; > > + > > + /** > > + * cycles: > > + * > > + * The total (cumulative) GPU cycles elapsed attributed to this > > + * context. > > + */ > > + uint64_t cycles; > > + > > /** > > * entities: > > * > > @@ -544,6 +560,9 @@ static inline void gpu_write64(struct msm_gpu *gpu, u32 lo, u32 hi, u64 val) > > int msm_gpu_pm_suspend(struct msm_gpu *gpu); > > int msm_gpu_pm_resume(struct msm_gpu *gpu); > > > > +void msm_gpu_show_fdinfo(struct msm_gpu *gpu, struct msm_file_private *ctx, > > + struct drm_printer *p); > > + > > int msm_submitqueue_init(struct drm_device *drm, struct msm_file_private *ctx); > > struct msm_gpu_submitqueue *msm_submitqueue_get(struct msm_file_private *ctx, > > u32 id);