Received: by 2002:a05:6358:11c7:b0:104:8066:f915 with SMTP id i7csp1469952rwl; Wed, 12 Apr 2023 13:25:23 -0700 (PDT) X-Google-Smtp-Source: AKy350aKhGox9LgL7xWJBmFD5XsDVGjCKszQ+vknUtIwEKL5yH6tPtqWEqXLluD2zviCoq2xRFNe X-Received: by 2002:a05:6a00:2e9e:b0:63b:1e80:5912 with SMTP id fd30-20020a056a002e9e00b0063b1e805912mr120581pfb.29.1681331123661; Wed, 12 Apr 2023 13:25:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1681331123; cv=none; d=google.com; s=arc-20160816; b=g5p4PS8U46e80JTZsRs1c4SaXDDBmUU9q9qx9/xVKRVV+cKkYLTIatfnGfFrNVcASp 4Z3vePGXsH2SdOo4TsYrY6PLf+CdbPw5pKPiYUhG7tMtIfuzdM5p4kONvqyKk+w1wp5+ 8wLBI7xqPPIBC8cjWbwZ6HmmpdAEhl35wuVhmKx38r34DnvRguyFLMi8z7c3Hohntlwf PkqDUVm/iHUzqgYXewueKtrn9xUtypBe9/+upiosfKqm2/R2i7ztv+VQTOi6OL26TvdS EZnNPuDYKxZSLxTb6xxLzw8OFmnHWc1VGX0uwYV4P7B0VMqF1ZmRaQGOh/U85+TRIdjt 8Hjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:cc:to:subject :message-id:date:from:in-reply-to:references:mime-version :dkim-signature; bh=UTBfRC0LvSg06QgiZSsEJ06WzqdIIIpng+1SlnJvGDg=; b=N3Bm4Gseo1EvjXZFtNLUV/sHmiCfd4a/Z3Yq9uw+70mQ6P7WMtNuXiCAOh7WA45fDt JU22SP25JRe8tYzIBRkRsjHARYBrQ+gGzexfSePuKD9ZtTk/gDdqiPcyLlWerEAD8MVf A0DNYGzbbZzrPKjkaarXRenXa5FeX21pUiCmy8tLI30WN04YCv1V1KHNbXKtdMfFmLWo DHew5huZpsJeL09NI7SUjYSX6gxDZuthye1tITtv2mVqs8341VdmrIzEuUbEwAjB3tSQ 1ykIzCst2gRe4zlGcoN/RJQlbPiTv9KlMrlZr4nhlbFp4ItZpgVnS0NaHO4B8LZBg9aX 9Z4g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=wnRJwBqm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b8-20020a63d808000000b0051948f1993fsi63946pgh.40.2023.04.12.13.25.09; Wed, 12 Apr 2023 13:25:23 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=wnRJwBqm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229968AbjDLUTg (ORCPT + 99 others); Wed, 12 Apr 2023 16:19:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33644 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229960AbjDLUTf (ORCPT ); Wed, 12 Apr 2023 16:19:35 -0400 Received: from mail-yw1-x1133.google.com (mail-yw1-x1133.google.com [IPv6:2607:f8b0:4864:20::1133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E77B159C5 for ; Wed, 12 Apr 2023 13:19:31 -0700 (PDT) Received: by mail-yw1-x1133.google.com with SMTP id 00721157ae682-54f21cdfadbso190254267b3.7 for ; Wed, 12 Apr 2023 13:19:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1681330771; x=1683922771; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=UTBfRC0LvSg06QgiZSsEJ06WzqdIIIpng+1SlnJvGDg=; b=wnRJwBqmiACuHtpVEqIwHDdg3GA5XSHjop9evKgRGBqje+3qMPEbYpxIcBTBgEJfut 8Egwyx/v2cAAFWyjEwYAISZZvbH+9Ftu1f3IsT+GapthMoVtLgGUZUJngah+8MNWV10L PIP6/EYq9fGAchHg/+UJZ3xWWsmb1sX+W94wm8T+n1f3lxaq7XBqAtwB3rJv/W2Qei+m 6ju/65zcJNmKXOMzro5s/V72TalgrZQPCTZPU+cPBHnRnmcXE+NcHVyXfuf+aiqXLAsq DJ6FwmPAmUQ9B8N9Y3Iv1PgasZnhVO2W0fP3g6egRhVfotUzQxIOIWnsSd3RChr6Ab8G FPdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681330771; x=1683922771; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UTBfRC0LvSg06QgiZSsEJ06WzqdIIIpng+1SlnJvGDg=; b=ltyW8EVOKrfjv7ihScWtPTsm0demkdjHlVxSRJCBSe4tzHQA5pbMQLjV47qTW3FUyI 2j6SMjhSFF61Ejb8xcUNJEe0kYtmK9siOfBLPeaF2ADlixv5dmPXVTcj6LsRFA4bPNYz Yiax9edwR77DzGzbsDKEOOFcgeybCifOJlO42z2Xmm2Ivxt39iWkOCBuLzF5hsFXBKLo a0KteFFyByn3tLgwZn/m4swptEG5dZD5oAFkcNCyEM+6GuOJgRq5MKMDkWjZ1fZAZQIo xIUcoPffDmHPm9Ug4bxDGltd2uILrzYnbXC+dRTTEReHlIr9/hPWg0JvK2DecAJqQWc5 tQGg== X-Gm-Message-State: AAQBX9e5bq2Ils6xjnMTlQCxx8Aa2qZXRxXot0YTZ8CzufEWT7rHoFXL Utdgxyil3SanQfoyDpRG43n7aGjzqUH+1/Qz8gPM0Q== X-Received: by 2002:a81:af0e:0:b0:54f:85a6:c80c with SMTP id n14-20020a81af0e000000b0054f85a6c80cmr2983965ywh.5.1681330771013; Wed, 12 Apr 2023 13:19:31 -0700 (PDT) MIME-Version: 1.0 References: <20230410210608.1873968-1-robdclark@gmail.com> In-Reply-To: From: Dmitry Baryshkov Date: Wed, 12 Apr 2023 23:19:19 +0300 Message-ID: Subject: Re: [Freedreno] [PATCH v2 0/2] drm: fdinfo memory stats To: Rob Clark Cc: Rodrigo Vivi , dri-devel@lists.freedesktop.org, Rob Clark , Tvrtko Ursulin , "open list:DOCUMENTATION" , linux-arm-msm@vger.kernel.org, Emil Velikov , Christopher Healy , open list , Sean Paul , Boris Brezillon , freedreno@lists.freedesktop.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, 12 Apr 2023 at 23:09, Rob Clark wrote: > > On Wed, Apr 12, 2023 at 5:47=E2=80=AFAM Rodrigo Vivi wrote: > > > > On Wed, Apr 12, 2023 at 10:11:32AM +0200, Daniel Vetter wrote: > > > On Wed, Apr 12, 2023 at 01:36:52AM +0300, Dmitry Baryshkov wrote: > > > > On 11/04/2023 21:28, Rob Clark wrote: > > > > > On Tue, Apr 11, 2023 at 10:36=E2=80=AFAM Dmitry Baryshkov > > > > > wrote: > > > > > > > > > > > > On Tue, 11 Apr 2023 at 20:13, Rob Clark w= rote: > > > > > > > > > > > > > > On Tue, Apr 11, 2023 at 9:53=E2=80=AFAM Daniel Vetter wrote: > > > > > > > > > > > > > > > > On Tue, Apr 11, 2023 at 09:47:32AM -0700, Rob Clark wrote: > > > > > > > > > On Mon, Apr 10, 2023 at 2:06=E2=80=AFPM Rob Clark wrote: > > > > > > > > > > > > > > > > > > > > From: Rob Clark > > > > > > > > > > > > > > > > > > > > Similar motivation to other similar recent attempt[1]. = But with an > > > > > > > > > > attempt to have some shared code for this. As well as = documentation. > > > > > > > > > > > > > > > > > > > > It is probably a bit UMA-centric, I guess devices with = VRAM might want > > > > > > > > > > some placement stats as well. But this seems like a re= asonable start. > > > > > > > > > > > > > > > > > > > > Basic gputop support: https://patchwork.freedesktop.org= /series/116236/ > > > > > > > > > > And already nvtop support: https://github.com/Syllo/nvt= op/pull/204 > > > > > > > > > > > > > > > > > > On a related topic, I'm wondering if it would make sense = to report > > > > > > > > > some more global things (temp, freq, etc) via fdinfo? So= me of this, > > > > > > > > > tools like nvtop could get by trawling sysfs or other dri= ver specific > > > > > > > > > ways. But maybe it makes sense to have these sort of thi= ngs reported > > > > > > > > > in a standardized way (even though they aren't really per= -drm_file) > > > > > > > > > > > > > > > > I think that's a bit much layering violation, we'd essentia= lly have to > > > > > > > > reinvent the hwmon sysfs uapi in fdinfo. Not really a busin= ess I want to > > > > > > > > be in :-) > > > > > > > > > > > > > > I guess this is true for temp (where there are thermal zones = with > > > > > > > potentially multiple temp sensors.. but I'm still digging my = way thru > > > > > > > the thermal_cooling_device stuff) > > > > > > > > > > > > It is slightly ugly. All thermal zones and cooling devices are = virtual > > > > > > devices (so, even no connection to the particular tsens device)= . One > > > > > > can either enumerate them by checking > > > > > > /sys/class/thermal/thermal_zoneN/type or enumerate them through > > > > > > /sys/class/hwmon. For cooling devices again the only enumeratio= n is > > > > > > through /sys/class/thermal/cooling_deviceN/type. > > > > > > > > > > > > Probably it should be possible to push cooling devices and ther= mal > > > > > > zones under corresponding providers. However I do not know if t= here is > > > > > > a good way to correlate cooling device (ideally a part of GPU) = to the > > > > > > thermal_zone (which in our case is provided by tsens / temp_ala= rm > > > > > > rather than GPU itself). > > > > > > > > > > > > > > > > > > > > But what about freq? I think, esp for cases where some "fw t= hing" is > > > > > > > controlling the freq we end up needing to use gpu counters to= measure > > > > > > > the freq. > > > > > > > > > > > > For the freq it is slightly easier: /sys/class/devfreq/*, devic= es are > > > > > > registered under proper parent (IOW, GPU). So one can read > > > > > > /sys/class/devfreq/3d00000.gpu/cur_freq or > > > > > > /sys/bus/platform/devices/3d00000.gpu/devfreq/3d00000.gpu/cur_f= req. > > > > > > > > > > > > However because of the components usage, there is no link from > > > > > > /sys/class/drm/card0 > > > > > > (/sys/devices/platform/soc@0/ae00000.display-subsystem/ae01000.= display-controller/drm/card0) > > > > > > to /sys/devices/platform/soc@0/3d00000.gpu, the GPU unit. > > > > > > > > > > > > Getting all these items together in a platform-independent way = would > > > > > > be definitely an important but complex topic. > > > > > > > > > > But I don't believe any of the pci gpu's use devfreq ;-) > > > > > > > > > > And also, you can't expect the CPU to actually know the freq when= fw > > > > > is the one controlling freq. We can, currently, have a reasonabl= e > > > > > approximation from devfreq but that stops if IFPC is implemented.= And > > > > > other GPUs have even less direct control. So freq is a thing tha= t I > > > > > don't think we should try to get from "common frameworks" > > > > > > > > I think it might be useful to add another passive devfreq governor = type for > > > > external frequencies. This way we can use the same interface to exp= ort > > > > non-CPU-controlled frequencies. > > > > > > Yeah this sounds like a decent idea to me too. It might also solve th= e fun > > > of various pci devices having very non-standard freq controls in sysf= s > > > (looking at least at i915 here ...) > > > > I also like the idea of having some common infrastructure for the GPU f= req. > > > > hwmon have a good infrastructure, but they are more focused on individu= al > > monitoring devices and not very welcomed to embedded monitoring and con= trol. > > I still want to check the opportunity to see if at least some freq cont= rol > > could be aligned there. > > > > Another thing that complicates that is that there are multiple frequenc= y > > domains and controls with multipliers in Intel GPU that are not very > > standard or easy to integrate. > > > > On a quick glace this devfreq seems neat because it aligns with the cpu= freq > > and governors. But again it would be hard to align with the multiple do= mains > > and controls. But it deserves a look. > > > > I will take a look to both fronts for Xe: hwmon and devfreq. Right now = on > > Xe we have a lot less controls than i915, but I can imagine soon there > > will be requirements to make that to grow and I fear that we end up jus= t > > like i915. So I will take a look before that happens. > > So it looks like i915 (dgpu only) and nouveau already use hwmon.. so > maybe this is a good way to expose temp. Maybe we can wire up some > sort of helper for drivers which use thermal_cooling_device (which can > be composed of multiple sensors) to give back an aggregate temp for > hwmon to report? The thermal_device already registers the hwmon, see below. The question is about linking that hwmon to the drm. Strictly speaking, I don't think that we can reexport it in a clean way. # grep gpu /sys/class/hwmon/hwmon*/name /sys/class/hwmon/hwmon15/name:gpu_top_thermal /sys/class/hwmon/hwmon24/name:gpu_bottom_thermal # ls /sys/class/hwmon/hwmon15/ -l lrwxrwxrwx 1 root root 0 Jan 26 08:14 device -> ../../thermal_zone15 -r--r--r-- 1 root root 4096 Jan 26 08:14 name drwxr-xr-x 2 root root 0 Jan 26 08:15 power lrwxrwxrwx 1 root root 0 Jan 26 08:12 subsystem -> ../../../../../class/hwmon -r--r--r-- 1 root root 4096 Jan 26 08:14 temp1_input -rw-r--r-- 1 root root 4096 Jan 26 08:12 uevent > Freq could possibly be added to hwmon (ie. seems like a reasonable > attribute to add). Devfreq might also be an option but on arm it > isn't necessarily associated with the drm device, whereas we could > associate the hwmon with the drm device to make it easier for > userspace to find. Possibly we can register a virtual 'passive' devfreq being driven by another active devfreq device. > > BR, > -R > > > > > > > I guess it would minimally be a good idea if we could document this, = or > > > maybe have a reference implementation in nvtop or whatever the cool t= hing > > > is rn. > > > -Daniel > > > > > > > > > > > > > > > > > BR, > > > > > -R > > > > > > > > > > > > > > > > > > > > What might be needed is better glue to go from the fd or fd= info to the > > > > > > > > right hw device and then crawl around the hwmon in sysfs au= tomatically. I > > > > > > > > would not be surprised at all if we really suck on this, pr= obably more > > > > > > > > likely on SoC than pci gpus where at least everything shoul= d be under the > > > > > > > > main pci sysfs device. > > > > > > > > > > > > > > yeah, I *think* userspace would have to look at /proc/device-= tree to > > > > > > > find the cooling device(s) associated with the gpu.. at least= I don't > > > > > > > see a straightforward way to figure it out just for sysfs > > > > > > > > > > > > > > BR, > > > > > > > -R > > > > > > > > > > > > > > > -Daniel > > > > > > > > > > > > > > > > > > > > > > > > > > BR, > > > > > > > > > -R > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] https://patchwork.freedesktop.org/series/112397/ > > > > > > > > > > > > > > > > > > > > Rob Clark (2): > > > > > > > > > > drm: Add fdinfo memory stats > > > > > > > > > > drm/msm: Add memory stats to fdinfo > > > > > > > > > > > > > > > > > > > > Documentation/gpu/drm-usage-stats.rst | 21 +++++++ > > > > > > > > > > drivers/gpu/drm/drm_file.c | 79 ++++++++++= +++++++++++++++++ > > > > > > > > > > drivers/gpu/drm/msm/msm_drv.c | 25 ++++++++- > > > > > > > > > > drivers/gpu/drm/msm/msm_gpu.c | 2 - > > > > > > > > > > include/drm/drm_file.h | 10 ++++ > > > > > > > > > > 5 files changed, 134 insertions(+), 3 deletions(-) > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > 2.39.2 > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Daniel Vetter > > > > > > > > Software Engineer, Intel Corporation > > > > > > > > http://blog.ffwll.ch > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > With best wishes > > > > > > Dmitry > > > > > > > > -- > > > > With best wishes > > > > Dmitry > > > > > > > > > > -- > > > Daniel Vetter > > > Software Engineer, Intel Corporation > > > http://blog.ffwll.ch --=20 With best wishes Dmitry