Received: by 2002:a25:4158:0:0:0:0:0 with SMTP id o85csp1976561yba; Sat, 27 Apr 2019 11:12:02 -0700 (PDT) X-Google-Smtp-Source: APXvYqz+gwfI6HWeyvLpdsPgJsb7Iz2UJ5ZRHts9HmS9bAWFl74p79Aaqqogh7GDlP9NWO9a7W2f X-Received: by 2002:a17:902:bb84:: with SMTP id m4mr16907454pls.302.1556388721900; Sat, 27 Apr 2019 11:12:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556388721; cv=none; d=google.com; s=arc-20160816; b=A06uecyWNIFBJkPrexH4qtrCE1ZVXI8f5dl+F21eT2a882Mta+KbYrAT20ddFI60hk xkf+1ydTE1ffW8D9+VItAgN3hxVUsqufVy0f7E4ayF1HTDhzR1SvFG7qtMowXcdqMhmk oPWFfahjIeLEVOtTevIDH21c9UmYcAyNh9h2y0A3NRCuoBsFK/0c53hdPitB0Qf7mMsq j/roiuyKk5hIYg/uGOaRQGetSfKWlESrRLac4La0xaE8ZLcfSIgSWbaZruuJDRpBzlVQ 3yc+X4mE/IGB/pXQioqyiOKywFh9eiqa/ezeEZsvb9ulM+Sy6sQbonDV6aybSFX1MyEG tBFQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:subject:cc:to:from:date :dkim-signature; bh=pc8/JJZ7CBDPbVXa77oSi9U2xzclmYzE3bpbxcu6gqY=; b=A5EgtDp+WCCBAnnXOs80Ijy31hoBmsehWlH86B0mtCUuRBGh6XxMfeCaSUHJTKf2oL Mgi5ezBQFHHxKQCM3V9bFDsrNZ8eXdagDKrVAFYT9oXvljbH+SArnHbLkmmGc/wJHZH+ FRqLUFlZSqryK/HcEO3AIWVnsEqC3j9oE++B9kPfJ4/vZJYLfycl4NoCo3nun9ug20fh XEEUESD9jbfT9zYiFyFXNL2gbBozrjXIS7yTbDjqxNxdNiTs0/xcCg31lMjWdLLZV8ZU ho6mHXshNNwbUERGK4f49XVCP1H5hpZh69HQpCk1XcVva51WVuS/C8phSoRBjXz82En3 Q61A== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@infradead.org header.s=casper.20170209 header.b=cw1nIeGz; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q1si220218pgp.25.2019.04.27.11.11.44; Sat, 27 Apr 2019 11:12:01 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=fail header.i=@infradead.org header.s=casper.20170209 header.b=cw1nIeGz; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726124AbfD0SJb (ORCPT + 99 others); Sat, 27 Apr 2019 14:09:31 -0400 Received: from casper.infradead.org ([85.118.1.10]:50916 "EHLO casper.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725942AbfD0SJb (ORCPT ); Sat, 27 Apr 2019 14:09:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:Content-Type: MIME-Version:References:In-Reply-To:Message-ID:Subject:Cc:To:From:Date:Sender :Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=pc8/JJZ7CBDPbVXa77oSi9U2xzclmYzE3bpbxcu6gqY=; b=cw1nIeGzqCbI+l9aaqLA9m7y/e +FeGBStQCyKTa8GUNI96KqYpLwOAEUNJspa/WODpqND/dbFlLJEx+oKSZdBAPL065lVHuQTv59+3h jVnhn3UtA7g4LuT7Oapw44/LQT82R2wj5PyQOw6L6pZbX5kG0N9I2bLPAfMLE8YFtYuP7hanJWrnF A18ZGi40MD9hhwRzhTqdB1r2iZD8OQrvZ3kG51yWSg7e17I83WkDSTp0S7l5qumK8GqGYpMgBAtfd n743PtJuVueKOfJi+y7PwjPELY3phwJQMRsuy/lIhAYxDZ5AScJiL0R5keq1W9gBUmqrETI33fPQ1 +/VspoiQ==; Received: from 177.17.250.151.dynamic.adsl.gvt.net.br ([177.17.250.151] helo=coco.lan) by casper.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hKRlI-0003pU-FN; Sat, 27 Apr 2019 18:09:21 +0000 Date: Sat, 27 Apr 2019 15:09:15 -0300 From: Mauro Carvalho Chehab To: Changbin Du Cc: Jonathan Corbet , tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, x86@kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH 17/27] Documentation: x86: convert resctrl_ui.txt to reST Message-ID: <20190427150915.75dacfb2@coco.lan> In-Reply-To: <20190426153150.21228-18-changbin.du@gmail.com> References: <20190426153150.21228-1-changbin.du@gmail.com> <20190426153150.21228-18-changbin.du@gmail.com> X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; x86_64-redhat-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Em Fri, 26 Apr 2019 23:31:40 +0800 Changbin Du escreveu: > This converts the plain text documentation to reStructuredText format and > add it to Sphinx TOC tree. No essential content change. > > Signed-off-by: Changbin Du > --- > Documentation/x86/index.rst | 1 + > .../x86/{resctrl_ui.txt => resctrl_ui.rst} | 913 ++++++++++-------- > 2 files changed, 490 insertions(+), 424 deletions(-) > rename Documentation/x86/{resctrl_ui.txt => resctrl_ui.rst} (68%) > > diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst > index 2fcd10f29b87..4e9fa2b046df 100644 > --- a/Documentation/x86/index.rst > +++ b/Documentation/x86/index.rst > @@ -23,3 +23,4 @@ Linux x86 Support > amd-memory-encryption > pti > microcode > + resctrl_ui > diff --git a/Documentation/x86/resctrl_ui.txt b/Documentation/x86/resctrl_ui.rst > similarity index 68% > rename from Documentation/x86/resctrl_ui.txt > rename to Documentation/x86/resctrl_ui.rst > index c1f95b59e14d..81aaa271d5ea 100644 > --- a/Documentation/x86/resctrl_ui.txt > +++ b/Documentation/x86/resctrl_ui.rst > @@ -1,33 +1,39 @@ > +.. SPDX-License-Identifier: GPL-2.0 > +.. include:: > + > +=========================================== > User Interface for Resource Control feature > +=========================================== > > -Intel refers to this feature as Intel Resource Director Technology(Intel(R) RDT). > -AMD refers to this feature as AMD Platform Quality of Service(AMD QoS). > +:Copyright: |copy| 2016 Intel Corporation > +:Authors: - Fenghua Yu > + - Tony Luck > + - Vikas Shivappa > > -Copyright (C) 2016 Intel Corporation > > -Fenghua Yu > -Tony Luck > -Vikas Shivappa > +Intel refers to this feature as Intel Resource Director Technology(Intel(R) RDT). > +AMD refers to this feature as AMD Platform Quality of Service(AMD QoS). > > This feature is enabled by the CONFIG_X86_CPU_RESCTRL and the x86 /proc/cpuinfo > -flag bits: > -RDT (Resource Director Technology) Allocation - "rdt_a" > -CAT (Cache Allocation Technology) - "cat_l3", "cat_l2" > -CDP (Code and Data Prioritization ) - "cdp_l3", "cdp_l2" > -CQM (Cache QoS Monitoring) - "cqm_llc", "cqm_occup_llc" > -MBM (Memory Bandwidth Monitoring) - "cqm_mbm_total", "cqm_mbm_local" > -MBA (Memory Bandwidth Allocation) - "mba" > +flag bits:: > + > + RDT (Resource Director Technology) Allocation - "rdt_a" > + CAT (Cache Allocation Technology) - "cat_l3", "cat_l2" > + CDP (Code and Data Prioritization ) - "cdp_l3", "cdp_l2" > + CQM (Cache QoS Monitoring) - "cqm_llc", "cqm_occup_llc" > + MBM (Memory Bandwidth Monitoring) - "cqm_mbm_total", "cqm_mbm_local" > + MBA (Memory Bandwidth Allocation) - "mba" I don't see any reason to convert this into a literal block. I would either convert it into a table or into something like: RDT (Resource Director Technology) Allocation - "rdt_a" CAT (Cache Allocation Technology) - "cat_l3", "cat_l2" CDP (Code and Data Prioritization ) - "cdp_l3", "cdp_l2" CQM (Cache QoS Monitoring) - "cqm_llc", "cqm_occup_llc" MBM (Memory Bandwidth Monitoring) - "cqm_mbm_total", "cqm_mbm_local" MBA (Memory Bandwidth Allocation) - "mba" A table seems to be the best approach, though: ============================================= ================================ RDT (Resource Director Technology) Allocation "rdt_a" CAT (Cache Allocation Technology) "cat_l3", "cat_l2" CDP (Code and Data Prioritization) "cdp_l3", "cdp_l2" CQM (Cache QoS Monitoring) "cqm_llc", "cqm_occup_llc" MBM (Memory Bandwidth Monitoring) "cqm_mbm_total", "cqm_mbm_local" MBA (Memory Bandwidth Allocation) "mba" ============================================= ================================ > > -To use the feature mount the file system: > +To use the feature mount the file system:: > > # mount -t resctrl resctrl [-o cdp[,cdpl2][,mba_MBps]] /sys/fs/resctrl > > mount options are: > > -"cdp": Enable code/data prioritization in L3 cache allocations. > -"cdpl2": Enable code/data prioritization in L2 cache allocations. > -"mba_MBps": Enable the MBA Software Controller(mba_sc) to specify MBA > - bandwidth in MBps > +* "cdp": Enable code/data prioritization in L3 cache allocations. > +* "cdpl2": Enable code/data prioritization in L2 cache allocations. > +* "mba_MBps": Enable the MBA Software Controller(mba_sc) to specify MBA > + bandwidth in MBps I would add a \n\t after each :, in order to show the options in bold and the explanation on a separate indented line, just like what you did with the next similar set of options. > > L2 and L3 CDP are controlled seperately. > > @@ -44,7 +50,7 @@ For more details on the behavior of the interface during monitoring > and allocation, see the "Resource alloc and monitor groups" section. > > Info directory > --------------- > +============== > > The 'info' directory contains information about the enabled > resources. Each resource has its own subdirectory. The subdirectory > @@ -56,77 +62,93 @@ allocation: > Cache resource(L3/L2) subdirectory contains the following files > related to allocation: > > -"num_closids": The number of CLOSIDs which are valid for this > - resource. The kernel uses the smallest number of > - CLOSIDs of all enabled resources as limit. > - > -"cbm_mask": The bitmask which is valid for this resource. > - This mask is equivalent to 100%. > - > -"min_cbm_bits": The minimum number of consecutive bits which > - must be set when writing a mask. > - > -"shareable_bits": Bitmask of shareable resource with other executing > - entities (e.g. I/O). User can use this when > - setting up exclusive cache partitions. Note that > - some platforms support devices that have their > - own settings for cache use which can over-ride > - these bits. > -"bit_usage": Annotated capacity bitmasks showing how all > - instances of the resource are used. The legend is: > - "0" - Corresponding region is unused. When the system's > +"num_closids": > + The number of CLOSIDs which are valid for this > + resource. The kernel uses the smallest number of > + CLOSIDs of all enabled resources as limit. > +"cbm_mask": > + The bitmask which is valid for this resource. > + This mask is equivalent to 100%. > +"min_cbm_bits": > + The minimum number of consecutive bits which > + must be set when writing a mask. > + > +"shareable_bits": > + Bitmask of shareable resource with other executing > + entities (e.g. I/O). User can use this when > + setting up exclusive cache partitions. Note that > + some platforms support devices that have their > + own settings for cache use which can over-ride > + these bits. > +"bit_usage": > + Annotated capacity bitmasks showing how all > + instances of the resource are used. The legend is: > + > + "0": > + Corresponding region is unused. When the system's > resources have been allocated and a "0" is found > in "bit_usage" it is a sign that resources are > wasted. > - "H" - Corresponding region is used by hardware only > + > + "H": > + Corresponding region is used by hardware only > but available for software use. If a resource > has bits set in "shareable_bits" but not all > of these bits appear in the resource groups' > schematas then the bits appearing in > "shareable_bits" but no resource group will > be marked as "H". > - "X" - Corresponding region is available for sharing and > + "X": > + Corresponding region is available for sharing and > used by hardware and software. These are the > bits that appear in "shareable_bits" as > well as a resource group's allocation. > - "S" - Corresponding region is used by software > + "S": > + Corresponding region is used by software > and available for sharing. > - "E" - Corresponding region is used exclusively by > + "E": > + Corresponding region is used exclusively by > one resource group. No sharing allowed. > - "P" - Corresponding region is pseudo-locked. No > + "P": > + Corresponding region is pseudo-locked. No > sharing allowed. > > Memory bandwitdh(MB) subdirectory contains the following files > with respect to allocation: > > -"min_bandwidth": The minimum memory bandwidth percentage which > - user can request. > +"min_bandwidth": > + The minimum memory bandwidth percentage which > + user can request. > > -"bandwidth_gran": The granularity in which the memory bandwidth > - percentage is allocated. The allocated > - b/w percentage is rounded off to the next > - control step available on the hardware. The > - available bandwidth control steps are: > - min_bandwidth + N * bandwidth_gran. > +"bandwidth_gran": > + The granularity in which the memory bandwidth > + percentage is allocated. The allocated > + b/w percentage is rounded off to the next > + control step available on the hardware. The > + available bandwidth control steps are: > + min_bandwidth + N * bandwidth_gran. > > -"delay_linear": Indicates if the delay scale is linear or > - non-linear. This field is purely informational > - only. > +"delay_linear": > + Indicates if the delay scale is linear or > + non-linear. This field is purely informational > + only. > > If RDT monitoring is available there will be an "L3_MON" directory > with the following files: > > -"num_rmids": The number of RMIDs available. This is the > - upper bound for how many "CTRL_MON" + "MON" > - groups can be created. > +"num_rmids": > + The number of RMIDs available. This is the > + upper bound for how many "CTRL_MON" + "MON" > + groups can be created. > > -"mon_features": Lists the monitoring events if > - monitoring is enabled for the resource. > +"mon_features": > + Lists the monitoring events if > + monitoring is enabled for the resource. > > "max_threshold_occupancy": > - Read/write file provides the largest value (in > - bytes) at which a previously used LLC_occupancy > - counter can be considered for re-use. > + Read/write file provides the largest value (in > + bytes) at which a previously used LLC_occupancy > + counter can be considered for re-use. > > Finally, in the top level of the "info" directory there is a file > named "last_cmd_status". This is reset with every "command" issued > @@ -134,6 +156,7 @@ via the file system (making new directories or writing to any of the > control files). If the command was successful, it will read as "ok". > If the command failed, it will provide more information that can be > conveyed in the error returns from file operations. E.g. > +:: > > # echo L3:0=f7 > schemata > bash: echo: write error: Invalid argument > @@ -141,7 +164,7 @@ conveyed in the error returns from file operations. E.g. > mask f7 has non-consecutive 1-bits > > Resource alloc and monitor groups > ---------------------------------- > +================================= > > Resource groups are represented as directories in the resctrl file > system. The default group is the root directory which, immediately > @@ -226,6 +249,7 @@ When monitoring is enabled all MON groups will also contain: > > Resource allocation rules > ------------------------- > + > When a task is running the following rules define which resources are > available to it: > > @@ -252,7 +276,7 @@ Resource monitoring rules > > > Notes on cache occupancy monitoring and control > ------------------------------------------------ > +=============================================== > When moving a task from one group to another you should remember that > this only affects *new* cache allocations by the task. E.g. you may have > a task in a monitor group showing 3 MB of cache occupancy. If you move > @@ -321,7 +345,7 @@ of the capacity of the cache. You could partition the cache into four > equal parts with masks: 0x1f, 0x3e0, 0x7c00, 0xf8000. > > Memory bandwidth Allocation and monitoring > ------------------------------------------- > +========================================== > > For Memory bandwidth resource, by default the user controls the resource > by indicating the percentage of total memory bandwidth. > @@ -369,7 +393,7 @@ In order to mitigate this and make the interface more user friendly, > resctrl added support for specifying the bandwidth in MBps as well. The > kernel underneath would use a software feedback mechanism or a "Software > Controller(mba_sc)" which reads the actual bandwidth using MBM counters > -and adjust the memowy bandwidth percentages to ensure > +and adjust the memowy bandwidth percentages to ensure:: > > "actual bandwidth < user specified bandwidth". > > @@ -380,14 +404,14 @@ sections. > > L3 schemata file details (code and data prioritization disabled) > ---------------------------------------------------------------- > -With CDP disabled the L3 schemata format is: > +With CDP disabled the L3 schemata format is:: > > L3:=;=;... > > L3 schemata file details (CDP enabled via mount option to resctrl) > ------------------------------------------------------------------ > When CDP is enabled L3 control is split into two separate resources > -so you can specify independent masks for code and data like this: > +so you can specify independent masks for code and data like this:: > > L3data:=;=;... > L3code:=;=;... > @@ -395,7 +419,7 @@ so you can specify independent masks for code and data like this: > L2 schemata file details > ------------------------ > L2 cache does not support code and data prioritization, so the > -schemata format is always: > +schemata format is always:: > > L2:=;=;... > > @@ -403,6 +427,7 @@ Memory bandwidth Allocation (default mode) > ------------------------------------------ > > Memory b/w domain is L3 cache. > +:: > > MB:=bandwidth0;=bandwidth1;... > > @@ -410,6 +435,7 @@ Memory bandwidth Allocation specified in MBps > --------------------------------------------- > > Memory bandwidth domain is L3 cache. > +:: > > MB:=bw_MBps0;=bw_MBps1;... > > @@ -418,17 +444,18 @@ Reading/writing the schemata file > Reading the schemata file will show the state of all resources > on all domains. When writing you only need to specify those values > which you wish to change. E.g. > +:: > > -# cat schemata > -L3DATA:0=fffff;1=fffff;2=fffff;3=fffff > -L3CODE:0=fffff;1=fffff;2=fffff;3=fffff > -# echo "L3DATA:2=3c0;" > schemata > -# cat schemata > -L3DATA:0=fffff;1=fffff;2=3c0;3=fffff > -L3CODE:0=fffff;1=fffff;2=fffff;3=fffff > + # cat schemata > + L3DATA:0=fffff;1=fffff;2=fffff;3=fffff > + L3CODE:0=fffff;1=fffff;2=fffff;3=fffff > + # echo "L3DATA:2=3c0;" > schemata > + # cat schemata > + L3DATA:0=fffff;1=fffff;2=3c0;3=fffff > + L3CODE:0=fffff;1=fffff;2=fffff;3=fffff > > Cache Pseudo-Locking > --------------------- > +==================== > CAT enables a user to specify the amount of cache space that an > application can fill. Cache pseudo-locking builds on the fact that a > CPU can still read and write data pre-allocated outside its current > @@ -442,6 +469,7 @@ a region of memory with reduced average read latency. > The creation of a cache pseudo-locked region is triggered by a request > from the user to do so that is accompanied by a schemata of the region > to be pseudo-locked. The cache pseudo-locked region is created as follows: > + > - Create a CAT allocation CLOSNEW with a CBM matching the schemata > from the user of the cache region that will contain the pseudo-locked > memory. This region must not overlap with any current CAT allocation/CLOS > @@ -480,6 +508,7 @@ initial mmap() handling, there is no enforcement afterwards and the > application self needs to ensure it remains affine to the correct cores. > > Pseudo-locking is accomplished in two stages: > + > 1) During the first stage the system administrator allocates a portion > of cache that should be dedicated to pseudo-locking. At this time an > equivalent portion of memory is allocated, loaded into allocated > @@ -506,7 +535,7 @@ by user space in order to obtain access to the pseudo-locked memory region. > An example of cache pseudo-locked region creation and usage can be found below. > > Cache Pseudo-Locking Debugging Interface > ---------------------------------------- > +---------------------------------------- > The pseudo-locking debugging interface is enabled by default (if > CONFIG_DEBUG_FS is enabled) and can be found in /sys/kernel/debug/resctrl. > > @@ -514,6 +543,7 @@ There is no explicit way for the kernel to test if a provided memory > location is present in the cache. The pseudo-locking debugging interface uses > the tracing infrastructure to provide two ways to measure cache residency of > the pseudo-locked region: > + > 1) Memory access latency using the pseudo_lock_mem_latency tracepoint. Data > from these measurements are best visualized using a hist trigger (see > example below). In this test the pseudo-locked region is traversed at > @@ -529,87 +559,97 @@ it in debugfs as /sys/kernel/debug/resctrl/. A single > write-only file, pseudo_lock_measure, is present in this directory. The > measurement of the pseudo-locked region depends on the number written to this > debugfs file: > -1 - writing "1" to the pseudo_lock_measure file will trigger the latency > + > +1: > + writing "1" to the pseudo_lock_measure file will trigger the latency > measurement captured in the pseudo_lock_mem_latency tracepoint. See > example below. > -2 - writing "2" to the pseudo_lock_measure file will trigger the L2 cache > +2: > + writing "2" to the pseudo_lock_measure file will trigger the L2 cache > residency (cache hits and misses) measurement captured in the > pseudo_lock_l2 tracepoint. See example below. > -3 - writing "3" to the pseudo_lock_measure file will trigger the L3 cache > +3: > + writing "3" to the pseudo_lock_measure file will trigger the L3 cache > residency (cache hits and misses) measurement captured in the > pseudo_lock_l3 tracepoint. > > All measurements are recorded with the tracing infrastructure. This requires > the relevant tracepoints to be enabled before the measurement is triggered. > > -Example of latency debugging interface: > +Example of latency debugging interface > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > In this example a pseudo-locked region named "newlock" was created. Here is > how we can measure the latency in cycles of reading from this region and > visualize this data with a histogram that is available if CONFIG_HIST_TRIGGERS > -is set: > -# :> /sys/kernel/debug/tracing/trace > -# echo 'hist:keys=latency' > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/trigger > -# echo 1 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/enable > -# echo 1 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure > -# echo 0 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/enable > -# cat /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/hist > - > -# event histogram > -# > -# trigger info: hist:keys=latency:vals=hitcount:sort=hitcount:size=2048 [active] > -# > - > -{ latency: 456 } hitcount: 1 > -{ latency: 50 } hitcount: 83 > -{ latency: 36 } hitcount: 96 > -{ latency: 44 } hitcount: 174 > -{ latency: 48 } hitcount: 195 > -{ latency: 46 } hitcount: 262 > -{ latency: 42 } hitcount: 693 > -{ latency: 40 } hitcount: 3204 > -{ latency: 38 } hitcount: 3484 > - > -Totals: > - Hits: 8192 > - Entries: 9 > - Dropped: 0 > - > -Example of cache hits/misses debugging: > +is set:: > + > + # :> /sys/kernel/debug/tracing/trace > + # echo 'hist:keys=latency' > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/trigger > + # echo 1 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/enable > + # echo 1 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure > + # echo 0 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/enable > + # cat /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/hist > + > + # event histogram > + # > + # trigger info: hist:keys=latency:vals=hitcount:sort=hitcount:size=2048 [active] > + # > + > + { latency: 456 } hitcount: 1 > + { latency: 50 } hitcount: 83 > + { latency: 36 } hitcount: 96 > + { latency: 44 } hitcount: 174 > + { latency: 48 } hitcount: 195 > + { latency: 46 } hitcount: 262 > + { latency: 42 } hitcount: 693 > + { latency: 40 } hitcount: 3204 > + { latency: 38 } hitcount: 3484 > + > + Totals: > + Hits: 8192 > + Entries: 9 > + Dropped: 0 > + > +Example of cache hits/misses debugging > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > In this example a pseudo-locked region named "newlock" was created on the L2 > cache of a platform. Here is how we can obtain details of the cache hits > and misses using the platform's precision counters. > +:: > > -# :> /sys/kernel/debug/tracing/trace > -# echo 1 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_l2/enable > -# echo 2 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure > -# echo 0 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_l2/enable > -# cat /sys/kernel/debug/tracing/trace > + # :> /sys/kernel/debug/tracing/trace > + # echo 1 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_l2/enable > + # echo 2 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure > + # echo 0 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_l2/enable > + # cat /sys/kernel/debug/tracing/trace > > -# tracer: nop > -# > -# _-----=> irqs-off > -# / _----=> need-resched > -# | / _---=> hardirq/softirq > -# || / _--=> preempt-depth > -# ||| / delay > -# TASK-PID CPU# |||| TIMESTAMP FUNCTION > -# | | | |||| | | > - pseudo_lock_mea-1672 [002] .... 3132.860500: pseudo_lock_l2: hits=4097 miss=0 > + # tracer: nop > + # > + # _-----=> irqs-off > + # / _----=> need-resched > + # | / _---=> hardirq/softirq > + # || / _--=> preempt-depth > + # ||| / delay > + # TASK-PID CPU# |||| TIMESTAMP FUNCTION > + # | | | |||| | | > + pseudo_lock_mea-1672 [002] .... 3132.860500: pseudo_lock_l2: hits=4097 miss=0 > > > -Examples for RDT allocation usage: > +Examples for RDT allocation usage > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +1) Example 1 > > -Example 1 > ---------- > On a two socket machine (one L3 cache per socket) with just four bits > for cache bit masks, minimum b/w of 10% with a memory bandwidth > -granularity of 10% > +granularity of 10%. > +:: > > -# mount -t resctrl resctrl /sys/fs/resctrl > -# cd /sys/fs/resctrl > -# mkdir p0 p1 > -# echo "L3:0=3;1=c\nMB:0=50;1=50" > /sys/fs/resctrl/p0/schemata > -# echo "L3:0=3;1=3\nMB:0=50;1=50" > /sys/fs/resctrl/p1/schemata > + # mount -t resctrl resctrl /sys/fs/resctrl > + # cd /sys/fs/resctrl > + # mkdir p0 p1 > + # echo "L3:0=3;1=c\nMB:0=50;1=50" > /sys/fs/resctrl/p0/schemata > + # echo "L3:0=3;1=3\nMB:0=50;1=50" > /sys/fs/resctrl/p1/schemata > > The default resource group is unmodified, so we have access to all parts > of all caches (its schemata file reads "L3:0=f;1=f"). > @@ -628,100 +668,106 @@ the b/w accordingly. > > If the MBA is specified in MB(megabytes) then user can enter the max b/w in MB > rather than the percentage values. > +:: > > -# echo "L3:0=3;1=c\nMB:0=1024;1=500" > /sys/fs/resctrl/p0/schemata > -# echo "L3:0=3;1=3\nMB:0=1024;1=500" > /sys/fs/resctrl/p1/schemata > + # echo "L3:0=3;1=c\nMB:0=1024;1=500" > /sys/fs/resctrl/p0/schemata > + # echo "L3:0=3;1=3\nMB:0=1024;1=500" > /sys/fs/resctrl/p1/schemata > > In the above example the tasks in "p1" and "p0" on socket 0 would use a max b/w > of 1024MB where as on socket 1 they would use 500MB. > > -Example 2 > ---------- > +2) Example 2 > + > Again two sockets, but this time with a more realistic 20-bit mask. > > Two real time tasks pid=1234 running on processor 0 and pid=5678 running on > processor 1 on socket 0 on a 2-socket and dual core machine. To avoid noisy > neighbors, each of the two real-time tasks exclusively occupies one quarter > of L3 cache on socket 0. > +:: > > -# mount -t resctrl resctrl /sys/fs/resctrl > -# cd /sys/fs/resctrl > + # mount -t resctrl resctrl /sys/fs/resctrl > + # cd /sys/fs/resctrl > > First we reset the schemata for the default group so that the "upper" > 50% of the L3 cache on socket 0 and 50% of memory b/w cannot be used by > -ordinary tasks: > +ordinary tasks:: > > -# echo "L3:0=3ff;1=fffff\nMB:0=50;1=100" > schemata > + # echo "L3:0=3ff;1=fffff\nMB:0=50;1=100" > schemata > > Next we make a resource group for our first real time task and give > it access to the "top" 25% of the cache on socket 0. > +:: > > -# mkdir p0 > -# echo "L3:0=f8000;1=fffff" > p0/schemata > + # mkdir p0 > + # echo "L3:0=f8000;1=fffff" > p0/schemata > > Finally we move our first real time task into this resource group. We > also use taskset(1) to ensure the task always runs on a dedicated CPU > on socket 0. Most uses of resource groups will also constrain which > processors tasks run on. > +:: > > -# echo 1234 > p0/tasks > -# taskset -cp 1 1234 > + # echo 1234 > p0/tasks > + # taskset -cp 1 1234 > > -Ditto for the second real time task (with the remaining 25% of cache): > +Ditto for the second real time task (with the remaining 25% of cache):: > > -# mkdir p1 > -# echo "L3:0=7c00;1=fffff" > p1/schemata > -# echo 5678 > p1/tasks > -# taskset -cp 2 5678 > + # mkdir p1 > + # echo "L3:0=7c00;1=fffff" > p1/schemata > + # echo 5678 > p1/tasks > + # taskset -cp 2 5678 > > For the same 2 socket system with memory b/w resource and CAT L3 the > schemata would look like(Assume min_bandwidth 10 and bandwidth_gran is > 10): > > -For our first real time task this would request 20% memory b/w on socket > -0. > +For our first real time task this would request 20% memory b/w on socket 0. > +:: > > -# echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata > + # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata > > For our second real time task this would request an other 20% memory b/w > on socket 0. > +:: > > -# echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata > + # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata > > -Example 3 > ---------- > +3) Example 3 > > A single socket system which has real-time tasks running on core 4-7 and > non real-time workload assigned to core 0-3. The real-time tasks share text > and data, so a per task association is not required and due to interaction > with the kernel it's desired that the kernel on these cores shares L3 with > the tasks. > +:: > > -# mount -t resctrl resctrl /sys/fs/resctrl > -# cd /sys/fs/resctrl > + # mount -t resctrl resctrl /sys/fs/resctrl > + # cd /sys/fs/resctrl > > First we reset the schemata for the default group so that the "upper" > 50% of the L3 cache on socket 0, and 50% of memory bandwidth on socket 0 > -cannot be used by ordinary tasks: > +cannot be used by ordinary tasks:: > > -# echo "L3:0=3ff\nMB:0=50" > schemata > + # echo "L3:0=3ff\nMB:0=50" > schemata > > Next we make a resource group for our real time cores and give it access > to the "top" 50% of the cache on socket 0 and 50% of memory bandwidth on > socket 0. > +:: > > -# mkdir p0 > -# echo "L3:0=ffc00\nMB:0=50" > p0/schemata > + # mkdir p0 > + # echo "L3:0=ffc00\nMB:0=50" > p0/schemata > > Finally we move core 4-7 over to the new group and make sure that the > kernel and the tasks running there get 50% of the cache. They should > also get 50% of memory bandwidth assuming that the cores 4-7 are SMT > siblings and only the real time threads are scheduled on the cores 4-7. > +:: > > -# echo F0 > p0/cpus > + # echo F0 > p0/cpus > > -Example 4 > ---------- > +4) Example 4 > > The resource groups in previous examples were all in the default "shareable" > mode allowing sharing of their cache allocations. If one resource group > @@ -732,157 +778,168 @@ In this example a new exclusive resource group will be created on a L2 CAT > system with two L2 cache instances that can be configured with an 8-bit > capacity bitmask. The new exclusive resource group will be configured to use > 25% of each cache instance. > +:: > > -# mount -t resctrl resctrl /sys/fs/resctrl/ > -# cd /sys/fs/resctrl > + # mount -t resctrl resctrl /sys/fs/resctrl/ > + # cd /sys/fs/resctrl > > First, we observe that the default group is configured to allocate to all L2 > -cache: > +cache:: > > -# cat schemata > -L2:0=ff;1=ff > + # cat schemata > + L2:0=ff;1=ff > > We could attempt to create the new resource group at this point, but it will > -fail because of the overlap with the schemata of the default group: > -# mkdir p0 > -# echo 'L2:0=0x3;1=0x3' > p0/schemata > -# cat p0/mode > -shareable > -# echo exclusive > p0/mode > --sh: echo: write error: Invalid argument > -# cat info/last_cmd_status > -schemata overlaps > +fail because of the overlap with the schemata of the default group:: > + > + # mkdir p0 > + # echo 'L2:0=0x3;1=0x3' > p0/schemata > + # cat p0/mode > + shareable > + # echo exclusive > p0/mode > + -sh: echo: write error: Invalid argument > + # cat info/last_cmd_status > + schemata overlaps > > To ensure that there is no overlap with another resource group the default > resource group's schemata has to change, making it possible for the new > resource group to become exclusive. > -# echo 'L2:0=0xfc;1=0xfc' > schemata > -# echo exclusive > p0/mode > -# grep . p0/* > -p0/cpus:0 > -p0/mode:exclusive > -p0/schemata:L2:0=03;1=03 > -p0/size:L2:0=262144;1=262144 > +:: > + > + # echo 'L2:0=0xfc;1=0xfc' > schemata > + # echo exclusive > p0/mode > + # grep . p0/* > + p0/cpus:0 > + p0/mode:exclusive > + p0/schemata:L2:0=03;1=03 > + p0/size:L2:0=262144;1=262144 > > A new resource group will on creation not overlap with an exclusive resource > -group: > -# mkdir p1 > -# grep . p1/* > -p1/cpus:0 > -p1/mode:shareable > -p1/schemata:L2:0=fc;1=fc > -p1/size:L2:0=786432;1=786432 > - > -The bit_usage will reflect how the cache is used: > -# cat info/L2/bit_usage > -0=SSSSSSEE;1=SSSSSSEE > - > -A resource group cannot be forced to overlap with an exclusive resource group: > -# echo 'L2:0=0x1;1=0x1' > p1/schemata > --sh: echo: write error: Invalid argument > -# cat info/last_cmd_status > -overlaps with exclusive group > +group:: > + > + # mkdir p1 > + # grep . p1/* > + p1/cpus:0 > + p1/mode:shareable > + p1/schemata:L2:0=fc;1=fc > + p1/size:L2:0=786432;1=786432 > + > +The bit_usage will reflect how the cache is used:: > + > + # cat info/L2/bit_usage > + 0=SSSSSSEE;1=SSSSSSEE > + > +A resource group cannot be forced to overlap with an exclusive resource group:: > + > + # echo 'L2:0=0x1;1=0x1' > p1/schemata > + -sh: echo: write error: Invalid argument > + # cat info/last_cmd_status > + overlaps with exclusive group > > Example of Cache Pseudo-Locking > -------------------------------- > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Lock portion of L2 cache from cache id 1 using CBM 0x3. Pseudo-locked > region is exposed at /dev/pseudo_lock/newlock that can be provided to > application for argument to mmap(). > +:: > > -# mount -t resctrl resctrl /sys/fs/resctrl/ > -# cd /sys/fs/resctrl > + # mount -t resctrl resctrl /sys/fs/resctrl/ > + # cd /sys/fs/resctrl > > Ensure that there are bits available that can be pseudo-locked, since only > unused bits can be pseudo-locked the bits to be pseudo-locked needs to be > -removed from the default resource group's schemata: > -# cat info/L2/bit_usage > -0=SSSSSSSS;1=SSSSSSSS > -# echo 'L2:1=0xfc' > schemata > -# cat info/L2/bit_usage > -0=SSSSSSSS;1=SSSSSS00 > +removed from the default resource group's schemata:: > + > + # cat info/L2/bit_usage > + 0=SSSSSSSS;1=SSSSSSSS > + # echo 'L2:1=0xfc' > schemata > + # cat info/L2/bit_usage > + 0=SSSSSSSS;1=SSSSSS00 > > Create a new resource group that will be associated with the pseudo-locked > region, indicate that it will be used for a pseudo-locked region, and > -configure the requested pseudo-locked region capacity bitmask: > +configure the requested pseudo-locked region capacity bitmask:: > > -# mkdir newlock > -# echo pseudo-locksetup > newlock/mode > -# echo 'L2:1=0x3' > newlock/schemata > + # mkdir newlock > + # echo pseudo-locksetup > newlock/mode > + # echo 'L2:1=0x3' > newlock/schemata > > On success the resource group's mode will change to pseudo-locked, the > bit_usage will reflect the pseudo-locked region, and the character device > -exposing the pseudo-locked region will exist: > - > -# cat newlock/mode > -pseudo-locked > -# cat info/L2/bit_usage > -0=SSSSSSSS;1=SSSSSSPP > -# ls -l /dev/pseudo_lock/newlock > -crw------- 1 root root 243, 0 Apr 3 05:01 /dev/pseudo_lock/newlock > - > -/* > - * Example code to access one page of pseudo-locked cache region > - * from user space. > - */ > -#define _GNU_SOURCE > -#include > -#include > -#include > -#include > -#include > -#include > - > -/* > - * It is required that the application runs with affinity to only > - * cores associated with the pseudo-locked region. Here the cpu > - * is hardcoded for convenience of example. > - */ > -static int cpuid = 2; > - > -int main(int argc, char *argv[]) > -{ > - cpu_set_t cpuset; > - long page_size; > - void *mapping; > - int dev_fd; > - int ret; > - > - page_size = sysconf(_SC_PAGESIZE); > - > - CPU_ZERO(&cpuset); > - CPU_SET(cpuid, &cpuset); > - ret = sched_setaffinity(0, sizeof(cpuset), &cpuset); > - if (ret < 0) { > - perror("sched_setaffinity"); > - exit(EXIT_FAILURE); > - } > - > - dev_fd = open("/dev/pseudo_lock/newlock", O_RDWR); > - if (dev_fd < 0) { > - perror("open"); > - exit(EXIT_FAILURE); > - } > - > - mapping = mmap(0, page_size, PROT_READ | PROT_WRITE, MAP_SHARED, > - dev_fd, 0); > - if (mapping == MAP_FAILED) { > - perror("mmap"); > - close(dev_fd); > - exit(EXIT_FAILURE); > - } > - > - /* Application interacts with pseudo-locked memory @mapping */ > - > - ret = munmap(mapping, page_size); > - if (ret < 0) { > - perror("munmap"); > - close(dev_fd); > - exit(EXIT_FAILURE); > - } > - > - close(dev_fd); > - exit(EXIT_SUCCESS); > -} > +exposing the pseudo-locked region will exist:: > + > + # cat newlock/mode > + pseudo-locked > + # cat info/L2/bit_usage > + 0=SSSSSSSS;1=SSSSSSPP > + # ls -l /dev/pseudo_lock/newlock > + crw------- 1 root root 243, 0 Apr 3 05:01 /dev/pseudo_lock/newlock > + > +:: > + > + /* > + * Example code to access one page of pseudo-locked cache region > + * from user space. > + */ > + #define _GNU_SOURCE > + #include > + #include > + #include > + #include > + #include > + #include > + > + /* > + * It is required that the application runs with affinity to only > + * cores associated with the pseudo-locked region. Here the cpu > + * is hardcoded for convenience of example. > + */ > + static int cpuid = 2; > + > + int main(int argc, char *argv[]) > + { > + cpu_set_t cpuset; > + long page_size; > + void *mapping; > + int dev_fd; > + int ret; > + > + page_size = sysconf(_SC_PAGESIZE); > + > + CPU_ZERO(&cpuset); > + CPU_SET(cpuid, &cpuset); > + ret = sched_setaffinity(0, sizeof(cpuset), &cpuset); > + if (ret < 0) { > + perror("sched_setaffinity"); > + exit(EXIT_FAILURE); > + } > + > + dev_fd = open("/dev/pseudo_lock/newlock", O_RDWR); > + if (dev_fd < 0) { > + perror("open"); > + exit(EXIT_FAILURE); > + } > + > + mapping = mmap(0, page_size, PROT_READ | PROT_WRITE, MAP_SHARED, > + dev_fd, 0); > + if (mapping == MAP_FAILED) { > + perror("mmap"); > + close(dev_fd); > + exit(EXIT_FAILURE); > + } > + > + /* Application interacts with pseudo-locked memory @mapping */ > + > + ret = munmap(mapping, page_size); > + if (ret < 0) { > + perror("munmap"); > + close(dev_fd); > + exit(EXIT_FAILURE); > + } > + > + close(dev_fd); > + exit(EXIT_SUCCESS); > + } > > Locking between applications > ---------------------------- > @@ -921,86 +978,86 @@ Read lock: > B) If success read the directory structure. > C) funlock > > -Example with bash: > - > -# Atomically read directory structure > -$ flock -s /sys/fs/resctrl/ find /sys/fs/resctrl > - > -# Read directory contents and create new subdirectory > - > -$ cat create-dir.sh > -find /sys/fs/resctrl/ > output.txt > -mask = function-of(output.txt) > -mkdir /sys/fs/resctrl/newres/ > -echo mask > /sys/fs/resctrl/newres/schemata > - > -$ flock /sys/fs/resctrl/ ./create-dir.sh > - > -Example with C: > - > -/* > - * Example code do take advisory locks > - * before accessing resctrl filesystem > - */ > -#include > -#include > - > -void resctrl_take_shared_lock(int fd) > -{ > - int ret; > - > - /* take shared lock on resctrl filesystem */ > - ret = flock(fd, LOCK_SH); > - if (ret) { > - perror("flock"); > - exit(-1); > - } > -} > - > -void resctrl_take_exclusive_lock(int fd) > -{ > - int ret; > - > - /* release lock on resctrl filesystem */ > - ret = flock(fd, LOCK_EX); > - if (ret) { > - perror("flock"); > - exit(-1); > - } > -} > - > -void resctrl_release_lock(int fd) > -{ > - int ret; > - > - /* take shared lock on resctrl filesystem */ > - ret = flock(fd, LOCK_UN); > - if (ret) { > - perror("flock"); > - exit(-1); > - } > -} > - > -void main(void) > -{ > - int fd, ret; > - > - fd = open("/sys/fs/resctrl", O_DIRECTORY); > - if (fd == -1) { > - perror("open"); > - exit(-1); > - } > - resctrl_take_shared_lock(fd); > - /* code to read directory contents */ > - resctrl_release_lock(fd); > - > - resctrl_take_exclusive_lock(fd); > - /* code to read and write directory contents */ > - resctrl_release_lock(fd); > -} > - > -Examples for RDT Monitoring along with allocation usage: > - > +Example with bash:: > + > + # Atomically read directory structure > + $ flock -s /sys/fs/resctrl/ find /sys/fs/resctrl > + > + # Read directory contents and create new subdirectory > + > + $ cat create-dir.sh > + find /sys/fs/resctrl/ > output.txt > + mask = function-of(output.txt) > + mkdir /sys/fs/resctrl/newres/ > + echo mask > /sys/fs/resctrl/newres/schemata > + > + $ flock /sys/fs/resctrl/ ./create-dir.sh > + > +Example with C:: > + > + /* > + * Example code do take advisory locks > + * before accessing resctrl filesystem > + */ > + #include > + #include > + > + void resctrl_take_shared_lock(int fd) > + { > + int ret; > + > + /* take shared lock on resctrl filesystem */ > + ret = flock(fd, LOCK_SH); > + if (ret) { > + perror("flock"); > + exit(-1); > + } > + } > + > + void resctrl_take_exclusive_lock(int fd) > + { > + int ret; > + > + /* release lock on resctrl filesystem */ > + ret = flock(fd, LOCK_EX); > + if (ret) { > + perror("flock"); > + exit(-1); > + } > + } > + > + void resctrl_release_lock(int fd) > + { > + int ret; > + > + /* take shared lock on resctrl filesystem */ > + ret = flock(fd, LOCK_UN); > + if (ret) { > + perror("flock"); > + exit(-1); > + } > + } > + > + void main(void) > + { > + int fd, ret; > + > + fd = open("/sys/fs/resctrl", O_DIRECTORY); > + if (fd == -1) { > + perror("open"); > + exit(-1); > + } > + resctrl_take_shared_lock(fd); > + /* code to read directory contents */ > + resctrl_release_lock(fd); > + > + resctrl_take_exclusive_lock(fd); > + /* code to read and write directory contents */ > + resctrl_release_lock(fd); > + } > + > +Examples for RDT Monitoring along with allocation usage > +======================================================= > Reading monitored data > ---------------------- > Reading an event file (for ex: mon_data/mon_L3_00/llc_occupancy) would > @@ -1009,17 +1066,17 @@ group or CTRL_MON group. > > > Example 1 (Monitor CTRL_MON group and subset of tasks in CTRL_MON group) > ---------- > +------------------------------------------------------------------------ > On a two socket machine (one L3 cache per socket) with just four bits > -for cache bit masks > +for cache bit masks:: > > -# mount -t resctrl resctrl /sys/fs/resctrl > -# cd /sys/fs/resctrl > -# mkdir p0 p1 > -# echo "L3:0=3;1=c" > /sys/fs/resctrl/p0/schemata > -# echo "L3:0=3;1=3" > /sys/fs/resctrl/p1/schemata > -# echo 5678 > p1/tasks > -# echo 5679 > p1/tasks > + # mount -t resctrl resctrl /sys/fs/resctrl > + # cd /sys/fs/resctrl > + # mkdir p0 p1 > + # echo "L3:0=3;1=c" > /sys/fs/resctrl/p0/schemata > + # echo "L3:0=3;1=3" > /sys/fs/resctrl/p1/schemata > + # echo 5678 > p1/tasks > + # echo 5679 > p1/tasks > > The default resource group is unmodified, so we have access to all parts > of all caches (its schemata file reads "L3:0=f;1=f"). > @@ -1029,47 +1086,51 @@ Tasks that are under the control of group "p0" may only allocate from the > Tasks in group "p1" use the "lower" 50% of cache on both sockets. > > Create monitor groups and assign a subset of tasks to each monitor group. > +:: > > -# cd /sys/fs/resctrl/p1/mon_groups > -# mkdir m11 m12 > -# echo 5678 > m11/tasks > -# echo 5679 > m12/tasks > + # cd /sys/fs/resctrl/p1/mon_groups > + # mkdir m11 m12 > + # echo 5678 > m11/tasks > + # echo 5679 > m12/tasks > > fetch data (data shown in bytes) > +:: > > -# cat m11/mon_data/mon_L3_00/llc_occupancy > -16234000 > -# cat m11/mon_data/mon_L3_01/llc_occupancy > -14789000 > -# cat m12/mon_data/mon_L3_00/llc_occupancy > -16789000 > + # cat m11/mon_data/mon_L3_00/llc_occupancy > + 16234000 > + # cat m11/mon_data/mon_L3_01/llc_occupancy > + 14789000 > + # cat m12/mon_data/mon_L3_00/llc_occupancy > + 16789000 > > The parent ctrl_mon group shows the aggregated data. > +:: > > -# cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy > -31234000 > + # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy > + 31234000 > > Example 2 (Monitor a task from its creation) > ---------- > -On a two socket machine (one L3 cache per socket) > +-------------------------------------------- > +On a two socket machine (one L3 cache per socket):: > > -# mount -t resctrl resctrl /sys/fs/resctrl > -# cd /sys/fs/resctrl > -# mkdir p0 p1 > + # mount -t resctrl resctrl /sys/fs/resctrl > + # cd /sys/fs/resctrl > + # mkdir p0 p1 > > An RMID is allocated to the group once its created and hence the > below is monitored from its creation. > +:: > > -# echo $$ > /sys/fs/resctrl/p1/tasks > -# > + # echo $$ > /sys/fs/resctrl/p1/tasks > + # > > -Fetch the data > +Fetch the data:: > > -# cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy > -31789000 > + # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy > + 31789000 > > Example 3 (Monitor without CAT support or before creating CAT groups) > ---------- > +--------------------------------------------------------------------- > > Assume a system like HSW has only CQM and no CAT support. In this case > the resctrl will still mount but cannot create CTRL_MON directories. > @@ -1078,27 +1139,29 @@ able to monitor all tasks including kernel threads. > > This can also be used to profile jobs cache size footprint before being > able to allocate them to different allocation groups. > +:: > > -# mount -t resctrl resctrl /sys/fs/resctrl > -# cd /sys/fs/resctrl > -# mkdir mon_groups/m01 > -# mkdir mon_groups/m02 > + # mount -t resctrl resctrl /sys/fs/resctrl > + # cd /sys/fs/resctrl > + # mkdir mon_groups/m01 > + # mkdir mon_groups/m02 > > -# echo 3478 > /sys/fs/resctrl/mon_groups/m01/tasks > -# echo 2467 > /sys/fs/resctrl/mon_groups/m02/tasks > + # echo 3478 > /sys/fs/resctrl/mon_groups/m01/tasks > + # echo 2467 > /sys/fs/resctrl/mon_groups/m02/tasks > > Monitor the groups separately and also get per domain data. From the > below its apparent that the tasks are mostly doing work on > domain(socket) 0. > +:: > > -# cat /sys/fs/resctrl/mon_groups/m01/mon_L3_00/llc_occupancy > -31234000 > -# cat /sys/fs/resctrl/mon_groups/m01/mon_L3_01/llc_occupancy > -34555 > -# cat /sys/fs/resctrl/mon_groups/m02/mon_L3_00/llc_occupancy > -31234000 > -# cat /sys/fs/resctrl/mon_groups/m02/mon_L3_01/llc_occupancy > -32789 > + # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_00/llc_occupancy > + 31234000 > + # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_01/llc_occupancy > + 34555 > + # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_00/llc_occupancy > + 31234000 > + # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_01/llc_occupancy > + 32789 > > > Example 4 (Monitor real time tasks) > @@ -1107,15 +1170,17 @@ Example 4 (Monitor real time tasks) > A single socket system which has real time tasks running on cores 4-7 > and non real time tasks on other cpus. We want to monitor the cache > occupancy of the real time threads on these cores. > +:: > > -# mount -t resctrl resctrl /sys/fs/resctrl > -# cd /sys/fs/resctrl > -# mkdir p1 > + # mount -t resctrl resctrl /sys/fs/resctrl > + # cd /sys/fs/resctrl > + # mkdir p1 > > -Move the cpus 4-7 over to p1 > -# echo f0 > p1/cpus > +Move the cpus 4-7 over to p1:: > + There are extra whitespaces at the tail of the above line. After fixing the above: Reviewed-by: Mauro Carvalho Chehab > + # echo f0 > p1/cpus > > -View the llc occupancy snapshot > +View the llc occupancy snapshot:: > > -# cat /sys/fs/resctrl/p1/mon_data/mon_L3_00/llc_occupancy > -11234000 > + # cat /sys/fs/resctrl/p1/mon_data/mon_L3_00/llc_occupancy > + 11234000 Thanks, Mauro