These series adds support few minor features.
1. Support assigning multiple tasks to control/mon groups in one command.
2. Add debug mount option for resctrl interface.
3. Add RMID and CLOSID in resctrl interface when mounted with debug option.
4. Moves the default control group creation during the mount instead of during init.
5. While doing these above changes, found that rftype flags needed some cleanup.
They were named inconsistently. Re-arranged them much more cleanly now and added
few comments. Hope it can help future additions.
---
v10:
Patches 1-2: no code changes. Added "Reviewed-by" from Fenghua.
Patch 3: Added the new flag RFTYPE_MON_BASE [comment for Reinette].
Removed the Reviewed as the patch has changed.
patch 4: No code change. Added "Reviewed-by" from Fenghua.
Patch 5: Removed empty lines in rdt_disable_ctx().
Patch 6: No changes. Added "Reviewed-by".
Patch 7: No changes.
Patch 8: No changes.
Patch 9: New patch. Added support for the files for MON groups only.
Patch 10: Modified. This patch only adds changes for "mon_hw_id" interface.
v9:
Changes since v8:
Split the RMID display in a separate patch. RMID is a special case here as it
should be printed only if monitoring is enabled.
Made rdtgroup_setup_root and rdtgroup_destroy_root as static functions.
Added code to print pid_str in case of task parse error.
v8:
Changes since v7:
Earlier moved both default group initialization and file creation on mount.
Now kept the group initialization as is and only moved the file creation on mount.
Re-organized the RFTYPE flags comments little more to avoid confusion. Thanks
to Reinette for feedback.
Re-factored the rdt_enable_ctx and added rdt_disable_ctx to unwind the errors.
And same call also used in rdt_kill_sb which simplifies the code.
Few other minor text changes.
v7:
Changes since v6:
While moving the default group file creation on mount, I also moved the
initialization of default group data structures. Reinette suggested to move
only the filesystem creation and keep the group initialization as is. Addressed it now.
Added a new function rdt_disable_ctx to unwind the context related features.
Few other minor text changes.
v6:
Changes since v5:
Moved the default group creation during mount instead of kernel init.
The rdt_root creation moved to rdt_get_tree as suggested by Reinette.
https://lore.kernel.org/lkml/[email protected]/
Needed to modify rdtgroup_setup_root to take care of this.
Re-arraged the patches to move the default group creation earlier.
Others are mostly text changes and few minor changes.
Patches are based on tip/master commit 1a2945f27157825a561be7840023e3664111ab2f
v5:
Changes since v4:
Moved the default group creation during mount instead of kernel init.
Tried to address most of the comments on commit log. Added more context and details.
Addressed feedback about the patch4. Removed the code changes and only kept the comments.
I am ok to drop patch4. But I will wait for the comment on that.
There were lots of comments. Hope I did not miss anything. Even if I missed, it is
not intentional.
v4: Changes since v3
Addressed comments from Reinette and others.
Removed newline requirement when adding tasks.
Dropped one of the changes on flags. Kept the flag names mostly same.
Changed the names of closid and rmid to ctrl_hw_id and mon_hw_id respectively.
James had some concerns about adding these files. Addressed it by making these
files x86 specific.
Tried to address Reinette's comment on patch 7. But due to current code design
I could not do it exact way. But changed it little bit to make it easy debug
file additions in the future.
v3: Changes since v2
Still waiting for more comments. While waiting, addressed few comments from Fenghua.
Added few more texts in the documentation about multiple tasks assignment feature.
Added pid in last_cmd_status when applicable.
Introduced static resctrl_debug to save the debug option.
Few minor text changes.
v2: Changes since v1
a. Removed the changes to add the task's threads automatically. It required
book keeping to handle the failures and gets complicated. Removed that change
for now.
b. Added -o debug option to mount in debug mode(comment from Fenghua)
c. Added debug files rmid and closid. Stephane wanted to rename them more
generic to accommodate ARM. It kind of loses meaning if is renamed differently.
Kept it same for now. Will change if he feels strong about it.
v9: https://lore.kernel.org/lkml/[email protected]/
v8: https://lore.kernel.org/lkml/[email protected]/
v7: https://lore.kernel.org/lkml/169178429591.1147205.4030367096506551808.stgit@bmoger-ubuntu/
v6: https://lore.kernel.org/lkml/168980872063.1619861.420806535295905172.stgit@bmoger-ubuntu/
v5: https://lore.kernel.org/lkml/168564586603.527584.10518315376465080920.stgit@bmoger-ubuntu/
v4: https://lore.kernel.org/lkml/168177435378.1758847.8317743523931859131.stgit@bmoger-ubuntu/
v3: https://lore.kernel.org/lkml/167778850105.1053859.14596357862185564029.stgit@bmoger-ubuntu/
v2: https://lore.kernel.org/lkml/167537433143.647488.9641864719195184123.stgit@bmoger-ubuntu/
v1: https://lore.kernel.org/lkml/167278351577.34228.12803395505584557101.stgit@bmoger-ubuntu/
*** BLURB HERE ***
Babu Moger (10):
x86/resctrl: Add multiple tasks to the resctrl group at once
x86/resctrl: Simplify rftype flag definitions
x86/resctrl: Rename rftype flags for consistency
x86/resctrl: Add comments on RFTYPE flags hierarchy
x86/resctrl: Unwind the errors inside rdt_enable_ctx()
x86/resctrl: Move default group file creation to mount
x86/resctrl: Introduce "-o debug" mount option
x86/resctrl: Display CLOSID for resource group
x86/resctrl: Add support for the files for MON groups only
x86/resctrl: Display RMID of resource group
Documentation/arch/x86/resctrl.rst | 22 ++-
arch/x86/kernel/cpu/resctrl/internal.h | 88 +++++++--
arch/x86/kernel/cpu/resctrl/rdtgroup.c | 253 ++++++++++++++++++-------
3 files changed, 281 insertions(+), 82 deletions(-)
--
2.34.1
resctrl uses RFTYPE flags for creating resctrl directory structure.
Definitions and directory structures are not documented. Add
comments to improve the readability and help future additions.
Reviewed-by: Reinette Chatre <[email protected]>
Reviewed-by: Fenghua Yu <[email protected]>
Signed-off-by: Babu Moger <[email protected]>
---
arch/x86/kernel/cpu/resctrl/internal.h | 58 ++++++++++++++++++++++++++
1 file changed, 58 insertions(+)
diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
index f71bc82c882f..14988c9f402c 100644
--- a/arch/x86/kernel/cpu/resctrl/internal.h
+++ b/arch/x86/kernel/cpu/resctrl/internal.h
@@ -240,6 +240,64 @@ struct rdtgroup {
/*
* Define the file type flags for base and info directories.
+ *
+ * RESCTRL filesystem has two main components
+ * a. info
+ * b. base
+ *
+ * /sys/fs/resctrl/
+ * |
+ * --> info (Top level directory named "info". Contains files that
+ * | provide details on control and monitoring resources.)
+ * |
+ * --> base (Root directory associated with default resource group
+ * as well as directories created by user for MON and CTRL
+ * groups. Contains files to interact with MON and CTRL
+ * groups.)
+ *
+ * Note: resctrl uses flags for files, not for directories.
+ * Directories are created based on the resource type. Added
+ * directories below for better understanding.
+ *
+ * info directory structure
+ * ------------------------------------------------------------------
+ * --> RFTYPE_INFO
+ * Directory: info
+ * --> RFTYPE_TOP (Files in top level of info directory)
+ * File: last_cmd_status
+ *
+ * --> RFTYPE_MON (Files for all monitoring resources)
+ * Directory: L3_MON
+ * Files: mon_features, num_rmids
+ *
+ * --> RFTYPE_RES_CACHE (Files for cache monitoring resources)
+ * Directory: L3_MON
+ * Files: max_threshold_occupancy,
+ * mbm_total_bytes_config,
+ * mbm_local_bytes_config
+ *
+ * --> RFTYPE_CTRL (Files for all control resources)
+ * Directories: L2, L3, MB, SMBA, L2CODE, L2DATA, L3CODE, L3DATA
+ * File: num_closids
+ *
+ * --> RFTYPE_RES_CACHE (Files for cache control resources)
+ * Directories: L2, L3, L2CODE, L2DATA, L3CODE, L3DATA
+ * Files: bit_usage, cbm_mask, min_cbm_bits,
+ * shareable_bits
+ *
+ * --> RFTYPE_RES_MB (Files for memory control resources)
+ * Directories: MB, SMBA
+ * Files: bandwidth_gran, delay_linear,
+ * min_bandwidth, thread_throttle_mode
+ *
+ * base directory structure
+ * ------------------------------------------------------------------
+ * --> RFTYPE_BASE (Files common for both MON and CTRL groups)
+ * Files: cpus, cpus_list, tasks
+ *
+ * --> RFTYPE_CTRL (Files only for CTRL group)
+ * Files: mode, schemata, size
+ *
*/
#define RFTYPE_INFO BIT(0)
#define RFTYPE_BASE BIT(1)
--
2.34.1
In x86, hardware uses CLOSID to identify a control group. When a user
creates a control group this information is not visible to the user.
It can help resctrl debugging.
Add CLOSID(ctrl_hw_id) to the control groups display in resctrl
interface. Users can see this detail when resctrl is mounted with
"-o debug" option.
Other architectures do not use "CLOSID". Use the names ctrl_hw_id
to refer to "CLOSID" in an effort to keep the naming generic.
For example:
$cat /sys/fs/resctrl/ctrl_grp1/ctrl_hw_id
1
Signed-off-by: Babu Moger <[email protected]>
---
Documentation/arch/x86/resctrl.rst | 4 ++++
arch/x86/kernel/cpu/resctrl/internal.h | 3 +++
arch/x86/kernel/cpu/resctrl/rdtgroup.c | 23 +++++++++++++++++++++++
3 files changed, 30 insertions(+)
diff --git a/Documentation/arch/x86/resctrl.rst b/Documentation/arch/x86/resctrl.rst
index 28d35aaa74b4..54691c8b832d 100644
--- a/Documentation/arch/x86/resctrl.rst
+++ b/Documentation/arch/x86/resctrl.rst
@@ -352,6 +352,10 @@ When control is enabled all CTRL_MON groups will also contain:
file. On successful pseudo-locked region creation the mode will
automatically change to "pseudo-locked".
+"ctrl_hw_id":
+ Available only with debug option. The identifier used by hardware
+ for the control group. On x86 this is the CLOSID.
+
When monitoring is enabled all MON groups will also contain:
"mon_data":
diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
index 68d1b7147291..a07fa4329b65 100644
--- a/arch/x86/kernel/cpu/resctrl/internal.h
+++ b/arch/x86/kernel/cpu/resctrl/internal.h
@@ -299,6 +299,9 @@ struct rdtgroup {
* --> RFTYPE_CTRL (Files only for CTRL group)
* Files: mode, schemata, size
*
+ * --> RFTYPE_DEBUG (Files to help resctrl debugging)
+ * File: ctrl_hw_id
+ *
*/
#define RFTYPE_INFO BIT(0)
#define RFTYPE_BASE BIT(1)
diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
index 150105c21fee..953b082cec8a 100644
--- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
+++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
@@ -779,6 +779,22 @@ static int rdtgroup_tasks_show(struct kernfs_open_file *of,
return ret;
}
+static int rdtgroup_closid_show(struct kernfs_open_file *of,
+ struct seq_file *s, void *v)
+{
+ struct rdtgroup *rdtgrp;
+ int ret = 0;
+
+ rdtgrp = rdtgroup_kn_lock_live(of->kn);
+ if (rdtgrp)
+ seq_printf(s, "%u\n", rdtgrp->closid);
+ else
+ ret = -ENOENT;
+ rdtgroup_kn_unlock(of->kn);
+
+ return ret;
+}
+
#ifdef CONFIG_PROC_CPU_RESCTRL
/*
@@ -1863,6 +1879,13 @@ static struct rftype res_common_files[] = {
.seq_show = rdtgroup_size_show,
.fflags = RFTYPE_CTRL_BASE,
},
+ {
+ .name = "ctrl_hw_id",
+ .mode = 0444,
+ .kf_ops = &rdtgroup_kf_single_ops,
+ .seq_show = rdtgroup_closid_show,
+ .fflags = RFTYPE_CTRL_BASE | RFTYPE_DEBUG,
+ },
};
--
2.34.1
On 9/15/23 15:42, Babu Moger wrote:
> In x86, hardware uses CLOSID to identify a control group. When a user
> creates a control group this information is not visible to the user.
> It can help resctrl debugging.
>
> Add CLOSID(ctrl_hw_id) to the control groups display in resctrl
> interface. Users can see this detail when resctrl is mounted with
> "-o debug" option.
>
> Other architectures do not use "CLOSID". Use the names ctrl_hw_id
> to refer to "CLOSID" in an effort to keep the naming generic.
>
> For example:
> $cat /sys/fs/resctrl/ctrl_grp1/ctrl_hw_id
> 1
>
> Signed-off-by: Babu Moger <[email protected]>
Reviewed-by: Fenghua Yu <[email protected]>
Thanks.
-Fenghua
Hi Babu,
On 9/15/2023 3:42 PM, Babu Moger wrote:
> In x86, hardware uses CLOSID to identify a control group. When a user
> creates a control group this information is not visible to the user.
> It can help resctrl debugging.
>
> Add CLOSID(ctrl_hw_id) to the control groups display in resctrl
> interface. Users can see this detail when resctrl is mounted with
> "-o debug" option.
>
> Other architectures do not use "CLOSID". Use the names ctrl_hw_id
> to refer to "CLOSID" in an effort to keep the naming generic.
>
> For example:
> $cat /sys/fs/resctrl/ctrl_grp1/ctrl_hw_id
> 1
>
> Signed-off-by: Babu Moger <[email protected]>
> ---
Reviewed-by: Reinette Chatre <[email protected]>
Reinette
Hi Babu,
I tested these features and ran the selftests/resctrl test set on Intel(R) Xeon(R) Gold 6254 CPU, there is no problem.
<Reviewed-by:[email protected]>
<Tested-by:[email protected]>
Best regards,
Shaopeng TAN
Hi Shaopeng,
On 9/28/23 00:28, Shaopeng Tan (Fujitsu) wrote:
> Hi Babu,
>
> I tested these features and ran the selftests/resctrl test set on Intel(R) Xeon(R) Gold 6254 CPU, there is no problem.
>
> <Reviewed-by:[email protected]>
> <Tested-by:[email protected]>
Thanks. I believe the normal format is "Tag: Name <email>"
Like this below. Please update next time.
Reviewed-by:Shaopeng Tan <[email protected]>
--
Thanks
Babu Moger
On Fri, 15 Sep 2023, Babu Moger wrote:
> resctrl uses RFTYPE flags for creating resctrl directory structure.
>
> Definitions and directory structures are not documented. Add
> comments to improve the readability and help future additions.
>
> Reviewed-by: Reinette Chatre <[email protected]>
> Reviewed-by: Fenghua Yu <[email protected]>
> Signed-off-by: Babu Moger <[email protected]>
> ---
> arch/x86/kernel/cpu/resctrl/internal.h | 58 ++++++++++++++++++++++++++
> 1 file changed, 58 insertions(+)
>
> diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
> index f71bc82c882f..14988c9f402c 100644
> --- a/arch/x86/kernel/cpu/resctrl/internal.h
> +++ b/arch/x86/kernel/cpu/resctrl/internal.h
> @@ -240,6 +240,64 @@ struct rdtgroup {
>
> /*
> * Define the file type flags for base and info directories.
> + *
> + * RESCTRL filesystem has two main components
> + * a. info
> + * b. base
> + *
> + * /sys/fs/resctrl/
> + * |
> + * --> info (Top level directory named "info". Contains files that
> + * | provide details on control and monitoring resources.)
> + * |
> + * --> base (Root directory associated with default resource group
> + * as well as directories created by user for MON and CTRL
> + * groups. Contains files to interact with MON and CTRL
> + * groups.)
> + *
> + * Note: resctrl uses flags for files, not for directories.
> + * Directories are created based on the resource type. Added
> + * directories below for better understanding.
> + *
> + * info directory structure
> + * ------------------------------------------------------------------
> + * --> RFTYPE_INFO
> + * Directory: info
> + * --> RFTYPE_TOP (Files in top level of info directory)
> + * File: last_cmd_status
> + *
> + * --> RFTYPE_MON (Files for all monitoring resources)
> + * Directory: L3_MON
> + * Files: mon_features, num_rmids
> + *
> + * --> RFTYPE_RES_CACHE (Files for cache monitoring resources)
> + * Directory: L3_MON
> + * Files: max_threshold_occupancy,
> + * mbm_total_bytes_config,
> + * mbm_local_bytes_config
> + *
> + * --> RFTYPE_CTRL (Files for all control resources)
> + * Directories: L2, L3, MB, SMBA, L2CODE, L2DATA, L3CODE, L3DATA
> + * File: num_closids
> + *
> + * --> RFTYPE_RES_CACHE (Files for cache control resources)
> + * Directories: L2, L3, L2CODE, L2DATA, L3CODE, L3DATA
> + * Files: bit_usage, cbm_mask, min_cbm_bits,
> + * shareable_bits
> + *
> + * --> RFTYPE_RES_MB (Files for memory control resources)
> + * Directories: MB, SMBA
> + * Files: bandwidth_gran, delay_linear,
> + * min_bandwidth, thread_throttle_mode
> + *
> + * base directory structure
> + * ------------------------------------------------------------------
> + * --> RFTYPE_BASE (Files common for both MON and CTRL groups)
> + * Files: cpus, cpus_list, tasks
> + *
> + * --> RFTYPE_CTRL (Files only for CTRL group)
> + * Files: mode, schemata, size
> + *
> */
> #define RFTYPE_INFO BIT(0)
> #define RFTYPE_BASE BIT(1)
>
Reviewed-by: Ilpo J?rvinen <[email protected]>
--
i.
On Fri, 15 Sep 2023, Babu Moger wrote:
> In x86, hardware uses CLOSID to identify a control group. When a user
> creates a control group this information is not visible to the user.
> It can help resctrl debugging.
>
> Add CLOSID(ctrl_hw_id) to the control groups display in resctrl
> interface. Users can see this detail when resctrl is mounted with
> "-o debug" option.
>
> Other architectures do not use "CLOSID". Use the names ctrl_hw_id
> to refer to "CLOSID" in an effort to keep the naming generic.
>
> For example:
> $cat /sys/fs/resctrl/ctrl_grp1/ctrl_hw_id
> 1
>
> Signed-off-by: Babu Moger <[email protected]>
> ---
> Documentation/arch/x86/resctrl.rst | 4 ++++
> arch/x86/kernel/cpu/resctrl/internal.h | 3 +++
> arch/x86/kernel/cpu/resctrl/rdtgroup.c | 23 +++++++++++++++++++++++
> 3 files changed, 30 insertions(+)
>
> diff --git a/Documentation/arch/x86/resctrl.rst b/Documentation/arch/x86/resctrl.rst
> index 28d35aaa74b4..54691c8b832d 100644
> --- a/Documentation/arch/x86/resctrl.rst
> +++ b/Documentation/arch/x86/resctrl.rst
> @@ -352,6 +352,10 @@ When control is enabled all CTRL_MON groups will also contain:
> file. On successful pseudo-locked region creation the mode will
> automatically change to "pseudo-locked".
>
> +"ctrl_hw_id":
> + Available only with debug option. The identifier used by hardware
> + for the control group. On x86 this is the CLOSID.
> +
> When monitoring is enabled all MON groups will also contain:
>
> "mon_data":
> diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
> index 68d1b7147291..a07fa4329b65 100644
> --- a/arch/x86/kernel/cpu/resctrl/internal.h
> +++ b/arch/x86/kernel/cpu/resctrl/internal.h
> @@ -299,6 +299,9 @@ struct rdtgroup {
> * --> RFTYPE_CTRL (Files only for CTRL group)
> * Files: mode, schemata, size
> *
> + * --> RFTYPE_DEBUG (Files to help resctrl debugging)
> + * File: ctrl_hw_id
> + *
> */
> #define RFTYPE_INFO BIT(0)
> #define RFTYPE_BASE BIT(1)
> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> index 150105c21fee..953b082cec8a 100644
> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> @@ -779,6 +779,22 @@ static int rdtgroup_tasks_show(struct kernfs_open_file *of,
> return ret;
> }
>
> +static int rdtgroup_closid_show(struct kernfs_open_file *of,
> + struct seq_file *s, void *v)
> +{
> + struct rdtgroup *rdtgrp;
> + int ret = 0;
> +
> + rdtgrp = rdtgroup_kn_lock_live(of->kn);
> + if (rdtgrp)
> + seq_printf(s, "%u\n", rdtgrp->closid);
> + else
> + ret = -ENOENT;
> + rdtgroup_kn_unlock(of->kn);
> +
> + return ret;
> +}
> +
> #ifdef CONFIG_PROC_CPU_RESCTRL
>
> /*
> @@ -1863,6 +1879,13 @@ static struct rftype res_common_files[] = {
> .seq_show = rdtgroup_size_show,
> .fflags = RFTYPE_CTRL_BASE,
> },
> + {
> + .name = "ctrl_hw_id",
> + .mode = 0444,
> + .kf_ops = &rdtgroup_kf_single_ops,
> + .seq_show = rdtgroup_closid_show,
> + .fflags = RFTYPE_CTRL_BASE | RFTYPE_DEBUG,
> + },
>
> };
Reviewed-by: Ilpo J?rvinen <[email protected]>
--
i.