Received-SPF: pass (google.com: domain of linux-kernel+bounces-89223-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Message-ID: <c73f444b-83a1-4e9a-95d3-54c5165ee782@intel.com>
Date: Fri, 1 Mar 2024 15:20:32 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v2 00/17] x86/resctrl : Support AMD Assignable Bandwidth
 Monitoring Counters (ABMC)
Content-Language: en-US
To: <babu.moger@amd.com>, James Morse <james.morse@arm.com>, <corbet@lwn.net>,
	<fenghua.yu@intel.com>, <tglx@linutronix.de>, <mingo@redhat.com>,
	<bp@alien8.de>, <dave.hansen@linux.intel.com>, Peter Newman
	<peternewman@google.com>
CC: <x86@kernel.org>, <hpa@zytor.com>, <paulmck@kernel.org>,
	<rdunlap@infradead.org>, <tj@kernel.org>, <peterz@infradead.org>,
	<yanjiewtw@gmail.com>, <kim.phillips@amd.com>, <lukas.bulwahn@gmail.com>,
	<seanjc@google.com>, <jmattson@google.com>, <leitao@debian.org>,
	<jpoimboe@kernel.org>, <rick.p.edgecombe@intel.com>,
	<kirill.shutemov@linux.intel.com>, <jithu.joseph@intel.com>,
	<kai.huang@intel.com>, <kan.liang@linux.intel.com>,
	<daniel.sneddon@linux.intel.com>, <pbonzini@redhat.com>,
	<sandipan.das@amd.com>, <ilpo.jarvinen@linux.intel.com>,
	<maciej.wieczor-retman@intel.com>, <linux-doc@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>, <eranian@google.com>
References: <20231201005720.235639-1-babu.moger@amd.com>
 <cover.1705688538.git.babu.moger@amd.com>
 <2f373abf-f0c0-4f5d-9e22-1039a40a57f0@arm.com>
 <474ebe02-2d24-4ce3-b26a-46c520efd453@amd.com>
 <b6bb6a59-67c2-47bc-b8d3-04cf8fd21219@intel.com>
 <3fe3f235-d8a6-453b-b69d-6b7f81c07ae1@amd.com>
 <9b94b97e-4a8c-415e-af7a-d3f832592cf9@intel.com>
 <1ae73c9a-cec4-4496-86c6-3ffcef7940d6@amd.com>
 <32a588e2-7b09-4257-b838-4268583a724d@intel.com>
 <088878bd-7533-492d-838c-6b39a93aad4d@amd.com>
 <9b20589b-6220-4ae7-bfc4-4a826b7114b1@intel.com>
 <5ddb7031-a828-4001-bfdf-5477cb85e9ed@amd.com>
 <1738493b-3248-4c9e-82a8-1599a033440d@intel.com>
 <369ab28a-f3fa-4359-8e73-4dcf214c9b6e@amd.com>
 <54687d59-d0e4-4fe7-b25f-dc1fead01ea1@intel.com>
 <11487a31-908e-d474-50c6-65617d417deb@amd.com>
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <11487a31-908e-d474-50c6-65617d417deb@amd.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dE9kSXRHa0U2OHUrQksrKzJ2dktXRlZUUDFMb3lFTXRvUUpRWStMbW9ZeDdR?=
 =?utf-8?B?WDUvT3JiWXJJU2lKclpVUDRMelkrTGFMK1l1Q1hRL2VtaG5mOFZIQ1J1WXZQ?=
 =?utf-8?B?QTVpZUxEWXR1SzZ2bXBJZ1B0bzJMNnVhSTU2bmUwMFpIT2V2cEFvWE15THdl?=
 =?utf-8?B?TXh3NFhwTE1xWDFSMXdVT1Fza3lFSkgzYXF6d3dtZHJwRG9wK0FqKzI0ZTl2?=
 =?utf-8?B?L1V3SHNYZklsbFlaVUhoZEN4dTA3QmFNcWFQNHF0ZHZxRWNIdmt6TFhGY2d0?=
 =?utf-8?B?R1M1NG1iUmttODV0QXRxeDJLRlB3eVZFYTlhMDk4Um0wODdoR3crQXlwemZi?=
 =?utf-8?B?a0VMa213RlhKYkRaeVkrTEVUazJLNkY1VnNzRmJkMUF3ZUoyYS9OcThSUXZL?=
 =?utf-8?B?b2dmRFp2R2JtQjhydU5yYi9ZMEFvdEhyK1QzZ2ZaeFNiNC9sUTlLZURRREpV?=
 =?utf-8?B?N1lBYk8zYzdFVWRzdkJKb0VUT3lxKzhmLzI5MVRSOTdsWG1pSmJDWVc3MlYx?=
 =?utf-8?B?a2VNUldPczBwcVB2VUhnbFFrSWsvOHdwdGlUYk9vV0JmM2RkSEZQRHQyZkZT?=
 =?utf-8?B?ckVxRmlNNTE1eU83SUw0SGY0cTlDUkdJWFNEMDNLeE1wajlFR1kzMW9KQ0JI?=
 =?utf-8?B?VGdrRWJXd1BXZjdaVHZnREc5dlc3Q3JxbENTL2hOL2JnZDJoTnF3YXpCSzBz?=
 =?utf-8?B?RDk2YkFRRDB1a25VbkRaWlFpZEhWNFNJNnByNllLd0E5VFgxdU5RTldocCtQ?=
 =?utf-8?B?bVo4VmxCWVlDRG1mRElpNW9iZHRJR0xZWjZTZDRkUDZwZjhTV1lVRUdzU0xr?=
 =?utf-8?B?b2RqVmFqcURaYU1MVEthYXRIdE53YWVyN1U5OU5FVkNvc0V2TEpUVWUwTTB5?=
 =?utf-8?B?Qzh2MGd3eVRJZUJPdE1QRWQvR0lVaFBtUGliN0NHZndjcHZpdSttei9EY3NM?=
 =?utf-8?B?Wmw0SVRpRitaNGdaTzVLVHg3UlExTDhBQTFaKzJiSmU2MVFFZnlUSWRDbE1s?=
 =?utf-8?B?TGNwbjFaa3Jqc2FndnVUektQcUhucnorMVZLREphS2dXUHRIeVVIakp1aTZH?=
 =?utf-8?B?ZHJPVWE2V2haMmo3eTR2dVVjcWF6MzdrVEQ0cEEvRVJGdzNiUVVleVNKckhm?=
 =?utf-8?B?S1dMbWgxUGJXbGRnQXZBcGg3S0V2YWpHT1JMSFZOUTVMdTVYdzZ0d0VsQXA1?=
 =?utf-8?B?UGRsNkhKSk1wd09aRXMwZE5YdlBPQ21vaVNTbmxMZHVyME4zQ3VnazEvVzIr?=
 =?utf-8?B?dWRDbG5HREQveXR3Z1ZsT09vbjk1L1QzaVF6MUJVS1d4UEd0d2hveHhPQ0tj?=
 =?utf-8?B?UVpoekhYYlBaOXJKd0Rxd3JxWElEYkNkbStSTDU4ckdCVXhEWG4xeU1mUzhH?=
 =?utf-8?B?c1E3Y0RPYjd2dFNGS2NkbkRvanBQZ3llUUJnTXYwdmxkMzQ4T2tnVjR2Ti9u?=
 =?utf-8?B?M05sTFRZb2JtWFV2WTZyUzRUT1paRXFXdUJBWWJZMVNNRTdJL2s0U090N1lL?=
 =?utf-8?B?dW1BYjNxUFFzelRzOTVOWjBScDU2RjU5RzRVT0hwbzkyTGtLb1RrTmVCMXlP?=
 =?utf-8?B?Yjl2WDh4aHF2SGllSGowdjNiblpMRGlkdHN1a24wOHJEOU5XZE5oNWxmMkZW?=
 =?utf-8?B?bHFqZlRmdFZrSWRRVU8relJCNE95K3o2VTc5MUt1bTFsUU51d0R3NVJTK21h?=
 =?utf-8?B?V05rTktrUEs4aTV6cVplTmJuQmVpR0VJdGNXeFBkQmQ2aDdrODB3SVlHNUZE?=
 =?utf-8?B?QnRlNGY1anhkYmgxOU5GN1ZaaDF4ck9WVmFoaVB1TEtsOWw4OERJd2JPVm1t?=
 =?utf-8?B?K2k1RG1TdmlyMzd1OXM5cHdNTTZTM3l2VnZDWjViTi9OTm9iOTliZVowSnJz?=
 =?utf-8?B?NVltOEJEb1J0UjNBeURLYkMxejFFSXFEQi9jYmFUNFJDajVodGZ4c3hGdzc5?=
 =?utf-8?B?dGZ5NVcvbEkzMzE1dEtMZk1PWFBUMXluVlA2RkpyY3JxMk5kenhzTTg5alNH?=
 =?utf-8?B?MG5sckNOSHFmUld2bEpnc1lUZ1UwQmc4dndWM1ZGVWU1TGtad2FkWXVPYU5q?=
 =?utf-8?B?M0VkTE54V3luckkvWklPUWpyc0lSSnp5V1hDaDJ4SlkxVzk1MkgrcTdBWkVj?=
 =?utf-8?B?Mkl0ZnBXRzVQdVE5STNBQTdzOEFSZThCb1hYeWpJUUNUcWd5TlA2Rk1DdnF2?=
 =?utf-8?B?YUE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: f245645d-052f-4636-c4e3-08dc3a46333d
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Mar 2024 23:20:36.3782
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Gvhje/q1vmrdVybUYAkBGsOuG7hYb33NnYEbAtQXvH5mntxUORz33g2keVkgc/lxdaPApGKKPkWi4LWyMrL4+qLUd+uHvFgJAclYhKmWwh8=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB8250
X-OriginatorOrg: intel.com

Hi Babu,

On 3/1/2024 12:36 PM, Moger, Babu wrote:
> On 2/29/24 15:50, Reinette Chatre wrote:
>> On 2/29/2024 12:37 PM, Moger, Babu wrote:

..

>>> To make more clear, let me list all the groups here based this.
>>>
>>> When none of the counters assigned:
>>>
>>> $cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control
>>> resctrl/00=none,none;01=none,none (#default control_mon group)
>>> resctrl/mon_a/00=none,none;01=none,none (#mon group)
>>> resctrl/ctrl_a/00=none,none;01=none,none (#control_mon group)
>>> resctrl/ctrl_a/mon_ab/00=none,none;01=none,none (#mon group)
>>
>> I am concerned that inconsistent use of "/" will make parsing hard.
> 
> Do you mean, you don't want to see multiple "/"?

No. I think that having a consistent number of "/" will be easier to
parse. In the above example, there are instances of 1, 2, as well as
three "/" among the lines. That seems complicated to parse.

I was thinking that it will make interpreting and parsing easier if there
consistently are just always two "/".

(You may find things to be different once you work on the parsing code
though.)

In summary:
* for monitoring of default CTRL_MON group: "//<flags>"
* for MON_GROUP inside default CTRL_MON group: "/<MON group>/<flags>"
* for monitoring of non-default CTRL_MON group: "<CTRL_MON group>//flags"
* for MON_GROUP within CTRL_MON group: "<CTRL_MON group>/<MON group>/<flags>"

What do you think?

> 
> resctrl/ctrl_a/mon_ab/
> 
> Change to
> 
> mon_ab/

rather:
ctrl_a/mon_ab/<flags>

> 
>>
>> I find "resctrl" and all the "none" redundant. It is not clear what
>> this improves.
>> Why have:
>> resctrl/00=none,none;01=none,none
>> when this could do:
>> //00=_;01=_
> 
> ok.
> 
> "//" meaning root of resctrl filesystem?

More specifically, monitoring of default control group. It is not intended to
specify a pathname.

>>> When some counters are assigned:
>>>
>>> $echo "resctrl/00=total,local" >
>>> /sys/fs/resctrl/info/L3_MON/mbm_assign_control (#assigning counter to
>>> default group)
>>>
>>> $echo "resctrl/mon_a/00=total;01=total" >
>>> /sys/fs/resctrl/info/L3_MON/mbm_assign_control (#assigning counter to mon
>>> group)
>>>
>>> $echo "resctrl/ctrl_a/00=local;01=local" >
>>> /sys/fs/resctrl/info/L3_MON/mbm_assign_control
>>>
>>> $echo "resctrl/ctrl_a/mon_ab/00=total,local;01=total,local" >
>>> /sys/fs/resctrl/nfo/L3_MON/mbm_assign_control
>>>
>>
>> We could learn some more lessons from dynamic debug (see Documentation/admin-guide/dynamic-debug-howto.rst). For example, "=" can be used to make an assignment while "+"
>> can be used to add a counter and "-" can be used to remove a counter.
>> "=_" can be used to remove counters from all events in that domain.
> 
> Yes. Looked at dynamic debug. I am still learning this interface. Some examples below based on my understanding.
> 
> To assign a counters to default group on domain 0.
> $echo "//00=+lt;01=+lt" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control

It should not be necessary to use both "=" and "+" in the same assignment.
I think of "=" as "assign" and "+" as append ("-" as remove).

An example of this, just focusing on default group.

#hypothetical start state of no counters assigned
$ cat mbm_assign_control 
#control_group/monitor_group/flags
//00=_;01=_

#assign counter to total MBM of both domains
$ echo "//00=t;01=t" > mbm_assign_control 
$ cat mbm_assign_control
#control_group/monitor_group/flags
//00=t;01=t

#add counter to local MBM of both domains without impacting total MBM counters
$echo "//00+l;01+l" > mbm_assign_control
$ cat mbm_assign_control
#control_group/monitor_group/flags
//00=tl;01=tl

#remove local MBM counters without impacting total MBM counters
$echo "//00-l;01-l" > mbm_assign_control
$ cat mbm_assign_control
#control_group/monitor_group/flags
//00=t;01=t

#assign local MBM counters, removing total MBM counters while doing so
$echo "//00=l;01=l" > mbm_assign_control
$ cat mbm_assign_control
#control_group/monitor_group/flags
//00=l;01=l

#remove all counters
$echo "//00=_;01=_" > mbm_assign_control
$ cat mbm_assign_control
#control_group/monitor_group/flags
//00=_;01=_


> 
> To assign a counters to mon group inside the default group.
> $echo "mon_a/00=+t;01=+t" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control

I think it will simplify parsing if number of "/" is consistent:
$echo "/mon_a/00=t;01=t" > ...

> 
> To assign a counters to control mon group inside the default group.

It is not clear to me what you mean with this.

> $echo "ctrl_a/00=+l;01=+l"  > /sys/fs/resctrl/info/L3_MON/mbm_assign_control

this looks similar to previous example, so I think it will be hard for parser
to know whether it is dealing with control group or monitor group.
I am not sure I understand your example, but this may perhaps be:

echo "ctrl_a//00=l;01=l > ...

> 
> To assign a counters to control mon group inside another control group.

I do not know what you mean with "another control group" 

> $echo "mon_ab/00=+lt;01=+lt" > /sys/fs/resctrl/nfo/L3_MON/mbm_assign_contro

How will parser know which control group? I was expecting:
$ echo "<CTRL_MON group>/<MON group>/<flags>"

> 
> To unassign a counters to control mon group inside another control group.
> $echo "mon_ab/00=-lt;01=-lt" > /sys/fs/resctrl/nfo/L3_MON/mbm_assign_control
> 
> To unassign all the counters on a specific group.
> $echo "mon_ab/00=_" > /sys/fs/resctrl/nfo/L3_MON/mbm_assign_control
> 
> It does not matter control group or mon group. We just need to name of the group in this interface.

It matters because users can have monitor groups with the same name within
different control groups.

Reinette