Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31;
Message-ID: <e36699cf-c73e-401b-b770-63eba708df38@intel.com>
Date:   Wed, 6 Dec 2023 10:49:13 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 00/15] x86/resctrl : Support AMD QoS RMID Pinning feature
Content-Language: en-US
To:     <babu.moger@amd.com>, <corbet@lwn.net>, <fenghua.yu@intel.com>,
        <tglx@linutronix.de>, <mingo@redhat.com>, <bp@alien8.de>,
        <dave.hansen@linux.intel.com>, James Morse <james.morse@arm.com>
CC:     <x86@kernel.org>, <hpa@zytor.com>, <paulmck@kernel.org>,
        <rdunlap@infradead.org>, <tj@kernel.org>, <peterz@infradead.org>,
        <seanjc@google.com>, <kim.phillips@amd.com>, <jmattson@google.com>,
        <ilpo.jarvinen@linux.intel.com>, <jithu.joseph@intel.com>,
        <kan.liang@linux.intel.com>, <nikunj@amd.com>,
        <daniel.sneddon@linux.intel.com>, <pbonzini@redhat.com>,
        <rick.p.edgecombe@intel.com>, <rppt@kernel.org>,
        <maciej.wieczor-retman@intel.com>, <linux-doc@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <eranian@google.com>,
        <peternewman@google.com>, <dhagiani@amd.com>
References: <20231201005720.235639-1-babu.moger@amd.com>
 <d97cbeba-af6d-4b64-b5c8-32dc437a67b6@intel.com>
 <71e85bf3-a451-4adf-ad5e-d39f7935efa0@amd.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <71e85bf3-a451-4adf-ad5e-d39f7935efa0@amd.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cVpWUlZHaGhlWjNrZ0VuWStiNjg5WGVmb2ZoRTBSRDJob2lNQ3gwUG9yWGlo?=
 =?utf-8?B?NTJCaEp5OFhPL05SMStteEJldTNveHgxWTd6N0tJU3NTdmEwTEhmQ1U3enJW?=
 =?utf-8?B?MUMzaVdxa0VnYktNR3ZpY3JuSkdteHl1QmdwVUNHakZ2bEFOU2Y1N1RnVWdD?=
 =?utf-8?B?QnVNM0JZMkw0aVEyRVRFdUl3WXpvc2xUVEZEa05scFRQWGNSd0xobG95RXZY?=
 =?utf-8?B?WHpKOGViODczbVE2cEsydGpTTDVJdkpNUFRBKzY2UWVsUUJMbVRVaUs5cUND?=
 =?utf-8?B?UndpYy9Qd1lMbzVpZllWRWNiMGowbnY1VngyWHVxd3A1UVFOV2pHQlIyd0gw?=
 =?utf-8?B?SXgvd2Q2R2xjdTFNNFdTOTJQYWRET0lydXFtbWdOc09sTEZ4NW9PTGV1aHU0?=
 =?utf-8?B?M0ovQnNnVFg1bWN3TUNXWWZOdk5BK1kvMUpYOWFBT2VrYXJ0ZXNMN0doMC9V?=
 =?utf-8?B?VVl2bzFFVllRWVBMQmlRLzFtRW9ITFlwOG5mSlgyUCt0S1BMZVYvSm1mUkI5?=
 =?utf-8?B?ajl5VUd2V1diTDFIc1ZITTErVEF5cC9oZlFOZytGUXJqYUNhNGtkMi92bXpr?=
 =?utf-8?B?MjA1bW15UDVqM0U4U3ltV3Bkb1ZGN1pWdVlEamgxc2pZN1BtNmc5d2tUc2Fu?=
 =?utf-8?B?TE0zWVk0ZHJPek9NK0hrVmM5cTczVFVieUtMdi9ibXlqWW1ISVN6Qk92RVRR?=
 =?utf-8?B?MHN6aVZFVG9XSXA1TVZMMTB0ZWo3bHF5MG9hSFl4RkFrZnJhQjZjaWt6LzJa?=
 =?utf-8?B?bFdZbXhLN3pRUStyTlQvWEY4UzZBenRyN2RKWCtWbU81WHZNTTNDc2VMVHRW?=
 =?utf-8?B?YWN5bXRiYmwrRWNYeE80emNJRkZPaVYyVTd6UXpVRFNYT2NxdlJSVGo4VmQz?=
 =?utf-8?B?MnN0WGYwN2VXdnZmQS9lcjJiL2dSR2dMMzV2akROMDZKVWJTVGVUU3RENm9Y?=
 =?utf-8?B?OGkvR0dhUFc0bUVDeDBmSnZ6WjJ5TnR3YW5SUGxvcitVVmZpMVZIbmRFcHhU?=
 =?utf-8?B?dU5iR2crRG5yYVdQYUhpeks4Z0M2OTJVbERJcDUzTHp1MFBjSnM1ckd5bDRX?=
 =?utf-8?B?NkM0bFVablhKbHY5Y0xaZE9FRytwZ0ZmMmdFL3RUOS90akQxeUc5SldkeE9I?=
 =?utf-8?B?ZDNXZnFrdUNVaWVVTEpaSnE0WTYvRFZkQ1o3MzZDOFh0Z2IyenpHMmZManVO?=
 =?utf-8?B?bjBwQkdGYWhQdlNmVk9wVFFoWVBxMTNGRVpuTk1DVjJSQndxaWEyNUJRS3hP?=
 =?utf-8?B?VFlOa05oREp6MzF5dkxwQWpJV1A2ZkNjSUkxS3Vmd2Zhcld5Rld4bjU0WnVz?=
 =?utf-8?B?bEhjOGhZNElIK25KOTgyelErVWxKWGZwMmQ3Nlc2eGR2dmpmbDNtbWNSczl1?=
 =?utf-8?B?RUFNejN6UXNZWUZFMnNUUFVoMG1QVXRUdko5RXFjL2RUSy92T01jQjc0Z0Vn?=
 =?utf-8?B?c2V0TU1EY1cvL0dpSnY5RjA4U25SQ2NCU1l0UVBwR0p4SEZUdlY4ZE14L3FD?=
 =?utf-8?B?TGtLbGc0L3RrbEQvYmk3YUVKS2taZ2c1bWVpbjUwTGFoL1JVSUJnckVIKytl?=
 =?utf-8?B?REVNcXprbE5iUEFsRFZ1QmtVeTFHYzQvZXVETVFDYVlTemw0Q3ZaeHF0NHNk?=
 =?utf-8?B?eDVpSndxald1aHQvMnc0ZFJZdmxvNDUrRFM1RmQ5RE5ZNUFDOFRnNG5jTGsw?=
 =?utf-8?B?WkJZcVY0SVpRQ0NZalVBbngrM2IxWXVTTzhvdmNmbFY0U3VLd3BrREREamF6?=
 =?utf-8?B?SkMxUkFoMnplYkVaT3JwMlYyRUZHNEJuN1M0TGdmQjRwY1M0Z2dBRDBVZHRU?=
 =?utf-8?B?YkZMMC9FTXIyaXRiRmpOM2wxSkh1ZjZRU0E3SCtXeFRKVGdSVnlnV2JPSExD?=
 =?utf-8?B?UzBTMlI4NHhiMlJNTUJBdGFzeEpLY2s2b2VJQ0JPczMrS3NkK0JlTVQ5a2tJ?=
 =?utf-8?B?N01rdlZPQjhXZzBqNnU1UW1SQmlNdFhISTBFOGpxWjRVUUdNbWJidDdxL3ky?=
 =?utf-8?B?QkJwN1A1U1RidFJXUzJsSTJ3OGtVQ0IyazRyaVJYdndlTUNoaTFSV0dUamJ0?=
 =?utf-8?B?LzdPL1ZEUGRpOEZNMndzM3Fkd28vRmFkdHhpbzFtUVVadzc5TmpyTnhZazBu?=
 =?utf-8?B?M3ZlTU42dGlCeTlaY1Y3T3U5R3ZpOGYzc0hlTmJqVEJYUkoycWtxL1ZOR0Zw?=
 =?utf-8?B?aXc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 4e7cd779-5900-4e2a-37e5-08dbf68c0bf1
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 18:49:16.2179
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 6oTp0JETj89i9wlUMnZz1giwThQ306cWO1NhsWweVDrtaT4cHFbKkO/nM1RuU0YkaBzdZh8NdIexLUxAOkOul4wsJ/XBLL+toA+59Fr/9UY=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR11MB7841
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Wed, 06 Dec 2023 10:50:17 -0800 (PST)

Hi Babu,

On 12/6/2023 7:40 AM, Moger, Babu wrote:
> Hi Reinette,
> 
> On 12/5/23 17:17, Reinette Chatre wrote:
>> (+James)
>>
>> Hi Babu,
>>
>> On 11/30/2023 4:57 PM, Babu Moger wrote:
>>> These series adds the support for AMD QoS RMID Pinning feature. It is also
>>
>> "These series" - is this series part of a bigger work?
> 
> No.
> There are some some plans to optimize rmid_reads. Peter is planning to
> work on that. But both are independent of each other.

I would propose that you use "This series" instead to avoid creating
wrong impression.

>>> a. Check if ABMC support is available
>>> 	#mount -t resctrl resctrl /sys/fs/resctrl/
>>> 	#cat /sys/fs/resctrl/info/L3_MON/mon_features 
>>> 	llc_occupancy
>>> 	mbm_total_bytes
>>> 	mbm_total_bytes_config
>>> 	mbm_local_bytes
>>> 	mbm_local_bytes_config
>>> 	abmc_capable ←  Linux kernel detected ABMC feature.
>>
>> (Please start thinking about a new name that is not the AMD feature
>> name. This is added to resctrl filesystem that is the generic interface
>> used to work with different architectures. This thus needs to be generalized
>> to what user requires and how it can be accommodated by the hardware ...
>> this is already expected to be needed by MPAM and having a AMD feature
>> name could cause confusion.)
> 
> Yes. Agree.
> 
> How about "assign_capable"?

Let's wait to learn more about other use case.

> 
>>
>>>
>>> b. Mount with ABMC support
>>> 	#umount /sys/fs/resctrl/
>>> 	#mount  -o abmc -t resctrl resctrl /sys/fs/resctrl/
>>> 	
>>
>> hmmm ... so this requires the user to mount resctrl, determine if the
>> feature is supported, unmount resctrl, remount resctrl with feature enabled.
>> Could you please elaborate what prevents this feature from being enabled
>> without needing to remount resctrl?
> 
> Spec says
> "Enabling ABMC: ABMC is enabled by setting L3_QOS_EXT_CFG.ABMC_En=1 (see
> Figure 19-7). When the state of ABMC_En is changed, it must be changed to
> the updated value on all logical processors in the QOS Domain.
> Upon transitions of the ABMC_En the following actions take place:
> All ABMC assignable bandwidth counters are reset to 0.
> The L3 default mode bandwidth counters are reset to 0.
> The L3_QOS_ABMC_CFG MSR is reset to 0."
> 
> So, all the monitoring group counters will be reset.
> 
> It is technically possible to enable without remount. But ABMC mode
> requires few new files(in each group) which I added when mounted with "-o
> abmc". Thought it is a better option.
> 
> Otherwise we need to add these files when ABMC is supported(not when
> enabled). Need to add another file in /sys/fs/resctrl/info/L3_MON to
> enable the feature on the fly.
> 
> Both are acceptable options. Any thoughts?

The new resctrl files in info/ could always be present. For example,
user space may want to know how many counters are available before
enabling the feature.

It is not yet obvious to me that this feature requires new files
in monitor groups.

>>> c. Read the monitor states. There will be new file "monitor_state"
>>>    for each monitor group when ABMC feature is enabled. By default,
>>>    both total and local MBM events are in "unassign" state.
>>> 	
>>> 	#cat /sys/fs/resctrl/monitor_state 
>>> 	total=unassign;local=unassign
>>> 	
>>> d. Read the event mbm_total_bytes and mbm_local_bytes. Note that MBA
>>>    events are not available until the user assigns the events explicitly.
>>>    Users need to assign the counters to monitor the events in this mode.
>>> 	
>>> 	#cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
>>> 	Unavailable
>>
>> How is the llc_occupancy event impacted when ABMC is enabled? Can all RMIDs
>> still be used to track cache occupancy?
> 
> llc_occupancy event is not impacted by ABMC mode. It can be still used to
> track cache occupancy.
> 
>>
>>> 	
>>> 	#cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes 
>>> 	Unavailable
>>
>> I believe that "Unavailable" already has an accepted meaning within current
>> interface and is associated with temporary failure. Even the AMD spec states "This
>> is generally a temporary condition and subsequent reads may succeed". In the
>> scenario above there is no chance that this counter would produce a value later.
>> I do not think it is ideal to overload existing interface with different meanings
>> associated with a new hardware specific feature ... something like "Disabled" seems
>> more appropriate.
> 
> Hardware still reports it as unavailable. Also, there are some error cases
> hardware can report unavailable. We may not be able to differentiate that.

This highlights that this resctrl feature is currently latched to AMD's 
ABMC. I do not think we should require that this resctrl feature is backed
by hardware that can support reads of counters that are disabled. A counter
read really only needs to be sent to hardware if it is enabled.

>> Considering this we may even consider using these files themselves as a
>> way to enable the counters if they are disabled. For example, just
>> "echo 1 > /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes" can be used
> 
> I am not sure about this. This is specific to domain 0. This group can
> have cpus from multiple domains. I think we should have the interface for
> all the domains(not for specific domain).

Are the ABMC registers not per CPU? This is unclear to me at this time
since changelog of patch #13 states it is per-CPU but yet the code
uses smp_call_function_any().

Even so, this needs to take other use cases into account. So far Peter
mentioned the scenario where enabling of one counter would do so for all
events associated with that counter and then there could also be a global
enable/disable.

> 
>> to enable this counter. No need for a new "monitor_state". Please note that this
>> is not an official proposal since there are two other use cases that still need to
>> be considered as we await James's feedback on how this may work for MPAM and
>> also how this may be useful on AMD hardware that does not support ABMC but
>> users may want to get similar benefits ([1])
> 
> Ok. Lets wait for James comments.

Reinette