Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <9965edff-c558-2962-4aad-3342480026bc@intel.com>
Date:   Fri, 26 Aug 2022 09:17:23 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.2.0
Subject: Re: [PATCH v3 02/10] x86/cpufeatures: Add Slow Memory Bandwidth
 Allocation feature flag
Content-Language: en-US
To:     <babu.moger@amd.com>, <fenghua.yu@intel.com>, <tglx@linutronix.de>,
        <mingo@redhat.com>, <bp@alien8.de>
CC:     <eranian@google.com>, <dave.hansen@linux.intel.com>,
        <x86@kernel.org>, <hpa@zytor.com>, <corbet@lwn.net>,
        <linux-kernel@vger.kernel.org>, <linux-doc@vger.kernel.org>,
        <bagasdotme@gmail.com>, "Luck, Tony" <tony.luck@intel.com>
References: <166117559756.6695.16047463526634290701.stgit@bmoger-ubuntu>
 <166117577662.6695.15496626554784059239.stgit@bmoger-ubuntu>
 <2b7e3511-8e69-2e47-0f11-7f1014b38940@intel.com>
 <1cf62647-20db-646d-ae83-c2ee61f257c7@amd.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <1cf62647-20db-646d-ae83-c2ee61f257c7@amd.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SkdhTUF1OVFMbVIwNjJoOE5wQ2tkUUNMRGh2UWUvUGVTWFdQYTRHNkU5YVdQ?=
 =?utf-8?B?cXM5UUk5dCtWTXlTbWdKSnNZRWVsdEg1M1ptdzNJaEJVd24remVFOG4ya0tV?=
 =?utf-8?B?Y1RhbmdWRkc1VEl2cGc2Qmh2amtUUlRrdlNPSXFMQWUvdXNTNXFvY3JXRG51?=
 =?utf-8?B?YTg4L05NZmpua1l6SjhnNTBpNjA3VkxXRkZQK1RiQkhOM2VTZjFCbG5BL01p?=
 =?utf-8?B?WXZkMlVDYm81UTRocDdaQUR5Zy9Idzg0VytMdTRBSVFvZXFSOEUyeHV4U3J0?=
 =?utf-8?B?SWJLWU0rd0F4UFl2VWFvWXkxNjViZkRWa3JOeUhPZU81dTlHQi9CZzRVZXZT?=
 =?utf-8?B?SlpFOEFCcG41M05BMHhiV1ZDVWJTc0hKd2ptRXNLQ2hMd2NxOEl5NHNBYTFN?=
 =?utf-8?B?bkc3eFlsRHdRSHNpdFMrd1hRYkRoQW5admRWN2RLbEJmVW9uZnp5ZWR1WXdY?=
 =?utf-8?B?YjRmVmRia09GUjJWbWF5ME1WMUFvTkJpY1lCMHVzdEh6RldLVmVFek94dFFT?=
 =?utf-8?B?VThLb2JvcXpUQ0Npdkl3cUlXOUlHaEJXdm5kbm9PbyswdVFKRC9yZ0pYWEVB?=
 =?utf-8?B?bjd5aDYzL2RLTEtRZTcvbnFGeng3TFdBVFUwb1pydW9IQkFpRGxPQjZaUkdw?=
 =?utf-8?B?WGdFVFBsWjNhc3k2U1U0Y0Zpc3JFMThGcDBEbVJzZ29UeERQTlhONTF3aGxS?=
 =?utf-8?B?WmQ3MG42TFRqeHlFRVNyL3NyTWtLNU5LcDlIRldCVG1NY3NtVnpZYnVSZ3Vq?=
 =?utf-8?B?LzAwSHVhRnZuRWQ3S2dicHZHcFI5bHF2SEZMek5GMkNTSndiZG5FMG1oSlRT?=
 =?utf-8?B?VVFBenhvTEZ6eStzMTUyRWYyMVJFTFdBRk5lbjhiWkxnTHR5SkVvVEQ0bTNo?=
 =?utf-8?B?ZEZOY0g3eHVjeCtqby9HWUtsNjI0cUJDUzkySlIyK2dtbEEveUhMRDdpMlJt?=
 =?utf-8?B?TkxKZFgzbmFSN0g0QlFNL0FsMTZod0RCemtTcmc3MWEya2dqMXd3dnlLZG4v?=
 =?utf-8?B?dzJMNHhBUEljbXprZGRmYzZpQWdHeVBtMCt6MlZIWkZ0NVpoNHhGTzJMNTY3?=
 =?utf-8?B?RDA3Qmxja2I3MjRNb3habjFoc1VPTXhvRkNBU1RJYndxNE1QNGNEd0hwSy9B?=
 =?utf-8?B?UGhHa3E3b1hOT0Q5OEpSR1JYak04NGVxT21KOS9BQWhjRm16L2JQWGhueS9i?=
 =?utf-8?B?UW9OVGdscFdsKzg4UWlKdkZ6aDJoWE1mSXRyeHRlTnNBbWI5dG1Nd1Rkai9Y?=
 =?utf-8?B?QitBcUxCaXJBcUo4NHBHcjF4aFZlWXBiUGk3NExiMExBVEhKRlliSTQxdTdP?=
 =?utf-8?B?ODBHRXVwVkMyUGUrYXJOdTdiRERScmJpM1YxRVFnS24xendEbHJ0WXNyMUNB?=
 =?utf-8?B?aU9xRnhRMm9PL2ZoaXlCWW9CTWpySUJiWks2bzJZeXR4K2dUQ0NXZmVwTzNT?=
 =?utf-8?B?cXV1czN4MVBnQ0RCTnY4bkpFU2lYSzBzdTN4dVB0SGFNbTJrWVZuSWF5ZWE4?=
 =?utf-8?B?RUgvOE0zcmt0YTc0RzhTeFhvYkd3aXUyMTc1VkhsaFZ0bk84NG1IQzFZb0s5?=
 =?utf-8?B?NzAvQXArZ1RyZTdVU3Z1Wm41RVI5bjZ4Yno0TXBqdWtYL1o2bCs3RWp0SjhS?=
 =?utf-8?B?UVprcFhJSDlBMHRXVzBjOXlTQ1graFRBaHpvZlo4bk1Va0NyWC9TclJMajF2?=
 =?utf-8?B?RDVtOGw0YU90akcrbWZLMlFmbHl6WnduUFUydTVlaVVTOXViaHFEbGVST2ZH?=
 =?utf-8?B?TGphdXdKcjdFdnlJYXg5TUVlTTRCZS9STEd1dGZENWxrbzF6MFZlNDBJR2pL?=
 =?utf-8?B?WFprSEtYVXBrOXYxWk1WMEI0RG5UMVYzVGpMUUtNWVYwZ0loTENtR042REZE?=
 =?utf-8?B?aHkrcnRwR2loc2s2b3lGYVR4anJ2Z3dKV0tXOGVENTVkNlVmbExQeUVHdlVo?=
 =?utf-8?B?MkNNVWJJTVh5eTBzRnJJVStNa3lpMlErdkJScWwzeWw5MnJ1U1NXTTErclM4?=
 =?utf-8?B?d2tvZkFvS25xMnRWdHl4ZTF2UC82N3g0c3djb00wOUdIaFJ5RUN5RnJnQlJY?=
 =?utf-8?B?RFkwczBlN3ZrZFduSHBGQlcrazlKRjlWd2ZOTVpMNjhsYi8xbnNkNzdlR3M1?=
 =?utf-8?B?VStZNnNnWitMMWRJaWtaNG5UWnk5d2FNNlJ3THd2M1p5V1gwZ2p6blVKSDhj?=
 =?utf-8?B?R2c9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: e014c934-e23f-4c3d-f2e3-08da877e7749
X-MS-Exchange-CrossTenant-AuthSource: CY4PR11MB1862.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2022 16:17:26.4872
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 1i05c4qKiFqDcAamRNovdoqYHCRHFmH7a6xfRf7dce0yFglBawLvWLkSqXszQ8VlkYFnTVSsutVID1A2HTJtQEdPcGtVL0aQCNfinmphANw=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR11MB6305
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Babu,

On 8/25/2022 3:42 PM, Moger, Babu wrote:
> 
> On 8/23/2022 5:47 PM, Reinette Chatre wrote:
>> Hi Babu,
>>
>> On 8/22/2022 6:42 AM, Babu Moger wrote:
>>> Adds the new AMD feature X86_FEATURE_SMBA. With this feature, the QOS
>>> enforcement policies can be applied to external slow memory connected
>>> to the host. QOS enforcement is accomplished by assigning a Class Of
>>> Service (COS) to a processor and specifying allocations or limits for
>>> that COS for each resource to be allocated.
>>>
>>> This feature is identified by the CPUID Function 8000_0020_EBX_x0.
>>>
>>> CPUID Fn8000_0020_EBX_x0 AMD Bandwidth Enforcement Feature Identifiers (ECX=0)
>>> Bits    Field Name      Description
>>> 2       L3SBE           L3 external slow memory bandwidth enforcement
>>>
>>> Feature description is available in the specification, "AMD64 Technology Platform Quality
>>> of Service Extensions, Revision: 1.03 Publication # 56375 Revision: 1.03 Issue Date: February 2022".
>>>

(snip modified links)

>>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>>> Reviewed-by: Ingo Molnar <mingo@kernel.org>
>>> ---
>> resctrl currently supports "memory bandwidth allocation" and this series adds
>> "slow memory bandwidth allocation". Could you please provide more detail about
>> what the difference is between "MBA" and "SMBA"? It is clear that the implementation

> In this case the slow memory means memory attached to CXL device.

When you say "in this case", is there another case?

Should "Slow Memory Bandwidth Allocation" thus be considered to be "CXL.mem
Memory Bandwidth Allocation"? Why not call it "CXL(.mem?) Memory Bandwith
Allocation"?

I am not familiar with CXL so please correct me where I am
wrong. From what I understand CXL.mem is a protocol and devices that implement
it can have different memory types ... some faster than others. So, even if
SMBA supports "CXL.mem" devices, could a system have multiple CXL.mem devices,
some faster than others? Would all be configured the same with SMBA (they
would all be classified as "slow" and throttled the same)?

>> treats them as different resources, but both resources are associated with L3 cache
>> domains and (from what I understand) throttling always occurs at the CPU. Can both
>> types of memory resources thus be seen as downstream from L3 cache? How can

> Yes. that is correct. They are seen as downstream from L3.


>> a user know what memory is considered when configuring MBA and what memory is
>> considered when configuring SMBA? Additionally, I do find the term "slow" to be
> 
> This memory completely transparent to OS with little bit higher latency that regular main memory.

I do not think these devices are invisible to the OS though (after
reading Documentation/driver-api/cxl/memory-devices.rst and
Documentation/ABI/testing/sysfs-class-cxl).

Is there not a way to provide some more clarity to users on what
would be throttled? 

> 
> Yes. I know slow word is bit vague. I am not an expert of CXL. But i see that word slow is being used to refer the CXL memory to differentiate it from regular memory.

What is very vague to me is how a user is intended to use this feature.
Would the "SMBA" resource be available only when CXL.mem devices are present
on the system? Since this is a CPU feature it is unclear to me whether
presence of CXL.mem devices would be known at the time "SMBA" is enumerated.
Could the "SMBA" resource thus exist without memory to throttle?

>> vague as a way to distinguish between different memory types. What is the
>> definition of "slow"? Would all "slow" memory on the system support SMBA?
> 
> Yes. All the slow memory in the system can support SMBA.
> 

How does a user know which memory on the system is "slow memory"?

It remains unclear to me how a user is intended to use this feature.

How will a user know which devices/memory (if any) are being
throttled by "SMBA"?

Reinette