Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <59259155-6c51-a750-216a-ebbb0702d200@intel.com>
Date:   Tue, 30 Aug 2022 09:39:37 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.2.0
Subject: Re: [PATCH v3 02/10] x86/cpufeatures: Add Slow Memory Bandwidth
 Allocation feature flag
To:     Babu Moger <babu.moger@amd.com>
CC:     <bagasdotme@gmail.com>, <bp@alien8.de>, <corbet@lwn.net>,
        <dave.hansen@linux.intel.com>, <eranian@google.com>,
        <fenghua.yu@intel.com>, <hpa@zytor.com>,
        <linux-doc@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        <mingo@redhat.com>, <tglx@linutronix.de>, <tony.luck@intel.com>,
        <x86@kernel.org>
References: <9965edff-c558-2962-4aad-3342480026bc@intel.com>
 <20220829232554.53763-1-babu.moger@amd.com>
Content-Language: en-US
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20220829232554.53763-1-babu.moger@amd.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?NHF4TTd2MlZ4UFRqTEx4NnRBZ3MzajZqY1JicnQ3bHZpRHhFZEFCWFNEMVpW?=
 =?utf-8?B?NXRsR0ZLVWprVEVBQWg1bmFEVXhUaloyYzJkY2lOQW12VTg5SkdmalBnUUtK?=
 =?utf-8?B?VDJXTElzVXdxUjFocXZPcDUyc3c3YWVuK0FPWk1vVUlEUHdKVVJYdC9Lcml2?=
 =?utf-8?B?dHZYTWVoL2o3R2crajJrVmFMZGthVUZ1WDBUNWZMV1dkdmlYVHEwaXExNG5S?=
 =?utf-8?B?bWcxbDZNb1RFSkdUeVUrU1ZLZXh5T1ovWkRKVzFVVFNYMENORnlENUdtQkFU?=
 =?utf-8?B?b1c5VDRncTRJT2V0Z2V2YldvTmttUzdrWDlCN1VCWHdNdkxNdTZ3TWExZyt2?=
 =?utf-8?B?a2s0Vzh2ZWtmRzY4RkFqWlV1RHEwT3dqMDBnVXJ5MmFtY3hOM0VjMSszV2ZI?=
 =?utf-8?B?b3E2dnVwd3hmREVLbTBabjVFVzNSL3VTbG5tOXN4WmxxVTUzWXd3Zm42bFJM?=
 =?utf-8?B?RlJaUWZleW1kWkJZVStweDFaNVg1UE8wTVNSVlgxaXUzaTRHbHVZZUwrV0hv?=
 =?utf-8?B?Smd5ZnVydGJNMnB3SFVXaSt0WDU4Y2VTRzlXR0dYMVhNWUd2ZFhLL3RMSk1z?=
 =?utf-8?B?czhTUlIvOXhPaFNtZWdxSUZ3dVZGYlV1cFJYYUVDZFVaaGtCaTN0Q3d5d1Vw?=
 =?utf-8?B?L1ZGS0tzSktvUm9pNUU4SHBwUWVGNFA2QkM3M3JrcllpTmpRWllNTUIxUTdx?=
 =?utf-8?B?YVpsK1J4OHVObjNnbVA2MWxYVFZpc0FlMGJQQTY4VFpRTG9XRGVDcXlGM3lN?=
 =?utf-8?B?TXBPRTh2T3hsalZhQ2N4TUZYNjc4ZDMrK0xWc0JoZHhmcjlZaE1YMmd2KzdJ?=
 =?utf-8?B?TnZpOEZodGlzbmZTMU9uUmQzbmNCVFhkZFJTVkRZVTJUYjZYd1RwQUlyN0ho?=
 =?utf-8?B?Uk83aWlHU29WSUhNUHMvWWlZRENUVWlHTHpkOG9kRGJOQVZocjIxODUyS09l?=
 =?utf-8?B?bzJDT0ZQT0c1ZjFzVEliN3pkUVlYVHpYeWR6ZjlDQmk1VXgydmF6azA1RytE?=
 =?utf-8?B?NmxmS3VZM3I0T1JoUkE0b2F4TWVEb3lVUWwxMUxSRld1MmlUOHBMTXN0MWhp?=
 =?utf-8?B?b1I2Zll4Yy9nWDEyQWN3N2t5WUNuYUZiQkJQcmlMZ2J5ZHlndjZKQngxVVor?=
 =?utf-8?B?VXhzNzJCVWdPbElkNmU3K3lmTDVzU0JlUldzdjBWQzFBMjlSbDg5TUxWWjZ4?=
 =?utf-8?B?M3RCMVA2MTRlZ0ZKelpVTFRFZ0RSTDlpbElNc0UyZzNLM2FQSzZWNk0zLzNV?=
 =?utf-8?B?OUxpblNjSEgwMGdWUVpYTVVNU3RwL0p5aUZoVmRRUGFkZjJySWM5MWoySGlD?=
 =?utf-8?B?b0ZzbTUvNWVRVXdaRW1YM0FsRTJpTUJZZTI2VUJrUm44MUw5ZU1xdXV1ZHEv?=
 =?utf-8?B?ais1U0plMUs0RHJiaitCck92bGNKWFZmOXp1OGh5NDhYRU5hUStyaURqdEk3?=
 =?utf-8?B?L0E0dEtJRk43YTdyY3FzdFM5SUJFcU92THpWMFpSN3dSSDd6eFN4YThpcmdM?=
 =?utf-8?B?aTFidjVDNTZJdzRIcmFaT1pSYkR3YjRCVldoUFVrSWVDY3JWNDZFelhPRlNq?=
 =?utf-8?B?UDM3SnMwMjk5TkNEMFJVWWJ4OEpMajRKOUlPaGdCN21wc0hqb0o4NlpyTkFT?=
 =?utf-8?B?dyt0ZmV6eldxb0dMOGVwU1ZqdmJBS005L0FpbHJJbUUvVm9OOUNiaVR3bVpF?=
 =?utf-8?B?cXdaMzdrSHBiR0NPa0NPUlFzOVRwNjdWRUtDeFloaTdWbGMxK3l0aDhPOGdT?=
 =?utf-8?B?cjZJS2tRMTBteU9WSnArTXAwWlJNZ1VwdnAwKzllZmhNZUVuaXBOazkrQ3dx?=
 =?utf-8?B?c2JkVzdDODUxQnR2M1Z3bzBCc0dVWHVxRlRtcnhySFNBOXdQWVZDSFdGVDJN?=
 =?utf-8?B?dndyZkgxTStIYkh5NDBSTDA1YnpBT0ppdXVKOW0vVE01MUcyWW1kc0RKUUkw?=
 =?utf-8?B?cTJwaUh0UHRmSmIxYkdBTU45aHp3dEFaWE1pZzVMMlJtZkFGaURvS3Rxd0pu?=
 =?utf-8?B?TkpHelV2RW54bTVJSUpNS1pNSXZuUExDeG54UFFQQkw4M0hidk9KdVFaVC91?=
 =?utf-8?B?dFlFajh6SkVEc1dpeHZXMU9QWmdBdmhCOXJFajVTeThjR3hXalZvZGhnMVh2?=
 =?utf-8?B?U2V1TWt0czB0SFBHdlAxOTdsQzIxRWR5QXNLK2pQNVBuMG8velRkeEY0ZTVP?=
 =?utf-8?B?OWc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: db89522b-32c7-4892-1bda-08da8aa63c0c
X-MS-Exchange-CrossTenant-AuthSource: CY4PR11MB1862.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Aug 2022 16:39:40.5008
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: jkgHxT+ENw/3tpdlcy/Y7ec1p1JnvQnTYNyyquRD6hsKfku+rEC8vYJhHPPpVli6t9RyZRKYgsCXpMMYZtM+uAyTCgwUxR50n+tECRdRO58=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR11MB0080
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SORTED_RECIPS,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Babu,

On 8/29/2022 4:25 PM, Babu Moger wrote:
> Hi Reinette,
>    Some reason this thread did not land in my mailbox. Replying using git sendmail to the thread 
> 
>> (snip modified links)
> 
> Link: https://www.amd.com/en/support/tech-docs/amd64-technology-platform-quality-service-extensions
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
> 
>> When you say "in this case", is there another case?
> 
> There is no other interface. It is only CXL memory device.
> 
>>
>> Should "Slow Memory Bandwidth Allocation" thus be considered to be "CXL.mem
>> Memory Bandwidth Allocation"? Why not call it "CXL(.mem?) Memory Bandwith
>> Allocation"?
> 
> Checked with our team here. The currently only supported slow memory is CXL.mem
> device. As for the naming, the "slow" memory landscape is still evolving.
> While CXL.mem is the only known type supported right now. The specs says
> "Slow Memory Bandwidth Allocation". So, we would prefer to keep it that way.

If you prefer to keep "Slow Memory Bandwidth Allocation" then please also
provide clear information to the user on what is managed via "Memory Bandwidth
Allocation" and what is managed via "Slow Memory Bandwidth Allocation". This
could be in the documentation.

>> I am not familiar with CXL so please correct me where I am
>> wrong. From what I understand CXL.mem is a protocol and devices that implement
>> it can have different memory types ... some faster than others. So, even if
>> SMBA supports "CXL.mem" devices, could a system have multiple CXL.mem devices,
>> some faster than others? Would all be configured the same with SMBA (they
>> would all be classified as "slow" and throttled the same)?
> 
> I have not tested the multiple devices with different memory speeds here.
> But checking with team here says it should just work the same way. It appears
> that the throttling logic groups all the slow sources together and applies
> the limit on them as a whole.

"the throttling logic groups all the slow sources together and applies
the limit on them as a whole.". This is valuable content for
the documentation about this feature. Could the changes to
Documentation/x86/resctrl.rst be updated to include a paragraph
describing SMBA and what is (or is not) considered a "slow resource"? 

>> I do not think these devices are invisible to the OS though (after
>> reading Documentation/driver-api/cxl/memory-devices.rst and
>> Documentation/ABI/testing/sysfs-class-cxl).
>>
>> Is there not a way to provide some more clarity to users on what
>> would be throttled? 
>>

I repeat the question you snipped from my email (please don't do that). Could
you please answer it?:
Would the "SMBA" resource be available only when CXL.mem devices are present
on the system? Since this is a CPU feature it is unclear to me whether
presence of CXL.mem devices would be known at the time "SMBA" is enumerated.
Could the "SMBA" resource thus exist without memory to throttle?

>> How does a user know which memory on the system is "slow memory"?
>>
>> It remains unclear to me how a user is intended to use this feature.
>>
>> How will a user know which devices/memory (if any) are being
>> throttled by "SMBA"?
>>
> This is a new technology. I am still learning. 
> 
> Currently, I have tested with CXL 1.1 type of device. CXL 1.1 uses a simple
> topology structure of direct attachment between host (such as a CPU or GPU)
> and CXL device.
> 
> #numactl -H
> available: 2 nodes (0-1)
> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
> node 0 size: 63678 MB
> node 0 free: 59542 MB
> node 1 cpus:             (CPU list is emply. Node 1 have CXL memory)
> node 1 size: 16122 MB    (There is 16GB CXL memory) 
> node 1 free: 15627 MB    
> node distances:
> node   0   1
>   0:  10  50
>   1:  50  10
> 
> The cpu-cxl node distance is greater than cpu-to-cpu distances.
> 
> We can also verify using lsmem
>  
> #lsmem --output RANGE,SIZE,STATE,NODE,ZONES,BLOCK
> RANGE                                 SIZE  STATE NODE  ZONES BLOCK
> 0x0000000000000000-0x000000007fffffff   2G online    0   None     0
> 0x0000000080000000-0x00000000ffffffff   2G online    0  DMA32     1
> 0x0000000100000000-0x0000000fffffffff  60G online    0 Normal  2-31
> 0x0000001000000000-0x000000107fffffff   2G online    0   None    32
> 0x0000001080000000-0x000000147fffffff  16G online    1 Normal 33-40
> 
> Memory block size:         2G
> Total online memory:      82G
> Total offline memory:      0B
> 
> 
> We can also verify using ACPI SRAT table and memory maps.

I think that adding (in general terms) that "SMBA throttles CXL.mem
devices" to Documentation/x86/resctrl.rst may be sufficient for
a user to understand what will be throttled without needing to go into
details about CXL device discovery. 

Reinette