Received-SPF: pass (google.com: domain of linux-kernel+bounces-127146-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <d6a649fd-0bd3-4777-acb3-2b9362131796@intel.com>
Date: Mon, 1 Apr 2024 15:44:03 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH] Documentation/x86: Document resctrl bandwidth control
 units are MiB
From: Reinette Chatre <reinette.chatre@intel.com>
To: Tony Luck <tony.luck@intel.com>
CC: <linux-doc@vger.kernel.org>, Fenghua Yu <fenghua.yu@intel.com>, "Peter
 Newman" <peternewman@google.com>, James Morse <james.morse@arm.com>, "Babu
 Moger" <babu.moger@amd.com>, Drew Fustini <dfustini@baylibre.com>,
	<x86@kernel.org>, <linux-kernel@vger.kernel.org>, <patches@lists.linux.dev>
References: <20240322182016.196544-1-tony.luck@intel.com>
 <56a93ec2-dc01-49be-b917-5134f5794062@intel.com>
 <Zgbe2FFwyHMmmsyM@agluck-desk3>
 <cf59f587-9ca2-4f0d-b412-69b559acbabb@intel.com>
Content-Language: en-US
In-Reply-To: <cf59f587-9ca2-4f0d-b412-69b559acbabb@intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?a1lGeFFCcmdNZTdITW1pTFJSRC9QUmcyNEF5elJKbVZId2FSaTYvK3BNVFEy?=
 =?utf-8?B?NnZjWFB0N3pnZjRMUWdMRXBDRmpURVpoRDQ2b1dMc2JoWHpOcUhkdEIrUVl0?=
 =?utf-8?B?NEFQT21qUG1HVzh0cm1RTStwQXhzV05qdjV3dElvTU1wMlVra0pZZURKZk9T?=
 =?utf-8?B?d3NrRUt4VmRNSllEUDNXTFhMNUUyWnZNVGpqYm05YnZKWW5YaXdmQXlWV29B?=
 =?utf-8?B?OW5lYll0VVh3aEtFVGFSNHBzUE1jOHptVFRPc01aMVRSSURYcEh0Z1YxbTYr?=
 =?utf-8?B?UG1RTjREM0NodXJJRkFvSkptOVo2ZzNHbjBFTGFiY0JzZzFsUGUyYlZKRU5R?=
 =?utf-8?B?S2g5RUpOSVAxSWFKTGpzZ04wSzBqREFGUzJKdjBEVmMzdzhOK3ZjbGtTSUs2?=
 =?utf-8?B?SDNwbllza0hJR0RKWFhxQUdHTzRSTWZUWXFLdDBuK0lRcVJyTmVpWjhmL3dq?=
 =?utf-8?B?Y0F1WGJOY1dHYWN2cFlIVGI5MHZiTDZuakozeHZ0ZXBUdHY3cUd4SzEydjVB?=
 =?utf-8?B?bk1UbCtPWlN4dnZBQW5jUERqeXRNaS9HWGFkcnZXb1psc3IzR0lNLzVzbWpL?=
 =?utf-8?B?dXBzMGl0Q2VKcXpydnl2a0FKTEVWd3pvMC90MmllSXdnQjBMUEZLV3l6VEJP?=
 =?utf-8?B?MUxZbk01QnFUYjgrZjJNck1HZ2dsTGFOTnhRYm1PS2hDcUVoR2JtQkdTcUNU?=
 =?utf-8?B?MmwvRVdJTS9qSm5lYll3ems1eW1LK2ZPeEI1MXNhZ21YdzZxMTlJNWVzQkVK?=
 =?utf-8?B?TTVoLy9CRlk4a2xIV3h2Q2VZeUdVM05ka25oc3ZBR25wWWVya1hPTnpLaEwv?=
 =?utf-8?B?R1VZNmc3dWtoVEZzK3JOckFiRWdoc2JzS2ZheVhLNHZ3aER4aWV4Z0c4bzBN?=
 =?utf-8?B?WnV2YmNpdHQwZitOS20vTVNkSkdmQm5rdjBRbHYwU2Z2L2dhNzRRWEUyUzZ0?=
 =?utf-8?B?N1p3SHJJMnZFN1NwaUFyZ3NzTDFQMFRjcndpTXd0Y1drTHdvU2RlN3NTeTJS?=
 =?utf-8?B?NEl3RmFFVmdxNUFFWTVGc3M0UVl0Ukh3WDF0NUgwS1crWDFWOG04bkRhWnIv?=
 =?utf-8?B?STNVbUR5OEhJMmhMVEJtamNZTTZST3A1K3ZpK1ZrcGdRU0ZuTzJoY20xVTNt?=
 =?utf-8?B?WDZQWVA0cDNFOFNDdEIyaUJnTmJENzdCUS8rdHVaeUNXU3BKRldyNFR5K3hW?=
 =?utf-8?B?bmtnMU9wdUlONTFiVFlLWTBDeC9LWUtKRE9JWkxkZC95NThSUnF6bUM5aUVq?=
 =?utf-8?B?cjIyY0VLMXhSUjhqdXB0MEpkUU5GVEt6TTIzZnAzT2cveGtINGdzQ2ZqMGw0?=
 =?utf-8?B?MUVmYlJaTnZjdzBiaVNxU24rOCtoY0E2SFMvUjhUZlVpN3FOTitEL3VMWnlT?=
 =?utf-8?B?WE92QkpPcUdmOWlVN2dFZ0wrclRQbjgrcFhJOEFoZGFaS0txRW9sdE4yQUVV?=
 =?utf-8?B?K0FpbmhkR2g3VkplTk1YckFYMmdnVHU4ajczU1dGeVBYQ0VnaFd4d2kzMDZX?=
 =?utf-8?B?enZUMHlCNjNWODBGYjI2Z3hGRWtNSUZJbHhPSmthd0VvQnNGNDVOVWdod2Ey?=
 =?utf-8?B?VndTTmx0SVVtUmZBOGlpTmc2K2xRcUhDMXZpV3dQQUlXRkdoNGxaSW5tVTFr?=
 =?utf-8?B?NThYbjJPNjFLdEEvQTlMZjN5NzY4M0pQVnl4NlVBSVVHZXBJZXFONlNOUnI0?=
 =?utf-8?B?Q3B2Y1hGRHgzdUhmdTJCcXhyclg4M3IyaHM5NHdFSjNJcGJEa1h0NlhuN1VO?=
 =?utf-8?B?bmFNNDE5K2gvRUo2MkdYQjNaNjVMYVVqUVJTMWhwaXRRM1F2MWQ1MjM5Rmcy?=
 =?utf-8?B?MWs4a0tKZzFXano1N2dsZjIxOTE4ak5zZldMQmxWNC81eUtaeEdVUnQ2RHFn?=
 =?utf-8?B?bzdpclZwVUFoWXJ0YWZtYUZDVGZBMXIzc3lSYWZzTmRBSDEyMk05bmN1NW5T?=
 =?utf-8?B?WHJLZ3dGQmkxTElnWWhKZUNxdTVwMEZUbjFXUUR0ZklRaUZZYlRobzd0RDc5?=
 =?utf-8?B?UWh0bzhuZnNyYkVuSWpacTRRU0J2L2tnYm93TnlSYnBuSWhEZDRVT1B0NCto?=
 =?utf-8?B?OEl0eVUyTXkvY0IxZC80bjg0RTEyVjd1T0EyMEN2QmRVS2QzVjNEMDYzNVBp?=
 =?utf-8?B?UVg5L04yaHVHNEVHVDBYV21PS2RaaTQyZ25ncTcrN2Mvc3ozaW9SSGhJNXNL?=
 =?utf-8?B?eGc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 24ab1d05-b9da-4085-0bce-08dc529d3c79
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Apr 2024 22:44:05.9501
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 1wZTkO7qPbiD+iSyoSzqyPAy8FqHK93XJrpjAg00QtC8WmKgiyb1aKjb9PeP2a56nUCfXudJcGVyj83XkYUKeee3M72wVp4j045w8DPus2E=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA2PR11MB4796
X-OriginatorOrg: intel.com

Hi Tony,

On 3/29/2024 9:37 AM, Reinette Chatre wrote:
> On 3/29/2024 8:31 AM, Tony Luck wrote:
>> On Thu, Mar 28, 2024 at 06:01:33PM -0700, Reinette Chatre wrote:
>>> On 3/22/2024 11:20 AM, Tony Luck wrote:
>>>> The memory bandwidth software controller uses 2^20 units rather than
>>>> 10^6. See mbm_bw_count() which computes bandwidth using the "SZ_1M"
>>>> Linux define for 0x00100000.
>>>>
>>>> Update the documentation to use MiB when describing this feature.
>>>> It's too late to fix the mount option "mba_MBps" as that is now an
>>>> established user interface.
>>>
>>> I see that this is merged already but I do not think this is correct.
>>
>> I was surprised that Ingo merged it without giving folks a chance to
>> comment.
>>
>>> Shouldn't the implementation be fixed instead? Looking at the implementation
>>> the intent appears to be clear that the goal is to have bandwidth be
>>> MBps .... that is when looking from documentation to the define
>>> (MBA_MAX_MBPS) to the comments of the function you reference above
>>> mbm_bw_count(). For example, "...and delta bandwidth in MBps ..."
>>> and "...maintain values in MBps..."
>>
>> Difficult to be sure of intent. But in general when people talk about
>> "megabytes" in the context of memory they mean 2^20. Storage capacity
>> on computers was originally in 2^20 units until the marketing teams
>> at disk drive manufacturers realized they could print numbers 4.8% bigger
>> on their products by using SI unit 10^6 Mega prefix (rising to 7.3% with
>> Giga and 10% with Tera).
> 
> This is not so obvious to me. I hear what you are saying about storage
> capacity but the topic here is memory bandwidth and here I find the custom
> to be that MB/s means 10^6 bytes per second. That is looking from how DDR
> bandwidth is documented to how benchmarks like
> https://github.com/intel/memory-bandwidth-benchmarks report the data, to
> what wikipedia says in https://en.wikipedia.org/wiki/Memory_bandwidth.
> 
> I also took a sample of what the perf side of things may look like
> and, for example, when looking at;
> tools/perf/pmu-events/arch/x86/sapphirerapids/spr-metrics.json
> I understand that the custom for bandwidth is MB/s. For example:
> 
>     {
>         "BriefDescription": "DDR memory read bandwidth (MB/sec)",
>         "MetricExpr": "UNC_M_CAS_COUNT.RD * 64 / 1e6 / duration_time",
>         "MetricName": "memory_bandwidth_read",
>         "ScaleUnit": "1MB/s"
>     },
> 

(Thanks to Kan Liang for explaining this to me.)

As an update on this, the perf side does not seem to be as consistent as
I first interpreted it to be. There appears to be a "kernel side" and
"user side" related to memory bandwidth data.

On the kernel side, users can refer directly to:
/sys/bus/event_source/devices/uncore_imc_*/events to read the
UNC_M_CAS_COUNT.RD and UNC_M_CAS_COUNT.WR data and this appears to
be intended to be consumed as MiB/s as per:
$ /sys/bus/event_source/devices/uncore_imc_0/events/cas_count_read.unit
MiB

On the user side, using perf from userspace the metrics are obtained
from the relevant json file as quoted above, and thus when using perf
from the command line the data is in MB/sec, for example:

$ perf list
[SNIP]
  llc_miss_local_memory_bandwidth_read
       [Bandwidth (MB/sec) of read requests that miss the last level cache (LLC) and go to local memory]
  llc_miss_local_memory_bandwidth_write
       [Bandwidth (MB/sec) of write requests that miss the last level cache (LLC) and go to local memory]
  llc_miss_remote_memory_bandwidth_read
       [Bandwidth (MB/sec) of read requests that miss the last level cache (LLC) and go to remote memory]
  llc_miss_remote_memory_bandwidth_write
       [Bandwidth (MB/sec) of write requests that miss the last level cache (LLC) and go to remote memory]
  loads_per_instr
       [The ratio of number of completed memory load instructions to the total number completed instructions]
  memory_bandwidth_read
       [DDR memory read bandwidth (MB/sec)]
  memory_bandwidth_total
       [DDR memory bandwidth (MB/sec)]
  memory_bandwidth_write
       [DDR memory write bandwidth (MB/sec)]
[SNIP]


It appears that there is no custom here and it may just be somebody's preference?

Reinette