Received-SPF: pass (google.com: domain of linux-kernel+bounces-125175-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <cf59f587-9ca2-4f0d-b412-69b559acbabb@intel.com>
Date: Fri, 29 Mar 2024 09:37:12 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH] Documentation/x86: Document resctrl bandwidth control
 units are MiB
To: Tony Luck <tony.luck@intel.com>
CC: <linux-doc@vger.kernel.org>, Fenghua Yu <fenghua.yu@intel.com>, "Peter
 Newman" <peternewman@google.com>, James Morse <james.morse@arm.com>, "Babu
 Moger" <babu.moger@amd.com>, Drew Fustini <dfustini@baylibre.com>,
	<x86@kernel.org>, <linux-kernel@vger.kernel.org>, <patches@lists.linux.dev>
References: <20240322182016.196544-1-tony.luck@intel.com>
 <56a93ec2-dc01-49be-b917-5134f5794062@intel.com>
 <Zgbe2FFwyHMmmsyM@agluck-desk3>
Content-Language: en-US
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <Zgbe2FFwyHMmmsyM@agluck-desk3>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?KzZYUDdSMVJXNUpWSXU1b0JrOXhRckNXcVllRGM1UnhGVFpQOUhwQlpBckRW?=
 =?utf-8?B?R0VZRnNKR3lnaTBsNVJwYlRvS25EUXBLK3VmRVQ4SWZyTjh2Uno4cVY3YkZl?=
 =?utf-8?B?MlNVVXZITHE4OFU4U2xET21qZ0wyRFU2U29aMzVIME9oNHhHVzIvd2o1blhx?=
 =?utf-8?B?SThNb01CTnZnVHVPU0JYN0Z0cDVKMzJnbS9pWGVwOFdldUh1a2lOSzJXOUkv?=
 =?utf-8?B?R1hXRjdqR3lWbnNqNWdCWnlBZlVGN3AyR1FDVWJTOWxhODgrdHdRSURIOGll?=
 =?utf-8?B?ZVBOTmwvTlFPUDlwbnR6VjZ3bnlZUG10NWM2SmRENXl2MEtyODJheEt0YnF2?=
 =?utf-8?B?TnFTZzlWZGxaZlFpWmlRZTdDMFFpbnk3OHJxN1NsZVZ4VE8ycDQyaHJXdW1P?=
 =?utf-8?B?NkdGMGxZRkhGNC9qWGtoazVVRmxLYjVsb3JHKzVmYjlsb3VrcU9WckFJdjNq?=
 =?utf-8?B?cDZGczBUOVZucFZjTlpQUklReWdlaFFqaXNPNk1vNTVwQi9hRW9yd1l6bDly?=
 =?utf-8?B?N1VFNXRWcWZ6RlNMM0VTcFdVUHJ6c2NUUXRhcGhXNnNIaCt1R3RxUkt2dzlH?=
 =?utf-8?B?M1VUZDRoQ2JYZ0VMREJhK3d1U09LZEpmS3FUVHdrV2hJNUwxTThIRWdXcEpN?=
 =?utf-8?B?b1h3Y2ptR0t5S3dqdlArUGFmRCtzRUdEaE1VRlJLZGlCUlZvSWYxTlNMaFVt?=
 =?utf-8?B?KzdOOHV1L2RRa0RFVVFPR3UweVYxdk5VRnlML0VNd2R6ZzFCcEJpdm5BNklY?=
 =?utf-8?B?eGN1ZjltT1VRUzkydlp4bFhISDV5ZFdDeExSWnZGWHRWbzZTT05Bb2UvZjZJ?=
 =?utf-8?B?L0IwVjlRNG8zZWFkVGFKWWFkeElEci9paC9vT2RndHlnSTYrcTRVMURVa0pk?=
 =?utf-8?B?TWJyQ2ZWZ1M5Y1FuN2N2KzY1enRvRlZPV3ZoUlNDT1FRVWFJRmt3ZFdSYTBF?=
 =?utf-8?B?NWUvcmd5bWdjbXpybG1PcHliZGZwLzFjcE0xaEkrall3eHBoc1lpVEU0akV2?=
 =?utf-8?B?eFp0S0pqcWRDR3laNlBSME03cEFhdG9WUXNmcVNhSFEzSEVkTm1UVUF4cTkz?=
 =?utf-8?B?L25zeVNwUUZrRU1QcDVkL1JxV2gySEphTngydkxIZ3NGMVJzOW5jdnVSUkUw?=
 =?utf-8?B?NUVtMGdZWDgzZnB1enJzTmNlRnRFYjFEeWNRNGtwVm90cVhySHZrMis3SVl1?=
 =?utf-8?B?MkFOOXlCMFRhRVdTRWZtTlFFV3JtYWpOcVgxZWkyaG8zM2tpM0FkT0syT0NI?=
 =?utf-8?B?WXU5citQeFhOdmh5NzdzdU9MUFo5ZE5IanhDdXRyeFNRTVRYTVNpS3RzUi9D?=
 =?utf-8?B?SFVoaC9GMktWVGFQRzlReUg1WjcyRVhaa3VaL3psZXlHK1dXVHpIQ2YraDd2?=
 =?utf-8?B?K0YwcjB0ZHRJMUpPSUxhcnE3T3dxTXlidUZRMlZSUEV3Zzc1RWFKMDlmelV0?=
 =?utf-8?B?cWRCVWxramF5ck4zWkJwU25jdmxSTG1sMUpHUjFRYmpZZHJDcjRmemI0OXcz?=
 =?utf-8?B?THQ4OThzUVFkUmdEU2hzQ1lFT0VVcUNBS1lyMGVxU3o4UmNhZ1Avb3Rac0VD?=
 =?utf-8?B?S2FzbFNGTDJmY2NBYnBZaGJWTlNmcStmR3U1dzJ3NlE0MTQ1djlwbHZnai9q?=
 =?utf-8?B?S0N3aVVDNk84TG44UG1Panp4OFdOS3ZVSkw2QUNHNUMxV0ZnNHpocnd4aU9Z?=
 =?utf-8?B?aTNmRVlkM1FGcUZ6SzNUWnV4NTNGS3pDeUFXMVZQS093N1YvcmZGK0I3Rk45?=
 =?utf-8?B?SldoZ3dWcVk2dXZZczF0bVEyMEJoSkJFY3VLYkhnbDJZQkc0UTNNM29pa2x6?=
 =?utf-8?B?eEIxaVFYNEF3akE1dEkvd3B6cXczZ092eFY4NmpQY1Q4YXYxK3dUZU1GcEVm?=
 =?utf-8?B?YUdjME8raHFVSU5sQ3EwczR6aVUyU3NEQlRNTlJmeWl4TWhMTzNFdkdSdjRF?=
 =?utf-8?B?Nk9LQnh5TS96Y1F3YVoyUjB5Q2tUOGx3NU9MWUR4eXJCVVdxek1USitTMUhQ?=
 =?utf-8?B?Z01vSmZqendiRDNVclpPSmQrOXpuRUFsWHZhVHR2NE5YVDRqYUpDME1qT25L?=
 =?utf-8?B?bWxwQmF1MldHRnNTZ3F2N21CSUZFL2NUb3graHNpSXNhcUtwSjBGNDBJeDdR?=
 =?utf-8?B?bzVxa2JnNUx1SjRBV3haaTlNdkxxbktZVW5FZ3NUZmhiWklOa1N3YTNKUU1N?=
 =?utf-8?B?Rmc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: c0214e82-ec9a-4dd5-b2d8-08dc500e7d68
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Mar 2024 16:37:14.4824
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: oYnZnSHBZjQXoipLZsQyfjCBQ37OJ5c2TlpJZqeIH45qswqF7o7ac5HUaGWVZNznqxD0bV91GhCvkNcwlsycT58JuFpqP8aOgOiUT/dfP+M=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR11MB7991
X-OriginatorOrg: intel.com

Hi Tony,

On 3/29/2024 8:31 AM, Tony Luck wrote:
> On Thu, Mar 28, 2024 at 06:01:33PM -0700, Reinette Chatre wrote:
>> Hi Tony,
>>
>> On 3/22/2024 11:20 AM, Tony Luck wrote:
>>> The memory bandwidth software controller uses 2^20 units rather than
>>> 10^6. See mbm_bw_count() which computes bandwidth using the "SZ_1M"
>>> Linux define for 0x00100000.
>>>
>>> Update the documentation to use MiB when describing this feature.
>>> It's too late to fix the mount option "mba_MBps" as that is now an
>>> established user interface.
>>
>> I see that this is merged already but I do not think this is correct.
> 
> I was surprised that Ingo merged it without giving folks a chance to
> comment.
> 
>> Shouldn't the implementation be fixed instead? Looking at the implementation
>> the intent appears to be clear that the goal is to have bandwidth be
>> MBps .... that is when looking from documentation to the define
>> (MBA_MAX_MBPS) to the comments of the function you reference above
>> mbm_bw_count(). For example, "...and delta bandwidth in MBps ..."
>> and "...maintain values in MBps..."
> 
> Difficult to be sure of intent. But in general when people talk about
> "megabytes" in the context of memory they mean 2^20. Storage capacity
> on computers was originally in 2^20 units until the marketing teams
> at disk drive manufacturers realized they could print numbers 4.8% bigger
> on their products by using SI unit 10^6 Mega prefix (rising to 7.3% with
> Giga and 10% with Tera).

This is not so obvious to me. I hear what you are saying about storage
capacity but the topic here is memory bandwidth and here I find the custom
to be that MB/s means 10^6 bytes per second. That is looking from how DDR
bandwidth is documented to how benchmarks like
https://github.com/intel/memory-bandwidth-benchmarks report the data, to
what wikipedia says in https://en.wikipedia.org/wiki/Memory_bandwidth.

I also took a sample of what the perf side of things may look like
and, for example, when looking at;
tools/perf/pmu-events/arch/x86/sapphirerapids/spr-metrics.json
I understand that the custom for bandwidth is MB/s. For example:

    {
        "BriefDescription": "DDR memory read bandwidth (MB/sec)",
        "MetricExpr": "UNC_M_CAS_COUNT.RD * 64 / 1e6 / duration_time",
        "MetricName": "memory_bandwidth_read",
        "ScaleUnit": "1MB/s"
    },

> 
> It is clear that the code uses 2^20 as it converts from bytes using
> a right shift by 20.

Right. This appears to be a bug.

> 
> Fixing the code would change the legacy API. Folks with a schemata
> file that sets a limit of 5000 MB/s would find their applications
> throttled by an addtional 4.8% on upgrading to a kernel with this
> "fix".

Wouldn't this be the right thing to do though? If the user sets a limit
of 5000MB/s then I believe the expectation is that it should not exceed
that bandwidth.

>> To me this change creates significant confusion since it now contradicts
>> with the source code and comments I reference above. Not to mention the
>> discrepancy with user documentation.
>>
>> If you believe that this should be MiB then should the
>> source and comments not also be changed to reflect that? Or alternatively,
>> why not just fix mbm_bw_count() to support the documentation and what
>> it appears to be intended to do. If users have been using the interface
>> expecting MBps then this seems more like a needed bugfix than 
>> a needed documentation change.
> 
> I agree that the comments need to be fixed. I will spin up a patch.

It is not just the comments but the constants and variables also. All point
to MBps.

Also note that there remain several examples in the resctrl doc (see
section "Memory bandwidth Allocation and monitoring") that continue to
use GB/s. Are you really proposing that all should be changed to GiB/s?

In addition to all above this change would fragment resctrl even more
with AMD's memory bandwidth control clearly being in GB/s.

I continue to find this change to be the source of confusion and diverges
resctrl from the custom.

Reinette