Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <c4560cfe-f116-9f9d-1e60-b7ee932fb4b6@intel.com>
Date:   Tue, 4 Oct 2022 09:15:21 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.3.1
Subject: Re: [PATCH v5 12/12] Documentation/x86: Update resctrl_ui.rst for new
 features
Content-Language: en-US
To:     <babu.moger@amd.com>, <corbet@lwn.net>, <tglx@linutronix.de>,
        <mingo@redhat.com>, <bp@alien8.de>
CC:     <fenghua.yu@intel.com>, <dave.hansen@linux.intel.com>,
        <x86@kernel.org>, <hpa@zytor.com>, <paulmck@kernel.org>,
        <akpm@linux-foundation.org>, <quic_neeraju@quicinc.com>,
        <rdunlap@infradead.org>, <damien.lemoal@opensource.wdc.com>,
        <songmuchun@bytedance.com>, <peterz@infradead.org>,
        <jpoimboe@kernel.org>, <pbonzini@redhat.com>,
        <chang.seok.bae@intel.com>, <pawan.kumar.gupta@linux.intel.com>,
        <jmattson@google.com>, <daniel.sneddon@linux.intel.com>,
        <sandipan.das@amd.com>, <tony.luck@intel.com>,
        <james.morse@arm.com>, <linux-doc@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <bagasdotme@gmail.com>,
        <eranian@google.com>
References: <166431016617.373387.1968875281081252467.stgit@bmoger-ubuntu>
 <166431042060.373387.8454967155095795574.stgit@bmoger-ubuntu>
 <71ca01ad-e563-3ac3-815d-0bbc3814ed6f@intel.com>
 <05f0ff0c-b328-46ac-c1fa-7aac09fbb9bc@amd.com>
 <39c3bf58-052d-fbab-3bbd-029f5507ff23@intel.com>
 <d60a73cd-3228-84bc-dd49-52ccb9fd0015@amd.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <d60a73cd-3228-84bc-dd49-52ccb9fd0015@amd.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UkpjQ3hrWEJ0MEF3emNEOUVzcFY1UkcwOVRnSEpiWms5Z1FEdlhWQkFjTVUx?=
 =?utf-8?B?SE1IRFMxZ3pzMFhnZ201cCtCNWROSngrYWh1aFpWb05KWkxhcjBvQW14ZVJQ?=
 =?utf-8?B?bHBoZDdnV1Vwcm9ReGhTOUZHWTcxRmF2K2RIRUZqbjI5UFJpdG5xc1REVFdG?=
 =?utf-8?B?SllLREpKTXU5cWowME4ydHErbFJwVW0rNS9XaERPcThiSkFUQ2pxeFBrVGkz?=
 =?utf-8?B?d0JYR2FVMVFkZFk2Z2tLbVdIYlJEcW9tRUFZcThIK1VMSloxQUVBK3M4T3lH?=
 =?utf-8?B?cmRiV1l0Y3RMSnIzTHZ5bS8yZkdmUGg4Y3VIRVZ1YzFwZHZYRjgyN21pSWlT?=
 =?utf-8?B?RzVLNzBXOGhWTDNFdEFkS3ZOTEM0cHloK1MvWDZXSE1XbE5MbXlxU2hocjZq?=
 =?utf-8?B?MloyNWtOcDM3VmRNak9TQXN3WWV5NC9ISTZ6L0VmNFFoeFFNdlJZbEF1ekR1?=
 =?utf-8?B?Vk5BZlMrUmxSaDJtbStpVUFhRUJqakNNYi9xQmZ0TGlMS0ZMSG9HbGRQWXR1?=
 =?utf-8?B?cEk0Mzg0aVZRQlpvR0NMN05BTUJ2WWd5U3BzUmgzV3pubnIxSkZHY24yVzkx?=
 =?utf-8?B?c2NTeEFEVDczSVlTeXJhb2t0UzRZOFprNHJydmlEVExqdkJvVVM5eHJqcEY0?=
 =?utf-8?B?Qkpua21jR0FpdkxEd3VzNFVzc3VBNXBYQzZZeXVDRm5qZ2hxWHF1U3JqTWpU?=
 =?utf-8?B?eVUrSm9XVWlEZ2pKKzlNazc5MjNTNWkyTDFtM29sZVlkeUlyV1l5SDVOdFd0?=
 =?utf-8?B?bmVwUThxSU5HTjN4ZElTWmVBUVlUVXRNbkJldzhzYUJxU1JkTUlqRDM4UHN6?=
 =?utf-8?B?bU8rWFFWV3Bna0VTR3hPYUdicmt0STEwaXhHQUNMVU5uWk85YjZSNHcybEtI?=
 =?utf-8?B?Rmpqc29VeVZzU0NkQm13eTFFTWh0UTAySHdyOVZ1aVk1N21jR0tUY3V3OHNL?=
 =?utf-8?B?RXlvUGdKMFZSUWRRbjYwQmMyLzBySSs5bEtrYzFsTU1NbmFxV1krZ2xsdzBZ?=
 =?utf-8?B?NjF3V2NxWU9SRW1RUXBMano2d3V3a3RPYno1RmxXbmpObS9hVE9aSEwwQ0VJ?=
 =?utf-8?B?MlpzSUxteENQdWRxSHhvYzZCa1JjNUMzeHYrYnZyN2EwM3JRbmNoN1lqTFFT?=
 =?utf-8?B?Vkt3cjF1NEhvQUphVWxpQ1g4ajlhZUN4cUdqUUZmM051SE1mWXRPMjlnQVdP?=
 =?utf-8?B?aEpjalJTV2x1WTZFRW03MGtRWGJEazgzbk9pdlhieDIxaUtGQVI3WFVUcDR0?=
 =?utf-8?B?clZ2N1ErU0VzSXlxcXVVY0NJaTNpVmgvTyt5UmZueW1GSUdkV0VOc1lWVita?=
 =?utf-8?B?a2JCZ0NzRVZFazdMTlFETWdWTmhOUWR0RWNuN0wybUIzUGZ3VGJOWkNpZW5V?=
 =?utf-8?B?SFFGNXo0d01KWVA1c09EaVBLbUpsVjBOOVBYL2NjS3dFUE5QTy9QaHZlTlJl?=
 =?utf-8?B?aWlLMU5NNDBOS0pTUmx2MEx2T1F4bVVOQ0QxWW1VaEdkT0tmcW9LbmpybHV1?=
 =?utf-8?B?Ym1YYW9RNTdDS3p5ajUzdktOZSs1eHJMdVFRckRsc0ZleHdacDRRdDFtVXVz?=
 =?utf-8?B?SkxvNmFJQW9TSzVBam5FUVdWSC9qb2NTYVdtS25ETkJCNUpkeUZISGp1aldk?=
 =?utf-8?B?VTJ0VXBmVXVLdzUyM2VxK3ZOdWR2NzlRemNuVHFBVmsyVW84NEhjTGJ3TkRO?=
 =?utf-8?B?WXdEbGZyaDNSODFuN0RyeGZHd3U5Wk50bzh3V1doNWhONm8xZWJGQ1pQbHB3?=
 =?utf-8?B?QnVWWHVRc0VDN0pqT1RCcCtvYzlLaDZ3cWhtNmdKOGYvbEtjQWtQTzVDZDQ4?=
 =?utf-8?B?M1VGRkwyeERXVlMrVVBZUWQ2WkFBU0VucVhlenpBd3F5WE5GQ2JtRjJ0eDFp?=
 =?utf-8?B?dWcvSHp1di9yZXB2MFZRcnZQZ3hUcWN1b21idGlOVjUvZnJ2QWFIUlJLOFRj?=
 =?utf-8?B?cUwvY3pvQStGMDdkeEhHcS9ZTXViTXdiMFVHVFIvclhHY0tRMmhkVFVNRGNh?=
 =?utf-8?B?NjNSWkczUWRNZjF3VVd1dmxCVWU2b05OZXVyMTB2bC9DWnFnQU5QazhqTzE5?=
 =?utf-8?B?WlQ5NnhMb25tcEEwMmFDaGNSL1VsaHZvUmU3dTRKQTZydXMxSVcvVFo3TWJn?=
 =?utf-8?B?dzZ0S2pVWnlaSTZvdC96bVhxVzdHeGlmNHJpTTBpQ2xBb2sxSzBtajFQWGYx?=
 =?utf-8?B?aXc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 0867389a-0db0-4f2d-bde2-08daa623a51f
X-MS-Exchange-CrossTenant-AuthSource: CY4PR11MB1862.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Oct 2022 16:15:25.3702
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: t/xqxKGMI8IYQ4PrSlUiH7CpVSbV7w8ps9X0jz4ZoupA4nLoE3QmFefVlQ8KCRkHTuFqA3KzTo1c1Es2yn3QCbJzcgTQqJ1jHhG3wtVEcOY=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR11MB5911
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Babu,

On 10/4/2022 7:00 AM, Moger, Babu wrote:
> On 10/3/22 10:36, Reinette Chatre wrote:
>> On 10/3/2022 7:28 AM, Moger, Babu wrote:
>>> On 9/29/22 17:10, Reinette Chatre wrote:
>>>> Hi Babu,
>>>>
>>>> In subject: resctrl_ui.rst -> resctrl.rst
>>>>
>>>> On 9/27/2022 1:27 PM, Babu Moger wrote:
>>>>> Update the documentation for the new features:
>>>>> 1. Slow Memory Bandwidth allocation (SMBA).
>>>>>    With this feature, the QOS  enforcement policies can be applied
>>>>>    to the external slow memory connected to the host. QOS enforcement
>>>>>    is accomplished by assigning a Class Of Service (COS) to a processor
>>>>>    and specifying allocations or limits for that COS for each resource
>>>>>    to be allocated.
>>>>>
>>>>> 2. Bandwidth Monitoring Event Configuration (BMEC).
>>>>>    The bandwidth monitoring events mbm_total_bytes and mbm_local_bytes
>>>>>    are set to count all the total and local reads/writes respectively.
>>>>>    With the introduction of slow memory, the two counters are not
>>>>>    enough to count all the different types are memory events. With the
>>>> types are memory events -> types of memory events?
>>> Ok Sure
>>>>>    feature BMEC, the users have the option to configure mbm_total_bytes
>>>>>    and mbm_local_bytes to count the specific type of events.
>>>>>
>>>>> Also add configuration instructions with examples.
>>>>>
>>>>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>>>>> ---
>>>> ...
>>>>
>>>>> +
>>>>> +"mbm_total_config", "mbm_local_config":
>>>>> +        These files contain the current event configuration for the events
>>>>> +        mbm_total_bytes and mbm_local_bytes, respectively, when the
>>>>> +        Bandwidth Monitoring Event Configuration (BMEC) feature is supported.
>>>>> +        The event configuration settings are domain specific. Changing the
>>>>> +        configuration on one CPU in a domain would affect the whole domain.
>>>> This contradicts the implementation done in this series where the
>>>> configuration is changed on every CPU in the domain.
>>> How about this?
>>>
>>> The event configuration settings are domain specific and will affect all the CPUs in the domain.
>> There remains a disconnect between this and the implementation that writes the
>> configuration to every CPU.
>>
>> You could make this change to the documentation but then the
>> implementation needs more than "Update MSR_IA32_EVT_CFG_BASE MSR on all
>> the CPUs in cpu_mask" - that comment needs to highlight that the
>> implementation does not follow the architecture and scope rules nor how
>> configuration changes are made in the rest of the driver and why. Previously [1]
>> you indicated that this is based on guidance from hardware team so perhaps you
>> could document it as a hardware quirk related to this feature? At the minimum
>> it should acknowledge the disconnect.
> 
> ok. I could document this in the code patch 9([PATCH v5 09/12]
> x86/resctrl: Add sysfs interface to write mbm_total_bytes event configuration.
> Something like this.
> 
> /*
> +        * Update MSR_IA32_EVT_CFG_BASE MSR on all the CPUs in cpu_mask.

Since multiple MSRs are impacted, how about:

"Update MSR_IA32_EVT_CFG_BASE MSRs ..."

> +        * The MSR MSR_IA32_EVT_CFG_BASE is domain specific. Writing the

"The MSRs offset from MSR MSR_IA32_EVT_CFG_BASE are scoped at the domain
level. Writing any of these MSRs on one CPU is supposed to be observed
by all CPUs in the domain."

> +        * MSR on one CPU will affect all the CPUs in the domain.

Since this is not the case, perhaps it should be " ...
is supposed to affect all the CPUs ..." instead?

> +        * However, the hardware team recommends to update the MSR on
> +        * all the CPU threads. It is not clear in the document yet.

To be consistent, could "CPU threads" be "CPUs"?

Could you please be specific about which document you refer to? Although,
I do not think that writing the last part about "the document" adds value
here. You are representing AMD with this submission and you document that
you are following the guidance from the hardware team in this regard. 
I think that is sufficient.
 

> *        * Doc will be updated in the next revision.

This is a change that will be made to the kernel source ... what does
"next revision" mean when somebody reads this comment in a few years?

Putting all of the above together, how about:

"Update MSR_IA32_EVT_CFG_BASE MSRs on all the CPUs in cpu_mask. The MSRs
offset from MSR MSR_IA32_EVT_CFG_BASE are scoped at the domain level.
Writing any of these MSRs on one CPU is supposed to be observed by all
CPUs in the domain. However, the hardware team recommends to update these
MSRs on all the CPUs in the domain."

Reinette