Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36;
Message-ID: <c1ccdfa6-ff8a-4602-bb06-f49bf34629ff@intel.com>
Date:   Thu, 7 Dec 2023 15:26:24 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 00/15] x86/resctrl : Support AMD QoS RMID Pinning feature
Content-Language: en-US
To:     <babu.moger@amd.com>, <corbet@lwn.net>, <fenghua.yu@intel.com>,
        <tglx@linutronix.de>, <mingo@redhat.com>, <bp@alien8.de>,
        <dave.hansen@linux.intel.com>, James Morse <james.morse@arm.com>
CC:     <x86@kernel.org>, <hpa@zytor.com>, <paulmck@kernel.org>,
        <rdunlap@infradead.org>, <tj@kernel.org>, <peterz@infradead.org>,
        <seanjc@google.com>, <kim.phillips@amd.com>, <jmattson@google.com>,
        <ilpo.jarvinen@linux.intel.com>, <jithu.joseph@intel.com>,
        <kan.liang@linux.intel.com>, <nikunj@amd.com>,
        <daniel.sneddon@linux.intel.com>, <pbonzini@redhat.com>,
        <rick.p.edgecombe@intel.com>, <rppt@kernel.org>,
        <maciej.wieczor-retman@intel.com>, <linux-doc@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <eranian@google.com>,
        <peternewman@google.com>, <dhagiani@amd.com>
References: <20231201005720.235639-1-babu.moger@amd.com>
 <d97cbeba-af6d-4b64-b5c8-32dc437a67b6@intel.com>
 <71e85bf3-a451-4adf-ad5e-d39f7935efa0@amd.com>
 <e36699cf-c73e-401b-b770-63eba708df38@intel.com>
 <5ce67d8f-e207-4029-8fb3-0bc7deab1e9f@amd.com>
 <390bbb7b-e709-4290-9bef-eed373e469b2@intel.com>
 <a6338e5d-ab94-8da8-3127-3ce90997bf65@amd.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <a6338e5d-ab94-8da8-3127-3ce90997bf65@amd.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?TVZHd1JReVdtWUh5TDdFYW9aRE9CQWJSNENqMklDYTJXSklzM1BPSW1lMDRi?=
 =?utf-8?B?QjhBcm90SlpjN1M0NzgwSyt3QlBPeU84OTRhTWhBdjVvbnMwejRxdlEwUE5l?=
 =?utf-8?B?M0V4UTNBSFpEQWtZSWZQNktEYmpnZFVrYlZmUEduY3dLa1F5SkFvdW5XaDRC?=
 =?utf-8?B?K1lQWTdFRUhwVjRGdXpyN0R1RzRkTWZ1WXZkZGFzMU13aXdpV1hJQnVBcjcy?=
 =?utf-8?B?dEhqMU9hVHVUc0ZwajcvanQ2OEF4eDdKZ005SlRQZ00wcmdNU29OTkdxSkdy?=
 =?utf-8?B?QUxkRmdtaEp2KzZIMGx5RFFRczEyOVpUbnhmOWNXMERWTmQ3TEtsWC91aWtu?=
 =?utf-8?B?OW43UC9qemcvNlp3ekVmS2dqMHZSdlNGMENpaVo5SjZvRkpnU3BUWnp1R3Zl?=
 =?utf-8?B?bG8rTlVUYlcweDlhOW1iTnI4dkk3L2NKYkZPOGFIK21Wei9EQlU4bnA2OHpL?=
 =?utf-8?B?eEFHN2J3ZG1QMWM5WjZDenAzQnJZV3dKZGJkVXZCMjc0dXJKekVOS3hEa24z?=
 =?utf-8?B?cXIyemlaaUZpd0RLSG5EUm9waERNd3lycnZwV1F2NVdMRi9wckJYTmNHTkhU?=
 =?utf-8?B?czlHand5MmM2emFoVEcxZzlOZFdBWm9DdWYwZzFlcVpNc1pOV2tycEJVTjRw?=
 =?utf-8?B?MWIyR3BRTVBjQncxYTZUTVB3QmwxRTBkbUdKOVlrYmlwUnpCNUd3K2VNQ2w5?=
 =?utf-8?B?clI0cU9IcXlIZHVGOWVDRzRZcTZMdXB5dFBVRS9oUzhUTm1FVzZseVdsTHZ5?=
 =?utf-8?B?Q3pMalNJK241NG05UVFJb3MwWXUrL0ZCVDdtKzR5amRTa2M4VVo3RmR3NkR3?=
 =?utf-8?B?YlhoUVVpU0lZNG9XU2FoeUFTbTV3b2NsQUU5U0dxL2QvM1hmYXVaZUJMeElq?=
 =?utf-8?B?dUFBUldiQ25jaDJxaGRGRk1VTzVNZmNpeUU3WitPR3VKakE4eVpBakhTK0c5?=
 =?utf-8?B?Y2t6cGcyb0ZKQ3RMdnYyR2RZc0UxaUpBVXRHUFYrUWdmU3RwN1l1UVEyZnhZ?=
 =?utf-8?B?cGRkZDBMSURjSUZVQkc0Rmt0UmVPKzFqNVFYb0ZUNGlVdDhvZTFkajN2Q0Vo?=
 =?utf-8?B?N0Vmem5Zdy9RY3UzVzdJRkJCNUlvKzBIQVpndXVmamh2UGl5TXp0ZXFpb0d1?=
 =?utf-8?B?eVpxdHFJOVZDeWlHM0dHK1ZubzRHSm00dEZkeTB0RVBpZ3JOdWt1WWxCaHlF?=
 =?utf-8?B?bG0wU2k3YW5KaHROL3NZMW1JT29xL2JsNDNpNUtXWVFubnUzOVN1cUVnY0w3?=
 =?utf-8?B?dzJJTUM2dmJ3eDFCa0dnWXUxR2hwc3pHM2NBMFVjekhzbVZqNmlwTVYxRjVs?=
 =?utf-8?B?MzBRL082L0VaaGg3YWdUUFZ4UWRuc1VHRDRRcU9pNHlKQzVibit0aFZVaEJV?=
 =?utf-8?B?WXgwMG1JeENZRExhR0xLYkVseXY0QVAvQTFxQmh2a2tFUW1FSXpCM2NUY1E2?=
 =?utf-8?B?T1BML0pxWWcyTFJ2WEVSanNWNGM3bWVPbmJKNUpDMC95N2VBWXczcU1hREVE?=
 =?utf-8?B?U2RqWkRyanBkSmtSQmhCNS9xdnJVSWdnaUdCNlFYdzJBNmcxVWtQNzJlOHE1?=
 =?utf-8?B?WGIwNmJSbFBOVm01aGNVOUl3bExVNHhVaEpmeG1COXpIQWpONmlHTGc4M2px?=
 =?utf-8?B?alRGSzlMK2pVTkUwdU5oV3V6TTQyTkdRNUd4MHBHb1lhdTlsUWY2OXZXV1R5?=
 =?utf-8?B?NDRKaGhHMkZPSmJ4aGsvYXhVVkdmc25UN09vSk1ML3BkaWM1ZmtHcTZ2QVl2?=
 =?utf-8?B?cTZEbURYU3dKQlkwR1lkQmc4RHd4UWhPSWFNSjAxMy9HN2E0VXc1TUFyVzlU?=
 =?utf-8?B?SElXQnhvcHlIZVpQWEwzTUtZYVp2aS9FSktlYWFJcGZoWEpKSkxDNUh1QzNp?=
 =?utf-8?B?NmhkN2wxNjJ6MnZ3WHB2djk3Uk9lSHh0U3R3T3l1aWRjN2lKNFljYytObld2?=
 =?utf-8?B?OERreTQrREdWS0RPWTFNaW5BQkxDNGZXNlJvS1k4T3FiZUNnWC9HNGdCK3Fr?=
 =?utf-8?B?dmh0bmVkUnViT3l0QkhqQjdKckp4b0gxNU80ajJEcUdXMElzeG1oVFNkdm1I?=
 =?utf-8?B?ZytSWmlWM3k2dFlucFVhU3ZLUDlUMGpnTzUwS25DS2dQNk0wQkQrSDdnazBH?=
 =?utf-8?B?RHdLSE5sem91UVhkQVliMVlMNDEzdzY2M3E4aHRSV1JKMkxhcTMrZXRRRjls?=
 =?utf-8?B?OEE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 2cbc9322-63f7-4e9f-57b6-08dbf77befa0
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Dec 2023 23:26:27.8356
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: U7CtxSMj7cBydvxX39aUUrBozpnfJanGptwteHz5QQL2KDo2+Ng0t7Y3mTkNKkbpv2y0t7rP+1XW0SdaYpu2eGzDq3J1BUw5XerRPE9Z1ZI=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR11MB6157
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Thu, 07 Dec 2023 15:26:52 -0800 (PST)

Hi Babu,

On 12/7/2023 3:07 PM, Moger, Babu wrote:
> On 12/7/2023 1:29 PM, Reinette Chatre wrote:
>> On 12/7/2023 8:12 AM, Moger, Babu wrote:
>>> On 12/6/23 12:49, Reinette Chatre wrote:
>>>> On 12/6/2023 7:40 AM, Moger, Babu wrote:
>>>>> On 12/5/23 17:17, Reinette Chatre wrote:
>>>>>> On 11/30/2023 4:57 PM, Babu Moger wrote:
>>
>>>>>>> b. Mount with ABMC support
>>>>>>>     #umount /sys/fs/resctrl/
>>>>>>>     #mount  -o abmc -t resctrl resctrl /sys/fs/resctrl/
>>>>>>>     
>>>>>> hmmm ... so this requires the user to mount resctrl, determine if the
>>>>>> feature is supported, unmount resctrl, remount resctrl with feature enabled.
>>>>>> Could you please elaborate what prevents this feature from being enabled
>>>>>> without needing to remount resctrl?
>>>>> Spec says
>>>>> "Enabling ABMC: ABMC is enabled by setting L3_QOS_EXT_CFG.ABMC_En=1 (see
>>>>> Figure 19-7). When the state of ABMC_En is changed, it must be changed to
>>>>> the updated value on all logical processors in the QOS Domain.
>>>>> Upon transitions of the ABMC_En the following actions take place:
>>>>> All ABMC assignable bandwidth counters are reset to 0.
>>>>> The L3 default mode bandwidth counters are reset to 0.
>>>>> The L3_QOS_ABMC_CFG MSR is reset to 0."
>>>>>
>>>>> So, all the monitoring group counters will be reset.
>>>>>
>>>>> It is technically possible to enable without remount. But ABMC mode
>>>>> requires few new files(in each group) which I added when mounted with "-o
>>>>> abmc". Thought it is a better option.
>>>>>
>>>>> Otherwise we need to add these files when ABMC is supported(not when
>>>>> enabled). Need to add another file in /sys/fs/resctrl/info/L3_MON to
>>>>> enable the feature on the fly.
>>>>>
>>>>> Both are acceptable options. Any thoughts?
>>>> The new resctrl files in info/ could always be present. For example,
>>>> user space may want to know how many counters are available before
>>>> enabling the feature.
>>>>
>>>> It is not yet obvious to me that this feature requires new files
>>>> in monitor groups.
>>> There are two MBM events(total and local) in each group.
>>> We should provide an interface to assign each event independently.
>>> User can assign only one event in a group. We should also provide an
>>> option assign both the events in the group. This needs to be done at
>>> resctrl group level.
>> Understood. I would like to start by considering how (if at all) existing
>> files may be used, thus my example of using mbm_total_bytes, before adding
>> more files.
>>
>>
>> ...
>>
>>>>>>>     #cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes
>>>>>>>     Unavailable
>>>>>> I believe that "Unavailable" already has an accepted meaning within current
>>>>>> interface and is associated with temporary failure. Even the AMD spec states "This
>>>>>> is generally a temporary condition and subsequent reads may succeed". In the
>>>>>> scenario above there is no chance that this counter would produce a value later.
>>>>>> I do not think it is ideal to overload existing interface with different meanings
>>>>>> associated with a new hardware specific feature ... something like "Disabled" seems
>>>>>> more appropriate.
>>>>> Hardware still reports it as unavailable. Also, there are some error cases
>>>>> hardware can report unavailable. We may not be able to differentiate that.
>>>> This highlights that this resctrl feature is currently latched to AMD's
>>>> ABMC. I do not think we should require that this resctrl feature is backed
>>>> by hardware that can support reads of counters that are disabled. A counter
>>>> read really only needs to be sent to hardware if it is enabled.
>>>>
>>>>>> Considering this we may even consider using these files themselves as a
>>>>>> way to enable the counters if they are disabled. For example, just
>>>>>> "echo 1 > /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes" can be used
>>>>> I am not sure about this. This is specific to domain 0. This group can
>>>>> have cpus from multiple domains. I think we should have the interface for
>>>>> all the domains(not for specific domain).
>>>> Are the ABMC registers not per CPU? This is unclear to me at this time
>>>> since changelog of patch #13 states it is per-CPU but yet the code
>>>> uses smp_call_function_any().
>>> Here are the clarifications from hardware engineer about this.
>>>
>>> # While configuring the counter, should we have to write (L3_QOS_ABMC_CFG)
>>> on all the logical processors in a domain?
>>>
>>> No.  In order to configure a specific counter, you only need to write it
>>> on a  single logical processor in a domain.  Configuring the actual ABMC
>>> counter is a side-effect of the write to this register.  And the actual
>>> ABMC counter configuration is a  global state.
>>>
>>> "Each logical processor implements a separate copy of these registers"
>>> identifies that if you write a 5 to L3_QOS_ABMC_CFG on C0T0, you will not
>>> read a 5 from the L3_QOS_ABMC_CFG register on C1T0.
>> Thank you for this information. Would reading L3_QOS_ABMC_DSC register on
>> C1T0 return the configuration written to L3_QOS_ABMC_CFG on C0T0 ?
> 
> Yes. Because the counter counter configuration is global. Reading L3_QOS_ABMC_DSC will return the configuration of the counter specified by
> 
> QOS_ABMC_CFG[CtrID].


To confirm, when you say "global" you mean within a domain?

> 
>>
>> Even so, you do confirm that the counter configuration is per domain. If I
>> understand correctly the implementation in this series assumes the counters
>> are programmed identically on all domains, but theoretically the system can support
>> domains with different counter configurations. For example, if a resource group
>> is limited to CPUs in one domain it would be unnecessary to consume the other
>> domain's counters.
> Yes. It is programmed on all the domains. Separating the domain
> configuration will require more changes. I am not planning to address
> in this series.

That may be ok. The priority is to consider how users want to interact with this
feature and create a suitable interface to support this. This version may not
separate domain configuration, but we do not want to create an the interface that
prevents such an enhancement in the future. Especially since it is already known
that hardware supports it.

Reinette