Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <5d17741a-c6a6-9ab0-fc05-673337b04a57@intel.com>
Date:   Wed, 30 Nov 2022 12:07:10 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.5.0
Subject: Re: [PATCH v8 05/13] x86/resctrl: Detect and configure Slow Memory
 Bandwidth Allocation
Content-Language: en-US
To:     <babu.moger@amd.com>, <corbet@lwn.net>, <tglx@linutronix.de>,
        <mingo@redhat.com>, <bp@alien8.de>
CC:     <fenghua.yu@intel.com>, <dave.hansen@linux.intel.com>,
        <x86@kernel.org>, <hpa@zytor.com>, <paulmck@kernel.org>,
        <akpm@linux-foundation.org>, <quic_neeraju@quicinc.com>,
        <rdunlap@infradead.org>, <damien.lemoal@opensource.wdc.com>,
        <songmuchun@bytedance.com>, <peterz@infradead.org>,
        <jpoimboe@kernel.org>, <pbonzini@redhat.com>,
        <chang.seok.bae@intel.com>, <pawan.kumar.gupta@linux.intel.com>,
        <jmattson@google.com>, <daniel.sneddon@linux.intel.com>,
        <sandipan.das@amd.com>, <tony.luck@intel.com>,
        <james.morse@arm.com>, <linux-doc@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <bagasdotme@gmail.com>,
        <eranian@google.com>
References: <166759188265.3281208.11769277079826754455.stgit@bmoger-ubuntu>
 <166759202922.3281208.6231300030834095574.stgit@bmoger-ubuntu>
 <d2209b3d-374c-efdd-a91f-217f1db3b4bf@intel.com>
 <764c8eac-7b33-3850-3052-e6444ce496d0@amd.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <764c8eac-7b33-3850-3052-e6444ce496d0@amd.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MStreEpoZ2pWeU9SRkU1OHRkR3h2NTFpQmdESFpia1ZCeDFLME00Mkk4M25m?=
 =?utf-8?B?RUM5alFQRWl1MEkyajRvWS9jQVNOTW1JT1lGQVpNdlk3Yk5KQWhlUHZkWXdP?=
 =?utf-8?B?cXBLVThCbzc1QVE1NnNlaWtSWEM4elFaUkhxR3FYY2dDNldQRksrUlhWOXpa?=
 =?utf-8?B?SnVBb0JHR01DMW9yaXJwT0JDVVByNjgvSFVRY0pWek84a0pOOVlWTFZ4YWp6?=
 =?utf-8?B?UExmSTFpTEVkVEVTOWFxL0h5VWcrN3VOakM4UjR4Y3Y0OGZpQTdmOWNPcG9N?=
 =?utf-8?B?ZTYyWVREQ3c0cGNlR0R5bDlScUhhd3V1ZURmdGdVZUxSdGk0UjgvWHcwazht?=
 =?utf-8?B?R0ZldGdEVVVXYUNwcmJyNzBXTExUckM3azdJTER1RXlQVitCL0FNV0FiaE9K?=
 =?utf-8?B?RjFZS0tEVlBuS1V5VkFGWXgzSDdGNkxXbFN3TUMxa2pyalJjbG42MlI3bWRh?=
 =?utf-8?B?OGtmaDE5dndRcUI2MlM1ZzRIUHJjMWxLM2JJZytlc3JyNUVnazJmS1FlTVpq?=
 =?utf-8?B?d0dPMHFtOU5rS05KNlpSNlltSldnd05FQkU5Q3liK0N3QXRCZVB6aE5JR29G?=
 =?utf-8?B?U24xcHplTE5yRWhVaUZ4TXlHRmlKdHRYYVg1Y1V4WkZkbHVWSUIxSC8rU3N0?=
 =?utf-8?B?NnlTTXR5bi9lZjFkUlJTT2g2bUJBU1VJRGM4aGtsU3JUM2x4L25HYjNIdGU4?=
 =?utf-8?B?VVVJM0phVVV2aVUyM2VaeHFWUjZlT0ovalRSYnlWMjNtSGdSSU5GOTYxaS9a?=
 =?utf-8?B?RjdBcUVwNElIa3dFajREeFozRlZlOStkS29LRzhJMVZ6WmtZMnZFclQxRGFt?=
 =?utf-8?B?di9JM1JuRVVZNG1iZ3pjOW1sbXEwNEhvSEo0NGZvODZ6Rmt1MkNGM2VNUGRk?=
 =?utf-8?B?enFOZHJBZjc5cXh6dXI2US9kMEpjeEtWK2tveWtra1BFOEF4ZzFWUFVjY204?=
 =?utf-8?B?WGdhUm5kWEdrR0pPb3FlOUlnL1NmQndtT3dTcnovWWtKa1NJQXNISUdha2l1?=
 =?utf-8?B?VWIvNWpYUVVoRVdVU3VvYzhVMjI5T3ZtVjFhS0ZLYzl5NkgvekpqU1NtZFp0?=
 =?utf-8?B?enpGa2VVWHZvUkJpMFk0dFRhVWtzZDZPRi9Ibm1XdWRHSkMzUC8rUUpzS2R6?=
 =?utf-8?B?QzNaYTl3OTd1eUcxSUdralhTNHd4VU5mdVBjVnR4QUFxcTlPVkNOK2xXUEE1?=
 =?utf-8?B?Y2lMUEpabUFFNlRJOXF3dkxXaVUwT3pQU0tNa1ZJZit0SGdyMWVWcXJNM0Q4?=
 =?utf-8?B?VFhBdXo4T21SVlUxYWFUMHU1Q2VkMXZKQ3ZMUGFLUHEwMFFic2UwZGVYYVk2?=
 =?utf-8?B?ckdUMWRlVXBLb1FHUkpvMkx2Zk1CS0szOGZPZzU1ZDhHK2p5NDZmZTd2WUF6?=
 =?utf-8?B?eitsWHJ1dXlWWVBNZUdBK2IreWlnSU1TTzkvVFdVZG5NSnpXL1BEUk5Zem9k?=
 =?utf-8?B?TW00S29GNjhyaDJMQ3RnR21rZTRCOWhJVzhXeG9xTnFOQUR3SzFoTExHWnc5?=
 =?utf-8?B?VUNuUFJiSnMvRC9wblRuSkYvdFBIUDd3R0VnbjJLT3ZvVkF5Wm56eEdCK1F6?=
 =?utf-8?B?Y1NvdXIwejhIZ0VJMXdHN1htSVVUY2hwL211djN3cVFxSzc2MjA3TXVOdHVq?=
 =?utf-8?B?K3Z4MzNXcjhKRzFramg5bHZPeklFSHprVXQwcVUxalJ1WmN1SWxQRWRmaXVr?=
 =?utf-8?B?NDgyVmpsekVLc01nS2k3NFBwS0hObXF0emNyMk8reFIxcEpmRTZOSmlNdE5Q?=
 =?utf-8?B?L2lvdkgySlY4M2VXWmxsb3pTZkpETUtXSVBCZk45TkJ2MzdJQWFLSTRqeTgx?=
 =?utf-8?B?WGM1VlVsUG1oY1k4V1Z3K3lCdTM0ejBQWnRVQzNrOWJwaWNEd0paK2Jidlh4?=
 =?utf-8?B?dFZYNURzNG8zVURrdVRGTXQxZlRSdUJ1R29PQjk1K3RUWTRwT0lFZVBsNnNk?=
 =?utf-8?B?ZFgxTzNwSnJtTzFJVjJIMUdHN2cyUGhZVkRRbWxwYTgwM3FWMGs5cUcyNTV4?=
 =?utf-8?B?dUNhTXBDYkltZHU3NVpSWkY5SFBreVFWSTM0b1c1TlhnQVkyQTRtaEVWQTBt?=
 =?utf-8?B?NVJ3RE5WOTVnN3pZVS9TUS81RUQxRDFOaHVZMWpVZG56Um9HRVhBWVRBU2F3?=
 =?utf-8?B?UnBha2l1cDF5N0l6MzByM0o5UWoxQjEyQjAyaEdDNUhzZGd4aGNUR1g3cUpx?=
 =?utf-8?B?aUE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: b016ab8e-8887-449b-33c5-08dad30e7915
X-MS-Exchange-CrossTenant-AuthSource: CY4PR11MB1862.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Nov 2022 20:07:14.3281
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: c4c7tdse3hwkbMMIEB1UCik5s663Da+EioktrjXwKsFQzYy35b7T6/vka1s6GZpZA1GLeIBfV2/pzDbxDvQneptqF8RtEAc3CfKRaAfYXaE=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB5119
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Babu,

On 11/30/2022 10:43 AM, Moger, Babu wrote:
> On 11/22/22 18:12, Reinette Chatre wrote:
>> On 11/4/2022 1:00 PM, Babu Moger wrote:
>>> The QoS slow memory configuration details are available via
>>> CPUID_Fn80000020_EDX_x02. Detect the available details and
>>> initialize the rest to defaults.
>>>
>>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>>> ---
>>>  arch/x86/kernel/cpu/resctrl/core.c        |   36 +++++++++++++++++++++++++++--
>>>  arch/x86/kernel/cpu/resctrl/ctrlmondata.c |    2 +-
>>>  arch/x86/kernel/cpu/resctrl/internal.h    |    1 +
>>>  arch/x86/kernel/cpu/resctrl/rdtgroup.c    |    8 ++++--
>>>  4 files changed, 41 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
>>> index e31c98e2fafc..6571d08e2b0d 100644
>>> --- a/arch/x86/kernel/cpu/resctrl/core.c
>>> +++ b/arch/x86/kernel/cpu/resctrl/core.c
>>> @@ -162,6 +162,13 @@ bool is_mba_sc(struct rdt_resource *r)
>>>  	if (!r)
>>>  		return rdt_resources_all[RDT_RESOURCE_MBA].r_resctrl.membw.mba_sc;
>>>  
>>> +	/*
>>> +	 * The software controller support is only applicable to MBA resource.
>>> +	 * Make sure to check for resource type again.
>>> +	 */
>> /again/d
>>
>> Not all callers of is_mba_sc() check if it is called for an MBA resource.
>>
>>> +	if (r->rid != RDT_RESOURCE_MBA)
>>> +		return false;
>>> +
>>>  	return r->membw.mba_sc;
>>>  }
>>>  
>>> @@ -225,9 +232,15 @@ static bool __rdt_get_mem_config_amd(struct rdt_resource *r)
>>>  	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
>>>  	union cpuid_0x10_3_eax eax;
>>>  	union cpuid_0x10_x_edx edx;
>>> -	u32 ebx, ecx;
>>> +	u32 ebx, ecx, subleaf;
>>>  
>>> -	cpuid_count(0x80000020, 1, &eax.full, &ebx, &ecx, &edx.full);
>>> +	/*
>>> +	 * Query CPUID_Fn80000020_EDX_x01 for MBA and
>>> +	 * CPUID_Fn80000020_EDX_x02 for SMBA
>>> +	 */
>>> +	subleaf = (r->rid == RDT_RESOURCE_SMBA) ? 2 :  1;
>>> +
>>> +	cpuid_count(0x80000020, subleaf, &eax.full, &ebx, &ecx, &edx.full);
>>>  	hw_res->num_closid = edx.split.cos_max + 1;
>>>  	r->default_ctrl = MAX_MBA_BW_AMD;
>>>  
>>> @@ -750,6 +763,19 @@ static __init bool get_mem_config(void)
>>>  	return false;
>>>  }
>>>  
>>> +static __init bool get_slow_mem_config(void)
>>> +{
>>> +	struct rdt_hw_resource *hw_res = &rdt_resources_all[RDT_RESOURCE_SMBA];
>>> +
>>> +	if (!rdt_cpu_has(X86_FEATURE_SMBA))
>>> +		return false;
>>> +
>>> +	if (boot_cpu_data.x86_vendor == X86_VENDOR_AMD)
>>> +		return __rdt_get_mem_config_amd(&hw_res->r_resctrl);
>>> +
>>> +	return false;
>>> +}
>>> +
>>>  static __init bool get_rdt_alloc_resources(void)
>>>  {
>>>  	struct rdt_resource *r;
>>> @@ -780,6 +806,9 @@ static __init bool get_rdt_alloc_resources(void)
>>>  	if (get_mem_config())
>>>  		ret = true;
>>>  
>>> +	if (get_slow_mem_config())
>>> +		ret = true;
>>> +
>>>  	return ret;
>>>  }
>>>  
>>> @@ -869,6 +898,9 @@ static __init void rdt_init_res_defs_amd(void)
>>>  		} else if (r->rid == RDT_RESOURCE_MBA) {
>>>  			hw_res->msr_base = MSR_IA32_MBA_BW_BASE;
>>>  			hw_res->msr_update = mba_wrmsr_amd;
>>> +		} else if (r->rid == RDT_RESOURCE_SMBA) {
>>> +			hw_res->msr_base = MSR_IA32_SMBA_BW_BASE;
>>> +			hw_res->msr_update = mba_wrmsr_amd;
>>>  		}
>>>  	}
>>>  }
>> I mentioned earlier that this can be moved to init of
>> rdt_resources_all[]. No strong preference, leaving here works
>> also.
> 
> I am little confused about this comment. Initialization of
> rdt_resources_all in core.c is mostly generic initialization. The msr_base
> and msr_update routines here are vendor specific. I would prefer to keep
> this in

This is a contradiction. Yes, rdt_resources_all[] initialization in core.c
is indeed generic initialization, so why is SMBA there? If this was really
generic initialization then the entire initialization of SMBA resource
should rather move to AMD specific code.

SMBA is an AMD only feature yet its resource initialization is fragmented
with one portion treated as generic and another portion treated as vendor
specific while it all is vendor specific.

The current fragmentation is not clear to me. Keeping the initialization
as you have in patch #2 is the simplest and that is what prompted me
to suggest the move to keep initialization together at that location.

> 
> rdt_init_res_defs_amd.Is that ok?

The generic vs non-generic initialization argument is not convincing to me. 
Could you please elaborate why you prefer it this way? I already mentioned
that I do not have a strong preference but I would like to understand what
the motivation for this split initialization is.

Reinette