Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <1b495f25-bc81-6181-e48d-729bf8211dc1@amd.com>
Date:   Wed, 30 Nov 2022 14:40:03 -0600
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.4.2
Reply-To: babu.moger@amd.com
Subject: Re: [PATCH v8 05/13] x86/resctrl: Detect and configure Slow Memory
 Bandwidth Allocation
Content-Language: en-US
To:     Reinette Chatre <reinette.chatre@intel.com>, corbet@lwn.net,
        tglx@linutronix.de, mingo@redhat.com, bp@alien8.de
Cc:     fenghua.yu@intel.com, dave.hansen@linux.intel.com, x86@kernel.org,
        hpa@zytor.com, paulmck@kernel.org, akpm@linux-foundation.org,
        quic_neeraju@quicinc.com, rdunlap@infradead.org,
        damien.lemoal@opensource.wdc.com, songmuchun@bytedance.com,
        peterz@infradead.org, jpoimboe@kernel.org, pbonzini@redhat.com,
        chang.seok.bae@intel.com, pawan.kumar.gupta@linux.intel.com,
        jmattson@google.com, daniel.sneddon@linux.intel.com,
        sandipan.das@amd.com, tony.luck@intel.com, james.morse@arm.com,
        linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org,
        bagasdotme@gmail.com, eranian@google.com
References: <166759188265.3281208.11769277079826754455.stgit@bmoger-ubuntu>
 <166759202922.3281208.6231300030834095574.stgit@bmoger-ubuntu>
 <d2209b3d-374c-efdd-a91f-217f1db3b4bf@intel.com>
 <764c8eac-7b33-3850-3052-e6444ce496d0@amd.com>
 <5d17741a-c6a6-9ab0-fc05-673337b04a57@intel.com>
From:   "Moger, Babu" <babu.moger@amd.com>
In-Reply-To: <5d17741a-c6a6-9ab0-fc05-673337b04a57@intel.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Q2V1ejVJYzNZWUNaZkFCbFZHZHZsK3ZLVVdmaXBwamFqY0ZMYVBrSkd4dWhX?=
 =?utf-8?B?Y2x3MkJudGE2cEI5MERlaGwzdDlOMWNxK0ZHQ2xLMktLK2s2WlUrR2VaN05t?=
 =?utf-8?B?M1ZtaDhWbU5YT0FZVFV1cXIydHRaYThrMW5MQTB4WUF6SjZYTk5FOW5KYWQv?=
 =?utf-8?B?VnFFSTcxQWszZjAwR1Z5ZHNLN0gxZEF2cmRBeStyalhZSHhWU0ErN0I2ZEpX?=
 =?utf-8?B?SUduMmxFaU9sVFhVenhTaW9MSitIeU1uT3RWb1hFclFXRElVUlpiRFVVcTZF?=
 =?utf-8?B?UFZ1V1NacTBmYW4wdzRGaHE5N0RYTWJQek5CdkVoZmlsczB1clEvQnh1WUhr?=
 =?utf-8?B?d051L1dzQU9kZUtrVjd0RVJhUEZnWjhaMzNqNEFtajZpcTZrQURteVFIcTFS?=
 =?utf-8?B?cmc5TUpDeXd1R25KVGFKRXpkUzJ2Mk1JZERKWHRpTXR2ZGthMENGMTliT0pM?=
 =?utf-8?B?UVJzZm05aU15ajZmeHo5RFc3YnZoSExWZGpvM2tSc3NSRlpYUU1hQ0hyYkdu?=
 =?utf-8?B?enBmV3RjamVGK0xMR3ZoQzJCaHdja1k3bDdkS3k2UDFEVGNBdmQzRkE1amww?=
 =?utf-8?B?TjU1d1oyTTJFa1FhQmw4QTZqKzlHNGd0TGt0bExBQ0dBRmxOZTM2RFpzUW9t?=
 =?utf-8?B?cGw3a1dncHFCeVBsejNoN0NXUWxVUk45RjBRQ2RVTDU3d2JDSmh0QzlHRE1k?=
 =?utf-8?B?NlY3S0JLVFlmb3o2RmRiQ1d5M0ZDRmY5b011VTZQY1BpcVJER1hWSENnZ2VH?=
 =?utf-8?B?UWN5UTAvS0ZjdTZiNWsydTNrS0JmdXZGaGViMVIrS2FiSlBPRWlOdG5HUGpp?=
 =?utf-8?B?RGcvNDkyUXY4YXNZMXp2Uk1wMXUwSmQ3bkwwMHIvR2ZaRXBIb2hhREt4ZWky?=
 =?utf-8?B?cVJSeTdjMnZQZEswVFNiV2R5d1lMZzA1T2VOc0k3MHgzbXdMM1N1emU5OWZu?=
 =?utf-8?B?MXBSQ0VrajlIbSs1ZzJlNURyM3IxTENSMWY2S3l4Um5aY0xVcy8xYkREUHBm?=
 =?utf-8?B?VzdVaVpvZjl4a3d4dVQrc0FlZXZpd1AxTG93TXp2R2paVjliMiszL240QXBz?=
 =?utf-8?B?cDBjM0pPZllVa1J2QjNiKytzc3c2YUVUTi9MOVdWcVZqWVF6RG9Jc0FNRzJJ?=
 =?utf-8?B?WW5ENTNEMjZDWis4QXdsOERGVjdwWUJpc1lUaldibi9jM2dRV09NZlErZWxq?=
 =?utf-8?B?K0tIc0wxVG82RkVhSGNkTjlJTUVadkNPb2xWTHM2RnY1aUlySzM2YzBkSU1B?=
 =?utf-8?B?cU5Hblc0RnZYUjNWRlBkcWNJMFFGZ3ZwNnlTaTdyVVNqWWkyYUw3T0ZLWm5r?=
 =?utf-8?B?VEVxZDRyVEZQbFNSR2RTOHpibzJXOHNLYUxweFNMODhjK1g3Z3ljSE9GM0RH?=
 =?utf-8?B?VU9hTWZwMnpLeVZ3QVhlNHN5NUM1OXAxV2d1czdpSUJWdTk2VEE5T1h5TFlk?=
 =?utf-8?B?amdURU84WVkxK0srTXU4Y0daak1mOU94Qy9zcHlFTjY1WGxXWTJjU1o1Ukx5?=
 =?utf-8?B?VjJLdlE4ekpUWE01Tm5VZndrcUpINFJydjc1SWpybVkzWDR5dVpJdTN3Uk5y?=
 =?utf-8?B?cUJmMi90MEpJQS9BeTd4OWtGbFE2ZVoyT3VQVUdmOTNaWjRHRWY0K0hFMEMz?=
 =?utf-8?B?SURndVFta25VSkZCeXAwcFZnWGVrclIxMlB4RW05Zldmc2hkTWdBak1XOGNF?=
 =?utf-8?B?VWlFelhvQXd1Q1JPRklha3BYOEJiZUIveTl5eFRNRW1JcEd5WXhENjFkaWZ6?=
 =?utf-8?B?dXBBNSt2dUEyVTRiMmhCS1ZvUExQRDIybk9WVU5ydFB6czhQaENtSXpucHNq?=
 =?utf-8?B?Vk05M2tkaTBHcG5XZlhBdnlNbCtLZFVRM082amFITDBpWjkvQmYxY3NBWW5w?=
 =?utf-8?B?MTN6SHYxQlZTb25Ra3NTZ1RjcllTYldKTit1Uk96dDU5ZXE4Zk1IZjA0UUpQ?=
 =?utf-8?B?UVllaDFXTnNHcUs5clB2c3hFRThUUlNIeGRBSDBRQkZpakNkd2JKM3RsZlJw?=
 =?utf-8?B?TlhYQ21SWE5NdFVBd0hNSTlMRVZFQW9DeTQ5UFRwMlhtRW95b0pxUHZtUWN4?=
 =?utf-8?B?RjJMUFZ5OWNVZGRQQUhad0FTZS9jWmFPMFdsVzNDcCs0amtXMXduYjFsNEYr?=
 =?utf-8?Q?0HJg=3D?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e13feb82-97b6-4325-3e49-08dad3131177
X-MS-Exchange-CrossTenant-AuthSource: MW3PR12MB4553.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Nov 2022 20:40:08.0025
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Pf8bi/TIHdKUc8Mnrpu226NkyA5YpIx88axSmcbT6uUnvY36Yg8IvNl/urkczyPa
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7329
X-Spam-Status: No, score=-2.4 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 11/30/22 14:07, Reinette Chatre wrote:
> Hi Babu,
>
> On 11/30/2022 10:43 AM, Moger, Babu wrote:
>> On 11/22/22 18:12, Reinette Chatre wrote:
>>> On 11/4/2022 1:00 PM, Babu Moger wrote:
>>>> The QoS slow memory configuration details are available via
>>>> CPUID_Fn80000020_EDX_x02. Detect the available details and
>>>> initialize the rest to defaults.
>>>>
>>>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>>>> ---
>>>>  arch/x86/kernel/cpu/resctrl/core.c        |   36 +++++++++++++++++++++++++++--
>>>>  arch/x86/kernel/cpu/resctrl/ctrlmondata.c |    2 +-
>>>>  arch/x86/kernel/cpu/resctrl/internal.h    |    1 +
>>>>  arch/x86/kernel/cpu/resctrl/rdtgroup.c    |    8 ++++--
>>>>  4 files changed, 41 insertions(+), 6 deletions(-)
>>>>
>>>> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
>>>> index e31c98e2fafc..6571d08e2b0d 100644
>>>> --- a/arch/x86/kernel/cpu/resctrl/core.c
>>>> +++ b/arch/x86/kernel/cpu/resctrl/core.c
>>>> @@ -162,6 +162,13 @@ bool is_mba_sc(struct rdt_resource *r)
>>>>  	if (!r)
>>>>  		return rdt_resources_all[RDT_RESOURCE_MBA].r_resctrl.membw.mba_sc;
>>>>  
>>>> +	/*
>>>> +	 * The software controller support is only applicable to MBA resource.
>>>> +	 * Make sure to check for resource type again.
>>>> +	 */
>>> /again/d
>>>
>>> Not all callers of is_mba_sc() check if it is called for an MBA resource.
>>>
>>>> +	if (r->rid != RDT_RESOURCE_MBA)
>>>> +		return false;
>>>> +
>>>>  	return r->membw.mba_sc;
>>>>  }
>>>>  
>>>> @@ -225,9 +232,15 @@ static bool __rdt_get_mem_config_amd(struct rdt_resource *r)
>>>>  	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
>>>>  	union cpuid_0x10_3_eax eax;
>>>>  	union cpuid_0x10_x_edx edx;
>>>> -	u32 ebx, ecx;
>>>> +	u32 ebx, ecx, subleaf;
>>>>  
>>>> -	cpuid_count(0x80000020, 1, &eax.full, &ebx, &ecx, &edx.full);
>>>> +	/*
>>>> +	 * Query CPUID_Fn80000020_EDX_x01 for MBA and
>>>> +	 * CPUID_Fn80000020_EDX_x02 for SMBA
>>>> +	 */
>>>> +	subleaf = (r->rid == RDT_RESOURCE_SMBA) ? 2 :  1;
>>>> +
>>>> +	cpuid_count(0x80000020, subleaf, &eax.full, &ebx, &ecx, &edx.full);
>>>>  	hw_res->num_closid = edx.split.cos_max + 1;
>>>>  	r->default_ctrl = MAX_MBA_BW_AMD;
>>>>  
>>>> @@ -750,6 +763,19 @@ static __init bool get_mem_config(void)
>>>>  	return false;
>>>>  }
>>>>  
>>>> +static __init bool get_slow_mem_config(void)
>>>> +{
>>>> +	struct rdt_hw_resource *hw_res = &rdt_resources_all[RDT_RESOURCE_SMBA];
>>>> +
>>>> +	if (!rdt_cpu_has(X86_FEATURE_SMBA))
>>>> +		return false;
>>>> +
>>>> +	if (boot_cpu_data.x86_vendor == X86_VENDOR_AMD)
>>>> +		return __rdt_get_mem_config_amd(&hw_res->r_resctrl);
>>>> +
>>>> +	return false;
>>>> +}
>>>> +
>>>>  static __init bool get_rdt_alloc_resources(void)
>>>>  {
>>>>  	struct rdt_resource *r;
>>>> @@ -780,6 +806,9 @@ static __init bool get_rdt_alloc_resources(void)
>>>>  	if (get_mem_config())
>>>>  		ret = true;
>>>>  
>>>> +	if (get_slow_mem_config())
>>>> +		ret = true;
>>>> +
>>>>  	return ret;
>>>>  }
>>>>  
>>>> @@ -869,6 +898,9 @@ static __init void rdt_init_res_defs_amd(void)
>>>>  		} else if (r->rid == RDT_RESOURCE_MBA) {
>>>>  			hw_res->msr_base = MSR_IA32_MBA_BW_BASE;
>>>>  			hw_res->msr_update = mba_wrmsr_amd;
>>>> +		} else if (r->rid == RDT_RESOURCE_SMBA) {
>>>> +			hw_res->msr_base = MSR_IA32_SMBA_BW_BASE;
>>>> +			hw_res->msr_update = mba_wrmsr_amd;
>>>>  		}
>>>>  	}
>>>>  }
>>> I mentioned earlier that this can be moved to init of
>>> rdt_resources_all[]. No strong preference, leaving here works
>>> also.
>> I am little confused about this comment. Initialization of
>> rdt_resources_all in core.c is mostly generic initialization. The msr_base
>> and msr_update routines here are vendor specific. I would prefer to keep
>> this in
> This is a contradiction. Yes, rdt_resources_all[] initialization in core.c
> is indeed generic initialization, so why is SMBA there? If this was really
> generic initialization then the entire initialization of SMBA resource
> should rather move to AMD specific code.
>
> SMBA is an AMD only feature yet its resource initialization is fragmented
> with one portion treated as generic and another portion treated as vendor
> specific while it all is vendor specific.
>
> The current fragmentation is not clear to me. Keeping the initialization
> as you have in patch #2 is the simplest and that is what prompted me
> to suggest the move to keep initialization together at that location.
>
>> rdt_init_res_defs_amd.Is that ok?
> The generic vs non-generic initialization argument is not convincing to me. 
> Could you please elaborate why you prefer it this way? I already mentioned
> that I do not have a strong preference but I would like to understand what
> the motivation for this split initialization is.
>
I dont have any strong argument. I was thinking, in case Intel supports
this resource in the future then they only have to change
rdt_init_res_defs_intel.

Thanks

Babu