Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1;
Message-ID: <bf9eced6-52ed-fc0f-ad72-73a4bde9f67c@amd.com>
Date:   Fri, 6 Oct 2023 09:24:05 -0500
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.5.0
Reply-To: babu.moger@amd.com
Subject: Re: [PATCH v4 2/4] x86/resctrl: Enable non-contiguous CBMs in Intel
 CAT
Content-Language: en-US
To:     Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com>,
        Fenghua Yu <fenghua.yu@intel.com>,
        Reinette Chatre <reinette.chatre@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>, x86@kernel.org,
        "H. Peter Anvin" <hpa@zytor.com>
Cc:     Peter Newman <peternewman@google.com>,
        =?UTF-8?Q?Ilpo_J=c3=a4rvinen?= <ilpo.jarvinen@linux.intel.com>,
        linux-kernel@vger.kernel.org
References: <cover.1696493034.git.maciej.wieczor-retman@intel.com>
 <bf80d39fe4faa58e3ab661fc2fce7bd1bd9285a5.1696493034.git.maciej.wieczor-retman@intel.com>
From:   "Moger, Babu" <bmoger@amd.com>
In-Reply-To: <bf80d39fe4faa58e3ab661fc2fce7bd1bd9285a5.1696493034.git.maciej.wieczor-retman@intel.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?OFY0bmZUMFdRR3JFOXoyaW9rNkxla2ViZXY1MnNnS0FubTlya3NJUnMwQ2pr?=
 =?utf-8?B?ejV0NEVOdU5KOFk2NU9YUmxURDdXSFI2U2lOV3JuTVd6RktlOGF5Q3BCdU50?=
 =?utf-8?B?NG5CdmxQWEUrWkY5L1AzOE1LNXpHeEw5MjhOZWRVd2lOVU5wUTFhNWpuWXEy?=
 =?utf-8?B?WVVoeHhJQ29oSkV4OHpxdEtzQllVR0NiVllzd2ZLRmxhRFBIZG9wV2hZVXhP?=
 =?utf-8?B?Mi93dVd5MkZ3Y2VSZjFjbC9laFdUeklGSXZJTFpTekM1YmtZdE9sNHVpWkdG?=
 =?utf-8?B?aVphTVBiai9CSDVibnJKZnZaQTBjaXIrZHFJa0hCZW9rZmZVdkpLTWhFT05r?=
 =?utf-8?B?Yi83WDYwTElMUjV4WUdkdzdYVVQwY2MxZHEzRjZLTGt5blQ2ZUVnYVJEeS9q?=
 =?utf-8?B?MStoTVV0enlzN3lRU2ZNMXdqYVRVNUc2aVdBL25NcnRkWTBFV2J0U0thcHlJ?=
 =?utf-8?B?OWI1VS9vM21wblFoS2dPbWcvNWJtckNDTlk3ai9Ud1c1aXE5UCtDS0d0T0Nr?=
 =?utf-8?B?YXNUMWJoVVdSWWFUK0Y2RnZ1M0ljamRhOG1vMllNVHM1cTk4T2JEVCtFVDJI?=
 =?utf-8?B?eDhic0UvVWdONDlRcUtRQThhMnFka214eUpNNHdmNnkxYzlBYXBUemdyWDBr?=
 =?utf-8?B?SDgwU2dJQ05pdW0ydlhVaWlQT1hwRVhqZjdJS2dBSE9EOWdyUjlOOGVtcUZq?=
 =?utf-8?B?QVplSHVsaVZ2ZThwSjRlclVINFBqdElmYjFnYTR4VHFlVW42S0RWb3pBRGVT?=
 =?utf-8?B?ckRCU0FRY1ZFYVVxQmZ2NUVHWHFmVm1DQ0hJcFRGZEVySzAyNTV5Y2dZVlBB?=
 =?utf-8?B?TU1jM0l4aUJzSkFyY1lveGJxSVAzY2lVT0RnOWlyVVhkc0Y4ZDhFdmRUalI2?=
 =?utf-8?B?Yno2NUE5ZE1wdGtub1oyNzZYSTN6aUZubExLV3VjanZPY2s4UHIwcWY4b0tu?=
 =?utf-8?B?bnJSVUh5YVFkUnlsc0hMbFlLWXE0OTUwVENhTFQzdHhPYXFla2NKRzVtMGRD?=
 =?utf-8?B?dFFWU2RZZzVFei9sN0p4WTlFeHRkZkxvVCthRGh1UHFWb0pZWkZDOENWRHls?=
 =?utf-8?B?UXRKWGhjM21IMnBWRjRDTFo2Y0lhTEo5WW5LYmIvRXdlNnhUeEJIY2JaVHM5?=
 =?utf-8?B?ZFV4dmM1Skk4cDh5QXRQQkpTMTl0NWxYSHFVK2g1R2JOWSt5a2FpNWFSdDFz?=
 =?utf-8?B?R0Z6RjBTcHJXZlc5bVFvdEpxMmpvUHdUUnhjdmgxZG9YZk1WNUwrampxSW5T?=
 =?utf-8?B?TE43YjZ2Q2MzTjZDM2NxSEVrMk9Na0cwYTg3UUtOQU1SMHp5Y3pnZW85Q3Fm?=
 =?utf-8?B?ME42dkxvU3hvVTVmOTY1cnJaUitWNXpRcGZLV09veEU2VUFrNWtXNU51Wk1C?=
 =?utf-8?B?d3E3Mi8vSnplSElxRzNUZHZqZGxhaVpuWms0Nk5xNjdOQU5tRG5CUWx1WHhr?=
 =?utf-8?B?UU1Ec1FKUjNma0NUSDZvRDBERnRhRmFUWmo4L0g1N0VWbG1zTVV4cGx2b3Vh?=
 =?utf-8?B?YU13S0VuVVJkbWdUVjdMc1dXNVJRWlhhbTNMc3FvWHlnWWhUaXI2dlQ3ZjJ4?=
 =?utf-8?B?QllaMnMrOHZFYk9ic2hyUm1rUWNRRzNiRWE5WWw0Q0dYRHBacExFZm4zbFk4?=
 =?utf-8?B?cUZyVW12NmIyandnWGNxY2dHZU5HUDFwS3VzQWFZTk5uTXBPbWNIbDlVVys4?=
 =?utf-8?B?SCtKTTl1aWZIeGpiUHVIM1NqVXVob2ZVbVpxM0ZKeG9iVm11RVdtQmd4UVVH?=
 =?utf-8?B?YUwxckdsQ3drNW9VNTZ6WFhOT0V4QSttMU5MVFd0R1ljVm5SR1VjVkZsb1o2?=
 =?utf-8?B?Z3pHeEQyeGFBc0I5OHk0OTBBRFlOUWpsRFJDaUJIYWhVOEFSSDVXNUlsWmpD?=
 =?utf-8?B?V1MyZGlMSHMxQW1FT0JpQ2RYQ2pJeVdiSmJQYm1GakxmUlVuT1dDcW5oWUdz?=
 =?utf-8?B?dWdzdGVsbk9FUUE1RTRXSFZUNGtWVXk0R2ZNM2JWd1VYem4wdUhGSzRSV3Qx?=
 =?utf-8?B?VEQ2NHBmYWFhdzZFcEd6bkJ2T1lQMzAxU0ZEdXJZcFBjMHVoV3hEM0F0UjRs?=
 =?utf-8?B?Wmh5d2w1WGgwNUtZTlVLbG02L1VrVk1SajN2eGs5WnhWQ2Q3b2ZkVGVydGgx?=
 =?utf-8?Q?ixEjvofKfQclf+Y4y4a+ZCXLf?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 786b3374-7cfa-4888-6a23-08dbc677e82f
X-MS-Exchange-CrossTenant-AuthSource: MW3PR12MB4553.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Oct 2023 14:24:10.3988
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: mkrcBspmnVsvR3wU7WxrMy/M2BR6G0jAf5J9+yqHYH/h6oI/2dDilGHboRa69B+l
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8904
X-Spam-Status: No, score=-0.1 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Fri, 06 Oct 2023 07:24:24 -0700 (PDT)


On 10/5/2023 3:15 AM, Maciej Wieczor-Retman wrote:
> The setting for non-contiguous 1s support in Intel CAT is
> hardcoded to false. On these systems, writing non-contiguous
> 1s into the schemata file will fail before resctrl passes
> the value to the hardware.
>
> In Intel CAT CPUID.0x10.1:ECX[3] and CPUID.0x10.2:ECX[3] stopped
> being reserved and now carry information about non-contiguous 1s
> value support for L3 and L2 cache respectively. The CAT
> capacity bitmask (CBM) supports a non-contiguous 1s value if
> the bit is set.
>
> Replace the hardcoded non-contiguous support value with
> the support learned from the hardware. Add hardcoded non-contiguous
> support value to Haswell probe since it can't make use of CPUID for
> Cache allocation.
>
> Originally-by: Fenghua Yu <fenghua.yu@intel.com>
> Signed-off-by: Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com>
> Tested-by: Peter Newman <peternewman@google.com>
> Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
> Reviewed-by: Peter Newman <peternewman@google.com>
> Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Reviewed-by: Babu Moger <babu.moger@amd.com>
> ---
> Changelog v4:
> - Add Ilpo's reviewed-by tag.
> - Add Reinette's reviewed-by tag.
>
> Changelog v3:
> - Add Peter's tested-by and reviewed-by tags.
> - Change patch subject to mention CBMs. (Babu)
>
> Changelog v2:
> - Rewrite part of a comment concerning Haswell. (Reinette)
>
>   arch/x86/kernel/cpu/resctrl/core.c        |  9 ++++++---
>   arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 10 ++++++----
>   arch/x86/kernel/cpu/resctrl/internal.h    |  9 +++++++++
>   3 files changed, 21 insertions(+), 7 deletions(-)
>
> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
> index c09e4fdded3c..19e0681f0435 100644
> --- a/arch/x86/kernel/cpu/resctrl/core.c
> +++ b/arch/x86/kernel/cpu/resctrl/core.c
> @@ -152,6 +152,7 @@ static inline void cache_alloc_hsw_probe(void)
>   	r->cache.cbm_len = 20;
>   	r->cache.shareable_bits = 0xc0000;
>   	r->cache.min_cbm_bits = 2;
> +	r->cache.arch_has_sparse_bitmasks = false;
>   	r->alloc_capable = true;
>   
>   	rdt_alloc_capable = true;
> @@ -267,15 +268,18 @@ static void rdt_get_cache_alloc_cfg(int idx, struct rdt_resource *r)
>   {
>   	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
>   	union cpuid_0x10_1_eax eax;
> +	union cpuid_0x10_x_ecx ecx;
>   	union cpuid_0x10_x_edx edx;
> -	u32 ebx, ecx;
> +	u32 ebx;
>   
> -	cpuid_count(0x00000010, idx, &eax.full, &ebx, &ecx, &edx.full);
> +	cpuid_count(0x00000010, idx, &eax.full, &ebx, &ecx.full, &edx.full);
>   	hw_res->num_closid = edx.split.cos_max + 1;
>   	r->cache.cbm_len = eax.split.cbm_len + 1;
>   	r->default_ctrl = BIT_MASK(eax.split.cbm_len + 1) - 1;
>   	r->cache.shareable_bits = ebx & r->default_ctrl;
>   	r->data_width = (r->cache.cbm_len + 3) / 4;
> +	if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL)
> +		r->cache.arch_has_sparse_bitmasks = ecx.split.noncont;
>   	r->alloc_capable = true;
>   }
>   
> @@ -872,7 +876,6 @@ static __init void rdt_init_res_defs_intel(void)
>   
>   		if (r->rid == RDT_RESOURCE_L3 ||
>   		    r->rid == RDT_RESOURCE_L2) {
> -			r->cache.arch_has_sparse_bitmasks = false;
>   			r->cache.arch_has_per_cpu_cfg = false;
>   			r->cache.min_cbm_bits = 1;
>   		} else if (r->rid == RDT_RESOURCE_MBA) {
> diff --git a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
> index ab45012288bb..beccb0e87ba7 100644
> --- a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
> +++ b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
> @@ -87,10 +87,12 @@ int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s,
>   
>   /*
>    * Check whether a cache bit mask is valid.
> - * For Intel the SDM says:
> - *	Please note that all (and only) contiguous '1' combinations
> - *	are allowed (e.g. FFFFH, 0FF0H, 003CH, etc.).
> - * Additionally Haswell requires at least two bits set.
> + * On Intel CPUs, non-contiguous 1s value support is indicated by CPUID:
> + *   - CPUID.0x10.1:ECX[3]: L3 non-contiguous 1s value supported if 1
> + *   - CPUID.0x10.2:ECX[3]: L2 non-contiguous 1s value supported if 1
> + *
> + * Haswell does not support a non-contiguous 1s value and additionally
> + * requires at least two bits set.
>    * AMD allows non-contiguous bitmasks.
>    */
>   static bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
> diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
> index 85ceaf9a31ac..c47ef2f13e8e 100644
> --- a/arch/x86/kernel/cpu/resctrl/internal.h
> +++ b/arch/x86/kernel/cpu/resctrl/internal.h
> @@ -492,6 +492,15 @@ union cpuid_0x10_3_eax {
>   	unsigned int full;
>   };
>   
> +/* CPUID.(EAX=10H, ECX=ResID).ECX */
> +union cpuid_0x10_x_ecx {
> +	struct {
> +		unsigned int reserved:3;
> +		unsigned int noncont:1;
> +	} split;
> +	unsigned int full;
> +};
> +
>   /* CPUID.(EAX=10H, ECX=ResID).EDX */
>   union cpuid_0x10_x_edx {
>   	struct {