Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [PATCH v1 05/20] x86/resctrl: Create mba_sc configuration in the
 rdt_domain
To:     James Morse <james.morse@arm.com>, <x86@kernel.org>,
        <linux-kernel@vger.kernel.org>
CC:     Fenghua Yu <fenghua.yu@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        H Peter Anvin <hpa@zytor.com>,
        Babu Moger <Babu.Moger@amd.com>,
        <shameerali.kolothum.thodi@huawei.com>,
        Jamie Iles <jamie@nuviainc.com>,
        "D Scott Phillips OS" <scott@os.amperecomputing.com>,
        <lcherian@marvell.com>, <bobo.shaobowang@huawei.com>
References: <20210729223610.29373-1-james.morse@arm.com>
 <20210729223610.29373-6-james.morse@arm.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
Message-ID: <638a0c73-ed1b-c4d6-f5f2-2af3c2e39a35@intel.com>
Date:   Wed, 1 Sep 2021 14:22:36 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
 Firefox/78.0 Thunderbird/78.13.0
In-Reply-To: <20210729223610.29373-6-james.morse@arm.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RlFaVlpwcXNQa25mTEFQTCtPZFNXQ3dqRzBuZ3pOTTRXT3BVcVcybUJyaFY5?=
 =?utf-8?B?QmMvU1JMbzY1RnRlU3J0MExvWk1PNDcxKzVuRFRmOGJQOFVaOWEvbXZHZzhw?=
 =?utf-8?B?a1VTejZwUGxFUm84dFBXaFBnOVpIajM0cmphc0phY3RSKzd4K0RzTEttNUt5?=
 =?utf-8?B?VmluUjdUdG01UVo0MkJxbzNwR0duQ2dIY2hFOHNBMHZIQUhzZEtBcWFOUGlR?=
 =?utf-8?B?UW5PaGJLeTlsWFpxTm5iMThPNjlDMjJkRG5NdFRHUlJhd1hpZ2R3aXUvSGZY?=
 =?utf-8?B?OFRkYkhLNVRKRFV0ekNBL20rWGV3dXRvN0xCK3Q0MzF0UTMzbHNNaURwVjRv?=
 =?utf-8?B?NXRzQ2xNcVMxOHIwNWtpODE1MTU1SHdWckFPdmRZN0RhelZRM240ZjErem1M?=
 =?utf-8?B?cDBsL3JEQ2Fram5VR2lmTEhMaitQeGhRL3hFK1FEWUx4WklJZnJaaUxjNTVJ?=
 =?utf-8?B?c0I1TnRzRVUxVzhDaEVJNGFRODF4YTQ3YmVJWGxLRXBleG5WVjBxSmo0ZlJ1?=
 =?utf-8?B?TjFrYVA1MWxSRExtZ2h6MXY2TVVwcS9UeHc5WXBJSENtcVVmMzNFdWdiRkY4?=
 =?utf-8?B?T1ZjcWdyaHpmVWphUHNPcGlERVh0MURVeE5WcmhscWRQUUVseWV1bG1OTjNZ?=
 =?utf-8?B?TzFxazVrMVZLeE8zSjBNcWRXNDByNXdobFVZOHNKREtoS2JxZWtDOXpJaHl6?=
 =?utf-8?B?TTR4TjVlUmhqKzI0WEQ4QS8zQ1BWUmhTenAwTG12S1d3dEs1WHNGbzFKMmM1?=
 =?utf-8?B?aVViZEVlVk40Ymg3MkR0clVOVHgwekVlcFdZOUd6alpYTHpiYnJOTytzN20x?=
 =?utf-8?B?M1pEeTNnSVJNT3N0ZTl5ZjROT3N5RFFDZzZRU3ZsL0ZHNHErOWhJY2FteTVu?=
 =?utf-8?B?QitoUW0rdHJ5ZU9JZi9UMWxMdWk4aE5kaUdZZlJjeW90cmpsQ1lNTkVKek1r?=
 =?utf-8?B?K1k4Q251NXV0Y2FwSmQ5N1FqUjlPYTV3YXJGWTNBamd3c0RjY1ZPaFdHK0tM?=
 =?utf-8?B?ckJnc2JXS251aUtvWXVzeDRxek5mL0ZaTlRxaEdOZ056cm1YUnFDeUt2bVR0?=
 =?utf-8?B?ZHk4Qmt1eXJuS2J6anZ5RnFReC9pTDNydnRnUWh0MVBlcjU1LzV4bjZ3d0tS?=
 =?utf-8?B?dmdhKzA0Z1F0cGFsK1Q1ejcvaVVaWGpYMng0YzJwSys4cEQzVEx2eHRJZ1cr?=
 =?utf-8?B?NmUzejJGMFFXWXJkZUJkUEo4YmVPOHBMbk8xd0FvWE5rNnI5ZVFIVDZkNHNC?=
 =?utf-8?B?eVdHdHA2NE1vazNHeWZnV0NOYkhiSzkzdDBNL3ljZ1pFR0VJSHZNaVhzKzh0?=
 =?utf-8?B?WUgwTnlBU0Y4dW1QZnVmMFdZcXlPcDYrKzZ0MHNETHo4L2pobkY0TkhaMTlr?=
 =?utf-8?B?amc0OUVvWHc3NzhJTEJvVXpVMklWRk9vS3F0RUZmMUlvNklpc3ByQ1NZNG9B?=
 =?utf-8?B?dzFaVHlES0xNSTM0MkxWR0dCZHBSKzVIbUp4VGtvWTFGWFBiK2FlWldPZ3Zj?=
 =?utf-8?B?ay9uNFI0ZDREWWFRNEsyaDNaYVZoaGxDR25ZdEsxQ0JrVDhpekJGOU41Q1Y3?=
 =?utf-8?B?UkpWdXdDdEdxdHgrVVJzWWxqQktkSzRoUnJjTEN5d3JCMTQyTnFjMFNESFow?=
 =?utf-8?B?L3JwOEZLWWtXSXh1TllBTHVURzRPL0t0d0VnUFVOS2JXV1RuYkhKQ1VjRyty?=
 =?utf-8?B?T0dpclptSUt6ZFNpNlcyZ1hOT1VCcDJoVVBPc3FxV1ZDRU15cmVGaUZ2dlBC?=
 =?utf-8?Q?ZsR027LVOxZL7F4AmIugeHwnXCyvgDzmr4VSUXt?=
X-MS-Exchange-CrossTenant-Network-Message-Id: a6f0343a-d8ba-4848-716f-08d96d8ea154
X-MS-Exchange-CrossTenant-AuthSource: BN0PR11MB5744.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Sep 2021 21:22:41.0230
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: G6dO+PuLRFGfVNkZ4bRpbwMFEdTywPGlo0JhZmFmrrGO6p815CWPYaEgSPbXNvp3JSuF8EhCPrSV9mfBzMf/MHiYwlYJRdoIcLSOH5dSls8=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR11MB5243
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi James,

On 7/29/2021 3:35 PM, James Morse wrote:
> To support resctrl's MBA software controller, the architecture must provide
> a second configuration array to hold the mbps_val from user-space.
> 
> This complicates the interface between the architecture code.

This sentence seems incomplete. I was expecting something like " 
complicates the interface between the architecture code and ..."

> 
> Make the filesystem parts of resctrl create an array for the mba_sc
> values when the struct resctrl_schema is created. The software controller
> can be changed to use this, allowing the architecture code to only
> consider the values configured in hardware.
> 
> Signed-off-by: James Morse <james.morse@arm.com>
> ---
>   arch/x86/kernel/cpu/resctrl/internal.h |  1 -
>   arch/x86/kernel/cpu/resctrl/rdtgroup.c | 69 ++++++++++++++++++++++++++
>   include/linux/resctrl.h                | 13 +++++
>   3 files changed, 82 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
> index e12b55f815bf..a7e2cbce29d5 100644
> --- a/arch/x86/kernel/cpu/resctrl/internal.h
> +++ b/arch/x86/kernel/cpu/resctrl/internal.h
> @@ -36,7 +36,6 @@
>   #define MBM_OVERFLOW_INTERVAL		1000
>   #define MAX_MBA_BW			100u
>   #define MBA_IS_LINEAR			0x4
> -#define MBA_MAX_MBPS			U32_MAX
>   #define MAX_MBA_BW_AMD			0x800
>   #define MBM_CNTR_WIDTH_OFFSET_AMD	20
>   
> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> index cf0db0b7a5d0..185f9bb992d1 100644
> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> @@ -2030,6 +2030,60 @@ static int mkdir_mondata_all(struct kernfs_node *parent_kn,
>   			     struct rdtgroup *prgrp,
>   			     struct kernfs_node **mon_data_kn);
>   
> +static int mba_sc_domain_allocate(struct rdt_resource *res,
> +				  struct rdt_domain *d)
> +{
> +	u32 num_closid = closid_free_map_len;
> +	int cpu = cpumask_any(&d->cpu_mask);
> +	int i;
> +
> +	d->mba_sc = kcalloc_node(num_closid, sizeof(*d->mba_sc),
> +				 GFP_KERNEL, cpu_to_node(cpu));
> +	if (!d->mba_sc)
> +		return -ENOMEM;
> +
> +	for (i = 0; i < num_closid; i++)
> +		d->mba_sc[i].mbps_val = MBA_MAX_MBPS;
> +
> +	return 0;
> +}
> +

I had the same initial reaction as Jamie and noted your answer to him. 
Considering the intricate flow here could you please add some comments 
to these functions that explains the calling flows in support of their 
safety?

...

>   /**
>    * struct rdt_domain - group of CPUs sharing a resctrl resource
>    * @list:		all instances of this resource
> @@ -53,6 +64,7 @@ struct resctrl_staged_config {
>    * @cqm_work_cpu:	worker CPU for CQM h/w counters
>    * @plr:		pseudo-locked region (if any) associated with domain
>    * @staged_config:	parsed configuration to be applied
> + * @mba_sc:	the mba software controller properties, indexed by closid
>    */
>   struct rdt_domain {
>   	struct list_head		list;
> @@ -67,6 +79,7 @@ struct rdt_domain {
>   	int				cqm_work_cpu;
>   	struct pseudo_lock_region	*plr;
>   	struct resctrl_staged_config	staged_config[CDP_NUM_TYPES];
> +	struct resctrl_mba_sc		*mba_sc;
>   };

Why is this additional abstraction needed? As I understand the usage 
struct resctrl_mba_sc would always only have the one member so why not 
have mbps_val within rdt_domain?

Reinette