Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [PATCH v2 18/23] x86/resctrl: Pass the required parameters into
 resctrl_arch_rmid_read()
To:     James Morse <james.morse@arm.com>, <x86@kernel.org>,
        <linux-kernel@vger.kernel.org>
CC:     Fenghua Yu <fenghua.yu@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        H Peter Anvin <hpa@zytor.com>,
        Babu Moger <Babu.Moger@amd.com>,
        <shameerali.kolothum.thodi@huawei.com>,
        Jamie Iles <jamie@nuviainc.com>,
        "D Scott Phillips OS" <scott@os.amperecomputing.com>,
        <lcherian@marvell.com>, <bobo.shaobowang@huawei.com>,
        <tan.shaopeng@fujitsu.com>
References: <20211001160302.31189-1-james.morse@arm.com>
 <20211001160302.31189-19-james.morse@arm.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
Message-ID: <0d3d0869-c303-2411-ab74-76f7e06f9721@intel.com>
Date:   Fri, 15 Oct 2021 15:30:09 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
 Firefox/78.0 Thunderbird/78.14.0
In-Reply-To: <20211001160302.31189-19-james.morse@arm.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RGxZNkx5Y2R1cHRBRHZuWHJsZTAzYllvSHJvbjNMLzZzQ1BFWkY1Nzh6T2Rx?=
 =?utf-8?B?eHNWc1c2NUwzeXY5YnNabUVxZWFOTjIxQWRwNk80VUF3YnYwaEFGWHk0MjR4?=
 =?utf-8?B?dDRUTzc2c1lENENIMU1oSWdrWEp2RHdCaEMzZ3MvYWh0MjJ3Y3A4cy90YTI4?=
 =?utf-8?B?RmVvd2ErODg3YzlWdk1tRW9xWU5UNzRMa3NadmNaTi9RY3FsM2tjYTdCbHRk?=
 =?utf-8?B?VGRleFJaNnZOaGVwejVvbGJ0QzMwYjRubXYwMklMNElNMGhLM2NId054S3RE?=
 =?utf-8?B?V3dySndsOFBIYTV1ZnE2L2pkL09uQ1VzNTYyWHg2eW9CQ3hvQ2c0UW81UXRU?=
 =?utf-8?B?NEt2K2taWVlGSEtjOGRoWDBtQVFSVG03L2JsNklxb1FZTTc4UndLY3U5NXNB?=
 =?utf-8?B?QTAyY0srWVppdUZYWHo4MkNRd0h6ckZqS3dzVkE5cStIVE9wUk1KclFieEVI?=
 =?utf-8?B?S3JPSkkyZkZBeVRUVlFzbXVjV0gyQzc1QlVtdThIQU9lOEphRzQrNGNkR2Rp?=
 =?utf-8?B?ZnVOOHd0TFExdGZ2QUcxeGIvVGJGLzJ0NHFEY1dUYzNwNXM5UjhEV1F6Qzlt?=
 =?utf-8?B?SXdCVk1YYUNoNjRrYU5mREd4eDkwa1VTdG5RMkY1TUJyUVJLSGFCOWpscCtz?=
 =?utf-8?B?T3RmVHY1cmg4TjU1bTAxb0V1SkVWMzJXRDVRSEZBSVI3R3RiWmJqUCtrOXgw?=
 =?utf-8?B?dUVuUmNYTEloWUttRHR0YUpIdzViRk1kM0F0VVBTb1NOUEQ1bGd4b3BEQmgy?=
 =?utf-8?B?MjJ4elVpOXdyaDNkcmpRUFp1Ui91SHRncDVlYVFMMHpQdGp4bXRBNkttcm4r?=
 =?utf-8?B?cTRoT1U5dk5DVU9saE9xaHhPVjZrNlhBZVl3YU9TbWZsbSt5elpDQzc2Skdl?=
 =?utf-8?B?bTkxZERMOS9ITjRxWDBPbVVSaGVpYmorODJZS1NMb2JrY0I0SkhHbCtLcktO?=
 =?utf-8?B?R0cyd3dTTjhKQjJYaU5HeEZSZkJ5bGJDMWNQRTlVbDJTbnU1RURLenJ0cTg2?=
 =?utf-8?B?Sk9kOEY4Qkw2NDBWdGVFNEdFYjFjbFBwUklTWldhbnh3SDYrUWVXRkhLcXpz?=
 =?utf-8?B?U083R0syUUNzTlFwRnJ0bHRmc2R0LzFuNUc0cjg5RHlzdXpMR3lwQ1VLazJu?=
 =?utf-8?B?UVZYYzFZQUtrN3ZPeHYrSlJ4NjRwY1JBUHZ2blFaUENRS1A2S3M3UzRWbHF2?=
 =?utf-8?B?RmRWbi91VlpyZ3k0SGl5ZTRqc1dBMGJGRWtiVEVKVW1BK3ZXc1NOR3BYeDdo?=
 =?utf-8?B?TXllRDluMW9GNTJ6WFZ4dmk2U04rZDl4RDQ2WCtpUVBSVjFCaTFmLzZhT3Jj?=
 =?utf-8?B?Z1FEdnVhK0RmOXpqb1pqdW1TZ1I5N1djVGVwVkorTW8xcWczUmxHOUtRRmlt?=
 =?utf-8?B?NVNNUWZUejd5eFBoMEpQWE03dHpSZFE1L25leHAwQytjYU5FRU9VZ2szMmdC?=
 =?utf-8?B?MjREV0xpcVM1MW1sanp4TXoxK0ZjbEc1eG4yYjhMZjN3V3VZeVJRdmhzcHA1?=
 =?utf-8?B?bWxLUU5icWpoSWhIWFh1MHZySDlFZTRsRHVlV0N6a0FrR1Z0T1lzUG1DOWpP?=
 =?utf-8?B?OVlBMlFkUU55WmV3K2VERkNyUFVIemhqUjk2UFZMREF4WTIyUlZ6bThyWWlu?=
 =?utf-8?B?THJIY3hhMEpJbmxwYkZxeWl1eGN2Z2w5aG44K2t6ZUdFNnExZkRsZTMxSk5V?=
 =?utf-8?B?cmFUb0lUVDRjd0wvVnNTdzN2TUE3K2hNZ0UvUW9FQ0lBWENobWR4S2pKRWIy?=
 =?utf-8?Q?M5JLx6rNb6lnpbXxwY5wk2jTgicdK3Ev9ASmNfb?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 9b8965b2-74d8-4c77-4159-08d9902b5b7d
X-MS-Exchange-CrossTenant-AuthSource: BN0PR11MB5744.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Oct 2021 22:30:14.4765
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: /R8lfR6x7EwJ1FKRP7JEpVevGtob+vhZRh0QIZ+OrkxVBZI2AG0YTCBL2ctBFAcO8zqjPf9IOnk4/m67CJuOgiV32xxPJF3m+2OVcm+OwxE=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR11MB1299
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi James,

On 10/1/2021 9:02 AM, James Morse wrote:
> resctrl_arch_rmid_read() is intended as the function that an
> architecture agnostic resctrl filesystem driver can use to
> read a value in bytes from a hardware register. Currently the function
> returns the mbm values in chunks directly from hardware.
> 
> To convert this to bytes, some correction and overflow calculations
> are needed. These depend on the resource and domain structures.
> Overflow detection requires the old chunks value. None of this
> is available to resctrl_arch_rmid_read(). MPAM requires the
> resource and domain structures to find the MMIO device that holds
> the registers.
> 
> Pass the resource and domain to resctrl_arch_rmid_read(). This make
> rmid_dirty() to big, instead merge it with its only caller, the name is

to big -> too big

> kept as a local variable.
> 
> Signed-off-by: James Morse <james.morse@arm.com>
> ---
> This is all a little noisy for __mon_event_count(), as the switch
> statement work is now before the resctrl_arch_rmid_read() call.
> ---
>   arch/x86/kernel/cpu/resctrl/monitor.c | 31 +++++++++++++++------------
>   include/linux/resctrl.h               | 15 ++++++++++++-
>   2 files changed, 31 insertions(+), 15 deletions(-)
> 
> diff --git a/arch/x86/kernel/cpu/resctrl/monitor.c b/arch/x86/kernel/cpu/resctrl/monitor.c
> index cf35eaf01042..f833bc01aeac 100644
> --- a/arch/x86/kernel/cpu/resctrl/monitor.c
> +++ b/arch/x86/kernel/cpu/resctrl/monitor.c
> @@ -167,10 +167,14 @@ void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_domain *d,
>   		memset(am, 0, sizeof(*am));
>   }
>   
> -int resctrl_arch_rmid_read(u32 rmid, enum resctrl_event_id eventid, u64 *val)
> +int resctrl_arch_rmid_read(struct rdt_resource	*r, struct rdt_domain *d,

Please do not use tabs in the function parameters.

> +			   u32 rmid, enum resctrl_event_id eventid, u64 *val)
>   {
>   	u64 msr_val;
>   
> +	if (!cpumask_test_cpu(smp_processor_id(), &d->cpu_mask))
> +		return -EINVAL;
> +
>   	/*
>   	 * As per the SDM, when IA32_QM_EVTSEL.EvtID (bits 7:0) is configured
>   	 * with a valid event code for supported resource type and the bits
> @@ -192,16 +196,6 @@ int resctrl_arch_rmid_read(u32 rmid, enum resctrl_event_id eventid, u64 *val)
>   	return 0;
>   }
>   
> -static bool rmid_dirty(struct rmid_entry *entry)
> -{
> -	u64 val = 0;
> -
> -	if (resctrl_arch_rmid_read(entry->rmid, QOS_L3_OCCUP_EVENT_ID, &val))
> -		return true;
> -
> -	return val >= resctrl_cqm_threshold;
> -}
> -
>   /*
>    * Check the RMIDs that are marked as busy for this domain. If the
>    * reported LLC occupancy is below the threshold clear the busy bit and
> @@ -213,6 +207,8 @@ void __check_limbo(struct rdt_domain *d, bool force_free)
>   	struct rmid_entry *entry;
>   	struct rdt_resource *r;
>   	u32 crmid = 1, nrmid;
> +	bool rmid_dirty;
> +	u64 val = 0;
>   
>   	r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;
>   
> @@ -228,7 +224,14 @@ void __check_limbo(struct rdt_domain *d, bool force_free)
>   			break;
>   
>   		entry = __rmid_entry(nrmid);
> -		if (force_free || !rmid_dirty(entry)) {
> +
> +		if (resctrl_arch_rmid_read(r, d, entry->rmid,
> +					   QOS_L3_OCCUP_EVENT_ID, &val))
> +			rmid_dirty = true;
> +		else
> +			rmid_dirty = (val >= resctrl_cqm_threshold);
> +
> +		if (force_free || !rmid_dirty) {
>   			clear_bit(entry->rmid, d->rmid_busy_llc);
>   			if (!--entry->busy) {
>   				rmid_limbo_count--;
> @@ -278,7 +281,7 @@ static void add_rmid_to_limbo(struct rmid_entry *entry)
>   	cpu = get_cpu();
>   	list_for_each_entry(d, &r->domains, list) {
>   		if (cpumask_test_cpu(cpu, &d->cpu_mask)) {
> -			err = resctrl_arch_rmid_read(entry->rmid,
> +			err = resctrl_arch_rmid_read(r, d, entry->rmid,
>   						     QOS_L3_OCCUP_EVENT_ID,
>   						     &val);
>   			if (err || val <= resctrl_cqm_threshold)
> @@ -336,7 +339,7 @@ static u64 __mon_event_count(u32 rmid, struct rmid_read *rr)
>   	if (rr->first)
>   		resctrl_arch_reset_rmid(rr->r, rr->d, rmid, rr->evtid);
>   
> -	rr->err = resctrl_arch_rmid_read(rmid, rr->evtid, &tval);
> +	rr->err = resctrl_arch_rmid_read(rr->r, rr->d, rmid, rr->evtid, &tval);
>   	if (rr->err)
>   		return rr->err;
>   
> diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
> index 01bdd8be590b..4215a0564206 100644
> --- a/include/linux/resctrl.h
> +++ b/include/linux/resctrl.h
> @@ -218,7 +218,20 @@ u32 resctrl_arch_get_config(struct rdt_resource *r, struct rdt_domain *d,
>   			    u32 closid, enum resctrl_conf_type type);
>   int resctrl_online_domain(struct rdt_resource *r, struct rdt_domain *d);
>   void resctrl_offline_domain(struct rdt_resource *r, struct rdt_domain *d);
> -int resctrl_arch_rmid_read(u32 rmid, enum resctrl_event_id eventid, u64 *res);
> +
> +/**
> + * resctrl_arch_rmid_read() - Read the eventid counter correpsonding to rmid

correpsonding -> corresponding

> + *			      for this resource and domain.
> + * @r:			The resource that the counter should be read from.
> + * @d:			The domain that the counter should be read from.
> + * @rmid:		The rmid of the counter to read.
> + * @eventid:		The eventid to read, e.g. L3 occupancy.
> + * @val:		The result of the counter read in chunks.
> + *

"The" prefix can be removed from all descriptions to match style of 
other descriptions, also in description of resctrl_arch_rmid_read() below.

> + * Returns 0 on success, or -EIO, -EINVAL etc on error.

I do not think this is valid kernel doc for the return section.

> + */
> +int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain *d,
> +			   u32 rmid, enum resctrl_event_id eventid, u64 *val);
>   
>   /**
>    * resctrl_arch_reset_rmid() - Reset any private state associated with rmid
> 

Reinette